Security and Trust: The Bottleneck of Deploying Agentic AI

Three categories of failures occur across deployed agent systems.

Insufficient guardrails

Agents with broad permissions exceed their intended scope. A Replit agent deleted an entire production database, then fabricated 4,000 fake records to cover the loss; the operator only noticed when query results stopped making sense. Amazon's Kiro coding agent reportedly deleted and recreated a live AWS environment, causing a 13-hour outage after inheriting an engineer's elevated permissions and bypassing the standard two-person approval, though Amazon disputes that the agent caused the outage. OpenClaw deleted 200+ emails from a Meta security researcher's inbox despite an explicit instruction to confirm before acting; context compaction had silently dropped the safety constraint, and she had to force-kill all processes on the host to stop it.

Cursor's Plan Mode, which was explicitly designed to prevent execution, deleted 70 files after acknowledging a "DO NOT RUN" instruction. All of these failures stemmed from within the trust boundary: operator constraints were present at the policy level, but the action layer ignored them at runtime.

Prompt injection

A developer's Cursor agent was connected to Supabase via the Model Context Protocol (MCP, the dominant standard for connecting agents to external services) with privileged database access. When it processed a support ticket containing injected SQL instructions, the agent exfiltrated integration tokens into a public thread. Simon Willison called this the "lethal trifecta": access to private data, exposure to untrusted input, and the ability to act on the outside world. Palo Alto's Unit 42 detected the first real-world prompt injection against a production AI system, where attackers embedded hidden instructions in web content to bypass automated ad review. The right mental model is dynamic code loading from an untrusted source. The tool ecosystem opens a second injection channel: MCP servers can be poisoned or swapped out mid-execution, and malicious tool implementations enable data exfiltration and privilege escalation. The largest empirical study to date, an indirect prompt injection competition with 464 participants and 271,000 attempts across 13 models, found per-attempt success rates of 0.5–8.5%, and identified universal attack strategies that transferred across 21 of 41 tested behaviors. Static defenses fail under adaptive pressure: one evaluation drove every one of the eight defenses it tested to attack success rates above 50%.

Prompt injection now persists through memory. Environment-injected memory poisoning shows that a single contaminated observation can persist across sessions and across sites, bypassing permission-based defenses. Memory control-flow attacks show that retrieved memory can dominate the agent's control flow on GPT-5 mini, Claude Sonnet 4.5, and Gemini 2.5 Flash, forcing unintended tool use against explicit user instructions. ChatGPT Atlas was found to be vulnerable to a cross-site request forgery (CSRF) attack chained with memory poisoning, in which tainted memory triggers remote code execution on subsequent invocation.

Silent policy violations

Agents complete tasks correctly while violating legal or internal policy constraints, with no indication to the operator that anything went wrong. Air Canada's chatbot fabricated a bereavement fare refund policy that did not exist, and a tribunal held the airline liable, rejecting the defense that the chatbot was a "separate legal entity." The agentic version is worse: in SHADE-Arena, agents with unmonitored chain-of-thought scratchpads complete hidden side objectives while evading external monitors.

These failures compound. Attack success rates increase 16% on average as interactions extend across turns. Long-horizon agents exhibit substantial deficiencies in risk assessment during planning, and multi-server MCP workflows further increase safety risk. OpenAI publicly acknowledged that prompt injection "is unlikely ever to be fully solved."

One structural problem lies beneath both accidental and adversarial failures: useful and dangerous capabilities occupy the same action space. An agent authorized to route payments looks identical at runtime to one that has been compromised to route an unauthorized transfer. "Send this reply" is legitimate when it originates from the user and adversarial when it originates from an injected instruction in an email body. The same action can be safe or unsafe depending on provenance and context, so static capability restrictions break down. The missing piece is runtime taint tracking: a system that labels where every piece of data came from and what risk it carries, then propagates those labels through the agent's data flow so every output retains the provenance of the inputs that produced it. Policy checks on scope and authorization are then run at every action.

This creates a dilemma for operators. Granting broad permissions accepts high-impact, irreversible failures. Restricting permissions reduces the agent to a supervised assistant. Neither position holds up as deployments scale.

The obvious response is to instruct the model to refuse dangerous actions. Measurement shows this approach fails. The model can articulate the rule in the abstract; acting on it consistently inside its own trajectories is the harder problem. Frontier models identify sudo rm -rf /* as dangerous in over 98% of isolated prompts, yet avoid the call in fewer than 26% of their own trajectories. Across six frontier models, one study found 219 cases in which the agent's text refused while its tool call executed the same forbidden action. On real bash, code, filesystem, and browser tools, OpenAgentSafety measures unsafe behavior on 51% (Claude-3.7-Sonnet) to 73% (o3-mini) of safety-vulnerable tasks. Safety enforcement must reside outside the LLM substrate. Reasoning training and longer context leave the gap open: on tool fabrication, the failure rate scales with reasoning capability, and on under-pressure compliance, some newer generations actually regress.

There are six categories of approaches, ranging from production-deployed to research-stage.

I/O filtering

The most deployed commercial category. Classifiers screen inputs and outputs for harmful content, data leakage, and policy violations. AWS Bedrock Guardrails, Azure AI Content Safety, Lakera (acquired by Check Point for ~$300M), and Prompt Security (acquired by SentinelOne for a reported $180–250M) all provide real-time classifiers for prompt injection and data leakage. These are table stakes, but they catch only patterns detectable at the request boundary.

Declarative runtime rules

Let developers define policy rules and enforcement hooks. NeMo Guardrails provides programmable rails for dialogue constraints. The OpenAI Agents SDK offers input/output guardrails with tripwire enforcement; an April 2026 update added multi-provider sandbox compatibility for controlled execution. The most credible big-vendor entrant is the Microsoft Agent Governance Toolkit (released MIT-licensed on April 2, 2026): a seven-package system spanning Python, TypeScript, Rust, Go, and .NET, with a stateless policy engine that reports sub-millisecond enforcement latency even at the slow tail and explicitly maps to all ten items in the OWASP Agentic Top 10 (the industry's standard agent-security checklist). Apono Agent Privilege Guard extends the same pattern to runtime privilege guardrails. Rule quality remains the binding constraint on coverage, and these substrates are now production-grade.

Sidecar monitoring

Puts external monitors on agent trajectories, enforcing constraints at the session level. SHIELDAGENT reports 90.1% rule recall with lower latency than rule-traversal baselines. ToolShield reduces multi-turn attack success rates by ~30% by screening tool interactions across full sessions. On the commercial side, Zenity provides step-level runtime monitoring for enterprise copilots, and Noma Security maps agent-to-agent connections and blast radius in production. LLM-judge sidecars cap out around 74% F1 on the independent agent risk-awareness benchmark R-Judge, well below the bar for irreversible-action gating. They function as a complement to deterministic enforcement.

Formal verification and red teaming

Encode policies in logic and check them against action sequences. AgentSpec achieves full compliance on tested autonomous-driving law scenarios. Agent-C uses SMT solving and constrained generation to enforce temporal constraints, lifting Claude Sonnet conformance from 77.4% to 100% while also improving task utility. Promptfoo (acquired by OpenAI) provides automated security evals for agentic workflows, and Virtue AI's AgentSuite tests multi-step attack trajectories across sandbox environments.

Tool governance

Applies capability, confidentiality, and trust labels to tools as a policy substrate for blocking dangerous access, enforced at the protocol layer before any individual call. Cloudflare's MCP Server Portals route MCP traffic through zero-trust enforcement with audit logging. Pomerium extends its access proxy to MCP. Cisco AI Defense provides an MCP Catalog for server inventory and risk management. Kong AI Gateway extended its MCP module to A2A traffic in April 2026, applying the same gateway pattern to agent-to-agent protocols. These thin proxies are the first enforcement points at the protocol layer, with full policy engines still to come.

Agent identity and reputation

Govern how agents acquire and present credentials to act on behalf of users, and how their trustworthiness is established across boundaries. Keycard issues identity-bound, task-scoped cryptographic tokens. Astrix provides just-in-time least-privilege credentials. Descope maps the delegation chain from human to agent to tool. For agents that must transact across organizational boundaries without a pre-existing relationship, ERC-8004 proposes on-chain identity, reputation, and validation registries so an agent can be discovered and vetted by portable reputation rather than by a shared employer. The OWASP Top 10 for Agentic Applications ranks identity and privilege abuse among the highest risks, and NIST launched an AI Agent Standards Initiative focused on agent identity, authorization, and access delegation.

Each of the categories above targets one slice of the failure surface, and each requires the others. The ideal production-grade agent security system comprises four layers.

Layer 1: Observation and interpretation

Record a canonical agent action trace (model I/O, memory reads/writes, tool calls/results, prompt assembly, escalation events). Taint-tracking tags data by sensitivity and origin, and classifiers flag risk signals such as exfiltration likelihood, payment intent, and credential use. In practice, these capabilities are already converging into single products: Arize and LangSmith capture multi-step traces, while Lakera and HiddenLayer classify threats in the same pipeline. This layer is critical as it enables every downstream layer to act on stateful multi-step traces. Poisoned retrieval memory that corrupts agent behavior without ever triggering a tool control is a Layer 1 failure. The same is true for memory control-flow attacks and environment-injected memory poisoning. Both depend on the absence of provenance metadata at the memory-write boundary. Everything so far reads the agent's observable surface. A newer and orthogonal signal reads the model's internals: a linear probe on activation deltas detects task drift the moment injected content diverts the agent, even on unseen jailbreaks, and other probes flag strategic deception such as concealed insider trading. In principle this is where the knowledge-action gap becomes partially recoverable: if the model represents sudo rm -rf /* as dangerous even while running it, a combination of probes could surface that representation rather than trust the model to act on it. However, the risk a probe reads is rarely a single clean direction but a composite that takes many probes to recover; and latent-space monitors are adversarially evadable, with one attack driving a harmfulness probe's recall from 100% to 0% and the evasion itself learnable end-to-end. There is a research gap here on making reliable and general probes of model internals.

Layer 2: Enforcement

A non-LLM checkpoint sits between the agent and every sensitive action, deciding whether to allow, deny, transform, or escalate. The LLM proposes; this layer disposes. The knowledge-action gap measured earlier is why that decision cannot be captured within the model. Enforcement operates on two axes: (1) data-flow control via taint analysis, which propagates sensitivity labels through the agent's computation and asks whether tainted data reached a forbidden sink without authorized declassification, and (2) effect control, which asks whether the agent is authorized to make a particular state change at a particular scope. Prompt assembly itself presents a control surface within this layer. Mixing untrusted content with trusted instructions at assembly time enables privilege escalation that is invisible to per-call filtering. Context accumulation across tool-return boundaries enables multi-turn takeover that per-call validation cannot detect.

Layer 3: Evaluation and assurance

Per-step enforcement is necessary but structurally incomplete because individually authorized actions can combine to form unauthorized trajectories. An agent that reads a sensitive record, reformulates it across context boundaries, and surfaces it in an outbound message has exfiltrated data through a chain of individually benign steps (a classic taint-propagation failure where the sensitivity label was lost across transformations). Each step passes the Layer 2 checkpoint, but the sequence violates policy. The 16% multi-turn increase in attack success cited earlier extends exactly this compositionality gap. The compositionality problem now extends across sessions: environment-injected memory poisoning demonstrates that a single contaminated observation, written to memory in session N, can drive the agent to attacker-controlled outcomes in session N+k against an entirely different site. Permission-based defenses miss this because every individual permission stays within policy. Layer 3 verifies enforcement at the trajectory and lifecycle level; SHIELDAGENT's trajectory monitoring and Agent-C's temporal-constraint solving, both introduced above, operate here. Continuous red teaming is moving into production (Promptfoo, Virtue AI), and the 464-participant IPI competition gives the field its first large-scale empirical floor for adaptive-attacker assumptions. Interpretability extends assurance below the behavioral surface: in Anthropic's blind auditing game, three of four teams recovered a model's deliberately hidden objective using sparse autoencoders alongside behavioral and training-data analysis. The operational caveat is a tool-to-agent gap: interpretability tools that score well in isolation often fail to lift an auditing agent's hit rate, and inside that agent, black-box probing currently outperforms white-box interpretation. Chain-of-thought monitoring reads the reasoning trace for intent to misbehave, a check that a multi-lab group of over forty researchers argued was worth preserving, but the signal is fragile: training against the monitor teaches the model to hide its intent. Even so, full-surface coverage under adaptive attack remains an open research problem. No production system today runs trajectory-level taint analysis or enforces temporal policy across a multi-turn agent lifecycle, let alone across sessions.

Layer 4: Governance

Policy authoring, tool registration, approval flows, and audit logging. Governance starts upstream: the tool-labeling substrate described in the previous section blocks dangerous access before the agent makes its first call. The Microsoft Agent Governance Toolkit is the first major vendor reference substrate to explicitly map governance packages to the OWASP Agentic Top 10. It is a useful test for our four-layer thesis: any claim about what production lacks should map to a specific AGT package or a specific hole in the toolkit.

A pattern emerged from this survey: every production system we examined makes local and per-step decisions. In contrast, the highest-risk failures are at the sequence level. Research points toward the next layer (temporal constraints, trajectory-aware monitoring, policy-aware tool governance), none of which is yet a production default.

Over $2 billion in AI security acquisitions during 2025–2026 were mostly concentrated in I/O classification and prompt injection detection: Protect AI by Palo Alto Networks, Prompt Security by SentinelOne, CalypsoAI by F5, and Promptfoo by OpenAI, among others. This amounts to perimeter security. Taint tracking, provenance-mediated enforcement, and effect-type differentiation exist only in research prototypes. The one system that applies dynamic taint analysis to agent data flows is the research prototype FIDES, which assigns confidentiality and integrity labels to all data and uses taint propagation to block actions when the data lineage violates the policy lattice. An emerging agent-identity category (including Astrix, Descope, ConductorOne, and Aembit) addresses authentication. The deeper problem of tracking what data influenced which action remains open.

Agents are already deployed in high-consequence domains with largely heuristic guardrails. A federal audit of 17 critical-infrastructure AI risk assessments found gaps across all 17 in how harm magnitude and probability were combined. The primary U.S. AI risk management framework broadly covers generative AI, leaving agentic-specific risks outside its scope. The MCP protocol, now the de facto standard for agent-to-tool communication with over 10,000 forks of the official server repository alone and over 67,000 server implementations indexed across public registries, leaves audit, sandboxing, and verification optional. Security researchers filed over 30 CVEs targeting MCP servers, clients, and infrastructure in the first 60 days of 2026 alone, including a CVSS 9.6 remote code execution in a package with 437,000 downloads. A community vulnerability database now tracks 50 MCP vulnerabilities, 13 of them critical. Among 2,614 MCP implementations surveyed, 82% had file operations vulnerable to path traversal, 67% had code injection risk, and 34% were susceptible to command injection. The 26% vulnerability rate across deployed agent skills comes from a survey of production marketplaces, not a lab simulation.

Agent-builder frameworks are themselves now exploitation surfaces. A CVSS 10.0 unauthenticated remote code execution vulnerability in Flowise's CustomMCP node was being actively exploited by April 2026 across more than 12,000 exposed instances. A multi-vector vulnerability chain in CrewAI combines RCE, SSRF, arbitrary file read, and sandbox escape: the Code Interpreter falls back to an in-process SandboxPython implementation when Docker is unavailable, and a proof-of-concept reaches it via prompt injection. Both of these live in the same toolchain that the production agents above depend on.

These security gaps are also openings. For product builders, the agentic products that actually get deployed will be the ones customers trust with real autonomy. Our advice to founders: pick a wedge, then ship the outcome.

Start with a vertical wedge

Pick a workflow where agent adoption is already underway, security is the gating factor, and a better product can lead to net-new deployments. While software engineering has successfully adopted agentic workflows, most of the other knowledge-work industries have not. The wedge can be back-office automation, email triage, calendar management, internal research, memo drafting, and customer support. In each of these, the value of autonomy is clear, yet prompt injection, unauthorized actions, and data leakage still impede broader rollout. Build the critical components required to ship the end-to-end outcome: provenance tracking along the data path, deterministic gates for irreversible actions, and trajectory-level auditing.

Ship the end outcome to customers

Rather than building fundamentally human-facing products that are intermediate to customers' end outcomes, agents need to assist customers in achieving these end outcomes directly. Questions to ask: How can an agent directly influence the KPIs of an organization? What system does the agent need access to? What human-in-the-loop processes do these agents need to have in the short term and long term? Can the technologies mentioned in this article be leveraged to remove these human-in-the-loop checkpoints?

Build toward network effects

Finally, there are potential paths toward network effects once an initial wedge has been established. Identity and reputation for agents and tools begin to compound as firms communicate and transact with one another, both within a use-case vertical and across verticals. The adoption path may look more like PDF (de facto standards emerging from adoption) than HTTPS (standards pushed onto mature products with distribution).

If you are building in this category, don't hesitate to reach out and chat with us.