AI Agent Security: How We Protect Oliver from Prompt Injection
AI agent security was the first thing I wrote on the whiteboard when we started building Oliver — before we wrote a single line of code. Not because I'm paranoid by nature, but because Oliver has real write access to production infrastructure. He can restart servers, push deployments, rotate credentials, and scale services up or down. That's not a chatbot. That's a principal in your system with blast radius comparable to a senior DevOps engineer who never sleeps and always says yes.
Then GitLost dropped this week — Noma Security's researchers tricked GitHub's AI agent into leaking private repositories through prompt injection. The post landed on Hacker News with 180 points and immediately validated every architectural decision we'd agonized over. It also exposed a few things I want to be direct about: there are two fundamentally different philosophies for securing AI agents, and most teams are building the wrong one.
Here's the comparison that actually matters: naive trust models vs. adversarial trust models. One assumes the LLM is a controlled tool. The other assumes the LLM is an attack surface. I'm going to break down both, explain what we got wrong when we built Oliver the first time, and tell you exactly which mitigations work.
The Two Philosophies of LLM Security
Naive Trust: Treating the LLM as a Controlled Tool
The naive trust model is where most teams start, including us. The mental model is: I control the system prompt, I control the tools, I control the input pipeline — therefore I control the agent. Security is a matter of access control at the tool level. If the tool only exposes safe operations, the agent can't do unsafe things.
This is GitHub's implicit model, and GitLost blew a hole through it. The attack worked because the agent processed content from untrusted sources — repository content, issue text, PR descriptions — and that content contained instructions that the LLM interpreted as legitimate directives. The model couldn't reliably distinguish between "instructions from my operator" and "instructions embedded in data I'm analyzing." Private repo contents exfiltrated through a crafted response pipeline.
The naive trust model has three core failure modes:
1. Data-plane / control-plane conflation. The LLM processes both your system prompt and arbitrary external content in the same attention context. There is no hardware boundary. There is no kernel ring separation. It's all tokens.
2. Tool-level access control is insufficient. If the agent has a "read file" tool and an "send HTTP request" tool, an attacker doesn't need to compromise your infrastructure — they need to trick the agent into chaining those two tools. The tools are individually safe. The composition is not.
3. Confidence in refusal. Teams test their system prompt with friendly inputs and declare it secure when the model refuses obvious attacks. This is like testing your SQL injection defenses by typing ' OR 1=1 into the search box yourself. Real adversarial inputs are obfuscated, context-dependent, and exploit the model's instruction-following behavior at a semantic level that no static rule catches.
Adversarial Trust: Treating the LLM as an Attack Surface
The adversarial trust model starts from a different axiom: the LLM will eventually be compromised by a sufficiently crafted input. Not maybe. Not in edge cases. Eventually, given enough surface area, the model will do something it shouldn't. Your job is to ensure that when it does, the consequences are bounded, auditable, and recoverable.
This is the model we built for Oliver after our first architecture failed a red-team exercise we ran internally. We had a moment where Oliver, given a crafted log file that contained embedded instructions, attempted to execute a shell command we hadn't anticipated. It didn't succeed — the tool call failed at the permission layer — but the fact that it tried was enough to tear down our entire first design.
The adversarial model has four pillars:
1. Minimal capability surface. Oliver doesn't have a generic "run command" tool. He has specific, named operations: restart_service(service_name), scale_replicas(service, count), tail_logs(service, lines). Each one is a function with typed parameters, server-side validation, and its own audit log entry. The LLM never touches a shell.
2. Structural separation of untrusted content. When Oliver ingests external data — log output, error messages, user-submitted tickets — that content is never concatenated directly into the instruction context. It goes through a separate summarization step first, with its own prompt that explicitly marks the content as untrusted data and instructs the model to extract structured facts only, not follow any embedded instructions.
3. Human-in-the-loop for irreversible actions. Reversible operations (read logs, check service status, list deployments) are autonomous. Irreversible or high-impact operations (push a deployment, restart a database, modify a firewall rule) require explicit human approval through a Slack confirmation step with a 60-second timeout. Oliver proposes; a human disposes.
4. Behavioral anomaly detection. We log every tool call Oliver makes, every parameter he passes, and the reasoning chain he produces when we have it. We built a lightweight classifier that flags unusual tool call sequences — for example, if Oliver calls tail_logs and then immediately attempts scale_replicas on a different service than the one he was asked about, that's a pattern worth alerting on.
Direct Comparison: Where Each Model Breaks Down
| Dimension | Naive Trust | Adversarial Trust |
|---|---|---|
| Threat model | External attackers | External attackers + the LLM itself |
| Primary defense | System prompt + tool access control | Structural isolation + behavioral bounds |
| Handles prompt injection? | No — data/control conflation | Partially — reduces surface, doesn't eliminate |
| Blast radius on compromise | High — full tool access | Low — bounded operations, approval gates |
| Operational complexity | Low | Medium-High |
| Auditability | Depends on logging setup | Built-in by design |
| Recovery from failure | Reactive | Proactive — failures are anticipated |
The naive model is faster to build and works fine for read-only agents or agents with trivially safe tools. If your AI agent is answering FAQ questions from a static knowledge base, you don't need Oliver's threat model. The adversarial model costs you real engineering time and ongoing operational overhead.
But if your agent touches production — if it can write, modify, deploy, or exfiltrate anything — the naive model isn't a starting point you'll harden later. It's a liability.
What We Got Wrong First
I want to be specific about our mistakes because "we built a threat model" sounds more competent in retrospect than it was in real time.
Mistake 1: We trusted structured output as a security boundary. Early Oliver used JSON-mode responses and we assumed that if the model was outputting structured JSON, it couldn't be injected. This is wrong. A model can produce valid JSON {"action": "restart_service", "service": "production-postgres"} because a crafted input told it to. Structure doesn't imply legitimacy.
Mistake 2: We underestimated the attack surface of log files. Log files feel like boring operational data. They're also written by every process on your system, including ones that process user input. We had a path where user-submitted job parameters ended up in logs that Oliver was asked to analyze. That's a full injection pipeline. We closed it, but it took an internal red-team exercise to find it.
Mistake 3: We conflated "the model refused in testing" with "the model will refuse in production." Model behavior is not deterministic. Temperature, context length, and the specific phrasing of surrounding content all influence whether a refusal fires. We now treat refusals as probabilistic, not guaranteed — which means we don't rely on them as a primary security control.
The Mitigations That Actually Work
Given the GitLost attack vector and our own experience, here's what meaningfully moves the needle:
Structural context isolation is the highest-leverage mitigation. Not prompt engineering, not "ignore previous instructions" counterprompts, not fine-tuning. Structurally separating untrusted content so it goes through a dedicated summarization/extraction step before touching your action-oriented context is the closest thing to a real defense. It doesn't make injection impossible — a sufficiently adversarial summarization target can still produce malicious structured output — but it raises the bar dramatically and gives you a choke point to monitor.
Typed, named tool interfaces over generic execution. Every tool parameter should be validated server-side against an allowlist or schema. Oliver's restart_service call validates the service name against a registry of known services before it executes anything. The LLM cannot invent a new service name and have it work.
Approval gates on irreversible actions. This is the one mitigation that survives even a fully compromised model. If a human has to confirm before a deployment goes out, a prompt injection attack has to also social-engineer that human in real time, which is a dramatically harder problem.
Audit everything at the tool layer, not the LLM layer. Log tool calls with their parameters at the point of execution, not as reported by the model. The model's reasoning chain is interesting context, but the ground truth of what Oliver actually did lives in the tool execution layer.
It's also worth noting that this problem space is getting more complex, not less. Unicode's transliteration rules were recently shown to be Turing-complete — meaning that obfuscated injection payloads encoded in Unicode normalization behaviors are a real and underexplored attack surface for LLMs that process international text. If Oliver were processing multilingual content at scale, that's the next threat vector I'd be modeling.
The Verdict
If you're building an AI agent with read-only access or trivially reversible operations, start with the naive trust model and iterate toward adversarial trust as your surface area grows. That's a defensible tradeoff.
If your agent has write access to anything production — infrastructure, databases, deployment pipelines, financial transactions — you do not have the luxury of iterating toward adversarial trust. You need to start there. The GitLost attack wasn't sophisticated. It was straightforward prompt injection against an agent that had too much trust in its own context. GitHub is a well-resourced company with serious security engineers, and they got caught by the fundamental data/control conflation problem.
Oliver isn't perfectly secure. No agent with real production access is. But the adversarial trust model means that when — not if — something goes wrong, the blast radius is bounded, the action is logged, and we have a recovery path. That's the actual goal. Not an agent that can never be fooled. An agent where being fooled has survivable consequences.
We're continuing to evolve Oliver's security model at Bedda.tech as we apply these patterns to client infrastructure. The threat model is a living document, not a checkbox. Treat it that way.