Oliver AI Agent: How We Handle False Positives

Our AI incident response agent gets it wrong. Not often, but when it does, the consequences are disproportionate to the mistake. That's the reality we don't talk about enough when we pitch autonomous ops tooling to clients.

Oliver is the internal name for the agent system we run across our production infrastructure. It watches metrics, correlates logs, classifies incidents, and in certain runbook-scoped scenarios, takes autonomous action. Restart a service. Scale a node group. Drain traffic from a region. The wins are real and they compound. But this post isn't about the wins.

This is about the false positives, the alert storms, and the two times Oliver got it badly wrong enough that we had to stop, do a post-mortem, and rebuild pieces of the decision pipeline from scratch.

The Scale of the Problem Nobody Advertises

Before I get into specifics, here's a number worth sitting with: the Atlassian State of Incident Management report has consistently shown that engineering teams spend somewhere between 25 and 40 percent of their on-call hours responding to alerts that don't require action. That's the baseline we were trying to beat when we built Oliver.

We beat it. But we traded one problem for another. Alert fatigue in a human-paged system is passive. You get paged, you look, you dismiss. In an autonomous agent system, a false positive isn't just noise. It's a trigger. The agent acts on it.

The difference in blast radius between "engineer glances at a noisy alert and goes back to sleep" and "agent scales down a healthy node group because it misread a metric spike as a memory leak" is not small.

What the Data Actually Shows About LLM-Based Triage

Oliver's classification layer runs on a fine-tuned model sitting behind a structured prompt chain. The architecture matters here because the failure modes are architectural, not just probabilistic.

The first version of Oliver used a single-stage classification prompt. Alert comes in, context gets assembled (recent logs, metric windows, related service topology), the LLM classifies severity and recommended action, and if confidence exceeds a threshold, the action executes. Clean. Fast. Dangerously simple.

In practice, single-stage classification has a specific failure mode: it's good at pattern-matching to training distribution and brittle to anything outside it. A cascading alert storm, where fifteen services start throwing errors because one upstream dependency hiccupped, looks nothing like a single-service incident in the context window. The token budget fills up with the first few alerts. The model classifies each one independently. It doesn't see the shape of the storm.

We saw this play out in what we internally call Incident Zero. A transient DNS resolution failure in one of our cloud regions caused a wave of connection timeout alerts across services that depended on a shared internal API. Oliver saw timeouts. Oliver classified them as service-level failures. Oliver started taking action on each one independently: restarting pods, rerouting traffic, scaling up replacement instances. By the time a human looked at the dashboard, Oliver had made the situation measurably worse. The DNS issue resolved on its own in under two minutes. Oliver's remediation actions took forty-five minutes to fully unwind.

That's the unglamorous version of "autonomous incident response."

The Architecture Changes After Incident Zero

The fix wasn't "turn off the agent." The fix was adding a correlation stage before classification.

We built what we call the storm detector. It's a sliding-window aggregation layer that runs before any alert reaches Oliver's classification prompt. If alert volume across a service cluster exceeds a threshold within a defined time window, the storm detector fires, and Oliver's autonomous action scope drops to read-only. It can still classify, still draft a runbook summary, still page the on-call engineer with a pre-assembled context package. But it won't touch infrastructure until a human confirms.

This sounds obvious in retrospect. It wasn't obvious when we were building the initial version because we were optimizing for the common case: single-service incidents with clean signal. The storm case is rarer and harder to simulate in testing.

The broader lesson is one that OpenAI's own documentation on agentic systems now explicitly calls out: autonomous agents need minimal footprint principles baked in from the start. Request only necessary permissions, prefer reversible actions, and err toward doing less when uncertainty is high. We learned that from production, not from reading the docs first. I wish we'd read the docs first.

The Second Time Oliver Got It Wrong

Incident Zero was a volume problem. The second major failure was a reasoning problem, and it was subtler.

Oliver has a memory layer. It maintains a rolling summary of recent incidents, resolutions, and service health trends. The idea is that context from past incidents improves classification accuracy for current ones. In most cases it does. But memory introduces a different failure mode: stale priors.

We had a service that had a recurring memory leak issue over several weeks. Oliver correctly identified and remediated it multiple times: flagging the service, triggering a restart, confirming recovery. The memory layer logged this pattern. "Service X: known memory leak, restart resolves."

Then we shipped a fix for the leak. The service was healthy. But a few days later, the service showed elevated memory usage for a completely different reason: a legitimate traffic spike from a new feature launch. Oliver saw the elevated memory, pulled the prior from its memory layer, classified it as the known leak pattern, and restarted the service mid-traffic spike.

The restart caused a brief outage during peak load. Not catastrophic. But avoidable.

The problem here is that Oliver's memory layer was storing pattern associations without any mechanism for invalidating them when the underlying conditions changed. It's a version of the same problem that makes LLM-based systems unreliable when world state changes faster than the model's knowledge: the context is stale but the confidence is high.

The fix was adding a TTL to memory layer entries and requiring that pattern-matched prior incidents be corroborated by at least two independent signal types before triggering autonomous action. Elevated memory alone is no longer enough. It needs elevated memory plus one of: error rate increase, latency degradation, or explicit OOM events in the logs. Single-signal autonomous action is now disabled for anything that touches running services.

What Good False Positive Handling Actually Looks Like

There's a broader conversation happening right now about AI safety and the risks of autonomous systems. A researcher's recent departure from Anthropic with a public warning about AI safety made the rounds this week. The concerns there are existential in scale, but the underlying principle maps directly to production ops: autonomous systems that act with high confidence on incomplete or misread information cause harm proportional to the scope of their permissions.

Oliver's permission scope is narrow by design. It can't modify databases. It can't touch security group rules. It can't push code. Every action it can take autonomously is reversible within minutes. That scope definition is the most important safety control we have, more important than the classification accuracy of the model itself.

Good false positive handling in an AI incident response agent isn't primarily about reducing the false positive rate. It's about minimizing the consequence of false positives that do occur. You tune the model to reduce them. You architect the system to survive them.

Here's what that looks like in practice for us:

Confidence thresholds with action tiers. Oliver has three action tiers. Tier one is observe-and-log, no autonomous action. Tier two is low-impact reversible actions (pod restarts, cache flushes). Tier three is infrastructure-level changes (scaling, traffic rerouting). Tier three requires either very high confidence plus corroborating signals, or explicit human approval. Most false positives that reach the classification stage land in tier one and cause zero impact.

Blast radius caps per incident window. Oliver tracks how many autonomous actions it has taken in a rolling time window. If it exceeds the cap, it locks itself into read-only mode regardless of confidence. This is the circuit breaker. It's what would have limited the damage in Incident Zero if we'd had it in place.

Mandatory human review for novel patterns. If Oliver's classification confidence is high but the incident pattern doesn't match anything in recent memory, it flags for human review before acting. High confidence on a novel pattern is a red flag, not a green light. Novel patterns are exactly where training distribution breaks down.

Post-action confirmation checks. After Oliver takes an action, it monitors for confirmation that the action resolved the issue. If the triggering signal doesn't improve within a defined window, Oliver does not take additional action. It pages a human. Escalation is always available; it's never bypassed.

The Honest Cost-Benefit

Running an autonomous AI incident response agent in production is worth it for us. The reduction in mean time to acknowledge, the overnight coverage, the pre-assembled context packages that make human response faster when it is required: these compound into real operational capacity.

But the cost isn't just compute and API tokens. The cost is the engineering time required to build the failure-handling infrastructure around the agent. The storm detector, the memory TTLs, the action tiers, the blast radius caps, the confirmation checks. None of that was in the initial build. All of it came from production failures.

If you're evaluating autonomous ops tooling and the vendor is showing you dashboards full of incidents resolved and MTTR improvements, ask them what happens when the agent is wrong. Ask them how many autonomous actions were taken on false positives last quarter. Ask them what the blast radius of a worst-case false positive looks like in their system.

If they don't have good answers, the system hasn't been in production long enough to have failed yet. It will.

The unglamorous reality of running an AI incident response agent is that the agent's judgment is only as good as the constraints you build around it. The model is one component. The architecture that decides when the model is allowed to act is the actual product.

We're still building ours. Oliver is better than it was six months ago. It will be better six months from now, mostly because of failures we haven't had yet.