AI Incident Response: Oliver Handles Prod Alone

---
title: 'AI Incident Response: Oliver Handles Prod Alone'
date: '2026-07-31'
description: 'How we built Oliver, an AI agent OS using Claude and SQLite, to automate production incident response — and what happened when it failed.'
author: 'Matthew J. Whitney'
tags: ['artificial intelligence', 'devops', 'cloud computing', 'infrastructure', 'ai integration']
category: 'cloud_devops'
published: true
---

**AI incident response automation** is having its inflection point right now — not because the models got smarter overnight, but because the tooling around agentic loops finally matured enough that a small team can ship something dangerous in a weekend. We did exactly that. We built Oliver, an LLM-powered triage agent running on Claude 3.5 Sonnet, backed by a SQLite incident ledger, wired directly into our production alerting pipeline. Then we let it run unsupervised for six weeks. This is the post-mortem.

I want to be precise about what "inflection point" means here. [Google just disclosed](https://blog.google/security/chrome-stronger-with-every-update/) that AI fixed more Chrome bugs in June 2026 than in the previous two years combined. That's not a marketing slide — that's a measurable productivity step-change at Google scale, in security-critical code, shipped to billions of users. Meanwhile, [DeepSeek-V4-Flash dropped this week](https://api-docs.deepseek.com/updates/) with benchmark scores that are making teams reconsider whether expensive frontier models are necessary for every agentic task. The economics and the capability are converging simultaneously. That's the moment we're in.

So what actually happens next? I've been running Oliver in production long enough to have opinions that aren't vibes. Here are five specific predictions — with the receipts to back them.

---

## Artificial Intelligence Agents Will Own Tier-1 Incident Triage Within 12 Months

**The prediction:** By mid-2027, the majority of engineering teams at Series B and above will have an AI agent handling first-response triage for at least 70% of their production alerts — not routing them to humans, but actually executing the first diagnostic pass and taking bounded remediation actions autonomously.

**The reasoning:** Oliver currently handles roughly 83% of our Tier-1 alerts without human intervention. That number started at 40% six weeks ago. The improvement wasn't model fine-tuning — it was prompt architecture, tool scoping, and building a decent incident memory layer in SQLite so Oliver could recognize recurrence patterns. The tooling is genuinely not that hard anymore. What took a team of five SREs to build three years ago at a previous company took me and one engineer two weekends on the Flow Z13.

The critical unlock is that agentic loop infrastructure stopped being a research problem. When I look at what [Marble OS is exploring for agent GUIs](https://marbleos.com/demo), the question has shifted from "can we build this" to "what does the operator experience look like when the agent is already running." That's a maturity signal.

**What would prove me wrong:** If a major, high-profile autonomous remediation causes a catastrophic cascading failure at a well-known company and the regulatory response is swift. One bad headline about an AI agent taking down a payment processor could freeze adoption for 18 months. I'd give that scenario a 20% probability.

---

## DevOps Teams Will Split Into "Agent Trainers" and "Agent Auditors"

**The prediction:** The traditional SRE/DevOps role bifurcates. One half of the team spends most of their time improving agent behavior — refining runbooks, curating incident examples, tuning tool permissions. The other half audits agent decisions and owns the cases where Oliver gets it wrong.

**The reasoning:** This is already happening on our team, even at our size. I spend more time reviewing Oliver's decision logs than I do reviewing Datadog dashboards directly. The work didn't disappear — it abstracted up one level. The interesting thing is that the "auditor" role requires deeper systems knowledge than the old on-call role did, because you're not debugging the incident, you're debugging the agent's reasoning about the incident. That's a harder problem.

There's also a tooling gap here that's about to get filled. [Ripple](https://www.reddit.com/r/programming/comments/1vbm2g5/show_ripple_detects_api_breaking_changes_across/) — which just launched and detects breaking API changes across OpenAPI, Proto, GraphQL, and AsyncAPI, then auto-opens fix PRs — is exactly the kind of tool that slots into an agent's toolkit. Oliver doesn't currently use it, but the pattern is identical: detect a class of problem, execute a bounded remediation, create an artifact for human review. The auditor's job becomes reviewing that PR, not finding the break in the first place.

**What would prove me wrong:** If model capability improves fast enough that agent errors drop below a threshold where dedicated auditing isn't cost-justified. I don't think we're within 24 months of that, but I've been wrong about model timelines before.

---

## False Positive Cascades Will Be the Defining Failure Mode — Not Model Hallucination

**The prediction:** The dominant incident category for AI incident response automation systems in 2027 won't be "the model hallucinated a bad command." It will be "the model correctly identified a pattern that was wrong in the current context, and the automated remediation made things worse."

**The reasoning:** This is the lesson Oliver taught us the hard way. Week three of the experiment, we had a database connection pool exhaustion event on Crowdia's API layer. Oliver had seen this pattern twice before — both times, the correct fix was restarting the connection pool manager service. So it did that. Except this time, the root cause was a misconfigured upstream load balancer that was hammering a single pod, and restarting the connection pool manager just meant that pod reconnected and got hammered again. Oliver restarted it four times in six minutes before I got paged. We went from a degraded state to a full outage.

The model didn't hallucinate. It pattern-matched correctly against its incident history. The problem was that the incident history was too thin and the action scope was too broad. This is the "[paradox of fancy tooling](https://fagnerbrack.com/the-paradox-of-fancy-tooling-b9af429789da)" in its most expensive form — the tool works exactly as designed, and that's the problem.

After that incident, we implemented what I call "blast radius scoping." Every tool available to Oliver has an explicit blast radius classification: `read-only`, `bounded-reversible`, `bounded-irreversible`, and `escalate`. The connection pool restart was classified `bounded-reversible` but should have been `bounded-irreversible` because of downstream state implications. We reclassified it and added a recurrence check — if Oliver has attempted the same remediation more than once in a 10-minute window without the alert resolving, it escalates instead of retrying.

The false positive cascade problem is structurally underappreciated because teams focus on whether the AI makes the right call in isolation. Production incidents don't happen in isolation.

**What would prove me wrong:** If the industry standardizes on conservative-by-default action scoping so quickly that this failure mode gets engineered out before it causes widespread damage. I hope I'm wrong. I doubt it.

---

## Cloud Infrastructure Providers Will Ship Native AI Incident Response Hooks Within 6 Months

**The prediction:** AWS, GCP, and Azure will all announce native integrations that allow LLM agents to interact with their incident management and remediation APIs through a standardized, permissioned interface — not through hacked-together Lambda functions and CLI wrappers the way we're doing it today.

**The reasoning:** Right now, Oliver's cloud interaction layer is a collection of boto3 wrappers, gcloud CLI subprocess calls, and a lot of defensive error handling. It's brittle. The authentication story is a mess — we're using a dedicated IAM role with carefully scoped permissions, but the tooling to define and audit those permissions for an agentic use case doesn't exist natively. I'm manually reviewing IAM policy diffs to make sure Oliver can't do anything catastrophic.

The cloud providers have every incentive to own this layer. If AI incident response automation becomes the standard, whoever controls the agent-to-infrastructure API owns the stickiest possible integration point. I'd expect AWS to move first — they have the broadest enterprise relationships and the most to lose if teams start building on abstraction layers that don't require AWS-native tooling.

The signal I'm watching: if AWS re:Invent 2026 doesn't include a major announcement in this space, I'll revise this timeline out. But I'd be surprised.

**What would prove me wrong:** If the open-source tooling (think: OpenTofu, Pulumi, etc.) moves faster than the cloud providers and establishes a de facto standard before the hyperscalers can ship their proprietary versions. That's happened before in infra tooling. It could happen here.

---

## The AI Integration Layer Will Become the Most Expensive Part of Your Incident Response Stack

**The prediction:** Within 12 months, teams running serious AI incident response automation will find that the LLM inference costs — not the infrastructure, not the tooling licenses — are the dominant line item in their incident response budget.

**The reasoning:** This surprised me with Oliver. Claude 3.5 Sonnet isn't cheap at the token volumes an active incident generates. A complex, multi-step triage event where Oliver is querying logs, cross-referencing the incident history, evaluating multiple remediation options, and writing a structured post-mortem can burn through 50-100K tokens. Multiply that by alert volume and you have a non-trivial monthly bill.

This is exactly why [DeepSeek-V4-Flash's performance numbers](https://artificialanalysis.ai/models/deepseek-v4-flash-ga) matter for this use case. If a model at a fraction of the inference cost can handle Tier-1 triage reliably — and the early benchmarks suggest it's competitive on structured reasoning tasks — then smart teams will route to cheaper models for routine incidents and escalate to frontier models only for complex or novel failures. We're already thinking about this routing layer for Oliver v2.

The implication is that **AI incident response automation** becomes a cost optimization problem, not just an engineering problem. Your on-call budget is now partly a model selection and routing problem.

**What would prove me wrong:** If inference costs drop faster than adoption grows, keeping the cost-per-incident low enough that optimization isn't necessary. Given the trajectory of both curves, I think costs will be a real constraint for at least the next 12 months even as prices fall.

---

## What You Should Do Right Now

If you're running production systems and you haven't started experimenting with autonomous triage, you're already behind the teams you're competing with for engineering talent and reliability benchmarks. But "start experimenting" doesn't mean "let an agent restart your database."

The most important thing I learned from Oliver's first six weeks: **scope the actions before you scope the intelligence.** Get obsessive about what the agent is allowed to do before you get clever about what it's allowed to think. Build the blast radius taxonomy for your specific infrastructure. Audit every tool you give the agent the way you'd audit a new junior engineer's access on day one — which is to say, conservatively, with explicit review checkpoints.

The second thing: build the incident memory layer first. Oliver's pattern recognition is only as good as its ledger. A flat alert stream with no memory is just an expensive on-call bot. The structured incident history — what happened, what was tried, what resolved it, what the blast radius was — is where the actual value compounds.

The teams that get this right in the next 12 months will have a structural reliability advantage that's genuinely hard to close. The teams that get it wrong will have a very expensive post-mortem and a board conversation they'd rather not have.

Oliver is still running. He got it wrong again last Tuesday — a false positive on a memory pressure alert that turned out to be a scheduled batch job. He escalated correctly this time, which means the blast radius scoping is working. That counts as progress.

We'll take it.