Oliver: AI Agent That Deploys While You Sleep
The prevailing belief about an autonomous AI deployment agent is seductive: you wire up an LLM to your infrastructure, give it some tools, and it handles everything — deploys, rollbacks, incident response — while you sleep. Zero-touch ops. The dream.
We built Oliver. It does deploy while we sleep. But the dream is only half right, and the half that's wrong will wreck your production environment if you don't understand it before you ship it.
This is the honest breakdown.
The Myth: "Give the AI Access and Get Out of the Way"
The hype cycle around agentic AI right now is loud. Every conference talk, every VC blog post, every breathless LinkedIn thread says the same thing: LLMs with tool access are the new DevOps engineer. Just give them shell access, a deployment pipeline, and a monitoring feed, and you're done.
I understand why people believe this. The demos are real. Claude's tool_use API is genuinely impressive — you define a schema for each tool, the model decides when to call it, parses structured arguments, and chains calls together with surprising coherence. I've watched it diagnose a memory leak, identify the offending service, and propose a targeted restart sequence in under forty seconds. That's not a toy. That's meaningful capability.
The belief crystallizes because the easy cases work so well that they crowd out your intuition about the hard cases. And in production, it's always eventually a hard case.
Why the Belief Sticks: The 3 AM Incident That Went Perfectly
Let me tell you about the night Oliver earned its name.
We were running a client platform — multi-tenant SaaS, about 40k active users on a given weekday, hosted across a handful of Fly.io machines with a Neon Postgres serverless backend. At 3:17 AM, connection pool exhaustion started cascading. PgBouncer was saturated. The primary was healthy but unreachable from the app tier.
Oliver caught it in the monitoring loop. Within ninety seconds it had:
- Queried Neon's metrics endpoint to confirm the connection count spike
- Identified the offending application instance via log correlation
- Issued a targeted restart to that single machine — not a full fleet rollout
- Verified recovery by polling the health endpoint on a thirty-second interval
- Written a structured incident report to our Neon-backed
incidentstable with root cause, actions taken, and resolution timestamp
Nobody woke up. The incident lasted four minutes. The postmortem wrote itself.
That night felt like proof. And it was proof — of something real. But it was also the night we started getting sloppy about trust boundaries, because success is the enemy of rigor.
The Actual Reality: Trust Boundaries Are the Whole Problem
Here's what the demos don't show you and the blog posts don't say: an autonomous AI deployment agent doesn't fail by being stupid. It fails by being confidently wrong at exactly the wrong moment.
We discovered this about six weeks into Oliver's operation. A deploy went sideways — a bad environment variable in a staging config that somehow made it into a production artifact. Oliver's monitoring loop detected elevated error rates. It correctly identified that the recent deployment was the likely cause. Then it did something we hadn't fully anticipated: it decided the appropriate remediation was to roll back to the previous artifact and restart the dependent microservice that consumed its API, because the tool schema we'd given it for "rollback" included a restart_dependents flag that defaulted to true.
The dependent service had its own state. Restarting it mid-transaction corrupted about 200 in-flight records.
Oliver's incident report was technically accurate. It had executed exactly what we'd defined. The tool schema was ours. The default was ours. The failure was ours.
This is the reality that the hype obscures: the agent is only as trustworthy as the boundaries you define for it. The LLM is not the risk surface. Your tool definitions are the risk surface.
The LLM Is Not the Problem. Your Tool Schemas Are.
When I look at how we initially designed Oliver's tool manifest, I see the fingerprints of optimism everywhere. We gave it a execute_deployment tool with a broad options object. We gave it a restart_service tool where the blast radius was implicit rather than explicit. We were thinking about capability, not constraint.
The shift that actually made Oliver trustworthy was redesigning every tool schema around minimum viable action. Each tool now has an explicit scope parameter — single_instance, availability_zone, full_fleet — that the model must set deliberately. The default is always the narrowest scope. Widening scope requires a separate tool call that logs an explicit justification string. That string goes into Neon alongside the action record.
This is conceptually similar to what the stinkpot project is doing with SQLite-backed shell history — the insight that persistent, queryable audit trails of what actually happened are more valuable than real-time dashboards. We applied the same logic to Oliver: every tool invocation is a row in Postgres. The agent's reasoning chain — the full Claude response including its chain-of-thought before the tool call — is stored alongside it. When something goes wrong, we have a complete forensic record.
State Is Not Optional
The other thing the demos elide is state management. A stateless agent is a dangerous agent, because it can't distinguish between "this is a novel incident" and "this is the third time I've tried this remediation and it hasn't worked."
Oliver's incident-response loop runs against a Neon Postgres state table that tracks:
- Active incidents with status and severity
- Remediation attempts per incident with outcomes
- A cooldown registry that prevents the same action from being taken twice within a configurable window
- A human-escalation flag that gets set after two failed remediation attempts or any action that exceeds a defined blast radius
That last point is critical. An autonomous AI deployment agent needs to know when to stop being autonomous. The escalation path — PagerDuty, in our case — is not a fallback for when the AI fails. It's a designed boundary that the AI itself respects. Oliver pages us. Oliver doesn't override the page.
What the AI Integration Actually Looks Like in Practice
I want to be specific about the Claude integration because the abstraction level matters enormously.
We're using Claude 3.5 Sonnet via the Anthropic API with tool_use enabled. The system prompt is doing significant work — it establishes the agent's role, its authority boundaries, and critically, a set of explicit prohibitions: no schema migrations without human approval, no fleet-wide restarts without human approval, no changes to authentication configuration under any circumstances.
The tool manifest has twelve tools. That number matters. Early versions had six, which felt clean but forced the model to overload individual tools with too much responsibility. Later versions had twenty-two, which created ambiguity about which tool was appropriate in a given situation — and ambiguity in tool selection is where agents make expensive mistakes. Twelve tools with clear, non-overlapping responsibilities and explicit scope parameters is where we landed after about three months of iteration.
The response loop runs on a sixty-second polling interval. That interval is itself a trust boundary: it gives us a window to manually intervene before a second action compounds a bad first action. A thirty-second loop sounds more responsive. In practice, it doubles your exposure to cascading automated mistakes.
The DevOps Reality: Humans Aren't Removed, They're Elevated
Here's the myth-busting conclusion that I think is actually optimistic, not pessimistic: the right mental model for an autonomous AI deployment agent isn't "replace the on-call engineer." It's "give the on-call engineer a capable first responder who handles the 3 AM connection pool incident so the engineer can focus on the architectural decisions that prevent the next one."
Hillelwayne's recent piece "We Are Not Special" makes a point that resonates here — software engineers have a tendency to believe our problems are uniquely complex in ways that exempt us from established disciplines. The agentic AI hype is doing this in reverse: we're claiming AI agents are so capable that established disciplines around change management, blast radius, and audit trails no longer apply.
They apply more than ever. The agent executes faster than a human. The blast radius of a bad decision expands proportionally.
What Oliver has actually done for us is shift the on-call burden from reactive firefighting to proactive boundary design. We spend less time at 3 AM restarting services. We spend more time in daylight reviewing Oliver's incident logs, refining tool schemas, and tightening the escalation criteria. That's a better use of engineering time. But it's not less engineering time — not yet, anyway.
The cloud computing promise was always "focus on your application, not your infrastructure." The AI integration promise is similar: "focus on your architecture, not your incidents." Both promises are partially true and require significant investment to realize safely.
What to Do Instead of the Hype
If you're building toward an autonomous AI deployment agent, here's the sequence that actually works:
Start with read-only tools. Let the agent observe, diagnose, and recommend for thirty days before it touches anything. You'll learn where its judgment is good and where it's confidently wrong. This is non-negotiable.
Design tool schemas around blast radius, not capability. Every destructive tool should have an explicit scope parameter. The default scope should always be the narrowest possible. Document what each scope level touches before you write the code.
Persist everything to a queryable store. We use Neon because the serverless scaling fits our incident load profile — we have bursts of activity during incidents and near-zero load between them. Every tool call, every reasoning chain, every outcome goes in. You will need this for postmortems.
Build the escalation path before the agent has production access. The human handoff is not a fallback. It's a designed feature. Define the exact conditions under which the agent pages a human and test them deliberately.
Audit the logs weekly. Oliver's incident log is the most valuable engineering artifact we produce. It shows us where our infrastructure is fragile, where our tool schemas have gaps, and where the model's reasoning diverges from what we'd actually want. That signal is worth more than any dashboard.
Oliver is real. It works. It deployed a hotfix at 4:43 AM last Tuesday while I was asleep, and the incident report in Neon was waiting for me with my first coffee. That's not hype — that's a well-designed system operating within carefully designed boundaries.
The myth is that the boundaries are optional. They're the whole product.
Building something like Oliver, or evaluating whether autonomous agents belong in your production stack? Bedda.tech works with teams on exactly this — AI integration that's designed for production from day one, not bolted on after the first incident.