17-Day Silent Outage: When Your Cron Lies to You

Production outage debugging is supposed to start with a symptom. An alert fires. A user complains. A dashboard goes red. What happens when none of that occurs, and your system is simply... quietly not working?

That's exactly what hit us. A scheduled job ran every night for 17 days, exited with code zero every single time, and wrote absolutely nothing to the database. No errors. No warnings. No Slack pings. Just silence dressed up as success.

The Cron That Lied

The job was part of a data ingestion pipeline. It pulled records from an external source, transformed them, and persisted the results to a Neon Postgres database using Drizzle ORM. Simple enough architecture. We'd run it in production for months without incident.

Then a schema migration happened. A column was added to the live database, the migration ran cleanly, and the confirmation came back green. What didn't happen was a corresponding update to the Drizzle schema definition in the application code. The TypeScript model and the actual table were now out of sync.

Drizzle, in its default configuration, doesn't throw when you insert a record and the schema definition is missing a column that exists in the database. It just inserts what it knows about. In our case, that column had a NOT NULL constraint with no default value. Postgres rejected every insert silently from Drizzle's perspective because the ORM was constructing queries that omitted the required field entirely, and the error handling in the job swallowed the exception, logged nothing meaningful, and returned success to the scheduler.

Seventeen days. Every night.

Why Schema Drift Is the Invisible Failure Mode

Schema drift is not a new problem, but it's one that modern ORM tooling has made easier to stumble into. The promise of code-first schema management is that your application code is the source of truth. The reality in any team environment is that the database and the code evolve on different timelines, often pushed by different people.

This is the specific failure pattern: a migration runs in production before the corresponding code change ships, or vice versa. In a CI/CD pipeline with fast deploys, that window might be seconds. In a cron-driven backend with weekly deploys, it can be weeks. We were in the second camp.

What made this particularly hard to catch during production outage debugging was the combination of three things happening at once. First, Drizzle's runtime behavior in the version we were running didn't surface schema mismatches as loud errors at startup. Second, the job's error handler was written to catch and suppress exceptions to prevent the scheduler from marking the job as failed repeatedly and triggering false-positive alerts. Third, Neon's serverless connection model means each job invocation spins up a fresh connection with no persistent state that might have revealed the issue over time.

The result was a system that looked healthy from every external vantage point. Scheduler: green. Connection pool: healthy. Database: responsive. Application logs: "Job completed successfully."

The Detection Problem in Cloud-Native Backends

There's a broader infrastructure problem here that goes beyond our specific stack. Cloud-native backend systems are increasingly composed of short-lived, event-driven processes: lambdas, cron jobs, queue workers. These processes are designed to be stateless and disposable, which is architecturally correct. But statelessness also means there's no warm process sitting in memory that would accumulate error state over time and eventually surface it.

A long-running server process that hits a schema mismatch will often crash or degrade in a way that's observable. A cron job that hits the same mismatch, handles the exception, and exits cleanly gives you nothing. The observability model for scheduled jobs needs to be fundamentally different from the model for persistent services, and most teams treat them the same way.

The Hacker News thread on Engrim, a local-first SQLite memory engine for AI CLIs, touched on a related principle this week: persistent local state is sometimes more valuable than people give it credit for, precisely because it accumulates context that ephemeral processes lose. That's not the right fix for a production cron, but the underlying insight applies. Ephemeral processes need external systems to hold the state they can't hold themselves.

For scheduled jobs, that external state is explicit outcome tracking. Not just "did the process exit zero" but "did the process actually do the thing it was supposed to do."

What the Fix Actually Looks Like

There are three layers to a real fix here, and you need all three.

Layer one: schema validation at startup. Drizzle has migration tooling that can compare your schema definition against the live database and surface drift before any application code runs. We now run drizzle-kit check as a preflight step in the job's initialization sequence. If the schemas don't match, the job fails loudly before touching any data. This single change would have caught the problem on day one.

Layer two: outcome-based health checks. Exit codes are not outcomes. An outcome for a data ingestion job is "N records were written this run." We added a post-run assertion: if the job claims to have processed records but the write count returned from the ORM is zero, that's an explicit failure condition that gets logged as an error and reported to our alerting system. The job is allowed to process zero records legitimately (when there's genuinely nothing new to ingest), but that case is now explicitly distinguished from "we tried to write and nothing happened."

Layer three: dead man's switch monitoring. Tools like Healthchecks.io exist specifically for this. The cron job pings an endpoint on successful completion. If the ping doesn't arrive within the expected window, an alert fires. This is the catch-all that survives any failure mode we haven't anticipated yet, including ones that don't involve schema drift at all.

The combination means a silent failure like this can't run for 17 days again. Schema validation catches drift before execution. Outcome assertions catch logical failures during execution. Dead man's switch monitoring catches anything that prevents execution from completing at all.

The Error Handling Anti-Pattern That Made It Worse

I want to be direct about something we got wrong in the original code. The error handler that swallowed exceptions was written with good intentions. We didn't want a transient network hiccup to mark the job as failed in the scheduler, trigger an alert at 3am, and wake someone up unnecessarily. That's a reasonable concern.

The mistake was treating all exceptions the same way. A network timeout is transient and worth suppressing after a retry. A schema mismatch is structural and should never be suppressed. The fix is categorizing exceptions, not flattening them. Operational errors (network, timeouts, rate limits) get retried and then suppressed if they resolve. Structural errors (schema mismatches, missing configuration, invalid credentials) get escalated immediately regardless of what time it is.

This is not a novel distinction. It maps closely to the difference between operational and programmer errors in Node.js error handling philosophy, and it applies equally to any backend runtime. The problem is that it requires discipline at the point of writing the error handler, not just at the point of writing the happy path. Under deadline pressure, that discipline is often the first thing to go.

What 17 Days of Silence Actually Costs

We caught the issue when a downstream system that consumed the ingested data flagged stale records during a manual review. Not an automated alert. A human noticed something looked old. That's a fragile detection mechanism for a production system.

The cost in this case was recoverable: we replayed the missing data once the fix was in place. But the 17-day window represents real data that downstream processes made decisions against, decisions based on a snapshot that should have been updated daily. Depending on what that data drives, the blast radius could have been significant.

Production outage debugging is always easier when you have a clear symptom and a tight timeline. "Something broke at 2:47pm and here's the error" is a tractable problem. "Something has been silently wrong for over two weeks and we don't know exactly when it started" is a different category of problem entirely. The investigation is harder, the impact assessment is harder, and the confidence in the fix is lower because you're not entirely sure you've identified the full scope of what went wrong.

The 17-day duration wasn't bad luck. It was the direct consequence of building a system where silent failure was architecturally possible. The fix isn't just technical. It's a commitment to making silent failure impossible as a design constraint, not an afterthought.

The Three Things That Matter Here

First: schema drift between your ORM definition and the live database is a silent killer for any backend process that doesn't validate at startup. Run the check. Make it a preflight gate.

Second: exit codes are not outcomes. Instrument your scheduled jobs to report what they actually did, not just whether they finished. Zero writes when writes were expected is a failure condition.

Third: dead man's switch monitoring is non-negotiable for any job that runs unattended. If the job doesn't check in, you need to know before 17 days pass.

The cron lied because we let it. We gave it every tool it needed to look successful while doing nothing. That's on the architecture, not the scheduler.