Drizzle Schema Drift: 17-Day Silent Cron Failure
Drizzle schema drift is the most dangerous class of bug I've shipped in the last three years, and the reason is simple: it doesn't look like a bug. No exceptions. No 500s. No Sentry alerts. Just a cron job that cheerfully runs every six hours, logs "completed successfully," and writes absolutely nothing to your database.
That's what happened on a production service I run on top of Neon — a serverless Postgres platform I've standardized on for most of my TypeScript backends. Seventeen days. The job ran 68 times. The success log fired 68 times. Zero rows were inserted.
I want to walk through exactly what happened, why Drizzle's design made it invisible, and why the fix I landed on — running drizzle-kit push with a timeout guard on every deploy — is the only sane default for any team that doesn't have a dedicated DBA watching migration diffs.
The TypeScript Backend That Lied to My Face
The service in question is part of KRAIN, an internal data aggregation layer I've been building. The cron job's job is simple: pull external records, normalize them, and upsert into a sync_records table. The stack is TypeScript, Drizzle ORM, Neon Postgres, deployed on Railway with a nixpacks-based build.
About three weeks before the incident, I added two new columns to sync_records: a checksum field for deduplication and a source_version integer for tracking schema versions of the upstream data. I updated the Drizzle schema file, ran drizzle-kit generate locally, reviewed the SQL, and committed everything. What I did not do was run drizzle-kit push or apply the migration to the production Neon branch before deploying.
The deploy went out. The cron job kept firing.
Here's the part that makes this insidious: Drizzle ORM does not validate your TypeScript schema against the live database at runtime. When you call db.insert(syncRecords).values(...), Drizzle constructs the SQL from your schema definition. If your schema definition includes columns that don't exist in the actual database, Postgres will reject the insert. But the error Postgres returns is a column-does-not-exist error, and in this particular job, I had a broad try/catch that logged the error internally and exited cleanly with code 0.
The monitoring saw exit code 0. The log line said "sync completed." Nothing paged.
This is structurally the same class of failure described in the recent LuaJIT NYI post making the rounds on r/programming, where a performance regression silently poisoned an unrelated hot loop without surfacing any obvious error signal. The pattern is universal: systems that fail quietly are orders of magnitude harder to debug than systems that fail loudly, because the absence of signal is itself invisible.
Silent Failures Are an Infrastructure Problem, Not a Code Problem
I want to push back hard on the reflex to blame the broad try/catch. Yes, I should have been more surgical with error handling in that specific job. But the root cause is an infrastructure gap, and patching the catch block is treating a symptom.
The core issue is that Drizzle's workflow, by design, separates schema definition from schema application. That separation is valuable in controlled environments. It's a liability in fast-moving solo or small-team projects where the person writing the migration is the same person deploying the service, and "run migrations" is a mental checklist item that competes with twelve other things during a deploy.
Neon's branching model makes this worse in a subtle way. Neon lets you create database branches for preview environments, which is genuinely excellent for staging. But it also means your production branch can silently diverge from your development branch, and if you're doing local development against a Neon dev branch, your local drizzle-kit push only touches that branch. The production branch stays untouched until you explicitly target it.
I had done exactly that. My local environment pointed at neon://...?branch=dev. My production Railway deploy pointed at the main branch connection string. I pushed schema changes to dev, tested locally, and deployed the code without ever running the migration against main.
The Neon documentation on branching is clear about this separation. I knew it intellectually. I still got burned.
The DevOps Fix: Auto-Migrate on Every Deploy
The solution I landed on is opinionated and some people will hate it: run drizzle-kit push as part of the deploy process, every single time, before the application starts.
In the Railway service configuration, I added a startCommand override that runs the push first:
npx drizzle-kit push && node dist/index.js
With a timeout guard via a wrapper script to prevent a hung migration from blocking the deploy indefinitely:
timeout 60 npx drizzle-kit push || { echo "Migration failed or timed out"; exit 1; }
node dist/index.js
If the migration fails, the deploy fails. If it times out, the deploy fails. The old version keeps running. No silent drift.
The objection I hear most often is "what about destructive migrations?" And that's fair. drizzle-kit push in non-interactive mode will not execute destructive changes by default; it will abort and tell you what it wanted to drop. That behavior is exactly what you want in a CI/CD pipeline. Destructive changes require human review. Additive changes (new columns, new tables, new indexes) apply automatically and safely.
For KRAIN, the vast majority of schema changes are additive. I'm adding columns, adding tables, adding indexes. The rare destructive change gets flagged, the deploy fails, I review it, and I apply it manually through the Neon console before re-deploying. That's the right workflow. The migration is no longer something that can be forgotten.
Cloud Infrastructure Requires Pessimistic Defaults
There's a broader principle here that I've been thinking about more since this incident. When you're running on serverless infrastructure like Neon, Railway, or Vercel, the contract between your code and your infrastructure is more fragile than it looks. Serverless Postgres connections pool and proxy in ways that can mask errors. Serverless compute environments can have cold start behaviors that affect how errors surface. The observability defaults are often optimistic.
Werner Vogels recently wrote about building scalable control planes and one of his recurring themes is that distributed systems require you to assume failure at every boundary. Schema drift is a failure at the boundary between your application's model of the world and the database's actual state. The only way to handle boundary failures in distributed systems is to make them explicit and loud, not to hope they don't happen.
The drizzle-kit push on deploy approach makes the boundary explicit. Every deploy is an assertion: "my schema and the database agree." If they don't, the deploy stops. That's a pessimistic default, and pessimistic defaults are the correct defaults for cloud infrastructure.
What I Changed in the Cron Job Itself
Beyond the deploy pipeline fix, I made two changes to the cron job that I should have had from the start.
First, I split the try/catch. The outer catch now re-throws any PostgresError with a code that indicates a schema mismatch (specifically 42703, which is undefined_column). That error now propagates to the job runner, which marks the run as failed and pages me.
Second, I added a startup health check that runs a lightweight SELECT 1 FROM sync_records LIMIT 1 before the job does any real work. If that query fails, the job aborts immediately with a non-zero exit code. This doesn't catch all schema drift scenarios, but it catches the most common one: a column referenced in the select list that doesn't exist.
These are defensive measures. The real fix is the migration in the deploy pipeline. The defensive measures exist because I no longer trust any single layer of protection.
Why I'm Not Backing Down on This
Seventeen days of silent failure is not a monitoring problem or a logging problem. It's a schema management problem, and the Drizzle community's default answer, "generate migrations and apply them manually," is wrong for most teams actually shipping on cloud infrastructure.
The argument for manual migrations is that it gives you control and reviewability. I agree with that for large teams, for financial systems, for anything where a bad migration has catastrophic consequences. For the other 80% of projects, the manual step is a footgun. The gap between "I updated the schema file" and "I applied the migration to production" is where Drizzle schema drift lives.
Auto-migrating on deploy with drizzle-kit push and a timeout guard closes that gap permanently. The deploy pipeline becomes the migration step. Schema drift becomes a deploy failure, not a silent runtime failure. Cron jobs stop lying about their success.
I shipped this pattern across every Drizzle-backed service I run after this incident. Two months in, it has caught three schema mismatches before they hit production, all of them additive changes I made locally and forgot to push to the production branch. Each one would have been another silent failure waiting to surface at the worst possible time.
The uncomfortable truth about Drizzle schema drift is that the ORM's flexibility is also its trap. Drizzle trusts you to keep the schema and the database in sync. On a team of one or two moving fast, that trust is misplaced. Automate the sync, make failures loud, and stop relying on memory to keep your production database honest.