Migrations That Don't Take Production Down With Them

There's a specific flavor of dread that comes from watching a deploy hang at the migration step while your error tracker starts filling up. I've felt it maybe four times in my career, and every single time the migration in question had run in under a second on my laptop.

My laptop had nine hundred rows in that table. Production had ninety million. That difference is the whole subject of this article.

The two lies your dev environment tells you

Lie one: migrations are fast. Adding a column, backfilling a value, adding an index — instant in development. In production, some of these operations take locks. On Postgres, an ALTER TABLE needs an ACCESS EXCLUSIVE lock, and here's the nasty part: even if the alteration itself is fast, the migration has to wait for that lock behind every running query on the table — and while it waits, every new query queues up behind it. One long-running report query plus one innocent add_column equals a frozen table and a very bad fifteen minutes.

Lie two: code and schema change at the same moment. They don't. During a deploy, old code runs against the new schema (migrations usually run before the new code boots), and with rolling deploys, old and new code run side by side for minutes. Any migration that assumes "the code that needs this schema is already live" — or worse, "the code that needed the old schema is already gone" — is a race you'll eventually lose.

The classic casualties

Removing a column the old code still loads

This one is famous for a reason. You delete a column and the migration runs fine — but ActiveRecord caches the column list, so the old code still running during the deploy generates SELECT statements naming the dead column. Every query on that model starts throwing UndefinedColumn. Not just the code path you changed — everything touching that table.

The fix is a two-deploy dance: first ship code that tells Rails to ignore the column, then remove it later.

# Deploy 1: code change only
class Order < ApplicationRecord
  self.ignored_columns += ["legacy_status"]
end

# Deploy 2: after deploy 1 is fully live
remove_column :orders, :legacy_status

Renames are the same problem wearing a different hat — a rename is a remove and an add. Do it as add-new, dual-write, backfill, switch reads, then remove-old. Tedious? Yes. Cheaper than the incident? Every time.

Adding a NOT NULL column with a default

On modern Postgres (11+), add_column with a static default is thankfully fast. But adding NOT NULL to an existing column still wants to scan the table to validate it — unless you add a CHECK constraint as NOT VALID first, then validate it separately, which takes a much friendlier lock. Know which operation you're actually asking the database to perform; "add a required field" spans both the cheap and expensive cases.

Adding an index without CONCURRENTLY

A plain CREATE INDEX on Postgres blocks writes to the table for the duration of the build. On a big table that's minutes of no inserts or updates. CREATE INDEX CONCURRENTLY avoids the write lock at the cost of being slower and requiring disable_ddl_transaction! in Rails:

class AddIndexToOrdersOnCustomerId < ActiveRecord::Migration[7.1]
  disable_ddl_transaction!

  def change
    add_index :orders, :customer_id, algorithm: :concurrently
  end
end

Backfills are not schema changes

This is the hill I'll politely die on: data migrations don't belong in schema migrations.

A backfill that updates ninety million rows inside a migration has three problems. It runs inside the deploy, so your deploy now takes an hour and can't be easily rolled back. It probably runs in one giant transaction, bloating locks and WAL. And if it fails at row 60 million, your deploy is wedged in a half-migrated state at the worst possible moment.

Backfills should be separate, resumable, and batched — a rake task or a background job that walks the table in chunks:

Order.where(fulfillment_state: nil).in_batches(of: 5_000) do |batch|
  batch.update_all(fulfillment_state: "legacy")
  sleep(0.1) # let replicas breathe
end

The full sequence for "new required column" looks like: add the column nullable → deploy code that writes it for new records → backfill old records out-of-band → verify → add the constraint. Five boring steps instead of one exciting one.

Guardrails beat vigilance

You will not remember all of this at 6 p.m. on the day the feature is due, and neither will I. That's what tooling is for.

• strong_migrations (or your stack's equivalent) statically catches dangerous operations and refuses to run them without an explicit acknowledgment. It's the single highest-value gem-shaped seatbelt I know.
• lock_timeout and statement_timeout on the migration connection turn "migration waits forever for a lock while queries pile up" into "migration fails fast and you retry off-peak." A failed migration is recoverable; a locked hot table is an incident.
• Review migrations as production events, not code. In review, I ask: how big is this table, what lock does this take, and what does the old code do while this is live? A diff that's syntactically perfect can still be operationally radioactive.

A word about generated code here, because it matters: coding assistants write textbook migrations — and the textbook assumes an empty database. Ask one for "add a required status column with an index" and you'll typically get add_column ... null: false, default: plus a plain add_index, sometimes with an inline update_all backfill for flavor. Nothing about it is wrong in development. All of it can be wrong on a 90-million-row table. The model doesn't know your row counts; you do. That context is exactly what human review is for.

The checklist I actually use

Before any migration ships to a table with real traffic:

1. How many rows, and how hot is the table? Under a million rows and low traffic, most of this article relaxes. Big and hot, all of it applies.
2. What lock does each statement take, and for how long? If you don't know, look it up before production teaches you.
3. Will the currently-deployed code work against the new schema? And will the new code work against the old schema, for the minutes both are live?
4. Is any data movement separated out into a batched, resumable, off-deploy task?
5. Indexes: concurrently? Column removals: ignored_columns first? Timeouts set?
6. Is strong_migrations (or equivalent) installed and not being bypassed with a shrug?

None of this is glamorous. Schema changes are plumbing. But plumbing is exactly the kind of work where "it ran fine on my laptop" and "it ran fine in production" are separated by ninety million rows — and knowing that difference is a big part of what senior means.