Your Background Job Will Run Twice. Plan for It.

The worst bug of my career was eleven lines long. It was a background job that charged customers for their monthly subscription, and it worked perfectly for about a year. Then one night a worker process got OOM-killed halfway through a batch — after the payment API call succeeded, before the job marked itself done. The queue did exactly what it promised: it saw an unfinished job and ran it again.

Forty-three customers got charged twice. I spent the next week writing apology emails and issuing refunds, and I've never designed a job the same way since.

"At least once" means what it says

Here's the mental model shift that took me embarrassingly long to internalize: almost every job queue you'll ever use — Sidekiq, ActiveJob backends, SQS, Kafka consumers — guarantees at-least-once delivery, not exactly-once. The system would rather run your job twice than risk not running it at all, and that's the correct trade. Jobs get re-run because:

• A worker crashes or gets OOM-killed mid-job.
• A deploy restarts workers and in-flight jobs get requeued.
• The job raises and the retry mechanism kicks in — including after it did real work.
• A network blip makes the queue think the ack never arrived.
• A human clicks "Retry" in the dashboard at 2 a.m. (Ask me how I know.)

Notice that most of these re-run the job after partial success. That's the killer detail. It's not "the job runs twice from the start"; it's "the job runs again from the start after already completing steps 1 through 3." Any design that doesn't survive that is a countdown timer.

What the fragile version looks like

Here's a lightly disguised version of my eleven lines of shame:

class ChargeSubscriptionJob
  def perform(subscription_id)
    sub = Subscription.find(subscription_id)
    PaymentGateway.charge(
      customer: sub.customer_token,
      amount: sub.price_cents
    )
    sub.update!(last_charged_at: Time.current)
    ReceiptMailer.send_receipt(sub).deliver_later
  end
end

Every line is reasonable. The sequence is deadly. If anything fails after the charge call — the database hiccups, the process dies, the mailer raises — the retry will charge again. The job assumes it runs once, and that assumption is not part of the contract.

Designing for the re-run

Idempotency is the property that running the job twice has the same effect as running it once. There are a few reliable ways to get there, and they stack.

1. Idempotency keys on the external call

Every serious payment API (and most external APIs worth their salt) accepts an idempotency key. Same key, same request — the provider returns the original result instead of acting twice:

PaymentGateway.charge(
  customer: sub.customer_token,
  amount: sub.price_cents,
  idempotency_key: "sub-#{sub.id}-#{billing_period.iso8601}"
)

The key must be deterministic from the work itself — subscription plus billing period, not SecureRandom.uuid generated inside the job. A random key regenerates on retry and protects nothing. I've flagged that exact mistake in review more than once, and lately I see it in AI-generated code constantly: the assistant knows idempotency keys exist, and then dutifully generates a fresh one per attempt, which is a lock that changes its own combination.

2. Record intent before acting

For anything involving money, I want a local record of the attempt before the external call — a charges table row with a unique constraint on (subscription_id, billing_period). The job becomes: claim the work, then do it.

charge = Charge.create!(subscription:, billing_period:, status: :pending)
# unique index makes a second attempt raise instead of double-charging

If the insert fails on the unique index, someone already claimed this period — look up that row and resume or bail. This also gives you an audit trail, which your future self on refund duty will treasure.

3. Make each step re-runnable or skippable

Structure the job as a series of steps that each check "did I already do this?" before acting. Guard the charge on the charge record's status. Guard the email on a receipt_sent_at timestamp. It feels like paranoia the first time you write it. It feels like wisdom the first time a retry sails through the completed steps and finishes the one that failed.

4. Keep jobs small and single-purpose

A job that charges, updates state, sends email, and syncs to the CRM has four ways to partially fail. Four small jobs — each idempotent, each enqueued by the previous one or by an event — mean a retry re-runs one cheap, guarded step instead of a fragile saga.

The subtle ones

A few non-obvious cases that bite people:

• increment! and counters. user.increment!(:login_count) run twice counts twice. Anything additive is non-idempotent by nature; either make it a derived value or guard it with an event record.
• Enqueuing other jobs. If your job enqueues ten child jobs and dies after enqueuing six, the retry enqueues sixteen. Child jobs need to be idempotent too, or the parent needs to record what it enqueued.
• Time-based logic. if sub.last_charged_at < 30.days.ago looks like a guard, but between the check and the write there's a window — and two concurrent retries can both pass the check. A unique constraint doesn't have that window. Prefer constraints over checks.
• "It only retries on failure." The queue's definition of failure is broader than yours. Timeouts, deploys, and dashboard clicks all count.

And one habit worth building: test the re-run explicitly. Call perform twice in the same test and assert one charge, one email, one row. It's a two-line test that encodes the entire contract. I almost never see it in generated test suites — assistants test the happy path once and move on — so it's on the reviewer to ask for it.

What I'd do instead

Before I approve any job that has side effects, I want answers to four questions:

1. What happens if this runs twice, including after partial success? If the answer involves the word "shouldn't," it's a no.
2. Is every external call carrying a deterministic idempotency key? Derived from the work, not from randomness.
3. Is there a unique constraint backing the "have we done this already?" check? Checks in Ruby have race windows; constraints don't.
4. Is there a test that performs the job twice?

Retries aren't the failure mode. Retries are the feature — they're the reason a flaky network doesn't lose your customer's order. The failure mode is writing code that treats a retry as an impossible event. It will run twice. The only question is whether you decided in advance what that means.