Every External Call Will Fail. The Only Question Is How Gracefully.

The outage that taught me the most wasn't caused by an error. It was caused by the absence of one.

Our shipping-rates provider had a bad Tuesday — not down, just slow. Requests that normally took 200ms started taking 45 seconds. Our checkout code dutifully waited. Every web worker that touched checkout got stuck holding a connection, waiting politely for an answer that was coming, eventually, technically. Within four minutes, every worker in the fleet was parked on that one call, and the entire application — including pages that had nothing to do with shipping — stopped responding.

The provider never returned a single error. We went down anyway. That's the day "it might fail" stopped meaning "it might return a 500" for me and started meaning something much broader: slow is down, and a dependency's bad day becomes your bad day unless you've drawn some boundaries in advance.

Timeouts: the boundary nobody sets

Here's the uncomfortable default: Ruby's Net::HTTP has timeouts of 60 seconds each for open and read. Many client libraries are similarly generous, and plenty of hand-rolled integrations never set anything at all. Sixty seconds is not a timeout; it's a hostage situation. No user is waiting a minute for a page. The only thing that 60-second window buys you is one worker held captive per request.

Every external call needs two numbers chosen by a human, on purpose:

• Connect/open timeout — how long to wait to establish the connection. Short. If you can't connect in 1–2 seconds, more waiting rarely helps.
• Read timeout — how long to wait for the response. Sized to the endpoint's real p99 plus headroom, not to hope.

conn = Faraday.new(url: SHIPPING_API) do |f|
  f.options.open_timeout = 2   # seconds to connect
  f.options.timeout      = 5   # seconds for the response
end

Five seconds still feels long? Good instinct. For anything in the request path, I want the total budget — all external calls a request can make, combined — comfortably under what a user will tolerate. If an endpoint genuinely needs 30 seconds, that's not a timeout problem; that's a "this belongs in a background job" problem.

When I review an integration, missing timeouts are the first thing I look for, and generated code makes this check more important, not less: assistants produce beautifully structured HTTP clients with retries and JSON parsing and no timeout configuration anywhere, because the happy path doesn't need one. The happy path never does.

Retries: helpful in small doses, arson at scale

A retry is a bet that the failure was transient. Often it is! Networks blip, load balancers rotate, a single node hiccups. One retry with a small delay resolves a huge fraction of real-world failures.

But retries have two sharp edges.

Edge one: retrying non-idempotent operations. Retrying a GET is free. Retrying a "charge the card" call after a timeout is terrifying — a timeout doesn't mean the operation failed; it means you don't know. Maybe the charge went through and the response got lost. Retry blindly and you've double-charged someone. Non-idempotent calls need idempotency keys before they've earned the right to be retried.

Edge two: retry storms. When the dependency is actually down (not blipping), retries multiply your traffic exactly when the dependency can least handle it. Three retries means 4x load on a service that's already drowning — and if every caller retries in lockstep, you get synchronized waves of traffic. This is how a two-minute outage becomes a forty-minute one.

The civilized version is bounded, backing off, and jittered:

def with_retries(max: 3)
  attempts = 0
  begin
    yield
  rescue Faraday::TimeoutError, Faraday::ConnectionFailed => e
    attempts += 1
    raise if attempts >= max
    sleep((2**attempts) * 0.1 + rand * 0.1)  # backoff + jitter
    retry
  end
end

Note what's rescued: transport failures only. Retrying a 422 will produce a 422, three times, slower. And a 429 is the service explicitly asking you to stop — honor it (and its Retry-After header) rather than negotiating.

Circuit breakers: knowing when to stop asking

Timeouts protect a single call. Retries recover a single call. Neither answers the fleet-level question: if the last 200 calls to this service failed, why are we still trying?

That's what a circuit breaker is for. The metaphor is electrical: after enough failures in a window, the breaker opens, and for the next while every call fails instantly — no connection, no waiting, straight to the fallback. Periodically it lets one probe request through (half-open); if the probe succeeds, the breaker closes and normal traffic resumes.

The wins are twofold. Your system stops burning workers on a dependency that's demonstrably down — failing in a millisecond instead of timing out in five seconds is the difference between a degraded feature and a site-wide outage. And the struggling dependency gets breathing room to recover instead of being trampled by your retries.

In Ruby, libraries like circuitbox or resiliency toolkits like semian give you this without ceremony. But the breaker only forces the question you should have answered anyway: what happens when it's open? Show cached shipping rates? A flat-rate fallback? Hide the feature? "Raise an exception" is an answer too, sometimes the right one — but it should be chosen, not discovered during the incident.

Putting it together

These three compose into a sensible posture for any external call:

1. Timeout bounds the cost of one slow call.
2. Bounded, jittered retry absorbs transient blips — for idempotent operations only.
3. Circuit breaker stops the bleeding when blips turn out to be an outage, and routes to a deliberate fallback.

And underneath all three: visibility. Log timeouts distinctly from errors, count retries, alert on open breakers. During my shipping-rates outage, the most damning part of the postmortem was that we had zero metrics distinguishing "slow dependency" from "our app is broken." We debugged our own healthy code for twenty minutes while the graph that would have told us the truth didn't exist.

The checklist I actually use

For every external call in a diff — human-written or AI-generated, no exemptions:

1. Are both timeouts set explicitly, and is the total request-path budget still tolerable for a user?
2. Is the failure mode decided? What does the user see when this call fails? (Someone has to choose. Choose now.)
3. Are retries bounded, backed off, jittered — and only on idempotent operations or ones protected by idempotency keys?
4. Does it retry on 4xx? It shouldn't. Does it honor 429? It should.
5. Is there a breaker (or at least a kill switch) for the fleet-level failure, with an explicit fallback behavior?
6. Will we be able to see this failing — distinct metrics for timeouts, retries, and open circuits?

None of this makes dependencies reliable. Nothing makes dependencies reliable. The goal is humbler and more achievable: when the next provider has its bad Tuesday, it costs you one feature for one hour — not the whole app, and not your whole evening.