Give each outbound dependency a retry budget, not a retry count

At 09:14 a gateway that usually answers an authorisation in 180 ms starts taking 2.4 seconds. Your client times out at 1 second and retries three times, so every checkout now sends four authorisations, and by 09:16 the gateway is resetting connections for everyone. This playbook replaces the retry count with a retry budget per dependency: a pool of retries that fills at a fixed fraction of first attempts, so a blip that fails 3% of calls still gets its second try and an outage that fails all of them gets almost none. How many retries a call may make should be decided by how many the dependency has absorbed in the last minute, not by a constant in the client.

Classify the failure before you count it

A retry on a response that cannot change wastes a token and, when the first attempt landed, makes a second side effect. A 400 for an amount with three decimals, a 401 for a rotated API key and a 404 for an unknown stored-card reference answer the same way every time. A 409 from an idempotency conflict means the first attempt was processed and the right move is to read the original result. A 429 or a 503 carrying Retry-After is the dependency asking for a later try. A timeout says nothing at all, because the gateway may have authorised the card after you closed the socket.

Put the classification in the adapter for that dependency, as one function that takes the raw error and returns a verdict next to the raw code, so the retry loop never reads status codes itself.

You know it worked when the retry counter, broken down by response class, shows zero retries on any 4xx other than 429.

Fill the budget from first attempts, not from a clock

A fixed count multiplies your load by one plus the count at the moment the dependency can least take it. A budget ties the retries you may send to the traffic you already sent: with a ratio of 0.1, 1,000 first attempts a minute earn 100 retries a minute. During a blip that fails 3% of calls, 30 retries are needed and 100 are available, so every failed call gets its second try. During an outage that fails all 1,000, the first 100 retries go out, the bucket empties, and every later call fails fast with its first error and the reason budget. The dependency never sees more than 1.1 times your normal traffic from retries.

I have built routing across 15+ gateways with automated failover, and the rule I apply is that the retry decision belongs to the dependency, never to the request. One bucket per dependency per process is enough, because each instance's retries are a fraction of that instance's first attempts, so the ratio holds however many instances run. Cap the bucket, otherwise a quiet night banks an hour of retries that all spend in the first minute of the morning outage.

flowchart TD
  F[Attempt failed] --> C{Retryable}
  C -- no --> R[Return error with code]
  C -- yes --> D{Wait fits deadline}
  D -- no --> X[Fail with reason deadline]
  D -- yes --> B{Budget token left}:::accent
  B -- no --> Y[Fail with reason budget]
  B -- yes --> W[Sleep jittered wait]
  W --> A[Send same request again]

// simplified: one bucket per dependency, per process
class RetryBudget {
  private tokens: number;
  constructor(private ratio: number, private cap: number) {
    this.tokens = cap;
  }
  recordFirstAttempt(): void {
    this.tokens = Math.min(this.cap, this.tokens + this.ratio);
  }
  tryTakeRetry(): boolean {
    if (this.tokens < 1) return false;
    this.tokens -= 1;
    return true;
  }
}

async function callWithBudget<T>(
  dep: Dependency, req: OutboundRequest, deadlineAt: number,
): Promise<T> {
  dep.budget.recordFirstAttempt();
  let attempt = 0;
  for (;;) {
    attempt += 1;
    try {
      return await dep.send<T>(req, { timeoutMs: deadlineAt - Date.now() });
    } catch (err) {
      const verdict = dep.classify(err); // { retryable, afterMs? }
      if (!verdict.retryable) throw err;
      const wait = verdict.afterMs ?? backoffMs(attempt);
      if (Date.now() + wait + dep.minResponseMs >= deadlineAt) {
        throw withReason(err, 'deadline', attempt);
      }
      if (!dep.budget.tryTakeRetry()) {
        throw withReason(err, 'budget', attempt);
      }
      await sleep(wait);
    }
  }
}

You know it worked when a dependency that fails every call for a full minute shows, in its request log, no more retries from you than the ratio times its normal minute of traffic.

Back off with full jitter and stop at the caller's deadline

Every client that timed out at the same instant retries at the same instant unless the wait is randomised, so the dependency takes a second spike exactly one timeout after the first. Full jitter picks the wait uniformly between zero and the capped exponential: attempt 1 waits between 0 and 200 ms, attempt 2 between 0 and 400 ms, attempt 3 between 0 and 800 ms, and nothing waits longer than 2 seconds. A Retry-After header overrides the computed wait, because the dependency knows when it will be ready and you do not.

The deadline matters more than the count. If checkout gave the payment service 5 seconds and the first attempt spent 1 second timing out, the retry gets what is left, and the loop refuses a retry whose wait plus the dependency's minimum response time would pass the deadline. A count of 3 with a 1 second timeout and no deadline check spends 4 of those 5 seconds on one dependency and leaves nothing for the failover gateway. The deadline check runs before the budget check, so a retry that could never finish never spends a token.

// simplified: full jitter, base 200 ms, cap 2 s
function backoffMs(attempt: number): number {
  const ceiling = Math.min(2000, 200 * 2 ** (attempt - 1));
  return Math.floor(Math.random() * ceiling);
}

You know it worked when the histogram of delays between a failed attempt and its retry is flat across each band rather than spiked at 200, 400 and 800 ms.

Rehearse the slow gateway, not the dead one

A dead dependency refuses the connection in 5 ms and costs you almost nothing. The slow one takes you down with it. Walk it through with a count of 3, a 1 second timeout and 1,000 authorisations a minute.

1. The gateway slows

   At 09:14:00 its median response goes from 180 ms to 2.4 seconds. Nothing is down. Every authorisation you send now outlives your 1 second timeout.

2. Each checkout sends four

   The first attempt times out at 1 second, and retries go out at about 1.2, 2.4 and 3.8 seconds. Closing your socket does not cancel the gateway's work, so it is now processing four authorisations per checkout, all for the same 42.00 GBP.

3. The pool fills

   Each instance holds 200 connections to that host. A checkout now holds one for 4 seconds instead of 1, so the pool is full within a minute, and refunds and status inquiries to the same host wait behind authorisations.

4. The customer sees the worst case

   A spinner for 5 seconds, then a failed payment. On the gateway side, an attempt you gave up on may still be approved, so the card carries a hold the order does not know about, and up to four of them if the retries did not share an idempotency key.

5. The budget changes the shape

   With ratio 0.1 and cap 100, the first 100 timeouts spend the whole bucket in the first six seconds. From then on the bucket refills at 0.1 per first attempt, so about one timeout in ten earns a retry and the other nine fail fast at 1 second with reason budget. The gateway sees 1.1 times normal load and each checkout holds a connection for 1 second, not 4.

What contains the rest sits outside the loop: the fast failure returns the classified error to the routing layer, which fails over to a second gateway with its own bucket, and every timed-out attempt is written as unknown so an inquiry job can find the holds placed after you gave up. Rehearse this with the sandbox gateway delayed to 2.4 seconds, not blocked: a blocked port never shows you the pool filling.

You know it worked when the sandbox run shows the slowed gateway receiving at most 1.1 times its normal request count in the first minute.

Alert on refused retries, not on the error rate

The error rate rises after the damage is done. A refused retry is earlier: it is the first moment your service decided a dependency was in enough trouble to stop helping it. Emit, per dependency, a gauge of tokens left and counters for first attempts, retries sent, retries refused for budget and retries refused for deadline, each labelled with the response class that triggered it. Alert when retries refused for budget stays above zero for two consecutive minutes, and page when it lasts five.

Log every refused retry with the dependency name, the attempt number, the reason and the correlation id, so the person on call can go from the alert to the checkout that failed fast without searching by time. Keep the ratio and the cap in the dependency's configuration next to its timeout; a ratio raised to 0.5 during an incident is a decision someone must find later.

You know it worked when a staged slowdown in the sandbox fires the budget alert before the checkout failure dashboard moves.

Build the classify function and the bucket first, wrap the one gateway with the worst latency tail, and ship the refused-retry metric the same day. The jitter and the deadline check can follow a day later; the budget is what stops the storm.