Unit 12.03: Retries that do not multiply the load

Retrying a struggling dependency multiplies the load on something already failing.

Bounded, jittered, and only for the right statuses

Three attempts with backoff and jitter, and which statuses are worth retrying.

The code shows the schedule and the classification.

ATTEMPTS = [(1, 0.5), (2, 1.0), (3, 2.0)]
JITTER = [0.31, 0.72, 0.08]

print(f"{'attempt':>8} {'base wait':>10} {'actual':>8}")
total = 0.0
for (attempt, base), jitter in zip(ATTEMPTS, JITTER):
    actual = base * (0.5 + jitter)
    total += actual
    print(f"{attempt:>8} {base:>10.1f} {actual:>8.2f}")
print(f"\n3 attempts, {total:.2f}s of waiting, then give up")

RETRYABLE = {"503": True, "504": True, "429": True,
             "400": False, "401": False, "422": False}
print(f"\nretry only: {[k for k, v in RETRYABLE.items() if v]}")

print("""
Retrying a failing dependency multiplies the load on something already
struggling. Bound the attempts, add jitter so every instance does not retry in
unison, and never retry a 4xx -- the payload will not become valid.
""")

Jitter stops every instance retrying in unison. Without it, a brief outage produces a synchronised thundering herd the moment the dependency recovers, which can knock it over again.

Never retry a 4xx. The payload will not become valid on the second attempt, so three tries cost three times as much and delay the real error.

The mistake this prevents

The mistake is an unbounded retry loop for resilience. A dependency outage lasts longer than any retry budget, so the loop turns a fifteen-minute outage into sustained load and a very large bill.

Takeaway

Bound the attempts, add jitter, and retry only statuses where something could differ next time. Retrying a 4xx is pure cost.

Mark complete