Unit 08.03: Rate limits and graceful degradation

The provider will rate limit you, and what happens next is a design decision you should make before it does.

A ladder, not a binary

Different upstream statuses call for different responses, and degradation has several rungs between working and broken.

The code lists four statuses and a four-rung ladder.

RESPONSES = [
    (200, "ok",              "serve"),
    (429, "rate limited",    "back off, retry with jitter, then degrade"),
    (503, "provider down",   "degrade immediately"),
    (408, "timeout",         "retry once, then degrade"),
]
print(f"{'status':>7} {'meaning':16} response")
for status, meaning, response in RESPONSES:
    print(f"{status:>7} {meaning:16} {response}")

DEGRADED = [
    ("serve from cache if a fresh-enough entry exists", "best"),
    ("return retrieved passages without a generated answer", "good"),
    ("state that the assistant is unavailable and offer the doc link", "acceptable"),
    ("show a spinner until it times out", "worst"),
]
print("\ndegradation ladder:")
for option, quality in DEGRADED:
    print(f"   {quality:11} {option}")

# The second rung is the useful one and is usually skipped. Retrieval still
# works when the model provider is down, and passages with citations are a real
# answer to most questions -- just not a written one.

The second rung is the useful one and is usually skipped: return the retrieved passages without a generated answer. Retrieval still works when the model provider is down, and passages with citations answer most questions - just not in prose.

The bottom rung is what most systems do by default. A spinner that times out gives the user nothing and costs them the wait, which is strictly worse than saying so immediately.

The mistake this prevents

The mistake is retrying a 503 the way you retry a 429. A rate limit clears with backoff; a provider outage does not, and retrying through it multiplies the load on a service that is already failing while the user waits.

Takeaway

Distinguish rate limits from outages and build a degradation ladder. Returning retrieved passages without a generated answer is a real fallback and the one most systems never build.

Mark complete