Unit 10.04: Rate limits and the state they need

A rate limiter needs state, and in-process state does not survive workers or restarts.

Five allowed, then 429 with Retry-After

A sliding-window counter in memory.

The code sends seven requests.

import time

WINDOW_SECONDS, LIMIT = 60, 5
buckets = {}


def allowed(caller: str, now: float) -> tuple[bool, int]:
    hits = [t for t in buckets.get(caller, []) if now - t < WINDOW_SECONDS]
    if len(hits) >= LIMIT:
        buckets[caller] = hits
        return False, int(WINDOW_SECONDS - (now - hits[0]))
    hits.append(now)
    buckets[caller] = hits
    return True, 0


now = time.time()
for i in range(1, 8):
    ok, retry_after = allowed("analyst", now)
    print(f"request {i}: {'200' if ok else f'429, Retry-After: {retry_after}s'}")

print("""
This counter lives in memory, so with four uvicorn workers a caller gets four
times the limit -- and a restart forgets everything.

A real limiter needs shared state: Redis, or the gateway in front of you. And
429 must carry Retry-After, or a well-behaved client cannot back off correctly.
""")

With four uvicorn workers this counter exists four times, so a caller gets four times the limit - and a restart forgets everything. That is the same shared-state problem as the deployment module, arriving early.

Retry-After is the other half. Without it a well-behaved client cannot back off correctly, and will retry immediately - which is precisely the behaviour the limit exists to prevent.

The mistake this prevents

The mistake is testing the limiter with one worker and shipping with four. The limit appears to work, and the symptom in production is a limit that seems to be set wrong rather than a design that cannot hold.

Takeaway

Rate limiting needs shared state - Redis or the gateway - because in-process counters multiply by the worker count. Always send Retry-After with a 429.

Mark complete