Unit 10.03: Timeouts, and what the user sees when one fires

A timeout the user can understand beats a gateway error they cannot.

Shorter than whatever is in front of you

An async call bounded by wait_for, at two timeouts.

The code shows both outcomes.

import asyncio


async def slow_model(seconds):
    await asyncio.sleep(seconds)
    return "an answer"


async def ask(timeout_s):
    try:
        return await asyncio.wait_for(slow_model(0.05), timeout=timeout_s)
    except asyncio.TimeoutError:
        return ("TIMEOUT: no answer within the budget. "
                "Your request id is r-8841 if you report this.")


for timeout in (0.2, 0.01):
    print(f"timeout {timeout:>5}s -> {asyncio.run(ask(timeout))}")

# The timeout has to be shorter than whatever is in front of you -- the load
# balancer, the browser -- or the user gets a generic gateway error instead of
# your message. And the message carries a request id so a report is actionable.

Your timeout has to be shorter than the load balancer's and the browser's, or the user receives a generic 504 instead of your message - and the request id you would have given them never reaches them.

That request id is what makes a user report actionable. "It timed out" is unactionable; "it timed out, r-8841" points at a specific trace.

The mistake this prevents

The mistake is setting the timeout from the model's typical latency. Set it from what the user will tolerate and what the infrastructure in front of you allows, then make the slow path a fallback rather than a wait.

Takeaway

Set your timeout shorter than the infrastructure in front of you, and include a request id in the message. A generic gateway error carries nothing anyone can act on.

Mark complete