Unit 07.00: The question you will ask at 2am

Decide what to instrument by writing down the questions you will ask during an incident. None of them can be added while it is running.

Seven questions, seven instruments

Scope, timing, cause, stage, upstream, user impact, and how many.

The code lists each with what it requires.

QUESTIONS = [
    ("is it broken for everyone or one user?", "error rate by user segment"),
    ("when did it start?",                     "timestamped metrics, not just now"),
    ("what changed?",                          "deploy and config version in every trace"),
    ("which stage is failing?",                "per-stage success rate"),
    ("is it the provider?",                    "upstream latency and error codes"),
    ("what did the user actually see?",        "the response, redacted, per request"),
    ("how many are affected?",                 "count, not rate"),
]
print(f"{'question at 2am':44} needs")
for question, need in QUESTIONS:
    print(f"{question:44} {need}")

print(f"""
{len(QUESTIONS)} questions. Every one implies something instrumented in advance,
and none can be added while the incident is running.

The third is the one most often missing and most often the answer.
""")

"What changed?" is the one most often missing and most often the answer. A deploy version and a config version in every trace turns that question into a comparison rather than an investigation.

"How many are affected?" needs a count rather than a rate. During an incident, "error rate is 3%" and "6,000 users have seen this" prompt very different responses, and the second is the one that decides whether to roll back.

The mistake this prevents

The mistake is instrumenting after the first incident, using whatever would have helped with that one. The incident you instrument for is never the incident you get, and all seven of these fields are cheap enough to add together.

Takeaway

Write down the 2am questions and instrument for all of them at once. Deploy and config version in every trace answers the one that matters most.

Mark complete