Unit 07.04: Sampling when you cannot keep everything

Sampling is necessary at volume. Sampling uniformly is a mistake.

Sample successes, never failures

Failures, gated actions and over-budget runs are kept in full. Successful runs are sampled.

The code applies that policy to two hundred thousand daily requests.

POLICY = [
    ("all failures",              1.00, "never sample away a failure"),
    ("all approval-gated actions", 1.00, "audit requirement"),
    ("all runs over the latency budget", 1.00, "these are the interesting ones"),
    ("successful runs",           0.02, "enough for a baseline"),
]
DAILY = 200_000
print(f"{'category':36} {'rate':>6} {'kept/day':>10}  why")
kept = 0
for name, rate, why in POLICY:
    share = {"all failures": 0.03, "all approval-gated actions": 0.01,
             "all runs over the latency budget": 0.05,
             "successful runs": 0.91}[name]
    n = DAILY * share * rate
    kept += n
    print(f"{name:36} {rate:>6.0%} {n:>10,.0f}  {why}")

print(f"\n{kept:,.0f} of {DAILY:,} traces kept ({kept / DAILY:.0%})")

# Sample successes, never failures. A uniform 2% sample keeps 2% of your
# incidents too, which means the trace you need during an incident is a coin
# flip you will usually lose.

The policy keeps roughly a tenth of traffic and all of the interesting part. A uniform 2% sample would keep 2% of your incidents too, which means the trace you need during an investigation is a coin flip you will usually lose.

Approval-gated actions are kept at 100% for a different reason: it is an audit requirement rather than a debugging one, and audit requirements do not accept sampling.

The mistake this prevents

The mistake is setting one sample rate for everything because it is simpler to configure. The whole value of a trace store is that it contains the request you need to look at, and uniform sampling optimises for the requests nobody will ever read.

Takeaway

Keep all failures, all gated actions and all over-budget runs; sample successes. Uniform sampling discards the same fraction of incidents as routine traffic.

Mark complete