Unit 02.04: Trading latency and cost deliberately

Thoroughness is bought with latency and money, on every request, including the ones that did not need it.

What the last point of correctness costs

Three configurations with their latency, cost and measured correctness side by side.

The code prices all three.

CONFIGS = [
    {"name": "cheap",    "k": 3,  "rerank": False, "p95_ms": 620,  "usd": 0.0011,
     "correctness": 0.86},
    {"name": "balanced", "k": 6,  "rerank": False, "p95_ms": 980,  "usd": 0.0031,
     "correctness": 0.92},
    {"name": "thorough", "k": 20, "rerank": True,  "p95_ms": 2400, "usd": 0.0094,
     "correctness": 0.93},
]
print(f"{'config':10} {'k':>3} {'p95':>7} {'$/req':>8} {'correct':>8} {'$/1k':>8}")
for c in CONFIGS:
    print(f"{c['name']:10} {c['k']:>3} {c['p95_ms']:>6}ms {c['usd']:>8.4f} "
          f"{c['correctness']:>8.0%} {c['usd'] * 1000:>8.2f}")

step = CONFIGS[2]["correctness"] - CONFIGS[1]["correctness"]
factor = CONFIGS[2]["usd"] / CONFIGS[1]["usd"]
print(f"\nbalanced -> thorough: +{step:.0%} correctness for {factor:.1f}x the cost"
      f" and {CONFIGS[2]['p95_ms'] / CONFIGS[1]['p95_ms']:.1f}x the latency")

# One point of correctness for three times the cost. That may be worth it; it
# should be a decision someone made, not a default someone inherited.

Going from balanced to thorough buys one point of correctness for three times the cost and two and a half times the latency. That may be worth it - for a medical or legal application it probably is - and it should be a decision someone made rather than a default someone inherited.

The correctness column is what makes the table usable. Without it you are comparing cost against an assumption, and the assumption is always that more retrieval helps.

The mistake this prevents

The mistake is choosing the configuration before measuring correctness for each. Thorough sounds better and often is not: past a point, extra retrieved documents dilute the relevant ones and correctness goes *down* while cost goes up.

Takeaway

Measure correctness for each configuration and price the difference. The last point of quality is usually the most expensive, and sometimes it is negative.

Mark complete