Compare quality, cost, and delay honestly

The candidate is faster on average and its recorded cost is lower. Somebody wants to declare it the winner. Before agreeing, ask what happened to the task, what the slowest experiences looked like, and whether the observations were comparable. Saving time on a response that fails the user's request is not automatically an improvement.

You will inspect matched synthetic observations across repeated trials, keep distinct cases separate from repeats, and apply independent constraints. This lesson calculates descriptive results only. No real model is run or timed, no cost is billed, and no statistical confidence claim is made.

Define the comparison before generating a score

In operations.json, trial_plan declares four holdout case IDs and three trial IDs before scoring. trials supplies the twelve pairs. Each pair stores a baseline and candidate record with a correctness boolean, delay value, and cost value. Matching the pair prevents a candidate being tested on easy messages while the baseline is tested on harder ones. The function requires the full planned Cartesian product: deleting one row, every third repeat, or an entire difficult case cannot silently redefine the experiment.

For a real service, you would also define a measurement boundary. Does delay begin when the user submits, when a queue receives work, or when a provider starts processing? Does it end at the first token or the complete usable response? Are retries, retrieval, tool calls, and queue time included? Different boundaries answer different questions. Our values are handwritten examples, so their boundary is explicitly “invented; not measured.”

Cost needs a similarly explicit convention. A provider's listed unit rate is not automatically the total cost of a completed task. Retries, extra calls, and review effort can change the denominator. Here each invented candidate observation costs 1,400 micro-USD and each baseline observation 1,800. One micro-USD is one millionth of a dollar. This arithmetic is not a provider price comparison.

Read the repeated observations

Run from project:

python3 -B opslab.py compare

The baseline has twelve correct observations. The candidate has ten. Its accuracy is 10/12 ≈ 0.8333, and its paired accuracy delta is -2/12 ≈ -0.1667. The change is a loss of about 16.7 percentage points on these observations, not a 16.7% estimate of future user harm.

Across the three trial IDs, candidate accuracy is 1.00, 0.75, and 0.75. Looking only at trial 1 would hide the two later failures. Repeating a case can expose variability in actual model behavior, but replaying our fixed fixture merely illustrates how such a table is analyzed.

There are four distinct cases, not twelve distinct users. The three observations of h3 share a case and context. Treating them as twelve independent sampled users would give a misleading impression of diversity. If you later estimate uncertainty, account for the sampling unit and dependence; do not copy a formula whose independence assumptions are unmet.

Work through the tradeoff

The baseline's mean invented delay is 562.5 ms. The candidate's is about 493.3 ms: an average paired change of about -69.2 ms. But one candidate observation has a delay of 900 ms. With twelve observations, our nearest-rank p95 selects the maximum, so its p95 is 900 ms.

The local comparison blocks despite lower mean delay and cost. We did not collapse these into one weighted score because the exercise defines quality and tail-delay limits separately. A weighted score can be useful for some decisions, but it allows one dimension to compensate for another. That is inappropriate when the owner has declared a condition non-negotiable.

The supplied gate has absolute ceilings. A relative improvement alone cannot satisfy them: a candidate faster than an unusably slow baseline may still be too slow. Conversely, a small cost increase could be acceptable if it remains within a justified budget and materially improves the task. The owner's constraints and failure consequences should determine the rule before the result is inspected.

Decide what more evidence would mean

Our table offers no confidence interval or significance test. Its descriptive range and distinct-case count are intentionally visible. A future real study might predeclare cases, repeated runs, pairing conditions, primary metrics, critical failure rules, and a stopping rule before execution. It should also preserve failures and avoid repeatedly adding trials until a preferred version happens to win.

A useful teaching stopping rule is: analyze the fixed three repeats per case, block if any declared hard condition fails, and investigate failures before proposing a new experiment version. It is not a universally sufficient sample size. Domain review, representative cases, and realistic operating conditions would still be missing.

The Google monitoring chapter explains why tail behavior can matter even when an average looks healthy. Our local percentile example is an illustration of that concern, not evidence about a deployed system.

Practice

1. Compute the paired quality delta and explain its denominator. State why twelve observations do not imply twelve independent users.
2. Explain how the candidate can have a lower mean delay but fail its p95 ceiling. Which single supplied observation creates the p95 problem?
3. In a copied list, remove the final repeat for h4, then call paired_comparison(rows, opslab.load()["trial_plan"]). Also try keeping only trial 1. Explain why both collections are rejected.
4. Write a predeclared comparison plan for a hypothetical authorized future run. Include a sampling unit, measurement boundary, quality floor, tail-delay limit, and stopping rule. Identify which values still require stakeholder evidence instead of borrowing our exercise numbers.

Your exit artifact is a comparison note that preserves quality, cost, delay, and uncertainty separately. Explained solutions.