Answers and explanations · Compare quality, cost, and delay honestly

1. Baseline 12/12, candidate 10/12, paired delta (10-12)/12 = -1/6, about -16.7 percentage points. Each row is a matched case/trial observation. Four cases repeated three times create dependence and do not represent twelve independently sampled users. No production estimate follows from these invented values.
2. Most candidate observations are faster, lowering the mean to about 493.3 ms from 562.5 ms. The h4, trial 3 candidate delay is 900 ms, and nearest-rank p95 on twelve values selects that maximum. The 700 ms ceiling still blocks. A faster average does not logically constrain the maximum below 700.
3. The remaining h4 trial IDs differ from those for the other cases. Keeping only trial 1 is balanced among survivors but still violates the declared three-repeat plan. Both raise InputError. Quiet omission could remove a difficult observation and flatter quality and latency. Keep incomplete evidence visible and repair the record before interpreting a score.

import copy
import opslab
rows = copy.deepcopy(opslab.load()["trials"])
rows.pop()
try:
    opslab.paired_comparison(rows, opslab.load()["trial_plan"])
except ValueError as error:
    print(error)

4. A defensible plan declares a complete task/conversation as its unit when context matters, uses the same sampled tasks for both versions, records separate repeats, measures submission-to-usable-response including retries, and sets quality/critical/tail limits before running. A stopping rule should say when analysis occurs and what blocks continuation. The real floors, ceilings, sample size, case mix, and permissible failure tradeoffs need task-specific evidence and an accountable owner; our 0.90 and 700 ms are not ready-made production targets.

Self-check: do not replace these limits with “the candidate is cheaper.” A complete decision explains why cost improvement does or does not meet the actual constraints.