Unit 10.02: Comparing two versions on the same set

The release decision is the broken list. The total is a summary of it.

Same score, different system

Per-case comparison between two versions over ten cases.

The code computes the fixed and broken lists.

A = {f"q{i}": r for i, r in enumerate(
    [1, 1, 0, 1, 1, 1, 0, 1, 1, 1], start=1)}
B_ = {f"q{i}": r for i, r in enumerate(
    [1, 0, 1, 1, 1, 1, 1, 1, 0, 1], start=1)}

fixed = sorted(q for q in A if not A[q] and B_[q])
broke = sorted(q for q in A if A[q] and not B_[q])

print(f"version A: {sum(A.values())}/{len(A)}")
print(f"version B: {sum(B_.values())}/{len(B_)}")
print(f"fixed    : {fixed}")
print(f"broke    : {broke}")
print(f"\nheadline change: {sum(B_.values()) - sum(A.values()):+d}")
print("per case: 2 fixed, 2 broken -- a different system, same score")

# Shipping on the headline number ships a change that broke two cases that used
# to work. The broke list is the release decision; the total is a summary of it.

Both versions score eight out of ten and the headline change is zero. Two cases were fixed and two were broken, which is a materially different system behind an identical number.

Shipping on the headline ships a change that broke two cases that used to work. Whether that is acceptable depends on which cases - and you cannot have that conversation without the list.

The mistake this prevents

The mistake is storing only the aggregate per run. It is one boolean per case per run, it cannot be reconstructed afterwards because the old code is gone, and without it the broken list does not exist.

Takeaway

Store per-case results and compare version to version. The broken list is the release decision; an unchanged total can hide a completely different system.

Mark complete