Unit 01.06: Rules, with their reason and their cost

Any exclusion that lives in your head or in a spreadsheet will change the headline figure without appearing anywhere.

Rules, with their reason and their cost

Analyses accumulate small decisions: drop the test accounts, exclude the straight-liners, ignore the first week while the pilot was starting. Each is defensible; together they can move a result substantially, and made by hand nobody can see them.

Written as code, each becomes a named constant with a comment giving its justification and a printed count of what it removed from each group. That is three lines, and it converts an invisible judgement into a reviewable one.

It also lets a reviewer test how sensitive your conclusion is, by changing the constant.

This block applies one exclusion rule and reports what it cost.

import pandas as pd

responses = pd.DataFrame({
    "site":      ["pilot"] * 6 + ["comparison"] * 6,
    "wellbeing": [62, 71, 68, 74, 55, 90, 58, 55, 60, 57, 62, 5],
    "seconds":   [240, 310, 190, 265, 8, 275, 205, 180, 230, 260, 9, 6],
})

# The rule, its reason and its cost -- all three in the code.
MIN_SECONDS = 15   # responses faster than this are straight-lining, per the plan

kept = responses[responses.seconds >= MIN_SECONDS]
dropped = responses[responses.seconds < MIN_SECONDS]

print(f"Rule: drop responses completed in under {MIN_SECONDS}s (straight-lining)")
print(f"Rows in: {len(responses)}   kept: {len(kept)}   dropped: {len(dropped)}")
print(dropped.groupby("site", observed=True).size().to_string())
print()
for label, frame in [("all rows", responses), ("after the rule", kept)]:
    means = frame.groupby("site", observed=True)["wellbeing"].mean().round(2)
    print(f"{label:15s} pilot {means['pilot']}   comparison {means['comparison']}")
print()
print("The exclusion changed the pilot mean by",
      round(kept[kept.site == 'pilot'].wellbeing.mean()
            - responses[responses.site == 'pilot'].wellbeing.mean(), 2))
print("Written as a rule it is reviewable; done by hand it is invisible.")

The rule drops responses completed in under 15 seconds as straight-lining: 3 of 12 rows, 2 from comparison and 1 from pilot. The effect on the headline is substantial — the pilot mean moves from 70.0 to 73.0 and the comparison mean from 49.5 to 57.5, so the *gap* between them shrinks by five points. The rule may well be right; the point is that a reader can see it, see its cost, and try a different threshold.

The mistake this prevents

The mistake is filtering in a spreadsheet before the data reaches Python. The script then starts from an already-edited file, and nothing in the project records what was removed or why.

Takeaway

Encode every exclusion as a named constant with its reason in a comment, print how many rows it removed from each group, and report the headline figure before and after.

Mark complete