Unit 02.05: Six checks, before any test

Run the checks before the test, not after the result surprises you.

Six checks, every time

The analysis table can be wrong in ways no test will complain about: duplicated units, a group column left as object, a group too small for the method, more missingness than you realised, an impossible value dragging a mean.

Encoding these as a dictionary of boolean checks makes them cheap enough to run every time and produces a record that they were run.

A failed check is not a line to comment out. It is a question for whoever produced the data, and answering it usually changes the analysis.

This block runs six checks against a table with several problems.

import numpy as np, pandas as pd

analysis = pd.DataFrame({
    "delivery_id": [1, 2, 3, 4, 5, 6, 7, 8, 9, 9],
    "plan": pd.Categorical(["old"] * 4 + ["new"] * 6),
    "minutes": [34, 41, 29, 38, 31, 27, 30, np.nan, 36, 36],
})

checks = {
    "One row per unit":
        not analysis.delivery_id.duplicated().any(),
    "Outcome is numeric":
        pd.api.types.is_numeric_dtype(analysis.minutes),
    "Group is categorical":
        isinstance(analysis.plan.dtype, pd.CategoricalDtype),
    "Both groups have >= 5 rows":
        bool((analysis.plan.value_counts() >= 5).all()),
    "Missingness under 10%":
        analysis.minutes.isna().mean() < 0.10,
    "No impossible outcome":
        bool((analysis.minutes.dropna() > 0).all()),
}
for name, ok in checks.items():
    print(f"[{'x' if ok else ' '}] {name}")

failed = [n for n, ok in checks.items() if not ok]
print(f"\nPassed: {sum(checks.values())} of {len(checks)}")
print("Failed:", "; ".join(failed))
print("Duplicated delivery_id:",
      analysis.loc[analysis.delivery_id.duplicated(), 'delivery_id'].tolist())
print("\nNone of these would have stopped a t-test from returning an answer.")

3 of 6 pass. Three fail: delivery_id 9 appears twice, one group has fewer than 5 rows, and missingness exceeds 10%. None of these would have stopped ttest_ind from returning a confident answer, and the duplicated id in particular would have counted one delivery twice.

The mistake this prevents

The mistake is treating a failed check as an obstacle to the analysis. It is information about the data, and the right response is to ask about it rather than to relax the check.

Takeaway

Run the check dictionary before every analysis and record which passed. Treat each failure as a question for the data owner, and re-run after any change to the table.

Mark complete