Unit 03.01: Carry n and the standard error; mind the dropped keys

A group mean without its standard error is a ranking presented as a fact — and pandas will quietly drop rows whose group is missing.

Carry n and the standard error, and mind the dropped keys

Grouped summaries invite comparison, and comparison needs uncertainty. The standard error — the standard deviation over the square root of n — says how precisely each group's mean is known, and it varies enormously between groups of different sizes.

A small group will frequently top a ranking by chance alone, so n and se belong in every grouped summary you publish.

The pandas-specific hazard is that groupby drops rows whose grouping key is NaN by default. Rows disappear from the summary without a word, and the totals no longer add up. dropna=False keeps them visible.

This block summarises three depots of very different sizes, then adds a row with a missing key.

import numpy as np, pandas as pd

rng = np.random.default_rng(23)
d = pd.DataFrame({
    "depot": np.repeat(["north", "south", "east"], [40, 35, 6]),
    "minutes": np.concatenate([rng.normal(34, 8, 40),
                               rng.normal(31, 7, 35),
                               rng.normal(45, 15, 6)]),
})

summary = (d.groupby("depot", observed=True)["minutes"]
           .agg(n="size", mean="mean", sd="std")
           .assign(se=lambda x: x["sd"] / np.sqrt(x["n"]))
           .round(2)
           .sort_values("mean", ascending=False))
print(summary.to_string())

east, north = summary.loc["east"], summary.loc["north"]
print(f"\nEast has the highest mean and the smallest n ({int(east.n)}).")
print(f"Its standard error is {east.se} against {north.se} for north --")
print(f"about {east.se / north.se:.1f} times as uncertain.")
print()
print("A pandas trap worth knowing: groupby drops rows whose KEY is missing.")
with_na = pd.concat([d, pd.DataFrame({"depot": [None], "minutes": [99.0]})])
print(f"  rows in: {len(with_na)}   rows counted by groupby:"
      f" {with_na.groupby('depot', observed=True).size().sum()}")
print("  pass dropna=False to see them:",
      with_na.groupby("depot", observed=True, dropna=False).size().sum())

East tops the table at a mean of 43.91 from 6 observations, with a standard error of 3.06 against north's 1.38 — about 2.2 times as uncertain. Then the key trap: with a missing-depot row added the frame has 82 rows and groupby counts only 81. Passing dropna=False recovers all 82.

The mistake this prevents

The mistake is sorting group means and reporting the top one. The smallest group wins that competition far more often than its size deserves, and if some rows had a missing key they are not in the table at all.

Takeaway

Include n, the standard deviation and the standard error in every grouped summary. Check that the group sizes sum to the row count, and pass dropna=False when a missing key is meaningful.

Mark complete