Unit 02.03: duplicated() answers a different question

Three deliveries by one driver are not three independent observations, and duplicated() will never tell you so.

Independence is an assumption you can check

Nearly every standard test assumes independent observations. Repeated measurements break that: two deliveries by the same driver are more alike than two by different drivers, so they carry less information than their count suggests.

The consequence is one-directional and therefore dangerous. Counting dependent rows as independent inflates n, shrinks the standard error, narrows the interval and lowers the p-value. It always makes a result look stronger.

duplicated() finds rows identical in every column, which repeated measurements are not — they are different values from the same unit. The check you need is nunique() on the unit identifier.

This block analyses the same deliveries both ways.

import numpy as np, pandas as pd
from scipy import stats

d = pd.DataFrame({
    "driver_id": [1, 1, 1, 2, 3, 3, 4, 5],
    "minutes":   [34, 41, 37, 28, 45, 43, 31, 29],
})

print(f"Rows: {len(d)}   distinct drivers: {d.driver_id.nunique()}")
print(f"Exact duplicate rows: {d.duplicated().sum()}\n")

def ci(x):
    res = stats.ttest_1samp(x, popmean=35)
    lo, hi = res.confidence_interval()
    return lo, hi

lo1, hi1 = ci(d.minutes)
per_driver = d.groupby("driver_id", observed=True)["minutes"].mean()
lo2, hi2 = ci(per_driver)

print(f"Every row independent : n = {len(d)}   CI [{lo1:.1f}, {hi1:.1f}]"
      f"   width {hi1 - lo1:.2f}")
print(f"One row per driver    : n = {len(per_driver)}   CI [{lo2:.1f}, {hi2:.1f}]"
      f"   width {hi2 - lo2:.2f}")
print()
print("Three deliveries by one driver are not three independent observations.")
print("Treating them as such inflates n, shrinks the standard error and")
print("narrows the interval. It always strengthens the result, never weakens it.")

Eight rows come from 5 drivers, with 0 exactly duplicated rows — so a duplicate check finds nothing at all. Treating every row as independent gives n = 8 and an interval of width 10.91. Aggregating to one row per driver gives n = 5 and a width of 16.71, more than half as wide again. Both used every observation; only the second respects where they came from.

The mistake this prevents

The mistake is checking df.duplicated().sum(), finding zero, and concluding the rows are independent. It answers a different question entirely.

Takeaway

Compare len(df) with df[unit].nunique() on every analysis table. Aggregate to one row per unit before testing, and report the number of units rather than the number of rows.

Mark complete