Unit 05.02: Smoothing, and what it removes

Smoothing removes noise, removes signal, and delays every turning point.

Three costs, all growing with the window

A noisy upward series smoothed at four window widths.

The code reports the residual noise, the points lost and the lag.

import numpy as np

rng = np.random.default_rng(0)
signal = np.linspace(100, 130, 40)
noisy = signal + rng.normal(0, 6, 40)


def moving_average(x, window):
    return np.convolve(x, np.ones(window) / window, mode="valid")


print(f"{'window':>7} {'points kept':>12} {'std of residual':>17} {'lag':>6}")
for w in (1, 3, 7, 15):
    smoothed = moving_average(noisy, w)
    residual = noisy[w - 1:] - smoothed
    print(f"{w:>7} {len(smoothed):>12} {residual.std():>17.2f} {(w - 1) // 2:>6}")

print("""
Each widening removes more noise, loses more points at the ends, and lags
further behind a turning point. A 15-point average shows a change seven points
after it happened, which for a monthly series is over half a year.

Smoothing is a claim that the removed variation did not matter. Say so.
""")

Each widening reduces the residual, loses more points at both ends, and lags further behind a change. A fifteen-point average shows a turning point seven periods after it happened - for monthly data, over half a year.

That lag is the cost people do not budget for. A smoothed chart used for monitoring will report a problem long after someone could have acted on it.

The mistake this prevents

The mistake is smoothing to make a chart look tidier. Smoothing asserts that the removed variation did not matter, which is a claim about the data - so state the window and why you chose it.

Takeaway

Smoothing trades noise for lag and lost endpoints. State the window, and do not use a smoothed series for anything time-critical.

Mark complete