Unit 08.03: The residuals are where the model tells you about itself

The residuals are where the model tells you about itself. R-squared is the least informative number in the output.

Fitted, residual, and what R-squared is not

For each observation the model produces a fitted value and a residual — the gap between what happened and what the line predicted. Residuals always sum to zero by construction, so their *pattern*, not their total, carries the information.

The residual standard deviation says how far a typical point sits from the line, in the outcome's own units. It is more useful than R-squared because it is interpretable.

R-squared is the share of variance in y that the line accounts for. A high value does not mean the model is correct — a badly curved relationship can produce one — and a low value does not mean there is no relationship, only that y varies a great deal around it.

This block reports the coefficients, three individual fits and the variance decomposition.

import numpy as np, pandas as pd
import statsmodels.formula.api as smf

rng = np.random.default_rng(504)
d = pd.DataFrame({"x": rng.uniform(0, 20, 60)})
d["y"] = 5 + 2.2 * d["x"] + rng.normal(0, 6, 60)
model = smf.ols("y ~ x", data=d).fit()

print(f"Intercept: {model.params['Intercept']:.3f}")
print(f"Slope    : {model.params['x']:.3f}\n")

print("First three observations:")
for i in range(3):
    print(f"  x = {d.x[i]:5.2f}   actual {d.y[i]:6.2f}"
          f"   fitted {model.fittedvalues[i]:6.2f}"
          f"   residual {model.resid[i]:+6.2f}")

print(f"\nResiduals sum to (essentially) zero: {model.resid.sum():.2e}")
print(f"Residual SD (sigma)               : {np.sqrt(model.scale):.2f}")
print(f"R-squared                         : {model.rsquared:.4f}")
print(f"  = 1 - {model.resid.var(ddof=1) / d.y.var(ddof=1):.4f}"
      "  (the share of variance the residuals still hold)\n")
print("A high R-squared does not mean the model is right, and a low one does")
print("not mean the relationship is absent -- only that y varies a lot")
print("around it. The residual SD is the number in the outcome's own units.")

The line is 6.011 + 2.238x. The first three observations show residuals of −7.85, +7.38 and −0.61 — the first point sits nearly eight units below the line. The residuals sum to −8.65e-13, zero to rounding, and their standard deviation is 5.48 in the units of y. R-squared is 0.8538, confirmed as 1 − 0.1462, the share of variance the residuals still hold.

The mistake this prevents

The mistake is judging a model by R-squared alone. It rises whenever the outcome has more variance to explain, and it says nothing about whether the line is the right shape.

Takeaway

Read the residual standard deviation as the practical accuracy of the model, and inspect residual patterns rather than their total. Treat R-squared as one summary among several, never as a verdict.

Mark complete