Unit 07.01: Fitting a line, and whether it earns its place

A fitted line is a claim, and it should appear only with the evidence for it.

How much is left after the line

A moderately correlated scatter with a fitted line and its residuals.

The code reports r-squared and the remaining spread.

import numpy as np

rng = np.random.default_rng(6)
x = rng.uniform(0, 100, 120)
y = x * 0.9 + rng.normal(0, 25, 120)

r = np.corrcoef(x, y)[0, 1]
slope, intercept = np.polyfit(x, y, 1)
residuals = y - (slope * x + intercept)

print(f"correlation           {r:.2f}")
print(f"r-squared             {r ** 2:.2f}   <- {r ** 2:.0%} of variance explained")
print(f"slope                 {slope:.2f} per unit")
print(f"residual std          {residuals.std():.1f}")
print(f"spread of y           {y.std():.1f}")

print("""
The line explains about half the variance, so half the vertical spread remains
after fitting it. Drawn without the points, that line asserts a precision the
data does not support.

Draw the line only when the points are also visible, and report r-squared next
to it.
""")

The line explains about half the variance, so roughly half the vertical spread survives the fit. Drawn alone, without the points, that line asserts a precision the data does not support.

Reporting r-squared next to the line is the cheap fix. It tells the reader immediately whether the line is a description of a tight relationship or of a wide cloud.

The mistake this prevents

The mistake is drawing the trend line and hiding the points because the cloud looks messy. The messiness is the finding - a clean line over a hidden cloud is a chart that misrepresents its own evidence.

Takeaway

Draw the line only with the points, and report r-squared beside it. Half the spread can survive a respectable-looking fit.

Mark complete