Unit 10.00: Why a straight line fails on a 0/1 outcome

A straight line has no way of knowing that a probability cannot be negative.

Why linear regression fails on a 0/1 outcome

A binary outcome takes two values, and what you want to model is the probability of one of them. Probabilities live in [0, 1]; a straight line does not, so smf.ols on a 0/1 column will happily predict −0.5.

That is not merely inelegant. Predictions outside the range are meaningless, and the model's error structure is wrong throughout, so the standard errors cannot be trusted either.

Logistic regression models the log-odds instead — a quantity running from minus infinity to plus infinity, which can therefore be a linear function of the predictors — and converts back to a probability at the end.

This block fits a linear model to a churn indicator and looks at what it predicts.

import numpy as np, pandas as pd
import statsmodels.formula.api as smf

rng = np.random.default_rng(701)
n = 400
d = pd.DataFrame({"tenure": rng.uniform(0, 48, n)})
p_true = 1 / (1 + np.exp(-(1.2 - 0.06 * d.tenure)))
d["churn"] = rng.binomial(1, p_true)

print(f"Outcome values   : {sorted(int(v) for v in d.churn.unique())}")
print(f"Overall churn rate: {d.churn.mean():.4f}\n")

# Why a straight line is the wrong model for a 0/1 outcome.
linear = smf.ols("churn ~ tenure", data=d).fit()
grid = pd.DataFrame({"tenure": [0, 24, 60, 90]})
preds = linear.predict(grid)
print("Linear model predictions at tenure 0, 24, 60, 90:")
print(" ", np.round(preds.values, 3))
print(f"Predictions outside [0, 1]: {((preds < 0) | (preds > 1)).sum()} of 4\n")

print("A probability cannot be negative or exceed 1, and a straight line has")
print("no way of knowing that. Logistic regression models the LOG-ODDS, which")
print("is unbounded, then converts back to a probability.")

The overall churn rate is 0.4225. The linear model predicts 0.757 at tenure 0, 0.419 at 24, and then −0.088 and −0.510 at 60 and 90 months. 2 of 4 predictions are impossible, and nothing in the fit objected.

The mistake this prevents

The mistake is running smf.ols on a 0/1 outcome because it is convenient. It produces a number for every case, some of which cannot be probabilities, and confidence intervals that do not mean what they say.

Takeaway

Use smf.logit for binary outcomes. Check that any predicted probability lies in [0, 1] — if it does not, the model is the wrong kind.

Mark complete