Compare regression rules and inspect their mistakes

A model family is a restriction on possible rules

Regression predicts a numerical quantity. In our case, a prediction of 7.4 means hours from intake to completion, not a probability and not an exact appointment. The model's family determines which patterns it can represent. That choice trades flexibility against data requirements, stability, computation and ease of inspection.

A linear model combines an intercept with weighted inputs. In readable notation, predicted duration equals intercept plus coefficient one times feature one, plus coefficient two times feature two, and so on. After one-hot encoding, categorical levels contribute indicator terms. This is linear in the transformed columns; it need not be linear in an original variable if you deliberately construct nonlinear features.

Ridge regression fits such a model while penalizing large squared coefficients. Its objective combines squared training error with alpha times the sum of squared coefficients. Alpha controls the strength of that penalty. Larger alpha generally shrinks coefficients more, though individual predictions need not change monotonically. Scaling matters because the penalty acts on numerical coefficient sizes. Ridge's official 1.8 documentation describes this objective.

A decision tree asks a sequence of threshold questions, such as whether queue size exceeds a value. Each leaf predicts from training jobs reaching it. A deep tree can memorize small peculiarities. A random forest averages many randomized trees, reducing some instability while retaining nonlinear splits and interactions. It is not guaranteed to win, and its extrapolation outside familiar ranges differs from a linear model's. The lab compares a median baseline, Ridge and a bounded random forest on the same temporal folds.

Worked miniature: calculate and criticize a prediction

Use this invented rule only for hand tracing: predicted hours equals 1 + 0.10 times distance + 0.40 times queued jobs. A job with distance 20 and queue 5 receives 1 + 2 + 2 = 5 hours. If it actually takes 7 hours, its residual, defined here as actual minus predicted, is +2 hours. Positive residuals mean underprediction; negative residuals mean overprediction. Always state your sign convention because plotting libraries and colleagues may use the opposite one.

Suppose three separate evaluation jobs have actual durations [3, 7, 10] and predictions [4, 5, 9]. Residuals are [−1, 2, 1]. MAE is (1 + 2 + 1)/3 = 4/3 hours. RMSE is the square root of (1 + 4 + 1)/3, about 1.414 hours. The mean residual is 2/3 hour, suggesting underprediction on this tiny set. That bias summary does not reveal how dispersed the individual errors are.

Now compare two four-job error patterns: A has absolute errors [1, 1, 1, 5], and B has [2, 2, 2, 2]. Both have MAE 2. A's RMSE is the square root of 7, about 2.646; B's is 2. RMSE penalizes concentrated large errors more strongly. Neither metric knows the actual staffing consequences. A declared practical requirement might care about the frequency of underprediction above three hours, which requires its own reported count.

Residuals make averages inspectable

Plot actual against predicted with a clearly labeled identity line. Plot residual against predicted duration, queue size and time. A curve can suggest a missing nonlinear relationship. Increasing residual spread may show that long jobs are harder to predict. Consecutive positive residuals late in time can indicate a changing process. These are diagnostic clues, not automatic diagnoses or proof that adding a feature will fix the issue.

NIST's model-validation discussion motivates examining residual patterns rather than relying on one goodness-of-fit statistic. For this course, plot patterns are especially useful because a single average can hide the synthetic process shift. Avoid classical uncertainty claims that assume independent, identically distributed errors when the plot or operating context suggests otherwise.

R-squared compares squared prediction error to predicting the evaluation sample's mean: one minus residual sum of squares divided by total squared deviation around that mean. It is unitless, can be negative on held-out data and becomes problematic for constant targets. A negative value means worse squared error than that comparison reference. R-squared is not the percentage of predictions that are correct and does not establish causality. MAE and RMSE retain the hours useful to the depot question.

Selection discipline matters more than a leaderboard

The course selects the regression family with the lowest mean MAE across four training-only temporal folds. It also records RMSE, rather than switching the winner according to whichever number looks attractive afterward. A small difference with substantial fold variation deserves caution. Later folds contain more training data and different time periods, so fold standard deviation is not a ready-made confidence interval for future performance.

Keep candidate complexity modest. The reference forest has a finite tree count, depth limit and minimum leaf size; it is an instructional CPU model, not an exhaustive search. A more flexible model may improve training scores while harming future scores. A simpler model may underfit an important interaction, such as queue effects becoming stronger under congestion. Inspect evidence before turning either possibility into a story.

Run python examples/05_regression.py. Locate each fold's MAE and RMSE in outputs/reference_results.json after the workflow is generated. Write the selected family, the rule used to select it and any inconsistent fold ordering. Then trace two predictions back to raw feature records and their residuals. Use actual output values, not the worked miniature. A plot copied from reference outputs is evidence about the reference run, not proof that your own implementation works.

Independent practice

1. 05.1 · Calculate an error report. Actual durations are [2, 6, 8, 12]; predictions are [3, 4, 9, 10]. Calculate signed residuals, MAE, RMSE and mean residual. Explain what each reveals and one thing each omits.
2. 05.2 · Choose a defensible next experiment. Ridge underpredicts only when queues are high; a forest has lower training error but worse later-fold MAE. Propose one limited training-only experiment and a comparison plan. Do not assume either model is correct.
3. 05.3 · Audit a model claim. Rewrite: “Our R-squared is 0.8, so 80% of jobs will be predicted correctly and increasing queue size causes longer jobs.” Separate predictive fit, individual error and causal interpretation.

Exit checks

• 05.A: Why can two models have equal MAE but different RMSE?
• 05.B: Why must the full preprocessing pipeline, rather than only the regressor, enter each temporal fold?