Answers and explanations · Error analysis and interpretation limits

Lesson 09 · Error analysis

Exercise 09.1

Residuals, defined as actual minus predicted, are −1, 1, 2 and 3 hours. Absolute errors are 1, 1, 2 and 3; overall MAE is 7/4 = 1.75 hours. Mean residual is 5/4 = 1.25 hours, indicating average underprediction in this tiny sample.

Standard-priority MAE is (1 plus 1 plus 2)/3 = 4/3, approximately 1.333 hours, across three jobs. Express MAE is 3/1 = 3 hours, across one job. Count-weighted reconciliation is ((3 times 4/3) plus (1 times 3))/4 = 7/4 = 1.75.

The unweighted mean of group MAEs is (4/3 plus 3)/2 = 13/6, approximately 2.167 hours. It gives the one express job's group as much weight as the three-job standard group, answering a different question. A one-job express result is especially weak evidence for a stable priority difference.

Exercise 09.2

The three loss increases are 0.3, 0.6 and 0.3 hours. Their mean is 0.4 hours; equivalently, shuffled mean MAE 2.2 minus baseline MAE 1.8 equals 0.4.

A supported claim is: “For this fitted pipeline and validation sample, shuffling distance increased MAE by an average 0.4 hours across three permutations.” This is a change in prediction error, not an estimated change in actual duration. No intervention on distance occurred, and other features or unmeasured conditions may account for its associations. Shuffling can also create unrealistic combinations, while correlated inputs can substitute for distance. Neither repeating the shuffle nor obtaining a positive importance establishes causality.

Exercise 09.3

One acceptable six-sentence response is:

“Two express jobs are too few to establish a stable slice-level weakness, so show their count and inspect more relevant evidence. Keep valid difficult jobs in the target population rather than removing them to improve the score. Final delay reason is unavailable at intake and must not become a prediction input. Compare the model's errors with the baseline on those same jobs before judging whether the model adds value there. Audit the intake queue snapshots, units and label quality, then evaluate any permitted change through the declared development protocol. Keep the final test closed until the revised procedure and interpretation plan are frozen.”

Other wording is acceptable if it covers all six required ideas without inventing a causal explanation or changing the target population silently.

Exit check 09.A

Positive and negative residuals can cancel. For example, residuals −5 and +5 have mean zero but MAE five hours. A zero signed mean does not mean accurate individual predictions; it only rules out net signed error in the observed average.

Exit check 09.B

Each input can supply information similar to the other. Shuffling one may leave the model able to rely on its substitute, so measured deterioration is small even though their shared signal is useful. Importance is model- and dataset-dependent; low individual values are not proof that both inputs are unnecessary.