Error analysis and interpretation limits

Turn mistakes into a question

A single error score compresses a collection of jobs into one number. That helps comparisons but hides where a model is fragile. Error analysis restores context: which observations were difficult, which patterns repeat, and which explanations remain guesses?

Your statistics course distinguished an observed relationship from a causal effect. Preserve that distinction here. A predictor can use queue size to anticipate a long duration without proving that changing the recorded queue size would shorten a job. Likewise, a depot with larger errors might have harder jobs, fewer observations or different recording practices. Prediction performance is not an assessment of staff quality.

Use development validation for investigation. These rows already support calibration and threshold decisions, so descriptions of their errors are exploratory. Repeatedly adjusting a model to these same cases makes its validation result increasingly optimistic. Keep the final 300 jobs closed until the complete procedure and reporting plan are frozen.

Worked trace: preserve the sign and the unit

Consider a miniature regression example with actual durations 4, 6, 8 and 10 hours, and predicted durations 5, 5, 7 and 7. Define residual as actual minus predicted. The residuals are −1, 1, 1 and 3 hours. Positive residuals mean the model underpredicted duration; negative residuals mean it overpredicted.

Absolute errors are 1, 1, 1 and 3. Mean absolute error is their sum, six, divided by four: 1.5 hours. Mean residual is four divided by four: 1.0 hour. Both are useful. A near-zero mean residual could conceal equally large positive and negative mistakes, whereas MAE does not allow cancellation. MAE definition

Suppose the first two jobs are standard and the last two express. Standard MAE is 1.0 hour; express MAE is 2.0 hours. The overall 1.5 lies between them because the groups are equally sized. This miniature result suggests a question about express jobs; it does not prove a stable group difference from two examples each.

An error table should retain the row identifier, intake time, permitted explanatory inputs, actual outcome, prediction and error. Outcomes belong in this audit table, never the intake feature matrix. When joining tables, remember: an investigation can accidentally turn an explanation recorded afterward into a future predictor.

Look at slices without hunting for a dramatic story

Start with a short list of operationally meaningful slices: priority, zone, depot, queue bands and time blocks. Record counts and the relevant denominator beside each metric. For classification, recall divides true positives by all actually late jobs, while precision divides true positives by all alerts. An empty denominator is undefined, not evidence of perfection.

Compare errors against a baseline on the same slice. A model can have worse errors on long-distance jobs while still improving substantially over the baseline there. Conversely, a strong overall result can conceal a slice where the baseline is preferable. Slice selection, minimum evidence requirements and repeated exploratory comparisons belong in the report.

Inspect several large errors and some ordinary cases. Looking only at the most extreme examples makes a vivid story easy to invent. Ask whether the input was available, valid and measured in the right unit; whether a label is correct; whether the case lies outside familiar ranges; and whether the same issue appears elsewhere. Do not delete a difficult but valid job merely because it hurts the score.

Worked trace: what permutation importance measures

Imagine a frozen regression pipeline with validation MAE 2.0 hours. Shuffle queued_jobs across these validation rows, keep the model fixed and recompute predictions. Suppose MAE becomes 2.7 hours. The loss increase is 0.7 hours. Under the neg_mean_absolute_error scorer, baseline score is −2.0 and shuffled score is −2.7, so score decrease is also 0.7.

Repeat the shuffle to show sensitivity to random rearrangements. Those repeats describe the permutation procedure on this dataset; they are not independent samples of future jobs or a confidence interval for a causal effect. Compute on raw permitted columns passed through the full pipeline so the object being shuffled remains understandable. Permutation-importance API

The practical interpretation is: this fitted model's measured performance depended on that column in this evaluation setting. Importance is conditional on the model, data and chosen score. A poor model's low importance is weak evidence that a variable lacks predictive information. Strongly correlated inputs can substitute for one another, hiding their shared importance. Permutation-importance limitations

Explain cautiously, then propose a test

Depot and zone might be closely related. Shuffling only one leaves a substitute available; shuffling it can also create combinations rarely seen in real work. Ordinary row permutations may disturb temporal relationships. Treat the course output as a diagnostic demonstration, not a complete dependence-aware inference method.

A fitted coefficient is also conditional on preprocessing and other variables. A coefficient for standardized distance concerns a one-standard-deviation change in that transformed input, not automatically one kilometre. Correlated features, regularization and omitted variables change coefficients. A tree split is a prediction rule, not evidence of a natural causal threshold.

Use language proportionate to the evidence: “On this validation period, the model underpredicted several high-queue express jobs.” Follow with a checkable proposal: “Audit queue snapshots and compare a predeclared alternative using training-only folds.” Avoid “Express priority causes delays” or “Changing priority will fix the model.” A proposed feature must exist at intake, as promised_hours does; final_delay_reason, label_available_at and close_cost remain forbidden predictors.

Changing a model after error analysis is legitimate development. Record the change and rerun the development protocol. If the final test has already been examined, label further work as a new iteration requiring fresh untouched evaluation evidence. The same old final score cannot certify the revised procedure.

Run and write the finding

Run python examples/09_errors.py and inspect outputs/reference_results.json for measured results. The script shuffles the logistic classifier using negative log loss; its importance units are log loss, unlike the miniature hours example. Check residual signs separately. Draft a short finding with the population, period, slice count, metric, baseline comparison and limitation. Include a next investigation that can actually distinguish competing explanations.

Independent practice

Exercise 09.1 · Reconcile a slice report

Four miniature jobs have actual durations 3, 5, 8, 12 and predictions 4, 4, 6, 9. The first three are standard; the last is express. Calculate residuals, overall MAE, mean residual and each priority's MAE. Reconcile the overall MAE using group counts. Explain why the unweighted average of group MAEs is different.

Exercise 09.2 · Interpret a shuffled feature

A validation pipeline has MAE 1.8 hours. Three shuffles of distance produce MAEs 2.1, 2.4 and 2.1. Calculate the mean loss increase. A teammate says, “Cutting distance will reduce duration by that amount.” Rewrite the claim correctly and name two reasons the causal statement is unsupported.

Exercise 09.3 · Design the next investigation

A validation slice has two express jobs, both underpredicted. A teammate proposes dropping express jobs and adding final_delay_reason. Write a six-sentence response covering counts, valid difficult cases, availability, a baseline comparison, a permitted investigation and final-test protection.

Exit checks

• 09.A: What can a mean residual of zero conceal that MAE would expose?
• 09.B: Why might two genuinely useful correlated inputs each receive low permutation importance?