Follow one complete prediction investigation

The deliverable is an evidence chain

A complete applied-ML investigation connects a question to a defensible decision. It does not stop when a model predicts successfully. In this lesson, follow the reference depot investigation from intake contract to a frozen evaluation plan. The examples combine concepts already introduced; no new algorithm rescues a weak information boundary.

The request is narrow: estimate duration and lateness for future accepted jobs at the represented fictional depots, using information available at intake. The intended classification action is a human review queue. The review-cost ratio and capacity used here are instructional assumptions: one cost unit per false alert, four per missed late job, and at most 40% of policy-period jobs flagged. They are not actual money, approved operational limits or evidence that review prevents delays.

Worked miniature: connect scores to an operational statement

Start with six invented policy jobs. Their late labels are [0, 1, 0, 1, 1, 0], and calibrated probabilities are [0.10, 0.35, 0.45, 0.60, 0.80, 0.90]. At threshold 0.50, jobs four, five and six are flagged. That gives two true positives, one false positive and one false negative. Cost is 1 + 4 times 1 = 5, but three alerts out of six is 50%, exceeding the 40% limit.

At threshold 0.70, only jobs five and six are flagged. There is one true positive, one false positive and two false negatives. Cost is 1 + 4 times 2 = 9. Two alerts out of six is approximately 33.3%, so this candidate is feasible. The higher cost does not justify silently discarding the capacity restriction. A valid choice is the best feasible candidate under the complete predeclared rule, not the unconstrained minimum.

A threshold between 0.60 and 0.80 produces the same two alerts in this miniature, depending on the inclusive boundary. If costs and alert counts tie, the course chooses the higher candidate threshold. For p exactly equal to the chosen threshold, the job is flagged because the rule is p greater than or equal to threshold. This small detail belongs in implementation tests.

Now imagine the next six jobs produce three alerts under the frozen threshold. The threshold was feasible on development data but is not guaranteed to satisfy capacity on every future batch. The honest report says the observed future alert fraction exceeded the development limit. It does not raise the threshold using those newly revealed answers and call the revised score an untouched test result. This is an integrated miniature, not the course's actual final outcome.

Trace the reference workflow

First, the data audit preserves job IDs and verifies chronology, uniqueness, target construction and label availability. label_available_at represents completion and label availability at the same instant in this synthetic dataset. Real systems may have a reporting delay; here the simultaneous timestamps are an explicit simplifying assumption. Duration is bounded at 23.5 hours, while the outer embargoes are full days.

Second, training consists of 894 rows through day 148. Four forward-looking training folds, each with a 12-row gap, compare full pipelines. The regression choice minimizes mean fold MAE among the median baseline, Ridge and random forest. The classification comparison records prior, logistic and forest behavior. The integration intentionally fixes the logistic family and tunes its C and class-weight choices with temporal-CV log loss. Family choice and parameter selection are separately stated rather than hidden in a single winner label.

Third, the selected logistic pipeline is refitted on training and frozen. The calibration segment contains 144 jobs on days 150–173. A sigmoid probability adjustment is learned there without refitting the base pipeline. After the day-174 gap, another 144 jobs on days 175–198 support threshold selection. The candidate grid is 0.10 through 0.90 in increments of 0.05. If no candidate satisfies capacity, the explicit threshold 1.01 creates no alerts. The fallback is a documented abstaining policy, not an ordinary probability cutoff.

Fourth, the report records raw and calibrated probabilities, candidate thresholds, confusion counts, regression residuals, zone summaries and descriptive permutation importance. These analyses use development information. They can reveal limitations, but they do not license a new feature search followed by claims that the same development segment was untouched. If the scope changes materially, record a new experiment and evaluation plan.

Read mixed evidence without flattening it

Run python -m courseml.run_all and inspect outputs/reference_results.json, outputs/validation_predictions.csv and outputs/frozen_plan.json. The development command does not load the final test partition. Inspect actual numerical outputs and compare them with the code's stated selection rule. An unexpectedly good score should prompt the same audit as an unexpectedly bad one.

Calibration can improve one proper probability score while worsening another on a finite sample. Report both rather than saying calibration helped universally. A zone with higher MAE also needs its sample size and outcome range. A permutation score describes how a fitted model uses a feature under a particular perturbation; it does not identify the effect of changing a depot process. The calibration guide and permutation-importance guide support these distinctions.

Finally, write a freeze record before opening final outcomes. It contains feature names, target definitions, split boundaries, selected models and parameters, calibration method, threshold, tie policy, source and development-data hashes, versions and intended report. Only after that record is complete should the reference final command run: python -m courseml.final_evaluation. That command evaluates the reference procedure; it does not automatically evaluate a learner's alternative pipeline.

Independent practice

1. 11.1 · Audit the miniature policy. For labels [0, 1, 0, 1, 1, 0] and probabilities [0.10, 0.35, 0.45, 0.60, 0.80, 0.90], calculate confusion counts, cost and alert fraction at 0.40 and 0.80. Apply the 40% capacity restriction and explain which of these two candidates is eligible.
2. 11.2 · Trace a real reference result. Using development outputs only, find the selected regression family, logistic parameters and threshold. For each, identify the exact rows and criterion that influenced it. Reconcile the selected policy confusion counts to its sample size.
3. 11.3 · Write the freeze note. Draft a concise freeze record and a final-report outline. Include the action if final capacity fails, how changed code would be handled and why reference results do not grade your own code.

Exit checks

• 11.A: Which stages learn base-model parameters, calibration parameters and the decision threshold, respectively?
• 11.B: Why can a capacity-constrained policy pass development checks and still fail the same operational limit later?