Separate reasoned solutions

Open this after attempting the exercises. The answers model reasoning and evidence, not wording to memorize. All small numerical examples are hand-worked miniatures unless explicitly identified as measured reference results. They are not substitutes for running your own code.

The final section contains reference final-test spoilers. If you inspect them before freezing your own procedure, disclose that exposure and treat the same test as practice evidence. A reference result does not grade a learner's implementation.

Lesson 01 · Prediction contract

Exercise 01.1

An acceptable contract says: “One row represents one accepted synthetic depot job. Predict duration in hours at intake using the eight approved intake measurements, including the original promised hours. The outcome becomes available after completion, with label availability recorded explicitly. The prediction supports a fictional human review/planning study. It does not authorize automatic staffing changes or consequential scoring of people. Evaluate on later jobs using the prescribed chronological protocol and report uncertainty and limitations.”

A mid-job model has a different prediction time, available information and likely remaining-duration target. An observed progress update might be legitimate then while being forbidden at intake. It requires a new contract and evaluation reconstruction, not an extra feature silently added to this model.

Exercise 01.2

The training sum is 30 and there are five values, so the mean predictor is 6 hours. The sorted middle value is 3, so the median predictor is 3 hours. For actual durations 2, 4 and 12, the mean predictor's absolute errors are 4, 2 and 6; MAE is 12/3 = 4 hours. The median predictor's errors are 1, 1 and 9; MAE is 11/3, approximately 3.667 hours.

The long training durations pull the mean upward more than the median. The median minimizes training absolute error among constant predictions, while the mean minimizes training squared error. This fact does not guarantee which constant wins on every future sample. Neither constant may be recalculated using these evaluation outcomes.

Exercise 01.3

An always-on-time baseline obtains 90% accuracy if 10% are late, while catching no delays. Accuracy alone therefore hides a complete failure of the intended late-job review function. Report recall to show the fraction of late jobs found, precision to show the fraction of alerts that are late, and alert count/fraction to quantify the review workload. At least one additional measurement is required by the exercise; a strong answer connects more than one to the actual decision. Probability quality is a separate consideration if predicted probabilities are communicated.

Exit check 01.A

Fitting can use historical labels that were available by its fit boundary. Intake prediction must use only approved intake inputs and the already learned procedure; it cannot require the new job's eventual duration, lateness or post-outcome fields. Historical labels also must have been available in the simulated historical fitting situation.

Exit check 01.B

A model was optimized on training data, so training improvement can include memorization or adaptation to noise. Future jobs may differ. An appropriate held-out design measures performance of the learned rule on observations that did not select it; a baseline makes the improvement interpretable.

Lesson 02 · Targets, provenance and leakage

Exercise 02.1

• queued_jobs: intake queue snapshot; legitimate predictor only if measured or reproducibly reconstructed at intake.
• label_available_at: audit timestamp for the observed outcome; available after completion in the synthetic lab, forbidden in X.
• priority: the original intake choice; permitted predictor. A later escalation status would not satisfy the same contract.
• close_cost: closing measurement influenced by completed work; available after the outcome and forbidden in X.
• job_id: assigned identifier, usually available at intake; retained for identity, joining and duplicate audits, excluded from the declared predictor list.

For example, close cost increases with realized duration in the generator. Using it would let the model reconstruct part of the answer from an unavailable closing value. Merely renaming it “operational value” would not change its availability.

Exercise 02.2

Only Q's label is available at Wednesday 09:00 because it was confirmed Tuesday 21:00. P completed before the fit but its label is not confirmed until Wednesday 10:00, so it cannot be used at that fit boundary. R remains unknown; do not label it on time or zero duration. Preserve P and R in the audit and apply a declared maturity/eligibility policy. Event time and availability time are distinct.

Exercise 02.3

First verify 100 unique intake IDs and inspect status counts per ID. A 260-row join is consistent with one-to-many status history, not 260 independent jobs. Determine which status version was valid and available at each intake; select or reconstruct that snapshot before the job-level join. Preserve later and conflicting versions in a separate audit rather than arbitrarily retaining the first row. Use a left join with relationship validation and a match indicator; inspect unmatched jobs and unexpected status-only IDs. Recheck 100 output jobs, uniqueness and no target-dependent filtering. A restored count without a valid snapshot rule is not enough.

Exit check 02.A

A historical label is intentionally constructed from an eventual outcome so the model can learn from known examples. The input must reconstruct what was available when that outcome was still unknown. One field can therefore be valid for retrospective labeling and invalid for prospective prediction.

Exit check 02.B

A gap constrains row or calendar separation, not actual reporting delay. Some outcomes can take longer or be confirmed later. Label-availability comparisons establish whether every included outcome existed at the relevant fit time; the embargo provides an additional design buffer.

Lesson 03 · Splits

Exercise 03.1

• Training: days 0–148, 149 days, 894 rows
• Outer embargo: day 149, 6 rows
• Calibration: days 150–173, 24 days, 144 rows
• Inner embargo: day 174, 6 rows
• Threshold policy: days 175–198, 24 days, 144 rows
• Outer embargo: day 199, 6 rows
• Final test: days 200–249, 50 days, 300 rows

Total is 894 + 6 + 144 + 6 + 144 + 6 + 300 = 1,500. The three validation sub-blocks reconcile to 294. The number of usable development rows is not the same as the full raw count because embargoed observations are deliberately not fitted or scored in those roles.

Exercise 03.2

(a) Use an earlier-to-later temporal split, with appropriate label-maturity checks, for later jobs at represented depots. It does not independently establish transfer to unrepresented depots.

(b) Hold entire depot groups out for the separate question of unseen-depot performance in comparable historical conditions. It does not by itself test temporal change; the shared time range can still conceal future drift.

(c) Design both restrictions: hold out relevant depot groups and evaluate them in a later, untouched period, with training available strictly earlier and no depot overlap. The design requires enough groups and later observations, and its populations must match the proposed transfer claim. More restrictive splitting can reduce precision; convenience is not a reason to replace the question. The course's actual final temporal holdout remains unchanged.

Exercise 03.3

The Sunday 10:00 label did not exist before the first Sunday 08:00 prediction. A row gap and Friday intake timestamp do not fix this. A documented maturity rule could exclude that unavailable training label, with the resulting selection consequences discussed. Alternatively move the fitting/evaluation boundary later, if that matches the intended use and is set before evaluation. Do not change the label timestamp or pretend the later export was available earlier.

Exit check 03.A

Stratification attempts to preserve class proportions; it does not preserve temporal availability or changing operating conditions. Mixing future rows into training answers a different sampling question, and making period late rates artificially similar can remove part of the deployment challenge.

Exit check 03.B

A depot holdout asks about new-group generalization. A chronological holdout asks about later conditions. The answer requires stating whether the same depots appear on both sides and whether future time is also held out.

Lesson 04 · Preprocessing

Exercise 04.1

Median imputation learns 4, giving queue counts [0, 4, 8]. Their mean is 4, population variance is (16 + 0 + 16)/3 = 32/3, and population scale is approximately 3.266. Standardized values are approximately [−1.225, 0, 1.225]. A new missing count becomes 4 and then 0. A new count of 12 becomes (12 − 4)/3.266, approximately 2.449.

Before scaling, values are counts of queued jobs. After scaling, values are dimensionless deviations measured in training scale units. The course also adds and scales a missingness-indicator column for this training-missing feature. Its pre-scaling training values are [0, 1, 0], mean 1/3 and scale square root of 2/9. Its standardized training values are approximately [−0.707, 1.414, −0.707]. This separate indicator does not alter the calculations for the value column.

Exercise 04.2

The imputer learned numerical statistics from held-out rows; the encoder learned their categories. Additional scaling or feature selection could leak too. Replace the precomputed table with a ColumnTransformer containing numerical and categorical preparation, followed by the classifier inside one Pipeline. Pass the entire unfitted pipeline to temporal cross-validation or grid search. Each fold fits its own preparation on its training rows and only transforms its held-out rows. Merely moving the classifier fit into a loop leaves preprocessing leakage intact.

Exercise 04.3

An unseen zone should follow the declared unknown-category rule, producing a finite prediction through the complete pipeline while logging unfamiliar-category exposure. A valid missing distance should use training-fitted imputation and the stored missingness handling. A missing required column should produce a clear input-contract failure rather than silently redefine the input. A forbidden extra column should never reach the model; the allowlist excludes it, and a schema audit should flag unexpected supplied fields if that is the declared intake interface. Verify behavior through feature names, transformed shape, assertions and recorded prediction/error output, not just absence of a crash.

Exit check 04.A

An all-zero indicator block is a fallback representation. The model did not observe that category's outcomes, so it cannot have learned a category-specific relationship. Successful execution and supported generalization are different claims.

Exit check 04.B

Every learned transformation, any feature-selection procedure and the estimator must be fitted separately on the permitted training portion. Fixed, independently specified schema checks do not learn data statistics, but their definitions still must not be rewritten to exploit held-out results.

Lesson 05 · Regression

Exercise 05.1

Actual minus predicted residuals are [−1, 2, −1, 2]. Absolute errors sum to 6, so MAE is 1.5 hours. Squared errors sum to 10, so RMSE is square root of 2.5, approximately 1.581 hours. Mean residual is 2/4 = 0.5 hour, indicating average underprediction in this small sample.

MAE summarizes average absolute magnitude but does not show direction or extreme-error concentration. RMSE emphasizes larger errors but is less directly representative of a typical absolute mistake. Mean residual reveals signed average bias but can hide cancellation. Individual residuals reveal specific mistakes without alone establishing a population pattern.

Exercise 05.2

One defensible experiment is a predeclared, modest nonlinear queue term in the Ridge pipeline, fitted and compared inside the same training-only temporal folds. Another is reducing forest complexity with a small declared grid. Choose one bounded experiment, retain the baseline and original candidates, and compare mean fold MAE plus fold patterns without using policy or final results to repeatedly search. High-queue underprediction suggests a hypothesis; it does not prove that queue curvature is the cause. The worse later-fold forest result argues against selecting it solely from training fit.

Exercise 05.3

A defensible rewrite is: “On the declared evaluation sample, R-squared was 0.8, meaning squared prediction error was 20% of the squared-error reference based on that sample's mean. This is not an 80% count of correct jobs; MAE, RMSE and residuals describe individual prediction errors. Queue size is associated with predictions under this fitted model and dataset. The study does not establish the effect of intervening on the queue.”

A constant-target sample requires additional care, and any reported score must identify the evaluation period rather than imply a universal result.

Exit check 05.A

Equal sums of absolute errors can have different concentrations. RMSE squares each error before averaging, so a few large errors contribute more than several smaller ones with the same total absolute magnitude.

Exit check 05.B

Imputers, scalers and encoders learn state too. If they see later fold rows, the regressor receives a representation influenced by held-out information even when its own fitting rows are correct.

Lesson 06 · Classification

Exercise 06.1

TP = 8, FN = 2, FP = 6 and TN = 34. The counts sum to 50. Accuracy is (8 + 34)/50 = 84%. Precision is 8/14, approximately 57.14%; recall is 8/10 = 80%. Specificity is 34/40 = 85%, so balanced accuracy is (80% + 85%)/2 = 82.5%. Alert fraction is 14/50 = 28%.

Always predicting on time produces 40/50 = 80% accuracy but zero recall. The model improves accuracy by four percentage points while also finding eight late jobs and requiring fourteen reviews. The workload and error tradeoff should be reported, not compressed into the four-point difference.

Exercise 06.2

A's squared probability errors are 0.04, 0.04 and 0.16; its Brier score is 0.24/3 = 0.08. B's are 0.16, 0.16 and 0.2025; its Brier score is 0.5225/3, approximately 0.17417. A has lower Brier loss on these three examples. Both systems produce hard labels [1, 0, 1] at threshold 0.5, so their thresholded confusion counts are identical. This illustrates information discarded by hard decisions, not population-level proof that A is universally better calibrated.

Exercise 06.3

High AUC supports a ranking claim in the measured population, not trustworthy numerical probabilities. Assess log loss/Brier score and a reliability analysis with bin counts on suitable held-out rows. It also does not specify review burden: choose a declared threshold and report alert count/fraction, confusion counts and capacity behavior on the relevant period. Even a threshold feasible during development may exceed capacity later. All claims need the actual sample and class prevalence.

Exit check 06.A

When on-time jobs dominate, calling every job on time correctly labels most observations. It still finds none of the late jobs, so its positive-class recall is zero.

Exit check 06.B

Ranking concerns ordering; probability quality concerns the numerical estimates relative to outcomes; the decision policy combines probabilities with a threshold, costs and constraints. Improving one does not automatically verify the others.

Lesson 07 · Validation

Exercise 07.1

With 30 rows and two splits, each validation block has 30 divided by 3, or 10 rows.

• Fold 1: training indices 0–6, gap 7–9, validation 10–19.
• Fold 2: training indices 0–16, gap 17–19, validation 20–29.

Training sizes are seven and seventeen, not ten and twenty: subtract the three-row gap. For each fold, every included training outcome must be available by the simulated fit boundary before that fold's validation predictions. Inspect actual availability timestamps rather than just intake order. Feature construction must also respect the intake cutoff. A row gap alone proves neither requirement.

Exercise 07.2

C totals 11.2 hours across the four fold MAEs, giving mean MAE 2.8. D totals 10.4, giving mean MAE 2.6. Their mean negative-MAE scores are −2.8 and −2.6. The maximized negative score and minimized positive error both select D under the declared rule.

C is much better in the earliest folds but deteriorates sharply; D is more stable and has the lower mean. Report that pattern rather than claiming D beats C in every period. The folds overlap in their training evidence, so their variability is not an automatic independent-sample confidence interval. Both candidates still require evaluation of the frozen procedure on untouched future data.

Exercise 07.3

A defensible five-step repair is:

1. Preserve a record that the original encoder used all rows and the original test selected settings. Those original scores do not estimate an untouched-test procedure.
2. Rebuild allowed intake features and explicit targets; keep audit-only label_available_at timestamps outside the predictor. Check availability and data quality.
3. Restore chronological development boundaries. Put preprocessing inside a pipeline and tune only through the specified training-only time folds, checking label maturity at every fit.
4. Use the declared calibration and threshold blocks for their separate roles. Document the fixed model, threshold and reporting plan; label any depot-group experiment as a different unseen-depot question.
5. Report results on the already exposed 300 rows as exploratory or previously used evidence. Obtain a genuinely fresh future holdout for a new final evaluation; changing the split name cannot restore the old holdout's independence.

The main contamination is both learned preprocessing from future data and test-driven model selection. A group split can answer a useful separate question, but choosing it for a better score does not repair the original evaluation.

Exit check 07.A

A pipeline refits learned preprocessing inside each fold, preventing later fold rows from determining transformations. It does not know when an outcome became available. If its supplied training labels were unknown at the simulated fit time, it will still learn from unavailable information. Timestamp audits and appropriate splitting remain necessary.

Exit check 07.B

The score describes the previously frozen procedure on that dataset. After using the result to change the model, that dataset has influenced development; the score cannot certify the revised model as independently evaluated. Preserve the earlier result, disclose the change and use fresh untouched evidence for the next final claim.

Lesson 08 · Calibration and thresholds

Exercise 08.1

Squared probability errors are 0.01, 0.36, 0.36 and 0.01. Their sum is 0.74; binary Brier loss is 0.74 divided by 4, or 0.185.

The low bin has mean prediction (0.10 plus 0.40) divided by 2 = 0.25 and observed event rate 1/2 = 0.50. The high bin has mean prediction (0.60 plus 0.90) divided by 2 = 0.75 and observed event rate 1/2 = 0.50. Each bin has only two observations. The gaps suggest what to examine with more appropriate evidence, but cannot establish population calibration or stable bin-level miscalibration.

Exercise 08.2

At threshold 0.50, alert jobs four, five and six. The counts are TP = 2, FP = 1, FN = 1, TN = 2. Precision is 2/3, recall is 2/3 and alert count is three. Error cost is 1 plus 4 times 1 = 5 units.

At threshold 0.30, alert jobs two through six. The counts are TP = 3, FP = 2, FN = 0, TN = 1. Precision is 3/5, recall is 3/3 = 1 and alert count is five. Error cost is 2 plus 4 times 0 = 2 units.

The lower threshold minimizes the stated error cost among these two options, but violates a three-alert capacity limit. The 0.50 threshold is the feasible candidate of the pair. State that constrained choice and its additional missed event; do not claim the lower-cost option satisfies capacity. These exercise counts differ from the course's percentage-based policy, which must be assessed on its own threshold block.

Exercise 08.3

Keep the base pipeline fitted on days 0–148 using training-only tuning. Fit its frozen-pipeline sigmoid calibrator on days 150–173, 144 jobs. Leave day 174 unused. Choose the threshold on days 175–198, another 144 jobs. Audit that calibration labels are known by the mapping's fit boundary, rather than relying only on the day-174 embargo.

Using all 294 validation labels to fit calibration means that the reported validation probability performance is partly in-sample. Choosing a threshold on the same labels introduces additional selection reuse. Reallocating the roles for a corrected run makes the procedure coherent, but does not erase the fact that the earlier analysis saw the labels. Document that history. The final 300 jobs may be opened only after all decisions and the evaluation plan are frozen, provided those test labels truly remained unused. They can then evaluate the final selected procedure independently of the development history.

Exit check 08.A

Changing a threshold changes the yes/no decisions, confusion counts and alert workload. It leaves the underlying probability estimates unchanged, so it does not improve their calibration. Calibration concerns probabilities versus outcome frequencies, not which side of an action boundary a job occupies.

Exit check 08.B

A low Brier score summarizes overall probability prediction quality; it does not specify the cost of a missed event, false alert, workload constraints or relevant future conditions. The analytic cost threshold also assumes calibrated probabilities, fixed error costs and the specified treatment of correct decisions. A practical policy must state and test these assumptions rather than infer them from one score.

Lesson 09 · Error analysis

Exercise 09.1

Residuals, defined as actual minus predicted, are −1, 1, 2 and 3 hours. Absolute errors are 1, 1, 2 and 3; overall MAE is 7/4 = 1.75 hours. Mean residual is 5/4 = 1.25 hours, indicating average underprediction in this tiny sample.

Standard-priority MAE is (1 plus 1 plus 2)/3 = 4/3, approximately 1.333 hours, across three jobs. Express MAE is 3/1 = 3 hours, across one job. Count-weighted reconciliation is ((3 times 4/3) plus (1 times 3))/4 = 7/4 = 1.75.

The unweighted mean of group MAEs is (4/3 plus 3)/2 = 13/6, approximately 2.167 hours. It gives the one express job's group as much weight as the three-job standard group, answering a different question. A one-job express result is especially weak evidence for a stable priority difference.

Exercise 09.2

The three loss increases are 0.3, 0.6 and 0.3 hours. Their mean is 0.4 hours; equivalently, shuffled mean MAE 2.2 minus baseline MAE 1.8 equals 0.4.

A supported claim is: “For this fitted pipeline and validation sample, shuffling distance increased MAE by an average 0.4 hours across three permutations.” This is a change in prediction error, not an estimated change in actual duration. No intervention on distance occurred, and other features or unmeasured conditions may account for its associations. Shuffling can also create unrealistic combinations, while correlated inputs can substitute for distance. Neither repeating the shuffle nor obtaining a positive importance establishes causality.

Exercise 09.3

One acceptable six-sentence response is:

“Two express jobs are too few to establish a stable slice-level weakness, so show their count and inspect more relevant evidence. Keep valid difficult jobs in the target population rather than removing them to improve the score. Final delay reason is unavailable at intake and must not become a prediction input. Compare the model's errors with the baseline on those same jobs before judging whether the model adds value there. Audit the intake queue snapshots, units and label quality, then evaluate any permitted change through the declared development protocol. Keep the final test closed until the revised procedure and interpretation plan are frozen.”

Other wording is acceptable if it covers all six required ideas without inventing a causal explanation or changing the target population silently.

Exit check 09.A

Positive and negative residuals can cancel. For example, residuals −5 and +5 have mean zero but MAE five hours. A zero signed mean does not mean accurate individual predictions; it only rules out net signed error in the observed average.

Exit check 09.B

Each input can supply information similar to the other. Shuffling one may leave the model able to rely on its substitute, so measured deterioration is small even though their shared signal is useful. Importance is model- and dataset-dependent; low individual values are not proof that both inputs are unnecessary.

Lesson 10 · Reproducibility and handoff

Exercise 10.1

A compact handoff needs:

• Intended use, prohibited uses, synthetic-data status and the person responsible for review.
• Raw input schema, units, allowed categories, missingness policy and intake availability requirements.
• Target definitions, row identifiers, data provenance or generator settings, input hashes and chronological ordering.
• Training, embargo, calibration, threshold and final-test boundaries, with label-maturity checks.
• Complete preprocessing-and-model procedure, selected settings, calibration mapping and threshold including its equality rule.
• Source revision, Python and dependency versions, random settings, execution commands and expected checks.
• Baseline and model metrics, evaluation populations, error-slice counts, limitations and the freeze/test history.

A seed controls particular random operations; it does not reconstruct missing code, input units, row ordering, environment or decision rules. A whole-pipeline smoke test passes a known raw job through preprocessing, prediction, calibration and thresholding and compares the resulting output with a saved expectation. Include a boundary case equal to the threshold and a separate valid missing or unfamiliar-category case according to the declared schema policy. Load persisted model files only from trusted, verified sources in an appropriate environment; do not load an unknown pickle merely to inspect it.

Exercise 10.2

Reference missingness is 8/200 = 0.04, or 4%. New missingness is 18/120 = 0.15, or 15%. The increase is 11 percentage points, and the relative rate is 15/4 = 3.75 times the reference rate. That relative ratio is not an 11% relative increase.

Among mature labels, the observed late fraction is 30/75 = 0.40, or 40%. Label coverage is 75/120 = 62.5%. First, the remaining 45 outcomes are unknown, not on-time cases. Second, the mature-label subset might differ systematically from pending jobs, especially if long durations delay label availability. The result does not establish the whole new cohort's late rate. Input missingness also does not by itself establish a change in model error.

Exercise 10.3

Preserve the original frozen procedure and its final evaluation. Tuning a threshold using those 300 labels would create a new development iteration; do not replace the earlier report or label the tuned result independent. If a revision is worthwhile, document its motivation, revise through a declared development process and obtain new untouched future evidence for its final assessment.

Unfamiliar depots are a separate schema and population question. Check whether they are genuine new depots or malformed identifiers, whether the declared encoder handles them and whether the intended-use population includes them. Report affected counts and test the existing unknown-category behavior without changing the policy silently. Escalate any unsupported cases to the stated review or fallback process. Neither an unfamiliar depot nor a test disappointment authorizes automatic retraining or threshold changes.

Exit check 10.A

Input drift concerns the distribution of observed predictors. Performance concerns errors against mature outcomes. The model may remain accurate under some input changes, or deteriorate because the input-to-outcome relationship changed while input distributions stayed similar. Each claim therefore needs its own evidence and denominator.

Exit check 10.B

The handed-off procedure must preserve the evaluated input contract, preprocessing, fitted models and settings, calibration, class interpretation, threshold and decision rule. The target, intended population and evaluation context must remain clear. Material revisions create a new procedure; the previous final-test result does not automatically validate it.

Lesson 11 · Worked integration

Exercise 11.1

At threshold 0.40, jobs three through six are alerted. TP = 2, FP = 2, FN = 1 and TN = 1. Cost is 2 + 4 times 1 = 6 units; alert fraction is 4/6, approximately 66.7%, which fails the 40% cap.

At threshold 0.80, jobs five and six are alerted, including the probability equal to the threshold. TP = 1, FP = 1, FN = 2 and TN = 2. Cost is 1 + 4 times 2 = 9 units; alert fraction is 2/6, approximately 33.3%, which passes. Of these two candidates, only 0.80 is eligible. This does not prove that 0.80 is optimal over the full declared grid; that would require checking all candidates and ties.

Exercise 11.2

For the supplied reference run, regression selects Ridge by lowest mean MAE across the four training-only temporal folds. Logistic regression selects C = 1 with no class weighting by training-CV negative log loss. Sigmoid calibration uses the separate 144-row calibration block. The threshold is 0.20, selected on the 144 policy rows by cost and capacity with the declared tie rules.

The recorded policy counts are TN = 84, FP = 20, FN = 3 and TP = 37; they sum to 144. Positives are 3 + 37 = 40, alerts are 20 + 37 = 57, and alert fraction is 57/144, approximately 39.583%. Cost is 20 + 4 times 3 = 32 units. Confirm these against outputs/reference_results.json in your run. A changed input or implementation must have its own results rather than copying these values.

Exercise 11.3

A good freeze note identifies the exact source revision, hashes, data partitions, target definitions, feature schema, pipeline(s), selected parameters, calibration block/method, threshold rule/value, seeds, environment and report plan. It states that no further choice will use final labels before the report. If final capacity fails, report the fraction and recommend withholding operational use pending a revised study with fresh evaluation evidence. If code changes after freeze, log the reason, which stages were affected and whether test outcomes were already seen. Reference results describe reference code; the learner must provide independently generated evidence from the learner implementation.

Exit check 11.A

Base parameters and selected settings use training and its inner folds. The probability mapping uses the calibration block while the base pipeline remains fixed. The threshold uses the later policy block after its separate embargo. The final test evaluates the frozen composite procedure.

Exit check 11.B

Capacity is measured on a particular sample and probability distribution. Future inputs, outcome relationships and scores can differ, and finite-sample variability also changes counts. A fixed threshold is not a hard queue-length control.

Lesson 12 · Capstone

Exercise 12.1

An acceptable manifest maps the contract and availability ledger to the intended-use claim; data/source hashes and audit output to provenance; split membership and maturity checks to information boundaries; learner source and test logs to implemented behavior; fold reports to model selection; calibration and threshold tables to probability/policy choices; and the freeze record to pre-test commitment. These must exist before final evaluation.

Final predictions and metrics, residual/slice evidence and the decision brief are created after evaluation. The environment specification and reproducible commands apply to both stages. State which script produces each artifact and identify the learner-owned output directory. A manifest that points only to supplied reference files is incomplete.

Exercise 12.2

The forbidden-predictor rehearsal should show the feature contract rejects or excludes the field before fitting, with explicit evidence that it cannot reach the estimator. Restore the approved list and retain the test log. An unseen depot should follow the declared unknown-category behavior and return finite output, with no claim that this proves accuracy at new depots. For no feasible ordinary threshold, return 1.01 and verify no alerts because valid probabilities are at most one. Preserve raw data; change only a test copy or explicit test fixture. A test that merely asserts a favorable final score does not address these risks.

Exercise 12.3

There is no single required recommendation or wording. Full-credit briefs identify the final period/sample, compare the learner model with baseline using actual units and counts, discuss probability performance separately from threshold burden, and state whether the predeclared operational gate passed. They distinguish synthetic evidence from real-world deployment, avoid causal claims, and propose a proportionate next research step. A material counterargument could be a capacity failure, weak small-slice evidence, inconsistent fold performance or lack of new-depot validation. Do not invent numerical uncertainty intervals or business benefits absent a valid design.

Exit check 12.A

Learner completion requires the learner's implemented functions/pipeline, independently generated audit and evaluation artifacts, freeze history, test results and explanations assessed with the rubric. Running the reference workflow or supplied solution tests only demonstrates that the reference executes in that environment.

Exit check 12.B

Report the failed gate, its magnitude, relevant denominator and effect on the recommendation. Preserve the frozen result. Further tuning is a new development iteration and needs fresh untouched evidence for another final claim. A clearly reasoned no-go can be an excellent capstone.

Reference final outcome · Spoiler section

This section summarizes the supplied reference run, not a learner grade or a deployment recommendation. Read it only after your own freeze and evaluation. Exact machine-readable results live in outputs/final_test_results.json; development details live in outputs/reference_results.json.

The reference final period contains 300 jobs. The selected Ridge duration model has MAE approximately 1.35949 hours, compared with approximately 2.41657 hours for the training-median baseline. That is a useful predictive improvement within the synthetic experiment, but it does not by itself establish acceptable individual errors, new-depot performance or a business effect.

The frozen classification threshold produces TN = 163, FP = 28, FN = 14 and TP = 95. These reconcile to 300 jobs and 109 late outcomes. There are 123 alerts, or 41% of the final sample. The predeclared development capacity limit was 40%, so the final alert fraction exceeds it by one percentage point. The right conclusion is promising predictive evidence with a failed capacity gate; do not launch the hypothetical policy or quietly retune on this same test.

Development calibration evidence is mixed: on the policy block, calibrated log loss is approximately 0.301738 versus 0.306761 raw, while calibrated Brier loss is approximately 0.094652 versus 0.092853 raw. A blanket claim that calibration improved everything is unsupported. Report each metric with its population and preserve the distinction between model development and the final evaluation.

A revised capacity policy could be studied in a later experiment with a new untouched future period and explicit handling of review workload. This educational exercise neither runs that future experiment nor authorizes changes to real operations.