Answers and explanations · Capstone: deliver an auditable intake prediction study

Lesson 12 · Capstone

Exercise 12.1

An acceptable manifest maps the contract and availability ledger to the intended-use claim; data/source hashes and audit output to provenance; split membership and maturity checks to information boundaries; learner source and test logs to implemented behavior; fold reports to model selection; calibration and threshold tables to probability/policy choices; and the freeze record to pre-test commitment. These must exist before final evaluation.

Final predictions and metrics, residual/slice evidence and the decision brief are created after evaluation. The environment specification and reproducible commands apply to both stages. State which script produces each artifact and identify the learner-owned output directory. A manifest that points only to supplied reference files is incomplete.

Exercise 12.2

The forbidden-predictor rehearsal should show the feature contract rejects or excludes the field before fitting, with explicit evidence that it cannot reach the estimator. Restore the approved list and retain the test log. An unseen depot should follow the declared unknown-category behavior and return finite output, with no claim that this proves accuracy at new depots. For no feasible ordinary threshold, return 1.01 and verify no alerts because valid probabilities are at most one. Preserve raw data; change only a test copy or explicit test fixture. A test that merely asserts a favorable final score does not address these risks.

Exercise 12.3

There is no single required recommendation or wording. Full-credit briefs identify the final period/sample, compare the learner model with baseline using actual units and counts, discuss probability performance separately from threshold burden, and state whether the predeclared operational gate passed. They distinguish synthetic evidence from real-world deployment, avoid causal claims, and propose a proportionate next research step. A material counterargument could be a capacity failure, weak small-slice evidence, inconsistent fold performance or lack of new-depot validation. Do not invent numerical uncertainty intervals or business benefits absent a valid design.

Exit check 12.A

Learner completion requires the learner's implemented functions/pipeline, independently generated audit and evaluation artifacts, freeze history, test results and explanations assessed with the rubric. Running the reference workflow or supplied solution tests only demonstrates that the reference executes in that environment.

Exit check 12.B

Report the failed gate, its magnitude, relevant denominator and effect on the recommendation. Preserve the frozen result. Further tuning is a new development iteration and needs fresh untouched evidence for another final claim. A clearly reasoned no-go can be an excellent capstone.

Reference final outcome · Spoiler section

This section summarizes the supplied reference run, not a learner grade or a deployment recommendation. Read it only after your own freeze and evaluation. Exact machine-readable results live in outputs/final_test_results.json; development details live in outputs/reference_results.json.

The reference final period contains 300 jobs. The selected Ridge duration model has MAE approximately 1.35949 hours, compared with approximately 2.41657 hours for the training-median baseline. That is a useful predictive improvement within the synthetic experiment, but it does not by itself establish acceptable individual errors, new-depot performance or a business effect.

The frozen classification threshold produces TN = 163, FP = 28, FN = 14 and TP = 95. These reconcile to 300 jobs and 109 late outcomes. There are 123 alerts, or 41% of the final sample. The predeclared development capacity limit was 40%, so the final alert fraction exceeds it by one percentage point. The right conclusion is promising predictive evidence with a failed capacity gate; do not launch the hypothetical policy or quietly retune on this same test.

Development calibration evidence is mixed: on the policy block, calibrated log loss is approximately 0.301738 versus 0.306761 raw, while calibrated Brier loss is approximately 0.094652 versus 0.092853 raw. A blanket claim that calibration improved everything is unsupported. Report each metric with its population and preserve the distinction between model development and the final evaluation.

A revised capacity policy could be studied in a later experiment with a new untouched future period and explicit handling of review workload. This educational exercise neither runs that future experiment nor authorizes changes to real operations.