Capstone: deliver an auditable intake prediction study

Your assignment

Produce your own small, reproducible investigation of the synthetic depot jobs. Deliver a duration predictor and a lateness review policy, compare them with appropriate baselines and explain whether the evidence supports a limited next experiment. The assignment is an educational evaluation. It does not authorize deployment, real-world messaging, staffing decisions or scoring people.

Work from the course's supplied raw partitions. Preserve the 894-row training period, the calibration and policy allocation within validation, all embargoes and the untouched 300-row final test. Use only the approved intake predictors, including the original promised_hours. Keep post-outcome fields for auditing and label construction only. You may use the taught CPU-friendly model families; a larger model is not extra credit by itself.

The completion standard is an evidence-backed artifact, not a favorable score. A careful no-go recommendation can earn full marks. A high score obtained by leakage cannot. Read the rubric in ASSESSMENT.md before beginning, but keep SOLUTIONS.md and supplied final-reference outputs closed until you have frozen your own choices and completed your evaluation.

Worked preparation example: turn a requirement into evidence

Suppose a fictional project requirement says, “Unknown depot labels must not crash prediction.” A weak submission says, “I used an encoder.” A stronger submission identifies the policy, constructs a copied intake row whose depot is D_NEW, passes it through the entire fitted pipeline and records a finite probability between zero and one. It also says that successful execution does not establish predictive quality at a genuinely new depot.

For a second requirement, “No future labels enter fitting,” a weak submission says, “I sorted dates.” A stronger submission saves each training/evaluation boundary, checks the maximum fitted label-availability time against the earliest predicted intake time, and explains the fixed label-availability assumption in the synthetic generator. If a check fails, the run stops and the issue is investigated. Altering the timestamp to satisfy the test would destroy the evidence.

These examples show the structure of a good submission: claim, check, observed result and limitation. They do not supply your model choice, threshold evidence or final recommendation. Use the same structure for each critical requirement.

Stage 1: define and audit

Write a one-page prediction contract, a feature-availability ledger and a target definition. Record how equality at the promise boundary is treated. Audit ID uniqueness, time order, missing inputs, target consistency, allowed feature selection and all partition counts. Include the difference between a valid absent value that can be imputed and a contract violation that should stop the workflow.

Describe the population represented by these 250 synthetic days and the transfer claims you will not make. A depot represented in training is different from an unseen depot. Future jobs are different from a random sample of historical jobs. The task's timeline is fixed so the comparison has one coherent interpretation.

Stage 2: implement and compare

Implement the metric functions in exercises/learner_functions.py and your classifier/regressor construction and threshold rule in exercises/capstone.py. Run python check_learner.py and python check_learner.py --capstone. The starters intentionally fail before implementation. Save the command and results for your own files; passing checks for supplied reference solutions is not learner evidence.

The public checks cover a few arithmetic cases, classifier and regressor pipeline smoke tests and threshold-policy cases. They cannot read your intent, reconstruct every data dependency or assess your written claims. Add your own boundary cases, including probability equal to threshold, no feasible ordinary threshold, missing numerical input and an unseen category. Explain what each test protects.

Compare full pipelines using the four training-only temporal folds and declared metrics. Select the regression family with mean CV MAE, report RMSE and retain fold-level evidence. For classification, state whether you follow the fixed logistic probability-policy case or predeclare another taught family before looking at policy results. Any alternative still needs valid probability estimates, calibration design and a documented selection criterion. Do not evaluate endless variants until one looks impressive. Your build_classifier() and build_regressor() factories must return unfitted full pipelines with settings chosen through your documented training-only comparisons; the runner does not perform that selection for you.

Stage 3: calibrate, choose policy and freeze

Refit selected base pipelines on training only. Fit probability calibration using days 150–173, leaving the base classifier unchanged. After day 174, apply calibration to days 175–198 and select a threshold with the exact stated grid, cost ratio, capacity limit and tie rules. Show candidate-level counts, including the explicit no-alert fallback if required.

Write a freeze record that another reader could use to reproduce the complete procedure. Include all fitted stages, feature schema, seeds, dependency versions, data/source hashes, threshold, metrics and the outcome of development checks. Freeze means decisions stop changing before final labels are inspected. If you discover a genuine bug afterward, disclose the incident and distinguish a repaired analysis from an untouched final evaluation.

Run python learner_run.py develop to fit your factories, calibrate on the assigned block and apply your policy selector. It writes development results and a freeze record to learner_outputs/ without opening test data. Complete your written decision record before running python learner_run.py final, which verifies the saved source/data/environment fingerprint and evaluates the frozen procedure without a new threshold search. The reference python -m courseml.final_evaluation evaluates reference code, not your capstone. Neither runner supplies the missing written audit or prevents deliberate test inspection.

Stage 4: evaluate once and hand over

Evaluate the frozen procedure on days 200–249. Report the actual sample and late count, baseline and model regression metrics, classification probability metrics, confusion matrix and alert fraction at the frozen threshold. Keep failed capacity or performance gates visible. Do not convert a failure into a pass by tuning on the same final outcomes.

Submit source, environment specification, audit, split evidence, comparison results, freeze record, final results, labeled residual/segment evidence and a short model card. The model card should describe intended use, limitations, monitoring signals, label delay, fallback behavior and who would review problems in a hypothetical next study. No real operational rollout is part of this assignment.

Independent practice

1. 12.1 · Prepare your evidence manifest. List each required artifact, the command or procedure that creates it and the claim it supports. Mark which files must exist before final evaluation.
2. 12.2 · Run a failure rehearsal. Introduce a forbidden predictor into a copy of your feature specification, then restore it. Separately test an unseen depot and the no-feasible-threshold case. Record whether each failure is blocked or safely handled, without editing raw evidence.
3. 12.3 · Defend your recommendation. Write a 250–400-word decision brief using your own final results. Explain baseline comparison, capacity, uncertainty, transfer limits and the next authorized research step. Include one finding that argues against an overly confident recommendation.

Exit checks

• 12.A: What evidence distinguishes a learner-owned completed capstone from running the supplied reference workflow?
• 12.B: What must you say if the frozen procedure fails a predeclared gate on the final test?