Reproducibility, handoff and drift

A model is only part of the deliverable

A receiver needs to know what rows the estimator expects, when inputs exist, how it was selected and where its claims stop. Reproducibility means preserving enough evidence to reconstruct and check the procedure. Handoff means making that evidence usable by another person. Neither means the model is ready for production.

Your Python course required rerunning source rather than trusting a screenshot. Your SQL course required explicit grains, join checks and cutoff rules. Your statistics course required a claim matched to its evidence. This final lesson combines those habits around a supervised, CPU-based synthetic experiment. It does not deploy a service, automate operational decisions or extend the course to high-stakes scoring.

The intended input is a depot job at intake. The permitted schema includes distance_km, queued_jobs, item_count, priority, zone, depot_id, weekday and promised_hours. State units, allowed categories, missing-value behavior and how unfamiliar categories are handled. The duration target is in hours; late means duration strictly exceeds the promise. label_available_at, final delay reason and close cost must remain outside prediction inputs.

Worked trace: follow one job through a handoff

Imagine an intake job with distance 12 km, queue 7, item count 3, standard priority, zone north, depot D1, weekday 0 (Monday) and a 10-hour promise. The receiver validates columns, types and units. It passes those raw fields into the saved preprocessing-and-model procedure. The classifier produces a late probability; the frozen calibration mapping adjusts it; the frozen threshold converts it into the declared alert rule. The regression pipeline produces duration separately.

If the receiver encodes “standard” using a new homemade mapping, it has changed the procedure even if the saved estimator is unchanged. If it fills missing queues with a newly calculated median, it has also changed the procedure. Keep preprocessing attached to the model and exercise the entire path with a small, known test input.

Suppose a saved miniature smoke-test probability is 0.40 and the threshold is 0.40 with an “at least” rule. The expected alert is true. A receiver using “greater than” produces false. This one boundary check can expose a materially different decision policy before any large batch is processed.

Preserve the evidence, not just a seed

Record the source revision, data-generation settings, input file hashes, sorted row identifiers, target definition, split ranges, label-availability audits, candidate settings, selected procedure and scoring definitions. Retain the freeze record and distinguish development outputs from final evaluation. A file hash detects changed bytes; it does not prove that those bytes describe a valid experiment.

The course environment is pinned to Python 3.12.14, NumPy 2.3.5, pandas 2.2.3, SciPy 1.17.0, scikit-learn 1.8.0; the supplied static charts use the standard library. Record actual versions from the environment you run. A different version is a changed condition to investigate, not permission to claim an exact match. Random seeds improve repeatability of randomized steps, but they cannot repair changed data, different split boundaries or unavailable labels. Randomness guidance

Use trusted local artifacts only. Pickle-based model files can execute code when loaded, and scikit-learn does not support arbitrary cross-version loading as a reliable compatibility strategy. Preserve the training environment and inspect provenance. Model-persistence guidance

Rehearse the whole workflow

Run python -m courseml.run_all for development, then python examples/10_handoff.py to compare input summaries and check for a freeze record. That script does not itself verify every handoff requirement. Find measured results in outputs/reference_results.json; do not substitute this lesson's miniature numbers. Verify whether expected outputs were produced and whether rerunning under the recorded conditions reproduces them within an explicitly justified numeric tolerance.

Freeze preprocessing, features, model settings, calibration, threshold and evaluation plan before python -m courseml.final_evaluation. Final evaluation estimates performance on days 200–249 for the frozen procedure. Report disappointing results honestly. If you change the procedure afterward, preserve the old report and seek fresh evaluation evidence for the new version.

A useful handoff summary states the intended use, training and evaluation populations, baseline comparison, important error slices, probability and threshold assumptions, known limitations and the person responsible for deciding what happens next. State that all depot records and outcomes are synthetic. A clean run of reference code is evidence about that code, not proof that a learner independently mastered the material.

Worked trace: distinguish drift from a verdict

In a miniature monitoring example, a reference window contains 100 jobs with five missing queue values. The missingness rate is 5%. A new window contains 80 jobs with 12 missing queue values: 15%. The increase is ten percentage points, and the relative rate is three times the reference rate. Keep denominators visible.

This change is input drift or a data-quality symptom worth investigating. It does not, by itself, prove that prediction error worsened. Perhaps a source-system field stopped arriving; perhaps the job mix changed. Check the pipeline and source before drawing a model conclusion.

Now suppose only 40 of those 80 new jobs have mature outcomes and 20 are late. The observed late fraction among mature outcomes is 50%. It is not legitimate to divide by 80 and call the rate 25%, because pending outcomes are unknown. Nor is 50% automatically representative of all 80: outcome availability may depend on duration. Wait for the declared maturity policy and report coverage.

Monitor questions with different evidence

Input monitoring can track missingness, invalid categories, unusual ranges and unfamiliar depots without labels. Prediction monitoring can track score distributions and alert volume. Performance monitoring needs mature outcomes: MAE, probability loss and threshold errors must use a defined, eligible cohort. A changed input distribution need not change performance; an unchanged input distribution does not guarantee that the outcome relationship stayed stable.

Predeclare review triggers and owners. An alert should start a diagnosis, not automatic retraining or silent threshold changes. Identify who would verify the source, inspect jobs, consider a fallback and authorize a new development cycle. Threshold changes can alter workload even without retraining. Version them as deliberate decisions and evaluate them appropriately.

Independent practice

Exercise 10.1 · Repair a handoff

A package contains an estimator file and “seed = 42,” but omits preprocessing, feature units, split dates, dependency versions, threshold and intended use. Draft a compact handoff checklist, explain why the seed is insufficient and specify one whole-pipeline smoke test. Include a safe rule for loading model files.

Exercise 10.2 · Audit monitoring arithmetic

A reference cohort has 200 jobs and eight missing distance values. A new cohort has 120 jobs and 18 missing values. Calculate rates, percentage-point increase and relative rate. Only 75 new jobs have mature labels; 30 are late. State the observed late fraction and two limitations on interpreting it.

Exercise 10.3 · Respond without hiding a failure

After final evaluation, a teammate proposes tuning the threshold on its 300 jobs and replacing the original result. Separately, unfamiliar depot categories appear in a later batch. Write a response that preserves the test history, separates the two issues and gives a bounded next step for each.

Exit checks

• 10.A: Why are changing input distributions and worsening predictive performance different claims?
• 10.B: What must remain fixed for a final-test result to describe the handed-off procedure?