Answers and explanations · Reproducibility, handoff and drift

Lesson 10 · Reproducibility and handoff

Exercise 10.1

A compact handoff needs:

• Intended use, prohibited uses, synthetic-data status and the person responsible for review.
• Raw input schema, units, allowed categories, missingness policy and intake availability requirements.
• Target definitions, row identifiers, data provenance or generator settings, input hashes and chronological ordering.
• Training, embargo, calibration, threshold and final-test boundaries, with label-maturity checks.
• Complete preprocessing-and-model procedure, selected settings, calibration mapping and threshold including its equality rule.
• Source revision, Python and dependency versions, random settings, execution commands and expected checks.
• Baseline and model metrics, evaluation populations, error-slice counts, limitations and the freeze/test history.

A seed controls particular random operations; it does not reconstruct missing code, input units, row ordering, environment or decision rules. A whole-pipeline smoke test passes a known raw job through preprocessing, prediction, calibration and thresholding and compares the resulting output with a saved expectation. Include a boundary case equal to the threshold and a separate valid missing or unfamiliar-category case according to the declared schema policy. Load persisted model files only from trusted, verified sources in an appropriate environment; do not load an unknown pickle merely to inspect it.

Exercise 10.2

Reference missingness is 8/200 = 0.04, or 4%. New missingness is 18/120 = 0.15, or 15%. The increase is 11 percentage points, and the relative rate is 15/4 = 3.75 times the reference rate. That relative ratio is not an 11% relative increase.

Among mature labels, the observed late fraction is 30/75 = 0.40, or 40%. Label coverage is 75/120 = 62.5%. First, the remaining 45 outcomes are unknown, not on-time cases. Second, the mature-label subset might differ systematically from pending jobs, especially if long durations delay label availability. The result does not establish the whole new cohort's late rate. Input missingness also does not by itself establish a change in model error.

Exercise 10.3

Preserve the original frozen procedure and its final evaluation. Tuning a threshold using those 300 labels would create a new development iteration; do not replace the earlier report or label the tuned result independent. If a revision is worthwhile, document its motivation, revise through a declared development process and obtain new untouched future evidence for its final assessment.

Unfamiliar depots are a separate schema and population question. Check whether they are genuine new depots or malformed identifiers, whether the declared encoder handles them and whether the intended-use population includes them. Report affected counts and test the existing unknown-category behavior without changing the policy silently. Escalate any unsupported cases to the stated review or fallback process. Neither an unfamiliar depot nor a test disappointment authorizes automatic retraining or threshold changes.

Exit check 10.A

Input drift concerns the distribution of observed predictors. Performance concerns errors against mature outcomes. The model may remain accurate under some input changes, or deteriorate because the input-to-outcome relationship changed while input distributions stayed similar. Each claim therefore needs its own evidence and denominator.

Exit check 10.B

The handed-off procedure must preserve the evaluated input contract, preprocessing, fitted models and settings, calibration, class interpretation, threshold and decision rule. The target, intended population and evaluation context must remain clear. Material revisions create a new procedure; the previous final-test result does not automatically validate it.