Answers and explanations · Cross-validation, tuning and test lockdown

Lesson 07 · Validation

Exercise 07.1

With 30 rows and two splits, each validation block has 30 divided by 3, or 10 rows.

• Fold 1: training indices 0–6, gap 7–9, validation 10–19.
• Fold 2: training indices 0–16, gap 17–19, validation 20–29.

Training sizes are seven and seventeen, not ten and twenty: subtract the three-row gap. For each fold, every included training outcome must be available by the simulated fit boundary before that fold's validation predictions. Inspect actual availability timestamps rather than just intake order. Feature construction must also respect the intake cutoff. A row gap alone proves neither requirement.

Exercise 07.2

C totals 11.2 hours across the four fold MAEs, giving mean MAE 2.8. D totals 10.4, giving mean MAE 2.6. Their mean negative-MAE scores are −2.8 and −2.6. The maximized negative score and minimized positive error both select D under the declared rule.

C is much better in the earliest folds but deteriorates sharply; D is more stable and has the lower mean. Report that pattern rather than claiming D beats C in every period. The folds overlap in their training evidence, so their variability is not an automatic independent-sample confidence interval. Both candidates still require evaluation of the frozen procedure on untouched future data.

Exercise 07.3

A defensible five-step repair is:

1. Preserve a record that the original encoder used all rows and the original test selected settings. Those original scores do not estimate an untouched-test procedure.
2. Rebuild allowed intake features and explicit targets; keep audit-only label_available_at timestamps outside the predictor. Check availability and data quality.
3. Restore chronological development boundaries. Put preprocessing inside a pipeline and tune only through the specified training-only time folds, checking label maturity at every fit.
4. Use the declared calibration and threshold blocks for their separate roles. Document the fixed model, threshold and reporting plan; label any depot-group experiment as a different unseen-depot question.
5. Report results on the already exposed 300 rows as exploratory or previously used evidence. Obtain a genuinely fresh future holdout for a new final evaluation; changing the split name cannot restore the old holdout's independence.

The main contamination is both learned preprocessing from future data and test-driven model selection. A group split can answer a useful separate question, but choosing it for a better score does not repair the original evaluation.

Exit check 07.A

A pipeline refits learned preprocessing inside each fold, preventing later fold rows from determining transformations. It does not know when an outcome became available. If its supplied training labels were unknown at the simulated fit time, it will still learn from unavailable information. Timestamp audits and appropriate splitting remain necessary.

Exit check 07.B

The score describes the previously frozen procedure on that dataset. After using the result to change the model, that dataset has influenced development; the score cannot certify the revised model as independently evaluated. Preserve the earlier result, disclose the change and use fresh untouched evidence for the next final claim.