Cross-validation, tuning and test lockdown

The question behind the score

You can now fit a preprocessing pipeline and compare a model with a baseline. The next question is whether your choice survives another period of work. A model that happened to suit one quiet week might fail during the next busy week. Cross-validation gives several rehearsals of the intended prediction situation. It does not manufacture new independent evidence or guarantee future performance.

In the SQL course, a feature cutoff separated information available now from information arriving later. Apply that same discipline to every training fold. Here one row is one synthetic depot job, predicted at intake. Regression predicts duration_hours; classification predicts whether duration_hours > promised_hours. Permitted inputs are distance, queue size, item count, priority, zone, depot, weekday and the promise known at intake. label_available_at audits outcome availability; this timestamp, final delay reason and close cost must never enter the predictor.

The full dataset contains 1,500 jobs: six per day for 250 days. Development training uses days 0–148, or 894 jobs. Day 149 is an embargo. Validation uses days 150–198, or 294 jobs. Day 199 is another embargo. Days 200–249 contain the final 300 test jobs. An embargo is a deliberate unused interval; it does not itself establish that earlier labels were available.

Separate three decisions

Fitting estimates model parameters from allowed training data. Tuning selects settings, such as a tree's minimum leaf size, by comparing candidate procedures. Evaluation measures the frozen procedure on evidence that did not select it. Repeatedly choosing models from their final-test scores converts that test into development data, even if no test row appears in a call to fit.

For this course, run TimeSeriesSplit(n_splits=4, gap=12) on the chronologically ordered 894 training rows only. Each fold trains on earlier rows and scores a later block; the training window expands. The gap is measured in rows, not days. Twelve excluded rows represent two days of jobs because this synthetic schedule has six arrivals per day. Overnight spacing differs from within-day spacing, so inspect elapsed-time boundaries as well as row counts. Irregular real arrivals require timestamp-based reasoning. TimeSeriesSplit documentation

A separate group-split thought experiment asks whether a model transfers to an unseen depot. Holding out depot identifiers answers that question, but does not preserve the main future-time evaluation automatically. Do not replace the declared final test with a group split because its result looks nicer. Cross-validation guidance

Worked trace: inspect boundaries before scores

Use this miniature arithmetic example, not the course dataset: 20 ordered rows, numbered 0–19, with three splits and a two-row gap. Default validation block size is 20 divided by 4, giving five rows.

• Fold 1: train 0–2; gap 3–4; validate 5–9.
• Fold 2: train 0–7; gap 8–9; validate 10–14.
• Fold 3: train 0–12; gap 13–14; validate 15–19.

Notice that an earlier validation row can become a training row in a later rehearsal. That is sensible only when its outcome has become known by that later fit boundary. Suppose row 2 arrived on day 2, but its completed duration became available on day 6. A fold fitted at the start of day 5 cannot use that label, even though row 2 precedes its two-row gap. The timestamp audit must reject this setup or apply a documented, appropriate maturity filter before fitting. Never silently accept unavailable labels.

For the actual 894-row training table, default fold validation size is 178. The candidate training sizes are 170, 348, 526 and 704; each gap contains 12 rows. The first validation indices are 182–359 and the last are 716–893. Check generated indices and actual label-availability timestamps; these counts alone are insufficient proof.

Keep preprocessing inside each rehearsal

Suppose a numeric imputer replaces a missing queue count with a median. If that median was computed using all 894 training rows before cross-validation, early folds have already learned about later observations. The outer table is called training data, but the later rows are still held out inside those folds. Fit imputation, scaling and encoding separately within each training fold through a pipeline. Apply the learned transformations to that fold's validation rows. Leakage and pipelines

Use a small, declared search. Regression selects among median, Ridge and random-forest procedures using mean training-CV MAE. In a hypothetical tree search, three leaf sizes across four folds would require twelve candidate fits. The actual logistic search uses three C values times two class-weight choices times four folds: twenty-four candidate fits, then a selected-model refit. Search cost grows through multiplication, not addition. Report the candidate grid and selection metric so the choice can be inspected. GridSearchCV reference

Worked trace: interpret a search result

In another miniature example, candidate A has fold MAEs of 2.0, 2.2, 3.8 and 4.0 hours. Its mean is 12.0 divided by 4, or 3.0 hours. Candidate B has 2.8, 2.9, 3.0 and 3.3, also averaging 3.0. Equal means conceal different behavior over time. A declared preference for stability might favor B; a declared complexity tie-breaker could favor a simpler procedure. Do not invent a tie-breaker after viewing the final test.

Scikit-learn search scorers maximize scores. neg_mean_absolute_error therefore represents MAE with a negative sign: −2.4 is better than −3.1. Convert back to positive hours when explaining error. Inspect fold scores as well as their average. Their standard deviation is descriptive spread, not an automatic confidence interval: folds share training observations and later periods may differ systematically.

Run, inspect and freeze

Run python examples/07_validation.py from the project root. Use outputs/reference_results.json for the actual measured results; none of the hand traces above claims to be model output. Inspect candidate settings, fold boundaries, label-maturity checks and score direction. A successful process exit is only a starting point for this audit.

Before final evaluation, freeze the feature list, cleaning rules, pipeline, selected settings, calibration method, threshold, metrics and interpretation plan. The development command python -m courseml.run_all belongs before that boundary. Run python -m courseml.final_evaluation only after the freeze. A disappointing final result is evidence to report, not permission to retune and call the same test untouched.

Independent practice

Exercise 07.1 · Trace a different split

For 30 ordered rows numbered 0–29, derive both folds of TimeSeriesSplit(n_splits=2, gap=3). State training, gap and validation indices. Then identify the additional timestamp condition required before either fit.

Exercise 07.2 · Choose with evidence

Miniature candidate C has fold MAEs 1.6, 1.8, 3.2, 4.6; D has 2.4, 2.5, 2.6, 2.9. Calculate both means, express their negative-MAE search scores, and recommend a candidate under a lowest-mean-MAE rule. Add one limitation the mean conceals.

Exercise 07.3 · Repair the experiment

A teammate fits an encoder on all 1,500 rows, tunes settings on days 200–249, then proposes holding out one depot instead because the score improves. Write a corrected five-step plan that preserves the original future-jobs question and identifies what evidence is already contaminated.

Exit checks

• 07.A: Why can a pipeline prevent one kind of leakage while still failing to prevent unavailable labels?
• 07.B: What can you honestly claim after inspecting a final test and then changing the model?