Choose a split that resembles the future

A score answers the question posed by its split

Training data teach the rule. Validation data support development choices. Test data provide a final, previously unused check of the frozen procedure. These are roles, not three random filenames. If you inspect test errors to choose features and then report a revised test score, the file has become development data regardless of its name.

The operational question is: how well will a model trained on earlier completed jobs predict later incoming jobs at the represented depots? A chronological split mimics that direction. A random split mixes earlier and later conditions and can make training unrealistically aware of future operating patterns. Shuffling does not destroy only order; it changes the question being measured.

Random splitting can be appropriate for an independent, exchangeable sample when future use resembles another draw from the same population. Stratification can preserve approximate class proportions in such a split. Neither is a universal default. A time-varying late rate may be part of the problem, so we do not reshuffle time periods until their rates match. The scikit-learn cross-validation guide distinguishes these designs and the dependencies they address.

Read the actual course boundaries

There are 1,500 synthetic jobs, six per day over 250 days numbered 0 through 249. Training uses days 0–148: 149 days times six jobs equals 894. Day 149 contributes six embargoed jobs. Validation uses days 150–198: 49 days times six equals 294. Day 199 contributes another six embargoed jobs. Final test uses days 200–249: 50 days times six equals 300. The counts reconcile: 894 + 6 + 294 + 6 + 300 = 1,500.

Validation is itself allocated before model selection. Days 150–173 provide 144 calibration jobs. Day 174 is a six-job embargo. Days 175–198 provide 144 threshold-policy jobs. Thus 144 + 6 + 144 = 294. The calibration segment teaches a probability adjustment; the later policy segment chooses a review threshold. They are not interchangeable chances to obtain a better score.

An embargo leaves time between a training or fitting period and the next evaluation period. Here generated durations are bounded at 23.5 hours, so a full-day outer gap supports outcome maturity. We still verify label timestamps. In real records, reporting delays or revised labels could exceed an assumed duration bound; an apparently generous gap could remain inadequate.

Worked miniature: chronology is not enough by itself

Suppose jobs arrive once per day. We propose training on days 0, 1 and 2, skipping day 3, and evaluating day 4 onward. The day-2 job arrives at 09:00 and its label becomes available on day 4 at 11:00. If our first evaluation prediction is at day 4 at 09:00, that label is unavailable for the fitted model. The row order looks correct and a day was skipped, but the information boundary still fails.

There are two defensible repairs: move the fitting/evaluation boundary later, or omit that not-yet-mature label from the training snapshot according to a predeclared rule. Retaining it because its eventual outcome is in the CSV is not a repair. Notice also that deleting every slow job can change the training population. A repair must explain selection consequences, not merely make an assertion pass.

Now consider six miniature jobs: depot A on days 1, 2 and 3 and depot B on days 1, 2 and 3. A chronological split evaluates later jobs from already represented depots. Holding all B jobs out instead asks about transfer to an unseen depot. Both are valid questions, but neither answers the other. For a combined future-and-new-depot claim, both restrictions must be designed together; a single convenient split is insufficient.

Where group splitting belongs

Repeated entities can share operating conditions. If the intended use were a new depot, training and evaluation should not share depot IDs. Group splitters separate whole groups, even when group sizes differ. Group leakage can also occur through repeated jobs or copies of the same original record.

For this course, a group holdout is a clearly marked hypothetical comparison, not a replacement for the temporal final holdout. A new-depot test would also make an unseen category appear at prediction time; preprocessing must cope without pretending it has learned that depot's effect. This connects split design to the next lesson's unknown-category policy.

Cross-validation stays inside training

Model comparisons need more than one lucky boundary, but must preserve the chosen information direction. The course uses TimeSeriesSplit(n_splits=4, gap=12) on the ordered 894 training jobs. Earlier portions train each candidate; later portions evaluate it. Twelve rows correspond to two days in this regular six-jobs-per-day dataset. In an irregular real stream, a row gap would not guarantee a particular elapsed time.

Do not read “time-series” as proof that every split respects whole days or label maturity. The splitter uses row positions, and its default fold sizes can cut within a day. Inspect actual indices, arrival times and label-availability timestamps. Its 1.8 reference specifies the meaning of gap and notes the importance of equally spaced samples for comparable time durations.

Run python examples/03_splits.py. Draw the full allocation and one inner fold before studying the printed output. Verify disjoint IDs, date direction, embargo exclusion, exact counts and target availability. A total count alone cannot detect a swap between two partitions. Save the allocation with the experiment so a later score can be connected to an exact evaluation population.

Independent practice

1. 03.1 · Reconcile the design. Make a seven-block timeline for training, outer embargo, calibration, inner embargo, policy, outer embargo and final test. Give inclusive day bounds and row counts for every block, then reconcile to 1,500.
2. 03.2 · Select the question. Propose split designs for (a) later jobs at current depots, (b) a new depot under otherwise similar conditions and (c) a new depot next year. Explain what each design can and cannot support. Do not change the course's actual temporal split.
3. 03.3 · Find the hidden future. An inner fold's training arrivals end Friday, evaluation starts Sunday at 08:00, and one training label is confirmed Sunday at 10:00. Explain the failure and two possible repairs. Include why a row-gap check alone is insufficient.

Exit checks

• 03.A: Why is stratifying the whole dataset by lateness not an acceptable substitute for the course's chronological holdout?
• 03.B: What new question is introduced when entire depots, rather than later dates, are held out?