Training, validation, and test sets

Unit ID: ML-M05-U01 Estimated active time: 20-35 minutes

Why this matters

Generalisation, Splits, and Leakage helps learners avoid a common mistake: using a modelling technique without knowing what the result is allowed to mean.

In this unit, we focus on training, validation, and test sets because it is one piece of the larger workflow.

The idea

Training data fits the model, validation data helps choose, and test data gives the final held-back check.

The classroom rule is simple:

Do the smallest honest experiment, then explain the limit.

For this topic, that means connecting the code or worksheet output to the decision context. A number, cluster, threshold, or model comparison is not useful until the learner can say what it means and what it does not mean.

Predict

Before running code, answer:

• What output do you expect?
• What could make that output misleading?
• What would you check before trusting it?

Run or inspect

Use the supplied synthetic dataset for Module 05. Read the column names first. Identify the target if the module has one. If there is no target, say clearly that the exercise is exploratory.

Keep the result small enough to inspect: a compact table, one metric summary, one cluster summary, or one threshold comparison.

Change one thing

Change one modelling choice or interpretation choice. For example, change a threshold, remove a risky feature, compare a baseline, scale the data, or inspect one subgroup.

Then ask:

Did the conclusion change, or only the number?

Practice

Compare a leaky experiment with a repaired one and explain why the score changed.

Write a short note that names at least one limitation.

Check and explain

Complete this sentence:

This result suggests ______, but it does not prove ______.

Takeaway

Make honest evaluation the center of the course. The model output is evidence to inspect, not a claim to overstate.