Turn a successful run into a reproducible experiment

A notebook that once printed a good score is a weak scientific record. To understand what happened later, you need the data-generating process, row assignments, preprocessing state, model definition, optimization settings, selection rule and measured result. Reproducibility makes the experiment inspectable; it does not make the underlying prediction problem valid automatically.

Our package records seed 2718 for generated data and 2719 for the split permutation. Each task contains 600 examples: 360 train, 120 validation and 120 test. The training seed and loader shuffle seed are also explicit. Data rows are given stable IDs before shuffling, so a split can be checked for overlap rather than trusted by description. data/manifest.json records shapes and CSV checksums. The reference program generates the same float32 data directly; CSV files are human-inspectable exports using enough significant digits for float32 round trips.

Randomness has several homes

A data generator, parameter initializer and minibatch shuffle each consume random numbers. If all three share one implicit stream, an innocent extra random draw can change the rest of the experiment. Dedicated generators make these dependencies clearer. The course uses an explicit generator for data, another for split IDs and another for each training loader. Global framework randomness is still seeded for model initialization and dropout.

A seed is not a promise that every machine, library release or hardware backend produces identical results. The verified path is CPU, one thread, Python 3.12 and PyTorch 2.9.1+cpu. Deterministic algorithms are requested. Different operating systems or library builds may change floating-point details; meaningful comparisons should use stated tolerances, not insist every decimal match. Keep environment version records and separate numerical regression checks from statistical claims.

Save the right state

A PyTorch model architecture describes operations. Its state dictionary contains learned tensors and registered buffers. To resume optimization exactly, optimizer state also matters: AdamW remembers running gradient statistics. An epoch counter and random-number generator states can matter too. The core course saves reports and retrains small models quickly rather than distributing binary checkpoints. This keeps the learner package source-only and avoids teaching untrusted model loading casually.

When keeping a best checkpoint in memory, a shallow reference to the current state can accidentally keep changing during later training. Our loop makes a deep copy when validation improves, then reloads it at the end. That is why the final reported validation loss matches the minimum saved history value even when the last epoch is worse.

If you later save and load weights, use trusted files and follow the current official serialization guidance, including restricted loading where appropriate. Do not download and execute arbitrary model artifacts simply because they have a familiar extension. The course requires no pretrained downloads.

Separate reproducibility from test reuse

The reference scripts may be rerun to verify the same frozen teaching experiment. This is not new evidence from fresh test data. The test partition remains the same 120 examples, and repeated executions do not increase its sample size. If you inspect the reference test score and then use it to redesign your own model, that partition is no longer a clean final assessment of your redesign.

For the capstone, document your protocol before running final evaluation. Keep a modest predeclared candidate set, select with validation and state any earlier exposure to the bundled test result. An instructor can generate a fresh hidden assessment set from a new seed after the model choice is frozen. It must have the same intended data process, and its labels must not become tuning feedback. This is a pedagogical assessment practice, not a substitute for a realistic deployment dataset.

Write a result someone can challenge

An honest report names the task, baseline, split, selected model, selection rule, result and limitations. “The network learned rings” is less useful than “On this fixed balanced class-conditional synthetic rings experiment, the validation-selected MLP achieved the reported accuracy on 120 held-out samples using a threshold fixed at 0.5.” State the actual measured value from your report, not one copied from a lesson. Give confusion counts alongside a rounded percentage so the reader can see the denominator.

The uncertainty of a result is not captured by a deterministic seed. The synthetic sample is only one sample, the generator is only one world, and no real deployment distribution has been established. Real applications also need privacy, fairness, monitoring, security, maintenance, cost and human oversight decisions. Those topics are important precisely because a good local metric does not settle them.

Exercises

• 10A · Missing record. A teammate saves only a model state dictionary and a test score. Name five missing pieces required to understand or reproduce the experiment.
• 10B · Best weights. Explain why storing best_state = model.state_dict() without a deep copy can be dangerous during continued training. Identify the relevant line in our implementation.
• 10C · Same test twice. You rerun the frozen reference experiment and obtain the same accuracy. Does the effective test sample become 240? Explain what this rerun actually verifies.

Exit check: distinguish run reproducibility, fresh-sample uncertainty and generalization to a new real-world population.