Defend one small neural experiment end to end

The capstone is a compact research report, not a race for the largest model. Your job is to make one prediction task understandable, reproducible and honestly evaluated. A modest score with a sound protocol is more valuable than a spectacular score obtained by tuning on the test set. You will build on the supplied components, but you must make and explain the important choices yourself.

Choose the regression task as the default path. Its two inputs are fictional standardized sensor-like measurements and its target is a continuous response. The generating process contains a linear contribution, a quadratic contribution and random noise. Because this is synthetic, you can compare the model's behavior with the known construction without pretending to have discovered a physical law. The optional extension is the rings classifier. Complete one coherent experiment before adding another.

Stage 1: make a contract before a model

Write a question in plain language: what does one row represent, what inputs are available, what target is predicted and what kind of new examples should performance describe? Record the IID assumption. Explain that a random row split would be inappropriate if several rows came from one person/device or if deployment predicts future periods with changing conditions. In those situations, return to the group/temporal evaluation material from Applied ML.

Declare split seeds, counts and selection metric. For regression, validation MSE is the selection criterion; RMSE and MAE help communicate performance in target units. Fix a small candidate set before using final test results. Include the train-mean baseline and a linear neural layer. Your nonlinear candidate must be small enough to train on CPU. Record every candidate you actually try.

Stage 2: implement and inspect

Complete the five public stages in exercises/stages.py. Run python check_learner.py; its default imports your functions, not the reference functions. Passing these checks is necessary but not sufficient. The checker verifies shape, preprocessing, model output and update mechanics. It cannot judge whether your written deployment question is sensible.

Next edit exercises/capstone.py, supplying your planned candidate configurations and report rationale. Run python -m exercises.capstone to produce training/validation evidence only. It should record row IDs, training-only scaling, candidate histories and the chosen index. Read those results and write your choice down. Do not run final evaluation simply because training finished if the protocol is still changing.

A common temptation is to fix the input scale using all 600 rows because “there are no labels in the mean.” That still lets the held-out feature distribution influence the fitted pipeline. Fit the scaler only on training rows. Reuse its mean and scale on validation, test and future inputs. The same principle applies to learned imputations and feature selection, not just normalization.

Stage 3: freeze, then assess

Finalize the candidate list and selected model using validation. Save the exact plan and its hash. Then run the explicit --final mode once to evaluate the selected configuration on the test partition and write the final report. The course runner includes guardrails against accidentally overwriting final results. These guardrails are not cryptographic secrecy: the bundled data and reference outcomes are visible. Disclose any prior test exposure rather than pretending the learner package provides a hidden benchmark.

A final score cannot tell you whether the architecture was well chosen for every plausible future task. Compare it with the naive baseline and your declared expectations, show the sample count, and explain whether the result supports the narrow question you asked. If test performance disappoints, report it. Further redesign requires a new evaluation plan and fresh held-out data, not quietly recycling this test set as validation.

Stage 4: explain an error and a limitation

Choose a diagnostic using validation data: inspect residuals across the first input, compare easy/hard regions of the rings, or inspect image errors by bar position in the optional extension. Identify one plausible cause and one experiment that could distinguish it from an alternative. Avoid claiming a causal explanation solely from a pattern in errors.

Your final report should be short enough for a reviewer to understand without reading every source file. Include a model shape trace, one correctly worked gradient example, learning curves, the selection rule, final metrics and a limitations paragraph. Link the exact configuration and generated outputs. State the actual environment tested. “Works on my CPU” is a useful verified statement; “works everywhere” requires evidence you do not yet have.

Exercises

• 12A · Protocol audit. A learner compares four candidates on test accuracy and reports only the best. Rewrite the procedure to preserve a final assessment boundary.
• 12B · Evidence package. List the files a reviewer needs to reproduce your training/validation selection and understand the final result. Distinguish them from a virtual environment or downloaded wheel, which should not be submitted.
• 12C · Responsible conclusion. Write a two-sentence conclusion that includes task, split, baseline comparison and one limitation. Use your actual measured results, not a promised improvement.

Exit check: another learner can reproduce your selection from the source and explain why no test result drove that selection. Use CAPSTONE.md for the full deliverable contract and rubric.