# Explained solutions · read after attempting the work

## Lesson 01

**01A.** The output has shape (1,2). First output: 2×2 + 0×(-1) + (-1)×0.5 + 0.5 = 4. Second: 2×(-1) + 0×0 + (-1)×1 - 0.5 = -3.5. A single observation still keeps a batch axis.

**01B.** Broadcasting gives (3,3), with rows [-1,-3,-5], [1,-1,-3], [3,1,-1]. Reshape targets with `unsqueeze(1)` to (3,1); corresponding-row subtraction gives three -1 differences. The fix is semantic pairing, not merely avoiding a runtime exception.

**01C.** Input (5,4), stored PyTorch weight (3,4), bias (3,), output (5,3). There are 12 weights plus 3 biases = 15 adjustable scalars. Batch size does not change parameter count.

## Lesson 02

**02A.** z=-0.9, h=0 after ReLU, prediction=0.2. Error relative to y=0.5 is -0.3. Half-squared loss is 0.045; MSE for this one scalar is 0.09. The factor of two belongs to the loss definition.

**02B.** Prices: floating output/target (6,1), MSELoss. Binary rings: raw-logit output (6,1), floating 0/1 targets (6,1), BCEWithLogitsLoss. Three classes: raw logits (6,3), integer targets (6,), values 0–2, CrossEntropyLoss. Do not pass an extra sigmoid/softmax into these logits losses.

**02C.** Exponentiating logits gives [2,1,1], normalized to [0.5,0.25,0.25]. Target class two has probability 0.25, so loss is -ln(0.25)=ln4≈1.386294. The highest-scoring class stays zero; changing the observed target changes which probability is penalized.

## Lesson 03

**03A.** Prediction remains -0.9, error=-0.4, half-squared loss=0.08. Gradients (w,b,v,c)=(0.8,0.4,-0.44,-0.4). The paths are unchanged; only the upstream loss derivative changes.

**03B.** z=-0.9 and h=0, so prediction=0.2 and error=-0.3. Gradients are (0,0,0,-0.3). The inactive ReLU blocks w/b paths; h=0 blocks the gradient to v; c still directly changes the output.

**03C.** Simultaneous updates use all gradients from the original parameter state. At learning rate 0.01, parameters become (0.472,0.086,-0.9846,0.214), prediction=-0.800138, loss≈0.845179409522. At 0.1, parameters=(0.22,-0.04,-0.846,0.34), prediction=0.0016, loss=0.12420128. At 1, parameters=(-2.3,-1.3,0.54,1.6); ReLU closes, prediction=1.6, loss=0.605. Rate 0.1 improves this step most among these choices; it is not universally optimal.

## Lesson 04

**04A.** Check counts 60/20/20, pairwise-empty intersections and union equal to all 100 original IDs. Deliberately inserting a training ID into validation must fail. Counts alone cannot detect overlap.

**04B.** Training mean=4; population variance=(4+0+4)/3=8/3; standard deviation≈1.632993. Transforming 20 gives (20-4)/1.632993≈9.797959. Using all four values gives mean 8, standard deviation sqrt50≈7.071068 and transformed value≈1.697056. The validation value has changed the fitted transform; withholding its label does not remove this leakage.

**04C.** Without dropping: 8,8,8,8,3. With dropping: 8,8,8,8 and three unmeasured rows. Evaluation should retain all relevant observations. Weight batch losses by their sample counts.

## Lesson 05

**05A.** Clearing after backward removes the gradients the optimizer needs. For an ordinary batch: zero_grad, forward, loss, backward, step. Deliberate gradient accumulation uses a different explicitly scaled schedule.

**05B.** Correct mean=(4×0.5+4×1+2×3)/10=1.2. Unweighted mean=(0.5+1+3)/3=1.5, over-weighting the smallest batch.

**05C.** Eval-mode Linear still produces a gradient-tracked output because parameters require gradients and recording is on. Inside no_grad, the computed output is not tracked. Module behavior and graph recording are separate controls.

## Lesson 06

**06A.** Linear(2,1) has 3 parameters. The 2–16–1 MLP has (32+16)+(16+1)=65. The displayed 2–16–16–1 MLP has 48+272+17=337. ReLU permits non-affine behavior; extra affine layers alone collapse to an affine map.

**06B.** A half-plane cannot include the entire inner circle while excluding the entire outer one. Multiple learned piecewise-linear regions can approximate an enclosing boundary. Finite noisy data still requires evaluation; representational possibility is not a performance guarantee.

**06C.** A sound plan fixes split IDs, train-only scaling, candidates, seeds and budget; selects checkpoint and model by validation loss; and uses the test set once after those choices. Accuracy/confusion can support interpretation. No score should be invented in advance.

## Lesson 07

**7A.** Output is 0 with probability 0.4 or 6/(1-0.4)=10 with probability 0.6. Expected output is 0×0.4+10×0.6=6. Evaluation returns 6. This preserves expectation, not each individual training value.

**7B.** Epoch 20 has the smallest validation loss, 0.31. Later training improvements may fit sample-specific noise. The declared rule restores epoch 20; it does not let a test score override the choice.

**7C.** The wide candidate uses 24 deliberately noisy rows, width 64, no dropout/decay and learning rate .01; the smaller candidate uses width 16, dropout .15, decay .01 and rate .005. In the recorded run, wide best validation MSE was about .5413 at epoch 22, while its final validation MSE rose to 3.1869 despite training MSE about .00979. The regularized candidate's best validation MSE was about .8293 at epoch 296. This illustrates overfit and the value of checkpointing; it does not show regularization winning here or isolate one regularizer's causal effect.

## Lesson 08

**8A.** The difference matrix has rows [0,-1,-2], [1,0,-1], [2,1,0]. Squared values sum to 12; mean over nine entries is 4/3. Correct corresponding-row differences are all zero. Shape success can conceal an unintended all-pairs comparison.

**8B.** Missing optimizer membership: inspect parameter IDs versus optimizer groups. Disconnected graph: inspect requires_grad and whether loss depends on the parameter; a detached output breaks that path. Clearing after backward: inspect gradients immediately before step. A zero gradient for one inactive unit/batch may be legitimate, so compare several meaningful cases.

**8C.** Accuracy=(60+30)/120=.75; precision=30/(30+10)=.75; recall=30/(30+20)=.6. A higher false-negative cost may justify another threshold, selected using validation and a stated cost tradeoff. It does not authorize test-set threshold tuning.

## Lesson 09

**9A.** Vertical center-column patch dotted with horizontal center-row kernel gives 1, at their intersection. Dotted with the vertical kernel gives 3. No kernel flip is applied in PyTorch's cross-correlation operation.

**9B.** No-padding convolution gives [B,4,6,6]; 2×2 pooling gives [B,4,3,3]; flatten gives [B,36]. Replace the head with Linear(36,2). Keeping Linear(64,2) would violate the matrix-multiplication contract.

**9C.** Horizontal flip preserves orientation and is label-preserving. A 90-degree rotation changes horizontal to vertical or vice versa, so the label must change. Added noise preserves the underlying orientation if it models plausible measurement noise and does not destroy the signal; its suitability is assumption-dependent.

## Lesson 10

**10A.** Missing pieces include architecture, feature order, data provenance and split IDs, preprocessing state, loss/optimizer/settings, selection rule, seed/runtime versions and prior test exposure. Optimizer state is additionally needed for exact training resumption. A score alone cannot identify the experiment.

**10B.** State dictionaries can contain tensor references connected to the live model; later updates can change what you thought was a frozen best state. The implementation uses `copy.deepcopy(model.state_dict())` at improvement, then reloads it after training.

**10C.** Test sample remains 120. Repeating verifies deterministic execution under the recorded environment, not independent sampling or improved statistical certainty.

## Lesson 11

**11A.** Frozen weights stop parameter gradients, while active dropout still randomly changes intermediate values. Inspect requires_grad/optimizer membership separately from module train/eval mode. Keep a fixed extractor in eval while a trainable head is in train mode.

**11B.** Equal scores give weights [0.5,0.5]. Weighted output is [1,2]. Attention forms a mixture; it need not select one whole value vector.

**11C.** Future ground-truth tokens reveal information unavailable at causal prediction time, making training/evaluation artificially easier. A causal mask prevents attention to those future positions before normalization. Correct masking is necessary but not alone sufficient for a complete language model.

## Lesson 12

**12A.** Compare candidates on training/validation only, fix the selected model and decision rules, then evaluate once on untouched test. If test has already driven choice, disclose contamination and obtain a new appropriate held-out assessment; do not relabel the old score as final evidence.

**12B.** Submit source, protocol, dependency pins, data generator/split IDs, completed stages, selection/final JSON, curve and report. Exclude environments/caches/wheels. This retains the reproducible decisions without distributing hundreds of megabytes of runtime binaries.

**12C.** Example using this package's recorded regression run: “On 120 held-out synthetic regression samples, the validation-selected MLP achieved RMSE 0.1612 target units, compared with 2.2854 for the train-mean baseline. This is one fixed generated experiment, with visible reference outcomes and no real deployment population, so it does not establish real sensor performance.” Verify the exact baseline against your own run; change the sentence when your results differ.

## Readiness answers

R1: ((2-1)²+(5-3)²)/2=2.5. R2: held-out information must not influence fitted preprocessing. R3: repeated machine-specific patterns can leak across a row split; split by machine or time according to deployment. R4: [7,2]. R5: no; test has become a selection set.

## Checkpoint answers

A1: 16 observations, 3 features per observation. A2: [4,4], likely unintended all-pairs comparisons. A3: substitution composes matrices and biases into one affine map. A4: 2.8; locally increasing w increases loss at the original point, so gradient descent decreases w. A5: float64 reduces rounding error; too-small differences cancel, while too-large steps are poor local approximations; avoid ReLU kinks. A6: 12 updates, eleven batches of 32 and one of 8.

B1: clear accumulated gradients; compute prediction/loss; backpropagate derivatives; update parameters. B2: eval changes mode-sensitive layers, no_grad suppresses graph recording. B3: overfit or a distribution/label-quality difference between partitions; also check comparable measurement modes. B4: minimum validation-loss checkpoint within the full epoch budget; computation does not stop early. B5: magnitude multiplies by 1.2, squared loss by 1.44. B6: zero by convention; undefined-denominator conventions affect reported summaries and should be explicit.

C1: convolution 4×(1×3×3+1)=40; head 64×2+2=130; total170. C2: orientation is the target. C3: no; module mode is separate. C4: weights [0.5,0.5], result [1,2]. C5: no, same observations. C6: it does not prove error-free future performance, real-image generalization, fairness, safety or calibration. Any two well-explained unsupported claims suffice.

## Staged code explanations

`solutions/stages.py` contains the five implementations. Stage 1 uses matrix multiplication plus bias. Stage 2 fits mean and population standard deviation only on supplied training rows, clamping constant-column scales. Stage 3 returns raw logits with ReLU hidden features. Stage 4 clears, computes, backpropagates and steps in the right order. Stage 5 changes module mode and disables gradient recording before converting logits to probabilities. The checker defaults to your starter and must fail until you implement these operations.