Answers and explanations · Treat preprocessing as part of the learned model

Lesson 04 · Preprocessing

Exercise 04.1

Median imputation learns 4, giving queue counts [0, 4, 8]. Their mean is 4, population variance is (16 + 0 + 16)/3 = 32/3, and population scale is approximately 3.266. Standardized values are approximately [−1.225, 0, 1.225]. A new missing count becomes 4 and then 0. A new count of 12 becomes (12 − 4)/3.266, approximately 2.449.

Before scaling, values are counts of queued jobs. After scaling, values are dimensionless deviations measured in training scale units. The course also adds and scales a missingness-indicator column for this training-missing feature. Its pre-scaling training values are [0, 1, 0], mean 1/3 and scale square root of 2/9. Its standardized training values are approximately [−0.707, 1.414, −0.707]. This separate indicator does not alter the calculations for the value column.

Exercise 04.2

The imputer learned numerical statistics from held-out rows; the encoder learned their categories. Additional scaling or feature selection could leak too. Replace the precomputed table with a ColumnTransformer containing numerical and categorical preparation, followed by the classifier inside one Pipeline. Pass the entire unfitted pipeline to temporal cross-validation or grid search. Each fold fits its own preparation on its training rows and only transforms its held-out rows. Merely moving the classifier fit into a loop leaves preprocessing leakage intact.

Exercise 04.3

An unseen zone should follow the declared unknown-category rule, producing a finite prediction through the complete pipeline while logging unfamiliar-category exposure. A valid missing distance should use training-fitted imputation and the stored missingness handling. A missing required column should produce a clear input-contract failure rather than silently redefine the input. A forbidden extra column should never reach the model; the allowlist excludes it, and a schema audit should flag unexpected supplied fields if that is the declared intake interface. Verify behavior through feature names, transformed shape, assertions and recorded prediction/error output, not just absence of a crash.

Exit check 04.A

An all-zero indicator block is a fallback representation. The model did not observe that category's outcomes, so it cannot have learned a category-specific relationship. Successful execution and supported generalization are different claims.

Exit check 04.B

Every learned transformation, any feature-selection procedure and the estimator must be fitted separately on the permitted training portion. Fixed, independently specified schema checks do not learn data statistics, but their definitions still must not be rewritten to exploit held-out results.