Treat preprocessing as part of the learned model

Learning begins before the estimator

A model rarely receives the original table unchanged. Some numerical values are missing, categorical strings need a numerical representation, and different scales affect regularized linear models. These preparation steps can learn from data. A median imputer learns a median; a scaler learns a mean and spread; an encoder learns category names. If they learn from held-out rows, information has crossed the boundary before the estimator's fit begins.

Separate fixed validation from learned preparation. A fixed rule can reject a negative item count because the contract defines counts as nonnegative. A learned rule that declares an unusually high count an outlier based on all 1,500 jobs has seen the final period. Its threshold must instead be fitted inside the relevant training portion, or be justified independently before analysis. “Cleaning” is not an exemption from evaluation discipline.

The course uses five numerical columns: distance, queued jobs, item count, weekday and promised hours. It uses three categorical columns: priority, zone and depot ID. Weekday is numerically coded in this baseline. A linear model consequently gives it a straight-line contribution, even though Sunday and Monday are adjacent in a cycle. That limitation is a reasoned feature-design question for training-only development, not permission to tune using test errors.

Worked miniature: fit once, transform later

Imagine training distances of 2, missing and 6 kilometers. Median imputation learns 4, producing 2, 4 and 6. Their mean is 4. The population variance used by StandardScaler is (4 + 0 + 4)/3 = 8/3, and the scale is its square root, about 1.633. Standardization subtracts the training mean and divides by that scale. The transformed training values are approximately −1.225, 0 and 1.225.

A later missing distance becomes 4 under the stored imputer and therefore 0 under the stored scaler. A later distance of 10 becomes (10 − 4)/1.633, approximately 3.674. We do not recompute the mean using 10. Being outside the training range is information to inspect, not a reason to silently redefine the representation. This hand example illustrates fitted state, not a promise that imputation recovers a true missing distance.

The reference numerical branch also adds a missingness indicator for columns that are missing during fitting. For this miniature distance column, its pre-scaling values are [0, 1, 0]. The hand calculations above concern only the distance-value column; the indicator is a separate transformed column. An indicator describes missingness, not the cause of missingness.

Suppose the training zones are north, south and north. A one-hot encoder creates a north indicator and a south indicator. North becomes [1, 0]; south becomes [0, 1]. An unseen west zone can become [0, 0] when handle_unknown="ignore" is used. That prevents a technical crash, but the model has no learned west-specific effect. A missing zone and an unseen zone also need not mean the same thing. Their handling must be documented.

Assemble one object with an inspectable boundary

A ColumnTransformer routes named numerical and categorical columns to separate preparation branches. A Pipeline connects the combined transformed matrix to an estimator. Fitting the pipeline fits its preparation on the supplied training rows, then fits the estimator on the transformed training matrix. Predicting applies the stored transformations and the stored estimator. These API roles are documented in the ColumnTransformer reference and OneHotEncoder reference.

The key unit to pass into cross-validation is the entire unfitted pipeline. Each fold then learns its own preparation from that fold's training rows. Precomputing one transformed table before cross-validation does not create independent fold preparation. The common-pitfalls guide explains why this placement prevents a common source of leakage.

Use an explicit feature allowlist. Selecting all numerical columns would accidentally admit duration, close cost or a timestamp converted to an integer. Dropping only a few known bad columns is fragile when the export gains new columns. The model's intended schema is smaller and more stable than the raw audit table.

Why scaling and encoding need interpretation

For Ridge and logistic regression, regularization penalizes coefficient size. When one feature is in kilometers and another in thousands of items, equal coefficient penalties do not represent comparable changes in the original measurements. Scaling creates a more comparable numerical basis, though it does not guarantee equal importance. Trees split on individual thresholds and usually do not need standardization for that reason. Keeping a clear pipeline is still worthwhile for consistent imputation and category handling.

One-hot encoding avoids imposing a false numerical order on depot labels. Encoding depot A as 1 and depot B as 2 would invite a linear model to treat their difference like a quantity. However, one-hot encoding does not make IDs causal or harmless. A depot category can capture location, equipment and unobserved processes simultaneously. A unique job ID has little reusable meaning and may enable memorization.

The transformation also changes shape. Eight raw predictors may become many matrix columns. Inspect names and dimensions, and check the matrix remains practical for CPU memory. The lab is deliberately small; a production table with millions of distinct IDs could make one-hot encoding inappropriate. Never assume a successful toy transformation establishes scale readiness.

Guided inspection

Run python examples/04_preprocessing.py. Locate the fitted numerical statistics, learned categories and transformed shape. Supply a copied intake record with one missing numerical value and one unseen category. Predict how it will transform before running. Do not edit the stored raw data to make the example convenient. Verify the same raw column order and names are accepted by the fitted pipeline and that forbidden columns are excluded by construction.

A safe failure is valuable. Missing a required input column should produce a clear contract error, not quietly substitute a new meaning. Saving only estimator coefficients loses the transformations that give those coefficients meaning. The complete fitted pipeline belongs in the eventual handoff, together with schema and version information.

Independent practice

1. 04.1 · Trace learned state. Training queue counts are 0, missing and 8. Work through median imputation, training mean, population scale and transformed values. Transform new counts missing and 12 without refitting. State the units before and after scaling.
2. 04.2 · Diagnose leakage. A notebook imputes all rows, encodes all categories and then cross-validates only the classifier. Explain which steps learned from held-out rows. Describe a replacement pipeline and exactly where it must be passed during validation.
3. 04.3 · Test the schema. Design four input tests: an unseen zone, a missing distance, a missing required column and a supplied forbidden column. For each, state the intended behavior and what evidence would show the behavior occurred.

Exit checks

• 04.A: Why does an unseen category producing an all-zero indicator block not mean the model understands that category?
• 04.B: Which objects must be refitted separately inside each cross-validation training fold?