# 01 · Start with a prediction-time contract

## The question comes before the estimator

A depot receives a job and must decide whether to put it on a human review list. At intake, the team knows the distance, queue size, item count, requested priority, zone, depot, weekday and promised hours. It does not yet know the job's duration. Our first task predicts `duration_hours`; our second estimates the probability that duration will exceed the promise recorded at intake. The decision is a fictional operational review, never a score about a person's worth, employment, credit, health or access to services.

You already know how to inspect a table from Python, preserve grain from SQL and question denominators from Statistics. Machine learning adds a different obligation: demonstrate that a rule learned from earlier observations remains useful on observations it could not have seen. A sophisticated model fitted to information from the future fails that obligation, even if its error is tiny.

A prediction-time contract is a short specification of the situation in which a prediction will exist. Write the population, one-row grain, prediction time, allowed information, target, outcome-observation time, intended action and evaluation plan. Here the population is accepted synthetic depot jobs, the grain is one job, and the prediction time is intake. An ID connects evidence but is not an explanatory measurement. The contract also says who would inspect an alert and what happens if data are invalid. A model is not the whole workflow.

## Features, targets and decisions are different objects

We use X to mean the input table and y to mean the target values. A fitted model is a reusable function: new X goes in, predictions come out. During fitting, the model uses paired historical X and y. During prediction, y must be unavailable; otherwise there would be no prediction problem.

`duration_hours` is a numerical target, making the first task regression. `late` is a binary target, making the second task classification. Its definition is duration greater than `promised_hours`, with equality counted as on time. The promise is both a target-definition ingredient and an approved predictor because it is recorded at intake. The classifier can distinguish an identical job with a short promise from one with a generous promise. Its availability must be verified, and the original intake promise must be preserved. If promise-setting practices change, the lateness problem can change even when delivery operations do not.

A probability is not a decision. A probability of 0.70 is a model's estimated risk under its learned data conditions. A review rule might flag probabilities at or above a threshold. The threshold depends on missed delays, unnecessary reviews and capacity; it is not automatically 0.50. We postpone its selection until we have separate data for it.

## Worked miniature: build a baseline before adding features

These four training durations are an arithmetic example, not results from the generated course data: 2, 4, 4 and 10 hours. The mean is 20 divided by 4, or 5. The median is 4. A mean baseline predicts 5 for every future job; a median baseline predicts 4. Neither needs distance or queue size. That simplicity is useful because it tells us what a feature-based model must improve on.

Now consider a separate miniature evaluation set with durations 3, 5 and 8. The mean baseline's absolute errors are 2, 0 and 3, giving mean absolute error, or MAE, of 5/3, approximately 1.67 hours. Its squared errors are 4, 0 and 9, giving mean squared error of 13/3. Taking the square root gives RMSE of approximately 2.08 hours. The median baseline has absolute errors 1, 1 and 4, so MAE is 2 hours. It would be a mistake to conclude that the training median was calculated incorrectly: a training-optimal constant need not win on every new sample.

Notice the direction of information. We calculated each constant on training durations, then evaluated it unchanged. Replacing it with the evaluation mean would make the score an estimate of a rule that was allowed to see the answers. In the executable workflow, `DummyRegressor` provides the same kind of reference. The [versioned API](https://scikit-learn.org/1.8/modules/generated/sklearn.dummy.DummyRegressor.html) documents the mean and median strategies.

For classification, imagine training contains 8 late jobs out of 40. A constant probability baseline can predict 0.20 for every job. A majority-label baseline always predicts on time. Its high accuracy on a mostly on-time sample says little about whether it catches delays. Later lessons compare both probability quality and review decisions against appropriate baselines.

## Define success without promising an outcome

Choose the primary metric before seeing candidate scores. MAE is easy to interpret in hours. RMSE reveals whether a few large errors dominate. For classification, ranking, probability quality and alert burden answer different questions. Write one main selection criterion and a small set of checks, rather than selecting whichever metric makes a favored model look good.

A model that beats a baseline by a negligible amount may not justify maintenance. Conversely, a useful average improvement can hide a poor result for a small zone. Evaluation should report the sample size, period and conditions as well as the number. The Statistics course's distinction between practical importance and numerical difference still applies. This course's simulated data establish a learning exercise, not business value at a real depot.

## Guided execution and interpretation

Run `python examples/01_contract.py` from the course root after setup. Before running, write which fields should be predictors and which should be retained only for auditing or label construction. Read the printed contract. Locate baseline results in the generated outputs when the complete workflow has run; actual values belong in `outputs/reference_results.json`. Do not substitute the miniature numbers above.

Keep a short decision log: question, chosen baseline, metric, reason and date. The log should let someone distinguish an assumption made before analysis from an explanation invented after the results. You will extend it rather than rewrite history at every lesson.

## Independent practice

1. **01.1 · Specify the moment.** Write a six-sentence contract for duration prediction, including grain, intake time, allowed inputs, outcome time, use and one non-use. Explain why an intake prediction and a mid-job prediction are different products.
2. **01.2 · Calculate without fitting software.** Training durations are 1, 3, 3, 9 and 14. Calculate the mean and median constant predictors. On new durations 2, 4 and 12, calculate each MAE. Explain why the two training constants differ.
3. **01.3 · Challenge the objective.** A colleague proposes selecting the classifier solely by accuracy because only 10% of jobs are late. Write a response that includes a baseline, a failure that accuracy can hide and one additional measurement needed for the review workflow.

## Exit checks

- **01.A:** What information may `fit` use that a real intake-time `predict` call must not require?
- **01.B:** Why does beating a baseline on training data fail to establish usefulness on future jobs?


# 02 · Make every label and feature earn its place

## A table is a history of measurements

A CSV looks simultaneous: every cell is visible when you open it. Operational data are not simultaneous. A job is created, its queue is measured, its promise may be amended, it completes, and its outcome may be confirmed later. A predictive table must reconstruct what was knowable at a specific moment. This is the practical continuation of the SQL course's cutoff and feature-window lesson.

Data provenance means being able to explain where a value came from and how it was produced. For each predictor, record its source, unit, valid range, availability timestamp and update behavior. `queued_jobs` should describe the queue at this job's intake, not today's queue or the largest queue encountered later. `priority` should be the intake value, not a final escalation status. A familiar column name is not evidence that the value existed in time.

Our approved inputs are `distance_km`, `queued_jobs`, `item_count`, `priority`, `zone`, `depot_id`, `weekday` and the intake value of `promised_hours`. `job_id` and timestamps remain useful for auditing and joining; they are not automatically predictors. `label_available_at`, `final_delay_reason` and `close_cost` are explicitly forbidden predictors because they become available after the outcome. In this synthetic dataset, `label_available_at` also records completion; a separate `completion_timestamp` in a real export would be forbidden for the same reason. The raw dataset retains traps so you can demonstrate exclusion rather than merely trust a conveniently cleaned table.

## Define the target precisely

Duration is elapsed time from intake to completion, expressed in hours. If timestamps carry time zones, normalize deliberately before subtraction. A negative elapsed time is an integrity problem, not a difficult prediction. The label `late` equals 1 when duration exceeds the intake promise and 0 otherwise. Equality is on time. A missing duration means the outcome is not yet known; it does not mean the job was on time.

Two times matter for a label. The event time says when the outcome happened. The label-availability timestamp says when the system could actually retrieve a trustworthy label. A completion can occur at 16:00 but only be confirmed at 18:00. A model fitted at 17:00 cannot use that label. A realistic training snapshot therefore checks label availability against its fit cutoff. A gap in calendar days is only a safeguard; the timestamp check is the evidence.

Unfinished jobs create a further problem. An export containing only completed jobs can preferentially omit long jobs near its endpoint. That selection makes the observed sample look faster. The synthetic lab provides completed outcomes, but a real system must define a maturity window or a method for censored observations. We do not turn unresolved jobs into zeros or claim to teach survival analysis in this course.

## Worked miniature: audit the timeline

Consider three hypothetical jobs. A arrives Monday 09:00, finishes 13:00 and has a confirmed label at 14:00. B arrives Monday 10:00, finishes Tuesday 10:00 and is confirmed Tuesday 12:00. C arrives Monday 11:00 and remains unfinished at the extraction cutoff. A model will be fitted Monday 18:00.

A's duration of 4 hours is available for that fit. B's eventual 24-hour duration is not, even if a later export contains it. C has no observed duration at the cutoff. Including B uses information from the future; labeling C as zero fabricates an outcome. If a prediction is made for B at Monday 10:00, its completion time is also forbidden as a feature. The same field can be appropriate for constructing a historical label while being inappropriate for constructing historical inputs.

Now suppose A's intake promise is 4 hours and B's is 20. A is on time because 4 is not greater than 4. B is late once its label becomes available. If someone changes A's promise to 3 hours after completion, constructing the target from that amended promise would change the question. Preserve the version used by the contract.

## Leakage has several routes

Target leakage occurs when a predictor directly or indirectly contains the answer. A final delay reason is nearly a label in another form. Temporal leakage occurs when a value was unavailable at the prediction moment. Train–evaluation leakage occurs when held-out observations influence preprocessing, feature selection, model fitting or repeated decisions. A perfectly timed feature can still participate in the last kind if its imputer was fitted on the entire table.

Duplicates can also leak. If the same job appears in two exports and one copy reaches training while the other reaches evaluation, the score partly measures memory. Grouping by `job_id`, checking uniqueness, and reconciling row counts matter before fitting. Do not remove duplicates by arbitrary first occurrence when records contain conflicting updates: establish which version was available at intake and preserve an issue log.

SQL join discipline carries over to pandas. Joining one intake row to three status rows creates three apparent jobs unless you select the valid status snapshot first. Use a declared key relationship and inspect unmatched rows. pandas provides merge validation and an indicator column for these audits; its [2.2 merge reference](https://pandas.pydata.org/pandas-docs/version/2.2/reference/api/pandas.DataFrame.merge.html) also warns that null keys can match each other, unlike usual SQL behavior. Do not casually equate pandas and SQL joins.

## Missingness and measurement are part of the model

A missing distance could mean an intake failure, an unmeasured route or a newly introduced zone. Imputation replaces a numerical gap for computation; it does not resolve which explanation is true. Record missing rates by time and source. A feature that is usually missing for express work can accidentally act as a priority proxy, and its meaning may change when a logging system is repaired.

Run `python examples/02_provenance.py`. Inspect the allowed-feature check, target boundary checks, row identities and timestamp assertions. State what each check establishes and what it cannot establish. A script can verify that a timestamp precedes another timestamp; it cannot establish that a source system's clock or business definition was truthful without outside evidence. Preserve the raw file unchanged and make accepted data a derived artifact.

## Independent practice

1. **02.1 · Write an availability ledger.** For `queued_jobs`, `label_available_at`, `priority`, `close_cost` and `job_id`, specify source role, earliest legitimate availability and whether it belongs in X. Give a concrete leakage route for one forbidden field.
2. **02.2 · Audit a fit cutoff.** A fit happens Wednesday 09:00. Job P completes Tuesday 18:00 and is confirmed Wednesday 10:00. Q completes Tuesday 20:00 and is confirmed Tuesday 21:00. R is still open. Decide which labels may train the model and explain each decision without inventing missing labels.
3. **02.3 · Repair a multiplied table.** An intake table has 100 unique jobs. Joining to status history yields 260 rows. Describe an audit and reconstruction plan that preserves the intake grain and history, including a check for unmatched jobs and conflicting records.

## Exit checks

- **02.A:** Why is a post-outcome field allowed in target construction but forbidden in intake predictors?
- **02.B:** Why does an embargo not remove the need to check label-availability timestamps?


# 03 · Choose a split that resembles the future

## A score answers the question posed by its split

Training data teach the rule. Validation data support development choices. Test data provide a final, previously unused check of the frozen procedure. These are roles, not three random filenames. If you inspect test errors to choose features and then report a revised test score, the file has become development data regardless of its name.

The operational question is: how well will a model trained on earlier completed jobs predict later incoming jobs at the represented depots? A chronological split mimics that direction. A random split mixes earlier and later conditions and can make training unrealistically aware of future operating patterns. Shuffling does not destroy only order; it changes the question being measured.

Random splitting can be appropriate for an independent, exchangeable sample when future use resembles another draw from the same population. Stratification can preserve approximate class proportions in such a split. Neither is a universal default. A time-varying late rate may be part of the problem, so we do not reshuffle time periods until their rates match. The [scikit-learn cross-validation guide](https://scikit-learn.org/stable/modules/cross_validation.html) distinguishes these designs and the dependencies they address.

## Read the actual course boundaries

There are 1,500 synthetic jobs, six per day over 250 days numbered 0 through 249. Training uses days 0–148: 149 days times six jobs equals 894. Day 149 contributes six embargoed jobs. Validation uses days 150–198: 49 days times six equals 294. Day 199 contributes another six embargoed jobs. Final test uses days 200–249: 50 days times six equals 300. The counts reconcile: 894 + 6 + 294 + 6 + 300 = 1,500.

Validation is itself allocated before model selection. Days 150–173 provide 144 calibration jobs. Day 174 is a six-job embargo. Days 175–198 provide 144 threshold-policy jobs. Thus 144 + 6 + 144 = 294. The calibration segment teaches a probability adjustment; the later policy segment chooses a review threshold. They are not interchangeable chances to obtain a better score.

An embargo leaves time between a training or fitting period and the next evaluation period. Here generated durations are bounded at 23.5 hours, so a full-day outer gap supports outcome maturity. We still verify label timestamps. In real records, reporting delays or revised labels could exceed an assumed duration bound; an apparently generous gap could remain inadequate.

## Worked miniature: chronology is not enough by itself

Suppose jobs arrive once per day. We propose training on days 0, 1 and 2, skipping day 3, and evaluating day 4 onward. The day-2 job arrives at 09:00 and its label becomes available on day 4 at 11:00. If our first evaluation prediction is at day 4 at 09:00, that label is unavailable for the fitted model. The row order looks correct and a day was skipped, but the information boundary still fails.

There are two defensible repairs: move the fitting/evaluation boundary later, or omit that not-yet-mature label from the training snapshot according to a predeclared rule. Retaining it because its eventual outcome is in the CSV is not a repair. Notice also that deleting every slow job can change the training population. A repair must explain selection consequences, not merely make an assertion pass.

Now consider six miniature jobs: depot A on days 1, 2 and 3 and depot B on days 1, 2 and 3. A chronological split evaluates later jobs from already represented depots. Holding all B jobs out instead asks about transfer to an unseen depot. Both are valid questions, but neither answers the other. For a combined future-and-new-depot claim, both restrictions must be designed together; a single convenient split is insufficient.

## Where group splitting belongs

Repeated entities can share operating conditions. If the intended use were a new depot, training and evaluation should not share depot IDs. Group splitters separate whole groups, even when group sizes differ. Group leakage can also occur through repeated jobs or copies of the same original record.

For this course, a group holdout is a clearly marked hypothetical comparison, not a replacement for the temporal final holdout. A new-depot test would also make an unseen category appear at prediction time; preprocessing must cope without pretending it has learned that depot's effect. This connects split design to the next lesson's unknown-category policy.

## Cross-validation stays inside training

Model comparisons need more than one lucky boundary, but must preserve the chosen information direction. The course uses `TimeSeriesSplit(n_splits=4, gap=12)` on the ordered 894 training jobs. Earlier portions train each candidate; later portions evaluate it. Twelve rows correspond to two days in this regular six-jobs-per-day dataset. In an irregular real stream, a row gap would not guarantee a particular elapsed time.

Do not read “time-series” as proof that every split respects whole days or label maturity. The splitter uses row positions, and its default fold sizes can cut within a day. Inspect actual indices, arrival times and label-availability timestamps. Its [1.8 reference](https://scikit-learn.org/1.8/modules/generated/sklearn.model_selection.TimeSeriesSplit.html) specifies the meaning of `gap` and notes the importance of equally spaced samples for comparable time durations.

Run `python examples/03_splits.py`. Draw the full allocation and one inner fold before studying the printed output. Verify disjoint IDs, date direction, embargo exclusion, exact counts and target availability. A total count alone cannot detect a swap between two partitions. Save the allocation with the experiment so a later score can be connected to an exact evaluation population.

## Independent practice

1. **03.1 · Reconcile the design.** Make a seven-block timeline for training, outer embargo, calibration, inner embargo, policy, outer embargo and final test. Give inclusive day bounds and row counts for every block, then reconcile to 1,500.
2. **03.2 · Select the question.** Propose split designs for (a) later jobs at current depots, (b) a new depot under otherwise similar conditions and (c) a new depot next year. Explain what each design can and cannot support. Do not change the course's actual temporal split.
3. **03.3 · Find the hidden future.** An inner fold's training arrivals end Friday, evaluation starts Sunday at 08:00, and one training label is confirmed Sunday at 10:00. Explain the failure and two possible repairs. Include why a row-gap check alone is insufficient.

## Exit checks

- **03.A:** Why is stratifying the whole dataset by lateness not an acceptable substitute for the course's chronological holdout?
- **03.B:** What new question is introduced when entire depots, rather than later dates, are held out?


# 04 · Treat preprocessing as part of the learned model

## Learning begins before the estimator

A model rarely receives the original table unchanged. Some numerical values are missing, categorical strings need a numerical representation, and different scales affect regularized linear models. These preparation steps can learn from data. A median imputer learns a median; a scaler learns a mean and spread; an encoder learns category names. If they learn from held-out rows, information has crossed the boundary before the estimator's `fit` begins.

Separate fixed validation from learned preparation. A fixed rule can reject a negative item count because the contract defines counts as nonnegative. A learned rule that declares an unusually high count an outlier based on all 1,500 jobs has seen the final period. Its threshold must instead be fitted inside the relevant training portion, or be justified independently before analysis. “Cleaning” is not an exemption from evaluation discipline.

The course uses five numerical columns: distance, queued jobs, item count, weekday and promised hours. It uses three categorical columns: priority, zone and depot ID. Weekday is numerically coded in this baseline. A linear model consequently gives it a straight-line contribution, even though Sunday and Monday are adjacent in a cycle. That limitation is a reasoned feature-design question for training-only development, not permission to tune using test errors.

## Worked miniature: fit once, transform later

Imagine training distances of 2, missing and 6 kilometers. Median imputation learns 4, producing 2, 4 and 6. Their mean is 4. The population variance used by `StandardScaler` is (4 + 0 + 4)/3 = 8/3, and the scale is its square root, about 1.633. Standardization subtracts the training mean and divides by that scale. The transformed training values are approximately −1.225, 0 and 1.225.

A later missing distance becomes 4 under the stored imputer and therefore 0 under the stored scaler. A later distance of 10 becomes (10 − 4)/1.633, approximately 3.674. We do not recompute the mean using 10. Being outside the training range is information to inspect, not a reason to silently redefine the representation. This hand example illustrates fitted state, not a promise that imputation recovers a true missing distance.

The reference numerical branch also adds a missingness indicator for columns that are missing during fitting. For this miniature distance column, its pre-scaling values are [0, 1, 0]. The hand calculations above concern only the distance-value column; the indicator is a separate transformed column. An indicator describes missingness, not the cause of missingness.

Suppose the training zones are north, south and north. A one-hot encoder creates a north indicator and a south indicator. North becomes [1, 0]; south becomes [0, 1]. An unseen west zone can become [0, 0] when `handle_unknown="ignore"` is used. That prevents a technical crash, but the model has no learned west-specific effect. A missing zone and an unseen zone also need not mean the same thing. Their handling must be documented.

## Assemble one object with an inspectable boundary

A `ColumnTransformer` routes named numerical and categorical columns to separate preparation branches. A `Pipeline` connects the combined transformed matrix to an estimator. Fitting the pipeline fits its preparation on the supplied training rows, then fits the estimator on the transformed training matrix. Predicting applies the stored transformations and the stored estimator. These API roles are documented in the [ColumnTransformer reference](https://scikit-learn.org/1.8/modules/generated/sklearn.compose.ColumnTransformer.html) and [OneHotEncoder reference](https://scikit-learn.org/1.8/modules/generated/sklearn.preprocessing.OneHotEncoder.html).

The key unit to pass into cross-validation is the entire unfitted pipeline. Each fold then learns its own preparation from that fold's training rows. Precomputing one transformed table before cross-validation does not create independent fold preparation. The [common-pitfalls guide](https://scikit-learn.org/stable/common_pitfalls.html) explains why this placement prevents a common source of leakage.

Use an explicit feature allowlist. Selecting all numerical columns would accidentally admit duration, close cost or a timestamp converted to an integer. Dropping only a few known bad columns is fragile when the export gains new columns. The model's intended schema is smaller and more stable than the raw audit table.

## Why scaling and encoding need interpretation

For Ridge and logistic regression, regularization penalizes coefficient size. When one feature is in kilometers and another in thousands of items, equal coefficient penalties do not represent comparable changes in the original measurements. Scaling creates a more comparable numerical basis, though it does not guarantee equal importance. Trees split on individual thresholds and usually do not need standardization for that reason. Keeping a clear pipeline is still worthwhile for consistent imputation and category handling.

One-hot encoding avoids imposing a false numerical order on depot labels. Encoding depot A as 1 and depot B as 2 would invite a linear model to treat their difference like a quantity. However, one-hot encoding does not make IDs causal or harmless. A depot category can capture location, equipment and unobserved processes simultaneously. A unique job ID has little reusable meaning and may enable memorization.

The transformation also changes shape. Eight raw predictors may become many matrix columns. Inspect names and dimensions, and check the matrix remains practical for CPU memory. The lab is deliberately small; a production table with millions of distinct IDs could make one-hot encoding inappropriate. Never assume a successful toy transformation establishes scale readiness.

## Guided inspection

Run `python examples/04_preprocessing.py`. Locate the fitted numerical statistics, learned categories and transformed shape. Supply a copied intake record with one missing numerical value and one unseen category. Predict how it will transform before running. Do not edit the stored raw data to make the example convenient. Verify the same raw column order and names are accepted by the fitted pipeline and that forbidden columns are excluded by construction.

A safe failure is valuable. Missing a required input column should produce a clear contract error, not quietly substitute a new meaning. Saving only estimator coefficients loses the transformations that give those coefficients meaning. The complete fitted pipeline belongs in the eventual handoff, together with schema and version information.

## Independent practice

1. **04.1 · Trace learned state.** Training queue counts are 0, missing and 8. Work through median imputation, training mean, population scale and transformed values. Transform new counts missing and 12 without refitting. State the units before and after scaling.
2. **04.2 · Diagnose leakage.** A notebook imputes all rows, encodes all categories and then cross-validates only the classifier. Explain which steps learned from held-out rows. Describe a replacement pipeline and exactly where it must be passed during validation.
3. **04.3 · Test the schema.** Design four input tests: an unseen zone, a missing distance, a missing required column and a supplied forbidden column. For each, state the intended behavior and what evidence would show the behavior occurred.

## Exit checks

- **04.A:** Why does an unseen category producing an all-zero indicator block not mean the model understands that category?
- **04.B:** Which objects must be refitted separately inside each cross-validation training fold?


# 05 · Compare regression rules and inspect their mistakes

## A model family is a restriction on possible rules

Regression predicts a numerical quantity. In our case, a prediction of 7.4 means hours from intake to completion, not a probability and not an exact appointment. The model's family determines which patterns it can represent. That choice trades flexibility against data requirements, stability, computation and ease of inspection.

A linear model combines an intercept with weighted inputs. In readable notation, predicted duration equals intercept plus coefficient one times feature one, plus coefficient two times feature two, and so on. After one-hot encoding, categorical levels contribute indicator terms. This is linear in the transformed columns; it need not be linear in an original variable if you deliberately construct nonlinear features.

Ridge regression fits such a model while penalizing large squared coefficients. Its objective combines squared training error with alpha times the sum of squared coefficients. Alpha controls the strength of that penalty. Larger alpha generally shrinks coefficients more, though individual predictions need not change monotonically. Scaling matters because the penalty acts on numerical coefficient sizes. Ridge's [official 1.8 documentation](https://scikit-learn.org/1.8/modules/linear_model.html#ridge-regression-and-classification) describes this objective.

A decision tree asks a sequence of threshold questions, such as whether queue size exceeds a value. Each leaf predicts from training jobs reaching it. A deep tree can memorize small peculiarities. A random forest averages many randomized trees, reducing some instability while retaining nonlinear splits and interactions. It is not guaranteed to win, and its extrapolation outside familiar ranges differs from a linear model's. The lab compares a median baseline, Ridge and a bounded random forest on the same temporal folds.

## Worked miniature: calculate and criticize a prediction

Use this invented rule only for hand tracing: predicted hours equals 1 + 0.10 times distance + 0.40 times queued jobs. A job with distance 20 and queue 5 receives 1 + 2 + 2 = 5 hours. If it actually takes 7 hours, its residual, defined here as actual minus predicted, is +2 hours. Positive residuals mean underprediction; negative residuals mean overprediction. Always state your sign convention because plotting libraries and colleagues may use the opposite one.

Suppose three separate evaluation jobs have actual durations [3, 7, 10] and predictions [4, 5, 9]. Residuals are [−1, 2, 1]. MAE is (1 + 2 + 1)/3 = 4/3 hours. RMSE is the square root of (1 + 4 + 1)/3, about 1.414 hours. The mean residual is 2/3 hour, suggesting underprediction on this tiny set. That bias summary does not reveal how dispersed the individual errors are.

Now compare two four-job error patterns: A has absolute errors [1, 1, 1, 5], and B has [2, 2, 2, 2]. Both have MAE 2. A's RMSE is the square root of 7, about 2.646; B's is 2. RMSE penalizes concentrated large errors more strongly. Neither metric knows the actual staffing consequences. A declared practical requirement might care about the frequency of underprediction above three hours, which requires its own reported count.

## Residuals make averages inspectable

Plot actual against predicted with a clearly labeled identity line. Plot residual against predicted duration, queue size and time. A curve can suggest a missing nonlinear relationship. Increasing residual spread may show that long jobs are harder to predict. Consecutive positive residuals late in time can indicate a changing process. These are diagnostic clues, not automatic diagnoses or proof that adding a feature will fix the issue.

NIST's [model-validation discussion](https://www.itl.nist.gov/div898/handbook/pmd/section4/pmd44.htm) motivates examining residual patterns rather than relying on one goodness-of-fit statistic. For this course, plot patterns are especially useful because a single average can hide the synthetic process shift. Avoid classical uncertainty claims that assume independent, identically distributed errors when the plot or operating context suggests otherwise.

R-squared compares squared prediction error to predicting the evaluation sample's mean: one minus residual sum of squares divided by total squared deviation around that mean. It is unitless, can be negative on held-out data and becomes problematic for constant targets. A negative value means worse squared error than that comparison reference. R-squared is not the percentage of predictions that are correct and does not establish causality. MAE and RMSE retain the hours useful to the depot question.

## Selection discipline matters more than a leaderboard

The course selects the regression family with the lowest mean MAE across four training-only temporal folds. It also records RMSE, rather than switching the winner according to whichever number looks attractive afterward. A small difference with substantial fold variation deserves caution. Later folds contain more training data and different time periods, so fold standard deviation is not a ready-made confidence interval for future performance.

Keep candidate complexity modest. The reference forest has a finite tree count, depth limit and minimum leaf size; it is an instructional CPU model, not an exhaustive search. A more flexible model may improve training scores while harming future scores. A simpler model may underfit an important interaction, such as queue effects becoming stronger under congestion. Inspect evidence before turning either possibility into a story.

Run `python examples/05_regression.py`. Locate each fold's MAE and RMSE in `outputs/reference_results.json` after the workflow is generated. Write the selected family, the rule used to select it and any inconsistent fold ordering. Then trace two predictions back to raw feature records and their residuals. Use actual output values, not the worked miniature. A plot copied from reference outputs is evidence about the reference run, not proof that your own implementation works.

## Independent practice

1. **05.1 · Calculate an error report.** Actual durations are [2, 6, 8, 12]; predictions are [3, 4, 9, 10]. Calculate signed residuals, MAE, RMSE and mean residual. Explain what each reveals and one thing each omits.
2. **05.2 · Choose a defensible next experiment.** Ridge underpredicts only when queues are high; a forest has lower training error but worse later-fold MAE. Propose one limited training-only experiment and a comparison plan. Do not assume either model is correct.
3. **05.3 · Audit a model claim.** Rewrite: “Our R-squared is 0.8, so 80% of jobs will be predicted correctly and increasing queue size causes longer jobs.” Separate predictive fit, individual error and causal interpretation.

## Exit checks

- **05.A:** Why can two models have equal MAE but different RMSE?
- **05.B:** Why must the full preprocessing pipeline, rather than only the regressor, enter each temporal fold?


# 06 · Read classification metrics as counts and questions

## Predicting lateness adds a probability layer

The label is 1 if `duration_hours > promised_hours` and 0 otherwise. A classifier can produce a score, a probability estimate or a hard label. These outputs serve different purposes. Ranking jobs requires a useful ordering; interpreting a value as a chance requires probability quality; creating a review list requires a decision threshold. A single accuracy number cannot verify all three.

Logistic regression starts with a weighted combination of prepared features, called a logit. The sigmoid converts it into a number between zero and one: probability equals one divided by one plus exp of negative logit. A logit of zero yields 0.5. A logit of approximately 1.386 yields 0.8. The curve changes smoothly, so the same coefficient does not add the same number of probability points everywhere. Despite its name, logistic regression is a classification model here.

The course compares a constant prior-probability baseline, logistic regression and a random forest classifier in training-only temporal validation. Its main probability-policy integration uses logistic regression as a deliberately fixed family. That is a pedagogical choice, not a claim that logistic regression beats every other family. Its regularization and class-weight options are chosen later with a declared training-CV criterion.

## Worked miniature: turn alerts into a confusion matrix

Imagine 20 new jobs, four of which are late. A threshold produces six alerts: three late jobs and three on-time jobs. We have three true positives, three false positives, one false negative and thirteen true negatives. Count all four cells: 3 + 3 + 1 + 13 = 20.

Accuracy is correct decisions divided by all jobs: (3 + 13)/20 = 0.80. Precision is late jobs among alerts: 3/6 = 0.50. Recall is alerted jobs among all late jobs: 3/4 = 0.75. Specificity is correctly unflagged on-time jobs divided by all on-time jobs: 13/16 = 0.8125. Balanced accuracy averages recall and specificity, giving 0.78125. F1 is twice precision times recall divided by their sum: 0.60.

The majority baseline also obtains 0.80 accuracy by calling every job on time. It produces no alerts and catches none of the four late jobs. The equal accuracies conceal different workflows. Precision tells the reviewer how often an alert is right on this sample; recall tells operations how many observed delays were caught. Neither measures the number of reviews directly, which is six, or 30% of jobs.

If there are no predicted positives, precision's denominator is zero. A library may return zero by an explicit convention, but the report should still say there were no alerts. Likewise, ROC AUC cannot be meaningfully estimated on a sample containing only one observed class. Record undefined situations rather than hiding them behind a convenient default.

## Ranking and probability quality ask different questions

ROC AUC evaluates ranking across positive-negative pairs. Informally, it asks how often a randomly selected late job receives a larger score than a randomly selected on-time job, with ties handled appropriately. It does not specify a review threshold, a probability interpretation or an alert budget. Under strong imbalance, the false-positive rate can look small while the absolute number of false alerts is large.

A precision-recall curve shows how precision and recall change as the threshold changes. Average precision summarizes the precision achieved at recall increases. scikit-learn's average precision is not simply the trapezoidal area under an interpolated curve; the [metrics guide](https://scikit-learn.org/1.8/modules/model_evaluation.html#precision-recall-f-measure-metrics) gives the precise distinction. The positive-class prevalence is essential context, because a task with many more late jobs has a different baseline precision.

Probability metrics use the probabilities themselves. Binary Brier score averages squared differences between probability and observed 0/1 outcome. For two miniature predictions, p = 0.8 with y = 1 and p = 0.3 with y = 0, the score is [(0.8 − 1) squared + (0.3 − 0) squared]/2 = 0.065. Log loss penalizes confident wrong probabilities especially strongly. For a positive outcome the contribution is negative log(p); for a negative outcome it is negative log(1 − p). Lower values are better for both metrics. Neither proves calibration by itself.

## Imbalance is not a command to rebalance everything

A rare positive outcome may be exactly what occurs in deployment. Balancing the evaluation sample would then change the prevalence and some operational interpretations. Keep evaluation representative of its intended period, and report its class counts. Weighted fitting can emphasize the minority class, but its probabilities may need calibration. The class-weight option belongs inside training-only model selection.

Resampling before splitting can copy or derive examples across evaluation boundaries. Synthetic oversampling also has assumptions about meaningful neighborhoods. This course does not require it. First establish a valid split, a baseline, a suitable loss and clear error counts. Complexity should answer an identified problem, not compensate for a misunderstood metric.

Run `python examples/06_classification.py`. Reconstruct a confusion matrix from the printed probabilities and threshold. Check that counts sum to the evaluation sample, recompute precision and recall by hand for one result, and state the positive class. Compare a probability baseline with a fitted model using the same rows. Record actual results from `outputs/reference_results.json`; the 20-job miniature is only a teaching trace.

## Independent practice

1. **06.1 · Rebuild the report.** Among 50 jobs, ten are late. A model alerts on eight late jobs and six on-time jobs. Calculate all four confusion counts, accuracy, precision, recall, balanced accuracy and alert fraction. Compare accuracy with an always-on-time baseline.
2. **06.2 · Score probabilities.** For outcomes [1, 0, 1], system A predicts [0.8, 0.2, 0.6] and system B predicts [0.6, 0.4, 0.55]. Calculate Brier scores. Explain why using threshold 0.5 alone would miss the difference between the systems.
3. **06.3 · Choose evidence for a claim.** A colleague says, “AUC is high, so probabilities are trustworthy and the review queue will stay small.” Identify the two unsupported conclusions and specify evidence needed for each, including an actual count or fraction.

## Exit checks

- **06.A:** How can an all-on-time classifier have high accuracy while being useless for finding late jobs?
- **06.B:** Which distinction separates ranking quality, probability quality and a thresholded decision policy?


# 07 · Cross-validation, tuning and test lockdown

## The question behind the score

You can now fit a preprocessing pipeline and compare a model with a baseline. The next question is whether your choice survives another period of work. A model that happened to suit one quiet week might fail during the next busy week. Cross-validation gives several rehearsals of the intended prediction situation. It does not manufacture new independent evidence or guarantee future performance.

In the SQL course, a feature cutoff separated information available now from information arriving later. Apply that same discipline to every training fold. Here one row is one synthetic depot job, predicted at intake. Regression predicts `duration_hours`; classification predicts whether `duration_hours > promised_hours`. Permitted inputs are distance, queue size, item count, priority, zone, depot, weekday and the promise known at intake. `label_available_at` audits outcome availability; this timestamp, final delay reason and close cost must never enter the predictor.

The full dataset contains 1,500 jobs: six per day for 250 days. Development training uses days 0–148, or 894 jobs. Day 149 is an embargo. Validation uses days 150–198, or 294 jobs. Day 199 is another embargo. Days 200–249 contain the final 300 test jobs. An embargo is a deliberate unused interval; it does not itself establish that earlier labels were available.

## Separate three decisions

Fitting estimates model parameters from allowed training data. Tuning selects settings, such as a tree's minimum leaf size, by comparing candidate procedures. Evaluation measures the frozen procedure on evidence that did not select it. Repeatedly choosing models from their final-test scores converts that test into development data, even if no test row appears in a call to `fit`.

For this course, run `TimeSeriesSplit(n_splits=4, gap=12)` on the chronologically ordered 894 training rows only. Each fold trains on earlier rows and scores a later block; the training window expands. The gap is measured in rows, not days. Twelve excluded rows represent two days of jobs because this synthetic schedule has six arrivals per day. Overnight spacing differs from within-day spacing, so inspect elapsed-time boundaries as well as row counts. Irregular real arrivals require timestamp-based reasoning. [TimeSeriesSplit documentation](https://scikit-learn.org/1.8/modules/generated/sklearn.model_selection.TimeSeriesSplit.html)

A separate group-split thought experiment asks whether a model transfers to an unseen depot. Holding out depot identifiers answers that question, but does not preserve the main future-time evaluation automatically. Do not replace the declared final test with a group split because its result looks nicer. [Cross-validation guidance](https://scikit-learn.org/1.8/modules/cross_validation.html)

## Worked trace: inspect boundaries before scores

Use this miniature arithmetic example, not the course dataset: 20 ordered rows, numbered 0–19, with three splits and a two-row gap. Default validation block size is 20 divided by 4, giving five rows.

- Fold 1: train 0–2; gap 3–4; validate 5–9.
- Fold 2: train 0–7; gap 8–9; validate 10–14.
- Fold 3: train 0–12; gap 13–14; validate 15–19.

Notice that an earlier validation row can become a training row in a later rehearsal. That is sensible only when its outcome has become known by that later fit boundary. Suppose row 2 arrived on day 2, but its completed duration became available on day 6. A fold fitted at the start of day 5 cannot use that label, even though row 2 precedes its two-row gap. The timestamp audit must reject this setup or apply a documented, appropriate maturity filter before fitting. Never silently accept unavailable labels.

For the actual 894-row training table, default fold validation size is 178. The candidate training sizes are 170, 348, 526 and 704; each gap contains 12 rows. The first validation indices are 182–359 and the last are 716–893. Check generated indices and actual label-availability timestamps; these counts alone are insufficient proof.

## Keep preprocessing inside each rehearsal

Suppose a numeric imputer replaces a missing queue count with a median. If that median was computed using all 894 training rows before cross-validation, early folds have already learned about later observations. The outer table is called training data, but the later rows are still held out inside those folds. Fit imputation, scaling and encoding separately within each training fold through a pipeline. Apply the learned transformations to that fold's validation rows. [Leakage and pipelines](https://scikit-learn.org/1.8/common_pitfalls.html)

Use a small, declared search. Regression selects among median, Ridge and random-forest procedures using mean training-CV MAE. In a hypothetical tree search, three leaf sizes across four folds would require twelve candidate fits. The actual logistic search uses three C values times two class-weight choices times four folds: twenty-four candidate fits, then a selected-model refit. Search cost grows through multiplication, not addition. Report the candidate grid and selection metric so the choice can be inspected. [GridSearchCV reference](https://scikit-learn.org/1.8/modules/generated/sklearn.model_selection.GridSearchCV.html)

## Worked trace: interpret a search result

In another miniature example, candidate A has fold MAEs of 2.0, 2.2, 3.8 and 4.0 hours. Its mean is 12.0 divided by 4, or 3.0 hours. Candidate B has 2.8, 2.9, 3.0 and 3.3, also averaging 3.0. Equal means conceal different behavior over time. A declared preference for stability might favor B; a declared complexity tie-breaker could favor a simpler procedure. Do not invent a tie-breaker after viewing the final test.

Scikit-learn search scorers maximize scores. `neg_mean_absolute_error` therefore represents MAE with a negative sign: −2.4 is better than −3.1. Convert back to positive hours when explaining error. Inspect fold scores as well as their average. Their standard deviation is descriptive spread, not an automatic confidence interval: folds share training observations and later periods may differ systematically.

## Run, inspect and freeze

Run `python examples/07_validation.py` from the project root. Use `outputs/reference_results.json` for the actual measured results; none of the hand traces above claims to be model output. Inspect candidate settings, fold boundaries, label-maturity checks and score direction. A successful process exit is only a starting point for this audit.

Before final evaluation, freeze the feature list, cleaning rules, pipeline, selected settings, calibration method, threshold, metrics and interpretation plan. The development command `python -m courseml.run_all` belongs before that boundary. Run `python -m courseml.final_evaluation` only after the freeze. A disappointing final result is evidence to report, not permission to retune and call the same test untouched.

## Independent practice

### Exercise 07.1 · Trace a different split

For 30 ordered rows numbered 0–29, derive both folds of `TimeSeriesSplit(n_splits=2, gap=3)`. State training, gap and validation indices. Then identify the additional timestamp condition required before either fit.

### Exercise 07.2 · Choose with evidence

Miniature candidate C has fold MAEs 1.6, 1.8, 3.2, 4.6; D has 2.4, 2.5, 2.6, 2.9. Calculate both means, express their negative-MAE search scores, and recommend a candidate under a lowest-mean-MAE rule. Add one limitation the mean conceals.

### Exercise 07.3 · Repair the experiment

A teammate fits an encoder on all 1,500 rows, tunes settings on days 200–249, then proposes holding out one depot instead because the score improves. Write a corrected five-step plan that preserves the original future-jobs question and identifies what evidence is already contaminated.

## Exit checks

- **07.A:** Why can a pipeline prevent one kind of leakage while still failing to prevent unavailable labels?
- **07.B:** What can you honestly claim after inspecting a final test and then changing the model?


# 08 · Calibration and threshold tradeoffs

## A probability and a decision are different outputs

A predicted late probability of 0.70 says something about uncertainty. An alert says someone should take an action. Connecting them requires a decision rule. Better probability estimates do not automatically produce a better rule, and a useful rule does not establish that every probability is trustworthy.

Our label remains late when `duration_hours > promised_hours`. Equality is on time. The synthetic example concerns planning attention for depot jobs; it does not authorize consequential scoring of people or real operations. Inputs include the promise known at intake. Never introduce the completed duration, `label_available_at`, final delay reason or close cost as a predictor to improve the result.

Calibration asks whether groups assigned similar probabilities experience events at similar frequencies. If a large, relevant collection of jobs receives probabilities near 0.20, roughly one fifth being late would support that part of the probability scale. One job is not “20% late”; four observations cannot establish reliable calibration. [Probability calibration](https://scikit-learn.org/1.8/modules/calibration.html)

## Worked trace: check probability quality

Consider four miniature predictions: probabilities 0.20, 0.20, 0.70 and 0.90, with outcomes 0, 1, 1 and 1. These are miniature arithmetic examples.

The Brier score for a binary event is the average of squared differences between probability and outcome. The four squared differences are 0.04, 0.64, 0.09 and 0.01. Their sum is 0.78; dividing by four gives 0.195. Lower is better for this loss. Confidently assigning a low probability to an event that happens produces a large penalty. [Brier score reference](https://scikit-learn.org/1.8/modules/generated/sklearn.metrics.brier_score_loss.html)

For a tiny reliability display, put the first two predictions in a low-probability bin. Their average prediction is 0.20, while the observed late fraction is one out of two, or 0.50. The other bin has mean probability 0.80 and observed fraction 1.00. These differences describe this sample. Two observations per bin make a strong “miscalibrated everywhere” conclusion unjustified.

Choose bins before comparing models and show bin counts. Empty bins contain no empirical event rate. A lower Brier score alone does not prove better calibration, because overall probability performance also depends on separating easy and difficult cases. Read probability loss, the reliability display and discrimination together.

## Allocate evidence before fitting calibration

The validation period has 294 jobs across days 150–198. Allocate days 150–173 to calibration: 24 days times six jobs gives 144. Leave day 174 as an embargo. Allocate days 175–198 to threshold choice: another 144 jobs. The arithmetic reconciles as 144 plus 6 plus 144 equals 294.

The logistic classifier tunes C across 0.1, 1, 10 and class weight across none or balanced using training-only negative log loss. Its family is fixed for this probability exercise; comparison with dummy and forest models does not imply logistic wins. Calibration fits a mapping using the base model's predictions and labels from the calibration block. The base classifier stays frozen. In scikit-learn 1.8, `FrozenEstimator` supports calibration of an already fitted estimator; disjoint fitting data remains the practitioner's responsibility. Sigmoid calibration is a restrained choice for this small teaching block; flexible isotonic calibration can overfit small samples. [Calibration implementation guidance](https://scikit-learn.org/1.8/modules/calibration.html)

Verify that training labels existed at the base-model fit boundary and calibration labels existed before fitting the mapping for the threshold period. A six-row embargo is not that verification. Reporting fit-block calibration as if it were independent evaluation is another error: the mapping has already used those labels.

## Worked trace: threshold changes the workload

Take four new miniature jobs with probabilities 0.15, 0.40, 0.55 and 0.85 and outcomes 0, 1, 0 and 1. Define an alert when probability is greater than or equal to the threshold. At threshold 0.50, alert jobs three and four. There is one true positive, one false positive, one false negative and one true negative.

At threshold 0.30, alert jobs two, three and four. Now there are two true positives, one false positive, no false negatives and one true negative. Recall rises from one half to one, while alerts rise from two to three. Precision changes from one half to two thirds. Lowering a threshold cannot remove existing alerts under this fixed rule, but precision need not change monotonically on a finite dataset.

Suppose this miniature exercise assigns three cost units to a missed late job and one unit to a false alert. The first threshold costs 3 times 1 plus 1 times 1, or four units. The second costs one unit. These are invented instructional costs, not measured depot economics. [Decision-threshold guidance](https://scikit-learn.org/1.8/modules/classification_threshold.html)

## Name the decision assumptions

For perfectly calibrated probabilities, zero cost for correct decisions and constant error costs, alerting has expected cost C_FP times (1 minus p); not alerting has expected cost C_FN times p. Alert when p is at least C_FP divided by (C_FP plus C_FN). The miniature costs above yield 0.25. This arithmetic is a useful starting point, not a universal operational optimum.

Real review capacity, changing costs, uncertain probabilities and differences between jobs can break those assumptions. The course threshold comparison therefore records false positives, false negatives, precision, recall and number of alerts. Define the primary criterion and any capacity constraint in advance. If no candidate meets both, report that conflict rather than quietly relaxing a requirement.

Selecting a threshold uses labels even though it does not change the classifier's weights. Keep the final test closed during that choice. The course searches 0.10–0.90 in 0.05 steps, minimizing FP plus 4FN with at most 40% alerts. Ties prefer fewer alerts, then higher thresholds. If none is feasible, the implementation records an explicit no-alert fallback. Test alert volume may exceed 40%; report it.

## Run and explain

Run `python examples/08_calibration.py`. Read the measured output in `outputs/reference_results.json`; distinguish calibration-block fit information from threshold-block evaluation. Confirm that the positive class is late, the probability column matches that class and equality uses the declared alert rule. Explain the chosen threshold in workload and error terms. 

## Independent practice

### Exercise 08.1 · Audit probabilities

For probabilities 0.10, 0.40, 0.60, 0.90 and outcomes 0, 1, 0, 1, calculate binary Brier loss. Use the first two and last two observations as bins; calculate each bin's mean prediction and observed rate. State why these bins cannot establish population calibration.

### Exercise 08.2 · Compare two decisions

Probabilities are 0.10, 0.35, 0.45, 0.60, 0.80, 0.90; outcomes are 0, 1, 0, 1, 0, 1. Use alerts at probability at least the threshold. Compare thresholds 0.50 and 0.30: confusion counts, precision, recall, alert count and cost with C_FN = 4 and C_FP = 1. Then impose a maximum of three alerts and explain the conflict.

### Exercise 08.3 · Repair reused evidence

A teammate calibrates on all 294 validation jobs, chooses a threshold on those same labels and calls that score independent validation. Reallocate the stated day ranges correctly and explain which claimed independence is lost. Identify when the final test may be opened.

## Exit checks

- **08.A:** Can changing only the threshold improve calibration? Explain what changes and what stays fixed.
- **08.B:** Why does a cost-optimal threshold require assumptions beyond a low Brier score?


# 09 · Error analysis and interpretation limits

## Turn mistakes into a question

A single error score compresses a collection of jobs into one number. That helps comparisons but hides where a model is fragile. Error analysis restores context: which observations were difficult, which patterns repeat, and which explanations remain guesses?

Your statistics course distinguished an observed relationship from a causal effect. Preserve that distinction here. A predictor can use queue size to anticipate a long duration without proving that changing the recorded queue size would shorten a job. Likewise, a depot with larger errors might have harder jobs, fewer observations or different recording practices. Prediction performance is not an assessment of staff quality.

Use development validation for investigation. These rows already support calibration and threshold decisions, so descriptions of their errors are exploratory. Repeatedly adjusting a model to these same cases makes its validation result increasingly optimistic. Keep the final 300 jobs closed until the complete procedure and reporting plan are frozen.

## Worked trace: preserve the sign and the unit

Consider a miniature regression example with actual durations 4, 6, 8 and 10 hours, and predicted durations 5, 5, 7 and 7. Define residual as actual minus predicted. The residuals are −1, 1, 1 and 3 hours. Positive residuals mean the model underpredicted duration; negative residuals mean it overpredicted.

Absolute errors are 1, 1, 1 and 3. Mean absolute error is their sum, six, divided by four: 1.5 hours. Mean residual is four divided by four: 1.0 hour. Both are useful. A near-zero mean residual could conceal equally large positive and negative mistakes, whereas MAE does not allow cancellation. [MAE definition](https://scikit-learn.org/1.8/modules/generated/sklearn.metrics.mean_absolute_error.html)

Suppose the first two jobs are standard and the last two express. Standard MAE is 1.0 hour; express MAE is 2.0 hours. The overall 1.5 lies between them because the groups are equally sized. This miniature result suggests a question about express jobs; it does not prove a stable group difference from two examples each.

An error table should retain the row identifier, intake time, permitted explanatory inputs, actual outcome, prediction and error. Outcomes belong in this audit table, never the intake feature matrix. When joining tables, remember: an investigation can accidentally turn an explanation recorded afterward into a future predictor.

## Look at slices without hunting for a dramatic story

Start with a short list of operationally meaningful slices: priority, zone, depot, queue bands and time blocks. Record counts and the relevant denominator beside each metric. For classification, recall divides true positives by all actually late jobs, while precision divides true positives by all alerts. An empty denominator is undefined, not evidence of perfection.

Compare errors against a baseline on the same slice. A model can have worse errors on long-distance jobs while still improving substantially over the baseline there. Conversely, a strong overall result can conceal a slice where the baseline is preferable. Slice selection, minimum evidence requirements and repeated exploratory comparisons belong in the report.

Inspect several large errors and some ordinary cases. Looking only at the most extreme examples makes a vivid story easy to invent. Ask whether the input was available, valid and measured in the right unit; whether a label is correct; whether the case lies outside familiar ranges; and whether the same issue appears elsewhere. Do not delete a difficult but valid job merely because it hurts the score.

## Worked trace: what permutation importance measures

Imagine a frozen regression pipeline with validation MAE 2.0 hours. Shuffle `queued_jobs` across these validation rows, keep the model fixed and recompute predictions. Suppose MAE becomes 2.7 hours. The loss increase is 0.7 hours. Under the `neg_mean_absolute_error` scorer, baseline score is −2.0 and shuffled score is −2.7, so score decrease is also 0.7.

Repeat the shuffle to show sensitivity to random rearrangements. Those repeats describe the permutation procedure on this dataset; they are not independent samples of future jobs or a confidence interval for a causal effect. Compute on raw permitted columns passed through the full pipeline so the object being shuffled remains understandable. [Permutation-importance API](https://scikit-learn.org/1.8/modules/generated/sklearn.inspection.permutation_importance.html)

The practical interpretation is: this fitted model's measured performance depended on that column in this evaluation setting. Importance is conditional on the model, data and chosen score. A poor model's low importance is weak evidence that a variable lacks predictive information. Strongly correlated inputs can substitute for one another, hiding their shared importance. [Permutation-importance limitations](https://scikit-learn.org/1.8/modules/permutation_importance.html)

## Explain cautiously, then propose a test

Depot and zone might be closely related. Shuffling only one leaves a substitute available; shuffling it can also create combinations rarely seen in real work. Ordinary row permutations may disturb temporal relationships. Treat the course output as a diagnostic demonstration, not a complete dependence-aware inference method.

A fitted coefficient is also conditional on preprocessing and other variables. A coefficient for standardized distance concerns a one-standard-deviation change in that transformed input, not automatically one kilometre. Correlated features, regularization and omitted variables change coefficients. A tree split is a prediction rule, not evidence of a natural causal threshold.

Use language proportionate to the evidence: “On this validation period, the model underpredicted several high-queue express jobs.” Follow with a checkable proposal: “Audit queue snapshots and compare a predeclared alternative using training-only folds.” Avoid “Express priority causes delays” or “Changing priority will fix the model.” A proposed feature must exist at intake, as `promised_hours` does; `final_delay_reason`, `label_available_at` and `close_cost` remain forbidden predictors.

Changing a model after error analysis is legitimate development. Record the change and rerun the development protocol. If the final test has already been examined, label further work as a new iteration requiring fresh untouched evaluation evidence. The same old final score cannot certify the revised procedure.

## Run and write the finding

Run `python examples/09_errors.py` and inspect `outputs/reference_results.json` for measured results. The script shuffles the logistic classifier using negative log loss; its importance units are log loss, unlike the miniature hours example. Check residual signs separately. Draft a short finding with the population, period, slice count, metric, baseline comparison and limitation. Include a next investigation that can actually distinguish competing explanations. 

## Independent practice

### Exercise 09.1 · Reconcile a slice report

Four miniature jobs have actual durations 3, 5, 8, 12 and predictions 4, 4, 6, 9. The first three are standard; the last is express. Calculate residuals, overall MAE, mean residual and each priority's MAE. Reconcile the overall MAE using group counts. Explain why the unweighted average of group MAEs is different.

### Exercise 09.2 · Interpret a shuffled feature

A validation pipeline has MAE 1.8 hours. Three shuffles of distance produce MAEs 2.1, 2.4 and 2.1. Calculate the mean loss increase. A teammate says, “Cutting distance will reduce duration by that amount.” Rewrite the claim correctly and name two reasons the causal statement is unsupported.

### Exercise 09.3 · Design the next investigation

A validation slice has two express jobs, both underpredicted. A teammate proposes dropping express jobs and adding `final_delay_reason`. Write a six-sentence response covering counts, valid difficult cases, availability, a baseline comparison, a permitted investigation and final-test protection.

## Exit checks

- **09.A:** What can a mean residual of zero conceal that MAE would expose?
- **09.B:** Why might two genuinely useful correlated inputs each receive low permutation importance?


# 10 · Reproducibility, handoff and drift

## A model is only part of the deliverable

A receiver needs to know what rows the estimator expects, when inputs exist, how it was selected and where its claims stop. Reproducibility means preserving enough evidence to reconstruct and check the procedure. Handoff means making that evidence usable by another person. Neither means the model is ready for production.

Your Python course required rerunning source rather than trusting a screenshot. Your SQL course required explicit grains, join checks and cutoff rules. Your statistics course required a claim matched to its evidence. This final lesson combines those habits around a supervised, CPU-based synthetic experiment. It does not deploy a service, automate operational decisions or extend the course to high-stakes scoring.

The intended input is a depot job at intake. The permitted schema includes `distance_km`, `queued_jobs`, `item_count`, `priority`, `zone`, `depot_id`, `weekday` and `promised_hours`. State units, allowed categories, missing-value behavior and how unfamiliar categories are handled. The duration target is in hours; late means duration strictly exceeds the promise. `label_available_at`, final delay reason and close cost must remain outside prediction inputs.

## Worked trace: follow one job through a handoff

Imagine an intake job with distance 12 km, queue 7, item count 3, standard priority, zone north, depot D1, weekday 0 (Monday) and a 10-hour promise. The receiver validates columns, types and units. It passes those raw fields into the saved preprocessing-and-model procedure. The classifier produces a late probability; the frozen calibration mapping adjusts it; the frozen threshold converts it into the declared alert rule. The regression pipeline produces duration separately.

If the receiver encodes “standard” using a new homemade mapping, it has changed the procedure even if the saved estimator is unchanged. If it fills missing queues with a newly calculated median, it has also changed the procedure. Keep preprocessing attached to the model and exercise the entire path with a small, known test input.

Suppose a saved miniature smoke-test probability is 0.40 and the threshold is 0.40 with an “at least” rule. The expected alert is true. A receiver using “greater than” produces false. This one boundary check can expose a materially different decision policy before any large batch is processed.

## Preserve the evidence, not just a seed

Record the source revision, data-generation settings, input file hashes, sorted row identifiers, target definition, split ranges, label-availability audits, candidate settings, selected procedure and scoring definitions. Retain the freeze record and distinguish development outputs from final evaluation. A file hash detects changed bytes; it does not prove that those bytes describe a valid experiment.

The course environment is pinned to Python 3.12.14, NumPy 2.3.5, pandas 2.2.3, SciPy 1.17.0, scikit-learn 1.8.0; the supplied static charts use the standard library. Record actual versions from the environment you run. A different version is a changed condition to investigate, not permission to claim an exact match. Random seeds improve repeatability of randomized steps, but they cannot repair changed data, different split boundaries or unavailable labels. [Randomness guidance](https://scikit-learn.org/1.8/common_pitfalls.html)

Use trusted local artifacts only. Pickle-based model files can execute code when loaded, and scikit-learn does not support arbitrary cross-version loading as a reliable compatibility strategy. Preserve the training environment and inspect provenance. [Model-persistence guidance](https://scikit-learn.org/1.8/model_persistence.html)

## Rehearse the whole workflow

Run `python -m courseml.run_all` for development, then `python examples/10_handoff.py` to compare input summaries and check for a freeze record. That script does not itself verify every handoff requirement. Find measured results in `outputs/reference_results.json`; do not substitute this lesson's miniature numbers. Verify whether expected outputs were produced and whether rerunning under the recorded conditions reproduces them within an explicitly justified numeric tolerance.

Freeze preprocessing, features, model settings, calibration, threshold and evaluation plan before `python -m courseml.final_evaluation`. Final evaluation estimates performance on days 200–249 for the frozen procedure. Report disappointing results honestly. If you change the procedure afterward, preserve the old report and seek fresh evaluation evidence for the new version.

A useful handoff summary states the intended use, training and evaluation populations, baseline comparison, important error slices, probability and threshold assumptions, known limitations and the person responsible for deciding what happens next. State that all depot records and outcomes are synthetic. A clean run of reference code is evidence about that code, not proof that a learner independently mastered the material.

## Worked trace: distinguish drift from a verdict

In a miniature monitoring example, a reference window contains 100 jobs with five missing queue values. The missingness rate is 5%. A new window contains 80 jobs with 12 missing queue values: 15%. The increase is ten percentage points, and the relative rate is three times the reference rate. Keep denominators visible.

This change is input drift or a data-quality symptom worth investigating. It does not, by itself, prove that prediction error worsened. Perhaps a source-system field stopped arriving; perhaps the job mix changed. Check the pipeline and source before drawing a model conclusion.

Now suppose only 40 of those 80 new jobs have mature outcomes and 20 are late. The observed late fraction among mature outcomes is 50%. It is not legitimate to divide by 80 and call the rate 25%, because pending outcomes are unknown. Nor is 50% automatically representative of all 80: outcome availability may depend on duration. Wait for the declared maturity policy and report coverage.

## Monitor questions with different evidence

Input monitoring can track missingness, invalid categories, unusual ranges and unfamiliar depots without labels. Prediction monitoring can track score distributions and alert volume. Performance monitoring needs mature outcomes: MAE, probability loss and threshold errors must use a defined, eligible cohort. A changed input distribution need not change performance; an unchanged input distribution does not guarantee that the outcome relationship stayed stable.

Predeclare review triggers and owners. An alert should start a diagnosis, not automatic retraining or silent threshold changes. Identify who would verify the source, inspect jobs, consider a fallback and authorize a new development cycle. Threshold changes can alter workload even without retraining. Version them as deliberate decisions and evaluate them appropriately.

## Independent practice

### Exercise 10.1 · Repair a handoff

A package contains an estimator file and “seed = 42,” but omits preprocessing, feature units, split dates, dependency versions, threshold and intended use. Draft a compact handoff checklist, explain why the seed is insufficient and specify one whole-pipeline smoke test. Include a safe rule for loading model files.

### Exercise 10.2 · Audit monitoring arithmetic

A reference cohort has 200 jobs and eight missing distance values. A new cohort has 120 jobs and 18 missing values. Calculate rates, percentage-point increase and relative rate. Only 75 new jobs have mature labels; 30 are late. State the observed late fraction and two limitations on interpreting it.

### Exercise 10.3 · Respond without hiding a failure

After final evaluation, a teammate proposes tuning the threshold on its 300 jobs and replacing the original result. Separately, unfamiliar depot categories appear in a later batch. Write a response that preserves the test history, separates the two issues and gives a bounded next step for each.

## Exit checks

- **10.A:** Why are changing input distributions and worsening predictive performance different claims?
- **10.B:** What must remain fixed for a final-test result to describe the handed-off procedure?


# 11 · Follow one complete prediction investigation

## The deliverable is an evidence chain

A complete applied-ML investigation connects a question to a defensible decision. It does not stop when a model predicts successfully. In this lesson, follow the reference depot investigation from intake contract to a frozen evaluation plan. The examples combine concepts already introduced; no new algorithm rescues a weak information boundary.

The request is narrow: estimate duration and lateness for future accepted jobs at the represented fictional depots, using information available at intake. The intended classification action is a human review queue. The review-cost ratio and capacity used here are instructional assumptions: one cost unit per false alert, four per missed late job, and at most 40% of policy-period jobs flagged. They are not actual money, approved operational limits or evidence that review prevents delays.

## Worked miniature: connect scores to an operational statement

Start with six invented policy jobs. Their late labels are [0, 1, 0, 1, 1, 0], and calibrated probabilities are [0.10, 0.35, 0.45, 0.60, 0.80, 0.90]. At threshold 0.50, jobs four, five and six are flagged. That gives two true positives, one false positive and one false negative. Cost is 1 + 4 times 1 = 5, but three alerts out of six is 50%, exceeding the 40% limit.

At threshold 0.70, only jobs five and six are flagged. There is one true positive, one false positive and two false negatives. Cost is 1 + 4 times 2 = 9. Two alerts out of six is approximately 33.3%, so this candidate is feasible. The higher cost does not justify silently discarding the capacity restriction. A valid choice is the best feasible candidate under the complete predeclared rule, not the unconstrained minimum.

A threshold between 0.60 and 0.80 produces the same two alerts in this miniature, depending on the inclusive boundary. If costs and alert counts tie, the course chooses the higher candidate threshold. For p exactly equal to the chosen threshold, the job is flagged because the rule is p greater than or equal to threshold. This small detail belongs in implementation tests.

Now imagine the next six jobs produce three alerts under the frozen threshold. The threshold was feasible on development data but is not guaranteed to satisfy capacity on every future batch. The honest report says the observed future alert fraction exceeded the development limit. It does not raise the threshold using those newly revealed answers and call the revised score an untouched test result. This is an integrated miniature, not the course's actual final outcome.

## Trace the reference workflow

First, the data audit preserves job IDs and verifies chronology, uniqueness, target construction and label availability. `label_available_at` represents completion and label availability at the same instant in this synthetic dataset. Real systems may have a reporting delay; here the simultaneous timestamps are an explicit simplifying assumption. Duration is bounded at 23.5 hours, while the outer embargoes are full days.

Second, training consists of 894 rows through day 148. Four forward-looking training folds, each with a 12-row gap, compare full pipelines. The regression choice minimizes mean fold MAE among the median baseline, Ridge and random forest. The classification comparison records prior, logistic and forest behavior. The integration intentionally fixes the logistic family and tunes its C and class-weight choices with temporal-CV log loss. Family choice and parameter selection are separately stated rather than hidden in a single winner label.

Third, the selected logistic pipeline is refitted on training and frozen. The calibration segment contains 144 jobs on days 150–173. A sigmoid probability adjustment is learned there without refitting the base pipeline. After the day-174 gap, another 144 jobs on days 175–198 support threshold selection. The candidate grid is 0.10 through 0.90 in increments of 0.05. If no candidate satisfies capacity, the explicit threshold 1.01 creates no alerts. The fallback is a documented abstaining policy, not an ordinary probability cutoff.

Fourth, the report records raw and calibrated probabilities, candidate thresholds, confusion counts, regression residuals, zone summaries and descriptive permutation importance. These analyses use development information. They can reveal limitations, but they do not license a new feature search followed by claims that the same development segment was untouched. If the scope changes materially, record a new experiment and evaluation plan.

## Read mixed evidence without flattening it

Run `python -m courseml.run_all` and inspect `outputs/reference_results.json`, `outputs/validation_predictions.csv` and `outputs/frozen_plan.json`. The development command does not load the final test partition. Inspect actual numerical outputs and compare them with the code's stated selection rule. An unexpectedly good score should prompt the same audit as an unexpectedly bad one.

Calibration can improve one proper probability score while worsening another on a finite sample. Report both rather than saying calibration helped universally. A zone with higher MAE also needs its sample size and outcome range. A permutation score describes how a fitted model uses a feature under a particular perturbation; it does not identify the effect of changing a depot process. The [calibration guide](https://scikit-learn.org/stable/modules/calibration.html) and [permutation-importance guide](https://scikit-learn.org/1.8/modules/permutation_importance.html) support these distinctions.

Finally, write a freeze record before opening final outcomes. It contains feature names, target definitions, split boundaries, selected models and parameters, calibration method, threshold, tie policy, source and development-data hashes, versions and intended report. Only after that record is complete should the reference final command run: `python -m courseml.final_evaluation`. That command evaluates the reference procedure; it does not automatically evaluate a learner's alternative pipeline.

## Independent practice

1. **11.1 · Audit the miniature policy.** For labels [0, 1, 0, 1, 1, 0] and probabilities [0.10, 0.35, 0.45, 0.60, 0.80, 0.90], calculate confusion counts, cost and alert fraction at 0.40 and 0.80. Apply the 40% capacity restriction and explain which of these two candidates is eligible.
2. **11.2 · Trace a real reference result.** Using development outputs only, find the selected regression family, logistic parameters and threshold. For each, identify the exact rows and criterion that influenced it. Reconcile the selected policy confusion counts to its sample size.
3. **11.3 · Write the freeze note.** Draft a concise freeze record and a final-report outline. Include the action if final capacity fails, how changed code would be handled and why reference results do not grade your own code.

## Exit checks

- **11.A:** Which stages learn base-model parameters, calibration parameters and the decision threshold, respectively?
- **11.B:** Why can a capacity-constrained policy pass development checks and still fail the same operational limit later?


# 12 · Capstone: deliver an auditable intake prediction study

## Your assignment

Produce your own small, reproducible investigation of the synthetic depot jobs. Deliver a duration predictor and a lateness review policy, compare them with appropriate baselines and explain whether the evidence supports a limited next experiment. The assignment is an educational evaluation. It does not authorize deployment, real-world messaging, staffing decisions or scoring people.

Work from the course's supplied raw partitions. Preserve the 894-row training period, the calibration and policy allocation within validation, all embargoes and the untouched 300-row final test. Use only the approved intake predictors, including the original `promised_hours`. Keep post-outcome fields for auditing and label construction only. You may use the taught CPU-friendly model families; a larger model is not extra credit by itself.

The completion standard is an evidence-backed artifact, not a favorable score. A careful no-go recommendation can earn full marks. A high score obtained by leakage cannot. Read the rubric in `ASSESSMENT.md` before beginning, but keep `SOLUTIONS.md` and supplied final-reference outputs closed until you have frozen your own choices and completed your evaluation.

## Worked preparation example: turn a requirement into evidence

Suppose a fictional project requirement says, “Unknown depot labels must not crash prediction.” A weak submission says, “I used an encoder.” A stronger submission identifies the policy, constructs a copied intake row whose depot is `D_NEW`, passes it through the entire fitted pipeline and records a finite probability between zero and one. It also says that successful execution does not establish predictive quality at a genuinely new depot.

For a second requirement, “No future labels enter fitting,” a weak submission says, “I sorted dates.” A stronger submission saves each training/evaluation boundary, checks the maximum fitted label-availability time against the earliest predicted intake time, and explains the fixed label-availability assumption in the synthetic generator. If a check fails, the run stops and the issue is investigated. Altering the timestamp to satisfy the test would destroy the evidence.

These examples show the structure of a good submission: claim, check, observed result and limitation. They do not supply your model choice, threshold evidence or final recommendation. Use the same structure for each critical requirement.

## Stage 1: define and audit

Write a one-page prediction contract, a feature-availability ledger and a target definition. Record how equality at the promise boundary is treated. Audit ID uniqueness, time order, missing inputs, target consistency, allowed feature selection and all partition counts. Include the difference between a valid absent value that can be imputed and a contract violation that should stop the workflow.

Describe the population represented by these 250 synthetic days and the transfer claims you will not make. A depot represented in training is different from an unseen depot. Future jobs are different from a random sample of historical jobs. The task's timeline is fixed so the comparison has one coherent interpretation.

## Stage 2: implement and compare

Implement the metric functions in `exercises/learner_functions.py` and your classifier/regressor construction and threshold rule in `exercises/capstone.py`. Run `python check_learner.py` and `python check_learner.py --capstone`. The starters intentionally fail before implementation. Save the command and results for your own files; passing checks for supplied reference solutions is not learner evidence.

The public checks cover a few arithmetic cases, classifier and regressor pipeline smoke tests and threshold-policy cases. They cannot read your intent, reconstruct every data dependency or assess your written claims. Add your own boundary cases, including probability equal to threshold, no feasible ordinary threshold, missing numerical input and an unseen category. Explain what each test protects.

Compare full pipelines using the four training-only temporal folds and declared metrics. Select the regression family with mean CV MAE, report RMSE and retain fold-level evidence. For classification, state whether you follow the fixed logistic probability-policy case or predeclare another taught family before looking at policy results. Any alternative still needs valid probability estimates, calibration design and a documented selection criterion. Do not evaluate endless variants until one looks impressive. Your `build_classifier()` and `build_regressor()` factories must return unfitted full pipelines with settings chosen through your documented training-only comparisons; the runner does not perform that selection for you.

## Stage 3: calibrate, choose policy and freeze

Refit selected base pipelines on training only. Fit probability calibration using days 150–173, leaving the base classifier unchanged. After day 174, apply calibration to days 175–198 and select a threshold with the exact stated grid, cost ratio, capacity limit and tie rules. Show candidate-level counts, including the explicit no-alert fallback if required.

Write a freeze record that another reader could use to reproduce the complete procedure. Include all fitted stages, feature schema, seeds, dependency versions, data/source hashes, threshold, metrics and the outcome of development checks. Freeze means decisions stop changing before final labels are inspected. If you discover a genuine bug afterward, disclose the incident and distinguish a repaired analysis from an untouched final evaluation.

Run `python learner_run.py develop` to fit your factories, calibrate on the assigned block and apply your policy selector. It writes development results and a freeze record to `learner_outputs/` without opening test data. Complete your written decision record before running `python learner_run.py final`, which verifies the saved source/data/environment fingerprint and evaluates the frozen procedure without a new threshold search. The reference `python -m courseml.final_evaluation` evaluates reference code, not your capstone. Neither runner supplies the missing written audit or prevents deliberate test inspection.

## Stage 4: evaluate once and hand over

Evaluate the frozen procedure on days 200–249. Report the actual sample and late count, baseline and model regression metrics, classification probability metrics, confusion matrix and alert fraction at the frozen threshold. Keep failed capacity or performance gates visible. Do not convert a failure into a pass by tuning on the same final outcomes.

Submit source, environment specification, audit, split evidence, comparison results, freeze record, final results, labeled residual/segment evidence and a short model card. The model card should describe intended use, limitations, monitoring signals, label delay, fallback behavior and who would review problems in a hypothetical next study. No real operational rollout is part of this assignment.

## Independent practice

1. **12.1 · Prepare your evidence manifest.** List each required artifact, the command or procedure that creates it and the claim it supports. Mark which files must exist before final evaluation.
2. **12.2 · Run a failure rehearsal.** Introduce a forbidden predictor into a copy of your feature specification, then restore it. Separately test an unseen depot and the no-feasible-threshold case. Record whether each failure is blocked or safely handled, without editing raw evidence.
3. **12.3 · Defend your recommendation.** Write a 250–400-word decision brief using your own final results. Explain baseline comparison, capacity, uncertainty, transfer limits and the next authorized research step. Include one finding that argues against an overly confident recommendation.

## Exit checks

- **12.A:** What evidence distinguishes a learner-owned completed capstone from running the supplied reference workflow?
- **12.B:** What must you say if the frozen procedure fails a predeclared gate on the final test?