# Assessment and completion standard

## What this assessment establishes

The intended outcome is a small, auditable CPU-based supervised-learning study using scikit-learn. The learner must explain why the data, validation design and decision rule match the prediction question. Executing examples, obtaining an attractive score or passing a few public checks does not demonstrate the complete outcome.

The course is instructional and uses fictional operations. It does not confer certification, establish production readiness or authorize consequential scoring. Timing and grading thresholds below are proposed teaching-design choices. They have not been validated in a learner pilot.

## Suggested study plan

Allow approximately 30–42 hours overall, including setup, reading, hand calculations, exercises, debugging and the capstone. Learners returning to Python or Statistics may need substantially longer. A possible allocation is:

- Setup and prerequisite review: 1–2 hours
- Lessons 01–03, contracts and information boundaries: 5–6 hours
- Lessons 04–06, pipelines and supervised evaluation: 6–8 hours
- Lessons 07–10, validation through handoff: 7–9 hours
- Lesson 11, integrated reference investigation: 2–3 hours
- Lesson 12 and capstone production: 8–12 hours
- Review and corrections: approximately 1–2 hours, more if a critical issue appears

These ranges overlap with practice time rather than implying fixed classroom periods. Take a break after an exercise exposes a misconception, then redo it on a changed input. Do not use the suggested schedule as evidence of mastery.

## Formative work: 36 exercises and 24 exit checks

Each of the 12 lessons contains exactly three independent exercises and two exit checks. Exercises mix calculation, design, diagnosis and communication. Worked examples precede the exercises; reasoned answers appear separately in `SOLUTIONS.md`.

For each exercise, an optional 0–2 self-check can guide revision:

- 0: incorrect result, missing attempt or an unsupported claim
- 1: substantially correct result but incomplete reasoning, missing boundary case or unclear evidence
- 2: correct reasoning and result, with the requested interpretation, check or limitation

This gives 72 possible formative points. A suggested readiness target is 60/72 with all information-boundary, target, split and test-reuse errors corrected. The threshold is not a validated exam cut score. Exit checks should be explained orally or in a short paragraph without looking at the solutions. All 24 should be correct before capstone sign-off; correction after feedback is encouraged.

Use three review pauses:

1. After Lesson 04, demonstrate prediction-time availability, exact split counts and a train-only fitted pipeline.
2. After Lesson 08, reconstruct a confusion matrix and explain the distinct data roles of model fitting, calibration and threshold selection.
3. After Lesson 11, trace a reference choice to its source rows, state what the test has not yet influenced and produce a freeze note.

An unresolved error at a pause should trigger targeted review, not more model tuning.

## Capstone submission package

Provide a learner-owned folder containing:

1. Prediction contract, intended use/non-use and feature/label availability ledger
2. Source-data identities or hashes, audit output, exact split membership/counts and maturity checks
3. Source code for preparation, model comparisons, calibration, threshold selection and final evaluation
4. Environment specification, seeds and complete commands
5. Baseline and candidate fold-level results with predeclared selection criteria
6. Calibration and threshold-policy evidence, including capacity and tie handling
7. Timestamped freeze record made before final outcome inspection
8. Final results, original predictions and labeled error/segment evidence
9. Model card, monitoring/fallback plan and a 250–400-word decision brief
10. Public-check results for the learner's files, additional boundary tests and a short limitations statement about those checks

Raw data must remain unchanged. Keep learner outputs separate from supplied reference outputs. A grader should be able to associate every reported number with source records, code, configuration and the evaluation period that generated it.

## Staged capstone rubric: 100 points

### Stage 1 · Question and trustworthy data — 20 points

**Contract and target, 8 points.** State population, job grain, intake prediction time, exact duration and late definitions, permitted features, proposed review use and excluded uses. Full credit requires the promise equality boundary and original-intake-value requirement.

**Audit and provenance, 12 points.** Preserve raw evidence, check IDs, join grain if applicable, ranges, missingness, label construction and label availability. Reconcile all partitions including embargoes. Distinguish actual implemented checks from assumptions that a script cannot prove. Credit depends on saved evidence, not a checklist of claims.

### Stage 2 · Learn and compare validly — 30 points

**Preparation and schema, 12 points.** Implement an explicit feature allowlist and a full fitted pipeline. Learned imputation, scaling and encoding must occur inside each training fold. Demonstrate missing-value and unseen-category behavior. Explain the limits of technical handling for an unseen depot.

**Baselines and metrics, 6 points.** Use suitable regression and classification baselines fitted only to training portions. Report metrics with units, positive-class definition, counts and undefined-case handling. Distinguish ranking, probability and threshold-policy measurements.

**Temporal comparison, 12 points.** Use four training-only forward folds with gap and timestamp checks. Save fold-level scores, predeclare family/parameter selection and give a bounded rationale. Explain why random and group splits answer different questions. No points for additional search breadth unless it improves a justified comparison.

### Stage 3 · Calibrate, choose policy and freeze — 20 points

**Calibration, 8 points.** Keep base fitting separate from calibration on days 150–173. Preserve the fitted base pipeline and explain what the calibration mapping learns. Assess probability evidence on later policy rows, acknowledge mixed or uncertain evidence and avoid claims that calibration guarantees future accuracy.

**Threshold policy, 8 points.** Correctly implement the inclusive threshold rule, candidate grid 0.10–0.90 by 0.05, illustrative FP + 4 FN cost, at-most-40% development alert fraction, tie rules and no-feasible-candidate fallback. Preserve all candidate results and test boundary cases. Explain that the capacity restriction is empirical, not guaranteed on future batches.

**Freeze evidence, 4 points.** Record feature schema, selected models and hyperparameters, calibration, threshold, source/data identities, versions and final-report plan before accessing final labels. Include an incident policy for a discovered post-freeze bug.

### Stage 4 · Evaluate honestly and hand over — 30 points

**Final evaluation, 10 points.** Evaluate the frozen learner procedure once on days 200–249. Report baseline comparison, regression errors, classification probability metrics, confusion counts and alert burden. Keep all failed gates visible. Distinguish reference-workflow execution from learner-workflow execution.

**Diagnostics and limits, 8 points.** Use residuals and at least one planned operational slice with sample size. Explain the limitations of small slices, dependent observations, feature importance and synthetic data. Avoid interpreting predictive association as an intervention effect.

**Reproducibility and model card, 8 points.** Supply runnable commands, environment pins, reproducible output locations and enough information to rebuild the learned transformations. State ownership, monitoring signals, label delay, fallback and escalation conditions for a hypothetical next study. Never load untrusted serialized Python model files.

**Decision brief and defense, 4 points.** Give a concise evidence-backed recommendation, one material counterargument and an appropriately bounded next step. A reasoned no-go can receive all four points. A learner should explain one prediction and one failure without reading generated prose.

## Safety and integrity gates

Suggested completion requires at least 80/100 and all integrity gates below. A grader should return a failed gate for correction with a written explanation, regardless of the numerical score:

- No unavailable, target-derived or forbidden predictors enter any fitting stage.
- Labels used at each fitting boundary were available then; the temporal test remains genuinely held out until freeze.
- Learned preparation and any feature selection are fitted only inside their allowed training stages.
- Raw data and evidence are not altered to satisfy checks; missing outcomes are not fabricated.
- Reported results come from the learner's code on declared partitions. Reference success is not represented as learner success.
- Test-guided changes are disclosed; a reused test is not described as untouched.
- No causal, deployment-ready or high-stakes claim exceeds the demonstrated evidence.
- Any failed operational gate is visible in the recommendation.

Distinguish assessment integrity from operational feasibility. A final alert fraction above 40% means the hypothetical policy does not meet that development capacity requirement in the final period. It does not automatically fail the learner: honestly detecting, quantifying and responding to it is strong applied-ML work. Hiding it or retuning on that same test while retaining an untouched-test claim is an integrity failure.

## What automated checks do and do not grade

`python check_learner.py` imports the learner's metric functions. `python check_learner.py --capstone` imports the learner's classifier/regressor pipeline constructors and policy function. The starter files deliberately raise `NotImplementedError` until completed. The capstone checker fits cloned pipelines, checks a small unseen-depot batch and compares policy behavior on a few fixed cases.

After documenting training-only model selection, `python learner_run.py develop` executes the learner factories, calibrates and freezes the policy in `learner_outputs/`. After completing the written freeze record, `python learner_run.py final` verifies the recorded fingerprint and evaluates those frozen learner procedures. The factories must return the preselected unfitted pipelines; the runner does not perform or grade model selection.

These checks do not grade written reasoning, audit all data dependencies, prove label availability, verify final-test independence or establish generalization. The reference final-evaluation command reconstructs the reference workflow and is not an alternate learner grader. Full capstone completion therefore requires both executable evidence and human review against this rubric.

## How to use the separate answers

Attempt the work, write down uncertainties, then consult only the relevant solution section. Close it and solve a changed input. For open-ended answers, compare the required reasoning rather than copy the example wording. Reference final outcomes are spoilers; keep them closed until your own protocol is frozen and evaluated. If you have already seen them, disclose that exposure and treat the same final set as a practice evaluation, not independent evidence.