Practice: explain the decision before changing the system

Attempt this after lesson 1, before opening the solutions. Use only the supplied synthetic files. No actual support messages, account records, or model calls are needed.

1. Trace the exchanged outcomes

For the baseline and regressed candidate, write the overall fraction and each slice fraction. Identify the one new failure and the one fixed case. In two sentences, explain why a reviewer who sees only the overall fraction could make the wrong decision.

2. Test whether the gates are independent

Predict the result if the local policy allowed one critical failure but still allowed zero new failures. Then predict the result if it allowed one new failure but still allowed zero critical failures. You may try this with temporary in-memory policy copies in Python. Do not edit the supplied policy to make a blocked candidate appear acceptable.

Explain why this experiment is a check of evaluator logic rather than evidence that either relaxed policy is appropriate for a real application.

3. Do not improve a score by dropping evidence

Suppose somebody removes boundary-02 from the candidate results without changing the dataset. What should the evaluator do: compute accuracy on eleven outputs, count the missing output as an ordinary miss, or report an input error? Choose one and explain the distinction between a valid failed case and incomplete evidence. Confirm the supplied evaluator's behavior using a temporary file or a copied in-memory dictionary.

4. Implement the small comparison helpers

Complete new_failure_ids and slice_accuracy in the Python starter. Their docstrings define the contracts. Preserve IDs when matching records; input order is not meaningful.

Run from the package root:

PYTHONDONTWRITEBYTECODE=1 python3 practice/check_exercise.py practice/01_lab.py

The starter intentionally raises NotImplementedError until you complete it. This optional practice check is separate from the finished evaluator's own tests. Use its errors as feedback, not as a reason to weaken the checks.

5. Find a blind spot

Invent one hypothetical output that gets the route right but would still be unacceptable for a support application. State why route equality misses it and propose one additional evaluation. Do not add personal data or actual harmful content to the dataset. A description of the missing check is enough.

6. Write a decision note

Write 80–140 words about the regressed candidate. Include the exact candidate identifier, evidence type, local decision, relevant case identifiers, conditions violated, and one consequential limitation. State what you would inspect next. Do not describe a technical test as a learner pilot or claim a real model was evaluated.

Optional extension: propose two new synthetic cases that challenge an ambiguity in the fictional contract. Record the expected route and rationale before writing a candidate result. If the right label is unclear, explicitly flag the ambiguity rather than guessing. Keep any extension as a separate dataset version.

Separate solutions and self-check rubric