Keep evidence useful as the application changes

Three months after a reassuring evaluation, the business changes its refund policy. The provider announces a replacement model. The incoming questions also contain more boundary requests. A report can remain perfectly reproducible while no longer answering today's decision.

You will separate an input-mix shift from evidence of quality loss, review changes against an explicit date, and assign maintenance work without creating a live schedule. The local program reads fixed synthetic counts and a fictional register; it does not watch a service or contact a provider.

Ask which assumption changed

An evaluation depends on a contract, cases, system, grader, policy, and operating context. A new business rule can change the right answer. A new model identifier can change outputs. A changed traffic mix can change how much a weak slice matters. A revised grader can change reported scores without changing output behavior.

Handle these as different changes. Preserve the old report as evidence about its original conditions. Create a new version and compare relevant artifacts under the changed assumption. Do not relabel yesterday's answers without a note, or assume a new model inherits an old model's result because the application code stayed the same.

The NIST AI RMF Playbook is a voluntary collection of suggested actions to adapt to a use case, not a universal checklist to follow in full. Our maintenance register is an original teaching mechanism; its dates and owners are not a compliance program.

Work through an input-mix change

In w1, the slice counts are routine 70, ambiguous 20, boundary 10. In w4, they are 45, 20, and 35. Both total 100. Convert them to shares and compute total variation distance:

0.5 × (|0.70−0.45| + |0.20−0.20| + |0.10−0.35|) = 0.25.

Run from project:

python3 -B opslab.py drift

The result is approximately 0.25; floating-point output may show 0.24999999999999994. Our illustrative review threshold is 0.15. The shift calls for investigation because the distribution of categories has changed. It does not show that answers got worse, that a model's reasoning changed, or that the shift caused the incident.

This calculation compares shares, so doubling every count without changing proportions gives distance zero. Categories must match, counts must be nonnegative, and both windows must have observations. If a new category appears, the category mapping itself needs review; silently dropping it would hide evidence.

An input distribution change is often called data drift. A changed relationship between inputs and correct outputs is a different concern, sometimes called concept drift. For example, a changed refund rule may alter which guidance is correct even if the input wording is unchanged. Our distance sees only category shares; it cannot detect that relationship change or semantic errors. A representative labeled review is still needed.

Turn change into owned work

Run:

python3 -B opslab.py maintenance

The function uses the fixture's as_of: 2026-10-17, not the computer clock. change-01 is an open policy review due on the 16th, so it is overdue. change-02 is a fictional provider-version review due on the 17th. change-03 is an input-mix review due on the 20th. A closed grader item is omitted from the open list.

An owner and an action make each item actionable. The date is a review expectation, not permission to change a service. The function returns a list; it does not install a reminder, alter an existing automation, renew a certificate, change retention, or invoke an API. In particular, none of the real platform's maintenance schedules are touched.

Decide when evidence expires

A useful maintenance record names what evidence is affected and what would refresh it. A policy change may require domain interpretation, new labels, and replay of both versions. A provider-version change may require new authorized observations and a new manifest. A drift finding may require sampling and label review rather than an immediate model replacement. A closed task should point to completion evidence, not just a tick mark.

Evidence can expire by event as well as by time. “Valid until the refund contract changes” is more informative than blindly rerunning the same synthetic cases every week. A recurring check only helps if its inputs remain relevant and someone owns the result. Scheduling and data access remain separately authorized work; this course gives you a plan, not an active monitor.

Practice

1. Compute the total variation distance by hand and explain why it is not an accuracy metric. Give a change it cannot detect.
2. Compare {routine: 7, boundary: 3} with {routine: 70, boundary: 30} using total_variation. Explain the result despite the different volumes.
3. Change only the in-memory as_of argument to 2026-10-21 and rerun maintenance. Which open items are now overdue? What did the program actually change?
4. Write an evidence-refresh plan for the fictional refund-policy change: owner, affected artifacts, required review, version update, and completion evidence. Distinguish proposed work from authorized live action.

Your exit artifact is a change register and a defensible refresh decision. Explained solutions.