Preserve the experiment you actually ran

Yesterday's report says the candidate passed. Today the same filename produces a different result. Was the prompt changed, the dataset relabeled, the grader revised, or the evaluator edited? Without a record of the ingredients, you have an argument about filenames instead of a reconstructable experiment.

This module gives you a practical evidence bundle: exact input bytes, code identity, declared context, and a replay. You will detect changed files without claiming that a hash authenticates their author or that deterministic bookkeeping makes a language model deterministic.

Separate identities that can change independently

A system version describes what produced an output: model identifier, prompt, application code, retrieval configuration, relevant dependency versions, and inference settings. A dataset version describes what was asked and how cases were labeled. A grader version describes how outputs were judged. A policy version describes which observed conditions pass or block. An execution record describes when, where, and under what measurement boundary the run occurred.

These identities are related but not interchangeable. If the output stays the same while a human rubric changes, the new score is evidence of a changed assessment. If a label is corrected, re-score both baseline and candidate on the same corrected dataset before attributing a difference to the candidate.

In our package, prompt labels such as fictional-triage-v2 are explicit fiction. No provider inference occurs. We can nevertheless preserve the actual evaluator, fixtures, and policy that determine the local result.

Hash the files, then preserve their meaning

Run from project:

python3 -B opslab.py manifest

The manifest includes SHA-256 digests for eight required files: operations fixtures, original cases, baseline results, both candidate result files, policy, gate.py, and opslab.py. It records the interpreter version, human-rubric revision, and an explicit timing boundary: no execution was timed; delay values are invented.

A digest identifies exact bytes. It helps you notice that a file changed. It does not tell you whether the contents are sensible, who approved them, or whether a malicious actor replaced both file and digest. A real approval trail needs trusted storage, ownership, and access controls outside this course's simulator. The local tool checks a complete required file set so removing an inconvenient input from the manifest does not quietly pass verification.

Whitespace affects a byte digest. A reformatted JSON file can change its digest without changing its interpreted values. This is useful when reconstructing the exact artifact, but reviewers should distinguish a formatting-only change from a changed label or policy. Recording both the bytes and a human-readable change note gives each kind of evidence a job.

Rehearse verification without editing a source file

Run this snippet from project with python3 -B - or in a local Python session:

import copy
import opslab
record = opslab.manifest()
print(opslab.verify_manifest(record))
altered = copy.deepcopy(record)
altered["files"]["policy.json"] = "0" * 64
print(opslab.verify_manifest(altered))

The first result is [], meaning the named bytes match the newly generated record. The second is ["policy.json"]. We changed the saved claim about the bytes in memory, not the policy itself. The result says the claim no longer matches; it does not infer why.

That first check compares a freshly made record with the same current files; it is not evidence that an older run used these bytes. The package also ships project/evidence/operations-manifest.json, a snapshot saved during final author verification. For a historical comparison, load that saved record with opslab.read_json(opslab.ROOT / "evidence/operations-manifest.json")[0] and pass it to verify_manifest. The capstone uses this saved snapshot. Neither a saved digest nor a self-check establishes trusted authorship.

Now remove policy.json from the copied manifest's files mapping and verify again. You should get an input error because the record is incomplete. This is different from a legitimate manifest that names a file whose bytes changed. A missing file on disk is reported as changed; an incomplete manifest cannot describe the required experiment at all.

Replaying saved outputs has a narrower guarantee

You can rerun module 1's evaluator on the same synthetic records and obtain the same decision. That demonstrates deterministic scoring for those records and that code. It does not show that a real model would generate the same outputs again. In a real study, preserve raw authorized outputs, relevant settings, sampling boundaries, and the evaluation result, then treat fresh generation as a new observation.

The Python JSON documentation describes hooks the original evaluator uses to reject duplicate object keys and non-finite constants. Rejecting ambiguous input prevents silent last-value replacement from obscuring what was scored. This bookkeeping is useful precisely because the application itself may be variable.

Evidence retention also deserves an explicit boundary. Our fixtures have no real learner or customer information and reports omit message inputs. A production plan should specify which fields are necessary, who may access them, how long they are needed, and how approved deletion or redaction affects replay. Do not start collecting raw conversations merely because a diagram has a box named “logs.”

Practice

1. Run the manifest rehearsal. Explain the difference between a mismatched digest and a manifest missing a required file.
2. Give one example each of a system change, dataset change, grader change, and policy change. State which comparisons must be repeated.
3. A teammate reformats cases.json and says a changed hash proves the application regressed. Explain the error and identify evidence that would distinguish formatting from meaning.
4. Draft a minimal experiment record for a hypothetical authorized future model evaluation. Include the timing boundary and one privacy/retention decision. Do not add actual credentials, logs, or model calls.

Your exit artifact is a manifest comparison plus an experiment record whose claims are bounded. Explained solutions.