ReleaseGate: an offline LLMOps teaching project

Evidence type: handwritten synthetic fixture replay. No LLM inference occurs. Routes, delay values, and costs are invented inputs so learners can inspect a release decision. This project has no network code, hosted service, external dependency, credentials, or deployment command. It reads the named inputs and writes a report only when --report is supplied.

Run

Use Python 3.11 or later. No installation step is required. From this project directory:

PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v
python3 gate.py --candidate fixtures/candidate-regressed.json --report evidence/regressed.json
python3 gate.py --candidate fixtures/candidate-repaired.json --report evidence/repaired.json

Run each command individually: the regressed candidate intentionally returns a nonzero exit code.

The default cases, baseline, and policy are resolved relative to gate.py, so the program also works from another directory when --candidate is an absolute path. --cases, --baseline, and --policy permit explicit alternate local inputs. A report path may overwrite an earlier report but cannot resolve to any input file. Reports do not include the case input messages. Source files are read only.

The fictional task

Juniper Workshop wants to route support messages. A public-policy question should be marked answer; a question missing necessary context should be marked clarify; requests for staff-only decisions, another attendee's private information, or account access changes should be marked escalate. Routing does not execute an action. For the boundary cases, escalate means the item needs a staff review and conveys no permission to disclose or change anything.

The project grades only this route label. It cannot tell whether a message accompanying escalate leaked private information, whether an answer is true, whether escalation succeeded, or whether a user was helped. Those are separate evaluations to design in later modules.

Files and input contract

• fixtures/cases.json: twelve cases, each with a unique id, slice, synthetic input, expected_route, boolean critical, and label rationale.
• fixtures/baseline.json: the reference record. It deliberately misses routine-04.
• fixtures/candidate-regressed.json: fixes routine-04 but breaks critical boundary-02.
• fixtures/candidate-repaired.json: all twelve route labels match. This is an edited fixture, not a newly trained or tested model.
• policy.json: named thresholds and their teaching rationale.
• gate.py: validates evidence, scores route equality, compares matched IDs, applies every gate, and optionally writes a report with SHA-256 input digests.
• tests/test_gate.py: positive cases, deceptive aggregate result, independent gate failures, incomplete evidence, invalid numbers, and CLI behavior.

Each run requires the same dataset identifier, exactly one result per case, a unique run identifier, provenance: handwritten-synthetic, and a system manifest. model: none-fixture prevents these inputs being mistaken for actual inference runs. Prompt and code revision strings are teaching labels; there is no real prompt engine behind them. Unknown fields, routes, duplicate IDs or JSON keys, omitted slices, negative measurements, boolean measurements, and incompatible dataset IDs are rejected. This strict contract makes accidental evidence loss visible; it is not a general-purpose ingestion service.

What the gate checks

All comparisons are inclusive. One micro-USD is one millionth of a US dollar; 2,000 micro-USD is $0.002. Values here are invented arithmetic inputs, not provider prices, bills, measurements, or a pricing recommendation. The fixtures all have a mean of 1,000 micro-USD and a nearest-rank p95 of 520 ms.

For percentile p and n values, this project sorts the values and selects position ceil(p*n), counting from one. For twelve observations, ceil(0.95*12)=12, so p95 is the largest value. This tiny sample should prompt caution, not a claim that the service has a stable production tail latency.

Expected evidence

The regressed candidate has 11/12 correct routes, just like the baseline. Its boundary slice is 3/4. It still meets the overall and slice floors, cost, and delay limits; the critical-route and new-case gates block it. The repaired candidate scores 12/12 and passes all eight checks. Every report retains release_authorized: false, its limitations, manifests, and input digests.

SHA-256 digests identify exact input bytes. They do not establish trusted authorship, prevent later tampering, provide a signed approval, or make synthetic data representative. JSON reports are deterministic for the same bytes and code; no timestamp is inserted so diffs focus on evidence changes.

Operations labs for modules 2–9

opslab.py reads fixtures/operations.json, an original collection of synthetic messages, labels, human ratings, repeated observations, count windows, stage plans, incident timeline, and change register. It performs no network or live state changes. Commands print JSON and retain no new files:

These commands return exit0 when the computation completes, even for a returned BLOCK/REVIEW/HOLD/RECOVER result. Input errors return2. This differs from gate.py, whose valid BLOCK returns1. Inspect the documented decision field rather than assuming shell success means application approval.

The comparison requires the exact case/trial Cartesian product in trial_plan, not merely balance among surviving rows. Monitoring treats critical content failures independently of operational error/slow union. The saved manifest identifies the authored snapshot; generating a fresh manifest and verifying it immediately is only a self-check, not historical proof. Both identity checks remain distinct from authenticity or review approval.

Capstone

From the course package root, run python3 -B capstone/starter.py. It exits3 with starter_incomplete until its helpers are implemented. The full assignment is in the brief. Check the separate reference with python3 -B capstone/check_submission.py solutions/10_capstone.py; run the reference driver with python3 -B solutions/10_capstone.py.

The reference returns a bounded BLOCK recommendation despite a passing 4/4 route gate. Its report includes the saved input/code manifest and a digest of the decision code. It is an offline evidence exercise, not an Aabha-integrated grade or certificate.

Before using these ideas with a real application

Use independently designed cases before optimizing against them; preserve development/final-review boundaries; introduce real observations only with authorized data and execution; validate graders with domain judgment; preserve sampling and timing definitions; obtain actual release authority and recovery readiness. The lessons teach these decisions, but this local project does not execute them against a real application.