Module 7 Test and operate the workflow

Time plan: 30 minutes teaching, 35 minutes example, 65 minutes practice and 20 minutes checks. Keep the complete test log, including failures. Prerequisite: Modules 1–6.

What you will be able to do

Build an evidence-based test suite, compare expected and actual outcomes, calculate honest metrics and write an operating plan with stop conditions and an owner.

Test the promises in your contract

A test case contains an input, source version, expected result, reason and pass condition. Decide the expected result before running the system. Otherwise it is easy to rationalise whatever the model produced. Preserve the actual result, not just a green tick. A reviewer must be able to distinguish a real observation from an authored example.

Test classes should reflect likely failures. Include a normal request; a value exactly on the policy boundary; one just above it; missing and invalid values; a guest or exception; a policy-date change; contradictory approved sources; an attempt to bypass instructions; a duplicate delivery; and an uncertain write outcome. A normal-case success cannot compensate for leaking another request or falsely claiming a booking.

Some outcomes may be expressed in different words while still being correct. Judge the required status, preserved facts, sources, uncertainty and authority. Do not fail a harmless wording difference. Conversely, a matching status does not rescue prose that says “reserved.” Check the whole output.

A useful test record

Record case ID, input reference, configuration version, workflow version, source pack, expected route, actual route, evidence type, original output, validation findings, side effects or queue references, pass/fail and corrective action. Evidence type is one of observed live, manual rehearsal, simulated failure or expected only.

A simulated failure means you deliberately injected a condition such as the loss of an acknowledgment in a paper trace. It does not prove that a vendor actually produced that error or that its error handler worked. Keep simulated and live results separately visible. The course-provided reference model tests validate authored rule logic only; they are not LLM evaluations or vendor execution evidence.

Worked metric example

Suppose a pilot receives 20 events. Two are exact duplicate deliveries and are correctly suppressed. Eighteen unique request revisions remain. Of those, 15 reach a valid intended disposition and three remain on technical hold. Four of the 18 require exception review, and two drafts contain a policy claim error before correction.

Safe-disposition rate is 15 divided by 18, or 83.3 percent. Duplicate suppression in this small set is two of two known replay events. Exception-review rate is four divided by 18, or 22.2 percent. The claim-error count is two drafts; to report a claim-error rate you must define the denominator, such as number of drafts checked or number of individual claims inspected. Do not silently call it two out of 20 if not all events produced a draft.

All 18 unique requests still require human review before a later action in this course. Exception review is a subset that needs additional judgment, not the entire review load. “Automation rate” would be misleading if it ignores the operator's validation and approval work. Record time spent reviewing, repairing, reconciling and maintaining the knowledge pack as well as time saved drafting.

Cost and capacity are part of operation

A workflow can charge for checks, branches, searches, model usage and retries differently across tools. The same ten requests may produce more than ten billable units. Use the current vendor's usage screen and documentation, then record observed consumption for a small test. Do not infer exact costs from the number of boxes on a diagram.

A useful budget model is planned runs multiplied by observed average units per run, plus a visible allowance for retries and monitoring. It is an estimate, not a bill guarantee. Set a stopping threshold below the permitted quota or budget. If a tool invites an upgrade, extra-credit purchase or auto-recharge, that is a separate decision outside this course.

Work a capacity estimate from a small usage log

Open practice/synthetic-usage-log.txt. Its learning units and price are invented for arithmetic practice; they are not Make credits, a provider quote or observed billing. Each batch contains ten unique requests. Batch A uses 40 base units, 0 retry units and 4 monitoring units. B uses 42, 6 and 4. C uses 38, 2 and 4. Total: 120 base + 8 retry + 12 monitoring = 140 units for 30 requests. Keep the categories separate so you do not add retry overhead twice.

The base average is 120 ÷ 30 = 4 units per request. Retry overhead is 8 ÷ 30, about 0.267 units per request. Monitoring is 4 units per ten-request batch in this fictional process. For 100 planned requests, estimate 400 base units + 27 retry units (rounded up) + 40 monitoring units = 467. Add a 20 percent uncertainty reserve, rounding 93.4 up to 94: the planning total is 561 units. At the fictional price of USD 0.01 per unit, that is USD 5.61. It is an estimate, not a measured bill.

The training allocation is 600 units, but the owner has set a conservative operating stop at 480 to preserve headroom. A 561-unit plan fits the allocation and still fails that operating rule. This is why “we have enough credits” and “we should run the whole batch” are different decisions.

An 80-request tranche needs 320 base + 22 retry + 32 monitoring = 374 units before reserve. Its reserve is 75, giving 449 units and a fictional cost of USD 4.49. It fits the 480-unit operating threshold, so the owner can choose that smaller tranche and reassess after observing usage. There is no guarantee future requests resemble these three small batches.

Before starting another batch, compare cumulative usage plus its conservative forecast with the stop threshold. At 455 units already used, a next batch forecast of 57 (rounded up from the reserved per-batch estimate) would project 512, so pause before starting it. Do not wait for the allocation to be exhausted.

Now independently estimate 60 requests using the same log and reserve. Then examine the proposed monitoring change in the practice card. Explain which assumption changes and whether a 100-request plan remains within the allocation and operating stop. Keep your arithmetic separate from any real provider's pricing.

Write a small operating plan

The runbook should say who owns the workflow, who reviews queued items, which source version is active, how often pending items are checked, what triggers a pause and how to recover. A queue with no reviewer and no inspection time is a hidden backlog, not a completed automation.

For the fictional pilot, choose a small supervised batch, one operator at a time. Review every output. Stop on any unauthorised action claim, duplicate side effect, missing source pack, unresolved policy conflict, access error, unexpected data exposure or exhausted retry. Pause the trigger before repairing configuration. Preserve enough non-sensitive evidence to understand the failure. After a repair, rerun the failing case and the relevant regression set before resuming.

Monitor both workflow activity and business outcome. “The scenario ran at 10:00” does not tell you whether the review record appeared or was checked. Conversely, zero events may be normal or may mean the trigger broke. Define an expected activity window and a manual check rather than assuming silence means success.

Read a filled fictional runbook

Here is a worked operating plan for the 80-request supervised training tranche. It contains decisions, not just headings:

- Owner: Mira, fictional desk owner. Reviewer: Sam, fictional desk trainer. One queue operator handles one request at a time. If Sam is unavailable, leave drafts pending and pause new batches once ten items are waiting.
- Active materials: configuration 1.0, workflow design 1.0, P1/F1 for requests as of 14 October 2026. The source manifest is checked before each session. A November session must switch to P2/F2 and run the change tests first.
- Trigger and schedule: manual batches of ten during the training day. No unattended schedule or connected sender exists. Sam checks the queue at 09:00, 12:00 and 15:30 UTC, and before the operator closes the session.
- Expected result: after a batch finishes, the operator accounts for every input as a confirmed queue item, a log-only replay or an explicit hold. If an expected queue item cannot be found after five minutes, investigate its operation key; do not create a replacement merely to make the count match.
- Capacity rule: 600 fictional units allocated, 480 operating stop, initial forecast 449 for 80 requests including reserve. Before each batch, pause if used units plus that batch's conservative forecast would exceed 480. Record actual usage separately; no auto-purchase is enabled.
- Safety and backlog stops: pause immediately on unsupported action claims, missing/conflicting sources, access errors, unexpected data exposure, duplicate effects or exhausted retries. Pause new work at ten pending items or any item left pending beyond one training business day, whichever comes first.
- Pause action: the operator does not press Run again and verifies any optional schedule is off. Existing pending records remain available to Sam. A pause does not erase the queue or mark items approved.
- Recovery evidence: retain the request/revision, original output, source/configuration version, operation key, attempt count and destination check. For unknown writes, Mira requires reconciliation to an existing record or a documented continuing hold. Temporary reads use the Module 6 bound; safe writes have a separate one-retry limit.
- Restart decision: Mira reviews the cause and repair. The operator reruns the failed case plus an ordinary case and the relevant duplicate/source-change regression. Resume only after the expected effect is verified, holds are accounted for, and the reviewer and remaining capacity are available. A new prompt alone is not restart evidence.
- Retention and access: use only synthetic course material in the permitted training location. Keep evidence for seven days after assessment, then have the owner apply the authorised cleanup process. Do not delete the ledger mid-test or silently broaden access. This plan is a human-run training procedure, not an implemented multi-user permission system.

To adapt the plan, change decisions rather than just names. If your reviewer is available only at 16:00, how many requests can safely wait, when will you pause and what will you tell the operator? If the period's remaining allocation is lower, recalculate the tranche rather than keeping 80 because the example did. Record the tradeoff: a lower backlog threshold reduces waiting but may stop intake sooner.

Practice

1. Build a test suite using all 12 Pack A events. State expected outcomes before processing them.
2. Add the four change-run cases from the capstone: two webcam boundaries, an unchanged projector and an unresolved approved-source conflict.
3. Execute through your chosen route, or rehearse with the provided records. Keep the evidence label per case. Do not fill the actual column from the answer key.
4. Add Failure F-B and F-E. Assess whether the queue effect and approval state remain safe.
5. Calculate safe-disposition and exception-review results from your own log with explicit denominators. If nothing ran live, name these rehearsal results.
6. Complete the 60-request and changed-monitoring estimates in the practice card. Adapt the worked runbook for a reviewer available only at 16:00 and a starting backlog of eight items. Choose a justified batch/backlog limit, pause action and restart evidence; do not copy the original timetable unchanged.
7. Make a release decision: ready for another supervised synthetic pilot, revise and retest, or blocked. Explain the evidence supporting it. A passing synthetic pilot is not a release to real users.

Expected artifact: test suite, completed evidence log, two metric rates, two independent capacity estimates, an adapted runbook and a reasoned release decision.

Check your understanding

M7 Q1. What is wrong with reporting “100 percent success” after excluding three technical holds from the denominator without disclosure?

M7 Q2. Why should the exact-boundary and just-above-boundary cases both be tested after a policy update?

M7 Q3. A scenario history shows a successful run, but no reviewer can find the record. Which outcome is still unverified?

Before you continue

Your evidence should let someone else reproduce the conclusion without trusting your memory. The capstone now combines the contract, configuration, sources, routing, recovery and operating plan.
