Module 10 Activity

Scenario

Something produced a wrong answer without raising. This is how you find it and how long you keep the evidence.

What you build

A span-per-step trace, a backwards debugging pass, run comparison, and per-field retention.

Steps

1. Emit one span per step with inputs, outputs and duration - including scores on the retrieval span.
2. Read a trace from a healthy run first, so you know what normal looks like.
3. Plant a retrieval fault and find it by reading backwards from the symptom. Note where reading forwards would have sent you.
4. Run the same input twice with one config field changed and attribute the difference.
5. Score retrieval and generation as separate booleans across your eval set.
6. Set retention per field: metrics and versions indefinitely, content on a short limit.

Evidence to hand in

• A full trace with retrieval scores.
• The healthy-run trace you read first.
• The planted fault, found backwards, with where forwards would have led.
• The two-run comparison and its attribution.
• Retrieval and generation scores as separate columns.
• The per-field retention table.

Review checklist

• Retrieval spans carry scores, not just ids.
• The backwards pass identified the first wrong value.
• The run comparison was possible because versions were recorded.
• Retrieval and generation are scored separately.
• Retention differs by field rather than one policy for the whole trace.

Mark complete