Capstone practice · defend the evidence path

Complete Lesson 10 and attempt the work before opening the separate solutions. Use synthetic data only. This capstone has no production submission endpoint, grade or certificate.

1. Establish the reference

Run capstone_reference.py from project/. Record Python version, input/code hashes and the policy parameters. Select one multi-source case and explain why its expected IDs are jointly needed. Select one unsupported case and explain why the closest-looking passage is insufficient. Do not infer relevance merely from the reference's labels: inspect the passage text.

2. Isolate the change

Compare k=1 with k=2 at a fixed 400-character budget. Then compare budgets 80 and 400 at fixed k=2. For each pair, predict one likely affected case, run both configurations, and keep the per-case rows. Distinguish retrieval misses from quotations dropped after retrieval. A result that contradicts your prediction needs an explanation, not a relabeled expected answer.

In the lesson's arithmetic example, both relevant sources are retrieved, their lengths are 70 and 90, and the budget is 100. Calculate retrieval recall and selected-evidence coverage. State whether either metric establishes claim support.

3. Complete the learner path

Implement the learner_select(hits, max_quote_chars) TODO in capstone_starter.py using the context-selection rules already taught. Loading, ranking and evaluation orchestration are provided by the reference runner. Preserve provenance and whole quotations; return its declared packet, budget-skipped and conflicting-version fields. Use the resulting report to distinguish “no relevant source is labeled,” “retrieval did not return a required source” and “context selection dropped a returned source.” Match the selector contract before adding an improvement of your own.

Create a small fictional extension and at least four original questions: a condition-bearing rule, a multi-source question, a stale/version-conflict trap and an unsupported request. Label evidence before tuning. Describe which questions were used for development and which were first inspected after you froze the policy. Do not describe the supplied course examples as an independent model benchmark.

4. Review one answer separately

Write a short human answer for one supported question and list each factual claim with its source. Mark which condition would change the advice. Write an intentionally plausible but unsupported answer for the unsupported case, then explain precisely why its citation does not support its claim. These are human-authored review exercises; do not present them as model output.

Explain what additional evidence would be needed for an actual LLM integration: model/configuration identity, actual returned answers, claim-level review, repeated or otherwise declared sampling where relevant, and fresh evaluation cases. No paid account or model experiment is required or authorized by this exercise.

5. Defend the recommendation

Submit a local folder with your completed starter, synthetic additions, labels, reproducible commands, concise per-case comparison and recommendation. Keep it free of real records and credentials. A second person should be able to reproduce the work without guessing which data or budget you used.

Use four equally weighted self-review dimensions: source/provenance correctness; retrieval and context reasoning; evaluation design and interpretation; clarity about claims and limitations. For each, describe concrete evidence for “needs revision,” “adequate” and “strong.” A convincing total cannot compensate for fabricated execution, hidden unsupported claims or calling synthetic fixtures a learner pilot. This is a formative self-review rubric, not an approved Aabha grading policy.

Before reading the solutions, answer: what did your program actually generate; what did a human supply; and which proposed step was never executed?