Lesson 1 practice

Use only the supplied synthetic corpus. Write your reasoning in a separate scratch file or on paper. The project is already runnable; these tasks assess your decisions, not your ability to copy the reference implementation. Solutions are separate.

A. Repair a claim, not just a citation

Question: “May I renew a book that another member has placed a hold on?”

Draft answer: “Yes. Books can be renewed once for seven days. Source: renewals-v1.”

Find the exact condition that the draft drops. Write a supported two-sentence answer and identify one fact about an actual member's situation that the corpus cannot establish. Explain why attaching the correct passage ID did not make the draft correct.

B. Trace evidence for an after-hours return

Before running code, choose which current public passage should support “Where can I return books when the desk is closed?” Name its ID, source version and the sentence that supports the answer. Run the CLI and compare the first returned passage with your prediction. If a second passage appears, decide whether it adds evidence or only shares wording.

Next run the same search with --max-quote-chars 40. Explain why the packet might have no evidence even when retrieval found candidates. Does that mean no relevant policy exists?

C. Calculate two metrics with different questions

Suppose a question has two relevant passages, hours-v2 and returns-v1. A proposed ranking is:

Calculate recall@2 and reciprocal rank@2 by hand. Calculate recall@3 and reciprocal rank@3. Explain which metric notices the second supporting passage and which stops caring after the first relevant hit.

D. Design a pair that can expose a shortcut

Write two new questions that are not in questions.json: one answerable from the current public corpus and one unsupported question sharing important words with it. For each, label all passages you consider necessary and explain why. Do not change the retriever yet. Run both questions, record whether the evidence is enough, and identify a possible failure type.

Treat these as development examples once you have inspected them. Describe how you would obtain separate, previously unseen questions for a later evaluation. Do not call your two-question exercise a learner pilot.

E. Write a bounded conclusion

A colleague says, “The tests pass and recall is 87.5%, so our RAG assistant is ready.” Write a three- or four-sentence correction that names what was actually tested, mentions both known failure cases, states whether a model was used and identifies the next measurement needed before making an answer-quality claim.

Self-check rubric

There are five criteria, each worth two points: preserve the renewal condition; trace source/version and explain budget loss; compute both metrics with denominators; label a useful answerable/unsupported question pair; state evidence limits honestly. Award one point for a partially correct explanation and two for a correct explanation with concrete evidence. Aim for 8/10 before continuing, and revisit any omitted policy condition regardless of total. This is a draft self-check rubric, not platform grading or a certificate criterion.