Answers and explanations · Start with an answer contract

Read after attempting the practice. Wording may vary if it preserves the evidence and its limits.

A. The condition changes the answer

The omitted condition is “if no other member has placed a hold.” A supported answer is: “Under the supplied policy, a book cannot be renewed while another member has a hold. The handbook permits one seven-day renewal only when no other member has placed a hold; ask the help desk before the due date (renewals-v1, version 1).”

The corpus cannot tell us whether a hold actually exists on a particular member's book. A citation points to a source; the reader must still compare the claim with what the source says. Here the draft contradicted the condition in the passage it cited.

B. Follow the whole path

The necessary passage is returns-v1, version 1, source synthetic://harbour/handbook/returns. Its supporting sentence is: “The outdoor return box accepts books when the desk is closed.” The normal CLI returns this passage first. It also returns borrowing-v1 at the default cutoff; that passage is about borrowing limits and adds no evidence about the return box.

With a quote budget of 40 characters, neither retrieved passage fits in full, so both are skipped and the packet has status: no_evidence. The ranker did find the relevant return rule, but context selection omitted it. This is a budget-selection problem, not evidence that the policy does not exist. The present evaluation command measures retrieval before this step, so it would not detect that loss by itself.

Cutting the passage arbitrarily to make it fit could remove a condition or object. A later context-design lesson should investigate better passage boundaries and a separate packet-coverage evaluation, rather than silently calling a partial quote complete.

C. Count all evidence, then locate the first useful rank

At k=2, only returns-v1 is present among the two relevant passages, so recall@2 is 1/2 = 0.5. The first relevant passage is at rank 2, so reciprocal rank@2 is 1/2 = 0.5.

At k=3, both relevant passages appear, so recall@3 is 2/2 = 1. The first relevant passage is still at rank 2, so reciprocal rank@3 stays 0.5. Recall notices the additional supporting passage. Reciprocal rank only reflects how early the first relevant result appeared. This is why the same number for both metrics on our supplied fixture is not a general identity.

D. One possible question pair

Answerable: “Must children attend the Saturday story circle with an adult?” Relevant evidence: story-v1, because it explicitly says children must attend with an adult.

Unsupported: “May adults attend a midnight story circle?” Relevant evidence: none under the given corpus. story-v1 describes a Saturday event at 11:00 and a condition for children; it establishes no midnight event or general adult attendance policy. A retriever may still return it for the shared “story circle” terms.

Other pairs are acceptable. “Unsupported” must mean the supplied sources do not establish the requested answer, not that the question is impossible or that a particular answer is false. Record every needed source for a compound question. Relevance labels reflect a stated interpretation, so document ambiguity rather than pretending annotators cannot disagree.

Once you have inspected the outputs, these questions are development examples. For a later test, ask another reviewer to draft and label fresh questions without showing the current failure list, agree the intended meaning, then reserve those cases before making further changes. A two-question technical exercise gives no evidence about learner completion or comprehension.

E. A bounded conclusion

“Local tests verified the offline program's stated contracts, and eight synthetic answerable questions gave mean recall@2 of 0.875. The same development fixture also contains a missed paraphrase and a laptop question that retrieves unrelated book-loan evidence. That quote-only retrieval run called no LLM and measured no generated-answer quality. We need independent questions and separate claim-support, completeness and abstention review of an actual answer stage before making that claim.”

The tests are useful because they check concrete contracts, not because every case succeeds at semantic retrieval. A regression test that preserves a known limitation can help us remember the limitation; it does not turn that behavior into a desired product result.