All courses · Course overview · Lesson sequence

Start with an answer contract

The help desk at our fictional Harbour Library receives a question: “May a book be renewed if another member has placed a hold?” Someone drafts a friendly reply: “Yes, you can renew your book for seven days.” The sentence is clear. It also gives the wrong advice.

The current handbook says:

A book may be renewed once for seven days if no other member has placed a hold. Ask the help desk before the due date.

This is an original synthetic policy written for the course. The exception is doing essential work. A system that notices “renewed” and “seven days” but drops “if no other member” changes the rule. Before choosing a model, we need to decide what an acceptable answer must preserve.

In this lesson, you will define an answer contract, follow evidence through a small local program and explain two retrieval failures. You need only Python file-running skills. We will derive the scoring mathematics later; you do not need to understand every line of the reference implementation before using this lesson.

First decide what would count as support

An answer contract describes what the system is allowed to claim and the evidence it must provide. For our question, a useful contract is:

Use only the current public Harbour handbook passages in the supplied dataset.
Preserve conditions and limits that change the advice.
Identify the source and its version so a person can inspect it.
If the passages do not support the requested fact, say the supplied handbook is insufficient and route the question to the help desk.
Do not perform account changes, renew a book or invent an access code.

The first rule defines our information scope. The second defines answer meaning. The third makes the decision inspectable. The fourth defines missing-evidence behavior. The fifth bounds the task to providing information. These are separate requirements: a source link does not repair a sentence that contradicts its source.

For this renewal question, a human-written supported answer would be: “Under the supplied renewal policy, a book cannot be renewed while another member has a hold. The handbook allows one seven-day renewal only when no other member has placed a hold; ask the help desk before the due date. Source: renewals-v1, version 1.”

Notice that the source is enough to answer this policy question. It does not tell us whether a particular book currently has a hold. We should not convert a policy explanation into a claim about a real member's account. There are no member records in this dataset.

Give retrieval and generation different jobs

Retrieval selects material from a collection in response to a question. It may produce a ranked list of passages. Generation produces new text, for example a concise answer written using those passages. A language model can be part of the second step; its output still needs checking against the answer contract.

Retrieval-augmented generation, often shortened to RAG, combines retrieved external material with a generative model. The original RAG research studied a trained architecture combining a neural retriever with a sequence-to-sequence language model. Our small learning project implements only a lexical retrieval foundation and copies source passages; it does not reproduce that architecture or call a language model. See Lewis et al., 2020.

Why start with retrieval? If the relevant rule never reaches a later generator, a carefully worded instruction cannot make that missing rule appear. Conversely, retrieving the right passage does not guarantee a later answer will preserve it. Separating these stages lets us locate a failure instead of treating every mistake as “the AI was wrong.”

Evidence before an answer — A question and a current public synthetic handbook enter retrieval. Retrieval returns ranked passages, which become a quote-only evidence packet preserving source identifiers and versions. An archived policy and a staff passage are excluded before indexing. A separate future generation step is outlined but not executed. A review asks whether the evidence supports every claim; no evidence means ask the help desk.

The figure shows a decision path, not a production architecture. In particular, the local “current/public” filter uses fields in a synthetic JSON file. It is not login, access control or a promise that a real system protects private documents. Keep real records out of this lab.

Inspect the collection before asking it questions

Open project/data/corpus.json. Each record has an id, a title, passage text, a source, a version, a status and an audience. An ID points to a particular passage version. The source identifier shows its fictional origin. These roles differ: the title is for reading, while a stable ID lets evaluation compare exactly which passage was returned.

There are nine records. Seven are current public passages. One archived hours policy gives an older, longer schedule. One harmless synthetic staff passage is outside the public handbook. Our index filters the last two out before calculating its term weights. The archived rule does not become correct because its wording resembles the question.

The synthetic:// source identifiers are labels, not websites. None should be opened or fetched. The policy credits, library name and operational details are invented; they are not Aabha learner policies or a real library's rules.

For a first manual check, find hours-v2 and hours-v1. Write down the two different schedules. If a system answered using the old schedule, would you classify that as poor wording, a source-selection failure or a missing policy? It is a source-selection failure: the current policy exists, but the system selected an ineligible version.

Run one question and read the evidence packet

From the package folder, run:

cd project
python3 -B lab.py search "May a book be renewed if another member has placed a hold?"

If your machine calls Python python, substitute that name. No package installation or key is required. The command reads local files, prints JSON and exits. It does not connect to Aabha, create an account or renew anything.

The top evidence entry is renewals-v1. Its quote field contains the entire renewal passage, including the hold condition and due-date instruction. Its source is synthetic://harbour/handbook/renewals, and its version is 1. A second entry may share question words while adding no evidence for this particular question; ranking is a candidate-selection step.

Read these two top-level fields before interpreting any result:

{
  "mode": "quote_only",
  "model_used": false
}

The program has not drafted a sentence answering the question. You are looking at evidence to review, not a hidden LLM response. The human-written answer earlier in this lesson was a worked example, not output from the program.

The retrieval_score compares the question's words with a passage's words after weighting and normalization. It is not “the probability this answer is true.” The index does not understand whether “if no other member has placed a hold” permits or forbids the requested renewal. It only finds wording that makes this passage worth inspecting.

Later, we will represent word weights as vectors and compare their directions. For now, keep one distinction: a similarity score concerns matching text; support concerns whether the source justifies the claim. A bag-of-words representation loses word order, a limit explained in the Stanford information-retrieval text.

A relevant-looking passage can still be insufficient

Predict the result of this question before running it:

python3 -B lab.py search "May I borrow a laptop for fourteen days?"

The first passage is borrowing-v1. It describes borrowing books for fourteen days. It says nothing about laptops. The query shares useful-looking words with that passage, so the retriever returns it. The unseen word “laptop” has no dimension in the index and contributes nothing to the score.

A confident “Yes, you may borrow a laptop for fourteen days” would be unsupported. Even a citation to borrowing-v1 would not make it valid. The claim changed the object from books to laptops.

What should happen instead? A human reviewer should say that the supplied handbook gives a book-loan rule but no laptop-loan policy. The next action is to ask the help desk. The current program cannot make that semantic judgment; it simply displays the passage and warns that retrieval is not answerability. Do not describe it as a safe automatic answering assistant.

Now try the opposite failure:

python3 -B lab.py search "What is the penalty for tardiness?"

The program returns no evidence. In the fictional library context, a human has labeled charges-v1 as the intended relevant passage: it describes overdue charges. But the question uses different words. This lexical baseline has no synonym mapping or learned semantic representation.

An empty result therefore does not prove that the collection lacks the fact. We have two different failures: missing a relevant passage and returning irrelevant evidence. We need both kinds in our practice set. Improving one can worsen the other; adding broad synonym rules might recover some paraphrases while creating new accidental matches.

Measure evidence separately from answer quality

Run:

python3 -B lab.py evaluate

questions.json contains ten questions with explicit relevance labels. Eight have at least one supporting passage; two have none. These are visible teaching cases authored alongside the program, not an unseen test set.

For each answerable question, recall@2 asks: “What fraction of the labeled relevant passages appeared in the first two results?” If a question needs two passages and we retrieve only one of them, its recall@2 is 1/2. Returning one useful item is progress, but it is not complete evidence.

The report also uses reciprocal rank@2: score 1 if the first relevant passage is first, 1/2 if it is second, and 0 if neither result is relevant. Averaging this across answerable questions describes how early a useful passage appears. It does not tell us whether every needed passage was found. Ranking evaluation uses an explicit cutoff so the result reflects a bounded list, as discussed in the information-retrieval evaluation chapter.

The supplied fixture yields mean recall@2 0.875 and mean reciprocal rank@2 0.875 across eight answerable questions. Both means are 7/8 because seven questions find a relevant passage first and the paraphrase finds none. This equality is specific to these cases; the metrics do not generally measure the same thing.

The two unsupported questions have a separate count. One produces no hits; the laptop question produces false evidence. We do not quietly remove those cases because the retrieval average looks good. We also do not give an unsupported question a fabricated recall score when its relevant-set denominator is zero.

These metrics do not grade generated answers, because there are none. A future generation exercise must separately inspect whether each claim is supported, whether conditions are preserved, whether the answer covers the question and whether missing evidence is handled appropriately. It must also evaluate what survives any context budget. Our current evaluation measures ranking before the quote budget.

Your turn

Before the independent budget exercise, try a smaller worked example. The CLI option --max-quote-chars limits the sum of source-quote characters, excluding JSON metadata. It copies whole passages in rank order and skips a passage that does not fit; it never cuts a quote to the limit. Imagine ranked passages of 70 and 30 characters and a budget of 50: the first is skipped, the second fits, and the packet contains 30 quote characters. A returned empty packet can therefore mean context selection lost the evidence, even when retrieval found a relevant passage.

Run the taught return question with --max-quote-chars 40. Both returned passages exceed that budget, so the packet has status: no_evidence. The policy still exists in the corpus. This worked check separates retrieval from a later budget decision and prepares you to explain the same distinction independently in practice.

Open the separate exercise sheet. You will review a flawed renewal answer, find evidence for an after-hours return, calculate a small ranking example and design a fresh question pair. Do the reasoning before opening the solutions.

Then run the code checks:

python3 -B -m unittest discover -s tests -v

The checks should pass. They verify specific program behavior, including a deliberately demonstrated false-evidence case. Passing them does not mean the retriever answers every question correctly, and it tells us nothing about real learner comprehension.

You are ready to continue when you can explain why a correct source can accompany an incorrect answer, why no retrieved evidence differs from no evidence existing, and why the program's model_used: false matters. Module 2 explains what changes when a generator enters this path.

Read separate answers and explanations