Defend an evidence-grounded capstone

The fictional Harbour Library's handbook has grown. Someone has added study-room and accessibility information, and a volunteer wants to reuse our evidence finder for those questions. The opening renewal example still works. That is encouraging, but it is not enough to tell the volunteer what the larger collection can support.

Your capstone is to answer that narrower engineering question: how does a declared retrieval and context policy behave on a frozen synthetic collection, and what would a person still have to check before using its evidence? You will compare alternatives, preserve failures and make a recommendation whose scope fits the work you actually ran. You will not deploy a library service, make reservations or claim that a language model answered the questions.

This lesson brings together the earlier modules. It does not introduce another retrieval framework. You will use the index, whole-passage context selector, labeled questions and source identities that you have already inspected. The challenge is to connect them without letting a good-looking summary conceal a weak stage.

Make the claim smaller than the demonstration

Begin your report with one sentence describing the actual program. A defensible version is: “This local Python project ranks current public synthetic handbook passages and selects whole quotations under a character budget.” That identifies the mechanism and the data boundary. The sentence “This system answers library questions accurately” would require evidence about answer generation and correctness that the quote-only program does not provide.

The distinction matters even after studying generation. Our count-based toy language model in Module 2 actually learns and decodes small token-transition counts; it is not an LLM. The fixed vectors in Module 5 are supplied numbers, not outputs of an embedding model. The reviewed renewal template in Module 7 copies an authored claim under an explicit source-identity condition. It is not a general question interpreter. Give each component the name that describes what ran.

Keep that boundary in your capstone. You may discuss a later model integration and specify how you would evaluate it, but an unexecuted design is not a benchmark result. A quote-only capstone can demonstrate evidence engineering. It cannot demonstrate the performance of an LLM-based answering system.

Freeze what the comparison means

The reference project extends the original fictional collection with study-room and accessibility passages. Inspect capstone_extension.json and capstone_questions.json before running anything. Read the passages themselves, including conditions and versions. A label that points to an ID is useful only after someone has decided that the passage supplies the required evidence.

A comparison needs a stable collection, stable relevance labels and a declared policy. Suppose you run one configuration, rewrite a passage to include the query's wording, and then run the second configuration. The difference now includes a corpus change. You cannot attribute it entirely to the ranking or context budget. The same problem arises if you change the expected labels after seeing a disappointing result.

Record the input files and their exact hashes, the code revision or code-file hashes, Python version, retrieval cutoff k, and quote-character budget. A hash lets another person identify the bytes you used. It does not establish that the policy is true, that a label is correct or that the collection is legally reusable. Those are separate review questions.

For your own extension, author questions and relevance judgments before inspecting candidate results. Use the existing examples to develop your approach; reserve a new set for a later comparison. Once you inspect the reserved results and tune against them, that set has become development material. There is no benefit in continuing to call it untouched test evidence.

Follow one failure through both stages

Consider a question that needs the desk's opening hours and its closed-hours return instructions. Its relevance judgment contains two source IDs. Imagine that the retriever returns both in its top two results. Retrieval recall at that cutoff is 2 / 2 = 1.

Now imagine that the two complete passages contain 70 and 90 characters and the quote budget is 100 characters. These lengths are a small arithmetic example, not a claim about the supplied files. The selector can include the first passage, using 70 characters, but cannot add the second: 70 + 90 exceeds 100. Because the selector preserves whole passages, it skips the second rather than cutting off a condition.

The resulting packet contains one of the two required evidence IDs. Its labeled evidence coverage is 1 / 2 = 0.5. The retriever did not miss the second source; the context policy dropped it. Reporting retrieval recall alone would hide the failure that the next stage actually receives.

Increasing the budget may restore both quotations. That improves this case's evidence coverage, but it does not make either quotation support every possible claim. A citation to the book-loan policy still cannot establish a laptop-loan rule. A packet with all expected IDs also does not prove that a later answer preserves their conditions. Keep retrieval, packet coverage and claim support as separate observations.

Run the complete reference before changing it

From the course's project directory, run:

python3 -B capstone_reference.py

This command runs the local lexical/context reference against its supplied synthetic inputs and exits. It does not start a service, call a model, store member records or perform a library action. Read the per-case rows as well as the summary. Find one question that needs more than one source and one question whose requested fact has no supporting passage.

Then run the same function with a deliberately restrictive policy:

python3 -B -c "import json; from capstone_reference import run_capstone; print(json.dumps(run_capstone(k=1, max_quote_chars=80), indent=2))"

Finally compare it with:

python3 -B -c "import json; from capstone_reference import run_capstone; print(json.dumps(run_capstone(k=2, max_quote_chars=400), indent=2))"

The first comparison changes both k and the budget, so it is useful as a demonstration of combined behavior. It does not isolate their individual effects. To investigate a cause, hold one fixed: compare k=1 and k=2 at 400 characters, then compare budgets 80 and 400 at k=2. Predict a case that each change could affect before you run it.

With the supplied, visible six-case capstone fixture, the reference produces this comparison. The recall and packet means each use the four answerable cases; the two unsupported cases are counted separately.

For cq-compound, k=1 retrieves study-room-v1, while k=2 retrieves both that passage and hours-v2. At 400 characters, both can reach the packet; at 80, neither complete passage fits. This explains why raising k alone improves retrieval but leaves packet coverage at zero under the small budget. Mean reciprocal rank is 1 at both cutoffs because every answerable case has a relevant first hit. It does not reveal that a multi-source question lacks its second source. No generated answer is being scored in this table.

You should not expect every larger setting to solve every problem. An unseen paraphrase can remain invisible to a lexical index. An unsupported query can still share words with an irrelevant passage. The reference supplies a reproducible baseline, not a hidden ideal answer to every question.

Build your own comparison

Open capstone_starter.py. Its intentional TODO marks the learner's implementation boundary: learner_select(hits, max_quote_chars) must select the context. The reference already supplies loading, retrieval and evaluation orchestration; you are not being asked to rebuild those parts in this hook. First explain the hits and budget inputs and the selector's output contract in your own words. Then implement the whole-passage selection policy taught in Module 6, preserving the packet, skipped-ID and version-conflict fields expected by the runner. Run the starter as documented before and after your changes so that an unimplemented placeholder cannot be mistaken for a finished capstone.

Your own report should preserve per-question evidence: the requested fact, expected source IDs and why they are relevant, retrieved IDs, selected quote IDs, omissions caused by budget or conflict, and the interpretation a person would make. Record unsupported questions separately. They have no relevant-document denominator, so do not turn an undefined recall into zero and mix it into the answerable mean without explaining that new metric.

Add a small original extension with a version conflict, a condition-bearing passage, a multi-source question and a question the collection cannot answer. Use only fictional data. The exercise is about reasoning through evidence, not importing real library records or broadening the collection until every development question matches a keyword.

Make a recommendation someone can inspect

A useful recommendation names the policy, what improved, what still failed and what the evidence cannot establish. For example: “For this frozen synthetic collection, I prefer the two-passage configuration with the larger quote budget because it preserves more of the labeled multi-source evidence in my comparison. The unsupported-object query still returns related wording, so a person must inspect support. No LLM, real user traffic, access boundary or production deployment was evaluated.” Replace the comparison claim with your actual observed result; the sentence structure is not a prefilled conclusion.

Attach the failures instead of hiding them in a limitation sentence. Show the expected and returned IDs for at least one case, explain whether the cause belongs to collection, retrieval, context or claim review, and propose a next experiment that could distinguish two explanations. A proposed embedding experiment should state what encoder and data would be needed; our fixed-vector arithmetic cannot supply that evidence.

The capstone practice gives the submission boundary and a formative rubric. Work through it before reading the explained solutions. These are local learning materials. They do not submit evidence to Aabha, award platform credit or issue a certificate.

Your finished capstone is an inspectable argument, supported by runnable work. Its strongest result may be a clear explanation of why a configuration should remain a local evidence aid. That conclusion is useful when it identifies the next test and prevents a demonstration from being mistaken for a trustworthy service.