Harbour handbook retrieval lab

Everything in data/ is fictional. synthetic:// identifiers document provenance; they are not network URLs. Nine source records include seven current public passages, an archived hours policy and a harmless staff-only scope example.

Run from this folder:

python3 -B lab.py search "When is the help desk open?"
python3 -B lab.py search "What is the penalty for tardiness?"
python3 -B lab.py search "May I borrow a laptop for fourteen days?"
python3 -B lab.py evaluate --k 2
python3 -B -m unittest discover -s tests -v

The search output is a quote-only evidence packet, not an answer. The evaluate output ranks evidence and compares IDs to questions.json; it does not grade generated text or the quote budget. The score is neither a probability nor calibrated confidence.

What each file does

• retrieval.py: strict file loader, current/public corpus filter, lexical index, whole-passage quote selection and retrieval metrics.
• lab.py: a command-line interface that prints JSON and exits.
• data/corpus.json: versioned synthetic source records. Each record already represents a passage; there is no automatic chunking.
• data/questions.json: ten development/teaching cases with human-authored relevance labels, including known failure cases. All were visible during authoring.
• tests/test_retrieval.py: fifteen correctness tests independent of any real learner account.
• methods.py: eight offline experiments for Modules2–9, including actual count-based toy decoding/chunking and explicitly labeled manual-vector/template/manual-label methods.
• capstone_reference.py: complete lexical/context comparison over the original corpus plus two synthetic extension records.
• capstone_starter.py: intentional learner TODO in learner_select; it exits2 with guidance until implemented. Orchestration is supplied by the reference.
• tests/test_methods.py and tests/test_capstone.py: additional method/capstone contracts, bringing the full suite to 36 tests.

Beyond the opening baseline

Run python3 -B methods.py NAME, where NAME is decoding, chunking, lexical, hybrid, context, generation, evaluation or failures. The exact mechanism is labeled in each result and explained in the corresponding lesson. The fixed-vector simulation has no text encoder. The answer template is human-authored for one question ID. The claim evaluator sums explicit manual labels rather than inferring support.

Run python3 -B capstone_reference.py --k 2 --max-quote-chars 400 for the full reference. Change one argument at a time to isolate a comparison. run_capstone(k, max_quote_chars, selector=select_context) is the programmatic hook. A selector returns the packet, budget-skipped IDs and source/version-conflict mapping taught in Module6. All six capstone questions were visible during authoring; none is a hidden evaluation set. See the capstone lesson.

The scoring rule, stated precisely

Filter to current/public records before computing index statistics. Lowercase each passage, extract [a-z0-9]+ words and remove the declared stop words. Index only passage text; titles are displayed as metadata and do not affect scores.

For each remaining term t, use raw occurrence count tf(t,d) and smoothed weight idf(t) = log((N + 1) / (df(t) + 1)) + 1, with natural logarithm. N is the eligible passage count and df(t) is the number of eligible passages containing the term. Multiply term count by IDF and divide each vector by its Euclidean length. Apply the same weighting to the query using the corpus IDF; omit query words absent from the vocabulary. A zero-length vector stays empty. Score with the dot product of normalized vectors, retain only strictly positive scores, sort descending, and resolve exact ties by document ID. This is an intentionally small TF-IDF/cosine variant. Weighting reference, normalization reference.

The packet copies entire passages in ranked order if their quote text fits the remaining character budget. An oversized passage is skipped. A lower-ranked shorter passage may then fit. The budget excludes JSON metadata and is not a token budget. Preserving a whole passage avoids cutting an exception in half; it does not establish that the passage supports an answer.

Metrics and denominators

For a question with a non-empty labeled relevant set, recall@k is the number of relevant IDs in the top k, divided by the total labeled relevant IDs. Reciprocal rank@k is 1/r if the first relevant ID occurs at rank r <= k, otherwise zero. The report averages each metric across the eight answerable cases, giving each question equal weight. The two-source question needs both labeled passages for full recall. Evaluation reference.

Questions with an empty relevant set use a separate denominator: did retrieval return no hits? One of two such questions passes. A non-empty result on an unsupported question is counted as a false_evidence_return, even though no answer is generated. If a report has no answerable cases, its mean metrics are JSON null, not zero or a fabricated perfect score.

No accuracy threshold is asserted for course release. Inspect individual cases before changing ranking parameters. Keep fresh, unseen questions aside before any future tuning; do not rename the supplied visible examples a test holdout.