Explain a lexical retrieval baseline

Our source passages now have stable identities. We still need to choose which ones to inspect for a question. The first temptation is to count matching words. That can work surprisingly well for an exact policy phrase, but a common word such as “desk” appears in many passages. We want a baseline that makes its weighting choices visible.

We will use TF-IDF and cosine similarity. These names describe arithmetic, not intelligence. The purpose of this baseline is to establish a reproducible comparison and expose errors before trying less inspectable representations.

Start with two very small documents

Let document A contain red blue and document B contain red. The query is blue. Counting occurrences gives A one match and B none. Now ask what makes blue informative: it occurs in only one of the two documents, while red occurs in both.

Term frequency, tf(t,d), is the number of occurrences of term t in document d in this implementation. Document frequency, df(t), counts eligible documents containing the term, once per document. With N eligible documents, our smoothed inverse document frequency is idf(t)=log((N+1)/(df(t)+1))+1. Weight is tf×idf. This is one TF-IDF variant; alternative formulas are possible. The family of weighting ideas is introduced in the primary IR textbook.

For our pair, N=2. Red has df=2, so idf(red)=log(3/3)+1=1. Blue has df=1, so idf(blue)=log(3/2)+1≈1.405465. Notice that our smoothing gives an everywhere-present term weight 1, not 0. Do not silently import a different IDF formula when predicting this program.

The weighted vector for A is (1,1.405465) in the ordered coordinates (red,blue). B is (1,0). These coordinates are named by vocabulary terms; there is no separate trained encoder.

Compare directions instead of raw lengths

Repeating every word in A twice would double both coordinates. We do not want that alone to double its similarity. Divide a vector by its Euclidean length, sqrt(sum(weight²)). A's length is approximately 1.724916; its normalized vector is approximately (0.579739,0.814802). B normalizes to (1,0).

Our query blue becomes (0,1) after the same weighting and normalization using the corpus IDF. The dot product multiplies corresponding coordinates and sums them. Query against A gives 0×0.579739+1×0.814802=0.814802; against B it gives 0. This is cosine similarity between the original non-zero vectors. The denominator removes their scale; it does not understand the relationship between words. The geometric definition and its bag-of-words limit are explained in the textbook's dot-product section.

Run:

python3 -B methods.py lexical

The first fields show these IDF values and A's blue-query score. Compare them to your calculation before reading the larger evaluation. The unit test uses this independent two-document calculation, so a mistake in normalization cannot hide behind an expected score copied from the implementation.

Account for every transformation

Index lowercases text, extracts ASCII words/numbers and removes the stop words declared in retrieval.py. It indexes passage text only; titles are displayed but not scored. It filters current/public passages before counting document frequencies. Otherwise, excluded records could change weights even if they never appeared in the result list.

The query uses the same vocabulary. Words absent from it are ignored. A query with no remaining known terms has a zero vector and returns no hits. We retain strictly positive scores, sort descending and resolve exact ties by document ID. Those choices make edge cases inspectable. A tie is not broken by whichever file happened to load first.

The tokenizer has deliberate limits. book and books are different terms. It has no multilingual model, phrase matching or understanding of negation. Keeping the word no in a vector does not mean the scorer understands “no hold.” Word occurrence is a signal for finding a passage whose meaning a reviewer must then inspect.

Read the cutoff as a resource choice

The evaluation section compares k=1 and k=2, the maximum number of returned passages. For the compound question about opening hours and after-hours returns, the first passage explains returns. The second supplies the opening schedule. At k=1, one of two labeled passages is retrieved: recall is 1/2. At k=2, both appear: recall is 1. The first useful passage was already at rank 1 in both cases, so reciprocal rank is unchanged.

Across the eight answerable development cases, mean recall rises from 0.8125 at k=1 to 0.875 at k=2. Mean reciprocal rank is 0.875 in both. The paraphrase still fails. Raising a cutoff adds candidates; it does not create a semantic connection for words that never match. It also puts more irrelevant text into the review path. Whether that tradeoff helps an eventual answer must be measured after context selection and generation, not inferred from recall alone.

Separate ranking score from a decision threshold

The score 0.814802 in the red/blue example is a geometric similarity. It is not an 81.48% chance of a correct answer. A threshold tuned on this tiny artificial corpus cannot justify a product-wide abstention rule. Changes in passage length, term frequency and vocabulary can alter score distributions.

The laptop example makes that danger concrete. Ignoring the unseen word laptop leaves familiar borrowing/duration words, producing a high-ranked book policy. The missing concept is exactly what would make the proposed answer unsupported. A larger score cannot restore it.

A useful baseline report therefore includes per-case IDs, the relevant-set denominator, metadata policy and known failure categories. It also identifies the visible examples as development data. A single aggregate score cannot explain whether a missing result came from a stale source, a lexical mismatch or a budget loss.

Complete the arithmetic and cutoff practice and check the solutions. You are ready for Module 5 when you can calculate a simple score and explain why a different representation might recover a paraphrase without automatically solving answer support.