All courses · Course overview · Lesson sequence

Compare semantic and hybrid retrieval

The lexical baseline misses “penalty for tardiness” because the handbook says “overdue charge.” A human connects the meanings. Could a representation learned from examples connect them too? That is a reasonable next hypothesis. It is not a reason to announce that a semantic system has already solved our problem.

In this module, we will separate three decisions: representing meaning, combining candidate lists and inspecting candidates more closely. The runnable semantic part uses manually supplied vectors. Its job is to make the geometry and combination rules visible, not to provide evidence about a neural model.

A learned representation is different from a vocabulary count

Our lexical vector has one coordinate per term. A dense retriever instead uses learned encoders to map questions and passages into numerical vectors, then ranks them by a chosen similarity. Related wording may occupy nearby regions because of the training procedure. The Dense Passage Retrieval paper describes a dual-encoder approach for question/passage retrieval. Its benchmark findings belong to its models and datasets; they are not results for our handbook.

Consider three manually chosen coordinates: a borrowing-topic axis, a charge-topic axis and an event-topic axis. Assign charges-v1 the vector (0,1,0) and borrowing-v1 (1,0,0). For the paraphrase, manually choose the query vector (0,1,0). Cosine now ranks the charge passage first even though no question word matches it.

We supplied that relationship ourselves. The program contains no text-to-vector function, training data, encoder weights or model download. Calling these “real semantic embeddings” would conceal that the correct topic was put into the input by the author. The coordinate names are also a teaching convenience; actual learned dense dimensions generally do not correspond to our three named topics.

Run:

python3 -B methods.py hybrid

lexical_ids is empty for the paraphrase. fixed_vector_ranking gives charges-v1 a cosine of 1 because the two manually supplied vectors are identical. The reported kind says manually_supplied_vector_simulation_no_encoder. That label is part of the result, so a screenshot or copied number cannot quietly become a model benchmark.

Cosine similarity compares directions and needs two nonzero vectors. A zero vector has no direction, so its mathematical cosine is undefined. Our cosine helper deliberately returns 0.0 when either vector contains only zeros. The fixed_vector_ranking helper then leaves out scores of zero or below and orders tied positive scores by document ID. These are choices in this teaching program; they do not show that it understands meaning.

Nearby meaning still does not establish a claim

Suppose two passages describe a fourteen-day book loan and a fourteen-day laptop loan. Both might lie near a broad borrowing-topic direction. Similarity is useful for finding candidates, but the requested object and conditions still matter. An embedding can encode distinctions poorly, or a scoring rule can favor a passage that shares a topic but contradicts the requested action. We need relevance labels and source review even when the representation is learned.

For a real comparison, we would freeze the same source versions, use the encoder's documented preprocessing, record model and tokenizer identifiers, state the similarity metric and label questions before tuning. We would compare lexical, dense and combined lists on the same questions. This course performs none of the neural steps; the experiment design is what you should be able to specify.

Fuse ranks when scores have incompatible meanings

A cosine from one retriever and a lexical score from another may have different distributions. Adding raw values assumes comparability we have not established. Reciprocal rank fusion, RRF, instead combines positions. Each occurrence of a document at rank r contributes 1/(c+r) to its fused score. Missing documents contribute zero. Our default constant is c=60; ranks start at 1. This follows the rule introduced by Cormack, Clarke and Büttcher, not a universal claim that fusion always improves results.

The demo includes independent arithmetic lists. One list is borrowing-v1, renewals-v1; another is charges-v1, borrowing-v1. Borrowing appears at ranks 1 and 2, so its total is 1/61+1/62≈0.032522. Charges appears once at rank 1, contributing about 0.016393. Renewals appears once at rank 2, contributing about 0.016129. Borrowing comes first because it has support from both lists.

Agreement can also repeat the same mistake. Two retrievers trained or designed similarly may favor the same wrong passage. Fusion cannot invent an unreturned source. If a source is absent from every candidate list, it has no fused score. Nor does the RRF number represent factual confidence.

Illustrative rank lists and one inspectable fusion rule — Two manually authored arithmetic lists contribute rank positions to reciprocal rank fusion. They are not the executed lexical and fixed-vector results for one question. Candidate review follows fusion, but neither fusion nor review can recover a document absent from both lists. No neural encoder runs.

Spend closer inspection on a bounded candidate set

Reranking applies another relevance decision to candidates already retrieved. A joint question–passage scorer, often called a cross-encoder, can examine their interaction rather than comparing independently encoded vectors. Passage Re-ranking with BERT is one primary example: it scores each question–passage input with a binary relevance classifier. This does not mean pairwise learning-to-rank loss. No BERT model or neural reranker runs in this package.

Our runnable alternative is an explicit exact-term filter. We pass the laptop candidates and manually require the term laptop. Both book-policy candidates are rejected because neither contains that token. This demonstrates a candidate-stage constraint, not automatic recognition of what a question means. Someone had to decide that laptop was required.

Try mentally substituting the required term book for a passage containing books. The current tokenizer does no stemming, so that exact constraint rejects it. A broader synonym rule could recover some good cases and admit bad ones. The proper response is to measure those changes, not to describe a simple filter as a semantic safety layer.

Choose the next experiment from the failure

For an exact policy name, lexical matching may be enough. For a paraphrase, a learned representation is worth evaluating when an approved local or external model is available. For incompatible score scales, rank fusion gives an inspectable combination baseline. For a missing required object, closer candidate inspection may help if that object can be recognized reliably. These are hypotheses matched to error types, not a ranking of methods from primitive to universally better.

Complete the vector/fusion practice and explained solutions. Your conclusion should distinguish actual lexical execution, manual vector simulation and a future model experiment. In Module 6, we will see why finding the right candidates still does not guarantee that the right evidence reaches an answer stage.

Read separate answers and explanations