Evaluate the whole question-to-answer path

An answer contains two citations, and both IDs exist in the packet. One sentence says renewals are always allowed; another offers a fourteen-day laptop loan. Both citations point to real synthetic passages. Both claims are unsupported. A citation-existence check would pass the answer while a source-support review would reject it.

Evaluation needs separate questions for separate stages. We have already measured whether retrieval finds labeled evidence and whether a budget preserves it. Now we will measure reviewed claims without pretending code can infer their meaning from an ID.

Start with a per-question record

For each question, record its intended meaning, whether the supplied collection supports an answer, all required evidence IDs and the aspects a complete answer should cover. Then record retrieved IDs, selected packet IDs and proposed claims. A reviewer labels each claim as supported or unsupported by the eligible evidence. If the question or source is ambiguous, record that ambiguity and resolve it before using a binary score.

In the renewal case, a required aspect is the hold condition attached to the one-seven-day-renewal rule. In the compound hours/returns case, both the schedule and return location matter. Relevance and completeness labels are decisions made by people for an evaluation purpose; they do not materialize from a similarity score.

The FActScore research motivates examining factual support at a finer claim level rather than treating a long answer as one all-or-nothing statement. Its method and datasets are not reproduced here. Our code simply aggregates explicit manual labels.

Run a deliberately bad answer through two checks

Run:

python3 -B methods.py evaluation

The first claim, with ID renewal-condition, says “Renewals are always allowed” and cites renewals-v1. The second, invented-laptop, offers laptop borrowing and cites borrowing-v1. The code receives manual support labels False for both. It does not read the sentences and decide those labels.

The result counts two valid citation-ID sets because both named sources exist. It counts zero manually supported claims, giving a supported-claim fraction of 0/2=0. Required supported-claim coverage is also 0: the required renewal-condition claim is present as an ID, but its manual label is unsupported. This prevents the presence of a required ID from automatically earning completeness.

Claim IDs themselves do not establish semantics. A reviewer must ensure an ID refers to the same intended aspect across the candidate and required set. If someone assigns the wrong label, the arithmetic faithfully summarizes a wrong judgment. review_claims rejects missing labels and duplicate claim IDs, but it cannot adjudicate meaning.

Keep each denominator visible

The supported-claim fraction asks how many proposed claims are supported among proposed claims. A cautious answer could achieve a perfect fraction by stating one safe fact while omitting most of the question. Required supported-claim coverage asks how many required aspects are both present and supported. Reporting both exposes that tradeoff.

For an empty answer, the proposed-claim denominator is zero, so our function returns null rather than a fabricated perfect fraction. If the answerable question required two aspects, empty output would still have required coverage 0. If no aspects are defined, coverage is null; that is missing applicability, not success.

Retrieval recall has its own denominator: labeled relevant passages for answerable questions. Unsupported questions have an empty relevant set and belong in a separate appropriate-abstention analysis. We should count both false evidence returns on unsupported questions and failures to answer when sufficient evidence is available. A higher refusal rate is not automatically better.

The course's quote-only outputs are not generated answers and should not receive a fabricated factual-generation score. The template's one reviewed claim is also not evidence of general neural generation quality. Name the actual mechanism in every report.

Separate development from evaluation evidence

All supplied questions were visible during authoring. They are development/teaching fixtures, including their known failures. Repeatedly changing a setting after seeing these cases adapts the method to them. Calling the same examples a held-out benchmark would conceal that feedback loop.

Before a later generalization claim, reserve new questions whose results have not influenced the settings. Keep near-duplicate/paraphrase families together rather than putting a rewrite of the same question in both development and test sets. Choose source-version and question populations that match the intended use. A future policy-change scenario may need a different split from a fixed-handbook search exercise.

For a small handmade set, publish counts and individual errors. Do not add impressive-looking uncertainty estimates that ignore how the questions were selected. Four easy examples cannot stand in for a representative population, regardless of whether an average is 1. A measured learner pilot would require actual participants and an approved study process; these local technical checks are not one.

Automatic judges require their own validation

Research systems such as ARES and Ragas study evaluation of RAG components using automated methods. That does not make an arbitrary model judge an objective truth oracle. An evaluation design must state what the judge sees, what labels it predicts and how its decisions compare with reviewed examples. No automated model judge runs in this course.

If two human reviewers disagree, inspect the underlying claim and source. One may have assumed “no laptop policy” means “laptops are forbidden”; the other may correctly distinguish absence of evidence from a prohibition. Resolve or retain the uncertainty before reporting an average. Disagreement can reveal a defective question or rubric rather than a defective reviewer.

Choose a repair from the trace

Low retrieval recall suggests investigating corpus coverage, metadata, boundaries or ranking. Good retrieval with low packet coverage suggests context selection. A complete packet with unsupported claims points to the answer stage or review contract. High support but low completeness suggests omissions. This stage diagnosis helps keep experiments causal: change one decision, rerun the affected cases and explain what the evidence supports.

Complete the evaluation practice and explained solutions. Module 9 will use this structure to investigate concrete failures and state a bounded readiness conclusion.