All courses · Course overview · Lesson sequence
Skip to lesson textAnswers and explanations · Evaluate the whole question-to-answer path
A. All three claims have non-empty citation sets whose IDs exist, so the valid-ID count is3. Two of three manual labels are supported, giving 2/3. Of required aspects {c1,c2}, only c1 is supported, giving coverage 1/2. The presence of a citation does not establish that its text supports the attached claim. To verify:
from methods import review_claims
claims = [{"id": key, "text": "Reviewed separately", "citation_ids": ["source-a"]}
for key in ("c1", "c2", "c3")]
print(review_claims(claims, {"source-a"},
{"c1": True, "c2": False, "c3": True}, {"c1", "c2"}))
The function aggregates the given labels. It does not inspect the placeholder sentences or determine support automatically.
B. Proposed-claim support fraction is null because the denominator is zero. Required coverage is 0/2=0. Calling the empty answer perfectly supported would conceal total failure to cover an answerable question. Appropriate abstention on unsupported questions needs its own cases and denominator.
C. Near-equivalent rewrites can share vocabulary and intent shortcuts, so a test built from the same family may reward adaptation rather than a broader skill. Group equivalent/paraphrase families before splitting; reserve fresh questions that have not guided settings, state the source/version population and label them independently. Review duplicates and ambiguity before measuring. The supplied fixtures are visible synthetic development/teaching examples, not unseen or representative learner questions.
D. Investigate context selection/budget first in the first trace: retrieval found the evidence but none survived. In the second, inspect the answer claim and generation/template contract against the complete source. The unconditional renewal sentence changes the rule despite available evidence. Keeping earlier stages fixed while testing those repairs makes it possible to attribute the effect. Changing the retriever at the same time could hide rather than explain the fault.