Module 12 Knowledge Check
5 questions. Pass mark 4 out of 5. Answer every question before checking the answer key below, then retry after reading the feedback.
1. A reliability report must open with…
- A. The architecture
- B. What the application is for and what a wrong answer would cost
- C. The model choice
- D. The cost
2. Reported metrics are only meaningful alongside…
- A. The model name
- B. The eval set they were measured on and its composition
- C. The date
- D. The cost
3. The report should state release gates as…
- A. Aspirations
- B. Specific thresholds that block a release when unmet
- C. Averages
- D. Targets for next year
4. Known weaknesses should be…
- A. Omitted
- B. Listed explicitly with their mitigations or accepted-risk rationale
- C. Fixed silently
- D. Marked resolved
5. The report should end with…
- A. A conclusion that it is reliable
- B. What is monitored in production and what would trigger re-evaluation
- C. A cost table
- D. A roadmap
---
Answer Key and Explanations
Check these only after attempting every question.
1. B - What the application is for and what a wrong answer would cost
Consequence determines which thresholds and controls are appropriate; everything else follows.
2. B - The eval set they were measured on and its composition
A score without its test set is unfalsifiable.
3. B - Specific thresholds that block a release when unmet
A gate that does not block anything is documentation, not a gate.
4. B - Listed explicitly with their mitigations or accepted-risk rationale
An honest weakness list is the strongest signal that the assessment is trustworthy.
5. B - What is monitored in production and what would trigger re-evaluation
Reliability is an ongoing property; the report should describe how it stays true.
Practical Check
Apply this module to your own work: complete the module activity for *Capstone: Reliability Report for an LLM App*, then write one sentence naming what your result shows and one naming what it does not.
Strong Answer Pattern
A strong answer names the task, the evidence used, the check performed, and the remaining limitation. It avoids "proved", "guaranteed", or "always" unless the evidence genuinely supports it.