Module 10 Knowledge Check
5 questions. Pass mark 4 out of 5. Answer every question before checking the answer key below, then retry after reading the feedback.
1. A trace's core value is showing…
- A. Total cost
- B. The actual inputs and outputs at each step, including the resolved prompt
- C. The model version
- D. The user
2. For a bad RAG answer, the trace should first tell you…
- A. The latency
- B. Which documents were retrieved and whether they contained the answer
- C. The token count
- D. The temperature
3. Production traces typically contain…
- A. Only metrics
- B. User content and retrieved documents, so they need access control and retention rules
- C. Nothing sensitive
- D. Only errors
4. Comparing traces across two prompt versions supports…
- A. Guessing
- B. Evidence-based iteration - you can see what actually changed in behaviour
- C. Faster inference
- D. Lower cost
5. Tracing should be enabled…
- A. Only in development
- B. In production too, with appropriate sampling and data controls
- C. Never in production
- D. Only on errors
---
Answer Key and Explanations
Check these only after attempting every question.
1. B - The actual inputs and outputs at each step, including the resolved prompt
Most 'the model is wrong' bugs turn out to be 'the prompt or context was not what I assumed'.
2. B - Which documents were retrieved and whether they contained the answer
This one check separates a retrieval bug from a generation bug immediately.
3. B - User content and retrieved documents, so they need access control and retention rules
Observability data is a second copy of your users' and corpus data.
4. B - Evidence-based iteration - you can see what actually changed in behaviour
Without side-by-side evidence, prompt tuning is superstition.
5. B - In production too, with appropriate sampling and data controls
Real failures happen on real traffic; you need the evidence when they do.
Practical Check
Apply this module to your own work: complete the module activity for *Tracing and Debugging with LangSmith*, then write one sentence naming what your result shows and one naming what it does not.
Strong Answer Pattern
A strong answer names the task, the evidence used, the check performed, and the remaining limitation. It avoids "proved", "guaranteed", or "always" unless the evidence genuinely supports it.