What Reliability Means for LLM Applications
By the end of this module you can define reliability for a system whose output is not deterministic, and say what 'working' means in terms you could test tomorrow.
LLMOps for Reliable AI Applications
A reliability report for an LLM app with eval cases, tests, traces, cost/latency notes, risk register, and release recommendation.
Quality Dimensions: Correctness, Grounding, Format, Safety, Latency, and Cost
By the end of this module you can break quality into dimensions you can measure separately - correctness, grounding, format, safety, latency, cost - instead of arguing about whether it feels good.
Eval Datasets and Test-Case Design
By the end of this module you can build an evaluation dataset that reflects real usage, including the awkward cases people actually send rather than the ones you hoped for.
Unit Tests for Prompts, Schemas, and Tools
By the end of this module you can unit-test prompts, schemas and tools, so a change that breaks output format fails in CI rather than in production.
RAG Evaluation: Retrieval, Citation, and Answer Quality
By the end of this module you can evaluate a RAG system's retrieval and its answers separately, and tell which of the two is responsible for a bad response.
Agent Evaluation: Tool Trajectory, Approvals, and Stopping
By the end of this module you can evaluate an agent on its trajectory - which tools it called, whether it stopped, whether it asked for approval - not only on its final answer.
Observability: Traces, Metrics, and Logs
By the end of this module you can instrument an LLM application with traces, metrics and logs that answer the question 'what happened on that request?' after the fact.
Cost, Latency, Caching, and Rate Limits
By the end of this module you can measure and control cost and latency, and use caching and rate limits without silently serving a stale answer.
Red-Team and Misuse Cases
By the end of this module you can red-team your own application - prompt injection, misuse, data exfiltration - and record what you found rather than what you hoped.
Release Gates and Regression Testing
By the end of this module you can define release gates that a change must pass, and run regression tests that catch the quality drop a model update introduced.
Incident Review and Continuous Improvement
By the end of this module you can run an incident review for an AI failure and turn it into a test case, so the same failure cannot ship twice.
Capstone: Reliability Report for an LLM App
In this capstone you will produce a reliability report for an LLM application: its evaluation results, its cost and latency profile, its known failure modes, and the gates it must pass to ship.
First-release boundary
This is a text-first course shell. Leonardo/image creation, image QC, dependency-heavy runtime QA, external review, and learner pilot evidence remain separate later quality steps.