All courses · Course overview · Lesson sequence

LLMOps for Reliable AI Applications

Make defensible release, monitoring and recovery decisions from synthetic cases and operational replay. Supplied outputs, timings and costs are invented teaching fixtures, not real model or production measurements.

Preparation: A working generative AI application and API development skills.

Plan your pace after trying the first lesson.

Start lesson 1

Lesson sequence

Practice, project and sources

Download and work locally

Includes separate solutions, capstone references and authored expected results. Attempt the tasks first. No runtime or installed dependencies are included.

Download project and reference materials. The Python source is downloaded, never executed on the website. Follow the included project README for commands and limitations.

The reading edition notes explain the practice limits and the historical draft notes retained in the source files.

Read the full authored syllabus and its limitations

Guest-reading edition: see edition scope and limitations. The authoring-status notes in this document describe the original 17 October snapshot.

Syllabus: LLMOps for Reliable AI Applications

Status: all ten modules and the capstone are fully authored as a local unpublished course draft. Each module has a written lesson, worked examples, meaningful practice, runnable offline work, and separate explained solutions. Subject-matter/learner validation and platform release remain pending. No learner-tested duration or completion rate is asserted.

The course follows one question: when an AI application changes, what evidence should let its owner continue, pause, or recover? Learners begin with a deterministic evidence replay, then extend their reasoning to real model variability, observation, and operational ownership. The teaching scenario is a fictional workshop support router; all supplied data is synthetic.

Module	Problem and intended capability	Meaningful practice and exit evidence	Authoring status
1. Make a release decision	An average stays the same while one important case gets worse. Separate a task contract, slice metric, critical condition, and regression.	Replay two candidates; identify the exchanged success and failure; write a bounded decision note; implement comparison helpers.	Full lesson, lab/starter/checker, original SVG, and solutions authored.
2. Decide what the dataset represents	A convenient test set misses the people or situations that matter. Define sampling units, labels, provenance, development vs held-out sets, and failure slices.	Design a synthetic case register and a permission-aware plan for collecting real examples; record ambiguities and label disagreements. Deliver a dataset card and a coverage gap table.	Full lesson, runnable audit, and solutions authored.
3. Check the judge as well as the answer	A score can reward the wrong behavior. Distinguish exact rules, rubric-based human review, and optional model grading; test grader agreement and blind spots.	Write anchored criteria, double-label examples, compare disagreements, and test a deliberately misleading correct-looking response. Deliver a grader validation note.	Full lesson, human-rubric/calibration mechanics, and solutions authored. No model judge is invoked.
4. Reproduce an experiment	A result cannot be reconstructed after a prompt or dependency changes. Version the system, data, grader, policy, and execution context.	Compare manifests, compute digests, reconstruct a saved run, and reject a mismatched dataset. Deliver a portable experiment record with retention boundaries.	Full lesson, manifest/replay example, and solutions authored.
5. Compare quality, cost, and delay honestly	A faster candidate appears better from one lucky run. Define measurement boundaries, repeated trials, tail latency, paired comparisons, and uncertainty.	Analyze supplied synthetic repeated observations, state what was timed, and evaluate a constrained tradeoff. Deliver a comparison with uncertainty and a stopping rule.	Full lesson, paired trial lab, and solutions authored.
6. Observe the application in use	Offline success does not show how service behavior changes. Relate task quality review to latency, traffic, errors, saturation, and explicit service objectives.	Design an event schema using synthetic request IDs; separate operational counters from sensitive content; define an alert with an owner and an action. Deliver a monitoring plan.	Full lesson, window replay, original SVG, and solutions authored. No live collector.
7. Change one thing with a recovery path	A passing candidate can still surprise users. Define change scope, staged exposure, stop criteria, rollback prerequisites, and decision authority.	Simulate a sequence of synthetic windows and decide when to hold or recover; rehearse a paper rollback. Deliver a release checklist and decision log.	Full lesson, pure stage/recovery simulator, and solutions authored.
8. Handle an incident	A complaint arrives without enough context to reproduce it. Connect triage, impact, containment, recovery, and follow-up cases.	Reconstruct a fictional incident from minimal sanitized evidence; distinguish observations from hypotheses; add a regression case. Deliver an incident review with owners.	Full lesson, synthetic replay, and solutions authored.
9. Keep evidence useful over time	Inputs, policies, providers, and reviewers change. Plan dataset refresh, drift investigation, deprecation review, and evidence expiry.	Review a fictional change notice and a changed input mix; decide what to re-evaluate and what needs permission. Deliver a maintenance register.	Full lesson, drift/register examples, and solutions authored.
10. Defend the decision	A dashboard says green, but reviewers need to understand remaining risk. Assemble evidence without overstating its scope.	Extend the local gate with independent held-out synthetic cases and a simulated incident; submit a release recommendation, limitations, and recovery plan.	Full lesson, capstone brief/starter/checker, original SVG, and explained reference authored.

Intended learning checks

This local course draft assesses in its exercises whether a learner can explain why a decision follows from a contract and evidence, detect misleading aggregates and missing records, preserve reproducibility, identify blind spots, and name an operational owner. A high score alone is insufficient if a critical condition is violated or provenance is missing.

Module self-check explanations are separate from practice; the capstone has a 20-point self-review rubric and eleven technical checks. Platform grading, course completion tracking, certificate logic, comments, and assessment generation remain outside this content package. This draft has not been imported into Aabha.

Scope boundaries

No model training, embedding implementation, retrieval tutorial, agent orchestration framework, or API backend construction is repeated here. Later instruction may examine saved outputs of such systems as operational artifacts. External inference, actual rollout, real logs, and learner studies would require separately authorized data handling and execution; none is needed to run this opening package.