Production & LLMOps · TRACK 05/05
Testing & Deployment
Shipping nondeterministic software without closing your eyes.
// THE TRACK
01 · START HERETesting LLM AppsLearn testing strategies that work when the same input can produce different output — assertions, evals, and snapshots.BEGINNERA/B Testing Prompts & ModelsLearn how to run statistically valid A/B tests on LLM prompts and models in production — from traffic splitting and sample sizing to reading results with confidence.INTERMEDIATESafe Model RolloutsLearn to ship model upgrades with canaries, feature flags, and rollback paths so a bad version never reaches everyone.INTERMEDIATEShadow ModeUnderstand how to run a candidate model silently on live traffic and compare results before any user sees them.INTERMEDIATEOffline vs Online EvalYou'll understand the difference between testing an LLM on a fixed dataset before release and measuring its quality on live traffic, and why you need both.BEGINNEREval-Driven DevelopmentYou'll understand how to build an eval set first so every prompt or model change is measured against real cases instead of judged by gut feeling.INTERMEDIATELoad TestingYou'll understand how to stress-test an LLM app realistically — measuring token throughput and concurrency limits, not just request counts — before launch.INTERMEDIATEBlue-Green vs CanaryYou'll understand the two main rollout strategies — blue-green and canary — and how to choose between them for shipping a new prompt or model.INTERMEDIATE