Production & LLMOps · TRACK 02/05
Observability & Monitoring
Tracing every call, scoring live traffic, catching drift early.
// THE TRACK
01 · START HERELLM ObservabilityLearn what to record on every LLM call and why ordinary application monitoring misses what matters in AI apps.BEGINNERLLM TracingUnderstand how traces and spans turn a multi-step AI pipeline into a debuggable tree you can replay step by step.INTERMEDIATELangfuse vs LangSmith vs HeliconeCompare the three most popular LLM observability tools and know which one fits your stack, budget, and hosting needs.INTERMEDIATEProduction MetricsKnow which latency, cost, and quality metrics belong on your LLM dashboard and which vanity numbers to ignore.INTERMEDIATECollecting User FeedbackLearn to capture explicit and implicit user feedback signals and turn them into eval data that improves your app.INTERMEDIATESLOs & Error BudgetsYou'll understand how to set measurable service-level objectives for an LLM app, including the tricky case of a quality SLO on nondeterministic output.INTERMEDIATETrace SamplingYou'll understand how to sample LLM traces so you keep the ones that matter — errors, slow calls, low-rated answers — without storing every request at full cost.INTERMEDIATECost AttributionYou'll understand how to instrument requests with tags so a single aggregate bill becomes a clear map of which feature, customer, or team is spending the tokens.INTERMEDIATELangSmith ObservabilityAfter reading, you'll understand what LangSmith is, how its tracing and evaluation features work, and when a team reaches for it to monitor LLM apps.INTERMEDIATELangfuseAfter reading, you'll understand what Langfuse is, how it traces and evaluates LLM apps, and why teams pick an open-source, self-hostable observability stack.INTERMEDIATEArize PhoenixAfter reading, you'll understand what Arize Phoenix is, how it lets you trace and evaluate LLM apps locally, and where it fits next to hosted observability tools.INTERMEDIATEComet OpikAfter reading, you'll understand what Comet Opik is, how its tracing, automated evals, and dashboards work, and where it fits among open-source observability tools.INTERMEDIATEOpenLLMetryAfter reading, you'll understand what OpenLLMetry is, how it adds OpenTelemetry instrumentation to LLM apps, and why a vendor-neutral standard matters for observability.INTERMEDIATE