Opik

// Overview

Opik is an open-source platform from Comet for building, testing, and monitoring applications built on large language models. It records the calls your app makes to an LLM, stores them as traces, and lets you inspect what went in and what came back, from early prototyping through production.

It is aimed at developers and teams working on RAG chatbots, code assistants, and agent systems who need to see why an app behaves the way it does. Alongside tracing, it offers evaluation tools, including LLM-as-a-judge metrics for things like hallucination detection, moderation, and RAG answer relevance.

As an LLM observability tool, Opik pairs a self-hostable server (run locally with Docker Compose) with client SDKs and many framework integrations, so you can collect traces, score them, and watch feedback scores, trace counts, and token usage over time in a dashboard.

// What it does

• Tracing of LLM calls, conversations, and agent activity, viewable in a UI
• LLM-as-a-judge metrics for hallucination detection, moderation, and RAG assessment (answer relevance, context precision)
• Datasets and experiments to automate evaluation, plus a PyTest integration for CI/CD
• Production monitoring dashboards with online evaluation rules; built to handle high trace volumes
• Many third-party framework integrations, including Google ADK, Autogen, and Flowise AI
• Self-hostable server via Docker Compose, plus the Opik Agent Optimizer and Guardrails

// Getting started

Install the Python SDK, optionally run the server locally, then wrap your LLM function with a decorator to start logging traces.

pip install opik

git clone https://github.com/comet-ml/opik.git
cd opik
./opik.sh

opik configure

import opik

opik.configure(use_local=True)

@opik.track
def my_llm_function(user_question: str) -> str:
    # Your LLM code here
    return "Hello"

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Debugging a RAG chatbot by inspecting each LLM call and its retrieved context in traces
• Scoring outputs for hallucinations or moderation issues using LLM-as-a-judge metrics
• Adding evaluation checks to a CI/CD pipeline with the PyTest integration
• Monitoring trace counts, token usage, and feedback scores for an agent app in production

// How Opik compares

Opik alongside other open-source observability & llmops tools AI/TLDR tracks, ranked by GitHub stars.