Overview
OpenLLMetry is a set of extensions built on top of OpenTelemetry that gives you observability over an LLM application. It ships standard OpenTelemetry instrumentations for LLM providers and vector databases, plus a Traceloop SDK that makes setup a one-liner. It is built and maintained by Traceloop under the Apache 2.0 license.
Because it uses OpenTelemetry under the hood, the traces it produces are standard OpenTelemetry data, so you can connect them to observability backends you already run, such as Datadog, Honeycomb, Grafana, New Relic, and many others. If you already have OpenTelemetry instrumented, you can add any of its instrumentations directly without the SDK.
It fits the LLM observability and tracing space: instead of a separate, proprietary dashboard, it routes LLM calls, vector DB queries, and framework activity into the same tracing system you use for the rest of your services. A JS/TS version is available separately as OpenLLMetry-JS.
What it does
- Auto-instruments LLM providers including OpenAI/Azure OpenAI, Anthropic, Bedrock, Cohere, Google Gemini, Mistral, Groq, Ollama, and more
- Instruments vector databases such as Chroma, Pinecone, Qdrant, and Weaviate
- Built on OpenTelemetry, so traces export to existing backends like Datadog, Honeycomb, Grafana, New Relic, SigNoz, and Sentry
- Traceloop SDK gets you tracing with a single Traceloop.init() call
- Works alongside standard OpenTelemetry instrumentations for your DB, API calls, and other services
- Apache 2.0 licensed; a separate OpenLLMetry-JS package covers JavaScript and TypeScript
Getting started
The easiest way to start is the Traceloop SDK, which wraps the OpenTelemetry instrumentations for you.
Install the SDK
Install the Traceloop SDK from PyPI.
pip install traceloop-sdkInitialize tracing
Add one line near the start of your app to begin tracing your LLM calls with OpenLLMetry.
from traceloop.sdk import Traceloop
Traceloop.init()See traces locally
When running locally, disable batch sending so traces appear immediately instead of being buffered.
Traceloop.init(disable_batch=True)Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Trace OpenAI or Anthropic calls in a production LLM app and send the spans to your existing Datadog or Honeycomb setup
- Debug a RAG pipeline by capturing both the LLM calls and the vector DB queries (Chroma, Pinecone, Qdrant, Weaviate) in one trace
- Add LLM observability to a service that already uses OpenTelemetry by dropping in the relevant instrumentations directly
- Inspect traces locally during development with batch sending disabled to iterate on prompt and retrieval behavior
How OpenLLMetry compares
OpenLLMetry alongside other open-source observability & llmops tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Langfuse | ★ 29.4k | A self-hostable platform for tracing LLM and agent calls, managing prompts, and running evaluations to debug and improve AI applications. |
| Opik | ★ 19.7k | An open-source platform from Comet for tracing, evaluating, and monitoring LLM applications, RAG systems, and agent workflows with dashboards and LLM-as-judge metrics. |
| TensorZero | ★ 11.7k | An open-source LLMOps platform that puts a single gateway in front of every major LLM provider and adds observability, evaluation, optimization, and A/B testing. |
| Evidently | ★ 7.6k | A monitoring and evaluation framework for ML and LLM systems that tracks output quality, drift, and test results over time with reports and dashboards. |
| OpenLLMetry | ★ 7.2k | OpenTelemetry-based tracing for LLM apps that plugs into your existing observability stack |
| Helicone | ★ 5.8k | A proxy-based observability platform that logs, monitors, and evaluates LLM API calls by routing requests through its endpoint with one line of code. |
| AgentOps | ★ 5.6k | An SDK for monitoring AI agents that tracks LLM cost, session replays, and performance across frameworks like CrewAI, LangChain, and the OpenAI Agents SDK. |
| Pydantic Logfire | ★ 4.3k | An observability platform from the Pydantic team that records LLM calls, agent runs, and tool invocations with tokens, cost, and latency attached. |