# TensorZero

> Open-source LLMOps stack: one gateway plus observability, evals, and optimization

- Category: Evaluation & Production Ops / Observability & LLMOps
- Language: Rust
- Licence: Apache-2.0
- Repository: https://github.com/tensorzero/tensorzero
- Website: https://tensorzero.com
- Canonical page: https://ai-tldr.dev/tools/tensorzero/

## In short

TensorZero is an open-source LLMOps platform that brings several pieces of the LLM workflow together. It pairs a fast gateway that reaches every major model provider through one unified API with observability, evaluation, optimization, and built-in experimentation. It sits in the Observability & LLMOps category of the Evaluation & Production Ops tools and is open-source, written in Rust and licensed Apache-2.0.

## Overview

TensorZero is an open-source LLMOps platform that brings several pieces of the LLM workflow together. It pairs a fast gateway that reaches every major model provider through one unified API with observability, evaluation, optimization, and built-in experimentation. You can adopt only the parts you need and add the rest over time.

The gateway is written in Rust for very low overhead, and it works with the OpenAI SDK, OpenTelemetry, and providers like Anthropic, OpenAI, Google, Mistral, and many more. It stores your inferences and feedback in your own database, so production data and human feedback can feed back into better prompts and models.

## What it does

- Unified gateway that calls any major LLM provider (API or self-hosted) through one OpenAI-compatible API, with sub-millisecond p99 latency overhead
- High availability features built in: routing, retries, fallbacks, load balancing, granular timeouts, plus tool use, structured JSON outputs, batching, embeddings, and multimodal inputs
- Observability that stores inferences and feedback in your own database, viewable in the open-source UI or programmatically, with OpenTelemetry (OTLP) and Prometheus export
- Evaluation of individual inferences with heuristics or LLM judges, and end-to-end workflow evaluations, runnable from the UI or the CLI
- Optimization of prompts, models, and inference strategies using production metrics and human feedback, including supervised fine-tuning and automated prompt engineering
- Built-in experimentation with adaptive A/B testing so you can compare prompts and models and ship with confidence

## Getting started

TensorZero ships as a single Docker container and speaks the OpenAI API, so you can point any OpenAI-compatible client at it and start routing inference through the gateway.

1. Deploy the gateway

   Run the TensorZero Gateway, which is a single Docker container. See the deployment guide for the full Docker Compose setup; the quickstart walks through it in about five minutes.

2. Point your OpenAI client at the gateway

   Update the base_url in your OpenAI-compatible client to the local gateway and leave the API key unused, since the gateway holds the real provider keys.

```python
from openai import OpenAI

# Point the client to the TensorZero Gateway
client = OpenAI(base_url="http://localhost:3000/openai/v1", api_key="not-used")
```

3. Run inference against any provider

   Pass a TensorZero model name to call any provider through the unified API. You can swap providers by changing the model string.

```python
response = client.chat.completions.create(
    model="tensorzero::model_name::anthropic::claude-sonnet-4-6",
    messages=[
        {"role": "user", "content": "Share a fun fact about TensorZero."}
    ],
)
```

4. Run evaluations from the CLI

   Once you have datasets and variants set up, run evaluations to compare prompts, models, and inference strategies using heuristics or LLM judges.

```bash
docker compose run --rm evaluations \
  --evaluation-name extract_data \
  --dataset-name hard_test_cases \
  --variant-name gpt_4o \
  --concurrency 5
```

## When to use it

- Put a single, fast gateway in front of many LLM providers so apps can switch models and add retries, fallbacks, and load balancing without code changes
- Capture every inference and piece of feedback in your own database to debug individual calls and watch aggregate metrics across models and prompts over time
- Turn production metrics and human feedback into better prompts and fine-tuned models through a data and learning flywheel
- Run evaluations and adaptive A/B tests to compare prompts, models, and inference strategies before shipping changes to production