# MLflow

> Open-source platform to trace, evaluate, and ship LLM and agent apps

- Category: Training & Fine-Tuning / Experiment Tracking
- Language: Python
- Repository: https://github.com/mlflow/mlflow
- Website: https://mlflow.org
- Canonical page: https://ai-tldr.dev/tools/mlflow/

## In short

MLflow is an open-source platform for building, debugging, evaluating, and deploying LLM applications and AI agents, alongside its long-standing tools for machine-learning experiment tracking. It captures traces of how your app runs, scores output quality, and helps you monitor cost and behavior once an app is in production. It sits in the Experiment Tracking category of the Training & Fine-Tuning tools and is open-source, written in Python.

## Overview

MLflow is an open-source platform for building, debugging, evaluating, and deploying LLM applications and AI agents, alongside its long-standing tools for machine-learning experiment tracking. It captures traces of how your app runs, scores output quality, and helps you monitor cost and behavior once an app is in production.

It is aimed at teams who want to ship AI applications with confidence and catch quality regressions before users do. It works with Python, TypeScript/JavaScript, Java, and other languages, and integrates with OpenTelemetry and MCP, so it fits into existing stacks rather than replacing them.

As an evaluation framework, MLflow lets you run systematic evaluations, track metrics over time, and compare model and agent outputs. You can pick from over 50 built-in metrics and LLM judges or define your own.

## What it does

- Tracing and observability for LLM apps and agents, built on OpenTelemetry and working with any provider or framework
- Evaluation with 50+ built-in metrics and LLM judges, plus support for custom metrics, to catch regressions before production
- Prompt registry and prompt optimization with version control and full lineage tracking
- AI Gateway: a unified, OpenAI-compatible API for routing requests, managing rate limits, fallbacks, and costs across providers
- Experiment tracking and model registry for the full ML lifecycle, covering parameters, metrics, and model versions
- Multi-language support (Python, TypeScript/JavaScript, Java) with native OpenTelemetry and MCP integration

## Getting started

Start an MLflow server, turn on autologging, then run your code and explore traces in the MLflow UI.

1. Start the MLflow server

   Launch a local MLflow tracking server. The UI is served at http://localhost:5000.

```bash
uvx mlflow server
```

2. Enable logging

   Point MLflow at the running server and enable autologging for your LLM client (OpenAI shown here).

```python
import mlflow

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.openai.autolog()
```

3. Run your code

   Call your LLM as usual; MLflow captures the trace automatically. Then explore traces and metrics in the UI at http://localhost:5000.

```python
from openai import OpenAI

client = OpenAI()
client.responses.create(
    model="gpt-5.4-mini",
    input="Hello!",
)
```

## When to use it

- Tracing an LLM or agent app end-to-end to debug why a response went wrong
- Running systematic evaluations and tracking quality metrics over time to catch regressions before release
- Versioning, testing, and optimizing prompts with full lineage tracking
- Routing requests across multiple LLM providers through one OpenAI-compatible gateway to manage cost and access
