What is MLflow?
MLflow is an open-source platform for building, debugging, evaluating, and deploying LLM applications and AI agents, alongside its long-standing tools for machine-learning experiment tracking. It captures traces of how your app runs, scores output quality, and helps you monitor cost and behavior once an app is in production. It sits in the Experiment Tracking category of the Training & Fine-Tuning tools and is open-source, written in Python.
Overview
MLflow is an open-source platform for building, debugging, evaluating, and deploying LLM applications and AI agents, alongside its long-standing tools for machine-learning experiment tracking. It captures traces of how your app runs, scores output quality, and helps you monitor cost and behavior once an app is in production.
It is aimed at teams who want to ship AI applications with confidence and catch quality regressions before users do. It works with Python, TypeScript/JavaScript, Java, and other languages, and integrates with OpenTelemetry and MCP, so it fits into existing stacks rather than replacing them.
As an evaluation framework, MLflow lets you run systematic evaluations, track metrics over time, and compare model and agent outputs. You can pick from over 50 built-in metrics and LLM judges or define your own.
What it does
- Tracing and observability for LLM apps and agents, built on OpenTelemetry and working with any provider or framework
- Evaluation with 50+ built-in metrics and LLM judges, plus support for custom metrics, to catch regressions before production
- Prompt registry and prompt optimization with version control and full lineage tracking
- AI Gateway: a unified, OpenAI-compatible API for routing requests, managing rate limits, fallbacks, and costs across providers
- Experiment tracking and model registry for the full ML lifecycle, covering parameters, metrics, and model versions
- Multi-language support (Python, TypeScript/JavaScript, Java) with native OpenTelemetry and MCP integration
Getting started
Start an MLflow server, turn on autologging, then run your code and explore traces in the MLflow UI.
Start the MLflow server
Launch a local MLflow tracking server. The UI is served at http://localhost:5000.
uvx mlflow serverEnable logging
Point MLflow at the running server and enable autologging for your LLM client (OpenAI shown here).
import mlflow
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.openai.autolog()Run your code
Call your LLM as usual; MLflow captures the trace automatically. Then explore traces and metrics in the UI at http://localhost:5000.
from openai import OpenAI
client = OpenAI()
client.responses.create(
model="gpt-5.4-mini",
input="Hello!",
)Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Tracing an LLM or agent app end-to-end to debug why a response went wrong
- Running systematic evaluations and tracking quality metrics over time to catch regressions before release
- Versioning, testing, and optimizing prompts with full lineage tracking
- Routing requests across multiple LLM providers through one OpenAI-compatible gateway to manage cost and access
How MLflow compares
MLflow alongside other open-source experiment tracking tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| MLflow | ★ 28.9k | Open-source platform to trace, evaluate, and ship LLM and agent apps |
| DVC | ★ 16k | A Git-like version control tool for datasets, models, and experiments that makes ML projects reproducible. |
| Weights & Biases (wandb) | ★ 11.3k | An open-source client library for logging experiments, running hyperparameter sweeps, and versioning models and datasets. |
| ClearML | ★ 6.9k | An open-source MLOps suite whose experiment manager auto-logs runs, metrics, and artifacts and ties them into data management and orchestration. |
| Aim | ★ 6.3k | A self-hosted, open-source experiment tracker that logs and compares thousands of training runs through a fast web UI. |
| Sacred | ★ 4.4k | A Python tool for configuring, organizing, logging, and reproducing computational research experiments. |
| Polyaxon | ★ 3.7k | An open-source platform for tracking, scheduling, and managing large-scale deep learning experiments on Kubernetes. |
| Trackio | ★ 1.8k | A lightweight open-source experiment tracker from Hugging Face that stores runs locally in SQLite as a drop-in alternative to wandb. |

