█

AI/TLDR

Buy me a coffee
Buy me a coffee

MLflow

Open-source platform to trace, evaluate, and ship LLM and agent apps

Experiment TrackingOpen source
Language
Python
$uvx mlflow server
IN SHORT

What is MLflow?

MLflow is an open-source platform for building, debugging, evaluating, and deploying LLM applications and AI agents, alongside its long-standing tools for machine-learning experiment tracking. It captures traces of how your app runs, scores output quality, and helps you monitor cost and behavior once an app is in production. It sits in the Experiment Tracking category of the Training & Fine-Tuning tools and is open-source, written in Python.

Overview

MLflow is an open-source platform for building, debugging, evaluating, and deploying LLM applications and AI agents, alongside its long-standing tools for machine-learning experiment tracking. It captures traces of how your app runs, scores output quality, and helps you monitor cost and behavior once an app is in production.

It is aimed at teams who want to ship AI applications with confidence and catch quality regressions before users do. It works with Python, TypeScript/JavaScript, Java, and other languages, and integrates with OpenTelemetry and MCP, so it fits into existing stacks rather than replacing them.

As an evaluation framework, MLflow lets you run systematic evaluations, track metrics over time, and compare model and agent outputs. You can pick from over 50 built-in metrics and LLM judges or define your own.

What it does

  • Tracing and observability for LLM apps and agents, built on OpenTelemetry and working with any provider or framework
  • Evaluation with 50+ built-in metrics and LLM judges, plus support for custom metrics, to catch regressions before production
  • Prompt registry and prompt optimization with version control and full lineage tracking
  • AI Gateway: a unified, OpenAI-compatible API for routing requests, managing rate limits, fallbacks, and costs across providers
  • Experiment tracking and model registry for the full ML lifecycle, covering parameters, metrics, and model versions
  • Multi-language support (Python, TypeScript/JavaScript, Java) with native OpenTelemetry and MCP integration

Getting started

Start an MLflow server, turn on autologging, then run your code and explore traces in the MLflow UI.

Start the MLflow server

Launch a local MLflow tracking server. The UI is served at http://localhost:5000.

bashbash
uvx mlflow server

Enable logging

Point MLflow at the running server and enable autologging for your LLM client (OpenAI shown here).

pythonpython
import mlflow

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.openai.autolog()

Run your code

Call your LLM as usual; MLflow captures the trace automatically. Then explore traces and metrics in the UI at http://localhost:5000.

pythonpython
from openai import OpenAI

client = OpenAI()
client.responses.create(
    model="gpt-5.4-mini",
    input="Hello!",
)

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

When to use it

  • Tracing an LLM or agent app end-to-end to debug why a response went wrong
  • Running systematic evaluations and tracking quality metrics over time to catch regressions before release
  • Versioning, testing, and optimizing prompts with full lineage tracking
  • Routing requests across multiple LLM providers through one OpenAI-compatible gateway to manage cost and access

How MLflow compares

MLflow alongside other open-source experiment tracking tools AI/TLDR tracks, ranked by GitHub stars.

ToolStarsWhat it does
MLflow★ 28.9kOpen-source platform to trace, evaluate, and ship LLM and agent apps
DVC★ 16kA Git-like version control tool for datasets, models, and experiments that makes ML projects reproducible.
Weights & Biases (wandb)★ 11.3kAn open-source client library for logging experiments, running hyperparameter sweeps, and versioning models and datasets.
ClearML★ 6.9kAn open-source MLOps suite whose experiment manager auto-logs runs, metrics, and artifacts and ties them into data management and orchestration.
Aim★ 6.3kA self-hosted, open-source experiment tracker that logs and compares thousands of training runs through a fast web UI.
Sacred★ 4.4kA Python tool for configuring, organizing, logging, and reproducing computational research experiments.
Polyaxon★ 3.7kAn open-source platform for tracking, scheduling, and managing large-scale deep learning experiments on Kubernetes.
Trackio★ 1.8kA lightweight open-source experiment tracker from Hugging Face that stores runs locally in SQLite as a drop-in alternative to wandb.