# lmms-eval

> Reproducible evaluation suite for large multimodal models across image, video, and audio

- Category: Evaluation & Production Ops / Benchmark Harnesses
- Language: Python
- Repository: https://github.com/EvolvingLMMs-Lab/lmms-eval
- Website: https://lmms-lab.github.io
- Canonical page: https://ai-tldr.dev/tools/lmms-eval/

## In short

lmms-eval is an evaluation toolkit for large multimodal models (LMMs) - models that work with images, video, and audio, not just text. It bundles 100+ benchmark tasks and 30+ model backends behind a single command-line interface, so you can score a model on many datasets without wiring up each one yourself. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python.

## Overview

lmms-eval is an evaluation toolkit for large multimodal models (LMMs) - models that work with images, video, and audio, not just text. It bundles 100+ benchmark tasks and 30+ model backends behind a single command-line interface, so you can score a model on many datasets without wiring up each one yourself.

It is built for model teams and researchers who need eval numbers they can act on. The project focuses on three goals: reproducible runs that give the same numbers every time, efficient evaluation that keeps GPUs busy at scale, and trustworthy results that go beyond a single accuracy score with confidence intervals and paired comparisons.

As a benchmark harness, it sits in the evaluation and testing stage of a model's lifecycle. You point it at a pretrained model and a list of tasks, and it handles dataset loading, generation, post-processing, and metric reporting in one pipeline.

## What it does

- 100+ built-in benchmark tasks spanning image, video, and audio evaluation
- 30+ model backends, including Qwen2.5-VL and other vision-language models
- Single unified CLI: choose a model and a task list, get consistent metrics
- Reproducible pipeline designed to return the same numbers on repeat runs
- Statistical reporting beyond raw accuracy: confidence intervals and paired comparisons
- Efficiency features like async serving, adaptive batching, and faster video I/O via TorchCodec

## Getting started

Clone the repo, install with uv, and run a small evaluation to confirm your environment works.

1. Install from source with uv

   Clone the repository and install it (with all extras) using uv, the recommended package manager for a consistent environment.

```bash
git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git
cd lmms-eval && uv pip install -e ".[all]"
```

2. Run your first evaluation

   Evaluate Qwen2.5-VL on the MME benchmark with a small sample limit. If it prints metrics, your setup is ready.

```bash
python -m lmms_eval \
  --model qwen2_5_vl \
  --model_args pretrained=Qwen/Qwen2.5-VL-3B-Instruct \
  --tasks mme \
  --batch_size 1 \
  --limit 8
```

3. Explore available options

   List the supported flags, models, and tasks to plan a fuller evaluation run.

```bash
uv run python -m lmms_eval --help
```

## When to use it

- Benchmark a new multimodal model across many vision, video, and audio tasks with one command
- Reproduce paper results for a model so two teams report the same numbers
- Compare two model checkpoints with confidence intervals and paired tests instead of single-number accuracy
- Run a quick smoke test on a handful of samples to confirm a model and environment are wired up correctly