# OpenCompass

> One-stop platform for benchmarking large language models across 100+ datasets

- Category: Evaluation & Production Ops / Benchmark Harnesses
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/open-compass/opencompass
- Website: https://opencompass.org.cn
- Canonical page: https://ai-tldr.dev/tools/opencompass/

## In short

OpenCompass is an open-source platform for evaluating large language models. It runs a model against more than 100 datasets that cover reasoning, knowledge, coding, math, and other tasks, then reports scores you can compare side by side. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

OpenCompass is an open-source platform for evaluating large language models. It runs a model against more than 100 datasets that cover reasoning, knowledge, coding, math, and other tasks, then reports scores you can compare side by side. The project also publishes public leaderboards (CompassRank) and a dataset hub (CompassHub).

It is built for researchers and engineers who need repeatable, comparable benchmark results rather than ad-hoc spot checks. You point it at a model from Hugging Face or an API, pick the datasets you care about, and it handles inference, answer extraction, and scoring through configuration files or simple command-line flags.

As a benchmark harness, OpenCompass sits in the evaluation stage of a model workflow. It supports multiple inference backends such as Hugging Face Transformers, vLLM, and LMDeploy, so you can evaluate the same model on different runtimes and reproduce official leaderboard results locally.

## What it does

- Runs models against 100+ datasets covering reasoning, knowledge, coding, math, and long-context tasks
- Works with Hugging Face models and API models, plus vLLM and LMDeploy inference backends
- Configuration-driven: define models, datasets, and summarizers in Python config files or pass them as CLI flags
- Public CompassRank leaderboards and the CompassHub dataset hub for comparing results
- LLM-as-judge and math-verification evaluators (GenericLLMEvaluator, MATHVerifyEvaluator) for open-ended and reasoning tasks
- Both perplexity (ppl) and generative (gen) evaluation modes per dataset

## Getting started

Install OpenCompass with pip, then run an evaluation by choosing a model and one or more datasets.

1. Install

   Install the latest release from PyPI. Python 3.10 is recommended.

```bash
pip install -U opencompass
```

2. Run an evaluation with the CLI

   Pass a model config and one or more datasets. This example evaluates a Hugging Face model on a demo GSM8K dataset.

```bash
opencompass --models hf_internlm2_5_1_8b_chat --datasets demo_gsm8k_chat_gen
```

3. Evaluate any Hugging Face model by path

   Use run.py with --hf-path to point at a model on the Hub and select datasets directly.

```bash
python run.py --datasets siqa_gen winograd_ppl \
  --hf-type base \
  --hf-path facebook/opt-125m
```

4. Use a config file

   For repeatable setups, define models, datasets, and summarizers in a Python config and pass it to run.py.

```bash
python run.py configs/eval_demo.py
```

## When to use it

- Benchmark a fine-tuned or newly released model against standard datasets before publishing results
- Reproduce official leaderboard scores (e.g. CompassAcademic) locally to validate a model
- Compare several candidate models on the same task suite to pick one for production
- Evaluate reasoning and math performance using LLM-judge or math-verification evaluators