# EvalPlus

> Rigorous correctness and efficiency benchmarks for code-generation LLMs

- Category: Evaluation & Production Ops / Benchmark Harnesses
- Language: Python
- Repository: https://github.com/evalplus/evalplus
- Website: https://evalplus.github.io/leaderboard.html
- Canonical page: https://ai-tldr.dev/tools/evalplus/

## In short

EvalPlus is an evaluation framework for code-generation language models. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python. To get started: install EvalPlus.

## Overview

EvalPlus is an evaluation framework for code-generation language models. It extends the well-known HumanEval and MBPP benchmarks with many more test cases: HumanEval+ adds about 80x more tests and MBPP+ about 35x more, so models that pass the originals but break on edge cases are caught.

It is meant for LLM researchers and engineering teams who train or compare code models and want a trustworthy pass@1 number. By measuring how much a score drops once the extra tests are applied, EvalPlus shows whether a model's generated code is solid or fragile. It also includes EvalPerf, which scores the efficiency of generated code, not just its correctness.

As a benchmark harness, it packages datasets, a CLI, a Docker image, and a public leaderboard. One command can run generation, post-processing, and evaluation, and it supports several inference backends including vLLM, Google Gemini, and Anthropic.

## What it does

- HumanEval+ and MBPP+ datasets with far more tests than the originals to expose edge-case failures
- EvalPerf benchmark that measures the runtime efficiency of generated code, not only correctness
- Single command runs generation, post-processing, and evaluation end to end
- Multiple inference backends, including vLLM, Google Gemini, and Anthropic
- Safe code execution inside the provided Docker image (ganler/evalplus)
- Public leaderboard with model rankings before and after the stricter tests

## Getting started

Install the package from PyPI, then run a model against HumanEval or MBPP with a single command. Use the Docker image when you want isolated, safe execution of generated code.

1. Install EvalPlus

   Install the latest stable release from PyPI. Use the vllm extra if you plan to run a model with the vLLM backend.

```bash
pip install "evalplus[vllm]" --upgrade
```

2. Run correctness evaluation

   Generate samples and evaluate them on HumanEval or MBPP in one command. Pick the dataset and backend, and add --greedy for deterministic decoding.

```bash
evalplus.evaluate --model "ise-uiuc/Magicoder-S-DS-6.7B" \
                  --dataset humaneval \
                  --backend vllm \
                  --greedy
```

3. Run generation and evaluation separately

   Generate code locally first, then evaluate the resulting samples inside Docker for safe execution.

```bash
evalplus.codegen --model "ise-uiuc/Magicoder-S-DS-6.7B" \
                 --dataset humaneval \
                 --backend vllm \
                 --greedy
```

## When to use it

- Benchmarking a newly trained code model on HumanEval+ and MBPP+ to report a reliable pass@1 score
- Comparing several code-generation models on the same stricter test suite before publishing results
- Checking how much a model's score drops under the extra tests to judge whether its code is fragile
- Measuring the runtime efficiency of generated code with EvalPerf alongside correctness