█

AI/TLDR

EvalPlus

Rigorous correctness and efficiency benchmarks for code-generation LLMs

GitHub★ 1.8kWebsiteevalplus.github.io/leaderboard.html
Benchmark HarnessesOpen source
Language
Python
$pip install "evalplus[vllm]" --upgrade

Overview

EvalPlus is an evaluation framework for code-generation language models. It extends the well-known HumanEval and MBPP benchmarks with many more test cases: HumanEval+ adds about 80x more tests and MBPP+ about 35x more, so models that pass the originals but break on edge cases are caught.

It is meant for LLM researchers and engineering teams who train or compare code models and want a trustworthy pass@1 number. By measuring how much a score drops once the extra tests are applied, EvalPlus shows whether a model's generated code is solid or fragile. It also includes EvalPerf, which scores the efficiency of generated code, not just its correctness.

As a benchmark harness, it packages datasets, a CLI, a Docker image, and a public leaderboard. One command can run generation, post-processing, and evaluation, and it supports several inference backends including vLLM, Google Gemini, and Anthropic.

What it does

  • HumanEval+ and MBPP+ datasets with far more tests than the originals to expose edge-case failures
  • EvalPerf benchmark that measures the runtime efficiency of generated code, not only correctness
  • Single command runs generation, post-processing, and evaluation end to end
  • Multiple inference backends, including vLLM, Google Gemini, and Anthropic
  • Safe code execution inside the provided Docker image (ganler/evalplus)
  • Public leaderboard with model rankings before and after the stricter tests

Getting started

Install the package from PyPI, then run a model against HumanEval or MBPP with a single command. Use the Docker image when you want isolated, safe execution of generated code.

Install EvalPlus

Install the latest stable release from PyPI. Use the vllm extra if you plan to run a model with the vLLM backend.

bashbash
pip install "evalplus[vllm]" --upgrade

Run correctness evaluation

Generate samples and evaluate them on HumanEval or MBPP in one command. Pick the dataset and backend, and add --greedy for deterministic decoding.

bashbash
evalplus.evaluate --model "ise-uiuc/Magicoder-S-DS-6.7B" \
                  --dataset humaneval \
                  --backend vllm \
                  --greedy

Run generation and evaluation separately

Generate code locally first, then evaluate the resulting samples inside Docker for safe execution.

bashbash
evalplus.codegen --model "ise-uiuc/Magicoder-S-DS-6.7B" \
                 --dataset humaneval \
                 --backend vllm \
                 --greedy

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

When to use it

  • Benchmarking a newly trained code model on HumanEval+ and MBPP+ to report a reliable pass@1 score
  • Comparing several code-generation models on the same stricter test suite before publishing results
  • Checking how much a model's score drops under the extra tests to judge whether its code is fragile
  • Measuring the runtime efficiency of generated code with EvalPerf alongside correctness

How EvalPlus compares

EvalPlus alongside other open-source benchmark harnesses tools AI/TLDR tracks, ranked by GitHub stars.

ToolStarsWhat it does
LM Evaluation Harness★ 14kEleutherAI's framework for few-shot evaluation of language models across 60+ academic benchmarks, used as the backend for many leaderboards.
OpenCompass★ 7.5kAn LLM evaluation platform that runs models against 100+ datasets covering reasoning, knowledge, coding, and domain tasks, with leaderboards and multi-model support.
SWE-bench★ 5.9kA benchmark and containerized harness that tests whether language models can resolve real GitHub issues by generating patches that pass a repository's tests.
simple-evals★ 4.6kOpenAI's lightweight library for running standard zero-shot, chain-of-thought benchmarks like MMLU, MATH, and GPQA to measure model accuracy.
lmms-eval★ 4.4kAn evaluation suite for large multimodal models that runs image, video, and audio benchmarks across many tasks with a unified, reproducible interface.
AgentBench★ 3.7kA benchmark that evaluates LLMs as agents across diverse interactive environments such as operating systems, databases, web browsing, and games.
HELM★ 2.9kStanford CRFM's Holistic Evaluation of Language Models framework for reproducible, transparent benchmarking of foundation and multimodal models across many scenarios and metrics.
EvalPlus★ 1.8kRigorous correctness and efficiency benchmarks for code-generation LLMs