EvalScope

// Overview

EvalScope is the evaluation framework maintained by the ModelScope community. One `evalscope eval` command points a benchmark suite at either a local checkpoint or any OpenAI-API-compatible endpoint and prints a scored table per dataset and subset. It ships industry-standard benchmarks out of the box — MMLU, C-Eval, GSM8K, ARC and many more — and covers large language models, vision-language models, embedding models, rerankers and AIGC models rather than text-only LLMs.

Beyond one-shot scoring it has an agent evaluation mode that drives benchmarks such as GSM8K, AIME and SWE-bench Agentic inside a controlled multi-turn AgentLoop with pluggable strategies, tools and a Docker sandbox, recording a full per-sample agent trace you can replay in the dashboard. An External Agent Bridge evaluates off-the-shelf agent CLIs — Claude Code, Codex — by transparently forwarding their Anthropic Messages, OpenAI Chat and OpenAI Responses traffic (including SSE streaming) to your evaluation model while recording the trajectory.

Three other pieces round it out. Multiple evaluation backends — OpenCompass, VLMEvalKit and RAGEval — plug in behind the same CLI. An inference performance stress tester measures throughput, time-to-first-token and time-per-output-token against a running service, with results recordable to Weights & Biases or SwanLab. And a web dashboard compares models across runs, browses report overviews and drills into individual predictions. Arena mode adds pairwise model battles with win rates and confidence intervals.

// What it does

• Built-in benchmarks including MMLU, C-Eval, GSM8K and ARC, with a growing catalogue covering agents, code, long context, multimodal and OCR
• Evaluates LLMs, VLMs, embedding models, rerankers and AIGC models through one CLI and one Python API
• Agent evaluation mode: a controlled multi-turn AgentLoop with pluggable strategies, tools and a Docker sandbox, recording a full agent trace per sample
• External Agent Bridge that evaluates agent CLIs such as Claude Code and Codex by forwarding their LLM traffic to your evaluation model
• Multi-backend integration with OpenCompass, VLMEvalKit and RAGEval behind a single interface
• Inference stress testing with TTFT and TPOT metrics, results recordable to wandb and SwanLab
• Interactive web dashboard for multi-model comparison, report overviews and prediction-level inspection
• Arena mode for pairwise model battles with win rates and confidence intervals, plus custom datasets, models and metrics

// Getting started

EvalScope is a Python package on PyPI and needs Python 3.10 or newer. Evaluating a hosted API endpoint needs no GPU.

pip install evalscope

evalscope eval \
 --model your-model-name \
 --api-url $OPENAI_API_BASE_URL \
 --api-key $OPENAI_API_KEY \
 --eval-type openai_api \
 --datasets gsm8k arc \
 --limit 5

evalscope eval \
 --model Qwen/Qwen2.5-0.5B-Instruct \
 --datasets gsm8k arc \
 --limit 5

from evalscope import run_task, TaskConfig

task_cfg = TaskConfig(
    model='your-model-name',
    api_url='https://your-openai-compatible-endpoint/v1',
    api_key='your_api_key',
    eval_type='openai_api',
    datasets=['gsm8k', 'arc'],
    limit=5
)

run_task(task_cfg)

pip install 'evalscope[service]'
evalscope service

evalscope eval \
 --model Qwen/Qwen3-0.6B \
 --model-args '{"revision": "master", "precision": "torch.float16", "device_map": "auto"}' \
 --generation-config '{"do_sample":true,"temperature":0.6,"max_tokens":512}' \
 --dataset-args '{"gsm8k": {"few_shot_num": 0, "few_shot_random": false}}' \
 --datasets gsm8k \
 --limit 10

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it to score a model you are serving against standard academic benchmarks without writing a harness
• Reach for it when the thing under test is an agent loop rather than a single completion, and you need the trajectory recorded
• Reach for it to compare several checkpoints or providers side by side in a dashboard instead of a pile of JSON
• Reach for it to stress-test a deployed inference service for throughput, TTFT and TPOT before it takes traffic

// How EvalScope compares

EvalScope alongside other open-source benchmark harnesses tools AI/TLDR tracks, ranked by GitHub stars.