Overview
The same model name served by two providers is not always the same model. Quantisation, a misconfigured chat template, a wrong tool-call schema or a truncated context window all produce an endpoint that answers plausibly while scoring measurably worse. Kimi-Vendor-Verifier is Moonshot AI's harness for detecting exactly that: you point it at an endpoint and it reports how that endpoint actually performs.
It runs two kinds of check. The benchmark suites — OCRBench, MMMU Pro Vision, AIME 2025, BEAM for 1M-token context, and DeepSWE for agentic work — measure capability across OCR, vision, math and long-context tasks. The pre-flight validation tests are cheaper and more diagnostic: they exercise API parameters, tool-call schemas, K3-specific features and prompt-token accounting, and they are what catch a misconfiguration before you spend money on a full benchmark run.
The harness works against the official Kimi API and against open-source deployments on vLLM, SGLang or KTransformers, which makes it as useful for validating your own serving stack as for comparing vendors. Reasoning effort is selectable (low, high, max) for K3 models, streaming is supported with automatic retries on network errors, and the repository publishes comparison results across providers.
What it does
- Benchmark suites spanning OCR (OCRBench), vision (MMMU Pro Vision), math (AIME 2025), 1M-token context (BEAM) and agentic work (DeepSWE)
- Pre-flight validation of API parameters, tool-call schemas, K3 features and prompt-token accuracy
- Runs against the official Kimi API or a self-hosted vLLM, SGLang or KTransformers deployment
- Selectable reasoning effort (low / high / max) for K3 models
- Streaming support with automatic retry on network errors
- Published cross-provider comparison results in the repository
Getting started
Install the package from a checkout with uv, point it at the endpoint you want to test, then run a benchmark.
Install
Resolves the environment and installs the harness in editable mode.
uv sync && uv pip install -e .Point it at an endpoint
Any OpenAI-compatible base URL works, whether that is the official API or your own vLLM/SGLang deployment.
export KIMI_API_KEY="your-api-key"
export KIMI_BASE_URL="your-base-url"Run a benchmark
Each suite is a subcommand of eval.py; --model is the model id as that provider names it.
uv run python eval.py ocrbench --model kimi/your-model-id --think-mode kimi --max-tokens 16384 --streamRun the pre-flight checks
The validation tests ship as a pytest suite and are much cheaper than a full benchmark — run them first when you suspect a misconfiguration.
Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Verify a third-party provider is serving an undegraded Kimi model before you route production traffic to it
- Regression-test your own vLLM or SGLang deployment after a config, template or quantisation change
- Diagnose tool-calling failures by checking the endpoint's schema conformance instead of blaming the prompt
- Compare providers on the same benchmarks rather than on their marketing numbers
How Kimi-Vendor-Verifier compares
Kimi-Vendor-Verifier alongside other open-source benchmark harnesses tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| LM Evaluation Harness | ★ 14k | EleutherAI's framework for few-shot evaluation of language models across 60+ academic benchmarks, used as the backend for many leaderboards. |
| OpenCompass | ★ 7.5k | An LLM evaluation platform that runs models against 100+ datasets covering reasoning, knowledge, coding, and domain tasks, with leaderboards and multi-model support. |
| SWE-bench | ★ 5.9k | A benchmark and containerized harness that tests whether language models can resolve real GitHub issues by generating patches that pass a repository's tests. |
| simple-evals | ★ 4.6k | OpenAI's lightweight library for running standard zero-shot, chain-of-thought benchmarks like MMLU, MATH, and GPQA to measure model accuracy. |
| lmms-eval | ★ 4.4k | An evaluation suite for large multimodal models that runs image, video, and audio benchmarks across many tasks with a unified, reproducible interface. |
| AgentBench | ★ 3.7k | A benchmark that evaluates LLMs as agents across diverse interactive environments such as operating systems, databases, web browsing, and games. |
| HELM | ★ 2.9k | Stanford CRFM's Holistic Evaluation of Language Models framework for reproducible, transparent benchmarking of foundation and multimodal models across many scenarios and metrics. |
| Kimi-Vendor-Verifier | ★ 154 | Check that the provider serving you a Kimi model is serving it correctly |