# Kimi-Vendor-Verifier

> Check that the provider serving you a Kimi model is serving it correctly

- Category: Evaluation & Production Ops / Benchmark Harnesses
- Language: Python
- Licence: MIT
- Repository: https://github.com/MoonshotAI/Kimi-Vendor-Verifier
- Canonical page: https://ai-tldr.dev/tools/kimi-vendor-verifier/
- Updated: 2026-04-20

## In short

Kimi-Vendor-Verifier is Check that the provider serving you a Kimi model is serving it correctly. Quantisation, a misconfigured chat template, a wrong tool-call schema or a truncated context window all produce an endpoint that answers plausibly while scoring measurably worse. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed MIT.

## Overview

The same model name served by two providers is not always the same model. Quantisation, a misconfigured chat template, a wrong tool-call schema or a truncated context window all produce an endpoint that answers plausibly while scoring measurably worse. Kimi-Vendor-Verifier is Moonshot AI's harness for detecting exactly that: you point it at an endpoint and it reports how that endpoint actually performs.

It runs two kinds of check. The benchmark suites — OCRBench, MMMU Pro Vision, AIME 2025, BEAM for 1M-token context, and DeepSWE for agentic work — measure capability across OCR, vision, math and long-context tasks. The pre-flight validation tests are cheaper and more diagnostic: they exercise API parameters, tool-call schemas, K3-specific features and prompt-token accounting, and they are what catch a misconfiguration before you spend money on a full benchmark run.

The harness works against the official Kimi API and against open-source deployments on vLLM, SGLang or KTransformers, which makes it as useful for validating your own serving stack as for comparing vendors. Reasoning effort is selectable (low, high, max) for K3 models, streaming is supported with automatic retries on network errors, and the repository publishes comparison results across providers.

## What it does

- Benchmark suites spanning OCR (OCRBench), vision (MMMU Pro Vision), math (AIME 2025), 1M-token context (BEAM) and agentic work (DeepSWE)
- Pre-flight validation of API parameters, tool-call schemas, K3 features and prompt-token accuracy
- Runs against the official Kimi API or a self-hosted vLLM, SGLang or KTransformers deployment
- Selectable reasoning effort (low / high / max) for K3 models
- Streaming support with automatic retry on network errors
- Published cross-provider comparison results in the repository

## Getting started

Install the package from a checkout with uv, point it at the endpoint you want to test, then run a benchmark.

1. Install

   Resolves the environment and installs the harness in editable mode.

```bash
uv sync && uv pip install -e .
```

2. Point it at an endpoint

   Any OpenAI-compatible base URL works, whether that is the official API or your own vLLM/SGLang deployment.

```bash
export KIMI_API_KEY="your-api-key"
export KIMI_BASE_URL="your-base-url"
```

3. Run a benchmark

   Each suite is a subcommand of eval.py; --model is the model id as that provider names it.

```bash
uv run python eval.py ocrbench --model kimi/your-model-id --think-mode kimi --max-tokens 16384 --stream
```

4. Run the pre-flight checks

   The validation tests ship as a pytest suite and are much cheaper than a full benchmark — run them first when you suspect a misconfiguration.

## When to use it

- Verify a third-party provider is serving an undegraded Kimi model before you route production traffic to it
- Regression-test your own vLLM or SGLang deployment after a config, template or quantisation change
- Diagnose tool-calling failures by checking the endpoint's schema conformance instead of blaming the prompt
- Compare providers on the same benchmarks rather than on their marketing numbers

## Version history

- 2026-04-20: Moonshot AI open-sourced Kimi Vendor Verifier under MIT to let users check that inference providers are not serving degraded or misconfigured Kimi models, running five benchmarks across OCR, vision, tool-call accuracy and agentic tasks.