# Kaggle Benchmarks

> Kaggle's Python library for writing model evaluations as decorated functions — prompt a model, assert on what comes back, and publish the run to a Kaggle leaderboard

- Category: Evaluation & Production Ops / Benchmark Harnesses
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/Kaggle/kaggle-benchmarks
- Website: https://www.kaggle.com/benchmarks
- Canonical page: https://ai-tldr.dev/tools/kaggle-benchmarks/
- Updated: 2026-06-04

## In short

Kaggle Benchmarks is a Python library for evaluating AI models on tasks you define yourself. A benchmark is an ordinary function wearing a `@kbench.task` decorator: it receives an `llm` handle, prompts it, and then asserts on the response using the library's assertion set. It sits in the Benchmark Harnesses category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

Kaggle Benchmarks is a Python library for evaluating AI models on tasks you define yourself. A benchmark is an ordinary function wearing a `@kbench.task` decorator: it receives an `llm` handle, prompts it, and then asserts on the response using the library's assertion set. That shape keeps an evaluation readable as code and reproducible as data — each run captures the exact inputs, outputs and model interactions for later review.

It is built for evaluations that go past string matching. Models can be given tools, including a built-in Python interpreter that executes code; inputs can be images, audio or video; outputs can be requested as structured `dataclass` or Pydantic objects rather than free text; and a task can be run across a whole dataset — a pandas DataFrame, for instance — to produce aggregate metrics instead of a single verdict. The assertion set is extensible, so a task-specific check is just another function.

The intended home is a Kaggle notebook, where the library and its dependencies are preinstalled and each task run emits a task file plus run files that build the benchmark entity and its public leaderboard. Since June 2026 the same authoring loop also works locally: a `benchmarks` subcommand on the Kaggle CLI, the `kaggle-benchmarks` SDK and a `write-kaggle-benchmarks` agent skill let you write, push and run evaluations from VS Code, Cursor or Antigravity. The library is Apache-2.0 licensed.

## What it does

- Define an evaluation as a plain Python function with the `@kbench.task` decorator
- Rich built-in assertions plus the ability to write your own for task-specific correctness checks
- Structured and multimodal I/O — dataclass or Pydantic outputs, and image, audio or video inputs
- Tool use, including a built-in Python interpreter the model can execute code with
- Dataset evaluation: run a task across a pandas DataFrame for aggregate performance rather than a single result
- Runs in a Kaggle notebook emit task and run files that generate a public Kaggle leaderboard

## Getting started

The path of least resistance is a Kaggle notebook, where nothing needs installing; local authoring is supported for developing tasks in your own editor.

1. Start in a Kaggle notebook

   No installation is required. Creating a new benchmark task opens a notebook with the library and its dependencies preinstalled.

```text
https://www.kaggle.com/benchmarks/tasks/new
```

2. Write a task

   Decorate a function, prompt the model through the llm handle, then assert on the response. The expectation string is what shows up when the assertion fails.

```python
import kaggle_benchmarks as kbench

@kbench.task(name="simple_riddle")
def solve_riddle(llm, riddle: str, answer: str):
    """Asks a riddle and checks for a keyword in the answer."""
    response = llm.prompt(riddle)

    kbench.assertions.assert_contains_regex(
        f"(?i){answer}", response, expectation="LLM should give the right answer."
    )
```

3. Run it

   Call .run() on the decorated task with the model handle and the task's arguments.

```python
solve_riddle.run(
    llm=kbench.llm,
    riddle="What gets wetter as it dries?",
    answer="Towel",
)
```

4. Set up local development

   Requires Python 3.11+, git and uv. Clone the repository, create a virtualenv and install it editable. Local runs need a MODEL_PROXY_URL and MODEL_PROXY_API_KEY in a .env file; HTTP response caching can be turned on to cut cost while iterating.

```bash
git clone https://github.com/Kaggle/kaggle-benchmarks.git
cd kaggle-benchmarks
uv venv
source .venv/bin/activate
uv pip install -e .

export ENABLE_LOCAL_CACHING=true
```

5. Let a coding agent write the task

   Kaggle publishes an agent skill for exactly this; the README calls it the fastest way to write a task and run it on Kaggle.

```text
https://github.com/Kaggle/kaggle-skills/blob/main/write-kaggle-benchmarks/SKILL.md
```

## When to use it

- Build a domain-specific evaluation for your own task rather than relying on a public benchmark that does not match it
- Compare several models on the same task with identical prompts, tools and assertions
- Test tool use and code execution, not just text output, by giving the model the built-in Python interpreter
- Score a model over an entire dataset and publish the result as a Kaggle leaderboard

## Version history

- 2026-06-04: Google opened Kaggle Benchmarks task authoring to local development: a benchmarks subcommand on the Kaggle CLI, the kaggle-benchmarks Python SDK and a write-kaggle-benchmarks agent skill let developers write, push and run evaluations from VS Code, Cursor or Antigravity instead of the web editor.