Kaggle Benchmarks

// Overview

Kaggle Benchmarks is a Python library for evaluating AI models on tasks you define yourself. A benchmark is an ordinary function wearing a `@kbench.task` decorator: it receives an `llm` handle, prompts it, and then asserts on the response using the library's assertion set. That shape keeps an evaluation readable as code and reproducible as data — each run captures the exact inputs, outputs and model interactions for later review.

It is built for evaluations that go past string matching. Models can be given tools, including a built-in Python interpreter that executes code; inputs can be images, audio or video; outputs can be requested as structured `dataclass` or Pydantic objects rather than free text; and a task can be run across a whole dataset — a pandas DataFrame, for instance — to produce aggregate metrics instead of a single verdict. The assertion set is extensible, so a task-specific check is just another function.

The intended home is a Kaggle notebook, where the library and its dependencies are preinstalled and each task run emits a task file plus run files that build the benchmark entity and its public leaderboard. Since June 2026 the same authoring loop also works locally: a `benchmarks` subcommand on the Kaggle CLI, the `kaggle-benchmarks` SDK and a `write-kaggle-benchmarks` agent skill let you write, push and run evaluations from VS Code, Cursor or Antigravity. The library is Apache-2.0 licensed.

// What it does

• Define an evaluation as a plain Python function with the `@kbench.task` decorator
• Rich built-in assertions plus the ability to write your own for task-specific correctness checks
• Structured and multimodal I/O — dataclass or Pydantic outputs, and image, audio or video inputs
• Tool use, including a built-in Python interpreter the model can execute code with
• Dataset evaluation: run a task across a pandas DataFrame for aggregate performance rather than a single result
• Runs in a Kaggle notebook emit task and run files that generate a public Kaggle leaderboard

// Getting started

The path of least resistance is a Kaggle notebook, where nothing needs installing; local authoring is supported for developing tasks in your own editor.

https://www.kaggle.com/benchmarks/tasks/new

import kaggle_benchmarks as kbench

@kbench.task(name="simple_riddle")
def solve_riddle(llm, riddle: str, answer: str):
    """Asks a riddle and checks for a keyword in the answer."""
    response = llm.prompt(riddle)

    kbench.assertions.assert_contains_regex(
        f"(?i){answer}", response, expectation="LLM should give the right answer."
    )

solve_riddle.run(
    llm=kbench.llm,
    riddle="What gets wetter as it dries?",
    answer="Towel",
)

git clone https://github.com/Kaggle/kaggle-benchmarks.git
cd kaggle-benchmarks
uv venv
source .venv/bin/activate
uv pip install -e .

export ENABLE_LOCAL_CACHING=true

https://github.com/Kaggle/kaggle-skills/blob/main/write-kaggle-benchmarks/SKILL.md

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Build a domain-specific evaluation for your own task rather than relying on a public benchmark that does not match it
• Compare several models on the same task with identical prompts, tools and assertions
• Test tool use and code execution, not just text output, by giving the model the built-in Python interpreter
• Score a model over an entire dataset and publish the result as a Kaggle leaderboard

// How Kaggle Benchmarks compares

Kaggle Benchmarks alongside other open-source benchmark harnesses tools AI/TLDR tracks, ranked by GitHub stars.