MTEB

// Overview

MTEB is the evaluation toolbox behind the Massive Text Embedding Benchmark. Picking an embedding model on one retrieval dataset tells you very little — a model that wins at semantic search can be mediocre at classification, clustering or bitext mining, and a model that looks strong in English can collapse in another language. MTEB answers that by running one model over a large, versioned catalogue of tasks and reporting a score per task, so the comparison is broad enough to act on.

It is a normal Python package. You install it with pip or uv, hand it a model — anything MTEB implements natively, or any `sentence-transformers` model it falls back to — select tasks or a whole named benchmark, and call `mteb.evaluate`. The same run is available from the command line with `mteb run`, which writes the per-task results to a folder you choose so they can be re-loaded, diffed between checkpoints, or submitted.

The project doubles as the source of the MTEB leaderboard hosted on Hugging Face Spaces, which is where most published embedding-model comparisons come from. Tasks, benchmarks and model definitions all live in the repository and are contributed through documented workflows, so adding your own dataset, registering a model, or defining a new benchmark is a pull request rather than a private fork.

// What it does

• One evaluation entry point (`mteb.evaluate`) covering many task types — retrieval, classification, clustering, reranking, semantic similarity, bitext mining and more
• Multilingual and multimodal task coverage, including low-resource languages
• Native model implementations with an automatic fall-back to any `sentence-transformers` model
• A CLI (`mteb run`) mirroring the Python API, with a results folder you can cache, reload and compare
• Named benchmarks so a run reproduces a published task selection instead of an ad-hoc list
• Feeds the public MTEB leaderboard on Hugging Face Spaces
• Documented contribution paths for adding a model, a dataset/task, or a whole benchmark

// Getting started

MTEB is a Python package; the only hard dependency for the example below is a model implementation such as sentence-transformers.

pip install mteb
# or, with uv
uv add mteb

import mteb
from sentence_transformers import SentenceTransformer

model_name = "sentence-transformers/all-MiniLM-L6-v2"
model = mteb.get_model(model_name)

tasks = mteb.get_tasks(tasks=["Banking77Classification.v2"])

results = mteb.evaluate(model, tasks=tasks)

mteb run \
    -m sentence-transformers/all-MiniLM-L6-v2 \
    -t "Banking77Classification.v2" \
    --output-folder results

https://huggingface.co/spaces/mteb/leaderboard

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when you have to choose an embedding model for a RAG stack and want more than one dataset's opinion
• Reach for it to check whether a fine-tuned embedding model actually improved, and where it regressed
• Reach for it when you need coverage in a specific language and want to see per-task scores rather than an average
• Reach for it to publish a new embedding model with results the community can reproduce and compare

// How MTEB compares

MTEB alongside other open-source benchmark harnesses tools AI/TLDR tracks, ranked by GitHub stars.