# VAKRA — IBM Research's Enterprise Agent Benchmark with 8,000+ Live APIs

> IBM Research's benchmark runs AI agents against 8,000+ live enterprise APIs to measure real tool-use and multi-hop reasoning.

- Org: IBM Research
- Date: 2026-04-15
- Category: benchmark
- Source: https://huggingface.co/blog/ibm-research/vakra-benchmark-analysis
- Canonical page: https://ai-tldr.dev/releases/ibm-vakra-benchmark/

## In short

IBM Research releases VAKRA: an executable benchmark for enterprise AI agents with 8,000+ locally-hosted APIs across 62 domains. Replay-based evaluation tests API chaining, tool selection, multi-hop reasoning, and policy adherence. Current frontier models are still failing significantly.

## What it is

VAKRA (eValuating API and Knowledge Retrieval Agents) is a benchmark from IBM Research for testing how well AI agents reason and act in enterprise-like environments. It uses over 8,000 locally-hosted APIs backed by real databases across 62 domains, plus domain-aligned document collections — designed for the multi-hop, multi-source task chains that appear in production enterprise deployments.

## How it works

Evaluation is replay-based: the benchmark records the agent's predicted tool-call trajectory and replays it against a live MCP environment, scoring each step for correctness, groundedness, and policy adherence. Tasks span four capability tiers from simple API chaining (2,077 instances) to multi-hop multi-source reasoning with policy constraints (644 instances requiring 3–7 reasoning steps). The leaderboard is open for public submission.

## Why it matters

Most agent benchmarks use static snapshots or simplified mock APIs. VAKRA uses live, executable APIs with real database state, making scores harder to game and closer to actual enterprise conditions. Results from GPT-OSS-120B and Gemini-3-flash-preview show substantial headroom, giving a realistic picture of where production agent reliability stands today.

## Who it's for

ML engineers and researchers evaluating enterprise AI agents

## Try it

https://ibm-research-vakra.hf.space/

## Links

- [Blog Post](https://huggingface.co/blog/ibm-research/vakra-benchmark-analysis)
- [Dataset](https://huggingface.co/datasets/ibm-research/VAKRA)
- [Leaderboard](https://ibm-research-vakra.hf.space/)
- [Repo](https://github.com/IBM/vakra)