# SkillEvaluator

> Test whether an agent skill is well-formed, original and actually helps the agent

- Category: Evaluation & Production Ops / Evaluation & Red-Teaming
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/NVIDIA/SkillEvaluator
- Website: https://docs.nvidia.com/skills/skillevaluator/
- Canonical page: https://ai-tldr.dev/tools/skillevaluator/
- Updated: 2026-11-13

## In short

SkillEvaluator is Test whether an agent skill is well-formed, original and actually helps the agent. Writing one is easy; knowing whether it is safe, whether it repeats guidance the agent already has, and whether it makes the agent any better at the task is not. It sits in the Evaluation & Red-Teaming category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

Agent skills are folders of instructions and supporting files, defined by the Agent Skills specification, that coding agents load on demand. Writing one is easy; knowing whether it is safe, whether it repeats guidance the agent already has, and whether it makes the agent any better at the task is not. SkillEvaluator is NVIDIA's open-source answer: it runs quality checks on a skill first, then compares agent runs with and without the skill. It is part of the NVIDIA Verified Skills pipeline.

The work is split into three independent tiers. Tier 1 (Validation) asks whether the skill is safe and well-formed, with schema, quality, PII and security checks; it fetches pinned SkillSpector and Gitleaks releases for scanning, and Semgrep is optional. Tier 2 (Deduplication) uses embeddings to find guidance that overlaps within a skill and, given a catalog, across skills. Tier 3 (Live Evaluation) runs a supported agent in a sandbox — Docker by default, powered by the open-source Harbor framework — on the same tasks with and without the skill, and reports the difference as Skill Lift.

Each tier is its own command, so a keyless static quality check can run with no API key, GPU or Docker, while a single `validate` command runs all three. Tier 3 can generate an evaluation dataset grounded in the skill's own evaluation guidance, a prompt file of tasks and outcomes, or retained traces; reports come out in the terminal and as HTML, JSON and Markdown, with budgets on tokens, judge calls and cost.

SkillEvaluator also ships ready-made GitHub Actions and GitLab CI templates, plus a `skillevaluator ci init` command that sets them up, so changed skills in a repository get checked on every pull request. It is Apache-2.0 licensed, written in Python (3.12–3.14), published on PyPI as `skillevaluator`, and accompanied by an arXiv paper. NVIDIA marks its support level as experimental and community-supported.

## What it does

- Tier 1 validation: schema, quality, PII and security checks, with SkillSpector and Gitleaks scanners fetched automatically
- Keyless `quality-check` that returns a 0–100 quality score and findings with no API key, GPU or Docker
- Tier 2 deduplication: embedding-based overlap detection within a skill and across a skill catalog
- Tier 3 live evaluation: sandboxed A/B agent runs with and without the skill, reported as Skill Lift with pass@k
- Evaluation-dataset generation grounded in author guidance, a prompt file or production traces, with human review before expectations are promoted
- Token, judge-call, attempt and cost budgets, plus version comparison and reusable no-skill baselines
- GitHub Actions and GitLab CI templates, set up by `skillevaluator ci init`

## Getting started

The quickest path uses the bundled `text-analyzer` example skill and needs only Git and uv — uv provisions Python 3.13 and the dependencies. Replace the example path with your own skill directory once it works.

1. Get the example

   Clone the repository; the example skill ships inside it.

```bash
git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
```

2. Run a quality check without an LLM

   No API key, scanner, GPU or Docker is needed. The example scores 84.5/100 (B) and saves a JSON report; the passing threshold defaults to 70.

```bash
uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality
```

3. Add model-backed rubric scores

   With an NVIDIA Build API key (free prototyping for eligible NVIDIA Developer Program members), the rubric check sends the skill's documentation to a model and saves its feedback. The same key covers Tier 2 embeddings and supported Tier 3 agents.

```bash
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
uv run --python 3.13 --extra llm skillevaluator rubric-eval \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-build
```

4. Run the three tiers

   Each tier is independent and needs its own extra. `doctor` checks readiness; `validate` runs all tiers (Tier 3 needs a provider, a supported agent and a running sandbox). Use `--tiers 1,2` to skip live evaluation.

```bash
uv run --extra security skillevaluator tier1 ./my-skill --no-llm
uv run --extra tier2 skillevaluator tier2 ./my-skill
uv run --extra tier3 skillevaluator tier3 ./my-skill

# or everything at once
uv run --python 3.13 --extra all skillevaluator doctor --env-mode docker
uv run --python 3.13 --extra all skillevaluator validate ./my-skill
```

5. Check skills in CI

   Inside a checked-out GitHub repository, the guided setup lets you choose checks and opens a setup pull request. Ready-made GitHub and GitLab templates are in the repository's `ci/bundles` folder if you prefer to copy one by hand.

```bash
pip install --upgrade skillevaluator
skillevaluator ci init
```

## When to use it

- Gate a team's agent-skill repository so every changed skill passes quality and security checks before merge
- Prove a new skill earns its context-window cost by measuring agent results with and without it
- Find skills in a large catalog that repeat each other's guidance before publishing them
- Compare two versions of a skill under a fixed token and cost budget

## Version history

- 2026-11-13 v0.9.0: Adds grounded evaluation-dataset generation, skill-version comparison with reusable baselines, token and cost budgets, and clearer evidence in reports; the Tier 3 backend moves to Harbor 0.25.0.
