█

AI/TLDR

Buy me a coffee
Buy me a coffee

SkillEvaluator

Test whether an agent skill is well-formed, original and actually helps the agent

Evaluation & Red-TeamingOpen source
Latest
v0.9.0
Updated
13 Nov 2026
Language
Python
License
Apache-2.0
Coverage
1 story
$git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git

What's new

v0.9.013 Nov 2026

Adds grounded evaluation-dataset generation, skill-version comparison with reusable baselines, token and cost budgets, and clearer evidence in reports; the Tier 3 backend moves to Harbor 0.25.0.

Latest news

IN SHORT

What is SkillEvaluator?

SkillEvaluator is Test whether an agent skill is well-formed, original and actually helps the agent. Writing one is easy; knowing whether it is safe, whether it repeats guidance the agent already has, and whether it makes the agent any better at the task is not. It sits in the Evaluation & Red-Teaming category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0.

Updated Markdown version ↗

Overview

Agent skills are folders of instructions and supporting files, defined by the Agent Skills specification, that coding agents load on demand. Writing one is easy; knowing whether it is safe, whether it repeats guidance the agent already has, and whether it makes the agent any better at the task is not. SkillEvaluator is NVIDIA's open-source answer: it runs quality checks on a skill first, then compares agent runs with and without the skill. It is part of the NVIDIA Verified Skills pipeline.

Diagram of the SkillEvaluator pipeline: a skill passes through Tier 1 Validation, Tier 2 Deduplication and Tier 3 Live Evaluation before reports with a verdict and lift
The three tiers, from static gates to measured agent lift; each tier can be run on its ownSkillEvaluator README ↗

The work is split into three independent tiers. Tier 1 (Validation) asks whether the skill is safe and well-formed, with schema, quality, PII and security checks; it fetches pinned SkillSpector and Gitleaks releases for scanning, and Semgrep is optional. Tier 2 (Deduplication) uses embeddings to find guidance that overlaps within a skill and, given a catalog, across skills. Tier 3 (Live Evaluation) runs a supported agent in a sandbox — Docker by default, powered by the open-source Harbor framework — on the same tasks with and without the skill, and reports the difference as Skill Lift.

Each tier is its own command, so a keyless static quality check can run with no API key, GPU or Docker, while a single `validate` command runs all three. Tier 3 can generate an evaluation dataset grounded in the skill's own evaluation guidance, a prompt file of tasks and outcomes, or retained traces; reports come out in the terminal and as HTML, JSON and Markdown, with budgets on tokens, judge calls and cost.

SkillEvaluator also ships ready-made GitHub Actions and GitLab CI templates, plus a `skillevaluator ci init` command that sets them up, so changed skills in a repository get checked on every pull request. It is Apache-2.0 licensed, written in Python (3.12–3.14), published on PyPI as `skillevaluator`, and accompanied by an arXiv paper. NVIDIA marks its support level as experimental and community-supported.

What it does

  • Tier 1 validation: schema, quality, PII and security checks, with SkillSpector and Gitleaks scanners fetched automatically
  • Keyless `quality-check` that returns a 0–100 quality score and findings with no API key, GPU or Docker
  • Tier 2 deduplication: embedding-based overlap detection within a skill and across a skill catalog
  • Tier 3 live evaluation: sandboxed A/B agent runs with and without the skill, reported as Skill Lift with pass@k
  • Evaluation-dataset generation grounded in author guidance, a prompt file or production traces, with human review before expectations are promoted
  • Token, judge-call, attempt and cost budgets, plus version comparison and reusable no-skill baselines
  • GitHub Actions and GitLab CI templates, set up by `skillevaluator ci init`

Getting started

The quickest path uses the bundled `text-analyzer` example skill and needs only Git and uv — uv provisions Python 3.13 and the dependencies. Replace the example path with your own skill directory once it works.

Get the example

Clone the repository; the example skill ships inside it.

bashbash
git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator

Run a quality check without an LLM

No API key, scanner, GPU or Docker is needed. The example scores 84.5/100 (B) and saves a JSON report; the passing threshold defaults to 70.

bashbash
uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality

Add model-backed rubric scores

With an NVIDIA Build API key (free prototyping for eligible NVIDIA Developer Program members), the rubric check sends the skill's documentation to a model and saves its feedback. The same key covers Tier 2 embeddings and supported Tier 3 agents.

bashbash
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
uv run --python 3.13 --extra llm skillevaluator rubric-eval \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-build

Run the three tiers

Each tier is independent and needs its own extra. `doctor` checks readiness; `validate` runs all tiers (Tier 3 needs a provider, a supported agent and a running sandbox). Use `--tiers 1,2` to skip live evaluation.

bashbash
uv run --extra security skillevaluator tier1 ./my-skill --no-llm
uv run --extra tier2 skillevaluator tier2 ./my-skill
uv run --extra tier3 skillevaluator tier3 ./my-skill

# or everything at once
uv run --python 3.13 --extra all skillevaluator doctor --env-mode docker
uv run --python 3.13 --extra all skillevaluator validate ./my-skill
SkillEvaluator CI report table listing skills/glossary-builder with a FAIL scan result, a 98.2 A quality score, 1 error, 3 warnings and a View report link
A repository CI report: the skill scores 98.2 for quality but fails the scan, which is a separate verdictSkillEvaluator CI examples ↗

Check skills in CI

Inside a checked-out GitHub repository, the guided setup lets you choose checks and opens a setup pull request. Ready-made GitHub and GitLab templates are in the repository's `ci/bundles` folder if you prefer to copy one by hand.

bashbash
pip install --upgrade skillevaluator
skillevaluator ci init

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

When to use it

  • Gate a team's agent-skill repository so every changed skill passes quality and security checks before merge
  • Prove a new skill earns its context-window cost by measuring agent results with and without it
  • Find skills in a large catalog that repeat each other's guidance before publishing them
  • Compare two versions of a skill under a fixed token and cost budget
VideoNVIDIA Developer livestream on evaluating agent skills, linked from the SkillEvaluator READMENVIDIA Developer ↗

How SkillEvaluator compares

SkillEvaluator alongside other open-source evaluation & red-teaming tools AI/TLDR tracks, ranked by GitHub stars.

ToolStarsWhat it does
Strix★ 75.9kStrix runs autonomous AI agents that act like hackers, dynamically running your code to find vulnerabilities and validate them with real proof-of-concepts.
iFixAi★ 46kAn agent auditor that runs up to 60 inspections across fabrication, manipulation, deception, unpredictability and opacity, and returns an A–F grade judged by a model from a different vendor.
promptfoo★ 27.3kA developer-first CLI and library for testing and comparing prompts and models, with red-teaming probes for prompt injection, PII leaks, and other vulnerabilities.
OpenAI Evals★ 19.8kA framework and open registry for building and running evaluations of LLMs and LLM-based systems, including prompt chains and tool-using agents.
DeepEval★ 19.5kAn open-source Python framework that tests LLM apps like unit tests, with 50+ metrics for RAG, agents, chatbots, and safety, and a Pytest integration for CI/CD.
Ragas★ 16.3kAn evaluation toolkit focused on retrieval-augmented generation that scores answer faithfulness, context precision/recall, and relevancy, often without needing ground-truth labels.
Arize Phoenix★ 12.3kAn open-source observability and evaluation tool for tracing LLM and agent behavior, running evals on traces, and troubleshooting issues in development and production.
SkillEvaluator★ 2.2kTest whether an agent skill is well-formed, original and actually helps the agent