What is SkillEvaluator?
SkillEvaluator is Test whether an agent skill is well-formed, original and actually helps the agent. Writing one is easy; knowing whether it is safe, whether it repeats guidance the agent already has, and whether it makes the agent any better at the task is not. It sits in the Evaluation & Red-Teaming category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0.
Overview
Agent skills are folders of instructions and supporting files, defined by the Agent Skills specification, that coding agents load on demand. Writing one is easy; knowing whether it is safe, whether it repeats guidance the agent already has, and whether it makes the agent any better at the task is not. SkillEvaluator is NVIDIA's open-source answer: it runs quality checks on a skill first, then compares agent runs with and without the skill. It is part of the NVIDIA Verified Skills pipeline.

The work is split into three independent tiers. Tier 1 (Validation) asks whether the skill is safe and well-formed, with schema, quality, PII and security checks; it fetches pinned SkillSpector and Gitleaks releases for scanning, and Semgrep is optional. Tier 2 (Deduplication) uses embeddings to find guidance that overlaps within a skill and, given a catalog, across skills. Tier 3 (Live Evaluation) runs a supported agent in a sandbox — Docker by default, powered by the open-source Harbor framework — on the same tasks with and without the skill, and reports the difference as Skill Lift.
Each tier is its own command, so a keyless static quality check can run with no API key, GPU or Docker, while a single `validate` command runs all three. Tier 3 can generate an evaluation dataset grounded in the skill's own evaluation guidance, a prompt file of tasks and outcomes, or retained traces; reports come out in the terminal and as HTML, JSON and Markdown, with budgets on tokens, judge calls and cost.
SkillEvaluator also ships ready-made GitHub Actions and GitLab CI templates, plus a `skillevaluator ci init` command that sets them up, so changed skills in a repository get checked on every pull request. It is Apache-2.0 licensed, written in Python (3.12–3.14), published on PyPI as `skillevaluator`, and accompanied by an arXiv paper. NVIDIA marks its support level as experimental and community-supported.
What it does
- Tier 1 validation: schema, quality, PII and security checks, with SkillSpector and Gitleaks scanners fetched automatically
- Keyless `quality-check` that returns a 0–100 quality score and findings with no API key, GPU or Docker
- Tier 2 deduplication: embedding-based overlap detection within a skill and across a skill catalog
- Tier 3 live evaluation: sandboxed A/B agent runs with and without the skill, reported as Skill Lift with pass@k
- Evaluation-dataset generation grounded in author guidance, a prompt file or production traces, with human review before expectations are promoted
- Token, judge-call, attempt and cost budgets, plus version comparison and reusable no-skill baselines
- GitHub Actions and GitLab CI templates, set up by `skillevaluator ci init`
Getting started
The quickest path uses the bundled `text-analyzer` example skill and needs only Git and uv — uv provisions Python 3.13 and the dependencies. Replace the example path with your own skill directory once it works.
Get the example
Clone the repository; the example skill ships inside it.
git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluatorRun a quality check without an LLM
No API key, scanner, GPU or Docker is needed. The example scores 84.5/100 (B) and saves a JSON report; the passing threshold defaults to 70.
uv run --python 3.13 skillevaluator quality-check \
src/skillevaluator/tier3/reference_skills/text-analyzer \
-r cli -r json -o reports/first-qualityAdd model-backed rubric scores
With an NVIDIA Build API key (free prototyping for eligible NVIDIA Developer Program members), the rubric check sends the skill's documentation to a model and saves its feedback. The same key covers Tier 2 embeddings and supported Tier 3 agents.
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
uv run --python 3.13 --extra llm skillevaluator rubric-eval \
src/skillevaluator/tier3/reference_skills/text-analyzer \
-r cli -r json -o reports/first-buildRun the three tiers
Each tier is independent and needs its own extra. `doctor` checks readiness; `validate` runs all tiers (Tier 3 needs a provider, a supported agent and a running sandbox). Use `--tiers 1,2` to skip live evaluation.
uv run --extra security skillevaluator tier1 ./my-skill --no-llm
uv run --extra tier2 skillevaluator tier2 ./my-skill
uv run --extra tier3 skillevaluator tier3 ./my-skill
# or everything at once
uv run --python 3.13 --extra all skillevaluator doctor --env-mode docker
uv run --python 3.13 --extra all skillevaluator validate ./my-skill
Check skills in CI
Inside a checked-out GitHub repository, the guided setup lets you choose checks and opens a setup pull request. Ready-made GitHub and GitLab templates are in the repository's `ci/bundles` folder if you prefer to copy one by hand.
pip install --upgrade skillevaluator
skillevaluator ci initCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Gate a team's agent-skill repository so every changed skill passes quality and security checks before merge
- Prove a new skill earns its context-window cost by measuring agent results with and without it
- Find skills in a large catalog that repeat each other's guidance before publishing them
- Compare two versions of a skill under a fixed token and cost budget

How SkillEvaluator compares
SkillEvaluator alongside other open-source evaluation & red-teaming tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| Strix | ★ 75.9k | Strix runs autonomous AI agents that act like hackers, dynamically running your code to find vulnerabilities and validate them with real proof-of-concepts. |
| iFixAi | ★ 46k | An agent auditor that runs up to 60 inspections across fabrication, manipulation, deception, unpredictability and opacity, and returns an A–F grade judged by a model from a different vendor. |
| promptfoo | ★ 27.3k | A developer-first CLI and library for testing and comparing prompts and models, with red-teaming probes for prompt injection, PII leaks, and other vulnerabilities. |
| OpenAI Evals | ★ 19.8k | A framework and open registry for building and running evaluations of LLMs and LLM-based systems, including prompt chains and tool-using agents. |
| DeepEval | ★ 19.5k | An open-source Python framework that tests LLM apps like unit tests, with 50+ metrics for RAG, agents, chatbots, and safety, and a Pytest integration for CI/CD. |
| Ragas | ★ 16.3k | An evaluation toolkit focused on retrieval-augmented generation that scores answer faithfulness, context precision/recall, and relevancy, often without needing ground-truth labels. |
| Arize Phoenix | ★ 12.3k | An open-source observability and evaluation tool for tracing LLM and agent behavior, running evals on traces, and troubleshooting issues in development and production. |
| SkillEvaluator | ★ 2.2k | Test whether an agent skill is well-formed, original and actually helps the agent |
