Giskard

// Overview

Giskard is an open-source Python library for testing and evaluating agentic systems. It can wrap an LLM, a black-box agent, or a multi-step pipeline, and check how it behaves on inputs where the same prompt may produce different valid answers.

The v3 release is a modular set of focused packages. `giskard-checks` (Beta) handles scenario-based evals and LLM-as-judge assessments, `giskard-scan` (in progress) does adversarial red teaming, and `giskard-rag` (planned) covers RAG evaluation and synthetic test data. Each package only pulls in the dependencies it needs.

It fits the evaluation and testing category for teams who need to catch regressions, validate RAG groundedness, enforce safety rules, and test full multi-turn conversations rather than single exchanges.

// What it does

• Scenario API for writing evals against non-deterministic LLM outputs, including multi-turn agent conversations
• Built-in checks: string matching, comparisons, regex, semantic similarity, and LLM-as-judge (Groundedness, Conformity, LLMJudge)
• Vulnerability scanner that auto-generates adversarial test suites from a plain-language description of your agent
• Red-teaming coverage for prompt injection, harmful content, stereotypes, and misinformation across OWASP LLM Top-10 categories
• Async-first, lightweight design that drops heavy dependencies for better efficiency
• Extensible with custom ScenarioGenerator instances for your own adversarial tests

// Getting started

Install Giskard with pip, then write a scenario that runs your model and checks its output. Giskard requires Python 3.12+.

pip install giskard

from openai import OpenAI
from giskard.checks import Scenario, Groundedness

client = OpenAI()

def get_answer(inputs: str) -> str:
    response = client.chat.completions.create(
        model="gpt-5-mini",
        messages=[{"role": "user", "content": inputs}],
    )
    return response.choices[0].message.content

scenario = (
    Scenario("test_dynamic_output")
    .interact(
        inputs="What is the capital of France?",
        outputs=get_answer,
    )
    .check(
        Groundedness(
            name="answer is grounded",
            context="France is a country in Western Europe. Its capital is Paris.",
        )
    )
)

result = await scenario.run()
result.print_report()

import asyncio
from giskard.scan import vulnerability_scan

async def main():
    await vulnerability_scan(
        target=my_agent,
        description="A customer support chatbot for an e-commerce platform.",
        languages=["en"],
    )

asyncio.run(main())

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Catch regressions in an LLM app by re-running scenario checks after each change
• Validate that a RAG system's answers stay grounded in retrieved context
• Red-team a customer-support agent for prompt injection and harmful content before launch
• Test full multi-turn agent conversations rather than single prompt-response pairs

// How Giskard compares

Giskard alongside other open-source evaluation & red-teaming tools AI/TLDR tracks, ranked by GitHub stars.