# DeepEval

> Unit-test your LLM apps with 50+ evaluation metrics

- Category: Evaluation & Production Ops / Evaluation & Red-Teaming
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/confident-ai/deepeval
- Website: https://deepeval.com
- Canonical page: https://ai-tldr.dev/tools/deepeval/

## In short

DeepEval is an open-source Python framework for evaluating large-language-model systems. It works much like Pytest, but is built for unit-testing LLM apps: you write test cases, attach metrics, and assert that your model's output meets a quality threshold. It sits in the Evaluation & Red-Teaming category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

DeepEval is an open-source Python framework for evaluating large-language-model systems. It works much like Pytest, but is built for unit-testing LLM apps: you write test cases, attach metrics, and assert that your model's output meets a quality threshold.

It ships with a large set of ready-to-use metrics powered by LLM-as-a-judge and other NLP models that run locally on your machine, including G-Eval, answer relevancy, faithfulness, hallucination, and task completion. Whether you build RAG pipelines, agents, or chatbots, with LangChain or the OpenAI SDK, you can score the parts that matter for each.

It fits in the evaluation and testing layer of an AI stack. Because tests run through the deepeval CLI, you can wire them into CI/CD to catch quality regressions, compare prompts and models, and decide whether a change is safe to ship.

## What it does

- 50+ ready-to-use metrics with explanations, including G-Eval, DAG, answer relevancy, faithfulness, hallucination, and task completion
- Pytest-style API: write test cases with LLMTestCase and assert them against metrics
- Runs metrics locally using LLM-as-a-judge and NLP models, with any LLM of your choice as the judge
- Dedicated metric groups for RAG, agentic, and multi-turn (chatbot) use cases
- CI/CD friendly via the deepeval test run CLI command
- Works with apps built on LangChain or OpenAI, so you can compare prompts, models, and architectures

## Getting started

Install DeepEval, set your judge model's API key, write a test case, and run it with the deepeval CLI.

1. Install DeepEval

   Install the package from PyPI.

```bash
pip install -U deepeval
```

2. Set your judge model key

   Many metrics use an LLM as a judge. Set the OpenAI API key (or configure another model) before running.

```bash
export OPENAI_API_KEY="..."
```

3. Write a test case

   Create a file (for example test_chatbot.py) with a metric and an LLMTestCase, then assert it.

```python
import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams

def test_case():
    correctness_metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
        threshold=0.5
    )
    test_case = LLMTestCase(
        input="What if these shoes don't fit?",
        actual_output="You have 30 days to get a full refund at no extra cost.",
        expected_output="We offer a 30-day full refund at no extra costs.",
        retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
    )
    assert_test(test_case, [correctness_metric])
```

4. Run the tests

   Use the deepeval CLI to run your test file.

```bash
deepeval test run test_chatbot.py
```

## When to use it

- Add quality checks to CI/CD so prompt or model changes that lower output quality fail the build
- Evaluate a RAG pipeline for answer relevancy, faithfulness, and contextual recall
- Score agent behavior such as task completion and tool correctness
- Compare prompts and models (for example moving from OpenAI to Claude) before shipping a change