Evidently

// Overview

Evidently is an open-source Python library for evaluating, testing, and monitoring machine learning and LLM systems across their whole lifecycle, from early experiments to production. It works with both tabular and text data and supports predictive tasks like classification as well as generative tasks like RAG.

It is built for data scientists and ML engineers who need to measure output quality, catch data drift, and check regressions over time. The library ships with over 100 built-in metrics, ranging from data drift detection to LLM-as-a-judge evaluators, and lets you add custom metrics through a Python interface.

As an LLM observability and monitoring tool, Evidently is modular. You can start with one-off Reports and Test Suites, then move up to a self-hosted Monitoring UI that tracks metrics and test results over time. Its open architecture lets you export results as JSON, HTML, or a Python dictionary and plug into existing tools.

// What it does

• Over 100 built-in metrics covering data drift, ML quality, and LLM evaluation
• Reports that compute and summarize data, ML, and LLM quality checks, viewable in Python or exportable to JSON, HTML, or a dictionary
• Test Suites that add pass/fail conditions to any Report for regression testing, CI/CD checks, and data validation
• Row-level descriptors such as Sentiment, TextLength, and Contains for evaluating text outputs
• Self-hostable Monitoring UI to visualize metrics and test results over time
• Python interface for custom metrics and support for LLM-as-a-judge evaluators

// Getting started

Install Evidently from PyPI, then run a small text-evaluation Report on a toy dataset.

pip install evidently

import pandas as pd
from evidently import Report
from evidently import Dataset, DataDefinition
from evidently.descriptors import Sentiment, TextLength, Contains
from evidently.presets import TextEvals

eval_df = pd.DataFrame([
    ["What is the capital of Japan?", "The capital of Japan is Tokyo."],
    ["Who painted the Mona Lisa?", "Leonardo da Vinci."],
    ["Can you write an essay?", "I'm sorry, but I can't assist with homework."]],
    columns=["question", "answer"])

eval_dataset = Dataset.from_pandas(pd.DataFrame(eval_df),
    data_definition=DataDefinition(),
    descriptors=[
        Sentiment("answer", alias="Sentiment"),
        TextLength("answer", alias="Length"),
        Contains("answer", items=['sorry', 'apologize'], mode="any", alias="Denials")
    ])

report = Report([
    TextEvals()
])

my_eval = report.run(eval_dataset)
my_eval
# my_eval.json()
# my_eval.dict()

uv run --with evidently evidently ui --demo-projects all

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Run LLM output evaluations such as sentiment, length, and denial checks during prompt and RAG experiments
• Detect data drift by comparing current production data against a reference dataset
• Add pass/fail Test Suites to CI/CD pipelines for ML regression testing and data validation
• Self-host a monitoring dashboard to track model and LLM quality metrics over time

// How Evidently compares

Evidently alongside other open-source observability & llmops tools AI/TLDR tracks, ranked by GitHub stars.