# RagaAI Catalyst

> A Python SDK that traces, evaluates and guards LLM and agent applications from one platform

- Category: Evaluation & Production Ops / Observability & LLMOps
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/raga-ai-hub/RagaAI-Catalyst
- Website: https://catalyst.raga.ai/
- Canonical page: https://ai-tldr.dev/tools/ragaai-catalyst/

## In short

RagaAI Catalyst is a Python SDK for managing and optimising LLM projects. It sits in the Observability & LLMOps category of the Evaluation & Production Ops tools and is open-source, written in Python and licensed Apache-2.0. To get started: install RagaAI Catalyst.

## Overview

RagaAI Catalyst is a Python SDK for managing and optimising LLM projects. Rather than covering a single slice of the lifecycle, it bundles project management, dataset management, evaluation, trace management, prompt management, synthetic data generation and guardrail management behind one client, so the same SDK that records what your application did also scores it and gates it.

Its agentic tracing module is aimed at multi-agent systems, where a failure is rarely traceable to one call. It tracks LLM interactions and token usage, tool utilisation and execution patterns, network activity and API calls, user interactions and feedback, and the agent's decision-making — with utilities for cost tracking, performance monitoring and debugging agent behaviour.

You authenticate with an access key and secret key generated from your profile settings, then create a project and attach traces, datasets and metrics to it. Evaluation works by declaring metrics such as Faithfulness or Hallucination against a schema mapping of your dataset columns, running them as an experiment, and reading back status and results; `append_metrics` re-scores only rows added since the last run.

## What it does

- Agentic tracing across LLM calls, tool use, network activity, user feedback and agent decisions, with cost and performance tracking
- Trace recording either as a context manager (`with tracer():`) or explicit `tracer.start()` / `tracer.stop()` calls, plus upload-status checks
- Metric evaluation (Faithfulness, Hallucination and others) declared with per-metric model, provider and threshold config against a dataset schema mapping
- Dataset management including creating datasets from CSV with explicit schema mapping
- Prompt management, synthetic data generation, guardrail management and red-teaming in the same SDK
- Auto-instrumentation via `init_tracing`, and `trace_llm` / `trace_tool` / `trace_agent` decorators for manual spans

## Getting started

Install the SDK, authenticate with keys generated from your profile settings, then create a project and start recording traces against it. Authentication is required before any other operation.

1. Install RagaAI Catalyst

   Install the package from PyPI.

```bash
pip install ragaai-catalyst
```

2. Authenticate

   Generate access and secret keys from your profile settings — Authenticate, then Generate New Key — and pass them to the client (or set them as environment variables).

```python
from ragaai_catalyst import RagaAICatalyst

catalyst = RagaAICatalyst(
    access_key="YOUR_ACCESS_KEY",
    secret_key="YOUR_SECRET_KEY",
    base_url="BASE_URL"
)
```

3. Create a project

   Projects are the container everything else hangs off — datasets, experiments and traces.

```python
project = catalyst.create_project(
    project_name="Test-RAG-App-1",
    usecase="Chatbot"
)

projects = catalyst.list_projects()
print(projects)
```

4. Record traces

   Create a tracer for the project and dataset, then record either with a context manager or with explicit start/stop calls.

```python
from ragaai_catalyst import Tracer

tracer = Tracer(
    project_name="Test-RAG-App-1",
    dataset_name="tracer_dataset_name",
    tracer_type="tracer_type"
)

with tracer():
    # Your code here
    ...
```

5. Turn on agentic tracing

   For agent systems, create the tracer with `tracer_type="Agentic"` and enable auto-instrumentation so LLM calls, tools and agent steps are captured without manual spans.

```python
from ragaai_catalyst import Tracer, init_tracing

tracer = Tracer(
    project_name=agentic_tracing_project_name,
    dataset_name="agentic_tracing_dataset_name",
    tracer_type="Agentic",
)

init_tracing(catalyst=catalyst, tracer=tracer)
```

6. Score a dataset with metrics

   Declare metrics against a schema mapping of your dataset's columns, then read back experiment status and results.

```python
from ragaai_catalyst import Evaluation

evaluation = Evaluation(
    project_name="Test-RAG-App-1",
    dataset_name="MyDataset",
)

schema_mapping = {
    'Query': 'prompt',
    'response': 'response',
    'Context': 'context',
    'expectedResponse': 'expected_response'
}

evaluation.add_metrics(metrics=[
    {"name": "Faithfulness",
     "config": {"model": "gpt-4o-mini", "provider": "openai", "threshold": {"gte": 0.232323}},
     "column_name": "Faithfulness_v1",
     "schema_mapping": schema_mapping},
])

print(evaluation.get_status())
print(evaluation.get_results())
```

## When to use it

- Debugging a multi-agent system by tracing which tools ran, what the LLM decided, and where the tokens and cost went
- Scoring a RAG application for faithfulness and hallucination against a versioned dataset before shipping a change
- Keeping tracing, evaluation, prompt management and guardrails in one SDK instead of stitching several vendors together
- Re-scoring only newly added dataset rows with `append_metrics` instead of re-running a whole experiment