RagaAI Catalyst

// Overview

RagaAI Catalyst is a Python SDK for managing and optimising LLM projects. Rather than covering a single slice of the lifecycle, it bundles project management, dataset management, evaluation, trace management, prompt management, synthetic data generation and guardrail management behind one client, so the same SDK that records what your application did also scores it and gates it.

Its agentic tracing module is aimed at multi-agent systems, where a failure is rarely traceable to one call. It tracks LLM interactions and token usage, tool utilisation and execution patterns, network activity and API calls, user interactions and feedback, and the agent's decision-making — with utilities for cost tracking, performance monitoring and debugging agent behaviour.

You authenticate with an access key and secret key generated from your profile settings, then create a project and attach traces, datasets and metrics to it. Evaluation works by declaring metrics such as Faithfulness or Hallucination against a schema mapping of your dataset columns, running them as an experiment, and reading back status and results; `append_metrics` re-scores only rows added since the last run.

// What it does

• Agentic tracing across LLM calls, tool use, network activity, user feedback and agent decisions, with cost and performance tracking
• Trace recording either as a context manager (`with tracer():`) or explicit `tracer.start()` / `tracer.stop()` calls, plus upload-status checks
• Metric evaluation (Faithfulness, Hallucination and others) declared with per-metric model, provider and threshold config against a dataset schema mapping
• Dataset management including creating datasets from CSV with explicit schema mapping
• Prompt management, synthetic data generation, guardrail management and red-teaming in the same SDK
• Auto-instrumentation via `init_tracing`, and `trace_llm` / `trace_tool` / `trace_agent` decorators for manual spans

// Getting started

Install the SDK, authenticate with keys generated from your profile settings, then create a project and start recording traces against it. Authentication is required before any other operation.

pip install ragaai-catalyst

from ragaai_catalyst import RagaAICatalyst

catalyst = RagaAICatalyst(
    access_key="YOUR_ACCESS_KEY",
    secret_key="YOUR_SECRET_KEY",
    base_url="BASE_URL"
)

project = catalyst.create_project(
    project_name="Test-RAG-App-1",
    usecase="Chatbot"
)

projects = catalyst.list_projects()
print(projects)

from ragaai_catalyst import Tracer

tracer = Tracer(
    project_name="Test-RAG-App-1",
    dataset_name="tracer_dataset_name",
    tracer_type="tracer_type"
)

with tracer():
    # Your code here
    ...

from ragaai_catalyst import Tracer, init_tracing

tracer = Tracer(
    project_name=agentic_tracing_project_name,
    dataset_name="agentic_tracing_dataset_name",
    tracer_type="Agentic",
)

init_tracing(catalyst=catalyst, tracer=tracer)

from ragaai_catalyst import Evaluation

evaluation = Evaluation(
    project_name="Test-RAG-App-1",
    dataset_name="MyDataset",
)

schema_mapping = {
    'Query': 'prompt',
    'response': 'response',
    'Context': 'context',
    'expectedResponse': 'expected_response'
}

evaluation.add_metrics(metrics=[
    {"name": "Faithfulness",
     "config": {"model": "gpt-4o-mini", "provider": "openai", "threshold": {"gte": 0.232323}},
     "column_name": "Faithfulness_v1",
     "schema_mapping": schema_mapping},
])

print(evaluation.get_status())
print(evaluation.get_results())

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Debugging a multi-agent system by tracing which tools ran, what the LLM decided, and where the tokens and cost went
• Scoring a RAG application for faithfulness and hallucination against a versioned dataset before shipping a change
• Keeping tracing, evaluation, prompt management and guardrails in one SDK instead of stitching several vendors together
• Re-scoring only newly added dataset rows with `append_metrics` instead of re-running a whole experiment

// How RagaAI Catalyst compares

RagaAI Catalyst alongside other open-source observability & llmops tools AI/TLDR tracks, ranked by GitHub stars.