AgentOps

// Overview

AgentOps is an open-source SDK and platform that helps developers build, evaluate, and monitor AI agents. You add it to a Python program to capture analytics on your LLM calls, replay agent sessions step by step, and track how much each run costs.

It is aimed at developers building agentic apps who need visibility into what their agents actually do at runtime. Instead of guessing why an agent failed or how much a workflow spent, you get execution graphs, input/output recording, and cost tracking across foundation-model providers.

As an LLM observability tool, AgentOps sits alongside your agent framework rather than replacing it. It offers native integrations with CrewAI, LangChain, LangGraph, AG2 (AutoGen), the OpenAI Agents SDK, and others, and can run as a hosted dashboard or be self-hosted on your own cloud.

// What it does

• Replay analytics and debugging with step-by-step agent execution graphs
• LLM cost management that tracks spend across foundation-model providers
• Session replays initialized in two lines of code
• Native integrations with CrewAI, AG2 (AutoGen), Agno, LangGraph, LangChain, and the OpenAI Agents SDK
• Decorators (@session, @agent, @operation, @task, @workflow) for tracing agents, tools, and functions
• Self-hosting option to run the full dashboard and API backend on your own cloud

// Getting started

Install the SDK, then initialize the client at the start of your program to capture analytics on all your LLM calls.

pip install agentops

import agentops

# Beginning of your program (i.e. main.py, __init__.py)
agentops.init(<INSERT YOUR API KEY HERE>)

...

# End of program
agentops.end_session('Success')

from agentops.sdk.decorators import session, agent, operation

@agent
class MyAgent:
    @operation
    def main_operation(self):
        return "done"

@session
def my_session():
    agent = MyAgent()
    return agent.main_operation()

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Debugging why a multi-step agent failed by replaying its execution graph step by step
• Tracking and controlling LLM spend across foundation-model providers in production
• Monitoring agents built with CrewAI, LangChain, LangGraph, or the OpenAI Agents SDK through native integrations
• Self-hosting the AgentOps dashboard and API backend to keep agent telemetry on your own cloud

// How AgentOps compares

AgentOps alongside other open-source observability & llmops tools AI/TLDR tracks, ranked by GitHub stars.