# LiteLLM

> Call 100+ LLM providers through one OpenAI-compatible API

- Category: Evaluation & Production Ops / Gateways & Routing
- Language: Python
- Repository: https://github.com/BerriAI/litellm
- Website: https://www.litellm.ai/
- Canonical page: https://ai-tldr.dev/tools/litellm/
- Updated: 2026-10-21

## In short

LiteLLM is an open-source AI gateway that gives you a single, unified interface to call 100+ LLM providers, including OpenAI, Anthropic, Gemini, Bedrock, and Azure, all using the OpenAI request format. It sits in the Gateways & Routing category of the Evaluation & Production Ops tools and is open-source, written in Python.

## Overview

LiteLLM is an open-source AI gateway that gives you a single, unified interface to call 100+ LLM providers, including OpenAI, Anthropic, Gemini, Bedrock, and Azure, all using the OpenAI request format. Instead of juggling a different SDK, auth pattern, and error type for every model, you write your code once and switch providers by changing the model name.

You can use it two ways. As a Python SDK, you import the `completion` function and call any model directly from your application. As an AI Gateway (proxy server), you deploy it as a central service that your whole team points at, with virtual keys, spend tracking, and load balancing handled in one place.

As an LLM gateway, LiteLLM sits between your apps and the model providers. It fits teams that want to standardize how they reach models, keep client code OpenAI-compatible, and add cost tracking, budgets, and fallbacks without rewriting each integration.

## What it does

- One unified API for 100+ LLMs, so you avoid provider-specific SDKs
- Drop-in OpenAI compatibility — swap providers by changing the model string, not your code
- Proxy server (AI Gateway) with virtual keys, spend tracking, guardrails, and load balancing
- Admin dashboard for managing keys and monitoring usage out of the box
- Supports many endpoint types: chat/completions, responses, embeddings, images, audio, batches, and rerank
- Can invoke A2A agents (LangGraph, Vertex AI Agent Engine, Bedrock AgentCore, Pydantic AI) via SDK or gateway

## Getting started

LiteLLM works as a Python SDK for direct calls or as a proxy server for your whole team. Pick one to start.

1. Install the Python SDK

   Add LiteLLM to your project.

```bash
uv add litellm
```

2. Make your first call

   Set the provider API keys you need, then call any model with the OpenAI-format `completion` function. Switch providers by changing the model string.

```python
from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])

# Anthropic
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])
```

3. Or run the AI Gateway (proxy server)

   Install the proxy extra and start it pointed at a model. It serves an OpenAI-compatible endpoint on port 4000.

```bash
uv tool install 'litellm[proxy]'
litellm --model gpt-4o
```

4. Call the gateway with the OpenAI client

   Point any OpenAI client at the local proxy base URL to route requests through LiteLLM.

```python
import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

## When to use it

- Build an app that can switch between OpenAI, Anthropic, and Gemini without rewriting client code
- Run a central gateway so a team shares one endpoint with virtual keys and per-team spend tracking
- Add fallbacks and load balancing across providers to keep requests flowing when one model is down
- Standardize calls to many endpoint types (chat, embeddings, images, audio, rerank) behind one OpenAI-format API

## Version history

- 2026-10-21 v1.107.0: Security incident: the release pipeline was compromised, and litellm 1.107.0 on PyPI plus the v1.107.0, latest, main-latest and main-stable Docker Hub and GHCR images shipped malware that steals credentials and loads remote-code-execution tooling. LiteLLM advises pinning 1.106.2, rebuilding affected environments and rotating every secret the host could reach.
- 2026-09-22 v1.102.0: post_call guardrail pipelines now execute on streaming responses, rewriting text and tool calls mid-stream, with legacy post-call hooks running as steps in that pipeline. Streaming latency routing ranks by raw time-to-first-token and adds percentile-based selection, an OCR core arrives with Mistral, Azure, Vertex and Reducto adapters, and an in-container pgbouncer pools connections across workers.
- 2026-09-15 v1.101.0: A 1,902-commit release built around routing: heuristic v2 complexity routing, a classifier circuit breaker, auto-escalation for stalled router tasks, and opt-in modality-based routing for image requests. The native MCP Gateway gains semantic tool search, cost tracking learns time-based off-peak pricing, and OpenAI calls can use OIDC token exchange.
- 2026-09-06 v1.100.0: LiteLLM v1.100.0 adds native Vertex AI Interactions API support, Grounding with Bing Search as a search provider, and day-0 routing for gemini-3.5-transcribe. Budgets can now be shared across a model access group, backed by a new per-window spend table, and prompt_token_calculator was deleted from utils.
- 2026-08-23 v1.98.0: LiteLLM v1.98.0 adds provisioned-throughput billing (flat cost for reserved capacity instead of per-token), shadow evals for the auto-router, and a per-key prompt caching switch.