LiteLLM

// Overview

LiteLLM is an open-source AI gateway that gives you a single, unified interface to call 100+ LLM providers, including OpenAI, Anthropic, Gemini, Bedrock, and Azure, all using the OpenAI request format. Instead of juggling a different SDK, auth pattern, and error type for every model, you write your code once and switch providers by changing the model name.

You can use it two ways. As a Python SDK, you import the `completion` function and call any model directly from your application. As an AI Gateway (proxy server), you deploy it as a central service that your whole team points at, with virtual keys, spend tracking, and load balancing handled in one place.

As an LLM gateway, LiteLLM sits between your apps and the model providers. It fits teams that want to standardize how they reach models, keep client code OpenAI-compatible, and add cost tracking, budgets, and fallbacks without rewriting each integration.

// What it does

• One unified API for 100+ LLMs, so you avoid provider-specific SDKs
• Drop-in OpenAI compatibility — swap providers by changing the model string, not your code
• Proxy server (AI Gateway) with virtual keys, spend tracking, guardrails, and load balancing
• Admin dashboard for managing keys and monitoring usage out of the box
• Supports many endpoint types: chat/completions, responses, embeddings, images, audio, batches, and rerank
• Can invoke A2A agents (LangGraph, Vertex AI Agent Engine, Bedrock AgentCore, Pydantic AI) via SDK or gateway

// Getting started

LiteLLM works as a Python SDK for direct calls or as a proxy server for your whole team. Pick one to start.

uv add litellm

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])

# Anthropic
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])

uv tool install 'litellm[proxy]'
litellm --model gpt-4o

import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Build an app that can switch between OpenAI, Anthropic, and Gemini without rewriting client code
• Run a central gateway so a team shares one endpoint with virtual keys and per-team spend tracking
• Add fallbacks and load balancing across providers to keep requests flowing when one model is down
• Standardize calls to many endpoint types (chat, embeddings, images, audio, rerank) behind one OpenAI-format API

// Version history

Every verified update to LiteLLM that AI/TLDR tracked, newest first — each links to our coverage and the official changeset.

1. 2026-09-15v1.101.0

   A 1,902-commit release built around routing: heuristic v2 complexity routing, a classifier circuit breaker, auto-escalation for stalled router tasks, and opt-in modality-based routing for image requests. The native MCP Gateway gains semantic tool search, cost tracking learns time-based off-peak pricing, and OpenAI calls can use OIDC token exchange.

   our coverage →changeset ↗
2. 2026-09-06v1.100.0

   LiteLLM v1.100.0 adds native Vertex AI Interactions API support, Grounding with Bing Search as a search provider, and day-0 routing for gemini-3.5-transcribe. Budgets can now be shared across a model access group, backed by a new per-window spend table, and prompt_token_calculator was deleted from utils.

   our coverage →changeset ↗
3. 2026-08-23v1.98.0

   LiteLLM v1.98.0 adds provisioned-throughput billing (flat cost for reserved capacity instead of per-token), shadow evals for the auto-router, and a per-key prompt caching switch.

   our coverage →changeset ↗

// LiteLLM in the news

1. 2026-09-15MAJORLiteLLM v1.101.0 — smarter complexity routing and a semantic MCP search
2. 2026-09-06MAJORLiteLLM v1.100.0 — Vertex AI Interactions API and shared team budgets
3. 2026-08-23MAJORLiteLLM v1.98.0 — reserved capacity gets flat-cost billing, not per-token
4. 2026-04-29MAJORLiteLLM CVE-2026-42208 — Critical SQL Injection in AI Gateway Exploited Within 36 Hours of Disclosure
5. 2026-04-25NOTABLEGoModel — Lightweight AI Gateway in Go, LiteLLM Alternative
6. 2026-04-15MAJORAnthropic MCP Design Flaw Enables RCE Across 150M+ Downloads
7. 2026-04-09MAJORMercor LiteLLM Supply-Chain Breach — 4TB Stolen, Meta Pauses AI Training Contracts

From the AI/TLDR release feed — every item is source-verified when it ships.

// How LiteLLM compares

LiteLLM alongside other open-source gateways & routing tools AI/TLDR tracks, ranked by GitHub stars.