Overview
Switchyard is NVIDIA's routing component for LLM traffic: it decides which model serves each individual call, rather than pinning an application to one model. It ships as an Apache-2.0 project under the NVIDIA-NeMo organisation with several algorithms — a staged router, an escalation router and classifier-based routers — and preserves native OpenAI and Anthropic API compatibility so the calling application does not have to change.
There are three ways to adopt it, and the repository presents them as self-contained paths. It can run inside a gateway you already operate: a native plugin for NeMo Relay loads a routes.toml deployment, and a routing plugin exists for LiteLLM's Router and proxy. It can be embedded as a library (pip install nemo-switchyard, or switchyard-libsy for Rust), in which case Switchyard picks the model and your harness still makes every call, leaving your transport, retries and credentials untouched. Or it can run as a standalone OpenAI- and Anthropic-compatible proxy in front of an agent — point Claude Code, Codex CLI or any SDK client at it and Switchyard decides per turn which model serves the request.
The project is explicitly pre-1.0 and rates its own components: switchyard-libsy is beta, switchyard-llm-client and switchyard-runner are alpha, and switchyard-server is labelled demo, for evaluation only and not production. APIs, configuration and routing behaviour can change between releases, so pin the version you integrate. The README also notes that the current library API is ahead of the published nemo-switchyard 0.2.0 on PyPI, and tells you to install from the repository until the next release.
What it does
- Per-call model selection across models and providers, with native OpenAI and Anthropic API compatibility preserved
- Several routing algorithms — staged, escalation and classifier-based — configured through a version-1 routes.toml
- Three integration paths: a NeMo Relay plugin, a LiteLLM Router/proxy plugin, or an embedded library
- Embedded mode where Switchyard only picks the model and your harness keeps ownership of transport, retries and credentials
- Python and Rust libraries (nemo-switchyard / switchyard-libsy) from the same codebase
- A standalone OpenAI- and Anthropic-compatible proxy for clients such as Claude Code or Codex CLI
- Published per-component stability ratings so you know which pieces are demo-only
Getting started
Pick the path that matches where routing should live. Note that the library API in the repository is newer than the 0.2.0 release on PyPI, so building from source is currently the documented route for the Step/LlmResponse API; a Rust toolchain is required.
Install the library from source
The published nemo-switchyard 0.2.0 wheel exposes an older LlmTarget-based interface; the README points at the repository until the next release.
pip install git+https://github.com/NVIDIA-NeMo/Switchyard.gitConstruct an algorithm in your harness
Target names are whatever your harness calls its models. The stage router is the one used in the repository's benchmark; random, llm_task_classifier and llm_classifier are built the same way.
from switchyard.libsy import LlmResponse, Step
from switchyard.libsy.algorithms import stage_routerOr run it as a standalone proxy
Rated demo/evaluation only. Point any OpenAI- or Anthropic-compatible client at the proxy and Switchyard decides per turn which model serves it.
cargo install --locked switchyard-server
switchyard-server --config routes.toml --port 4000Or load the plugin into NeMo Relay
Build and register the plugin, write the deployment to /etc/switchyard/routes.toml, point the plugin at it, then enable and validate. Requires NeMo Relay >=0.8.1,<0.9.0 and a Rust toolchain.
nemo-relay plugins enable nvidia.switchyard
nemo-relay plugins validate nvidia.switchyardOr use it from LiteLLM
A routing plugin for LiteLLM's Router and proxy lives in the repository's examples directory.
examples/litellm/README.mdCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Cut inference spend on an agent that sends every step to a frontier model when most steps do not need one
- Add per-call routing to an existing NeMo Relay or LiteLLM deployment without touching application code
- Embed model selection in your own gateway or harness while keeping control of the actual HTTP calls
- Benchmark cost against accuracy for different routing policies before committing to a model mix
How NeMo Switchyard compares
NeMo Switchyard alongside other open-source gateways & routing tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| OmniRoute | ★ 79.9k | A local-first AI gateway that fronts hundreds of model providers with one OpenAI-compatible endpoint, quota-aware auto-fallback, many routing strategies and prompt compression. |
| Free Claude Code | ★ 62.7k | A local proxy that points Claude Code, Codex, Pi, OpenCode, Cline and other agent CLIs at 50 free, paid, subscription or local model providers, with automatic failover and an admin UI. |
| LiteLLM | ★ 61.8k | A Python SDK and proxy server that gives one OpenAI-compatible API to 100+ LLM providers, with cost tracking, budgets, fallbacks, rate limiting, and an admin UI. |
| Kong Gateway | ★ 44.3k | A Lua/NGINX API gateway whose AI Gateway plugins put one universal endpoint in front of OpenAI, Anthropic, Gemini, Bedrock and other providers, with semantic routing, caching and MCP traffic governance. |
| 9Router | ★ 31.2k | Local OpenAI-compatible router for coding CLIs that tracks per-provider quota, falls back from subscription to cheap to free models, and compresses tool results to cut tokens. |
| magpie | ★ 19.5k | Menu bar app and local gateway that lets Claude Code, Codex, Gemini CLI and dozens of other coding agents run on any provider's model, with routing groups that fail over across accounts when a quota runs out. |
| OpenCodex | ★ 17.8k | Local proxy that translates the Codex Responses API to 40+ providers, so Codex CLI, Claude Code and Grok Build can run on any model, with routing combos, failover and account pooling. |
| NeMo Switchyard | ★ 3.4k | Routes each LLM call to the cheapest model that can still do the job, without changing your agent |