Overview
Switchyard is NVIDIA's routing component for LLM traffic: it decides which model serves each individual call, rather than pinning an application to one model. It ships as an Apache-2.0 project under the NVIDIA-NeMo organisation with several algorithms — a staged router, an escalation router and classifier-based routers — and preserves native OpenAI and Anthropic API compatibility so the calling application does not have to change.
There are three ways to adopt it, and the repository presents them as self-contained paths. It can run inside a gateway you already operate: a native plugin for NeMo Relay loads a routes.toml deployment, and a routing plugin exists for LiteLLM's Router and proxy. It can be embedded as a library (pip install nemo-switchyard, or switchyard-libsy for Rust), in which case Switchyard picks the model and your harness still makes every call, leaving your transport, retries and credentials untouched. Or it can run as a standalone OpenAI- and Anthropic-compatible proxy in front of an agent — point Claude Code, Codex CLI or any SDK client at it and Switchyard decides per turn which model serves the request.
The project is explicitly pre-1.0 and rates its own components: switchyard-libsy is beta, switchyard-llm-client and switchyard-runner are alpha, and switchyard-server is labelled demo, for evaluation only and not production. APIs, configuration and routing behaviour can change between releases, so pin the version you integrate. The README also notes that the current library API is ahead of the published nemo-switchyard 0.2.0 on PyPI, and tells you to install from the repository until the next release.
What it does
- Per-call model selection across models and providers, with native OpenAI and Anthropic API compatibility preserved
- Several routing algorithms — staged, escalation and classifier-based — configured through a version-1 routes.toml
- Three integration paths: a NeMo Relay plugin, a LiteLLM Router/proxy plugin, or an embedded library
- Embedded mode where Switchyard only picks the model and your harness keeps ownership of transport, retries and credentials
- Python and Rust libraries (nemo-switchyard / switchyard-libsy) from the same codebase
- A standalone OpenAI- and Anthropic-compatible proxy for clients such as Claude Code or Codex CLI
- Published per-component stability ratings so you know which pieces are demo-only
Getting started
Pick the path that matches where routing should live. Note that the library API in the repository is newer than the 0.2.0 release on PyPI, so building from source is currently the documented route for the Step/LlmResponse API; a Rust toolchain is required.
Install the library from source
The published nemo-switchyard 0.2.0 wheel exposes an older LlmTarget-based interface; the README points at the repository until the next release.
pip install git+https://github.com/NVIDIA-NeMo/Switchyard.gitConstruct an algorithm in your harness
Target names are whatever your harness calls its models. The stage router is the one used in the repository's benchmark; random, llm_task_classifier and llm_classifier are built the same way.
from switchyard.libsy import LlmResponse, Step
from switchyard.libsy.algorithms import stage_routerOr run it as a standalone proxy
Rated demo/evaluation only. Point any OpenAI- or Anthropic-compatible client at the proxy and Switchyard decides per turn which model serves it.
cargo install --locked switchyard-server
switchyard-server --config routes.toml --port 4000Or load the plugin into NeMo Relay
Build and register the plugin, write the deployment to /etc/switchyard/routes.toml, point the plugin at it, then enable and validate. Requires NeMo Relay >=0.8.1,<0.9.0 and a Rust toolchain.
nemo-relay plugins enable nvidia.switchyard
nemo-relay plugins validate nvidia.switchyardOr use it from LiteLLM
A routing plugin for LiteLLM's Router and proxy lives in the repository's examples directory.
examples/litellm/README.mdCommands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Cut inference spend on an agent that sends every step to a frontier model when most steps do not need one
- Add per-call routing to an existing NeMo Relay or LiteLLM deployment without touching application code
- Embed model selection in your own gateway or harness while keeping control of the actual HTTP calls
- Benchmark cost against accuracy for different routing policies before committing to a model mix
How NeMo Switchyard compares
NeMo Switchyard alongside other open-source gateways & routing tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| RTK | ★ 79.1k | A single-binary Rust CLI proxy that filters, groups, truncates and dedupes the output of 100+ dev commands before a coding agent reads it, cutting token use by 60–90%. |
| Headroom | ★ 69.1k | Local context-compression layer that shrinks tool outputs, logs, files and RAG chunks before they reach the model, usable as a library, a drop-in proxy or an MCP server. |
| OmniRoute | ★ 61.9k | A local-first AI gateway that fronts hundreds of model providers with one OpenAI-compatible endpoint, quota-aware auto-fallback, many routing strategies and prompt compression. |
| LiteLLM | ★ 58.2k | A Python SDK and proxy server that gives one OpenAI-compatible API to 100+ LLM providers, with cost tracking, budgets, fallbacks, rate limiting, and an admin UI. |
| Apache APISIX | ★ 17.1k | A cloud-native API gateway whose AI plugins add multi-provider LLM proxying, load balancing, retries and fallbacks, token-based rate limiting, and content moderation. |
| Portkey AI Gateway | ★ 12.9k | An LLM gateway that routes calls to 100+ providers through one API and adds logging, tracing, caching, and fallbacks for production AI traffic. |
| Higress | ★ 9.3k | An AI-native API gateway built on Istio and Envoy that proxies and governs traffic to many LLM providers, with token rate limiting, caching, and MCP server hosting. |
| NeMo Switchyard | ★ 2.7k | Routes each LLM call to the cheapest model that can still do the job, without changing your agent |