NVIDIA-NeMo Switchyard: Rust Routing for LLM Traffic
NVIDIA-NeMo’s Switchyard is a pre-alpha Rust proxy and library that routes LLM traffic across providers while preserving native OpenAI and Anthropic API formats.
It translates requests and responses, exposes Prometheus metrics, and supports random, classifier, stage-based, and escalation routing. The library separates routing decisions from the HTTP/model client, so it can embed in an existing gateway or agent runtime; the standalone proxy can target vLLM, NIM, Ollama, or other compatible endpoints. APIs and algorithms are still expected to change before 1.0.