# NVIDIA ModelExpress — Rust sidecar makes vLLM cold-loads 48× faster

> NVIDIA's open-source Rust sidecar loads model weights into GPUs up to 48× faster than a cold Hugging Face pull.

- Org: NVIDIA
- Date: 2026-07-24
- Category: tool, repo
- Source: https://developer.nvidia.com/blog/modelexpress-distributing-model-artifacts-at-the-speed-of-light/
- Canonical page: https://ai-tldr.dev/releases/nvidia-modelexpress/

NVIDIA open-sources ModelExpress, an Apache-2.0 Rust component that P2P-transfers model weights next to vLLM, SGLang, TensorRT-LLM, and Dynamo. It reports a 48× cold-load speedup on DeepSeek V4 Pro.

## What it is

ModelExpress is an Apache-2.0 Rust component NVIDIA open-sourced on July 24, 2026 that sits next to an inference server and speeds up model loading. It caches weight artifacts, routes them across GPUs, and hands them to vLLM, SGLang, TensorRT-LLM, or NVIDIA Dynamo without changing the serving stack.

## How it works

Given a request to load a model, ModelExpress picks the fastest path available — GPU-to-GPU P2P over RDMA, GPUDirect Storage off a local NVMe cache, a host-staged POSIX read, or a fresh Hugging Face download — then streams the checkpoint into GPU memory. A Helm-deployed agent shares a fleet-wide cache so JIT kernel builds and weight artifacts survive scale-outs.

## Why it matters

Cold-loading a large model from Hugging Face bottlenecks every autoscale and blue/green rollout. ModelExpress cuts that startup from minutes to seconds — NVIDIA reports 48× vs cold HF pull on DeepSeek V4 Pro — which shrinks the tail latency SREs plan around and lowers the cost of running fleets of vLLM or SGLang replicas.

## Who it's for

SREs and platform teams running large-model inference at scale

## Try it

pip install vllm>=0.23.0 && vllm serve --load-format modelexpress <model>

## Quick facts

- Maker: NVIDIA (ai-dynamo)
- License: Apache-2.0
- Language: Rust (Python client)
- Reported speedup: 48× vs cold Hugging Face pull on DeepSeek V4 Pro
- Integrations: vLLM 0.23+, SGLang, TensorRT-LLM (beta), NVIDIA Dynamo
- Deployment: Kubernetes / Helm sidecar
- Repo: github.com/ai-dynamo/modelexpress

## FAQ

### How much faster is ModelExpress than cold-loading from Hugging Face?

NVIDIA reports ModelExpress delivers up to a 48× speedup versus a cold Hugging Face pull for DeepSeek V4 Pro when it can use GPU-to-GPU P2P transfer. The exact gain depends on which loading path — P2P RDMA, GPUDirect Storage, host-staged POSIX I/O, or a fresh HF download — the loader picks based on what is already cached across the fleet.

### Which inference servers can use ModelExpress today?

ModelExpress ships native integrations with vLLM 0.23+ via `--load-format modelexpress`, SGLang as a `remote_instance` backend, TensorRT-LLM in beta, and NVIDIA Dynamo's vLLM and SGLang runtimes with P2P weight and JIT kernel-cache transfer. Other servers can still call it as a sidecar since ModelExpress runs alongside the inference process.

### What license is ModelExpress and where does the code live?

ModelExpress is Apache-2.0 licensed and hosted at github.com/ai-dynamo/modelexpress. The core is written in Rust with a Python client that inference frameworks link against, and the repo ships Helm charts for the Kubernetes sidecar deployment path NVIDIA recommends.

### How does ModelExpress fit into a Kubernetes serving cluster?

ModelExpress runs as a sidecar next to each inference pod, so replicas pull checkpoints and JIT kernel caches from each other over P2P RDMA or GPUDirect Storage instead of hitting Hugging Face on every scale-out. Helm charts in the ai-dynamo/modelexpress repo cover the shared-cache deployment pattern.

## Links

- [Announcement](https://developer.nvidia.com/blog/modelexpress-distributing-model-artifacts-at-the-speed-of-light/)
- [Repository](https://github.com/ai-dynamo/modelexpress)
- [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo)