How much faster is ModelExpress than cold-loading from Hugging Face?

NVIDIA reports ModelExpress delivers up to a 48× speedup versus a cold Hugging Face pull for DeepSeek V4 Pro when it can use GPU-to-GPU P2P transfer. The exact gain depends on which loading path — P2P RDMA, GPUDirect Storage, host-staged POSIX I/O, or a fresh HF download — the loader picks based on what is already cached across the fleet.

Which inference servers can use ModelExpress today?

ModelExpress ships native integrations with vLLM 0.23+ via `--load-format modelexpress`, SGLang as a `remote_instance` backend, TensorRT-LLM in beta, and NVIDIA Dynamo's vLLM and SGLang runtimes with P2P weight and JIT kernel-cache transfer. Other servers can still call it as a sidecar since ModelExpress runs alongside the inference process.

What license is ModelExpress and where does the code live?

ModelExpress is Apache-2.0 licensed and hosted at github.com/ai-dynamo/modelexpress. The core is written in Rust with a Python client that inference frameworks link against, and the repo ships Helm charts for the Kubernetes sidecar deployment path NVIDIA recommends.

How does ModelExpress fit into a Kubernetes serving cluster?

ModelExpress runs as a sidecar next to each inference pod, so replicas pull checkpoints and JIT kernel caches from each other over P2P RDMA or GPUDirect Storage instead of hitting Hugging Face on every scale-out. Helm charts in the ai-dynamo/modelexpress repo cover the shared-cache deployment pattern.

NVIDIA · 2026-07-24 · major

NVIDIA ModelExpress — Rust sidecar makes vLLM cold-loads 48× faster

NVIDIA open-sources ModelExpress, an Apache-2.0 Rust component that P2P-transfers model weights next to vLLM, SGLang, TensorRT-LLM, and Dynamo. It reports a 48× cold-load speedup on DeepSeek V4 Pro.

Diagram of ModelExpress routing model weights between GPUs, host cache, and Hugging Face

NVIDIA's open-source Rust sidecar loads model weights into GPUs up to 48× faster than a cold Hugging Face pull.

Key specs

License	Apache-2.0
Cold pull speedup (deep seek v4 pro)	48×

Quick facts

Maker	NVIDIA (ai-dynamo)
License	Apache-2.0
Language	Rust (Python client)
Reported speedup	48× vs cold Hugging Face pull on DeepSeek V4 Pro
Integrations	vLLM 0.23+, SGLang, TensorRT-LLM (beta), NVIDIA Dynamo
Deployment	Kubernetes / Helm sidecar
Repo	github.com/ai-dynamo/modelexpress

What is it?

ModelExpress is an Apache-2.0 Rust component NVIDIA open-sourced on July 24, 2026 that sits next to an inference server and speeds up model loading. It caches weight artifacts, routes them across GPUs, and hands them to vLLM, SGLang, TensorRT-LLM, or NVIDIA Dynamo without changing the serving stack.

How does it work?

Given a request to load a model, ModelExpress picks the fastest path available — GPU-to-GPU P2P over RDMA, GPUDirect Storage off a local NVMe cache, a host-staged POSIX read, or a fresh Hugging Face download — then streams the checkpoint into GPU memory. A Helm-deployed agent shares a fleet-wide cache so JIT kernel builds and weight artifacts survive scale-outs.

Why does it matter?

Cold-loading a large model from Hugging Face bottlenecks every autoscale and blue/green rollout. ModelExpress cuts that startup from minutes to seconds — NVIDIA reports 48× vs cold HF pull on DeepSeek V4 Pro — which shrinks the tail latency SREs plan around and lowers the cost of running fleets of vLLM or SGLang replicas.

Who is it for?

SREs and platform teams running large-model inference at scale

Frequently asked questions

How much faster is ModelExpress than cold-loading from Hugging Face?: NVIDIA reports ModelExpress delivers up to a 48× speedup versus a cold Hugging Face pull for DeepSeek V4 Pro when it can use GPU-to-GPU P2P transfer. The exact gain depends on which loading path — P2P RDMA, GPUDirect Storage, host-staged POSIX I/O, or a fresh HF download — the loader picks based on what is already cached across the fleet.
Which inference servers can use ModelExpress today?: ModelExpress ships native integrations with vLLM 0.23+ via `--load-format modelexpress`, SGLang as a `remote_instance` backend, TensorRT-LLM in beta, and NVIDIA Dynamo's vLLM and SGLang runtimes with P2P weight and JIT kernel-cache transfer. Other servers can still call it as a sidecar since ModelExpress runs alongside the inference process.
What license is ModelExpress and where does the code live?: ModelExpress is Apache-2.0 licensed and hosted at github.com/ai-dynamo/modelexpress. The core is written in Rust with a Python client that inference frameworks link against, and the repo ships Helm charts for the Kubernetes sidecar deployment path NVIDIA recommends.
How does ModelExpress fit into a Kubernetes serving cluster?: ModelExpress runs as a sidecar next to each inference pod, so replicas pull checkpoints and JIT kernel caches from each other over P2P RDMA or GPUDirect Storage instead of hitting Hugging Face on every scale-out. Helm charts in the ai-dynamo/modelexpress repo cover the shared-cache deployment pattern.

Try it

pip install vllm>=0.23.0 && vllm serve --load-format modelexpress <model>