ryanzhou · 2026-08-04 · notable
DeepSeek V4-Flash on one MI300X — 168 tok/s decode, no quantization
ryanzhou publishes a production stack that runs DeepSeek V4-Flash unquantized on a single AMD MI300X GPU. It hits 168 tok/s single-stream decode, 830 tok/s across 64 streams, and 256K validated context.
Docker Compose stack that runs the full 304B DeepSeek V4-Flash unquantized on a single AMD MI300X, hitting 168 tok/s decode.
Key specs
| License | Apache-2.0 |
|---|---|
| Stars (day 1) | 64 |
| Hn points | 286 |
| Decode | 168.6 tok/s |
What is it?
deepseek-v4-flash-mi300x is an Apache-2.0 vLLM overlay from developer ryanzhou that runs the full 304B DeepSeek V4-Flash model on one AMD MI300X GPU without weight quantization. The whole 156.67 GiB of weights fits in HBM3.
How does it work?
The stack fixes AMD's FNUZ FP8 cache-writer for the MI300X, patches MXFP4 routing corruption that hurt tool-calling accuracy, ships 21 GEMM shapes tuned for gfx942, and pairs a 20 GB GPU KV cache with a 96 GiB CPU KV tier. DSpark speculative decoding adds probabilistic drafting with causal verification.
Why does it matter?
The MI300X has been an option on paper for months, but vanilla vLLM misbehaved on FP8 and MoE routing. This Apache-2.0 stack drops a ready-to-run Docker Compose setup that rents for about $1.99/hr on AMD Developer Cloud, giving self-hosted teams a cheaper, un-quantized alternative to NVIDIA for the flagship DeepSeek model.
Who is it for?
AMD GPU users, self-hosted inference teams
Try it
git clone https://github.com/ryanzhou/deepseek-v4-flash-mi300x