Cohere · 2026-09-08 · notable
Cohere megakernel — one CUDA file serves North Mini Code faster than vLLM
Cohere open-sourced a decode megakernel for North Mini Code that runs 1.25x to 1.41x faster end to end than vLLM on a single H100. It fuses the whole decode step into one persistent CUDA kernel with no compiler dependencies.
Cohere fused North Mini Code's whole decode step into a single persistent CUDA kernel.
Key specs
| Decode throughput | 292 tok/s on one H100 |
|---|---|
| Speed vs v llm (batch 1) | 1.58x |
What is it?
Cohere published a serving engine for North Mini Code built around a decode megakernel, and released the CUDA source on GitHub under Apache-2.0. North Mini Code is Cohere's 30B mixture-of-experts coding model with 3.3B parameters active per token. The engine targets a single NVIDIA H100 with batch sizes up to eight, and needs CUDA 13, Python 3.12 and Linux.
How does it work?
A megakernel fuses the many small GPU operations of a decode step into one persistent kernel, so the GPU never sits idle between launches. Cohere reports 292 tokens per second at batch size 1 on an H100 — 62% of the theoretical speed-of-light for that hardware — and 1.58x the decode speed of vLLM v0.24. Measured end to end across AIME 2025, GPQA, MMLU-Pro, SciCode and LiveCodeBench, the engine is 1.25x to 1.41x faster, with accuracy inside the usual run-to-run spread.
Why does it matter?
Serving a small mixture-of-experts model well is mostly a launch-overhead problem, not a maths problem, and this shows how much is left on the table: same H100, same weights, up to 41% more end-to-end throughput. Cohere ships the kernel as a single CUDA file with no compiler dependencies, so it also reads as a worked example for anyone writing their own persistent kernel.
Who is it for?
inference engineers and CUDA developers
Try it
git clone --recurse-submodules https://github.com/cohere-ai/cohere-megakernel