Kog · 2026-05-28 · major
Kog Ships Inference Engine Tech Preview — Paris Startup Hits 3,000 Tokens/s Per Request on 8× AMD MI300X and 2,100 on 8× NVIDIA H200 With a Single Persistent CUDA Kernel
Kog opens a public tech preview of its FP16 inference engine: 3,000 output tokens/s per request on 8× MI300X, 2,100 on 8× H200, no speculative decoding, on a 2B Laneformer model.
A Paris startup squeezes 3,000 tokens/s out of a single inference request on standard datacenter GPUs.
Key specs
| Parameters | 2B |
|---|---|
| Tokens per sec mi300x | 3,000 |
| Tokens per sec h200 | 2,100 |
| Cross gpu latency us | 3 |
| Hn points | 203 |
What is it?
Kog Inference Engine (KIE) is a from-scratch C++/CUDA LLM inference engine focused on single-request decode latency rather than aggregate batch throughput. The public tech preview runs a custom 2B Laneformer model on 8× AMD MI300X and 8× NVIDIA H200 nodes, with a live coding playground at playground.kog.ai. Support for large third-party MoE models is the next milestone.
How does it work?
Decode runs as one persistent GPU program (a monokernel) so each new token avoids the launch overhead of separate kernels. A custom collective library, KCCL, cuts inter-GPU sync to under 3 microseconds. The Laneformer architecture adds Delayed Tensor Parallelism so cross-GPU communication overlaps with computation instead of blocking the critical path.
Why does it matter?
Agentic, voice, and gaming workloads depend on single-stream token speed, where batched inference engines like vLLM and TensorRT-LLM leave bandwidth on the table. Kog claims roughly 3.5× faster generation than vLLM and TensorRT-LLM on identical AMD workloads, which compresses long reasoning chains into product-grade latency on hardware enterprises already own.
Who is it for?
infra engineers running agents, voice, and gaming workloads on MI300X or H200
Try it
https://playground.kog.ai