FlashInfer

// Overview

FlashInfer is a library and kernel generator for LLM inference that provides unified APIs for attention, GEMM and Mixture-of-Experts operations. Rather than binding you to one hand-written kernel, it wraps several backend implementations — FlashAttention-2/3, cuDNN, CUTLASS and TensorRT-LLM — and selects the one that suits your hardware and workload.

The attention side is built for real serving rather than for benchmarks on fixed shapes: paged and ragged KV-cache layouts for dynamic batches, separate decode, prefill and append kernels, native Multi-Latent Attention as used by DeepSeek, cascade attention that shares a KV-cache across requests with a common prefix, block-sparse patterns, and POD-attention which fuses prefill and decode for mixed batching.

Beyond attention it covers the rest of the serving hot path: BF16/FP8/FP4 GEMM including grouped GEMM for LoRA and expert routing, fused MoE kernels with DeepSeek-V3, Llama-4 and standard top-k routing, sorting-free Top-K/Top-P/Min-P sampling, chain speculative sampling, RoPE, RMSNorm and LayerNorm, and AllReduce with multi-node NVLink and NVSHMEM support. Kernels are CUDAGraph and torch.compile compatible, and the project documents support from SM75 (Turing) through Blackwell.

// What it does

• Unified attention API over FlashAttention-2/3, cuDNN, CUTLASS and TensorRT-LLM backends, chosen automatically for the hardware and workload
• Serving-shaped attention: paged and ragged KV-cache, separate decode/prefill/append kernels, MLA, cascade attention for shared prefixes, block-sparse and POD-attention
• FP8 and FP4 low-precision compute for attention, GEMM and MoE, plus grouped GEMM for LoRA and multi-expert routing
• Fused MoE kernels with DeepSeek-V3, Llama-4 and standard top-k routing, and block-wise scaled quantized expert weights
• Sorting-free Top-K / Top-P / Min-P sampling and chain speculative sampling
• CUDAGraph- and torch.compile-compatible, with AllReduce, multi-node NVLink (MNNVL) and NVSHMEM for distributed inference

// Getting started

FlashInfer ships as a Python package that compiles or downloads kernels on first use. Optional companion wheels pre-compile them so the first request does not pay a JIT cost.

pip install flashinfer-python

pip install flashinfer-python
flashinfer install-cubin-wheel
flashinfer install-jit-cache-wheel

# Blackwell (SM100+) CuTe DSL kernels
pip install flashinfer-python[cu13]

flashinfer show-config

import torch
import flashinfer

# Single decode attention
q = torch.randn(32, 128, device="cuda", dtype=torch.float16)  # [num_qo_heads, head_dim]
k = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)  # [kv_len, num_kv_heads, head_dim]
v = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)

output = flashinfer.single_decode_with_kv_cache(q, k, v)

git clone https://github.com/flashinfer-ai/flashinfer.git --recursive
cd flashinfer

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Speed up an inference server's attention path without hand-writing a kernel per GPU generation, letting FlashInfer pick between FlashAttention, cuDNN, CUTLASS and TensorRT-LLM
• Serve a Mixture-of-Experts model with fused, quantized expert kernels and DeepSeek-V3 or Llama-4 routing
• Cut memory on workloads where many requests share a long system prompt, using cascade attention over a shared KV-cache
• Run FP8 or FP4 attention and GEMM on Blackwell hardware while keeping the same Python API used on older architectures

// How FlashInfer compares

FlashInfer alongside other open-source gpu kernels & compilers tools AI/TLDR tracks, ranked by GitHub stars.