# FlashInfer

> Attention, GEMM and MoE kernels for LLM serving, with the backend picked for your GPU

- Category: Models & Inference / GPU Kernels & Compilers
- Language: CUDA
- Licence: Apache-2.0
- Repository: https://github.com/flashinfer-ai/flashinfer
- Website: https://flashinfer.ai
- Canonical page: https://ai-tldr.dev/tools/flashinfer/

## In short

FlashInfer is a library and kernel generator for LLM inference that provides unified APIs for attention, GEMM and Mixture-of-Experts operations. Rather than binding you to one hand-written kernel, it wraps several backend implementations — FlashAttention-2/3, cuDNN, CUTLASS and TensorRT-LLM — and selects the one that suits your hardware and workload. It sits in the GPU Kernels & Compilers category of the Models & Inference tools and is open-source, written in CUDA and licensed Apache-2.0.

## Overview

FlashInfer is a library and kernel generator for LLM inference that provides unified APIs for attention, GEMM and Mixture-of-Experts operations. Rather than binding you to one hand-written kernel, it wraps several backend implementations — FlashAttention-2/3, cuDNN, CUTLASS and TensorRT-LLM — and selects the one that suits your hardware and workload.

The attention side is built for real serving rather than for benchmarks on fixed shapes: paged and ragged KV-cache layouts for dynamic batches, separate decode, prefill and append kernels, native Multi-Latent Attention as used by DeepSeek, cascade attention that shares a KV-cache across requests with a common prefix, block-sparse patterns, and POD-attention which fuses prefill and decode for mixed batching.

Beyond attention it covers the rest of the serving hot path: BF16/FP8/FP4 GEMM including grouped GEMM for LoRA and expert routing, fused MoE kernels with DeepSeek-V3, Llama-4 and standard top-k routing, sorting-free Top-K/Top-P/Min-P sampling, chain speculative sampling, RoPE, RMSNorm and LayerNorm, and AllReduce with multi-node NVLink and NVSHMEM support. Kernels are CUDAGraph and torch.compile compatible, and the project documents support from SM75 (Turing) through Blackwell.

## What it does

- Unified attention API over FlashAttention-2/3, cuDNN, CUTLASS and TensorRT-LLM backends, chosen automatically for the hardware and workload
- Serving-shaped attention: paged and ragged KV-cache, separate decode/prefill/append kernels, MLA, cascade attention for shared prefixes, block-sparse and POD-attention
- FP8 and FP4 low-precision compute for attention, GEMM and MoE, plus grouped GEMM for LoRA and multi-expert routing
- Fused MoE kernels with DeepSeek-V3, Llama-4 and standard top-k routing, and block-wise scaled quantized expert weights
- Sorting-free Top-K / Top-P / Min-P sampling and chain speculative sampling
- CUDAGraph- and torch.compile-compatible, with AllReduce, multi-node NVLink (MNNVL) and NVSHMEM for distributed inference

## Getting started

FlashInfer ships as a Python package that compiles or downloads kernels on first use. Optional companion wheels pre-compile them so the first request does not pay a JIT cost.

1. Install the core package

   flashinfer-python compiles or downloads kernels the first time each one is used.

```bash
pip install flashinfer-python
```

2. Pre-compile for faster startup and offline use

   The cubin and JIT-cache wheels carry pre-built kernels for the supported architectures, so a cold process does not compile. On Blackwell (SM100+), install the CUDA 13 extra instead to enable the CuTe DSL kernels.

```bash
pip install flashinfer-python
flashinfer install-cubin-wheel
flashinfer install-jit-cache-wheel

# Blackwell (SM100+) CuTe DSL kernels
pip install flashinfer-python[cu13]
```

3. Verify the install

   Prints the detected GPU, backends and kernel providers FlashInfer will use.

```bash
flashinfer show-config
```

4. Run a decode attention call

   The single-request decode kernel takes a query plus the key and value cache and returns the attention output.

```python
import torch
import flashinfer

# Single decode attention
q = torch.randn(32, 128, device="cuda", dtype=torch.float16)  # [num_qo_heads, head_dim]
k = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)  # [kv_len, num_kv_heads, head_dim]
v = torch.randn(2048, 32, 128, device="cuda", dtype=torch.float16)

output = flashinfer.single_decode_with_kv_cache(q, k, v)
```

5. Build from source

   Clone recursively — the build pulls in submodules. See docs.flashinfer.ai for the full API reference and tutorials.

```bash
git clone https://github.com/flashinfer-ai/flashinfer.git --recursive
cd flashinfer
```

## When to use it

- Speed up an inference server's attention path without hand-writing a kernel per GPU generation, letting FlashInfer pick between FlashAttention, cuDNN, CUTLASS and TensorRT-LLM
- Serve a Mixture-of-Experts model with fused, quantized expert kernels and DeepSeek-V3 or Llama-4 routing
- Cut memory on workloads where many requests share a long system prompt, using cascade attention over a shared KV-cache
- Run FP8 or FP4 attention and GEMM on Blackwell hardware while keeping the same Python API used on older architectures