# EAGLE

> Speculative decoding that drafts tokens from an LLM's own features to speed up generation without changing its output

- Category: Models & Inference / Serving & Deployment
- Language: Python
- Licence: Apache-2.0
- Repository: https://github.com/SafeAILab/EAGLE
- Canonical page: https://ai-tldr.dev/tools/eagle/

## In short

EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) is a speculative-decoding method for making large language models generate text faster, and this repository is its official implementation from SafeAI Lab. It sits in the Serving & Deployment category of the Models & Inference tools and is open-source, written in Python and licensed Apache-2.0.

## Overview

EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) is a speculative-decoding method for making large language models generate text faster, and this repository is its official implementation from SafeAI Lab. Instead of running a separate small model as the drafter, EAGLE attaches a lightweight draft head to the target model that works from the model's own internal feature vectors to guess several tokens ahead; the full model then verifies those guesses in one pass. Because verification decides what is kept, the method is designed to preserve the distribution of the text the original model would have produced — the speed-up comes without a change in output quality.

The repository covers three generations of the idea. EAGLE-1 extrapolates the second-top-layer features of the LLM. EAGLE-2 uses the draft model's confidence scores to approximate acceptance rates and reshapes the draft tree on the fly. EAGLE-3 drops the feature-prediction constraint, simulates it during training with a "training-time test", and fuses low-, mid- and high-level features instead of relying on the top layer alone. The main branch implements EAGLE-3 and EAGLE-2; EAGLE-1 lives on the `v1` branch.

In practice you pair a base model from Hugging Face with a matching EAGLE draft checkpoint. The project publishes official EAGLE-3 weights for Vicuna-13B, Llama 3.1 8B, Llama 3.3 70B and DeepSeek-R1-Distill-Llama-8B, plus EAGLE-1 weights for Vicuna, Llama 2/3 chat, Mixtral and Qwen2 models, and its README lists community checkpoints for Qwen3, Llama 4, GPT-OSS and others. You can then chat with the pair through a Gradio web UI, call an `eagenerate` method that mirrors Hugging Face's `generate`, benchmark it on MT-bench, or train your own draft head.

The method has spread beyond this codebase: the README lists EAGLE support in serving and optimization frameworks including vLLM, SGLang, NVIDIA TensorRT-LLM, TensorRT Model Optimizer, NeMo, MLC-LLM, AWS NeuronX, AMD ROCm and PaddleNLP. For training EAGLE-3 heads to use with SGLang, the maintainers recommend SpecForge. The repository itself is Python and Apache-2.0.

## What it does

- Official implementations of EAGLE-1, EAGLE-2 (dynamic draft trees) and EAGLE-3 (multi-level feature fusion with training-time test)
- Designed to keep the base model's output distribution while generating several tokens per verification step, including non-greedy sampling
- `EaModel.from_pretrained` plus `eagenerate`, a drop-in counterpart to Hugging Face `generate`, with weights spread across multiple GPUs automatically
- A Gradio web UI for chatting with a base model and its EAGLE head, with a tunable number of draft tokens (or `-1` to let EAGLE-2 choose)
- DeepSpeed training code for EAGLE-3 draft heads and MT-bench scripts that measure speed against vanilla autoregressive decoding
- Published draft weights for Vicuna, Llama 2/3/3.1/3.3, Mixtral, Qwen2 and DeepSeek-R1-Distill models, with community checkpoints for many more

## Getting started

The repository runs as a Python package in its own virtual environment. You need a base model and a matching EAGLE checkpoint; the README's tables list which draft weights go with which base model.

1. Install

```bash
git clone https://github.com/SafeAILab/EAGLE.git
cd EAGLE
python -m venv ~/venvs/ea_env
source ~/venvs/ea_env/bin/activate
pip install -r requirements.txt
```

2. Try it in the web UI

   Point the UI at an EAGLE checkpoint and its base model. When loading finishes, a URL is printed in the terminal to open in your browser. `--total-token` is the number of draft tokens; larger values suit smaller models and faster GPUs, and `-1` lets EAGLE-2 configure it automatically.

```bash
python -m eagle.application.webui --ea-model-path [path of EAGLE weight] \
  --base-model-path [path of the original model] \
  --model-type [vicuna\llama2\llama3] \
  --total-token [int]
```

3. Generate from code

   `eagenerate` is used like Hugging Face's `generate`. EAGLE-3 is the default; pass `use_eagle3=False` to `from_pretrained` to load EAGLE-1 weights. Chat models need their correct chat template, or output quality and speed-up both suffer.

```python
from eagle.model.ea_model import EaModel
from fastchat.model import get_conversation_template

model = EaModel.from_pretrained(
    base_model_path=base_model_path,
    ea_model_path=EAGLE_model_path,
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
    device_map="auto",
    total_token=-1
)
model.eval()
conv = get_conversation_template("vicuna")
conv.append_message(conv.roles[0], "Hello")
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()
input_ids = model.tokenizer([prompt]).input_ids
input_ids = torch.as_tensor(input_ids).cuda()
output_ids = model.eagenerate(input_ids, temperature=0.5, max_new_tokens=512)
output = model.tokenizer.decode(output_ids[0])
```

4. Measure the speed-up

   Run MT-bench with EAGLE and then with plain autoregressive decoding; each command writes a .jsonl file with outputs and wall time, and `evaluation/speed.py` computes the ratio. Models download automatically, which may require `huggingface-cli login`.

```bash
python -m eagle.evaluation.gen_ea_answer_llama3chat --ea-model-path yuhuili/EAGLE3-LLaMA3.1-Instruct-8B --base-model-path meta-llama/Llama-3.1-8B-Instruct --use_eagle3

python -m eagle.evaluation.gen_baseline_answer_llama3chat --ea-model-path yuhuili/EAGLE3-LLaMA3.1-Instruct-8B --base-model-path meta-llama/Llama-3.1-8B-Instruct
```

5. Train your own draft head

   The EAGLE-3 trainer runs under DeepSpeed. For out-of-the-box EAGLE-3 training aimed at SGLang, the maintainers recommend SpecForge instead.

```bash
cd eagle/traineagle3
deepspeed main.py --deepspeed_config ds_config.json
```

## When to use it

- Reach for it when generation latency is the bottleneck and you want faster decoding from an existing LLM without accepting a change in its outputs
- Reach for it to train an EAGLE draft head for a model that has no published checkpoint, then deploy it in a serving engine that supports EAGLE
- Reach for it to benchmark speculative decoding on your own hardware and model before enabling it in production
- Reach for it when you are researching speculative decoding and need the reference code behind EAGLE-1, EAGLE-2 and EAGLE-3