How much faster is TRL v1.13.0 in practice?

Measured tokens per second per GPU at 16,384 tokens per step: Qwen3-8B full fine-tuning on 2xH100 FSDP2 rises from 3,554 to 6,009 (1.69x), the same model on LoRA r16 from 4,531 to 7,125 (1.57x), gemma-3-270m from 24,036 to 31,609 (1.32x), and Qwen3-30B-A3B with LoRA attention from 4,251 to 5,101 (1.20x).

What should I use now that PPOTrainer is gone?

TRL v1.13.0 deletes `PPOTrainer`, `PPOConfig` and `modeling_value_head.py` along with their tests, docs and examples. The release notes say the trainer predates every pull request in the repo but had stayed unmaintained with little recorded usage. Pipelines that still need it must pin an earlier TRL version or move to another trainer such as GRPO.

Does TRL v1.13.0 still need liger-kernel installed?

Yes, for the model kernels. TRL v1.13.0 vendors the fused linear DPO, KTO, GRPO and JSD losses from Liger-Kernel v0.8.2 into `trl.losses`, keeping the BSD-2 notice and bitwise-identical behaviour. There is no new config flag — `use_liger_kernel=True` still selects the fused loss, and transformers' Trainer still patches model kernels with liger-kernel.

What are the limits of the 1M-token training path?

TRL's long-context guide documents full attention only, with no sequence packing, and requires transformers 5.16 or newer for gradient-checkpointing offload. The reference run is Qwen3-8B in bf16 with `per_device_train_batch_size=1` and `loss_type="chunked_nll"` on one 8xH100 node; it runs end to end and saves a checkpoint.

Hugging Face · 2026-09-10 · major

TRL v1.13.0 — million-token training, and PPOTrainer is removed

TRL v1.13.0 documents training at 1,048,576 tokens per step on one 8xH100 node and makes the chunked cross-entropy projection run on tensor cores, up to 1.69x faster end to end. PPOTrainer, the library's oldest trainer, is deleted.

GitHub card for the Hugging Face TRL post-training library

A one-line dtype fix pulls TRL's default loss back onto tensor cores, and the library's oldest trainer is finally deleted.

Quick facts

Maker	Hugging Face
Version	v1.13.0
Released	September 10, 2026
License	Apache-2.0
Long-context run	Qwen3-8B at 1,048,576 tokens, 380 s/step, 56.2 GB per GPU
Removed	PPOTrainer, PPOConfig, value-head models
New floors	peft>=0.13.0, deepspeed>=0.18.6, vLLM 0.28.0

What is it?

Book-length training is now documented in TRL v1.13.0: a long-context guide plus a runnable example that trains Qwen3-8B at 1,048,576 tokens per step on a single 8xH100 node, measured at 380 seconds per step and 56.2 GB per GPU in bf16. The guide walks the levers in the order you hit them — `chunked_nll`, gradient-checkpointing offload, YaRN RoPE — and names the constraints: full attention only, no packing.

How does it work?

The speedup comes from deleting an upcast. The inner loop of the default `chunked_nll` loss was computing `h.float() @ w.float().t()` on operands that were already bf16, which pushed the matrix multiply off the tensor cores onto the fp32 SIMT path and rebuilt a 2.03 GB fp32 copy of the lm_head weight for every chunk. In an 8xH100 profile of `trl sft` on Qwen3.6-35B-A3B, those two fp32 kernels were 21.6% of all GPU kernel time. One chunk now runs in 3.86 ms instead of 23.37 ms.

Why does it matter?

Post-training jobs get cheaper without any config change: Qwen3-8B full fine-tuning on 2xH100 FSDP2 goes from 3,554 to 6,009 tokens per second per GPU. The cost lands on anyone still importing PPO. `PPOTrainer`, `PPOConfig` and the value-head wrappers in `modeling_value_head.py` are gone, so those pipelines must move to another trainer or pin an older TRL. Dependency floors also rise to peft 0.13.0 and deepspeed 0.18.6, and vLLM 0.19.0 support is dropped.

Who is it for?

teams running post-training and RLHF

Frequently asked questions

How much faster is TRL v1.13.0 in practice?: Measured tokens per second per GPU at 16,384 tokens per step: Qwen3-8B full fine-tuning on 2xH100 FSDP2 rises from 3,554 to 6,009 (1.69x), the same model on LoRA r16 from 4,531 to 7,125 (1.57x), gemma-3-270m from 24,036 to 31,609 (1.32x), and Qwen3-30B-A3B with LoRA attention from 4,251 to 5,101 (1.20x).
What should I use now that PPOTrainer is gone?: TRL v1.13.0 deletes `PPOTrainer`, `PPOConfig` and `modeling_value_head.py` along with their tests, docs and examples. The release notes say the trainer predates every pull request in the repo but had stayed unmaintained with little recorded usage. Pipelines that still need it must pin an earlier TRL version or move to another trainer such as GRPO.
Does TRL v1.13.0 still need liger-kernel installed?: Yes, for the model kernels. TRL v1.13.0 vendors the fused linear DPO, KTO, GRPO and JSD losses from Liger-Kernel v0.8.2 into `trl.losses`, keeping the BSD-2 notice and bitwise-identical behaviour. There is no new config flag — `use_liger_kernel=True` still selects the fused loss, and transformers' Trainer still patches model kernels with liger-kernel.
What are the limits of the 1M-token training path?: TRL's long-context guide documents full attention only, with no sequence packing, and requires transformers 5.16 or newer for gradient-checkpointing offload. The reference run is Qwen3-8B in bf16 with `per_device_train_batch_size=1` and `loss_type="chunked_nll"` on one 8xH100 node; it runs end to end and saves a checkpoint.

Try it

pip install -U trl