Miles

// Overview

Miles is an open-source reinforcement-learning framework from RadixArk for post-training large language and vision-language models. The project describes it as "a high-performance, enterprise-ready reinforcement learning framework for large-scale model post-training" that combines SGLang for rollout with Megatron-LM for training, adding the precision, stability and observability an RL run needs at trillion-parameter scale. It is a fork of THUDM's slime and, in RadixArk's words, co-evolves with it rather than replacing it.

The design point is asynchrony. Rollout workers and training workers are decoupled and run fully async, so generation does not stall while a gradient step runs. Tokens pass between the two halves directly — the project calls this token-in-token-out, or TITO — which removes a detokenize/retokenize round trip and the subtle mismatches it can introduce. Updated weights reach the inference engines in seconds even at trillion-parameter scale, and a mechanism called Rollout Routing Replay (R3) removes mixture-of-experts routing mismatches between the rollout and training copies of a model.

Beyond the core loop, Miles covers the parts of an RL run that break in production. Low-precision training spans MXFP8, NVFP4, FP8, INT4 quantization-aware training, BF16 and FP16. LoRA and multi-LoRA work at frontier scale, so a run does not have to update every parameter. SGLang engines that die are recovered automatically, and the rollout path does its own request routing and health-checking. Training runs on Megatron-LM as the primary backend or PyTorch FSDP2 as an alternative.

Hardware and model coverage are unusually broad for RL training code. NVIDIA GB300, GB200, B300, B200, H200, H100 and A100 are supported, as are AMD MI300X, MI325, MI350 and MI355X through ROCm, with a separate ROCm Docker image. Day-zero model support covers DeepSeek-V4, Kimi-K3, GLM-5.2, Inkling and Nemotron, with wider compatibility including Kimi-K2.6 and Qwen3.5. The recipe set spans GRPO, GSPO, PPO, REINFORCE++, supervised fine-tuning and on-policy distillation.

// What it does

• Fully asynchronous RL — rollout and training workers are decoupled, so generation does not block on gradient steps
• Token-in-token-out (TITO) passes tokens straight between rollout and training, with no detokenize/retokenize cycle
• Weight updates that reach the inference engines in seconds even for trillion-parameter models
• Low-precision training across MXFP8, NVFP4, FP8, INT4 QAT, BF16 and FP16, plus LoRA and multi-LoRA at frontier scale
• Rollout Routing Replay (R3) removes mixture-of-experts routing mismatches between rollout and training
• Runs on both NVIDIA (GB300 through A100) and AMD (MI300X through MI355X) accelerators, with fault tolerance that restarts failed SGLang engines

// Getting started

Docker is the recommended path — the image ships the patched SGLang and Megatron dependencies. The quick start runs a complete GRPO loop on Qwen3-4B with a math dataset and AIME as the evaluation set.

nvidia-smi -L
docker run --rm --gpus all ubuntu nvidia-smi
df -h $(docker info -f '{{.DockerRootDir}}')

docker pull radixark/miles:latest

docker run --rm --gpus all --ipc=host --shm-size=32g \
  -it radixark/miles:latest /bin/bash

cd /root/miles
git pull --rebase
pip install -e . --no-deps

python -c "import miles; print('Miles import OK')"

git clone https://github.com/radixark/miles.git
cd miles
pip install -r requirements.txt
pip install -e . --no-deps

# three `hf download` commands fetch the model, the math
# training set and the AIME eval set, then:
python tools/convert_hf_to_torch_dist.py ${MODEL_ARGS[@]} \
  --hf-checkpoint /root/Qwen3-4B \
  --save /root/Qwen3-4B_torch_dist

python scripts/run_qwen3_dense.py --model-name Qwen3-4B

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Reach for it when you are running RL post-training at frontier scale and need rollout and training to run asynchronously rather than in lockstep
• Reach for it when your cluster is AMD — Miles ships a ROCm image and supports MI300X through MI355X, which most RL training code does not
• Reach for it when you want GRPO, GSPO, PPO, REINFORCE++, SFT and on-policy distillation from one framework instead of several projects stitched together
• Reach for it when a mixture-of-experts model is drifting between rollout and training — Rollout Routing Replay exists specifically for that mismatch

// How Miles compares

Miles alongside other open-source rlhf & alignment tools AI/TLDR tracks, ranked by GitHub stars.