minWM

// Overview

minWM is a full-stack open-source framework for building action-conditioned video world models. Rather than shipping one model, it walks the complete path — data, training, distillation, inference — for turning a bidirectional text-to-video foundation model into a few-step autoregressive world model you can steer with camera actions in real time. Every stage exposes its input and output checkpoints, so you can stop, swap or fork anywhere in the pipeline instead of taking it as a black box.

The training half covers FSDP with sequence parallelism, single- and multi-node runs, and the full distillation ladder: bidirectional supervised fine-tuning, then teacher-forced autoregressive diffusion, causal ODE and causal consistency distillation, and finally asymmetric DMD with self-rollout to reach a 4-step student. Two backbones are carried end to end through all four stages — Wan 2.1 (1.3B, cross-attention DiT) and HunyuanVideo 1.5 (8B, MMDiT) — and they share the same trainer, loss and dataset abstractions, so adding a third is largely a wrapper-and-config exercise. Inference runs 4-step DMD with multi-GPU sequence parallelism and camera-trajectory control expressed as pose strings like `"a*4,w*8,s*7"` or as JSON.

The project is explicit that it is aimed at newcomers as much as at researchers: it ships example data, runnable scripts, onboarding material, and Claude skills that package the maintainers' own experience — `debug-world-model` for diagnosing collected failure modes (loss NaN, frame-to-frame jitter, camera drift, memory attenuation, distillation collapse), `integrate-new-backbone` for plugging in a new video DiT against the HunyuanVideo and Wan reference integrations, and `onboarding-world-model` covering the minimal background plus the non-obvious pitfalls the team hit. It is Apache-2.0, with checkpoints on Hugging Face and a technical report on arXiv.

// What it does

• The complete data → training → inference pipeline open-sourced, with input/output checkpoints exposed at every stage
• Full distillation ladder from bidirectional SFT through teacher-forced AR diffusion, causal ODE/CD and asymmetric DMD to a 4-step student
• Two backbones carried through all four stages — Wan 2.1 (1.3B) and HunyuanVideo 1.5 (8B) — behind shared trainer, loss and dataset abstractions
• Camera-trajectory control at inference via pose strings or JSON, with translate and rotate keys per latent frame
• FSDP plus sequence parallelism for single- and multi-node training, and sequence-parallel sampling at inference
• Claude skills for debugging the training pipeline, integrating a new backbone and onboarding to world models

// Getting started

minWM installs as an editable Python package; the demo path is download the base and 4-step DMD checkpoints, then run one inference entrypoint. Full requirements and troubleshooting are in INSTALL.md.

conda create -n minwm python=3.12 -y
conda activate minwm
pip install -r requirements/base.txt
pip install flash-attn --no-build-isolation
pip install -e .

hf download Wan-AI/Wan2.1-T2V-1.3B --local-dir ./ckpts/Wan2.1-T2V-1.3B

hf download MIN-Lab/minWM --local-dir ./ckpts --include "Wan21/Action2V/dmd/*"
ln -sfnT dmd ./ckpts/Wan21/Action2V/stage3_ar_dmd

torchrun --nproc_per_node=1 tools/infer_mwm.py \
    --config-file configs/wan21/action2v/infer/stage3_ar_dmd.py \
    inference.benchmark=assets/example_t2v.json inference.limit=2 \
    inference.output_dir=./outputs/quickstart_wan_action2v

"trajectory": "d*8,i*5,l*6"

Commands and code are distilled from the project's own documentation — always check the official repo for the latest.

// When to use it

• Turn an existing text-to-video diffusion model into an interactive, camera-controllable world model instead of starting from scratch
• Study or reproduce the distillation path from a bidirectional diffusion teacher to a 4-step autoregressive student
• Integrate your own video DiT backbone against two worked reference integrations
• Onboard into world-model research without reverse-engineering a research repo first

// How minWM compares

minWM alongside other open-source video world models tools AI/TLDR tracks, ranked by GitHub stars.