Overview
OpenWorldLib is an attempt to make world-model code callable in a consistent way. The project defines a world model as a perception-centred model or framework with interaction and long-term memory, used to understand and predict a complex world — which makes multimodal understanding, visual action prediction and visual generation all sub-tasks of the same thing. Rather than each method shipping its own bespoke scripts, OpenWorldLib puts them behind one standardised structure and curates the surrounding literature alongside the code.
The task coverage is broad: navigation video generation (lingbot, matrix-game, hunyuan-worldplay, genie3), long video generation (sora-2, veo-3, wan), egocentric interaction generation, 3D scene generation (flash-world, vggt), visual question answering with world-understanding VLMs, and vision-language-action models (pi-0, pi-0.5, giga-brain). Separate extension repositories cover 3D generation, VLA and a simulator.
The code is arranged around a small set of classes users compose directly. A `pipeline` handles weight loading and module initialisation through `from_pretrained()` and is what you normally call; `operator` validates and preprocesses input and interaction signals; `synthesis`, `reasoning` and `representation` produce multimodal, text and 3D outputs; and in multi-turn interaction a `memory` class retrieves context before a turn and records results after it, which is what `stream()` drives. Installation is per-method — a base conda environment plus a default install script, with special scripts under `scripts/setup` for methods that need them. Testing so far has used GPUs with 80 GB and 141 GB of VRAM.
What it does
- One pipeline API over many published world models, loaded with from_pretrained() and called directly
- Covers navigation and long video generation, egocentric interaction, 3D scene generation, VQA and vision-language-action models
- Explicit memory module for multi-turn interaction — select() retrieves context before a turn, record() stores the result after it
- Separate synthesis, reasoning and representation classes for multimodal, text and 3D outputs
- Ready-made inference and streaming test scripts per method under scripts/test_inference and scripts/test_stream
- Benchmark evaluation code and test data shipped in the repository
- Extension repositories for 3D generation, VLA and a simulator, plus a curated world-model literature list
- Apache-2.0, with a development guide that acts as a template for contributing a new method
Getting started
Installation is a base conda environment plus a default script; individual methods can have their own requirements, and all installation scripts live under ./scripts/setup. The full guide is docs/installation.md.
Create the environment
OpenWorldLib targets Python 3.10.
conda create -n "openworldlib" python=3.10 -y
conda activate "openworldlib"Install
Run the default install script from the repository root. Methods with special requirements have their own scripts in the same directory.
cd OpenWorldLib
bash scripts/setup/default_install.shRun a first generation
Test navigation video generation with matrix-game-2, then the multi-turn interactive form of the same method.
cd OpenWorldLib
bash scripts/test_inference/test_nav_video_gen.sh matrix-game-2
bash scripts/test_stream/test_nav_video_gen.sh matrix-game-2Compose it in your own code
Call a pipeline directly — it loads weights and initialises modules. Input goes through the operator for validation and preprocessing, then to synthesis, reasoning or representation depending on the output you want. In multi-turn use, stream() wraps each turn with a memory select and record.
Commands and code are distilled from the project's own documentation — always check the official repo for the latest.
When to use it
- Reach for it to compare several published world models without adapting to each project's own scripts
- Reach for it when a multi-turn interactive session needs context carried across turns rather than regenerated each time
- Reach for it to run navigation or long-video world models and 3D scene generation from one installation
- Reach for it as the integration target when publishing a new world-model method you want others to call the same way
How OpenWorldLib compares
OpenWorldLib alongside other open-source video world models tools AI/TLDR tracks, ranked by GitHub stars.
| Tool | Stars | What it does |
|---|---|---|
| LongLive | ★ 2.6k | NVIDIA's training and inference infrastructure for real-time interactive long video generation, with NVFP4 4-bit precision, sequence parallelism and few-step distillation. |
| OpenWorldLib | ★ 872 | One inference stack for world models across video, 3D, reasoning and action |
| minWM | ★ 830 | A full-stack framework for turning a bidirectional text-to-video diffusion model into an action-conditioned, few-step autoregressive world model, covering data, training, distillation and real-time inference. |