S-Lab, NTU · 2026-09-03 · notable
Puffin-World — an open 3D world model with physics, depth and camera
Puffin-World is a multimodal model that handles camera understanding, 3D world generation and reconstruction in one architecture. The team published code, three checkpoints and Puffin-16M, a set of 15M vision-language-camera triplets.
One open model reads camera pose, depth and physics from images, then generates and reconstructs 3D scenes.
What is it?
Puffin-World models three native world states — physics, geometry and appearance — inside a single multimodal architecture, together with what the authors call an Omni-Camera representation. It covers camera understanding, free-viewpoint spatial simulation, 3D world generation and reconstruction without calling external offline modules. Three checkpoints are public: a 7B Base, a 1.5B Pro and a 0.8B understanding-only Caption model.
How does it work?
The Omni-Camera representation lets the model propagate physical dynamics across future frames while keeping the scene consistent. Training used Puffin-16M, a new dataset of 15 million vision-language-camera triplets and 1 million motion trajectories, plus physical annotations covering about 44.5 million images drawn from 28 public datasets.
Why does it matter?
Camera-centric 3D work normally means stitching a pose estimator, a depth model and a generator together. The paper reports state-of-the-art camera understanding on Stanford2D3D, MegaDepth, TartanAir and LaMAR from a single model, and the code, weights and data are published under the NTU S-Lab License 1.0.
Who is it for?
3D vision and robotics researchers
Try it
python scripts/demo/world_modeling.py --model Puffin-World --checkpoint checkpoints/Puffin-World-Pro.pth --dataset re10k