JD.com · 2026-08-24 · major
EchoWM — a world model you can walk through, with sound and speech
EchoWM is an omnimodal world model from JD.com's Joy Future Academy that generates 720p video together with matching environmental sound, music and speech, while a viewer steers the camera through the scene step by step.
An enterable world model that keeps picture, ambient sound and speech in sync while you navigate the scene.
Key specs
| Wbench navigation (avg) | 81.7 |
|---|---|
| Interaction score | 87.2 |
Quick facts
| Maker | JD.com Joy Future Academy with academic collaborators |
|---|---|
| Type | Omnimodal world model for navigable scenes |
| Outputs | 720p video, environmental sound, music, speech |
| Control | Shared metric-scale relative 6-DoF camera trajectory |
| Paper | arXiv 2608.23189, submitted August 24, 2026 |
| Code repo | jd-opensource/JoyAI-Echo (1,901 stars) |
What is it?
Walking through a generated scene, rather than watching a fixed clip, is what EchoWM adds. The model responds to continuous navigation input and jointly produces 720p video, environmental sound, music and speech for wherever the camera goes. It handles both first-person scenes, where the camera is the observer, and third-person scenes, where the relationship between camera and character is learned from data instead of hand-built controllers.
How does it work?
Interaction in EchoWM is organised around camera intent. Discrete commands and continuous poses are both mapped onto one shared metric-scale relative 6-DoF trajectory, and a dataset-level calibration step keeps motion magnitude consistent across training sources that measure movement differently. Training runs in stages — a progressive schedule to learn audio-visual generation and trajectory control together, followed by autoregressive post-training so the model can keep going over long horizons and multi-turn continuations.
Why does it matter?
Most generative video gives you a clip you cannot enter, and audio is bolted on afterwards. By treating navigation and synchronized sound as first-class outputs, EchoWM points at interactive media — game-like scenes, simulators, training environments — that a person or an agent can move through. The team reports an 81.7 average on WBench Navigation across 158 cases, with the project page publishing playable-style demos of both camera modes.
Who is it for?
world-model and video-generation researchers
Frequently asked questions
- Can I download the EchoWM weights right now?
- The JoyAI-Echo Hugging Face repository publishes echo-longvideo-release.safetensors, the checkpoint for the JoyAI-Echo 1.5 long-video model, and the GitHub repository links the Echo-WM paper. The project does not state a release date for EchoWM's own checkpoint, so treat the world model as published research with code and demos available rather than as downloadable weights today.
- What license covers the JoyAI-Echo code and checkpoints?
- JoyAI-Echo is released for academic and research purposes only. The project is built on LTX-2 by Lightricks Ltd and remains subject to the LTX-2 Community License Agreement, with the original copyright notices retained. Commercial deployment is not permitted without approval from Lightricks. Anyone planning product work on top of EchoWM or JoyAI-Echo needs to check that license first.
- How is EchoWM different from a text-to-video model?
- A text-to-video model renders a fixed clip from a prompt. EchoWM instead accepts continuous navigation while generating, so the scene responds as the camera moves and keeps producing new frames over long horizons and multi-turn continuations. It also generates environmental sound, music and speech in sync with the picture, which typical video generators leave to a separate audio pass.
- What hardware do the Echo models need?
- The published JoyAI-Echo 1.5 checkpoint is roughly 46GB on its own, and it additionally requires the google/gemma-3-12b-it text encoder, around 24GB more. The Hugging Face repository indicates a requirement in the region of 46-50GB of GPU memory to run inference, which puts the released Echo family firmly in datacenter-GPU territory rather than consumer hardware.
Try it
Read the paper at arxiv.org/abs/2608.23189; inference code for the Echo family lives at github.com/jd-opensource/JoyAI-Echo.