Real-Time Interactive World Generation via Diffusion Distillation
Distill an autoregressive video world model using distribution matching to collapse multi-step sampling into a real-time generation pipeline.
Source paper
Jisu Nam, Yicong Hong, Chun-Hao Paul Huang, Feng Liu, JoungBin Lee, Jiyoung Kim, Siyoon Jin, Yunsung Lee, Jaeyoon Jung, Suhwan Choi, Seungryong Kim, Yang Zhou
Generated by AI. The paper’s authors did not write these.
Distill an autoregressive video world model using distribution matching to collapse multi-step sampling into a real-time generation pipeline.
Replace Euclidean distance memory retrieval with depth-aware field-of-view overlap metrics to prevent spatial confusion in complex multi-level environments.
Expand a camera-conditioned diffusion transformer with a secondary entity embedding stream to model independent motion in dynamic environments.
The paper introduces WorldCam, an interactive gaming world model based on video Diffusion Transformers (DiTs) that uses camera pose as a unifying geometric representation. By mapping user actions to a continuous Lie algebra action space and retrieving historical frames via global camera poses, the model achieves precise action control and long-horizon 3D consistency.
WorldCam translates user input actions into 6-DoF relative camera poses using the SE(3) manifold (Lie algebra) to accurately capture coupled translation and rotation. These poses are converted into Plücker embeddings and injected into the intermediate layers of a progressive autoregressive DiT. Long-horizon consistency is maintained by using global camera poses as spatial indices to retrieve and concatenate relevant historical latents from a long-term memory pool.