Skip to content

Source paper

OmniForcing: Unleashing Real-time Joint Audio-Visual Generation

Yaofeng Su, Yuming Li, Zeyue Xue and 7 more

Yaofeng Su, Yuming Li, Zeyue Xue, Jie Huang, Siming Fu, Haoran Li, Ying Li, Zezhong Qian, Haoyang Huang, Nan Duan

arXiv 2603.11647 · 202620 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces OmniForcing, the first framework to distill a heavy, bidirectional dual-stream audio-visual diffusion model into a real-time streaming autoregressive generator. By mitigating training instabilities caused by severe modality asymmetry and token sparsity, the framework achieves state-of-the-art streaming generation at 25 FPS on a single GPU while preserving synchronization and visual quality.

How it works (AI summary)

OmniForcing uses a three-stage distillation pipeline (Bidirectional DMD, Causal ODE Regression, and Joint Self-Forcing) to convert an offline diffusion model into a causal engine. Key innovations include Asymmetric Block-Causal Alignment with a Global Prefix to handle VAE stride mismatch, an Audio Sink Token mechanism with Identity RoPE to prevent gradient explosion from token sparsity, and a Modality-Independent Rolling KV-Cache for parallel inference.