Sliding-Window Attention with Global Context for Long-Form Synthesis
A system that uses sliding-window attention with a global context cache to generate long-form audiovisual sequences without temporal drift or synchronization loss.
Source paper
Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, Andrew Kvochko, Avishai Berkowitz, Daniel Shalem, Daphna Lifschitz, Dudu Moshe, Eitan Porat, Eitan Richardson, Guy Shiran, Itay Chachy, Jonathan Chetboun, Michael Finkelson, Michael Kupchick, Nir Zabari, Nitzan Guetta, Noa Kotler, Ofir Bibi, Ori Gordon, Poriya Panet, Roi Benita, Shahar Armon, Victor Kulikov, Yaron Inger, Yonatan Shiftan, Zeev Melumian, Zeev Farbman
Generated by AI. The paper’s authors did not write these.
A system that uses sliding-window attention with a global context cache to generate long-form audiovisual sequences without temporal drift or synchronization loss.
A method that applies spatial cross-attention masks derived from face tracking to bind distinct audio streams to specific speakers in multi-speaker video generation.
A method that trains language-specific adapters with a lip-reading loss to improve lip-sync and audio quality in underrepresented languages.
This paper presents LTX-2, an efficient, open-source multimodal foundation model capable of jointly generating synchronized video and audio from text prompts. By utilizing an asymmetric dual-stream transformer architecture with bidirectional cross-attention, the model achieves state-of-the-art audiovisual alignment and inference speed, overcoming the latency and coherence issues found in sequential (video-then-audio) generation pipelines.
The model employs an asymmetric dual-stream Diffusion Transformer (DiT) consisting of a 14B parameter video stream and a 5B parameter audio stream. These streams process decoupled latents from modality-specific VAEs and communicate via bidirectional cross-attention layers and cross-modality AdaLN for timestep conditioning. Text conditioning is handled by a Gemma 3 encoder using a multi-layer feature extractor and 'thinking tokens' to enhance prompt adherence.