Modality-Dependence Reweighting and Interleaved Sparse Attention for Omni-Modal Audio-Visual-Speech Models
Generalize modality-dependence loss reweighting and interleaved sparse attention to audio-visual and streaming speech models so speculative decoding accelerates omni-modal generation.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Appendix F (Limitations & Future Work), the authors note that MMFlash is designed and evaluated solely for VLMs taking image or video inputs and generating text. Its visual dependence score (Equation 1) only contrasts with-image vs. without-image distributions, and its sparse attention pattern (Section 4.2) assumes contiguous visual tokens followed by text tokens.
“Another direction is to extend MMFlash to omni-modal models [59, 68], which also take audio as input and generate streaming speech.”
Limitation
Omni-modal architectures (such as Qwen2.5-omni and Qwen3.5-omni) take continuous audio alongside video in long interleaved contexts and generate streaming speech tokens in addition to text; text-only block-diffusion drafters and image-only KL reweighting cannot isolate audio-dependent or cross-modal audio-visual tokens nor handle interleaved multi-rate audio-visual KV caches.
Proposed Solution
Extend Equation 1 to multi-perturbation modality dependence by measuring the maximum or Shapley-style KL divergence of target next-token distributions under audio-ablated, visual-ablated, and jointly ablated prefixes, enabling separate reweighting of audio-grounded and joint audio-visual tokens. Pair this with an interleaved sparse attention mask that preserves temporal alignment across audio and video frames via synchronized striding and sink tokens, and adapt the block-diffusion drafter to output dual-stream text and discrete speech-codec tokens.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Extending modality-aware diffusion drafting to omni-modal models is expected to unlock lossless speculative decoding speedups for real-time voice and video assistants, where streaming speech latency is critical.
How to Get Started
How to test it: Test whether multi-perturbation modality-dependence reweighting improves draft acceptance length and end-to-end streaming speedup over unweighted DFlash and autoregressive baselines on an open-source omni-modal model (e.g., Qwen2.5-Omni-7B) across audio-visual QA and speech-generation benchmarks. Stop if speech-token acceptance remains too low (<2 tokens/block) to yield >1.3x real-time speedup.
- Profile token acceptance rates of an unweighted DFlash-style drafter on Qwen2.5-Omni-7B across audio-only, video-only, and joint audio-visual QA tasks, grouped by audio- and visual-ablation KL quartiles.
- Extend MMFlash's loss reweighting to use audio-ablated and visual-ablated KL dependence scores on regenerated audio-visual training samples.
- Design an interleaved audio-visual sparse attention pattern that retains synchronized audio-video anchors, all text tokens, and a local decoding window.
- Evaluate lossless text and speech-codec decoding speedups (tok/s and time-to-first-audio-chunk) under single-request and concurrent serving.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
General omni-modal models and benchmarks like OmniJigsaw, LongCat-Flash-Omni, and Hallo-Live are surveyed or introduced in recent literature, but none evaluate Shapley-style modality-dependence reweighting or interleaved sparse attention for dual-stream text and speech-codec block-diffusion drafters.
- OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering · 2026 · 1 citations
- LongCat-Flash-Omni Technical Report · 2025 · 46 citations
- Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillation · 2026 · 16 citations
We searched OpenAlex and arXiv for “speculative decoding omni modal audio visual streaming speech” and “Modality-Dependence Reweighting and Interleaved Sparse Attention for Omni-Modal Audio-Visual-Speech Models” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
MMFlash: Faster Speculative Decoding for Vision-Language ModelsSicheng Feng, Zigeng Chen, Song Wang +2 more · 2026
MMFlash introduces a diffusion-based speculative decoding framework designed to accelerate inference in vision-language models (VLMs) while preserving exact target output distributions. After diagnosing that draft token acceptance drops sharply on visually dependent tokens even with multimodal training, the paper proposes a visually grounded loss reweighting scheme and a sparse attention pattern for long video contexts. Across seven image and three video benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58x on Qwen3.5 (and up to 4.71x on Qwen3.8-27B), outperforming autoregressive and text-focused diffusion drafters.
Related Ideas
- Decoupled Asymmetric Modality Branching in Vision-Language TransformersMulti-Branch Transformer🔴Hard
- Latent Visual-Audio Contrastive BindingDreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation🟠Medium
- Adaptive Token-Level Gating for Heterogeneous Modality AlignmentInternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing🟡Fairly easy
- End-to-End Speech-to-Speech Integration via Latent S3 Token AlignmentFunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs🔴Hard
- Dual-Stream Vision Fusion for Concurrent Motion Dynamics and Text/OCR RecognitionV-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations🟡Fairly easy