Modality-Adaptive Flow Matching
Apply modality-specific rectified flow matching with independent velocity fields to optimize joint audio-video generation paths simultaneously.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 3.3 describes a complex 'Multi-Task Progressive Training' curriculum required because naive joint training fails due to conflicting objectives. The current symmetric diffusion process (Eq. 2 & 3) treats audio and video noise schedules identically despite their different informational densities.
“A naive joint training approach often suffers from conflicting learning objectives, where the generative objective of creating diverse content can interfere with the fidelity objective of adhering to strong conditional constraints.”
Limitation
The symmetric diffusion formulation creates optimization conflicts where one modality may dominate the gradient, necessitating a slow, multi-stage training pipeline to prevent collapse.
Proposed Solution
Replace the standard diffusion objective with Rectified Flow Matching using modality-specific velocity fields. Define separate flow trajectories for audio and video latents that share a global time variable but evolve with different curvatures (velocities), allowing the model to learn optimal transport paths for each modality simultaneously.
Why This Matters
Would likely remove the need for the complex three-stage training curriculum, stabilize convergence, and potentially improve generation quality by respecting the distinct manifold properties of audio vs. video.
How to Get Started
How to test it: Hypothesis: modality-specific rectified flow trajectories prevent gradient domination. Baseline: symmetric diffusion objective. Metric: convergence stability and training loss balance. Result: modality collapse or training divergence.
- Formulate separate ordinary differential equations and velocity field parameterizations for audio and video latent spaces.
- Modify the standard diffusion training pipeline to support distinct curvatures and noise schedules per modality under a shared time variable.
- Implement the modality-adaptive flow matching objective within the core model training loop.
- Compare generation sample quality and training convergence speed against the baseline symmetric diffusion model.
Code linked to this paper on Hugging Face: github.com/Guoxu1233/DreamID-Omni
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like SyncFlow and MOVA explore joint audio-video generation, and ModaFlow explores modality-aware flow matching, but they do not explicitly detail separate flow trajectories with modality-specific curvatures sharing a global time variable as proposed.
- SyncFlow: Toward Temporally Aligned Joint Audio-Video Generation from Text · 2024 · 28 citations
- MOVA: Towards Scalable and Synchronized Video-Audio Generation · 2026 · 60 citations
- ModaFlow: Modality-Aware Flow Matching for High-Fidelity Virtual Try-On · 2026 · 1 citations
We searched OpenAlex for “Apply modality-specific rectified flow matching with independent velocity fields to optimize joint audio-video generation paths simultaneously.” and “Modality-Adaptive Flow Matching” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video GenerationXu Guo, Fulong Ye, Qichao Sun +7 more · 2026
This paper presents DreamID-Omni, a unified generative framework capable of handling three human-centric tasks—reference-based audio-video generation (R2AV), video editing (RV2AV), and audio-driven animation (RA2V)—within a single model. By treating these tasks as variations of conditional generation, the authors achieve state-of-the-art performance in multi-speaker scenarios while ensuring consistent identity and voice timbre control.
Related Ideas
- Extending Decoupled Semantic-Detail Representations to Video ModalitiesCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generationâ›”Very hard
- Distilling Generative Video Depth Priors into Lightweight Linear-Complexity ModelsDVD: Deterministic Video Depth Estimation with Generative Priorsđź”´Hard
- Motion-Aware Transition Modeling via Multi-Frame ConditioningStoryMem: Multi-shot Long Video Storytelling with Memory🟡Fairly easy
- End-to-End Joint Optimization of MLLM and MMDiTUniVideo: Unified Understanding, Generation, and Editing for Videosđź”´Hard
- Optimization of Identity-Coherence via Direct Preference LearningDreamID-V:Bridging the Image-to-Video Gap for High-Fidelity Face Swapping via Diffusion Transformerđźź Medium