End-to-End Joint Optimization of MLLM and MMDiT
Fine-tune both the multimodal language model and the video diffusion generator end-to-end using parameter-efficient adapters so the understanding module aligns its hidden states with generative physics.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
In Section 3.3.1 and Section C, the authors explicitly state the MLLM is kept frozen to preserve understanding capabilities. This results in an 'assembled' system where the understanding module cannot adapt its internal representations to better suit the specific requirements of the generative diffusion process.
Limitation
Because the MLLM is frozen, the semantic features it produces are optimized for next-token prediction (text generation) rather than being optimal conditioning signals for video generation, potentially creating a semantic bottleneck.
Proposed Solution
Implement an end-to-end training strategy where the MLLM is unfrozen or fine-tuned using parameter-efficient methods (e.g., LoRA) alongside the MMDiT. This allows the MLLM to learn to output hidden states that are specifically aligned with the manifold of the video generator, bridging the gap between linguistic understanding and generative physics.
Why This Matters
Enabling the MLLM to adapt to generation tasks would create a truly 'native' multimodal model, likely improving instruction adherence and the handling of complex visual prompts.
How to Get Started
- Download an open-source MLLM and a compatible video diffusion model codebase.
- Implement a training script integrating LoRA into both the vision-language encoder and the diffusion transformer backbone.
- Set up a small video-text dataset for a multi-gpu cluster to test stability and monitor loss convergence.
- Evaluate the jointly trained model on standard video generation benchmarks to measure visual quality and alignment.
Code linked to this paper on Hugging Face: github.com/KlingTeam/UniVideo
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like Exploring MLLM-Diffusion Information Transfer with MetaCanvas and JoyAI-Image explore the intersection of multimodal language models and diffusion/generative architectures for understanding and generation. However, none of the listed papers specifically details the end-to-end joint fine-tuning of an MLLM with an MMDiT using parameter-efficient methods like LoRA to bridge linguistic understanding and generative physics.
- Exploring MLLM-Diffusion Information Transfer with MetaCanvas · 2025 · 6 citations
- JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generation · 2026 · 34 citations
We searched OpenAlex for “Fine-tune both the multimodal language model and the video diffusion generator end-to-end using parameter-efficient adapters so the understanding module aligns its hidden states with generative physics.” and “End-to-End Joint Optimization of MLLM and MMDiT” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
UniVideo: Unified Understanding, Generation, and Editing for VideosCong Wei, Quande Liu, Zixuan Ye +5 more · 2025
UniVideo introduces a unified framework capable of video understanding, generation, and editing within a single system by integrating a Multimodal Large Language Model (MLLM) with a Multimodal Diffusion Transformer (MMDiT). The approach allows the model to process complex multimodal instructions (text, images, video) via the MLLM and project semantic guidance to the diffusion generator, achieving state-of-the-art results across multiple video tasks without task-specific adapters.
Related Ideas
- Adaptive Token-Level Gating for Heterogeneous Modality AlignmentInternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing🟡Fairly easy
- Decoupled Asymmetric Modality Branching in Vision-Language TransformersMulti-Branch Transformer🔴Hard
- Scaling Unified Multimodal Models via Large Pre-trained Vision-Language InitializationsCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation🔴Hard
- Extending Decoupled Semantic-Detail Representations to Video ModalitiesCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation⛔Very hard
- Modality-Dependence Reweighting and Interleaved Sparse Attention for Omni-Modal Audio-Visual-Speech ModelsMMFlash: Faster Speculative Decoding for Vision-Language Models🔴Hard