Skip to content

Source paper

UniVideo: Unified Understanding, Generation, and Editing for Videos

Cong Wei, Quande Liu, Zixuan Ye and 5 more

Cong Wei, Quande Liu, Zixuan Ye, Qiulin Wang, Xintao Wang, Pengfei Wan, Kun Gai, Wenhu Chen

arXiv 2510.08377 · 202571 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

UniVideo introduces a unified framework capable of video understanding, generation, and editing within a single system by integrating a Multimodal Large Language Model (MLLM) with a Multimodal Diffusion Transformer (MMDiT). The approach allows the model to process complex multimodal instructions (text, images, video) via the MLLM and project semantic guidance to the diffusion generator, achieving state-of-the-art results across multiple video tasks without task-specific adapters.

How it works (AI summary)

The model utilizes a dual-stream architecture: an 'understanding stream' based on a frozen Qwen2.5-VL MLLM and a 'generation stream' based on HunyuanVideo MMDiT. The MLLM processes inputs and outputs hidden states, which are aligned via a trained MLP connector to condition the MMDiT. Training occurs in three stages: connector alignment, generator fine-tuning, and multi-task training on text-to-video, image-to-video, and in-context editing tasks.