Skip to content

Source paper

OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions

Yuanhao Cai, He Zhang, Xi Chen and 11 more

Yuanhao Cai, He Zhang, Xi Chen, Jinbo Xing, Yiwei Hu, Yuqian Zhou, Kai Zhang, Zhifei Zhang, Soo Ye Kim, Tianyu Wang, Yulun Zhang, Xiaokang Yang, Zhe Lin, Alan Yuille

arXiv 2506.23361 · 20251 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces OmniVCus, a feedforward diffusion transformer framework designed for subject-driven video customization that supports multiple subjects and multimodal control signals (depth, mask, camera, text) without requiring test-time tuning. The authors propose a data pipeline, VideoCus-Factory, to generate training pairs from raw videos, and introduce Lottery Embedding (LE) and Temporally Aligned Embedding (TAE) to enable scalable multi-subject inference and precise spatiotemporal control.

How it works (AI summary)

The method uses a Diffusion Transformer (DiT) where reference images and control signals are tokenized and concatenated with noise latents. Key innovations include Lottery Embedding (LE), which randomly selects subject slots during training to allow inference with more subjects than seen in training, and Temporally Aligned Embedding (TAE), which aligns dense control tokens (masks/depth) with noise tokens to enhance structural guidance. An Image-Video Transfer Mixed (IVTM) training strategy is used to learn editing capabilities from image datasets.