Disentangled Part-Based Subject Encoding
Incorporate semantic parsing into subject image encoders to disentangle invariant facial features from variant clothing tokens, enabling targeted cross-attention editing without identity leakage.
Source paper
Yuanhao Cai, He Zhang, Xi Chen, Jinbo Xing, Yiwei Hu, Yuqian Zhou, Kai Zhang, Zhifei Zhang, Soo Ye Kim, Tianyu Wang, Yulun Zhang, Xiaokang Yang, Zhe Lin, Alan Yuille
Generated by AI. The paper’s authors did not write these.
Incorporate semantic parsing into subject image encoders to disentangle invariant facial features from variant clothing tokens, enabling targeted cross-attention editing without identity leakage.
Generate synthetic video pairs with procedurally swapped assets in a 3D engine to train models on motion-consistent instructive editing without domain gaps.
Introduce a global interaction attention block to model physical contact and mutual shadowing between multiple customized subjects in video generation.
This paper introduces OmniVCus, a feedforward diffusion transformer framework designed for subject-driven video customization that supports multiple subjects and multimodal control signals (depth, mask, camera, text) without requiring test-time tuning. The authors propose a data pipeline, VideoCus-Factory, to generate training pairs from raw videos, and introduce Lottery Embedding (LE) and Temporally Aligned Embedding (TAE) to enable scalable multi-subject inference and precise spatiotemporal control.
The method uses a Diffusion Transformer (DiT) where reference images and control signals are tokenized and concatenated with noise latents. Key innovations include Lottery Embedding (LE), which randomly selects subject slots during training to allow inference with more subjects than seen in training, and Temporally Aligned Embedding (TAE), which aligns dense control tokens (masks/depth) with noise tokens to enhance structural guidance. An Image-Video Transfer Mixed (IVTM) training strategy is used to learn editing capabilities from image datasets.