Interaction-Aware Multi-Subject Modeling
Introduce a global interaction attention block to model physical contact and mutual shadowing between multiple customized subjects in video generation.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
In Section 3.2, the 'Lottery Embedding' (LE) mechanism enables the model to handle more subjects than it was trained on by assigning them available embedding slots. However, the training data and objective treat subjects largely as independent entities to be placed in the scene. The paper demonstrates spatial composition (Fig 1e) but lacks examples of complex physical interactions (e.g., hugging, handshakes) where boundaries and lighting between subjects are critical.
Limitation
The current slot-based embedding approach scales the number of subjects but does not explicitly model physical interactions or occlusions between multiple custom subjects.
Proposed Solution
Augment the VideoCus-Factory to specifically filter for 'high-interaction' clips using action recognition models. Introduce a 'Global Interaction Attention' block in the DiT architecture that computes attention specifically between the token sets of different subjects before integrating them into the spatial latent. This forces the model to resolve boundary interactions and mutual shadowing between customized identities.
Why This Matters
Enabling realistic physical interactions between multiple custom subjects is crucial for narrative video generation, moving beyond simple collage-style composition.
How to Get Started
- Filter public video action datasets like Kinetics or AVA to isolate high-interaction multi-subject clips such as hugs and handshakes.
- Implement a Global Interaction Attention module that computes cross-attention between token sets of different subjects prior to spatial latent integration.
- Establish evaluation baselines using standard multi-subject video generation metrics on the filtered dataset.
- Train the modified diffusion transformer architecture on the interaction-heavy video subset and measure boundary fidelity.
Code linked to this paper on Hugging Face: github.com/caiyuanhao1998/Open-OmniVCus
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed papers describe a 'Global Interaction Attention' block in a DiT architecture designed to resolve boundary interactions and mutual shadowing between customized identities. Works like CoVEBench and SubjectAnchor discuss video editing and multi-shot storytelling, but not this exact architectural mechanism.
- CoVEBench: Can Video Editing Models Handle Complex Instructions? · 2026 · 4 citations
- SubjectAnchor: Subject-Aware Memory-to-Video for Multi-Shot Storytelling · 2026 · 5 citations
We searched OpenAlex for “Introduce a global interaction attention block to model physical contact and mutual shadowing between multiple customized subjects in video generation.” and “Interaction-Aware Multi-Subject Modeling” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control ConditionsYuanhao Cai, He Zhang, Xi Chen +11 more · 2025
This paper introduces OmniVCus, a feedforward diffusion transformer framework designed for subject-driven video customization that supports multiple subjects and multimodal control signals (depth, mask, camera, text) without requiring test-time tuning. The authors propose a data pipeline, VideoCus-Factory, to generate training pairs from raw videos, and introduce Lottery Embedding (LE) and Temporally Aligned Embedding (TAE) to enable scalable multi-subject inference and precise spatiotemporal control.
Related Ideas
- Multi-Entity Action-State Modeling for Interactive EnvironmentsWorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling🔴Hard
- Spatially-Guided Attention Masks for Multi-Speaker DisentanglementLTX-2: Efficient Joint Audio-Visual Foundation Model🔴Hard
- Multi-Agent World Modeling: Extending Camera-Conditioned DiTs to Dynamic EntitiesWorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation⛔Very hard
- Extending Spatial Curation to Dynamic Scenes via 4D RepresentationHoli-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence🔴Hard
- Hierarchical Dynamic Memory Aggregation for Long-Form Video ReasoningV-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations🟠Medium