Text-Guided Multimodal Memory Retrieval
Add a cross-attention layer to the memory bank that queries text embeddings to prevent identity swaps in multi-character video generation.
Source paper
Kaiwen Zhang, Liming Jiang, Angtian Wang, Jacob Zhiyuan Fang, Tiancheng Zhi, Qing Yan, Hao Kang, Xin Lu, Xingang Pan
Generated by AI. The paper’s authors did not write these.
Add a cross-attention layer to the memory bank that queries text embeddings to prevent identity swaps in multi-character video generation.
Condition diffusion models on multi-frame motion latents extracted from previous shots to preserve video speed and motion continuity.
Integrate entity segmentation models into the memory update pipeline to store cropped character latents separately from background noise.
The paper introduces StoryMem, a framework that enables pre-trained single-shot video diffusion models to generate coherent, multi-shot long videos. It employs a Memory-to-Video (M2V) mechanism that stores and retrieves visual keyframes from previous shots to maintain narrative consistency across minute-long sequences.
The method adapts the Wan2.2-I2V diffusion transformer (DiT) using LoRA fine-tuning. It utilizes a dynamic memory bank of keyframes, selected via CLIP semantic similarity and HPSv3 aesthetic filtering, which are injected into the generation process through latent concatenation and a novel negative Rotary Positional Embedding (RoPE) shift strategy to represent past context.