Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationNo close match found

Text-Guided Multimodal Memory Retrieval

Add a cross-attention layer to the memory bank that queries text embeddings to prevent identity swaps in multi-character video generation.

Time

several weeks

Compute level

Single GPU or less

Resources

Standard GPU

Prerequisites

transformer architectures, multimodal embeddings (CLIP/LLM)

Scope

modify existing code

The Gap

Appendix C highlights that the current memory update function is purely visual and discards textual information from previous prompts. Consequently, in multi-character scenarios (e.g., Fig. 7), the model fails to map specific prompt entities (e.g., 'Sheldon') to the correct visual memory, leading to identity swaps.

Limitation

The retrieval mechanism is ambiguous because it relies solely on visual similarity, lacking the semantic labels necessary to distinguish between different characters or objects stored in the memory bank.

Proposed Solution

Implement a multimodal memory bank that stores pairs of (visual_latent, text_embedding) for each keyframe. Modify the attention mechanism in the DiT to include a cross-attention layer where the current shot's text prompt queries the stored text embeddings. Use the resulting attention weights to selectively modulate the injection of visual memory latents, ensuring the model attends to the specific character or object requested by the text.

Why This Matters

This would solve the critical failure mode of identity confusion in complex, multi-subject narratives, significantly improving the model's utility for detailed storytelling.

How to Get Started

  1. Modify the memory bank data structure to store text-visual latent pairs for keyframes.
  2. Implement a text-to-visual cross-attention module inside the diffusion transformer blocks.
  3. Fine-tune the modified model using a multi-character video dataset.
  4. Evaluate identity consistency metrics against baseline models without textual memory querying.

Code linked to this paper on Hugging Face: github.com/Kevin-thu/StoryMem

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

None of the listed results implement a multimodal memory bank with text-queried cross-attention for DiT-based video generation. While papers like 'Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understanding' and 'Search over the Visual World: Persistent Visual Memory, Layered Indexes, and Source-Grounded Evidence' explore retrieval or memory for understanding tasks, they do not address modulating visual latent injection via text queries in diffusion transformers.

We searched OpenAlex and arXiv for “Add a cross-attention layer to the memory bank that queries text embeddings to prevent identity swaps in multi-character video generation.” and “Text-Guided Multimodal Memory Retrieval” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

StoryMem: Multi-shot Long Video Storytelling with Memory

Kaiwen Zhang, Liming Jiang, Angtian Wang +6 more · 2025

The paper introduces StoryMem, a framework that enables pre-trained single-shot video diffusion models to generate coherent, multi-shot long videos. It employs a Memory-to-Video (M2V) mechanism that stores and retrieves visual keyframes from previous shots to maintain narrative consistency across minute-long sequences.

Related Ideas

Is this a good idea?

Share onXLinkedIn