Hierarchical Dynamic Memory Aggregation for Long-Form Video Reasoning
Aggregate compact spectral clip tokens across long video sequences using dynamic key-value memory compression to enable hour-long temporal dynamics understanding.
Source paper
Generated by AI. The paper’s authors did not write these.
Aggregate compact spectral clip tokens across long video sequences using dynamic key-value memory compression to enable hour-long temporal dynamics understanding.
Combine frozen V-JEPA motion representations with text-aligned SigLIP features via dual-stream spectral projection to preserve dynamic reasoning while restoring static visual knowledge.
Synthesize action-conditioned counterfactual video outcomes in robotic physics environments to train video LLMs on physical causality and trajectory feasibility.
This paper investigates why Video Large Language Models (video LLMs) fail to read visual motion dynamics even when powered by predictive world-model encoders like V-JEPA. The authors demonstrate that standard text captioning and QA supervision allow language models to rely on single-frame static shortcuts rather than dynamic features. To address this, they propose counterfactual supervision using fourteen pixel-level video transformations that isolate change components, enabling state-of-the-art motion and temporal reasoning across eleven video benchmarks with significantly lower compute and data requirements.
The model attaches a frozen V-JEPA 2.1 predictive video encoder (ViT-L/ViT-g) to an open language model (Qwen3 or Llama-3) using a spectral bridge that compresses clips into 1,024 tokens. Training is performed in a single stage on 4.53 million public samples, augmented by counterfactual video pairs generated programmatically via six pixel-transformation families (sign flip, order, position, magnitude, count, and stop/cut) without language model annotations.