Infinite-Horizon Attention via Compressed Memory Caching in Hierarchical Denoising
Integrate continuous memory compression into hierarchical video denoising to maintain an infinite-horizon context alongside a dense recent sliding window.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In the Implementation Details (Section 4.1), the autoregressive generation relies on a sliding-window KV cache with a strict, constant attention window of 5 seconds.
“At inference time, we employ a sliding-window KV cache with a constant attention window of 5 s.”
Limitation
Truncating the context to exactly 5 seconds means the model suffers from complete amnesia regarding earlier events. This prevents the generation of long-term narrative dependencies, returning characters, or cyclic actions spanning longer than the 5-second window.
Proposed Solution
Integrate a continuous memory compression module (such as streaming summarization tokens or a localized Token Turing Machine) into the hierarchical denoising pipeline. This allows the KV cache to store a dense 5-second window alongside a compressed, infinite-horizon representation of all prior blocks.
Why This Matters
This would transition the model from simply generating 'infinite continuous video' to generating 'narratively coherent infinite video', which is a critical step toward true interactive world models.
How to Get Started
How to test it: Hypothesize that incorporating compressed long-horizon memory tokens into the KV cache reduces character identity drift over 30s+ generation. Compare against HiAR with standard 5s sliding window using long-video VBench semantic consistency and drift metrics. Stop if character consistency does not improve or inference throughput drops by >20%.
- Examine the baseline sliding-window KV cache implementation and memory usage profiles in video foundation models.
- Implement a continuous memory compression module using streaming summarization tokens or localized memory banks.
- Integrate the compressed memory caching mechanism into the hierarchical denoising autoregressive pipeline.
- Test long-horizon generation stability and memory footprint reduction against standard fixed-window baselines.
Code linked to this paper on Hugging Face: github.com/Jacky-hate/HiAR
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokens and Addressable Memory for Video World Models explore memory caching and long-horizon video generation mechanisms. However, they do not specifically integrate a continuous memory compression module combining a dense short-term window with a compressed infinite-horizon representation in hierarchical denoising as proposed.
- ReCA: Multi-Shot Long Video Extrapolation via Recursive Context Allocation · 2026 · 8 citations
- MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokens · 2026 · 32 citations
- Addressable Memory for Video World Models · 2026 · 21 citations
- Long-Horizon Streaming Video Generation via Hybrid Attention with Decoupled Distillation · 2026 · 9 citations
We searched OpenAlex and arXiv for “Integrate continuous memory compression into hierarchical video denoising to maintain an infinite-horizon context alongside a dense recent sliding window.” and “Infinite-Horizon Attention via Compressed Memory Caching in Hierarchical Denoising” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
HiAR: Efficient Autoregressive Long Video Generation via Hierarchical DenoisingKai Zou, Dian Zheng, Hongbo Liu +3 more · 2026
This paper proposes HiAR, an autoregressive framework for long video generation that mitigates distribution drift by utilizing hierarchical denoising. By conditioning generated blocks on context at matched noise levels instead of fully clean frames, it effectively reduces error propagation while enabling pipelined parallel inference.
Related Ideas
- Sliding-Window Attention with Global Context for Long-Form SynthesisLTX-2: Efficient Joint Audio-Visual Foundation Model🟠Medium
- Causal-SLA for Autoregressive Video GenerationTurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times🔴Hard
- Hierarchical Dynamic Memory Aggregation for Long-Form Video ReasoningV-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations🟠Medium
- Hierarchical Decay Mechanisms for Infinite-Context MemorytttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction🟡Fairly easy
- Dynamic Semantic Attention Sinks for Long-Horizon StreamingLiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation🟠Medium