Adaptive Attention-Based Memory Compression
A method that replaces fixed history compression kernels with a learnable token merging mechanism to dynamically preserve high-importance context tokens during inference.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 3.3.1 describes 'Multi-Term Memory Patchification,' which partitions history into short, mid, and long-term buckets with fixed, hard-coded compression kernels (e.g., 4x, 2x, 1x). This heuristic assumes all history within a timeframe is equally important.
Limitation
The fixed compression kernels inevitably discard semantically critical details in the 'long-term' history while potentially preserving redundant information in the 'short-term' history, limiting the model's ability to recall specific past events in complex narratives.
Proposed Solution
Replace the fixed convolution kernels with a Learnable Token Merging (ToMe) or KV-cache compression mechanism. Use a lightweight scoring network to calculate the 'recall importance' of history tokens dynamically during inference. High-importance tokens should be preserved at full resolution, while low-importance background tokens are heavily compressed, regardless of their temporal distance.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Moving from static to dynamic memory compression would allow the model to maintain coherence over much longer durations (e.g., 10+ minutes) by efficiently storing only relevant context, rather than blindly compressing older frames.
How to Get Started
How to test it: Replace static Multi-Term Memory Patchification kernels with lightweight learnable token merging (ToMe) based on dynamic recall importance scores. Compare against Helios-Base on 1440-frame long video prompts from HeliosBench using the Drifting Semantic and Drifting Aesthetic metrics. Stop if dynamic token merging increases token overhead beyond real-time latency budgets (<15 FPS).
- Implement a lightweight scoring network to evaluate the recall importance of history tokens across different temporal buckets.
- Integrate a learnable token merging or dynamic KV-cache compression mechanism into the attention layers.
- Fine-tune the compression network alongside the core model using standard language modeling objectives.
- Evaluate memory consumption and task performance against fixed-kernel baselines on long-context benchmarks.
Code linked to this paper on Hugging Face: github.com/PKU-YuanGroup/Helios
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as 'Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs' and 'Reducing Token Redundancy in LVLMs: A Systematic Review of Token Pruning Methods' explore dynamic token compression and KV-cache management based on importance scoring. However, they do not specifically establish the proposed combination of a lightweight scoring network with Learnable Token Merging for arbitrary history token preservation.
- V-CoLA: Vision Token Compression with Linear Attention · 2026 · 0 citations
- Reducing Token Redundancy in LVLMs: A Systematic Review of Token Pruning Methods · 2026 · 0 citations
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs · 2023 · 482 citations
We searched OpenAlex and arXiv for “A method that replaces fixed history compression kernels with a learnable token merging mechanism to dynamically preserve high-importance context tokens during inference.” and “Adaptive Attention-Based Memory Compression” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Helios: Real Real-Time Long Video Generation ModelShenghai Yuan, Yuanyang Yin, Zongjian Li +3 more · 2026
Helios introduces a 14-billion parameter autoregressive video generation model capable of synthesizing minute-scale videos at 19.5 FPS on a single H100 GPU. The authors achieve this by adapting a pre-trained bidirectional transformer with novel token compression techniques, drift-mitigation strategies that avoid expensive training-time rollouts, and a hierarchical adversarial distillation pipeline.
Related Ideas
- Importance-Weighted Semantic Memory Compaction for Ultra-Long Horizon TasksGamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent🟡Fairly easy
- Visual Token Compression for Extended Context HistoryFara-7B: An Efficient Agentic Model for Computer Use🔴Hard
- Learned Query-Aware Importance PredictionInfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams🟠Medium
- Prompt-Relevant Historical Memory Retrieval and Entropy-Calibrated Layer-Head KV CompressionMovieRoll: A Systematic Framework for Rolling Out Long-Form Videos🟡Fairly easy
- Hierarchical Dynamic Memory Aggregation for Long-Form Video ReasoningV-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations🟠Medium