Learned Query-Aware Importance Prediction
Train a lightweight Multi-Layer Perceptron to predict token importance scores from keys and query statistics, avoiding expensive full attention matrices.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section 3.2 justifies using Key-similarity for pruning because calculating full attention weights is too expensive (O(N2)) and breaks FlashAttention optimization.
Limitation
Using only Keys for pruning is heuristic and 'blind' to what the model actually needs. A token might be unique (high diversity) but irrelevant to the current query (e.g., a distractor object). This leads to suboptimal cache usage compared to true attention-based pruning.
Proposed Solution
Train a lightweight 'Importance Head' (a small MLP) that takes the current Key and a running average of recent Query statistics to predict a scalar importance score. This approximates the attention weight without materializing the full matrix, allowing the cache to prioritize tokens that are likely to be attended to.
Why This Matters
Optimizes the information-per-byte ratio in the cache, allowing for longer history retention or smaller memory footprints by discarding diverse but useless data.
How to Get Started
- Collect a dataset of key tensors and running query statistics from a pre-trained model run.
- Design a small MLP architecture that takes keys and query statistics as input and outputs a scalar importance score.
- Train the MLP using mean squared error against true full-attention weights as supervision targets.
- Integrate the trained importance head into the caching mechanism and measure inference speedup versus the standard key-similarity baseline.
Code linked to this paper on Hugging Face: github.com/AutoLab-SAI-SJTU/InfiniteVGGT
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like Delta Attention and On the Limits of Learned Importance Scoring for KV Cache Compression explore approximation methods and learned importance scoring for attention and KV cache compression. However, none explicitly detail the exact proposal of using a lightweight MLP combining current Keys and query statistics to approximate attention weights without full matrix materialization.
- Delta Attention: Fast and Accurate Sparse Attention Inference by Delta Correction · 2025 · 11 citations
- On the Limits of Learned Importance Scoring for KV Cache Compression · 2026 · 1 citations
We searched OpenAlex for “Train a lightweight Multi-Layer Perceptron to predict token importance scores from keys and query statistics, avoiding expensive full attention matrices.” and “Learned Query-Aware Importance Prediction” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
InfiniteVGGT: Visual Geometry Grounded Transformer for Endless StreamsShuai Yuan, Yantai Yang, Xiaotian Yang +4 more · 2026
This paper introduces InfiniteVGGT, a transformer-based model for continuous 3D geometry reconstruction from infinite video streams without memory overflow. The authors propose a rolling memory mechanism that dynamically prunes the KV cache based on key-diversity (cosine similarity) rather than attention weights, enabling compatibility with FlashAttention and long-term stability.
Related Ideas
- Granular Importance-Aware Sparse RetrievalLMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference🔴Hard
- Adaptive Attention-Based Memory CompressionHelios: Real Real-Time Long Video Generation Model🟠Medium
- Importance-Weighted Semantic Memory Compaction for Ultra-Long Horizon TasksGamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent🟡Fairly easy
- Efficient Input-Dependent Queries for Attention ResidualsAttention Residuals🟠Medium
- End-to-End Trainable Sparse Attention Indexers for RLGLM-5: from Vibe Coding to Agentic Engineering🔴Hard