Compressive Memory for Agentic Contexts
Replace heuristic history dropping with a compressive memory mechanism that encodes older context chunks into soft tokens retained in the attention cache.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 4.2.4, the paper admits that model accuracy 'degrades substantially' beyond 100k tokens in agentic loops, despite being trained on 200k contexts. Their current solution is a heuristic 'Keep-recent-k' strategy that simply discards older history, causing the loss of early constraints or observations.
“Prior work [26] has introduced context management, where Discard-all resets the context by removing the entire history of tool calls.”
Limitation
The reliance on heuristic context truncation (Discard-all/Keep-recent) lobotomizes the agent's long-term memory, preventing it from recalling critical information from the start of a long session.
Proposed Solution
Replace heuristic dropping with a 'Compressive Memory' mechanism. Fine-tune the model to compress older context chunks into soft 'memory tokens' that are retained in the DSA attention cache. This allows the model to maintain semantic access to the entire history without the linear computational cost of full attention.
Why This Matters
Solving the 'lost-in-the-middle' problem for agents is crucial for autonomous systems that need to operate over days or weeks without forgetting user instructions.
How to Get Started
How to test it: We hypothesize that compressive memory fine-tuning retains long-term session context better than heuristic truncation. Baseline is Keep-recent-k/Discard-all; metric is accuracy on BrowseComp and long-context retrieval tasks. We stop if performance falls below the heuristic baseline.
- Reproduce the context degradation phenomenon using a long-horizon agentic task benchmark like AgentBench.
- Implement a memory compression module that aggregates older KV cache states into fixed-size soft memory tokens.
- Fine-tune an open-source language model (such as Llama-3) on long-context QA datasets using the compressive memory loss.
- Benchmark the compressed memory approach against the standard 'Keep-recent-k' strategy in terms of accuracy and latency.
Code linked to this paper on Hugging Face: github.com/zai-org/GLM-5
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Several listed works explore memory compression and context management for agents and language models, including What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents and A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts. However, none specifically propose fine-tuning a model to compress older context chunks into soft memory tokens retained in a DSA attention cache as detailed here.
- What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents · 2026 · 7 citations
- A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts · 2024 · 143 citations
We searched OpenAlex for “Replace heuristic history dropping with a compressive memory mechanism that encodes older context chunks into soft tokens retained in the attention cache.” and “Compressive Memory for Agentic Contexts” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
GLM-5: from Vibe Coding to Agentic EngineeringGLM-5 Team, Aohan Zeng, Xin Lv +183 more · 2026
This paper introduces GLM-5, a 744B parameter Mixture-of-Experts foundation model optimized for agentic engineering tasks. It employs DeepSeek Sparse Attention (DSA) for efficient long-context processing and utilizes a novel asynchronous reinforcement learning pipeline that decouples generation from training to achieve state-of-the-art performance in reasoning and coding benchmarks.
Related Ideas
- Neural Explicit Memory for Consistent Multi-Agent World SimulationSolaris: Building a Multiplayer Video World Model in Minecraft🟠Medium
- Semantic Compressive Memory for Long-Horizon ContextMiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling🟠Medium
- Recurrent Visual Memory for Infinite-Context TrajectoriesMobile-Agent-v3: Foundamental Agents for GUI Automation🔴Hard
- Hierarchical Retrieval-Augmented Self-Improvement Memory for Extended Optimization HorizonsSelf-Improvement Memory Enables Continual Recursive Self-Improvement🟡Fairly easy
- Dynamic Working-Memory Pruning for Search Agents to Overcome Reinforcement Learning SaturationK2-Horizon: A Frontier Fleet of Open Source Language Models🟡Fairly easy