Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationRelated work found

Dynamic Working-Memory Pruning for Search Agents to Overcome Reinforcement Learning Saturation

Prune redundant retrieved text dynamically during search agent rollouts to prevent context saturation and reward plateaus during long-horizon reinforcement learning.

Time

several weeks

Compute level

Multiple GPUs

Resources

Multi-GPU node with 4-8 GPUs, local Wikipedia RAG server (such as the 26M passage FAISS index used in the paper), search RL environments.

What the paper used

8-GPU nodes of NVIDIA H200s, running RL360 with Miles rollout manager, SGLang inference engines, and Megatron-LM trainer.

Prerequisites

Reinforcement learning (GRPO/PPO), dense retrieval, context window management, transformer KV caching.

Scope

modify existing code

The Gap

In Section 6.3.1 and Figure 41, the authors analyze the training trajectory of the 375B-A23B search expert under GRPO. The search expert improves in base reward initially (55.4% to 61.8%) but flattens completely after rollout step 20, with trajectory length reaching 62.0k tokens.

“This expert is in fact context-bound: its longest trajectory sits at 97.6k tokens at every rollout step (97,608–97,712 across all 40) against a 96k window and truncation rises 14.3% → 22.4%, so the exported checkpoint sits on a plateau rather than on a rise.”
From the paper, Section 6.3.1, page 97. Matched against the PDF text.

Limitation

Search agent trajectories become context-bound because approximately 84% of the trajectory consists of raw retrieved context rather than reasoning, causing truncation to rise from 14.3% to 22.4% and plateauing performance gains.

Proposed Solution

Replace static raw document ingestion with an integrated dynamic context pruner and memory compression step. During rollout execution, after each search or fetch tool call, apply an attention-saliency or extractive summarization step that retains only query-relevant facts and extracts before appending to the running context. Incorporate an explicit context frugality penalty or efficiency reward into the GRPO objective to train the agent to discard irrelevant retrieved passages prior to next-step planning.

Why This Matters

Preventing context saturation would allow search agents to continue exploring complex, multi-hop queries over dozens of turns without hitting hard context limits or suffering from off-policy truncation bias.

How to Get Started

How to test it: Train the K2-Horizon-7B search expert using the proposed dynamic pruning against the baseline RL recipe from Table 29; the method succeeds if the average truncation rate stays below 5% and the final test accuracy on BrowseComp or complex multi-hop search tasks improves by at least 3 percentage points.

  1. Clone the public RL360 repository and configure the search expert environment with the local Wikipedia FAISS index.
  2. Instrument the search harness to log per-turn context accumulation and replicate the truncation plateau on a 7B model.
  3. Implement a post-retrieval extractive pruning tool that compresses fetched documents to under 1k tokens based on cross-attention scores.
  4. Run comparative GRPO training runs with and without dynamic pruning across 50 optimization steps.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers such as Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learning and SWE-Pruner explore context pruning and reinforcement learning for agents, but they do not specifically integrate attention-saliency working-memory pruning with efficiency rewards under GRPO for search agents.

We searched OpenAlex and arXiv for “reinforcement learning search agent context pruning truncation” and “Dynamic Working-Memory Pruning for Search Agents to Overcome Reinforcement Learning Saturation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

K2-Horizon: A Frontier Fleet of Open Source Language Models

K2 Team, Zhengzhong Liu, Haonan Li +92 more · 2026

This technical report presents K2-Horizon, a family of six fully open language models ranging from 0.9B dense to a 375B mixture-of-experts (MoE) flagship activating 23B parameters, optimized for agentic and reasoning tasks. The authors open-source the complete development artifacts, including intermediate checkpoints, pre-training corpora with 10T synthetic tokens, the xLLM training framework, and the RL360 reinforcement learning stack. Key architectural innovations include Grouped RMSNorm, Mixture-of-Value Attention (MoVA), softplus attention gating, and Uno discrete diffusion adapters for accelerated decoding.

Related Ideas

Is this a good idea?

Share onXLinkedIn