Dynamic Multi-Step Retrieval for Long-Horizon Tasks
Empower long-horizon agents to query memory autonomously during task execution by training them to emit an active retrieval token when confidence drops.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Currently, ReMe employs a fixed retrieval strategy where experiences are fetched only once at the beginning of a task (Section 3.3). However, benchmarks like AppWorld involve long execution horizons where the task context and sub-goals shift significantly after initial tool interactions.
“First, ReMe currently employs a fixed retrieval strategy, where experiences are retrieved once at the beginning of each task.”
Limitation
The single-turn retrieval mechanism fails to provide relevant guidance for intermediate steps in multi-turn tasks. If an agent encounters a specific error or sub-problem halfway through a trajectory, the experiences retrieved based on the initial query are likely irrelevant, leading to 'blind' trial-and-error in later stages.
Proposed Solution
Implement an 'Active Retrieval' mechanism where the agent can autonomously trigger memory lookups during execution. This involves fine-tuning the agent to emit a special token (e.g., <RETRIEVE>) when confidence is low or an error occurs, using the current conversation history and tool output as the query to fetch context-relevant experiences dynamically.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
This would significantly improve success rates on complex, multi-step benchmarks like AppWorld by ensuring the agent has access to the right procedural knowledge exactly when it is needed, rather than relying solely on initial context.
How to Get Started
How to test it: We hypothesize that active retrieval improves intermediate step success over static initial retrieval. Baseline: ReMe with fixed single-turn retrieval. Metric: task success rate on multi-turn subsets. Stopping condition: failure to exceed baseline task success by at least 2% after active intervention fine-tuning.
- Define a special retrieval token and construct fine-tuning data with simulated low-confidence triggers in AppWorld or BFCL.
- Modify the agent decoding loop to intercept the retrieval token and execute dynamic memory queries using the conversation history.
- Fine-tune the agent policy on mixed standard and retrieval-augmented execution traces.
- Evaluate the active retrieval agent on multi-step benchmarks to measure error recovery and task success rates.
Code linked to this paper on Hugging Face: github.com/agentscope-ai/ReMe
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like 'ActiveMem: Dynamic Latent Memory Trees for Long-Horizon Agents' and 'STaR: Scalable Task-Conditioned Retrieval for Long-Horizon Multi-Modal Robot Memory' touch upon dynamic retrieval mechanisms and active memory management for agents. However, the specific token-triggered active retrieval mechanism using conversation history and tool outputs leaves room for unique implementation details not fully resolved by the listed papers.
- STaR: Scalable Task-Conditioned Retrieval for Long-Horizon Multi-Modal Robot Memory · 2026 · 10 citations
- ActiveMem: Dynamic Latent Memory Trees for Long-Horizon Agents · 2026 · 2 citations
We searched OpenAlex for “Empower long-horizon agents to query memory autonomously during task execution by training them to emit an active retrieval token when confidence drops.” and “Dynamic Multi-Step Retrieval for Long-Horizon Tasks” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent EvolutionZouying Cao, Jiaji Deng, Li Yu +4 more · 2025
This paper introduces ReMe (Remember Me, Refine Me), a framework for autonomous agent evolution that replaces static memory accumulation with a dynamic lifecycle of experience extraction, reuse, and refinement. By distilling keypoint-level insights from trajectories and employing utility-based pruning, ReMe enables smaller models to outperform larger baselines on complex reasoning tasks without parameter updates.
Related Ideas
- Importance-Weighted Semantic Memory Compaction for Ultra-Long Horizon TasksGamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent🟡Fairly easy
- Hierarchical Retrieval-Augmented Self-Improvement Memory for Extended Optimization HorizonsSelf-Improvement Memory Enables Continual Recursive Self-Improvement🟡Fairly easy
- Dynamic Working-Memory Pruning for Search Agents to Overcome Reinforcement Learning SaturationK2-Horizon: A Frontier Fleet of Open Source Language Models🟡Fairly easy
- Dynamic Step-Wise Skill RetrievalSkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning🟠Medium
- Dynamic Context Summarization for Long-Running TasksAutoDev: Automated AI-Driven Development🟡Fairly easy