Sample-Efficient Online Policy Optimization for Rapid Agent Adaptation
Replace standard reinforcement learning policy loops with offline-to-online algorithms like Implicit Q-Learning and synthetic data augmentation using hindsight experience replay.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The Opportunistic policy optimization loop requires accumulating a sufficiently large query buffer of post-adaptation trajectories to ensure stable GRPO-based RL updates, as launching RL with too few samples causes high-variance gradients.
“This means policy optimization naturally lags behind skill-driven adaptation by days or longer, reinforcing the asymmetry between the two timescales: skills evolve continuously, while the policy improves in discrete, data-gated steps.”
Limitation
Because of the sample efficiency bottleneck inherent to standard RL, weight-level policy optimization lags behind skill-driven adaptation by days or weeks, delaying the internalization of complex execution behaviors that prompt injection alone cannot solve.
Proposed Solution
Replace standard GRPO with a sample-efficient offline-to-online RL algorithm, such as Implicit Q-Learning (IQL), combined with synthetic data augmentation. The skill evolver can be repurposed to aggressively generate synthetic successful rollouts from failed ones using hindsight experience replay, artificially inflating the training buffer to trigger reliable LoRA updates much sooner.
Why This Matters
Accelerating the RL training cycle would bridge the temporal gap between prompt-based skill acquisition and weight-level execution reliability, allowing agents to internalize complex tasks in hours rather than days.
How to Get Started
How to test it: Hypothesis: replacing GRPO with IQL and skill-evolver hindsight synthetic rollouts reduces the trajectory threshold for stable LoRA updates and accelerates rule internalization on Kimi-K2.5. Compared against the MetaClaw (Full) GRPO baseline on MetaClaw-Bench Part II (days to reach >50% file-check completion and final accuracy), stop if synthetic rollouts fail to beat GRPO before Day 8 or final completion falls below 45%.
- Construct an offline dataset baseline using historical agent failure trajectories from public software engineering benchmarks like SWE-Bench.
- Implement Implicit Q-Learning (IQL) using standard codebase libraries such as Stable-Baselines3 or CleanRL.
- Write a synthetic data generator script leveraging a base LLM to transform failed execution rollouts into successful demonstrations via hindsight relabeling.
- Evaluate sample efficiency and LoRA update stability on a standard single GPU against standard GRPO baselines.
Code linked to this paper on Hugging Face: github.com/aiming-lab/MetaClaw
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like 'Group-in-Group Policy Optimization for LLM Agent Training' and 'Q-PrOP: Sample-efficient policy gradient with an off-policy critic' explore policy optimization and sample efficiency enhancements for agents. However, none of the listed papers combine implicit Q-learning with hindsight experience replay synthetic rollouts specifically for rapid LoRA updates as proposed.
- Group-in-Group Policy Optimization for LLM Agent Training · 2025 · 623 citations
- Q-PrOP: Sample-efficient policy gradient with an off-policy critic · 2016 · 339 citations
We searched OpenAlex and arXiv for “Replace standard reinforcement learning policy loops with offline-to-online algorithms like Implicit Q-Learning and synthetic data augmentation using hindsight experience replay.” and “Sample-Efficient Online Policy Optimization for Rapid Agent Adaptation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the WildPeng Xia, Jianwen Chen, Xinyu Yang +10 more · 2026
The paper introduces MetaClaw, a continual meta-learning framework designed to enable deployed Large Language Model (LLM) agents to autonomously adapt to shifting task distributions in the wild. By bridging fast skill distillation and slow weight optimization, it allows agents to continuously improve from failure trajectories without requiring service downtime.
Related Ideas
- Integration of Online Reinforcement Learning (PPO/GRPO)BitNet b1.58 2B4T Technical Report🟠Medium
- Off-Policy Correction for Asynchronous Agent RLGLM-5: from Vibe Coding to Agentic Engineering🟠Medium
- Interactive Policy Refinement via World Model RLGigaBrain-0: A World Model-Powered Vision-Language-Action Model⛔Very hard
- Token-Level Importance-Weighted Off-Policy Correction for Asynchronous Agent RLTeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and Frameworks🟠Medium
- Efficiency-Weighted Reward Modeling for GRPOMiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling🟠Medium