Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationClosely matching work found

Token-Level Importance-Weighted Off-Policy Correction for Asynchronous Agent RL

A method that applies token-level off-policy importance weighting to partial rollout trajectories to avoid discarding entire query groups during asynchronous multi-turn agent reinforcement learning.

Time

several weeks

Compute level

Multiple GPUs

Resources

Multi-GPU or multi-NPU server (e.g., 8x A100 or Ascend NPUs) with open-source agent environments and LLM RL infrastructure

What the paper used

Pretraining evaluated on a 1,024-NPU Ascend profiling cluster and an 8,192 Ascend 910C cluster for the 285B MoE model; inference evaluated on 32 Ascend 910C cards.

Prerequisites

reinforcement learning, off-policy policy gradients, LLM agent training

Scope

modify existing code

The Gap

Section 4.4 describes the asynchronous reinforcement learning workflow with partial rollout in TeleGym, where heterogeneous task durations cause agent trajectories to span multiple policy updates. To address policy lag, TeleGym employs a stale-data filtering heuristic that rejects an entire query group if any constituent trajectory is identified as old, in order to preserve the group membership required for group-relative reward normalization.

“Optional stale-data filtering uses observed policy versions to identify old trajectories and removes the entire query group when a constituent trajectory is rejected, preserving the membership used for group-relative reward processing. This is a trajectory-level heuristic rather than a strict token-wise age bound.”
From the paper, Section 4.4, page 26. Matched against the PDF text.

Limitation

The stale-data filtering heuristic operates at a coarse trajectory level by discarding entire query groups whenever a single trajectory becomes stale, causing substantial training sample waste and discarding useful intermediate tokens collected during multi-turn interactions.

Proposed Solution

Implement a token-level or segment-level off-policy correction mechanism, such as truncated importance sampling or V-trace adapted for group-relative policy optimization (GRPO). Instead of dropping entire query groups when policy version drift occurs, compute the importance sampling ratio between the current policy and the sampling-time policy log-probabilities recorded by the server for each generated token segment. Adjust the group-relative baseline and advantage weighting using importance-corrected sample weights, allowing valid prefixes and completed steps from lagging trajectories to contribute gradient signals without corrupting optimization.

Why This Matters

This approach is expected to significantly increase sample efficiency in agent RL pipelines, reducing wasted compute on discarded rollouts and allowing long-horizon multi-turn tasks to train stably even under frequent policy updates.

How to Get Started

How to test it: Evaluate on multi-turn code and reasoning benchmarks (such as LiveCodeBench and GSM8K) under high simulated policy lag; the hypothesis is that token-level importance weighting achieves equal or superior sample efficiency while utilizing at least 30% more generated rollout tokens than trajectory-level group dropping, stopping if importance weight variance destabilizes policy convergence.

  1. Set up an asynchronous RL pipeline using an open-source framework such as Slime or OpenRLHF configured with group-relative policy optimization.
  2. Implement artificial policy publication delays to reproduce TeleGym's trajectory-level group dropping behavior as an evaluation baseline.
  3. Add token-wise importance ratio calculation and clipping to the loss function using recorded behavior-policy log probabilities.
  4. Benchmark gradient variance, sample retention rate, and task performance across varied rollout latency distributions.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Closely matching work found

Papers such as 'COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning' and 'Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization' directly address token-level or coupled off-policy corrections for asynchronous LLM reinforcement learning. These works explore similar mechanisms to handle stale or lagging trajectories and logits during training.

We searched OpenAlex and arXiv for “asynchronous reinforcement learning agent off-policy importance weighting” and “Token-Level Importance-Weighted Off-Policy Correction for Asynchronous Agent RL” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

TeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and Frameworks

Xingchen AGI Lab, Baotian Hu, Bingkai Yang +136 more · 2026

This technical report presents Xing 4, a family of large language models scaling up to hundreds of billions of parameters trained natively on Ascend 910C NPUs using MindSpore HyperParallel. The report establishes design principles combining sparse attention, wider-and-shallower MoE topologies, and speculative decoding, supported by systems optimizations including hybrid sharded data parallelism, owner-based Muon execution, and CPU core isolation. It also introduces TeleGym, a unified execution and reinforcement learning platform for long-horizon agent and reasoning workloads.

Related Ideas

Is this a good idea?

Share onXLinkedIn