Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Dense Process Reward Optimization for Long-Horizon GUI Agents

Provide dense, step-by-step feedback to long-horizon GUI agents by training a dedicated process reward model on step-level critic data.

Time

3-5 days

Compute level

Large cluster

Resources

A training cluster with multiple GPUs for asynchronous environment rollouts and policy updates.

What the paper used

Qwen2.5-VL-7B and Qwen2.5-VL-32B initialized models trained on OSWorld-Verified and AndroidWorld benchmarks using a scalable RL infrastructure.

Prerequisites

reinforcement learning

Scope

modify existing code

The Gap

In Section 3.1.2, the authors describe using Trajectory-aware Relative Policy Optimization (TRPO) where the advantage estimate from a single terminal reward R(tau) is 'uniformly distributed to every action taken within that trajectory'.

“This approach circumvents the challenge of assigning per-step rewards, a task that is nearly impossible to perform accurately in complex GUI interactions.”
From the paper, Page 9, Section 3.1.2. Matched against the PDF text.

Limitation

Uniform credit assignment is inefficient for long-horizon tasks (e.g., 20+ steps). If an agent performs 19 correct steps and fails on the 20th, TRPO penalizes all previous correct actions equally, slowing down convergence and failing to reinforce partial successes.

Proposed Solution

Train a dedicated Process Reward Model (PRM) using the data already generated by the 'Step-Level Critic' (described in Section 6.3). Integrate this PRM into the RL loop to provide dense, step-wise reward signals combined with the sparse terminal reward. This allows the policy to distinguish between a 'bad plan' and a 'good plan with one bad execution step'.

Why This Matters

Improving credit assignment is critical for scaling GUI agents to complex, real-world workflows where tasks may span dozens of screens and actions.

How to Get Started

How to test it: Hypothesis: Integrating a PRM will improve long-horizon task completion over sparse rewards. Baseline: The current sparse-reward TRPO policy model. Metric: Success rate on OSWorld-Verified. Stop if success rate improvement is less than 1.5% absolute.

  1. Extract step-level critic trajectories and intermediate annotations from existing training logs.
  2. Construct a dataset of positive and negative action prefixes to train a binary classification process reward model.
  3. Integrate the trained process reward model into the policy optimization loop to combine step-wise and terminal rewards.
  4. Benchmark the updated agent on complex GUI tasks against a baseline using only terminal rewards.

Code linked to this paper on Hugging Face: github.com/X-PLUG/MobileAgent

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Works like 'GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks' and 'PRO-CUA: Process-Reward Optimization for Computer Use Agents' explore process reward modeling and optimization for GUI tasks. However, these specific references do not detail training a dedicated PRM using data generated by a specific 'Step-Level Critic' configuration as proposed. The listed works provide context on step-level filtering and verification but leave the exact architectural integration open.

We searched OpenAlex for “Provide dense, step-by-step feedback to long-horizon GUI agents by training a dedicated process reward model on step-level critic data.” and “Dense Process Reward Optimization for Long-Horizon GUI Agents” and an AI compared the idea with the top 9 results. A small search can miss relevant work.

Source Paper

Mobile-Agent-v3: Foundamental Agents for GUI Automation

Jiabo Ye, Xi Zhang, Haiyang Xu +12 more · 2025

This paper presents GUI-Owl and Mobile-Agent-v3, a foundational end-to-end multimodal model and multi-agent framework designed for cross-platform GUI automation (Mobile, PC, Web). The authors introduce a self-evolving data production pipeline that synthesizes large-scale training data through model rollouts, critic-based filtering, and VLM-generated guidance, followed by a scalable reinforcement learning framework using Trajectory-aware Relative Policy Optimization (TRPO) to handle long-horizon tasks.

Related Ideas

Is this a good idea?

Share onXLinkedIn