Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Proactive User-in-the-Loop Clarification for High-Ambiguity Daily Tasks

Add a clarification action to the model's policy and train it via reinforcement learning to trigger dialogue when encountering high-ambiguity daily GUI tasks.

Time

a couple of months

Compute level

Multiple GPUs

Resources

High-end GPUs (e.g., A100s or H100s) for RL training, dataset containing ambiguous instructions and clarification dialogues (likely requiring synthetic generation)

Prerequisites

Reinforcement Learning (specifically RLVR/PPO), Transformer architectures, GUI agent protocols, Large Language Model fine-tuning

Scope

Significant code extension: modifying action space, designing custom reward functions, and integrating multi-turn dialogue capabilities

The Gap

The AndroidDaily benchmark results (Section 4, Table 6) demonstrate a significant performance drop for Step-GUI-8B when moving from low-ambiguity tasks to high-ambiguity tasks (success rates falling by over 12 points).

Limitation

The current action space (Section 2.2.1) is purely execution-oriented (Click, Swipe, Type) and lacks a mechanism for the agent to identify underspecified user intents and seek clarification.

Proposed Solution

Extend the model's action space to include a 'CLARIFY' action. Train the model using RLVR with a reward function that penalizes incorrect terminal actions in high-uncertainty states but provides a positive reward for triggering a clarification dialogue when the task instruction entropy exceeds a threshold. This requires integrating a multi-turn conversation head into the GUI-MCP protocol.

Why This Matters

Practical GUI agents cannot assume perfect user instructions; the ability to ask for missing parameters (e.g., 'Which email address should I send this to?') is critical for real-world reliability.

How to Get Started

  1. Extend the model's discrete action space to include a 'CLARIFY' token within the GUI-MCP protocol.
  2. Synthesize a dataset of high-ambiguity instructions paired with appropriate clarification dialogue trees.
  3. Implement a reinforcement learning reward function that penalizes incorrect terminal actions and rewards proper clarification triggering.
  4. Fine-tune Step-GUI-8B using the reward function and evaluate success rates on the AndroidDaily benchmark.

Code linked to this paper on Hugging Face: github.com/stepfun-ai/gelab-zero

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation and Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction explore proactive behaviors and interactive agent evaluations. However, they do not specifically integrate a reinforcement learning mechanism with explicit task instruction entropy thresholds and a 'CLARIFY' action head inside a GUI-MCP framework as proposed.

We searched OpenAlex for “Add a clarification action to the model's policy and train it via reinforcement learning to trigger dialogue when encountering high-ambiguity daily GUI tasks.” and “Proactive User-in-the-Loop Clarification for High-Ambiguity Daily Tasks” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Step-GUI Technical Report

Haolong Yan, Jia Wang, Xin Huang +94 more · 2025

The paper introduces Step-GUI, a family of multimodal models (4B/8B) specifically optimized for GUI automation through a self-evolving training pipeline. This pipeline uses a Calibrated Step Reward System (CSRS) to convert model-generated trajectories into high-quality training data at significantly lower costs while introducing GUI-MCP for secure, cross-platform deployment. Evaluation on the new AndroidDaily benchmark shows Step-GUI achieves state-of-the-art performance in real-world mobile and desktop scenarios.

Related Ideas

Is this a good idea?

Share onXLinkedIn