Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Trajectory-Consistency Preference Optimization (TCPO)

Align offline reinforcement learning with long-term task success by optimizing trajectories using direct preference optimization over complete sequences.

Time

2-3 weeks

Compute level

Large cluster

Resources

Multi-node GPU cluster for sequence-level preference alignment and running full-trajectory generations.

What the paper used

10 billion tokens across 30+ datasets with Offline-RL and GRPO online optimization

Prerequisites

Reinforcement Learning, DPO/PPO

Scope

Modify existing code

The Gap

Section 2.2.2 highlights a 'Discrepancy Between Step and Trace Success Rates' during Offline RL, where step-level success increases while trace-level success eventually declines because the reward system optimizes greedy actions.

“As shown in Figure 6, while step-level success rates increase steadily, trace-level success rates eventually peak and then decline. We attribute this behavior to an inherent limitation of Offline-RL: step-level rewards only optimize individual actions and fail to guide the successful composition of a full multi-step trace.”
From the paper, Section 2.2, page 8. Matched against the PDF text.

Limitation

The Offline RL objective optimizes individual steps in isolation using immediate rewards (Rformat, Raction), failing to model the long-horizon dependencies required for successful task completion before the expensive Online RL phase.

Proposed Solution

Implement a trajectory-level objective during the offline phase, such as Direct Preference Optimization (DPO) applied to full sequences. Construct preference pairs consisting of (successful complete trace, failed complete trace) and optimize the policy to assign higher likelihood to the coherent successful sequence rather than just maximizing immediate step rewards.

Why This Matters

Improving the offline phase's ability to model long-horizon tasks reduces the 'cold start' problem for Online RL, leading to faster convergence and less reliance on expensive online interactions.

How to Get Started

How to test it: Hypothesis: Trajectory-level Direct Preference Optimization (DPO) on offline traces will prevent the peak-and-decline phenomenon seen in step-level Offline-RL. Baseline: UI-Venus-1.5 after task-specific Offline-RL using step rewards. Metric: Peak and final trace success rate on AndroidWorld and WebVoyager evaluation sets. Stopping condition: If trajectory-level DPO fails to eliminate the performance drop or leads to unstable training loss.

  1. Parse existing training datasets to construct preference pairs consisting of successful versus failed complete action traces.
  2. Implement a sequence-level direct preference optimization objective to penalize greedy short-term actions.
  3. Fine-tune the offline reinforcement learning policy on the preference pairs and evaluate trace-level success rates.

Code linked to this paper on Hugging Face: github.com/inclusionAI/UI-Venus

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Direct Preference Optimization: Your Language Model is Secretly a Reward Model introduces the foundational DPO method, while Offline Reinforcement Learning for LLM Multi-step Reasoning touches upon multi-step sequence alignment. However, neither paper explicitly implements the proposed Trajectory-Consistency Preference Optimization using complete successful versus failed traces for offline trajectory-level preference tuning.

We searched OpenAlex for “Align offline reinforcement learning with long-term task success by optimizing trajectories using direct preference optimization over complete sequences.” and “Trajectory-Consistency Preference Optimization (TCPO)” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

UI-Venus-1.5 Technical Report

Veuns-Team, Changlong Gao, Zhangxuan Gu +24 more · 2026

This paper introduces UI-Venus-1.5, an end-to-end multimodal GUI agent designed to navigate mobile and web environments via a unified model. The system is trained through a three-stage pipeline: large-scale Mid-Training on 10B tokens of GUI data, task-specific Offline Reinforcement Learning (RL), and massive Online RL on a Device-as-a-Service infrastructure, culminating in a model merging strategy that unifies specialized capabilities.

Related Ideas

Is this a good idea?

Share onXLinkedIn