Trajectory-Consistency Preference Optimization (TCPO)
Align offline reinforcement learning with long-term task success by optimizing trajectories using direct preference optimization over complete sequences.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 2.2.2 highlights a 'Discrepancy Between Step and Trace Success Rates' during Offline RL, where step-level success increases while trace-level success eventually declines because the reward system optimizes greedy actions.
“As shown in Figure 6, while step-level success rates increase steadily, trace-level success rates eventually peak and then decline. We attribute this behavior to an inherent limitation of Offline-RL: step-level rewards only optimize individual actions and fail to guide the successful composition of a full multi-step trace.”
Limitation
The Offline RL objective optimizes individual steps in isolation using immediate rewards (Rformat, Raction), failing to model the long-horizon dependencies required for successful task completion before the expensive Online RL phase.
Proposed Solution
Implement a trajectory-level objective during the offline phase, such as Direct Preference Optimization (DPO) applied to full sequences. Construct preference pairs consisting of (successful complete trace, failed complete trace) and optimize the policy to assign higher likelihood to the coherent successful sequence rather than just maximizing immediate step rewards.
Why This Matters
Improving the offline phase's ability to model long-horizon tasks reduces the 'cold start' problem for Online RL, leading to faster convergence and less reliance on expensive online interactions.
How to Get Started
How to test it: Hypothesis: Trajectory-level Direct Preference Optimization (DPO) on offline traces will prevent the peak-and-decline phenomenon seen in step-level Offline-RL. Baseline: UI-Venus-1.5 after task-specific Offline-RL using step rewards. Metric: Peak and final trace success rate on AndroidWorld and WebVoyager evaluation sets. Stopping condition: If trajectory-level DPO fails to eliminate the performance drop or leads to unstable training loss.
- Parse existing training datasets to construct preference pairs consisting of successful versus failed complete action traces.
- Implement a sequence-level direct preference optimization objective to penalize greedy short-term actions.
- Fine-tune the offline reinforcement learning policy on the preference pairs and evaluate trace-level success rates.
Code linked to this paper on Hugging Face: github.com/inclusionAI/UI-Venus
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Direct Preference Optimization: Your Language Model is Secretly a Reward Model introduces the foundational DPO method, while Offline Reinforcement Learning for LLM Multi-step Reasoning touches upon multi-step sequence alignment. However, neither paper explicitly implements the proposed Trajectory-Consistency Preference Optimization using complete successful versus failed traces for offline trajectory-level preference tuning.
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model · 2023 · 10906 citations
- Offline Reinforcement Learning for LLM Multi-step Reasoning · 2025 · 79 citations
We searched OpenAlex for “Align offline reinforcement learning with long-term task success by optimizing trajectories using direct preference optimization over complete sequences.” and “Trajectory-Consistency Preference Optimization (TCPO)” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
UI-Venus-1.5 Technical ReportVeuns-Team, Changlong Gao, Zhangxuan Gu +24 more · 2026
This paper introduces UI-Venus-1.5, an end-to-end multimodal GUI agent designed to navigate mobile and web environments via a unified model. The system is trained through a three-stage pipeline: large-scale Mid-Training on 10B tokens of GUI data, task-specific Offline Reinforcement Learning (RL), and massive Online RL on a Device-as-a-Service infrastructure, culminating in a model merging strategy that unifies specialized capabilities.
Related Ideas
- Token-Level Importance-Weighted Off-Policy Correction for Asynchronous Agent RLTeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and Frameworks🟠Medium
- Sample-Efficient Online Policy Optimization for Rapid Agent AdaptationMetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wild🟠Medium
- Dense Process Reward Optimization for Long-Horizon GUI AgentsMobile-Agent-v3: Foundamental Agents for GUI Automation🟠Medium
- Integration of Online Reinforcement Learning (PPO/GRPO)BitNet b1.58 2B4T Technical Report🟠Medium
- Attention-Guided Temporal Credit Assignment for Segment-Based Online Reinforcement LearningGamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent🟠Medium