Reinforcement Learning for Long-Horizon Reasoning
Apply reinforcement learning via GRPO on top of supervised fine-tuned vision-language models using multi-modal verifier rewards to optimize long-horizon web navigation reasoning.
Source paper
Ahmed Awadallah, Yash Lara, Raghav Magazine, Hussein Mozannar, Akshay Nambi, Yash Pandya, Aravind Rajeswaran, Corby Rosset, Alexey Taymanov, Vibhav Vineet, Spencer Whitehead, Andrew Zhao
Generated by AI. The paper’s authors did not write these.
Apply reinforcement learning via GRPO on top of supervised fine-tuned vision-language models using multi-modal verifier rewards to optimize long-horizon web navigation reasoning.
Construct a sandbox simulator with mocked payment and login interfaces to enable automated data generation and agent fine-tuning past critical interaction points.
Implement visual token merging to compress historical screenshots into sparse feature vectors, maintaining long-term visual memory without exceeding context window limits.
This paper presents Fara-7B, a 7-billion parameter vision-language model specialized for computer use agents (CUAs), trained on a novel synthetic dataset generated by the FaraGen engine. FaraGen overcomes data scarcity by using a multi-agent system to propose, solve, and verify 145k web interaction trajectories, enabling Fara-7B to achieve state-of-the-art performance among 7B models on benchmarks like WebVoyager and the newly introduced WebTailBench.
The authors utilize a "pixel-in, action-out" architecture based on Qwen2.5-VL-7B, fine-tuned via Supervised Fine-Tuning (SFT) on synthetic trajectories comprising screenshots, Chain-of-Thought reasoning, and coordinate-based actions. The data generation pipeline employs an Orchestrator (planner), WebSurfer (actor), and multiple Verifiers (LLM-based judges) to create high-quality, verified training examples from real-world websites without human annotation.