Efficiency-Aware Reward Shaping to Mitigate Redundant Tool Use
Shape agent rewards using dynamic penalties for redundant tool calls to prevent agents from gaming linear step bonuses without doing useful work.
Source paper
Shitian Zhao, Shaoheng Lin, Ming Li, Haoquan Zhang, Wenshuo Peng, Kaipeng Zhang, Chen Wei
Generated by AI. The paper’s authors did not write these.
Shape agent rewards using dynamic penalties for redundant tool calls to prevent agents from gaming linear step bonuses without doing useful work.
Incorporate execution failure trajectories into training with penalized rewards and self-correction loops to teach agents how to recover from runtime errors.
Expand the agent's Python execution environment with temporal-native computer vision primitives to process continuous motion dynamics directly.
This paper introduces PyVision-RL, a framework for training open-weight multimodal agents (based on Qwen2.5-VL) to effectively use Python tools for image and video understanding. To address 'interaction collapse'—where models stop using tools after RL fine-tuning—the authors propose an oversampling-filtering-ranking training strategy and an accumulative tool reward mechanism.
The model uses Group Relative Policy Optimization (GRPO) with Python as a primitive tool. Key techniques include 'On-Demand Context Construction' for videos (keeping video in runtime and sampling frames via code) and a modified reward function that incentivizes multi-turn interactions. Training data is selected via a curriculum based on reward variance.