Granular Credit Assignment via Learned Value Functions
Implement a learned Value Model using Generalized Advantage Estimation over trajectory dependency graphs to provide granular credit assignment for long-horizon agents.
Source paper
Xufang Luo, Yuge Zhang, Zhiyuan He, Zilong Wang, Siyun Zhao, Dongsheng Li, Luna K. Qiu, Yuqing Yang
Generated by AI. The paper’s authors did not write these.
Implement a learned Value Model using Generalized Advantage Estimation over trajectory dependency graphs to provide granular credit assignment for long-horizon agents.
Incorporate a centralized critic using Multi-Agent PPO to stabilize training across independent language model agents operating in a shared environment.
Augment the reward function with intrinsic motivation signals like episodic curiosity to prevent agents from exploiting sparse reward tools.
The paper introduces "Agent Lightning," a framework designed to decouple Reinforcement Learning (RL) training from agent execution, enabling the optimization of LLMs within diverse agent frameworks (e.g., LangChain, AutoGen) with minimal code changes. By formulating agent execution as a Markov Decision Process (MDP) and utilizing a hierarchical RL algorithm called "LightningRL," the system decomposes complex multi-turn trajectories into individual transitions, allowing for efficient, scalable policy updates using standard single-turn RL methods.
The framework utilizes a "Training-Agent Disaggregation" architecture where a centralized Server manages model updates and a distributed Client executes agent logic to collect data. The core algorithm, LightningRL, treats agent execution as a set of isolated transitions (input, output, reward); it employs a credit assignment module that broadcasts episode-level rewards to all steps in a trajectory, enabling the use of value-free RL methods like GRPO on complex, multi-step agent workflows.