Centralized Training with Decentralized Execution (CTDE) for Multi-Agent Systems
Incorporate a centralized critic using Multi-Agent PPO to stabilize training across independent language model agents operating in a shared environment.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 3.2.2 suggests extending the framework to multi-LLM settings by treating each LLM as an 'independent MDP' and optimizing them separately. This ignores the non-stationarity introduced when multiple agents (e.g., a generator and a reviewer) update their policies simultaneously, which often leads to unstable convergence or cyclic behaviors.
“When multiple distinct LLMs with their own parameters must be jointly optimized, a straightforward approach treats each LLM as an independent MDP and optimizes them separately.”
Limitation
Treating interacting agents as independent learners fails to capture the joint dynamics required for effective multi-agent coordination.
Proposed Solution
Integrate Multi-Agent Reinforcement Learning (MARL) algorithms such as MAPPO (Multi-Agent PPO). Modify the Lightning Server to support a centralized Critic that observes the global state (inputs/outputs of all agents in the workflow) during training to stabilize the updates, while keeping execution decentralized on the Client side.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Essential for robustly training multi-agent systems (e.g., Red Teaming, Debate protocols) where agents must learn complementary strategies.
How to Get Started
How to test it: We hypothesize that using MAPPO with a centralized critic for multi-agent workflows will stabilize updates and yield higher trajectory rewards than independent optimization. We will measure the success rate on multi-turn SQL queries and stop if validation reward improvement plateaus for 3 consecutive epochs.
- Modify the Lightning Server architecture to support multi-agent rollouts with shared global state observations.
- Implement the Multi-Agent PPO (MAPPO) algorithm with a centralized Critic network.
- Set up a multi-LLM benchmark environment featuring generator-reviewer workflows, such as ChatDev or MetaGPT tasks.
- Compare the training stability and convergence rate of MAPPO against independent MDP optimization baselines.
Code linked to this paper on Hugging Face: github.com/microsoft/agent-lightning
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The concept of Centralized Training with Decentralized Execution (CTDE) and variants like MAPPO are extensively studied in multi-agent reinforcement learning, as overviewed in 'An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning' and 'Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?'.
- An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning · 2024 · 201 citations
- Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL? · 2023 · 63 citations
We searched OpenAlex for “Incorporate a centralized critic using Multi-Agent PPO to stabilize training across independent language model agents operating in a shared environment.” and “Centralized Training with Decentralized Execution (CTDE) for Multi-Agent Systems” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Agent Lightning: Train ANY AI Agents with Reinforcement LearningXufang Luo, Yuge Zhang, Zhiyuan He +5 more · 2025
The paper introduces "Agent Lightning," a framework designed to decouple Reinforcement Learning (RL) training from agent execution, enabling the optimization of LLMs within diverse agent frameworks (e.g., LangChain, AutoGen) with minimal code changes. By formulating agent execution as a Markov Decision Process (MDP) and utilizing a hierarchical RL algorithm called "LightningRL," the system decomposes complex multi-turn trajectories into individual transitions, allowing for efficient, scalable policy updates using standard single-turn RL methods.
Related Ideas
- Optimizing High-Concurrency Database and Inference Pipelines for Million-Agent SimulationsOASIS: Open Agent Social Interaction Simulations with One Million Agentsđźź Medium
- Self-Correction via Test-Driven Reinforcement LearningOpenDevin: An Open Platform for AI Software Developers as Generalist Agentsâ›”Very hard
- Off-Policy Correction for Asynchronous Agent RLGLM-5: from Vibe Coding to Agentic Engineeringđźź Medium
- Reinforcement Learning for Long-Horizon GUI Agent NavigationPenguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encodersâ›”Very hard
- Sample-Efficient Online Policy Optimization for Rapid Agent AdaptationMetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wildđźź Medium