Task-Specific Reward Experts for MR-GRPO
Expand the reinforcement learning reward system into a mixture of specialized weak supervisor experts dynamically activated by prompt keywords.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The Reinforcement Learning stage uses MR-GRPO with a mixture of three broad rewards: VLM Preference (general quality), OCR (text accuracy), and CLIP (semantic alignment). The weights are adjusted based on prompt category (text-rendering vs. general).
Limitation
The current reward set is too coarse for fine-grained editing and spatial reasoning. A general VLM preference reward may overlook subtle spatial errors (e.g., 'cup left of spoon' vs 'spoon left of cup') or counting errors, as these global metrics often prioritize texture and style over precise structural logic.
Proposed Solution
Expand the reward model into a Mixture of Experts (MoE) system. Integrate specialized, lightweight 'weak supervisor' models as reward functions, such as an Object Detector (for counting/position), a Depth Estimator (for spatial relations), and a segmentation model. Dynamically activate these rewards based on keywords in the user prompt.
Why This Matters
This directly addresses the 'hallucination' problem in complex instruction following. By providing precise, disentangled feedback signals, the model can learn specific capabilities (like counting or relative positioning) much more effectively than via a noisy global reward.
How to Get Started
How to test it: Hypothesis: specialized reward experts improve fine-grained spatial and counting accuracy. Baseline: standard global VLM preference reward and OCR reward. Metric: UniREditBench and RISE spatial/logical scores. Stop if training becomes unstable or reward hacking occurs.
- Collect lightweight pretrained expert models including an object detector, a depth estimator, and a segmentation model.
- Design a keyword-matching router to assign weights to each expert reward based on the user prompt.
- Integrate the multi-expert reward framework into the existing MR-GRPO training pipeline.
- Benchmark the reward-optimized model against the baseline on spatial reasoning and counting tasks.
Code linked to this paper on Hugging Face: github.com/DeepGenTeam/DeepGen
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed literature contains works applying GRPO to normative reasoning, molecular optimization, and clinical reliability, along with general surveys on reward models. However, none of these papers cover expanding reward models into an MoE system with specialized perception supervisors activated via prompt keywords.
We searched OpenAlex for “Expand the reinforcement learning reward system into a mixture of specialized weak supervisor experts dynamically activated by prompt keywords.” and “Task-Specific Reward Experts for MR-GRPO” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
DeepGen 1.0: A Lightweight Unified Multimodal Model for Advancing Image Generation and EditingDianyi Wang, Ruihang Li, Feng Han +17 more · 2026
The paper introduces DeepGen 1.0, a lightweight (5B parameter) unified multimodal model capable of image generation, editing, and reasoning. By employing a novel Stacked Channel Bridging (SCB) architecture to fuse Vision-Language Model (VLM) features with a Diffusion Transformer (DiT) and utilizing a three-stage training pipeline culminating in Reinforcement Learning (MR-GRPO), the model achieves performance competitive with significantly larger closed-source and open-source models.
Related Ideas
- Attention-Guided Temporal Credit Assignment for Segment-Based Online Reinforcement LearningGamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent🟠Medium
- Curriculum-Based Full-Scale Reinforcement LearningNextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation🟠Medium
- Integration of Online Reinforcement Learning (PPO/GRPO)BitNet b1.58 2B4T Technical Report🟠Medium
- Granular Credit Assignment via Learned Value FunctionsAgent Lightning: Train ANY AI Agents with Reinforcement Learning🟠Medium
- Efficiency-Weighted Reward Modeling for GRPOMiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling🟠Medium