Stabilizing On-Policy Distillation via Critic Regularization
Stabilize on-policy distillation by applying critic regularization techniques like Spectral Normalization and target networks to prevent generator collapse.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section 3.2 describes a 'delicate interdependency' between the generator and critic in DMD, leading to a 'peak-then-degrade' pattern (Section 3.3). The authors currently mitigate this by using an 'aggressive learning rate schedule' to converge before the model collapses.
Limitation
The reliance on aggressive learning rates to 'race' against mode collapse is a heuristic workaround. It indicates that the underlying adversarial optimization game is unstable for complex multimodal distributions, limiting the model's potential performance ceiling.
Proposed Solution
Implement regularization techniques for the critic network, such as Spectral Normalization or Gradient Penalty, to constrain the Lipschitz constant. Additionally, investigate using a 'Target Network' (similar to Double Q-Learning) that updates slowly to provide stable scoring signals to the generator, preventing the feedback loop from spiraling into collapse.
Why This Matters
Stabilizing the training dynamic would remove the need for hyper-sensitive tuning of learning rate schedules and allow the model to train longer, potentially unlocking higher visual quality and better multimodal alignment.
How to Get Started
- Set up the baseline distribution matching distillation training pipeline on a standard GPU.
- Implement spectral normalization and gradient penalty methods within the critic network architecture.
- Introduce a slowly updating target network for scoring and measure stability improvements over training epochs.
Code linked to this paper on Hugging Face: github.com/GAIR-NLP/LiveTalk
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed papers specifically combine critic Lipschitz regularization via spectral normalization or gradient penalty with slowly updating target networks for stabilizing on-policy distillation. While works like Lipschitzness is all you need to tame off-policy generative adversarial imitation learning explore Lipschitz constraints in adversarial imitation learning, and HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation or KL for a KL: On-Policy Distillation with Control Variate Baseline study distillation, none of these specific results address the exact proposed combination.
- Lipschitzness is all you need to tame off-policy generative adversarial imitation learning · 2022 · 26 citations
- HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation · 2026 · 49 citations
- KL for a KL: On-Policy Distillation with Control Variate Baseline · 2026 · 32 citations
We searched OpenAlex and arXiv for “Stabilize on-policy distillation by applying critic regularization techniques like Spectral Normalization and target networks to prevent generator collapse.” and “Stabilizing On-Policy Distillation via Critic Regularization” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy DistillationEthan Chern, Zhulin Hu, Bohao Tang +4 more · 2025
The paper presents LiveTalk, a real-time system for interactive multimodal avatar video generation using a novel video diffusion distillation recipe. By refining multimodal data quality, ensuring full ODE convergence before distillation, and using an aggressive optimization schedule, the authors achieve 20x acceleration and sub-second latency while maintaining visual quality comparable to larger bidirectional models.
Related Ideas
- Dynamic Adaptive Renoising Schedules (DARS) for Multi-Step DistillationDecoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shield🟠Medium
- Adaptive Staleness Control via Online Meta-LearningAReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning🟡Fairly easy
- Curriculum-Based Full-Scale Reinforcement LearningNextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation🟠Medium
- Temporal-Consistent Reinforcement Learning for Video AutoregressionHelios: Real Real-Time Long Video Generation Model🔴Hard
- Evaluating Cautious Weight Decay in Deep Reinforcement Learning and Diffusion ModelsCautious Weight Decay🟠Medium