Closed-Loop Autoregressive Training to Mitigate Prerouter Feature Drift
Implement a closed-loop training curriculum inspired by DAgger to replace ground-truth execution one-hots with model predictions during prerouter distillation.
Source paper
Yu Lin, Yiming Wang, Runyuan Cai, Hanze Liu, Xiaodong Zeng
Generated by AI. The paper’s authors did not write these.
Implement a closed-loop training curriculum inspired by DAgger to replace ground-truth execution one-hots with model predictions during prerouter distillation.
Adopt static computation graph capture and persistent memory pointer swapping to eliminate CPU-side computational graph construction overhead during decoding.
Develop a targeted distillation pipeline combining complex reasoning datasets with step-level process reward models or Monte Carlo tree search rollouts.
Design an expert-aware continuous batching scheduler that groups requests predicting overlapping sets of experts within the same decode cycle.
The paper introduces EDGE0, a streaming inference framework designed to serve large mixture-of-experts (MoE) models (such as 35B and 8B variants) on consumer devices constrained by limited RAM. It overcomes SSD loading latency by using a cross-token prerouter that predicts routing one token ahead, executes the prediction directly as the actual routing to completely overlap SSD expert streaming with compute, and restores accuracy loss from int4 quantization and routing replacement via an unmerged distillation LoRA.
EDGE0 stores int4-quantized MoE expert parameters on NVMe SSDs and memory-maps them into a fixed-slot double-buffered streaming pool. To hide disk latency without pipeline bubbles, per-layer MLP heads predict layer N+1's expert routing one token in advance from layer N's hidden representations; this predicted set is adopted directly as the routed set ('prediction-as-routing'). To recover output quality without re-quantizing deltas into 4-bit weights, a parallel unmerged recovery LoRA is trained over the student routing path via multi-phase supervised fine-tuning and on-policy reverse-KL distillation.