Skip to content

Source paper

The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

Yu Lin, Yiming Wang, Runyuan Cai and 2 more

Yu Lin, Yiming Wang, Runyuan Cai, Hanze Liu, Xiaodong Zeng

arXiv 2609.18063 · 202633 upvotes on Hugging Face

Research ideas from this paper (4)

Generated by AI. The paper’s authors did not write these.

Closed-Loop Autoregressive Training to Mitigate Prerouter Feature Drift

Implement a closed-loop training curriculum inspired by DAgger to replace ground-truth execution one-hots with model predictions during prerouter distillation.

The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
🟡Fairly easy💡Moderate InnovationMultiple GPUsNo close match found#Training#Efficiency

Direct GPU Graph Recording and Persistent Memory Pointers to Eliminate Decode Dispatch Overhead

Adopt static computation graph capture and persistent memory pointer swapping to eliminate CPU-side computational graph construction overhead during decoding.

The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
🟠Medium💡Moderate InnovationSingle GPU or lessNo close match found#Efficiency#Training

Process-Supervised Reasoning-Targeted Distillation for Routing-Replaced MoEs

Develop a targeted distillation pipeline combining complex reasoning datasets with step-level process reward models or Monte Carlo tree search rollouts.

The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
🟠Medium🔬Significant ResearchMultiple GPUsRelated work found#Reasoning#Training

Continuous Batching and Expert Co-scheduling for Multi-User SSD Streaming

Design an expert-aware continuous batching scheduler that groups requests predicting overlapping sets of experts within the same decode cycle.

The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
🔴Hard🔬Significant ResearchSingle GPU or lessRelated work found#Efficiency#Retrieval & memory

AI summary

The paper introduces EDGE0, a streaming inference framework designed to serve large mixture-of-experts (MoE) models (such as 35B and 8B variants) on consumer devices constrained by limited RAM. It overcomes SSD loading latency by using a cross-token prerouter that predicts routing one token ahead, executes the prediction directly as the actual routing to completely overlap SSD expert streaming with compute, and restores accuracy loss from int4 quantization and routing replacement via an unmerged distillation LoRA.

How it works (AI summary)

EDGE0 stores int4-quantized MoE expert parameters on NVMe SSDs and memory-maps them into a fixed-slot double-buffered streaming pool. To hide disk latency without pipeline bubbles, per-layer MLP heads predict layer N+1's expert routing one token in advance from layer N's hidden representations; this predicted set is adopted directly as the routed set ('prediction-as-routing'). To recover output quality without re-quantizing deltas into 4-bit weights, a parallel unmerged recovery LoRA is trained over the student routing path via multi-phase supervised fine-tuning and on-policy reverse-KL distillation.