Continuous Batching and Expert Co-scheduling for Multi-User SSD Streaming
Design an expert-aware continuous batching scheduler that groups requests predicting overlapping sets of experts within the same decode cycle.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 6, the authors state that EDGE0 serves one request at a time via FIFO serialization because batching fundamentally alters the expert working set in ways not supported by their per-request slot management and prerouting heads.
“Concurrency belongs to a serving layer, not the engine, and batching changes the expert working set in ways our per-request profiles do not model.”
Limitation
The framework cannot support multi-tenant or concurrent serving; concurrently active sequences cause expert union explosion across layers, causing SSD bandwidth saturation and catastrophic cache thrashing.
Proposed Solution
Design an expert-aware continuous batching scheduler. Extend the prerouter heads to output batch-wide expert priority scores, and implement an expert affinity-based request dispatcher that co-schedules sequences predicting overlapping sets of experts within the same decode cycle, bounding the total active expert footprint to fixed buffer limits.
Why This Matters
Enables EDGE0 to scale from single-user desktop utilities to multi-user on-premise local servers, multiplying aggregate token serving throughput across concurrent users.
How to Get Started
How to test it: Hypothesis: An expert-affinity continuous batching scheduler driven by prerouter scores bounds the per-step union of active experts and increases aggregate decode throughput over FIFO serving within a 24 GB memory budget. Baseline: FIFO-serialized Edge0-35B and naive uncoordinated batching on Mac mini M4 Pro 24 GB; Metric: Aggregate tok/s across batch sizes 2-8, MiB read from SSD per step, and peak active memory; Stop if batching causes SSD thrashing or active memory growth that yields less than 1.2x FIFO aggregate throughput or causes OOM/swapping on 24 GB.
- Modify the prerouter heads to output batch-wide expert priority scores rather than single-request selections.
- Implement an expert affinity-based request dispatcher queue that groups sequences requiring shared expert blocks.
- Integrate the scheduler into the local SSD streaming evaluation framework to handle concurrent user requests.
- Benchmark throughput, expert cache hit rates, and latency on a workstation equipped with high-speed PCIe NVMe SSDs.
Code linked to this paper on Hugging Face: github.com/Edge0-AI/edge0
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as LayerScope, HarMoEny, and HCRMap discuss expert-aware scheduling, cross-layer scheduling, and expert residency mapping for Mixture-of-Experts inference. However, they do not specifically implement the proposed continuous batching scheduler with prerouter output scores for multi-user SSD streaming.
- LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers · 2026 · 12 citations
- HarMoEny: Efficient multi-GPU inference of Mixture of Experts models by scheduling experts across accelerators · 2025 · 0 citations
- HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference · 2026 · 0 citations
We searched OpenAlex and arXiv for “Design an expert-aware continuous batching scheduler that groups requests predicting overlapping sets of experts within the same decode cycle.” and “Continuous Batching and Expert Co-scheduling for Multi-User SSD Streaming” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing PredictionYu Lin, Yiming Wang, Runyuan Cai +2 more · 2026
The paper introduces EDGE0, a streaming inference framework designed to serve large mixture-of-experts (MoE) models (such as 35B and 8B variants) on consumer devices constrained by limited RAM. It overcomes SSD loading latency by using a cross-token prerouter that predicts routing one token ahead, executes the prediction directly as the actual routing to completely overlap SSD expert streaming with compute, and restores accuracy loss from int4 quantization and routing replacement via an unmerged distillation LoRA.
Related Ideas
- Hybrid Kernel Scheduling for Prefill OptimizationBitnet.cpp: Efficient Edge Inference for Ternary LLMs🟡Fairly easy
- Empirical Production Serving Benchmarking of MLA-DSA Attention with IndexShareTeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and Frameworks🔴Hard
- Universal Asynchronous Optimization for Server-Side AgentsMetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wild🔴Hard
- Dynamic Block Sizing for Multi-Architecture ServingEfficient Memory Management for Large Language Model Serving with PagedAttention🔴Hard
- Dynamic Bucket Resizing via Control TheoryPyTorch Distributed: Experiences on Accelerating Data Parallel Training🟡Fairly easy