Mixture-of-Experts (MoE) DiT for Latency-Constrained Scaling
A system that replaces dense feed-forward networks in diffusion backbones with Mixture-of-Experts layers to scale parameters while maintaining real-time latency.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section 7 explicitly identifies the 5B parameter size as a constraint limiting generation quality, yet notes that simply scaling up the dense model would violate the real-time latency requirement (currently 12 FPS on A100).
Limitation
The current dense Transformer architecture forces a linear trade-off between model capacity (knowledge/quality) and inference speed, preventing the use of larger models needed to resolve complex artifacts.
Proposed Solution
Replace the dense feed-forward networks in the diffusion backbone with Mixture-of-Experts (MoE) layers. Implement a router mechanism to activate only a sparse subset of parameters (e.g., top-2 experts) per token during inference. This would allow scaling the total parameter count to 20B+ to improve world knowledge and physics adherence while keeping the active parameter count (and thus FLOPs) within the 5B budget required for real-time performance.
Why This Matters
Enables the generation of significantly higher fidelity and more physically accurate worlds without sacrificing the real-time interactivity that is central to the paper's contribution.
How to Get Started
- Modify the DiT backbone architecture to include sparse Mixture-of-Experts layers.
- Implement a top-2 token routing mechanism using standard deep learning frameworks.
- Benchmark inference throughput and active parameter counts against the dense 5B baseline on A100 GPUs.
Code linked to this paper on Hugging Face: github.com/stdstu12/YUME
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing and Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts directly implement mixture-of-experts layers within diffusion transformers to scale parameters while controlling active compute budgets. They leave open specific configurations for real-time video generation constraints.
- EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing · 2024 · 24 citations
- Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts · 2025 · 10 citations
We searched OpenAlex for “A system that replaces dense feed-forward networks in diffusion backbones with Mixture-of-Experts layers to scale parameters while maintaining real-time latency.” and “Mixture-of-Experts (MoE) DiT for Latency-Constrained Scaling” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Yume-1.5: A Text-Controlled Interactive World Generation ModelXiaofeng Mao, Zhen Li, Chuanhao Li +6 more · 2025
This paper presents Yume1.5, a multimodal framework designed to generate infinite, interactive, and photorealistic video worlds from a single image or text prompt. The authors address limitations in latency and context length by introducing a Joint Temporal-Spatial-Channel Modeling (TSCM) strategy for efficient history compression and a Self-Forcing training paradigm to mitigate autoregressive error accumulation.
Related Ideas
- Sparse Scaling via Multimodal Mixture-of-ExpertsNextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation🟠Medium
- Synthetic Trajectory Distillation for Specialized Open-Weight ModelsDeepCode: Open Agentic Coding🟠Medium
- Causal Diffusion Decoding for 25Hz Semantic TokensQwen3-TTS Technical Report🔴Hard
- Real-Time Interactive World Generation via Diffusion DistillationWorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation🟠Medium
- Native Integration of Adaptive Gradient CompressionPyTorch Distributed: Experiences on Accelerating Data Parallel Training🟠Medium