Skip to content
AI-generated research ideađźź Mediumđź’ˇModerate InnovationNo close match found

Direct GPU Graph Recording and Persistent Memory Pointers to Eliminate Decode Dispatch Overhead

Adopt static computation graph capture and persistent memory pointer swapping to eliminate CPU-side computational graph construction overhead during decoding.

Time

2-4 weeks to implement static graph capture and persistent pointer swapping within the backend facade, verify math contract parity, and run paired A/B throughput benchmarks.

Compute level

Single GPU or less

Resources

One Mac mini M4 Pro 24 GB (and optionally a 24 GB consumer CUDA GPU such as an RTX 4090 if validating CUDA Graphs via the backend facade) with the released 19.5 GB 35B int4 checkpoint and adapters.

What the paper used

Decode throughput is measured single-device on a Mac mini M4 Pro 24 GB (20.4 tok/s for 35B, 28.0 tok/s for 8B) and a MacBook M2 16 GB using MLX 0.30.4-0.30.6; training uses bf16 35B/8B models with ~2M and ~200k rows (training GPU hardware not stated).

Prerequisites

Low-level MLX/CUDA runtime internals, memory virtual address management, static graph execution.

Scope

Modify existing code in the execution engine backend.

The Gap

In Section 6 and Appendix B, the authors identify that decoding speed is CPU-bound rather than storage-bound, with 44 ms per step consumed exclusively by CPU-side computational graph construction across the 40 decoder layers.

“The streaming engine’s decode is CPU-side, not storage-side: 44 ms per step of graph building in the 40-layer forward is the floor, and no storage-side optimization moves it.”
From the paper, Section 6, page 11. Matched against the PDF text.

Limitation

Rebuilding and dispatching dynamic computational graphs at every single decode step imposes a hard latency floor of 44 ms that prevents the system from exceeding ~20 tok/s even when SSD prefetching is optimal.

Proposed Solution

Adopt static computation graph capture (such as CUDA Graphs or MLX-level static graph compilation with persistent slot references). Pre-record the entire forward pass graph once, keeping kernel launches and memory offsets invariant while updating only the underlying backing pointers of the sticky-slot expert memory buffers via pointer-swapping or zero-copy virtual address re-mapping.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Removing the 44 ms CPU graph construction tax per step could dramatically increase token throughput, potentially doubling generation speed on consumer Apple Silicon or PC hardware.

How to Get Started

How to test it: Hypothesis: Capturing a static computation graph with persistent sticky-slot pointer swapping eliminates the 44 ms per-step CPU graph-build floor and raises 35B decode speed above 25 tok/s. Baseline: Edge0-35B staged incr_stack on Mac mini M4 Pro 24 GB (20.4 tok/s); Metric: Median decode tok/s, per-step CPU dispatch latency, and element-wise relative L2 < 1% math contract parity; Stop if static graph capture breaks the <1% L2 parity contract or reduces total decode step latency by less than 15%.

  1. Profile the current decoding pipeline using PyTorch profiler or Apple Metal System Trace to isolate CPU-side graph construction bottlenecks.
  2. Implement CUDA Graphs capture or MLX static graph compilation for the model's forward pass loops.
  3. Develop a zero-copy virtual address re-mapping mechanism to update sticky-slot expert memory buffers without rebuilding the graph.
  4. Benchmark end-to-end token generation latency on consumer hardware or Apple Silicon M-series devices against the baseline FIFO implementation.

Code linked to this paper on Hugging Face: github.com/Edge0-AI/edge0

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

The retrieved papers cover graph processing accelerators, memory safety, and GPU task abstractions, but none of them address static computation graph capture or pointer-swapping techniques for eliminating decode dispatch overhead in MoE or LLM serving frameworks.

We searched OpenAlex and arXiv for “Adopt static computation graph capture and persistent memory pointer swapping to eliminate CPU-side computational graph construction overhead during decoding.” and “Direct GPU Graph Recording and Persistent Memory Pointers to Eliminate Decode Dispatch Overhead” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

Yu Lin, Yiming Wang, Runyuan Cai +2 more · 2026

The paper introduces EDGE0, a streaming inference framework designed to serve large mixture-of-experts (MoE) models (such as 35B and 8B variants) on consumer devices constrained by limited RAM. It overcomes SSD loading latency by using a cross-token prerouter that predicts routing one token ahead, executes the prediction directly as the actual routing to completely overlap SSD expert streaming with compute, and restores accuracy loss from int4 quantization and routing replacement via an unmerged distillation LoRA.

Related Ideas

Is this a good idea?

Share onXLinkedIn