Non-Autoregressive Local Decoding for Real-Time Inference
Replace the autoregressive local transformer with a non-autoregressive module to predict residual codebook tokens in parallel and reduce real-time inference latency.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 3.1, the Local Transformer (θloc) models intra-frame dependencies autoregressively, predicting residual codebook tokens (k=1 to K-1) sequentially conditioned on previous layers (hl,<k). In Section 3.5, this dependency contributes to high latency and excessive kernel launches, requiring complex optimizations like CUDA Graphs to achieve reasonable speeds.
“Given HeartMuLa's Global-Local architecture, where a lightweight Local Transformer predicts a multi-layer RVQ, the computational latency added by increasing the RVQ layers is significantly lower than the linear growth caused by increasing the frame rate.”
Limitation
The autoregressive modeling of residual codebook layers creates a computational bottleneck, preventing true parallel generation of acoustic details and complicating the inference pipeline.
Proposed Solution
Replace the autoregressive Local Transformer with a Non-Autoregressive (NAR) module, such as a MaskGIT-style bidirectional transformer or a Flow Matching network. This module would predict all residual codebooks for a sequence of global tokens in parallel (or in a fixed number of iterative refinement steps) rather than sequentially per layer/token, drastically reducing GPU kernel launches.
Why This Matters
This would decouple the generation speed from the number of codebook layers, enabling faster-than-real-time generation on standard hardware without fragile system-level optimizations like custom CUDA graphs.
How to Get Started
How to test it: Hypothesis: A non-autoregressive local decoder reduces inference latency compared to the autoregressive local transformer while maintaining acoustic quality. Baseline: The original autoregressive Local Transformer. Metric: Real-Time Factor (RTF) and Perceptual Quality (PQ) score. Stop condition: If RTF decreases by at least 30% but PQ score drops by more than 0.2 points, stop and reject the modification.
- Identify the baseline autoregressive local transformer codebase and measure current generation latency.
- Design a MaskGIT-style bidirectional transformer or flow matching module to replace the sequential layer.
- Train the non-autoregressive module using teacher forcing on pre-computed codebook representations.
- Benchmark the inference speedup and reconstruction quality against the original autoregressive baseline.
Code linked to this paper on Hugging Face: github.com/HeartMuLa/heartlib
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed works discuss standard autoregressive skip decoding, batching, caching, and non-autoregressive machine translation. None of them cover applying MaskGIT-style or Flow Matching non-autoregressive modules specifically for residual codebook prediction in local transformer decoding.
We searched OpenAlex for “Replace the autoregressive local transformer with a non-autoregressive module to predict residual codebook tokens in parallel and reduce real-time inference latency.” and “Non-Autoregressive Local Decoding for Real-Time Inference” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
HeartMuLa: A Family of Open Sourced Music Foundation ModelsDongchao Yang, Yuxin Xie, Yuguo Yin +25 more · 2026
HeartMuLa introduces a comprehensive open-source ecosystem for music generation, featuring a low-frame-rate audio tokenizer (HeartCodec), a lyric aligner, and a 3B parameter autoregressive generation model. By factorizing generation into global structural modeling and local acoustic refinement, and employing a four-stage training pipeline including Direct Preference Optimization (DPO), the system achieves state-of-the-art long-form music synthesis (up to 6 minutes) with high lyrical intelligibility and style control.
Related Ideas
- Semantic-Aware Variable-Rate TokenizationNextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation🔴Hard
- Hardware-Aware Kernels for Zero-Overhead InferenceBitNet b1.58 2B4T Technical Report⛔Very hard
- Native Integration of Adaptive Gradient CompressionPyTorch Distributed: Experiences on Accelerating Data Parallel Training🟠Medium
- Hybrid Sliding-Window and Fixed-State Depth Memory for Deeply Looped TransformersLoopback Transformer: Towards a Unified View of Residual Looped Transformers🔴Hard
- Dynamic Network-Compute Latency ArbitratorLMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference🟡Fairly easy