Streaming-Compatible Architecture via Block-Causal Attention
Enable real-time streaming speech recognition by replacing global encoder attention with a block-causal or sliding-window mechanism.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section 2.1 describes the model as an encoder-decoder transformer. While it avoids padding, standard transformer encoders typically utilize global attention, requiring the entire audio utterance to be processed before decoding begins.
“Low latency ASR models that are small enough to deploy on-device can empower a new generation of voice-driven applications”
Limitation
The requirement to process the full audio context introduces latency proportional to the input duration, which prevents true real-time streaming (e.g., live captioning) on edge devices.
Proposed Solution
Modify the encoder to use block-causal attention or a sliding-window attention mechanism (similar to Streaming Whisper or Emformer). This allows the encoder to process audio in chunks and pass representations to the decoder incrementally, enabling the model to output text while the user is still speaking.
Why This Matters
For edge devices used in live translation or smart assistants (key use cases mentioned in the intro), minimizing the time-to-first-token is crucial for user experience.
How to Get Started
How to test it: We hypothesize that incorporating block-causal attention will enable real-time streaming capability with negligible degradation in accuracy. Baseline is the standard full-context encoder model. Metric is real-time factor (RTF) and Word Error Rate (WER). We stop if WER drops by more than 2% relative to the baseline.
- Analyze the current encoder architecture to identify global attention layers that require full utterance processing.
- Implement block-causal attention or sliding-window mechanisms inspired by streaming architectures like Whisper.
- Evaluate the modified model on streaming benchmarks to measure latency improvements and word error rate trade-offs.
Code linked to this paper on Hugging Face: github.com/moonshine-ai/moonshine
Suggested by AI. Adapt it to your setup.
Has This Been Done?
StreamFlow explores streaming flow matching using block-wise guided attention masks, while AvatarForcing utilizes local-future sliding-window denoising. However, neither explicitly implements the specific block-causal encoder architecture combined with incremental text output for the proposed ASR setup.
- StreamFlow: Streaming Flow Matching with Block-Wise Guided Attention Mask for Speech Token Decoding · 2026 · 4 citations
- AvatarForcing: One-Step Streaming Talking Avatars via Local-Future Sliding-Window Denoising · 2026 · 7 citations
We searched OpenAlex for “Enable real-time streaming speech recognition by replacing global encoder attention with a block-causal or sliding-window mechanism.” and “Streaming-Compatible Architecture via Block-Causal Attention” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Flavors of Moonshine: Tiny Specialized ASR Models for Edge DevicesEvan King, Adam Sabra, Manjunath Kudlur +2 more · 2025
This paper introduces 'Moonshine,' a family of tiny (27M parameter) monolingual ASR models optimized for edge devices, specifically targeting underrepresented languages like Arabic, Vietnamese, and Ukrainian. By leveraging a data strategy that combines public datasets, pseudo-labeling, and synthetic data generation, the authors demonstrate that specialized tiny models can significantly outperform comparably sized multilingual models (like Whisper Tiny) and match larger ones (Whisper Medium).
Related Ideas
- Streamable SAN-M Architecture for Real-Time InteractionFunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs🔴Hard
- Homogeneous Mamba-Based Audio TokenizationMOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models🔴Hard
- Causal Diffusion Decoding for 25Hz Semantic TokensQwen3-TTS Technical Report🔴Hard
- Hierarchical Active Perception EncodersOmniGAIA: Towards Native Omni-Modal AI Agents🔴Hard
- Sliding-Window Attention with Global Context for Long-Form SynthesisLTX-2: Efficient Joint Audio-Visual Foundation Model🟠Medium