Optimizing Parameter Efficiency via Language-Specific Tokenization
Reduce parameter count and inference latency by training a custom language-specific tokenizer with a smaller vocabulary to replace the default multilingual one.
Source paper
Evan King, Adam Sabra, Manjunath Kudlur, James Wang, Pete Warden
Generated by AI. The paper’s authors did not write these.
Reduce parameter count and inference latency by training a custom language-specific tokenizer with a smaller vocabulary to replace the default multilingual one.
Enable zero-shot automatic speech recognition for ultra-low resource dialects by training entirely on heavily augmented synthetic speech data.
Enable real-time streaming speech recognition by replacing global encoder attention with a block-causal or sliding-window mechanism.
This paper introduces 'Moonshine,' a family of tiny (27M parameter) monolingual ASR models optimized for edge devices, specifically targeting underrepresented languages like Arabic, Vietnamese, and Ukrainian. By leveraging a data strategy that combines public datasets, pseudo-labeling, and synthetic data generation, the authors demonstrate that specialized tiny models can significantly outperform comparably sized multilingual models (like Whisper Tiny) and match larger ones (Whisper Medium).
The authors utilize the Moonshine architecture, an encoder-decoder transformer using Rotary Position Embeddings (RoPE) and variable-length input processing to avoid unnecessary padding computation. They employ a three-stage data approach: aggregating public ASR data, pseudo-labeling raw audio from the web, and synthesizing speech from text-only datasets using TTS models.