Homogeneous Mamba-Based Audio Tokenization
Replace Causal Transformer blocks in audio tokenizers with Mamba selective state space models to achieve linear computational complexity and capture global dependencies.
Source paper
Yitian Gong, Kuangwei Chen, Zhaoye Fei, Xiaogui Yang, Ke Chen, Yang Wang, Kexin Huang, Mingshu Chen, Ruixiao Li, Qingyuan Cheng, Shimin Li, Xipeng Qiu
Generated by AI. The paper’s authors did not write these.
Replace Causal Transformer blocks in audio tokenizers with Mamba selective state space models to achieve linear computational complexity and capture global dependencies.
Apply a structured disentanglement objective with gradient reversal to isolate semantic text information from speaker identity within vector-quantized audio codebooks.
Integrate a Connectionist Temporal Classification loss head into early quantization layers to enforce frame-level semantic alignment with phonemes.
This paper introduces MOSS-Audio-Tokenizer, a unified discrete audio tokenizer designed for audio foundation models. It utilizes a homogeneous, purely Transformer-based architecture (CAT) without CNNs, trained end-to-end on 3 million hours of audio to achieve state-of-the-art reconstruction and generation capabilities.
The model (CAT) replaces traditional CNN-based encoders with a causal Transformer architecture using sliding-window attention and patchifying operations. It employs Residual Vector Quantization (RVQ) and is optimized jointly using reconstruction loss, adversarial discriminators, and a semantic alignment loss via an auxiliary LLM.