Multilingual Generalization via Universal Tokenization
Replace language-specific tokenizers with a universal multilingual tokenizer and language ID embeddings to expand speech models beyond Chinese and English.
Source paper
Wei Deng, Siyi Zhou, Jingchen Shu, Jinchao Wang, Lu Wang
Generated by AI. The paper’s authors did not write these.
Replace language-specific tokenizers with a universal multilingual tokenizer and language ID embeddings to expand speech models beyond Chinese and English.
Apply Direct Preference Optimization using an emotional scoring reward model to steer autoregressive speech generation toward rich prosodic variance and specific emotions.
Augment speech generation models with special tokens for paralinguistic expressions like laughter so synthesized speech includes natural non-verbal sounds.
This paper presents IndexTTS, an industrial-level zero-shot Text-to-Speech system designed for high efficiency, controllability, and stability, particularly in Chinese and English scenarios. The authors introduce a hybrid character-pinyin modeling approach to handle polyphonic ambiguity and replace standard Vector Quantization with Finite-Scalar Quantization (FSQ) to ensure high codebook utilization.
The model utilizes a decoder-only Transformer (SpeechLLM) conditioned on a Conformer-encoded speech prompt to generate acoustic tokens from text. It employs a hybrid text tokenizer (mixing BPE and pinyin), FSQ for acoustic tokenization to prevent codebook collapse, and a BigVGAN2 decoder to reconstruct high-quality waveforms from the latent representations.