Enhancing Perceptual Speech Quality via Flow-Matching Acoustic Decoders
Replace deterministic VAE decoders with a conditional continuous normalizing flow to map LLM latent acoustic vectors into high-quality continuous waveforms.
Source paper
Trung Dang, Sharath Rao, Ananya Gupta, Christopher Gagne, Panagiotis Tzirakis, Alice Baird, Jakub Piotr Cłapa, Peter Chin, Alan Cowen
Generated by AI. The paper’s authors did not write these.
Replace deterministic VAE decoders with a conditional continuous normalizing flow to map LLM latent acoustic vectors into high-quality continuous waveforms.
Inject global speaker embedding projections and cross-attention layers into the language model backbone to maintain speaker consistency in long-form synthesis.
Implement a sparse Mixture-of-Experts routing architecture within language model transformer blocks to preserve text reasoning while handling parallel acoustic features.
This paper introduces TADA, a generative framework that unifies text and speech modeling within a Large Language Model by establishing a one-to-one synchronization between continuous acoustic features and discrete text tokens. This approach addresses the frequency mismatch between text and audio, virtually eliminating content hallucinations and enabling highly efficient, single-stream autoregressive speech synthesis.
The framework utilizes an alignment-aware Variational Autoencoder (VAE) to compress high-frequency acoustic features into latent vectors that map directly to text tokens via Connectionist Temporal Classification (CTC) and Viterbi forced alignment. A Llama-based LLM is then augmented with a flow-matching head to autoregressively predict these synchronized continuous acoustic latents and their temporal durations in a unified stream.