Cross-Lingual Voice Cloning Generalization via Reinforcement Learning
Build a specialized Direct Preference Optimization pipeline targeting cross-lingual zero-shot voice cloning to maximize speaker similarity and minimize word error rates.
Source paper
Shijia Liao, Yuxuan Wang, Tianyu Li, Yifan Cheng, Ruoyi Zhang, Rongzhi Zhou, Yijin Xing
Generated by AI. The paper’s authors did not write these.
Build a specialized Direct Preference Optimization pipeline targeting cross-lingual zero-shot voice cloning to maximize speaker similarity and minimize word error rates.
Introduce an explicit emotion-aware context module into the Slow Transformer to generate stable and conditioned emotional text-to-speech outputs.
Deploy low-rank adaptation layers on the Slow Transformer cross-attention modules to improve low-resource language speech adaptation without catastrophic forgetting.
This paper introduces Fish-Speech, a novel multilingual Text-to-Speech (TTS) framework that leverages Large Language Models (LLMs) to bypass traditional grapheme-to-phoneme conversions. Its primary contribution is enhancing sequence generation stability and multilingual voice cloning capabilities through a dual autoregressive architecture and an advanced Grouped Finite Scalar Vector Quantization vocoder.
The system utilizes a Dual Autoregressive (Dual-AR) architecture consisting of a Slow Transformer for global semantic token generation and a Fast Transformer for detailed acoustic codebook processing. It is paired with FF-GAN, an enhanced vocoder that employs Grouped Finite Scalar Vector Quantization (GFSQ) and ParallelBlock convolutions to achieve near 100% codebook utilization and high-fidelity audio reconstruction.