MIMO-Vibe: Multi-Stream Latent Diffusion for Overlapping Speech
Extend audio generation models to a multi-input multi-output framework with joint latent denoising to synthesize overlapping speech segments.
Source paper
Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, Yutao Sun, Li Dong, Yi Zhu, Weijiang Xu, Hangbo Bao, Zehua Wang, Shaohan Huang, Yan Xia, Furu Wei
Generated by AI. The paper’s authors did not write these.
Extend audio generation models to a multi-input multi-output framework with joint latent denoising to synthesize overlapping speech segments.
Replace speech-specific acoustic tokenizers with universal audio tokenizers and scene latent inputs to synthesize background noise and environmental sounds alongside speech.
Implement a language-agnostic phoneme-to-latent mapping layer using a frozen multilingual LLM and LoRA adapters for zero-shot cross-lingual speech synthesis.
Implement an asynchronous look-ahead denoising strategy combined with consistency distillation to reduce generation latency for interactive speech applications.
VIBEVOICE is a novel speech synthesis framework designed for generating long-form, multi-speaker conversational audio, supporting sequences up to 90 minutes. It introduces an ultra-high 3200x compression causal speech tokenizer that enables efficient processing of long sequences within a Large Language Model (LLM) framework using next-token diffusion.
The architecture integrates separate acoustic and semantic tokenizers with a pre-trained LLM (Qwen2.5) that processes interleaved voice font features and text scripts. A lightweight, four-layer token-level diffusion head is then conditioned on the LLM's hidden states to iteratively predict and refine continuous Variational Autoencoder (VAE) features for high-fidelity audio reconstruction.