Skip to content

Source paper

FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs

Tongyi SpeechTeam
arXiv 2407.04051 · 202439 upvotes on Hugging Face

Research ideas from this paper (4)

Generated by AI. The paper’s authors did not write these.

Semantic-Driven Prosody Inference for Implicit Emotion Control

Map LLM semantic hidden states directly into speech synthesis style spaces to automate expressive emotional tone generation without explicit tags.

FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
🟠Medium💡Moderate InnovationSingle GPU or lessRelated work found#Speech & audio#Generative media

End-to-End Speech-to-Speech Integration via Latent S3 Token Alignment

Replace modular speech-to-text-to-speech pipelines with unified architectures that feed acoustic speech tokens directly into language model embeddings.

FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
🔴Hard🔬Significant ResearchLarge clusterRelated work found#Speech & audio#Multimodal

Streamable SAN-M Architecture for Real-Time Interaction

Enable real-time streaming speech recognition by modifying memory-equipped self-attention encoders with causal chunk-based masking and Transducer losses.

FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
🔴Hard💡Moderate InnovationSingle GPU or lessNo close match found#Speech & audio#Efficiency

Cross-Lingual Phonetic Transfer for Under-Resourced ASR

Transfer universal phonetic representations to under-resourced languages using parameter-efficient fine-tuning on language-agnostic features.

FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
🟠Medium💡Moderate InnovationSingle GPU or lessRelated work found#Speech & audio#Training

AI summary

FunAudioLLM is a foundation model family comprising SenseVoice for multilingual speech understanding and CosyVoice for natural voice generation. These models aim to bridge the gap between human voice interaction and large language models (LLMs) by providing high-speed ASR, emotion recognition, and zero-shot voice cloning capabilities.

How it works (AI summary)

The framework utilizes a non-autoregressive SAN-M architecture for low-latency speech recognition (SenseVoice-Small) and an autoregressive Transformer encoder-decoder for high-precision multilingual tasks (SenseVoice-Large). For generation, CosyVoice employs a supervised semantic speech tokenizer (S3), an autoregressive language model for token prediction, and a flow-matching-based diffusion model for Mel-spectrogram reconstruction.