End-to-End Speech-to-Speech Integration via Latent S3 Token Alignment
Replace modular speech-to-text-to-speech pipelines with unified architectures that feed acoustic speech tokens directly into language model embeddings.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section 5 and 6 describe the current framework as a pipeline where SenseVoice performs ASR, an LLM processes text, and CosyVoice generates audio. The authors identify that this modular approach leads to error propagation, where a transcription error in SenseVoice causes a logic error in the LLM.
Limitation
High latency and cascading error propagation inherent in modular speech-to-text-to-speech pipelines.
Proposed Solution
Develop a unified multimodal architecture where the S3 tokens produced by the SenseVoice encoder are directly consumed by the LLM's embedding space. The LLM would be trained to predict target-language S3 tokens directly, bypassing the intermediate text representation entirely to preserve paralinguistic nuances.
Why This Matters
This transition to a true end-to-end speech model would reduce inference latency and prevent the loss of emotional or speaker-specific information that occurs during the text-bottleneck stage.
How to Get Started
- Gather large-scale aligned speech-text and speech-speech datasets for multi-modal training.
- Extract discrete speech tokens using an audio encoder like SenseVoice.
- Modify the language model embedding layer to accept speech tokens alongside text tokens.
- Train the unified model to predict target-language speech tokens autoregressively and evaluate end-to-end latency and accuracy.
Code linked to this paper on Hugging Face: github.com/funaudiollm/cosyvoice
Suggested by AI. Adapt it to your setup.
Has This Been Done?
VITA-Audio discusses fast interleaved cross-modal token generation for speech-language models, while other papers deal with speech tokenizers generally. However, they do not specifically target direct S3 token alignment bypassing text representation as proposed.
- Speech Tokenizers are Vulnerable: Transferable Semantic Attack and Robust Tokenizer · 2026 · 0 citations
- VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model · 2025 · 43 citations
We searched OpenAlex for “Replace modular speech-to-text-to-speech pipelines with unified architectures that feed acoustic speech tokens directly into language model embeddings.” and “End-to-End Speech-to-Speech Integration via Latent S3 Token Alignment” and an AI compared the idea with the top 6 results. A small search can miss relevant work.
Source Paper
FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMsTongyi SpeechTeam · 2024
FunAudioLLM is a foundation model family comprising SenseVoice for multilingual speech understanding and CosyVoice for natural voice generation. These models aim to bridge the gap between human voice interaction and large language models (LLMs) by providing high-speed ASR, emotion recognition, and zero-shot voice cloning capabilities.
Related Ideas
- CTC-Informed Semantic Alignment for ASRMOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models🟡Fairly easy
- Cross-Lingual Timbre Transfer for Low-Resource TTS AlignmentFish Audio S2 Technical Report🟠Medium
- Token-Based Paralinguistic InjectionIndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System🟡Fairly easy
- Universal Phonemic Mapping for Zero-Shot Cross-Lingual SynthesisVibeVoice Technical Report🔴Hard
- MIMO-Vibe: Multi-Stream Latent Diffusion for Overlapping SpeechVibeVoice Technical Report🔴Hard