Resolving Speaker Drift in Long-Form Synthesis via Explicit Global Conditioning
Inject global speaker embedding projections and cross-attention layers into the language model backbone to maintain speaker consistency in long-form synthesis.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
During the evaluation of long-form expressive generation using the EARS dataset (Section 6.2), the system relies heavily on online rejection sampling.
“In contrast to short-form generation, we observed occasional speaker drifting, which is reflected in the SIM scores.”
Limitation
The purely autoregressive modeling of continuous acoustic features causes occasional speaker drifting during long continuous outputs, necessitating computationally wasteful online rejection sampling to filter out degraded segments.
Proposed Solution
Integrate a global speaker embedding projection combined with cross-attention layers injected into the LLM backbone. This explicitly forces every autoregressive generation step to attend to a global reference speaker identity, decoupling speaker consistency from the localized, causal acoustic context.
Why This Matters
This modification will improve the stability of long-form speech generation by inherently preserving speaker identity, thus removing the computational bottleneck and latency overhead caused by online rejection sampling.
How to Get Started
How to test it: Hypothesis: Injecting global reference speaker embedding cross-attention layers into the Llama 3.2 1B/3B backbone will prevent long-form speaker drift without online rejection sampling or text-free guidance. Evaluate against unguided TADA-3B and TADA-3B + Online RS on EARS long-form synthesis using SIM, sSIM, CER, oMOS, and RTF, stopping if SIM fails to match TADA-3B + Online RS (74.7) or if CER degrades beyond 2.74.
- Extract global reference speaker embeddings using pre-trained speaker verification networks on the EARS dataset.
- Incorporate cross-attention layers into the language model backbone to ingest global speaker representations.
- Modify the autoregressive generation loop to condition every token prediction on the global speaker embedding.
- Evaluate long-form synthesis stability and speaker drift reduction compared to online rejection sampling baselines.
Code linked to this paper on Hugging Face: github.com/HumeAI/tada
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers such as A Dual-Stream Discrete Neural Codec with Fixed-Length Global Speaker Tokens and SwanTale study multi-speaker and long-form tasks using global speaker tokens or conditioning. However, none of these specific results focus directly on injecting cross-attention layers into the LLM backbone to resolve speaker drift as proposed.
- A Dual-Stream Discrete Neural Codec with Fixed-Length Global Speaker Tokens and Dynamic Frame Rates for Low-Bitrate Speech Tokenization · 2026 · 0 citations
- SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks · 2026 · 5 citations
We searched OpenAlex and arXiv for “Inject global speaker embedding projections and cross-attention layers into the language model backbone to maintain speaker consistency in long-form synthesis.” and “Resolving Speaker Drift in Long-Form Synthesis via Explicit Global Conditioning” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual AlignmentTrung Dang, Sharath Rao, Ananya Gupta +6 more · 2026
This paper introduces TADA, a generative framework that unifies text and speech modeling within a Large Language Model by establishing a one-to-one synchronization between continuous acoustic features and discrete text tokens. This approach addresses the frequency mismatch between text and audio, virtually eliminating content hallucinations and enabling highly efficient, single-stream autoregressive speech synthesis.
Related Ideas
- Authorized Vocal Identity Cloning via Adapter ModulesHeartMuLa: A Family of Open Sourced Music Foundation Models🟡Fairly easy
- State-Space Memory Integration for Unbounded Audio ContextOmniForcing: Unleashing Real-time Joint Audio-Visual Generation⛔Very hard
- Sliding-Window Attention with Global Context for Long-Form SynthesisLTX-2: Efficient Joint Audio-Visual Foundation Model🟠Medium
- Cross-Lingual Voice Cloning Generalization via Reinforcement LearningFish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis🔴Hard
- Hierarchical Disentanglement for Controllable TokenizationMOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models🟠Medium