Skip to content
AI-generated research idea🟠Medium🔧Minor TweakRelated work found

Resolving Speaker Drift in Long-Form Synthesis via Explicit Global Conditioning

Inject global speaker embedding projections and cross-attention layers into the language model backbone to maintain speaker consistency in long-form synthesis.

Time

2 to 4 weeks for full two-stage training (context 192 then 1024/2048) and long-form benchmark evaluation.

Compute level

Large cluster

Resources

A multi-node GPU training cluster (e.g., 32 to 64 H100 GPUs) to retrain the Llama 3.2 1B/3B backbone with added cross-attention layers over the two-stage 400k-step schedule on 270k to 905k hours of speech-text data.

What the paper used

Inference benchmarking used a single H100 GPU; training hardware is not stated in the paper. Training used 270k hours of English and 635k hours of non-English speech, a Wav2Vec2-large aligner (300k steps, batch size 64), a 6-layer transformer plus DAC CNN VAE, and Llama 3.2 1B/3B backbones (200k steps at batch size 256, context 192, plus 200k steps at batch size 64, context 1024/2048).

Prerequisites

transformer architectures, voice cloning techniques

Scope

modify existing code

The Gap

During the evaluation of long-form expressive generation using the EARS dataset (Section 6.2), the system relies heavily on online rejection sampling.

“In contrast to short-form generation, we observed occasional speaker drifting, which is reflected in the SIM scores.”
From the paper, Section 6.2, page 7. Matched against the PDF text.

Limitation

The purely autoregressive modeling of continuous acoustic features causes occasional speaker drifting during long continuous outputs, necessitating computationally wasteful online rejection sampling to filter out degraded segments.

Proposed Solution

Integrate a global speaker embedding projection combined with cross-attention layers injected into the LLM backbone. This explicitly forces every autoregressive generation step to attend to a global reference speaker identity, decoupling speaker consistency from the localized, causal acoustic context.

Why This Matters

This modification will improve the stability of long-form speech generation by inherently preserving speaker identity, thus removing the computational bottleneck and latency overhead caused by online rejection sampling.

How to Get Started

How to test it: Hypothesis: Injecting global reference speaker embedding cross-attention layers into the Llama 3.2 1B/3B backbone will prevent long-form speaker drift without online rejection sampling or text-free guidance. Evaluate against unguided TADA-3B and TADA-3B + Online RS on EARS long-form synthesis using SIM, sSIM, CER, oMOS, and RTF, stopping if SIM fails to match TADA-3B + Online RS (74.7) or if CER degrades beyond 2.74.

  1. Extract global reference speaker embeddings using pre-trained speaker verification networks on the EARS dataset.
  2. Incorporate cross-attention layers into the language model backbone to ingest global speaker representations.
  3. Modify the autoregressive generation loop to condition every token prediction on the global speaker embedding.
  4. Evaluate long-form synthesis stability and speaker drift reduction compared to online rejection sampling baselines.

Code linked to this paper on Hugging Face: github.com/HumeAI/tada

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers such as A Dual-Stream Discrete Neural Codec with Fixed-Length Global Speaker Tokens and SwanTale study multi-speaker and long-form tasks using global speaker tokens or conditioning. However, none of these specific results focus directly on injecting cross-attention layers into the LLM backbone to resolve speaker drift as proposed.

We searched OpenAlex and arXiv for “Inject global speaker embedding projections and cross-attention layers into the language model backbone to maintain speaker consistency in long-form synthesis.” and “Resolving Speaker Drift in Long-Form Synthesis via Explicit Global Conditioning” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment

Trung Dang, Sharath Rao, Ananya Gupta +6 more · 2026

This paper introduces TADA, a generative framework that unifies text and speech modeling within a Large Language Model by establishing a one-to-one synchronization between continuous acoustic features and discrete text tokens. This approach addresses the frequency mismatch between text and audio, virtually eliminating content hallucinations and enabling highly efficient, single-stream autoregressive speech synthesis.

Related Ideas

Is this a good idea?

Share onXLinkedIn