Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchClosely matching work found

Mitigating the Modality Gap via Mixture-of-Experts (MoE) Routing

Implement a sparse Mixture-of-Experts routing architecture within language model transformer blocks to preserve text reasoning while handling parallel acoustic features.

Time

2 to 4 weeks for MoE upcycling, two-stage multimodal training, and speech/language evaluation.

Compute level

Large cluster

Resources

A multi-node GPU training cluster (e.g., 32 to 64 H100 GPUs) with fast interconnect (InfiniBand) for MoE all-to-all communication and distributed training of the expanded 1B/3B MoE model over 400k steps on 270k to 905k hours of data.

What the paper used

Inference benchmarking used a single H100 GPU; training hardware is not stated in the paper. Training used 270k hours of English and 635k hours of non-English speech, a Wav2Vec2-large aligner (300k steps, batch size 64), a 6-layer transformer plus DAC CNN VAE, and Llama 3.2 1B/3B backbones (200k steps at batch size 256, context 192, plus 200k steps at batch size 64, context 1024/2048).

Prerequisites

large language models, Mixture-of-Experts, parameter-efficient fine-tuning

Scope

new model architecture

The Gap

In the Spoken Language Modeling evaluation on benchmarks like sSC and tSC (Section 6.3), the model shows a drop in text-only accuracy compared to the base Llama model.

“On sSC and tSC, we observe an accuracy drop when comparing TADA operating in text-only mode to the base Llama models, suggesting the model trades off some of its text pretraining capacity to accommodate the speech modality.”
From the paper, Section 6.3, page 8. Matched against the PDF text.

Limitation

Training the unified model on text-speech data causes a degradation in pure text-only reasoning capacity, indicating that the LLM trades off some of its foundational linguistic pretraining to accommodate the new speech modality.

Proposed Solution

Implement a sparse Mixture-of-Experts (MoE) routing architecture within the LLM transformer blocks. Dedicated text-only experts can be frozen or regularized to preserve base reasoning, while new speech-aware experts are dynamically routed to handle the parallel acoustic feature modeling.

Why This Matters

This prevents catastrophic forgetting of pure language intelligence, allowing unified multimodal systems to scale speech capabilities without sacrificing state-of-the-art text reasoning.

How to Get Started

How to test it: Hypothesis: Converting Llama 3.2 1B/3B FFN blocks into sparse MoE layers with frozen text experts and trainable speech-routed experts will eliminate text-only capability degradation while preserving multimodal quality. Compare against base Llama-1B/3B-Instruct and TADA-1B/3B-ML (with SFG) on SI PPL, sSC, tSC, CER, SIM, and oMOS, stopping if text-only sSC/tSC drops noticeably below base Llama-3B-Instruct (79.4/98.0) or text-speech sSC/tSC falls below TADA-3B-ML w/ SFG (66.7/94.7).

  1. Download the base Llama model and inspect its transformer block definitions.
  2. Implement a routing mechanism that splits incoming tokens between frozen text experts and newly introduced speech experts.
  3. Set up the sSC and tSC evaluation benchmarks to measure text-only accuracy alongside speech modeling performance.
  4. Train the modified model on a combined speech-text dataset while regularizing the base experts against catastrophic forgetting.

Code linked to this paper on Hugging Face: github.com/HumeAI/tada

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Closely matching work found

MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts directly targets mixing speech and text using modality-aware mixture-of-experts architectures. Additionally, Uni-MoE and MoE-LLaVA explore scaling multimodal LLMs with mixture of experts to bridge modality gaps.

We searched OpenAlex and arXiv for “Implement a sparse Mixture-of-Experts routing architecture within language model transformer blocks to preserve text reasoning while handling parallel acoustic features.” and “Mitigating the Modality Gap via Mixture-of-Experts (MoE) Routing” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

TADA: A Generative Framework for Speech Modeling via Text-Acoustic Dual Alignment

Trung Dang, Sharath Rao, Ananya Gupta +6 more · 2026

This paper introduces TADA, a generative framework that unifies text and speech modeling within a Large Language Model by establishing a one-to-one synchronization between continuous acoustic features and discrete text tokens. This approach addresses the frequency mismatch between text and audio, virtually eliminating content hallucinations and enabling highly efficient, single-stream autoregressive speech synthesis.

Related Ideas

Is this a good idea?

Share onXLinkedIn