Expanding and Balancing the Fish Audio Instruction Benchmark for Robust TTS Evaluation
Expand a text-to-speech instruction benchmark with multi-domain audio data and active learning to improve underrepresented acoustic tag evaluation and open-weights model scoring.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
In the creation of the Fish Audio Instruction Benchmark (Appendix A.4), the authors acknowledge that their evaluation dataset, built from MELD and a game character dataset, is limited in scope and domain.
“As a first release, it still has limitations, including limited data diversity, imbalanced acoustic-tag distributions, and early-stage human–model alignment analysis.”
Limitation
The current benchmark suffers from limited data diversity and imbalanced acoustic-tag distributions, which biases evaluation toward common vocalizations and fails to capture the full spectrum of nuanced human speech.
Proposed Solution
Develop an expanded, multi-domain instruction dataset by scraping and segmenting open-source podcasts, clinical interviews, and audiobooks. Apply active learning to specifically mine and annotate underrepresented acoustic tags (e.g., [stutter], [sigh], [crying]). Additionally, fine-tune a specialized, open-weights LLM to replace Gemini 3 Pro as an evaluator, improving human-model alignment for absolute scoring.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
A robust, diverse, and balanced benchmark is essential for accurately identifying model weaknesses in fine-grained emotion control, ensuring that future TTS models can handle complex, real-world conversational scenarios rather than just theatrical dialogue.
How to Get Started
How to test it: We hypothesize that fine-tuning an open LLM on an expanded, balanced dataset improves absolute scoring alignment with human experts, measured via Pearson correlation and Quadratic Weighted Kappa against professional annotations, stopping if the correlation does not exceed 0.65.
- Scrape and segment open-source podcasts, clinical interviews, and audiobooks to build a diverse raw audio corpus.
- Apply active learning and keyword mining techniques to isolate and annotate underrepresented acoustic tags like stutters and sighs.
- Fine-tune an open-weights LLM baseline on human-annotated scoring preferences to act as an automated evaluation model.
- Benchmark existing text-to-speech models using the newly expanded dataset and evaluate scoring correlation.
Code linked to this paper on Hugging Face: github.com/fishaudio/fish-speech
Suggested by AI. Adapt it to your setup.
Has This Been Done?
InstructTTSEval benchmarks natural-language instruction following in TTS systems, addressing a similar evaluation domain. However, the listed papers do not cover expanding a multi-domain dataset via active learning for specific acoustic tags or replacing Gemini with a specialized open-weights evaluator.
- InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems · 2025 · 54 citations
We searched OpenAlex and arXiv for “Expand a text-to-speech instruction benchmark with multi-domain audio data and active learning to improve underrepresented acoustic tag evaluation and open-weights model scoring.” and “Expanding and Balancing the Fish Audio Instruction Benchmark for Robust TTS Evaluation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Fish Audio S2 Technical ReportShijia Liao, Yuxuan Wang, Songting Liu +11 more · 2026
This paper introduces Fish Audio S2, an open-source text-to-speech (TTS) system that achieves multi-speaker, multi-turn generation and fine-grained natural language control over vocal features (e.g., emotion, prosody). The core contribution is an end-to-end framework that seamlessly integrates large-scale data curation, a highly optimized inference engine, and a novel reinforcement learning alignment process that reuses data filtering models as reward signals.
Related Ideas
- Enhancing Emotional Stability in Dual-AR TTS ArchitecturesFish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis🟠Medium
- Token-Based Paralinguistic InjectionIndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System🟡Fairly easy
- RLHF for Controllable Emotional SynthesisIndexTTS: An Industrial-Level Controllable and Efficient Zero-Shot Text-To-Speech System🟠Medium
- Latent Visual-Audio Contrastive BindingDreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation🟠Medium
- End-to-End Speech-to-Speech Integration via Latent S3 Token AlignmentFunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs🔴Hard