Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationRelated work found

Expanding and Balancing the Fish Audio Instruction Benchmark for Robust TTS Evaluation

Expand a text-to-speech instruction benchmark with multi-domain audio data and active learning to improve underrepresented acoustic tag evaluation and open-weights model scoring.

Time

3 weeks

Compute level

Large cluster

Resources

8x NVIDIA H100 GPUs for 2 weeks for dataset scraping, active learning annotation, and LLM fine-tuning.

Prerequisites

basic ML knowledge and data curation

Scope

large-scale data collection

The Gap

In the creation of the Fish Audio Instruction Benchmark (Appendix A.4), the authors acknowledge that their evaluation dataset, built from MELD and a game character dataset, is limited in scope and domain.

“As a first release, it still has limitations, including limited data diversity, imbalanced acoustic-tag distributions, and early-stage human–model alignment analysis.”
From the paper, Section A.4, page 19. Matched against the PDF text.

Limitation

The current benchmark suffers from limited data diversity and imbalanced acoustic-tag distributions, which biases evaluation toward common vocalizations and fails to capture the full spectrum of nuanced human speech.

Proposed Solution

Develop an expanded, multi-domain instruction dataset by scraping and segmenting open-source podcasts, clinical interviews, and audiobooks. Apply active learning to specifically mine and annotate underrepresented acoustic tags (e.g., [stutter], [sigh], [crying]). Additionally, fine-tune a specialized, open-weights LLM to replace Gemini 3 Pro as an evaluator, improving human-model alignment for absolute scoring.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

A robust, diverse, and balanced benchmark is essential for accurately identifying model weaknesses in fine-grained emotion control, ensuring that future TTS models can handle complex, real-world conversational scenarios rather than just theatrical dialogue.

How to Get Started

How to test it: We hypothesize that fine-tuning an open LLM on an expanded, balanced dataset improves absolute scoring alignment with human experts, measured via Pearson correlation and Quadratic Weighted Kappa against professional annotations, stopping if the correlation does not exceed 0.65.

  1. Scrape and segment open-source podcasts, clinical interviews, and audiobooks to build a diverse raw audio corpus.
  2. Apply active learning and keyword mining techniques to isolate and annotate underrepresented acoustic tags like stutters and sighs.
  3. Fine-tune an open-weights LLM baseline on human-annotated scoring preferences to act as an automated evaluation model.
  4. Benchmark existing text-to-speech models using the newly expanded dataset and evaluate scoring correlation.

Code linked to this paper on Hugging Face: github.com/fishaudio/fish-speech

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

InstructTTSEval benchmarks natural-language instruction following in TTS systems, addressing a similar evaluation domain. However, the listed papers do not cover expanding a multi-domain dataset via active learning for specific acoustic tags or replacing Gemini with a specialized open-weights evaluator.

We searched OpenAlex and arXiv for “Expand a text-to-speech instruction benchmark with multi-domain audio data and active learning to improve underrepresented acoustic tag evaluation and open-weights model scoring.” and “Expanding and Balancing the Fish Audio Instruction Benchmark for Robust TTS Evaluation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Fish Audio S2 Technical Report

Shijia Liao, Yuxuan Wang, Songting Liu +11 more · 2026

This paper introduces Fish Audio S2, an open-source text-to-speech (TTS) system that achieves multi-speaker, multi-turn generation and fine-grained natural language control over vocal features (e.g., emotion, prosody). The core contribution is an end-to-end framework that seamlessly integrates large-scale data curation, a highly optimized inference engine, and a novel reinforcement learning alignment process that reuses data filtering models as reward signals.

Related Ideas

Is this a good idea?

Share onXLinkedIn