Skip to content
AI-generated research idea🔴Hard💡Moderate InnovationNo close match found

Streamable SAN-M Architecture for Real-Time Interaction

Enable real-time streaming speech recognition by modifying memory-equipped self-attention encoders with causal chunk-based masking and Transducer losses.

Time

A couple of months

Compute level

Single GPU or less

Resources

High-end consumer GPU or Cloud GPU instance (e.g., NVIDIA A100), public ASR datasets (e.g., LibriSpeech, Common Voice)

Prerequisites

Deep Learning, Automatic Speech Recognition (ASR), Self-Attention Networks, Transducer (RNN-T) architectures, Streaming inference logic

Scope

Significant architectural modification (converting batch attention to causal chunk-based attention, implementing state caching), adding RNN-T loss, and retraining

The Gap

The SenseVoice-Small model (Section 2.2) uses a memory-equipped self-attention network (SAN-M) designed for non-autoregressive, end-to-end processing. However, the authors state in the limitations that the model is not designed for streaming transcription.

Limitation

The current architecture requires the entire audio segment to be processed before returning a result, making it unsuitable for live captions or real-time human-agent translation.

Proposed Solution

Modify the SAN-M encoder to support chunk-based processing with causal attention masks. Implement a streaming Transducer (RNN-T) loss alongside the existing CTC loss to enable the model to emit tokens incrementally as audio frames are received.

Why This Matters

Enabling streaming capabilities is essential for real-world deployment in low-latency environments such as live telecommunications, accessibility tools, and interactive voice assistants.

How to Get Started

  1. Set up a baseline training pipeline using public ASR datasets like LibriSpeech and Common Voice.
  2. Modify the memory-equipped self-attention network encoder to support chunk-based processing with causal attention masks.
  3. Implement an RNN-T loss function alongside the existing CTC loss in the training framework.
  4. Evaluate real-time streaming latency, token emission speed, and word error rate on benchmark validation sets.

Code linked to this paper on Hugging Face: github.com/funaudiollm/cosyvoice

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

The listed works cover surveys on attention-based ASR and label-synchronous neural transducers for online recognition and translation, but none propose modifying the SAN-M architecture with chunk-based processing and streaming RNN-T loss.

We searched OpenAlex for “Enable real-time streaming speech recognition by modifying memory-equipped self-attention encoders with causal chunk-based masking and Transducer losses.” and “Streamable SAN-M Architecture for Real-Time Interaction” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs

Tongyi SpeechTeam · 2024

FunAudioLLM is a foundation model family comprising SenseVoice for multilingual speech understanding and CosyVoice for natural voice generation. These models aim to bridge the gap between human voice interaction and large language models (LLMs) by providing high-speed ASR, emotion recognition, and zero-shot voice cloning capabilities.

Related Ideas

Is this a good idea?

Share onXLinkedIn