Learnable Adaptive SSTA (Soft-Gating)
Replace hard top-k block pruning with a learnable soft-gating mechanism using Gumbel-Softmax to prevent abrupt attention drop-offs in complex scenes.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Algorithm 1 'Selective and Sliding Tile Attention' relies on a hard top-k selection based on static block similarity scores (`Selected_indexes ← Topk(Score)`). This binary pruning (keep vs. drop) is non-differentiable and can lead to abrupt attention cut-offs where relevant context is dropped if it falls just below the threshold, causing discontinuity in complex scenes.
Limitation
The heuristic, hard-coded top-k pruning in the attention mechanism is rigid and may discard subtle but crucial long-range dependencies.
Proposed Solution
Replace the hard Top-k selection with a learnable soft-gating mechanism (e.g., using Gumbel-Softmax or a lightweight predictor network). Train a small auxiliary 'router' network that predicts the importance of block pairs dynamically based on the current hidden state, allowing the model to learn *where* to be sparse rather than relying on fixed Q-K similarity heuristics.
Why This Matters
Enhances the model's ability to handle complex, chaotic scenes where 'importance' is not strictly correlated with simple Q-K dot products, potentially improving generation consistency.
How to Get Started
- Extract the attention-based baseline model and locate the hard top-k selection logic within the sliding tile attention module.
- Design a lightweight router network that predicts block importance based on hidden states.
- Implement the Gumbel-Softmax estimator to enable end-to-end differentiable soft-gating.
- Benchmark the updated model on long-context language tasks to measure perplexity and training stability improvements.
Code linked to this paper on Hugging Face: github.com/Tencent-Hunyuan/HunyuanVideo-1.5
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The listed search results cover vision transformer compression surveys and continuous fine-tuning, but none of them discuss learnable soft-gating mechanisms such as Gumbel-Softmax routers to replace hard Top-k selection in SSTA.
We searched OpenAlex for “Replace hard top-k block pruning with a learnable soft-gating mechanism using Gumbel-Softmax to prevent abrupt attention drop-offs in complex scenes.” and “Learnable Adaptive SSTA (Soft-Gating)” and an AI compared the idea with the top 2 results. A small search can miss relevant work.
Source Paper
HunyuanVideo 1.5 Technical ReportBing Wu, Chang Zou, Changlin Li +78 more · 2025
This paper introduces HunyuanVideo 1.5, an open-source 8.3B parameter video generation model that achieves state-of-the-art performance comparable to proprietary models like Sora and Kling. The authors propose a unified Diffusion Transformer (DiT) architecture optimized with a novel Selective and Sliding Tile Attention (SSTA) mechanism and a high-compression 3D VAE, trained via a progressive multi-stage pipeline involving text-to-image, text-to-video, and image-to-video tasks.
Related Ideas
- End-to-End Trainable Sparse Attention Indexers for RLGLM-5: from Vibe Coding to Agentic Engineering🔴Hard
- Learned Query-Aware Importance PredictionInfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams🟠Medium
- Granular Credit Assignment via Learned Value FunctionsAgent Lightning: Train ANY AI Agents with Reinforcement Learning🟠Medium
- Differentiable Relaxation of Written Associative Memories for Few-Shot Fine-TuningWeight-Space Programming Doesn't Need Transformers🟡Fairly easy
- Learnable Temporal-Semantic Affinity for ConsolidationSimpleMem: Efficient Lifelong Memory for LLM Agents🟠Medium