Linear-Complexity Depth-Wise Attention for Scalable Full Attention Residuals
A method that replaces depth-wise softmax attention with linear-complexity kernelized attention along the depth dimension, allowing networks to scale residual connections efficiently without quadratic memory overhead.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 6.1, the authors note that they rely on standard softmax attention across depth because current layer counts are relatively small. However, as networks grow deeper, softmax attention scales quadratically with depth O(L^2) and requires full O(Ld) memory, preventing the use of Full AttnRes at massive scale and necessitating the compressed 'Block' variant.
“Because Full AttnRes must access all preceding layer outputs at every layer, the memory footprint of cross-layer aggregation grows as O(Ld), which is prohibitive for large-scale models on current hardware.”
Limitation
Full AttnRes has prohibitive memory and pipeline communication overhead O(Ld) for very deep models, forcing the use of Block AttnRes, which sacrifices fine-grained, layer-wise resolution.
Proposed Solution
Replace the depth-wise softmax attention with linear-complexity attention mechanisms (e.g., kernelized linear attention like GLA or Delta Rule) along the depth dimension. This would allow each layer to maintain a fixed-size compressed state of all prior layers, enabling Full AttnRes-like layer-wise granularity without the O(Ld) caching overhead.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
This would allow models to achieve the superior validation loss of Full AttnRes over Block AttnRes without hitting the memory and communication bottlenecks during distributed pipeline parallel training.
How to Get Started
How to test it: Hypothesis: replacing depth-wise softmax attention with a linear-complexity depth recurrence (e.g., GLA or Delta Rule) achieves validation loss close to Full AttnRes (1.737) without O(Ld) activation caching or cross-stage communication; compare against PreNorm, Block AttnRes (N=8, 1.746), and Full AttnRes on the 16-layer MoE ablation setup measuring validation loss, per-layer I/O, and pipeline overhead. Stop if validation loss fails to beat Block AttnRes (1.746) at matched compute or if state-expansion overhead exceeds Block AttnRes's 5.5d I/O and <4% training overhead.
- Set up a standard transformer baseline utilizing full depth-wise attention residual blocks (AttnRes).
- Implement kernelized linear attention mechanisms, such as Generalized Linear Attention (GLA) or the Delta Rule, adapted for the depth dimension.
- Incorporate the linear depth-wise attention into a deep transformer model while tracking peak memory consumption.
- Benchmark the scaled architecture on standard language modeling tasks to compare performance and O(Ld) caching overhead against quadratic baselines.
Code linked to this paper on Hugging Face: github.com/MoonshotAI/Attention-Residuals
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works like SoLA-Vision: Fine-grained Layer-wise Linear Softmax Hybrid Attention and SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation explore layer-wise linear softmax hybrid attention and attention residuals, but do not specifically solve the depth-wise full attention residual caching overhead with kernelized linear attention along the depth dimension. They leave open the direct combination of linear attention mechanics applied to the depth-wise residual stream.
- SoLA-Vision: Fine-grained Layer-wise Linear Softmax Hybrid Attention · 2026 · 2 citations
- MiniMax-01: Scaling Foundation Models with Lightning Attention · 2025 · 249 citations
- Fast Conformer With Linearly Scalable Attention For Efficient Speech Recognition · 2023 · 300 citations
- Polarized self-attention: Towards high-quality pixel-wise mapping · 2022 · 573 citations
We searched OpenAlex for “A method that replaces depth-wise softmax attention with linear-complexity kernelized attention along the depth dimension, allowing networks to scale residual connections efficiently without quadratic memory overhead.” and “Linear-Complexity Depth-Wise Attention for Scalable Full Attention Residuals” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Attention ResidualsKimi Team, Guangyu Chen, Yu Zhang +34 more · 2026
This paper introduces Attention Residuals (AttnRes) to address the limitations of standard residual connections, which uniformly aggregate all prior layer outputs and cause uncontrolled hidden-state growth. The authors propose replacing fixed accumulation with learned, depth-wise softmax attention, allowing layers to selectively retrieve information from earlier layers to improve downstream performance.
Related Ideas
- Hybrid Sliding-Window and Fixed-State Depth Memory for Deeply Looped TransformersLoopback Transformer: Towards a Unified View of Residual Looped Transformers🔴Hard
- Multi-Scale Hierarchical Masking for Dense Prediction TasksIn Pursuit of Pixel Supervision for Visual Pre-training🟠Medium
- Controlled Block-Level Comparison of Content-Gated Recurrent Depth ReadersLoopback Transformer: Towards a Unified View of Residual Looped Transformers🟡Fairly easy
- End-to-End Trainable Sparse Attention Indexers for RLGLM-5: from Vibe Coding to Agentic Engineering🔴Hard
- Dual-Axis KV Sharing: Integrating Joint-KV with Cross-Layer and Prefix-Shared AttentionMulti-Branch Transformer🟠Medium