Skip to content
AI-generated research idea🔴Hard🔬Significant ResearchRelated work found

Linear-Complexity Depth-Wise Attention for Scalable Full Attention Residuals

A method that replaces depth-wise softmax attention with linear-complexity kernelized attention along the depth dimension, allowing networks to scale residual connections efficiently without quadratic memory overhead.

Time

3-4 weeks on 64-128 H100-class GPUs for kernel/pipeline integration and ablation runs (or ~1.5-2 months if scaled to the 1.4T-token 48B/3B run).

Compute level

Large cluster

Resources

At the paper's ablation and scaling-law scale (436M-528M active parameters, 87.9B-119.0B tokens per run, ~1.5-5.6 PFLOP/s-days each), testing 3-4 linear-depth variants plus baselines requires ~6-15 PFLOP/s-days, or roughly 64-128 H100-class GPUs (plus subsequent 1.4T-token 48B/3B validation if successful).

What the paper used

Specific GPU hardware is not stated in the paper. It reports scaling-law runs across 5 MoE sizes (194M-528M activated parameters, 38.7B-119.0B tokens, 8192 context, up to ~5.6 PFLOP/s-days), a 25-point architecture sweep (~6.5e19 FLOPs, ~2.3e8 active parameters), 16-layer ablations (436M active parameters, 87.9B tokens), and main runs on Kimi Linear (48B total / 3B activated parameters, 54 layers, 1.4T tokens).

Prerequisites

transformer architectures, linear attention

Scope

new model architecture

The Gap

In Section 6.1, the authors note that they rely on standard softmax attention across depth because current layer counts are relatively small. However, as networks grow deeper, softmax attention scales quadratically with depth O(L^2) and requires full O(Ld) memory, preventing the use of Full AttnRes at massive scale and necessitating the compressed 'Block' variant.

“Because Full AttnRes must access all preceding layer outputs at every layer, the memory footprint of cross-layer aggregation grows as O(Ld), which is prohibitive for large-scale models on current hardware.”
From the paper, Conclusion, page 16. Matched against the PDF text.

Limitation

Full AttnRes has prohibitive memory and pipeline communication overhead O(Ld) for very deep models, forcing the use of Block AttnRes, which sacrifices fine-grained, layer-wise resolution.

Proposed Solution

Replace the depth-wise softmax attention with linear-complexity attention mechanisms (e.g., kernelized linear attention like GLA or Delta Rule) along the depth dimension. This would allow each layer to maintain a fixed-size compressed state of all prior layers, enabling Full AttnRes-like layer-wise granularity without the O(Ld) caching overhead.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

This would allow models to achieve the superior validation loss of Full AttnRes over Block AttnRes without hitting the memory and communication bottlenecks during distributed pipeline parallel training.

How to Get Started

How to test it: Hypothesis: replacing depth-wise softmax attention with a linear-complexity depth recurrence (e.g., GLA or Delta Rule) achieves validation loss close to Full AttnRes (1.737) without O(Ld) activation caching or cross-stage communication; compare against PreNorm, Block AttnRes (N=8, 1.746), and Full AttnRes on the 16-layer MoE ablation setup measuring validation loss, per-layer I/O, and pipeline overhead. Stop if validation loss fails to beat Block AttnRes (1.746) at matched compute or if state-expansion overhead exceeds Block AttnRes's 5.5d I/O and <4% training overhead.

  1. Set up a standard transformer baseline utilizing full depth-wise attention residual blocks (AttnRes).
  2. Implement kernelized linear attention mechanisms, such as Generalized Linear Attention (GLA) or the Delta Rule, adapted for the depth dimension.
  3. Incorporate the linear depth-wise attention into a deep transformer model while tracking peak memory consumption.
  4. Benchmark the scaled architecture on standard language modeling tasks to compare performance and O(Ld) caching overhead against quadratic baselines.

Code linked to this paper on Hugging Face: github.com/MoonshotAI/Attention-Residuals

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Works like SoLA-Vision: Fine-grained Layer-wise Linear Softmax Hybrid Attention and SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation explore layer-wise linear softmax hybrid attention and attention residuals, but do not specifically solve the depth-wise full attention residual caching overhead with kernelized linear attention along the depth dimension. They leave open the direct combination of linear attention mechanics applied to the depth-wise residual stream.

We searched OpenAlex for “A method that replaces depth-wise softmax attention with linear-complexity kernelized attention along the depth dimension, allowing networks to scale residual connections efficiently without quadratic memory overhead.” and “Linear-Complexity Depth-Wise Attention for Scalable Full Attention Residuals” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Attention Residuals

Kimi Team, Guangyu Chen, Yu Zhang +34 more · 2026

This paper introduces Attention Residuals (AttnRes) to address the limitations of standard residual connections, which uniformly aggregate all prior layer outputs and cause uncontrolled hidden-state growth. The authors propose replacing fixed accumulation with learned, depth-wise softmax attention, allowing layers to selectively retrieve information from earlier layers to improve downstream performance.

Related Ideas

Is this a good idea?

Share onXLinkedIn