Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Efficient Input-Dependent Queries for Attention Residuals

A method that introduces low-rank projections and speculative decoding to input-dependent queries in attention residuals, reducing parameter cost and eliminating sequential decoding bottlenecks.

Time

2-3 weeks on 64-128 H100-class GPUs (1-2 weeks for training ablations and 1 week for batched lookahead inference profiling).

Compute level

Large cluster

Resources

Consistent with the paper's Section 5.3 ablations (16-layer MoE, 436M active parameters, 87.9B tokens per run), sweeping rank and lookahead offset k across 4-5 runs requires ~6-8 PFLOP/s-days, or roughly 64-128 H100-class GPUs plus multi-GPU inference benchmarking.

What the paper used

Specific GPU hardware is not stated in the paper. It reports scaling-law runs across 5 MoE sizes (194M-528M activated parameters, 38.7B-119.0B tokens, 8192 context, up to ~5.6 PFLOP/s-days), a 25-point architecture sweep (~6.5e19 FLOPs, ~2.3e8 active parameters), 16-layer ablations (436M active parameters, 87.9B tokens), and main runs on Kimi Linear (48B total / 3B activated parameters, 54 layers, 1.4T tokens).

Prerequisites

basic ML knowledge, transformer architectures, inference optimization

Scope

modify existing code

The Gap

In Section 5.3's component ablation, the authors reveal that projecting the query from the current hidden state (input-dependent query) lowers validation loss to 1.731, outperforming the 1.737 baseline of Full AttnRes. However, they abandoned this because it introduces a d x d projection per layer and forces sequential memory access during autoregressive decoding.

“This further lowers loss to 1.731, but introduces a d × d projection per layer and requires sequential memory access during decoding, so we default to the learned query.”
From the paper, Section 5.3, page 12. Matched against the PDF text.

Limitation

The current pseudo-query is a static, learned parameter for each layer. It cannot dynamically determine what prior layer features to fetch based on the specific token's current representation.

Proposed Solution

Implement a low-rank (LoRA-style) projection for the input-dependent query to reduce the parameter cost. To solve the sequential decoding penalty, utilize speculative decoding or a lookahead projection mechanism where layer l's query is predicted from layer l-k's state, allowing batched query computation while retaining input-dependence.

Why This Matters

Unlocks an additional performance boost by making depth-wise retrieval dynamic to the token context, while preserving the inference speed necessary for production deployment.

How to Get Started

How to test it: Hypothesis: a low-rank query projection predicted from layer l-k enables batched Phase 1 inter-block reads while recovering most of the loss gain of full-rank input-dependent queries (1.731 vs. 1.737/1.746); compare against static Block AttnRes (1.746) and full input-dependent Full AttnRes (1.731) on the 16-layer MoE setup using validation loss, parameter overhead, and decoding latency. Stop if validation loss does not improve over static Block AttnRes (1.746) or if decoding latency overhead exceeds 5% (eroding Block AttnRes's <2% budget).

  1. Implement a low-rank (LoRA-style) projection mechanism for input-dependent queries within an existing attention residual codebase.
  2. Establish a baseline evaluating the validation loss and sequential decoding latency of the input-dependent query configuration.
  3. Integrate a lookahead prediction mechanism to estimate layer l's query from layer l-k's hidden state.
  4. Benchmark the modified model on autoregressive generation speed to ensure batched query computation improves inference throughput.

Code linked to this paper on Hugging Face: github.com/MoonshotAI/Attention-Residuals

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers such as ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models and SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation investigate recurrent states and speculative mechanisms in hybrid linear attention models. However, they do not specifically implement a low-rank projection for input-dependent queries paired with a lookahead prediction mechanism to bypass the sequential decoding penalty in attention residuals. They leave the exact query prediction formulation for attention residuals open.

We searched OpenAlex for “A method that introduces low-rank projections and speculative decoding to input-dependent queries in attention residuals, reducing parameter cost and eliminating sequential decoding bottlenecks.” and “Efficient Input-Dependent Queries for Attention Residuals” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Attention Residuals

Kimi Team, Guangyu Chen, Yu Zhang +34 more · 2026

This paper introduces Attention Residuals (AttnRes) to address the limitations of standard residual connections, which uniformly aggregate all prior layer outputs and cause uncontrolled hidden-state growth. The authors propose replacing fixed accumulation with learned, depth-wise softmax attention, allowing layers to selectively retrieve information from earlier layers to improve downstream performance.

Related Ideas

Is this a good idea?

Share onXLinkedIn