Skip to content

Source paper

Loopback Transformer: Towards a Unified View of Residual Looped Transformers

Xuan Ouyang, Zefan Cai, Chenlong Zhang and 3 more

Xuan Ouyang, Zefan Cai, Chenlong Zhang, Cheng Luo, Ke Wan, Junjie Hu

arXiv 2611.07226 · 202626 upvotes on Hugging Face

Research ideas from this paper (5)

Generated by AI. The paper’s authors did not write these.

Hybrid Sliding-Window and Fixed-State Depth Memory for Deeply Looped Transformers

Compress older layer updates into a fixed-size recurrent state while keeping recent updates explicit to cut memory traffic and latency in parameter-shared transformers.

Loopback Transformer: Towards a Unified View of Residual Looped Transformers
🔴Hard🔬Significant ResearchMultiple GPUsNo close match found#Efficiency#Training

Variable-Visit Training and Depth-Relative Query Routing for Test-Time Loop Scaling

Train parameter-shared transformers with randomized loop counts and depth-relative attention queries so extra inference iterations improve predictions instead of degrading accuracy.

Loopback Transformer: Towards a Unified View of Residual Looped Transformers
🟡Fairly easy💡Moderate InnovationMultiple GPUsRelated work found#Training#Reasoning

Token-Conditioned Input Injection and Adaptive Update Budgets in Loopback Residuals

Replace fixed input weights and static depth queries with lightweight token-conditioned gates so shared layers dynamically balance original inputs and new updates.

Loopback Transformer: Towards a Unified View of Residual Looped Transformers
🟠Medium💡Moderate InnovationMultiple GPUsNo close match found#Training#LLMs

Controlled Block-Level Comparison of Content-Gated Recurrent Depth Readers

Benchmark recurrent residual readers at block granularity with content-dependent gating and matched heads to isolate when fixed-size memory can rival explicit history attention.

Loopback Transformer: Towards a Unified View of Residual Looped Transformers
🟡Fairly easy💡Moderate InnovationMultiple GPUsRelated work found#Training#Efficiency

Multi-Corpus Pretraining and Post-Training Reasoning Evaluation of Loopback Transformers

Evaluate parameter-shared language models across multiple pretraining corpora and instruction-tuned reasoning tasks using execution and generation metrics rather than likelihood alone.

Loopback Transformer: Towards a Unified View of Residual Looped Transformers
🟠Medium💡Moderate InnovationLarge clusterNo close match found#LLMs#Reasoning

AI summary

This paper introduces a depth-memory framework that unifies residual connections in looped Transformers with linear attention and recurrent memory via write, transport, and read operations. Building on this analysis, the authors propose the Loopback Transformer, which achieves lower validation loss and higher thirteen-task zero-shot accuracy than depth-matched standard Transformers across 191M to 3B scales while using 40-75% of their Transformer parameters.

How it works (AI summary)

The model uses a prefix-middle-coda looped architecture that stores individual middle-block updates separately from the shared stack's input, keeping the input on a dedicated unit-coefficient residual path. At each application, geometric iteration-end decay (retention lambda = 1/4) defines a recency prior and bounds the total update weight, while a single-head softmax reader with learned per-step queries redistributes that fixed total weight across normalized stored updates using a fused Triton kernel.