Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationRelated work found

Variable-Visit Training and Depth-Relative Query Routing for Test-Time Loop Scaling

Train parameter-shared transformers with randomized loop counts and depth-relative attention queries so extra inference iterations improve predictions instead of degrading accuracy.

Time

several weeks

Compute level

Multiple GPUs

Resources

One 8x80GB H100 GPU node, ClimbMix shards, and lm-evaluation-harness 0.4.12

What the paper used

ClimbMix pretraining at 20 tokens per parameter across 191M (3.82B tokens) to 3B (59.77B tokens) on one node with eight 80 GB GPUs (H100s for timing and 191M/374M/3B; A100s for 778M), bf16 FSDP.

Prerequisites

Looped transformers, test-time compute scaling, multi-depth training objectives

Scope

modify existing Loopback training loop, query parameterization, and evaluation script

The Gap

In Section 5.2.3 and Appendix H, Loopback is trained at a fixed loop visit count R (e.g., R=4 at 191M, R=2 at 374M-3B) with application-specific learned queries q_s. When extra visits are added at inference without retraining by reusing the last trained visit's queries, validation loss increases across all tested scales (Fig. 6a-c: +0.060 at 191M with 8 visits, +0.074 at 374M with 4 visits, +0.028 at 1.4B with 4 visits), and 3B zero-shot gains are inconsistent across tasks (Table 14: 5 of 13 tasks drop while RTE contributes 0.56 of the +0.72 point mean increase).

“Shared weights allow extra visits at inference, but excessive recurrence can degrade predictions, a problem known as overthinking (Bansal et al., 2022).”
From the paper, Section 5.2.3, page 11. Matched against the PDF text.

Limitation

Fixed-visit training and step-indexed queries prevent Loopback from generalizing reliably to additional loop iterations at test time, failing to turn extra compute into consistent validation-loss or downstream-task improvements.

Proposed Solution

Train with stochastic visit counts R sampled per batch or step from [R_min, R_max] (applying loss at the sampled exit or via depth-wise self-distillation) and replace fixed-index queries q_s with queries generated from a shared base embedding plus sinusoidal or normalized-depth embeddings of (r/R, k/K) conditioned on the current hidden state. Combine this with an inference-time early-exit or KL-convergence criterion so tokens stop looping before overthinking degrades representations.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Enabling monotonic or controllable quality improvements with extra test-time iterations would unlock genuine latent-reasoning compute scaling without needing separate checkpoints per depth.

How to Get Started

How to test it: Hypothesis: variable-visit training with depth-relative queries eliminates validation-loss degradation up to 2x the nominal visit count while matching fixed-visit loss at the base depth. Evaluate 191M (R in 2..8) and 374M (R in 1..4) models on ClimbMix validation loss and the 13-task lm-evaluation-harness suite against fixed-visit Loopback; stop if base-depth validation loss worsens by more than 0.005 nats or extra visits still monotonically increase loss.

  1. Reproduce the 191M [2L -> 4L (x4) -> 2L] and 374M [2L -> 10L (x2) -> 2L] Loopback baselines and Figure 6a-b extra-visit curves.
  2. Replace per-step queries q_s in the Loopback reader with a lightweight MLP mapping continuous depth features (r/R, k/K) to query vectors in R^d.
  3. Implement per-step uniform sampling of R during 20-TPP ClimbMix training, ensuring FSDP gradient synchronization across dynamic loop lengths.
  4. Evaluate trained checkpoints across R = 2 to 8 (191M) and R = 2 to 4 (374M) on ClimbMix validation loss and all 13 zero-shot benchmarks.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Scaling Test-Time Compute with Latent Reasoning [2] and LatentLoop [9] discuss test-time compute and variable depth, and some papers like [4] investigate truncated backpropagation. However, the specific combination of training with stochastic visit counts alongside a sinusoidal depth-relative query routing mechanism conditioned on hidden states is not explicitly detailed in these works.

We searched OpenAlex and arXiv for “looped transformer variable depth training test-time scaling overthinking” and “Variable-Visit Training and Depth-Relative Query Routing for Test-Time Loop Scaling” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Loopback Transformer: Towards a Unified View of Residual Looped Transformers

Xuan Ouyang, Zefan Cai, Chenlong Zhang +3 more · 2026

This paper introduces a depth-memory framework that unifies residual connections in looped Transformers with linear attention and recurrent memory via write, transport, and read operations. Building on this analysis, the authors propose the Loopback Transformer, which achieves lower validation loss and higher thirteen-task zero-shot accuracy than depth-matched standard Transformers across 191M to 3B scales while using 40-75% of their Transformer parameters.

Related Ideas

Is this a good idea?

Share onXLinkedIn