Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationRelated work found

Difficulty- and Saturation-Aware Adaptive Task-Level Recursion Budgeting

Dynamically allocate test-time exploration budgets using early rollout divergence and early-stopping checks so agents avoid wasting inference tokens on easy or saturated tasks.

Time

several weeks

Compute level

Single GPU or less

Resources

Workstation with Docker container runtime, API access to Gemini 3.5 Flash/GLM-5 and Claude Opus 4.8 for multi-budget rollouts, and WorkBuddy-Bench plus JobBench benchmarks.

What the paper used

Frozen API executors (Gemini 3.5 Flash, GLM-5) and editor/selector (Claude Opus 4.8 at high reasoning effort) run inside OpenCode/Codex Docker containers (200-turn and 3,600s rollout caps), spending roughly 1-100M tokens per task across WorkBuddy-Bench (100 test tasks), JobBench (34 test tasks), and LAB (100 test tasks) over 3 runs.

Prerequisites

Test-time compute scaling, LLM agent evaluation, and adaptive stopping heuristics.

Scope

modify existing code

The Gap

In Section 2.2.2 and Section 4 (Figure 6), task-level recursion fixes both the recursion structure (parallel or sequential) and the exploration budget k (e.g., k=1 or k=5) in advance for every incoming task in the stream.

“Future work could allocate exploration according to task difficulty and the relevance of prior experience, or stop when additional probes yield little new evidence.”
From the paper, Appendix G, page 29. Matched against the PDF text.

Limitation

Applying a uniform fixed budget k across all tasks incurs substantial inference cost (up to tens of millions of tokens per task on JobBench) on straightforward tasks where the base harness already succeeds, while failing to adaptively switch between parallel width and sequential depth when probes stop yielding new evidence.

Proposed Solution

Replace the static budget k with a two-stage adaptive controller: (1) execute a small initial pilot of k_0=2 base rollouts plus a lightweight static/trajectory diagnostic (checking tool-error rates, empty deliverables, and inter-rollout behavioral divergence) alongside library retrieval confidence; (2) bypass refinement entirely (returning the verified base output) when pilot rollouts agree cleanly with zero tool or structural errors and high confidence; (3) otherwise trigger iterative contrastive steps that compute a marginal information-gain score across consecutive probe diffs and trajectory contrasts, terminating recursion early when candidate behavioral contrasts converge or switching to sequential refinement if initial parallel probes expose multi-step procedural dependencies.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

This is expected to improve the compute-performance Pareto frontier by reducing average token consumption per task while reserving deeper exploration budgets for genuinely difficult or novel tasks.

How to Get Started

How to test it: Test the hypothesis that adaptive pilot gating and contrast-saturation early stopping match or beat fixed k=5 Harness^2 scores at significantly lower average token cost per task on JobBench and WorkBuddy-Bench. Plot the Pareto frontier against fixed k=1 and k=5 Harness^2 using task score and total tokens per task; stop if adaptive budgeting degrades mean score by >1.5 points relative to fixed k=5 at comparable compute.

  1. Analyze per-task trajectories from k=1 and k=5 runs in github.com/google-research/harness2 on WorkBuddy-Bench and JobBench to quantify how often base runs already match k=5 performance.
  2. Design a lightweight divergence and error-signal classifier over k_0=2 base trajectories (measuring tool-error counts, artifact completeness, and deliverable agreement) without seeing ground-truth scores.
  3. Implement early-exit gating and marginal-contrast stopping inside the parallel and sequential recursion loops.
  4. Benchmark the adaptive controller against fixed k=1, k=3, and k=5 Harness^2 on the compute-performance Pareto curve.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers [1], [2], and [3] investigate adaptive test-time compute and budget allocation for reasoning tasks. The proposed idea adds a specific two-stage pilot and information-gain-based termination logic that is not covered by these existing adaptive budgeting approaches.

We searched OpenAlex and arXiv for “adaptive test-time compute allocation early stopping agents” and “Difficulty- and Saturation-Aware Adaptive Task-Level Recursion Budgeting” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Harness^2: Recursive Agent Harnessing for an Open World

Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu +6 more · 2026

This paper introduces Harness^2, a plug-and-play recursive agent harnessing framework that enables LLM agents to optimize their execution harnesses at test time in open-world settings without ground-truth verifiers. It couples task-level contrastive refinement (Propose-Probe-Compose) with harnessing-level recursion that distills reusable harness components and lessons into an evolving improvement library across tasks.

Related Ideas

Is this a good idea?

Share onXLinkedIn