Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationClosely matching work found

Closed-Loop Evidence-Gated Handoff Verification for Coding Agents

Require coding agents to execute automated self-generated unit test contracts verifying each acquired requirement before submitting final handoff reports to eliminate ungrounded completion claims.

Time

several weeks

Compute level

Single GPU or less

Resources

Standard compute environment running SWE-CoWork Docker containers and LLM API credits.

What the paper used

Runs capped at 8 hours wall-clock; Claude-Opus-5.5 completed runs in a median of 5.0 hours with 49 premature stops having 2.8 hours median remaining budget.

Prerequisites

software testing, test synthesis, agentic workflows

Scope

modify existing code

The Gap

Section 6.3.4 and Figure 8a evaluate the correspondence between the agent's final handoff claims and the actual private executable test outcomes across 2,399 requirements.

“Among 826 requirements reported as completed at handoff, 255 (31%) fail their corresponding tests (Figure 8a).”
From the paper, Section 6.3.4, page 19. Matched against the PDF text.

Limitation

Agents exhibit poor calibration and high handoff hallucination: 31% of requirements reported as completed fail their private test nodes, and agents terminate early with actionable work remaining in 49 of 101 tasks.

Proposed Solution

Implement a mandatory self-verification protocol where the agent must formally translate every confirmed requirement into an executable reproduction script or test case in the repository before writing code. Prior to invoking the terminal handoff tool, a meta-evaluator tool parses the handoff markdown report, extracts each claim of completion, executes the associated test script, and blocks submission if any claimed feature fails its assertion.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Preventing ungrounded handoffs would bridge the gap between claimed and verified completion, reducing false-positive delivery notes and forcing agents to utilize remaining budget on failed requirements.

How to Get Started

How to test it: Test whether evidence-gated verification reduces the rate of failing 'done' claims from 31% to under 10% on SWE-CoWork while increasing F2P pass rates, stopping if test synthesis introduces regressions in preserved pass-to-pass (P2P) tests.

  1. Extract the requirement disclosure logs from SWE-CoWork runs to identify the syntax of requirements disclosed by colleagues.
  2. Build a prompt module instructing the agent to author a dedicated test node in a separate sandbox file immediately upon receiving a requirement disclosure.
  3. Construct an automated handoff validator tool that checks execution logs of newly authored tests before writing the delivery note.
  4. Benchmark the modified scaffold on tasks from Kimai, PostCSS, and Stencil to inspect failure attribution shifts.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Closely matching work found

The proposed mechanism of evidence-gated verification and executable test-based handoff is explicitly covered by 'Loop Engineering: Version-Bound, Evidence-Gated Coordination for Production Coding Agents' and 'Proof-or-Stop: Don't Trust the Agent, Trust the Evidence'. These papers propose exactly the workflow of requiring executable proof before terminal handoff to prevent silent failures.

We searched OpenAlex and arXiv for “automated test synthesis agent self verification calibration” and “Closed-Loop Evidence-Gated Handoff Verification for Coding Agents” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

Xinping Lei, Minghao Liu, Ruijie Zhu +23 more · 2026

The paper introduces SWE-CoWork, a benchmark evaluating software engineering agents as digital employees within simulated organizations of 26 rule-governed colleagues across 101 repositories, 11 languages, and 12 domains. Agents must inspect evolving records, converse with colleagues to recover distributed requirements, implement complex changes, and deliver a verified handoff. The study reveals a critical gap between partial code generation (~63% fail-to-pass tests) and end-to-end task success (<8%), primarily caused by breakdowns in long-horizon requirement tracking, abandoned clarifications, and hallucinated handoff completion.

Related Ideas

Is this a good idea?

Share onXLinkedIn