Closed-Loop Evidence-Gated Handoff Verification for Coding Agents
Require coding agents to execute automated self-generated unit test contracts verifying each acquired requirement before submitting final handoff reports to eliminate ungrounded completion claims.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 6.3.4 and Figure 8a evaluate the correspondence between the agent's final handoff claims and the actual private executable test outcomes across 2,399 requirements.
“Among 826 requirements reported as completed at handoff, 255 (31%) fail their corresponding tests (Figure 8a).”
Limitation
Agents exhibit poor calibration and high handoff hallucination: 31% of requirements reported as completed fail their private test nodes, and agents terminate early with actionable work remaining in 49 of 101 tasks.
Proposed Solution
Implement a mandatory self-verification protocol where the agent must formally translate every confirmed requirement into an executable reproduction script or test case in the repository before writing code. Prior to invoking the terminal handoff tool, a meta-evaluator tool parses the handoff markdown report, extracts each claim of completion, executes the associated test script, and blocks submission if any claimed feature fails its assertion.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Preventing ungrounded handoffs would bridge the gap between claimed and verified completion, reducing false-positive delivery notes and forcing agents to utilize remaining budget on failed requirements.
How to Get Started
How to test it: Test whether evidence-gated verification reduces the rate of failing 'done' claims from 31% to under 10% on SWE-CoWork while increasing F2P pass rates, stopping if test synthesis introduces regressions in preserved pass-to-pass (P2P) tests.
- Extract the requirement disclosure logs from SWE-CoWork runs to identify the syntax of requirements disclosed by colleagues.
- Build a prompt module instructing the agent to author a dedicated test node in a separate sandbox file immediately upon receiving a requirement disclosure.
- Construct an automated handoff validator tool that checks execution logs of newly authored tests before writing the delivery note.
- Benchmark the modified scaffold on tasks from Kimai, PostCSS, and Stencil to inspect failure attribution shifts.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
The proposed mechanism of evidence-gated verification and executable test-based handoff is explicitly covered by 'Loop Engineering: Version-Bound, Evidence-Gated Coordination for Production Coding Agents' and 'Proof-or-Stop: Don't Trust the Agent, Trust the Evidence'. These papers propose exactly the workflow of requiring executable proof before terminal handoff to prevent silent failures.
- Loop Engineering: Version-Bound, Evidence-Gated Coordination for Production Coding Agents · 2026 · 0 citations
- Proof-or-Stop: Don't Trust the Agent, Trust the Evidence -- Loop Engineering for Verifiable Evidence-Gated Lifecycle Control · 2026 · 12 citations
We searched OpenAlex and arXiv for “automated test synthesis agent self verification calibration” and “Closed-Loop Evidence-Gated Handoff Verification for Coding Agents” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
SWE-CoWork: Benchmarking Coding Agents as Digital Employees in OrganizationsXinping Lei, Minghao Liu, Ruijie Zhu +23 more · 2026
The paper introduces SWE-CoWork, a benchmark evaluating software engineering agents as digital employees within simulated organizations of 26 rule-governed colleagues across 101 repositories, 11 languages, and 12 domains. Agents must inspect evolving records, converse with colleagues to recover distributed requirements, implement complex changes, and deliver a verified handoff. The study reveals a critical gap between partial code generation (~63% fail-to-pass tests) and end-to-end task success (<8%), primarily caused by breakdowns in long-horizon requirement tracking, abandoned clarifications, and hallucinated handoff completion.
Related Ideas
- Requirement-Coverage and Execution-Trace-Aware Candidate Deliverable SelectionHarness^2: Recursive Agent Harnessing for an Open World🟢Easy
- Adversarial Specification Probing and Multi-Seed Contrast Verification for Verifier-Free Harness EditsHarness^2: Recursive Agent Harnessing for an Open World🟠Medium
- Human-in-the-Loop Protocol for Critical Point HandoffsFara-7B: An Efficient Agentic Model for Computer Use🟡Fairly easy
- Executable Programmatic Verifiers for Robust Micro-Skill Lifecycle ManagementTokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents🟡Fairly easy
- Artifact-to-Claim Cross-Verification and Process Consistency Rewards for Agentic DeliverablesSage S2.1 Flash Technical Report🟡Fairly easy