Requirement-Coverage and Execution-Trace-Aware Candidate Deliverable Selection
Augment final output selection with trajectory-derived behavioral invariants and requirement-coverage checklists to prevent selectors from picking incomplete or non-conforming deliverables.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 2.2.2 (Prompt 5) and Appendix E.6 (Figure 14), when k > 1, a terminal selector (Claude Opus 4.8) chooses among anonymized final deliverables solely by inspecting context/task.md, context/source/, and rendered artifacts, completely blinded to execution trajectories and harness metadata.
Limitation
Blindly judging only rendered deliverable text causes severe score regressions (such as dropping from 96 to 55 on WB-code, 100 to 70 on WB-web, and 54 to 26 on LAB) because the selector over-weights surface polish or precision over requirement coverage and misses execution-time failures like unexposed runtime state or failing checks.
Proposed Solution
Upgrade Prompt 5 into a two-phase structured selection pipeline: (1) decompose context/task.md into an explicit atomic requirement-coverage matrix (listing every requested entity, file, state transition, and format rule) before viewing candidates, scoring each candidate strictly on requirement coverage alongside factual precision; and (2) attach sanitized, harness-agnostic execution summaries (e.g., whether runtime/test commands exited cleanly, browser DOM/console check results, and tool error counts, while still masking harness diffs and candidate provenance) so the selector can filter out deliverables produced by broken executions without introducing harness-complexity bias.
Why This Matters
Reducing blind-selection regressions could recover substantial performance already achieved by candidate harnesses in the pool without requiring additional generation rollouts.
How to Get Started
How to test it: Test whether requirement-coverage decomposition and sanitized execution-summary signals reduce selection regret (difference between oracle best-of-pool score and selected score) on WorkBuddy-Bench and LAB at k=5. Compare against Prompt 5 on selection regret and overall benchmark score; stop if selection regret does not decrease or precision-sensitive tasks regress.
- Collect the k=5 candidate deliverable pools and ground-truth retrospective scores from WorkBuddy-Bench (Code, Web) and LAB runs using github.com/google-research/harness2.
- Measure baseline selection regret (oracle pool max minus Prompt 5 pick) to isolate selection failures from proposal failures.
- Implement a pre-selection task-decomposition step generating an explicit requirement checklist and append sanitized tool/exit-code summaries to candidate views.
- Re-run selection offline across stored pools first, then integrate into online k=5 runs on WorkBuddy-Bench and LAB.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Existing research in [7] and [9] touches on trace-aware reasoning and evidence tracking for agents, while [0] and [4] discuss LLM-based evaluation metrics. The proposal to combine a structured requirement-coverage matrix with harness-agnostic execution summaries specifically to minimize harness-complexity bias is not explicitly detailed in the listed works.
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods · 2024 · 895 citations
- From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents · 2026 · 76 citations
- Execution-State-Aware LLM Reasoning for Automated Proof-of-Vulnerability Generation · 2026 · 8 citations
We searched OpenAlex and arXiv for “LLM judge blind candidate selection requirement coverage execution trace” and “Requirement-Coverage and Execution-Trace-Aware Candidate Deliverable Selection” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Harness^2: Recursive Agent Harnessing for an Open WorldRuiyao Xu, Yanfei Chen, Zhongying CuiZhu +6 more · 2026
This paper introduces Harness^2, a plug-and-play recursive agent harnessing framework that enables LLM agents to optimize their execution harnesses at test time in open-world settings without ground-truth verifiers. It couples task-level contrastive refinement (Propose-Probe-Compose) with harnessing-level recursion that distills reusable harness components and lessons into an evolving improvement library across tasks.
Related Ideas
- Hierarchical Specification-to-Patch Synthesis for Large-Scale Repository ChangesSWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations🟠Medium
- Executable Programmatic Verifiers for Robust Micro-Skill Lifecycle ManagementTokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents🟡Fairly easy
- NFR-Aware Benchmarking for Agentic CodingAgent READMEs: An Empirical Study of Context Files for Agentic Coding🟡Fairly easy
- Hermetic Harness-Agent Environment Isolation Against Dependency-Mediated Answer LeakageSage S2.1 Flash Technical Report🟡Fairly easy
- Domain-Grounded Test-Time Compute Routing for Policy WorkflowsEnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings🟡Fairly easy