Adversarial Specification Probing and Multi-Seed Contrast Verification for Verifier-Free Harness Edits
Cross-check proposed agent edits with specification-derived adversarial probes and multi-seed reruns to prevent shared model assumptions from passing self-written verification scripts.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
In Section 2.2.1, Appendix E.6 (Figure 14), and Appendix F, candidate harnesses are tested via a single probe rollout per candidate, and behavioral contrasts plus self-written verification scripts (e.g., validator cache and header injection cases on WB-code) guide composition without external ground truth.
“These signals help assess the effects of a proposed change, but remain sensitive to execution variability and shared model biases.”
Limitation
Single-rollout probes conflate execution stochasticity with genuine edit effects, and when an editor introduces a faulty domain assumption or writes a matching self-checker script, the flawed premise propagates across composed candidates and passes self-checks while failing hidden tests.
Proposed Solution
Introduce decoupled falsification probing into the Probe and Compose stages: (1) generate independent specification-derived edge-case micro-tests (e.g., case-sensitivity, structural equivalence, and boundary inputs extracted solely from task.md and source schemas without seeing the candidate's scripts) to stress-test candidate code/scripts; (2) when an editor proposes a hard prescriptive guardrail or domain rule that contradicts any base rollout's behavior, require a paired counterfactual probe (running with vs. without the restrictive rule on matched seeds) before composition; and (3) allocate targeted multi-seed replication (e.g., 2-3 seeds) specifically to edits showing high behavioral variance rather than uniform single-rollout probing.
Why This Matters
Decoupling verification from editor assumptions and controlling for rollout variance should lower regression rates caused by restrictive priors and unverified self-checkers.
How to Get Started
How to test it: Test whether specification-derived falsification tests and targeted counterfactual re-probes reduce negative-delta regressions (tasks scoring below base harness) on WorkBuddy-Bench Code and LAB without exceeding the k=5 token budget. Compare regression frequency and mean score against standard parallel k=5 Harness^2; stop if overhead from extra probes outweighs regression reductions.
- Identify all tasks across WorkBuddy-Bench and LAB where Harness^2 underperforms the base harness and categorize whether failures stem from shared restrictive rules or rollout variance.
- Implement a separate specification-tester prompt that reads only task.md and source schemas to output executable edge-case assertions for scripts and deliverables.
- Modify the Compose stage to flag restrictive rules that forbid behaviors seen in base rollouts and trigger a single matched counterfactual check before merging.
- Evaluate regression rate and net benchmark score on WorkBuddy-Bench and LAB under matched total token budgets.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
While [0] and [8] discuss adversarial correctness and verified detection of anomalies in agents, and [9] provides foundational work on natural-language harnesses, the specific mechanism of using decoupled specification-derived micro-tests to perform counterfactual verification of harness edits is novel relative to these results.
- Adversarial Agent Collaboration for Correctness Improvements of C to Safe Rust Translation · 2025 · 12 citations
- Verified Detection and Prevention of Concurrency Anomalies in Multi-Agent Large Language Model Systems · 2026 · 11 citations
- Natural-Language Agent Harnesses · 2026 · 88 citations
We searched OpenAlex and arXiv for “verifier-free agent self-improvement adversarial probing shared model bias” and “Adversarial Specification Probing and Multi-Seed Contrast Verification for Verifier-Free Harness Edits” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Harness^2: Recursive Agent Harnessing for an Open WorldRuiyao Xu, Yanfei Chen, Zhongying CuiZhu +6 more · 2026
This paper introduces Harness^2, a plug-and-play recursive agent harnessing framework that enables LLM agents to optimize their execution harnesses at test time in open-world settings without ground-truth verifiers. It couples task-level contrastive refinement (Propose-Probe-Compose) with harnessing-level recursion that distills reusable harness components and lessons into an evolving improvement library across tasks.
Related Ideas
- Simulation-Based Ground Truth ValidationRemember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution🔴Hard
- Executable Programmatic Verifiers for Robust Micro-Skill Lifecycle ManagementTokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents🟡Fairly easy
- Artifact-to-Claim Cross-Verification and Process Consistency Rewards for Agentic DeliverablesSage S2.1 Flash Technical Report🟡Fairly easy
- Closed-Loop Adaptive Red-Teaming and Adversarially Trained Span-Veto SanitizationSPILLWAY: Separating Target Admission from Sanitization to Defend Tool-Using Agents against Prompt Injection🔴Hard
- Hierarchical Specification-to-Patch Synthesis for Large-Scale Repository ChangesSWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations🟠Medium