Retrieval-Augmented Cross-Kernel Experience Transfer to Mitigate Agent Harness Sensitivity
Replace fixed heuristic lesson imports with computation-matched experience retrieval so optimization insights transfer reliably across kernels and reduce sensitivity to agent scaffolding.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 4.2.1 reveals high sensitivity to agent scaffolding even when the model and 100% task pass rate are identical: GPT-5.6 achieves 1.69x GM speedup with Codex versus 1.05x with Claude Code. Meanwhile, in Section 4.2.2 and Appendix B, the Default harness's warm mode imports five static lessons—drawing from other kernels for CANN tasks while reusing same-kernel cold-run lessons for fused_gdn_gating, hc_post, and hc_pre_sinkhorn—without adaptive retrieval or negative-lesson filtering.
“GPT-5.6 passes all tasks with each of the three harnesses, yet its GM speedup differs substantially: approximately 1.69× with Codex versus 1.05× with Claude Code (Figures 7 and 5).”
Limitation
Optimization outcomes are strongly confounded by undocumented harness scaffolding differences, and the built-in experience mechanism relies on static five-lesson injection that mixes cross-kernel transfer with same-kernel memorization.
Proposed Solution
Decompose harness behaviors (context truncation, diff/edit tools, rollback rules, and reflection frequency) via controlled ablations inside AuraKernel, and implement a retrieval-augmented experience bank indexed by kernel computation graph features (operator class, tensor shapes, memory/compute intensity) and failure modes. Filter stored lessons by cross-kernel generalization lift, enforcing strict leave-one-kernel-out cross-kernel transfer separate from same-kernel warm-starting.
Why This Matters
Standardizing retrieval-driven experience transfer would better isolate intrinsic model capability from harness artifacts and improve sample efficiency under tight token budgets.
How to Get Started
How to test it: Hypothesis: strict leave-one-kernel-out retrieval of computation-matched lessons improves 10M-token STS over both cold and fixed 5-lesson warm baselines without same-kernel leakage. Baseline: Default cold and strict cross-kernel Default warm with GPT-5.6 Luna on Batch 1; metric: STS at 10M tokens and cross-harness GM speedup variance; stop if dynamic retrieval fails to beat fixed 5-lesson cross-kernel transfer on at least 60% of Batch 1 tasks.
- Audit the Default (AuraKernel) warm configuration in AscendGYM to separate strict cross-kernel lessons from the three tasks using same-kernel cold lessons (fused_gdn_gating, hc_post, hc_pre_sinkhorn).
- Analyze saved search traces from Codex/GPT-5.6 and Claude Code/GPT-5.6 on Batch 1 to pinpoint which harness mechanisms (context management, tool-call granularity, rollback policy) drive the 1.69x vs. 1.05x gap.
- Build a structured experience store indexing validated optimization patterns and anti-patterns by operator category, shape regime, and bottleneck type using metadata + embedding retrieval.
- Benchmark leave-one-kernel-out retrieved lessons against cold and fixed-warm Default configurations at 10M, 20M, and 50M token budgets.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Studies like Code as Agent Harness and Meta-Harness focus on optimizing model harnesses and agent behaviors, while retrieval-augmented generation is widely studied in general contexts. The listed works leave open the specific decomposition of harness behaviors via controlled ablations paired with a retrieval-augmented experience bank indexed by computation graph features and failure modes for cross-kernel transfer.
- Code as Agent Harness · 2026 · 107 citations
- Meta-Harness: End-to-End Optimization of Model Harnesses · 2026 · 433 citations
- Evolving Agents in the Dark: Retrospective Harness Optimization via Self-Preference · 2026 · 40 citations
We searched OpenAlex and arXiv for “coding agent harness sensitivity experience retrieval kernel optimization” and “Retrieval-Augmented Cross-Kernel Experience Transfer to Mitigate Agent Harness Sensitivity” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUsAI Training Platform Team, Bingwei Zhang, Jing Li +24 more · 2026
This paper introduces AscendGYM, a benchmark and execution sandbox (AscendForge) for evaluating how effectively LLM coding agents optimize existing Ascend NPU kernel implementations from real-world LLM training and inference workloads. Unlike prior kernel-generation benchmarks, AscendGYM measures optimization gains relative to production baselines alongside token cost using a new Speedup-Token Score (STS) metric.
Related Ideas
- Sample-Efficient Online Policy Optimization for Rapid Agent AdaptationMetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wild🟠Medium
- Parameter-Efficient Action Adaptation and Multi-Layer Feature Readout for Frontier VLMsScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters🔴Hard
- Utility-Tracked Consolidation and Failure-Aware Retrieval for Harness Improvement LibrariesHarness^2: Recursive Agent Harnessing for an Open World🟡Fairly easy
- Closed-Loop Adaptive Red-Teaming and Adversarially Trained Span-Veto SanitizationSPILLWAY: Separating Target Admission from Sanitization to Defend Tool-Using Agents against Prompt Injection🔴Hard
- End-to-End Trainable Sparse Attention Indexers for RLGLM-5: from Vibe Coding to Agentic Engineering🔴Hard