Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationRelated work found

Retrieval-Augmented Cross-Kernel Experience Transfer to Mitigate Agent Harness Sensitivity

Replace fixed heuristic lesson imports with computation-matched experience retrieval so optimization insights transfer reliably across kernels and reduce sensitivity to agent scaffolding.

Time

several weeks

Compute level

Multiple GPUs

Resources

Ascend 910C worker pool and LLM API budget (~50M tokens per harness ablation across Batch 1) consistent with the paper's Default evaluation setup

What the paper used

Ascend 910C NPUs (1 NPU per evaluation request, CANN 9.0.1, msprof 26.0.0), 17 Batch 1 tasks (from 48 total), up to 80 candidate evaluations per task with 10M/20M/50M token budgets (up to 140.4M mean tokens and 11.1 hours per task observed), across 7 LLMs and 4 harnesses.

Prerequisites

LLM agent architectures, retrieval-augmented memory systems, and controlled experimental design

Scope

modify existing code and implement retrieval memory module

The Gap

Section 4.2.1 reveals high sensitivity to agent scaffolding even when the model and 100% task pass rate are identical: GPT-5.6 achieves 1.69x GM speedup with Codex versus 1.05x with Claude Code. Meanwhile, in Section 4.2.2 and Appendix B, the Default harness's warm mode imports five static lessons—drawing from other kernels for CANN tasks while reusing same-kernel cold-run lessons for fused_gdn_gating, hc_post, and hc_pre_sinkhorn—without adaptive retrieval or negative-lesson filtering.

“GPT-5.6 passes all tasks with each of the three harnesses, yet its GM speedup differs substantially: approximately 1.69× with Codex versus 1.05× with Claude Code (Figures 7 and 5).”
From the paper, Section 4.2.1, page 11. Matched against the PDF text.

Limitation

Optimization outcomes are strongly confounded by undocumented harness scaffolding differences, and the built-in experience mechanism relies on static five-lesson injection that mixes cross-kernel transfer with same-kernel memorization.

Proposed Solution

Decompose harness behaviors (context truncation, diff/edit tools, rollback rules, and reflection frequency) via controlled ablations inside AuraKernel, and implement a retrieval-augmented experience bank indexed by kernel computation graph features (operator class, tensor shapes, memory/compute intensity) and failure modes. Filter stored lessons by cross-kernel generalization lift, enforcing strict leave-one-kernel-out cross-kernel transfer separate from same-kernel warm-starting.

Why This Matters

Standardizing retrieval-driven experience transfer would better isolate intrinsic model capability from harness artifacts and improve sample efficiency under tight token budgets.

How to Get Started

How to test it: Hypothesis: strict leave-one-kernel-out retrieval of computation-matched lessons improves 10M-token STS over both cold and fixed 5-lesson warm baselines without same-kernel leakage. Baseline: Default cold and strict cross-kernel Default warm with GPT-5.6 Luna on Batch 1; metric: STS at 10M tokens and cross-harness GM speedup variance; stop if dynamic retrieval fails to beat fixed 5-lesson cross-kernel transfer on at least 60% of Batch 1 tasks.

  1. Audit the Default (AuraKernel) warm configuration in AscendGYM to separate strict cross-kernel lessons from the three tasks using same-kernel cold lessons (fused_gdn_gating, hc_post, hc_pre_sinkhorn).
  2. Analyze saved search traces from Codex/GPT-5.6 and Claude Code/GPT-5.6 on Batch 1 to pinpoint which harness mechanisms (context management, tool-call granularity, rollback policy) drive the 1.69x vs. 1.05x gap.
  3. Build a structured experience store indexing validated optimization patterns and anti-patterns by operator category, shape regime, and bottleneck type using metadata + embedding retrieval.
  4. Benchmark leave-one-kernel-out retrieved lessons against cold and fixed-warm Default configurations at 10M, 20M, and 50M token budgets.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Studies like Code as Agent Harness and Meta-Harness focus on optimizing model harnesses and agent behaviors, while retrieval-augmented generation is widely studied in general contexts. The listed works leave open the specific decomposition of harness behaviors via controlled ablations paired with a retrieval-augmented experience bank indexed by computation graph features and failure modes for cross-kernel transfer.

We searched OpenAlex and arXiv for “coding agent harness sensitivity experience retrieval kernel optimization” and “Retrieval-Augmented Cross-Kernel Experience Transfer to Mitigate Agent Harness Sensitivity” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs

AI Training Platform Team, Bingwei Zhang, Jing Li +24 more · 2026

This paper introduces AscendGYM, a benchmark and execution sandbox (AscendForge) for evaluating how effectively LLM coding agents optimize existing Ascend NPU kernel implementations from real-world LLM training and inference workloads. Unlike prior kernel-generation benchmarks, AscendGYM measures optimization gains relative to production baselines alongside token cost using a new Speedup-Token Score (STS) metric.

Related Ideas

Is this a good idea?

Share onXLinkedIn