Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationNo close match found

Execution-Time and Hardware-Cost-Aware Scoring Beyond Token Budgets in Agentic Kernel Optimization

Incorporate compilation, profiling, and hardware execution wall-clock time into the optimization score so agents are rewarded for fast turnaround rather than low token usage alone.

Time

several weeks

Compute level

Single GPU or less

Resources

Existing AscendGYM optimization trajectory logs plus a single Ascend 910C node for time-capped verification runs

What the paper used

Ascend 910C NPUs (1 NPU per evaluation request, CANN 9.0.1, msprof 26.0.0), 17 Batch 1 tasks (from 48 total), up to 80 candidate evaluations per task with 10M/20M/50M token budgets (up to 140.4M mean tokens and 11.1 hours per task observed), across 7 LLMs and 4 harnesses.

Prerequisites

benchmark metric design, systems profiling, and agent harness orchestration

Scope

modify existing code

The Gap

Section 3.3.1 defines the primary Speedup-Token Score (STS) solely in terms of cumulative LLM token consumption (B_k / B_max), ignoring wall-clock latency, interaction rounds, and NPU compilation or profiling time. In Section E.2, Qwen-35B achieves nearly identical GM speedup (1.06x with OpenCode vs. 1.05x with Codex) and comparable token consumption, yet requires 11.1 hours per task with OpenCode compared to 2.2 hours with Codex.

“Qwen-35B achieves GM speedups of approximately 1.06× with OpenCode and 1.05× with Codex, with comparable mean token consumption, yet takes 11.1 and 2.2 hours per task, respectively (Figures 5, 10, and 12).”
From the paper, Section E.2, page 19. Matched against the PDF text.

Limitation

A token-only joint metric conceals fivefold differences in wall-clock turnaround and NPU occupancy, causing slow, compilation-heavy agent trajectories to score identically to fast, cluster-efficient trajectories.

Proposed Solution

Formulate a multi-resource Speedup-Resource Score (SRS) that augments or replaces B_k / B_max in STS with a normalized composite cost combining API token cost (weighted by model pricing), cumulative NPU sandbox wall-clock time (host/device build plus msprof execution), and failed-build penalties. Expose real-time remaining compute and time budgets in the AscendForge status payload so agents can dynamically balance heavy compile-profile loops against static analysis.

Why This Matters

Adopting a hardware- and wall-clock-aware metric would align agent optimization incentives with real cluster economics and operational deployment windows.

How to Get Started

How to test it: Hypothesis: augmenting STS with normalized NPU wall-clock time and API dollar cost shifts model-harness rankings and improves Pareto efficiency when agents receive budget-remaining feedback. Baseline: standard token-only STS on Batch 1 trajectories; metric: Pareto frontier hypervolume of GM speedup versus total compute dollar cost and wall-clock hours; stop if time-aware feedback reduces wall-clock hours by less than 15% without preserving speedup.

  1. Parse the released AscendGYM Batch 1 run records (token consumption, interaction rounds, build durations, and wall-clock timestamps) across all 18 model-harness combinations.
  2. Define a composite resource cost model parameterized by Ascend 910C node-hour rates and LLM API token prices, and re-score all existing trajectories to quantify rank inversions relative to STS.
  3. Modify the AscendForge status interface and shared evaluation prompt to report remaining wall-clock and NPU-execution budgets after each submission.
  4. Run controlled evaluations with Claude Code/Opus 5 and OpenCode/Qwen-35B under fixed wall-clock caps (e.g., 2 hours per task) to test whether budget awareness cuts wasted compile cycles.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

No close match in 10 results

While papers such as KernelEvolve and BudgetThinker discuss budget-aware reasoning or agentic kernel coding for heterogeneous accelerators, none of the listed results propose a multi-resource Speedup-Resource Score (SRS) combining API token costs, NPU wall-clock build/execution time, and failed-build penalties within an AscendForge framework.

We searched OpenAlex and arXiv for “cost-aware benchmark metric wall-clock time token budget coding agents” and “Execution-Time and Hardware-Cost-Aware Scoring Beyond Token Budgets in Agentic Kernel Optimization” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs

AI Training Platform Team, Bingwei Zhang, Jing Li +24 more · 2026

This paper introduces AscendGYM, a benchmark and execution sandbox (AscendForge) for evaluating how effectively LLM coding agents optimize existing Ascend NPU kernel implementations from real-world LLM training and inference workloads. Unlike prior kernel-generation benchmarks, AscendGYM measures optimization gains relative to production baselines alongside token cost using a new Speedup-Token Score (STS) metric.

Related Ideas

Is this a good idea?

Share onXLinkedIn