Execution-Time and Hardware-Cost-Aware Scoring Beyond Token Budgets in Agentic Kernel Optimization
Incorporate compilation, profiling, and hardware execution wall-clock time into the optimization score so agents are rewarded for fast turnaround rather than low token usage alone.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Section 3.3.1 defines the primary Speedup-Token Score (STS) solely in terms of cumulative LLM token consumption (B_k / B_max), ignoring wall-clock latency, interaction rounds, and NPU compilation or profiling time. In Section E.2, Qwen-35B achieves nearly identical GM speedup (1.06x with OpenCode vs. 1.05x with Codex) and comparable token consumption, yet requires 11.1 hours per task with OpenCode compared to 2.2 hours with Codex.
“Qwen-35B achieves GM speedups of approximately 1.06× with OpenCode and 1.05× with Codex, with comparable mean token consumption, yet takes 11.1 and 2.2 hours per task, respectively (Figures 5, 10, and 12).”
Limitation
A token-only joint metric conceals fivefold differences in wall-clock turnaround and NPU occupancy, causing slow, compilation-heavy agent trajectories to score identically to fast, cluster-efficient trajectories.
Proposed Solution
Formulate a multi-resource Speedup-Resource Score (SRS) that augments or replaces B_k / B_max in STS with a normalized composite cost combining API token cost (weighted by model pricing), cumulative NPU sandbox wall-clock time (host/device build plus msprof execution), and failed-build penalties. Expose real-time remaining compute and time budgets in the AscendForge status payload so agents can dynamically balance heavy compile-profile loops against static analysis.
Why This Matters
Adopting a hardware- and wall-clock-aware metric would align agent optimization incentives with real cluster economics and operational deployment windows.
How to Get Started
How to test it: Hypothesis: augmenting STS with normalized NPU wall-clock time and API dollar cost shifts model-harness rankings and improves Pareto efficiency when agents receive budget-remaining feedback. Baseline: standard token-only STS on Batch 1 trajectories; metric: Pareto frontier hypervolume of GM speedup versus total compute dollar cost and wall-clock hours; stop if time-aware feedback reduces wall-clock hours by less than 15% without preserving speedup.
- Parse the released AscendGYM Batch 1 run records (token consumption, interaction rounds, build durations, and wall-clock timestamps) across all 18 model-harness combinations.
- Define a composite resource cost model parameterized by Ascend 910C node-hour rates and LLM API token prices, and re-score all existing trajectories to quantify rank inversions relative to STS.
- Modify the AscendForge status interface and shared evaluation prompt to report remaining wall-clock and NPU-execution budgets after each submission.
- Run controlled evaluations with Claude Code/Opus 5 and OpenCode/Qwen-35B under fixed wall-clock caps (e.g., 2 hours per task) to test whether budget awareness cuts wasted compile cycles.
Suggested by AI. Adapt it to your setup.
Has This Been Done?
While papers such as KernelEvolve and BudgetThinker discuss budget-aware reasoning or agentic kernel coding for heterogeneous accelerators, none of the listed results propose a multi-resource Speedup-Resource Score (SRS) combining API token costs, NPU wall-clock build/execution time, and failed-build penalties within an AscendForge framework.
- BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens · 2026 · 47 citations
- KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta · 2026 · 64 citations
- Hawk: Harnessing Hardware-Aware Knowledge for High-Performance NPU Kernel Generation · 2026 · 2 citations
We searched OpenAlex and arXiv for “cost-aware benchmark metric wall-clock time token budget coding agents” and “Execution-Time and Hardware-Cost-Aware Scoring Beyond Token Budgets in Agentic Kernel Optimization” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUsAI Training Platform Team, Bingwei Zhang, Jing Li +24 more · 2026
This paper introduces AscendGYM, a benchmark and execution sandbox (AscendForge) for evaluating how effectively LLM coding agents optimize existing Ascend NPU kernel implementations from real-world LLM training and inference workloads. Unlike prior kernel-generation benchmarks, AscendGYM measures optimization gains relative to production baselines alongside token cost using a new Speedup-Token Score (STS) metric.
Related Ideas
- Full-Trajectory Deployment Cost and Retry-Overhead Profiling for Medical AI AgentsInclusionMed: Benchmarking AI for Inclusive Healthcare🟡Fairly easy
- Empirical Production Serving Benchmarking of MLA-DSA Attention with IndexShareTeleChat to Xing4: Advancing Frontier Agentic Models on Domestic Chips and Frameworks🔴Hard
- Concurrency- and Confidence-Adaptive Block-Size Scheduling for High-Batch VLM ServingMMFlash: Faster Speculative Decoding for Vision-Language Models🟡Fairly easy
- Hybrid Kernel Scheduling for Prefill OptimizationBitnet.cpp: Efficient Edge Inference for Ternary LLMs🟡Fairly easy
- Token-Normalized and Convergence-Controlled Evaluation of VLM Scaling Laws in RoboticsScaleVLA: Scaling Vision-Language-Action Models to 552 Billion Parameters🟡Fairly easy