Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Multi-Shape Generalization and End-to-End Workload Evaluation for NPU Kernel Optimization

Extend kernel optimization evaluation to dynamic tensor shapes, fused multistage operators, and end-to-end model throughput so agents cannot overfit single static input configurations.

Time

a couple of months

Compute level

Multiple GPUs

Resources

Multi-NPU Ascend 910C server pool, frontier LLM API budget (~100M tokens per model-harness across 48 tasks), and vLLM-Ascend/MindSpeed-LLM workload traces

What the paper used

Ascend 910C NPUs (1 NPU per evaluation request, CANN 9.0.1, msprof 26.0.0), 17 Batch 1 tasks (from 48 total), up to 80 candidate evaluations per task with 10M/20M/50M token budgets (up to 140.4M mean tokens and 11.1 hours per task observed), across 7 LLMs and 4 harnesses.

Prerequisites

Ascend C kernel programming, LLM inference/training profiling, and distributed execution frameworks

Scope

modify existing code and expand benchmark evaluation suite

The Gap

Section 3.1 and Appendix E evaluate only the 17 basic memory-access kernels in Batch 1, leaving the 31 multistage and fused attention kernels in Batches 2 and 3 unreported. Moreover, each task test case binds a kernel to a single static deployment configuration (for example, AddRMSNorm fixed to batch size 1 and sequence length 32 in Qwen3-0.6B prefill) and measures kernel latency in isolation from surrounding graph execution.

“This report evaluates the 17 kernel optimization tasks in Batch 1; results for Batches 2 (17 kernels) and 3 (14 kernels) will be reported in future updates.”
From the paper, Section 3.1, page 3. Matched against the PDF text.

Limitation

Evaluating isolated basic kernels on a single static tensor shape allows agents to over-specialize code for one input configuration while failing to reflect dynamic prefill/decode shape regimes, complex fused operators, or end-to-end model throughput.

Proposed Solution

Complete the evaluation benchmark across Batches 2 and 3 and augment each task manifest with a distribution of real workload shapes (varying batch sizes, prefill and decode sequence lengths, and tensor-parallel configurations) weighted by profiling invocation frequency. Add an end-to-end replay harness inside AscendForge using vLLM-Ascend and MindSpeed-LLM execution traces to verify that kernel-level speedups translate to full-model step-time reductions without regressing non-target shapes.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

If successful, this extension would prevent shape overfitting and ensure that agent-optimized kernels deliver reliable speedups across realistic production inference and training workloads.

How to Get Started

How to test it: Hypothesis: single-shape Batch 1 kernel optimizations degrade or fail to generalize on unseen prefill/decode shapes and fused Batch 2/3 operators, whereas multi-shape-evaluated candidates sustain positive geometric mean speedup. Baseline: single-shape Batch 1 winners from Claude Code/Opus 5 and Codex/GPT-5.6; metric: frequency-weighted multi-shape GM speedup and end-to-end vLLM-Ascend step latency; stop if multi-shape evaluation adds over 5x NPU profiling overhead without altering kernel rankings (Spearman rho > 0.95).

  1. Clone the released AscendGYM repository and verify baseline compilation and execution on Ascend 910C for all 48 tasks across Batches 1, 2, and 3.
  2. Profile Qwen3-0.6B and DeepSeek-V4 inference and training runs in vLLM-Ascend and MindSpeed-LLM to extract dynamic prefill and decode tensor shape distributions for each operator.
  3. Extend the AscendForge test driver and input manifests to score candidates using a workload-frequency-weighted geometric mean across multiple tensor shapes.
  4. Re-evaluate the top Batch 1 kernels optimized by Claude Code/Opus 5 and Codex/GPT-5.6 against the multi-shape suite and end-to-end forward passes to measure shape overfitting.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers like BladeDISC and TVM cover end-to-end compiler optimizations and workload evaluations, while work on Ascend NPUs touches upon dynamic shapes and operator optimization. However, none of the listed results combine a full evaluation benchmark across multiple batches with workload-frequency weighting and vLLM-Ascend/MindSpeed-LLM replay harnesses for NPU kernels.

We searched OpenAlex and arXiv for “kernel optimization benchmark dynamic shapes end-to-end workload evaluation” and “Multi-Shape Generalization and End-to-End Workload Evaluation for NPU Kernel Optimization” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

AscendGYM: A Cost-Aware Kernel Optimization Benchmark for Real-World Workloads on Ascend NPUs

AI Training Platform Team, Bingwei Zhang, Jing Li +24 more · 2026

This paper introduces AscendGYM, a benchmark and execution sandbox (AscendForge) for evaluating how effectively LLM coding agents optimize existing Ascend NPU kernel implementations from real-world LLM training and inference workloads. Unlike prior kernel-generation benchmarks, AscendGYM measures optimization gains relative to production baselines alongside token cost using a new Speedup-Token Score (STS) metric.

Related Ideas

Is this a good idea?

Share onXLinkedIn