Skip to content
AI-generated research idea🟠Medium🔬Significant ResearchRelated work found

Retrieval-Augmented Feedback for Code Generation

Integrate a retrieval-augmented generation system into compiler-based feedback loops to supply relevant hardware manuals during code generation mutation.

Time

3-5 days including compiler/profiler environment setup and ablation runs

Compute level

Single GPU or less

Resources

1x NVIDIA V100/A100 GPU for CUDA kernel compilation and profiling, AMD XDNA2 NPU compiler/simulator environment for NPUEval, a lightweight vector index for technical manuals, and ~$150-$300 in GPT-4o API credits for up to 3,000 rollouts.

What the paper used

GPT-4o evaluated on NPUEval (25+ kernels, Sequential10 up to 10 refinements) and 35 tasks from KernelBench representative subset (Sequential5, up to ~3,000 rollouts) targeting AMD XDNA2 NPU and NVIDIA V100 GPUs.

Prerequisites

RAG, Code generation, Compiler toolchains

Scope

modify existing code

The Gap

In Section 5.1, GEPA is applied to code generation (NPUEval) using compiler errors as feedback. The authors note that 'kernel development expertise—often codified in technical manuals—can be selectively surfaced' but their current implementation is preliminary.

“Specifically, kernel development expertise—often codified in technical manuals and documentation—can be selectively surfaced by retrieving relevant manual sections based on rollout failures (e.g., compiler error messages).”
From the paper, Section 5.1, page 12. Matched against the PDF text.

Limitation

The current feedback function relies primarily on the LLM's internal knowledge and raw error messages. When optimizing for obscure hardware (like NPUs) or complex APIs, the LLM may lack the specific syntax knowledge required to fix the error based on the trace alone.

Proposed Solution

Integrate a Retrieval-Augmented Generation (RAG) system into the feedback function (mu_f). When a compiler error occurs, use the error message to retrieve relevant snippets from the specific hardware documentation or API manual. Feed this retrieved context into the reflective mutation step to guide the LLM in generating syntactically correct kernel code.

Why This Matters

This extends GEPA from general reasoning tasks to specialized engineering domains where LLMs traditionally hallucinate, enabling automated optimization of high-performance code for new hardware architectures.

How to Get Started

How to test it: Note that the paper already implements error-driven RAG inside the feedback function mu_f for NPUEval and KernelBench in Section 5.1. To isolate and validate the RAG-in-mu_f contribution on an unseen hardware target or API, compare GEPA with RAG-augmented mu_f against GEPA with raw compiler-error mu_f (no RAG) on NPUEval and a held-out kernel suite using mean vector utilization and fast_1; stop if RAG-augmented mu_f does not improve vector utilization or fast_1 over raw compiler feedback.

  1. Construct a vector database containing technical manuals and API documentation for target hardware kernels.
  2. Modify the reflective mutation function to capture compiler error messages as search queries.
  3. Retrieve top-k relevant documentation snippets and append them to the prompt context sent to the large language model.
  4. Evaluate the updated code generation pipeline on NPUEval using syntactic correctness and execution success metrics.

Code linked to this paper on Hugging Face: github.com/gepa-ai/gepa

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

AgenticCANN: Automated Ascend C Operator Generation via Knowledge-Augmented Agentic Evolution explores operator generation, and Improving LLM-Assisted Secure Code Generation through Retrieval-Augmented-Generation and Multi-Tool Feedback uses RAG with feedback for code generation. However, they do not specifically target using compiler errors to retrieve hardware documentation for iterative kernel mutation.

We searched OpenAlex for “Integrate a retrieval-augmented generation system into compiler-based feedback loops to supply relevant hardware manuals during code generation mutation.” and “Retrieval-Augmented Feedback for Code Generation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning

Lakshya A Agrawal, Shangyin Tan, Dilara Soylu +14 more · 2025

This paper introduces GEPA (Genetic-Pareto), a prompt optimization algorithm for compound AI systems that outperforms Reinforcement Learning (RL) methods like GRPO. GEPA uses natural language reflection to mutate prompts based on execution traces and maintains a Pareto frontier of diverse candidates to avoid local optima, achieving higher performance with significantly fewer rollouts.

Related Ideas

Is this a good idea?

Share onXLinkedIn