NFR-Aware Benchmarking for Agentic Coding
Create an evaluation framework paired with explicit non-functional requirements to benchmark coding agents on both functional correctness and code quality constraints.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section 4.3 and Table 3 reveal a significant gap where non-functional requirements (NFRs) like security and performance are only present in 14.5% of context files, despite their criticality for production-grade software.
Limitation
Existing benchmarks like SWE-bench primarily measure functional task completion but fail to evaluate whether an agent adheres to quality guardrails or NFR constraints specified in a project's persistent context.
Proposed Solution
Create an evaluation framework where coding tasks are paired with agent context files containing explicit NFR constraints (e.g., 'Do not use wildcard imports' or 'Sanitize all SQL inputs'). Use a dual-scoring metric that evaluates both functional correctness and strict adherence to the provided NFR context.
Why This Matters
By penalizing agents for generating insecure or inefficient code that violates project-level instructions, researchers can shift the focus from 'making code work' to 'building code well.'
How to Get Started
- Select an existing coding benchmark such as SWE-bench as the evaluation foundation.
- Annotate a subset of tasks with explicit non-functional requirements like security rules and performance constraints.
- Develop a dual-scoring automated evaluation script combining functional test runners and static analysis tools.
- Run baseline coding agents through the new benchmark framework to measure adherence rates.
Code linked to this paper on Hugging Face: github.com/openai/agents.md
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers like 'DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents' explore joint security and functionality benchmarking, and 'Quality assurance of LLM-generated code: Addressing non-functional quality characteristics' discusses non-functional attributes. However, they do not specifically evaluate coding tasks paired with agent context files containing explicit NFR constraints using a dual-scoring metric.
- DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents · 2026 · 5 citations
- Quality assurance of LLM-generated code: Addressing non-functional quality characteristics · 2026 · 34 citations
We searched OpenAlex and arXiv for “Create an evaluation framework paired with explicit non-functional requirements to benchmark coding agents on both functional correctness and code quality constraints.” and “NFR-Aware Benchmarking for Agentic Coding” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Agent READMEs: An Empirical Study of Context Files for Agentic CodingWorawalan Chatlatanagulchai, Hao Li, Yutaro Kashiwa +8 more · 2025
This paper presents the first large-scale empirical study of agent context files (such as CLAUDE.md and AGENTS.md), analyzing 2,303 files from 1,925 repositories. The study characterizes these files as complex, 'configuration-as-code' artifacts that are frequently maintained but lack critical non-functional requirements like security and performance.
Related Ideas
- Requirement-Coverage and Execution-Trace-Aware Candidate Deliverable SelectionHarness^2: Recursive Agent Harnessing for an Open World🟢Easy
- Hermetic Harness-Agent Environment Isolation Against Dependency-Mediated Answer LeakageSage S2.1 Flash Technical Report🟡Fairly easy
- Bidirectional Dynamic Planning for Evolving SpecificationsDeepCode: Open Agentic Coding🟠Medium
- Generalizing OpenSeeker: Integrating Heterogeneous Tool CapabilitiesOpenSeeker: Democratizing Frontier Search Agents by Fully Open-Sourcing Training Data🔴Hard
- Persistent Agentic Heuristics through Post-Task ReflectionDeepCode: Open Agentic Coding🟡Fairly easy