Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationRelated work found

Evaluation on Repository-Scale Benchmarks (SWE-bench)

Extend the evaluation harness of autonomous development agents to support repository-scale benchmarks so systems can be tested on real GitHub issues and pull requests.

Time

3 weeks

Compute level

Large cluster

Resources

A multi-GPU cluster or cloud-based Docker execution environment with access to the GPT-4 API to run hundreds of multi-turn repository-level benchmark tasks.

What the paper used

Evaluated on HumanEval dataset comprising 164 handwritten programming problems using GPT-4 model (gpt-4-1106-preview).

Prerequisites

Docker, Python, Software Testing Frameworks

Scope

modify existing code

The Gap

The paper evaluates AutoDev primarily on HumanEval (Section 3 and 6.2), a dataset of small, self-contained algorithmic problems. Section 6.2 acknowledges that current metrics fall short in capturing essential programming aspects of real-world codebases.

“While our current evaluation relies on HumanEval for assessing the effectiveness of AutoDev in two coding tasks, our future work aims to extend this assessment to more challenging and real-world datasets, such as those offered by the Copilot Evaluation Harness.”
From the paper, Section 6.2, page 8. Matched against the PDF text.

Limitation

Evaluating on isolated functions fails to test the system's ability to navigate complex dependency graphs, handle large file contexts, or modify legacy code in existing repositories, which are the primary challenges in real-world software engineering.

Proposed Solution

Extend the AutoDev evaluation harness to support SWE-bench, a benchmark consisting of real GitHub issues and Pull Requests. This involves adapting the Docker environment to dynamically install repository-specific dependencies and modifying the retrieval tools to handle codebase-wide context searching.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Validates the framework's practical utility for actual software maintenance tasks rather than just algorithmic puzzles, potentially bridging the gap to commercial adoption.

How to Get Started

How to test it: The hypothesis is that integrating SWE-bench will measure repository-level navigation capabilities. The baseline is current function-level HumanEval performance. The evaluation metric is pass rate on SWE-bench tasks. A passing result showing robustness would maintain at least 40% resolution rate.

  1. Clone the SWE-bench repository and review the evaluation harness and Docker environment specifications.
  2. Integrate the agent execution framework with the SWE-bench Docker container to handle dynamic dependency installation.
  3. Update the agent's codebase retrieval and search tools to support repository-wide context indexing and lookup.
  4. Run the evaluation pipeline on a subset of SWE-bench Lite issues to measure resolve rates and parsing failures.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? introduces the SWE-bench benchmark for evaluating language models on real GitHub issues and pull requests, requiring dynamic environment setup and retrieval strategies. However, the listed papers do not specifically detail the integration of this benchmark into the AutoDev harness as proposed.

We searched OpenAlex for “Extend the evaluation harness of autonomous development agents to support repository-scale benchmarks so systems can be tested on real GitHub issues and pull requests.” and “Evaluation on Repository-Scale Benchmarks (SWE-bench)” and an AI compared the idea with the top 6 results. A small search can miss relevant work.

Source Paper

AutoDev: Automated AI-Driven Development

Michele Tufano, Anisha Agarwal, Jinu Jang +2 more · 2024

AutoDev is a fully automated AI-driven software development framework that enables LLM-based agents to autonomously plan and execute complex software engineering tasks. Unlike standard coding assistants that only suggest snippets, AutoDev agents operate within a secure Docker environment to edit files, retrieve context, build code, run tests, and execute git commands iteratively until objectives are met.

Related Ideas

Is this a good idea?

Share onXLinkedIn