Skip to content

Source paper

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

Xinping Lei, Minghao Liu, Ruijie Zhu and 23 more

Xinping Lei, Minghao Liu, Ruijie Zhu, Xiaohui Hu, Xingqiao Li, Ao Zhang, Wenhao Chai, Han Li, Tao Wang, Jingyuan Zhang, Enduo Zhao, Yunfeng Shi, Yiming Liang, Zetao Yang, Xingjian Wang, Yifan Yao, Xinkai Ma, Yingru Li, Tianhan Yang, Huan Zhou, Weiran Shi, Shen Yan, Jiaheng Liu, Stephen Huang, Eli Cheung, Weihao Xuan

arXiv 2611.20218 · 202651 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

The paper introduces SWE-CoWork, a benchmark evaluating software engineering agents as digital employees within simulated organizations of 26 rule-governed colleagues across 101 repositories, 11 languages, and 12 domains. Agents must inspect evolving records, converse with colleagues to recover distributed requirements, implement complex changes, and deliver a verified handoff. The study reveals a critical gap between partial code generation (~63% fail-to-pass tests) and end-to-end task success (<8%), primarily caused by breakdowns in long-horizon requirement tracking, abandoned clarifications, and hallucinated handoff completion.

How it works (AI summary)

The benchmark embeds 101 real open-source repositories within simulated organizational workplace environments containing project boards, documents, chat histories, pull requests, and 26 digital colleagues backed by GPT-5.6 Terra with explicit knowledge boundaries and behavioral rules. Agents interact via a tool interface to query records and message colleagues to discover unstated requirements, execute edits in the repository, and undergo automated evaluation against paired fail-to-pass (F2P) and pass-to-pass (P2P) executable test suites alongside an Oracle-Spec condition.