METR Study Finds Open-Source Maintainers Reject 24 Percent More SWE-Bench Patches Than Automated Graders

Software engineering researchers at Berkeley-based evaluation institute METR released an empirical study on Sunday, October 18, demonstrating that automated coding benchmarks substantially overstate the real-world readiness of AI-generated pull requests. Led by investigators along Shattuck Avenue in Berkeley, the study compared automated SWE-bench Verified grader scores against blind code reviews by open-source maintainers across 296 AI-generated pull requests (PRs) and 47 human-written golden patches from three repositories. Enterprise engineering departments, open-source foundation boards, and developer council leaders in San Francisco, Oakland, San Ramon, and Pleasanton noted the findings reshape how teams measure autonomous coding tools.

During the blind evaluation program, four active open-source repository maintainers inspected each submission without knowing whether it was written by a human developer or a frontier model. While human-written golden patches achieved a 68 percent baseline merge acceptance rate from the maintainers, actual maintainer merge decisions for AI-generated patches averaged 24 percentage points lower than their automated SWE-bench Verified test suite scores. Maintainers rejected benchmark-passing AI patches primarily due to core functionality gaps, subtle regressions that broke adjacent code paths, or violations of repository architecture and code quality standards.

In parallel research examining longer-horizon software tasks, METR researcher David Rein and collaborators at Epoch AI published preliminary findings from a benchmark program called MirrorCode. Their experiments demonstrate that frontier coding agents can now complete certain multi-week software engineering tasks when given structured repository environments and iterative testing feedback. Software architecture committees across Contra Costa and Alameda counties observed that bridging the 24 percent maintainer quality gap could save enterprise teams over $100,000 annually in manual code review overhead.

Complementing the MirrorCode study, METR researcher Nikola Jurkovic detailed qualitative code-audit lessons drawn from evaluating Claude Opus 4.6 command-line reimplementations of the popular games Balatro and Slay the Spire. Jurkovic's line-by-line inspection showed that while autonomous models rapidly scaffold playable game loops and interface menus, they frequently take shortcuts on edge-case rule interactions unless guided by strict verification suites. University computer science district instructors in Berkeley and Walnut Creek are already incorporating these code-audit case studies into autumn software testing courses.

Open-source maintainers, engineering managers, and AI researchers can download the full SWE-bench maintainer dataset and MirrorCode documentation online at metr.org. Technical questions regarding benchmark grant funding or repository evaluation partnerships may be directed to METR's Berkeley research office at 2150 Shattuck Avenue or by calling (510) 982-3110 between 9:00 AM and 5:00 PM Monday through Friday. Additional collaborative benchmark metrics are available at epochai.org.