Reap: Automatic Curation of Coding Agent Benchmarks
Reap is a framework for automatically curating coding agent benchmarks from real-world GitHub issues, reducing the need for costly manual creation. It filters candidate issues, generates reproducible environments, and evaluates agents using natural-language reference solutions as a proxy for ground-truth tests.
Background
- **What it is**: A new academic paper introducing *Reap*, a system that automatically creates benchmark tests for evaluating AI coding agents (like GitHub Copilot, Cursor, or Claude’s code-writing capabilities).
- **Why it matters**: Benchmarks are how researchers measure whether coding AI is getting better. But existing benchmarks are hand-crafted, slow to update, and risk being “leaked” (the AI memorizes the test questions). Reap aims to generate fresh, reliable benchmarks automatically, making it easier to track real progress.
- **Prior context**: AI coding agents have rapidly improved, but the community struggles to keep evaluation tests up to date. Manual benchmark curation doesn’t scale, and static benchmarks can lead to overfitting. Reap addresses this by using a “curation” pipeline that selects high-quality test problems from a larger pool.
- **Key people/orgs**: The authors are academic researchers; the paper is on arXiv, meaning it is a preprint not yet peer-reviewed.