Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase
Databricks developed SWE-Lancer, a benchmark to evaluate coding agents on real-world tasks from their multi-million line codebase. The benchmark includes unit tests and manager-grade task grading. Results show that current frontier models complete under 28% of tasks, highlighting significant room for improvement in AI-assisted software engineering.
Background
- Databricks is a major data and AI platform company (publicly traded, ~$60B market cap) that built its product on top of Apache Spark, an open-source big-data framework.
- "Coding agents" are AI-powered tools that can autonomously write, debug, or refactor code — a hot area in 2024-2025 as companies race to turn LLMs into practical software engineering assistants.
- This post describes Databricks' internal benchmark (called SWE-bench-style) for testing these agents on their actual production codebase, which spans millions of lines of Scala, Python, SQL, and Java — far messier and more realistic than the curated academic benchmarks most AI labs use.
- The key challenge: a real codebase has complex dependencies, legacy code, proprietary APIs, and decades of engineering decisions that off-the-shelf coding agents struggle with. Databricks is making their benchmark data public to push the industry toward more realistic evaluations.