Empirical Computation: Prompting versus Programming [pdf]
The paper presents an empirical comparison between prompting large language models and traditional programming for computational tasks, analyzing their strengths, weaknesses, and appropriate use cases to guide developers in choosing between the two approaches.
Background
The paper compares two approaches to building software: traditional programming (writing explicit code in languages like Python or Java) vs. prompting large language models (LLMs) like ChatGPT or Claude to generate or modify code. The key question is whether describing a task in natural language can outperform writing precise instructions, especially for tasks like fixing bugs or refactoring. The author, Marcel Böhme, is a computer scientist at MPI-SP (Max Planck Institute for Security and Privacy) known for work on software testing and empirical methods in security. This paper contributes to the ongoing debate about whether LLMs replace programmers or serve as different tools for different jobs—it uses controlled experiments rather than anecdotes to measure trade-offs in correctness, cost, and effort.