Using DSPy to evaluate and improve Datasette Agent's SQL system prompts
Simon Willison used DSPy to evaluate and improve Datasette Agent's SQL system prompts. DSPy identified that the schema listing lacked column names and that advice to avoid calling describe_table caused column-name guessing and error retries.
Background
Simon Willison (creator of the Datasette open-source tool for exploring data) used DSPy to improve the system prompt for Datasette Agent, an AI assistant that converts user questions into SQL queries. DSPy is a framework from Stanford that systematically tests and optimizes LLM prompts, rather than relying on manual trial-and-error. Here, DSPy ran automated evaluations using GPT-4.1 mini/nano against the Datasette Agent's prompt and found a concrete issue: the prompt listed table names but not column names, causing the AI to guess column names (and fail) instead of calling the intended lookup function. The fix — including columns in the schema listing — was surfaced entirely through automated testing. This illustrates a broader trend: treating prompts as code that can be debugged and optimized via programmatic evaluation, rather than as "magic" incantations.