使用DSPy评估与改进Datasette Agent的SQL系统提示词
Simon Willison在AIE主题演讲的启发下，尝试使用DSPy框架来评估和优化Datasette Agent的SQL系统提示词。他通过Claude Code和Claude Fable 5发起异步研究任务，Fable选用GPT 4.1 mini和nano进行测试，发现了多个改进方向。其中一个关键发现是：当前提示词中只列出了表名而未包含列名，加上"已掌握信息就不要调用describe_table"的建议，导致模型猜测列名（如page_count、o.order_id、first_name）并陷入错误重试循环。解决方案是在提示词的模式描述中直接包含列名，或软化相关建议。
背景速读
- **Simon Willison**: well-known tech blogger and creator of Datasette, an open-source tool for exploring and publishing data. **Datasette Agent** is an AI-powered feature that lets users ask natural-language questions and have the AI generate read-only SQL queries against a database.
- **DSPy (Declarative Self-improving Python)**: a framework from Stanford NLP for systematically optimizing LLM prompts and program logic — think "auto-tuning for prompt engineering," rather than manually rewriting prompts by trial and error.
- **Claude Fable 5 / Claude Code for web**: Anthropic's experimental coding agent. Willison used it here to autonomously install tools, run experiments, and analyze results.
- **What happened**: Willison asked Fable to use DSPy to evaluate and improve Datasette Agent's SQL-generation system prompt. The agent found that the original prompt omitted column names in schema listings, causing the LLM to guess column names and fail; it suggested fixes like including column names explicitly. This demonstrates a workflow where an LLM drives prompt improvement via automated evaluation.