Simon Willison · 2026-07-02 · notable
Simon Willison — using DSPy to fix Datasette Agent's SQL prompts
Simon Willison lets Claude Code run DSPy against Datasette Agent's SQL system prompt and finds one culprit: the schema listing skips column names, which pushes the model into guessing and error-retry loops.
Simon hands Claude Code a DSPy research task on Datasette Agent's prompt and pins a real regression to one line about describe_table.
What is it?
'Using DSPy to evaluate and improve Datasette Agent's SQL system prompts' is Simon Willison's July 2 write-up of an automated prompt-engineering experiment. He asked Claude Code to install Datasette, datasette-agent and DSPy, then use DSPy to iterate on the system prompt that steers Datasette Agent's read-only SQL query tool.
How does it work?
DSPy treats the prompt as parameters and searches for wording that improves an evaluation score. In this run, the winning change traced back to a single sentence: the schema listing gives only table names, and the prompt's 'don't call describe_table if you already have the information' advice was pushing the model into guessing column names and burning tokens on error-retry loops.
Why does it matter?
This is one of the cleanest published examples of using an agentic loop plus DSPy to debug a real production prompt end to end — install, run, evaluate, blame a specific line. Anyone shipping a Text-to-SQL or schema-aware agent can lift the loop directly and probably spot the same class of bug.
Who is it for?
LLM app developers, prompt engineers, agent framework maintainers
Try it
Research repo: https://github.com/simonw/research/tree/main/dspy-datasette-agent-prompts