First Principles of Model Routing
The article explores model routing from first principles, breaking down how AI systems can intelligently direct queries to the most suitable model based on task requirements. It covers core concepts like cost-performance trade-offs, latency considerations, and the architectural decisions behind building effective routing systems for large language models.
Background
- Model routing means automatically sending each user query to the best-suited AI model rather than using one model for everything. The goal: use small/cheap models for easy tasks and save expensive frontier models (GPT-4, Claude Opus) for hard ones.
- This article comes from Try Works, a company building an open-source routing platform. It argues routing should be based on "first principles" — fundamental properties of the query — not crude heuristics like word count.
- Key ideas: "domain of resolution" (the range of plausible correct answers — narrow for math, wide for creative writing); "proxy tasks" (measuring something easy like confidence scores to predict real performance); and matching a query's required capabilities (reasoning, recall, creativity) to a model's known strengths.
- The authors warn that routing can create "selection bias" in evaluation data, making routers appear more accurate than they truly are.