We tracked 1M LLM API calls – 62% were using the wrong model
An analysis of one million LLM API calls found that 62% of them used an inappropriate model, leading to unnecessary costs. The study highlights common over-provisioning where simpler tasks were handled by expensive models, wasting money without improving quality.
Background
- Large Language Models (LLMs) like OpenAI's GPT-4 and Anthropic's Claude are accessed by developers via paid API calls, priced per "token" (roughly per ~0.75 word). Models vary widely in capability, speed, and cost — from cheap, fast "small" models to expensive, slow "frontier" ones.
- Many developers automatically route all queries to the most powerful model (e.g., GPT-4), even for simple tasks like summarization or classification — analogous to using a race car to drive to the corner store.
- "Prompt routing" or "model routing" is an emerging technique: analyzing each request and dynamically sending it to the cheapest model that can handle it well, cutting costs without losing quality.
- The article's data from 1M real API calls suggests that nearly two-thirds of requests currently use an unnecessarily expensive model, implying most developers are overpaying significantly.