TokenBudgeting: Our Conversations with Enterprises on Token Spend
The article examines how enterprises are approaching token budgeting for large language models, discussing strategies to manage and optimize token spend based on conversations with various businesses. It highlights the need for cost tracking, efficient prompt design, and the trade-offs between model capability and expense.
Background
- The "token" in AI refers to the basic unit of text (roughly a word or part of a word) that a language model processes. Every API call to services like OpenAI or Anthropic costs money per token — both for input (your prompt) and output (the model's response).
- "Token budgeting" is a new area of enterprise planning: companies are now formally budgeting for AI model usage much like they budget for cloud compute or software licenses, tracking prompt volumes, output lengths, and caching strategies to control costs.
- This matters because generative AI has moved from experimental to production use, and many enterprises were surprised by how quickly token costs scale when hundreds of employees run AI queries daily.
- Key players mentioned include large-scale AI users (tech firms, financial services, healthcare) and infrastructure providers (cloud platforms and model API services). The piece draws on direct conversations, not public financial disclosures.