Token accounting and cancellation in durable agent workflows
The article discusses techniques for token accounting and cancellation in durable agent workflows when using streaming large language models (LLMs). It addresses the challenge of managing costs and system resources by accurately tracking token usage and implementing effective cancellation mechanisms for long-running or unnecessary streaming requests.
Background
Large language model (LLM) agents are AI programs that can take multi-step actions (e.g., browsing the web, calling APIs) rather than just answering one question. "Streaming" means the agent outputs tokens (word fragments) one by one in real time, rather than waiting for the full response.
- "Durable agent workflows" are systems designed to survive crashes — if the server restarts mid-task, the agent should resume where it left off, not start over.
- "Token accounting" is the problem of tracking exactly how many tokens an agent has consumed and billed for, which becomes tricky when a workflow is paused, cancelled, or restarted mid-stream.
- "Cancellation" here refers to stopping an agent's execution cleanly — making sure no partial output leaks to the user, no tokens are double-counted, and no side effects (like database writes) are left hanging.
- This blog post addresses a real engineering gap: popular LLM frameworks (LangChain, AutoGen, etc.) handle simple chatbot streaming well, but have no standard way to manage token billing and cancellation for long-running, restartable agent tasks.