AI feature cost is driven by tokens per interaction, interactions per user and hidden multipliers — resent conversation history, retries, retrieved context and agent loops. A simple spreadsheet built before launch, with a per-user cap in code, prevents the expensive surprises.
Token pricing invites a reassuring mental estimate. A fraction of a cent per request, thousands of requests, a modest monthly figure. Then the first invoice arrives and the figure has a different number of digits.
The gap is never the headline price. It is the multipliers.
Count interactions, not requests
One thing a user does is often several calls: a classification, a retrieval embedding, a rerank, the answer, perhaps a validation pass. Modelling the answer call alone understates by a factor you will not notice until it matters.
Write down every call in one user-visible action, with its typical input and output length. That table is the cost model.
Conversation history is the quiet one
A chat resends its history on every turn, so turn ten pays for turns one through nine again. Total cost across a conversation grows with the square of its length, not linearly — and long conversations are exactly what a good assistant encourages.
The multipliers to include
Retries on rate limits and validation failures. Retrieved context, which is often larger than the question by an order of magnitude. Agent loops, where a single request can become fifteen calls. Streaming cancellations that were billed anyway.
None of these is exotic. All of them are missing from the first estimate.
Where to get the savings
Prompt caching on a stable prefix. A smaller model for the mechanical steps. Summarising conversation history instead of resending it. Retrieving five chunks rather than thirty. Deduplicating identical requests.
Each is a modest percentage; together they routinely halve the bill without a quality change anyone can detect.
Cap it in code
A per-user budget per period, enforced before the call, is the thing that turns an unbounded liability into a known one. Warn as it approaches, degrade to a cheaper model rather than refusing outright, and make the limit visible.
Write this before launch. It is a small feature beforehand and an emergency afterwards.
Frequently asked questions
What usually blows the budget?
Long conversations and agent loops. A chat that resends its full history pays for turn one again on every later turn, and an agent with no step cap can spend a hundred times a single call without anything appearing broken.
How do I cap spending without ruining the feature?
Budget per user per period rather than per request, warn before the limit, and degrade to a smaller model rather than cutting off entirely. A hard stop with no warning reads as a bug to the person it happens to.
