A vendor page quotes a price per million tokens. Your product makes a request with a system prompt, three retrieved documents, a tool schema, the user's message, and a tool result, then repeats the whole thing on the next step. The number on the vendor page is an ingredient, not the bill.
This guide is the arithmetic between the two. It pairs with how to read model pricing, which explains what the fields mean; here the question is what your workload will cost and where to put the guardrails.
Price is a unit; the bill is a workload
Start with one real request. Count four quantities:
- input tokens sent, including the system prompt, retrieved text, tool definitions, and conversation history;
- output tokens generated, including any hidden reasoning the vendor bills;
- the number of requests per month;
- the number of attempts per accepted answer.
The monthly estimate is:
monthly cost = requests × attempts × (
input_tokens × input_price / 1,000,000
+ output_tokens × output_price / 1,000,000
)
Plug in one request you have actually sent, not the demo from the vendor's page. If the request includes a 12,000-token document and a 900-token tool schema, that is 12,900 input tokens before the user types anything. Multiply by twelve steps in an agent loop and the same conversation can send six figures of input tokens for one task.
Count tokens, then multiply
Do not estimate tokens from word count and call it a budget. English prose is roughly four characters per token, which is close enough to spot an order of magnitude and wrong enough to miss a 20 percent difference. Code, JSON, Chinese, and emoji all move the ratio.
Use the vendor's tokenizer when one is published, or the tokenizer shipped with the model family. Count the whole payload, including the tool schema and the system prompt, because the bill includes them. If the API returns usage numbers, log them; after a week you will have a better model of your workload than any calculator.
The input is the bill
In a chat product with short answers, input dominates. In an agent loop it dominates twice: every step resends the accumulated context, and tool results add new text. A loop that runs ten steps does not cost ten times a single call; it costs the sum of a growing prefix, which is closer to quadratic in the number of steps.
Three habits keep the input down:
- Retrieve, do not paste. Send the section that answers the question, not the whole manual.
- Trim history. Keep the last few turns and a summary of the rest. The model does not need the entire transcript to answer the current message.
- Cap tool output. A tool that returns a 40,000-token log will be paid for on every subsequent step. Truncate at the tool boundary and return a path to the full file.
A large context window is not a license to fill it. Models attend unevenly across long inputs, and a detail buried in the middle is recovered less reliably than the same detail near the end. Paying for tokens the model underuses is the worst of both.
Caching changes the arithmetic
Prompt caching lets a vendor reuse a stable prefix across requests. The discounts are real but asymmetric: at the vendors we checked, a cache read costs about a tenth of the base input price, a short-lived cache write costs about 1.25 times the base, and a longer-lived write costs about twice the base. The exact multipliers move; the shape does not.
The consequence is a break-even rule. A cache write at 1.25 times base costs an extra quarter of one request. Each later read saves nine tenths. One reuse pays for a short-lived write, if the cache is still alive. A longer-lived write at twice the base needs two or more reuses before it beats paying the base price each time. If your traffic is one request every ten minutes and the cache lives five minutes, you will pay the write premium and never collect.
Caching rewards a stable prefix: the same system prompt, the same tool schema, the same document preamble, in the same order. It punishes prompts that shuffle retrieved chunks or inject a timestamp near the top. Restructure the request so the volatile part comes last, and measure the hit rate rather than assuming it.
Batch is a different product
Several vendors price asynchronous batches at half the standard rate, on input and output. That is a large discount with a large condition: results arrive later, often within hours and sometimes within a day. Batch is right for offline evaluation, backfills, classification of a stored corpus, and report generation. It is wrong for an interactive agent, a support widget, or anything a person is waiting on.
Do not compare a batch price with an interactive price in the same table. They are different products with different latency contracts. If your workload can tolerate the delay, batch is the cheapest lever available; if it cannot, the discount is irrelevant. Batch can also stack with caching when the batch shares a prefix, which makes an offline job cheaper still.
Retries are a cost line
The formula above has an attempts term for a reason. A model that fails one request in five, or produces an answer you regenerate, costs more than its unit price. So does a model that needs a longer prompt to follow the format. Measure the retry rate on your own tasks; do not assume it is zero because the demo was clean.
The same logic applies to routing. A cheap model that handles 70 percent of traffic and escalates the rest can be cheaper than a frontier model for everything, but only if the escalation rule is precise. Route by consequence: a call whose wrong answer is caught by a schema or a test can go cheap; a call whose wrong answer reaches a customer should not.
A one-page worksheet
Before launch, write these numbers in the repository:
| Field | Example |
|---|---|
| Requests per month | 50,000 |
| Attempts per accepted answer | 1.2 |
| Input tokens per request | 20,000 |
| Output tokens per request | 400 |
| Input price per 1M | $2.00 |
| Output price per 1M | $10.00 |
| Monthly estimate | 50,000 × 1.2 × ($0.040 + $0.004) = $2,640 |
Then add the second column: what happens at three times the traffic, and what happens if the retry rate doubles. If either number breaks the budget, the fix belongs in the design, not in a dashboard alert.
What to monitor after launch
Log tokens and cost per request, not just latency. Break the number down by route, by step in the loop, and by model version. Watch three ratios:
- cost per accepted answer, which catches retries and silent failures;
- cache hit rate, which catches prompts that stopped being cacheable;
- input-to-output ratio, which catches context that keeps growing.
Set a ceiling per request and per account, and make the system fail closed when it is crossed. A bill you discover at the end of the month is not a budget; it is an anecdote. The model directory carries the price fields this arithmetic needs, and the comparison pages show how much the same workload costs across rows.