"One million tokens" has become a spec line, like megapixels on a phone. It sounds like more room is strictly better. In practice, a large context window is a budget with three costs: the price of filling it, the latency of reading it, and the risk that the model attends to the wrong part of it.
Understanding those costs changes how you design a prompt. The goal is not to use the whole window. The goal is to put the right tokens in it.
Cost one: every token is billed before the answer starts
Most APIs charge for input tokens separately from output tokens, and input is usually cheaper per token. Cheap is not free. If you paste a 200-page contract into every request, you pay for 200 pages every time, even when the question is about one clause.
A simple example makes the scale visible. Suppose input costs one unit per million tokens and output costs four units per million. A prompt with 500,000 input tokens costs 0.5 units before the model writes a word. Ten such requests cost five units. The same ten questions against a retrieved 4,000-token excerpt cost 0.04 units. The difference is not a rounding error; it is the difference between a viable feature and an expensive demo.
Caching helps, and some providers discount repeated prefixes. Caching is not a substitute for scoping. You still pay to build the cache, you still risk stale content, and you still send the tokens over the network.
Cost two: latency grows with the input
Time to first token scales with prefill work. A short prompt feels instant; a long one can pause for seconds before the first word appears. If the interface is a chat box, users tolerate a pause. If the interface is a search bar, they do not.
There is a second-order effect: long inputs make streaming less useful. The user waits for the model to read, not to think, and the wait is hard to explain because nothing is appearing on screen. A progress indicator helps; a smaller prompt helps more.
Latency also compounds in agents. An agent that calls the model five times with a full repository in context spends most of its time re-reading the same files. Retrieval, summaries, and a stable scratchpad are cheaper than repeating the corpus.
Cost three: attention is not uniform
A model can accept a million tokens and still struggle to use the middle of them. Research on long-context behavior has repeatedly found that information near the beginning and end of a prompt is recalled more reliably than information in the middle. The effect is not a hard boundary, but it is real enough to design around.
This is why "just put everything in context" fails on long documents. The answer may be present and still not be used. Chunking with headings, putting the question before and after the evidence, and extracting the relevant section first all improve reliability more than adding another hundred thousand tokens.
A better default: retrieve, then reason
The practical pattern is:
- Find the smallest set of text that can answer the question. A search over headings, a keyword filter, or an embedding lookup.
- Send that set, plus a short statement of the task. Include the source labels so the model can cite them.
- Ask for the answer and the evidence. If the evidence is missing, the model should say so rather than fill the gap.
- Log the token counts. A feature that quietly grows from 4,000 to 40,000 tokens per request will show up on the bill before it shows up in the dashboard.
This is more engineering than pasting a file. It is also the difference between a feature that survives a pricing page and one that does not.
When the big window is right
There are legitimate cases for a large window: a one-off analysis of a document you cannot chunk without losing structure, a code review where cross-file references matter, a batch job where latency is irrelevant and the input is genuinely the whole corpus. In those cases, use the window and measure the result. The mistake is treating the maximum as the default.
A useful rule: the window is a ceiling for exceptional inputs, not a target for ordinary ones. If your median request uses less than a tenth of the window, the window is not the feature. The retrieval is.
What to measure
Track four numbers per feature: median input tokens, p95 input tokens, cost per successful task, and time to first token. If input tokens are climbing while task success is flat, the prompt is accumulating text that does not earn its place. If cost per successful task is rising faster than usage, the feature is getting less efficient as it grows.
The context window is a tool for fitting the work, not a measure of quality. A model that reads the right page beats a model that skims the whole book. For the surrounding decisions, see how to read model pricing and context windows.