Model cards compress a lot into a few numbers. A context window, a parameter count, a benchmark score, and a price per million tokens look comparable across vendors. They usually are not. The same words describe different billing rules, different limits, and different failure modes.

This guide explains how to read the numbers in the model directory without treating them as a single ranking. It is not a pricing table; vendors change prices. It is a way to ask the right question before you put a model behind a product.

The context window is not the output limit

A context window is the total number of tokens the model can consider in one request: input plus output, and sometimes reasoning tokens that are not shown to you. The window is a ceiling on the whole conversation, not a promise that you can generate that many tokens.

Look for a separate maximum output. A model with a very large context window may still cap a single response at a few thousand tokens. If your job is "summarize a long report into a long report," the output limit matters as much as the input window.

Three practical consequences:

  • A large window does not mean the model attends to every part of it equally. Long-context performance varies by task and by position in the prompt.
  • A large window can make a request more expensive even when the answer is short, because you pay for the input you send.
  • A large window encourages lazy prompting: pasting an entire repository instead of selecting the files that matter. More context is not always better context.

For a coding agent, the useful question is not "how big is the window" but "which files and instructions fit inside it alongside the conversation." The first week with a coding agent guide treats context as a budget.

Price per million tokens is a unit, not a bill

Vendors usually quote separate prices for input and output tokens, per million. The headline number is rarely the number you pay.

  • Input-heavy work (classification, extraction, search) is dominated by input price.
  • Output-heavy work (long drafts, code generation, reasoning) is dominated by output price.
  • Agent loops multiply both: every step resends context, and every step may produce tool calls and reasoning tokens.

A model that is cheap per input token but expensive per output token can cost more than a "pricier" model for a writing task. A model that is cheap for both can still be expensive if it needs three attempts to produce a usable answer.

Do the arithmetic on one real request before you compare headline rates. Take the input tokens you actually send, the output tokens you actually receive, and multiply by the vendor's current prices. The result is the only comparable number.

Cached input, batch discounts, and reasoning tokens

Three billing details change the comparison more than the headline price.

Cached input. Some vendors charge less when a prefix of the prompt repeats across requests. This rewards stable system prompts and stable document context, and it punishes prompts that reshuffle the same material on every call. If your workload sends the same long instructions repeatedly, check whether caching applies and how long the cache lasts.

Batch pricing. Some vendors offer a discount for asynchronous batches with a longer turnaround. This is useful for offline evaluation and useless for an interactive agent. Do not compare a batch price with an interactive price.

Reasoning tokens. A reasoning model may generate tokens you never see, and those tokens may be billed as output. A model that looks affordable per visible answer can be expensive per request. Check whether the vendor bills hidden reasoning, and whether you can cap it.

There is also the retry multiplier. A model that fails one in five requests, or produces an answer you have to regenerate, is more expensive than its unit price suggests. Reliability is part of the cost.

Parameters, quantization, and the price of self-hosting

A parameter count is a rough measure of model size, not a measure of quality. It tells you something about memory requirements and serving cost, especially when you host the model yourself.

For self-hosting, the bill is different:

  • GPU memory must hold the weights, the KV cache, and the runtime overhead.
  • Quantization reduces memory and can reduce quality; the same model name at different precisions is not the same product.
  • Throughput depends on batching, sequence length, and the inference stack, not just the parameter count.
  • Someone has to operate it: upgrades, monitoring, failover, and security.

A hosted API price includes those costs. A self-hosted model is only cheaper when you have enough steady utilization to justify the hardware and the attention. For a small team, the cheap-looking local model often costs more in engineering time than the API it replaces.

The open source and proprietary models guide covers the data-handling side of the same decision.

Benchmarks are not a price-performance curve

Benchmark scores on a model card are usually vendor-reported or taken from a public leaderboard. They are useful for a rough sanity check and dangerous as a selection tool.

  • Scores are sensitive to prompt format, sampling settings, and the harness used to run them.
  • A benchmark can be contaminated if its questions appear in training data.
  • A single aggregate score hides the distribution: a model can be excellent at code and poor at instruction following.
  • A newer model with a lower score may still be the better product for your task.

Use benchmarks to build a shortlist, then test the shortlist on your own tasks. The test a model on your own tasks guide has a small evaluation routine that costs less than a day.

A worked comparison

Suppose you are choosing between two models for a support assistant that answers from a documentation set.

  • Model A: low input price, high output price, 8k output limit, no cached input.
  • Model B: higher input price, lower output price, 32k output limit, cached input for repeated context.
  • Model C: cheap for both, but often needs a second attempt to follow the format.

Your request sends 20,000 tokens of documentation and returns 400 tokens. Model A looks cheap on input and expensive on output; for this request it may still win. Model B's caching changes the math if the documentation prefix repeats across requests. Model C's retry rate can erase its price advantage.

The correct answer depends on the workload, not on the table. Measure one real request for each model, then multiply by your monthly volume. That is the comparison the model card cannot make for you.

A short checklist

Before you choose a model from a card:

  • Does the context window cover input plus output plus reasoning, and what is the separate output limit?
  • What are the input and output prices today, and do cached or batch rates apply to your workload?
  • Are hidden reasoning tokens billed, and can you cap them?
  • How often does the model need a retry for your task?
  • If self-hosting, what does the hardware and operations cost at your utilization?
  • Does the benchmark reflect your task, or only a leaderboard average?

A model is a recurring cost and a data-handling decision, not a single score. Read the numbers as a starting point, then measure the request you actually send.