Downloading a model is easy. Downloading the wrong size is also easy. A card says "30B" and a number appears in your head; the process starts, the GPU allocates, and a few minutes later the runtime dies with an out-of-memory error. The parameter count was only the first term.
This guide is the arithmetic before the download. It covers the two numbers that decide whether a model fits, the context length that changes one of them, and the difference between fitting and serving well. The glossary entry on VRAM is the short version; this is the worksheet.
Two numbers decide whether it fits
GPU memory holds three things while a model runs:
- Weights. The parameters, at whatever precision the file uses.
- KV cache. The attention state for every token in every active sequence.
- Runtime overhead. Kernels, activations, temporary buffers, CUDA graphs, and fragmentation.
The weights are fixed for a given file. The KV cache grows with context length and batch size. The overhead is the number people forget, and it is usually the difference between a comfortable fit and a crash at the end of a long prompt.
Weights: parameters times bytes
At full sixteen-bit precision, each parameter costs two bytes, so a billion parameters cost about two gigabytes. The rule scales linearly:
| Precision | Bytes per parameter | 7B model | 30B model | 70B model |
|---|---|---|---|---|
| FP16 / BF16 | 2 | ~14 GB | ~60 GB | ~140 GB |
| FP8 / INT8 | 1 | ~7 GB | ~30 GB | ~70 GB |
| FP4 / INT4 | 0.5 | ~3.5 GB | ~15 GB | ~35 GB |
These are decimal gigabytes for the weights alone. Add ten to twenty percent for the runtime before you compare against the card's capacity. A 24 GB card does not hold a 24 GB model; it holds a smaller one with room to work.
Mixture-of-experts models complicate the count. Only some experts are active per token, which helps compute, but all of the experts still have to be resident in memory. A model with a huge total parameter count and a small active count is still a huge download. Do not divide the total by the active number and expect the weights to shrink.
KV cache: the number that surprises people
Every token in the context gets a key and a value vector in every layer. The size per token is:
bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_element
The leading 2 is for the key and the value. kv_heads is the number of key/value heads, which is often smaller than the attention head count on modern models because of grouped-query attention. head_dim is the dimension per head, usually hidden_size / num_attention_heads.
For a model with 32 layers, 8 KV heads, a head dimension of 128, and FP16 cache:
2 × 32 × 8 × 128 × 2 = 131,072 bytes per token
At 32,768 tokens that is 4 GiB for one sequence. Two concurrent sequences at the same length are 8 GiB. This is why long context is expensive even when the weights fit: the cache scales with the product of context length and concurrency.
Some runtimes quantize the KV cache to FP8 or INT8, which halves or quarters this term. Check whether yours does, and check the quality cost on your task. Quantized weights do not imply a quantized cache; they are separate settings.
A worked example
Suppose the model is 30B parameters at four-bit precision, and you want two concurrent conversations at 32,000 tokens.
- Weights: about 15 GB.
- KV cache: 4 GiB per sequence × 2 = 8 GiB.
- Runtime overhead: about 2 GB.
Total: roughly 25 GB. That fits a 32 GB card with headroom. It does not fit a 24 GB card once the driver and the desktop take their share. If you drop to one sequence at 32,000 tokens, the total falls to about 21 GB, which is still tight on 24 GB. If you drop the context to 8,000 tokens, the cache falls to 1 GiB per sequence and the same model becomes comfortable.
This is the trade you are actually making: context length and concurrency, not just parameter count. Decide those two before you shop for hardware.
Quantization is not a free swap
Four-bit weights are smaller, and on ordinary text they can be close to the sixteen-bit parent. The difference shows up on long reasoning chains, precise formatting, and code that must compile. Two four-bit files from different toolchains are not the same model; they choose which weights to approximate differently.
Treat a quantized build as a different artifact. Re-run the prompt you graded against the full-precision model. Record the file hash next to the download. If the answers diverge on the cases you care about, the memory you saved is not worth the regression. The open source and proprietary models guide covers the license side of the same download.
Throughput is a separate question
Fitting is necessary, not sufficient. Token generation during decoding is usually limited by memory bandwidth: every token requires reading the weights and the cache. Prefill, the phase that processes the prompt, is more compute-bound. That is why a model can fit and still feel slow, and why a long prompt can produce a long time to first token even on a fast card.
Three numbers describe serving, and they trade against each other:
- Time to first token, which is what an interactive user feels.
- Tokens per second per sequence, which is reading speed.
- Total throughput, which is what batching buys at the cost of per-request latency.
Measure all three with your real context length. A benchmark at 512 tokens says little about a 32,000-token request.
When the card does not list parameters
Many repositories omit the parameter count. Read config.json instead: num_hidden_layers, hidden_size, num_attention_heads, num_key_value_heads, and the torch dtype give you most of the arithmetic. The safetensors index lists every tensor and its shape, so you can sum the bytes without loading the model.
Do not infer size from the name. "Flash," "mini," and "pro" are marketing, and a name with a number in it may refer to a version, a context length, or nothing at all. If a listing has no parameter count and no config, treat the size as unknown and measure before you commit.
Measure before you buy
Rent an hour on the GPU class you are considering and run the model at the context length you need. Watch peak memory, time to first token, and tokens per second. Then decide whether the hardware is the right purchase or whether a hosted endpoint is cheaper at your utilization. For a small team, the local model often costs more in engineering time than the API it replaces, even when the GPU is already paid for.
The directory's model pages show VRAM where a source provides it, and the comparison tool puts those numbers side by side. Treat a published VRAM figure as a floor for the weights, not a promise about your context length.