While a model generates, it stores the key and value vectors for every token in the context so it does not recompute them at each step. That store is the KV cache. Its size per token is roughly 2 × layers × kv_heads × head_dim × bytes_per_element; the 2 is for the key and the value, and kv_heads is the number of key/value heads, which is often smaller than the attention head count because of grouped-query attention.
The cache is why long context is expensive even when the weights fit. A model with 32 layers, 8 KV heads, a head dimension of 128, and a sixteen-bit cache needs about 128 kilobytes per token, or 4 gibibytes for a single 32,000-token sequence. Two concurrent sequences double it. Quantizing the cache to eight bits halves the term, at a quality cost you should measure.
The local sizing guide works through the arithmetic. The short version: size for the context and concurrency you actually need, then add ten to twenty percent for runtime overhead.