Context windows & the KV cache

The context window has a price tag, and it is paid in VRAM. Every token
kept in context forces the model to remember two small vectors per layer,
and that memory — the KV cache — grows with every token and quietly eats
the headroom you thought you had. It explains the strangest local-LLM
failures: the OOM at turn forty, the model that seemed to get slower as
the chat grew, the 8 GB card that runs "128k context" in the brochure
only.

What the cache stores

Attention needs each past token's key and value projections before it can
attend to that token. They are computed once during prefill and kept for
the life of the request: per layer, per token, one K vector and one V
vector of kv_heads × head_dim numbers each. That store is the KV cache,
and it is why context costs memory linearly, forever, for every request
being served.

The formula

bytes per token = 2 (K and V) × layers × kv_heads × head_dim × 2 bytes (fp16)

Worked example, llama-3-8B-class GQA (32 layers, 8 KV heads, head_dim 128):

2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KB per token.

At 32k context that is 32,768 × 128 KB ≈ 4 GB of KV cache — comparable to
the Q4 weights sitting next to it. Leaner configs with 4 KV heads land
near 2 GB per 32k. Order of magnitude for 8B-class models at fp16 KV:
budget 0.5–4 GB per 32k depending on architecture. Exact numbers for a
checkpoint come straight from its config.json — num_key_value_heads,
num_hidden_layers, and head size.

GQA is why long context got cheap

Grouped-query attention shares a handful of KV heads across many query
heads — say 32 query heads served by 8 KV heads — shrinking the cache 4×
versus full multi-head attention at negligible quality cost. Without GQA
the table below would be 4–8× larger and long context on a 24 GB card
would be a fantasy. It is the single biggest architectural gift to local
long-context work, and the reason modern 7–9B models ship 32k–128k
windows at all.

How fast it grows

An 8B-class GQA model at fp16 KV, 128 KB per token:

Memory grows linearly per token, but attention compute grows with the
square of context — so long contexts cost VRAM and time-per-token at the
same time.

Levers when KV explodes

• Quantize the cache. q8_0 halves it, q4_0 quarters it, at modest
  quality cost — often the best capacity trade in the whole stack. In
  llama.cpp, KV quantization requires flash attention, shown below.
• Serve shorter context. -c 8192 costs nothing if your prompts are
  2k. Allocating the advertised 128k "just in case" reserves VRAM you
  will never productively use.
• Sliding-window models. Some architectures cap the KV window per
  layer regardless of nominal context — long-context behavior without the
  linear bill, at some recall cost on old material.
• Flash attention. Fundamentally a compute and kernel-efficiency win
  — exact attention without the quadratic activation blow-up — and in
  llama.cpp it is also the gate that unlocks KV quantization.

llama-server -m model-8b-q4_k_m.gguf -c 65536 -fa --cache-type-k q8_0 --cache-type-v q8_0

Advertised vs effective context

A model advertising 128k will run at 128k — the cache fits and the math
works. Whether it reasons at 128k is a different claim. Stretched RoPE
and long-context fine-tuning keep needle-retrieval scores high while
broader comprehension often sags well before the limit. Treat the
advertised window as a physical maximum, assume quality softens in the
last half, and look for community long-context evals of the specific
checkpoint before betting a workflow on it.

Lost in the middle, and hygiene habits

Position bias is real: models attend most strongly to the beginning and
end of the prompt and sag in the middle. Put instructions and critical
facts first or last; never bury the spec the model must follow inside a
pasted document. Habits that keep contexts small and sharp:

• Trim or summarize old turns instead of replaying the whole chat.
• Retrieve and inject the relevant chunk, not the entire file.
• Keep the system prompt short; it is re-sent with every request.
• Watch VRAM during long sessions — KV growth is the silent variable
  (VRAM math), and the serving flags that control it
  are catalogued in Engine params.

Related: VRAM math · Engine params ·
Speed