VRAM math

Every "will this run on my card" question is arithmetic, not folklore.
Total VRAM is the weights, plus the KV cache, plus a fixed slab of
activations and engine overhead. If you can multiply parameters by bits,
you can size a machine before spending anything on it. This page gives the
formula, worked tables, and a 16 GB walkthrough you can copy for any card.

The formula

total VRAM   = weights + KV cache + overhead
weights (GB) = parameters (B) × bits per weight ÷ 8
KV cache     = bytes per token × context length   (see [KV cache](/guide/kv-cache))
overhead     ≈ 1–2 GB (activations, driver context, buffers, fragmentation)

Effective bits per weight for the common GGUF quants: Q4_K_M ≈ 4.8,
Q5_K_M ≈ 5.7, Q6_K ≈ 6.6, Q8_0 ≈ 8.5. Multiply parameters by that, divide
by 8, and you are within a few percent of the file size on disk.

Rules of thumb worth memorizing, in GB per billion parameters:

• Q4_K_M ≈ 0.6 GB — the everyday workhorse
• Q5_K_M ≈ 0.7 GB
• Q6_K ≈ 0.8 GB — near-lossless to most eyes
• Q8_0 ≈ 1.1 GB — transparent, roughly double Q4 cost
• fp16/bf16 = 2.0 GB — training and serious fine-tuning territory

Worked examples

KV cache size depends on the architecture. Modern GQA models run roughly
128–256 KB per token at fp16 KV for the 8B–32B classes, and around 320 KB
for 70B-class. The exact formula lives on KV cache;
here we just add it in. Overhead is budgeted at 1.5 GB per row.

Q4_K_M (~4.8 bits per weight):

Q6_K (~6.6 bits per weight):

Two patterns to internalize. Q6_K costs ~35–40% more than Q4_K_M — often
the difference between one GPU tier and the next. And KV at 32k context is
not a rounding error: on small models it rivals the weights themselves.
Old MHA designs (Llama-2 era) carried 4–8× larger KV caches, which is why
32k context used to demand exotic hardware and GQA models do not.

OOM at load vs OOM mid-generation

• At load: weights alone exceed free VRAM. The engine refuses to start,
  or swaps layers to CPU and crawls. Honest, immediate, easy to diagnose.
• Mid-generation: weights fit, but the KV cache grows token by token.
  A fresh session works fine; forty turns later, after a RAG dump or a
  pasted file, the cache eats your headroom and the process dies
  mid-answer. This is the sneaky one — always size for the context you
  will actually reach, not the first prompt.

Watch it happen live:

watch -n1 nvidia-smi --query-gpu=memory.used,memory.total --format=csv

Sizing worksheet: the 16 GB walkthrough

1. Usable budget: 16 GB × 0.85 ≈ 13.6 GB. Desktop, browser, and driver
   already reserve 0.5–1 GB; 10–20% headroom is the rule, not a nicety.
2. Dream: 32B Q4_K_M — 19.2 GB of weights alone. Dead on arrival.
3. Realistic: 14B Q4_K_M at 8k context: 8.4 + 1.6 + 1.5 = 11.5 GB. Fits
   with ~15% headroom. Ship it.
4. Now you want 32k context: 8.4 + 6.4 + 1.5 = 16.3 GB. Over budget.
5. Three fixes — pick one or combine:
   • Quantize the KV cache to q8_0: 6.4 → 3.2 GB, total 13.1 GB. In.
   • Drop weights to Q4_K_S or IQ4_XS, saving another ~0.5–1 GB.
   • Drop to 8B Q6_K: 6.6 + 4.2 + 1.5 = 12.3 GB, roomy.

The same five steps work for an 8 GB card, a 24 GB card, or a 48 GB
dual-GPU box. Budget, dream, subtract, adjust.

Leave headroom, always

Engines allocate scratch buffers per batch slot, fragmentation accumulates
over a long-running server, and concurrency multiplies KV per active
request. A config that "just fits" in a one-shot bench test will OOM in
production. 10–20% free is the tax for staying up. For what fits at each
card tier in practice, see GPU tiers; for why the
quant bits are what they are, see Quantization.

Related: Quantization · KV cache ·
GPU tiers