Quantization, in depth

Quantization shrinks weights from 16-bit floats to fewer bits, trading a little quality for a lot of memory. The one-line version (Q8 ≈ lossless, Q4_K_M ≈ sweet spot, below Q3 hurts) is enough to get running. This page goes one level down: what the tags actually do, how K-quants and I-quants differ, and where quality loss shows up first.

Why 4-bit works at all

Trained weights are not uniform noise. Values in each tensor cluster in a narrow, roughly bell-shaped distribution, and the network's behavior is carried by the shape of that distribution — not by the last decimal digit of any single weight. That makes weights compressible: store each weight as a small integer plus a shared scale factor, and you lose only fine detail.

One real complication: outlier channels. A small fraction of channels carry activations far larger than the rest. A single scale for the whole tensor spends its entire range on those outliers and crushes everything else into mud. Every serious quant scheme is fundamentally an outlier-handling scheme — per-block scales, activation-aware protection (AWQ), or calibration against real data (imatrix).

Block quantization mechanics

The core trick: chop each tensor into small blocks that share one scale. Dequantizing a weight is a single multiply:

w ≈ s · q                       # s: block scale, q: stored integer

Q4_0:     block = 32 weights → 1×fp16 scale + 32×4-bit ints ≈ 4.5 bpw
K-quants: super-block = 256 weights
          1×fp16 super-scale + 8×fp8 sub-scales + ints ≈ 4.8 bpw

bpw = bits per weight, including scale overhead. That overhead is why nothing is exactly 4.0: a 4-bit weight is half a byte, and a 2-byte scale spread over 32 weights adds 0.5 bpw. K-quants amortize the cost with two-level scaling — 256-weight super-blocks whose 8 sub-blocks get cheap fp8 scales — and land at ~4.8 bpw with better quality than Q4_0 at similar size.

K-quants and the _M / _S suffixes

Q4_K_M is not "Q4_K with a hat" — the suffix is a mix. The quantizer bumps specific tensors up to a higher quant while the bulk stays at Q4_K: typically attention projections (wv, wo) and sometimes the token embedding table go to Q6_K. The suffix ranks the mix:

• _S (small) — fewest tensors bumped, smallest file
• _M (medium) — the default recommendation, best size/quality balance
• _L (large) — more tensors at higher precision, diminishing returns

The mix exists because layers degrade unevenly: attention projections and embeddings are more sensitive than the big feed-forward matrices. A few hundred extra MB there buys real quality — which is why _M is the community default rather than plain Q4_K.

I-quants: calibrated, smaller, CPU-slow

IQ quants (IQ4_XS, IQ3_XXS, IQ2_XS…) add a calibration step. First compute an importance matrix (imatrix) by running the model over a sample of representative text; then place the quantization levels where the weights that matter actually live. The payoff: a given quality at smaller size than the equivalent K-quant — IQ4_XS undercuts Q4_K_S at equal-or-better quality, and IQ2/IQ3 make sub-3 GB 7–8B models possible at all.

The tradeoff is decode speed. I-quant dequantization is not a plain multiply — it needs non-power-of-two lookups and structured reconstruction. On CPU that extra math is visible in tokens/sec; on GPU it hides behind memory latency and costs almost nothing. Rule of thumb: K-quants for CPU boxes, I-quants fine on GPU, IQ2/IQ3 only when VRAM forces your hand.

The quality ladder, honestly

Approximate file sizes for a 7–8B model. Scale roughly linearly with parameters — a 14B is about double:

Treat exact percentages as folklore-grade: the loss depends on the model, the language, and the task. The ordering is stable even when the magnitudes aren't.

Where quantization hurts first

Damage is not uniform across capabilities. Roughly in order of falling:

1. Factual recall — names, dates, numbers drift first. A Q3 model misattributes confidently.
2. Math and multi-step reasoning — small errors compound through the chain.
3. Long-context coherence — retrieval from deep in a long prompt gets shakier than short-prompt behavior suggests.
4. Lower-resource languages — Korean degrades earlier than English at heavy quants; Korean tokens were rarer in training, so there is less precision slack. If Korean matters to you, hold Q5+ where you can.
5. Chat fluency and style — barely shows at all. A Q2 model still sounds fine, which is the trap: fluency is a bad proxy for damage.

Small models suffer more

Quantization error eats redundancy, and small models have the least of it. A 70B at Q4 loses nearly nothing; a 3B at Q4 loses visibly more; below ~2B, heavy quants can gut the model. Practical rule: Q6_K or Q8_0 for anything under ~4B parameters — the absolute GB saved by crushing a small model are tiny anyway. Get the size-vs-capability baseline in parameters before deciding where to spend bits.

The KV cache is a separate decision

Quantizing the weights does nothing to the KV cache — that is its own memory line (fp16 by default, separately quantizable to q8_0/q4_0 in llama.cpp). At long context the cache can rival the weights in size; budget it independently in KV cache.

A practical decision ladder

Worked example — 12 GB card, 8B dense model, 8k context:

weights    Q4_K_M                 4.9 GB
KV cache   8k tokens, fp16        1.0 GB    (8B GQA ≈ 128 KiB/token)
overhead   compute buffers + ctx  ~1.5 GB
─────────────────────────────────────────
total                             ~7.4 GB  → fits with room to spare

Given VRAM math, the ladder:

1. Budget: VRAM − context − overhead = what's left for weights.
2. Q6_K fits → run Q6_K. Only Q4_K_M fits → run Q4_K_M without guilt.
3. Drop to IQ4_XS/Q3 only to fit context or avoid CPU offloading — offload costs more speed than one quant step costs quality.
4. If your target model doesn't fit at Q4, drop a size class rather than quantizing to Q2: a solid 8B at Q4 usually beats a crushed 14B in practice.

Related: VRAM math · KV cache · Parameters · Model formats