The physics of tokens/sec

Token generation speed is not mysterious: during decode, every single token requires reading essentially all the active weights from memory once. That makes decode memory-bandwidth-bound, and it means you can predict your tok/s with division. Everything people argue about — quantization, MoE, offloading, batching — is this formula wearing different hats.

The one formula

tokens/sec ≈ effective memory bandwidth ÷ bytes of weights read per token

RTX 4090, 8B Q4_K_M:
  1008 GB/s ÷ 4.9 GB ≈ 205 tok/s theoretical ceiling
  observed: ~120–150 tok/s (overhead, attention, KV reads)

Sanity-check table for one common case (8B at Q4_K_M, single stream — order-of-magnitude numbers, not specs; measure your own):

Why the gap between ceiling and reality? Kernel launch overhead, attention work that grows with context, and KV cache reads on top of weights. The formula gives you the ceiling; you typically realize 60–75% of it single-stream.

One card, four models

The formula also predicts how model size trades against speed on fixed hardware. Ceiling on a ~1,000 GB/s card, then 60–75% realized (illustrative):

Two lessons fall out. Doubling parameters roughly halves speed on the same card. And a 32B on a 24 GB card is still fast enough for chat — memory-bound is not hopeless. That pairing is why 24 GB cards and 32B Q4 quants show up together constantly (see GPU tiers).

Quantization is a speed feature, not just compression

Same formula, fewer bytes: an 8B model at fp16 reads ~16 GB per token; at Q4_K_M it reads ~4.9 GB. On the same card, Q4 runs roughly 3× the fp16 speed and fits where fp16 does not. This is why Q4-class quants are the default recommendation for local use (see quantization) — quality loss is small, and the speed gain is structural, not incidental.

Context slows you down

As the conversation grows, each new token must also read the KV cache for every previous token — those reads scale with context length. Near an empty context an 8B Q4 might do 140 tok/s on a 4090-class card; deep into a 64k-token session the same setup can sag to a half or a third of that. Attention kernels also stop fitting cache-friendly shapes. Long-context workloads are where flash attention and KV quantization pay for themselves (see KV cache and engine knobs).

A common confusion: "the model got slower as the chat went on" is usually KV growth, not the server degrading (see KV cache).

MoE: fast-for-capability

Mixture-of-experts models activate only a few expert blocks per token, so bytes read per token track the active parameters, not the total. A 30B-total/3B-active model at Q4 reads only a couple of gigabytes per token — it decodes like a small model while carrying a large model's knowledge. That is the whole appeal of the MoE families for local use (see model families): you pay memory to store the experts, but bandwidth only for the ones you use.

Concrete shape as of this writing: the Qwen3-30B-A3B class reads only its ~3B active parameters per token — large-model knowledge at small-model speed, for ~18 GB of VRAM at Q4.

Speculative decoding

Pair a small draft model with the big target: the draft proposes k tokens cheaply, the target verifies all k in one forward pass, accepted tokens come out at the target's quality. Ongreedy-ish workloads — code completion, JSON, extraction at low temperature — expect 2–3× decode speedup. At high temperature the draft guesses wrong more often and the gain shrinks. llama.cpp exposes this via --model-draft and draft-length flags; exact flag names drift between versions, check --help.

Costs: the draft must share the target's tokenizer, and both models sit in memory.

Parallelism: the reality

Two GPUs do not give you 2× a single stream. Tensor parallelism (vLLM -tp 2) splits each layer across cards for throughput, at the cost of interconnect traffic; llama.cpp's default split is pipeline (layer-wise), which helps capacity more than single-stream latency. Realistic expectation for a good two-card split: 1.2–1.6× single-stream, not 2×. If someone promises linear scaling per GPU, they are selling something.

Adding cards does scale capacity nicely, though — that is the server story in concurrency.

The CPU-offload cliff

When layers stay on CPU, every token's forward pass hops across PCIe (~10–25 GB/s) into system RAM and back. The loop runs at the speed of its slowest resident part: a model with 80% of layers on GPU runs much closer to CPU speed than to "80% of GPU speed." If a model almost fits, drop one quant step instead of half-offloading — see engine knobs. The tell-tale of accidental partial offload is 3–10 tok/s on GPU-class hardware.

Prefill is compute-bound: TTFT is a different measurement

Processing your prompt is parallel across tokens and limited by compute, not bandwidth — a 4090-class card can prefill an 8B Q4 at a few thousand tok/s. So a 12k-token paste costs ~3–5 s before the first output token, even though decode afterwards runs at 140 tok/s. Time-to-first-token and steady-state tok/s are different numbers measuring different bottlenecks; when you read a benchmark, check which one it is showing.

Measure your own

Numbers on pages (including this one) are ceilings and ranges. Your setup is measured in two minutes:

1. Generate ~200 tokens from a short prompt; ignore the very first run (cold load).
2. Take steady-state decode: tokens generated ÷ seconds after the first token.
3. Separately note time-to-first-token on a long paste — it is a different bottleneck.

llama-server prints prompt-eval and eval tok/s per request; LM Studio shows decode tok/s live; nvidia-smi confirms the GPU is actually busy while you test.

How fast feels

The thresholds are surprisingly low because reading speed is only ~5–10 tok/s for humans — but waiting on a completion at 5 tok/s is misery. Feel it yourself: the simulator renders the same answer at different token rates, and one minute there teaches more than any table.

Below ~2–3 tok/s a streaming UI feels broken rather than slow — people close the tab, so treat that as a hard floor for interactive use.

Related: Hardware overview · VRAM math · GPU tiers · Engine knobs