Prefill vs decode: how generation runs

Press Enter and generation runs in two phases with completely different physics. Prefill processes your entire prompt in one parallel pass — compute-heavy, fast per token, but you wait for all of it before the first word appears. Decode then emits the answer one token at a time — every single step re-reads essentially all model weights from memory to produce one token. "First token slow, then it streams" is not a quirk of your setup; it is these two phases.

Phase 1: prefill — the prompt

• All prompt tokens go through the model simultaneously: big parallel matrix multiplies, exactly what GPUs are built for. Utilization is high; tokens per second are high.
• You feel it as time-to-first-token (TTFT): the silence after you hit Enter.
• The pass also fills the KV cache with the key/value vectors of every prompt token — the scratchpad decode will read.
• Longer prompts, linearly longer TTFT: big system prompts, RAG documents, and chat history all prefill every time unless your engine caches prefixes.

Phase 2: decode — the answer

• One token per forward pass. That pass needs every weight; producing a single token means streaming the entire model from VRAM.
• The ceiling is memory bandwidth, not compute: roughly decode tok/s ≈ effective bandwidth ÷ bytes of weights. 8 GB of weights on a card with ~400 GB/s effective ≈ 50 tok/s ceiling.
• This is why quantization speeds up decode: fewer bits per weight means fewer bytes to stream each step (see speed).

Why the asymmetry: prefill processes thousands of tokens per weight-read (huge matrices, arithmetic intensity high — compute-bound), decode processes exactly one (every byte of weights fetched for a single token — bandwidth-bound). Same weights, opposite bottlenecks.

total_time ≈ prompt_tokens / prefill_tok_s  +  output_tokens / decode_tok_s

Worked example

prompt:  1,000 tokens    prefill 2,000 tok/s → 0.5 s   (TTFT)
output:    300 tokens    decode    40 tok/s → 7.5 s
total ≈ 8.0 s — 94% of wall-clock is decode

• The first half-second processed 1,000 tokens; the next 7.5 seconds produce 300. Same model, same GPU, 50× difference — different phase, not different effort.
• 40 tok/s is roughly 160 English characters per second (≈4 chars/token): faster than anyone types, which is why streaming reads comfortably (tokens).
• The same 40 tok/s in Korean is roughly 40–60 chars/s: identical token rate, less text per token.
• Output length is a decode-side dial: max_tokens caps how long generation runs, stop sequences end it early. Both change wall-clock linearly (engine parameters).
• Streaming changes perception, not physics: you watch decode happen instead of waiting for it, at the same 40 tok/s.

Worked example 2: the RAG case

prompt: 12,000 tokens (system + 20 retrieved chunks)  prefill 1,500 tok/s → 8.0 s
output:   500 tokens                                    decode    40 tok/s → 12.5 s
total ≈ 20.5 s — TTFT alone is 39% of the wait

This is the shape of retrieval-heavy local setups: fat prompts mean the model "thinks" for seconds before typing. Cures, in order of leverage:

• Retrieve less — top 3–5 chunks usually beat top 20 (RAG).
• Keep the system prompt stable so the engine can reuse a cached prefix.
• Trim history: old turns can be summarized instead of replayed in full.

The KV cache: the growing scratchpad

Without a cache, generating token N would recompute attention over all N−1 earlier tokens from scratch — every step redoing identical work, total cost quadratic in length. Instead:

1. Prefill computes keys and values for all prompt tokens and stores them once.
2. Each decode step computes only the new token's query/key/value and appends its K/V to the cache.
3. Attention for the new token is its query dotted against all stored keys.
4. One decoded token, in order: embed it, run the layers reading the cache, project to vocab, sample — append its K/V, repeat.

Now each decode step is linear in context length (read the cache) instead of quadratic (recompute everything). The price is the cache itself, which grows with every token:

KV bytes ≈ 2 × layers × kv_heads × head_dim × bytes_per_value × context

8B-class model (32 layers, 8 KV heads, 128 head dim, FP16):
  2 × 32 × 8 × 128 × 2 B ≈ 128 KiB per token
  → ~2.6 GB at 20k context, ~17 GB at 128k context

That cache competes with weights for VRAM — long-context chats crash boxes that run the same model fine with short prompts (KV cache, VRAM math).

Batching changes the economics

Reading 8 GB of weights to emit one token is wasteful; the same reading can serve many sequences at once. Batch 8 requests and one weight-stream produces 8 tokens — throughput climbs near-linearly until compute saturates. This is the entire trick behind serving (serving APIs, concurrency). Solo on a laptop you pay full freight per token; servers amortize it across simultaneous users.

Reading the meters

Measure before tuning: every serious engine logs prompt-eval and generation speeds per request. One minute of logs tells you which phase eats your latency; guessing does not.

Tuning follows the phase: shorter prompts and prefix caching fix TTFT; quantization and smaller weights fix decode (engine parameters, troubleshooting).

Related: speed · KV cache · concurrency · tokens