Running on CPU only

CPU-only inference is viable, and the numbers are less depressing than
lore suggests — as long as you understand it is the same bandwidth game as
GPU decode, played with a tenth of the pipe. A modern desktop with
dual-channel DDR5 has ~80–100 GB/s of memory bandwidth. That is your
entire budget; nothing about the CPU cores themselves changes it much.

The formula, worked

decode tok/s ≈ effective bandwidth (GB/s) ÷ model size (GB)

Dual-channel DDR5-5600 moves ~90 GB/s on paper; call it 60–80 GB/s
effective once cores and prefetchers take their share. An 8B Q4 model is
~5 GB, so:

90 GB/s ÷ 5 GB ≈ 18 tok/s theoretical ceiling → 8–14 tok/s in practice.

That is not a bug — it is the same arithmetic a big discrete GPU uses,
with a tenth of the bandwidth.

What to expect by hardware

Comfort tiers on a dual-channel desktop:

• Pleasant: 7–8B Q4 chat and code help at 10+ tok/s.
• Usable: 13–14B Q4 at ~5–8 tok/s. You wait, not forever.
• Patience: 24–32B Q4 at ~2–4 tok/s. Overnight batches, not chat.
• Don't: 70B Q4 under 1 tok/s on dual-channel. Quad-channel
  workstations only, and even then think twice.

The escape hatch: partial offload

If a small GPU is present, llama.cpp splits layers between CPU and GPU —
say 20 of 32 layers on an 8 GB card. The GPU-resident fraction runs at VRAM
bandwidth, the rest at RAM bandwidth, so speed lands between the two
extremes. It beats either pure mode when the weights fit nowhere whole,
and it beats watching a "full GPU offload" thrash into shared memory.

Prefill is the real pain

Decode gets all the attention, but prefill is where CPU hurts: it is
compute-bound, and a desktop CPU has a small fraction of a GPU's FLOPs. A
4,000-token prompt on an 8B model can cost tens of seconds before the
first token appears. RAG-style pipelines with large injected contexts pay
it on every single request. Mitigations: keep prompts short, give batch
prefill more threads than decode, and treat indexing as a one-time cost
(see RAG).

Practical flags

# decode threads = physical cores; batch prefill can use logical cores too
llama-server -m qwen2.5-7b-instruct-q4_k_m.gguf -c 8192 -t 8 --threads-batch 16

# multi-socket box: interleave memory across NUMA nodes
numactl --interleave=all ./llama-server -m model.gguf -t 32

# what SIMD does this CPU have? AVX2 is the floor; AVX-512 helps prefill
lscpu | grep -o 'avx[0-9a-z_]*' | sort -u

• -t set to physical cores usually beats logical/SMT counts for decode;
  benchmark both on your machine, the crossover is workload-specific.
• AVX2 is the baseline everywhere; AVX-512 (Zen 4/5, recent Xeon) gives
  real prefill gains. llama.cpp detects support at build time.
• On multi-socket machines, NUMA pinning can be worth 20%+ — memory on
  the far socket is slow memory.
• Laptop DDR4 carries roughly half of desktop DDR5 bandwidth. Same
  formula, worse constant: expect 5–8 tok/s where the desktop gives 12.

When CPU-only is genuinely the right answer

• Embeddings and RAG indexing — throughput matters, latency does not;
  a few hours of chunk-embedding overnight is free.
• Tiny models in scripts — 0.5–3B classifiers, extractors, and
  rerankers answer in tens of milliseconds even on CPU.
• Batch background jobs — summarize the backlog at 3 tok/s while you
  sleep; nobody is watching the stream.
• Trying before buying — validate that a model family even does the
  job before committing to a GPU purchase.

If interactive chat is the goal and any GPU is available, put the model
there. Use First run to get moving today, and
Speed to understand what the numbers you see mean.

Related: First run · Speed ·
RAG