What hardware matters (and what doesn't)

Local LLM performance comes down to three resources, and they are not
equal. VRAM capacity decides what you can run at all. Memory bandwidth
decides how fast generation feels. Raw compute mostly decides how long you
wait right after you press Enter. Rank them correctly and every hardware
decision becomes obvious; rank them wrong and you pay for teraflops you
never use.

The ranking: capacity, bandwidth, compute

1. VRAM capacity (GB) — the binding constraint. A model that does not
   fit does not run, or spills to system RAM and crawls. Capacity decides
   model size, quantization level, and context length simultaneously, and
   nothing else on the card compensates for missing it.
2. Memory bandwidth (GB/s) — how fast tokens come out. Every generated
   token reads essentially the whole model from VRAM, so decode speed is
   bandwidth divided by model size in bytes. Compute sits mostly idle
   during generation.
3. Compute (TFLOPS) — matters for prefill: digesting your prompt
   before the first token appears. More FLOPs means shorter
   time-to-first-token and more batched throughput when several requests
   share the GPU. It barely moves single-stream tokens per second.

Which resource buys which experience

Two formulas carry almost all of the intuition:

decode tok/s   ≈ effective bandwidth (GB/s) ÷ model size (GB)
prefill FLOPs  ≈ 2 × parameters × prompt tokens

Worked example. An 8B model at Q4_K_M is about 5 GB of weights. On a GPU
with ~1,000 GB/s of bandwidth the decode ceiling is 1,000 ÷ 5 ≈ 200 tok/s;
real engines land at 100–150 once overheads are paid. That same 2,000-token
prompt needs roughly 2 × 8×10⁹ × 2×10³ ≈ 32 TFLOPs of prefill — under a
second on any modern discrete GPU, slow enough to feel on CPU. The
simulator lets you poke at these trade-offs with real numbers.

Why a new GPU generation barely speeds up decode

Each GPU generation roughly doubles compute: more cores, higher clocks,
better tensor units. Prefill and TTFT improve accordingly — long-prompt RAG
and codebase-scale agents feel visibly snappier on newer cards. But the
memory system moves far less: GDDR6 to GDDR6X to GDDR7 buys maybe 20–50%
more bandwidth per generation, and bandwidth is what decode runs on. So a
new card digests prompts faster, then generates at nearly the same speed as
the previous tier with similar bandwidth. When comparing cards for local
LLMs, read the GB/s and VRAM columns first, TFLOPS last.

What mostly does not matter

• PSU headroom. LLM inference draws well below the card's gaming peak.
  A quality PSU rated for the card's official requirement is plenty; do not
  budget for a 1,600 W monster you will never load.
• Cooling. Only sustained multi-hour serving matters, where thermal
  throttling shaves clocks. Any decent dual- or triple-fan card holds up.
• PCIe lanes. x8 vs x16, Gen 4 vs Gen 5 — affects model load time and
  CPU-offload traffic, not generation. Ignore it unless you offload layers.
• NVLink and pro drivers. Hobby serving shuffles almost nothing between
  cards during decode. Nice to have, never worth paying for here.
• NPUs. The NPU in modern laptops and desktop SoCs is genuinely good at
  small vision and audio models, but as of this writing there is no
  practical LLM serving stack targeting it. Plan on GPU, CPU, or Apple
  Silicon — not NPU — for chat models.

Sizing mindset: model first, card second

Do not ask "what can my GPU run." Ask what tier of model you actually want,
then buy the cheapest card that holds it with headroom:

1. Pick the workload: casual 8B chat, a serious 32B coding/reasoning
   assistant, or 70B-class quality where budget allows.
2. Compute its footprint at a good quant with the context you really use —
   VRAM math has the formula and worked tables.
3. Match it to a card tier with 10–20% headroom to spare —
   GPU tiers maps tiers to models.
4. Only then optimize bandwidth within that tier.

Working forward from a GPU budget is how people end up with a fast card
that only fits models too small to be useful, or a giant card used at 40%
capacity. Budget the model, then the memory, and let compute sort itself
out. And before upgrading anything, check what you actually have:

nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv

Related: VRAM math · GPU tiers ·
Speed · CPU inference