GPU tiers: what actually fits

VRAM tier is the whole game. Bandwidth decides how fast things run
inside a tier, but capacity decides which models are even on the menu.
Below is the practical map as of this writing: what genuinely fits at each
tier, with representative cards. Tiers are the stable knowledge here —
specific SKUs get renamed every product cycle, the VRAM brackets have
barely moved in a decade.

The tier table

Tier notes:

• 6–8 GB is the entry tier. 7–8B Q4 assistants are genuinely useful;
  context beyond 8k starts to pinch before anything else does.
• 12 GB is the price sweet spot. 14B Q4 is the first serious quality
  jump over the 8B class, and it fits with real context.
• 16 GB gets you 14B at Q6 (near-lossless) or a 24B at low-bit quants.
  32B at IQ3 runs, but 3-bit on 32B is where quantization damage shows.
• 24 GB is the local-LLM classic. 32B Q4/Q5 with 16k of context is the
  best quality-per-gigabyte point for a single user. 70B at IQ2_XS fits
  the brochure and disappoints the ears — too few bits for that many
  parameters.
• 32–48 GB+ exists for 70B at Q4 or better, and for serving several
  users at once (see Concurrency).

The used market

The RTX 3090 (24 GB, ~936 GB/s) remains the classic value pick as of this
writing: 24 GB for a fraction of a new flagship, and 32B Q4 fits with
headroom. Caveats: 350 W under load, blow-through coolers dump heat into
the case, and mining-era cards may need repasting. Honorable budget
mentions: RTX 3060 12 GB new, or a used 2080 Ti with the 22 GB memory mod
if you enjoy risk. Used datacenter cards (P40, Tesla T4 era) are cheap VRAM
but slow bandwidth, noisy, and lack modern feature support — fine for
tinkering, tiring for daily use.

Multi-GPU: capacity yes, speed mostly no

Splitting across two cards buys capacity, not bandwidth:

• Layer split (pipeline) — each GPU holds contiguous blocks of layers;
  only small activations cross the bus. Near-linear capacity gain, modest
  decode penalty. llama.cpp does this automatically when you spread
  layers across devices.
• Tensor parallel — every matmul is split across cards; needs fast
  inter-GPU links to pay off, and hobby stacks implement it imperfectly.
• Neither approach reaches the sum of both cards' bandwidth. Two 24 GB
  cards run a 70B at roughly one card's decode speed, not double. For a
  single user, one bigger card is almost always the better purchase than
  two smaller ones — buy the pair only for capacity you cannot get
  otherwise, or for concurrent serving.

Laptop GPUs

A laptop RTX 4070 fits exactly what a desktop 4070 fits — capacity follows
the VRAM config, not the chassis. What differs is power: sustained clocks
sit lower under a shared thermal budget, so expect decode roughly 20–40%
slower than the desktop sibling, and prefill proportionally slower. Fine
for travel and demos; not a reason to skip a desktop if this is daily
work.

Pro cards and NVLink

Workstation cards buy ECC, certified drivers, blower coolers, and large
VRAM configs — the capacity premium can be worth it, the rest is
irrelevant to inference. NVLink exists for tensor-parallel training
traffic; hobby serving is bandwidth-bound decode that barely crosses
between cards. Skip both unless a specific capacity (48/96 GB) demands
them.

Check before you buy

nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv

Then run the VRAM math for the models you actually
want before trusting any tier table — architectures shift, the formula
does not.

Related: VRAM math · Concurrency ·
Apple Silicon