Apple Silicon & unified memory

Apple Silicon plays a different game. The CPU and GPU share one pool of
unified memory, and the GPU can address almost all of it — so on a Mac,
buying RAM is buying VRAM. A 64 GB Mac holds every 32B and most 70B quants
that no consumer discrete card can. The catch is bandwidth: you get
laptop-class memory pipes, not terabyte-per-second GDDR.

Unified memory, concretely

On a PC, system RAM and VRAM are separate pools; spilling means PCIe round
trips at a fraction of VRAM speed. On Apple Silicon there is one physical
pool, so the GPU reads weights directly from "system" RAM with no copy and
no PCIe tax. Recent macOS versions cap GPU-usable RAM at roughly
two-thirds to three-quarters of physical by default; you can raise the
cap:

sysctl iogpu.wired_limit_mb                 # current GPU memory limit
sudo sysctl iogpu.wired_limit_mb=49152      # e.g. 48 GB on a 64 GB Mac

Leave the OS several GB regardless — pinning everything to the GPU makes
the whole machine miserable.

Bandwidth by chip class

Decode speed is bandwidth ÷ bytes per token, same arithmetic as anywhere
else. An 8B Q4 model is ~5 GB, so the ceiling is bandwidth ÷ 5:

Worked example: a ~550 GB/s Max chip has a ceiling of ~110 tok/s on 8B Q4
and delivers 70–95 in practice. Base chips are fine for small models and
painful at 30B+; Max and Ultra are the classes that feel like a discrete
GPU. Ranges are approximate — check the exact chip's spec before buying.

Second example, the one Macs exist for: a 32B Q4_K_M model is ~19.2 GB of
weights plus ~2.1 GB of KV at 8k, plus the OS. A 36 GB machine runs it
with a thin margin; 48–64 GB runs it with real context headroom. The same
model needs 24 GB of discrete VRAM before you count anything else — and
very few single cards have that. Capacity per unit cash is the Mac's whole
argument: the 36–128 GB bracket is territory consumer discrete cards
simply do not cover.

The ecosystem: MLX and llama.cpp

Both paths are healthy, and you will likely end up using both:

• llama.cpp (Metal backend) — runs everywhere, gets exotic quant
  formats first, and uses the same GGUF files and flags as your
  NVIDIA/AMD boxes.
• MLX — Apple's array framework. mlx-lm generates, serves, and
  converts models, and it is often first with Apple-specific kernels.
  Hugging Face hosts MLX-community conversions of most popular families.

Neither requires CUDA — which is also the structural limitation: vLLM,
SGLang, and TensorRT-LLM do not run on macOS as of this writing. For
OpenAI-compatible serving, use llama.cpp's built-in server or
mlx-lm server.

The limits

• Thermals. Max-class chips in a MacBook throttle under sustained
  generation; a Mac Studio holds clocks indefinitely. Laptops are for
  bursts, Studios for serving.
• No CUDA. If a tutorial says vLLM or SGLang, that path is closed.
  llama.cpp and MLX cover chat, embeddings, and serving well enough that
  it rarely matters — until it does.
• Quant lag. Bleeding-edge quants and kernels land in llama.cpp first;
  MLX conversions follow within days to weeks.
• Swap is not VRAM. When memory pressure pushes weights into swap,
  performance does not degrade — it collapses. Size models below physical
  RAM minus OS overhead. Full stop, no exceptions worth making.

Buying guidance

• 16 GB: 7–8B Q4 only. Casual.
• 24–36 GB: 14B comfortable; 32B Q4 on 36 GB with careful context.
• 48–64 GB: 32B Q4/Q5 with real context; 70B IQ3/Q4 at 64 GB with
  patience for its slowness relative to the weights moved.
• Max or Ultra for bandwidth — that is what decode speed buys. Base M for
  small models, Whisper, and vision.

Non-LLM bonus: these machines are excellent for Whisper transcription and
vision encoders. Unified memory plus strong SIMD units make audio and
image pipelines disproportionately fast per watt — a quiet Mac Studio is
a legitimate multimodal workhorse.

Related: CPU inference · Speed ·
Quantization