Inference engines compared

The engine is the program that turns weights into tokens: it loads the file, runs the transformer forward pass, and usually speaks HTTP on top. The good news is that almost every engine runs the same model at roughly the same speed on the same hardware — engine choice is about workflow, format support, and concurrency, not raw tok/s. The bad news is the same: benchmark-shopping between them is mostly wasted effort.

The field

Engine	What it is	Formats	Hardware reach	Built for
llama.cpp	C/C++ inference core with a built-in server	GGUF	CPU, CUDA, ROCm, Metal, Vulkan — the broadest reach	The foundation; runs nearly everywhere
Ollama	llama.cpp wrapped with a model library, CLI, and REST	GGUF	Whatever llama.cpp does	Zero-friction desktop use
LM Studio	Polished GUI plus an OpenAI-compatible server	GGUF (and MLX on Mac)	Windows/macOS/Linux GUI	People who want buttons, not flags
vLLM	Python serving engine: paged attention, continuous batching	safetensors	CUDA (plus some ROCm/TPU support)	Datacenter-grade concurrent serving
SGLang	Serving engine with RadixAttention prefix caching	safetensors	CUDA	Fast structured output; agents sharing prefixes
exllamav2	GPU-only loader tuned for decode speed	EXL2, GPTQ	CUDA only	Maximum single-user tok/s on one consumer NVIDIA GPU
MLX	Apple's array framework; `mlx-lm` runs models on it	MLX	Apple silicon	The fast practical route on recent Macs
transformers (HF)	The reference implementation everything is ported from	safetensors	CPU, CUDA	Correctness checks, experiments, training plumbing

One lineage note: Ollama and LM Studio both run llama.cpp under the hood. Comparing them to llama.cpp is comparing packaging, not inference.

Also worth knowing: llama.cpp ships more than a server — quantization and conversion tools, a benchmark harness, and bindings for a dozen languages.

Deep cuts

vLLM was born from the paged-attention insight (see concurrency): treat GPU memory like an OS treats RAM, and far more requests fit. It is the default answer to "I have a GPU server and real traffic."

SGLang caches prefixes across requests (RadixAttention): if 40 agents share the same two-page system prompt, the prompt is processed once, not 40 times. It is also known for fast constrained decoding — see structured output.

exllamav2 is the sharp edge for exactly one niche: a single user on a single NVIDIA GPU who wants the most tok/s per bit of quantization. The server story and hardware breadth are thin; the niche is real.

MLX is Apple's ML framework, built for unified memory. As of this writing mlx-lm often beats llama.cpp's Metal path on larger models — close enough that on a Mac it is worth trying both (more in Apple silicon).

transformers is the slow, faithful reference: when a quantized engine does something odd, running the same model through it answers "is this the engine or the model?" It is also the plumbing under fine-tuning tooling (see fine-tuning).

Choose by need

You are...	Start with
Chatting at your own desk	Ollama or LM Studio
Scripting from the shell (pipes, cron, small tools)	Ollama's REST API, or `llama-server`
Building an app backend with light traffic	`llama-server` — one process, zero ops
Serving many concurrent users	vLLM or SGLang
Running agent fleets that share long system prompts	SGLang or vLLM; see agents
An all-Mac household	Ollama/LM Studio first; MLX if you want more speed

Two notes on the table:

"Start with" is not "settle for" — every desktop engine reads the same GGUF, so trying a second one costs five minutes, not a migration.
When you do compare, hold everything constant: same model, same quant, same prompt, model already warm.

The format split mirrors the desktop/server split

GGUF versus safetensors (see model formats) is not just a container preference — it decides your engine. GGUF is the single-file, pre-quantized, runs-anywhere format, and the entire desktop ecosystem (llama.cpp, Ollama, LM Studio) is built on it. safetensors is the sharded-tensor format of the Hugging Face ecosystem; the high-throughput server engines (vLLM, SGLang, transformers) consume it, usually quantized on the fly (AWQ/GPTQ variants) rather than pre-packed.

So the real decision tree is short: desktop and single-user → GGUF engines; server and concurrent → safetensors engines. Everything else is detail.

When something "cannot run," check the format first — half of "engine X cannot run model Y" is a GGUF/safetensors mismatch, not a capability gap.

Overhead is not the story

On identical hardware with an identical quant, engines land within roughly 10–20% of each other for single-stream decode — often because they share a backend. What actually moves your experienced speed by multiples, not percent:

Whether the model is already loaded — a cold load costs seconds to a minute.
Whether the whole conversation is resent every turn — context cost, see tokens.
Whether requests queue behind each other — see concurrency.
Whether layers sit on the GPU — see engine knobs.

Same GPU, same 8B Q4_K_M, single stream (illustrative, not specs):
llama.cpp / Ollama / LM Studio    ~130–145 tok/s   (same engine underneath)
exllamav2 EXL2 ~4bpw              ~160–180 tok/s   (loader tuned for decode)

Same GPU, 8 concurrent chat streams:
llama-server serial queue         each stream waits; total throughput flat
vLLM                              all streams near single-stream speed

Pick the engine that matches your workflow, then spend your energy on I/O patterns instead of engine benchmarks.

One file, many engines

A GGUF you downloaded for Ollama also runs in llama.cpp directly and in LM Studio — engines do not convert or re-download; you point them at the same file (Ollama can import a local path with a one-line Modelfile). The same holds server-side for safetensors between vLLM and SGLang. Practical consequence: there is no file-level vendor lock, and being wrong about an engine costs you minutes.

That is why this page keeps pushing workflow over benchmarks: when your needs change, the models you already downloaded move with you.

When to graduate from Ollama

Ollama's defaults are excellent until they are not. Move to a serving engine when:

You need sustained concurrency beyond a handful of streams (see concurrency).
You want prefix caching across many agents sharing a system prompt (SGLang).
You must serve AWQ/GPTQ safetensors quants directly from the HF ecosystem.
You are load-testing a product feature before shipping it.
You need multi-GPU tensor parallelism to fit one big model across cards.

Until one of those is true, the convenience is worth more than the throughput.