Choosing the right model

Choosing a model is a function of three inputs: your memory budget, your use case, and your quality floor. Benchmarks only inform the third; the first two are arithmetic. Optimizing any one while ignoring the others is how people end up with a model that benchmarks beautifully and can't do Tuesday's actual task.

The three inputs

1. Budget — VRAM on a GPU, unified memory on Apple Silicon. Rule of thumb: weights at Q4_K_M ≈ 0.6 GB per billion parameters, then add KV cache and overhead — the full method is VRAM math.
2. Use case — what the model must do daily; decode below.
3. Quality floor — the level under which output is useless to you. A writing companion tolerates more slop than a code generator whose output must compile.

Budget also caps context: a huge-context plan on a small card just means trading KV-cache pressure against weight quality. Everything trades.

Worked example, 16 GB GPU → ~13.5 GB usable after the OS takes its cut: a 14B at Q4_K_M (~8.5 GB weights) with generous context, or an 8B at Q6_K (~6.6 GB) with huge context. Both fit; the use case decides between them.

Start from the use case

• General chat and writing — a mid-size instruct (8–14B class) at Q5–Q6 covers most people. Past that tier, style preference matters more than benchmark points.
• Coding assistant — favor the strong coder lineages (Qwen Coder, DeepSeek-Coder descendants) and keep context long: the value is filling in mid-file with repo context around it. Real code also punishes sloppier quants sooner than chat does.
• Long documents — verify that claimed context is usable context. Many "128k" models attend well to the front and back and mush the middle. Test before trusting; see the protocol below.
• Korean / multilingual — Qwen first, Gemma second. Both handle Korean respectably; English-first families show cracks in grammar and honorifics. Details in model families.
• Reasoning-heavy work — prefer reasoning-tuned checkpoints (Qwen3 thinking mode, R1-distills); they trade latency for correctness on math and planning, and quantization hurts them more — keep the quant high.
• Cheap bulk extraction — a small instruct (1.5–3B) with structured output beats a big model at several times the cost per document. Don't spend an 8B on regex-shaped work.
• Agents and tool use — the real requirement is instruction-following and format discipline, not raw IQ. In practice that means ≥7–9B class, more if you can afford it: one malformed tool call can wreck a whole chain. See agents.

Reading benchmarks without being fooled

Habits that keep you honest: prefer relative deltas on the same harness over absolute numbers; treat sub-2-point gaps as noise; distrust claims without a reproducible setup; and read the contamination discussion around a benchmark, which is often more informative than the leaderboard itself.

The cheapest defense is triangulation: a genuinely good model shows it across independent signals — lab evals, arena, community threads, and your own prompt set. When signals disagree, trust the ones closest to your workload. Your 20 prompts outrank every leaderboard for your purposes, because they sample your actual distribution.

Local-community signal

Benchmarks run on labs' prompts. Your better signal is people running the same file you're about to download:

• Hugging Face trending and download counts — slow-moving popularity, expensive to game for long.
• Which releases the community quantizers (bartowski, mradermacher) quantize across the full ladder — a decent importance filter.
• r/LocalLLaMA threads — search the model name, sort by recent; the recurring praise and complaints are real. Apply "as of this writing" doubly here: local consensus has a half-life of weeks.
• Quant-specific reports — quality of model X at Q4 is a different question from model X at FP16, and both differ from what the lab reported.

Recommendation matrix

As-of-this-writing starting points by VRAM tier, assuming roughly Q4_K_M quants and moderate context — cross-check hardware in GPU tiers:

Treat the matrix as a search starting point, not gospel — the field re-ranks every quarter. The stable advice within any tier: prefer newer generations, and prefer bigger-at-Q4 over smaller-at-Q8 down to the Q4 floor (below that, quantization damage wins — see quantization).

One more stable rule: when two models tie, pick the friendlier license. Apache-2.0 and MIT never come back to bite; community licenses sometimes do — see open vs closed.

Try before you commit

A disciplined 30 minutes beats a week of vibes:

1. Build a fixed prompt set of 10–20 — your real hardest cases: a Korean email to rewrite, a buggy diff to fix, a document summary with one checkable fact buried in it.
2. Run every candidate on the same prompts, same sampling settings, same quant level. Vary one thing at a time.
3. Blind A/B: shuffle outputs and judge without knowing which model made which. A spreadsheet and a friend works; so does your strongest current model as judge.
4. Check Korean specifically: honorific consistency (존댓말/반말), natural particles, no English calques. Models that "know Korean" often write translated English in Korean clothing.
5. Check long context: bury a fact 70% deep in a long paste and ask for it. That single test disqualifies more "128k" models than any leaderboard.
6. Re-run ties at temperature 0 — sampling noise masquerades as model differences more often than anyone admits.

The harness can be four lines of shell against a running llama-server:

while IFS= read -r prompt; do
  curl -s http://localhost:8080/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"messages":[{"role":"user","content":"'"$prompt"'"}],"temperature":0.7}' \
    >> out-model-a.jsonl
done < prompts.txt

Same loop, one output file per model; then read them side by side with the labels hidden.

When a closed API still wins

Local is about control, privacy, and marginal cost at volume — not frontier capability. As of this writing, the strongest closed APIs still beat the strongest open weights on hard reasoning and long agentic chains, with zero setup: no VRAM, no engine tuning, no quant math.

On cost, hedged numbers: a local 8B at Q4 is effectively free per token once the hardware exists; a frontier API charges on the order of a few dollars per million tokens. At hobby volume both round to zero; at millions of tokens a day, local wins on price alone. The capability gap — not the price — is the honest reason to keep an API key around.

A durable pattern is hybrid: local model as the default, API for the few prompts that stump it. Weigh the tradeoff consciously in open vs closed.

Related: GPU tiers · Model families · Agents · Open vs closed