What parameter count buys you

Parameter count is the best one-number summary of a model: it buys knowledge capacity and skill, and it charges you in memory and speed. Below ~30B you can feel every rung of the ladder; above it, returns flatten. This page maps the ladder and tells you which rung your task actually needs.

The capability ladder

A 30–40B at 4-bit is the local sweet spot as of this writing: it fits 24 GB cards and stays close enough to frontier quality for most work. MoE models break the size heuristic — judge them by active parameters for speed, total parameters for memory.

What parameters buy: knowledge capacity

Facts live in weights — overwhelmingly in the FFN layers (see inside the transformer). Capacity scales with parameter count:

• Famous facts (capital cities, common APIs): reliable from ~3B up.
• Working knowledge (mid-popularity libraries, secondary topics): want 14B+.
• Long tail (obscure specs, niche history): 30B+, and even then expect gaps.

Small models do not just know less — they patch the gaps with fluent invention, so hallucination rate rises as size falls. For long-tail questions on small models, retrieval beats parameters (RAG).

Read the suffix: base vs instruct

The same parameter count ships at different behavioral stages:

• Base: bare family name (Qwen2.5-7B). Next-token completion. Ask it a question and it may answer with more questions, mimic a forum thread, or ramble onward.
• Instruct/chat: suffixed -instruct, -it, or -chat (Qwen2.5-7B-Instruct). Answers questions, follows instructions, stops. Tuned for it in later training stages (see training).

prompt: "How do I boil an egg?"

base:
  "How do I boil an egg? 10 points for best answer!?"
  (continues the document — you gave it text, it gave you more text)

instruct:
  "Place eggs in a pot and cover with cold water. Bring to a
   boil, cover, and remove from heat: 6 min soft, 10–12 hard."
  (answers, formats, stops)

95% of local use wants the instruct variant. Base models earn their keep for completion-style workloads: autocomplete, text infilling, creative continuation, and as the substrate for your own fine-tuning.

More suffixes you will meet in repo names:

• -base: explicitly the pretrained stage (fine-tuning substrate).
• -coder / -math / -vl: tuned for code, math, or vision-language input.
• MoE naming like 235B-A22B: total parameters, then active per token — read it as "235B resident, 22B working."
• -gguf or a quant tag: a repackaging, not a new model (see model formats).

Diminishing returns per doubling

Each doubling of parameters adds fewer benchmark points than the last, while memory cost grows linearly and decode speed drops with every gigabyte — all of it streams through VRAM per token during decode (prefill vs decode). The 7B → 14B step is transformative; 70B → 140B is marginal for most tasks. The rule: run the smallest model that passes your tests.

What size costs in speed

Decode streams the whole model per token, so on the same card bigger means slower, roughly inversely with weight bytes. Order-of-magnitude single-user decode on one 24 GB card (~1 TB/s bandwidth, 4-bit weights):

Numbers move with bandwidth, drivers, and batch size; the shape does not: twice the weights, half the tokens per second (speed, GPU tiers).

Small models are more quantization-sensitive

A 30B model at 4-bit loses almost nothing; a 1B model at 4-bit loses a lot. Redundancy grows with size, so big weights absorb coarse rounding that small weights cannot:

• 30B+: Q4 is nearly free, Q3 often fine.
• 7–14B: Q4/Q5 are safe; watch Q3 closely and benchmark it.
• Under 4B: prefer Q6/Q8/FP16 — the savings are small anyway.

Details and per-format behavior in quantization; how to pick per family in choosing.

Related: quantization · choosing · model families · training