Sampling settings

Sampling is the last inch of inference: the model scores every token in the vocabulary, and your settings decide how that distribution becomes one token. The knobs interact, so the order of operations matters more than any single value.

The pipeline, in order

Most engines apply samplers in roughly this sequence:

• Logits — raw scores, one per vocabulary token (50k–150k entries).
• Repetition penalties — tokens seen recently get their logits pushed down.
• Temperature — logits divided by T, sharpening or flattening everything.
• top_k — hard cap: only the k best tokens survive.
• top_p — nucleus: the smallest top set covering p of the probability mass.
• min_p — relative floor: drop tokens below min_p × P(top token).
• Sample — pick one token from what is left, append it, repeat.

A filter applied early changes what every later filter sees. Keep that in mind when stacking.

Temperature

Temperature T divides the logits before the softmax. Lower T sharpens the distribution toward the top token; higher T flattens it toward uniform. Concrete: logits [5.0, 3.0, 1.0] over three tokens.

At T → 0 this collapses to greedy: always argmax, fully deterministic.

Use temperature 0 (or 0.000001 where an engine disallows 0) for extraction, classification, and anything you want reproducible run-to-run. Chat wants 0.6–0.8; creative writing tolerates 0.9–1.2.

top_k: a hard cap

Keep only the k highest-probability tokens and sample among them. top_k 40 makes token 41 and below impossible, no matter what.

It is a blunt instrument. k=40 is roomy after "The capital of France is" and absurdly wasteful on rare-token positions where only two candidates matter. Common default: 40. Disable it (0 or -1, engine-dependent) when you use min_p.

top_p: the nucleus

Sort by probability, keep the smallest set whose cumulative probability reaches p, renormalize, sample.

Worked: probabilities [0.867, 0.117, 0.016] with top_p 0.9. The top token alone covers 0.867 < 0.9; adding the second reaches 0.984 ≥ 0.9. You sample from two tokens, renormalized to [0.88, 0.12].

The candidate set shrinks automatically when the model is confident and grows when it is unsure. Typical range: 0.9–0.95.

min_p: the local favorite

Keep only tokens whose probability is at least min_p × P(top token). The floor is scale-relative:

• Model 99% confident → floor = 0.05 × 0.99 ≈ 0.05: only real contenders survive.
• Top token at 30% → floor = 0.05 × 0.30 = 0.015: diversity survives.

That kills the classic top_k/top_p failures — over-pruning a confident distribution, under-pruning a flat one — and min_p composes safely with high temperature. The "creative but coherent" recipe: temp 1.0–1.5, min_p 0.05–0.1. Common local setting: min_p 0.05 with top_k and top_p off.

repeat_penalty (llama.cpp family)

Before temperature is applied, the logits of tokens that appeared in recent context get divided by the penalty. Values below 1.0 boost repetition instead. Sensible range: 1.0–1.1.

Side effects are real, because code and data repeat tokens on purpose:

• Identifiers legitimately repeat: count inside counter.count.
• JSON keys and closing braces repeat across every object in an array.
• Penalty 1.2+ breaks code and JSON (dropped braces, "corrected" quotes) long before it fixes prose loops.

Modern instruct models rarely need more than 1.05. For code and structured output, 1.0 is a legitimate choice.

frequency and presence penalties (API style)

OpenAI-compatible APIs expose frequency_penalty (grows with each repetition) and presence_penalty (flat penalty for any already-seen token) where llama.cpp uses repeat_penalty. Same intent, different math. Not every local engine honors both — check what yours implements before stacking them on top of each other: engine params.

Stacking samplers into a corner

temp 0.1 + top_k 10 + top_p 0.5 + min_p 0.2 + repeat_penalty 1.3, applied together, can mask every candidate token. Symptoms: NaN logprob warnings, empty generations, "sampling returned zero candidates" aborts.

That is not a broken model — it is an empty candidate set. Loosen the stack, one knob at a time. Full triage: troubleshooting.

Presets that work

The chat row as llama.cpp flags:

llama-server -m model.gguf --temp 0.7 --top-p 0.95 --min-p 0.05 --repeat-penalty 1.05

Change one thing at a time

Treat the presets as starting points, not laws. If you move temperature and top_p together and output improves, you have learned nothing about which knob mattered.

1. Fix a test set: 5–10 prompts that exercise what you care about.
2. Save the outputs per configuration; diff them side by side.
3. Change exactly one knob per run.
4. Save the winner as a named preset — every engine and UI stores them.

For how these knobs map to engine flags and API fields: engine params.

Related: engine params · prompting · structured output · troubleshooting