Engine knobs that matter

Most engine defaults are sane for a demo and wrong for real use. Three or four knobs change outcomes: context length, GPU offload, KV cache type, and flash attention. Everything else is tuning at the margin. This page uses llama.cpp flag names, with Ollama equivalents at the end — the concepts transfer to every engine.

Context length: the silent-default trap

Engines ship conservative defaults: llama.cpp historically 512 then 4096, Ollama 2048–4096 depending on version (as of this writing OLLAMA_CONTEXT_LENGTH lands around 4k). Meanwhile your model may natively support 32k–128k. Paste a 20k-token document into a 4k context and one of two things happens, both bad: a hard error ("request exceeds available context") or silent truncation of the oldest messages. Neither error message explains itself to a beginner.

Worked example: you load a contract of ~12k tokens plus a question, into a default 4096 window. The engine keeps the most recent material, drops the beginning of the document, and the model confidently answers questions about the wrong half of the file. This looks like a stupid model; it is a truncated prompt.

The fix is mechanical: pass the context size explicitly on every launch, sized to your real prompts plus expected output, then verify the startup log shows n_ctx = 16384 (or whatever you chose). Korean note: Korean text often tokenizes less densely than English in the same tokenizer — a page of Korean can cost more tokens than a page of English (see tokens) — so context pressure arrives sooner than you expect.

# the trap: defaults differ per engine and per version
llama-server -m model.gguf                  # starts with n_ctx = 4096 or so

# the fix: explicit, and verified in the startup log
llama-server -m model.gguf -c 16384         # log prints: n_ctx = 16384

GPU offload: n_gpu_layers

-ngl / --n-gpu-layers sets how many transformer layers live on the GPU; use a large number like 99 (or -1 where supported) to mean "all". The symptom of partial offload is distinctive: 3–10 tok/s on a machine with a decent GPU. Verify with the load log line, e.g. offloaded 33/33 layers to GPU. If VRAM cannot hold model plus KV cache, the honest options are a smaller quant, a quantized KV cache, or fewer layers offloaded — but know that every layer left on CPU drags the whole loop to CPU rhythm (see speed). Half-offloading a model that almost fits is usually worse than a quant step down.

Sanity-check in the other direction too: -ngl 0 forces pure CPU, and if that runs at the same speed as your "GPU" config, the GPU was not doing anything.

Threads (CPU mode only)

In CPU mode set -t to the number of physical cores, not SMT threads: on an 8-core/16-thread chip, use 8. More threads than physical cores causes contention and slows decode. Prompt processing has a separate knob (--threads-batch); the same rule applies.

Memory: mmap and --mlock

llama.cpp memory-maps the model file by default: startup is fast and untouched layers cost no RAM. The trade-off is that the OS may evict pages under memory pressure, causing mysterious pauses hours into a session. --mlock pins the model so it cannot be evicted — useful on busy machines, at the cost of locking gigabytes away from everything else. Ollama handles this internally; raw llama.cpp users on shared machines should know the flag exists.

Flash attention

-fa / --flash-attn reduces memory traffic in the attention step and is on (or auto) by default in recent llama.cpp builds. It matters most at long context, and quantizing the V cache requires it in current builds. If a fresh build misbehaves on your GPU backend, this is the first flag to toggle — and on any recent build, the first thing to check is that it did not silently turn itself off.

KV cache quantization

--cache-type-k q8_0 --cache-type-v q5_0 stores the KV cache in 8-bit/5-bit instead of 16-bit floats, cutting KV memory roughly in half or better at minor quality cost — which means longer context in the same VRAM. What the KV cache is and the exact size math: KV cache.

Typical win: q8_0 K plus q5_0 V roughly halves KV size with barely visible change on chat tasks — verify on your own workloads before trusting it.

Prompt batch size

-b / --batch-size is the chunk size for prefill (prompt processing). Bigger batches mean faster time-to-first-token on long prompts, at the cost of a little peak VRAM. If long pastes feel slow to start, raise it before blaming the model.

Stretching context: rope scaling and YaRN

--rope-scaling yarn --rope-factor 2.0 stretches a 32k-native model toward 64k by rescaling positional frequencies. It works, with a quality slope: degradation is small near the native window and grows as you stretch past it. For anything important, prefer a model whose native context already covers your need over stretching a smaller window (see choosing).

As of this writing Ollama does not expose rope scaling in its Modelfile; stretched-context experiments mean launching llama.cpp directly.

An annotated launch line

llama-server -m qwen3-8b-q4_k_m.gguf --host 127.0.0.1 --port 8080 -c 16384 -ngl 99 -fa on -b 2048 --cache-type-k q8_0 --cache-type-v q5_0

Ollama equivalents

# per-model, in a Modelfile:
PARAMETER num_ctx 16384
PARAMETER num_gpu 99

# or globally, before the service starts:
export OLLAMA_CONTEXT_LENGTH=16384
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0

Ollama logs a warning when a request exceeds num_ctx — read the log once after changing any of these. Changes via env vars require restarting the service.

To see what Ollama actually applied: ollama show model --modelfile prints the resolved parameters, and OLLAMA_DEBUG=1 makes the service log the real context and offload counts.

Samplers are not engine knobs

Temperature, top-p, repetition penalty — those shape which token is chosen, not how fast or how much fits in memory. They live in sampling. If output is weird but speed and memory are fine, you want that page, not this one.

The rule

Set context explicitly. Verify with a log line — n_ctx = ... and offloaded N/N layers — every time you change a launch config. Defaults are tuned so the first demo starts, not so your workload is served.

llm_load_tensors: offloaded 33/33 layers to GPU
n_ctx = 16384

Both lines print within seconds of launch. Grep for them once and configuration stops being folklore.

Related: Speed · KV cache · Sampling · Serving an endpoint