localaiforum
Log in
Guide › Working with models

Troubleshooting: symptom → cause → fix

Local inference fails in patterns. The same dozen symptoms account for almost every help request, and each has a boring cause and a two-minute fix. Scan the table, apply the fix, then read the numbers to confirm you are actually fixed.

The diagnostic table

Symptom Likely cause Fix
Output contains role tokens (user:, `< im_start >`)
Instant empty reply Template renders a "finished" chat, or wrong EOS token Fix the template; check EOS: prompting
Repeats your question instead of answering Base model, not an instruct model Use an instruct/chat GGUF
Out of memory while loading Weights + quant exceed VRAM Smaller quant or smaller model: VRAM math
Out of memory mid-generation KV cache grows past headroom Shorter context, KV cache q8_0: KV cache
1–3 tok/s despite having a GPU Layers still on CPU Full offload (n_gpu_layers): engine params
Slow first token, fast stream after Prefill-bound: very long prompts Shorten prompts, batch, faster prefill: speed
Endless repetition, phrase loops Sampler too permissive, penalties off temp ≤ 0.8, repeat_penalty 1.05–1.1: sampling
Korean output is gibberish or stilted Model weak in Korean Multilingual families (Qwen, Gemma): model families
Output truncated mid-sentence Context window full or max_tokens too low Trim context or raise the output limit
Invalid JSON from a good model Free-form sampling Constrained decoding: structured output
Model ignores your instructions Lost in the middle on long prompts Goal first, payload last, repeat the ask: prompting
Crash or coredump on load Corrupted download Verify the checksum, redownload

Read it symptom-first, not cause-first: match what you see, apply the fix, measure again. Most "my model is broken" reports end at row 1, 2, or 6.

The five-minute pre-flight

Before any deep debugging, collect the facts:

nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv   # GPU + VRAM
ls -lh model.gguf                                                   # size vs VRAM
llama-server --version                                              # engine build

If the model file is bigger than free VRAM, row 4 is your answer and no flag will save you. If memory.used climbs steadily during generation, it is the KV cache, not a leak.

What normal looks like

Before you debug performance, know the ballpark. Decode speed for a 7–8B Q4_K_M model, batch 1:

Hardware Decode tok/s
Midrange GPU (RTX 3060/4060 class) 30–60
High-end GPU (RTX 3090/4090 class) 100–150
Apple M-series (unified memory) 20–40
CPU only (modern 8-core) 3–8

Two formulas explain most of it:

decode tok/s ≈ memory bandwidth (GB/s) ÷ model size (GB)
VRAM needed  ≈ weights + KV cache + 1–2 GB overhead

A 4.5 GB 7B quant on a 288 GB/s card tops out near 64 tok/s theoretically; expect ~45–55 real. If you are within 2× of the table, your hardware is not the bug.

Time-to-first-token (TTFT) should be well under a second for short prompts; it grows with prompt length divided by prefill throughput (which runs 10–100× decode speed). A 10k-token prompt that takes 20 s before the first token is prefill-bound, not broken.

VRAM accounting: if the weights loaded fine but you OOM later, it is the KV cache or context length, not the quant — KV cache.

The workflow: swap one variable

Bugs hide when you change two things at once. Isolate along this chain — model → engine → config → prompt:

  1. Baseline. Known-good model, known-good engine, default flags. Does chat work? Then the stack is fine.
  2. Swap the model (same engine, same flags). Breaks? It is the file or its template.
  3. Swap the engine (same model). Breaks? Engine bug or flag translation.
  4. Bisect your flags. Restore defaults, add flags back one at a time until it breaks.
  5. Dump the rendered prompt. When output is weird-but-running, the rendered template is where the truth lives.

Keep notes as you go — "works at temp 0.8, breaks with min_p 0.2" is a diagnosis; "tried stuff, still bad" is not.

How to ask for help well

A good help post is a lab report. Include:

  • Model file: exact name, quant, and source (qwen2.5-7b-instruct-q4_k_m.gguf, which repo).
  • Engine and version: llama.cpp build date, Ollama version, vLLM release — engines change weekly.
  • Exact flags or API call: the whole command line or request body, not a paraphrase.
  • Hardware: GPU + VRAM (or Mac + RAM), CPU if it is carrying layers.
  • Observed numbers: tok/s, TTFT, VRAM used, and the exact error text if any.
  • What you already tried and what changed as a result.

Copy-paste skeleton:

Model:    qwen2.5-7b-instruct-q4_k_m.gguf (bartowski repo)
Engine:   llama.cpp b4321 (llama-server)
Flags:    -ngl 99 -c 8192 --temp 0.7
Hardware: RTX 4060 8GB / Ryzen 7 / 32GB RAM
Observed: loads in 21s, ~35 tok/s, TTFT ~1.2s, then OOM at ~6k ctx
Tried:    -c 4096 → no crash; q8_0 KV → no crash

With those six facts, most problems map to a row in the table above in one reply. This forum exists for exactly these posts — bring the lab report and you will get the fix.

Related: engine params · vram math · kv cache · sampling

Guide