Prompts & chat templates

Every message you send gets wrapped in markup before the model ever sees it. The chat template turns your role-tagged messages into one flat token string, and the model was trained to read that exact string — not your tidy API objects. Get the envelope right and a 3B instruct model behaves; get it wrong and a 70B model emits word soup.

The envelope: one chat, three dialects

Your API exposes roles: system, user, assistant. The model sees special tokens. The same two-message conversation, rendered three ways.

ChatML — Qwen-family models and Hermes-lineage finetunes:

<|im_start|>system
You are a concise assistant.<|im_end|>
<|im_start|>user
Summarize this file.<|im_end|>
<|im_start|>assistant

Llama 2 style — system markers nested inside the first instruction:

<s>[INST] <<SYS>>You are a concise assistant.<</SYS>>

Summarize this file. [/INST] The file implements a tokenizer... </s>

Gemma style — there is no system role; instructions ride on the first user turn, and the assistant is called model:

<bos><start_of_turn>user
You are a concise assistant.

Summarize this file.<end_of_turn>
<start_of_turn>model

Same conversation, three incompatible envelopes. A model trained on ChatML does not "figure out" Llama markup — it treats stray markup as ordinary text and responds in kind.

Engines read the template from the model file

GGUF metadata embeds a chat_template (a Jinja snippet). llama.cpp, Ollama, LM Studio, and Open WebUI read it and render your messages automatically, which is why chat works out of the box with a proper GGUF. Two classic traps: the raw completion endpoint (/completion) skips the template and hands the model bare text — use the chat endpoint; and a GGUF can ship with a missing or wrong template — pass one explicitly (--chat-template in llama.cpp, a Modelfile in Ollama) or grab another quant of the same model from a source that got it right.

Symptoms of a wrong template

• Instant empty reply — the rendered prompt already looks like a finished conversation, so the model emits EOS immediately.
• Role bleed — the output reads user: sure thing! assistant: — the model imitates the envelope instead of answering.
• Garbage — base-model word soup from a model that chatted fine an hour ago, because you swapped files without swapping templates.

To diagnose, dump the rendered prompt — the llama.cpp server logs it verbosely, OLLAMA_DEBUG=1 shows it in Ollama. Then fix the template or the file: troubleshooting.

System prompts and precedence

The system role sets persistent behavior: persona, language, output format, safety posture. It is context, not firmware — a long conversation can wash it out, and small models weight it less than you hope. Two practical rules: put anything load-bearing (language, format contract, hard constraints) in the system prompt and restate it in the final user message; instructions closest to generation win.

Zero-shot vs few-shot

Zero-shot asks; few-shot shows. Extracting structured fields from bug reports with a 3B model, zero-shot — "Extract the title, component, and severity as JSON" — gets valid, consistently-keyed JSON about two-thirds of the time, with key names drifting (severity one turn, priority the next).

Few-shot: same model, two worked examples, third report left open:

Report: "Login button does nothing on Safari 17."
{"title": "Login button unresponsive", "component": "auth", "severity": "medium"}

Report: "App crashes when uploading 4K video."
{"title": "Crash on 4K upload", "component": "upload", "severity": "high"}

Report: "Dark mode makes settings text unreadable."

The examples carry the format contract, so the sampler chooses content instead of inventing schema. On this class of task the format failure rate typically drops from ~30% to a few percent. Two or three examples is the knee of the curve; beyond that you are paying context tokens. For strict schemas, skip the guessing entirely: structured output.

Patterns that survive small models

The middle of the context is where instructions die

Attention is strongest at the start and the end of the prompt — the "lost in the middle" effect, stable across model families. Practical placement: goal and format contract at the top, the document to process at the bottom, and for long documents a one-line instruction repeat after the payload. Position is as load-bearing as content — and every extra turn of context costs memory: KV cache.

Stop sequences

Generation ends when the model emits its EOS token — the template defines which token that is. Extra stop sequences (the stop parameter on the llama.cpp server and OpenAI-compatible APIs) truncate at a string instead: useful when a chatty model starts writing your next user turn. Choose stops that can only mean "turn over" (\n<|im_start|> on ChatML models), never strings that occur inside legitimate answers — a bare newline will clip multi-line code mid-block.

Brevity beats verbosity below ~7B

Small instruct models follow a 40-word instruction better than a 400-word rubric: their limited attention spreads over whatever you hand them. Structure and examples substitute for prose rules. Save the essay prompts for the big models; they are the only ones that read every line.

Reusable snippet templates

Extraction:

Extract from the text below: person names, dates, money amounts.
Return a JSON array, one object per item, keys: name, date, amount.
No prose before or after the JSON.
Text: {text}

Summarization:

Summarize in exactly 3 bullet points, each under 15 words,
for someone who skipped the meeting. Most important decision first.
Text: {text}

Code review:

Review the diff below. List at most 5 findings, ordered by severity.
Format each as: [severity] file:line — problem — suggested fix.
Skip praise and summary. Diff: {diff}

Related: tokens · structured output · engine params · troubleshooting