How models are trained (and why you won't)

Every chat model you can download went through the same three-stage pipeline. You will almost certainly never run stage one — that is datacenter territory — but you might run cheap cousins of stages two and three. Knowing the stages explains base-vs-instruct behavior, why models hallucinate, and what a "merged" model on Hugging Face actually is.

Stage 1: pretraining — the expensive part

One objective at unthinkable scale: predict the next token across trillions of tokens of web text, code, and books. This is where knowledge and skill enter the weights, and where the overwhelming majority of the compute goes.

The output is a base model: brilliant at continuation, useless as an assistant. Asked a question, it answers — then keeps going with another question, a list, a page break, because that is what text on the internet does.

Stage 2: supervised fine-tuning (SFT)

Tens of thousands to a few million curated (instruction → response) pairs, a few GPU-days. Rounding error next to pretraining, but it rewires behavior: answer the instruction, in a helpful format, then stop. This is the stage that turns a base model into something you can chat with.

Stage 3: preference tuning

• RLHF: humans rank pairs of responses; a reward model learns the ranking; the policy is optimized against the reward (PPO). The classic recipe, heavyweight machinery.
• DPO: skips the reward model and optimizes preference pairs directly. Simpler, stable, the open-source favorite.
• Effect: style, helpfulness, refusals, better-calibrated claims. Also sycophancy — the model learns that people reward agreement.

What SFT data actually looks like — plain JSONL of instruction/response pairs, nothing exotic:

{"instruction": "Summarize the changelog in three bullets.", "response": "- Adds ..."}
{"instruction": "답변을 존댓말로 작성하세요.", "response": "네, 알겠습니다 ..."}

Thousands to millions of these, a few epochs, and the base model learns the assistant format.

Same weights, different stages

prompt: "Write a haiku about GPUs."

base (after stage 1):
  "Write a haiku about GPUs.
   Write a sonnet about CPUs.
   1. Choose your topic carefully..."     ← continues the document

instruct (stages 1+2+3):
  "A thousand cores wake,
   each chasing one quiet token—
   fans hold their breath."               ← answers, formats, stops

The repo suffix tells you the stage: bare name = base; -instruct / -it / -chat = tuned (see parameters).

One more vocabulary item: the knowledge cutoff — the date the pretraining corpus was collected. Ask about anything later and you get either an honest "I don't know" or, worse, confident fiction; both are correct behavior for frozen weights (what is an LLM).

Training vs fine-tuning: about 1000× apart

Fine-tuning adjusts the dials; it does not add knowledge. Teaching a 7B your company's tone: LoRA, an evening. Teaching it your 50k internal documents: that is retrieval, not training (RAG, fine-tuning).

What fine-tuning can and cannot do, so you aim it correctly:

• Can: output format, tone, terminology, refusal style, tool-call syntax, a narrow task done reliably.
• Can: distill one narrow task from a big model into a small one.
• Cannot (reliably): add new facts, fix weak reasoning, extend knowledge cutoffs, make a small model big.
• Rule: if the problem is "doesn't know," use retrieval; if it is "doesn't behave," fine-tune.

Checkpoints, merges, and what you download

• Training saves checkpoints along the way; releases are usually the final instruct-stage weights — you get the finished assistant, not the process that built it.
• A "merge" averages or interpolates multiple fine-tunes of the same base model (model soups). No training happens; it is weight arithmetic that often lands between its parents. Popular on Hugging Face, cheap, occasionally excellent.
• Unclear lineage means unclear license: merges inherit every parent's terms (open weights vs closed APIs).
• The base weights behind a chat model are sometimes published too — grab those only for completion workloads or as fine-tuning substrate.
• Why open releases stop at the final stage: pretraining data and intermediate checkpoints are the expensive competitive assets; the finished weights are the demo.
• Community instruct models share post-training recipes and datasets, which is why so many feel interchangeable: same base, same stage-2/3 playbook.
• Prefer releases whose model card lists data and stages; a documented final checkpoint ages better than a hyped one.

Related: fine-tuning · RAG · open weights vs closed APIs · parameters