Fine-tuning with LoRA & QLoRA

Fine-tuning tunes behavior, not knowledge. It is the right tool when the model knows the material but performs the wrong dance: wrong output format, wrong tone, wrong domain dialect, wrong refusal calibration. It is the wrong tool for "knows my documents" — that is retrieval: RAG.

Fine-tune or not

If a prompt or a schema can do it, prefer the prompt or the schema. Tune when you catch yourself pasting the same 300 tokens of instructions into every call.

LoRA in one paragraph

Freeze every base weight. For each target weight matrix, add two small trainable matrices — A (r×d) and B (d×r) — and train only those. The update is low-rank:

W' = W + (alpha / r) · B·A      # A is r×d, B is d×r, rank r << d

Concrete: a d = 4096 matrix at rank r = 16 needs 2 × 4096 × 16 = 131k adapter parameters instead of 4096² ≈ 16.8M dense ones — 0.8% of that matrix. Across a 7–8B model, adapters total ~40–80M parameters: 0.5–1% of the base, a few hundred MB. That is enough capacity to move style and format hard, because those behaviors live in strongly low-rank directions.

QLoRA: one consumer card

QLoRA keeps the frozen base quantized to 4-bit (NF4) during training and runs the adapters in 16-bit. Working numbers on current stacks: a 7–8B model QLoRA-trains on a single 24GB GPU; a 3B trains comfortably in 8GB. Gradient checkpointing and paged optimizers are handled by the tooling — what you supply is the budget math: VRAM math.

Data beats everything

500–5,000 high-quality pairs beat 100k scraped pairs, every time. The rules that matter:

• One consistent format. Use the exact chat template you will serve with — a train/serve template mismatch undoes the whole run.
• Quality over volume. Each pair should be an output you would accept from the model. Deduplicate; drop lazy examples.
• Coverage. Include the cases you want the model to decline or handle specially — otherwise it overgeneralizes its new behavior onto everything.
• Contamination checks. Make sure eval questions do not appear in the training data — otherwise you are grading memorization.
• Held-out eval set. 50–100 examples you never train on, graded before and after.

What a pair looks like (the messages format most tools consume):

{"messages": [{"role": "system", "content": "You are a 1998-era equipment manual assistant."},
              {"role": "user", "content": "How do I reset the service counter?"},
              {"role": "assistant", "content": "Press MENU and hold for 3 seconds..."}]}

Tools

As of this writing all three are actively maintained: unsloth, axolotl, llama-factory.

Hyperparameters that actually matter

In order of impact:

• Learning rate — 1e-4 to 2e-4 for QLoRA is the sane band; too high is the usual cause of mush.
• Epochs — 1–3. More is where the yes-man comes from.
• Rank / alpha — r 8–32 with alpha 16–32 covers most behavior tweaks; bigger rank for domain-heavy shifts.
• Context length — set to your longest real example, not the trainer default.
• Batch size — 1–8 is normal for QLoRA on one card; it is a throughput knob, not a quality one.

If a run is disappointing, fix data first, learning rate second, rank last.

Time and cost

QLoRA on 2,000 pairs, 3 epochs, 7–8B model: expect 1–3 hours on one RTX 3090/4090-class card. That is electricity, not API bills, and iteration is overnight-at-worst. This is why local fine-tuning is a hobby, not a procurement process.

Budget for iteration: the first run teaches you the data problems, the second is the one you keep.

Evaluate before you ship

Run a fixed suite of 30–50 prompts — including the held-out set — before and after; diff the outputs side by side. Then regression-check general chat: the classic failure is a domain-tuned model that answers everything in bug-report format. If general quality dropped, use fewer epochs, a lower learning rate, or mix 10–30% general instruction data into the run.

Loss curves are for spotting divergence, not for grading: training loss should fall and flatten. The eval suite is the arbiter.

Export and serve

Two paths from a trained adapter:

• Merge LoRA into the base at fp16, then convert to GGUF for llama.cpp — the forum-friendly artifact: one file, any engine: model formats, quantization.
• Keep the adapter separate for transformers/vLLM, which load base + adapter at runtime — cheaper to iterate, engine-specific.

Failure modes

• Overfit / "yes-man" — echoes your prompt, agrees with everything; training loss low, eval worse. Fix: fewer epochs, lower LR, more diverse data.
• Catastrophic forgetting — domain skill up, general ability down. Fix: mix general data in; lower LR.
• Template drift — trained with one chat template, served with another; symptoms are the classics in troubleshooting.

DPO after SFT

When format is right but preferences are off — too verbose, wrong tone, over-refusing — a small DPO run on preference pairs (chosen vs rejected responses) adjusts behavior without another SFT pass. Cheap, stable, and the standard second stage of the SFT-then-DPO recipe.

Related: rag · model formats · quantization · vram math