Guide
Fundamentals
1
What is an LLM, really?Next-token prediction, billions of frozen weights, and why every product is orchestration around a text-in/text-out loop.
2
Tokens & tokenizersSubword vocabularies, why Korean eats context 3–4× faster than English, and why letter-counting questions fail.
3
Inside the transformerEmbeddings, self-attention, FFN, and GQA — the whole architecture with nothing heavier than dot products and a softmax.
4
Prefill vs decode: how generation runsTwo phases with different physics: one parallel pass over the prompt, then one-token-at-a-time generation that re-reads all weights.
5
What parameter count buys youThe capability ladder from 1B toys to 70B+ frontier class — knowledge capacity, naming conventions, and matching size to task.
6
How models are trained (and why you won't)Pretraining, SFT, and preference tuning — what each stage costs, what it changes, and what's actually doable on one GPU.
7
Open weights vs closed APIsThe license spectrum from Apache-2.0 to API-only, the gotchas that bite commercial use, and how to read a Hugging Face repo name.
Hardware
8
What hardware matters (and what doesn't)Capacity, bandwidth, compute — ranked by what they actually buy you, plus the spec-sheet items that don't matter.
9
VRAM mathWeights + KV cache + overhead: the formula, worked tables for 8B–70B, and a 16 GB sizing walkthrough.
10
GPU tiers: what actually fitsWhat fits at each VRAM tier, used-market picks, and why two smaller cards lose to one bigger one.
11
Apple Silicon & unified memoryUnified memory makes RAM into VRAM: bandwidth by chip class, MLX vs llama.cpp, and the swap trap.
12
Running on CPU onlyBandwidth arithmetic for DDR4/DDR5, comfort tiers, prefill pain, and when CPU-only is genuinely the right call.
13
Context windows & the KV cacheWhy context costs VRAM: the per-token formula, GQA, quantized KV, and advertised vs effective windows.
Models & formats
14
Model file formatsGGUF, safetensors, AWQ/GPTQ, EXL2, ONNX — what each container is for, which engine eats which, and how to convert.
15
Quantization, in depthPast the cheat table: block scales, K-quant mixes, imatrix I-quants, where quality breaks first, and a VRAM-aware decision ladder.
16
The open-weight family treeQwen, Llama, Gemma, Mistral, Phi, DeepSeek — sizes, licenses, base vs instruct naming, MoE total-vs-active, and the fine-tune jungle.
17
Choosing the right modelUse case first, benchmarks second: reading leaderboards without being fooled, a VRAM-tier recommendation matrix, and a try-before-commit protocol.
Running & serving
18
Your first model in 10 minutesFrom empty machine to chatting with a local model, via terminal or GUI — plus the health checks and the three classic first failures.
19
Inference engines comparedllama.cpp, Ollama, LM Studio, vLLM, SGLang, exllamav2, MLX, transformers — what each is for, and why the differences matter less than your I/O pattern.
20
Engine knobs that matterContext length, GPU offload, threads, flash attention, KV cache dtype, YaRN — the handful of flags that change outcomes, and the log lines that verify them.
21
The physics of tokens/secDecode speed is memory bandwidth divided by bytes read per token — from that one formula comes quantization-as-speedup, MoE gains, the CPU-offload cliff, and what tok/s actually feels like.
22
Serving an OpenAI-compatible endpointThe chat-completions shape is the lingua franca every client speaks — serve it with llama-server, Ollama, or vLLM, point SDKs at localhost, and lock the door.
23
UIs & integrationsOpen WebUI, LibreChat, text-generation-webui, Continue.dev, terminal tools — pick by role; they all speak OpenAI-compat so any engine plugs in.
24
Batching & multiple usersSerial serving wastes the GPU; batching shares each weight read across requests. Continuous batching, paged attention, capacity math, and when to leave Ollama for vLLM.
Working with models
25
Prompts & chat templatesChat templates are the envelope your prompt travels in — ChatML vs Llama vs Gemma markup, few-shot contracts, and the patterns small models can actually follow.
26
Sampling settingsTemperature, top_k, top_p, min_p, and repeat_penalty — what each knob does to the logits, in what order, and presets that work per task.
27
Structured output & function callingFrom prompt-only JSON to grammar-masked decoding that makes invalid tokens impossible — plus the tool-calling conventions agents run on.
28
RAG: giving the model your dataChunk, embed, retrieve, cite — a full local RAG pipeline with worked numbers, hybrid search, reranking, and how to tell retrieval misses from generation misses.
29
Fine-tuning with LoRA & QLoRAAdapters on one consumer GPU: what tuning changes (behavior) vs what it can't (facts), QLoRA budgets, the data rules that dominate, and export to GGUF.
30
Agents & tool useThe plan–act–observe loop, tools as JSON schemas, MCP, context discipline for long runs, and prompt injection — the actual threat model.
31
Troubleshooting: symptom → cause → fixA symptom-first diagnostic table for the classics — template mismatches, OOM at load and mid-run, 1 tok/s, repetition loops, gibberish, truncation — plus what normal numbers look like.