RAG: giving the model your data

An LLM's knowledge froze at its training cutoff; your documents change weekly. Retrieval-augmented generation (RAG) thaws it: at query time, fetch the relevant slices of your data, paste them into the prompt, and let the model answer from evidence with citations. The model does reasoning; the index does memory.

The pipeline, one worked example

Take a 12,000-token maintenance manual and walk it through end to end:

1. Chunk. Split on headers first (sections survive intact), then re-split any section over ~400 tokens, keeping ~60 tokens of overlap (10–15%). Sensible chunk sizes run 200–800 tokens. Result: roughly 35 chunks.
2. Embed. Each chunk becomes one vector — bge-m3 emits 1,024 floats per chunk. On CPU this takes seconds for the whole manual.
3. Index. Store vectors + text + metadata (source file, header path) in a vector store.
4. Retrieve. Embed the question, fetch the top k = 5 chunks by cosine similarity (4–10 is the sane range).
5. Generate. Assemble the prompt: instructions, the 5 chunks with ID tags, the question.

Chunking on semantic boundaries beats fixed windows. A table split from its header row, or a numbered procedure cut at step 4, is unretrievable garbage no matter how good the embedder is.

Two extras that pay for themselves immediately: store the header path (manual › §4 › Bolts) as metadata so you can filter and cite precisely, and embed the query exactly as the user typed it — preprocessing "cleanup" usually makes retrieval worse, not better.

Embeddings: the multilingual requirement

The embedder decides what "similar" means. For Korean and mixed Korean/English corpora the choice is narrow — English-only embedders fail silently on Korean queries, returning plausible-but-wrong chunks:

Embedding models are tiny (0.1–0.6B parameters); a CPU core chews through thousands of chunks — CPU inference. Embed the corpus once, embed queries at ask time. Keep the embedder frozen per index: swapping models mid-index makes vectors from different spaces, and similarity scores become noise.

Vector stores

All three run locally. Start with sqlite-vec; move when you actually hit its wall, not before.

Assembling the prompt

Answer only from the excerpts below. Cite every claim as [n].
If the excerpts do not contain the answer, reply "not in the docs".

[1] (manual §4.2) Torque the M8 bolt to 12 Nm after lubrication...
[2] (manual §7.1) Do not re-use stretch bolts...

Question: what torque spec applies to the M8 bolt?

Citations let users verify instead of trusting. The "not in the docs" exit matters: without it, the model invents an answer rather than fail the format.

Hybrid search and reranking

Pure vector search misses exact strings — error codes, function names, part numbers — that keyword search nails. Hybrid = BM25 + vectors, merged with reciprocal rank fusion; it is nearly free and reliably better than either alone.

For quality, add a reranker: retrieve 20–50 candidates, rerank down to the top 5 with a cross-encoder (the bge-reranker family — same multilingual story as bge-m3). The reranker reads chunk and query together, which a single-vector embedder cannot. Cheap recipe that covers most needs: hybrid retrieve 25, rerank to 5.

Diagnose: retrieval miss or generation miss?

Before tuning anything, answer one question: did the right chunk get retrieved?

1. Print the top-k chunks for a failing query, with scores.
2. If the answer-bearing chunk is not in the list, it is a retrieval problem — fix chunking, embeddings, or raise k.
3. If it is in the list but the answer is wrong, it is a generation problem — bigger model, better prompt, rerank to push distractors out.

This one test saves hours of tuning the wrong half of the pipeline.

Failure modes

Measure, do not vibe

Keep a golden set of 20–50 real questions with known answers. Score two numbers separately: retrieval hit rate (is the right chunk in the top-k?) and answer correctness. Track them after every change to chunk size, embedder, or k. RAG work without a golden set is astrology.

When to fine-tune instead

Fine-tuning changes behavior — format, style, terminology. It is a poor way to add facts: the model cannot cite what it was tuned on, and fresh documents mean a fresh training run. Rule of thumb: facts → RAG, behavior → LoRA, both only when you have exhausted prompting: fine-tuning.

Local stacks, smallest first

Open WebUI ships document upload and hybrid retrieval built in — start there and you may never write pipeline code. The honest core of RAG is ~50 lines; write it once so you know what the frameworks are doing:

chunks = chunk_by_header(read("manual.md"), max_tokens=400, overlap=60)
index = [(embed(c), c) for c in chunks]          # done once

def answer(question):
    hits = topk_by_cosine(embed(question), index, k=5)
    ctx = "\n\n".join("[{}] {}".format(i, c) for i, (v, c) in enumerate(hits))
    prompt = TEMPLATE.replace("{ctx}", ctx).replace("{q}", question)
    return chat(prompt)                           # small local model is plenty

Reach for LlamaIndex or LangChain when you need permissions, scheduled refresh, or evaluation harnesses — not for the loop above. For Korean documents specifically, multilingual embeddings (bge-m3) are not optional, and header-based chunking should respect Korean document structure (장/절 headings) the same way it respects markdown ## headings.

Related: fine-tuning · prompting · structured output · cpu inference