Inside the transformer

Everything you can run locally is a transformer, and the architecture fits on a napkin: turn tokens into vectors, push them through N identical layers where tokens look at each other, project the final vector back to vocabulary scores. No calculus needed — dot products and a softmax carry the entire show. This page walks the path once, slowly.

The residual stream

Picture a conveyor belt running through the model. At each layer, sub-modules read the belt and add their contribution back onto it — nothing replaces, everything accumulates:

x = embed(token_ids) + positions      # tokens become vectors (dim ~4k)
for layer in range(N):                # N ≈ 32 for an 8B model
    x = x + attention(x)              # tokens exchange information
    x = x + ffn(x)                    # each token transforms itself
logits = unembed(x[-1])               # last position → score per vocab entry
p = softmax(logits)                   # probability over every next token

• Embedding: each token ID indexes a learned vector table. Position information is mixed in (rotary embeddings in modern models), so "dog bites man" differs from "man bites dog."
• Layers are identical in shape, different in content. Early layers handle syntax and surface patterns; deeper layers carry meaning, facts, and task behavior.
• Unembedding: the final vector is compared against every vocabulary entry; softmax turns scores into probabilities. Sampling picks one (sampling).

Self-attention in four tokens

The exchange mechanism. Each token's vector produces three derived vectors: a query (what am I looking for), a key (what do I advertise), and a value (what I hand over if selected). Sentence: "The cat sat on". What should the position of "sat" pull in?

token:          The     cat     sat     on
q·k scores:     1.2     4.1     3.0     0.4      (dot products, illustrative)
softmax →        0.05    0.60    0.31   0.04     (weights sum to 1.0)

output("sat") = 0.05·v(The) + 0.60·v(cat) + 0.31·v(sat) + 0.04·v(on)

"sat" attends mostly to "cat" — the subject it needs. Every position runs this simultaneously. One constraint makes it a language model: causal masking — each token may only look at earlier tokens, never later ones, so one pass trains every next-token prediction in the sequence at once.

That is the entire mathematical toolkit: similarity scores (dot products), normalized into a mix (softmax), over value vectors.

As a recipe, one attention head at one position:

1. Compute query, key, and value vectors for every token.
2. Dot the query with each earlier token's key; scale down by √(head_dim).
3. Softmax the scores into weights that sum to 1.
4. Output the weighted sum of those tokens' values.
5. Concatenate all heads' outputs and project — that is the layer's attention contribution.

Multi-head: patterns in parallel

One head = one query/key/value triple = one kind of relationship it can express (subject↔verb, adjective↔noun, coreference). Layers run many heads in parallel (e.g. 32) on lower-dimensional slices and concatenate the results. More heads means more relationship types per layer — breadth of pattern, not depth of thought.

• Heads are fixed at training time; "thinking" across positions is what attention writes back into the residual stream.
• Head dim (e.g. 128) × heads (e.g. 32) = hidden dim (4096): heads are parallel slices of the same conveyor belt, not separate models.

The FFN: where the parameters live

Each layer's second half is a two-matrix network applied to every token independently: expand the vector (~4×), nonlinearity, project back. In a typical dense model, roughly two-thirds of all parameters live here, not in attention. The useful intuition: key-value memory — the first matrix holds patterns to detect, the second holds what to add to the belt when a pattern fires. Much of what people call "the model's facts" is stored in FFN weights (see parameters).

Count the parameters yourself

Model cards publish five numbers; multiply them and you land on the parameter count. For a real 8B-class open model (32 layers, hidden dim 4096, 32 heads with 8 KV heads, FFN width 14336, vocab ~128k):

attention/layer:  q: 4096×4096 + k,v: 2×(4096×1024) + o: 4096×4096
                  ≈ 42M per layer × 32 layers            ≈ 1.3B
FFN/layer:        3 × (4096×14336), gated ≈ 176M × 32     ≈ 5.6B
embeddings:       2 × (128k × 4096), input + output head  ≈ 1.1B
total                                                     ≈ 8.0B

The arithmetic matches the model card's "8B" almost exactly. Two lessons fall out: the FFN dominates the parameter count (hence the memory budget), and a large vocabulary costs about a billion parameters before the first layer even runs — part of why big-vocab multilingual models weigh more than their dense siblings suggest.

GQA: attention's cost fix

During generation, each new token must read the keys and values of every previous token — stored in the KV cache. Original multi-head attention stores one K and one V per head per token. Grouped-query attention (GQA) shares a handful of KV heads across many query heads (e.g. 32 query heads, 8 KV heads), shrinking the cache 4× at negligible quality cost. Multi-query attention (MQA) shares a single KV head. Effectively every modern open model uses GQA; those savings are a big part of why long context is usable at all (KV cache).

Context length = attention span

Attention lets every token see every earlier token, so prefill compute grows quadratically with prompt length and the cache grows linearly with every generated token. A "128k context" model gets there with positional tricks and trained extension, and quality near the far end of the window is usually weaker than near the start. Fill long windows only when you must.

Test it yourself: put a fact at the start of the window, fill the middle with filler, ask about it. Most models pass near the beginning and end of the window and dip in the middle — the well-known "lost in the middle" pattern. Treat advertised context as capacity, not a quality guarantee.

Why this is enough

From this structure alone you can derive the practical facts: memory need = weights + KV cache (VRAM math); decode speed = how fast you can re-read the weights (prefill vs decode); capability tracks parameter count (parameters); context cost = attention span. The architecture is simple — the billions of trained weights inside it are the part nobody could write by hand.

Related: what is an LLM · KV cache · prefill vs decode · parameters