Tokens & tokenizers

Models never see words or letters. They see token IDs — integers indexing a fixed vocabulary of subword pieces, built with byte-pair encoding (BPE) or a close variant. Tokenization is the border crossing for everything: your prompt, the reply, the context window, the tokens-per-second meter, API bills. Once you think in tokens, half the folklore about LLM behavior dissolves.

How a vocabulary is built

BPE starts from raw bytes and repeatedly merges the most frequent adjacent pair until the vocabulary hits its budget. The result: common words become single tokens, rare words become 2–5 pieces, and nothing is unrepresentable — byte fallback covers every string, including typos and emoji. Vocabularies range from ~30k entries (older models) to ~250k (modern multilingual):

A tokenizer trained mostly on English merges English fragments aggressively; Korean syllable pairs get far fewer useful merges. The vocab decides your Korean tax long before the model itself is good or bad at Korean.

Worked example: English

"Tokenization turns text into pieces."

→ ["Token", "ization", " turns", " text", " into", " pieces", "."]
→ 37 characters → 7 tokens ≈ 5.3 chars/token

Note the details: the common word "Token" is one piece; the suffix gets its own; and the leading space belongs to the following token (" turns", not "turns"). That is why "hello", " hello", and "Hello" are different tokens — prompts that differ by a space or a capital tokenize differently.

Worked example: Korean

Same sentence, worse economics. Hangul is written in syllable blocks, and Korean merges are rarer in English-heavy vocabularies (splits below are plausible, not exact):

"안녕하세요, 만나서 반가워요."        # 17 characters

strong multilingual vocab (~150k entries):
["안녕", "하세요", ",", " 만나", "서", " 반", "가", "워", "요", "."]
→ ~10 tokens ≈ 1.7 chars/token

weak vocab (~32k entries, jamo-level splits):
["안", "녕", "하", "세", "요", ",", " 만", "나", "서", ...]
→ 15+ tokens ≈ 1.1 chars/token

English averages ~4 chars/token. Korean commonly lands at 1–1.5. Same meaning, 3–4× the tokens. This is why:

• CJK prompts consume context windows alarmingly fast — an 8k-token window holds roughly 3–4× more English than Korean.
• Generation "feels" slower in Korean at identical tok/s: more tokens per sentence, less text per token.
• Korean costs more on per-token APIs, and reasoning traces written in Korean burn budget the same way.

Why letter-counting questions fail

Ask "how many r's in strawberry" and even large models fumble. Reason: the model never sees letters. "strawberry" arrives as roughly ["straw", "berry"] — two integers. There is no letter count inside an integer, any more than a page number tells you the page's spelling. The model can recite memorized spellings or reason step by step, and both are unreliable. If you need it done: force the model to spell the word one letter per line first, then count the lines.

The vocabulary ships with the model

• Tokenizer files (vocab, merges) are part of the release; GGUF embeds them, safetensors repos carry them alongside the weights.
• Engines (llama.cpp, LM Studio, vLLM, ...) load the right tokenizer automatically. You never pick one.
• Token counts are model-relative: the same text may be 1,000 tokens under one family's vocab and 1,150 under another's. Never hardcode token budgets across models.
• Vocab quality and model quality are separate axes: a 32k-vocab model can handle Korean (it understands its tokens); it just spends more tokens per sentence.

Everything is priced in tokens

Special tokens and the chat template

Besides text pieces, the vocab reserves control IDs: beginning-of-sequence, end-of-sequence, and role markers. When your chat UI sends a message, it is not sending raw text — it fills a chat template that interleaves special tokens like <|im_start|>user ... <|im_end|> around your words. That wrapping is why the token count the engine reports is always a bit higher than your message's text alone, and why mixing templates from different families confuses models.

Estimation rules: English chars ÷ 4. Korean chars ÷ 1.0–1.5 — assume the worst. The simulator uses the ~4 chars/token approximation and is calibrated for English; mentally inflate for Korean.

Counting tokens yourself

pip install tiktoken
python -c "import tiktoken as t; e = t.get_encoding('o200k_base'); print(len(e.encode('hello world')))"

• tiktoken (https://github.com/openai/tiktoken) covers OpenAI-family encodings; other families differ but land within roughly ±15% for English.
• Hugging Face model pages include a tokenizer viewer; tokenizer playgrounds let you paste text and watch the splits live.
• Local engines print prompt token counts in their logs — check once against your real prompts, then estimate by ratio.

Related: inside the transformer · prefill vs decode · KV cache