Model file formats

A model release is not one file — it's a checkpoint that gets repackaged per engine. This page maps the formats you'll actually meet locally: GGUF for the llama.cpp world, safetensors for the PyTorch/HF stack, engine-native GPU quants, and ONNX for edge runtimes. Pick the engine first; the format follows.

GGUF: one file, everything in it

A .gguf file packs model weights, tokenizer vocabulary, chat template, and per-tensor metadata into a single binary. One download, zero configuration, and every GGUF-speaking app runs the same file: llama.cpp and llama-server, Ollama, LM Studio, Jan, KoboldCpp. The format is memory-mappable, so weights stream from disk page by page — startup is fast and the OS page cache does the caching for free.

Filename anatomy, field by field, for Qwen3-8B-Q4_K_M.gguf:

Inspect any GGUF without loading it:

python gguf-py/gguf/scripts/gguf_dump.py model.gguf

general.architecture     : qwen3
general.name             : Qwen3 8B
qwen3.context_length     : 32768
tokenizer.ggml.model     : gpt2 (BPE)
tokenizer.chat_template  : {%- ... a Jinja template ... %}

That metadata block is why a GGUF "just works" across five different apps: the engine reads the architecture, builds the graph, loads the tokenizer, and chat front-ends apply the embedded template. No config hunting, no vocab mismatch, no separate tokenizer download.

Safetensors: the Hugging Face native

Safetensors is a deliberately boring container: a JSON header of tensor names, shapes, and dtypes followed by raw bytes. It stores only tensors — no executable code, unlike pickle-flavored .bin checkpoints that are unsafe to load blindly — and it supports zero-copy mmap, which is why PyTorch startup is fast. Transformers, vLLM, SGLang, TRL, and the whole fine-tuning ecosystem speak it natively.

Because it stores only tensors, a working model is the whole repo directory:

model-00001-of-00004.safetensors   ← weight shards (~5 GB each)
model-00002-of-00004.safetensors
model-00003-of-00004.safetensors
model-00004-of-00004.safetensors
model.safetensors.index.json        ← shard map: tensor name → shard file
config.json                         ← architecture + hyperparameters
generation_config.json              ← default sampling settings
tokenizer.json                      ← the tokenizer itself
tokenizer_config.json               ← chat template lives here

Sharding exists because a 70B FP16 checkpoint is ~140 GB — one file would be unmanageable. If you serve with vLLM you download all shards; there is no "just the 4-bit file" shortcut inside a safetensors release. Quantized serving is a separate repo (an AWQ or GPTQ variant of the same model).

Engine-native GPU quants: AWQ, GPTQ, EXL2

These formats exist for GPU inference servers, where throughput matters more than file convenience:

• AWQ — activation-aware quantization. Calibration prompts find the ~1% of weight channels that activations depend on most; those get protected scaling, everything else goes to 4-bit. Near-original quality at 4-bit with fast kernels; the default 4-bit choice in vLLM.
• GPTQ — the classic GPU quant, descended from optimal brain quantization: quantize layer by layer, compensating each layer's error against the not-yet-quantized weights. 4-bit, GPU-only kernels, still everywhere in serving configs.
• EXL2 — exllamav2's format with variable bits per weight. One repo can sit at, say, 4.65 bpw by mixing 2–8 bit tensors so the total lands exactly under your VRAM cap. The tool for when Q4 doesn't fit but Q3 would hurt.

All three require calibration data and a one-time conversion run, and none of them run on CPU.

ONNX, MLX, raw FP16/BF16

ONNX is the portable compute-graph format: export a transformers model once, run it anywhere onnxruntime runs — Windows boxes without CUDA, phones, browser WebGPU demos. onnxruntime can also quantize the graph to INT8 for CPU. You pay setup friction and often peak speed; you gain portability to machines with no PyTorch stack. Edge and embedded are its home turf, not desktop chat.

MLX is Apple's array framework for Apple Silicon; the mlx-community org repacks most major releases into MLX format, 4-bit and 8-bit included. On an M-series Mac running mlx-lm, that's your format — see Apple Silicon.

Raw FP16/BF16 checkpoints matter when you want a lossless reference, a fine-tuning base, or a conversion source. BF16 keeps FP32's 8-bit exponent range and trades mantissa bits for it — far more stable for training than FP16, at the same 2 bytes per weight. A 7–8B model at FP16/BF16 is ~16 GB of weights alone, before any context.

Which format for which engine

Mixing formats is the classic beginner mistake: pointing vLLM at a .gguf, or llama.cpp at safetensors shards. Neither works — the engine either errors out or needs a conversion step first.

Conversion paths

The standard path from a Hugging Face repo to a local file is two steps — convert the checkpoint to a 16-bit GGUF, then quantize down:

git clone https://github.com/ggerganov/llama.cpp
pip install -r llama.cpp/requirements.txt

# 1. convert the HF checkpoint (safetensors) to GGUF
python llama.cpp/convert_hf_to_gguf.py ./Qwen3-8B --outtype bf16 --outfile qwen3-8b-bf16.gguf

# 2. quantize to the target size
llama-quantize qwen3-8b-bf16.gguf qwen3-8b-Q4_K_M.gguf Q4_K_M

In practice you rarely convert yourself: quantizers like bartowski and mradermacher publish every size of every notable release within hours. Convert on your own when you have a private fine-tune, want an imatrix-calibrated I-quant, or need a nonstandard bit target — details in quantization.

Embedding models — the retrieval half of RAG — ship in the same formats: GGUF for llama.cpp's embedding endpoint, safetensors for sentence-transformers. Same rule as everything above: match the format to the engine you'll actually run.

Related: Quantization · Model families · Engines · RAG