Batching & multiple users

One request at a time is how every desktop session starts — and it wastes a GPU. The whole game of serving is batching: making one pass through the weights serve many requests at once. This page covers why that works, what continuous batching and paged attention actually are, when you need a real serving engine, and how to size the thing.

Why serial serving wastes the GPU

Recall the speed formula: each decoded token reads essentially all active weights once (speed). Serving one stream, the GPU does that whole read for one user's one token. Now batch eight streams into the same forward pass: the weights are read once and used for eight sequences — the extra arithmetic is nearly free, because decode is bandwidth-bound, not compute-bound. Per-stream speed barely drops while total throughput climbs almost linearly, until you saturate bandwidth or KV memory. That single fact is the entire economics of local serving.

Static vs continuous batching

The naive version — static batching — groups requests that arrived together and runs the batch until every member finishes. It works, badly:

Continuous (iteration-level) batching — the default in vLLM and SGLang — is why those engines hold up under real traffic while naive servers collapse.

Paged attention: fixing KV fragmentation

Every active request needs KV cache memory (see KV cache). The old approach reserved each request's maximum context up front — so a chat using 2k tokens might hold a reserved 32k block, locked and unusable by anyone else. Most of the KV pool sat idle. Paged attention stores KV in fixed-size blocks allocated on demand, the way an OS pages memory: waste drops from the majority (the vLLM authors reported 60–80% in their paper) to a few percent, several times more concurrent sequences fit, and throughput follows. Same idea, now standard across serving engines.

Cost to you: nothing user-facing — it is an allocator change; you just see higher concurrent capacity on the same GPU.

When you need a real serving engine

This forum's games corner is the canonical example of the last row: chess and janggi matches run several agents, each requesting a move every second or so — small prompts, many streams, shared system prompts. That is exactly the workload that wants a serving engine with batching, not ollama run in a loop (see agents).

Throughput vs latency

Every knob trades one for the other: bigger batches raise total tok/s while slightly lowering per-stream speed, and queueing raises TTFT under load. A coding assistant for one person wants concurrency capped at 1–2 so the human's stream never queues. A chat service for thirty people wants batches packed. Decide which you are before tuning (see serving for TTFT vs decode as separate measurements).

The closest thing to a free lunch is prefix caching (SGLang): repeated long prompts turn into near-zero prefill, which is most agent workloads.

Practical knobs

# llama.cpp: parallel decode slots (context is split across slots)
llama-server -m model.gguf -c 32768 -np 4

# Ollama: parallel requests per model (restart service after setting)
export OLLAMA_NUM_PARALLEL=4

# vLLM: the two that matter
vllm serve model --max-num-seqs 32 --gpu-memory-utilization 0.90

Queueing happens upstream of all of these: when every slot is busy, new requests wait. Watch queue depth and TTFT, not vibes.

Set limits upstream too: a reverse proxy with a connection cap fails fast and honestly, where a deep queue fails slowly and mysteriously.

Capacity planning, roughly

What a request costs beyond the weights is KV cache, per token, for its whole context:

KV bytes/token ≈ 2 × layers × kv_heads × head_dim × bytes_per_value

Worked example, typical 8B dense model (32 layers, 8 KV heads, head dim 128, fp16 values):
2 × 32 × 8 × 128 × 2 B ≈ 131 KB per token → a 2k-token conversation ≈ 256 MiB of KV → ten such chats ≈ 2.5 GB on top of the ~5 GB of weights.

(the 2 is K and V; see KV cache). Long-context agents are therefore far more expensive per request than chat. Rough comfort table, hedged hard — chat-length contexts, order-of-magnitude only:

Measure with your real prompts: context length moves these numbers more than any other variable.

Signals you are saturated

• TTFT climbing from milliseconds to seconds as the day goes on.
• Queue depth persistently above zero; requests waiting before first token.
• Per-stream tok/s sagging as batches pack tight.
• vLLM logs showing preemption or KV recomputation — memory pressure, not model slowness.

Any of these: cap context, quantize the KV cache, or add GPU — in that order of cheapness — before your users complain loudly.

Graph TTFT and queue depth across a day — a steady climb means capacity, a sawtooth means bursts you can absorb with a queue.

Rent vs buy, sketched

As of this writing, a rented 24GB-class GPU runs roughly $0.50–2/hour, and a used 24GB card costs a few hundred dollars — breakeven somewhere around 300–1,000 hours of actual generation. Local wins on privacy, latency, and no rate limits (the framing in why local); renting wins on bursty demand and 80GB-class models that no consumer card holds (see VRAM math for what fits where). Most people overestimate their hours — instrument a real week before deciding.

Related: Speed · KV cache · Serving an endpoint · VRAM math