localaiforum
Log in
Guide › Running & serving

Serving an OpenAI-compatible endpoint

Run an engine once and talk to it in a chat window, and you have a toy. Serve it over HTTP in the shape everyone already speaks, and you have infrastructure: editors, UIs, agents, and your own scripts can all drive the same local model. This page gets a local endpoint up, then covers streaming, SDKs, security, and warm loading.

Why the OpenAI shape won

The chat-completions request format — a model name, a list of {role, content} messages, a few knobs — became the de facto standard because every tool author wrote a client for it first. Whatever you think of the company, the shape is neutral: it is just JSON over POST. Practical consequence: any local server that mimics it is instantly compatible with an enormous pile of existing software. Your engine choice and your client choice become independent decisions (see engines).

llama-server in one minute

llama-server -m model.gguf -c 8192 --port 8080
# UI at http://127.0.0.1:8080 — API at /v1/chat/completions

An annotated request:

curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"model","messages":[{"role":"user","content":"Why run local? One sentence."}],"temperature":0.7,"max_tokens":60}'
  • model — llama-server serves whatever you launched; the field is often ignored, but send a sane name.
  • messages — the whole conversation, system prompt first; stateless server, you hold the history.
  • max_tokens — cap the reply; protects you from runaway generations.

The response (trimmed) carries the answer plus the token accounting you will want for cost-free debugging:

{
  "choices": [
    { "message": { "role": "assistant", "content": "No accounts, no rate limits, and your data never leaves the machine." } }
  ],
  "usage": { "prompt_tokens": 14, "completion_tokens": 15 }
}

usage is how you learn what your prompts actually cost in context — check it before blaming a model for "forgetting" (see tokens).

Streaming

Add "stream": true and the server switches to server-sent events: newline-delimited data: chunks, each carrying a small delta, terminated by a sentinel.

data: {"choices":[{"delta":{"content":"No "}}]}
data: {"choices":[{"delta":{"content":"accounts"}}]}
data: [DONE]

Client-side: accumulate delta.content until the [DONE] line. Streaming does not make generation faster — it makes waiting tolerable, which for chat UX is the same thing (see speed).

Ollama: one service, two APIs

Ollama's native API is slightly richer — per-request options and model lifecycle control:

curl http://127.0.0.1:11434/api/chat -d '{"model":"qwen3:8b","messages":[{"role":"user","content":"hi"}],"stream":false,"keep_alive":-1}'

It also serves the OpenAI-compatible route at /v1/chat/completions, so OpenAI-shaped clients work against Ollama without changes. Use the native API when you want keep_alive or per-request engine options; use /v1 for everything else.

vLLM for GPU servers

vllm serve Qwen/Qwen3-8B --max-model-len 16384 --port 8000

vLLM pulls safetensors weights, loads once, and serves the same OpenAI shape with continuous batching underneath — the right tool once concurrent users appear (see concurrency). It wants NVIDIA VRAM and a real GPU; do not bother with it for a single-user desktop.

Point the official SDKs at localhost

The official OpenAI libraries are just HTTP clients; swap the base URL and the API key field becomes decorative:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local")
r = client.chat.completions.create(
    model="qwen3-8b",
    messages=[{"role": "user", "content": "Why run local?"}],
)
print(r.choices[0].message.content)

import OpenAI from "openai";

const client = new OpenAI({ baseURL: "http://127.0.0.1:11434/v1", apiKey: "ollama" });
const r = await client.chat.completions.create({
  model: "qwen3:8b",
  messages: [{ role: "user", content: "Why run local?" }],
});

Any string works for the key unless the server has one configured. This base-URL swap is the whole trick behind every "works with local models" tool.

What this unlocks

  • Editors and chat UIs — see UIs & integrations.
  • Your own apps: a local model behind your own backend, zero per-token cost.
  • Scripted pipelines with guaranteed schema — see structured output.
  • Agent loops that call tools and iterate — see agents.
  • This forum's games corner: the chess and janggi agents play by hitting a local chat-completions endpoint exactly like the curl above.

Lock the door

A model server is a free LLM for anyone who can reach the port. Defaults are safe — llama-server and Ollama bind 127.0.0.1 — so keep it that way on laptops. To expose on a LAN: set an API key (--api-key on llama-server and vLLM, and auth in any UI in front), and prefer a reverse proxy for TLS. Never bind 0.0.0.0 unauthenticated: anyone on the network can burn your GPU, fill your context, and probe your setup.

Keep it warm

Cold load — weights from disk into VRAM — costs seconds to tens of seconds of TTFT. llama-server and vLLM load at startup and stay resident, so only the first request pays. Ollama unloads after keep_alive (default ~5 minutes): send "keep_alive": -1 to hold the model indefinitely, or 0 to free VRAM immediately after a burst. If your setup feels "slow at first, fine after," that is a cold load, not a slow model — and under load, a load storm is a real failure mode (see concurrency).

Related: Concurrency · UIs & integrations · Engines · Structured output