Structured output & function calling

Ask a model for JSON and it will happily produce almost-JSON: a trailing comma, an unquoted key, a friendly sentence before the first brace. Free-form sampling can always emit an invalid token, because the sampler knows nothing about your schema. Structured output closes that gap, and function calling is the same mechanism wearing a different hat.

Why free-form JSON breaks

Every generated token is a sample from a distribution. At each step some probability mass — small, but never zero at temperature above 0 — sits on tokens that would break your format: a closing brace before its opener, a comma after the last array element, a lowercase true spelled True (a Python habit baked into training data). Multiply "almost always valid" over 200 output tokens and you get intermittent, non-reproducible breakage — the worst kind of bug. Even temperature 0 only makes output deterministic, not schema-valid.

The solutions ladder

Each rung costs flexibility and buys determinism. Use the top rung whenever the consumer is code.

Grammar-constrained decoding

Compile the schema into a grammar. At every decoding step, each token that could not extend a valid document gets its logit set to −infinity before sampling. Invalid output is not improbable — it is impossible. llama.cpp ships GBNF grammars; a decimal-number grammar is one line:

root ::= "-"? [0-9]+ ("." [0-9]+)?

The same idea wraps JSON Schema. Against a local llama.cpp server (OpenAI-compatible endpoint):

import json, requests

schema = {
    "type": "object",
    "properties": {
        "title":     {"type": "string"},
        "component": {"type": "string",
                      "enum": ["auth", "upload", "ui", "api"]},
        "severity":  {"type": "string",
                      "enum": ["low", "medium", "high"]}
    },
    "required": ["title", "component", "severity"],
    "additionalProperties": False
}

r = requests.post("http://localhost:8080/v1/chat/completions", json={
    "messages": [{"role": "user",
                  "content": "Extract the bug: " + report}],
    "response_format": {
        "type": "json_schema",
        "json_schema": {"name": "bug", "strict": True,
                        "schema": schema}
    },
    "temperature": 0
})
bug = json.loads(r.json()["choices"][0]["message"]["content"])

Ollama does the same with a JSON Schema in the format field; vLLM with guided_json. Same mechanism, three spellings — your engine's exact field names live in engine params.

Constrained is not correct

The grammar guarantees shape, not truth. The model can emit {"severity": "low"} for a crash report, or pad a string field with filler. Masking can also hurt open-ended fields: when the model wants a paragraph break and the grammar forbids the newline, quality drops. Practical rules:

• Narrow enums instead of free strings whenever the value space is known.
• required for every field you consume; additionalProperties: false to stop inventions.
• Keep string fields short; route long prose out of the constrained response.
• Put one worked example of the desired output in the prompt anyway — it steers content, which the grammar cannot.

Function and tool calling

Tool calling is structured output plus a convention: the model returns which function to call and with what arguments; your code executes it and appends the result. The OpenAI tools shape is the de facto dialect local engines imitate:

{
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Current weather for a city",
      "parameters": {
        "type": "object",
        "properties": {
          "city": {"type": "string"},
          "unit": {"type": "string", "enum": ["C", "F"]}
        },
        "required": ["city"]
      }
    }
  }]
}

The model replies with tool_calls in the message; you return output as a role: "tool" message and the loop continues. Models are trained for specific tool dialects — Hermes- and Qwen-lineage templates are the best supported on local engines (llama.cpp with --jinja, Ollama, vLLM with automatic tool choice). A model without tool training produces plausible calls with hallucinated argument names no matter what you constrain, so pick the model for the job first: choosing a model.

Verify both parse and content

Two checks after every call. First, parsing — json.loads succeeds (the grammar usually guarantees this; JSON mode does not always). Second, semantics: required keys present, enums in range, numbers plausible, strings non-empty. On failure, retry once with the validation error fed back into the prompt — constrained retries are cheap and usually succeed on the first bounce.

This exact loop — emit a call, validate, execute, observe — is what agents run all day: agents.

Related: agents · prompting · sampling · rag