Structured Output and Tool Calling

Asking nicely for JSON isn't the same as requiring it

Ask a model to "return JSON" in plain English, and most of the time it will, wrapped in a markdown code fence, prefaced with "Sure, here's the JSON:", or one trailing comma away from actually parsing.

1response = client.chat.completions.create(
2    model="gpt-4o",
3    messages=[{
4        "role": "user",
5        "content": "Extract the name and age from: 'John is 34 years old.' Return as JSON.",
6    }],
7)
8print(response.choices[0].message.content)

Feed that string straight to json.loads() and it throws, not because the model misunderstood the task, the extracted values are both correct, but because a trailing comma after 34 isn't valid JSON, and the code fence and preamble around it aren't JSON at all.

The model is predicting the next token of text, the same as it does for any other request. "Return JSON" is a request made in English, and English requests get interpreted, not enforced. Getting output a parser can actually rely on means moving the requirement out of the prompt and into the API call itself.

JSON mode fixes syntax. It doesn't fix shape.

Most providers offer a structured output mode that constrains generation itself, not the wording of the prompt, so that the model is only allowed to produce tokens that form valid JSON. No code fence, no preamble, no trailing comma, because those aren't reachable outputs anymore.

1response = client.chat.completions.create(
2    model="gpt-4o",
3    messages=[{
4        "role": "user",
5        "content": "Extract the name and age from: 'John is 34 years old.'",
6    }],
7    response_format={"type": "json_object"},
8)
9print(response.choices[0].message.content)

Guaranteed valid JSON is not the same as guaranteed matching JSON. Nothing stops this mode from returning {"person_name": "John", "years_old": 34} instead, syntactically perfect, and still not the shape your code downstream expects. The model is free-forming the keys unless you constrain those too.

1class PersonExtraction(BaseModel):
2    name: str
3    age: int
4
5data = PersonExtraction.model_validate_json(response.choices[0].message.content)

Two separate layers, doing two separate jobs: the API's structured output mode guarantees syntax, a schema validator guarantees shape. Skipping the second layer because the first one already sounds like "structured" is how a wrong field name reaches production instead of failing loudly in a validator where it belongs.

Tool calling is the same mechanism, aimed at a decision instead of a blob

Instead of constraining the model to any valid JSON object, tool calling constrains it to one of a specific set of named functions, called with arguments matching that function's own schema.

1tools = [{
2    "type": "function",
3    "function": {
4        "name": "get_weather",
5        "description": "Get the current weather for a city",
6        "parameters": {
7            "type": "object",
8            "properties": {"city": {"type": "string"}},
9            "required": ["city"],
10        },
11    },
12}]
13
14response = client.chat.completions.create(
15    model="gpt-4o",
16    messages=[{"role": "user", "content": "What's the weather in Lisbon?"}],
17    tools=tools,
18)
19
20call = response.choices[0].message.tool_calls[0]
21print(call.function.name, call.function.arguments)

The model didn't answer the weather question, it isn't connected to a weather service and has no way to know today's forecast. It decided which function answers this kind of question and produced the arguments to call it with. Nothing has executed yet, the model only emitted a structured decision.

Your code runs the real function, and the result goes back into the message history as its own message, not folded into the model's output:

1result = get_weather("Lisbon")
2messages.append({"role": "assistant", "tool_calls": [call]})
3messages.append({
4    "role": "tool",
5    "tool_call_id": call.id,
6    "content": json.dumps(result),
7})
8# call the API again with this appended; the model now has a real result to answer from

Decide, execute, feed the result back, let the model continue: that loop, repeated, is the entire mechanism behind an agent. Agents and Tool Use, later in this series, builds directly on it.

The model decides. Your code still has to check.

This is the real API surface for building anything past a chatbot. Once a model can reliably emit a structured, schema-matched decision, you can wire it into a database, an internal service, a browser, whatever your product needs, because the interface between "the model decided" and "something happened" is just a function call like any other.

Just like any other function call is exactly how it needs to be treated. The arguments in a tool call are model output, not verified input, the model can hallucinate a plausible but wrong city name, misread the user's intent, or, once the Guardrails article covers prompt injection, be manipulated by text it read into calling something it shouldn't. "The model asked for it" is not authorization, and a tool call's arguments deserve the same treatment any other untrusted input reaching your code does: validated types and ranges, an allowlist where one applies, and a real permission check against the actual user behind the request, not just against whether the model produced well-formed JSON.

That includes never letting a tool's arguments get concatenated into a raw SQL query or a shell command, the same rule for the same reason as any other untrusted string reaching that code, model-generated or not. The model's job is to decide what should happen. Your code's job is still to decide whether it's allowed to.