Latency, Cost, and Streaming

The tokens exist one at a time, so the user can see them one at a time

What Is an LLM API covered output being generated token by token, which is also why a response can be shown to the user as it's generated instead of held until the whole thing finishes. Total generation time doesn't change, the model still takes as long as it takes, but perceived latency drops sharply: the first token appearing in a few hundred milliseconds reads as fast, even for a response that takes ten full seconds to finish.

1stream = client.chat.completions.create(
2    model="gpt-4o",
3    messages=messages,
4    stream=True,
5)
6
7for chunk in stream:
8    token = chunk.choices[0].delta.content
9    if token:
10        print(token, end="", flush=True)

Streaming trades one thing for the perceived-speed win: you can't validate or parse the full response until it's done arriving, which matters for the structured-output and tool-calling cases from earlier in this series, where the output needs to be complete and valid JSON before anything downstream can safely act on it. Stream for a chat interface a person is reading. Wait for the full response where something else is about to parse it.

Paying full price to reprocess the same system prompt on every call is a choice

A long system prompt, or a large document included as context, is often identical across many calls, the same instructions, the same reference material, only the user's message actually changing. Reprocessing that identical prefix from scratch on every single call is wasted work, and most providers offer prompt caching specifically to avoid it.

A cache hit is both cheaper and faster for the cached portion, but only for the exact prefix that matches, byte for byte, a previous call. That constraint changes how a prompt gets structured in practice: the stable content, system instructions, reference documents, tool definitions, belongs at the start, and the part that changes every call, the user's actual message, belongs at the end. Put the variable part first and every call invalidates the cache before it can help.

This is the same discipline as the prompt assembly pipeline from Context Engineering, just with an extra constraint: order isn't only about what the model attends to well, it also decides whether you're paying to reprocess the same ten thousand tokens on every one of ten thousand calls.

Most requests are easy. Only the hard ones need the expensive model.

Sending every request to the largest, most capable, most expensive model available is the simplest thing to build and often the most wasteful, because most real traffic is the easy majority of cases a cheaper, faster model already handles correctly.

1result = run_with_model("gpt-4o-mini", request)
2
3if not passes_confidence_check(result):
4    result = run_with_model("gpt-4o", request)  # escalate only the hard cases
5
6return result

The confidence check is doing the real work here, and it's the same kind of check this series has already covered elsewhere: a schema validation failure, a low score from an LLM-as-judge pass, or a rule specific to the task, like an unusually short or generic response. Get that check wrong and a cascade either escalates almost everything, erasing the cost savings, or escalates almost nothing, quietly shipping the cheap model's mistakes. It's worth tuning against the same golden set the rest of this series builds pipelines around, not set once and left alone.

The average latency was fine. The users complaining weren't hitting the average.

An average latency number can look completely healthy while a real fraction of requests take five times as long, and those are exactly the requests a frustrated user remembers. Track p50 and p95 separately, not just the mean, the same way any latency-sensitive system does, an LLM call is no exception just because the workload is different.

Cost deserves the same scrutiny past the raw total spend. Cost per request is easy to track and easy to misread: a cheap pipeline that's wrong often enough to generate support tickets and manual review can cost more in total than a slightly pricier one that's reliable, which is why cost per successful outcome, not cost per call, is the number actually worth optimizing.

None of these numbers are useful without somewhere to actually see them, per request, over time, broken down by which model handled it. That instrumentation is the subject of the Observability article later in this series.