A chat completion is a list of messages in, one message out
Most people's first mental model of "calling an LLM" is calling any other API: send a request, get a response, done. That model holds up right up until the bill arrives, or the fifth message in a conversation costs noticeably more than the first one did. An LLM API isn't a function that answers a question. It's a function that continues a list of messages, and everything in this article follows from that.
A single chat completion call
1response = client.chat.completions.create(2 model="gpt-4o",3 messages=[4 {"role": "system", "content": "You are a helpful assistant."},5 {"role": "user", "content": "What's the capital of France?"},6 ],7)89print(response.choices[0].message.content)
Three things worth noticing before the rest of this series builds on them. messages is a list, not a single prompt string, each entry tagged with a role (system, user, assistant), the actual subject of the next article in this series. model picks which model answers, and different models on the same API can vary wildly in cost, speed, and capability. choices is a list too, because you can ask for more than one candidate completion back from a single call.
This shape isn't universal by accident. Most providers converged on something close to it, and plenty of open-weight model servers deliberately expose an OpenAI-compatible endpoint even when the model underneath has nothing to do with OpenAI, specifically so existing client code keeps working unchanged. Learn this shape once and most of what you build transfers across providers with a changed base URL and a changed model name, not a rewrite.
Tokens are what you're billed for, and what you're waiting on
A traditional API charges per request, or doesn't charge per call at all. An LLM API charges per token, for both the tokens you sent and the tokens it generated back, and a token isn't a word.
Counting tokens, not words
1enc = tiktoken.encoding_for_model("gpt-4o")2print(len(enc.encode("Hello, world!")))
Two words, four tokens: the tokenizer (covered in depth in the LLMs from Scratch series if you want the internals) splits text into subword chunks, not words, and punctuation and whitespace often get their own tokens too. A rough working estimate for English text is about four characters per token, close enough to reason about cost and context limits day to day, not close enough to bill against precisely.
The pricing split matters as much as the count. Output tokens are typically several times more expensive per token than input tokens, because generating each one requires a full forward pass through the model, one token at a time, while input tokens get processed in parallel in a single pass. A request that reads a long document and answers in one word is cheap. A request that reads one sentence and writes three paragraphs back is not, even though the second one "feels" like less work went into the prompt.
Tokens are also the latency unit, not just the cost unit. Output is generated one token at a time, so asking for a longer response doesn't just cost more, it takes proportionally longer to finish, in a way a traditional API call generally doesn't.
The context window is the one resource you actually manage
The API call in the first section had no memory of anything. Every model behind one of these APIs is stateless between calls: it doesn't remember your last message, your last conversation, or that you exist at all, unless you send that history back yourself, every single time.
A four-message conversation, resent in full on the next call
1messages = [2 {"role": "system", "content": "You are a helpful assistant."},3 {"role": "user", "content": "What's 2+2?"},4 {"role": "assistant", "content": "4."},5 {"role": "user", "content": "And that times 10?"},6]7# every one of these four messages gets sent, and billed, on this call
By turn ten of a real conversation, you're resending and re-paying for the first nine turns on every single call, not just the newest message. This is what the context window actually is: the maximum number of tokens, input and output combined, that one call can hold. It varies by model, and it's a hard limit, not a soft one.
Some providers offer a "thread" or "session" API as a convenience wrapper, where you don't manually resend history yourself. Underneath, the same thing is still happening: the full history is still being assembled and sent to the model on your behalf, still counted, still billed, still bounded by the same limit. The wrapper hides the bookkeeping. It doesn't change what the model actually receives.
What happens when a naive chat loop hits the limit
Put the last two sections together and the failure mode writes itself.
A chat function that keeps every message forever
1history = [{"role": "system", "content": "You are a helpful assistant."}]23def chat(user_message):4 history.append({"role": "user", "content": user_message})5 response = client.chat.completions.create(model="gpt-4o", messages=history)6 reply = response.choices[0].message.content7 history.append({"role": "assistant", "content": reply})8 return reply
history only ever grows. Cost per call climbs turn over turn, long before anything breaks, because every earlier turn is still being resent. Eventually a call doesn't come back slower or more expensive, it fails outright: the request gets rejected for exceeding the model's maximum context length. Not a warning on the way there, a hard error, usually arriving exactly when a long-running conversation has become the most valuable to whoever's having it.
Nothing about this function is unreasonable to write, it's the first version almost anyone building a chat feature writes, and it's exactly why the rest of this series exists. Deciding what actually belongs in that growing message list, instead of everything that's ever been said, is its own discipline, covered in Context Engineering two articles from here. Deciding what to remember across an entire relationship with a user, not just within one growing list, is covered later in Memory for AI Applications. Both exist because of the one constraint this article has been building to: the window is finite, and something has to decide what fills it.