Context Engineering

A bigger window doesn't mean every token in it gets equal attention

The natural response to a model losing track of something is to paste in more context, the whole document instead of a page of it, the whole conversation instead of a summary. Research on long-context models keeps finding the same shape of problem: performance on information in the middle of a long context degrades measurably compared to information near the start or end, well before the model hits its stated context limit.

Put a clear system instruction first, fifty pages of pasted reference material next, and the user's actual question last, and the instruction from the very start of the prompt is exactly the thing most likely to get followed loosely by the time the model reaches the end. Not because the model "forgot", everything is still technically in the context, but because more tokens competing for the same attention mechanism means every individual token gets a thinner slice of it.

This is the actual argument for context engineering as its own discipline. The context window isn't a bucket you fill until it's full. It's a limited resource where what you put in it, and in what order, changes how reliably the model uses any of it.

Paste the relevant ten lines, not the file they live in

Asking a question about one function and pasting the ten-thousand-line file it lives in is the path-of-least-resistance move, and it's the exact shape of problem the last section described: the function you care about is one small island in a sea of tokens the model has to weigh against everything else in the file.

1# not needed: the other 200 functions in the file
2# needed: the function itself, and the two callers that invoke it
3# needed: the type definitions its arguments and return value use
4# not needed: the import statements for unrelated modules

The discipline is deciding, deliberately, what's actually relevant to the question being asked, not what's easiest to copy and paste. At small scale that's a judgment call you make by hand. At the scale of an entire codebase or document library, making that judgment call automatically, for every query, is exactly what retrieval systems exist to do, covered starting with the next two articles in this series.

The final prompt is assembled, not typed

Past a simple chatbot, the messages sent to the model rarely come from one place. A system instruction, retrieved documents, conversation history, and a tool result all need to land in the same request, and gluing them together with string concatenation wherever they happen to be needed is how a token budget gets blown without anyone deciding it should.

1def assemble_prompt(system, retrieved_docs, history, user_message, max_tokens=8000):
2    reserved_for_system = 300
3    reserved_for_user_message = count_tokens(user_message) + 50
4    remaining = max_tokens - reserved_for_system - reserved_for_user_message
5
6    docs_budget = int(remaining * 0.6)
7    history_budget = remaining - docs_budget
8
9    docs = fit_to_budget(retrieved_docs, docs_budget)
10    trimmed_history = fit_to_budget(history, history_budget, keep="most_recent")
11
12    return build_messages(system, docs, trimmed_history, user_message)

Nothing here is complicated, it's arithmetic. The point is that the budget is decided once, explicitly, as part of building the prompt, instead of discovered later as a context-length error in production or, worse, never discovered at all because everything technically still fit under the limit while quietly diluting the parts that actually mattered.

When the budget doesn't fit everything, something has to lose first

Given a fixed budget and more candidate content than fits, the order you cut in isn't arbitrary. Some rules hold up across almost every application.

The system instruction and the user's current message are the two things that never get cut, they're small and they define what the request even is. Older conversation turns are usually the safest thing to trim first, the tenth message back is rarely as load-bearing as the first one or the most recent one, and a summary of what was cut can recover most of the value at a fraction of the token cost, the exact technique the Memory article in this series covers in depth.

Large reference material is the trickier case: dropping it entirely loses information the user might actually need, but including all of it is what caused the problem in the first place. The better move, in almost every case, is retrieving only the relevant slice instead of choosing between "all of it" and "none of it", which is precisely the problem the rest of this series turns to next.