The context window is a budget, not a memory
A context window is how much text a model can consider in one request, and it resets every time. Filling it is not free: cost and latency rise with length, and retrieval accuracy degrades for material buried in the middle. Treat the window as a budget to spend deliberately rather than a space to fill.
When context windows grew from a few thousand tokens to hundreds of thousands, a lot of people concluded the memory problem was solved. It was not. The window got bigger; it is still refilled from empty on every request.
Every request starts from nothing
A model retains nothing between calls. The reason a chat feels continuous is that the application resends the conversation each turn, so the model can condition on it.
Once you see it that way, the engineering question becomes obvious: as a conversation grows, what do you keep? Everything, until it no longer fits? The last N turns? A running summary plus the recent turns? That choice is the memory design, and the model has no opinion on it.
Long contexts are not free
You pay for every token you send, on every turn. A conversation that carries its full history grows quadratically in total cost — turn ten resends nine turns of history.
Latency follows the same curve. A user who notices the assistant getting slower as the conversation goes on is not imagining it.
Prompt caching changes the arithmetic
Where a provider supports caching a stable prefix, the economics shift substantially: put the unchanging material first, keep the variable part at the end, and the repeated portion is billed at a fraction of the rate. Designing your prompt so a long prefix stays byte-identical is worth real money.
The middle is where things get lost
Recall is not uniform across a long context. Material near the beginning and near the end is used reliably; material in the middle is the most likely to be overlooked. This holds even when the window is nowhere near full.
The practical response is to place what matters at the edges. Instructions at the top, the most relevant retrieved passage close to the question, and anything merely supporting in between.
Spend the budget deliberately
Treat every token as a choice. Retrieve five relevant chunks rather than fifty mediocre ones. Summarise old turns instead of carrying them verbatim. Cut the boilerplate preamble that has been copied between prompts since the first prototype.
A smaller, better context usually outperforms a larger one. The window is how much you may spend, not how much you should.
Frequently asked questions
If the window is large, can I just send everything?
You can, and quality often drops. Long contexts dilute attention, bury the relevant passage among irrelevant ones, and cost more on every turn. Sending the right three paragraphs generally beats sending thirty.
How does a chat assistant remember earlier messages?
It does not. The application resends the conversation with each request. When the history outgrows the window it has to be summarised or trimmed, and that trimming decision is a real piece of product design.
