Understanding Context Windows and Token Limits: A Developer's Guide
Why a model that 'read the whole file' can still miss something in it, what a token actually is, and the practical rules for managing context in long AI-assisted coding sessions.
"Just paste the whole codebase in" runs into a hard limit fast, and the limit isn't measured in files or characters — it's measured in tokens. Understanding what a token is and how context windows actually behave explains a whole category of confusing AI-assistant behavior: why long sessions get worse, why the model "forgot" something you told it twenty messages ago, and why pasting more context isn't always better.
What a token actually is
A token is roughly three-quarters of a word in English — common words are often one token, longer or less common words split into pieces ("tokenization" might become "token" + "ization"). Code tokenizes less efficiently than prose: punctuation-heavy syntax, long variable names, and indentation all cost tokens. As a rough rule of thumb, 100 tokens is about 75 words of English prose, or noticeably less code.
What the context window actually limits
The context window is the maximum number of tokens the model can attend to in a single request — and critically, this includes everything: the system prompt, every prior message in the conversation, every file you've pasted, and the response the model is about to generate. A long conversation doesn't get a fresh budget each turn; it's all one running total.
| What counts against the context window | Common mistake |
|---|---|
| Every previous user and assistant message in the session | Assuming a long conversation history is "free" because you already paid for it in earlier turns |
| Every file or code snippet pasted, including ones no longer relevant | Leaving stale context (an old version of a file you've since changed) in the conversation, competing for attention with the current version |
| The system prompt / tool definitions in an agentic coding tool | Not realizing a coding assistant's tool descriptions and instructions already consume a meaningful chunk of the budget before your first message |
| The model's own output as it generates | Asking for a huge single-pass output (e.g. "write the whole module") that itself risks hitting the limit mid-generation |
Why "more context" isn't strictly better
Even within the limit, models attend more strongly to context near the end of the prompt than context buried in the middle — a phenomenon often called "lost in the middle." Pasting an entire 2,000-line file when only one function is relevant doesn't just cost tokens; it dilutes the model's attention away from the part that actually matters, and increases the odds it references or gets confused by the irrelevant parts.
Paste the smallest context that fully specifies the task — the relevant function plus its direct dependencies, not the whole file; the whole file, not the whole module. Quote exact code rather than describing it from memory when precision matters (a debug session, an API signature) — summaries lose the details that matter most.
Managing context in long sessions
- Start a fresh conversation for a new, unrelated task rather than continuing an old one — old context that isn't relevant anymore still competes for attention and eats budget.
- When a file changes, re-paste the current version rather than assuming the model will track the diff from an earlier paste — it won't unless the tool you're using explicitly re-reads the file.
- For genuinely large tasks, decompose into steps that each fit comfortably in context, rather than one prompt trying to hold the entire task's context at once — this is also why chain-of-thought and explicit planning prompts work well for big tasks.
- If a coding assistant supports it, prefer tools that read files on demand over pasting everything up front — you only pay the token cost for what's actually needed.
Context window size isn't the whole story
A larger context window (some current models support hundreds of thousands of tokens) raises the ceiling but doesn't remove the "lost in the middle" effect or the cost of irrelevant context. Fitting something in the window and getting good attention on it are different properties — the discipline of giving precise, relevant context matters at any window size, it just becomes optional to ignore at small scale and mandatory to respect at large scale.
The Foundations program covers this as part of the core mental model of how LLMs process text — it's the same underlying mechanism that explains hallucination, prompt structure, and why quoting beats summarizing when precision matters.