Most homes have a drawer containing three dead batteries, a mystery key, takeout menus from restaurants that closed, and exactly one useful screwdriver.
A very large prompt can become the software version of that drawer. The answer is in there. Probably. We added every document, chat message, tool result, policy, and log line so the model could not possibly complain about missing context. Then we asked it to find the screwdriver.
The junk drawer test
Long context windows are a real capability. They make new workflows possible: reading a codebase, comparing contracts, carrying a long conversation, or reasoning across a research packet without compressing it into dust.
But teams often use capacity as a substitute for selection. Retrieval feels fallible, so they retrieve everything. Summaries lose detail, so they append the full history. Tool output might matter later, so it stays forever.
This is not “more context.” It is postponed information architecture.
Capacity is not recall
The Lost in the Middle experiments showed that language models did not use information equally across long inputs; relevant facts placed in the middle could be harder to recover than facts near the beginning or end.
Models improve, and the exact curve changes. The broader lesson survives: a maximum context length describes what fits through the door. It does not promise equal attention to everything inside.
Every token charges rent
Irrelevant context has several costs. It increases latency and inference spend. It can introduce conflicting instructions, stale facts, and accidental data exposure. It makes traces harder for humans to inspect. Most subtly, it creates more plausible wrong paths for the model to follow.
The most expensive token is not always the one with the highest API price. It is the irrelevant token that distracts the model, causes a bad action, and sends a person searching through 200,000 tokens to understand why.
Pack for the task
Treat context like packing for a trip:
- put the current goal and constraints where they are easy to find,
- retrieve evidence for this step rather than every possible step,
- separate instructions from untrusted content,
- summarize old interaction while preserving links to the originals, and
- drop tool output once its durable consequence is recorded.
A well-packed 20,000-token prompt can outperform a 200,000-token attic. More room is useful. Knowing what not to bring is the actual skill.