Context Windows, Context Selection, and Context Compression

A context window is the fixed slice of text — system prompt, conversation, tool output, and retrieved documents together — that a model can actually see while generating one response; context selection and context compression are the two disciplines that decide what earns a seat inside that slice. 🧠

This sounds like a plumbing detail until an agent silently forgets a decision it made forty turns ago, a support bot misses the one clause that mattered in a 40-page contract, or a token bill triples because every turn re-sends a full document nobody re-read. Teams that treat the window as infinite storage instead of a finite, shared, and increasingly degradable resource end up debugging "the model got dumber" issues that are really context design issues in disguise. 💸

📑 In This Post

1. What a Context Window Actually Is
2. Context Rot: Why More Tokens Isn't Automatically Better
3. Context Selection: Choosing What Gets In
4. Context Compression and Compaction
5. Prompt Caching: The Companion Discipline
6. Hands-On Lab: Feel Context Rot Yourself
7. Rolling This Out at Enterprise Scale
8. Common Mistakes
9. FAQ
10. References & Further Reading
11. Summary

🔀 Quick Comparison

Approach	What It Actually Does	Token Cost Pattern	Where It Breaks
Full-Context Stuffing	Pastes every document, log, or turn into the prompt and lets the model sort it out	Grows linearly with everything ever seen; no reuse	Hits window limits and context rot at the same time — cost and quality degrade together
Context Selection (RAG)	Searches a larger corpus and pulls in only the passages relevant to this turn	Small, roughly constant per query, independent of corpus size	Retriever misses the right chunk, or query phrasing doesn't match the stored text
Context Compression / Compaction	Shrinks or offloads what's already in the window: trims stale tool output, summarizes old turns, or writes facts to external memory	Grows sub-linearly across a long session instead of linearly	Summarization drops a detail the agent needed three steps later

1. What a Context Window Actually Is

🧒 Kid analogy: imagine you're allowed to bring exactly one backpack to school, and everything you might need that day — books, lunch, homework, a note from your mom — has to fit inside it. You don't get a second backpack halfway through the day. Whatever isn't in the bag when the bell rings, you simply don't have. A context window is that backpack for a language model: whatever isn't packed into it for this particular turn doesn't exist for the model, no matter how "smart" the model is in general.

Anthropic's own documentation describes the context window as everything the model can reference while generating a response — the system prompt, every prior message, tool results, images, documents, and the model's own output for that turn — all sharing one token budget, and explicitly separate from the much larger body of data the model was trained on. Training data is what the model generally knows; the context window is what it can see right now, for this exact request.

Real-world example: Anthropic's engineering team frames this constraint concretely in their guide to long-running agents. A coding agent working across many hours or days cannot simply keep one context window open the entire time — the window is finite, and most non-trivial projects can't be completed inside a single one. Their solution treats each work session like a shift change between engineers: a new session starts with zero memory of the last one, so the team built a persistent claude-progress.txt file alongside git history specifically so a fresh context window can reconstruct "where we left off" without re-reading everything that happened before.

✅ Worked example: A customer-support agent handling one ticket needs the current ticket text, the last few turns, and maybe one knowledge-base article. That's a few thousand tokens — comfortably inside almost any window, with room to spare for reasoning.

💡 Harder case: The same agent, after four hours of continuous operation across dozens of tickets, has accumulated every ticket, every tool call, and every draft reply it ever produced — even though only the current ticket matters right now. The backpack is full of yesterday's lunch.

Mechanically, three things matter about window size in production: (1) the limit is a hard ceiling — a request that exceeds it simply fails, it doesn't get politely truncated by default; (2) the limit includes the model's own output space, so a long expected answer eats into the same budget as the input; and (3) window sizes and pricing shift frequently across providers and model versions, so a number you hard-code today is a support ticket waiting to happen next quarter — always check the live model listing rather than a cached number in your head or in an old blog post.

🎯 Use this when: sizing any new integration — before writing a single line of retrieval or compression logic, know what the actual per-request budget is for the model you're targeting, and treat it as a moving target, not a constant.

2. Context Rot: Why More Tokens Isn't Automatically Better

🧒 Kid analogy: picture that same backpack, except now it's crammed absolutely full — every notebook from every subject this year, every worksheet you've ever gotten, three broken pencils. Technically nothing fell out, technically it all still fits. But now when your teacher asks "where's today's math homework?", it takes you five minutes of digging through a mess to find one sheet of paper that's buried in the middle. Having the paper isn't the same as being able to find it fast.

Anthropic's documentation names this phenomenon directly: as token count grows, a model's accuracy and recall degrade, even while the content technically remains inside the window. They call it context rot, and it's one reason their own guidance is explicit that "more context isn't automatically better" — curating what's in the window matters just as much as how large the window is.

Real-world example: Anthropic's guide to effective context engineering treats long-horizon coherence as an engineering problem to be solved deliberately, not something a bigger window fixes on its own — it explicitly frames context as a finite resource with diminishing marginal returns, arguing that teams should design for the smallest set of high-signal tokens that gets the job done rather than the largest set that technically fits. Their internal comparisons of agent harnesses back this up in practice: agents built to actively curate what stays in context outperform ones that simply accumulate everything and hope the model sorts it out.

✅ Worked example: Placing the single most important instruction or fact at the very start or the very end of a long prompt — the two positions the rot curve above treats most reliably — instead of burying it in paragraph fourteen of twenty.

💡 Key warning: Context rot is not the same failure as running out of window space. A request can be well under the token limit and still produce a worse answer than a shorter, more curated version of the same request — this is exactly why the support-agent backpack from Section 1 needs active cleanup, not just a bigger backpack.

🎯 Use this when: diagnosing a quality regression on a long-context task — before assuming the model got worse, check whether the fact it missed was buried in the middle of a bloated prompt.

3. Context Selection: Choosing What Gets In

🧒 Kid analogy: if you're writing a report on frogs, you don't check out every book in the library and lug the whole stack home — you ask the librarian to help you find the two or three books that are actually about frogs, and you bring those. Context selection is asking a librarian instead of hauling the whole library.

Technically, this is retrieval-augmented prompt construction: a search step runs over a large corpus (documents, tickets, code, memory files) and returns only the pieces relevant to the current query, which then get inserted into the prompt instead of the corpus itself. The design goal is to keep recall high at the search stage — cast a wide net over the whole corpus — while keeping token cost low at the prompt-construction stage, by only forwarding what actually mattered.

Real-world example: LangChain's own engineering blog documents exactly this pattern as a named abstraction: a ContextualCompressionRetriever that wraps a base retriever and passes its results through a document compressor before they ever reach the model. One compressor variant uses an LLM call to extract only the sentences relevant to the query from each retrieved document; a second, cheaper variant embeds the query and each document and discards anything below a similarity threshold. The stated motivation from the team was direct: it lets an application be generous at the search stage — return more candidate documents, favoring recall — precisely because the compression stage is responsible for stripping out the irrelevant parts before they cost tokens.

Anthropic's own agent-design guidance describes a closely related idea for agentic tool use rather than document retrieval: instead of preloading an entire codebase or corpus into context up front, store lightweight references — file paths, document IDs, query hashes — and fetch the full body only when the current plan actually calls for it. A coding agent, for instance, can be handed a list of file paths and a "read file" tool rather than having every file's full contents pasted into every turn whether or not that turn touches them.

Here is the general shape of that pipeline, stage by stage:

1. Index once, ahead of time: chunk the corpus and embed each chunk into a vector store, so search doesn't require re-reading everything per query.
2. Retrieve broadly per query: run a similarity search and pull back more candidates than you expect to need — over-fetching here is cheap because nothing has hit the model's context yet.
3. Compress or filter the candidates: extract only the query-relevant spans, or drop candidates below a similarity threshold, so what remains is short and on-topic.
4. Insert into the prompt with provenance: attach a source or citation to each surviving excerpt so downstream answers can be checked against it.
5. Fetch full bodies just-in-time, if needed: if the model's next step genuinely requires the complete document rather than the excerpt, retrieve it on demand instead of upfront.

Without this stage, a production system either truncates arbitrarily when a corpus outgrows the window — silently dropping whatever happened to fall off the end — or it stuffs the full corpus in every time and pays full-context prices, and full-context rot, on every single query.

✅ Worked example: here's the shape of a minimal retrieval-plus-compression step, walked through before you look at the code. It takes a user's query and does four things in order: (1) asks the retriever for the 12 documents most similar to the query, casting a wide net on purpose; (2) throws out any candidate whose relevance score to the query is 0.72 or lower, so only genuinely on-topic documents survive; (3) from the survivors, keeps just the top 4 and, for each one, pulls out only the sentences that actually answer the query rather than the whole document; and (4) joins those short excerpts together and drops them into a final prompt template, right next to the original question. The result handed to the model is a few short paragraphs, not the 12 full documents it started from — recall stayed wide in step 1, but only the useful, filtered text made it into the context window by step 4.

candidates = retriever.search(query, top_k=12)
kept = [doc for doc in candidates
        if compressor.relevance_score(doc, query) > 0.72]
excerpt = "\n\n".join(c.extract_relevant_span(query) for c in kept[:4])
prompt = f"Answer using only this context:\n{excerpt}\n\nQuestion: {query}"

💡 Contrasting case: If the same retriever's similarity threshold is set too loosely, it starts returning near-duplicates and tangentially related chunks — you've replaced "the whole library" with "half the library," which still triggers the context rot from Section 2, just at a smaller scale.

🎯 Use this when: the source material (a knowledge base, a codebase, a ticket history) is larger than a single context window could ever hold — selection isn't optional at that point, it's the only way in.

4. Context Compression and Compaction

🧒 Kid analogy: selection is choosing which books to bring home. Compression is what you do once you're already carrying them and your arms are getting tired: you might summarize a long chapter into three bullet points on an index card, put a book you're done with back on the shelf but keep the index card, or ask a friend to hold a book for you until you actually need it again. The information isn't gone — it's just not all riding in your arms at once.

This matters most in long-running, multi-turn agent sessions, where the window doesn't start full — it fills up gradually as tool calls, search results, and reasoning steps accumulate turn after turn, long after most of that material has already served its purpose.

Real-world example: Anthropic ships this as a concrete set of API-level mechanisms rather than leaving it purely to prompt design. Context editing (a beta capability enabled with the context-management-2025-06-27 header) automatically clears old tool-use/result pairs and older extended-thinking blocks once conversation context passes a configured threshold, replacing each cleared item with a placeholder so the model knows something was removed even though it can no longer see what. A separate compaction mechanism takes a coarser approach: instead of surgically removing individual items, it has the model summarize the earlier conversation and continues the session from that summary. A memory tool addresses a third failure mode — durable facts that shouldn't vanish at all — by letting the model write to and read from a file-based memory directory that persists outside the context window entirely, so an agent can accumulate project knowledge across sessions without keeping every detail loaded at once. In a self-reported 100-turn web-search evaluation, Anthropic found that combining context editing with the memory tool let an agent complete workflows that would otherwise fail from context exhaustion, while substantially cutting token consumption over the course of the run — a reminder that this is self-reported benchmark data from the vendor's own announcement, useful as a directional signal but worth validating against your own workload before treating the exact percentage as guaranteed.

Here's the general compaction flow, stage by stage, for a session approaching its budget:

1. Monitor consumption continuously: track token usage per turn rather than discovering the problem only when a request fails.
2. Trigger at a threshold, not at the ceiling: start trimming or summarizing well before the hard limit — commonly discussed guidance is to act once history plus retrieved content approaches roughly 70–80% of the effective budget, leaving headroom for the current turn's reasoning and output.
3. Trim what's genuinely disposable first: stale tool output and old reasoning traces the model has already acted on are usually safe to clear or placeholder-out before anything else.
4. Summarize what's still relevant but bulky: collapse older turns into a short statement of decisions made, open questions, and files touched — not a verbatim transcript.
5. Offload what must survive long-term: write durable facts to external memory rather than either keeping them in-window forever or losing them in a summary that didn't preserve them.

✅ Worked example: Returning to the support agent from Section 1's callout — once it crosses its threshold, it clears the full text of resolved tickets' tool calls, keeps a one-line summary of each ("Ticket #4821: refund approved, customer notified"), and writes any policy exception it had to apply to memory so future tickets can reference it without re-deriving it.

💡 Key warning: Summarization is lossy by construction. If the summary step drops the one constraint the agent actually needed three steps later — the exact refund amount, the exact file path — compaction has traded a context-window problem for a correctness problem, which is often worse because it's silent.

🎯 Use this when: a single session is expected to run dozens of turns or tool calls — below that, the added complexity of a compaction pipeline usually isn't worth it yet.

5. Prompt Caching: The Companion Discipline

🧒 Kid analogy: if you write the same permission-slip header on every single form your parents have to sign this year, that's wasted effort — better to print it once and just swap out the date each time. Prompt caching is printing the header once.

Caching doesn't change what's in the window the way selection and compression do — it changes how expensively the model has to reprocess the parts that repeat unchanged from one request to the next. Anthropic's published guidance is to structure a prompt with static content first — system instructions, few-shot examples, tool definitions — and variable, query-specific content last, because the static prefix can then be cached and reused across calls instead of being reprocessed from scratch every time, with Anthropic reporting substantial reductions in both cost and latency from this pattern in production workloads. It pairs naturally with compaction: keep the stable prefix cache-friendly, and let the variable tail be where trimming, summarizing, and retrieval happen.

🎯 Use this when: the same system prompt, tool definitions, or reference document gets sent on many calls in a row — caching is close to free performance for content that isn't changing turn to turn.

6. Hands-On Lab: Feel Context Rot Yourself

This is a small, disposable exercise you can run in any chat interface with a model — no API key, no code, nothing you'll need to clean up afterward. The goal is to see the difference between full-context stuffing, selection, and compression on the same question, so the sections above stop being abstract.

1

Write or paste a long piece of throwaway text — 3,000–5,000 words is plenty. A public-domain novel excerpt, a long meeting transcript, or several unrelated Wikipedia articles pasted together all work. Bury one specific, made-up fact in the exact middle of it, like "The warehouse key is under the third flowerpot."
Expect to see: a document where that single sentence is surrounded on both sides by roughly equal amounts of unrelated text.

2

Paste the whole document into a fresh chat and ask only: "Where is the warehouse key?" This is the full-context-stuffing condition — nothing selected, nothing compressed.
Expect to see: the model usually gets this right on a short document like this one, but note how much of the reply is generic acknowledgment of the surrounding text it didn't need.
Troubleshooting: if the model refuses or gets it wrong even here, your document is too short to demonstrate the effect — lengthen it and try again.

3

Now simulate selection: copy out only the one paragraph containing the flowerpot sentence, paste just that paragraph into a new chat, and ask the same question.
Expect to see: a faster, shorter, more directly-grounded answer — this is what a retriever-plus-compressor pipeline is trying to hand the model automatically, instead of you doing it by hand.

4

Finally, simulate compression: ask the model to summarize the full document into three bullet points first, in a separate turn, then ask "Where is the warehouse key?" using only that summary as context.
Expect to see: this step usually fails unless you explicitly told the summarizer to preserve specific facts — which is exactly the lossy-summarization warning from Section 4's callout, now something you've watched happen firsthand rather than just read about.

Bridging to production: steps 3 and 4 here were you playing the role of the retriever, the compressor, and the summarizer by hand. In a real system, that hand-work is exactly what a vector search plus a document compressor (Section 3) and a compaction policy with explicit "always preserve" instructions (Section 4) automate — at a scale of thousands of documents and hundreds of turns, not one paragraph and one chat.

7. Rolling This Out at Enterprise Scale

Context strategy stops being a per-project decision once dozens of teams are shipping prompts, retrievers, and agents against shared models and shared budgets. A handful of governance questions come up repeatedly at that scale:

• Ownership and versioning: a prompt template, a retriever's compression threshold, and a compaction trigger are all configuration that changes behavior — they need an owner, a changelog, and a review step, the same as application code.
• CI-gated regression testing: before any change to retrieval thresholds, compaction triggers, or the prompt template itself ships, run it against a fixed regression suite of representative queries and known-good answers, and block the merge on regressions rather than discovering them from a user complaint.
• Access control and data governance: retrieved chunks and memory files frequently contain real customer or employee data. Whoever can edit a retriever's data source or a memory tool's storage backend can effectively change what real user context a model sees — that access needs the same controls as direct database access, not the lighter review a "prompt tweak" might otherwise get.
• Cost governance: token spend at scale is a function of context design as much as usage volume — a retriever that over-fetches, or a cache-unfriendly prompt structure that puts variable content first, multiplies cost across every call. Track cost per resolved task, not just raw token counts, so a "cheaper per call" change that requires more calls to succeed doesn't look like a win it isn't.
• Observability across model version upgrades: context-window sizes, caching behavior, and compaction mechanics are all model-version-specific — cached content tied to one model version generally does not carry over to a new one, and compaction or context-editing thresholds tuned for one model's behavior may need re-validation after an upgrade. Treat a model version bump like a dependency upgrade: re-run the regression suite before rolling it out broadly.
• Alerting for quality regressions: instrument retrieval hit-rate, average context size per request, and downstream answer-quality signals (thumbs-down rate, escalation rate) so a slow drift — a retriever's index going stale, a compaction summary quietly dropping a field — surfaces as an alert instead of a slow-burning support backlog.

🎯 Use this when: more than one team is building against the same model and the same data sources — at that point, undocumented context decisions in one team's prompt become an invisible dependency for everyone else's.

8. Common Mistakes

These show up again and again in production systems, and each one has a specific reason it happens:

• Treating "fits in the window" as "will be used well." A team confirms a document fits under the token limit and stops there, without accounting for context rot — the fact that recall degrades as content grows even within the limit. Fitting is necessary, not sufficient.
• Stuffing the window with irrelevant material instead of curating it. It's easier to paste everything than to build a retriever, so teams default to full-context stuffing "just to be safe" — which is precisely the condition that produces the worst cost-to-quality ratio, as the Quick Comparison table above shows.
• Hardcoding a prompt or threshold tuned for one model version with no regression suite. A compaction trigger or few-shot example set that was hand-tuned against one model's behavior can silently underperform after a model upgrade, because context-window size, caching behavior, and attention characteristics are all version-specific — without a regression suite, nobody notices until users do.
• Ignoring token cost and latency as first-class design constraints. Retrieval and compression decisions get made purely on "does the answer look right," with cost and response-time treated as an afterthought to optimize later — later rarely comes, and the bill or the latency becomes the real bottleneck.
• Testing a retriever or compaction policy only on happy-path inputs. A handful of clean, well-phrased test queries pass, so the system ships — but adversarial or oddly-phrased real queries are exactly where a retriever's similarity threshold or a summarizer's "preserve important facts" instruction gets exposed.
• Letting context strategy drift out of sync with the product. The retriever was built for last quarter's document set, or the compaction summary template was written before the agent gained a new tool — and nobody updated the context pipeline when the product changed underneath it.

❓ FAQ

Is a context window the same thing as a model's training data?

No. Training data is the much larger body of text the model learned general knowledge and language patterns from, long before your conversation started. The context window is the small, per-request slice of text — your prompt, the conversation, any retrieved documents — that the model can actually reference while generating this specific response.

What exactly is "context rot," and why does it matter if my request is well under the token limit?

Context rot is the documented tendency for a model's accuracy and recall to decline as the amount of content in its window grows, even when that content is nowhere near the hard token ceiling. It matters because "under the limit" and "well-used" are different properties — a shorter, curated prompt can outperform a longer one that technically still fits.

Are context selection (RAG) and context compression the same technique?

No, though they're often used together. Selection decides which material gets into the window in the first place, by searching a larger corpus. Compression shrinks or offloads material that's already in the window — trimming stale tool output, summarizing old turns, or moving facts to external memory — regardless of how it got there.

If my model has a huge context window, do I still need retrieval or compression?

Often yes. A large window raises the ceiling on what technically fits, but it doesn't remove context rot, and it doesn't lower the token cost of stuffing everything in on every call. For a corpus that's genuinely larger than any window, or a session that runs for hundreds of turns, selection and compression stay relevant regardless of window size.

How does prompt caching relate to context compression?

They solve different problems and combine well. Caching reduces the cost of reprocessing a static prefix that repeats unchanged across calls. Compression reduces how much variable, accumulating content is in the window at all. Structuring a prompt with static content first keeps caching effective even as the variable tail gets trimmed, summarized, or refreshed by selection and compaction.

🔗 References & Further Reading

Official / primary sources relied on for this post:

• Anthropic — Context windows (Claude Platform Docs)
• Anthropic — Effective context engineering for AI agents (Anthropic Engineering)
• Anthropic — Effective harnesses for long-running agents (Anthropic Engineering)
• Anthropic — Context editing (Claude Platform Docs)
• Anthropic — Memory tool (Claude Platform Docs)
• Anthropic — Managing context on the Claude Developer Platform (Claude Blog)
• Anthropic — Prompt engineering best practices (Claude Blog)
• LangChain — Improving Document Retrieval with Contextual Compression (LangChain Blog)

All product and company names (Anthropic, Claude, LangChain, and others referenced) are trademarks of their respective owners. 

📝 Summary

• A context window is a fixed, shared token budget for one request — not infinite storage, and separate from training data.
• Context rot means accuracy can degrade well before the window is technically full, so curation matters as much as capacity.
• Context selection (RAG-style retrieval plus compression) keeps only relevant material in the window, sourced from a much larger corpus.
• Context compression and compaction shrink or offload what's already accumulated in a long session — trimming, summarizing, and externalizing memory.
• Prompt caching is a companion discipline: structure static content first so unchanged prefixes are cheap to reuse.
• The hands-on lab lets you feel context rot and the value of curation directly, before automating it with real retrieval and compaction pipelines.
• At enterprise scale, context decisions need the same ownership, testing, and observability discipline as application code.
• The most common mistakes all trace back to one habit: treating "it fits" as good enough, instead of actively designing what belongs in the window.

That's context windows, selection, and compression end to end — from the backpack analogy all the way to CI-gated regression suites. Go build something that only packs what it actually needs. 🎒