Question-Aware Context Compression: 2026 RAG & Agents Guide

TL;DR

Question-aware context compression is a technique that uses the user’s specific question to decide which parts of a long context to keep and which to discard before sending it to an LLM. Unlike generic compression that treats all text equally, it adapts its output to the query, keeping only the spans that matter for answering that particular question. This makes it possible to compress aggressively (often 4x to 32x) while maintaining or even improving answer accuracy, cutting both cost and latency in RAG, agent, and long-document workflows.

What is Question-Aware Context Compression?

Question-aware context compression is an online prompt optimization technique in Retrieval-Augmented Generation (RAG) and LLM agent pipelines that evaluates document context against a user’s specific query to prune non-relevant tokens, sentences, or chunks before generation. By executing after retrieval and before LLM inference, it reduces context bloat by 4x to 32x, lowers input token costs, reduces time-to-first-token (TTFT) latency, and eliminates "context rot" and "lost in the middle" accuracy drops.

How It Works

Question-aware context compression sits at a specific point in the LLM pipeline: after retrieval (or context assembly) and before the generation call. The compression model receives two inputs, the long context and the user’s question, and uses the question to score each span, sentence, or token for relevance. Only the spans that carry evidence for answering the question survive. Everything else gets dropped.

The output is natural-language text, readable and auditable. This is sometimes called “hard prompt” compression because the compressed result is a plain text prompt, not a latent vector or cached KV state.

Consider a 200-page SEC 10-K filing. If the user asks “What was Boeing’s revenue last year?”, the compressor keeps revenue figures, the income statement, and related footnotes. It discards risk factors, board composition, legal proceedings, and boilerplate. If a different user asks “What are the pending lawsuits?”, the same filing compresses to a completely different output, preserving the legal sections instead.

This per-query adaptiveness is the core idea. As the Lacuna research overview puts it: “Instead of asking ‘What is important in this document?’, it asks ‘What is important in this document given this specific question?’”

A NeurIPS 2024 paper formalized this intuition using information theory, demonstrating that query-aware compression is the only paradigm where a method can outperform the theoretical limit of the best query-agnostic strategy. Their Adaptive QuerySelect approach achieved this, confirming what practitioners had observed empirically.

Try question-aware compression in action →

Question-Aware vs. Question-Agnostic Compression

The distinction between these two paradigms is fundamental, and the terminology can be confusing. “Question-aware,” “query-aware,” “query-guided,” “query-conditioned,” and “task-aware” all refer to the same idea. The literature uses these interchangeably. Query-specific compression is another common synonym.

Question-agnostic compression removes redundancy based on statistical signals like entropy or perplexity. It doesn’t know what the user will ask, so it tries to preserve everything that “seems important” based on general language patterns. The advantage: it can be done offline, once, and reused. The disadvantage: it can’t compress very aggressively without risking the removal of critical information.

Question-aware compression adapts the compressed output to the specific query. It must run online, per question. But because it knows exactly what matters, it can compress far more aggressively. A detail that looks statistically unimportant to a generic compressor might be the exact answer to the user’s question.

According to the prompt compression characterization survey, methods break down clearly along this line. Among token pruning approaches, LongLLMLingua and attention-based pruning are query-aware, while Selective-Context and LLMLingua-2 are query-agnostic. Among abstractive methods, RECOMP is query-aware. Among extractive methods, both RECOMP and rerankers are query-aware.

The practical upshot: generic compression suffers from a fundamental flaw. Importance is not an intrinsic property of text. A boilerplate paragraph about forward-looking statements is noise for a revenue question but critical context for a legal-risk question. Without the question, a compressor either preserves everything (weak compression) or gambles on what to cut.

Why It Matters

Accuracy

This may be counterintuitive, but compressing context can actually improve LLM accuracy. The reason is context rot, the degradation in output quality that happens as input context grows longer. Chroma’s research tested 18 frontier models and found that every single one gets worse as input length increases, even when the context window isn’t close to full.

There’s also the “lost in the middle” problem. Research from Stanford showed that performance can degrade by more than 30% when relevant information sits in the middle of the context window rather than at the beginning or end. This is a transformer architecture bias that compression directly addresses by stripping away the irrelevant material that pushes answer-bearing text into those dead zones.

Perplexity deployed query-aware context compression in production in May 2026 and published their results. They treat snippet generation as a compression problem: “Models looking for specific information do not benefit from generalized summaries or excerpts of a document. Rather, they need the smallest, most surgically extracted piece of source information to respond to the user’s request.” Their single-step setting achieved 95% accuracy with an average of only 200 tokens per document.

Cost

Most RAG applications retrieve 3 to 5 times more context than the model meaningfully uses. Embedding similarity pulls in chunks that are topically related but full of headers, boilerplate, redundant definitions, and filler. You’re paying for tokens the model ignores.

Question-aware compression cuts tokens at the exact point where money is spent: input tokens, which make up the bulk of LLM API costs in retrieval-heavy applications. For a deeper breakdown of where those costs accumulate, the guide on RAG over-retrieval and token costs covers this in detail.

Latency

LLM prefill time scales with input length. Fewer input tokens means faster time-to-first-token. For user-facing applications where responsiveness matters, cutting a 10,000-token context down to 2,000 tokens doesn’t just save money. It makes the application feel noticeably faster.

Common Confusion Points

This is not summarization. Extractive question-aware compression keeps original text spans verbatim. It doesn’t generate new text or paraphrase. This matters for source fidelity and citation accuracy. Abstractive variants exist, but a 2024 study comparing methods on multi-document QA found that extractive reranker-based compression achieved +7.89 F1 points on 2WikiMultihopQA at 4.5x compression, while abstractive compression at similar ratios decreased performance by 4.69 F1 points.

This is not a replacement for retrieval. Question-aware compression happens after retrieval. You still need a retriever to find relevant documents. Compression then trims those documents down to the answer-bearing parts. Think of retrieval as finding the right haystacks, and compression as finding the needles within them.

This is different from prompt caching. Prompt caching (offered by providers like Anthropic and OpenAI) reuses cached KV states for repeated prompt prefixes. It reduces cost for repeated context but doesn’t reduce what’s in the prompt. Compression actually removes tokens. The two techniques are complementary, not competing. For a fuller comparison, see the prompt caching vs. compression breakdown.

This is not always the right choice. For very short contexts under roughly 500 tokens, the API overhead of running a compressor may exceed the savings. And for contexts that will be reused across many different questions without modification, query-agnostic compression may be more cost-effective since you only compress once.

Research Lineage

The field has moved quickly since 2023. Here are the key papers and their contributions:

How to Apply Question-Aware Compression

The technique fits naturally into several common LLM architectures.

In RAG pipelines: Compress retrieved chunks before passing them to the answer model. This is the highest-impact use case because RAG pipelines routinely pass thousands of tokens of retrieved context when only a fraction contains the answer. A practitioner on Reddit described piggybacking context compression on every tool call in agentic workflows and getting 3x longer task completion on small models with zero extra LLM calls.

In agent workflows: Compress tool outputs and accumulated chat history each turn. As agents run longer, chat history balloons with prior tool calls, observations, and reasoning traces. Compressing per-turn keeps the agent within its context window without losing track of earlier findings. The guide on reducing tokens in agent tool calls walks through this pattern.

In long-document QA: Compress the full document against the user’s question. This is where the most dramatic compression ratios appear, since a single question typically only needs a small fraction of a long document.

Integration points: Major frameworks support compression as middleware. LangChain, LlamaIndex, LangGraph, and LiteLLM all have integration patterns for inserting compression between retrieval and generation.

Get started with the quick-start guide →

As the SitePoint developer resource notes: “Token compression remains the fastest lever to reduce LLM operating costs in retrieval-augmented and agentic systems.” A buildmvpfast.com analysis makes the complementary point that “retrieval doesn’t solve context bloat. RAG retrieves relevant chunks, sure. But those chunks still contain headers, boilerplate, redundant definitions, and filler. Compression handles what RAG leaves on the table.”

Compresr provides a query-aware compression API with Python and TypeScript SDKs, offering $10 in free credits on signup with no credit card required.

See pricing and free credits →

How to Implement Question-Aware Compression

To integrate question-aware compression into a production RAG pipeline, place the compressor module directly between vector retrieval and final prompt assembly:

1. Retrieve initial context: Query your vector store or retriever to gather the top candidate chunks (for example, top 10 to 20 documents).

2. Pass context and query to compressor: Supply both the raw retrieved text chunks and the user's specific prompt to a compression engine (such as LongLLMLingua, QGC, or a dedicated compression API).

3. Set target parameters: Define your target token count or compression ratio (for example, compressing 10,000 input tokens down to a 2,000-token budget).

4. Extract preserved spans: The compressor ranks spans based on query relevance, discards non-essential tokens, and outputs a condensed plain-text prompt.

5. Send to generation model: Pass the compressed text payload into your main LLM (e.g., OpenAI, Anthropic, or open-source models) for final output generation.

This post-processing step ensures that your main LLM receives only answer-bearing evidence while discarding irrelevant headers, boilerplate, and filler.

Frequently Asked Questions

What is the difference between “question-aware” and “query-aware” compression?

Nothing. These terms are synonyms. The academic literature uses “query-aware,” “question-aware,” “query-guided,” “query-conditioned,” and “task-aware” interchangeably. They all describe the same idea: the compressor has access to the user’s question and uses it to decide what to keep.

Does question-aware context compression work with all LLMs?

Yes. Because it produces plain text output (a shorter prompt), it works with any LLM regardless of provider or architecture. The compressed text is sent as a normal prompt to OpenAI, Anthropic, Google, open-source models, or any other provider.

How much can question-aware compression reduce token counts?

It depends on the content and question. For focused factual questions against long documents, compression ratios of 10x to 30x are common. For questions that require reasoning across many sections, 2x to 5x is more typical. Perplexity’s production system achieves 95% accuracy with roughly 200 tokens per document in their single-step setting.

Can compression hurt accuracy?

At aggressive compression ratios, yes. There is always a tradeoff between how much you compress and how much information you preserve. However, at light to moderate compression (2x to 5x), question-aware compression often improves accuracy by removing distracting and irrelevant material that would otherwise trigger context rot or the lost-in-the-middle effect.

When should I use question-agnostic compression instead?

When the same compressed context will be reused across many different questions. For example, if you pre-process a knowledge base that many users will query with different questions, compressing once (query-agnostic) and caching the result may be more cost-effective than running query-aware compression per request. Also, for very short contexts under about 500 tokens, the overhead of any compression step likely exceeds the savings.

How does question-aware compression fit with prompt caching?

They’re complementary. Prompt caching reuses previously computed KV states for identical prompt prefixes, saving compute on repeated prompts. Question-aware compression reduces the number of tokens in the prompt itself. You can compress first, then benefit from prompt caching on the compressed output.

Is question-aware compression the same as extractive summarization?

No. Extractive summarization selects sentences that capture the “gist” of a document for a general reader. Question-aware compression selects spans that are relevant to a specific question. The same document produces different outputs for different questions, which is not how summarization works. And unlike abstractive summarization, extractive question-aware compression preserves the original text verbatim, maintaining source fidelity.

What types of content benefit most from question-aware compression?

Long, information-dense documents where only a small portion is relevant to any given question. Financial filings, legal contracts, medical records, technical documentation, and multi-page research papers are prime candidates. The longer the document and the more specific the question, the higher the compression ratio without accuracy loss.

What is the latency overhead of running a question-aware compressor?

The latency overhead depends on the compressor model size. Compact compressors (such as 0.5B to 1.5B parameter models or specialized cross-encoders) typically process context in 20 ms to 100 ms. Because reducing prompt length from 10,000 tokens to 2,000 tokens saves several hundred milliseconds in LLM prefill time, the net time-to-first-token (TTFT) is usually significantly faster overall.

How does variable-rate compression differ from fixed-ratio compression?

Fixed-ratio compression forces every document or chunk to shrink by a set percentage (such as keeping strictly 25% of tokens across all inputs). Variable-rate compression evaluates the density of relevant information dynamically: passages dense with answer-bearing evidence keep more tokens, while sparse or noisy passages are compressed aggressively or dropped entirely.