Query-Aware vs Query-Agnostic Compression: 2026 Guide

TL;DR

Query-aware compression takes both the document and the user’s question as inputs, keeping only the parts relevant to that specific question. Query-agnostic compression compresses the document once without seeing any question, relying on general redundancy removal. Query-aware methods typically score 3 to 6 accuracy points higher on QA benchmarks, but query-agnostic methods are faster and can be cached for repeated use. The right choice depends on whether you know the query at compression time and how many times each document will be queried.

Direct Answer: The core difference between query-aware and query-agnostic compression comes down to when the user’s question is known. Query-aware compression processes both the document and the query together at runtime to remove irrelevant text, achieving 3 to 6 points higher accuracy on QA benchmarks. Query-agnostic compression removes redundant text from the document alone, allowing the output to be cached and reused across thousands of different queries.

---

What Is Query-Aware Compression?

Query-aware compression is a prompt compression strategy where the compressor receives both the source document and the user’s query, then selectively retains the tokens, sentences, or passages most relevant to answering that specific question. Everything else gets removed.

The key insight is that different questions about the same document need different pieces of it. A financial analyst asking “What was Boeing’s revenue in Q3?” and another asking “Describe Boeing’s supply chain risks” would produce two completely different compressed versions of the same 10-K filing.

Perplexity’s engineering team describes this well: models looking for specific information do not benefit from generalized summaries. They need “the smallest, most surgically extracted piece of source information,” and “everything else can and must be removed.”

Canonical examples of query-aware methods include LongLLMLingua, attention-based pruning, SnapKV, and the extractive and abstractive variants of RECOMP.

What Is Query-Agnostic Compression?

Query-agnostic compression operates on the document alone, without access to any downstream question or task. Because it cannot tailor its output to a specific query, it works by exploiting redundancy in natural language: removing filler words, deduplicating repeated information, and distilling the text to its densest form.

The defining advantage is that compression happens once and can be stored. A contract compressed query-agnostically can serve dozens of different questions without reprocessing. This makes it the natural fit for indexing pipelines, offline batch processing, and KV cache reuse scenarios.

LLMLingua-2 is a prominent example. It trains a classifier to identify and remove redundant tokens without ever seeing a query. PromptSAW takes a knowledge-graph approach, deduplicating similar information elements regardless of the downstream task. KVzip compresses KV caches by 3 to 4x while maintaining near-perfect task accuracy at just 30% cache retention.

Side-by-Side Matrix: Query-Aware vs. Query-Agnostic

This comparison highlights the core trade-off in prompt optimization: query-aware compression maximizes accuracy per request, while query-agnostic compression amortizes compute costs across high-volume workloads.

How Each Approach Works

Query-Aware Flow

1. The user submits a query alongside a long context (retrieved documents, chat history, tool output).

2. The compressor scores each segment of the context against the query.

3. Low-relevance segments are pruned or summarized. High-relevance segments are preserved.

4. The compressed context, now tailored to that specific question, is sent to the LLM.

Because the compressed output is query-specific, it cannot be reused if a different question arrives. This is the fundamental cost: every new query triggers a new compression pass.

Query-Agnostic Flow

1. A document enters the compression pipeline with no associated query.

2. The compressor identifies and removes redundant tokens, repetitive phrases, or low-information passages based on language statistics alone.

3. The compressed output is stored (in a cache, index, or database).

4. When queries arrive later, they operate against this pre-compressed version.

The compressed representation is the same regardless of what question is eventually asked. This makes it compatible with prefix caching, where the LLM provider stores the processed prefix and reuses it across requests.

What the Research Shows About Performance

Accuracy: Query-Aware Has a Clear Edge

The canonical taxonomy paper by Jha et al. (2024) found that query-aware abstractive compression outperforms query-agnostic abstractive compression by 3 to 6 points on NarrativeQA, MultiFieldQA, and HotpotQA. That gap is consistent across datasets and compression methods.

The evidence gets sharper with ablation studies. When query guidance is removed from a query-aware system, accuracy drops by 13.85 to 18.83 points, a dramatic degradation that underscores how much work the query signal is doing.

Adaptive QuerySelect, a query-aware adaptation of LLMLingua-2, is the only method tested that outperforms the optimal query-agnostic strategy, according to Nagle et al. at NeurIPS 2024. The authors note that this “highlights the importance of conditioning on the query.”

Meanwhile, NEC Labs’ LeanContext uses reinforcement learning to dynamically select how many key sentences to extract per query. Despite cost reductions of 37% to 68%, ROUGE-1 scores dropped only 1.4% to 2.7%, a remarkably small penalty for significant savings.

Speed and Reuse: Query-Agnostic Wins

Query-aware compression boosts accuracy at the cost of increased processing time. Query-agnostic methods are simply faster because they run offline and once per document.

The economic argument is compelling for multi-query scenarios. As the KVzip authors explain, the business case for a compressed KV cache is reuse: compress a document once, answer many future questions against it. Think of a contract interrogated by dozens of questions, or a codebase queried throughout the workday. In that deployment pattern, compression must happen before any question is seen.

Query-aware schemes are also incompatible with multi-query prefix caching. Each new question would evict different prefix tokens, so the compressed prefix cannot be reused across prompts.

The Evaluation Protocol Illusion

Here is a sharp, underreported finding. A July 2026 KV-cache audit revealed that the evaluation protocol itself, whether query-aware or query-agnostic, fundamentally changes which methods appear to work. Under query-aware evaluation, four of five tested methods beat trivial baselines. Under agnostic evaluation, the picture inverted: only KeyDiff survived, and SnapKV averaged below a simple “keep the start and the recent window” baseline.

This means many published benchmarks, which use query-aware evaluation by default, overstate how well methods perform in real-world query-agnostic deployment. If you are choosing a compression method for production, pay attention to which protocol generated the numbers you are reading.

Taxonomy: Which Methods Are Query-Aware?

Based on the classification in Jha et al., here is how major prompt compression methods map:

The pattern is clear: token pruning methods split roughly evenly, while extractive and abstractive methods lean query-aware. KV cache methods traditionally tend toward query-agnostic because their whole value proposition rests on reuse.

The Theoretical Foundation: Rate-Distortion Theory

The difference between query-aware and query-agnostic compression has a formal basis in rate-distortion theory. A query-agnostic compressor must choose one summary that protects against all possible queries simultaneously. A query-conditioned compressor can adapt its representation to the actual query before transmitting anything, averaging per-query costs rather than guarding against worst cases.

This is why empirical analysis consistently shows a large gap between current query-agnostic methods and the theoretical optimum. The query-aware approach gets to cheat, in a good way, by knowing what matters before it compresses.

When to Use Which: A Decision Guide

Choose Query-Aware Compression When:

• The query is known at compression time. This is the standard case for RAG pipelines, chat applications, and agent tool outputs.

• Accuracy is the priority. The 3 to 6 point advantage matters in high-stakes domains like finance, healthcare, and legal.

• Each context is queried once. Single-turn QA, one-shot document questions, or ephemeral chat contexts where reuse is unlikely.

• Documents are long and heterogeneous. When different queries need completely different passages, query-agnostic approaches “fail to reliably preserve evidence for localized, complex, or retrieval-oriented queries.”

Choose Query-Agnostic Compression When:

• The query is unknown at compression time. Pre-processing, indexing, or building a searchable document store.

• One document serves many queries. The “compress once, query many” pattern where amortized cost wins.

• Throughput and latency are critical. Batch offline compression for high-volume pipelines.

• Prefix caching is required. Query-aware eviction breaks prefix reuse, making it incompatible with provider-level caching optimizations.

Architectural Decision Flow: Selecting Your Compression Strategy

When deciding between query-aware and query-agnostic compression for production systems, evaluate your workload requirements against this three-step workflow:

1. Single-Turn Agentic Workflows -> Use Query-Aware

   When context is generated dynamically per user interaction (such as real-time web scraping or tool outputs), use query-aware tools like LongLLMLingua or RECOMP. The query is already known, and maximum accuracy is required.

2. Enterprise Knowledge Bases & Static Search -> Use Query-Agnostic

   When processing standardized policy documents, legal contracts, or code repositories served to multiple users, use query-agnostic tools like LLMLingua-2 or KVzip. Compress the documents once during database ingestion to allow prefix caching across all incoming queries.

3. High-Throughput RAG Pipelines -> Use a Two-Stage Hybrid Flow

   For large-scale RAG systems, combine both approaches:

   • Stage 1: Pre-compress raw source text by 2x using query-agnostic token pruning during index creation.

   • Stage 2: Apply a light 2x query-aware compression pass on the retrieved context at query time.

     This hybrid approach yields up to 4x total context reduction while retaining prompt-level accuracy.

Hybrid and Emerging Approaches

The query-aware vs query-agnostic compression divide is not always binary. A growing category of “task-aware but query-agnostic” methods is emerging. These approaches precompute a compressed cache tailored to a broader task context without per-query recomputation. Instead of recompressing the input for every query, they capture the general intent of a task class (like “answer financial questions about this filing”) and compress accordingly.

Some methods can flexibly switch between task-aware and task-agnostic modes by setting prefix prompts and dynamically selecting compression ratios. This represents the frontier of compression research: getting closer to query-aware accuracy while preserving query-agnostic reusability.

Dynamic ratio selection is another practical middle ground. Rather than applying a fixed compression rate to all inputs, the compressor auto-selects how aggressively to compress each chunk. Dense, information-rich chunks keep more context while sparse chunks compress more aggressively. This reduces manual tuning and handles the natural variation in document density.

Practical Considerations From the Field

Token-Pruned Fragments and Modern LLMs

Practitioners report that the token-probability approach used by some query-agnostic pruning methods produces fragments that confuse newer models like Claude Sonnet 4.6 and GPT-4o. These models were trained on coherent text, and compressed fragments can trigger unexpected behaviors, especially in instruction-following scenarios. This is a real caution for teams using purely statistical token-pruning methods. Query-aware compression partially mitigates this by preserving coherent, relevant passages rather than scattered high-probability tokens.

The Cost-Reduction Sweet Spot

Light prompt compression of 2 to 3x delivers roughly 80% cost reduction with less than 5% accuracy impact. This is the safest starting point for most teams. Moderate compression of 5 to 7x can achieve 85 to 90% cost reduction, though with trade-offs that need evaluation for each use case.

The Context Rot Connection

When query-agnostic compression leaves large amounts of irrelevant information in the prompt, it can cause context rot: long streams of irrelevant tokens waste model capacity and impair the LLM’s ability to address the actual user request. Query-aware compression directly targets this problem by removing irrelevant content based on what the user actually needs.

Key Statistics at a Glance

FAQ

Does query-aware compression always beat query-agnostic?

Not always. Query-aware methods win on per-query accuracy, typically by 3 to 6 points on QA benchmarks. But when a single document must answer many different questions, the cost of recompressing per query can outweigh the accuracy gain. In multi-query deployment with KV cache reuse, query-agnostic approaches like KVzip actually maintain higher accuracy because query-aware eviction breaks cache consistency.

Can query-agnostic compressed output be cached?

Yes, and this is its primary advantage. Because the compressed output does not depend on any specific question, it can be stored and reused across unlimited future queries. This makes query-agnostic compression ideal for indexing, prefix caching, and “compress once, query many” workflows where amortized cost matters most.

What if I don’t know the query at compression time?

Then query-agnostic compression is your only option. Query-aware methods require the query as an input, and their performance degrades significantly when the query is unavailable or misspecified. For pre-processing and indexing pipelines, query-agnostic methods or the newer hybrid “task-aware” approaches are the right choice.

How do I choose between the two for a RAG pipeline?

In a typical RAG pipeline, the query is known before retrieval and compression happen, making query-aware compression the natural fit. The retrieved documents vary per query anyway, so there is little reuse to preserve. Query-aware compression in RAG has emerged as a central paradigm for efficient, scalable inference.

Can I combine both approaches?

Yes. A practical pattern is to apply query-agnostic compression during indexing (removing obvious redundancy and noise) and then apply lighter query-aware compression at query time on the already-reduced context. This two-stage approach captures some of the caching benefits of query-agnostic methods while still tailoring the final context to the specific question.

Do these concepts apply to KV cache compression too?

Absolutely. SnapKV is a query-aware KV cache method, while KVzip is query-agnostic. The July 2026 KV-cache audit showed that the choice of evaluation protocol (query-aware vs agnostic) dramatically changes which methods appear effective, so understanding this distinction is critical when evaluating KV cache solutions.

Ready to see how query-aware compression performs on your own contexts? Explore pricing and get $10 in free credits.