Token Waste in AI Pipelines: 17 Fixes That Work (2026)

TL;DR

Token waste in AI pipelines is the consumption of LLM input or output tokens that contribute nothing to the accuracy, quality, or usefulness of the final result. It is the structural cost problem of production AI. Agentic AI workflows consume 5 to 30 times more tokens per task than standard chatbot exchanges, and research suggests 50% to 99% of that usage is redundant. This waste hides inside RAG retrieval bloat, tool call outputs, conversation history, inflated system prompts, runaway thinking budgets, and over a dozen other structural problems. This article maps the 17 biggest sources, quantifies each one, and prescribes specific fixes, from semantic compression and adaptive context selection to model routing, workflow decomposition, and observability instrumentation.

What Is Token Waste in AI Pipelines?

Token waste refers to the consumption of Large Language Model (LLM) input or output tokens that do not contribute to the accuracy, quality, or context of the final output. In production AI systems, token waste typically manifests as raw, unfiltered database strings, overly dense documentation schemas, conversational repetitions, uncapped reasoning chains, redundant session state, or natural language preambles.

Quick Takeaways:

• The Agentic Multiplier: Agentic workflows consume 5 to 30 times more tokens than isolated chatbot exchanges due to iterative loops and persistent state tracking.
• Context Overhead vs. Value: In agentic development pipelines, 85% to 95% of total spend is eaten by systemic context overhead (system prompts, tool definitions, dynamic context), leaving only 5% to 15% for actual generation. Understanding token usage by prompt component makes this breakdown concrete.
• The Remediation Layer: Implementing a precision layer, such as query aware semantic compression, removes the noise before it hits the model, yielding significant cost reductions while protecting execution accuracy.

The Scale of the Problem

Goldman Sachs Research forecasts that token consumption will multiply 24 times between 2026 and 2030, reaching 120 quadrillion tokens per month. That growth is not just more users asking more questions. It is agentic workflows burning through context windows at rates nobody budgeted for.

Gartner’s 2026 analysis found that a single agentic AI task consumes 5 to 30 times more tokens than a standard chatbot exchange. One agent answering a support ticket can burn the same tokens as 30 chat conversations. A Stanford Digital Economy Lab paper pushed that number further: agentic tasks are “uniquely expensive, consuming 1000x more tokens than code reasoning and code chat,” with input tokens driving the bulk of the cost.

Here is the number that should change how you think about this: in agentic coding workflows, actual code generation accounts for only 5 to 15% of total tokens consumed. Everything else is context overhead. System prompts, tool schemas, retrieved documents, conversation history, formatting noise. Most of it contributes nothing to output quality.

The financial stakes are real. A 100 developer organization mixing inline and agentic AI coding tools now faces $400,000 to $600,000 in annual token costs before governance infrastructure. And 95% of enterprise AI pilots deliver zero measurable ROI, with cost sustainability cited as a primary driver of abandonment.

Token waste in AI pipelines is not a minor inefficiency. It is the structural cost problem of production AI.

Try Compresr’s free demo to see how query aware compression reduces token bloat in real time.

Measuring What Matters: Key Metrics Before You Optimize

Before diving into specific waste sources, teams need the right metrics in place. Optimizing without measurement is guessing.

Baseline Token Logging

The first step is knowing what you spend and where. Baseline token logging means recording input tokens, output tokens, and (if applicable) thinking tokens at the level of each individual LLM call, not just aggregate monthly totals. Without this granularity, every optimization is a shot in the dark.

Practitioners on Reddit’s r/LangChain report that simply instrumenting per step token counts often reveals that one or two steps in a workflow consume 60% to 80% of total tokens. Those are your targets.

Cost Per Useful Output

Raw token counts tell you volume. Cost per useful output tells you efficiency. This metric divides total token cost (including retries, failed attempts, and overhead) by the number of successful, quality outputs produced. A pipeline that spends 100,000 tokens to produce 10 good answers has very different economics than one that spends 100,000 tokens to produce 2.

For a formula and worked examples, the AI cost per task guide breaks this down by workflow type.

Retrieval Precision Metric

For RAG pipelines specifically, retrieval precision measures what fraction of retrieved chunks actually contributed to the final answer. If a pipeline retrieves 10 chunks but only 2 contain relevant information, retrieval precision is 20%. The other 8 chunks are pure token waste. Tracking this metric over time reveals whether retrieval quality is improving or degrading, and whether compression is preserving the right content.

A/B Testing Compression Ratios

Compression is not one size fits all. The right ratio depends on the task, the data, and the model. A/B testing different compression levels against your evaluation set (measuring accuracy, completeness, and cost) is the only reliable way to find the sweet spot. On the FinanceBench benchmark, Compresr’s query aware compression at roughly 2x improved accuracy from 73% to 77% compared to full context. But at 10x compression, the tradeoff shifts. Testing reveals where that inflection point sits for your specific use case. The compression ratio vs accuracy guide covers this tradeoff in detail.

At a Glance: 17 Waste Sources and Their Fixes

Now let’s break each one down.

1. RAG Context Bloat

Best for understanding: Why retrieval augmented generation is often retrieval augmented waste.

RAG pipelines retrieve document chunks based on vector similarity, then stuff them wholesale into the prompt. The problem is that retrieval is not relevance filtering. A chunk matched because it contains a relevant paragraph will also carry headers, boilerplate, definitions, and tangential content that have nothing to do with the query.

Retrieved documents routinely add 2,000 to 12,000 tokens per query. Most of that content is irrelevant to the specific question being asked. You are paying for the LLM to read (and be distracted by) content that does not help it answer.

Retrieval chunking is the first line of defense. How you split documents before indexing them determines the ceiling on retrieval precision. Chunks that are too large (entire pages or sections) guarantee irrelevant content in every retrieval hit. Chunks that are too small lose necessary context. Semantic chunking, which splits on topic boundaries rather than fixed character counts, consistently outperforms naive splitting. But even with perfect chunking, similarity search will return chunks that partially match the query.

Hierarchical retrieval adds another layer. Instead of searching a flat index of chunks, organize documents into a tree: document level summaries at the top, section summaries in the middle, granular chunks at the leaves. The retrieval system first identifies relevant documents, then relevant sections, then relevant chunks. This narrows the search space dramatically and avoids pulling chunks from completely unrelated sections that happen to share vocabulary with the query.

The fix: Query aware compression that retains only the spans relevant to the user’s query before the context reaches the LLM. Unlike naive truncation or generic summarization, query aware compression evaluates each span against the actual question and preserves what matters. This is a form of semantic compression: reducing content based on meaning and relevance rather than character count or position.

Practitioners on Reddit’s r/LangChain report dramatic results. One developer cut query costs by 60% just by adding a compression step before retrieval results hit the model. The waste was entirely in how much raw context got packed into each prompt.

There is a counterintuitive finding worth noting: on the FinanceBench benchmark, Compresr’s query aware compression at roughly 2x improved accuracy from 73% to 77% compared to full context. Removing distracting content did not just save money. It made the answers better. This is a direct demonstration of lost in the middle mitigation: LLMs struggle to attend equally to all parts of a long context, and removing irrelevant material lets the model focus on what actually matters. For teams struggling with over retrieval costs, this is often the single highest ROI fix.

For a deeper walkthrough, see the RAG compression guide.

2. Tool Call Output Bloat

Best for understanding: Why your agent’s API calls are bleeding tokens.

When an agent queries an API, reads a file, or runs a database query, the raw output gets appended to context. A single JSON blob returned with 50 fields, when the agent only needs 3, consumes thousands of tokens for zero additional value.

This is one of the most common sources of token waste in AI pipelines, and it scales linearly with agent complexity. More tools means more raw outputs means more wasted context.

The fix: Two layers work together here. First, field filtering at the tool level: configure your API wrappers to return only the fields the agent actually needs. This alone can reduce payload tokens by 80 to 90%. Second, for outputs that cannot be pre filtered (file contents, search results, dynamic responses), compress them before they enter the prompt. Tool spec gating takes this further: define per tool output specifications that declare which fields are required, optional, or irrelevant for each downstream task type. The gating layer strips fields before the output ever reaches the context window. For detailed tactics on trimming tool call tokens, there is a dedicated guide worth reading.

Compresr provides LangChain middleware specifically for tool output compression, so the compression fires automatically in the right spot without manual plumbing on every tool call.

3. Unbounded Conversation History

Best for understanding: Why multi turn agents get exponentially more expensive per turn.

Multi turn agents replay the full message history on every request. After 10 turns, conversation history alone can exceed 15,000 tokens. After 20 turns, you might be spending more on history replay than on the actual task.

This is straightforward compounding. Each turn adds its own content plus re sends everything that came before. The early turns, which established context the model has long since incorporated into its working state, get re processed at full cost every single time.

The fix: Three approaches, in order of sophistication: sliding window (keep only the last N turns), manual summarization (condense older turns into a human reviewed or LLM generated summary), or query aware history compression (keep the spans from history that are relevant to the current turn). Manual summarization works well for workflows where a human operator periodically reviews and condenses the conversation state, but it does not scale for fully automated pipelines. The third approach, adaptive context selection, is the most token efficient because it dynamically evaluates which historical spans are relevant to the current turn rather than applying a fixed rule.

Practitioners report 60 to 72% reductions in history related token usage with compression based approaches, without degrading the model’s ability to maintain conversational coherence. For teams already managing conversation history costs, conversation history is often the single biggest line item to attack.

4. System Prompt Creep

Best for understanding: Why your system prompt quietly became a cost center.

System prompts accumulate instructions the way codebases accumulate dead code. A developer adds a rule for edge case handling. Another adds formatting guidance. A third adds persona instructions. Nobody removes anything. A system prompt that started at 500 tokens balloons to 5,000.

The cost is not just the 4,500 extra tokens. It is that those tokens get sent on every single request. At 1,000 requests per day, that is 4.5 million wasted tokens daily, just from instructions that might be contradictory or obsolete.

The fix: Regular audits. Schedule quarterly system prompt reviews the same way you schedule dependency updates. Test each instruction block independently: does removing it change output quality on your eval set? Anthropic’s own benchmarks have shown that concise prompts often improve task completion, not just reduce cost. Structured distillation can help here too: take a verbose system prompt and distill it into a compact, structured format (numbered rules, JSON constraints) that conveys the same behavioral requirements in fewer tokens.

There is no tool that fixes this automatically. It is a discipline problem. But it is also one of the cheapest wins in the list because it requires zero infrastructure changes. Teams tracking system prompt costs often discover that half their instructions are redundant.

5. MCP Tool Schema Inflation

Best for understanding: Why the Model Context Protocol creates a hidden per request tax.

If your AI agent has 30 registered MCP tools, that is 3,000+ tokens of JSON schema injected into every single request before a single line of user input appears. The agent needs to “see” all available tools to decide which one to use, and each tool definition includes its name, description, and full parameter schema.

Recent analysis shows MCP servers use 35x more tokens than CLI tools performing the same task, with reliability actually dropping from 100% to 72% as tool complexity grows. The token overhead is not just costly, it actively degrades performance by diluting the context with schema definitions the agent does not need for the current task.

The fix: Progressive disclosure, sometimes called tool spec gating. Instead of loading all tool schemas upfront, implement a two stage approach: give the agent a lightweight tool catalog (names and one line descriptions), then load full schemas only for tools the agent selects. This mirrors how developers work: you browse a directory before reading full documentation.

Tool pruning helps too. If an agent uses 8 of its 30 tools in 95% of requests, consider splitting into specialized agents with smaller toolsets.

6. Uncompressed Web Search Payloads

Best for understanding: Why search augmented agents waste the most tokens per operation.

Most AI agents that search the web fetch entire pages, dump thousands of tokens of raw HTML stripped text into the context window, and hope the model figures out what matters. A single web search result can add 2,000 to 8,000 tokens, and agents often pull multiple results per query.

Running content through a readability parser before passing it to the model cuts token count by 60 to 80% for article pages. But readability parsing only removes structural noise. It does not remove content that is irrelevant to the query.

The fix: Two stage processing. First, readability parsing to strip navigation, ads, and boilerplate HTML. Second, query aware compression to keep only the spans that answer the actual question. The combination routinely achieves 80%+ reduction while preserving the information the agent needs. Chunk routing can further optimize this: after parsing, split the page into chunks and route only those chunks whose embedding similarity exceeds a threshold to the compression step, discarding obviously irrelevant sections before compression even runs.

This matters especially for research agents and RAG pipelines that pull from live sources. Without compression, a single research task that triggers 5 web searches can consume 20,000 to 40,000 tokens of search content alone.

7. Context Re ingestion at Every Agent Step

Best for understanding: Why multi step workflows create exponential cost curves.

In multi step agentic workflows, the full accumulated context gets re read at every step. Step 1 reads the original context. Step 2 reads the original context plus step 1’s output. Step 3 reads everything from steps 1 and 2. The cost compounds exponentially.

GitHub’s own agentic workflows illustrate this. A single Auto Triage PR workflow consumed 15.7 million tokens across just 5 runs, averaging roughly 186 turns per run. When GitHub systematically optimized their workflows in April 2026, they achieved a 62% reduction in effective token usage across 109 runs.

This compounding is also where context rot becomes a quality problem, not just a cost problem. As the context grows with each step, earlier information gets diluted, the model’s attention spreads thinner, and output quality degrades. This is the “lost in the middle” effect at scale: critical information from earlier steps gets buried under growing volumes of intermediate output.

The fix: State management between steps. Instead of passing raw context forward, extract structured intermediate results at each step and pass only those. This is a form of structured distillation: converting verbose natural language outputs into compact, structured representations (JSON objects, key value pairs, bullet summaries) that preserve the essential information at a fraction of the token cost. Add context compaction between steps: compress the accumulated context before the next step reads it. This breaks the exponential curve and turns it into something closer to linear growth.

Request chaining formalizes this pattern. Rather than a single long running context that accumulates everything, each step in the chain receives a curated input (the structured output of the previous step plus any new context it specifically needs) and produces a structured output for the next. The chain passes data, not conversation history.

8. Frontier Model Overuse

Best for understanding: Why model selection is a cost multiplier, not a quality guarantee.

Routing every request through GPT 4o or Claude Opus when many tasks (classification, extraction, formatting, simple Q&A) work fine on smaller models is one of the most expensive mistakes in production pipelines. The price differential between model tiers is not small: an analysis of 2.4 billion enterprise API calls found that organizations with tiered model architecture paid $2.31 per million tokens, while those routing everything to frontier models paid $18.40 per million. That is an 87% gap.

Starting June 15, 2026, Anthropic moved Claude Code and third party agent harnesses from flat subscriptions to credit metered billing at full API rates, precisely because agentic tool calling was consuming disproportionate compute per seat. The era of “just use the best model for everything” is ending.

The fix: Implement task type routing. Classify incoming requests by complexity, then route to the appropriate model tier. Simple extraction goes to Haiku or Flash. Complex reasoning goes to Opus or GPT 4o. This is not about sacrificing quality. It is about not overpaying for tasks that do not need frontier intelligence.

For a broader framework on managing these costs, the model routing cost guide covers model routing alongside other unit economics considerations.

Estimate your compression savings with Compresr’s transparent pricing at $0.10 per million tokens.

9. Unstructured Output Verbosity

Best for understanding: Why letting models freeform their responses costs you 30 to 50% extra tokens.

Without output constraints, models produce verbose natural language. They add preamble (“Sure, I’d be happy to help with that!”), restate the question, provide caveats, and wrap answers in unnecessary prose. On coding and extraction tasks, this verbosity accounts for 30 to 50% of response tokens.

Since output tokens are typically 2 to 4 times more expensive than input tokens per unit, this waste hits harder than the raw count suggests. The verbosity also creates downstream problems: if another agent or system consumes this output, it now has to parse through the prose to find the actual answer.

The fix: Use structured output schemas. JSON schemas, function calling formats, and explicit length constraints strip the verbose preamble and force the model to return only what you asked for. Structured distillation applies here too: rather than asking for a natural language explanation, request a structured object (JSON, YAML, typed fields) that captures exactly what the downstream consumer needs. Structured outputs are machine parseable by default, which eliminates retry costs from parsing failures. On coding and extraction tasks, structured output modes reduce response tokens by 30 to 50%.

10. Retry Loops from Ambiguous Errors

Best for understanding: Why poor error handling creates a hidden token multiplier.

When an agent encounters a vague error, it retries. Each retry re sends the full prompt context plus generates new output. Three retries on a single step means 4x the token cost of that step. In complex workflows, retries can cascade: a failed step triggers retries, which consume tokens, which might fail again if the root cause is not addressed.

The insidious part is that retries often succeed eventually (the model gets lucky on attempt 3), so teams do not notice the waste. The billing shows up as higher aggregate usage, but nobody traces it back to specific retry storms.

The fix: Structured error messages that give the agent enough information to change its approach rather than blindly retry. Validation before LLM calls catches obvious issues (malformed inputs, missing parameters) without burning tokens. Implement max retry limits with exponential backoff, and log retry events so you can identify which tools and which error types trigger the most waste.

11. Formatting Noise

Best for understanding: Why invisible characters are eating your token budget.

Colored terminal output (ANSI escape codes), bold and dim formatting, cursor control sequences, HTML tags, excessive whitespace: all of it gets serialized into the context and burns tokens for zero informational value. Practitioners on r/ClaudeCode report that git diff output alone, with its ANSI formatting and structural markers, can account for significant context waste.

This category of token waste in AI pipelines is easy to overlook because the noise is invisible in most interfaces. You see a neatly formatted diff; the model sees hundreds of escape sequences.

The fix: Strip ANSI codes, convert HTML to markdown, remove boilerplate whitespace before injecting any tool output into context. The AXON project and similar preprocessing pipelines claim 60 to 95% token reduction for structured data after cleaning. These are simple regex level transformations that should be standard in every tool output pipeline.

12. No Observability

Best for understanding: Why you cannot fix what you cannot see, and why this is the most important item on the list.

Most teams track aggregate token spend. They see a monthly bill going up. They do not know which workflow, which step, which tool call is responsible. According to a 2026 Stanford HAI study, teams that instrument token consumption at the tool call level identify optimization opportunities 3x faster than those relying on aggregate billing reports.

Without per step observability, every other item on this list remains invisible. You cannot prioritize fixes without knowing which waste sources dominate your specific pipeline.

The fix: Instrument everything. Log tokens consumed per workflow, per step, per tool call. Calculate efficiency ratios (cost per useful output, retrieval precision). Run daily audits on your top consuming workflows. GitHub built a dedicated token audit agent for exactly this purpose, and it is what enabled their 62% reduction.

The FinOps convergence is making this urgent. In 2025, 31% of FinOps practitioners managed AI spend. By 2026, that figure hit 98%. Engineering leaders are now expected to explain token spend line by line, not just defend a monthly total. For a full framework on AI FinOps governance, the discipline is maturing fast.

13. LLM Tool Calls That Should Be Direct API or CLI Calls

Best for understanding: Why routing deterministic operations through an LLM is pure waste.

This one is surprisingly common and rarely discussed in optimization guides. Teams build agentic workflows where the LLM decides to call a tool, generates a structured request, sends it, receives the result, and then interprets it. For many operations, the LLM’s involvement in the middle steps adds nothing.

Consider a file listing operation. An agent using an MCP server to list files sends the full tool schema, generates a tool call request, receives the directory listing, then reads and summarizes it. That entire roundtrip might burn 1,500 to 3,000 tokens. A direct ls or os.listdir() call returns the same information for zero tokens and executes in milliseconds instead of seconds.

Practitioners on Reddit’s r/LocalLLaMA frequently point out that many “agent” workflows are just expensive wrappers around operations that could be a simple function call. One developer shared that after auditing their agent’s tool usage, roughly 40% of tool calls were deterministic lookups or CRUD operations where the LLM added no reasoning value whatsoever.

The fix: Audit every tool call in your agent workflow and ask: does the LLM need to reason about this, or is the action fully determined by the preceding step? For deterministic operations (file reads, database lookups, API calls with known parameters, status checks), replace the LLM tool call with a direct programmatic call. The LLM should only be involved when it needs to decide what to do or interpret ambiguous results.

This does not mean removing tools from your agent entirely. It means building a hybrid architecture where deterministic steps execute as code and only ambiguous, reasoning heavy steps go through the LLM. The savings are dramatic because you eliminate both the input tokens (tool schema, request generation) and the output tokens (result interpretation) for every replaced call.

14. Externalize Logs and State to a Database

Best for understanding: Why your agent’s context window is not a database, and treating it like one is expensive.

Many agentic workflows accumulate operational state inside the conversation context. Execution logs, intermediate results, debugging traces, status updates: all of it piles up in the message history that gets replayed on every turn. An agent running a 15 step workflow might carry 5,000 to 10,000 tokens of log output that the model never needs to reference again.

This pattern emerges naturally because it is the easiest way to build an agent. Just append everything to the messages array. But the context window is the most expensive storage medium in your entire stack. Storing a kilobyte of state in context costs orders of magnitude more than storing it in Redis or Postgres, because you pay to re process it on every subsequent call.

One engineering manager shared in a YouTube walkthrough of their agent architecture that moving execution logs and intermediate state into a SQLite database (with only a short summary injected into context) cut their per task token usage by over 70%. The agent could still query the database when it needed historical state, but it was not paying to re read logs it would never look at again.

The fix: Architect your agent with an external state store from the start. Execution logs go to a database or structured log file. Intermediate computation results get stored in a key value store. Only a compact summary or pointer gets injected into the LLM context. When the agent needs to reference historical state, it queries the store explicitly rather than scanning through thousands of tokens of accumulated context.

This approach also improves debuggability. Structured logs in a database are searchable and queryable. Logs buried in a conversation transcript are not.

15. Cap Thinking Token Budgets

Best for understanding: Why unconstrained reasoning chains are the fastest growing source of hidden cost.

Models with extended thinking capabilities (Claude’s extended thinking, OpenAI’s o series reasoning, Gemini’s thinking mode) can burn enormous numbers of tokens on internal reasoning before producing a single word of visible output. A straightforward classification task that needs maybe 200 tokens of reasoning might consume 10,000 or more thinking tokens if no budget is set.

The problem is that thinking tokens are billed but invisible in the final output. Teams see the result, think the cost is reasonable, and never realize the model spent 15,000 tokens deliberating about something trivial. Practitioners on forums report cases where Claude’s extended thinking consumed 30,000+ thinking tokens on a task that a simple prompt without thinking mode handled in 500 total tokens.

This is particularly wasteful in agentic loops where thinking mode is enabled globally. Every step, including simple tool call decisions and formatting steps, triggers a full reasoning chain. The cost per task can spike by 5x to 10x without any corresponding quality improvement.

The fix: Set explicit max_tokens or budget_tokens limits on thinking/reasoning for each task type. Simple classification and extraction tasks should have thinking budgets of 500 tokens or less. Complex multi step reasoning might warrant 5,000 to 10,000. The key is matching the budget to the task complexity, not leaving it unbounded.

For OpenAI’s o series models, use the max_completion_tokens parameter. For Claude’s extended thinking, set budget_tokens in the thinking configuration. For agents built on frameworks like LangGraph, implement a pre call classifier that sets the thinking budget based on the detected task type.

Monitor thinking token usage in your observability dashboard. If a task type consistently uses less than 20% of its thinking budget, lower the cap.

16. Start New Sessions Per Topic

Best for understanding: Why long running sessions accumulate dead context that inflates every subsequent call.

Developers and agents commonly maintain a single long running session across multiple unrelated tasks. A coding agent that debugs a database issue, then refactors a UI component, then writes unit tests carries the full context of all three tasks in every subsequent call. By the third task, the session might contain 20,000+ tokens of context about database schemas and CSS styling that are completely irrelevant to writing test assertions.

This is different from the unbounded chat history problem (fix #3). History compression can help, but the more fundamental issue is architectural: the session itself has no concept of topic boundaries. Everything bleeds together.

Practitioners on r/ClaudeCode and r/cursor report that one of the simplest and most effective cost optimizations is just starting a fresh session when switching tasks. No compression needed, no summarization, no sliding windows. Just a clean slate. One user documented a 60% reduction in average tokens per request simply by splitting their work into topic scoped sessions instead of marathon conversations.

The fix: Implement session boundaries tied to task or topic changes. For interactive agents, detect topic shifts (using a lightweight classifier or explicit user signals) and start a new session. For automated pipelines, scope each workflow invocation to a single task with its own session.

If continuity between sessions matters, extract a compact handoff summary at session end: 200 to 500 tokens of structured state that captures decisions, constraints, and outcomes. Inject that summary into the new session’s system prompt. This preserves the essential context at 1% of the cost of dragging the full history forward.

17. Divide and Conquer Workflow Planning

Best for understanding: Why monolithic agent workflows duplicate context across every subtask.

When an agent receives a complex task like “research competitor pricing, draft a comparison table, and write recommendations,” the naive approach is to stuff all context into a single workflow that processes everything sequentially. Each step inherits the full context of every previous step, including context that is irrelevant to the current subtask.

The research step does not need the formatting preferences for the comparison table. The recommendation step does not need the raw search results that were already synthesized. But in a monolithic workflow, all of it travels together, inflating every call.

This pattern is especially wasteful in agentic coding scenarios. A developer shared on YouTube that their refactoring agent was processing an entire codebase’s context for each individual file change, even though each file edit was largely independent. Splitting the workflow into per file tasks with only relevant context cut their token usage by over 60%.

The fix: Decompose complex tasks into independent subtasks, each with its own minimal context scope. A planning step (which can be cheap, using a smaller model) breaks the task into subtasks and identifies what context each one needs. Each subtask then runs with only its required context, not the full accumulated state.

This mirrors how effective human teams work. A project manager creates a brief for each team member containing only what that person needs, not the entire project archive.

For implementation, frameworks like LangGraph support subgraph patterns that make this decomposition natural. The orchestrator node plans the subtasks, spawns subgraphs with scoped context, and aggregates results. The savings compound because each subtask processes a fraction of the total context, and subtasks that do not depend on each other can even run in parallel.

Latency Optimization: The Other Half of the Equation

Most discussion of token waste focuses on cost. But tokens are also the primary driver of latency in LLM calls. Prefill time (the time the model spends processing input tokens before generating the first output token) scales roughly linearly with input length. A prompt with 20,000 tokens of context takes meaningfully longer to start generating than one with 5,000 tokens.

This matters most for user facing applications where time to first token determines perceived responsiveness, and for agentic loops where each step waits for the previous one to complete. A 10 step agent workflow where each step carries unnecessary context is not just expensive. It is slow. Cutting 60% of input tokens does not just cut cost by 60%. It meaningfully reduces latency at every step, and those improvements compound across the workflow.

Compresr’s Boeing 10 K demo illustrates the dual benefit: compressing 112,552 tokens down to 498 tokens produced an 86% cost reduction and approximately 24% latency improvement on that call. For latency sensitive pipelines, the token cost reduction solution covers both dimensions.

Putting It All Together

Token waste in AI pipelines is structural, not incidental. It will not disappear with better prompts or cheaper models. The five highest impact moves, in order:

1. Instrument your pipelines at the tool call level so you know where the waste is. Establish baseline token logging and track cost per useful output as your north star metrics.
2. Compress context before it hits the LLM, using query aware semantic compression for RAG documents, tool outputs, chat history, and web search results. A/B test compression ratios against your evaluation set to find the right tradeoff.
3. Route tasks to appropriately sized models instead of defaulting to frontier.
4. Eliminate unnecessary LLM calls by replacing deterministic operations with direct code execution and externalizing state to databases.
5. Decompose workflows into scoped subtasks with bounded context and capped thinking budgets.

One clarification worth making: prompt caching and context compression solve different problems. Caching avoids reprocessing constant content. Compression eliminates wasteful content. Caching will not fix bloated context; it just makes the bloat cheaper to repeat. Most production pipelines benefit from doing both. For a detailed comparison, see prompt caching vs compression.

At $0.10 per million tokens compressed, the cost of compression is 10 to 180x cheaper than the waste it prevents. The math is straightforward.

Get started with Compresr’s quick start guide and start cutting token waste today.

FAQ

How much of the tokens in agentic AI workflows are wasted?

Research indicates that 50% to 99% of token usage in standard agentic workflows is redundant. In agentic coding specifically, actual code generation accounts for only 5 to 15% of total tokens. The rest is context overhead: system prompts, tool schemas, retrieved documents, conversation history, thinking tokens, and formatting noise.

What is the most common source of token waste in AI pipelines?

RAG context bloat and tool call output bloat are typically the largest per request sources. Retrieved documents routinely add 2,000 to 12,000 tokens per query, most of which is irrelevant to the specific question. But context re ingestion in multi step workflows produces the largest absolute waste because costs compound exponentially across steps.

Does context compression hurt LLM output quality?

Not necessarily. Query aware semantic compression, which keeps only the spans relevant to the current query, can actually improve accuracy by removing distracting context. On the FinanceBench benchmark, compression at roughly 2x improved accuracy from 73% to 77% compared to full context. This is partly a lost in the middle mitigation effect: shorter, more focused context lets the model attend to what matters. Naive truncation hurts quality; intelligent compression preserves or improves it.

How does prompt caching compare to context compression for reducing token costs?

They solve different problems and work well together. Prompt caching avoids reprocessing constant content (system prompts, static instructions) so you pay less per repetition. Context compression eliminates wasteful content entirely, so there is less to cache or process in the first place. Caching will not fix bloated context; it just makes the bloat cheaper to repeat.

Why do MCP servers use so many more tokens than CLI tools?

MCP servers inject full JSON schemas for every registered tool into every request. With 30 typed tools, that is 3,000+ tokens of schema definition before any user input. Analysis shows MCP servers use 35x more tokens than CLI tools on the same task, with reliability dropping as tool count grows. Progressive disclosure and tool spec gating (loading full schemas only for selected tools) are the primary mitigations.

What is the cost difference between tiered model routing and using frontier models for everything?

An analysis of 2.4 billion enterprise API calls found organizations with tiered model architecture achieved $2.31 per million tokens, while those routing everything to frontier models paid $18.40 per million. That is an 87% cost gap for workloads where simpler models produce equivalent results on non complex tasks.

How fast can teams find optimization opportunities with per step token observability?

According to a 2026 Stanford HAI study, teams that instrument token consumption at the tool call level identify optimization opportunities 3x faster than those relying on aggregate billing reports. Per step logging reveals exactly which workflows, steps, and tool calls are responsible for waste, turning a vague cost problem into an actionable engineering backlog.

Should I replace all LLM tool calls with direct API calls?

No. Only replace tool calls where the LLM adds no reasoning value, meaning the action is fully deterministic given the preceding step. Lookups, file reads, status checks, and CRUD operations with known parameters are good candidates. Keep LLM tool calls for operations requiring interpretation, decision making, or handling ambiguous inputs.

How much can capping thinking tokens actually save?

Thinking token budgets can reduce reasoning costs by 40% to 80% depending on the task mix. The key insight is that most tasks in an agentic pipeline are simple (tool selection, formatting, extraction) and do not need extended reasoning. Setting per task type budgets prevents the model from overthinking trivial operations while preserving deep reasoning capacity for tasks that genuinely need it.

What is the 5 to 15% rule in agentic coding?

It is a diagnostic principle drawn from practitioner data: in agentic coding workflows, actual code generation typically accounts for only 5 to 15% of total tokens consumed. The remaining 85 to 95% is context overhead. This framing helps teams understand that optimizing the code generation step is far less impactful than optimizing the context that surrounds it.

What is the difference between semantic compression and manual summarization?

Semantic compression algorithmically identifies and retains the spans of text most relevant to a given query, preserving original wording while discarding irrelevant content. Manual summarization (whether human written or LLM generated) rephrases and condenses content into new text. Semantic compression is faster, more scalable, and preserves exact phrasing, which matters for factual accuracy. Manual summarization can be more aggressive in compression ratio but introduces the risk of paraphrasing errors and requires either human time or additional LLM calls (which themselves cost tokens). For most production pipelines, semantic compression is the better default, with manual summarization reserved for cases where extreme compression or editorial control is needed.

How should I A/B test compression ratios?

Run your evaluation dataset at multiple compression levels (for example, 2x, 4x, and 8x) and measure accuracy, completeness, and cost at each level. Plot the results to find the inflection point where accuracy begins to drop meaningfully. Most teams find that light compression (2x to 3x) matches or improves baseline accuracy, medium compression (4x to 6x) trades a small accuracy reduction for large cost savings, and aggressive compression (8x+) is appropriate only for cost sensitive, error tolerant workloads.