Token Waste in AI Pipelines: 17 Fixes That Work (2026)

TL;DR

Token waste in AI pipelines is the consumption of LLM input or output tokens that contribute nothing to the accuracy, quality, or usefulness of the final result. It is the structural cost problem of production AI. Agentic AI workflows consume 5 to 30 times more tokens per task than standard chatbot exchanges, and research suggests 50% to 99% of that usage is redundant. This waste hides inside RAG retrieval bloat, tool call outputs, conversation history, inflated system prompts, runaway thinking budgets, and over a dozen other structural problems. This article maps the 17 biggest sources, quantifies each one, and prescribes specific fixes, from query aware context compression to model routing to workflow decomposition to observability instrumentation.

What is Token Waste in AI Pipelines?

Token waste refers to the consumption of Large Language Model (LLM) input or output tokens that do not contribute to the accuracy, quality, or context of the final output. In production AI systems, token waste typically manifests as raw, unfiltered database strings, overly dense documentation schemas, conversational repetitions, uncapped reasoning chains, redundant session state, or natural language preambles.

Quick Takeaways:

• The Agentic Multiplier: Agentic workflows consume 5 to 30 times more tokens than isolated chatbot exchanges due to iterative loops and persistent state tracking.
• Context Overhead vs. Value: In agentic development pipelines, 85% to 95% of total spend is eaten by systemic context overhead (system prompts, tool definitions, dynamic context), leaving only 5% to 15% for actual generation.
• The Remediation Layer: Implementing a precision layer, such as query aware context compression, removes the noise before it hits the model, yielding significant cost reductions while protecting execution accuracy.

The Scale of the Problem

Goldman Sachs Research forecasts that token consumption will multiply 24 times between 2026 and 2030, reaching 120 quadrillion tokens per month. That growth isn't just more users asking more questions. It's agentic workflows burning through context windows at rates nobody budgeted for.

Gartner's 2026 analysis found that a single agentic AI task consumes 5 to 30 times more tokens than a standard chatbot exchange. One agent answering a support ticket can burn the same tokens as 30 chat conversations. A Stanford Digital Economy Lab paper pushed that number further: agentic tasks are "uniquely expensive, consuming 1000x more tokens than code reasoning and code chat," with input tokens driving the bulk of the cost.

Here's the number that should change how you think about this: in agentic coding workflows, actual code generation accounts for only 5 to 15% of total tokens consumed. Everything else is context overhead. System prompts, tool schemas, retrieved documents, conversation history, formatting noise. Most of it contributes nothing to output quality.

The financial stakes are real. A 100 developer organization mixing inline and agentic AI coding tools now faces $400,000 to $600,000 in annual token costs before governance infrastructure. And 95% of enterprise AI pilots deliver zero measurable ROI, with cost sustainability cited as a primary driver of abandonment.

Token waste in AI pipelines is not a minor inefficiency. It is the structural cost problem of production AI.

Try Compresr's free demo to see how query aware compression reduces token bloat in real time.

At a Glance: 17 Waste Sources and Their Fixes

Now let's break each one down.

1. RAG Context Bloat

Best for understanding: Why retrieval augmented generation is often retrieval augmented waste.

RAG pipelines retrieve document chunks based on vector similarity, then stuff them wholesale into the prompt. The problem is that retrieval is not relevance filtering. A chunk matched because it contains a relevant paragraph will also carry headers, boilerplate, definitions, and tangential content that have nothing to do with the query.

Retrieved documents routinely add 2,000 to 12,000 tokens per query. Most of that content is irrelevant to the specific question being asked. You're paying for the LLM to read (and be distracted by) content that doesn't help it answer.

The fix: Query aware compression that retains only the spans relevant to the user's query before the context reaches the LLM. Unlike naive truncation or generic summarization, query aware compression evaluates each span against the actual question and preserves what matters.

Practitioners on Reddit's r/LangChain report dramatic results. One developer cut query costs by 60% just by adding a compression step before retrieval results hit the model. The waste was entirely in how much raw context got packed into each prompt.

There's a counterintuitive finding worth noting: on the FinanceBench benchmark, Compresr's query aware compression at roughly 2x compression improved accuracy from 73% to 77% compared to full context. Removing distracting content didn't just save money. It made the answers better. For teams struggling with over retrieval costs, this is often the single highest ROI fix.

For a deeper walkthrough, see the RAG compression guide.

2. Tool Call Output Bloat

Best for understanding: Why your agent's API calls are bleeding tokens.

When an agent queries an API, reads a file, or runs a database query, the raw output gets appended to context. A single JSON blob returned with 50 fields, when the agent only needs 3, consumes thousands of tokens for zero additional value.

This is one of the most common sources of token waste in AI pipelines, and it scales linearly with agent complexity. More tools means more raw outputs means more wasted context.

The fix: Two layers work together here. First, field filtering at the tool level: configure your API wrappers to return only the fields the agent actually needs. This alone can reduce payload tokens by 80 to 90%. Second, for outputs that can't be pre filtered (file contents, search results, dynamic responses), compress them before they enter the prompt. For detailed tactics on trimming tool call tokens, there's a dedicated guide worth reading.

Compresr provides LangChain middleware specifically for tool output compression, so the compression fires automatically in the right spot without manual plumbing on every tool call.

3. Unbounded Conversation History

Best for understanding: Why multi turn agents get exponentially more expensive per turn.

Multi turn agents replay the full message history on every request. After 10 turns, conversation history alone can exceed 15,000 tokens. After 20 turns, you might be spending more on history replay than on the actual task.

This is straightforward compounding. Each turn adds its own content plus re sends everything that came before. The early turns, which established context the model has long since incorporated into its working state, get re processed at full cost every single time.

The fix: Three approaches, in order of sophistication: sliding window (keep only the last N turns), summarization (condense older turns into a summary), or query aware history compression (keep the spans from history that are relevant to the current turn). The third approach is the most token efficient because it adapts to what's actually being discussed.

Practitioners report 60 to 72% reductions in history related token usage with compression based approaches, without degrading the model's ability to maintain conversational coherence. For teams already managing conversation history costs, conversation history is often the single biggest line item to attack.

4. System Prompt Creep

Best for understanding: Why your system prompt quietly became a cost center.

System prompts accumulate instructions the way codebases accumulate dead code. A developer adds a rule for edge case handling. Another adds formatting guidance. A third adds persona instructions. Nobody removes anything. A system prompt that started at 500 tokens balloons to 5,000.

The cost isn't just the 4,500 extra tokens. It's that those tokens get sent on every single request. At 1,000 requests per day, that's 4.5 million wasted tokens daily, just from instructions that might be contradictory or obsolete.

The fix: Regular audits. Schedule quarterly system prompt reviews the same way you schedule dependency updates. Test each instruction block independently: does removing it change output quality on your eval set? Anthropic's own benchmarks have shown that concise prompts often improve task completion, not just reduce cost.

There's no tool that fixes this automatically. It's a discipline problem. But it's also one of the cheapest wins in the list because it requires zero infrastructure changes. Teams tracking system prompt costs often discover that half their instructions are redundant.

5. MCP Tool Schema Inflation

Best for understanding: Why the Model Context Protocol creates a hidden per request tax.

If your AI agent has 30 registered MCP tools, that's 3,000+ tokens of JSON schema injected into every single request before a single line of user input appears. The agent needs to "see" all available tools to decide which one to use, and each tool definition includes its name, description, and full parameter schema.

Recent analysis shows MCP servers use 35x more tokens than CLI tools performing the same task, with reliability actually dropping from 100% to 72% as tool complexity grows. The token overhead isn't just costly, it actively degrades performance by diluting the context with schema definitions the agent doesn't need for the current task.

The fix: Progressive disclosure. Instead of loading all tool schemas upfront, implement a two stage approach: give the agent a lightweight tool catalog (names and one line descriptions), then load full schemas only for tools the agent selects. This mirrors how developers work: you browse a directory before reading full documentation.

Tool pruning helps too. If an agent uses 8 of its 30 tools in 95% of requests, consider splitting into specialized agents with smaller toolsets.

6. Uncompressed Web Search Payloads

Best for understanding: Why search augmented agents waste the most tokens per operation.

Most AI agents that search the web fetch entire pages, dump thousands of tokens of raw HTML stripped text into the context window, and hope the model figures out what matters. A single web search result can add 2,000 to 8,000 tokens, and agents often pull multiple results per query.

Running content through a readability parser before passing it to the model cuts token count by 60 to 80% for article pages. But readability parsing only removes structural noise. It doesn't remove content that's irrelevant to the query.

The fix: Two stage processing. First, readability parsing to strip navigation, ads, and boilerplate HTML. Second, query aware compression to keep only the spans that answer the actual question. The combination routinely achieves 80%+ reduction while preserving the information the agent needs.

This matters especially for research agents and RAG pipelines that pull from live sources. Without compression, a single research task that triggers 5 web searches can consume 20,000 to 40,000 tokens of search content alone.

7. Context Re ingestion at Every Agent Step

Best for understanding: Why multi step workflows create exponential cost curves.

In multi step agentic workflows, the full accumulated context gets re read at every step. Step 1 reads the original context. Step 2 reads the original context plus step 1's output. Step 3 reads everything from steps 1 and 2. The cost compounds exponentially.

GitHub's own agentic workflows illustrate this. A single Auto Triage PR workflow consumed 15.7 million tokens across just 5 runs, averaging roughly 186 turns per run. When GitHub systematically optimized their workflows in April 2026, they achieved a 62% reduction in effective token usage across 109 runs.

This compounding is also where context rot becomes a quality problem, not just a cost problem. As the context grows with each step, earlier information gets diluted, the model's attention spreads thinner, and output quality degrades.

The fix: State management between steps. Instead of passing raw context forward, extract structured intermediate results at each step and pass only those. Add context compaction between steps: compress the accumulated context before the next step reads it. This breaks the exponential curve and turns it into something closer to linear growth.

8. Frontier Model Overuse

Best for understanding: Why model selection is a cost multiplier, not a quality guarantee.

Routing every request through GPT 4o or Claude Opus when many tasks (classification, extraction, formatting, simple Q&A) work fine on smaller models is one of the most expensive mistakes in production pipelines. The price differential between model tiers is not small: an analysis of 2.4 billion enterprise API calls found that organizations with tiered model architecture paid $2.31 per million tokens, while those routing everything to frontier models paid $18.40 per million. That's an 87% gap.

Starting June 15, 2026, Anthropic moved Claude Code and third party agent harnesses from flat subscriptions to credit metered billing at full API rates, precisely because agentic tool calling was consuming disproportionate compute per seat. The era of "just use the best model for everything" is ending.

The fix: Implement task type routing. Classify incoming requests by complexity, then route to the appropriate model tier. Simple extraction goes to Haiku or Flash. Complex reasoning goes to Opus or GPT 4o. This isn't about sacrificing quality. It's about not overpaying for tasks that don't need frontier intelligence.

For a broader framework on managing these costs, the model routing cost guide covers model routing alongside other unit economics considerations.

9. Unstructured Output Verbosity

Best for understanding: Why letting models freeform their responses costs you 30 to 50% extra tokens.

Without output constraints, models produce verbose natural language. They add preamble ("Sure, I'd be happy to help with that!"), restate the question, provide caveats, and wrap answers in unnecessary prose. On coding and extraction tasks, this verbosity accounts for 30 to 50% of response tokens.

Since output tokens are typically 2 to 4 times more expensive than input tokens per unit, this waste hits harder than the raw count suggests. The verbosity also creates downstream problems: if another agent or system consumes this output, it now has to parse through the prose to find the actual answer.

The fix: Use structured output schemas. JSON schemas, function calling formats, and explicit length constraints strip the verbose preamble and force the model to return only what you asked for. Structured outputs are machine parseable by default, which eliminates retry costs from parsing failures. On coding and extraction tasks, structured output modes reduce response tokens by 30 to 50%.

10. Retry Loops from Ambiguous Errors

Best for understanding: Why poor error handling creates a hidden token multiplier.

When an agent encounters a vague error, it retries. Each retry re sends the full prompt context plus generates new output. Three retries on a single step means 4x the token cost of that step. In complex workflows, retries can cascade: a failed step triggers retries, which consume tokens, which might fail again if the root cause isn't addressed.

The insidious part is that retries often succeed eventually (the model gets lucky on attempt 3), so teams don't notice the waste. The billing shows up as higher aggregate usage, but nobody traces it back to specific retry storms.

The fix: Structured error messages that give the agent enough information to change its approach rather than blindly retry. Validation before LLM calls catches obvious issues (malformed inputs, missing parameters) without burning tokens. Implement max retry limits with exponential backoff, and log retry events so you can identify which tools and which error types trigger the most waste.

11. Formatting Noise

Best for understanding: Why invisible characters are eating your token budget.

Colored terminal output (ANSI escape codes), bold and dim formatting, cursor control sequences, HTML tags, excessive whitespace: all of it gets serialized into the context and burns tokens for zero informational value. Practitioners on r/ClaudeCode report that git diff output alone, with its ANSI formatting and structural markers, can account for significant context waste.

This category of token waste in AI pipelines is easy to overlook because the noise is invisible in most interfaces. You see a neatly formatted diff; the model sees hundreds of escape sequences.

The fix: Strip ANSI codes, convert HTML to markdown, remove boilerplate whitespace before injecting any tool output into context. The AXON project and similar preprocessing pipelines claim 60 to 95% token reduction for structured data after cleaning. These are simple regex level transformations that should be standard in every tool output pipeline.

12. No Observability

Best for understanding: Why you can't fix what you can't see, and why this is the most important item on the list.

Most teams track aggregate token spend. They see a monthly bill going up. They don't know which workflow, which step, which tool call is responsible. According to a 2026 Stanford HAI study, teams that instrument token consumption at the tool call level identify optimization opportunities 3x faster than those relying on aggregate billing reports.

Without per step observability, every other item on this list remains invisible. You can't prioritize fixes without knowing which waste sources dominate your specific pipeline.

The fix: Instrument everything. Log tokens consumed per workflow, per step, per tool call. Calculate efficiency ratios (output quality per token). Run daily audits on your top consuming workflows. GitHub built a dedicated token audit agent for exactly this purpose, and it's what enabled their 62% reduction.

The FinOps convergence is making this urgent. In 2025, 31% of FinOps practitioners managed AI spend. By 2026, that figure hit 98%. Engineering leaders are now expected to explain token spend line by line, not just defend a monthly total. For a full framework on AI FinOps governance, the discipline is maturing fast.

Estimate your compression savings with Compresr's transparent pricing at $0.10 per million tokens.

13. LLM Tool Calls That Should Be Direct API or CLI Calls

Best for understanding: Why routing deterministic operations through an LLM is pure waste.

This one is surprisingly common and rarely discussed in optimization guides. Teams build agentic workflows where the LLM decides to call a tool, generates a structured request, sends it, receives the result, and then interprets it. For many operations, the LLM's involvement in the middle steps adds nothing.

Consider a file listing operation. An agent using an MCP server to list files sends the full tool schema, generates a tool call request, receives the directory listing, then reads and summarizes it. That entire roundtrip might burn 1,500 to 3,000 tokens. A direct ls or os.listdir() call returns the same information for zero tokens and executes in milliseconds instead of seconds.

Practitioners on Reddit's r/LocalLLaMA frequently point out that many "agent" workflows are just expensive wrappers around operations that could be a simple function call. One developer shared that after auditing their agent's tool usage, roughly 40% of tool calls were deterministic lookups or CRUD operations where the LLM added no reasoning value whatsoever.

The fix: Audit every tool call in your agent workflow and ask: does the LLM need to reason about this, or is the action fully determined by the preceding step? For deterministic operations (file reads, database lookups, API calls with known parameters, status checks), replace the LLM tool call with a direct programmatic call. The LLM should only be involved when it needs to decide what to do or interpret ambiguous results.

This doesn't mean removing tools from your agent entirely. It means building a hybrid architecture where deterministic steps execute as code and only ambiguous, reasoning heavy steps go through the LLM. The savings are dramatic because you eliminate both the input tokens (tool schema, request generation) and the output tokens (result interpretation) for every replaced call.

14. Externalize Logs and State to a Database

Best for understanding: Why your agent's context window is not a database, and treating it like one is expensive.

Many agentic workflows accumulate operational state inside the conversation context. Execution logs, intermediate results, debugging traces, status updates: all of it piles up in the message history that gets replayed on every turn. An agent running a 15 step workflow might carry 5,000 to 10,000 tokens of log output that the model never needs to reference again.

This pattern emerges naturally because it's the easiest way to build an agent. Just append everything to the messages array. But the context window is the most expensive storage medium in your entire stack. Storing a kilobyte of state in context costs orders of magnitude more than storing it in Redis or Postgres, because you pay to re process it on every subsequent call.

One engineering manager shared in a YouTube walkthrough of their agent architecture that moving execution logs and intermediate state into a SQLite database (with only a short summary injected into context) cut their per task token usage by over 70%. The agent could still query the database when it needed historical state, but it wasn't paying to re read logs it would never look at again.

The fix: Architect your agent with an external state store from the start. Execution logs go to a database or structured log file. Intermediate computation results get stored in a key value store. Only a compact summary or pointer gets injected into the LLM context. When the agent needs to reference historical state, it queries the store explicitly rather than scanning through thousands of tokens of accumulated context.

This approach also improves debuggability. Structured logs in a database are searchable and queryable. Logs buried in a conversation transcript are not.

15. Cap Thinking Token Budgets

Best for understanding: Why unconstrained reasoning chains are the fastest growing source of hidden cost.

Models with extended thinking capabilities (Claude's extended thinking, OpenAI's o series reasoning, Gemini's thinking mode) can burn enormous numbers of tokens on internal reasoning before producing a single word of visible output. A straightforward classification task that needs maybe 200 tokens of reasoning might consume 10,000 or more thinking tokens if no budget is set.

The problem is that thinking tokens are billed but invisible in the final output. Teams see the result, think the cost is reasonable, and never realize the model spent 15,000 tokens deliberating about something trivial. Practitioners on forums report cases where Claude's extended thinking consumed 30,000+ thinking tokens on a task that a simple prompt without thinking mode handled in 500 total tokens.

This is particularly wasteful in agentic loops where thinking mode is enabled globally. Every step, including simple tool call decisions and formatting steps, triggers a full reasoning chain. The cost per task can spike by 5x to 10x without any corresponding quality improvement.

The fix: Set explicit max_tokens or budget_tokens limits on thinking/reasoning for each task type. Simple classification and extraction tasks should have thinking budgets of 500 tokens or less. Complex multi step reasoning might warrant 5,000 to 10,000. The key is matching the budget to the task complexity, not leaving it unbounded.

For OpenAI's o series models, use the max_completion_tokens parameter. For Claude's extended thinking, set budget_tokens in the thinking configuration. For agents built on frameworks like LangGraph, implement a pre call classifier that sets the thinking budget based on the detected task type.

Monitor thinking token usage in your observability dashboard. If a task type consistently uses less than 20% of its thinking budget, lower the cap.

16. Start New Sessions Per Topic

Best for understanding: Why long running sessions accumulate dead context that inflates every subsequent call.

Developers and agents commonly maintain a single long running session across multiple unrelated tasks. A coding agent that debugs a database issue, then refactors a UI component, then writes unit tests carries the full context of all three tasks in every subsequent call. By the third task, the session might contain 20,000+ tokens of context about database schemas and CSS styling that are completely irrelevant to writing test assertions.

This is different from the unbounded chat history problem (fix #3). History compression can help, but the more fundamental issue is architectural: the session itself has no concept of topic boundaries. Everything bleeds together.

Practitioners on r/ClaudeCode and r/cursor report that one of the simplest and most effective cost optimizations is just starting a fresh session when switching tasks. No compression needed, no summarization, no sliding windows. Just a clean slate. One user documented a 60% reduction in average tokens per request simply by splitting their work into topic scoped sessions instead of marathon conversations.

The fix: Implement session boundaries tied to task or topic changes. For interactive agents, detect topic shifts (using a lightweight classifier or explicit user signals) and start a new session. For automated pipelines, scope each workflow invocation to a single task with its own session.

If continuity between sessions matters, extract a compact handoff summary at session end: 200 to 500 tokens of structured state that captures decisions, constraints, and outcomes. Inject that summary into the new session's system prompt. This preserves the essential context at 1% of the cost of dragging the full history forward.

17. Divide and Conquer Workflow Planning

Best for understanding: Why monolithic agent workflows duplicate context across every subtask.

When an agent receives a complex task like "research competitor pricing, draft a comparison table, and write recommendations," the naive approach is to stuff all context into a single workflow that processes everything sequentially. Each step inherits the full context of every previous step, including context that's irrelevant to the current subtask.

The research step doesn't need the formatting preferences for the comparison table. The recommendation step doesn't need the raw search results that were already synthesized. But in a monolithic workflow, all of it travels together, inflating every call.

This pattern is especially wasteful in agentic coding scenarios. A developer shared on YouTube that their refactoring agent was processing an entire codebase's context for each individual file change, even though each file edit was largely independent. Splitting the workflow into per file tasks with only relevant context cut their token usage by over 60%.

The fix: Decompose complex tasks into independent subtasks, each with its own minimal context scope. A planning step (which can be cheap, using a smaller model) breaks the task into subtasks and identifies what context each one needs. Each subtask then runs with only its required context, not the full accumulated state.

This mirrors how effective human teams work. A project manager creates a brief for each team member containing only what that person needs, not the entire project archive.

For implementation, frameworks like LangGraph support subgraph patterns that make this decomposition natural. The orchestrator node plans the subtasks, spawns subgraphs with scoped context, and aggregates results. The savings compound because each subtask processes a fraction of the total context, and subtasks that don't depend on each other can even run in parallel.

Putting It All Together

Token waste in AI pipelines is structural, not incidental. It won't disappear with better prompts or cheaper models. The five highest impact moves, in order:

1. Instrument your pipelines at the tool call level so you know where the waste is.
2. Compress context before it hits the LLM, using query aware compression for RAG documents, tool outputs, chat history, and web search results.
3. Route tasks to appropriately sized models instead of defaulting to frontier.
4. Eliminate unnecessary LLM calls by replacing deterministic operations with direct code execution and externalizing state to databases.
5. Decompose workflows into scoped subtasks with bounded context and capped thinking budgets.

One clarification worth making: prompt caching and context compression solve different problems. Caching avoids reprocessing constant content. Compression eliminates wasteful content. Caching won't fix bloated context; it just makes the bloat cheaper to repeat. Most production pipelines benefit from doing both. For a detailed comparison, see prompt caching vs compression.

At $0.10 per million tokens compressed, the cost of compression is 10 to 180x cheaper than the waste it prevents. The math is straightforward.

Get started with Compresr's quick start guide and start cutting token waste today.

FAQ

How much of the tokens in agentic AI workflows are wasted?

Research indicates that 50% to 99% of token usage in standard agentic workflows is redundant. In agentic coding specifically, actual code generation accounts for only 5 to 15% of total tokens. The rest is context overhead: system prompts, tool schemas, retrieved documents, conversation history, thinking tokens, and formatting noise.

What is the most common source of token waste in AI pipelines?

RAG context bloat and tool call output bloat are typically the largest per request sources. Retrieved documents routinely add 2,000 to 12,000 tokens per query, most of which is irrelevant to the specific question. But context re ingestion in multi step workflows produces the largest absolute waste because costs compound exponentially across steps.

Does context compression hurt LLM output quality?

Not necessarily. Query aware compression, which keeps only the spans relevant to the current query, can actually improve accuracy by removing distracting context. On the FinanceBench benchmark, compression at roughly 2x improved accuracy from 73% to 77% compared to full context. Naive truncation hurts quality; intelligent compression preserves or improves it.

How does prompt caching compare to context compression for reducing token costs?

They solve different problems and work well together. Prompt caching avoids reprocessing constant content (system prompts, static instructions) so you pay less per repetition. Context compression eliminates wasteful content entirely, so there's less to cache or process in the first place. Caching won't fix bloated context; it just makes the bloat cheaper to repeat.

Why do MCP servers use so many more tokens than CLI tools?

MCP servers inject full JSON schemas for every registered tool into every request. With 30 typed tools, that's 3,000+ tokens of schema definition before any user input. Analysis shows MCP servers use 35x more tokens than CLI tools on the same task, with reliability dropping as tool count grows. Progressive disclosure (loading full schemas only for selected tools) is the primary mitigation.

What is the cost difference between tiered model routing and using frontier models for everything?

An analysis of 2.4 billion enterprise API calls found organizations with tiered model architecture achieved $2.31 per million tokens, while those routing everything to frontier models paid $18.40 per million. That's an 87% cost gap for workloads where simpler models produce equivalent results on non complex tasks.

How fast can teams find optimization opportunities with per step token observability?

According to a 2026 Stanford HAI study, teams that instrument token consumption at the tool call level identify optimization opportunities 3x faster than those relying on aggregate billing reports. Per step logging reveals exactly which workflows, steps, and tool calls are responsible for waste, turning a vague cost problem into an actionable engineering backlog.

Should I replace all LLM tool calls with direct API calls?

No. Only replace tool calls where the LLM adds no reasoning value, meaning the action is fully deterministic given the preceding step. Lookups, file reads, status checks, and CRUD operations with known parameters are good candidates. Keep LLM tool calls for operations requiring interpretation, decision making, or handling ambiguous inputs.

How much can capping thinking tokens actually save?

Thinking token budgets can reduce reasoning costs by 40% to 80% depending on the task mix. The key insight is that most tasks in an agentic pipeline are simple (tool selection, formatting, extraction) and don't need extended reasoning. Setting per task type budgets prevents the model from overthinking trivial operations while preserving deep reasoning capacity for tasks that genuinely need it.

What is the 5 to 15% rule in agentic coding?

It's a diagnostic principle drawn from practitioner data: in agentic coding workflows, actual code generation typically accounts for only 5 to 15% of total tokens consumed. The remaining 85 to 95% is context overhead. This framing helps teams understand that optimizing the code generation step is far less impactful than optimizing the context that surrounds it.