9 Proven Ways to Cut Conversation History Token Costs (2026)

TL;DR

Conversation history token costs compound because the model rereads prior turns, tool outputs, and system prompts on every call, not just your user’s latest message. A 20-turn agent session can quietly accumulate over 270,000 input tokens even when each individual message is short. The fix is not “write shorter prompts.” It is treating different types of context differently: cache stable prefixes, compress large relevant context, compact tool outputs, externalize raw state, retrieve durable memory selectively, and measure cost per successful task. This guide ranks nine practical methods by savings potential, quality risk, and best-fit workload.

The Model Is Rereading More Than Your User Typed

Most teams discover conversation history token costs the hard way. A user sends a 15-word question. The bill shows thousands of input tokens. The confusion is understandable, but the math is straightforward: the model does not remember previous turns. It receives them again as part of the input. That means every prior message, every system instruction, every tool schema, every tool result, and every assistant reply gets counted as input tokens on every single call.

OpenAI’s help center defines token usage categories as input tokens, output tokens, and cached tokens, with pricing per token varying by model and category. Tokenization also differs across models, so a lower price per million tokens does not always translate to lower total cost.

This creates a pattern that catches builders off guard. Practitioners on Reddit describe Claude agents where message 1 uses around 6,000 input tokens, message 5 uses around 10,000, message 10 around 15,000, and message 20 exceeds 22,000 tokens. The user’s messages stayed short. The context surrounding them did not.

For a deeper look at the input versus output cost split, see our guide on input token costs.

Agents make the problem worse. In a ReAct-style tool-calling loop, each step appends new messages, tool calls, tool results, and observations. One developer documented a QuickBooks agent where the fourth call ballooned to 24,612 tokens, with non-cached input representing 86% of the cost. The model was mostly rereading old context, not doing new reasoning.

A LinkedIn practitioner reported that across 1,127 agent runs, 52.1% of spend came from context rereads, more than new input, output, tool fees, or retries combined.

The rest of this article ranks nine methods to reduce conversation history token costs, explains when each one works, and warns you when it does not.

Try query-aware compression on your own chat histories and RAG payloads to see how much context you can safely remove.

Quick Answer: How to Cut Conversation History Token Costs

To lower multi-turn LLM and agent token costs, optimize how past context is handled rather than shortening user prompts:

1. Cache Static Context: Use provider prompt caching on system prompts and tool schemas to save up to 80% to 90% on repeated inputs.

2. Compact Tool Results: Offload raw JSON/API responses to external storage and pass only short handles or summaries into history.

3. Compress Dynamic History: Apply query-aware context compression or sliding window trimming to drop dead spans before calling the model.

4. Isolate Durable Memory: Extract user facts into a database or vector store and retrieve them selectively instead of replaying full transcripts.

5. Route by Intent: Pass simple turns to smaller, cheaper models and reserve long-context frontier models for complex multi-step reasoning.

At-a-Glance Comparison: 9 Methods to Reduce Token Costs

Are You Actually Charged for Conversation History Tokens?

Yes. This is the most common source of confusion, and it trips up experienced developers too.

When you use a chat API, you send the conversation so far as part of the request. The model processes all of those tokens. You pay for all of those tokens. A longer history means a larger input on every subsequent turn.

“But I’m using a thread ID / conversation state / Assistants API. Doesn’t the provider handle memory?”

Provider-managed state simplifies your code. It does not automatically make prior context free. OpenAI community discussions show persistent confusion around this point, with users discovering that Assistants-style APIs can still reflect full context in billing. Cached tokens may get a discount, but a discount is not the same as deletion. You still need a token-budgeting strategy.

Three billing myths worth clearing up:

• Myth: “Thread-based APIs only bill the new message.” Reality: The model still processes relevant prior context. Provider-managed state reduces your client-side work, not necessarily your token bill.

• Myth: “Longer context windows make history cheaper.” Reality: A larger window lets you fit more tokens. It does not make them free. Research on long-context models shows accuracy can degrade when relevant information is buried in the middle of a long input.

• Myth: “Prompt caching eliminates conversation history costs.” Reality: Caching helps with stable, repeated prefixes. Dynamic chat history changes every turn, which limits cache hits.

Why Conversation History Token Costs Grow Faster Than Expected

The growth is not linear. It is closer to quadratic.

In a multi-turn conversation, the model receives the full history on each call. If S is your static prefix (system prompt + tool schemas), H is the average new tokens added per turn, and N is the number of turns, the rough cumulative input tokens look like this:

Total input ≈ N × S + H × N(N - 1) / 2

Here is a concrete example:

S = 6,000 tokens (system prompt + tools)
H = 800 new tokens per turn
N = 20 turns

Total input ≈ 20 × 6,000 + 800 × 20 × 19 / 2
Total input ≈ 120,000 + 152,000
Total input ≈ 272,000 input tokens

A naive estimate of “20 turns times 6,800 tokens” would give you 136,000 tokens. The actual number is roughly double that because of the compounding tail. Augment Code frames this mathematically: naive agent loops rebill prior context at every step, making input token cost grow at O(N²).

Tool outputs make it even worse. A 4,000-token tool result added at step 3 of a 12-step run gets reprocessed on steps 4 through 12. That single result costs 36,000 repeated input tokens across those nine future calls. This is what practitioners call the “tool output tail tax,” and it is the biggest blind spot in generic cost articles.

A Reddit user building an autonomous security-testing agent described hitting 50,000 to 100,000 tokens per request after 20 to 30 turns. Each individual turn added a modest number of tokens. The cumulative effect was brutal.

For a broader view of where tokens go to waste, see our breakdown of token waste in AI pipelines.

The Five Types of Context (and Why They Need Different Treatment)

Most articles treat “conversation history” as a single blob. It is not. Your prompt contains at least five distinct types of context, and each one has a different optimal treatment:

Now let’s walk through nine methods to bring those costs under control.

1. Query-Aware Compression

Best for: Long chat history, RAG documents, web search results, and tool outputs where relevance to the current query matters.

When the expensive part of your input is large but only partly relevant, query-aware compression is the best default first move. Unlike generic summarization, it removes spans that are irrelevant to the current query while keeping the parts the model actually needs to answer.

Why it matters for conversation history: Old turns often contain information the model no longer needs. A support conversation about a billing issue three turns ago is irrelevant when the user is now asking about API rate limits. Query-aware compression identifies and removes those dead spans without losing the parts that still matter.

Compresr is a query-aware LLM context compression API and SDK that compresses long prompts, chat histories, RAG documents, and tool outputs before they reach the LLM. It is especially relevant when you cannot simply truncate history because old context may still contain the answer.

Key features:

• Two compression models (latte_v1 and latte_v2), with latte_v2 supporting dynamic ratio selection

• Coarse paragraph-level and fine token-level compression

• Batch and streaming endpoints

• Python and TypeScript SDKs

• First-party integrations for LangChain, LlamaIndex, LangGraph, and LiteLLM

• Web search tooling for Tavily, Brave, and Amazon Bedrock AgentCore payloads

• Markdown-aware endpoint for structure-preserving compression

• On-prem GPU deployment for regulated workloads

Pricing: $0.10 per 1M tokens compressed. $10 free credits on signup, no credit card required.

Break-even math:

Input before compression: 10,000 tokens
Input after compression: 3,000 tokens
Tokens removed: 7,000

Downstream model input price: $3/1M tokens
Downstream savings: 7,000 × $3 / 1,000,000 = $0.021

Compresr cost: 10,000 × $0.10 / 1,000,000 = $0.001

Net savings per call: ~$0.020

Actual savings depend on your downstream model price, compression ratio, and whether quality holds. For very short contexts under roughly 500 tokens, API overhead may outweigh savings, so use minimum-token thresholds to skip compression on small inputs.

Practitioner evidence: Developers on Reddit explicitly ask for tool-result compression and sliding-window history in agent workflows. Commenters recommend stripping tool results down to the fields the agent actually reads, which is exactly the kind of targeted removal query-aware compression automates.

The “Lost in the Middle” paper supports this approach from a quality angle: sending more context is not always better because models can underperform when relevant information is buried in long inputs.

Tradeoffs:

• Provider-native server search results may be opaque and not interceptable; client-side web search tools are required for compressible results

• On-prem output may not be byte-identical to cloud due to a different attention kernel, though compression targets are designed to match closely

• Evaluate compression ratio against answer quality on real transcripts, not toy examples

Get started with the quick-start guide or estimate your savings.

2. Prompt Caching

Best for: Stable system prompts, shared tool definitions, and long repeated prefixes that do not change between turns.

Prompt caching gives you a discount on context the provider has already processed. OpenAI’s prompt caching applies automatically for prompts longer than 1,024 tokens when common prefixes repeat. Anthropic offers explicit cache controls with separate pricing for cache writes and cache hits.

Pricing example (Anthropic Claude Sonnet 5): $2/MTok base input, $2.50/MTok 5-minute cache writes, $0.20/MTok cache hits, $10/MTok output. The 5-minute cache write costs 25% more than regular input, but cache hits cost 90% less. Break-even happens after just two uses of the same prefix.

Break-even math:

5-minute cache, 2 uses:
  Cache write: 1.25 units
  Cache hit:   0.10 units
  Total:       1.35 units (vs. 2.0 without cache)
  Savings:     32.5%

For a deeper comparison of when caching beats compression (and vice versa), see our article on prompt caching cost savings.

Tradeoffs:

• Cache invalidation is fragile. Anthropic’s docs show that changes to tool definitions, system prompt content, or message structure can invalidate the cache

• Dynamic content (timestamps, changing retrieved context, randomized JSON key order) breaks prefix matching

• A Reddit user working on autonomous agents noted that exact prefix matches are required and branching can invalidate savings

• Cache TTLs matter: OpenAI’s implementation clears caches after 5 to 10 minutes of inactivity; Anthropic offers optional 1-hour retention at higher write cost

• One-shot prompts with cache writes can actually cost more than no caching at all

Practical cache hygiene:

• Keep dynamic content after the cacheable prefix, never before it

• Avoid timestamps or session IDs in cacheable blocks

• Keep tool schema ordering stable across calls

• Monitor cache hit/miss fields in production responses

Bottom line: Prompt caching is a discount on repeated context, not a license to keep dumping everything into history.

3. Tool-Output Compaction

Best for: SaaS agents calling APIs, web search agents, coding agents reading files, and data agents querying databases.

Tool outputs are often the single largest contributor to conversation history token costs, and they are the one most developers overlook. A JSON response from a CRM API, a web page scraped during research, a database query result, or a file read in a coding agent can easily run 2,000 to 10,000 tokens. Once that result enters the conversation history, it gets reprocessed on every future call.

The pattern that works:

Raw tool output:
  → Store in database/object store
  → Assign handle: tool_result_9831

History message:
  → tool: campaigns.get
  → handle: tool_result_9831
  → key fields: campaign_id, spend, impressions, ROAS, date range
  → summary: "Campaign A spent $1,240 over 7 days; ROAS 2.1; below target."

Reddit commenters recommend treating tool output as ephemeral data, storing raw payloads server-side with handles, and writing only compact summaries into history. One commenter emphasized using stable IDs for entities and date ranges and never resending large JSON unless the model specifically asks for a handle.

Tradeoffs:

• Requires discipline in tool design and a rehydration mechanism if the agent later needs raw data

• Bad summaries can omit fields needed downstream

• Raw data must still exist outside the model for auditability

• If compaction is lossy and the agent must refetch data, savings shrink

Strong recommendation: Tool outputs should be persisted, referenced, and summarized. They should not be appended verbatim to the conversation forever.

4. Sliding Window Trimming

Best for: Short customer-support chats, task-specific sessions, and coding-assistant threads where the user switches tasks frequently.

Sliding windows are the simplest approach: keep the last N turns (or the last N tokens of history), drop the rest. LangChain’s trim_messages utility can reduce chat history to a specified token count while keeping the system message and preserving valid message ordering. LlamaIndex offers ChatMemoryBuffer as a similar approach.

Cost: Free in most frameworks. No external API calls required.

Pluralsight’s developer guide captures the philosophy well: start fresh conversations when switching tasks, summarize only the decision that mattered, and do not treat conversation history as sacred.

Tradeoffs:

• Can drop important commitments, constraints, or facts from earlier turns

• Dangerous for regulated or audit-sensitive workflows if dropped context is not stored elsewhere

• Tool-call history needs careful structural trimming: tool messages usually need to remain paired with the assistant tool call that produced them

• Poor fit for long-lived personalization or multi-session continuity

Sliding windows are cheap and useful, but they are a forgetting strategy, not a memory strategy.

5. Rolling Summarization

Best for: Long single-session workflows, support agents that need continuity but not verbatim history, and coding sessions where previous decisions matter.

Instead of keeping all old turns or dropping them entirely, you can summarize them. Keep the last 4 to 6 turns verbatim for immediate coherence, then replace everything older with a structured summary block. Regenerate the summary every few turns.

OpenAI’s Codex agent used a version of this in production: the agent generated a summary as the new input for subsequent turns, showing that summarization is a real production pattern, not just a blog tactic.

Recommended summary schema:

[Goal]
[Current state]
[Decisions made]
[Constraints]
[Entities and IDs]
[Open questions]
[Tool-result handles]
[Do not forget]

Cost: Extra LLM calls for summarization. MindStudio recommends compressing history every 5 to 10 turns and notes that the summarization call only makes economic sense above a threshold, such as when accumulated output exceeds 1,000 tokens.

Tradeoffs:

• Summaries are lossy. Repeated summarization can compound errors (“summary drift”)

• Important exact wording, like a user’s specific constraint or a regulatory requirement, may be lost

• One Reddit practitioner warned that mid-task summarization did not help their team because quality loss appeared downstream and cost more to fix in retries

• Evaluate summarization on real transcripts, not toy examples

6. Persistent Memory

Best for: Cross-session personalization, CRM/sales assistants, customer support across sessions, and agents that need durable user preferences or project facts.

Persistent memory systems extract durable facts from conversations (preferences, decisions, entities) and store them in a database. On each new turn or session, the system retrieves only the relevant memories instead of replaying the entire transcript.

Mem0 illustrates the math simply: a 10-turn session with 50,000 accumulated context tokens creates 500,000 input tokens across the session. Retrieved memory at 3,000 tokens per turn creates 30,000 input tokens. That is a 94% reduction for repeated context.

Pricing: Mem0 ranges from free (Hobby) to $249/month (Pro). Zep offers 10,000 free credits/month for prototyping, with paid plans starting at $104/month billed annually.

Tradeoffs:

• Memory extraction can miss nuance or create contradictions

• Privacy and deletion requirements are real, especially for healthcare and finance

• Over-broad memory can pollute future prompts with irrelevant facts

• A Reddit commenter argued the real question is not “how much to store” but “what to store and when to forget”

• Not ideal for one-off long documents or raw tool payloads

Persistent memory is for durable facts. It should not be a fancy way to replay every chat message.

7. External State and Retrieval

Best for: Agents with large logs, RAG over static documents, workflows with repeated user intents, and tool-heavy workflows where raw payloads are too large for history.

The principle is simple: your database is cheaper, more reliable, and more auditable than your prompt. Move raw logs, intermediate state, action history, and large documents out of the prompt. Retrieve only the rows, chunks, or handles relevant to the current step.

For repeated questions, semantic caching can avoid LLM calls entirely by storing responses with vector embeddings and returning pre-generated answers when a new query is similar enough. Redis positions this as useful for customer support and FAQ-style interactions.

For teams already working with retrieval pipelines, our RAG compression guide shows how to compress retrieved chunks before they hit the model.

Tradeoffs:

• Retrieval quality becomes a new failure mode

• Semantic cache thresholds can return stale or wrong answers for unique queries

• More moving parts in your architecture

• Requires observability to know what context was actually retrieved versus what was relevant

8. Model Routing and Thinking Budgets

Best for: High-volume production apps with mixed query complexity, agent workflows with extraction/formatting/classification steps, and cost-sensitive SaaS products.

Not every message needs a frontier model processing 20,000 tokens of context. A Reddit practitioner said that “boring” routing moved the needle most: mechanical work went to small models, frontier models were reserved for planning and review, and subagents returned conclusions rather than raw dumps.

Another practitioner reported a 38% token reduction without changing models by adding an intent classifier that gave simple queries a trimmed 4k context and complex queries the full retrieval stack.

Key tactics:

• Intent classifier before the main model call

• Route simple lookups, formatting, and extraction to smaller, cheaper models

• Cap output length and reasoning/thinking budgets where supported

• Use deterministic code for tasks that do not need an LLM at all

For a deeper dive on when routing beats compression (and when to combine them), see our guide on context compression and model routing.

Tradeoffs:

• Misrouting can cause wrong answers, retries, or escalation to expensive models

• Requires evaluation infrastructure

• Savings may come from lower price per token, not fewer tokens overall

9. Observability and Token Budgets

Best for: Any production LLM app, multi-tenant SaaS, and teams deciding whether compression, caching, or memory actually works.

You cannot optimize conversation history token costs from your monthly invoice. You need per-turn and per-task traces.

What to track:

• Input tokens, output tokens, cached tokens, and cache-write tokens per call

• Cost by model, user, feature, and task

• Tool-call count and retry count

• Context length by turn number

• Cache hit rate

• Compression ratio (if using compression)

• Cost per successful task, not just cost per request

Pricing: LangSmith starts free (5,000 traces/month), with paid plans at $39/seat. Helicone offers a free tier with 10,000 requests. LiteLLM provides spend tracking, budgets, and retry logic across 100+ LLMs.

Tradeoffs:

• Observability does not reduce cost by itself, but it enables every other optimization

• Trace storage becomes a bill at scale

• Sensitive prompts may require self-hosting or redaction

• Dashboards are useless unless tied to budgets and owners

A Reddit practitioner working on agent automation said cost visibility becomes mandatory as agents grow more autonomous, because retries, subagents, tool calls, and failure loops all accumulate silently.

Tradeoff Matrix: Cost, Latency, and Quality Impact

Not every token-reduction method is free of trade-offs. Some reduce token billing at the expense of added API latency or risk of losing critical nuances.

How to Choose the Right Strategy

The right method depends on where your tokens actually go. Use this decision tree:

If most repeated tokens are stable instructions or tool schemas: Start with prompt caching.

If most repeated tokens are long chat history, RAG docs, web results, or tool outputs: Use query-aware compression and tool-output compaction.

If the conversation changes tasks often: Start fresh sessions and carry forward only a task summary.

If the agent needs user facts across sessions: Use persistent memory.

If the agent carries logs or raw API payloads: Move them out of the prompt.

If simple requests get the full expensive stack: Add routing and smaller models.

If you do not know where tokens go: Add observability first.

Most production systems will use three or four of these methods together. They are complementary, not interchangeable.

Recommended Implementation Order

Do not try to implement everything at once. This sequence moves from highest-visibility to highest-effort:

1. Instrument token usage per turn. You cannot fix what you cannot see.

2. Separate your context into the five types. Know what is stable, what is recent, what is retrieved, what is tool output, and what is durable memory.

3. Cache stable prefixes. Quick win for system prompts and tool schemas.

4. Compress large context before model calls. Addresses the biggest driver for most agents.

5. Compact tool outputs and store raw payloads externally. Stops the tool-output tail tax.

6. Trim or summarize old short-term history. Use sliding windows for short chats, rolling summaries for longer ones.

7. Add persistent memory for durable facts. Only for cross-session continuity, not transcript replay.

8. Add routing and budgets. Reserve frontier models for tasks that need them.

9. Measure cost per successful task. The real metric is not tokens per request. It is what you paid for a correct, complete outcome.

For a complete walkthrough of this sequence, our AI cost optimization checklist covers each step with implementation details.

The Bottom Line

If your agent is expensive, do not start by shaving words off the system prompt. First, find out what it rereads. Cache stable prefixes. Compress large query-relevant context. Keep raw tool outputs and logs outside the prompt. Retrieve durable memory selectively. Trim or summarize only when the risk is acceptable. Then measure cost per successful task, not just tokens per request.

For teams whose biggest cost driver is long chat history, RAG documents, or tool outputs, test query-aware compression before rebuilding the entire agent stack. At $0.10 per million tokens compressed with $10 in free credits, the break-even point is almost immediate for any workload sending more than a few thousand tokens per call.

FAQ

Do I have to send the whole conversation history every time?

Not manually, but the model needs enough context to answer coherently. If full history is included as input, expect those tokens to affect billing. Provider-managed state (thread IDs, conversation state) simplifies your code but does not automatically eliminate prior-context costs. Cached tokens may get a discount depending on the provider.

Does prompt caching solve conversation history token costs?

Only partially. Caching helps when the repeated content is stable and appears at the beginning of the prompt. It is weaker for dynamic chat history, changing tool outputs, and branchy agent workflows where the prefix shifts between calls. Use caching for system prompts and tool schemas. Use compression or trimming for the dynamic parts.

Is summarization better than truncation?

They serve different purposes. Summarization preserves a rough state of the conversation at the cost of extra LLM calls and potential information loss. Truncation (sliding windows) drops old turns entirely, which is cheaper but riskier for long-running tasks. Many production systems combine both: recent turns verbatim, older turns summarized, and very old turns dropped but stored externally.

Is persistent memory better than compression?

They solve different problems. Persistent memory is best for durable facts (user preferences, decisions, entities) that survive across sessions. Compression is better for shrinking large, single-use context (RAG chunks, tool outputs, long chat history) relevant to a specific query. Most serious agent architectures use both.

Can a larger context window reduce costs?

No. A larger window lets you fit more tokens into a single call. It does not reduce the number of tokens billed. It can also hurt accuracy: the “Lost in the Middle” paper found that models can underperform when relevant information sits in the middle of very long inputs.

Why did my agent costs spike even though each user message is short?

Because the expensive part is not the user’s message. It is the system prompt, tool schemas, tool results, retrieved documents, assistant replies, and prior turns being re-sent as input. In a 20-step agent loop with a 6,000-token base prompt and 800 new tokens per step, cumulative input can reach 272,000 tokens. Each individual step looks modest; the cumulative effect is not.

What is the single most impactful change for reducing conversation history token costs?

There is no single magic trick. Practitioners on Reddit consistently report that the combination of routing, context discipline, and tool-output compaction moved the needle more than any one optimizer tool. If forced to pick one starting point, instrument your token usage first so you know which context type is actually the biggest cost driver. Then apply the right method for that specific type.

Should I use a compression tool or build my own summarization pipeline?

Building summarization pipelines works but adds LLM call costs, introduces summary drift, and requires evaluation infrastructure. Query-aware compression APIs like Compresr operate at the token level without generating new text, which avoids the compounding-error problem of repeated summarization. For most teams, starting with an API is faster than building and maintaining a custom pipeline.

Do I have to send the whole conversation history every time?

Yes. Standard chat completion APIs are stateless. The model requires prior messages passed as input tokens on every turn to maintain context.

Does prompt caching solve conversation history token costs?

Only partially. Caching reduces costs for static prefixes like system prompts, but dynamic turn-by-turn chat history invalidates simple prefix caches.

What is the tool output tail tax in AI agents?

The tool output tail tax occurs when large JSON or API outputs added early in an agent run get reprocessed as input tokens on every subsequent turn.