Input Token Costs vs Output Token Costs: 2026 Guide

TL;DR

Output tokens cost 2 to 6 times more than input tokens per million across every major LLM provider because generating text sequentially is far more GPU-intensive than reading a prompt in parallel. But here’s the catch: input tokens often dominate your total bill because multi-turn conversations and RAG workloads compound input volume with every API call. Understanding both sides of this equation, plus hidden costs from reasoning tokens, is the key to controlling LLM spend.

Every LLM API pricing page splits costs into two line items: input tokens and output tokens. If you’ve looked at your bill and wondered why these are priced differently, or which one is actually eating your budget, you’re asking the right question. The gap between input token costs and output token costs reflects real differences in how GPUs process language, and misunderstanding it leads teams to optimize the wrong thing.

This guide breaks down what each token type is, why the pricing gap exists, and the counterintuitive reality that the cheaper token type often generates the bigger bill. If you’re already looking to cut LLM costs, explore compression pricing to see how it compares to what you’re paying per million input tokens today.

Key Takeaway: Why Output Tokens Cost More Than Input Tokens

Output tokens cost 2x to 6x more than input tokens because of core GPU processing mechanics:

• Input Tokens (Prefill Phase): Processed simultaneously in parallel. This phase is compute-bound, resulting in high GPU hardware utilization (80%+).

• Output Tokens (Decode Phase): Generated sequentially, one token at a time. Each new token requires reading an expanding Key-Value (KV) cache. This phase is memory-bandwidth-bound, reducing GPU utilization to 20%–40%.

The Spend Paradox: Despite lower per-unit prices, input tokens frequently generate 70%–90% of total API bills in RAG pipelines and multi-turn chatbots because full conversation histories and document contexts must be resent on every request.

What Are Input Tokens?

Input tokens are everything you send to the model before it starts generating a response. That includes your system prompt, user message, conversation history, retrieved documents (in RAG), tool definitions, and function call results. If you’re unfamiliar with how text gets split into tokens, think of them as word fragments, roughly 0.75 words per token for English text.

From a technical standpoint, the model processes your input during what’s called the “prefill” phase. The GPU reads all input tokens in parallel in a single forward pass. This is efficient. The hardware is doing what it was designed for: crunching a large matrix operation in one shot. Arithmetic intensity during prefill runs between 200 and 400 operations per byte, which keeps GPU utilization high.

That efficiency is why input tokens are cheaper. As of 2026, input token pricing ranges from $0.10 per million tokens (GPT-4.1 Nano) up to about $5 per million for frontier models.

What Are Output Tokens?

Output tokens are everything the model generates back to you: the answer, the code block, the JSON object, the summary. Unlike input processing, generation happens during the “decode” phase, and it’s fundamentally different.

The model produces output one token at a time. Each new token requires a full forward pass through the model, plus reading a growing key-value cache that expands with every token generated. This is sequential, memory-bandwidth-bound work. GPU utilization drops to 20 to 40%, and arithmetic intensity falls to 60 to 80 operations per byte, a fraction of what prefill achieves.

Output token prices reflect this inefficiency. They range from $0.40 per million tokens on budget models up to $60 per million for reasoning-heavy models like o3. The ratio holds: output tokens cost 2 to 6 times more than input tokens across providers.

There’s also a third category that surprises many teams. Reasoning tokens, generated by models like OpenAI’s o3, o4-mini, and Anthropic’s extended thinking, are billed at output token rates. OpenAI’s reasoning tokens are hidden from the developer entirely. A simple question might produce 500 visible output tokens but consume 3,000 internal reasoning tokens you never see. Your actual cost is 7 times what you’d estimate from the visible output alone.

Why Output Tokens Cost More Than Input Tokens

The simplest analogy: reading a book is fast because your eyes scan many words at once. Writing a book is slow because each word depends on the one before it.

That’s the difference between prefill and decode at the GPU level. Prefill is compute-bound and parallelizable. Decode is memory-bandwidth-bound and sequential. Research from the Sarathi paper found that at batch size 1, decode can cost as much as 200 times more per token than prefill in raw GPU time.

Technical Comparison: Prefill (Input) vs. Decode (Output)

2026 Model Pricing Comparison (per 1 Million Tokens)

Note: For models supporting prompt caching, hit rates reduce input costs significantly, bringing the effective ratio closer to 1:1 on repetitive prompts.

Newer entrants like DeepSeek compress the ratio closer to 1.6×, but the pattern is universal: output always costs more per token. The overall LLM API price range spans a 600× spread from the cheapest input tier to the most expensive output tier.

The Input Token Paradox: Why Cheaper Tokens Generate Bigger Bills

Here’s where most pricing guides stop, and where reality gets interesting.

Despite costing less per unit, input tokens frequently dominate total LLM spend. The FinOps Foundation documented this directly: while individual output tokens are priced around 300% more than input tokens, the compounding volume of input tokens means they will almost always dominate total spend in any conversational application, including conversations between LLMs in agentic systems.

Why does input volume compound so aggressively? LLM APIs are stateless. To maintain a coherent multi-turn conversation, you must resend the entire conversation history with every new message. Imagine repeating your entire conversation from the beginning every time you want to add a single sentence. That’s exactly what happens inside your API calls.

A 10-turn chat doesn’t cost 10 times a single turn. It costs roughly 55 times, because turns 1 through 10 are all resent as input on the final call. This phenomenon, sometimes called context rot, creates runaway input costs that dwarf the per-token savings.

Real-World Cost Breakdown: 10-Turn Multi-Turn Session

The table below illustrates how re-sending context history in a multi-turn conversation causes input costs to compound rapidly compared to output costs (assumes GPT-4.1 pricing: $2.00/1M input, $8.00/1M output).

Takeaway: By Turn 10, input tokens represent 82% of the session cost, even though output tokens cost 4× more per unit.

The problem gets worse with agents. Gartner’s March 2026 analysis found that agentic models require between 5 and 30 times more tokens per task than a standard chatbot. Every tool call, every intermediate result, every planning step gets packed into the input context.

Practitioners on Reddit and engineering forums consistently report that their RAG systems generate surprisingly small outputs (a paragraph or two) while consuming tens of thousands of input tokens per query from retrieved documents. The input-to-output ratio in a typical RAG pipeline can easily hit 50:1 or 100:1.

For a deeper look at how these costs add up across different workloads, the LLM API cost optimization guide walks through the math in detail.

The Reasoning Token Trap

Reasoning models have introduced a hidden multiplier that makes the input-vs-output cost picture even more complex.

When you use OpenAI’s o3 or o4-mini, the model generates internal “thinking” tokens that are billed at output rates but never shown to the developer. A single complex query can generate 10,000 thinking tokens. On a frontier model at $30 per million output tokens, those invisible tokens cost $0.30, while the visible answer might cost $0.006. You just paid 50 times more for the thinking than the answer.

Anthropic handles this differently. Their extended thinking tokens are visible to the developer (returned as thinking blocks), but they’re still billed at output rates.

Either way, reasoning tokens make output costs spike unpredictably. One LinkedIn practitioner noted that teams often discover this only after their first month of o3 usage, when the bill comes in at 5 to 7 times what they budgeted based on visible token counts.

The practical takeaway: if you’re using reasoning models, track total billed output tokens (not just visible completion tokens) and set thinking budgets explicitly through the API parameters.

How to Reduce Input Token Costs

Since input tokens often dominate total spend, this is where many teams should start optimizing.

Prompt compression. Query-aware compression removes low-information tokens from LLM inputs before sending them, reducing token count by 20 to 80% while preserving the semantic content the model needs to answer accurately. Unlike naive truncation, query-aware approaches keep only the spans relevant to the specific question being asked.

Prompt caching. If your system prompt or common document prefix stays stable across calls, prompt caching at providers like Anthropic and OpenAI gives up to 90% off those cached input tokens. Caching and compression complement each other: cache the stable parts, compress the variable parts.

Smarter retrieval. In RAG workloads, send only the most relevant chunks to the model, not full documents. A well-tuned RAG pipeline with compression applied to retrieved results can cut input tokens dramatically without hurting answer quality.

Context window management. For multi-turn applications, summarize old conversation turns instead of carrying raw history forward. Prune tool call results once the model has used them.

One critical warning: naive compression can backfire. The CAVEWOMAN paper (arXiv, 2026) found that simply stripping function words from prompts saves a small number of input tokens, but the model compensates by generating longer, more verbose output. Since output tokens cost several times more, the net cost actually increases by about 15%. This is exactly why query-aware compression, which preserves the information density the model needs, matters more than blunt truncation.

Try compressing a real prompt to see how much reduction is possible on your actual workload.

How to Reduce Output Token Costs

When your workload is output-heavy (code generation, long-form writing, data extraction), these strategies target the expensive side of the bill.

Set max_tokens explicitly. Don’t let the model ramble. If you need a 200-word answer, cap the response. For structured output like JSON or classifications, explicit length constraints can reduce output tokens by 50 to 80%.

Request structured output. Asking for JSON instead of prose cuts output length and makes parsing easier. A classification that might take three sentences in natural language takes one line in JSON.

Model routing. Send simple tasks (summarization, classification, reformatting) to cheap, fast models and reserve frontier models for complex reasoning. This approach can cut overall costs by 60 to 80% while maintaining quality where it matters. Several practitioners on AI engineering forums describe running a lightweight classifier that routes queries to the cheapest model capable of handling them.

Batch processing. For workloads that aren’t latency-sensitive (bulk classification, dataset annotation, report generation), batch APIs from OpenAI and Anthropic offer a flat 50% discount.

Control reasoning effort. On reasoning models, set explicit thinking budgets. Not every query needs 10,000 reasoning tokens.

Optimizing by Workload Type

Knowing whether your workload is input-heavy or output-heavy tells you where to focus. Here’s a quick reference:

RAG and document QA. These are input-heavy. You’re injecting thousands of retrieved tokens as context for every query, often generating short answers. Compress retrieved context before it reaches the model, and use prompt caching for stable system prompts. Input token costs will dominate here.

Chatbots and multi-turn assistants. Input compounds every turn because of history re-sending. Compress conversation history aggressively, prune old turns, or summarize them. The longer the average session, the more input costs balloon.

Code generation and creative writing. These are output-heavy. The model produces large volumes of text or code. Cap max_tokens, use model routing for simpler generation tasks, and request structured formats where possible.

Agentic workflows. Both sides explode. Agents generate chains of tool calls (output), then ingest all results as context (input) for the next step. Compress tool outputs between steps, route simple sub-tasks to cheaper models, and prune tool definitions the agent doesn’t need for the current step.

For a complete walkthrough of how to calculate and reduce costs across these patterns, the AI cost per task formula guide provides worked examples with real numbers.

Self-Hosted Models: A Different Cost Structure

One detail that rarely appears in pricing discussions: if you self-host an open model on your own GPUs, there’s no separate input and output line item on a bill. You pay for GPU time regardless.

But the underlying economics are identical. Prefill (input) saturates your GPU efficiently. Decode (output) underutilizes it. The same token generates very different infrastructure costs depending on which phase produced it. Self-hosting doesn’t eliminate the input token cost vs output token cost asymmetry. It just moves it from a per-token API charge to a GPU utilization problem.

Teams evaluating self-hosting should model their expected input-to-output ratio and decode throughput carefully. A workload that’s 95% input (like RAG) will see much better GPU utilization than one that’s 50/50.

The Bigger Picture: Prices Are Falling, But Volume Is Rising

LLM API prices dropped roughly 80% between 2025 and 2026. That’s significant. But enterprise AI API spend still hit $8.4 billion in 2025 and is projected to double again, because usage is growing faster than prices are falling.

The teams that control costs aren’t just chasing cheaper models. They’re reducing token volume at the source, through compression, smarter retrieval, context management, and routing. Understanding the mechanics of input token costs vs output token costs is the foundation for every one of those strategies.

Ready to cut your LLM API spend? Try Query-Aware Compression Free or talk to our engineering team about on-prem options for enterprise workloads.

FAQ

Why are output tokens more expensive than input tokens?

Output tokens require sequential generation, one token at a time, with each token needing a full forward pass through the model. Input tokens are processed in parallel during a single pass. This makes output generation far less GPU-efficient, and providers price accordingly. The ratio is typically 2 to 6 times higher for output.

What are reasoning tokens, and how are they billed?

Reasoning tokens are internal “thinking” tokens generated by models like OpenAI’s o3 and Anthropic’s extended thinking. They’re billed at output token rates. OpenAI’s reasoning tokens are hidden from the developer, while Anthropic’s are visible. Either way, they can multiply your effective output cost by 7 to 50 times what you’d estimate from the visible response alone.

Which costs more in total, input or output tokens?

It depends on your workload. For multi-turn chatbots, RAG systems, and agentic workflows, input tokens typically dominate total spend because conversation history and retrieved documents compound with every call. For code generation or long-form writing, output tokens may dominate. Check your actual input-to-output ratio before deciding where to optimize.

What is prompt compression, and how does it reduce input token costs?

Prompt compression removes low-information tokens from LLM inputs while preserving the meaning needed for accurate responses. Query-aware compression can reduce input token counts by 20 to 80%. It’s especially effective for RAG workloads where large document chunks are injected as context.

Can reducing input tokens accidentally increase output costs?

Yes. Research from the CAVEWOMAN paper found that naive compression (like stripping function words) causes models to compensate with longer, more verbose output. Since output tokens cost more per unit, total cost can actually rise. Query-aware compression avoids this by preserving the information density the model needs, preventing it from “filling in the blanks” with extra output.

How do prompt caching and prompt compression work together?

Prompt caching gives discounts (up to 90%) on input tokens that stay the same across calls, like system prompts. Compression reduces the variable parts, like retrieved documents or conversation history. The two strategies stack. Using both can bring effective per-call cost down to roughly 25% of standard rates.

What’s a good input-to-output token ratio to track?

There’s no universal target, but tracking this ratio reveals your workload profile. RAG systems often run 50:1 or higher (input-heavy). Chatbots start balanced but drift input-heavy over long sessions. Code generation might run 1:3 or 1:5 (output-heavy). Measuring this ratio tells you which side of the bill to optimize first.

Do self-hosted models have different input vs output costs?

Self-hosted models don’t have separate per-token pricing, but the GPU economics are the same. Prefill (input processing) uses GPU compute efficiently. Decode (output generation) underutilizes it. The cost asymmetry shows up as lower throughput and higher GPU-hours-per-token on the output side rather than as a line item on a bill.