Compress JSON for LLMs: The 2026 Guide to Lower Token Costs

TL;DR

Compressing JSON for LLMs means reducing the token count of structured data before it enters a language model. This spans two dimensions: format compression (stripping syntactic overhead like repeated keys, braces, and whitespace) and content compression (removing data irrelevant to the current query). Format tricks save 10–60% depending on data shape, while query-aware content compression can cut 50–90%+ of tokens by keeping only what the LLM actually needs to answer the question.

Key Takeaway: Compressing JSON for LLMs reduces token consumption via two methods:

1. Format Compression (Lossless): Minification, key abbreviation, and converting tabular JSON to alternative formats like TOON or CSV save 10–60% on syntactic overhead (braces, quotes, whitespace).

2. Content Compression (Lossy): Query-aware filtering drops fields irrelevant to the specific prompt, saving 50–90%+ of tokens.

   Combining query-aware content filtering first, followed by format minification, delivers maximum token efficiency and minimizes latency.

What Does “Compress JSON for LLMs” Mean?

Compressing JSON for LLMs is the practice of reducing the token count of JSON data before sending it to a language model. The goal is straightforward: spend fewer tokens on the same information, which lowers cost, reduces latency, and frees up context window space for content that matters.

But “compress” is doing double duty here. Most articles on the topic treat it as a format problem: strip whitespace, shorten keys, maybe convert to a different notation. That covers one dimension. The other dimension, largely missing from the conversation, is content compression: removing the fields, records, and values that are irrelevant to the specific task. The difference matters enormously. Format compression might save 30–60% on syntax. Content compression can remove 80%+ of the payload by dropping data the LLM never needed to see.

This guide covers both.

---

Why JSON Is Token-Expensive

JSON was designed for machines to parse and humans to read. It was not designed for BPE tokenizers to process efficiently. Several mechanics make it uniquely costly when passed to LLMs.

Structural Characters Don’t Compress Well in BPE

BPE tokenizers build their vocabulary by merging frequently co-occurring byte pairs. JSON brackets, colons, URL slashes, and code indentation patterns rarely merge aggressively during training, so those sequences stay expensive in tokens relative to the characters they contain. A single curly brace or quotation mark often consumes an entire token.

Research shows that whitespace and indentation alone account for roughly 24.5% of tokens across programming languages. When you’re outputting or consuming structured formats like JSON, where braces, quotes, and indentation stack on top of actual content, that overhead compounds fast.

Repeated Keys Multiply the Problem

Consider an array of 1,000 sensor readings. Field names like device_id, temperature, and timestamp appear 1,000 times each. Every repetition costs tokens. Add the structural punctuation (braces, brackets, colons, commas) and you’re burning thousands of tokens on information that could be stated once.

A 500-word JSON payload routinely uses 800+ tokens. A kubectl get pods response in JSON runs 2.5–3x more tokens than the same data in a compact format. Studies on agentic and structured benchmarks show that wrapping unstructured text in JSON adds 18% to 30% more tokens on average. JSON doesn’t just fail to compress content; it actively bloats it.

Understanding this overhead is the first step toward reducing input token costs across your LLM pipeline.

Format-Level Compression Techniques

These approaches reduce JSON’s syntactic overhead while preserving all the original data. Think of them as lossless compression for LLM inputs.

Quick Wins (No Dependencies Required)

• Minification: Strip all whitespace, newlines, and indentation. In one test case, an unminified JSON object tokenized to 10 tokens while minification cut it to 6, a 40% reduction. That’s an extreme best case, but 5–20% savings are typical. Models do not need indentation. Indentation exists for human developers.

• Key abbreviation: Replace verbose key names with short aliases and include a mapping header. "device_id" becomes "d", "temperature" becomes "t". When keys repeat across hundreds or thousands of records, this adds up quickly.

• UUID shortening: A UUID like "550e8400-e29b-41d4-a716-446655440000" costs 4–5 tokens. A short alias like "u-1" costs 1. If the LLM doesn’t need the actual UUID to answer the question, replace it.

• Removing nulls and empty values: Fields with null, "", or [] values carry zero information but still cost tokens. Drop them.

These techniques combined typically deliver 10–30% savings with minimal implementation effort.

Alternative Formats: TOON, CSV, and Markdown Tables

For tabular or uniform data, converting away from JSON entirely yields bigger gains.

TOON (Token-Oriented Object Notation) is an alternative designed for LLMs. It states field names once as a header row, then lists values as rows, eliminating the repeated-key problem. Benchmarks show impressive results for uniform data shapes:

• Uniform employee records (100 rows): 60.7% fewer tokens than formatted JSON, 36.8% fewer than minified JSON

• Time-series analytics data: 59% and 35.8% savings respectively

• Model accuracy: Across four major LLMs, TOON used 39.6% fewer tokens while achieving 73.9% accuracy versus JSON’s 69.7%

• Production implementation: Halodoc’s engineering team reported 35–45% token reduction with roughly 95% LLM compatibility across OpenAI, Anthropic, Google, and AWS Bedrock.

CSV and Markdown tables perform similarly for flat tabular data. One benchmark on MCP server responses found TSV and CSV both achieved 55% token reduction, with Markdown tables at 44%.

When Alternative Formats Break Down

TOON’s limitations are real and often omitted from discussions promoting it.

• Nested data is a problem: For deeply nested or non-uniform JSON, TOON can actually cost more tokens than the original. If your data has nested objects, you need to flatten before encoding, which adds complexity and can lose structural information.

• Model compatibility varies wildly: In benchmark evaluations, Mistral-Small-24B’s accuracy dropped from 89% under JSON to 53% under TOON. That’s a severe regression that breaks the model’s ability to reason about the data. Always test with your specific model.

• Small payloads don’t benefit: For payloads under ~500 tokens, the conversion cost likely exceeds the savings.

• TOON is specialized: It works best for high-volume, internally bounded LLM pipelines where both the prompt and the parser are under your control. For a single prompt with a small dataset, JSON with minification is fine.

Format Compression Decision Criteria

The right approach depends on your data shape and the compression ratio you need:

• Small payloads (<500 tokens): Don’t bother. The optimization overhead isn’t worth it.

• Tabular/uniform arrays: TOON or CSV. Expect 30–60% savings.

• Deeply nested structures: Stick with minified JSON.

• Mixed payloads: Minify the JSON and abbreviate keys. Expect 15–30%.

Rule of thumb: Use TOON for high-volume tabular input (40% fewer tokens), JSON with constrained decoding for output (guaranteed valid), and a two-step approach (free reasoning then structured formatting) when accuracy matters.

Content-Level Compression: The Bigger Win

Format compression attacks syntax. Content compression attacks relevance. For most real-world JSON payloads, irrelevant content is the bigger problem.

When an API returns 50 fields but the user’s question only needs 3, format tricks save 30–60% on the syntax wrapping those 50 fields. Query-aware compression can remove 80%+ of the payload by dropping the 47 irrelevant fields entirely.

This matters most in three scenarios:

• Tool outputs in agent workflows: An agent calls an API that returns a massive JSON response. Most of that response is irrelevant to what the agent was actually asked. Passing all of it to the downstream model wastes tokens and hurts accuracy.

• RAG retrieved documents: Retrieved chunks wrapped in JSON metadata often carry fields (timestamps, source IDs, embedding scores) that the LLM doesn’t need for answering the question.

• Chat history: Long conversation histories serialized as JSON contain many exchanges irrelevant to the current turn.

Passing irrelevant data creates context rot: even powerful models have finite capacity, and long streams of irrelevant information degrade their reasoning.

Query-Aware vs. Query-Agnostic Compression

Context compression methods fall into two categories:

• Query-agnostic methods: Exploit redundancy in the text itself, compressing uniformly regardless of the task.

• Query-aware methods: Compress differently depending on the specific question, keeping only the fields and values relevant to the current query.

For JSON payloads, query-specific compression is almost always the better choice. A JSON tool output from a financial API might contain dozens of fields, but if the user asked "What was last quarter's revenue?", only a few fields matter. Query-aware compression identifies and preserves those fields while aggressively cutting everything else.

The typical savings range is 50–80%, with some payloads compressing by 90%+.

Combining Format and Content Compression

Optimal Two-Step Compression Pipeline:

Raw JSON Payload ---> [Step 1: Query-Aware Content Filter] ---> [Step 2: Format Optimization] ---> Token-Efficient Prompt

The maximum compression on JSON payloads comes from layering both approaches:

1. Content compression first: Run the JSON payload through query-aware compression to strip irrelevant fields and values.

2. Format optimization second: If the remaining data is still large (particularly if it’s tabular), apply format-level techniques: minify, abbreviate keys, or convert to TOON/CSV.

Why content first? Format compression operates on every byte of data provided. If you minify a 10,000-token JSON payload and then strip 80% of the content, you wasted effort minifying 8,000 tokens of data that got thrown away. Compress content first, then optimize the format of what remains.

Measuring Results

• Track tokens-per-record (TPR): Measures baseline efficiency independent of payload size. A JSON array of 100 user records might start at 45 TPR, drop to 30 TPR after minification, and drop to 6 TPR after query-aware compression strips irrelevant fields.

• Measure accuracy: Run your evaluation suite after every compression change. Compression is useless if the LLM cannot answer correctly with the compressed input.

For teams managing wasted context across pipelines, combining format and content compression yields the highest return on optimization effort.

Common Mistakes When Compressing JSON for LLMs

• Compressing short payloads: If your JSON is under 500 tokens, the API overhead of a compression step exceeds the savings. Set a minimum-token threshold and skip compression for small inputs.

• Assuming all models handle alternative formats equally: The Mistral accuracy drop (89% to 53%) when switching from JSON to TOON highlights the risks. Test with your specific model and data shape before deploying non-standard formats.

• Trusting format conversion alone when content is the real problem: If your JSON payload is 10,000 tokens because it contains 200 fields from an API response and the user’s question only touches 5 of them, TOON will save ~40% on syntax. Query-aware compression will save 90%+ by removing the 195 irrelevant fields.

• Not measuring accuracy after compression: Run structured evaluations. Some compression approaches improve accuracy (by removing distracting information) while others degrade it (by removing context the model needed).

• Forgetting about output costs: Compressing JSON inputs is half the picture. If you’re asking the LLM to produce JSON output, evaluate whether you need full JSON or if a simpler format suffices for downstream consumers.

When Each Approach Makes Sense

---

FAQ

Does minifying JSON hurt LLM accuracy?

No. Current models process minified JSON just as well as pretty-printed JSON, provided the schema definition is clear in the prompt. Minification is a free win that every pipeline should implement.

Is TOON always better than JSON for LLM prompts?

Not at all. TOON excels at uniform, flat, tabular data where field names repeat across many rows. For deeply nested or non-uniform JSON, TOON can increase token count. Some models (notably smaller ones like Mistral-Small-24B) show significant accuracy drops when switching from JSON to TOON. Always benchmark with your specific model and data.

What’s the difference between format compression and content compression for JSON?

Format compression reduces syntactic overhead (whitespace, repeated keys, braces) while preserving all original data. Content compression removes fields and values that are irrelevant to the current query. Format compression is lossless and capped at roughly 60% savings for ideal data shapes. Content compression is lossy but targeted, yielding 50–90%+ savings by keeping only what the LLM needs to answer the question.

How do I know if my JSON payloads are worth compressing?

Measure the token count. If a payload is under 500 tokens, compression overhead likely exceeds the savings. If you’re sending the same schema repeatedly at high volume, or if your payloads regularly exceed 1,000 tokens, compression will pay for itself quickly. Track tokens-per-record as your baseline metric.

Can gzip or similar byte-level compression help with LLM token costs?

No. Gzip compresses bytes for network transfer, but the LLM still needs to process the decompressed text. LLM costs are based on token count, not byte count. You need to reduce the actual text that reaches the model, not just compress it in transit.

What about using Protocol Buffers or MessagePack instead of JSON?

These are binary formats that reduce network payload size, but LLMs cannot read binary data. You must serialize back to text before sending to the LLM, which defeats the purpose. The compression needs to happen at the text/token level, not the wire protocol level.

How does compressing JSON for LLMs affect latency?

Fewer input tokens means less prefill time. LLM inference latency scales with input length, so a 50% reduction in tokens often translates to a measurable latency improvement, particularly on longer inputs. One illustrative example showed a 24% latency reduction alongside an 86% cost decrease on a financial filing.

Should I compress JSON inputs or outputs first?

Start with inputs. Input tokens are where the waste is largest, especially for tool outputs, RAG contexts, and API responses. Output compression is a secondary optimization that matters most when output token costs are a significant share of your bill.