How to Reduce LLM API Costs: Why Output Compression Beats Prompt Compression in Agent Loops
Reduce LLM API costs in agent loops by targeting output tokens, not prompts. Learn why output compression cuts compounding costs across every turn.
TL;DR: In agent loops, the most effective way to reduce LLM API costs is to compress model outputs, not prompts. Verbose responses accumulate as context and are re-injected at every subsequent step, compounding cost across the full task. Prompt compression saves tokens once on the cheaper input side. Output compression saves tokens that multiply across every turn, making it the dominant lever for production agent workflows.
Key Takeaways
- In agent loops, verbose model responses feed back as context into every subsequent call, compounding cost across the entire task rather than just a single turn.
- Cost per completed task (CPCT), an author-defined metric used throughout this guide, is the only measure that captures multi-turn compounding. Per-call metrics hide the problem entirely.
- Compressing your system prompt without controlling output length can trigger compensatory verbosity, which at the output price premium can cost more than the input saving recovered.
- Output compression through schema enforcement, explicit token budgets, typed response contracts, and post-processing routing is the primary lever for reducing LLM API costs in agent loops.
- Prompt compression tools target the cheaper input tier and are most useful after output length is already under control, not as a substitute for it.
- Measure CPCT before and after every intervention. Never assume a reduction in input token count equals a reduction in total API spend.
Why does output token pricing make prompt compression the wrong default for agent loops?
Prompt compression is the wrong default for agent loops because it optimizes the cheaper side of a structurally asymmetric pricing equation while leaving the more expensive side unconstrained. Across frontier models as of mid-2026, output tokens carry a per-token price premium over input tokens. That asymmetry is the structural fact that makes input-side optimization an unreliable primary strategy.
Most teams reach for prompt compression first, and nearly every published guide in 2026 reinforces that instinct. The problem is that in multi-step agent loops, compressing the prompt can trigger compensatory verbosity. When a compressed prompt strips context the model relies on to calibrate its response, the model may compensate by hedging, restating assumptions, and explaining its reasoning inline. A meaningful prompt reduction that produces a proportionally larger output increase can cost more than it saved, once the output price premium is applied. And in agent loops, that verbose output does not disappear after the call. It feeds back as input to the next step.
How do agent loops turn a single verbose response into a compounding cost problem?
In a multi-step agent loop, every verbose response becomes part of the next call's input, so output length inflates cost at every subsequent turn, not just once.
In a multi-step tool-calling loop, Step 1's output becomes Step 2's context, Step 2's becomes Step 3's. Surplus tokens from early steps accumulate as carried context throughout the task. Verbose output is a multiplier that grows with task depth. A 300-token surplus at Step 1 of a 10-step loop is not a 300-token problem. It is a problem that propagates, at the input price, through every step that follows.
The pattern stays invisible at the per-call level. Per-call spend can drop while the monthly bill climbs, and an observability dashboard reporting per-call token counts will report that the optimization is working. That is what makes this failure mode easy to miss and costly to diagnose after the fact.
What is the correct metric and method for auditing LLM agent costs?
Cost per completed task (CPCT) is the only metric that captures agent loop economics fully. CPCT is defined in this guide as total API spend, covering all steps from the first call to the final output, for one end-to-end task completion.
The CPCT Audit Protocol
CPCT = Σ(input_tokens_n × input_price + output_tokens_n × output_price)
...summed across all N steps required to complete one task. Run this four-step audit before modifying any prompt or compression setting:
- Log full token counts (input and output separately) at every step in the loop.
- Tag logs by task ID so all calls belonging to a single task completion can be aggregated together.
- Calculate CPCT across a representative sample of completed tasks to establish a reliable baseline.
- Re-measure CPCT before and after any compression intervention. Never optimize blind.
Observability tools can surface per-call token data, but CPCT rollup typically requires custom aggregation on top of their exports.
Table 1: LLM Cost Metric Comparison
| Metric | What it measures | What it misses | When it is useful |
|---|---|---|---|
| Tokens per call | Single-call token volume | Multi-turn compounding | Single-inference workloads |
| Cost per call | Single-call dollar spend | Agent loop amplification | Batch jobs with fixed structure |
| Cost per completed task (CPCT) | Full end-to-end task cost | Nothing - this is the target metric | Production agent loops |
| Input token reduction % | Prompt compression effectiveness | Output token inflation from verbosity | Input-heavy single-turn use cases |
How do you implement output compression to actually reduce LLM API costs?
Output compression, defined in this guide as constraining what the model returns through format enforcement, length budgets, and schema-bound responses, is the single highest-impact technique for reducing costs in production agent loops. Four implementation approaches worth testing:
- Schema-bound output: Force JSON or structured schemas that eliminate prose wrapping. A model returning
{"status": "done", "result": "X"}structurally cannot pad with explanatory text. - Explicit token budget instructions: Include a token ceiling in the system prompt and enforce it with the
max_tokensparameter simultaneously. The instruction alone is not enough. The parameter is the hard stop. - Role-specific response contracts: Define what each agent step may return. A tool-calling step that only needs to emit a function call should never return prose. Treat each step's output specification as a typed interface.
- Output audit step: Route a lightweight post-processing step to a smaller, cheaper model that strips surplus tokens before they enter the next context window. The routing cost is low. The downstream savings compound.
Prompt compression tools target the cheaper input tier. In the author's view, they are most useful after output length is already under control, not as a substitute for it.
Frequently Asked Questions
Why does compressing my system prompt sometimes increase my total API bill?
Stripping context from your system prompt can cause the model to compensate with more verbose output: restating assumptions, hedging conclusions, and explaining reasoning that a fuller prompt would have made implicit. Because output tokens carry a per-token price premium over input tokens (as of mid-2026), that inflation can more than offset the input saving. In agent loops the effect compounds because that verbose output is re-injected as context at every subsequent step.
What is cost per completed task, and why does it matter more than tokens per call?
CPCT, as defined in the CPCT Audit Protocol in this guide, is total API spend across all steps required to finish one agent task. Per-call metrics hide multi-turn compounding: a change that reduces per-call token counts can simultaneously increase the real monthly bill by inducing output verbosity that propagates through the loop. CPCT is the only metric that makes this dynamic visible.
Do prompt compression tools actually reduce LLM costs in agent workflows?
They reduce input token counts, which is the cheaper side of the pricing equation. If the compressed prompt induces more verbose responses, the output cost increase can exceed the input saving, and in agent loops that surplus output carries forward into every subsequent step. Measure CPCT before and after. Never assume that a reduction in input token count equals a reduction in total API spend.
Conclusion
Compressing prompts optimizes the cheaper half of a structurally asymmetric pricing equation. In agent loops, that approach is not neutral. It can be actively counterproductive if it induces output verbosity that compounds across turns. The recommended priority stack, as a practical rule of thumb from this guide:
- Implement CPCT logging before touching any prompt or compression setting.
- Identify the worst-offending agent steps by output token volume.
- Apply output compression: schema enforcement, explicit token budgets, typed response contracts, and post-processing routing.
- Re-measure CPCT to confirm the direction of change.
- Only then apply input-side techniques to whatever cost remains.
The published guides promising large cost reductions through prompt trimming are optimizing the receipt, not the spend. Instrument your next agent task completions with CPCT logging. The number you get back will tell you immediately whether output or input is your real cost driver.
Learn from me
Agent Engineering Bootcamp: Developers Edition, my Maven cohort. Advanced agentic RAG, multi-agent orchestration, memory, evals, and guardrails. Take agents from prototype to production. Join the next cohort →
Hire us
Traversaal.ai. We're a team of forward deployed engineers solving the toughest AI problems for Fortune 100 companies: document intelligence, agentic data platforms, and real-time web intelligence, deployed in production. Work with our team to deploy your next agentic ecosystem. Talk to Traversaal.ai →
Join us
Want to solve these problems with us? We're always looking for forward deployed engineers who want to ship production AI. jobs@traversaal.ai
