AI API Pricing Decoded: Input Tokens, Output Tokens, and the Bill You Didn’t Expect
Everyone modelling AI API costs for the first time makes the same estimate and gets the same surprise. The estimate is reasonable. The bill is three to five times higher.
The gap is not caused by hidden fees. It is caused by five mechanics of token billing that are all publicly documented and all easy to miss. Once you understand them the arithmetic becomes predictable, and most of the overage becomes avoidable.
The base structure
AI APIs bill per token, with input and output priced separately. Output is typically three to five times more expensive than input.
Representative rates for a mainstream mid-tier model:
- Input: $3.00 per million tokens
- Output: $15.00 per million tokens
A token is roughly ¾ of an English word. So a million input tokens is about 750,000 words — a substantial pile of text for three dollars.
Which is exactly why the first estimate is always too low. At those numbers, AI looks nearly free. The five mechanics below are what turn nearly-free into a real line item.
Mechanic 1: Conversations re-bill their entire history
This is the big one, and it is the one that surprises everyone.
Models are stateless. Every turn in a conversation resends the whole prior exchange as input. A twenty-turn conversation does not bill twenty messages — it bills the cumulative context, twenty times.
The growth is quadratic, not linear:
| Turn | Context sent | Cumulative input tokens |
|---|---|---|
| 1 | 500 | 500 |
| 5 | ~4,500 | ~12,500 |
| 10 | ~9,500 | ~50,000 |
| 20 | ~19,500 | ~200,000 |
A twenty-turn conversation consumes roughly 200,000 input tokens rather than the 20,000 a naive model predicts. Ten times the estimate, from a mechanic nobody mentions in the quickstart.
The fix: cap conversation length, summarise and restart long threads, and trim history to what is actually needed rather than sending everything by default.
Mechanic 2: Documents are input tokens, and they are large
Attaching a file means sending its full text as input, every turn it remains in context.
| Document | Approximate input tokens | Cost per send at $3/M |
|---|---|---|
| 20-page report | ~13,000 | $0.04 |
| 300-page book | ~150,000 | $0.45 |
| 500-line code file | ~8,000 | $0.02 |
| Hour-long transcript | ~10,000 | $0.03 |
Individually trivial. The problem is repetition: a 150,000-token document held across a ten-turn conversation is 1.5 million input tokens, or about $4.50 for one session. Run that workflow four hundred times a month and you have a $1,800 line item from a task that felt like it cost nothing.
The fix: extract the relevant sections before sending, use retrieval instead of full-document context for repeated queries, and drop documents from context once you are done with them.
Mechanic 3: Reasoning tokens are billed as output
Models that “think before answering” generate intermediate reasoning. On most providers those tokens are billed at output rates, even when they are not shown to you.
This can multiply the output cost of a single response several times over. A response that displays 500 tokens may have generated several thousand internally.
The fix: use reasoning modes deliberately, for problems that genuinely need them. Routing simple tasks to a reasoning model is the single most common source of unexplained cost.
Mechanic 4: System prompts are billed every single call
Your system prompt — instructions, examples, formatting rules, tone guidance — is input on every request.
A well-engineered 2,000-token system prompt across 100,000 monthly calls is 200 million input tokens. At $3 per million that is $600 a month, before any user content at all.
The fix: two things. Trim the system prompt ruthlessly; most are twice as long as they need to be. Then use prompt caching if your provider offers it — cached prefix tokens are typically billed at a large discount, often 90% off, and system prompts are the ideal cache candidate because they are identical every call.
Mechanic 5: Retries and failures bill in full
Failed requests consume tokens. Timeouts, malformed outputs that trigger a retry, and validation failures all bill.
A pipeline with a 15% retry rate is paying 15% more than its nominal cost, and retry rates climb quietly as prompts drift and edge cases accumulate.
The fix: log the retry rate as a first-class metric. If it exceeds a few percent, fix the prompt rather than paying the surcharge indefinitely.
The four levers that actually reduce cost
In order of impact for most workloads:
1. Route by tier. Model prices vary by more than 10× across a provider’s own range. Sending classification and formatting work to a frontier model is the largest avoidable cost in most pipelines. Establish the cheapest model that passes your accuracy bar and route bulk work there.
2. Cache aggressively. Prompt caching on system prompts and stable context is close to free money. Structure prompts so the stable part comes first and is cacheable.
3. Cap output length. Output costs several times input. Explicit length limits in the prompt, plus a hard max_tokens, prevent the model from producing a thousand tokens where two hundred would do.
4. Use batch endpoints for anything non-urgent. Providers commonly discount asynchronous batch processing by around 50%. If your job can tolerate a delay, this is a straight halving.
Comparing across providers
Rates vary widely between labs and between tiers within a lab, and they change often enough that any figure you memorise will be stale within a quarter. Compare against a maintained reference — a current AI pricing guide that lists per-model input and output rates side by side is more reliable than a vendor’s own page, because it puts the alternatives next to each other.
When comparing, check the ratio as well as the absolute rate. A model with cheap input and expensive output suits analysis workloads, where you send a lot and receive a little. A model with the opposite profile suits generation workloads. The cheapest model on paper is frequently not the cheapest for your actual shape of work.
When the API is the wrong choice
For interactive human use, subscriptions usually beat API access on cost and always beat it on effort. A person chatting all day generates enough tokens that a flat plan is cheaper, and the API gives you nothing you want — you would have to build chat history, file upload, and an interface yourself.
The comparison worth running is your realistic monthly volume priced both ways. An AI subscription calculator gives you the subscription side quickly by letting you select the plans you would otherwise buy, and comparing that total against your modelled API spend usually settles the question in a couple of minutes.
The rough rule: API for programmatic and high-volume automated work, subscriptions for humans typing. Teams that get this backwards either build a chat interface they did not need or pay per-seat rates for a batch pipeline.
Frequently asked questions
Why is AI output more expensive than input? Output is generated one token at a time and cannot be parallelised the way input processing can. The pricing reflects a real compute asymmetry rather than a margin decision.
What is prompt caching? Reusing a previously processed prompt prefix instead of reprocessing it. Cached tokens are billed at a steep discount, often around 90% off. Best applied to system prompts and other stable context repeated across calls.
How do I estimate AI API costs before building? Take one representative request, count input and output tokens, multiply by the published rates, then multiply by realistic monthly volume — and then multiply by two to three for conversation history, retries, and growth. The final multiplier is where most estimates fail.
Are batch APIs always cheaper? Where offered, yes, typically around half price, at the cost of latency. Suitable for anything that does not need an immediate response.
Is the API cheaper than a subscription? For low-volume programmatic use, usually. For a person using AI all day, almost never. Model both against your realistic volume rather than assuming.
The estimate that actually holds
Take your naive calculation and multiply by three. That accounts for conversation history, system prompt repetition, retries, and the growth that always happens once people find the tool useful.
Then apply the four levers — route by tier, cache, cap output, batch what can wait — and you will typically get most of that multiplier back. The mechanics are all documented. They are just documented somewhere other than the pricing page.
