AI running costs tend to creep rather than spike. A feature ships, usage grows, and the monthly bill grows with it until somebody notices. When we review these, the same handful of savings are available almost every time.
1. Route by difficulty
Most workloads have a difficulty distribution. A large share of requests are straightforward, a small share are genuinely hard. Sending everything to the largest available model means paying the hard case price for the easy majority.
A classifier or a simple heuristic in front, routing easy cases to a smaller model and escalating on low confidence, is usually the single largest saving available. The measurement that matters is whether quality holds on the routed cases, which is what the eval set is for.
2. Cache aggressively
Identical or near identical requests recur far more than people expect, especially in support and internal knowledge use cases. An exact match cache is trivial. A semantic cache keyed on embedding similarity catches considerably more, at the cost of needing a similarity threshold you have to tune.
3. Use prompt caching where the provider offers it
If your prompts share a large static prefix, which most production prompts do, provider side prompt caching reduces the cost of that prefix substantially. It requires structuring the prompt so the static part comes first, which is a small refactor with a large payoff.
4. Stop retrieving more context than the model uses
Retrieval systems that pass twenty chunks into the context window are paying for twenty chunks. Rerank and pass four. This usually improves answer quality at the same time, which makes it the rare change that is better and cheaper.
Trimming retrieved context is the one optimization that reliably improves quality and reduces cost at the same time.
5. Constrain output length
Output tokens are the expensive ones. A model asked an open question will produce a thorough answer. If the consuming system only uses three fields from it, specify a structured output with those three fields and cap the length. Verbose output that gets parsed down to a boolean is pure waste.
6. Batch what is not interactive
Anything running on a schedule rather than in front of a user can go through a batch endpoint where the provider offers one, typically at a significant discount for accepting higher latency. Classification, enrichment, and summarization jobs are almost always eligible and almost never converted.
Measure per feature, not in total
A single monthly bill tells you nothing actionable. Tag every call with the feature that made it and report cost per feature per month, and cost per successful outcome.
| Measure | Why it matters |
|---|---|
| Cost per feature per month | Shows you where the money actually goes, which is rarely where people assume |
| Cost per successful outcome | A feature with high cost and high value is fine. High cost and low value is the one to fix |
| Cache hit rate | Directly proportional to savings, and easy to improve once visible |
| Escalation rate to the large model | If routing is working, this stays low and stable |
Set a spend alert per feature before you optimize anything. An unwatched inference bill is an incident waiting to happen.
What good looks like
On the reviews we have done, a combination of routing, caching, and output constraints typically produces a substantial reduction in monthly spend with no measurable quality change on the eval set. The eval set is the important half of that sentence. Cost reduction without measurement is just quality reduction you have not detected yet.
Written from work we have actually delivered. If your situation looks like the one described here, the quote form takes about two minutes and there is no sales sequence attached to it.
Get a free quote