AI features have a cost structure most engineering teams have never dealt with: the marginal cost of a request is meaningful, it scales linearly with usage, and a well-received feature can produce an alarming invoice. The good news is that most production LLM spend is straightforwardly reducible without users noticing anything.
01Find out what you are actually paying for
Before optimising, attribute. Log tokens in and out, model, latency and cost per request, tagged by feature and by customer. Almost every team that does this discovers something unexpected - one feature dominating the bill, a handful of users generating a disproportionate share, or a retry loop quietly tripling volume.
Cost per request matters less than cost per outcome. A more expensive model that succeeds first time can be cheaper than a cheap one that needs two retries and a human correction, and only end-to-end attribution shows you that.
| Source | Typical share | Fix |
|---|---|---|
| Oversized context on every call | High | Retrieve less, cache the stable prefix |
| Using the largest model for everything | High | Route by task difficulty |
| Retries on malformed output | Moderate | Constrained decoding, schema validation |
| Regenerating identical responses | Moderate | Semantic cache |
| Verbose system prompts | Low but constant | Trim, then cache |
02Routing is the biggest single lever
Most applications send every request to the most capable model available. In practice a large share of requests are simple - classification, extraction, short rewrites - and a smaller, cheaper model handles them at equal quality for a fraction of the cost.
Route by task rather than by user or by guess. Classify the request, send the easy cases to the small model, and escalate to the large one when confidence is low or the task is genuinely hard. Measure quality per route so you can prove the cheap path is not degrading anything.
03Cache the prefix and cache the answer
Prompt caching, where the provider supports it, charges substantially less for a repeated prefix. Structure prompts so the stable part - system instructions, few-shot examples, fixed context - comes first and the variable part comes last. That single reordering can cut input costs sharply for high-volume features.
Separately, cache responses. Exact-match caching catches genuinely repeated queries. Semantic caching, where a sufficiently similar question returns a stored answer, catches far more but needs a similarity threshold you have tested - too loose and users get answers to questions they did not ask.
| Technique | Effort | Typical reduction |
|---|---|---|
| Trim retrieved context | Low | 20-40% of input tokens |
| Reorder for prompt caching | Low | Large on repeated prefixes |
| Route simple tasks to a small model | Moderate | 30-60% overall |
| Exact-match response cache | Low | 5-20%, workload dependent |
| Semantic cache | Moderate | 10-30% |
| Constrained output to stop retries | Moderate | Removes retry waste |
04Stop paying for retries
Requests that fail schema validation and get retried cost double or triple for one useful result. The usual cause is asking for structured output in prose and hoping. Constrained decoding or native structured output support removes most of that class of failure.
Also cap output length deliberately. A model asked an open question with no limit will produce a lot of tokens, most of which the interface truncates anyway. You paid for all of them.
05Put a ceiling on it
Per-user and per-tenant spend limits prevent a single customer or a runaway loop from producing a bill nobody approved. This matters more than it sounds: the failure mode of an unbounded AI feature is not degraded service, it is an invoice.
Alert on cost per active user rather than on total spend. Total spend rising alongside growth is fine; cost per user rising means the economics are drifting, and that is the number that determines whether the feature is sustainable.
Topics
Aarav Patel
Principal Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer