Where LLM inference spend actually goes: cost patterns from production systems

The levers, ranked

Not all optimisations are equal, and the ordering is fairly stable across the workloads we see. The first two routinely deliver more than everything else combined, which matters because teams often start with the hardest one.

Caching is the largest and least-used lever

Most production prompts are mostly static - a long system prompt, tool definitions, retrieved context - with a short variable tail. Provider-side caching bills that stable prefix at a steep discount, and enabling it is often a matter of reordering the prompt so stable content comes first.

The failure mode is specific and worth designing against: a single volatile token near the top of the prompt - a timestamp, a request identifier, a session ID - invalidates the entire prefix. We have seen this silently reduce hit rates to near zero while the code appeared correct, which is why cache hit rate belongs on a dashboard rather than being assumed.

Spend concentrates in a small number of features

Across the engagements we have instrumented, a small subset of features - typically 10 to 20% - accounts for the large majority of inference spend. This is an observation from our own work rather than a published benchmark, and the specific ratio varies, but the shape has been consistent enough to be worth checking for.

The practical implication is that attribution beats optimisation as a first step. Teams that optimise before instrumenting frequently improve a feature that was never material, while the actual driver goes unexamined. Cost per feature and per customer is the reporting that makes this visible.

• Instrument model, token counts and latency per call before optimising.
• Attribute cost by feature and by tenant, not just in aggregate.
• Check what proportion of calls are retries or health checks.
• Alert on cost per request, since totals hide regressions behind growth.

Routing works when it is an explicit policy

Published rate cards show roughly an order of magnitude between frontier and small model tiers as of August 2026. Classification, extraction, short summarisation and format conversion are handled well by the cheaper tier, and sending them to the most capable model is straightforwardly wasteful.

What makes routing safe is treating it as a testable policy with an escalation path, rather than conditionals scattered through a codebase. When cheap-tier output fails validation, retry once on the stronger model. Without that path, routing trades cost for quality invisibly, and the regression surfaces as user complaints rather than as a metric.

Self-hosting rarely pays at mid-market scale

Self-hosting is the optimisation teams reach for first and should usually reach for last. Running inference means GPU capacity planning, autoscaling that tolerates cold starts measured in minutes, quantisation trade-offs, model updates and an on-call rotation with CUDA familiarity. That is a platform team, not a configuration change.

The crossover exists but sits higher than most expect: sustained, predictable, high-volume traffic on a model that does not need frequent changing. Bursty or moderate workloads are almost always cheaper on an API, because you are buying someone else's utilisation. Run the comparison with fully loaded engineering cost included, not GPU hourly rates alone.

What we would instrument first

Before any optimisation: model, input and output token counts, latency, cache status, and the feature and tenant that triggered the call. That is enough to find where spend actually concentrates, and it usually contradicts the team's expectation.

The most common surprises are retries counted as legitimate traffic, debug or evaluation calls left running in production, and a single enterprise customer whose usage pattern distorts unit economics for everyone else.

• Model, tokens, latency and cache status on every call.
• Feature and tenant tags so cost maps to product decisions.
• Separate retry traffic from first-attempt traffic.
• Track cost per customer to catch pathological usage early.