LLM inference cost optimization: the levers that actually move the bill

01Measure before you optimise

You cannot optimise what you cannot attribute. Before touching a single parameter, instrument every call with the model used, input and output token counts, latency, the feature that triggered it, and whether it hit cache. Most teams discover that 10-20% of their features drive 80% of their spend, and that a surprising share of calls are retries, health checks, or debug traffic nobody meant to ship.

Attribute cost per feature and per customer, not just in aggregate. A single enterprise customer running a pathological workload can distort your entire unit economics, and you will never see it in a monthly total.

• Log model, input tokens, output tokens, latency and cache status on every call.
• Tag by feature and tenant so cost maps to product decisions.
• Alert on cost per request, not just total spend - totals hide regressions behind growth.
• Track the ratio of retries to first-attempt successes.

02The levers, ranked

Not all optimisations are equal. In our experience the ordering below holds for most product workloads, and the first two routinely deliver more than the rest combined. Start at the top and stop when the savings stop justifying the engineering time.

03Caching is the cheapest win available

Most production prompts are mostly static - a long system prompt, tool definitions, retrieved context - with a short variable tail. Provider-side prompt caching bills that static prefix at a steep discount, and the change is often a matter of reordering the prompt so the stable content comes first and the variable content last.

The failure mode is subtle: a single volatile token near the top of the prompt, a timestamp or a request ID, invalidates the entire prefix and silently drops your hit rate to zero. Log cache hit rates per prompt template and treat a sudden drop as a production incident, because financially it is one.

04Routing: not every request needs the best model

The instinct to send everything to the most capable model is expensive and usually unnecessary. Classification, extraction, routing decisions, short summaries and format conversion are handled well by smaller, cheaper models. Reserve the frontier tier for genuinely hard reasoning, long-context synthesis and customer-visible generation where quality is the product.

Implement routing as an explicit, testable policy rather than scattered conditionals. A small router - rules first, a classifier only if rules prove insufficient - keeps the decision auditable. Critically, build an escalation path: when the cheap model's output fails validation, retry once on the stronger model rather than shipping a bad answer.

• Rules before classifiers - most routing is a switch statement, not ML.
• Validate cheap-tier output and escalate on failure.
• Track quality per tier, not just cost, or you will optimise into a worse product.
• Re-evaluate routing whenever a new model tier ships; the economics move fast.

05When self-hosting actually pays

Self-hosting is the optimisation teams reach for first and should usually reach for last. Running your own inference means GPU capacity planning, autoscaling that handles cold starts measured in minutes, quantisation trade-offs, model updates, and an on-call rotation that understands CUDA. That is a platform team, not a config change.

The crossover point is real but higher than most expect: sustained, predictable, high-volume traffic on a model you do not need to change often. Bursty or low-volume workloads are almost always cheaper on an API, because you are buying someone else's utilisation. Run the numbers with fully loaded engineering cost included, not just GPU hourly rates.