Running open-weight models in production: when self-hosting beats an API and what it really costs

01Utilisation is the whole argument

A hosted API charges per token, so cost tracks usage exactly and drops to zero overnight. A GPU charges by the hour whether or not anything is inferring. That difference means the comparison is entirely determined by utilisation, and utilisation is determined by your traffic shape, not by your model choice.

Interactive business applications have brutal traffic shapes for this: a working-day peak, near-zero at night and at weekends, and a peak-to-mean ratio commonly between five and ten. Size for the peak and you pay for idle capacity most of the week. Size for the mean and you queue during the peak. Batch workloads are the opposite - they can run flat out on a schedule, and that is where self-hosting economics are genuinely compelling.

02The costs that never make the spreadsheet

GPU hourly rate is the visible cost and rarely the largest one. The hidden costs are engineering time and the redundancy you need before this can carry production traffic - a single instance means a single point of failure with a multi-minute model load time on restart, so you are buying at least two.

Then there is the ongoing work: upgrading serving frameworks that move fast and break things, re-evaluating quality when you change quantisation, handling driver and CUDA version conflicts, and capacity planning for launches. None of that appears in a per-token comparison and all of it lands on a person who was previously shipping features.

03The reasons that actually justify it

Cost is the stated reason and usually the weakest one. The durable justifications are structural: data that legally or contractually cannot leave your infrastructure, latency requirements that demand co-location with your data, workloads at genuinely high sustained volume, and the need to serve a fine-tuned model that no provider hosts.

There is also a strategic argument about dependency - not being subject to a provider deprecating the model version your prompts were tuned against. That is real, but note it cuts both ways: you now own the upgrade treadmill instead, and open-weight releases move quickly enough that staying current is continuous work rather than a one-off migration.

• Regulated data that cannot leave a boundary you control - the strongest case.
• Sustained high-volume batch inference with predictable load.
• A fine-tuned or domain-adapted model with no hosted equivalent.
• Latency budgets that a network hop to an external provider breaks.
• Not: 'the per-token price looked high in a spreadsheet'.

04Serving is where the throughput actually comes from

Running a model with a naive loop and a single request at a time will produce numbers so bad that people conclude self-hosting does not work. Modern serving frameworks are the difference between that and viable economics - continuous batching keeps the GPU fed by admitting new requests as others finish, and paged attention stops the key-value cache from fragmenting memory and capping concurrency.

The practical consequence is that throughput at a given latency is a property of your serving configuration, not just your hardware. Two teams on identical GPUs can differ by an order of magnitude in tokens per second. Before benchmarking against an API price, make sure you are measuring a properly configured server under concurrent load, because a single-stream benchmark tells you almost nothing about production capacity.

05Quantisation buys throughput and costs quality you must measure

Quantising weights to 8 or 4 bits reduces memory footprint and increases throughput, often letting a model fit on a smaller and much cheaper GPU. Eight-bit quantisation is usually close to lossless for general use. Four-bit is more aggressive and the degradation is uneven - it tends to show up on reasoning-heavy and long-context tasks rather than on the short prompts people test with.

This is where an eval suite stops being optional. The quality difference between quantisation levels is not visible in casual testing and is very visible to users on the hard cases. Run your evaluation set at each level and decide with data, because the pattern of failure is task-specific and no general benchmark will predict it for your workload.

06A pragmatic sequence

Start on a hosted API. It gives you traffic data, a quality baseline and an eval suite, all of which you need before you can evaluate self-hosting honestly. Teams that self-host first end up comparing their new stack against nothing and cannot tell whether quality regressed.

Revisit once you have three months of real usage. Compute utilisation from actual traffic, price a redundant deployment rather than one instance, and add a realistic engineering cost. If the answer is close, stay on the API - the operational simplicity is worth the difference. Where the answer is clearly favourable, the common outcome is a hybrid: self-host the high-volume, well-understood workload, keep the API for spikes, long-tail tasks and anything needing frontier capability.