Most LLM comparisons rank models on benchmarks. In a bank, an insurer or a hospital that is close to irrelevant, because the constraint is not capability - it is that a specific auditor will ask where the data went, who could see it, and how you know. That question eliminates most options before quality is even discussed, and it is the reason platforms like watsonx exist alongside the frontier APIs.
01Start from the data boundary, not the leaderboard
The first question in a regulated build is which jurisdiction the inference happens in and whether the prompt leaves your control. If the answer is that customer data crosses into a third party's infrastructure, everything downstream - the DPA, the residency clause, the retention commitment, the sub-processor list - becomes the actual project.
This is why teams end up on watsonx, on self-hosted open-weight models, or on a cloud provider's in-region managed offering. Not because the model is stronger, but because the boundary is somewhere they can point to on a diagram and defend.
| Shape | Data boundary | Effort | Suits |
|---|---|---|---|
| Frontier API | Vendor infrastructure | Lowest | Non-sensitive workloads |
| In-region managed | Named region, vendor-run | Low | Most regulated cases |
| watsonx | Choice incl. on-prem | Moderate | Governance-heavy sectors |
| Self-hosted open weights | Entirely yours | Highest | Strict residency or air-gap |
02What Granite is for
IBM's Granite family is smaller than the frontier models and is not trying to beat them on general reasoning. The pitch is different: models with documented training data provenance, permissive licensing, and sizes that run economically on hardware you control.
Provenance is the part that matters in this context. When an auditor or a customer asks what the model was trained on, 'we do not know and cannot find out' is an uncomfortable answer for a regulated deployment. A documented corpus is a defensible one, and for a great many enterprise tasks - classification, extraction, summarisation over your own documents - the smaller model is entirely sufficient.
03Governance is the actual product
watsonx.governance exists because in regulated sectors the model is a small part of the obligation. You need a record of which model version served which request, what it was prompted with, what it returned, who approved its deployment, and how it performs against drift and bias measures over time.
Teams routinely underestimate this. Building the inference path takes weeks; building the evidence trail that satisfies a model risk management framework takes considerably longer, and it cannot be retrofitted convincingly after the fact because the records simply do not exist for the period before you started keeping them.
04The cost model inverts at volume
Per-token API pricing is dramatically cheaper at low volume. There is no infrastructure to run, no GPUs sitting idle overnight, and no platform team. For pilots and for spiky workloads this is the correct economic choice and it is not close.
Self-hosted or platform-hosted inference has high fixed cost and low marginal cost, so it wins only past a sustained volume threshold. The mistake is comparing them at pilot scale, concluding the API is cheaper, and never revisiting it once production volume has grown by an order of magnitude.
05Retrieval quality dominates model choice
In practice most enterprise AI disappointments are retrieval failures wearing a model's clothes. If the right passage never reaches the context window, no model recovers - and swapping to a stronger one produces a marginally more fluent wrong answer.
Spend the effort on chunking, on hybrid search, on reranking and on evaluating retrieval separately from generation. A modest model with excellent retrieval consistently outperforms a frontier model with careless retrieval, and it is cheaper and easier to keep inside your data boundary.
06How to decide
If your data is not sensitive and volume is uncertain, use a frontier API and revisit at scale. Building infrastructure for a workload you have not validated is the most common and most expensive error in this space.
If a regulator, a residency requirement or a customer contract puts the boundary inside your estate, that decides it - and the real work becomes governance and retrieval rather than model selection. Choose the smallest model that passes your evaluations, because every increment of size is paid for on every request forever.
Topics
Aarav Patel
Principal Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.