Fine-tuning vs RAG vs prompting: choosing the right tool for the actual problem

01The distinction that decides everything

Ask what is actually wrong with the current output. If the model does not know something - your product catalogue, this quarter's policies, a specific customer's history - that is a knowledge gap, and retrieval solves it. If the model knows the material but responds in the wrong format, tone, structure or style, that is a behaviour gap, and fine-tuning solves it.

Fine-tuning to inject knowledge fails in a particularly frustrating way: the model learns the shape of your data and confidently produces plausible-looking answers with wrong specifics. It has learned that responses look like this, not that these facts are true. And every knowledge update requires retraining, whereas retrieval updates the moment the source document does.

02Exhaust prompting first

Prompting is the cheapest option, iterates in seconds, and handles more than teams expect. Before concluding that a model cannot do something, verify the prompt actually specifies the task clearly, includes several examples of correct output, and defines the output structure explicitly.

Few-shot examples are consistently under-used. Three or four well-chosen examples of the exact input-output transformation you want frequently resolve behaviour problems that teams were preparing to fine-tune for. With prompt caching, the token cost of carrying those examples is far lower than it used to be.

• Specify the task, the audience and the output format explicitly.
• Include three to five examples covering normal and edge cases.
• Show what a refusal or an unknown answer should look like.
• Iterate against an evaluation set, not on individual impressions.

03The realistic comparison

Each approach carries different setup effort, ongoing cost and update characteristics. The update column matters more than teams weigh it, because knowledge changes constantly and retraining cadence becomes an operational commitment.

04When fine-tuning genuinely wins

There are real cases. Highly specialised output formats that resist prompting. Domain language and conventions that would consume enormous prompt space to explain. Latency and cost reduction, where a small fine-tuned model matches a much larger prompted one for a narrow task at a fraction of the price per call.

That last case is the most commercially compelling and the most overlooked. If one high-volume task dominates your inference bill, fine-tuning a smaller model on that specific task can cut cost dramatically while improving latency. The economics only work when volume is high and the task is stable.

05Data quality is the whole job

Fine-tuning effort is not the training run - that is often minutes and inexpensive. It is assembling several hundred to several thousand examples that are correct, consistent and representative, which is slow expert work that cannot be delegated to whoever is available.

Inconsistency in training data is worse than having less of it. If two examples handle similar inputs differently, the model learns the inconsistency and reproduces it unpredictably. Have one person or a tightly aligned pair curate the set, and review it for contradictions before training rather than diagnosing them afterwards from odd outputs.

• Prioritise consistency over volume - contradictions teach contradiction.
• Cover edge cases deliberately; the model learns the distribution you show it.
• Hold out a genuine test set before training, not a random split afterwards.
• Budget most of the timeline for curation, not compute.

06They compose

These are not competing choices. The strongest production systems combine them: a fine-tuned model that has learned your domain's output conventions, fed retrieved context for current facts, with a carefully engineered prompt tying the two together.

Sequence the adoption though. Get prompting right, add retrieval when knowledge is the constraint, and consider fine-tuning only once you have production data showing a specific behaviour gap that prompting cannot close. That order means each step is informed by evidence rather than assumption.