RAG vs Fine-Tuning vs Prompting: Which Should You Use?
Three different ways to get an LLM to know something it doesn't already know, or behave in a way it doesn't already behave — and they solve different problems, not the same one at different price points.
"Should we fine-tune or just prompt it?" is usually the wrong question, because it treats these as competing options for the same problem. Prompting, RAG, and fine-tuning solve three different problems: giving instructions, giving current/specific knowledge, and changing behavior patterns. Most real systems need one or two of these, rarely all three, and picking the wrong one wastes real engineering time.
What each one actually does
| Approach | What it solves | What it doesn't solve |
|---|---|---|
| Prompting | Telling the model what to do, right now, for this specific request — instructions, context, constraints, format | Doesn't give the model knowledge it wasn't trained on, and doesn't change its underlying behavior patterns across requests |
| RAG (retrieval-augmented generation) | Giving the model current, specific, or private information it doesn't have from training — by fetching relevant documents and inserting them into the prompt | Doesn't change how the model reasons or its general behavior — it's still the same model, just with better information to work from |
| Fine-tuning | Changing the model's default behavior — tone, format habits, domain-specific reasoning patterns — by further training it on examples | Doesn't reliably teach new facts (the model can still hallucinate about content from fine-tuning data), and doesn't help with information that changes after training |
The decision in practice
- Start with prompting. It's the cheapest, fastest to iterate, and solves the majority of real problems — most "we need fine-tuning" conversations turn out to be solvable with better prompt structure.
- Add RAG when the task needs information the model can't know — your company's internal docs, current data, anything proprietary or that changes after the model's training cutoff. This is almost always the right choice for 'the model doesn't know about our specific X' problems.
- Consider fine-tuning only when prompting and RAG both fail to produce the consistent behavior you need — usually a narrow, high-volume, well-defined task where you need a specific output style or format reliably at scale, and the cost of fine-tuning (data curation, training, ongoing maintenance) is justified by that volume.
"The model doesn't know our product catalog" is a RAG problem, not a fine-tuning problem — fine-tuning teaches behavior patterns, not reliable fact recall, and a fine-tuned model can still hallucinate details from its training data the same way a base model does. Retrieval, where the actual current data is fetched and placed in context, is the fix.
A concrete example: a customer support assistant
| Requirement | Right tool |
|---|---|
| Answer using our current help-center articles, which change weekly | RAG — retrieve the relevant article at query time, don't bake it into the model |
| Follow our specific tone and response structure consistently | Prompting (system prompt with explicit style constraints) — usually sufficient; fine-tuning only if prompting proves inconsistent at your actual volume |
| Handle a domain-specific reasoning pattern used thousands of times a day in a narrow, well-defined way (e.g. a specific classification task) | Fine-tuning becomes worth considering here — high volume, narrow task, where consistency at scale justifies the setup cost |
These aren't mutually exclusive — a production system commonly prompts a model, retrieves relevant context via RAG, and in rare high-volume narrow cases also uses a fine-tuned model for one specific sub-task. The Foundations program covers this decision explicitly as part of the prompting-fundamentals module, since choosing the wrong one is a common early-career mistake that costs real engineering time to walk back.