RAG, fine-tuning or better prompts? How to choose.

Start with better prompts and a small test set. Add retrieval (RAG) when the model needs facts it doesn't have, such as your documents or anything that changes often. Consider fine-tuning only when the model already knows enough but its format, tone or task behavior stays inconsistent after careful prompting.

That order is cheap first, expensive last, and each step only happens when a test says the previous one wasn't enough. The rest of this note explains how to tell which problem you actually have.

What problem is each one solving?

The three options fix different failures, so the useful first step is to name the failure you are seeing.

• Better prompts fix unclear instructions. Many weak results come from a prompt that never says what a good answer looks like, what to do when unsure, or what format to return.
• RAG fixes missing knowledge. The model can't know your pricing sheet, your policy manual or last week's tickets. Retrieval finds the relevant passages and puts them in front of the model at answer time.
• Fine-tuning fixes inconsistent behavior. You train the model on examples of the exact output you want, so it learns a pattern: a house style, a strict structure, a classification scheme.

OpenAI's accuracy guide frames this as two axes. Context optimization helps when the model lacks the right information. LLM optimization helps when the model has the information but behaves inconsistently. Most real projects need some of the first and very little of the second.

Why start with prompts?

Writing a good prompt forces you to define success. You have to say what a correct answer contains, what tone it takes and what it should refuse. That definition becomes your test set, and you need a test set whichever path you take.

Anthropic's prompt engineering guide makes the same point from the other direction: before tuning a prompt, have clear success criteria and a way to test against them. It also notes that some problems, like latency or cost, are often solved faster by choosing a different model than by rewriting the prompt.

A useful first test set is small. A set of 20 to 50 real questions with the answers you'd accept is enough to see patterns. Run every change against it.

When do you need RAG?

You need retrieval when the failures are about facts. Signs to look for:

• The model gives confident, plausible answers that are wrong about your business.
• The right answer lives in documents, a database or a help center.
• The information changes weekly or monthly, so anything baked into a model would go stale.
• You need to show where an answer came from, for trust or for compliance.

When RAG fails, check retrieval before blaming the model. If the right passage never reached the prompt, no amount of prompting will help. Look at chunk size, how documents are split, whether search uses keywords, meaning or both, and whether results are re-ranked. If the right passage did arrive and the answer is still wrong, the problem is the instructions, and you are back to prompting.

With today's long context windows, a small and stable set of documents can sometimes go straight into the prompt with no retrieval layer at all. That is simpler to build and easy to test, so try it first when the material is short.

When is fine-tuning worth it?

Fine-tuning earns its cost in a narrow band of cases:

• A repeated, well-defined task (classify, extract, rewrite in a fixed style) at high volume.
• A solid set of high-quality examples of the exact output you want. Quality matters more than quantity.
• A test set showing that good prompts plus examples still leave the behavior inconsistent.
• A reason to shorten prompts, such as cost per call or speed at scale.

It is a weak tool for teaching facts. Facts change, and retraining every time they do is slow and expensive. It also adds work you have to own: a training pipeline, versioned datasets, and re-evaluation whenever the base model is updated.

Decision checklist

Work down the table. Stop at the first row that fixes your test results.

If your test shows...	Then try	You'll need
Vague, off-format or inconsistent answers on simple tasks	Clearer prompt: role, rules, examples, output format	A test set and a few hours
Wrong or invented facts about your business	RAG over your documents or data	Clean source content, a search setup, retrieval tests
Right passage retrieved, wrong answer	Better instructions on using context; cite sources	Prompt changes, answer-level tests
Correct facts, but format or style drifts at scale	More examples in the prompt, then fine-tuning	A curated example set, a training and eval loop
Too slow or too costly	A different or smaller model; caching; shorter prompts	Cost and latency measurements
Answers are good but nobody trusts them	Source citations, a "not sure" path, human review	Product and UX work on how answers are shown

Two rules sit above the table. Measure before and after every change. And design for failure: an assistant that says "I don't know" and hands off to a person is safer than one that guesses.

How I'd approach it for you

I'd begin with a short discovery: what the system should answer, for whom, and what a wrong answer costs you. From that we write a test set of real questions together. Then I build the simplest version that could pass, usually a strong prompt over a good model, and measure it. Retrieval or tuning only come in when the numbers ask for them, and you see the scores at each step, so you know what you're paying for and why.

You own the prompts, data pipelines and test set at the end, so the system can be checked and improved without me.

If you have a problem like this, Start a conversation.

Sources

• OpenAI, Optimizing LLM accuracy (opens in a new tab) (context vs model optimization; start with prompt engineering; evaluation-driven iteration).
• Anthropic, Prompt engineering overview (opens in a new tab) (success criteria and tests before prompt engineering; when a different model is the better fix).