Fine-tuning vs RAG: how to choose

The most common strategic mistake in enterprise AI is reaching for fine-tuning when the problem is knowledge access — or bolting on retrieval when the problem is behaviour. The two techniques solve different problems.

What each technique actually changes

Retrieval-augmented generation (RAG) leaves the model untouched. At query time, relevant passages from your documents are fetched and placed in the prompt, so answers are grounded in current, citable sources. Update a document and the system knows it immediately.

Fine-tuning changes the model's weights using your examples. It shifts how the model behaves: its format, tone, domain vocabulary and task reliability. It does not give the model a live view of your documents, and retraining is needed to reflect new knowledge.

The decision rule we use

• The answer lives in documents that change: RAG. Policies, contracts, product docs, tickets, research.
• The problem is output behaviour: fine-tuning. Strict formats, house style, domain shorthand, classification consistency.
• The model misreads your domain's language even with good context: fine-tuning, usually parameter-efficient (LoRA/QLoRA).
• You need citations and auditability: RAG, because every answer can point to its source passage.
• Both symptoms at once: combine them — a lightly tuned model reading from a retrieval pipeline is the most common production pattern we deploy.

Cost and maintenance realities

RAG is cheaper to start and cheaper to keep current, but it lives or dies on retrieval quality: chunking, embeddings and ranking need real engineering. Fine-tuning has an upfront training cost and needs an evaluation suite so you can prove the tuned model beats the base model on your tasks — if you cannot measure that, do not fine-tune yet.

A worked example

A firm wants an assistant that answers questions about its 4,000 internal policies in a fixed response template. Knowledge (the policies) changes weekly: that part is RAG. The rigid template and terminology: that part is a small LoRA fine-tune, trained once on a few hundred formatted examples. Neither technique alone would have hit the quality bar.

Bottom line

Ask one question first: is the failure about what the model knows or how the model behaves? Knowledge problems are retrieval problems. Behaviour problems are training problems. Diagnose before you spend.