The short answer
RAG and fine-tuning solve different problems. RAG puts your documents in front of the model at answer time. Fine-tuning changes the model's weights with your examples. If answers are wrong because the model lacks current or private facts, RAG is usually the first move. If it has the facts but writes in the wrong shape, tone, or label set, fine-tuning fits better.
A useful test: give the model a question that needs a document it has not seen. If it fails, you have a knowledge problem. Paste the document into the prompt. If it now answers well, RAG can fix that. If it still rambles or misses the schema, you have a behavior problem.
Start with a clean prompt and a small eval set before either. You need to know what good looks like. Add RAG for facts and citations. Add fine-tuning for consistent structure, a narrower task, or a smaller model. Many systems use both, but rarely on day one.
| Aspect | RAG | Fine-tuning |
|---|---|---|
| What changes | Adds external knowledge at query time through retrieved chunks. | Changes model weights with task examples. |
| Best fit | Facts, citations, fresh documents, per-user access. | Format, tone, labels, tool-call patterns, narrow tasks. |
| Data freshness | Update the index when a document changes. | Retrain and redeploy when behavior or data changes. |
| Traceability | Can cite source and section. | Usually cannot point to a source. |
| Access control | Filters chunks by user permission at retrieval. | Data sits in weights, hard to remove or scope. |
| Start effort | Ingestion, chunking, index, retrieval eval. | Labeled examples, training runs, held-out eval. |
| Runtime cost | Retrieval plus more input tokens per answer. | Often shorter prompts, but training and hosting costs. |
| Main failure | Wrong or stale chunks reach the prompt. | Overfits, forgets, or invents facts. |
How RAG works
A RAG pipeline starts with ingestion. Pull from the source of truth, not a stale export. Clean headers and footers. Remove boilerplate, and split the text along document structure: a section, a table, a support article. Chunk size around 300 to 800 tokens is a reasonable start, but test it. Embed each chunk and write the vector, text, source, date, and access tags to an index such as pgvector, OpenSearch, Qdrant, or Pinecone.
At query time, embed the question with the same model. Retrieve the top-k chunks, then filter by tenant, product, or permission. A reranker helps when many near-duplicates compete. Build the prompt with a short system instruction, the retrieved chunks, and the question. Ask for citations by source and section. Keep the index fresh: reindex changed pages, delete removed ones, and log queries that found nothing.
How fine-tuning works
Fine-tuning starts with examples. Collect input and output pairs that show the task: a ticket and its category, a clause and its extracted fields, a rough note and the polished reply. Write them as JSONL, with a clear system prompt if the model supports one. Split train, validation, and test sets before you train. Keep a held-out test set that nobody tunes against.
For most teams, LoRA or QLoRA is enough. You train an adapter on a base model instead of updating every weight. Set epochs, learning rate, and batch size, then watch validation loss and task metrics. Stop when validation stops improving. Save the adapter with the base model name, data version, and prompt template, or you cannot reproduce the result. Fine-tuning shapes behavior, not a live fact store.
When to choose RAG
Choose RAG when answers depend on documents that change: manuals, policy pages, contracts, support tickets, internal wikis. Choose it when users need a source link or a quote. Choose it when access differs by user, because you can filter chunks by permission at retrieval time. That is much harder when facts sit in the weights.
RAG also fits when retraining is too slow. A price change or release note can land in the index in minutes. You keep the model fixed and swap the knowledge. Rollback is simple: remove the bad document and the answer changes. Watch the retrieval layer. Bad chunking, missing metadata, and duplicate pages are common failure points. Set a fallback when nothing relevant is found.
When to choose fine-tuning
Choose fine-tuning when the model knows enough but cannot follow your format. Examples: extracting fields into a fixed JSON schema, classifying messages into your internal labels, writing replies in a controlled tone, turning terse notes into a standard report. The task is narrow and you can show many examples. Fine-tuning can also make a smaller model viable, with lower latency and shorter prompts for high volume.
Do not fine-tune to store a knowledge base. If the model must answer from a new policy each month, RAG is cheaper to maintain. Fine-tuning also has a data removal problem: once a fact is in the weights, deleting it means retraining or filtering outputs. Use fine-tuning for skills, not for a document store. Test the tuned model on unrelated prompts to catch forgetting.
Costs and effort to consider
RAG costs come from ingestion, embedding calls, vector storage, retrieval latency, and index maintenance. The hidden work is connectors and permissions. Every source has its own API, auth model, and update pattern. You also need an eval set for retrieval and answer quality. Budget for reindexing and for someone to review bad answers.
Fine-tuning costs come from labeling, training runs, experiment cycles, and hosting. GPU hours for LoRA are modest, but the data work is not. You need clean examples, a held-out test set, and a retraining plan when behavior drifts. Both approaches still need a prompt, a fallback, and monitoring. If you want help ranking use cases and sizing the run cost before you commit, the AI readiness assessment from Cloudverg is $5,500/month, fixed scope over two weeks. It is one option among others.
Frequently asked questions
Can I use RAG and fine-tuning together?
Yes. A common split is fine-tuning for output format and tone, with RAG for facts and citations. Keep separate eval sets so you can tell which change helped. Add one at a time, because debugging both at once is slow.
Does fine-tuning teach a model new facts?
Not reliably. It can make a model repeat phrases from training data, but it cannot cite a source and it goes stale when facts change. For current policies or prices, use RAG with a fresh index.
How do I test which one works for my project?
Build a small test set of real questions with expected answers and sources. Run the base model, a RAG version, and a fine-tuned version on the same set. Score correctness and format, plus citation quality. Pick the simplest option that passes.