Fine-Tuning vs RAG vs Prompt Engineering
Fine-tuning teaches behavior, RAG supplies knowledge, prompting shapes both. What the published comparisons found, and how to pick without burning a GPU budget.
Fine-tuning teaches behavior, RAG supplies knowledge, prompting shapes both. What the published comparisons found, and how to pick without burning a GPU budget.
Why AI chat can be free, what happens to what you type, and the short checklist worth running before anything from work goes into a chat box.
When owning your hardware beats renting a virtual machine, when it does not, and the questions to ask a colocation facility before you commit to anything.
Object storage explained: how it differs from a server disk, what it is good and bad at, and the signs that tell you it is time to use a bucket.
Self-hosting an open-weight model or calling a hosted API? The drivers on each side, the break-even point, and the hidden costs.
A walkthrough of how to build semantic search with embeddings using RAG AI to index and chat with your docs, and when keyword search still wins.
Prompt engineering changes what you ask. Retrieval-augmented generation, or RAG, changes what the model can see when it answers. Fine-tuning changes the model itself. They solve different problems, and the most common mistake in applied AI work is reaching for the third when you needed the second.
if the model does not know a fact, give it the fact. If the model knows the fact but answers in the wrong shape, change the model. Facts belong in retrieval. Behavior, format, tone, and task structure belong in fine-tuning. Prompting is where you start and where a surprising amount of work finishes.
Retrieval does, and fine-tuning largely does not. When researchers compared the two head to head on knowledge-intensive tasks, retrieval won consistently, and the models struggled to absorb new facts through fine-tuning at all. This is the single most expensive misconception in the field, because fine-tuning is also the most expensive option.
Ovadia et al. tested this directly in "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs." Their conclusion, stated in the abstract: while unsupervised fine-tuning offers some improvement, "RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge." They add the mechanism: "LLMs struggle to learn new factual information through unsupervised fine-tuning, and that exposing them to numerous variations of the same fact during training could alleviate this problem" (arXiv:2312.05934).
To teach a model one fact by fine-tuning, you need that fact restated many different ways in your training data. To teach it the same fact by retrieval, you put the document in an index. One of those is an afternoon.
There is a second reason retrieval wins for facts, and it has nothing to do with benchmark scores. Facts change. A fine-tuned model has your April data baked into weights, and updating it means another training run. A retrieval index is updated by editing a document. If your knowledge has a shelf life, and almost all business knowledge does, fine-tuning is the wrong storage medium regardless of how well it scores.
Each operates on a different part of the system. Prompting changes the instruction and the examples you send with each request. RAG changes the context by fetching relevant documents and putting them in front of the model at query time. Fine-tuning changes the weights, which changes the model's default behavior everywhere, permanently.
Prompt engineering is instruction, structure, and examples. You are working within a model that already exists, spending tokens instead of compute. Few-shot examples in the prompt are the cheapest form of teaching there is: show the model three correctly formatted outputs and it will frequently produce the fourth. Cost is per request and per token, and iteration takes seconds.
How to turn scanned images and PDFs into typed fields your app can store, using local OCR and a language model, with the validation that keeps poor quality out.
RAG adds a retrieval step before generation. A user question gets embedded, matched against an index of your documents, and the closest chunks get pasted into the prompt as context. The model then answers from material it can see rather than from what it memorized in training. It can cite its sources, because it has sources. The mechanics of the retrieval half are covered in how to build semantic search with embeddings.
Fine-tuning continues training the model on your examples so the behavior you want becomes its default. Nothing needs to be in the prompt because it is in the weights. This is the only one of the three that requires GPUs, a dataset, and a training run, and it is the only one that produces an artifact you then have to host, version, and re-do when the base model improves.
Prompting is instructions to an employee, RAG is handing them the file, and fine-tuning is training them for the role. You would not send someone to a training course to tell them one customer's address.
A clear instruction, an explicit output format, two or three examples, and a statement of what to do when the answer is not available will solve a large share of applied tasks. Try this first, always, because it is the only option where a failed experiment costs you ten minutes.
What to try before concluding prompting has failed:
summary and confidence" outperforms "summarize this" by a wide margin.Stanford's 2025 AI Index reports that "the inference cost for a system performing at the level of GPT-3.5 dropped over 280-fold between November 2022 and October 2024" (Stanford HAI). A longer prompt used to be a cost problem worth engineering around. Increasingly it is cheaper than the engineering.
Prompting stops being enough when the instruction becomes unwieldy, when you need output consistency a prompt cannot enforce, or when the model needs information that does not exist in it.
Use RAG when the answer depends on information the model was never trained on, or on information that changes. Internal documentation, product catalogs, support histories, policies, contracts, and anything with a date on it. If a correct answer requires reading a specific document, that document has to be retrievable.
The signals:
RAG's honest failure mode is that it is a search problem wearing an AI costume. If retrieval returns the wrong chunks, the model answers from the wrong chunks, confidently. Most disappointing RAG systems are not model failures; they are chunking, indexing, and ranking failures. Budget your effort accordingly: the retrieval half deserves more of your attention than the generation half.
Fine-tune to change behavior, not to add knowledge. Consistent output format across thousands of calls, a specific voice, a narrow classification task, a domain-specific style of reasoning, or a structured output the base model keeps drifting away from. Also fine-tune when you want a smaller model to do one job as well as a large model does, which is where the cost savings live.
The cases where it pays:
The cost has come down enough that this is no longer a research-lab activity. LoRA reports reducing "the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times" against full fine-tuning of GPT-3 175B with Adam (arXiv:2106.09685), and QLoRA reports enough further savings "to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance" (arXiv:2305.14314). The sizing math behind those numbers is in how much VRAM you need to fine-tune an LLM.
What fine-tuning costs: assembling and cleaning a dataset of a few hundred to a few thousand high-quality examples, running and evaluating the training, hosting the resulting weights, and repeating the whole exercise when a better base model ships in four months. That last one is the recurring tax, and it is why fine-tuning should be a decision you can defend rather than a default.
Work up the ladder, cheapest first, and stop at the rung that solves the problem. Each step costs more to build and more to maintain than the one before it, so skipping ahead is how projects acquire expensive machinery they did not need.
Build the evaluation set before step one. Thirty to fifty representative inputs with known-acceptable outputs, scored the same way each time. Without it, every step above is a matter of opinion, and "it seems better" is how teams end up fine-tuning to fix a chunking bug.
Our production systems do, and the combination in our experience is usually the better architecture. Retrieval supplies the facts, a fine-tuned model supplies the behavior, and the prompt assembles them. Each layer does the job it is suited to, and none is compensating for another.
A mature system tends to look like: user question comes in, retrieval pulls the relevant documents with permission filtering applied, a prompt template assembles the question and the retrieved context with the output contract stated, and a model fine-tuned for the domain's output format generates the answer with citations back to the retrieved chunks.
The fine-tuned model is not being asked to remember the documents. The retrieval layer is not being asked to enforce output format. Each failure has one obvious place to look, which matters more for maintenance than any benchmark.
None of these three fixes a model that is not capable enough for your task. If the base model cannot reason through your problem, retrieval gives it better material to be wrong about and fine-tuning teaches it to be wrong in your house style. Test capability first with a handful of hard examples and a strong prompt before building anything.
If your task is classification with a fixed set of labels and clean rules, or extraction from consistently formatted documents, conventional code is faster, cheaper, deterministic, and testable. Reach for a language model where the input is unstructured and the judgment is fuzzy. Everywhere else, the boring solution is still the better engineering.
We host open-weight models on our own US infrastructure and serve them through an OpenAI-compatible API, so the retrieval-and-prompting layers described here run against Carpathian AI with a base URL change and no GPUs for you to manage. The public chat is free, with no account, no message cap, and no card, which makes it a reasonable place to test whether a prompt change fixes your problem before you build anything.
If you would rather not work through this alone, that is the work our AI application development team does: picking the right rung, building the retrieval layer properly, and being honest when the answer is that you did not need a model for this part.