Updated October 2026
The AI assistant was wrong.
That part was obvious.
What wasn’t obvious was what the team should do about it.
One engineer wanted to rewrite the prompt.
Another wanted to build a retrieval system.
A third suggested fine-tuning the model.
All three ideas sounded reasonable.
Only one of them addressed the actual problem.
This is one of the most useful distinctions to understand when building production AI:
Prompt engineering, RAG and fine-tuning are not three versions of the same solution.
They solve different kinds of failure.
Sometimes the model already has everything it needs but the instructions are poor.
Sometimes the model is missing current or private information.
Sometimes the information is available and the instructions are good, but the model still does not perform a specialized task consistently enough.
Those situations can look almost identical to the person using the product:
“The AI gave me a bad answer.”
But they require very different fixes.
So instead of defining prompt engineering, retrieval-augmented generation and fine-tuning separately, let’s diagnose three failures inside one fictional AI product.
Meet Atlas Support
Imagine a software company called Northstar Systems.
Northstar sells project-management software to businesses.
Its support team handles thousands of customer questions every month:
“How do I configure SSO?”
“Why did my invoice increase?”
“Does the Business plan support SCIM?”
“How long are audit logs retained?”
“My API request returns error 403. What should I check?”
Northstar builds an AI support assistant called Atlas.
Atlas has three jobs:
- answer product questions,
- help customers troubleshoot,
- classify support cases for the correct internal team.
The prototype looks good.
Then production testing begins.
Atlas fails in three different ways.
Table of Contents
Failure 1: The Answer Is Correct, but the Response Is Terrible
A customer asks:
“How do I reset an API token?”
Atlas produces twelve paragraphs.
It explains:
authentication,
credential hygiene,
token rotation,
API security,
least privilege,
and why secrets should not be committed to source control.
Technically, almost everything it says is correct.
The problem is that the customer needed this:
- Open Settings.
- Select API Access.
- Revoke the old token.
- Generate a new token.
- Update integrations using the previous token.
The model does not lack knowledge.
It has not failed retrieval.
It does not need to memorize more examples permanently.
The failure is simpler:
Atlas was not given precise enough instructions about how it should respond.
This is where prompt engineering belongs.
Fix 1: Improve the Prompt
Northstar replaces:
“You are a helpful customer support assistant.”
with something more useful:
“Answer troubleshooting questions using short numbered steps. Start with the action the customer should take. Avoid background explanation unless it is necessary to solve the problem. If the solution requires more than five steps, group related steps. If you do not know the answer, say so.”
The same question is tested again.
This time Atlas returns five concise steps.
No retraining.
No vector database.
No new model.
The system simply received better instructions.
That is the first important lesson:
If the model already has the information and capability required, fix the instructions before changing the architecture.
Google describes prompt engineering in similar terms: prompts provide models with instructions, context and examples that influence the response they generate.
What Prompt Engineering Is Actually Good At
Prompt engineering is especially useful when the problem sounds like:
“The answer is too long.”
“Use this output format.”
“Always return valid JSON.”
“Explain this for a beginner.”
“Classify this request into one of these five categories.”
“Use our brand tone.”
“Show your recommendation first and the explanation second.”
“Ask a clarification question when information is missing.”
Prompts are good at defining:
- task
- role
- constraints
- output format
- examples
- tone
- decision criteria
- context available during the request
Few-shot examples can also be included.
For example:
Input
Payment failed after card update.
Desired output
Category: BILLING_PAYMENT
Priority: MEDIUM
Next action: Verify payment method
A few representative examples may be enough to make the model understand the pattern.
But Then Atlas Fails Again
A customer asks:
“Does the Business plan include unlimited audit-log retention?”
Atlas answers:
“Yes. Business accounts include unlimited audit-log retention.”
The support engineer immediately knows that is wrong.
Northstar changed the policy two months ago.
Business accounts now receive 180 days.
Enterprise contracts can have longer retention.
The model is using old information.
The team tries adding this to the prompt:
“Always use the latest Northstar product information.”
That sounds reasonable.
It does nothing.
The model cannot use information that was never provided to it.
Now we have a different problem.
Failure 2: The Model Is Missing the Information
Northstar’s product information changes constantly.
There are:
- pricing documents
- plan limits
- release notes
- security documentation
- integration guides
- internal troubleshooting pages
- billing policies
This information is not a behavior the model needs to learn.
It is knowledge the system needs to retrieve when required.
That is where RAG becomes useful.
Fix 2: Give the Model the Right Knowledge at Runtime
Northstar creates a searchable knowledge base containing its approved documentation.
Now the workflow becomes:
Customer question
↓
Search company knowledge
↓
Retrieve relevant passages
↓
Send those passages with the question to the model
↓
Generate an answer grounded in the retrieved information
The same customer asks:
“Does the Business plan include unlimited audit-log retention?”
The retrieval system finds:
Business plan — audit-log retention: 180 days.
Atlas now answers:
“No. The Business plan currently includes 180 days of audit-log retention. Enterprise retention can vary by contract.”
The underlying language model did not need to memorize the new policy.
It was given the right information when answering the question.
That is retrieval-augmented generation.
Google describes RAG as combining information retrieval with generative models so responses can use external, current or organization-specific information rather than relying only on what was learned during model training.
A Mental Model That Makes the Difference Clear
I think about the three approaches like this.
Prompt Engineering
Instructions
What should the model do?
RAG
Knowledge
What should the model know for this request?
Fine-Tuning
Learned behavior
What should the model become consistently better at doing?
That distinction is more useful than treating them as competing AI features.
RAG Is Like an Open-Book Exam
Imagine a student taking an exam.
Prompt engineering is similar to giving the student clearer instructions:
“Answer in three sentences and show the calculation.”
RAG is more like giving the student the correct reference book while answering.
Fine-tuning is different.
Fine-tuning is closer to training the student repeatedly before the exam so that they become better at a particular kind of task.
The analogy is imperfect, but it reveals why each approach solves a different problem.
When RAG Is Usually the Better Starting Point
RAG becomes attractive when requirements sound like:
“The assistant needs our internal documentation.”
“Information changes regularly.”
“Answers must use current policies.”
“Users need citations.”
“We have thousands of documents.”
“Different users are allowed to see different documents.”
“The model needs information that was never in its training data.”
Common RAG applications include:
- internal knowledge assistants
- customer-support systems
- enterprise search
- legal research
- policy assistants
- product documentation
- research tools
The important clue is:
The answer exists outside the model.
RAG Also Makes Updating Knowledge Easier
Suppose Northstar changes its refund policy tomorrow.
With RAG, the company can update the approved document and refresh the search index.
Future requests can retrieve the new policy.
Trying to encode every frequently changing business fact into model weights would create a very different operational problem.
This is why RAG is especially natural for knowledge that is:
- private
- changing
- large
- traceable
- permission-controlled
It also makes citations more practical because the system knows which source it retrieved.
But RAG Has Its Own Failure Modes
RAG does not automatically make answers correct.
Suppose the correct document exists but search retrieves the wrong one.
The language model may confidently answer using irrelevant evidence.
So a RAG system has at least three distinct quality questions.
Retrieval
Did we find the information required?
Grounding
Did the answer stay faithful to the retrieved information?
Usefulness
Did the final answer actually help the user?
If Atlas retrieves a 2024 policy instead of the current 2026 policy, changing the language model may not solve the problem.
The retrieval layer needs fixing.
Google’s current RAG guidance likewise emphasizes that retrieval quality is critical: irrelevant retrieved information can produce grounded-looking but still incorrect or off-topic answers.
Then Atlas Fails a Third Time
Northstar also wants Atlas to classify every incoming support request.
The company has 42 internal categories.
Examples include:
- BILLING_REFUND
- BILLING_INVOICE
- AUTH_SSO
- AUTH_SCIM
- API_PERMISSION
- API_RATE_LIMIT
- SECURITY_AUDIT
- EXPORT_DATA
- INTEGRATION_SLACK
A customer writes:
“Our identity provider says provisioning succeeded, but no users are appearing in Northstar.”
The correct category is:
AUTH_SCIM
Sometimes Atlas gets it right.
Other times it chooses:
AUTH_SSO
The team improves the prompt.
It adds category definitions.
It includes good examples.
Performance improves.
But production evaluation still shows enough inconsistency that routing errors would create significant operational work.
Now the problem is neither missing information nor obviously bad instructions.
Atlas needs to become more consistent at a stable, specialized task.
Failure 3: The Model Understands the Task but Does It Inconsistently
Northstar has an advantage.
It already possesses years of support data.
Thousands of tickets were correctly classified by experienced support agents.
That dataset contains examples of:
customer input → correct support category
Now fine-tuning becomes a reasonable option.
Fix 3: Fine-Tune for the Specialized Behavior
Northstar creates a carefully cleaned training dataset.
Example:
Input
“Users are being created in our IdP but aren’t appearing in Northstar.”
Output
AUTH_SCIM
Another:
Input
“SAML login redirects back to our login page.”
Output
AUTH_SSO
The goal is not to teach Atlas yesterday’s product documentation.
It is to make Atlas more reliable at a repeated behavior:
Classify Northstar support requests according to Northstar’s taxonomy.
Google’s current model-adaptation guidance similarly positions fine-tuning as useful when prompting and in-context examples are not sufficient for specialized or highly consistent behavior.
What Fine-Tuning Actually Changes
Fine-tuning trains a model on examples representing the behavior you want.
Potential uses include:
- classification
- specialized terminology
- structured outputs
- consistent formatting
- recurring domain tasks
- stylistic consistency
- task-specific behavior
The model’s parameters are adjusted during training.
That is fundamentally different from RAG, where additional information is provided during each request.
A simple distinction is:
RAG changes the context.
Fine-tuning changes the model.
Fine-Tuning Is Not a Database
This distinction prevents a lot of bad architecture.
Imagine Northstar has a 500-page product handbook.
Someone says:
“Let’s fine-tune the model on the handbook so it knows everything.”
That might sound efficient.
But what happens when:
pricing changes,
a product limit changes,
a security policy changes,
a feature is removed,
or a customer requires a citation?
A knowledge-retrieval system is much easier to update and audit.
Fine-tuning should not generally be treated as a substitute for a document store or search layer.
The more frequently knowledge changes, the less attractive it becomes to encode that knowledge primarily through model training.
The Three Failures Side by Side
Northstar experienced three bad outputs.
But the causes were completely different.
| What went wrong | Root cause | Best first move |
|---|---|---|
| Correct answer, wrong format | Instructions | Prompt engineering |
| Outdated product information | Missing knowledge | RAG |
| Inconsistent specialized classification | Behavioral consistency | Fine-tuning |
This is why:
“Our AI is wrong” is not a diagnosis.
It is merely a symptom.
The Digital Stackroom Diagnostic
When an AI system underperforms, I would ask these questions in order.
Question 1: Does the model have the information required?
If no:
consider:
RAG
or:
API / database / tool access
If yes, continue.
Question 2: Does the model understand what you want it to do?
If the problem can be fixed using:
- clearer instructions
- better structure
- examples
- output constraints
start with:
Prompt Engineering
If performance remains inadequate, continue.
Question 3: Is the desired behavior stable and repeated?
Examples:
- classification
- translation style
- structured transformation
- highly consistent writing style
- specialized domain reasoning
If yes—and you have representative high-quality training data—consider:
Fine-Tuning
Question 4: Does the application require both specialized behavior and changing knowledge?
Then the answer may be:
Fine-Tuning + RAG
There is no rule saying you must choose only one technique.
Start With the Least Complicated Architecture That Works
There is a temptation in AI projects to equate complexity with maturity.
A team hears about RAG and thinks:
“Production AI needs RAG.”
Another hears about model tuning and thinks:
“We need a custom model.”
But architecture should solve actual problems.
If four lines of better instructions solve the issue, those four lines may be the best architecture.
Every additional component introduces:
- cost
- latency
- infrastructure
- monitoring
- testing
- failure modes
- maintenance
So a reasonable default sequence is:
Establish an evaluation baseline
↓
Improve the prompt
↓
Add external knowledge if information is missing
↓
Consider tuning if specialized behavior remains inconsistent
↓
Evaluate again
OpenAI’s current optimization guidance makes essentially the same broader point: prompting, RAG and fine-tuning address different failure types, and teams should diagnose whether they need better context or better model behavior rather than automatically moving through the techniques as a fixed ladder.
Tools and APIs Belong in This Conversation Too
Suppose a Northstar customer asks:
“What is my current invoice balance?”
RAG is not necessarily the right answer.
The correct information already exists in Northstar’s billing database.
Atlas should query the billing system.
Now suppose the customer says:
“Cancel my subscription.”
That requires an action.
The assistant needs an authorized tool or API.
This gives us four useful architectural layers.
Prompt
Instructions
RAG
Knowledge
Fine-Tuning
Behavior
Tools / APIs
Live data and actions
Many modern AI applications combine all four.
A Rule I Would Not Give to the Model Alone
Imagine Northstar has this policy:
Refunds above $1,000 require manager approval.
Should Atlas simply receive that instruction and be trusted to remember it every time?
I would not rely on that.
This is a deterministic business rule.
The application should enforce it with normal software logic.
For example:
Refund ≤ $1,000
→ continue workflow
Refund > $1,000
→ require manager approval
This is another important design principle:
Use AI for interpretation. Use deterministic software for controls that must always be enforced.
A Worked Example: Internal HR Assistant
Consider an HR assistant.
Employees ask:
“How much annual leave do I have?”
“What’s our parental-leave policy?”
“Can I work from another country?”
“Draft an email to my manager requesting Friday off.”
Which techniques belong where?
Current HR policies
Use:
RAG
because policies can change and employees may need citations.
Concise, professional answers
Use:
Prompt Engineering
because this is response behavior.
Very consistent classification of HR requests
Potentially:
Fine-Tuning
if prompt-based classification cannot meet the required reliability.
Employee’s actual remaining leave balance
Use:
HR system API
because the answer is live transactional data.
Submit a leave request
Use:
Tool/API
because an action must occur.
One assistant.
Four different mechanisms.
A Worked Example: Marketing Copy
Now imagine a company generating product descriptions.
The product information is already sent with every request.
The problem is inconsistent writing style.
RAG?
Probably unnecessary.
The information is already available.
Start with:
better prompts + examples
If the business generates millions of descriptions and needs extremely consistent brand behavior, tuning may become useful.
The same technology choices produce completely different answers depending on the failure.
A Worked Example: Technical Support
A customer asks:
“What does error PX-441 mean?”
That error code was introduced yesterday.
The solution exists in a new technical document.
That points strongly toward:
RAG
Now imagine Atlas retrieves the correct documentation but writes a troubleshooting response in the wrong format every time.
That points toward:
prompting or fine-tuning
The final user experience is one answer.
The engineering problem can exist at several layers.
The Freshness Test
One simple question helps eliminate many bad designs:
How quickly does the information change?
Current bank balance
API/database.
Today’s inventory
API/database.
Product documentation updated weekly
RAG.
Stable support-ticket taxonomy
Potential fine-tuning.
Output tone
Prompting or fine-tuning.
Permanent business rule
Application logic.
The faster factual information changes, the stronger the case for keeping that information outside model weights.
The Citation Test
Ask:
Does the user need to see evidence?
A research assistant probably should.
A legal assistant probably should.
A policy assistant probably should.
A product-support assistant often should.
That makes retrieval attractive because the system knows which sources were used.
A model may produce information learned during training, but tracing a particular statement back to a specific authoritative source is much harder.
The Training-Data Test
Fine-tuning often sounds attractive until someone asks:
“What would we train it on?”
Northstar might have 100,000 historical tickets.
That sounds excellent.
But perhaps:
20% have outdated categories,
15% were incorrectly labelled,
and older tickets follow a taxonomy that no longer exists.
The real question is not:
“How much data do we have?”
It is:
How much clean, representative data do we have?
Current fine-tuning guidance from both Google and OpenAI emphasizes the importance of representative examples and evaluation data rather than simply maximizing dataset size.
The Evaluation Test
Before Northstar adds any architecture, it builds a test set.
Imagine 300 representative support cases.
For each case, the team knows the desired:
- answer
- support category
- citation
- escalation decision
Now it can compare:
Prompt A
vs.
Prompt B
vs.
Prompt + RAG
vs.
Prompt + tuned model
vs.
Prompt + RAG + tuned model
Without evaluations, architecture decisions become opinions.
One engineer says:
“This feels better.”
Another says:
“The previous version sounded smarter.”
That is not enough.
Article #24 covered evaluation in detail for exactly this reason.
Prompt Engineering: Advantages
Prompt engineering is usually:
- fast
- inexpensive
- easy to iterate
- low infrastructure
- easy to experiment with
It is often the best place to start.
But prompts can become fragile as requirements accumulate.
Eventually you may see instructions like:
Always perform A except when B, unless C is true, but if D then use E…
That is a sign that some logic may belong in application code rather than another paragraph of prompt text.
RAG: Advantages
RAG gives an AI system access to:
- private knowledge
- large document collections
- current information
- traceable sources
But it adds another system to operate.
You now need to think about:
- ingestion
- chunking
- metadata
- embeddings
- search
- ranking
- access control
- document freshness
- duplicate content
Google’s current RAG infrastructure guidance explicitly treats ingestion, transformation, indexing, retrieval and observability as parts of the architecture rather than just “attach a vector database.”
Fine-Tuning: Advantages
Fine-tuning can improve:
- consistency
- specialized task performance
- domain-specific behavior
- classification
- structured responses
It can also sometimes make smaller models viable for specialized high-volume tasks.
But it introduces its own lifecycle:
Collect data
↓
Clean examples
↓
Train
↓
Evaluate
↓
Version
↓
Monitor
↓
Retrain when necessary
It is not a one-click upgrade.
Implementation availability also varies significantly by model provider, so teams should treat fine-tuning as an architectural concept rather than assume every frontier model supports the same tuning workflow.
Why RAG + Fine-Tuning Can Work Together
Suppose Atlas needs:
current Northstar documentation
and:
highly consistent Northstar support behavior
RAG can provide the current documentation.
Fine-tuning can improve how the model handles the support task.
Then prompts define the immediate instructions.
The architecture becomes:
System Instructions
↓
Retrieve Current Knowledge
↓
Specialized Model Behavior
↓
Business Rules
↓
Final Response
That combination is perfectly reasonable.
But each layer should have a purpose.
More Components Do Not Automatically Mean More Accuracy
There is an important trap here.
A team may think:
“Prompt + RAG + fine-tuning must be better than prompt alone.”
Not necessarily.
Extra retrieved information can introduce noise.
Poor fine-tuning data can reduce quality.
A retrieval layer can return irrelevant documents.
A longer prompt can introduce conflicting instructions.
OpenAI’s optimization guidance even documents cases where adding retrieval reduced performance because the task’s main problem was behavioral consistency rather than missing context.
The lesson is not that RAG is bad.
The lesson is:
Every optimization technique should solve an observed problem.
Digital Stackroom Decision Matrix
| Need | Prompt Engineering | RAG | Fine-Tuning | Tools/API |
|---|---|---|---|---|
| Change response tone | Excellent | Not needed | Strong | Not needed |
| Change output format | Excellent | Not needed | Strong | Not needed |
| Add current private knowledge | Limited | Excellent | Poor fit | Good |
| Produce source-backed answers | Limited | Excellent | Weak | Good |
| Use transactional data | Weak | Sometimes | Poor fit | Excellent |
| Take real-world actions | No | No | No | Excellent |
| Improve repeated specialized behavior | Moderate | Weak | Excellent | No |
| Enforce deterministic business rules | Weak | Weak | Weak | Application logic |
| Update knowledge frequently | Poor | Excellent | Poor | Excellent |
| Start quickly | Excellent | Moderate | Slowest | Moderate |
The table is not a law.
It is a diagnostic starting point.
The Simplest Question May Be the Best One
If you’re stuck between RAG and fine-tuning, ask:
Is the problem knowledge or behavior?
If the model needs information:
context problem → RAG
If the model has the information but behaves inconsistently:
behavior problem → prompting or fine-tuning
If the model needs a live value or needs to perform an action:
system problem → API/tool
This mental model resolves a surprising number of architecture debates.
A Five-Minute Design Exercise
Take one real failure from your AI application.
Write it down exactly.
Not:
“The chatbot isn’t accurate.”
Something concrete:
“It tells customers that the Business plan has unlimited audit-log retention.”
Then ask:
Did the model receive the correct information?
If no:
investigate retrieval/context.
Did it receive the right information but ignore the requested format or instruction?
Investigate prompting/behavior.
Does it repeatedly fail the same specialized task despite strong prompts?
Investigate fine-tuning.
Did it require data from another system?
Investigate tools/APIs.
Should the action always follow a fixed rule?
Put the rule in normal application logic.
That is architecture driven by failure analysis rather than fashion.
The Final Northstar Architecture
After several iterations, Atlas ends up with all four layers.
Prompt
Defines:
- tone
- response structure
- escalation behavior
- safety instructions
RAG
Supplies:
- product documentation
- pricing
- security policies
- troubleshooting guides
Fine-Tuned Behavior
Improves:
- ticket classification
- Northstar terminology
- consistent internal categorization
Tools
Provide:
- customer subscription status
- billing details
- support-ticket creation
- approved actions
And deterministic application logic handles rules that must never depend solely on model judgment.
That is what mature AI architecture often looks like.
Not one magical technique.
A set of components solving different problems.
Frequently Asked Questions
Is RAG better than fine-tuning?
Neither is universally better. RAG is generally more appropriate when the model needs external, private or frequently changing knowledge. Fine-tuning is more appropriate when the goal is to change or specialize model behavior.
Should I try prompt engineering before fine-tuning?
Usually yes. Good prompts and evaluations provide a cheaper baseline and help reveal whether the real problem is instructions, missing knowledge or behavioral consistency.
Can RAG and fine-tuning be used together?
Yes. A tuned model can still receive retrieved information. This can be useful when an application needs both specialized behavior and current external knowledge.
Can fine-tuning teach a model company knowledge?
It can influence model behavior using company examples, but frequently changing factual knowledge is often better handled through retrieval or APIs because it can be updated and traced more easily.
Does RAG eliminate hallucinations?
No. RAG can reduce unsupported answers by providing relevant evidence, but poor retrieval, irrelevant context or incorrect generation can still produce bad results.
When should I use an API instead of RAG?
Use APIs when the AI needs live structured data or must perform actions—for example, checking an account balance, submitting a request or changing a subscription.
Does fine-tuning remove the need for prompts?
No. Fine-tuned systems still use prompts and instructions. Training changes model behavior; prompts still define the current task and context.
Is a vector database required for RAG?
Not always. Retrieval can use vector search, keyword search, hybrid search, databases or other retrieval mechanisms. The goal is to provide the model with relevant external context.
Final Thoughts
Prompt engineering, RAG and fine-tuning are easier to understand once we stop treating them as competing upgrades.
They operate on different parts of the problem.
Prompt engineering asks:
What should the model do?
RAG asks:
What information should the model have?
Fine-tuning asks:
What behavior should the model learn more consistently?
Tools ask:
What data or action exists outside the language model?
And normal software asks:
Which rules should never be probabilistic in the first place?
Northstar’s biggest mistake would have been starting with:
“Should we use RAG or fine-tuning?”
The better starting point was:
Why exactly did Atlas fail?
Once that question was answered, the architecture became much easier.
The same principle applies to almost every production AI system.
Don’t choose the most sophisticated technique.
Choose the technique that fixes the failure you can actually measure.
Official Sources & Further Reading
Google Cloud — Prompt Engineering: Overview and Guide
Current guidance on using instructions, context, examples and structured prompts to influence model responses.
Google Cloud — What Is Retrieval-Augmented Generation (RAG)?
Overview of retrieval, external knowledge, grounding and the role of search quality in RAG systems.
Google Cloud — Develop a Generative AI Application
Architecture guidance covering prompt design, grounding, RAG and model tuning.
Google Cloud — Mastering Model Adaptation: A Guide to Fine-Tuning
Practical discussion of when prompt engineering is sufficient and when specialized model adaptation may be useful.
Google Cloud — RAG Infrastructure Architecture
Production architecture guidance covering ingestion, vector search, retrieval, generation, logging and monitoring.
OpenAI — Optimizing LLM Accuracy
A practical framework distinguishing context problems from behavior problems and comparing prompt engineering, RAG and fine-tuning.
OpenAI — Prompt Engineering
Guidance on instructions, examples, contextual information and retrieval-augmented generation.
OpenAI — Fine-Tuning Best Practices
Guidance on representative training data, training/test separation and evaluating model customization.