What Is RAG? How Retrieval-Augmented Generation Works and When Your Product Needs It

RAG, short for retrieval-augmented generation, means that before an AI model answers a question, your system finds the passages in your own documents or data that match it and hands them to the model together with the question. The model answers from those passages and can show where each answer came from. You need RAG when answers must come from your own content and that content changes often. With us, a pilot for a feature like this costs $5,000-15,000 and takes 2-4 weeks.

Why plain AI models get your business wrong

A language model knows what was in its training data. It doesn’t know your price list, your refund policy, last week’s release notes or the contract you signed yesterday. Ask it anyway and it will often produce a fluent answer that sounds right and isn’t.

You could retrain the model on your documents, but that is slow and costly, and it has to be redone every time a document changes. RAG takes the other route: leave the model as it is and give it the right pages at the moment of the question.

The idea comes from a 2020 research paper by Patrick Lewis and colleagues, which paired a pre-trained language model with a searchable index of Wikipedia. Products today do the same with a help center, a policy library or a customer database.

How RAG works, step by step

1. Prepare: split your documents into short passages and store them in a search index. Often that is a vector index, which finds passages by meaning; often it is combined with keyword search.
2. Retrieve: when a question comes in, find the few passages that match it best, out of the ones this user is allowed to see.
3. Generate: send the question and those passages to the model, with an instruction to answer only from them and to say which passage each claim came from.
4. Check: log the question, the passages and the answer, so people can review what went wrong when something does.

Most of the engineering sits in the first two steps: how documents are split, what is stored with each passage (its source, its date and who may see it) and how matches are ranked. The model in step 3 is the easiest part to swap later.

When you need RAG and when you don’t

Your situation	What fits
Answers must come from your documents, and they change every week	RAG
You want the model to write in a set style or format	Better instructions first, fine-tuning if that isn’t enough
A few short documents that fit in one prompt	Put them in the prompt and skip the search step
Exact live numbers, like an order status or an account balance	Query your database directly, not a document search
General questions that don’t depend on your business	A plain model

Many products end up mixing these. A support assistant might use RAG for policy questions and a direct database lookup for “where is my order”.

A small example from our work

IPOPBOOKS, a bookkeeping platform we built for US small businesses, has a chat on its website that answers questions about plans and setup at any hour, from notes the founder can edit. When the founder changes a note, the next answer changes with it. No model has to be retrained. The idea is the same as in RAG: the answers come from content the business controls. The IPOPBOOKS case study covers the rest of the platform. For a first feature like this, a small set of documents that someone keeps up to date is worth more than a large archive nobody owns.

In GhostMinutes, an AI meeting notes app, a chat answers questions about each meeting and drafts the follow-up. Its answers are about that one meeting.

What to test in a RAG pilot

A RAG demo is easy. Trust is harder, and it comes from answers people can check. In the 2026 Stack Overflow Developer Survey, 48% of respondents said they trust AI when they can easily validate the answers, and only 16% trust it for most tasks that aren’t important work decisions. Showing the source of each answer is how a RAG feature earns that trust.

A RAG feature your customers can rely on needs these checks before launch:

• Retrieval: for a set of real questions, does the right passage come back? If it doesn’t, the answer is built on the wrong facts.
• Faithfulness: does the answer stick to the passages, and does it say “I don’t know” when they don’t cover the question?
• Access rules: one customer’s documents must never reach another customer’s answer. OWASP lists this under vector and embedding weaknesses and recommends permission-aware retrieval.
• Hidden instructions: a document can contain text that tries to steer the model. OWASP calls this indirect prompt injection and notes that RAG doesn’t fully prevent it, so test with booby-trapped documents.
• Freshness: how long does it take for an edited document to show up in answers?
• Cost and speed: what does one answer cost, and how long does it take?
• Data: where your documents go and which outside services see them. We agree this before the pilot starts.

If your users are in the EU, add one more check. The European Commission’s AI Act page says the Act’s transparency rules come into effect in August 2026, and its example is that people talking to a chatbot should be told they are dealing with a machine. A plain label on the chat window covers the obvious case. Ask a lawyer about the rest; this isn’t legal advice.

What RAG costs to run

The build is one bill. A live RAG feature also costs money every day, and those costs grow with use.

• Model calls: every answer sends the question and the retrieved passages to a model, so longer passages mean a bigger bill per answer.
• Keeping the index fresh: each new or edited document has to be split and indexed again.
• Hosting the search index and storing the documents.
• Review time: someone has to read a sample of answers every week, at least at the start.

Three habits keep these costs down. Retrieve fewer, shorter passages once tests show that the answers hold up. Cache answers to questions people ask again and again. And where a decision repeats, remember it: IPOPBOOKS sorts the next charge from a merchant a bookkeeper already approved from memory, with no AI call at all.

For the build itself, you can run your idea through our project estimate. It turns your brief into user stories, such as “a customer asks about plan limits and gets an answer with its source”. You confirm what is in scope, and every hour in the budget traces back to a story you kept. We check the hours the AI suggests against what our team actually books on finished projects, then multiply by our published rate of $39-55 an hour. Running costs such as model calls are a separate line, so list them next to the build budget.

Which of our services fits

RAG is a technique, and it shows up in different kinds of builds.

If you want a chat, a search box or an assistant inside your app that answers from your content, that is ChatGPT and LLM integration. We compare models on your real examples and pick by quality, speed and price, and our AI R&D department also fine-tunes models when instructions aren’t enough.

If the AI should also act, for example sort what comes in, draft a reply for a person to approve and pass the result to your other tools, that is an AI agent, and RAG is one of its tools. In IPOPBOOKS, AI suggests a category for each bank transaction with a reason, and a bookkeeper approves it before it reaches the books.

Our AI development page shows all of it in one place. Every project there starts the same way: a pilot for $5,000-15,000, ready in 2-4 weeks. If the pilot works, we price the production version then. Other work is $39-55 an hour.

What we would do

1. Pick one question type that real people ask often, such as plan questions from customers or policy questions from staff.
2. Collect the documents that answer them, and note who owns each one and who may see it.
3. Write down real questions with the right answers, to use as a test set.
4. Build retrieval and answers with sources, and measure both against the test set.
5. Add the access rules and the hidden-instruction tests before any customer sees it.
6. Look at accuracy and cost per answer, then decide on production.

FAQ

What is RAG in simple terms?

RAG, retrieval-augmented generation, is a way to make an AI answer from your own documents. Your system finds the passages that match a question, gives them to the model with the question, and the model answers from them, ideally showing its sources.

What is the difference between RAG and fine-tuning?

RAG gives the model the right documents at the moment of each question, so answers change as soon as the documents do. Fine-tuning trains the model further to change its style or behavior. Teaching it facts that way means training it again whenever the facts change. Many products need only RAG and good instructions.

Does RAG stop AI hallucinations?

It helps when the search finds the right passage, but it doesn’t guarantee a correct answer. The model can still misread a passage, and OWASP notes that RAG doesn’t fully prevent prompt injection. Test it on real questions with known answers before launch.

How much does a RAG pilot cost?

At Rocketech, a pilot costs $5,000-15,000 and takes 2-4 weeks. You pick one task, we build it and test it on real examples with your team, and then you decide whether to take it further.

Is our data safe in a RAG system?

It depends on the setup, so we agree where your data goes before the pilot starts. During the IPOPBOOKS build, only one person at Rocketech could see client data, and Subflare keeps all its data on AWS in Europe.