LLMs & Retrieval (RAG)
Retrieval-augmented generation (RAG) finds the right extracts from your own documents and data and gives them to a large language model before it answers. The result is answers grounded in your information, with sources you can check.
Instead of answering from memory, the AI first looks up the right pages in your organisation's files and then answers using what it found.
The short version, in a few minutes.
A plain-English walkthrough of this page. Prefer reading? Everything in the video is covered below.
What it is
A large language model (LLM) learns general knowledge during training, but it knows nothing about your contracts, policies, products or customers, and its knowledge stops at a certain date. Retrieval-augmented generation fixes this by searching your own information first, then handing the most relevant passages to the model along with the question.
It is the technology behind most “ask our documents” assistants, internal knowledge search, support tools that quote policy and many AI features in business software. For keeping a model up to date with your facts, it is usually cheaper, quicker to update and easier to control than retraining the model on your data.
How it works
First, your content is prepared. Documents are split into passages, and each passage is turned into an embedding: a list of numbers that captures its meaning. These are stored in a vector index. When someone asks a question, the question becomes an embedding too, and vector search finds the passages closest in meaning, even if they use different words. Good systems usually combine this with ordinary keyword search, then rank the results so the best come first.
The top passages are added to the model's input, with an instruction to answer from them and cite them. Live facts, such as stock levels or order status, are better fetched directly from your systems through an API or database query than from documents. Models can now read far more text at once than they used to, but retrieval still matters: it keeps costs down, keeps answers current and controls what each person is allowed to see.
Where it creates real value
- Internal knowledge. Let staff ask questions of policies, procedures and manuals and get an answer with the source attached.
- Customer and service teams. Draft accurate replies from product information, terms and past cases.
- Technical documents. Find the right section of a specification, contract or standard in seconds.
- Onboarding. Help new starters find answers without interrupting colleagues for every question.
- Bids and compliance. Pull together approved wording and evidence from previous work, with references.
Where it's commonly misapplied
- Assuming it cannot make things up. RAG reduces invented answers but does not eliminate them, because the model can still misread, mix up or go beyond the passages it is given.
- Ignoring permissions. If retrieval does not respect who may see what, the assistant will happily show salary files or board papers to anyone who asks.
- Pointing it at a mess. Out-of-date, duplicated and contradictory documents produce confident, contradictory answers.
- Using documents for live numbers. Questions like “how much stock do we have?” should be answered from your live systems, not from a document that was true last month.
- Never measuring quality. Without a set of test questions with checked answers, nobody knows whether a change made things better or worse.
Good answers start with good retrieval.
Most RAG quality problems are retrieval problems, not model problems. Decide which sources are in scope and who owns them, how often they are refreshed, and how documents are split and labelled. Choose between a dedicated vector database, the search features of a database you already run, or the retrieval built into a platform you already use. Combine meaning-based and keyword search, and keep the index in step with the sources so deleted or replaced documents disappear from answers.
Security needs particular care. Enforce access control at the moment of retrieval, using the same permissions as the source system, so each person only sees passages they are allowed to see. Treat retrieved content as untrusted, because a document can contain hidden instructions. Log questions, sources and answers for review. Costs grow with usage and with the amount of text sent to the model, so track cost per answer as well as quality.
What this means for your organisation.
RAG is often the quickest route to real value from generative AI, because it works with information you already have. The hard part is not the technology. It is deciding which content is authoritative, removing what is out of date and making sure someone owns each source.
Set expectations honestly. Staff should treat answers as a well-referenced first draft and check the source for anything that matters. Make it easy to flag a wrong answer, and use those reports to fix the content or the retrieval.
Questions to ask your team or vendor
- Which sources does it search, and who is responsible for keeping them accurate?
- Does it respect the same access permissions as the original systems?
- Does every answer show its sources, and can people open them?
- How do we measure answer quality, and what happens when it gets something wrong?
- How quickly do changes and deletions in the sources reach the assistant?
- Where is our data stored and processed, and is any of it used to train the vendor's models?
How it connects to the rest.
Generative AI
RAG grounds generative AI in your own information instead of its general training.
Explore →Natural Language Processing
Embeddings and meaning-based search grew out of decades of natural language processing.
Explore →Data Foundations for AI
Retrieval is only as good as the content and data it searches.
Explore →Agentic AI & AI Agents
Agents use retrieval to find the facts they need before taking action.
Explore →I help you design retrieval that respects permissions, connects to the systems you already run and can be measured. Start with AI adoption, or see how I approach AI security and data leakage.
Does RAG stop AI making things up?
It reduces made-up answers, but it does not stop them. The model can still misread a passage, combine sources wrongly or fill gaps with guesses. Showing sources, and checking important answers, remain essential.
Do we need to train our own model to use our data?
Usually not. For most question-and-answer and search tasks, retrieval with a standard model is cheaper, easier to keep current and easier to control. Fine-tuning is better suited to teaching a model a style or a specialised task than to teaching it facts.
Is it safe to put confidential documents into a RAG system?
It can be, with the right design: access control enforced at retrieval, clear data processing terms with your provider, encryption and logging. Without those, it can expose information to the wrong people, so security has to be part of the design from day one.