Retrieval (RAG)
AI that looks things up in your own documents before it answers, and shows where each answer came from.
What it is
Retrieval-augmented generation, or RAG, lets a language model answer from your organisation's own content: policies, procedures, product sheets, contracts and case notes. Instead of relying on what the model learnt in training, the system first searches your content, then gives the model the most relevant passages and asks it to answer from those alone, with references.
It has two halves. Ingestion splits documents into passages, records where each came from and who may see it, and indexes them for both keyword and meaning-based (vector) search. At question time, the system searches, ranks the results, removes anything the user isn't allowed to see, and passes the best passages to the model, which writes an answer and cites them.
In an enterprise, RAG is usually the knowledge layer behind something else: a staff assistant, a suggested reply for a contact-centre adviser, or an agent checking what a policy covers. Most of the work isn't the model. It's getting the content clean, current and owned, and making sure search respects the same permissions as the system the documents came from.
Use it when, and think twice when
Use it when
- The answers already exist in documents people can't find quickly
- Content changes often, so retraining a model would never keep up
- People need to see the source to trust the answer
- You can name an owner for each body of content
Think twice when
- The content is out of date or contradicts itself
- The answer needs a calculation or live data, not a document
- Document permissions are messy and can't be mirrored in the index
- A wrong answer would be acted on without anyone checking the source
Reference architecture
01Channels
- Staff assistant or intranet
- Customer chat
- Another agent or app
02Intake
- API GatewayAPI ManagementKong, or your existing API gatewayQuestion API
- Amazon S3Blob StorageCeph, or IONOS S3 Object StorageContent store
03Intelligence
- Bedrock Knowledge BasesFoundry IQ (Azure AI Search)OpenSearch, or PostgreSQL with pgvectorSearch and rank
- Amazon BedrockMicrosoft FoundryOpen-weight models served with vLLMEmbed and answer
- Bedrock GuardrailsFoundry guardrails (Azure AI Content Safety)Open-source guardrails (e.g. NeMo Guardrails, Llama Guard)Input and output checks
04Orchestration
- AWS Step FunctionsLogic Apps or Durable FunctionsTemporal, or n8nIngestion pipeline
- Amazon EventBridgeEvent GridApache Kafka or RabbitMQDocument change events
05Systems of record
- Document management and intranet
- Policy and product library
- Directory: who may see what
06People
- Person
Content owners keep sources current - Person
Users flag wrong answers
- Person
- Cloud or platform service (AWSAzuresovereign / on-prem)
- AI agent
- A person decides
- Your existing systems
- Outside your organisation
Components, side by side
| Layer | AWS | Azure | Sovereign / on-prem |
|---|---|---|---|
| Content storeOne copy of each source document, with its owner, version and access list. | Amazon S3 | Blob Storage | Ceph, or IONOS S3 Object Storage |
| Parsing and chunkingTables and scanned pages need proper parsing, or the answers will be wrong. | Bedrock Knowledge Bases parsing, or Bedrock Data Automation for complex layouts | Azure AI Search indexers, with Azure Content Understanding for complex layouts | Open-source parsing (e.g. Docling) |
| Vector and keyword searchHybrid search (keywords plus meaning) handles policy numbers and product codes better than either alone. | Bedrock Knowledge Bases, or Amazon OpenSearch Service | Foundry IQ, built on Azure AI Search | OpenSearch, or PostgreSQL with pgvector |
| Answer generation | Amazon Bedrock | Microsoft Foundry | Open-weight models served with vLLM on your own GPUs, or IONOS AI Model Hub |
| PermissionsThe index must never return a passage the user couldn't open in the source system. | IAM Identity Center, with access filters on every query | Entra ID, with security filters on every query | Keycloak or your existing Active Directory, with filters on every query |
| Guardrails and grounding checksCheck that the answer is supported by the passages retrieved. | Bedrock Guardrails | Foundry guardrails (Azure AI Content Safety) | Open-source guardrails (e.g. NeMo Guardrails, Llama Guard) |
| Freshness pipelineRe-index when a document changes, and remove it when it's withdrawn. | Amazon EventBridge, AWS Step Functions | Event Grid, Logic Apps or Durable Functions | Apache Kafka; Temporal |
| Monitoring and auditLog the question, the passages used and the answer. | CloudWatch, CloudTrail | Azure Monitor, Log Analytics | Prometheus, Grafana, OpenTelemetry |
| UK data residencyCheck that each service and model you pick is offered in your UK region. | eu-west-2 (London) | UK South / UK West | Your own UK data centre, or a UK or EU provider |
On a phone, the table shows the platform you pick. Sovereign / on-prem means open, portable components you can run in your own data centre or with a UK or European provider; IONOS is shown as one example, and others exist. What sovereign cloud means →
Operations
Keeping it reliable
AI fails quietly. These three habits make failure visible before customers see it.
Evaluation
Test search and answers separately
Build a set of real questions with the right answer and the right source document. Measure whether search finds the source first, then whether the answer is correct and cites it. Most failures are search failures, so fix search before changing the model.
Monitoring
Watch what people can't find
Log questions with no good match, answers users flag, and documents never retrieved. Unanswered questions show gaps in your content. A document that suddenly stops being retrieved usually means a broken sync.
Cost
Index once, query cheaply
Embedding and indexing cost most at the start and on large re-indexes. Query cost depends on how much text you send to the model, so send fewer, better passages. Search services charge for capacity even when idle, so size them to real use.
Security and failure risks
Prompt injection through content
A document in the index contains instructions, and the model follows them when that passage is retrieved.
How the design avoids it
Index only owned sources, treat retrieved text as data, run guardrails on inputs and outputs, and give the answering step no tools that change anything.
Permission leakage
The index holds everything, so a user sees a passage from an HR file or board paper they couldn't open in the source system.
How the design avoids it
Carry each document's access list into the index and filter every query by the user's identity. Test it with accounts that should be refused.
Confident answers from old content
A withdrawn policy is still in the index, and the AI quotes it accurately and confidently.
How the design avoids it
Re-index on change, remove withdrawn documents automatically, show each source with its date, and give every collection an owner.
Answers beyond the sources
The model fills a gap with general knowledge that sounds right but isn't your policy.
How the design avoids it
It's told to say it can't find the answer when the sources don't cover it, grounding is checked before replying, and the 'not found' rate is reported.
Playbooks that use this pattern
All 39 playbooks using Retrieval (RAG) →Complaints handling
Every complaint caught, whatever channel it came through, answered fairly within the DISP deadlines, and traced back to its root cause. A handler writes and signs every final response.
- Retrieval (RAG)
- Agents with tools
- Speech and voice
- Human in the loop
Spotting vulnerable customers
Possible signs of vulnerability noticed the first time a customer mentions them, a trained person offered the right support, and outcomes checked so no group of customers is left worse off.
- Retrieval (RAG)
- Predictive ML
- Speech and voice
- Human in the loop
Underwriting submission review
Every broker submission read, its risk data extracted and checked against appetite on the day it arrives, with gaps flagged and a referral note drafted. The underwriter decides, prices and speaks to the broker.
- Document AI
- Retrieval (RAG)
- Agents with tools
- Human in the loop
Referral and clinical letter processing
Every referral and letter read, summarised and routed within minutes of arriving. The consultant sees a one-screen summary and a suggested priority, and makes every triage and clinical decision.
- Document AI
- Retrieval (RAG)
- Agents with tools
- Human in the loop
- Event-driven integration
Patient enquiry assistant
Routine questions answered straight away, by phone or web chat, from content the trust has approved. Anything clinical or urgent goes to a person, with clear signposting to NHS 111 and 999.
- Retrieval (RAG)
- Speech and voice
- Agents with tools
- Human in the loop
Care notes and incident analysis
Better care notes in less time, often by voice, and early warning of patterns such as falls, weight loss, missed medicines and safeguarding signs, put in front of a manager who decides what to do.
- Speech and voice
- Predictive ML
- Human in the loop
- Retrieval (RAG)
Citizen casework and correspondence
Every letter and email read, sorted and routed on the day it arrives, with a sourced draft reply waiting for an official to edit and sign. People decide what the department says and what happens to each case.
- Retrieval (RAG)
- Document AI
- Agents with tools
- Human in the loop
Freedom of Information request handling
Every request logged, checked and on the clock the day it arrives, records found faster, and suggested exemptions and redactions laid out with reasons. The FOI officer decides what is released.
- Retrieval (RAG)
- Document AI
- Agents with tools
- Human in the loop
Benefits and grants application review
Every application read, checked against the scheme rules and made ready for an officer within a day. Missing evidence asked for early, fraud and error checks done fairly, and an officer deciding every award and every refusal.
- Document AI
- Agents with tools
- Retrieval (RAG)
- Human in the loop
Patterns
All eight building blocks
Document AI
Read forms, letters and PDFs into structured data.
Retrieval (RAG)
Answer questions from your own documents, with sources.
Agents with tools
AI that takes small, controlled actions across your systems.
Human in the loop
Put people on the decisions that matter, with the evidence.
Predictive ML
Score, rank and forecast from your own historical data.
Event-driven integration
Connect AI to core systems safely, through events and queues.
Computer vision
Inspect photos, video and camera feeds for defects and damage.
Speech and voice
Turn calls and conversations into text, notes and actions.
Thinking about retrieval (rag) in your organisation?
Talk through where it fits, what it would plug into, and which decisions should stay with people.
