In plain English
A large language model on its own knows only what it learned in training. Ask it about your holiday policy or your payment terms and it will guess. RAG fixes that by adding a search step. Your documents are indexed so the system can quickly find the passages that relate to a question. Those passages are handed to the AI along with the question, and it answers from them.
Taking the name apart: retrieval means finding the right pages, augmented means adding them to the question, and generation means the AI writing the answer. It’s the difference between asking a new colleague something off the top of their head and asking them after handing over the right file.
Microsoft calls its version “grounding”. With the paid Microsoft 365 Copilot licence, Copilot looks for relevant content across your emails, chats and files before it answers.
An example from a small business
Picture a hypothetical 25-person accountancy practice with a staff handbook, a procedures manual and years of internal how-to notes. New starters ask the same questions again and again: how do I record time on a fixed-fee job, who signs off a VAT return, where’s the engagement letter template? A knowledge assistant built on RAG answers from those documents and links to the relevant section. When the answer isn’t in the documents, it says so instead of guessing, and the question is logged so someone can write the missing guidance.
What to watch out for
- It’s only as good as your documents. Out-of-date or contradictory procedures produce out-of-date or contradictory answers, so a tidy-up usually comes first.
- Permissions matter. The assistant should only answer from documents the person asking is allowed to see.
- It reduces mistakes, it doesn’t remove them. Answers that cite their source are much easier to check, so insist on citations (see AI hallucination).
- Know where your documents go. Your content is processed by the AI service, so check the data processing agreement.
Related terms
- Large language model (LLM): the AI that writes the answer.
- AI hallucination: the problem RAG helps to reduce.
- Microsoft 365 Copilot: Microsoft’s assistant that grounds answers in your files.
- Data processing agreement (DPA): the contract covering your documents.
Where AlphaOps can help
A company knowledge assistant, answering staff questions from your own handbook and procedures, is part of the standard pack on our Monthly AI Support plans. See our AI assistant page for how it works, or book a free automation audit.