Large language models write fluently, but they only know what was in their training data, and that knowledge has a cut-off point. Ask about a company's latest returns policy or a document written last week and the model may simply guess. Retrieval Augmented Generation, usually shortened to RAG, is a widely used design pattern that addresses this by letting the model look things up before it answers, a bit like a student allowed to bring notes into an exam.
Two halves of one system
RAG combines a retriever and a generator. The retriever searches a collection of documents for passages related to the question. The generator, a language model, receives the question together with those passages and writes a response grounded in them. Neither half is new on its own; the value comes from wiring them together so the model works from relevant, current material instead of memory alone.
Where vector databases fit
Keyword search struggles when the question and the answer use different words. "How do I send something back?" and "returns procedure" mean the same thing but share almost no vocabulary. Vector search handles this with embeddings: numerical representations of text, produced by a model, in which passages with similar meaning end up close together in a high-dimensional space.
A vector database stores those embeddings and is built to find nearest neighbours quickly, even across very large collections. When a question arrives, it is embedded with the same model, and the database returns the passages whose vectors sit closest to it, ranked by a similarity measure such as cosine similarity.
The pipeline step by step
- Ingest: collect documents such as manuals, help articles, policies or notes.
- Chunk: split them into passages small enough to be specific but large enough to keep context.
- Embed: turn each chunk into a vector and store it with its source text and metadata.
- Retrieve: embed the user's question and fetch the closest chunks, optionally filtered by date, product or language.
- Generate: pass the question and retrieved chunks to the model with instructions to answer from them and cite sources where possible.
Each stage involves choices. Chunk size, overlap between chunks, the embedding model and the number of passages retrieved all affect answer quality, and teams typically test several combinations against real questions.
Where it shows up
Customer support assistants use RAG to answer from a help centre rather than improvising. Internal search tools let staff query handbooks and project documentation in plain language. Research assistants pull passages from large libraries of papers or reports, and developer tools retrieve snippets from codebases and documentation. In each case, updating the knowledge means adding or replacing documents, not retraining the model.
Limits worth keeping in mind
Retrieval reduces guessing but does not eliminate it. If the right passage is missing from the collection, or the retriever overlooks it, the model can still produce a confident but wrong answer. Poorly chunked documents return fragments without context. Permissions matter too: a system that indexes sensitive files must respect who is allowed to see them. Well-run RAG systems are evaluated continually, with sample questions, source citations users can check, and a clear fallback when nothing relevant turns up.
Tell us what you think.
We read every message.
