Retrieval-Augmented Generation (RAG) improves an LLM’s answer by first retrieving relevant information from an external knowledge source and adding it to the prompt, so the model answers from that context instead of only from its training memory.
Main steps of a RAG pipeline
- Ingest: collect documents (policies, manuals, FAQs, PDFs) and clean them.
- Chunk: split them into passages of a few hundred tokens, with some overlap.
- Embed: turn each chunk into a vector with an embedding model.
- Index: store vectors with their text and metadata in a vector database.
- Retrieve: embed the user’s question and find the top-k most similar chunks (often hybrid with keyword search, then re-ranked).
- Augment: build a prompt from the instructions, retrieved chunks and the question.
- Generate: the LLM writes the answer, ideally citing its sources.
- Evaluate and update: measure retrieval and answer quality; re-index when documents change.
Role of the vector database (e.g. Pinecone, Milvus, pgvector, FAISS): stores embeddings and does fast approximate nearest-neighbour search by cosine similarity, so passages with the same meaning are found even when the words differ. It also filters by metadata (department, date, access rights) and scales to millions of chunks.
How RAG improves accuracy and relevance
- Grounds answers in real, current documents, which reduces hallucination.
- Gives up-to-date and private/domain knowledge without retraining the model.
- Answers can cite sources, so they can be verified.
- Updating knowledge is cheap: re-index the documents instead of fine-tuning.
- Access control at retrieval keeps users to what they are allowed to see.
— rhs