Software architecture
RAG architecture
RAG architecture, or retrieval-augmented generation, is a system design that fetches relevant documents and gives them to a language model to ground its answer.
Teams build RAG systems because a language model only knows what it was trained on, and what they need it to answer about is their own documentation, tickets and contracts. Retrieval puts the right passages in front of the model at the moment it answers, which keeps the knowledge current and lets every answer show where it came from.
Why does RAG architecture matter?
The model is the easiest part of a RAG system to swap. The parts that decide whether answers are correct are the ones around it.
- Current knowledge. Re-index a document and the next answer reflects it. No retraining.
- Citations. The model is told which passages it was given, so the answer can link to them.
- Access control. Retrieval can filter by who is asking, so the model never sees documents the user may not.
What are the parts of a RAG architecture?
The hero diagram shows an internal documentation assistant. Every RAG system has the same two paths.
Ingestion pipeline
- Document loading. Pull text out of wikis, PDFs, tickets and code, with the metadata needed for filtering: source, owner, date, access level.
- Chunking. Split each document into passages small enough to embed and retrieve on their own, usually a few hundred tokens with some overlap.
- Embedding. Run each chunk through an embedding model to get a vector that captures its meaning.
- Vector store. Index the vectors with the chunk text and metadata so similar chunks can be found quickly.
Query path
- Query embedding. Embed the question with the same model used at ingestion. A different model produces vectors in a different space and retrieval silently fails.
- Retrieval. Find the top k chunks nearest the question, often filtered by metadata first.
- Reranking. Score the candidates with a more accurate model and keep the best few. Optional, but it fixes many near misses.
- Prompt assembly and generation. Build a prompt from the instructions, the selected passages and the question. The model answers from the context and cites the passages it used.
How does RAG work?
Follow the two paths in the diagram at the top of this page. Ahead of time, the ingestion pipeline takes the engineering wiki, splits each page into chunks, embeds them and writes the vectors into the vector store. It runs again whenever a page changes.
When an engineer asks “how do I rotate the signing key?”, the query path embeds that question, asks the vector store for the twenty nearest chunks and sends them through a reranker that keeps the five most relevant. Those five, with the question, become the prompt. The model writes the steps and cites the two wiki pages they came from. If the top chunks had been about a different key, the answer would be wrong however good the model, which is why retrieval gets most of the work.
What are the types of retrieval in RAG?
| Approach | How it finds passages | Strength | Weakness |
|---|---|---|---|
| Keyword search | Matches terms, with scoring such as BM25 | Exact names, codes and error strings | Misses synonyms and paraphrase |
| Vector search | Nearest embeddings by meaning | Questions phrased differently from the text | Misses rare exact terms |
| Hybrid search | Runs both and merges the ranked lists | The best of each, which is why most production systems use it | Two indexes to maintain |
How do you design a RAG architecture?
- Collect real questions. Twenty questions people actually ask, with the passages that answer them, become the evaluation set for everything that follows.
- Choose the sources and their metadata. Decide what gets indexed, how often it changes and what filters retrieval needs.
- Pick a chunking strategy. Start with sections or paragraphs of a few hundred tokens, then adjust using the evaluation set.
- Choose the embedding model and store. Use the same model on both paths and a store that supports the filters you need.
- Add hybrid search and a reranker. Both are cheap and both usually improve retrieval more than a bigger model does.
- Design the prompt and iterate. Tell the model to answer only from the context and to cite passages, then re-run the evaluation set after every change.
What are common RAG architecture mistakes?
- Chunking by character count. Splitting mid-sentence or mid-table produces passages that mean nothing on their own. Split on structure.
- Different embedding models on the two paths. Vectors from different models do not compare. Pin the version and re-index when it changes.
- Retrieving too much. Thirty passages in the prompt bury the right one. Retrieve many, rerank, send few.
- No access filtering. If retrieval ignores permissions, the model will summarize a document the user should not see.
How does RAG compare with other approaches?
| Approach | What it does | Use it when |
|---|---|---|
| RAG | Retrieves passages at query time and grounds the answer in them | Knowledge changes, is private or must be cited |
| Fine-tuning | Trains the model on examples to change its behavior | You need a style, format or task the base model does poorly |
| AI agent architecture | Lets a model plan and call tools in a loop | The task needs actions, not just an answer; RAG is often one of the tools |
RAG beats fine-tuning whenever the question is about facts rather than behavior. A model fine-tuned on this month’s documents half-remembers them and cannot say where anything came from. The two combine well: a model fine-tuned to follow your answer format, with RAG supplying the facts.
In production the two paths are deployed separately. Ingestion runs as a batch or event-driven job whenever documents change, and the query path runs as a service behind an API gateway, often in a microservices architecture.
What tools help with RAG architecture?
- RAG frameworks. Libraries that provide loaders, chunkers and retriever interfaces so you can assemble a pipeline quickly.
- Evaluation tools. Services that run a question set through the pipeline and score retrieval and faithfulness, which is the only way to know whether a change helped.
- AI diagramming tools. Tools like Eraser turn a description of your pipeline into an architecture diagram you can keep editing, so both paths, the stores and the model calls are in one picture for review.
For more on this kind of diagram, see architecture diagrams.
Frequently asked questions
-
What does RAG stand for?
RAG stands for retrieval-augmented generation. Retrieval is the search step that finds relevant passages, and generation is the language model writing an answer from them. The term comes from a 2020 research paper and now covers any system that puts retrieved text into a prompt.
-
What is the difference between RAG and fine-tuning?
RAG gives the model facts at query time, so the knowledge can change daily and every answer can cite its source. Fine-tuning changes the model's weights with examples, which teaches style and format but is a poor way to store facts. Systems that need current, private or citable knowledge use RAG, sometimes with a fine-tuned model inside it.
-
Do you need a vector database for RAG?
Not always. Vector search finds passages by meaning, which keyword search misses, but a relational database with a vector extension or a search engine with hybrid support often handles a few million chunks. A dedicated vector database earns its place at larger scale or for built-in filtering and multi-tenancy.
-
How do you evaluate a RAG system?
Evaluate the two paths separately. For retrieval, build a set of questions with known relevant passages and measure how often they appear in the top results. For generation, judge whether the answer is faithful to the context and answers the question. Retrieval failures cause most bad answers, so test it first.
