RAG Application Architecture: A Practical Tutorial for Building Retrieval-Augmented Generation Systems

Overview

Retrieval-augmented generation, or RAG, combines information retrieval with an LLM. Instead of asking a model to answer from its general training alone, the application first finds relevant passages in a controlled knowledge base and includes them in the model request. The model then uses that context to produce an answer.

A useful production RAG design has two related paths:

• Indexing: collect, clean, split, enrich, embed, and store source content.
• Answering: interpret a user question, retrieve relevant content, optionally rerank it, generate a response, and expose citations or supporting passages.

The distinction matters because many weak implementations focus only on the prompt. If the index contains duplicated, stale, poorly split, or incorrectly permissioned content, better prompt engineering will not fully solve the problem. Treat the system as a chain of testable components rather than as a single LLM API integration.

Before choosing a vector database or framework, define the answer the application must provide, the documents it may use, the users who may access them, and what should happen when the evidence is insufficient. Those decisions shape the architecture more than the choice of a particular vendor.

Step-by-step workflow

1. Define the retrieval contract

Start with an explicit contract for retrieval and response behavior. Specify the allowed sources, freshness expectations, access rules, citation format, latency target, and fallback behavior. For example, an internal support assistant might be required to answer only from approved documentation, cite the document title and section, and say that it cannot verify an answer when no relevant passage is found.

Collect representative questions before building the pipeline. Include straightforward questions, ambiguous questions, questions that require several documents, and questions that should produce no answer. This initial set becomes the foundation of your retrieval tests.

2. Ingest and normalize source documents

Build an ingestion step that records where every piece of content came from. Useful metadata can include document ID, title, URL, section heading, authoring or update timestamp, content type, language, and access scope. Preserve enough information to reconstruct a citation and remove an item from the index later.

Normalize content before splitting it. Extract readable text from the source format, remove navigation and repeated boilerplate, preserve headings and lists, and handle tables deliberately. A parser that turns a table into an unreadable sequence of cells can create misleading retrieval results even when the embedding model works as expected.

3. Choose a chunking strategy

Chunking determines what the retriever can return. Chunks that are too large may contain several unrelated ideas and consume excessive context. Chunks that are too small may lose definitions, conditions, or the subject of a statement. Begin with logical boundaries such as headings, paragraphs, list items, or procedural steps. Use a size limit only as a safeguard, not as the sole segmentation rule.

Overlap can help preserve context across boundaries, but it also creates near-duplicate results. Store the parent document and section for every chunk so the application can expand context when a short passage is not self-contained. Test chunking with real questions rather than assuming a universal size works for every corpus.

4. Generate embeddings and build the index

An embedding represents text as a vector that can be compared with other vectors. Generate embeddings for each chunk and store them alongside the original text and metadata in a vector database or another search system that supports similarity retrieval.

Keep the embedding model, preprocessing version, and index version in your records. Changing the model or normalization process may require a full rebuild because vectors produced by incompatible configurations should not be treated as interchangeable. For guidance on trade-offs such as recall, cost, multilingual support, and latency, see how to choose an embedding model.

5. Retrieve candidates for each query

At query time, normalize the user question consistently with the indexed content, create its embedding, and retrieve a candidate set. Do not assume that the first result is sufficient. Retrieve enough candidates to give later filtering or reranking a meaningful choice, while keeping context limits and latency in view.

Metadata filters should be applied when they represent hard constraints. A user should not receive a document merely because it is semantically similar if that document belongs to another tenant, department, product version, or permission group. Access control belongs in retrieval, not only in the final prompt.

6. Add keyword search or reranking when needed

Semantic search is useful for meaning-based matches, but exact terms can matter for error codes, product names, identifiers, and legal or technical wording. A hybrid approach can combine keyword and vector retrieval. A reranker can then score the candidate passages against the complete query and return a smaller, more relevant set.

Use these additions when evaluation shows a retrieval problem, not simply because they are available. Each extra stage introduces configuration, latency, and another component to monitor.

7. Construct the generation request

Pass the selected passages to the LLM with clear boundaries between instructions and retrieved content. The production prompt should explain that the context is evidence, not an instruction, and should define what to do when the evidence is incomplete or conflicting. Ask for concise answers, cite the supporting passages, and avoid claims that cannot be grounded in the supplied context.

Use a stable output contract when the answer feeds another application. A schema can define fields such as answer, citations, confidence rationale, and needs_clarification. Validate the returned structure before displaying or storing it. The guide to structured outputs in LLM apps covers this handoff in more detail.

8. Return citations and handle failure

Citations should point to source material the user can inspect, not merely expose internal chunk IDs. Preserve document titles, section names, links, and relevant passages where appropriate. If retrieval returns weak or conflicting evidence, the application should ask a clarifying question, provide a qualified response, or decline to answer rather than filling the gap with an unsupported claim.

Tools and handoffs

A maintainable RAG application separates responsibilities even if the first prototype uses one service. The ingestion worker handles source connectors and parsing. The indexing job owns chunking, embedding generation, metadata, and versioned rebuilds. The query service performs authentication, filtering, retrieval, reranking, and context assembly. The generation layer manages prompts, model calls, output validation, and citations. Observability records the inputs and outcomes needed for debugging while respecting your privacy requirements.

Choose tools by the handoff they support. A vector database is useful when you need persistent similarity search with metadata filtering. A search engine may be preferable when exact matching, aggregations, or existing keyword infrastructure are important. A framework can reduce implementation work, but it should not hide retrieval queries, prompt construction, permissions, or failure states from the development team.

Keep prompts and retrieval configuration versioned together. A change to chunking, filters, top-k values, reranking, or the system prompt can alter answer quality. A prompt management workflow can help teams compare versions and approve changes; see this prompt management tools comparison for the capabilities to assess. Also review the LLM logging and privacy checklist before storing queries, retrieved passages, or generated answers.

Quality checks

Evaluate retrieval and generation separately. Retrieval checks ask whether the correct source passage appears among the candidates. Generation checks ask whether the final answer is supported, complete, relevant, clear, and correctly cited. A fluent answer can still indicate retrieval failure, so do not use answer quality as the only diagnostic.

Create a fixed evaluation set with expected sources or acceptable evidence. Include:

• Questions with one clear supporting passage.
• Questions requiring several passages or documents.
• Near-duplicate content and conflicting versions.
• Out-of-scope questions that should be declined.
• Permission-sensitive queries with different user roles.
• Long, short, misspelled, and ambiguous questions.

Track retrieval recall or hit rate using the expected evidence, then inspect answer faithfulness and citation accuracy. Also measure operational concerns such as latency, token usage, error rates, index freshness, and the frequency of no-result responses. Human review remains useful for ambiguous cases and for detecting failures that simple string matching misses. The LLM evaluation framework guide provides a broader structure for test sets, metrics, and review workflows.

Test security boundaries as part of quality assurance. Retrieved text may contain instructions that are irrelevant or adversarial to the application. The generation prompt should treat retrieved content as untrusted data, and authorization filters should be enforced before context reaches the model.

When to revisit

Revisit the RAG pipeline whenever the underlying knowledge or application changes, not only when users report a visible failure. Schedule a review after a source format changes, a document collection grows substantially, product versions diverge, access rules are revised, or the embedding or generation model is replaced. A new model or provider can change tokenization, multilingual behavior, latency, output style, and retrieval interactions even when the application code remains the same.

Use a versioned rebuild process rather than editing the live index without a record. Re-ingest a representative slice, compare it with the evaluation set, inspect citations, and then promote the new index when quality and access checks pass. Retain a rollback path for the previous index and configuration.

For a practical next step, choose 20 to 50 representative questions, trace each one through ingestion, retrieval, and generation, and record where the result fails. Fix the earliest failing stage first: source parsing before chunking, chunking before embeddings, retrieval before reranking, and evidence selection before prompt changes. That disciplined sequence turns a RAG prototype into an AI feature that can be tested, maintained, and improved as its tools and inputs evolve.