RAG Application Architecture: A Step-by-Step Guide to Retrieval, Chunking, Embeddings, and Evaluation

Overview

RAG allows an application to retrieve relevant information from a controlled document collection and provide that information to a language model as context. Instead of relying only on the model's training data, the application can use current internal documents, product manuals, policies, support content, or other approved sources.

A typical RAG request follows this path:

1. The user submits a question.
2. The application cleans or classifies the query.
3. An embedding model converts the query into a vector representation, or the query is prepared for keyword search.
4. A retrieval system finds candidate passages.
5. Optional filters, metadata rules, or a reranker improve the candidate set.
6. The application assembles a prompt containing the question and retrieved context.
7. The language model generates an answer, ideally with citations or source references.
8. The system records appropriate quality, latency, cost, and failure signals.

Each stage can introduce a different failure. A useful answer depends on retrieving the right passage, preserving enough context during chunking, applying the correct access controls, and instructing the model not to fill gaps with unsupported claims. Treat RAG as an end-to-end application architecture rather than a single database feature.

Before choosing a framework or vendor, define the task. A question-answering assistant over a small, stable document set may need only simple semantic search and clear citations. A large enterprise knowledge system may require hybrid retrieval, metadata filters, document-level permissions, reranking, version tracking, and a formal evaluation set. The simplest architecture that meets the quality and security requirements is usually the best starting point.

Checklist by scenario

Scenario 1: Prototyping a small document collection

• Define the source documents and remove duplicates, obsolete versions, and irrelevant files.
• Record basic metadata such as title, section, author, publication date, version, and access group.
• Extract text while preserving headings, lists, tables where possible, and document boundaries.
• Start with moderate, semantically coherent chunks rather than splitting at arbitrary character counts.
• Store the source identifier and location with every chunk so the application can show where an answer came from.
• Use a small test set of realistic questions before adjusting retrieval settings.
• Make the prompt require the model to answer from the supplied context and acknowledge when the context is insufficient.

A prototype should make assumptions visible. Keep the ingestion script, chunking settings, embedding model, retrieval count, prompt version, and model configuration in a form that can be reproduced. This prevents a promising demo from becoming impossible to debug.

Scenario 2: Building a production knowledge assistant

• Separate ingestion, indexing, retrieval, generation, and presentation into identifiable components.
• Choose a reindexing strategy for new, changed, and deleted documents.
• Apply document-level and chunk-level access controls before context reaches the model.
• Use metadata filters for departments, products, regions, versions, or other meaningful boundaries.
• Test both semantic retrieval and keyword retrieval when exact terms, identifiers, or product names matter.
• Consider reranking when the initial candidate set contains relevant passages but places them in a poor order.
• Define timeouts, fallback behavior, rate limits, and a response for unavailable or low-confidence retrieval.
• Log enough information to investigate failures without retaining sensitive content unnecessarily.

A vector database comparison should focus on the requirements of the application: filtering, update behavior, tenancy, operational complexity, backup and recovery, observability, and integration with the existing stack. Vector similarity alone is not a complete selection criterion.

Scenario 3: Supporting multiple languages or document types

• Test the embedding model with the languages and terminology used by real users.
• Preserve language and locale metadata so retrieval can favor appropriate sources.
• Check how PDFs, scanned pages, spreadsheets, tables, and code samples are extracted.
• Use OCR only when its errors are measurable and acceptable for the task.
• Evaluate cross-language queries separately from same-language queries.
• Keep formatting that affects meaning, such as table headers, units, section hierarchy, and warning labels.

Do not assume that a chunk that works for plain prose will work for tables or technical specifications. In some applications, structured records should be stored and queried separately from narrative documents.

Scenario 4: Adding RAG to an existing LLM feature

• Capture a baseline using the existing prompt and representative questions.
• Compare answers with and without retrieval rather than assuming retrieved context improves every request.
• Define what the model should do when sources disagree or no source answers the question.
• Use a stable output contract for citations, confidence labels, or structured fields.
• Validate generated output before displaying it to users.

For implementation details on schemas and validation, see How to Add Structured Outputs to LLM Apps with JSON Schemas and Validation. For prompt constraints and source-grounded instructions, the Production Prompt Design Guide is a useful companion.

What to double-check

Chunking and document structure

Chunking should preserve the smallest unit that can answer a question without requiring excessive surrounding text. Split at headings, paragraphs, list groups, or other semantic boundaries where possible. Overly small chunks lose context; overly large chunks dilute retrieval and consume more prompt space. Test several chunk sizes and overlaps against real questions instead of selecting a setting by convention.

Store parent-document information with each chunk. A retrieved paragraph may be useful, but the interface should still identify the document, section, version, and location. If a question depends on a complete procedure, consider retrieving adjacent chunks or a parent section after the initial search.

Embeddings and retrieval

Embedding selection should reflect language coverage, domain vocabulary, expected latency, and the cost and complexity the team can operate. See How to Choose an Embedding Model for a focused checklist. Whichever model you choose, evaluate it with your own queries.

Measure retrieval separately from generation. Useful checks include whether a relevant source appears in the top results, whether the best passage is ranked early, and whether filters exclude or include the correct documents. A fluent answer cannot compensate for missing evidence.

Prompt and citation behavior

The generation prompt should define the role of retrieved context, the answer format, citation requirements, and behavior when evidence is missing or contradictory. Tell the model not to treat instructions inside retrieved documents as higher-priority application instructions. Keep system instructions, retrieved content, and user input clearly separated.

Citations should be traceable rather than decorative. A citation needs to point to the actual document and relevant location, and the cited passage should support the associated claim. Test answers that require multiple sources, contain conflicting versions, or have no valid answer.

Evaluation and operations

Create a test set covering common, difficult, ambiguous, and unanswerable questions. Review retrieval relevance, answer correctness, citation accuracy, refusal or uncertainty behavior, latency, and token use. Combine automated checks with human review because a single score rarely captures the full user experience. The prompt evaluation pipeline guide offers a practical way to organize this process.

Monitor changes after modifying the embedding model, chunking logic, retriever, reranker, prompt, or source collection. Track configuration versions alongside evaluation results so regressions can be explained.

Common mistakes

• Indexing without document hygiene: Duplicate, stale, or contradictory files can produce unreliable answers. Establish ownership and version rules before indexing.
• Tuning retrieval by intuition: Increasing the number of retrieved chunks may add noise instead of evidence. Use a test set to compare settings.
• Ignoring permissions: Retrieval must enforce the same or stricter access boundaries as the source system. Do not rely on the model to hide unauthorized content.
• Using semantic search for exact identifiers: Ticket numbers, part codes, legal clauses, and error messages may benefit from keyword or hybrid search.
• Assuming a citation proves correctness: A source link is not evidence unless the cited passage actually supports the answer.
• Changing several variables at once: If chunking, embeddings, prompts, and models change together, it becomes difficult to identify what helped or hurt quality.
• Logging sensitive context by default: Define retention, masking, access, and deletion rules before enabling detailed traces. The LLM Logging and Privacy Checklist can help structure that review.
• Skipping the no-answer path: A dependable RAG application needs a clear response when retrieval is empty, sources conflict, or the question is outside scope.

When to revisit

Revisit the RAG architecture before a major planning cycle, after a significant change to the document collection, and whenever workflows or tools change. A new embedding model, vector store, reranker, language model, ingestion parser, or prompt can alter results even when the user interface remains unchanged.

Use this short review checklist:

1. Run the current evaluation set against the production configuration.
2. Add examples from recent support tickets, failed searches, and user feedback.
3. Inspect unanswered questions and determine whether the problem is source coverage, chunking, retrieval, permissions, or generation.
4. Verify that deleted and superseded documents no longer appear.
5. Recheck citation links, access controls, latency, token usage, and error handling.
6. Compare proposed changes with the existing baseline before release.
7. Record the new configuration and keep a rollback path.

When the system grows, resist adding complexity without a diagnosed problem. Improve one stage at a time, measure the effect, and preserve the evidence behind each decision. That habit turns RAG from a fragile demo into a maintainable part of an LLM application.