RAG shows up in nearly every serious LLM application that needs to answer questions about specific, current, or private data. Here's what it actually does under the hood.
The problem it solves
A language model's knowledge is frozen at training time and limited to what was in its training data. Ask it about your company's internal documentation, a document uploaded five minutes ago, or anything published after its cutoff, and it either says it doesn't know — or worse, confidently makes something up.
RAG's fix: instead of relying purely on the model's trained-in knowledge, retrieve relevant information at query time and hand it to the model as context, then ask the model to answer using that context.
The pipeline, step by step
- Ingest: your documents (docs, PDFs, support tickets, whatever) are split into chunks and converted into vector embeddings — numerical representations that capture semantic meaning — then stored in a vector database.
- Query: when a user asks a question, that question is also converted into an embedding.
- Retrieve: the system finds the stored chunks whose embeddings are most similar (closest in vector space) to the question's embedding — typically the top 3-10 most relevant chunks.
- Augment: those chunks get inserted into the prompt sent to the LLM, usually with instructions like "answer using only the following context."
- Generate: the model produces an answer grounded in the retrieved text, ideally citing which chunk it came from.
User question
│
▼
Embed question ──► Vector search ──► Top-k relevant chunks
│
▼
Prompt = chunks + question ──► LLM ──► Answer
Why not just fine-tune the model on your data instead?
Fine-tuning bakes information into the model's weights — it's slower to update (retrain every time your data changes), more expensive, and doesn't give you a way to show which source an answer came from. RAG keeps your data external and swappable: update a document, and the next query immediately has access to the new version, no retraining required. It also lets you cite sources, which matters a lot for trust in any answer-facing product.
Where RAG actually breaks down
- Bad chunking. Splitting documents at arbitrary character counts instead of natural boundaries (paragraphs, sections) can cut a relevant sentence in half across two chunks, and neither chunk alone contains the full answer.
- Retrieval that misses the right chunk. If the top-k search doesn't surface the chunk that actually answers the question — common with vague queries or documents using different terminology than the question — the model never sees the right information and either says "I don't know" or guesses.
- Context window overload. Stuffing in too many retrieved chunks "just in case" dilutes the signal and can push out genuinely relevant information, or exceed the model's effective context length where it reliably pays attention.
The practical takeaway
RAG isn't a single technique so much as a pattern — the quality of a RAG system is determined far more by the retrieval step (chunking strategy, embedding model, search relevance) than by which LLM sits at the end of the pipeline. Most production RAG problems are debugging what got retrieved, not the model's reasoning.
