Companion post to the video above. This is the deeper reference version — the configs, code, and sources the video didn’t have time for. If you just want the mental model, watch the video first; come back here when you’re actually building.
Why RAG demos fall apart in production
Most RAG tutorials are tested against one clean PDF or a Wikipedia dump. That’s the wrong test. The moment you point the same pipeline at real company documents — inconsistent formatting, support tickets, sprawling internal wikis — answers start coming back incomplete, hallucinated, or confidently wrong.
Two mental models explain why, and they hold for almost every production RAG failure:
- RAG is a retrieval system first, a generation system second. If retrieval surfaces the wrong context, the LLM has no real chance of producing a correct answer, no matter how good the model is.
- System quality is bounded by the worst stage, not the average. A strong embedding model cannot rescue a chunking strategy that cut a sentence in half.
The two-phase pipeline
Every RAG system splits into an offline indexing phase and an online query phase:
Indexing (offline): documents → chunks → embeddings → vector store
Query (online): question → embedding → retrieval → context assembly → LLM
Production systems add caching, reranking, guardrails, monitoring, and fallback logic on top — but this is the foundation everything else sits on. Get the foundation wrong and no amount of tooling fixes it downstream.
Chunking: the highest-leverage decision in the pipeline
Chunking is where most systems fail, and it’s usually invisible until you’re debugging a wrong answer three stages downstream. It matters because embedding models work best on coherent spans of text, LLM context windows are finite, oversized chunks introduce noise, and undersized chunks lose the surrounding information a correct answer depends on.
Fixed-size chunking (with overlap). Split every N tokens, overlap 10–20%. Simple and predictable, but indifferent to sentence and paragraph boundaries — it will cut a definition in half without noticing. Common starting point, rarely optimal.
Recursive / hierarchical chunking. Try natural boundaries first — double newlines, then single newlines, then sentences, then words — falling back only when a section is still too large. This is the default in most frameworks now (see LangChain’s RecursiveCharacterTextSplitter), and it’s a meaningful step up from pure fixed-size.
Structure-aware chunking. Respect the document’s actual structure — headers, sections, code blocks, tables. For technical docs and markdown this usually wins outright. Store parent-child relationships so you can retrieve a small, precise chunk but expand to the full parent section when the answer needs surrounding context.
Semantic chunking. Use an embedding model or an LLM to detect where the topic actually shifts, and chunk there. Produces the most coherent chunks, at real extra cost (an embedding or LLM call per boundary decision) — usually reserved for corpora where retrieval quality is worth the spend.
Working defaults:
- Start with recursive or structure-aware chunking — not fixed-size.
- Keep chunks in the 300–800 token range for most use cases; tune against your own eval set, not a blog post’s number.
- Always store metadata on the chunk: source document, section title, page/paragraph number. You need this later for filtering and for citations in the final answer.
- Build a small set of real questions and inspect what actually gets retrieved before touching the embedding model. Bad chunking is a silent killer — fix it before you start tuning anything downstream.
For the actual implementations behind each strategy, LangChain’s text splitter docs and LlamaIndex’s node parser docs are the two references worth having open while you build this.
Embeddings and vector stores
Embedding models map text into a high-dimensional space where semantic similarity becomes geometric closeness, typically scored with cosine similarity or dot product. The model you pick matters, but in practice data quality and chunking usually matter more than which embedding model is in the pipeline.
Broad categories: general-purpose open models, domain-specific embeddings (legal, medical, code), and proprietary APIs. Pick based on your domain’s vocabulary, not benchmark leaderboards alone.
A vector store’s actual job is fast Approximate Nearest Neighbor (ANN) search — exact (brute-force) search doesn’t scale past a small collection. Under the hood, most stores use graph-based indexes like HNSW (Hierarchical Navigable Small World) or cluster-based indexes like IVF to avoid comparing your query against every vector in the collection; that’s the tradeoff you’re implicitly accepting when you pick “approximate” over “exact” — a small, tunable chance of missing the true nearest neighbor in exchange for sub-linear query time. You rarely need to tune this yourself early on, but it’s worth knowing it’s there before you’re debugging a “why didn’t it retrieve the obviously-correct chunk” issue.
When choosing a vector store, the questions that matter are: does it support metadata filtering, does it support hybrid search (vector + keyword/BM25) natively, how easy is it to run locally versus managed, and what do persistence, replication, and scaling actually look like at your data volume. Common options: Qdrant, Weaviate, Pinecone, Chroma, pgvector, OpenSearch’s k-NN.
Early on, the specific vector store matters less than: clean data, good chunking, proper metadata, and the ability to combine vector and keyword search. Hybrid search shows up in nearly every serious production system because pure vector search reliably misses exact-match tokens — error codes, product SKUs, IDs — that a keyword/BM25 pass catches trivially. Weaviate’s hybrid search documentation is a clean reference for how the fusion between dense and sparse scores actually works if you want to see it under the hood.
# Conceptual shape of a hybrid query — exact API varies by vector store
results = vector_store.hybrid_search(
query=user_question,
alpha=0.5, # weight between dense (1.0) and keyword (0.0) search
filters={"doc_type": "runbook", "product": "checkout-service"},
top_k=25,
)
Retrieval quality is not the same as “top-k similarity”
Retrieving the top-k most similar chunks is the starting point, not the finish line. In production you’ll consistently hit: the most similar chunk isn’t the most useful one, an answer’s information is split across multiple chunks, the “lost in the middle” effect where models under-attend to information buried in the middle of a long context even when it’s technically present, and embedding models that simply don’t understand your domain’s vocabulary.
Techniques that move the needle, roughly in order of ROI:
-
Reranking. Pull a wider candidate set (top 20–50, sometimes top 100) with cheap vector search, then re-score with a cross-encoder or a small LLM before truncating to what actually goes in the prompt. This is consistently one of the highest-ROI additions to a RAG pipeline — see Pinecone’s rerankers writeup for the mechanics of why a cross-encoder outperforms bi-encoder similarity alone, or Cohere’s Rerank docs if you want a reranker you can drop in without training your own.
The reason this is two stages and not one: a cross-encoder scores a query against a document jointly, which is far more accurate than comparing two independently-computed embedding vectors — but it means running the model once per candidate document, which doesn’t scale to your full collection. Bi-encoder search is cheap and approximate; it narrows the field. The cross-encoder is expensive and precise; it only has to look at what’s left.
- Hybrid search. Combine dense vectors with sparse keyword search so exact-match tokens aren’t left to chance.
- Query transformation. Rewrite the user’s question, generate a hypothetical answer document and search on that (HyDE-style), or decompose a complex question into sub-questions retrieved independently.
- Metadata filtering. Narrow the candidate pool by document type, date range, or product before vector search runs, not after.
- Parent document retrieval. Retrieve a small, precise chunk for matching accuracy, but return its larger parent section for context completeness.
Measuring retrieval, separately from the final answer
You need an evaluation set, not a vibe. Build 20–50 realistic questions, note which chunk(s) should be retrieved for each, and measure directly: does the correct chunk appear in the top 5 or top 10 (Recall@k)? Layer an LLM-as-judge for relevance scoring once you trust manual inspection, not before. If you never measure retrieval quality independently of final answer quality, you will spend real time optimizing the wrong stage of the pipeline.
You don’t have to build this evaluation harness entirely by hand — Ragas is the most widely used open framework for RAG-specific metrics, and Qdrant’s guide to RAG evaluation is a practical walkthrough of building the eval set itself, not just the metrics math.
Framework choice: LlamaIndex vs. LangChain, conceptually
LlamaIndex is data-centric — strong, opinionated abstractions around indexing, retrieval, and query engines. It tends to feel more natural when the core problem is genuinely “I have a lot of documents and need good retrieval.”
LangChain is general-purpose — it started with chains and agents, and has a much larger integration ecosystem. More flexible, and correspondingly heavier when all you need is retrieval. LangChain’s own framework comparison is a reasonable starting point if you want the maintainers’ framing directly.
Neither is magic — both are orchestration layers over the same fundamental patterns above. Pick one, learn its concepts deeply, and don’t get religious about it: the architecture knowledge transfers regardless of which framework’s syntax you’re writing.
Recap
- Indexing and query are cleanly separate phases — design them independently.
- Chunking is a first-class design decision, not a default you accept from a framework.
- Vector stores are infrastructure — the leverage is in filtering and hybrid search, not the store’s brand.
- Retrieval quality has to be measured on its own, separate from final answer quality.
- Frameworks are tools, not the architecture. Learn the patterns; the tool is interchangeable.
If you’ve seen the “RAG is dead, just use a bigger context window” takes going around — that’s the next post. Short version: bigger context windows fix retrieval recall’s easier failure mode and do nothing for LLM recall’s harder one, and there’s a real difference between the two. Full breakdown next.
After that: building safe LLM interfaces — guardrails, policy enforcement, and preventing unsafe or non-compliant outputs before they ship.
References
- Liu, N. F. et al. — Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172)
- Pinecone — Rerankers and Two-Stage Retrieval
- Cohere — Rerank overview
- Weaviate — Hybrid Search documentation
- Weaviate — Hybrid Search Explained
- LangChain — Text splitter integrations
- LlamaIndex — Node Parser usage pattern
- Ragas — ragas.io
- Qdrant — Best Practices in RAG Evaluation
- LangChain — LangChain vs. LlamaIndex (LangChain’s own framing — worth balancing against LlamaIndex’s docs directly)