RAG Chunking Strategies: The Definitive Guide to Building Smarter AI Products

Picture this: you’ve just deployed an AI assistant for your company’s technical documentation or legal knowledge base. You’re using one of the best models available, but when users ask concrete questions about regulations or procedures, the chatbot hedges, skips key clauses, or makes things up. After hours of prompt tweaking, you reach the same conclusion: the model isn’t the problem. The problem is how you’re splitting the context before feeding it to the model. When building a Retrieval-Augmented Generation (RAG) system, your chunking strategy determines how good the system can get. If information is cut in half, the model can’t reconstruct it. By the end of this article you’ll know which strategy to use based on your data type, budget, and latency requirements.

1. What is RAG and why does it matter for AI products?

Retrieval-Augmented Generation (RAG) is the architecture that connects an LLM with external, up-to-date knowledge bases without retraining the model.

Real-world product use cases

• Internal documentation chatbot: queries manuals, wikis, and corporate PDFs
• Legal assistant: answers based on contracts, regulations, and official bulletins
• Support copilot: answers using past tickets and up-to-date FAQs
• Semantic search: natural language queries over a product catalog

RAG vs. Fine-tuning: when to use each

If you need the model to know company-specific or domain-specific information, start with RAG. Fine-tuning is for adjusting how the model responds, not what it knows.

2. The chunking problem: why context size matters so much

A chunk is the unit of text that becomes a vector and gets stored in a vector database. When a user asks a question, the system searches for the most similar chunks and passes them to the LLM as context. The most common trap: using the default tutorial configuration, rigid 512-token fragments with 50-100 token overlap.

[--- Chunk 1 (512 tokens) ---]
                     [--- Chunk 2 (512 tokens) ---]

Chunk size creates a trade-off with no universal answer:

If you cut a legal clause or technical procedure in half, the resulting embeddings represent incomplete fragments and the search returns useless results. The ideal chunk contains exactly the information needed to answer the question, and nothing more.

3. The main chunking strategies

3.1. Fixed-Size Chunking (by characters or tokens)

Splits the text at a fixed number of characters or tokens, regardless of semantic boundaries. Fast and easy to implement, but breaks sentences and paragraphs in half. Useful for rapid prototyping or bulk processing of unstructured data.

from langchain_text_splitters import CharacterTextSplitter
text_splitter = CharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separator=""
)
chunks = text_splitter.split_text(document)

3.2. Recursive Character Text Splitting

The default strategy in most production pipelines. Tries to split by natural separators in hierarchical order: paragraphs (\n\n), lines (\n), spaces, and finally characters. Good for general purposes, articles, guides, and continuous prose. Start here if you’re not sure where to begin.

from langchain_text_splitters import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=400,
    chunk_overlap=50,   # 10-20% of chunk_size is a good starting point
    separators=["\n\n", "\n", " ", ""]
)
chunks = text_splitter.create_documents([prose_text])

Start with chunk_size between 400 and 1000 and 15% overlap, then tune based on real metrics.

3.3. Semantic Chunking (embedding-based)

Instead of fixing a character count, it calculates semantic similarity between consecutive sentences using embeddings. When similarity drops below a threshold, it detects a topic change and inserts a break. Works well on documents with frequent, unpredictable topic shifts (annual reports, heterogeneous corpus). The tradeoff: it requires an embedding call per sentence during ingestion, which makes it slower and more expensive than the previous options.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
chunker = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=85
)
chunks = chunker.create_documents([contract_text])

3.4. Document-Structure Chunking (Markdown, HTML, PDF)

Splits by respecting the document’s native hierarchy: headers, sections, lists, tables. Each chunk inherits metadata about its position in the structure. The most precise option when documents have a defined structure. Works for technical documentation, manuals, official regulations, and structured contracts.

from langchain_text_splitters import MarkdownHeaderTextSplitter
headers = [
    ("#", "H1"),
    ("##", "H2"),
    ("###", "H3"),
]
splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers,
    strip_headers=False
)
chunks = splitter.split_text(markdown_content)
# Each chunk includes metadata: {"H1": "Section name", "H2": "Subsection"}

For complex PDFs with tables, images, or multi-column layouts, use Docling or Unstructured.io instead of reading plain text. For very long sections, combine structure chunking with recursive splitting as a fallback.

3.5. Parent-Document Retrieval and Propositional Chunking (advanced)

Two strategies that resolve the granularity/context tradeoff in different ways. Parent-Document Retrieval indexes small chunks (~100 tokens) in the vector store for high semantic precision during search, but returns the larger “parent” fragment (400-800 tokens) to the LLM to ensure enough context. Propositional Chunking decomposes the document into atomic logical propositions: independent, self-contained assertions. Each chunk represents a single idea. Inspired by the Dense X Retrieval paper.

# Pseudocode: proposition extraction with LLM
propositions = llm.invoke(f"""
Decompose the following text into independent propositions.
Each proposition should be self-contained and verifiable on its own.
Text: {text}
Respond with a JSON list of propositions.
""")

These strategies make sense in enterprise systems where precision justifies the additional computational cost, for example legal or medical assistants.

3.6. GraphRAG: Knowledge Graph-Based Retrieval (advanced)

When plain text chunking isn’t enough for cross-cutting or complex reasoning questions, GraphRAG is the alternative. Instead of indexing isolated text blocks, it extracts the relationship structure of the corpus to build a Knowledge Graph: entities become nodes and their relationships become edges. This overcomes the main limitation of classic vector RAG: multi-hop reasoning, connecting entity A to entity C through B across different documents, and answering questions like “what are the main themes across the entire corpus?”.

How Microsoft GraphRAG works (the reference implementation):

1. An LLM extracts entities and relationships from the full corpus (subject-predicate-object)
2. Tightly connected node communities are detected (Leiden algorithm)
3. Hierarchical summaries are generated at different abstraction levels
4. At query time, the system combines subgraph traversal with community summaries

In production, the best practice is to combine vector search for fast precision on specific passages with the knowledge graph for contextualizing global relationships.

This approach fits corpus with high entity interconnection (case law, medical records, software architecture), multi-hop questions like “how does article X affect entity Y mentioned in section Z?”, and cross-cutting analysis of large document collections.

The cost is real: graph construction requires 3 to 5 times more time and tokens than classic vector RAG, and it’s considerably harder to maintain. Reference libraries: Microsoft GraphRAG, LlamaIndex Property Graph.

4. Strategy comparison

5. Which strategy should you choose?

Variables to evaluate before deciding:

1. Document type: Markdown, PDF, HTML, plain text?
2. Volume: thousands of documents means prioritizing speed; few but critical means prioritizing quality.
3. Query type: specific factual questions vs. broad queries?
4. Token budget: how much can you spend on indexing and retrieval?
5. Latency: does the user expect real-time responses?

6. How to evaluate your RAG pipeline

Don’t try to guess which strategy works best by intuition. Use quantitative metrics with frameworks like RAGAS, DeepEval, or TruLens:

To A/B test chunking in production:

1. Index your corpus with two different strategies in separate vector stores
2. For a percentage of traffic, capture the chunks retrieved by each strategy
3. Evaluate Context Precision and Answer Relevancy with RAGAS
4. Pick the winner and migrate 100% of traffic

7. Recommended stack

For a production-ready RAG pipeline:

• Orchestration: LangChain, LlamaIndex, or Semantic Kernel (Microsoft, especially recommended if your stack is .NET/C#)
• Embeddings: OpenAI text-embedding-3-small or open-source like nomic-embed-text
• Vector store: Qdrant (powerful metadata filters, very fast) or pgvector (great if you already use PostgreSQL)
• Hybrid search: combine vector search with BM25 and add a reranking model (Cohere Rerank or cross-encoder/ms-marco-MiniLM) to reorder results before sending them to the LLM
• Evaluation: RAGAS from day one

The most common mistake is implementing RAG without measuring. Set up evaluation from the start, even with a small set of test questions. It’s the only way to know whether your changes improve or degrade the system.

💬 Which chunking strategy are you using in production? Have you tried any combination that works particularly well?

Found this useful? Share it with your team. Most RAG problems have a solution at the chunking layer.