Definition
A chunking strategy defines how documents are split into smaller pieces (chunks) for storage in vector databases and retrieval in RAG systems. The strategy determines chunk size, overlap, and boundaries—critical decisions that significantly impact retrieval quality and the relevance of generated responses.
Why it matters
Effective chunking is foundational to RAG system performance:
- Retrieval precision — properly sized chunks improve semantic matching accuracy
- Context preservation — good boundaries keep related information together
- Token efficiency — optimal sizes balance context richness with LLM limits
- Answer quality — better chunks lead to better generated responses
- Cost management — appropriate sizing reduces unnecessary API calls
Poor chunking is one of the most common causes of RAG system underperformance.
How it works
┌────────────────────────────────────────────────────────────┐
│ CHUNKING STRATEGIES │
├────────────────────────────────────────────────────────────┤
│ │
│ FIXED-SIZE CHUNKING │
│ ┌──────────┬──────────┬──────────┬──────────┐ │
│ │ 500 tok │ 500 tok │ 500 tok │ 500 tok │ │
│ └──────────┴──────────┴──────────┴──────────┘ │
│ Simple but may cut mid-sentence │
│ │
├────────────────────────────────────────────────────────────┤
│ │
│ OVERLAPPING CHUNKS │
│ ┌──────────────┐ │
│ │ Chunk 1 │ │
│ └────────┬─────┴───────┐ │
│ │ Chunk 2 │ 50-100 token overlap │
│ └────────┬────┴───────┐ │
│ │ Chunk 3 │ │
│ └────────────┘ │
│ │
├────────────────────────────────────────────────────────────┤
│ │
│ SEMANTIC CHUNKING │
│ ┌─────────────────┐ ┌────────────┐ ┌──────────────────┐ │
│ │ Complete idea A │ │ Idea B │ │ Complete idea C │ │
│ └─────────────────┘ └────────────┘ └──────────────────┘ │
│ Splits at natural boundaries (paragraphs, sections) │
│ │
├────────────────────────────────────────────────────────────┤
│ │
│ HIERARCHICAL CHUNKING │
│ Document → Section → Paragraph → Sentence │
│ Multiple granularities stored together │
│ │
└────────────────────────────────────────────────────────────┘Key parameters:
- Chunk size — typically 256-1024 tokens; depends on content type
- Overlap — usually 10-20% prevents information loss at boundaries
- Splitting method — character, token, sentence, paragraph, or semantic
- Metadata — source, position, and hierarchy information preserved
Common questions
Q: What’s the best chunk size?
A: It depends on your content. Technical docs often work well with 500-1000 tokens. Q&A content may need shorter chunks (256-500). Test different sizes with your actual queries to find the optimum.
Q: Should chunks overlap?
A: Usually yes. 50-100 token overlap helps preserve context that spans chunk boundaries. Without overlap, sentences or important context can be cut in half.
Q: What’s semantic chunking?
A: Instead of fixed sizes, semantic chunking splits at natural boundaries—paragraphs, sections, or even detected topic changes. It keeps coherent ideas together but produces variable-size chunks.
Q: How does chunking affect retrieval?
A: Too large = diluted relevance, may exceed context limits. Too small = fragmented information, missing context. Finding the right balance for your use case is essential.
Related terms
- RAG — system that uses chunked documents
- Embeddings — vectors generated from chunks
- Vector Database — stores chunk embeddings
- Context Window — limits how many chunks fit
References
Lewis et al. (2020), “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, NeurIPS. [4,000+ citations]
Gao et al. (2024), “Retrieval-Augmented Generation for Large Language Models: A Survey”, arXiv. [500+ citations]
Karpukhin et al. (2020), “Dense Passage Retrieval for Open-Domain Question Answering”, EMNLP. [3,500+ citations]
Izacard & Grave (2021), “Leveraging Passage Retrieval with Generative Models for Open Domain QA”, EACL. [1,500+ citations]