In the context of Retrieval-Augmented Generation, what is the use of chunking of raw…
- User
In the context of Retrieval-Augmented Generation, what is the use of chunking of raw input data?
- Assistant
Why “Chunk” the Data at All?
Retrieval-Augmented Generation (RAG) works in two-phases:
- Retrieve relevant pieces of knowledge.
- Pass the retrieved text + the user question into the LLM to generate an answer.
Raw source material—PDFs, web pages, logs—can be hundreds or thousands of tokens long.
Putting those whole documents into an embedding model, a vector store, or the LLM itself causes multiple problems:Problem if you DON’T chunk How Chunking Helps 1. Model limits – Most embedding models top out at 512–4 096 tokens, and GPT-4/Claude/Mistral prompts are capped. A 300-token chunk easily fits into embedding models and the LLM’s context window. 2. Diluted semantics – Long passages discuss many topics; the embedding becomes an “averaged” representation that is fuzzy for any one sub-topic. Smaller, topically-consistent chunks create sharper, high-precision embeddings, improving similarity search. 3. Low retrieval granularity – If you retrieve a 10-page doc for a single sentence, you waste context space and force the LLM to sift through irrelevant noise. The retriever can “laser-pick” only the paragraphs likely to answer the query, freeing up prompt budget. 4. Higher hallucination risk – Extra, off-topic text in the context window can steer the LLM off course. Tighter, semantically focused chunks anchor the LLM on the right evidence. 5. Update & storage costs – Re-embedding a whole doc when one paragraph changes is wasteful. Only the affected chunk needs re-embedding.
How Chunking Is Usually Done
-
Sliding‐window token chunks
• Pick a window size (e.g., 256 or 512 tokens).
• Step forward with optional overlap (e.g., 20–30 %). Overlap preserves continuity so that sentences spanning two chunks aren’t broken. -
Semantic / phrase-level chunking
• Split on headings, paragraphs, or sentences until a max-token threshold is hit.
• More coherent but slightly heavier to implement (needs parsing). -
Hybrid (semantic + sliding)
• Split by headings first, then apply a sliding window on unusually long segments.
Typical hyper-parameters you’ll see in production:
chunk_size = 200–400 tokens # depends on the embedding model chunk_overlap = 10–20 % # 20-40 tokens
Trade-offs & Tuning Tips
• Small chunks (≤100 tokens)
- High precision; low prompt cost.
– May lose global context; recall can drop.
• Large chunks (≥800 tokens)
- Better context; fewer embeddings.
– Lower precision, risk of hitting context limits.
• Overlap too high (50 %+)
– Duplicated content in vector DB, slower retrieval, higher storage.• Overlap too low (0 %)
– Boundary sentences split awkwardly, hurting recall.A/B test different combinations while measuring: • Retrieval precision/recall on a labelled test set.
• End-to-end answer accuracy & hallucination rate.
• Latency & cost (embedding, storage, token usage).
One-Sentence Summary
Chunking in RAG breaks large documents into “bite-size”, semantically coherent pieces so that the retriever can find, embed, and supply only the most relevant evidence to the language model—boosting precision, lowering cost, and staying within context-length limits.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.