Chunking

Chunking is the process of splitting a document into smaller, semantically coherent segments before they are embedded and indexed for retrieval. In a retrieval-augmented generation (RAG) pipeline, chunk size and boundaries determine what a language model actually "sees" when it answers a question about a document. Get chunking wrong and even a perfect extraction upstream produces vague or incorrect answers downstream.

How it works

A chunker takes raw text, or structured output from document parsing, and breaks it into pieces sized for an embedding model's context window, typically a few hundred tokens. The simplest approach is fixed-size chunking: cut every N characters or tokens, sometimes with overlap so context isn't lost at the seams. It's fast but ignores meaning, and can split a clause in a contract or a line item in a table right down the middle.

More useful strategies respect document structure. Recursive chunking splits along natural breakpoints (paragraphs, then sentences, then words) only when a section is too large. Semantic chunking groups sentences by topic similarity, using embeddings to decide where one idea ends and another begins. Layout-aware chunking goes further: it uses the output of document layout analysis to keep tables, headers, and their surrounding text together, rather than cutting across a table extraction result or separating a key from its value in key-value extraction.

Overlap between chunks (usually 10-20% of chunk length) helps preserve context that spans a boundary, at the cost of some redundancy in the index. Metadata (page number, section title, document type from document classification) is usually attached to each chunk so the retrieval step can filter and the generation step can cite sources accurately.

Why it matters for document processing

Chunking sits between extraction and retrieval, and it's often the weakest link. A contract clause split across two chunks may retrieve incompletely, leading a model to miss an obligation or a termination date. A table cut mid-row destroys the relationship between a value and its label. For structured or semi-structured documents, generic text chunkers built for prose (blog posts, articles) simply don't transfer well.

This matters most in regulated, high-stakes document work: insurance claims, contracts, KYC files. If a retrieval system misses a clause because of a bad chunk boundary, the error looks like a model failure but is really a preprocessing failure. Good chunking strategy for these domains starts with accurate document layout analysis and table extraction, not with tuning chunk size in isolation. The choice of chunking approach should be treated as a first-class design decision, not an afterthought bolted onto a RAG pipeline.

Related terms

• Retrieval-Augmented Generation (RAG)
• Document layout analysis
• Document parsing
• Table extraction
• Contract analysis

FAQ

What chunk size should I use for RAG on business documents?

There's no universal number. It depends on the embedding model's context window and the document type. Dense contracts often need larger chunks (300-500 tokens) to preserve full clauses, while invoices or forms benefit from smaller, layout-aware chunks tied to individual fields or line items rather than fixed character counts.

Does chunking strategy matter more than the embedding model?

Often, yes. A strong embedding model applied to poorly chunked, structurally broken text will still retrieve incomplete or misleading context. Fixing chunk boundaries, especially around tables and clauses, tends to produce bigger accuracy gains than swapping embedding models, particularly for structured business documents.