Chunking is the step where you split documents into pieces before embedding them into a vector database, and it remains the single highest-leverage decision in most retrieval-augmented generation (RAG) pipelines. Get it wrong and your retriever returns fragments that lack context or blobs so large they dilute the embedding signal. Get it right and even a modest embedding model outperforms a bigger model paired with careless segmentation. This guide covers the main strategies, when each one makes sense, the numbers that matter, and the mistakes teams keep repeating.
What Chunking Actually Does to Retrieval Quality
Also worth reading: How Can B2B Teams Enable Secure Vector Database Access Without Slowing Down AI Workflows? · How does churn intervention uplift modeling work for B2B SaaS retention strategies? · How Do B2B Signal Scoring Models Actually Work in 2026?
When you embed a chunk, the resulting vector encodes the semantic content of that specific span of text. A 1,000-token chunk about five different topics produces an embedding that sits somewhere in the middle of all of them, which makes it a poor match for any specific query. A 100-token chunk focused on one topic produces a much sharper vector. The tradeoff is context: a very small chunk may not contain enough information for the language model to answer a question even after retrieval.
This is why chunk size is not a knob you set once. Research and vendor guidance from IBM, NVIDIA, and AWS all converge on the same point: the best chunk size depends on your documents, your queries, and your embedding model's context window. Most production systems land somewhere between 200 and 800 tokens per chunk, with 256 to 512 tokens being the most common starting range. Embedding models like OpenAI's text-embedding-3 series handle up to 8,191 tokens, but that ceiling is not a recommendation — retrieval precision degrades well before you hit it.
There is also an overlap dimension. Adjacent chunks that share 10 to 20 percent of their text (typically 50 to 100 tokens of overlap) prevent sentences and ideas from being severed at arbitrary boundaries. Without overlap, a key sentence split across two chunks may not rank highly in either chunk's embedding.
Fixed-Size Chunking: The Baseline Everyone Starts With
Fixed-size chunking splits text every N tokens or characters, usually with overlap. It is the default in nearly every framework — LangChain's RecursiveCharacterTextSplitter, LlamaIndex's token splitters, and most vector database ingestion tutorials use some variant of it. The appeal is obvious: it is deterministic, fast, costs nothing to compute, and produces predictable chunk counts for capacity planning.
The weakness is equally obvious. Fixed-size splits ignore sentence and paragraph boundaries, so a chunk can end mid-sentence or mid-thought. Recursive character splitting mitigates this by trying to split on paragraph breaks first, then sentences, then words, only falling back to raw characters as a last resort. In practice, recursive splitting with a target of 400 to 600 tokens and 10 to 15 percent overlap is a defensible default for heterogeneous text corpora. If you have no time to tune anything, start there and measure.
Where fixed-size fails badly is structured or highly formatted content. Legal contracts, code files, and tables get shredded into meaningless fragments. If your corpus is dominated by those formats, move on to structural or semantic methods rather than trying to tune token counts.
Semantic Chunking: Splitting on Meaning Instead of Count
Semantic chunking embeds individual sentences, then splits wherever the cosine similarity between consecutive sentence embeddings drops below a threshold — commonly around the 50th percentile of pairwise similarity scores in the document, or an absolute threshold in the 0.5 to 0.8 range depending on the embedding model. The idea is that a drop in similarity signals a topic shift, which is the natural place to cut.
The results are genuinely better for long-form prose: essays, reports, documentation, and transcripts where topics shift at irregular intervals. Teams typically report retrieval precision gains in the 5 to 15 percent range over fixed-size chunking on prose-heavy benchmarks, though your mileage varies enormously by corpus. The cost is real, too. Semantic chunking requires one embedding pass per sentence during ingestion, which can multiply your embedding compute by 3 to 10 times compared to fixed-size splitting. For a 100,000-document corpus, that difference shows up directly in your ingestion bill and ingestion time.
A pragmatic middle ground is to use semantic chunking only for your highest-value documents — the ones that answer the most customer or user queries — and fixed-size chunking for the long tail. The open-source library Chonkie, which launched on Hacker News in 2025 as part of the YC X25 batch, has made semantic and late chunking accessible without heavy framework dependencies, and it is a reasonable starting point if you want to experiment cheaply.
Structural and Document-Aware Chunking
Structural chunking respects the document's own boundaries: headings, paragraphs, list items, table rows, and code functions. For Markdown and HTML, this means splitting on heading levels and keeping each section intact up to a token cap. For PDFs, it means using layout-aware parsers that reconstruct reading order before splitting — a step teams skip at their peril, because naive PDF extraction scrambles multi-column layouts and tables.
Structured data deserves special mention. A 2025 HackerNoon analysis of chunking strategies for structured data in RAG systems argued that text-to-SQL approaches often fall short when questions require joining fragmented rows, and that chunking tables row-wise with schema context attached to each row outperforms embedding whole tables. The pattern generalizes: attach metadata (table name, column headers, section title, source document, date) to every chunk. Metadata filtering at query time frequently matters more than the chunking method itself, because it lets the vector search operate on a pre-narrowed subset.
For code, chunk by function or class rather than by token count. For support tickets and chat logs — a common corpus for product and support teams — chunk by conversation or by resolved issue, not by message count, so each vector represents one complete problem-and-solution unit.
Comparing the Main Strategies Side by Side
| Feature | Fixed-Size (Recursive) | Semantic Chunking | Structural Chunking |
|---|---|---|---|
| Ingestion cost | Lowest — one pass | Highest — per-sentence embeddings | Low to moderate |
| Best corpus type | Mixed, heterogeneous | Long-form prose, transcripts | Docs, code, tables, PDFs |
| Typical chunk size | 400–600 tokens | 100–400 tokens (variable) | Varies by section |
| Retrieval precision | Baseline | +5–15% on prose | +10–25% on structured docs |
| Implementation effort | Minutes | Hours, threshold tuning | Hours, parser work |
| Failure mode | Mid-sentence cuts | Over-fragmentation | Missed boundaries in messy PDFs |
| Overlap needed | Yes, 10–15% | Usually no | Rarely |
How to Evaluate a Chunking Strategy Properly
The teams that get chunking right treat it as an empirical problem, not a preference. Build a small golden set of 50 to 200 real questions with known correct source passages — for a support team, these are actual tickets where you know which help-center article resolved them. Then measure retrieval metrics: hit rate (did the correct chunk appear in the top-k results), mean reciprocal rank, and, if you are running full RAG, answer faithfulness judged by an LLM or by humans.
Run each candidate chunking strategy over the same evaluation set and compare. A change that improves hit rate at k=5 by even 5 percentage points is usually worth shipping, because downstream answer quality tracks retrieval quality almost linearly. Re-evaluate whenever you change embedding models, since a chunk size tuned for one model's embedding geometry can be wrong for another. Also re-evaluate when your corpus shifts materially — adding a new product line or documentation set can invalidate prior tuning.
One practical tip: evaluate chunking in isolation before adding rerankers. A cross-encoder reranker can mask a bad chunking strategy, which feels like a win but means you are paying reranking compute to compensate for an upstream problem you could fix for free.
Common Mistakes That Quietly Destroy Retrieval
The most common mistake is treating chunk size as the only variable. Overlap, boundary placement, metadata, and the embedding model interact, and tuning one in isolation produces misleading results. The second mistake is chunking tables and code with a text splitter, which produces vectors that match queries but return unusable content. The third is ignoring document quality upstream: duplicated content, boilerplate headers and footers repeated on every page, and broken PDF layouts all poison the index regardless of chunking method. Deduplicate and strip boilerplate before you split.
A fourth mistake is over-fragmenting. Teams chasing precision sometimes cut chunks to 50 or 100 tokens and discover the retrieved passages no longer contain enough context for the LLM to answer. A useful mitigation is small-to-big retrieval: embed small chunks for precision but return the surrounding parent section (500 to 1,500 tokens) to the model. This parent-document retrieval pattern, supported natively by LlamaIndex and implementable in any vector store with metadata links, is one of the highest-return upgrades available.
Finally, teams forget that queries and chunks should live in similar semantic space. If your users ask short questions but your chunks are 800-token essays, consider adding a query-expansion step or generating a hypothetical answer to bridge the gap.
When to Act, and What It Costs
Act now if your retrieval hit rate on your golden set is below roughly 80 percent at k=5, or if users complain the assistant cites irrelevant or truncated sources. Chunking work is cheap relative to its impact: for most teams it is one to three engineer-days to implement and evaluate a new strategy, with no ongoing infrastructure cost unless you adopt semantic chunking at scale.
Cost-wise, embedding is the main line item. At OpenAI's text-embedding-3-small pricing (roughly $0.02 per million tokens as of 2026), a 1 million-token corpus costs about $0.02 per full re-embedding — trivial. Semantic chunking multiplies ingestion embeddings several-fold, but even a 10x increase on a small corpus is pennies. The real costs are engineering time and the discipline to maintain an evaluation set. Commercial chunking tooling exists — Bedrock's custom chunking, various RAG platform tiers — but the open-source route (Chonkie, LangChain, LlamaIndex) covers the vast majority of needs at zero license cost.
For teams building signal-inbox products — say, a B2B tool that clusters customer feedback, support tickets, and community posts into themes — chunking strategy maps directly to product quality. Chunk each ticket or thread as a unit, attach metadata like product area, sentiment, and customer tier, and the vector search becomes a theme-detection engine rather than a text matcher. That is the difference between a dashboard that shows noise and one that tells you what to build next.
The Bottom Line
Start with recursive fixed-size chunking at 400 to 600 tokens with 10 percent overlap, build a 100-question evaluation set, and only then invest in semantic or structural methods where your corpus demands it. Attach metadata to every chunk, use parent-document retrieval to restore context, and re-evaluate after every embedding model change. Chunking is unglamorous, but it is where most RAG quality is won or lost — and the teams who measure it consistently beat the teams who argue about it.