Chunking & Passage Retrieval
Consensus definition
Chunking divides source documents into discrete segments — typically a few hundred tokens — that are each encoded as a dense vector and stored in an index1. At query time the system compares the query vector against chunk vectors and returns the top matches, and a model generates its answer grounded in those passages2. Because retrieval and citation happen at the passage level, one well-scoped document can supply many independently retrievable passages — but only if each holds enough standalone context3. Strategies range from fixed-size to sentence-aware, semantic and proposition-based chunking14. Chunk size is a trade-off: smaller chunks sharpen precision for factual queries; larger ones preserve context for analytical ones4. The same logic underlies passage-level ranking in classic search — individual passages are evaluated against a query, not only whole pages.
rhinegold operator refinement
Rhinegold's reframe: the splitter is provider-internal — you cannot configure it. The only lever you control is structure. Write each section so it stands alone: its own subject, context and conclusion, without needing the paragraph before it. Short focused sections, explicit headings, one idea per paragraph, complete entity–attribute–value statements — these maximise the odds the right chunk is retrieved and cited. A 3,000-word page built as ten standalone passages is retrievable for ten distinct queries; the same words as a flowing narrative may surface for none.
Operational use
Audit content at the passage level, not the page level: each H2/H3 section should answer its target query without surrounding context. Quick test — read any paragraph in isolation; if it needs the previous one to make sense, it is not a retrievable unit. Apply one-claim-per-sentence writing in the definition, comparison and specification sections most likely to be cited.
Measurement boundary
The chunk size, overlap and splitting algorithm used by AI-search providers are proprietary and undisclosed, and you cannot see which chunk was retrieved for a given citation. Observable proxies are citation frequency by content section and grounded-response rate at the query-cluster level — they can signal better passage retrievability but cannot isolate chunking as the sole cause.
Distinct from
From RAG: that is the overall retrieve-then-generate system; chunking is one preparation step inside it. From the embedding: chunking decides what text gets encoded, the embedding decides how it is encoded. From structured data: schema markup signals semantic roles to crawlers and complements passage structure, but markup does not replace self-contained prose.
Common mistakes
- Writing long flowing prose without clear section breaks — one 2,000-word block becomes one poorly scoped chunk.
- Assuming page-level ranking equals passage-level retrievability: a high-ranking page does not guarantee the right passage is retrieved for a specific query.
- Relying on tables or lists without accompanying prose — many pipelines mishandle HTML tables, leaving the information unreachable as text.
- Dropping the subject from mid-document paragraphs ("It also offers…"), so the passage loses meaning when extracted as a standalone chunk.
Sources & deeper reading
- 2Karpukhin et al. (2020) — "Dense Passage Retrieval for Open-Domain Question Answering" (EMNLP 2020)
- 1Weaviate — "Chunking strategies to improve LLM RAG pipeline performance" (fixed/semantic/sentence/proposition chunking)
- 4Unstructured — "Chunking strategies for RAG: best practices and key methods" (chunk-size trade-offs)