Chunking

Chunking splits a document collection into retrieval units. In RAG, the chunk is often the item that is embedded, indexed, retrieved, and packed into context.

Given a document $d$, chunking produces:

$$C(d)=\{c_1,\dots ,c_n\}$$

where each chunk $c_i$ has text, metadata, source location, and token length ${\ell }_i$.

Trade-off

Small chunks improve precision but can lose context. Large chunks preserve context but can retrieve irrelevant material and waste the token budget.

The practical objective is:

$$\underset{C}{\max }\mathrm{support}(C,q)\text{s.t.}\sum _{c_i\in C_q}{\ell }_i\le B_{\mathrm{retrieval}}$$

where $C_q$ is the retrieved chunk set for query $q$.

Chunk Types

• Fixed-size token chunks: simple, but may cut sections or formulas.
• Paragraph chunks: good for prose, weak for tables and code.
• Section chunks: preserve headings and intent.
• Semantic chunks: split by topic or discourse boundary.
• Structured chunks: tables, equations, code blocks, figures, metadata, or paper sections.

Checks

• Is the chunk unit aligned with the evidence needed by the answer?
• Does the chunk include source title, section, and stable location metadata?
• Are tables, equations, code blocks, and captions preserved as coherent units?
• Is overlap used deliberately rather than as a default patch?
• Does retrieval evaluation measure chunk recall separately from answer quality?
• Are private or unpublished chunks excluded from public indexes?

Related

• Embedding retrieval
• Context packing
• Retrieval-augmented generation
• Evidence-grounded generation
• Retrieval
• LLM Wiki