Chunking Decides What Is Findable
An embedding has a finite window. Text past it is not represented at all — not ranked lower, not weakly matched. Absent. Whatever falls outside the window cannot be retrieved by any query, no matter how well the rest of the pipeline works.
That failure is quiet, which is what makes it expensive. Measured on a real 65,000-message mail archive: appending PDF attachment text to message bodies and embedding the first 3,000 characters left 14 million characters of successfully-extracted text outside every vector. The parser did its job. The text was in the output. It simply was not in any embedding, and so it was invisible to search.
Four Strategies v0.27+
| Strategy | Boundary | Best for |
|---|---|---|
Auto |
Picked from the document's actual composition | The default when you do not want to think about it |
FixedSize |
Token budget, with configurable overlap | Flat prose with no headings to lean on |
ByStructure |
A new chunk at each heading | Contracts, reports, specs — anything with a hierarchy |
BySection |
One chunk per slide, sheet, or attachment | PPTX, XLSX, and emails with attachments |
Auto inspects what the document actually contains rather than trusting its file extension: tabular and slide formats go to BySection, documents with headings go to ByStructure, and everything else falls back to FixedSize.
Why there is no BySimilarity
Semantic chunking is a reasonable idea and it is deliberately absent. It needs an AI round-trip, and a chunker that silently costs money on every document is the wrong default. Chunking here stays deterministic and free — same input, same chunks, no API key, no per-document charge.
What It Guarantees
Blocks Are Never Split
A table stays whole even when it exceeds the token budget. An oversized intact table beats two halves of one — half a payment schedule is not half as useful, it is misleading.
Tables Keep Their Contents
Chunk text uses a chunk-specific renderer rather than the summary renderer, which prints a table as the placeholder [table]. Without this, a chunk would claim to cover a payment schedule while containing none of it.
Deterministic
No model call, no sampling, no variability. The same document produces the same chunks on every run, so your index is reproducible and diffable.
Free
Chunking adds no cost to a parse. It is pure computation over blocks you already have.
tables.docx went from 10 to 61 estimated tokens once cell contents were included in chunk text. The earlier number was a chunk that pointed at a table without carrying any of it.Citation Provenance
A retrieved passage is much less useful when the reader cannot see which clause it came from. Every chunk carries the context needed to cite it:
{
"id": 3,
"text": "The Seller shall indemnify the Buyer against all claims...",
"tokenCount": 184,
"startBlockIndex": 12,
"endBlockIndex": 15,
"headingContext": "7. Indemnities",
"sectionKind": "",
"hasTable": false
}headingContext is the nearest heading above the chunk, so a passage lifted out of a contract can be shown with the clause heading it sits under. startBlockIndex and endBlockIndex map the chunk back to the exact block range in the parsed document, and hasTable lets a retrieval layer treat tabular chunks differently — rendering them as a grid, say, rather than as a wall of prose.
Token Counts Are Estimates
Counts come from words × 1.3. There is no tokenizer in the parser and no model call — that is the trade that keeps chunking free and deterministic, and it is accurate enough to size chunks sensibly.
It is not accurate enough to sit flush against a hard model limit. If you are packing chunks into a context window with no slack, leave headroom rather than treating tokenCount as exact.
Try It
Chunking operates on a parsed document, so it composes with every format the parser supports — the same call chunks a DOCX contract, a PPTX deck, or an email with a PDF attachment.
import docparse/services/chunker (chunkDocument, Auto, ByStructure, Chunk)
-- Let the document decide
let chunks = chunkDocument(doc, Auto({maxTokens: 512, overlap: 64}))
-- Or pin a strategy for a known corpus
let chunks = chunkDocument(doc, ByStructure({maxTokens: 512, groupUnderHeadings: true}))FAQ
How do I chunk documents for a RAG pipeline?
Parse the document, then call chunkDocument with a strategy. Because chunking runs over the typed Block ADT rather than raw text, boundaries land on real structure. It is deterministic and free — no AI call.
Does chunking split tables in half?
No. Blocks are never split. A table stays whole even when it exceeds the token budget, because an oversized intact table is more useful than two halves of one.
Are chunk token counts exact?
No — they are estimates (words × 1.3). There is no tokenizer and no model call. Leave headroom when sizing against a hard model limit.
Can I cite the source of a retrieved chunk?
Yes. Each chunk carries headingContext, sectionKind, a start and end block index, and hasTable, so a passage can be shown with the clause it came from.
Why is there no semantic (embedding-based) chunking?
It needs an AI round-trip, and a chunker that silently costs money on every document is the wrong default. Chunking stays deterministic and free.
Format Guides
Chunking runs on any parsed document. See what gets extracted first: