Concept dictionary
Chunking
Chunking is the process of dividing a document, dataset, or stream into smaller units for storage, search, retrieval, or processing.
- Review state
- Reviewed in POC
- Freshness
- Current
- Version
- 3
Also called
- Chunking
- Chunk
Remember it this wayChunk where the meaning can travel together.
In plain language
Chunking divides a document, dataset, or stream into smaller units that a system can store, search, retrieve, or process. The boundaries may follow length, structure, meaning, time, or a combination of these.
Why it matters
Chunk boundaries determine what context stays together. Poor chunks can split evidence from its qualifier, create noisy retrieval, or exceed a model’s context limits.
Example
A policy manual is divided by section headings, with a small overlap between adjacent sections, before each chunk is embedded for retrieval.
Common misconception
Chunking is not simply cutting text into equal character counts. Fixed-size chunks are one method, and they may damage meaning when structure matters.
Related terms
Sources
- Patrick Lewis and colleagues, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”shows retrieval over indexed passages as external model context.
Learn more
- Compare [[retrieval-augmented-generation|Retrieval-Augmented Generation]], [[embedding|Embedding]], [[context-window|Context Window]], and [[search-retrieval|Search / Retrieval]] to place Chunking in the surrounding system.
- Follow the cited source material above for the method, standard, or research context behind this draft.