How to Chunk Documents for RAG: Meaning Beats Equal Size

Why chunk size is a business setting

In a RAG system, the model rarely reads an entire corporate document. A retrieval layer first selects several fragments, and a generative model then builds an answer from them. The chunking policy therefore determines whether the system sees a section heading, a discount condition, a continued table, and an exception to the rule at the same time.

The simplest implementation cuts text into equal numbers of characters or tokens with a small overlap. It is a convenient technical baseline, but a poor universal policy. One fragment may end immediately before the word “except,” another may contain a table row without its header, and a third may include a pronoun without the object it refers to. Retrieval can be fast while confidently returning incomplete context.

The useful management question is not “what chunk size is best?” It is: what is the smallest unit retrieval must find, and how much context does a person or model need to make a safe decision? Those two units are often different.

Retrieval unit and answer-context unit

Small fragments help precise retrieval: one procedure step, one product record, or one contract paragraph. Their vectors are less likely to blend unrelated subjects. Yet an answer may require the parent section, neighbouring conditions, or a complete table.

This leads to a practical “small retrieval, wider context” pattern:

• the index stores small child fragments;
• each child knows its parent identifier, heading path, and document position;
• retrieval ranks the child fragments;
• before generation, the system expands a hit into its parent section or a neighbouring window;
• duplicate parents are merged and the final context stays within a defined budget.

LlamaIndex demonstrates a related sentence-window technique: individual sentences are indexed, then a retrieved sentence is replaced with a window of surrounding sentences before it is sent to the model. It is not a universal production recipe, but it clearly separates the two jobs. A narrow fragment finds the location; the window restores meaning.

Start with document structure

When the source format allows headings, paragraphs, lists, and tables to be detected, preserve those elements first and apply size limits afterwards. Unstructured documents this approach: chunking operates on elements produced during partitioning, and normal chunks combine complete elements. Text splitting is used inside an element only when that element exceeds the hard maximum.

The `by_title` strategy is useful for procedures, manuals, and commercial terms. A new heading starts a new chunk even when the previous chunk has spare capacity. This is generally safer than merging the end of “Warranty” with the start of “Returns” merely to obtain equal lengths.

Maintain two limits:

• a soft size after which a chunk should preferably close;
• a hard maximum that cannot be exceeded because of embedding-model context or processing limits.

Overlap should not automatically be applied across every normal boundary. Unstructured notes that overlap between already coherent semantic elements can pollute chunks. It is often better reserved for a forced split of an oversized element and evaluated on real questions.

Different documents need different policies

One setting for the entire knowledge base almost always creates avoidable errors. A practical policy catalogue may look like this.

Procedures and manuals

Boundaries follow headings and numbered steps. Metadata includes the full heading path, version, effective date, and owner. At answer time, the retrieved step is returned with its parent section and nearby exceptions.

Contracts and regulatory material

A clause is the minimum unit. A definition, appendix, or reference to another clause cannot be detached without a way to expand the relevant context. Automated answers should cite the source and must not replace a legal decision.

Catalogues and price lists

Index a structured product or service record rather than a string rendering of a complete table. Price, currency, effective period, branch, and availability remain separate fields and filters. Descriptions can be searched semantically, while commercial constraints are checked deterministically.

Tables

Headers, units, notes, and rows must stay connected. A small table fragment should inherit its table title and header context. When an answer needs aggregation, code or a database should calculate it instead of a language model reasoning over a random set of rows.

When late chunking can help

The standard pipeline splits a document first and encodes each fragment independently with an embedding model. A phrase such as “this tariff” can therefore lose the tariff name that appeared in the preceding paragraph. The Late Chunking paper proposes a different order: a long text first passes through a long-context embedding model, and token representations are then pooled into vectors for the individual chunks. Each fragment can retain information from its place in the document.

The authors report gains across several retrieval tasks, particularly with smaller chunks. They also describe cases where ordinary chunking was comparable or better with large chunks, and where unrelated surrounding context did not help. Late chunking is therefore a candidate for an experiment, not a mandatory new default.

An implementation should verify:

• whether the embedding model supports a sufficiently long input and the required pooling method;
• whether an important section is truncated;
• how indexing time and memory change;
• whether boundaries remain reproducible after parser changes;
• whether retrieval improves on the company's own Russian-language documents.

Pipeline architecture

Reliable indexing starts before the chunker and ends after the vector database:

1. Ingestion records the source, owner, permissions, version, and checksum.
2. A parser extracts elements, reading order, and tables; uncertain pages go to quarantine.
3. A classifier selects a chunking policy for the document type.
4. Every fragment receives a stable identifier, parent reference, heading path, page range, and version.
5. The indexer writes vectors and a full-text index idempotently.
6. Retrieval applies access filters before ranking, then combines dense and lexical signals.
7. A context builder expands parents or neighbouring windows, removes duplicates, and enforces a context budget.
8. The answer cites specific fragments; low confidence or conflicting versions causes refusal or human escalation.

Compare strategies on your own questions

Collect 100–200 real questions from support, sales, production, or the internal knowledge base. For every question, a domain owner marks the mandatory document, relevant section, and minimum facts required for an answer. Include deliberately difficult cases:

• the answer crosses a subsection boundary;
• a rule and its exception are separated;
• a table continues on another page;
• the same term appears in several products;
• only the latest version is valid;
• the correct answer is “the sources do not contain enough information.”

Compare at least three variants: fixed-size chunks, structure-aware chunks, and small-child retrieval with parent context. Add late chunking as a fourth option when the stack supports it.

Retrieval measures come first:

• did the mandatory fragment appear in top-k;
• was the exception retrieved together with the rule;
• how much irrelevant text consumed the context budget;
• how many distinct documents were sent to the model;
• were version and access requirements respected.

Then measure answer grounding, correct refusals, and employee corrections. Ragas lists RAG metrics including context precision and recall, response relevancy, faithfulness, and noise sensitivity. Automated evaluations are useful for regression testing, but a process owner should validate business-critical examples.

The economics of fragments

Smaller fragments increase the vector count, index size, reindexing time, and number of candidates for a reranker. Larger fragments reduce those costs but often send irrelevant text to the generative model and may weaken retrieval precision.

Consider an illustrative corpus with 20 million characters of cleaned text. Fixed chunks of 1,000 characters with 10% overlap produce about 22,000 fragments. Chunks of 400 characters with the same overlap produce about 56,000. Indexing, backups, and reranking will nevertheless increase materially.

Measure cost per accepted answer rather than cost per vector:

• parsing and embedding compute;
• index storage and backups;
• reranking and generation;
• response latency;
• human review and correction;
• the business cost of an omitted condition.

Sometimes the more expensive “small retrieval, wider context” design pays for itself through fewer corrections. Sometimes simple `by_title` chunking performs almost as well. The evaluation set, not an architect's preference, should decide.

A two-week pilot

In week one, select one document type and collect questions with labelled sources. Pin the parser, embedding model, and retrieval settings so that only the chunking policy changes. Build three indexes and compare retrieval before enabling answer generation.

In week two, connect the same answer model, access controls, and refusal rules to every variant. Run a shadow test on new questions and measure quality, latency, fragment count, and employee review time. Classify failures as lost heading, split table, overly broad parent, wrong version, or insufficient context.

For the first decision, choose the policy that consistently retrieves mandatory facts within budget. Do not search for one permanent chunk size. Version the chunking policy, keep a reproducible evaluation set, and maintain a reindexing path. That makes RAG improvement controlled rather than a reaction to the next mysterious customer answer.