Chunking strategy for RAG systems with trustworthy citations

Quick answer

There is no universal best token count. Legal clauses, support articles, code, tables, and transcripts have different semantic boundaries. Start with the smallest unit that can answer a target question without losing qualifications, and keep enough parent context to interpret it. This tutorial is for RAG builders diagnosing weak retrieval or unreliable citations.

Select a starting strategy

Corpus	Starting boundary	Preserve
Policies and contracts	Section or clause	heading path, version, effective date, page
Product documentation	Heading and procedure	product/version, code block, previous step
Support articles	Article subsection	article ID, locale, updated date
Tables	Whole logical table or row group	headers, units, footnotes
Transcripts	Speaker turn plus topic window	speaker, timestamp, meeting
Source code	Symbol or logical block	file, symbol, language, imports

Token windows are a fallback, not the information architecture.

Run the experiment

Create 50–100 questions with known supporting passages, including cases that require a qualification or multiple sections. Index three variants—for example 300 tokens, 700 tokens, and structure-aware chunks—with identical embeddings and retrieval settings. Measure:

• support recall in the retrieved top-k;
• citation precision at the displayed source unit;
• answer faithfulness and completeness;
• duplicate evidence caused by overlap;
• p95 latency, index size, and reranking cost.

Use Ragas for retrieval-oriented metrics and inspect LlamaIndex or Haystack when comparing parsing pipelines. Keep human review for citation usefulness.

Example: overlap creates false confidence

A policy is split every 500 tokens with 100-token overlap. One exception appears in three neighboring chunks, so the retriever returns all three and the answer looks strongly supported. In reality, every result repeats the same evidence. Deduplicate by source span and use parent sections for context instead of treating overlap as independent support.

Metadata and citation contract

Every chunk should carry a stable source ID, canonical URL, title, section path, version or updated date, and a precise locator. The interface should cite the unit a reader can open and verify. If a chunk cannot be mapped back to a human-readable source, it is unsuitable for high-trust answers.

Risks and limits

Larger chunks may improve context but dilute similarity and raise cost. Smaller chunks increase precision but can strip exceptions or definitions. OCR, tables, access controls, and stale versions can dominate chunk-size effects. Fix source quality and permissions before tuning retrieval.

Diagnose retrieval before changing chunk size

Inspect failed questions and label the cause. A missing source is an ingestion problem. A source present but ranked too low suggests embedding, query, filter, or reranking work. A retrieved chunk without the necessary exception is a boundary problem. An answer that ignores good evidence is a generation or prompt problem. Changing chunk size for every failure can increase cost without addressing the actual layer.

Create a gold set at the level readers verify: document, section, page, table, or code symbol. Record all acceptable supporting passages when several sources answer the question. Evaluate retrieval separately from answer generation so a good model cannot hide poor search and a weak answer cannot make useful retrieval look bad.

Handle structures that token windows damage

Keep table headers, units, notes, and row groups together. A row without its header may reverse meaning. For procedures, preserve prerequisites and warnings with the relevant step or attach a parent-section reference. For code, split on symbols and retain imports, signatures, file paths, and language. For policies, include definitions and exceptions through a parent-child retrieval pattern rather than duplicating them into every chunk.

For an evaluation that starts with parser fidelity, multi-column reading order, scanned OCR, and source-page reconstruction, use the PDF chunking strategy comparison. PDF layout failures must be separated from ordinary text-boundary tuning before an index can be trusted.

Use overlap only when it fixes a measured boundary failure. Store source spans so overlapping results can be deduplicated. Otherwise the retriever may return several copies of one passage and crowd out independent evidence. When adjacent chunks are retrieved, merge them for presentation without counting them as separate citations.

Versioning, access, and freshness

Attach document version, effective date, locale, product, and access-control labels during ingestion. Apply authorization before or during retrieval, not after the model has seen content. If a source is replaced, expire its chunks and verify that stale vectors no longer appear. A canonical identifier should survive re-indexing so citations and deletion requests remain traceable.

Record parser, chunker, embedding model, metadata schema, and index revision. Run the gold set before promoting a new index and keep the previous revision available for comparison or rollback. Measure freshness lag from source update to searchable chunk, especially for policy and support corpora.

Cost and operating tradeoffs

Small chunks create more vectors, retrieval candidates, and reranking work. Large chunks increase context tokens and may place irrelevant text around the answer. Parent-child retrieval can improve context but adds storage and orchestration. Report index size, ingestion time, query latency, reranking cost, and context tokens alongside quality.

Sample production queries after launch and inspect zero-result searches, low-confidence answers, repeated citations, and stale sources. Add adjudicated failures to the question set. Re-tune when the corpus changes materially; a strategy optimized for short support articles may fail after contracts or tables are added.

Promotion record

Before re-indexing, record the hypothesis, corpus sample, strategies, fixed retrieval settings, question-set revision, metrics, and tradeoffs. Include examples where the candidate loses. State migration duration, dual-index plan, rollback threshold, and how documents arriving during the switch are handled.

Validate citations in the rendered interface. The backend may return a precise span while the product links only to a document home page or exposes a title the reader cannot access. Test users from different permission groups and confirm counts, snippets, and errors do not leak restricted sources. After promotion, compare live query and file-format distributions with the evaluation assumptions.

Keep the old index until the new revision passes reconciliation and freshness checks; an irreversible in-place rebuild makes retrieval regressions harder to isolate.

Decision

Adopt the strategy that improves supported answers and verifiable citations on your question set. Version the parser and re-index only after the candidate clears the regression gate.