RAG Chunk Size: Overlap and Citation Checks

Selecting a RAG chunk size and chunk overlap requires aligning your text splitter with your embedding model tokenizer, subtracting prepended metadata tokens from the chunk limit, and verifying retrieval precision and citation entailment to prevent truncation and retrieval failure before deployment. In OpenAI's tiktoken library, the cl100k_base encoding is used by text-embedding-ada-002, text-embedding-3-small, text-embedding-3-large, gpt-4-turbo, gpt-4, and gpt-3.5-turbo, whereas o200k_base is used by gpt-4o and gpt-4o-mini.

For data platform teams designing retrieval pipelines across our AI Data Architecture coverage—after selecting between vector retrieval, graph indexing, and weight adaptation in the Agentic RAG vs GraphRAG and Fine-Tuning Guide—this reference specifies the exact token budget equations, overlap efficiency benchmarks, contextual embedding protocols, and citation offset checks required for production ingestion.

Embedding Token Limits and Metadata-Aware Splitter Mechanics

Configuring a text splitter around character heuristics rather than model-specific tokenizers risks silent sequence truncation or unexpected chunk fragmentation. With tiktoken, the exact token count of a text string for a given encoding is computed by loading the encoding with tiktoken.get_encoding(encoding_name) and taking len(encoding.encode(string)). The maximum input text length for the Azure OpenAI text-embedding-3-small model is 8,191 tokens (equivalent to around 6,000 words at roughly four characters per token for common OpenAI models), and staying under the token limit prevents data loss due to truncation. In SentenceTransformers, the SentenceTransformer max_seq_length property returns the maximal input sequence length for the model (Optional[int]), and any longer inputs will be truncated.

LlamaIndex's SentenceSplitter.split_text_metadata_aware calculates metadata_len = len(self._tokenizer(metadata_str)) and subtracts it from chunk_size via effective_chunk_size = self.chunk_size - metadata_len, raising a ValueError if effective_chunk_size <= 0 and warning if effective_chunk_size < 50 tokens.

Production orchestration frameworks enforce distinct boundary rules, separator hierarchies, and overlap eviction loops when assembling chunks:

RAG Chunk Overlap and Retrieval Efficiency Benchmarks

Setting a fixed chunk size and overlap ratio trades off index cardinality against redundant token retrieval across neighboring chunks:

• For fixed-size parsing with overlap, the Microsoft Azure Architecture Center recommends using bidirectional encoder representations from transformers (BERT) tokens instead of character counts because BERT tokens are based on meaningful units of language and preserve more semantic information than character counts.
• When chunking data based on fixed size, Azure AI Search recommends starting with a chunk size of 512 tokens (approximately 2,000 characters) and an initial overlap of 25%, which equals 128 tokens, and notes that fixed-size paragraphs of 200 words or 600 characters can also use 10-15% overlap.
• In Azure AI Search's Text Split skill, textSplitMode supports pages (default, multiple sentences per chunk) and sentences (single sentence per chunk); for pages mode, pageOverlapLength must be less than half of maximumPageLength, maximumPagesToTake defaults to 0 (take all pages/chunks), and the recommended character default is maximumPageLength=2000 with pageOverlapLength=500.
• On NASA's 200-page Earth at Night PDF (average 189 cl100k_base tokens per page, maximum 1,583 tokens), Azure AI Search's Text Split skill in pages mode produces 172 chunks at maximumPageLength=1000 with 0 overlap vs. 216 chunks with 200 overlap, 85 chunks at 2000 with 0 overlap vs. 113 chunks with 500 overlap, 34 chunks at 5000 with 0 overlap vs. 38 chunks with 500 overlap, and 13,361 chunks in sentences mode.

Evaluating retrieval quality solely at the document level hides how much irrelevant text or duplicate overlap is injected into the generator context window. For a query q with relevant excerpt tokens t_e and retrieved chunk tokens t_r in chunked corpus C, Chroma defines token-level Intersection over Union as IoU_q(C) = |t_e cap t_r| / (|t_e| + |t_r| - |t_e cap t_r|), Precision_q(C) = |t_e cap t_r| / |t_r|, Recall_q(C) = |t_e cap t_r| / |t_e|, and F_1 = 2 * |t_e cap t_r| / (|t_e| + |t_r|). Chroma's chunking evaluation dataset spans 5 corpora totaling 328,208 cl100k_base tokens and 472 filtered queries: State of the Union (10,444 tokens, 76 queries), Wikitext (26,649 tokens, 144 queries), Chatlogs (7,727 tokens, 56 queries), Finance (166,177 tokens, 97 queries), and Pubmed (117,211 tokens, 99 queries). Chroma found that LangChain's RecursiveCharacterTextSplitter default separators ['\n\n', '\n', ' ', ''] commonly produced very short chunks that performed poorly compared to TokenTextSplitter, so Chroma modified the separator list to include sentence punctuation: ['\n\n', '\n', '.', '?', '!', ' ', ''].

In Chroma's token-level IoU formula, each relevant excerpt token t_e among t_r is counted only once in the numerator while all retrieved tokens are counted in |t_r| in the denominator to penalize redundant excerpt tokens appearing in multiple overlapping chunks, and Precision_Omega measures precision when all chunks containing excerpt tokens are retrieved.

Empirical benchmarks across Chroma, LlamaIndex, and Amazon Bedrock Knowledge Bases show how chunk size and overlap affect recall, token efficiency, and response faithfulness:

Preserving Cross-Chunk Context with Contextual Retrieval and Late Chunking

Splitting a document prior to embedding severs pronouns, entity references, and temporal scope across chunk boundaries unless document-wide context is restored before indexing. Anthropic's Contextual Retrieval and Jina AI's Late Chunking address cross-chunk context loss at two different stages of the indexing pipeline:

• Traditional RAG chunking destroys context when an isolated chunk such as 'The company's revenue grew by 3% over the previous quarter.' lacks the company name (ACME Corp) or relevant time period (Q2 2023 and previous quarter revenue $314 million); Anthropic's Contextual Retrieval uses Claude 3 Haiku to prepend usually 50-100 tokens of chunk-specific context to each chunk before embedding (Contextual Embeddings) and before creating the BM25 index (Contextual BM25).
• Anthropic notes that if a knowledge base is smaller than 200,000 tokens (about 500 pages of material), developers can include the entire knowledge base directly in the prompt without RAG, using prompt caching to reduce latency by > 2x and costs by up to 90%; for larger knowledge bases, assuming 800 token chunks, 8k token documents, 50 token context instructions, and 100 tokens of context per chunk, the one-time cost to generate contextualized chunks with prompt caching is $1.02 per million document tokens.
• Measured by 1 minus recall@20 (the percentage of relevant documents failing to be retrieved within the top 20 chunks, after testing 5, 10, and 20 chunks and reranking the top 150 initial chunks down to the top 20), Anthropic's Contextual Embeddings reduced the top-20-chunk retrieval failure rate by 35% (5.7% to 3.7%), combining Contextual Embeddings with Contextual BM25 reduced it by 49% (5.7% to 2.9%), and adding Cohere reranking reduced it by 67% (5.7% to 1.9%).
• In Jina AI's Late Chunking Berlin example using jina-embeddings-v2-small, when sentences containing co-references like 'its' and 'the city' are embedded after naive splitting vs. late chunking, cosine similarity to the query 'Berlin' rises from 0.8486 to 0.8495 on the first sentence ('Berlin is the capital and largest city of Germany'), jumps from 0.7084 to 0.8249 on the second sentence ('Its more than 3.85 million inhabitants...'), and jumps from 0.7535 to 0.8498 on the third sentence ('The city is also one of the states of Germany...').
• Unlike naive chunking (which splits text into sentences or paragraphs first and embeds each chunk independently), Late Chunking (Algorithm 1) first tokenizes the full text up to the model's maximum token length (8192 tokens for jina-embeddings-v2-small, roughly ten pages of standard text), runs the transformer layers once to produce contextualized token-level representations ϑ1, ..., ϑm, and then applies mean pooling over each chunk's token span (cue_start to cue_end) to produce context-aware chunk embeddings.
• For documents longer than an embedding model's maximum token length l_max, Long Late Chunking (Algorithm 2) splits the document into macro chunks of l_max tokens augmented with an overlap length of omega tokens (vartheta_{i_start + omega}, ..., vartheta_{i_end}) that provide supplementary context during late chunking.
• Averaged across three models (jina-embeddings-v2-small, jina-embeddings-v3, and nomic-embed-text-v1) and four BeIR datasets (SciFact, NFCorpus, FiQA, and TRECCOVID) in Table 2, late chunking improves mean nDCG@10 over naive chunking by 3.63% relative (1.9% absolute, 52.4% to 54.3%) for 5-sentence chunks, 3.46% relative (1.8% absolute, 52.2% to 54.0%) for 256-token fixed-size chunks, and 2.70% relative (1.5% absolute, 52.4% to 53.8%) for semantic sentence boundaries.
• In Jina AI's evaluation on documents truncated at 8192 tokens, late chunking consistently beats naive chunking on NFCorpus and for small chunk sizes, whereas on synthetic Needle-8192 and Passkey-8192 tasks—where short relevant information is placed inside a document of unrelated text—late chunking is not useful because the additional document context is irrelevant.

RAG Citation Accuracy: Offset Tracking and NLI Entailment Checks

Chunk granularity directly governs whether a model can attribute generated statements to exact source spans and pass natural language inference entailment checks. Anthropic Claude Citations returns three location schemas with 0-indexed document_index: char_location for plain text with 0-indexed start_char_index and exclusive end_char_index (such as 0 to 20 for 'The grass is green. ' and 20 to 36 for 'The sky is blue.'), page_location for PDFs with 1-indexed start_page_number and exclusive end_page_number, and content_block_location for custom content with 0-indexed start_block_index and exclusive end_block_index.

To audit RAG citation accuracy across retrieval pipelines and API responses, combine the Princeton NLP ALCE entailment definitions, Anthropic Claude Citations block rules, and structured document extraction pointers:

Enterprise document pipelines pair these API citation mechanics with pre-ingestion layout extraction and source-position pointers:

• In the Microsoft Build 2026 DEM331 session ('Agents Can Reason But They Cannot Read') indexed on OnlineSummit, the Microsoft Content Understanding team notes that handing raw scanned PDFs or complex tables to a model degrades quality and increases token cost simultaneously (1:40), whereas running a parse, classify and extract pipeline before the model has a vendor-claimed 85 per cent reduction in tokens (20:22).
• In the Microsoft Build 2026 Foundry Tools session indexed on OnlineSummit, Joe Filcik highlights that each extracted document field carries both a confidence score and a pointer back to its position in the source document (29:29) to enable selective human review, and distinguishes answers that exist in a document and need finding from answers that must be constructed by reasoning across a chain of amendments or sections (30:57).

Production Chunking and Citation Verification Worksheet

Before locking your indexing pipeline or shipping user-facing citations, verify your configuration across the five engineering gates below:

Once your team has validated chunk token budgets, overlap efficiency, and NLI citation entailment, compare higher-level retrieval control loops in the Agentic RAG vs GraphRAG and Fine-Tuning Guide, review our verification workflow on Methodology, and browse related engineering sessions in AI Data Architecture.

Sources

• OpenAI Cookbook, How to Count Tokens with Tiktoken.
• SentenceTransformers Documentation, SentenceTransformer Model Reference.
• LlamaIndex Core Specification, SentenceSplitter Node Parser Source.
• LangChain Text Splitters Specification, Base TextSplitter, TokenTextSplitter, and RecursiveCharacterTextSplitter Source.
• Microsoft Azure Architecture Center, Develop a RAG Solution: Chunking Phase.
• Microsoft Azure AI Search Documentation, Chunk Large Documents for Vector Search.
• Chroma Technical Report (Brandon Smith and Anton Troynikov), Evaluating Chunking Strategies for Retrieval.
• LlamaIndex Engineering Blog, Evaluating the Ideal Chunk Size for a RAG System Using LlamaIndex.
• AWS Amazon Bedrock User Guide, How Document Chunking Works in Knowledge Bases.
• Anthropic Research, Introducing Contextual Retrieval.
• Jina AI Research (Michael Günther et al.), Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models.
• Princeton NLP (Tianyu Gao et al.), Enabling Large Language Models to Generate Text with Citations.
• Anthropic Claude API Documentation, Citations Guide.
• OnlineSummit Talk Archive, Microsoft Build Agent Content Velocity and Agent Tools and Documents Sessions.