LlamaIndex Chunk Size: Tune chunk_size and chunk_overlap

LlamaIndex chunk size is configured through Settings.chunk_size or a node parser such as SentenceSplitter. This guide covers where to set chunk_size and chunk_overlap, how to apply a splitter to one index, and how to compare configurations with LlamaIndex evaluators.

For document boundaries and overlap choices across frameworks, start with RAG Stack Guide’s RAG chunking strategy: picking chunk size and overlap. For the pipeline around the parser, see how documents become retrievable nodes.

What chunk_size actually controls

The default LlamaIndex document-to-index path uses SentenceSplitter to create Node objects for embedding and retrieval. It tries to preserve sentence and paragraph boundaries, with documented defaults of chunk_size=1024 and chunk_overlap=20 (LlamaIndex node parser docs).

from llama_index.core.node_parser import SentenceSplitter
from llama_index.core import Settings, VectorStoreIndex

splitter = SentenceSplitter(
    chunk_size=512,
    chunk_overlap=50,
)

# apply globally
Settings.text_splitter = splitter

# or per-index via transformations
index = VectorStoreIndex.from_documents(
    documents,
    transformations=[splitter],
)

chunk_size is a token budget using the splitter’s tokenizer, not a character count. chunk_overlap sets the overlap budget between adjacent chunks; inspect the generated nodes because sentence boundaries can affect the actual overlap.

Settings.chunk_size or per-index transformations

To keep the default splitter, set Settings.chunk_size = 512 and Settings.chunk_overlap = 50 before constructing the index. To choose the splitter itself, use Settings.text_splitter as above. These are shared defaults; passing transformations=[splitter] to VectorStoreIndex.from_documents() makes the parser choice explicit for that index. The Settings guide documents both approaches.

Apply a changed splitter when ingesting documents again. Loading previously persisted nodes does not split them with the new settings. Keep each evaluation index separate so a sweep compares the nodes generated by each configuration.

SentenceSplitter and the other LlamaIndex parsers

SentenceSplitter isn’t the only option, and the choice of parser matters as much as the chunk_size value supplied to it (LlamaIndex node parser modules reference):

• TokenTextSplitter: splits on raw token count with a configurable separator, same 1024/20 defaults, no sentence awareness.
• SemanticSplitterNodeParser: uses embedding similarity between sentences to place breakpoints adaptively, controlled by breakpoint_percentile_threshold instead of a fixed size.
• SentenceWindowNodeParser: indexes single sentences (window_size=3 of context stored in metadata), meant to pair with MetadataReplacementPostProcessor so retrieval is precise but synthesis still gets surrounding context.
• HierarchicalNodeParser: builds parent/child chunks at multiple sizes (chunk_sizes=[2048, 512, 128] by default) so AutoMergingRetriever can retrieve small and return the enclosing parent when several children match.

Using SentenceSplitter with a single fixed chunk_size selects the simplest option, not necessarily the best one for a given retrieval pattern.

Metadata consumes part of the chunk_size budget

SentenceSplitter.split_text_metadata_aware() subtracts the metadata token count from chunk_size before splitting the text. If that leaves no text budget, it raises a ValueError; a remaining budget below 50 tokens produces a warning. The SentenceSplitter API reference shows these checks. Inspect long metadata values when a small chunk_size produces unexpectedly short nodes. Increase the budget or shorten unnecessary metadata while preserving fields needed for filtering and citations.

Evaluate chunk_size with LlamaIndex response evaluators

LlamaIndex’s published chunk-size evaluation compares faithfulness, relevancy, and response time using a financial filing. Treat its setup as an example of an evaluation process; it does not establish the right parameter for another corpus.

The evaluation documentation separates response evaluation from retrieval evaluation. FaithfulnessEvaluator checks whether the answer is supported by retrieved context, while RelevancyEvaluator checks relevance to the query. Retrieval metrics such as hit rate and MRR answer a different question: whether the expected evidence was retrieved. Record both kinds of evidence when deciding which splitter configuration to keep.

Running a chunk-size sweep

Use representative questions from your own documents. The following loop assumes documents, eval_questions, Settings.llm, and Settings.embed_model are already configured. It illustrates a sweep, not a measured result. Keep the corpus, questions, models, and evaluator prompts fixed between configurations:

from llama_index.core import VectorStoreIndex, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.evaluation import FaithfulnessEvaluator, RelevancyEvaluator

faithfulness_eval = FaithfulnessEvaluator(llm=Settings.llm)
relevancy_eval = RelevancyEvaluator(llm=Settings.llm)

results = {}
for chunk_size, top_k in [(256, 8), (512, 4), (1024, 2), (2048, 1)]:
    splitter = SentenceSplitter(chunk_size=chunk_size, chunk_overlap=int(chunk_size * 0.1))
    index = VectorStoreIndex.from_documents(documents, transformations=[splitter])
    query_engine = index.as_query_engine(similarity_top_k=top_k)

    scores = []
    for question in eval_questions:
        response = query_engine.query(question)
        f = faithfulness_eval.evaluate_response(response=response)
        r = relevancy_eval.evaluate_response(query=question, response=response)
        scores.append((f.passing, r.passing))

    results[chunk_size] = scores

The loop keeps chunk_size * similarity_top_k at 2,048 tokens. This is a nominal retrieved-text budget, not the final prompt length: metadata, overlap, short nodes, and synthesis affect what reaches the model. LlamaIndex’s Basic Strategies guide similarly halves chunk size and doubles similarity_top_k. Record both parameters rather than attributing a result to chunk size alone. The example also holds the overlap ratio near 10%; this is an illustrative choice, not a documented optimum.

Interpreting the results

Inspect failed questions alongside their retrieved nodes. If an answer loses a qualification at a chunk boundary, compare the neighboring nodes and the configured overlap. If the correct evidence is present but the answer is unsupported, inspect synthesis as well as the splitter. The score alone does not identify which stage failed.

Compare response quality, retrieval metrics, and latency on the same questions. Treat the sweep as a way to select candidates for further evaluation, not as a reason to expect one score curve or a universal winning range.

Caveats

Corpus-dependence is real, not a hedge. A 10-K filing (LlamaIndex’s benchmark corpus) is long-form, section-structured prose. Chat transcripts, code, tables, and short FAQ entries all chunk differently, and a 1024-token default can span multiple unrelated FAQ answers in one chunk. The sweep should be rerun on the target data before any published number, including this one, is trusted.

LLM evaluation is not ground truth. Review a sample of the evaluator’s passing and failing judgments. Keep the judge configuration fixed and avoid selecting a parameter solely on a small score difference.

Inspect overlap in the actual nodes. Check whether repeated passages consume the retrieved context and whether boundary-spanning evidence survives. Compare chunk_overlap settings on the same questions before increasing overlap across the corpus.

Chunk size should not be tuned in isolation from retrieval architecture. If a pipeline shows failures that look like a chunking problem, such as answers missing context that is clearly in the source document, the team should check whether SentenceWindowNodeParser or HierarchicalNodeParser with AutoMergingRetriever solves it structurally before continuing to hand-tune a single chunk_size integer. For teams running retrieval pipelines in production long enough to need drift monitoring on embedding distributions or query patterns, sentryml.com covers the ML observability side of that problem once the pipeline is live, not just at indexing time.