LlamaIndex Ingestion, Indexing and Retrieval Explained

LlamaIndex is a data framework for connecting your own documents to a large language model. Its job is the pipeline around retrieval augmented generation: loading source data, splitting it into retrievable units, indexing those units, retrieving the relevant ones at query time, and assembling them into a prompt. Each stage has decisions that matter more than the choice of model.

This page is the conceptual map. For the working code that implements it, see the LlamaIndex quickstart; for the failure modes each stage produces, see retrieval troubleshooting.

Documents Become Nodes

Ingestion starts with loaders that pull from files, databases, APIs, or web pages and produce Document objects. A parser then splits each document into nodes, the atomic units that get embedded and retrieved.

Node size is the single most consequential setting in the pipeline. Small nodes embed precisely but arrive without the surrounding context needed to interpret them. Large nodes carry context but produce diffuse embeddings that match many queries weakly. Splitting along document structure such as headings, sections, or code blocks preserves meaning better than splitting at a fixed character count.

Overlap between adjacent nodes prevents a sentence that spans a boundary from being lost, at the cost of duplicated content in the index.

Metadata attached to nodes is not decoration. Source, section title, document type, and timestamps let you filter retrieval to a subset of the corpus, which is often more effective than improving the embedding. Metadata also makes citation possible, and an answer that cannot cite its source is hard to trust.

Index Types Serve Different Queries

A vector index embeds every node and retrieves by similarity. It is the default and suits questions answerable from a few passages.

A summary index walks every node in sequence. It is expensive but correct for questions that require reading the whole corpus, such as summarization, where similarity search would sample arbitrarily.

Knowledge graph indexes extract entities and relationships and retrieve by traversing them. They handle multi hop questions that connect facts across documents, which vector similarity handles badly, but extraction quality determines whether the graph is useful at all.

Real systems often combine these behind a router that picks an index per query.

Retrieval and Response Synthesis

Retrieval returns candidate nodes. How many you return is a tradeoff: too few and the answer lacks support, too many and the important passage is buried in a long prompt.

Pure semantic similarity misses exact tokens such as identifiers, error codes, and product names. Hybrid retrieval combining keyword and vector search recovers those. A reranking step over the merged candidates then reorders by relevance to the actual question rather than embedding proximity, which usually improves precision more than any tweak to the embedding stage. LlamaIndex exposes that stage as node postprocessors, including a similarity cutoff, several rerankers, and a context reordering step.

Response synthesis assembles retrieved nodes into an answer. Compact mode stuffs everything into one prompt. Refine mode passes over nodes sequentially, updating the answer, which handles more context than fits in a single call at the cost of more calls and more latency.

Vector Stores and Persistence

Any vector store works as long as it holds embeddings and metadata, but embeddings are only comparable when produced by the same model. Changing embedding models requires reindexing everything, so record which model produced an index alongside the data.

Persist the index rather than rebuilding it on every start. Embedding is the slow and expensive part of the pipeline, and rebuilding from scratch on each run is an avoidable cost that is easy to miss in a RAG deployment.

Index size follows directly from these choices: chunk count scales with corpus size divided by chunk size, and memory scales with chunk count times embedding dimensions. The chunking and vector store sizer works the arithmetic so a 384-dimension local model can be compared against a 3,072-dimension hosted one before anything is provisioned.

Where This Sits

None of these decisions are unique to LlamaIndex; the same four stages appear in every retrieval framework, which is why the framework choice matters less than it feels. That argument is worked through in LlamaIndex vs LangChain.