LlamaIndex Quickstart: Build a RAG Pipeline in Python

A working LlamaIndex retrieval pipeline is about five lines of Python. The interesting part is everything those five lines hide: which package you installed, where the embeddings went, what happens on the second run, and why the answer is wrong when the source document plainly contains it.

This walkthrough follows the framework’s own starter example, then covers the three decisions that turn a demo into something you can run twice.

Before you install anything

LlamaIndex describes itself as a framework for building LLM-powered agents over your data, and its own documentation breaks a RAG application into five stages: loading, indexing, storing, querying, and evaluation. Keep the first four in mind while you build, because every error you hit belongs to exactly one of them, and knowing which stage failed is most of the debugging.

You need three things to start:

• Python and a virtual environment.
• An API key for whatever model you intend to use. The starter example reads OPENAI_API_KEY from the environment.
• A folder of documents. Anything the default reader can parse: text, Markdown, PDF, Word.

Install: one package or many

Since version 0.10 the project is not a single library. The ecosystem is a set of namespaced packages: llama-index-core holds the framework, and every model provider, vector store, and reader ships separately as llama-index-llms-openai, llama-index-vector-stores-qdrant, and so on. The plain llama-index name is a starter bundle, documented as core plus llama-index-llms-openai, llama-index-embeddings-openai and llama-index-readers-file, which is why a beginner tutorial works after one install and a production service usually pins core plus only the integrations it actually imports.

pip install llama-index
export OPENAI_API_KEY=sk-...

Install the bundle to learn the shape of the API. Move to llama-index-core plus explicit integrations when you care about what is in your image.

The five lines

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What does this corpus say about X?")
print(response)

That is the whole loop, and it maps one-to-one onto the documented stages.

SimpleDirectoryReader is the loading stage. It walks the folder, picks a parser per file extension, and returns Document objects. It is deliberately dumb about structure: a PDF becomes text, and if the PDF was a scanned image the text is empty. Print the length of documents before doing anything else, because an empty list here produces a confident, useless answer four lines later.

VectorStoreIndex.from_documents does the indexing stage, and it does more than the name suggests. It splits each document into nodes with the default sentence splitter, sends every node to an embedding model, and stores the vectors. This is the step that costs money and time. In the starter configuration the embeddings come from OpenAI and the vectors live in memory.

as_query_engine() builds the querying stage: a retriever, an optional set of node postprocessors, and a response synthesizer. Calling it with no arguments accepts every default.

query() embeds your question, retrieves the closest nodes, packs them into a prompt with the question, and returns the model’s answer along with the source nodes it used. Always look at response.source_nodes while you are learning. It is the difference between “the model is bad” and “the retriever handed the model the wrong paragraph”, and those have completely different fixes.

Configure once, with Settings

The first thing most people want to change is the model. The current way is the global Settings object:

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.chunk_size = 512

If you find a tutorial that builds a ServiceContext and passes it around, that tutorial predates version 0.10. ServiceContext was deprecated in favour of Settings in 0.10 and removed in 0.11, so the code will not run on a current install. This is the single most common reason a copied snippet fails with an import error, and the migration guide in the sources exists specifically for it.

Settings attributes are lazily instantiated, so setting a value you never use costs nothing. Per-call overrides still work: anything you pass directly to a query engine or index beats the global.

Persist the index, or pay twice

The five-line version rebuilds the entire index on every run. Embedding is the slow, metered part of the pipeline, and re-embedding an unchanged corpus at every process start is pure waste.

index.storage_context.persist("storage")

and on the next run:

from llama_index.core import StorageContext, load_index_from_storage

storage_context = StorageContext.from_defaults(persist_dir="storage")
index = load_index_from_storage(storage_context)

Wrap those in a check for whether storage/ exists and the second run becomes near-instant. Two rules go with this. Record which embedding model produced the index, because vectors from different models are not comparable and a swapped model silently returns nonsense. And treat the persisted directory as a build artifact tied to a corpus version, not as a database you edit by hand.

Moving to a real vector store

The default store keeps vectors in the Python process. That is correct for a prototype and wrong for anything with more than one worker. A dedicated vector store separates the index lifecycle from the app lifecycle, and Qdrant is the usual first step in this stack:

pip install llama-index-vector-stores-qdrant

import qdrant_client
from llama_index.core import StorageContext, VectorStoreIndex
from llama_index.vector_stores.qdrant import QdrantVectorStore

client = qdrant_client.QdrantClient("<qdrant-url>", api_key="<qdrant-api-key>")
vector_store = QdrantVectorStore(client=client, collection_name="documents")
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)

The application code above this line does not change. That is the argument for the abstraction: the retriever, postprocessors, and query engine are unaware of which store is underneath.

Before you provision anything, it helps to know roughly how large the index will be. Chunk count scales with corpus size divided by chunk size, and memory scales with chunk count times embedding dimensions times four bytes, plus graph overhead for an approximate index. The site’s chunking and vector store sizer does that arithmetic so you can compare a 384-dimension local embedding model against a 1,536-dimension hosted one before committing to an instance size.

Query engine or chat engine

as_query_engine() answers one question with no memory of the last one. That is the right default for search-style traffic and for anything you evaluate, because each call is independent and reproducible.

as_chat_engine() adds conversation state, which changes the retrieval problem rather than just the interface. A follow-up such as “and what about the second one?” has almost no retrievable content on its own, so the engine has to rewrite it against the history before retrieval can work. If a chatbot answers the first question well and the third question badly, suspect that rewriting step before you suspect the index.

Start with a query engine. Add chat only when the product genuinely needs multi-turn, and evaluate both separately.

The four errors everyone hits first

Error	What it means	Fix
ImportError on ServiceContext	Snippet predates version 0.10	Use the Settings object instead
ModuleNotFoundError on a vector store or LLM import	Integration package not installed	Install the specific llama-index-* package
Confident answers with no relation to your files	Loader returned nothing	Check len(documents) and node content
Startup slow on every run	Index rebuilt each time	Persist and load from storage/

Three of the four are loading or packaging problems, not model problems. That ratio holds well beyond the first day.

Evaluate before you believe it

A RAG pipeline that answers fluently and wrongly looks identical to one that works. LlamaIndex ships evaluation modules for both halves of the problem: retrieval metrics such as hit rate and mean reciprocal rank for whether the right node came back, and response evaluators for faithfulness, context relevancy, answer relevancy, and correctness for whether the answer used it properly. Many of these do not require ground-truth labels, and the framework can synthetically generate question and context pairs from your own text, so there is no excuse for shipping with zero measurements.

Build the evaluation set on day one, while the corpus is small. It is the only way to tell whether a change to chunk size helped.

What to read next

• The concepts behind the five lines, in one page: LlamaIndex ingestion, indexing and retrieval explained.
• Picking the framework at all: LlamaIndex vs LangChain, which to use for RAG.
• When the pipeline runs but the answers are wrong: LlamaIndex retrieval troubleshooting.