LlamaIndex Query Engine vs Chat Engine for Production

A RAG endpoint can look healthy at p50 and still develop a nasty p99 tail as conversations get longer. The llamaindex query engine vs chat engine choice defines much of that failure surface: LlamaIndex describes the query engine as an interface for standalone questions, while the chat engine is its stateful, multi-turn analogue. That is a capacity and isolation decision, not merely a UI preference.

Short answer

Use a query engine when every request should stand alone: search, document Q&A, batch evaluation, an API called by another service, or a retrieval tool exposed to an agent. The caller sends a complete question; the engine retrieves context and synthesizes a response.

Use a chat engine when later turns depend on earlier ones: “Compare it with the previous plan,” “Why did that fail?”, or “Show the source for the second option.” It owns or receives conversation history so it can interpret those references.

A chat engine is not automatically a better query engine. Its behavior depends on chat_mode. The official chat-engine usage guide distinguishes context, condense_question, condense_plus_context, and simple. The simple mode talks to the LLM without using the knowledge base. The condense modes rewrite a follow-up into a standalone retrieval query, which can improve retrieval but can also add model latency and distort intent.

Query engine vs chat engine side by side

Concern	Query engine	Chat engine
Primary call	query() / aquery()	chat() / achat() / stream_chat()
State	No conversation state between independent calls	Conversation history scoped to a session
Retrieval input	The request’s query string	Raw message or a history-aware rewritten query, depending on mode
Typical path	retrieve → synthesize	optionally condense → retrieve → synthesize → update memory
Best fit	Q&A APIs, batch jobs, evals, agent tools	support assistants, research copilots, guided exploration
Main operational risk	retrieval or generation regression	the same risks, plus memory growth and session isolation

Both interfaces can sit on the same VectorStoreIndex, HNSW index, embedding model, and response synthesizer. That shared retrieval layer is the RAG part: generation is conditioned on records fetched from non-parametric storage, the basic architecture formalized in the original RAG paper. The difference is what happens before retrieval and what state survives afterward.

Do not share one mutable chat-engine instance across tenants. Keep history under a session key, enforce tenant metadata filters at retrieval, and cap or summarize memory. A query engine is usually the cleaner shared service object because the complete input arrives with each request.

The metric that matters: p99 latency by turn depth

Track end-to-end p99 latency grouped by engine and bounded turn bucket. Formally, p99 is the smallest duration t for which at least 99% of observed requests complete in time t or less:

p99 = inf { t : F̂(t) ≥ 0.99 }

This beats mean latency and an unsegmented p50. A condense-mode chat request may perform a history rewrite before retrieval and generation, and prompt size can grow with the session. Those slow paths disappear in the average. Bucketing turns as 1, 2-4, 5-8, and 9+ exposes the slope without turning every turn number into a new time series. Prometheus recommends server-side quantile calculation from histograms when results must be aggregated across replicas; averaging precomputed quantiles is statistically invalid (Prometheus histogram guidance).

Latency is not the quality gate. On a golden set of follow-up conversations, also track MRR over the first relevant retrieved chunk: MRR = (1/|Q|) Σ 1/rankᵢ, assigning zero when no relevant chunk is returned. LlamaIndex’s evaluation documentation lists MRR, hit rate, and precision for retrieval evaluation. Compute it separately for first turns and follow-ups; otherwise easy standalone questions hide failed reference resolution.

Wiring it up

This minimal setup creates both paths over the same index and exports low-cardinality Prometheus metrics. Create one chat engine per session or attach persistent session memory in the serving layer; the example deliberately does not put a global chat object behind every user.

from typing import Literal

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
from prometheus_client import Counter, Histogram, start_http_server

REQUEST_SECONDS = Histogram(
    "llamaindex_request_duration_seconds",
    "End-to-end LlamaIndex request latency",
    ("engine", "turn_bucket"),
    buckets=(0.25, 0.5, 1.0, 2.0, 5.0, 10.0),
)
REQUESTS = Counter(
    "llamaindex_requests_total",
    "Completed LlamaIndex requests",
    ("engine", "status"),
)


def bucket_turn(turn: int) -> str:
    if turn == 1:
        return "1"
    if turn <= 4:
        return "2-4"
    if turn <= 8:
        return "5-8"
    return "9+"


documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine(similarity_top_k=4)


def new_chat_session():
    return index.as_chat_engine(
        chat_mode="condense_plus_context",
        similarity_top_k=4,
    )


def invoke(
    kind: Literal["query", "chat"], engine, prompt: str, turn: int = 1
) -> str:
    status = "error"
    try:
        with REQUEST_SECONDS.labels(kind, bucket_turn(turn)).time():
            response = engine.query(prompt) if kind == "query" else engine.chat(prompt)
        status = "ok"
        return str(response)
    finally:
        REQUESTS.labels(kind, status).inc()


start_http_server(9108)

Plot p99 across replicas with:

histogram_quantile(
  0.99,
  sum by (le, engine, turn_bucket) (
    rate(llamaindex_request_duration_seconds_bucket[5m])
  )
)

For component attribution, add LlamaIndex OpenTelemetry traces and inspect the condense, retriever, reranker, and LLM spans rather than blaming the vector database on sight. The broader LlamaIndex observability guide covers OpenTelemetry, MLflow, Weights & Biases Weave, and Arize Phoenix integrations. A practical dashboard should also carry QPS, errors, prompt tokens, output tokens, time-to-first-token (TTFT), and tokens/sec; SentryML covers the surrounding model-monitoring patterns.

What you’ll see

A healthy query path has similar p50/p95/p99 across repeated independent requests after caches warm. A healthy chat path may cost more on follow-ups, but its latency should plateau once memory is bounded. Follow-up MRR should justify that cost.

A bad chart has chat p99 rising with every turn bucket while query latency stays flat. Check prompt-token growth and the condense span first. If both paths degrade together, look downstream at the embedding service, vector index, reranker, or vLLM/Triton serving tier. If latency is fine but follow-up MRR falls, inspect rewritten queries and stale history rather than adding GPUs.

Which one should you deploy?

Start with a query engine unless conversational context is part of the product contract. It is simpler to cache, replay, shadow, and regression-test. Add a separate chat endpoint when the golden set contains genuine context-dependent turns, then shadow it on sanitized transcripts and compare follow-up MRR, p99, TTFT, and cost per successful conversation before a canary deploy.

Caveats

• False alarms: Do not compare blocking query latency with streaming chat TTFT, or a warm query cohort with cold chat sessions. Match model, retriever, similarity_top_k, response mode, and traffic shape.
• Sampling cost: Full OpenTelemetry traces can store prompts, retrieved chunks, and outputs. Sample routine traffic, retain errors and slow traces, and scrub secrets and personal data.
• Cardinality blowup: Never put session_id, user_id, raw query text, document ID, or exact turn number in Prometheus labels. Keep those in sampled traces or logs.
• Label leakage: Preserve raw multi-turn prompts in the eval set. Supplying the expected standalone rewrite to the retriever tests the answer key, not the chat engine.
• Security: Prior turns and retrieved documents are untrusted input. Isolate tenant memory and filters, and apply prompt-injection controls before either path can call tools; GuardML tracks defensive patterns for these pipelines.

Related across the network

• Best AI Monitoring Tools 2026: LLM Observability Compared — aiincidents.org
• RAG vs Fine-Tuning Explained: A Production Decision Guide — ragstackguide.com
• 6 Model Monitoring Tools Compared for 2026 — sentryml.com
• Model Monitoring in Production: Metrics and Triggers — sentryml.com
• OpenAI Named a Leader in Enterprise Coding Agents by Gartner — sentryml.com