RAG latency budget: where the time goes

OriginChain Team · May 5, 2026

rag performance latency ann tutorial

TL;DR - A conversational RAG loop has a 1-3 second user-perceived budget, and 500-2000ms of it is the LLM call. Embedding, ANN search, hydration and prompt construction should sit under 250ms combined. When they don’t, hydration is usually the reason: serial GETs instead of a parallel batch or an mget.

The budget

Pick your latency target first. Common targets:

Conversational AI: <2s end-to-end. Anything longer feels broken.
Search-with-AI-summary: <3s. People expect search to be slow-ish.
Background agent: <30s. Latency doesn’t matter; reliability does.
Streaming output: <800ms to first token. The rest streams.

This post focuses on the conversational-AI target: 2 seconds end-to-end.

Where the time goes

A typical RAG request has 5 stages:

Embed the query → ~50-200ms (depends on the model + provider)
ANN search → 1-50ms (depends on corpus size + index quality)
Hydrate matched records → 1-100ms (depends on storage + parallelism)
Construct the prompt → ~5-20ms (string ops + token counting)
LLM generation → 500-2000ms (this is most of your budget)

Add latency variance from network round-trips, cold caches, and the occasional 99th-percentile outlier.

The LLM is your bottleneck. Stage 5 is the hard floor. Everything else combined should sit under 250ms or you’re losing real time.

Realistic ANN search numbers

Corpus size	OriginChain p50	OriginChain p99
10k vectors	<1ms	~3ms
100k vectors	~1ms	~5ms
1M vectors	~2ms	~8ms
10M vectors	~5ms	~15ms
100M vectors	(post-optimiser)	(post-optimiser)

These are end-to-end from the SDK call to the result, including the HTTP round-trip from a co-located client. Cross-region adds whatever your network latency is.

What this tells you: ANN itself is rarely the bottleneck below ~10M vectors. If your search is taking 100ms, the time is going somewhere else (network, hydration, embedding regeneration).

The hydration step

Most RAG implementations get back a list of ids from ANN search and then need to fetch the actual content. This is the step that’s surprisingly easy to screw up.

Wrong: serial GETs.

const ids = await oc.vectorSearch("article-embedding", { query: q, top_k: 10 });
const articles = [];
for (const m of ids) {
 articles.push(await oc.get("article", m.id)); // 10 round-trips!
}
// → 50-100ms wasted on round-trip overhead

Right: parallel batch.

const ids = await oc.vectorSearch("article-embedding", { query: q, top_k: 10 });
const articles = await Promise.all(
 ids.map((m) => oc.get("article", m.id))
);
// → 1 round-trip's worth of latency, ~3-5ms

Even better when supported: mget.

const ids = await oc.vectorSearch("article-embedding", { query: q, top_k: 10 });
const articles = await oc.mget("article", ids.map((m) => m.id));
// → 1 actual round-trip + server-side parallelism, ~2-3ms

That’s a 30x improvement on the hydration step from one syntax change. We’ve seen real applications discover this and find half their latency budget by fixing the loop.

The embedding step

Embedding the query is often outside your database - you’re calling OpenAI or an internal model. This is the easiest place to optimize:

Cache aggressively. If the same query was embedded recently, reuse the embedding. A 5-minute cache on common queries is free latency.
Pre-compute embeddings for known structured queries (FAQ entries, common templates).
If your model supports it, use a smaller embedding model for the query and a larger one for the corpus. Asymmetric models exist.

A 100ms embedding call you can avoid is worth 100ms of any other optimization.

The prompt construction step

Looks innocent, often isn’t. Watch for:

Token counting on long contexts. If you’re trimming retrieved content to fit a model’s context window, the trimming logic should be O(n) on the trimmed length, not the original.
String concatenation on long context. Use array join instead of += accumulation.
Reserialization. If the retrieved record is already a JSON string, don’t re-JSON.stringify it.

These look like 5ms problems each but they compound when you’re concatenating dozens of retrieved records.

The LLM step

Mostly out of your control, but a few levers:

Stream the response. Time-to-first-token is what users perceive; total time is less important.
Pick the smallest sufficient model.
Use prompt caching if your provider supports it. The system prompt + retrieved-context structure is often cacheable across requests.
Parallelize tool calls if your loop can. Independent tool invocations are a tree, not a list.

Measuring your own loop

The single most useful thing you can do: instrument every stage with Date.now (or your tracer of choice) and log the per-stage durations on every request. After 100 requests, your bottleneck is obvious.

const t0 = Date.now;
const queryEmbedding = await embed(query);
const t1 = Date.now;
const matches = await oc.vectorSearch("article-embedding", { query: queryEmbedding, top_k: 10 });
const t2 = Date.now;
const articles = await oc.mget("article", matches.map((m) => m.id));
const t3 = Date.now;
const prompt = buildPrompt(query, articles);
const t4 = Date.now;
const response = await llm.complete(prompt);
const t5 = Date.now;

logStageTiming({
 embed_ms: t1 - t0,
 ann_ms: t2 - t1,
 hydrate_ms: t3 - t2,
 prompt_ms: t4 - t3,
 llm_ms: t5 - t4,
});

Plot the percentiles. Find the stage with the highest p99. Fix that one. Repeat.

A worked budget

For a 2-second conversational target, our default budget allocation:

Stage	Target p99	What if it’s higher
Embed	200ms	Cache common queries; smaller model for queries
ANN	20ms	Corpus is too large; consider filter-then-search
Hydrate	10ms	You’re not parallelizing or batching
Prompt	20ms	Token counting / serialization issue
LLM	1500ms	Smaller model; streaming; prompt caching
Network/overhead	250ms	Move client closer to substrate

If your numbers look very different from this, it tells you where to look first.

FAQ

How fast is OriginChain ANN at production scale?

About 1ms p50 at 100k vectors, ~2ms at 1M and ~5ms at 10M, measured end-to-end from a co-located client. 100M is post-optimiser work. Full table above.

Should I co-locate my application with OriginChain?

Yes. A 50ms cross-region RTT on every database call eats your budget fast. Run your application in the same region as your tenant.

Is hydration really 30x faster as parallel batch?

Yes, in real loops we’ve seen. The fix is ~3 lines of code.

Does caching the embedding break personalization?

If the cache key is the raw query string, no - same query string produces the same embedding regardless of user. If you want per-user embeddings, key the cache by (user, query).

What about latency variance?

The p99 is what your users complain about, not the p50. Always measure both.