# The RAG Cost Curve > When does a retrieval index actually beat live search? Measured break-even points, not vendor claims: 334 queries a day on a short agent history, never on code the model already knows, and 24 on code it has never seen. Author: Simon Hearne (https://simonhearne.com/about/) Published: 2026-09-15 Canonical: https://simonhearne.com/2026/rag-cost-curve/ Tags: VectorDB, Zilliz, Milvus, talk, conference Slides: https://talks.simonhearne.com/rag-cost-curve/ Link: Reproduce every number: https://github.com/simonhearne/rag-cost-curve --- My abstract promised a number. Measuring it changed the claim. ## About the talk "Just use RAG" and "RAG is dead" are both answers to a question nobody asked precisely. An index costs money to build, to store and to keep current. Live search costs money every time you run it. Which one wins is not a matter of taste, it is a crossover point, and it moves depending on what you are searching and how often. This talk puts a number on that crossover for two workloads, conversation memory and code search, and shows the measurements behind it. Every figure comes from a committed CSV, every chart renders from that CSV at build time, and the whole study runs on a laptop with Docker. The honest finding is that the index does not always win. It loses badly in one of the cases I measured, and the talk spends time there rather than skipping past it. ## What's covered - **Defining break-even.** What goes into the cost of an index against the cost of live search, and why "worth it" is meaningless without naming what you are comparing against. - **Quantisation and dimensionality reduction.** Scalar, product and RaBitQ quantisation, PCA and Matryoshka embeddings, what each costs you in recall, and how refinement claws precision back. SQ8 won on its own. - **Does recall even matter?** Recall barely moves answer quality until it falls off a cliff, mostly because most questions never had a choice. Aim for recall@k of about 0.95 and stop optimising. - **Two measured workloads.** Conversation memory, and code search against three repositories: one the model already knows, one it has never seen, and the same repository stuffed into context instead of grepped. - **The cost curves, assembled.** Where each workload crosses over, and where it never does. ## What the numbers said **Live search wins** when the corpus is small and the model already knows it: - A short agent history, at 98k tokens of conversation memory, needs **334 queries a day** before a dedicated index pays for itself. - On `fastapi`, a repository the model already knows, break-even is **infinite at every size**. No query volume repays the index, because grep is cheaper every time. - Hold the prompt cache at 99% rather than the measured rate and the crossover is still **232 queries a day**, so the cache assumption is not doing all the work. **The index wins** as the corpus grows, or when the model has never seen it: - Conversation memory collapses fast: **15 queries a day** at 392k tokens and **10** at 1.2M, against 334 at 98k. - On code the model has never seen, the index is **40% cheaper per correct answer** than grep, and more accurate with it: 39 of 40 correct against 36. - Against stuffing the same repository into context rather than grepping it, the index repays at **12 queries a day**: $0.29 a query re-sent against $0.067 indexed. That last pair is the sharpest thing in the talk. Same repository, same index, same box, same prices. The only thing that changes is what the live arm does, and the answer flips from "no volume repays this" to "twelve queries a day". The question is never "is retrieval worth it", it is "worth it against what". ## Reproduce it Everything in the talk runs on your own machine, given Docker with 16GB of RAM or more: ```bash make setup && source .venv/bin/activate make up # docker compose up -d + Milvus health wait make data-1m # or data-10m jupyter lab # notebooks 01 to 06b in order ``` If you want the tooling rather than the study, [memsearch](https://github.com/zilliztech/memsearch) gives Claude semantic recall over its own conversation history, and [claude-context](https://github.com/zilliztech/claude-context) adds semantic code search to coding agents over MCP. Presented at [Haystack Europe 2026](https://haystackconf.com/speaker/simon-hearne/) in Berlin on 15 September 2026.