# Comparing RAGs, Part 2: the benchmark

In [Part 1](https://dev.to/skucherenko/comparing-rags-part-1-the-framework-e9p), I laid out why "better" RAG isn't about raw accuracy: it's about whether your
data stays inside your own infrastructure or ends up as a standing index on someone else's. Six approaches got tested
head-to-head against the same 10-question golden set, `payments-rag`'s own hand-rolled build, `openai-file-search`,
Google `NotebookLM`, and three library builds (`Haystack`, `LlamaIndex`, `LangChain`/`LangGraph`). Here's how each was
actually built, and what happened when they ran.

## Development

The source code lies in a [GitHub repo](https://github.com/KucherenkoSerhiy/payments-rag).
But the inner works across all six implementations can be summarized in the following sections.

### System Design

![System design: one test client drives all six systems into a shared adapter, judge and score](https://serhiykucherenko.dev/images/comparing-rags/p2-system-design-dark.png)

This project consists of the following components:

1. **Corpus**: the raw PDFs (three SEPA/ISO 20022 rulebooks), the input every system indexes.
2. **Indexing** (one-time, per system): chunk the corpus, embed each chunk, store as vectors.
   `payments-rag`/`Haystack`/`LlamaIndex`/`LangChain` all do this themselves; `openai-file-search` and `NotebookLM` do
   it
   inside the vendor's own managed service instead.
3. **Embedding**: turns text (a question, or a chunk) into a vector. The same OpenAI model across every system except
   `NotebookLM`, whose embedding step is entirely internal/opaque.
4. **Retrieval**: given a question's embedding, find the closest stored chunks. `pgvector` similarity search for
   `payments-rag`; each framework's own equivalent for the rest.
5. **Generation**: question + retrieved chunks → an LLM → the answer. `Claude` for `payments-rag`, `gpt-4o` for most
   framework builds and `openai-file-search`, `Gemini` for `NotebookLM`.
6. **Adapter** (comparison-only, not part of any system's real architecture): a thin wrapper per system normalizing
   each one's very different calling convention into one common shape: question in, `{answer, contexts, cost, latency}`
   out. This is what makes the next layer possible across six otherwise-incompatible systems.
7. **Judge/Eval** (offline, never per-query): the normalized output gets scored two ways: `RAGAS` (
   faithfulness/relevancy/precision/recall) and a cross-model judge (correctness vs. ground truth). Runs after the fact,
   on the fixed 10-question golden set.
8. **Client vs. pipeline test client**: a real user's question only ever touches `payments-rag`'s own `Embedder` →
   `Retriever` → `Generator` chain and never reaches eval. The pipeline test client (the golden-set harness) is the only
   thing that reaches `Adapter`. It's also the only thing that talks to the other five systems.
9. **Reaching the other five systems**: four of them (`openai-file-search`, `Haystack`, `LlamaIndex`, `LangChain`) have
   a real API, so the test client calls them directly, same as it calls into `payments-rag`. `NotebookLM` doesn't, and
it's reached through a `Browser` instead.

### Indexing and Retrieving Text

**1. `payments-rag`**

![payments-rag sequence: embed, pgvector search, Claude answer](https://serhiykucherenko.dev/images/comparing-rags/p2-seq-payments-rag-dark.png)

**2. `openai-file-search`**

![openai-file-search sequence: one-time upload to a persistent OpenAI vector store, then per-question retrieval inside the API](https://serhiykucherenko.dev/images/comparing-rags/p2-seq-openai-file-search-dark.png)

**3. Google `NotebookLM`**

![NotebookLM sequence: manual upload and manual copy-paste per question](https://serhiykucherenko.dev/images/comparing-rags/p2-seq-notebooklm-dark.png)

**4. `Haystack`**

![Haystack sequence: PyPDF load, split, embed into in-memory store, retrieve top 5, generate](https://serhiykucherenko.dev/images/comparing-rags/p2-seq-haystack-dark.png)

![Haystack bug: one malformed 12,000-word 'sentence' exceeds the embedding size limit and the chunk is silently dropped](https://serhiykucherenko.dev/images/comparing-rags/bug1-haystack-dark.png)

`Haystack`'s sentence-based chunker treated a malformed PDF-extracted block as one giant "sentence," pushing it past
OpenAI's embedding size limit. As a consequence, `Haystack` silently dropped that chunk instead of raising an error,
quietly losing a third of the index. Fixed by switching to `Haystack`'s own word-based chunking default.

**5. `LlamaIndex`**

![LlamaIndex sequence: pypdf load, sentence split, VectorStoreIndex, query engine](https://serhiykucherenko.dev/images/comparing-rags/p2-seq-llamaindex-dark.png)

![LlamaIndex bug: raw PDF metadata and binary noise leaked into extracted text, outranking real answers](https://serhiykucherenko.dev/images/comparing-rags/bug2-llamaindex-dark.png)

`LlamaIndex`'s default PDF reader leaked raw PDF metadata into extracted text and mangled some pages into unreadable
binary noise, both silently, without an error thrown. The noise sometimes outranked the real answer during retrieval.
Fixed by loading PDFs with `pypdf` directly, the same library `Haystack` already used cleanly.

**6. `LangChain` / `LangGraph`**

![LangChain/LangGraph sequence: pypdf load, recursive splitter, retrieve and generate nodes in a compiled graph](https://serhiykucherenko.dev/images/comparing-rags/p2-seq-langchain-dark.png)

## Results

![Results: judge score, RAGAS metrics, cost and latency for all six systems](https://serhiykucherenko.dev/images/comparing-rags/results-dark.png)

`LlamaIndex` is the best in both judge and faithfulness scores.

OpenAI file search cost $0.3526 for the same ten questions, which is the most expensive system in the comparison
by a wide margin: over 8x pricier than the next-priciest option (`Haystack`, $0.0413), and roughly 13x pricier than
`payments-rag`'s own hand-rolled approach ($0.0276). Likely because it stuffs more retrieved context into every request
than expected.

`NotebookLM` costs nothing in dollars, but it required 11 min of setup work of clicking
before the first question could even be asked. Meaning, it just moves the cost from money to time.

`NotebookLM` ties `LlamaIndex` for the highest answer-relevancy of all six, despite its other `RAGAS` metrics being
unusable. `LangChain`, the most popular framework by mindshare, placed second-worst, just above `Haystack`.
`Haystack`'s remaining weakness is a retrieval bias toward one source document on standard-SCT questions, not a
grounding failure.


## Conclusion

Some numbers worth pulling forward before the caveats:

- own built `payments-rag` placed 4th of 6 on judge-scored accuracy (84.8), behind `LlamaIndex` (96.5),
  `NotebookLM` (92.5), and `openai-file-search` (90.2).
- it was also the cheapest ($0.0276) and fastest (2.68s) system in the whole comparison. For instance, OpenAI markup
  caused it to cost about 8-13x, while `NotebookLM` took 11 minutes to set up.
- silent bugs may creep simply because of how the systems are wired up as it happened to `LlamaIndex` and `Haystack`. In
  this case it was about clean corpus and a sane chunk size.

The obvious flaw of this stage is a small set of questions (or maybe topics of questions) of a golden set.
Additionally, the compared options could be expanded dimension-wise by combining different models
(both embedding, llm, judge).

However, this intel brings some clarity on why we might end up having RAGs everywhere as well as concerns about both
correctness and security. Building a working and useful RAG is one effort. Building one at scale is a different thing.
And building one for privacy is yet another problem. The only way to fully avoid exposure is self-hosting all
the layers: the embedding, the generation models, and the retrieval layer. `payments-rag`'s own hand-rolled setup still
sends chunks to OpenAI and Anthropic at embed- and generation-time (see the exposure diagram in [Part 1](https://dev.to/skucherenko/comparing-rags-part-1-the-framework-e9p)).

## Out of Scope

- Bigger and more diverse golden sets, additional metrics beyond `RAGAS`/judge
- Testing every embedding/LLM/judge model combination for full experimental cleanliness: this run fixed one model set
  throughout
- `NotebookLM`'s paid Enterprise tier: investigated, doesn't change the conclusion (still no way to ask it a question
  programmatically)

Also, that was made on Python. I'd imagine someone doing it, say, on C++ or Rust (whatever is better) would make it
faster.

## Sources

- [GitHub repo](https://github.com/KucherenkoSerhiy/payments-rag)
- [live demo](https://rag.serhiykucherenko.dev)
- [RAGAS issue #2753](https://github.com/vibrantlabsai/ragas/issues/2753): the scoring-library bug
- [NotebookLM Enterprise API docs](https://docs.cloud.google.com/gemini/enterprise/notebooklm-enterprise/docs/api-notebooks):
  no query endpoint, any tier
- [NotebookLM privacy policy](https://support.google.com/notebooklm/answer/17004255): training-data use
- [Haystack telemetry docs](https://docs.haystack.deepset.ai/docs/telemetry)