A small, reproducible harness for evaluating Retrieval-Augmented Generation (RAG) pipelines end-to-end: chunking → embedding → retrieval → generation → metrics.
No API keys, no model downloads, no external data. numpy is the only runtime
dependency; the whole evaluation runs in under a second and gives identical
numbers on every machine.
pip install -e ".[dev]"
pytest # 11 tests
rag-eval # run the evaluation grid
Example output:
| config | recall@k | MRR | nDCG@k | answer F1 |
|---|---|---|---|---|
| chunk1/ovl0/k5/dim512 | 1.000 | 1.000 | 1.000 | 0.530 |
| chunk3/ovl1/k5/dim512 | 1.000 | 1.000 | 1.000 | 0.530 |
| chunk6/ovl2/k5/dim512 | 1.000 | 1.000 | 1.000 | 0.530 |
| chunk3/ovl1/k5/dim128 | 1.000 | 0.992 | 0.994 | 0.530 |
| chunk3/ovl1/k1/dim512 | 1.000 | 1.000 | 1.000 | 0.503 |
I build and operate RAG systems in production (healthcare and pharmacy domains — patient-facing, on-premise, under Swiss data-protection constraints). Those systems are proprietary and covered by trade-secret agreements, so I cannot publish them. This repository demonstrates the method behind them on fully synthetic data: how I structure a pipeline so that every design decision — chunk size, overlap, embedding dimension, retrieval depth — is a measured choice rather than a guessed one.
The design positions I consider non-negotiable in real deployments, all reflected here:
- Evaluation is part of the pipeline, not an afterthought. If you cannot score a configuration, you cannot defend it — to an engineer, an auditor, or a citizen.
- Baselines before models. The built-in embedder is hashed TF-IDF and the default generator is purely extractive. They are deliberately simple: any LLM or neural embedder has to beat a measured floor to justify its cost, latency and governance burden.
- Determinism and reproducibility. Seeded corpus generation, stateless feature hashing, brute-force exact retrieval. When a number changes, it is because the system changed — not the environment.
- Grounding over fluency. The (optional) LLM generator receives only the retrieved chunks and is instructed to abstain when the context does not contain the answer. Answer F1 is scored token-level, which is strict enough to catch a wrong number — in administrative and clinical settings, the failure mode that actually matters.
corpus.py synthetic "public administration" articles + gold QA pairs
chunking.py sentence windows with overlap (an evaluated parameter, not a constant)
embed.py Embedder protocol + hashed-TF-IDF baseline (deterministic, stateless)
index.py exact cosine retrieval (brute force, so quality is attributable)
rag.py RagPipeline; ExtractiveGenerator baseline + OllamaGenerator sketch
metrics.py recall@k, MRR, nDCG@k (retrieval) + token F1 (answer quality)
harness.py the composition: run a config grid, produce a comparable report
cli.py `rag-eval` entry point
Swapping in a real embedder or generator means implementing a two-method protocol — the harness, corpus and metrics stay identical, which is the point: the yardstick must not move when the system under test changes.
The generated articles cycle through a small set of topics, so the corpus contains many near-duplicates ("Merkblatt 3" vs "Merkblatt 7" on the same subject). Retrieving the right document, not just the right topic, is what separates a usable knowledge-management assistant from a plausible-sounding one — and it is exactly what recall@k and MRR measure here.
Fact sentences are written self-contained (topic and document identifier repeated per sentence), mirroring the chunk-hygiene practice that makes real RAG chunks retrievable out of context.
MIT — see LICENSE.