FAQ RAG Benchmark

Does model size matter when you have RAG?

For a customer service chatbot handling FAQ, using a frontier model felt like using a sledgehammer for a nail. The hypothesis: a smaller model with RAG can match a larger model's accuracy at significantly lower cost.

Case study: https://rival.my.id/case-study/why-you-dont-need-a-bigger-model
Live demo: https://tokofiktif.rival.my.id

Setup

Models (via Groq free tier):

• Small: llama-3.1-8b-instant
• Large: llama-3.3-70b-versatile

Vector DB: Qdrant (local) + BAAI/bge-m3 embeddings
Dataset: 100 FAQ items from a fictional Indonesian e-commerce company ("TokoFiktif")
Evaluator: qwen/qwen3.6-27b — different model family to avoid bias

4 conditions tested:

Config	Model	RAG
8b_no_rag	llama-3.1-8b-instant	❌
8b_rag	llama-3.1-8b-instant	✅
70b_no_rag	llama-3.3-70b-versatile	❌
70b_rag	llama-3.3-70b-versatile	✅

Results

Config	Accuracy	Avg Latency	Total Cost (USD)
70b_rag	100%	564ms	$0.0203
8b_rag	97%	333ms	$0.0023
70b_no_rag	35%	421ms	$0.0119
8b_no_rag	17%	261ms	$0.0014

Analysis

RAG is the dominant factor — not model size. Without RAG, both models fail badly (17–35%). With RAG, the accuracy gap between 8B and 70B shrinks to 3%.

8b_rag vs 70b_rag:

• 88% cheaper ($0.0023 vs $0.0203)
• 41% lower latency (333ms vs 564ms)
• Only 3% accuracy difference

For a constrained FAQ use case, 8b_rag is the better engineering choice.

Stack

• Inference: Groq API
• Embeddings: sentence-transformers (BAAI/bge-m3)
• Vector DB: Qdrant (local)
• Evaluation: qwen/qwen3.6-27b via Groq

Running

cp .env.example .env
# Fill in GROQ_API_KEY
uv sync
uv run benchmark.py
uv run graph.py