← back to the tree

RAG Showcase

Seven RAG approaches compared side-by-side in one multi-model chat, all on Atlas.

7 strategies3 datasets380/380 answers scored

git clone --recursive https://github.com/thekaveh/rag-showcase && ./scripts/start-all.sh

Seven colored retrieval paths — vanilla to lazy-graph — converging through one knowledge graph into measured comparisons.
Seven retrieval paths converging on one measured comparison. Brand banner, 2026 — RAG Showcase repository.

There are a dozen ways to do retrieval-augmented generation and no honest way to compare them when each runs on its own embedding model and corpus. RAG Showcase makes the comparison fair by construction. Seven approaches run side by side in Open WebUI’s multi-model chat, every one reading the same corpus through the same embedding model, all hosted on Atlas (vendored as a Git submodule). Each approach is an OpenAI-compatible endpoint that Atlas validates, routes, and makes selectable in the chat with no admin setup. Open a multi-model chat, pick the approaches you want, and one prompt fans out into answers you can read next to each other, each carrying its retrieved context and metrics. The chat is only the interactive face, though: the same endpoints drive a headless evaluation harness, so the full comparison runs — and has run — from the command line with no web UI involved, and the results are written up on the documentation site. The project doubles as a deliberate test-drive of Atlas as reusable infrastructure.

Highlights

  • Seven retrieval strategies, side by side — vanilla, hybrid (Weaviate BM25 + dense with a reranker), contextual, graph RAG over Neo4j, an agentic ReAct loop, an n8n adaptive workflow, and an experimental lazy-graph-RAG
  • Fair by construction — one embedding model, one shared corpus, and a local-first role-to-model map, so no approach gets an unearned edge
  • A committed three-dataset evaluation — all 380 answers completed across a curated baseline, a graph-native set, and a MITRE ATT&CK cyber-threat corpus, scored by Ragas and a blinded two-model judge panel, with full leaderboards and a narrative comparison published on the docs site
  • Query-time flavors — the same approach re-run under reproducible parameter overrides (hybrid-rag-high-recall, hybrid-rag-fast), so tuning is itself a selectable, comparable thing
  • One-command bring-up — start-all.sh backfills the environment, validates the Compose manifest, starts the stack, assembles the corpus, and prints the live chat URL
  • Headless, resumable evaluation harness — the committed results come from a command-line runner that drives every approach through the same endpoints with no web UI in the loop, persisting every cell to append-safe JSONL and keeping deterministic operational metrics separate from the optional judge panel
  • Atlas-reuse assessment — a written account of what reused cleanly, where the friction was, and what got fed back upstream

Stack & Architecture

The seven approaches are FastAPI routes mounted inside the Atlas backend through a generic plugin seam, and a plugin.yml manifest is the source of truth Atlas validates for each route, health check, auth policy, and dependency. Atlas compiles them into LiteLLM aliases, so Open WebUI and the comparison harness reach every approach through one OpenAI-compatible endpoint. Retrieval leans on Weaviate for hybrid search and Neo4j with LightRAG for the graph work, with Atlas vendored under infra/.

Status

Actively developed, with a full documentation site covering the evaluation methodology, the dataset ladder, complete leaderboards, and a written narrative comparison of the approaches.