RAG Cache Assistant

A support assistant for an online store that answers customer questions from a knowledge base (RAG) and puts a semantic cache in front of it — so repeated questions skip both retrieval and the LLM entirely.

Two problems, two layers:

1. Grounded answers (RAG). Answers come from the store's actual policy docs, not from whatever the model happens to know. Wrong-but-confident answers are the usual failure mode of a bare chatbot; retrieval fixes that.

2. Cost & latency (semantic cache). A store's support gets the same questions all day in different wordings. The cache matches on meaning, so "where's my order", "how do I track my package", and "my order hasn't arrived" are answered once and served from cache after that — no model call, no token cost, milliseconds not seconds.

How it works

question ──> embed ──> 1. check the cache (nearest past question)
                              │
                   similar & fresh? ── yes ──> return cached answer   (cost $0, ~ms)
                              │
                              no
                              ▼
                   2. retrieve top-k chunks from the knowledge base
                              ▼
                   3. ask the LLM, grounded in those chunks
                              ▼
                   4. store the answer in the cache, return it

A cache hit short-circuits at step 1. A miss pays for retrieval + one LLM call, then every similar question afterwards is a hit.

The caching strategy (what, how many, how fresh)

• What gets cached: the grounded answer to every cache miss. The cache key is the question's embedding, so paraphrases collapse to one entry.

• How many: capped at MAX_CACHE_ENTRIES (default 1000). When full, the least-recently-used entry is evicted — hot questions stay, cold ones fall out.

• How fresh: entries expire after CACHE_TTL_SECONDS (default 24h). A stale hit is dropped and treated as a miss, so policy changes propagate.

• What to be careful with: the similarity threshold (default 0.92) is the safety knob — too low and a different question gets the wrong cached answer. For user-specific or time-sensitive questions ("where is my order #4471") you'd skip caching entirely; here the demo questions are all generic policy ones, which is exactly what's safe to cache.

The knowledge base

A small set of the store's support policies (app/kb_docs.py) — shipping times, international delivery, returns window, refund timing, payment methods, address changes, warranty, and so on. On startup each chunk is embedded into a Qdrant collection; on a miss, the most relevant ones are retrieved and handed to the model.

How to Run it

pip install -r requirements.txt
cp .env.example .env        # paste your OPENAI_API_KEY
uvicorn app.main:app --reload

Ask a question:

curl -s localhost:8000/chat -H 'content-type: application/json' \
  -d '{"prompt": "How do I return an item?"}'

First time it's a miss (retrieved + answered). Ask it a different way and it's a hit:

curl -s localhost:8000/chat -H 'content-type: application/json' \
  -d '{"prompt": "can I send a product back?"}'

Prove the savings

benchmark.py fires groups of reworded support questions. Only the first in each group reaches the LLM.

python benchmark.py

query                                                result      sim       ms
------------------------------------------------------------------------------
Where is my order?                                   miss          -   1120.4
how do I track my package                            HIT       0.967     11.9
my order hasn't arrived yet, where is it?            HIT       0.948     11.4
How do I return an item?                             miss          -    980.1
what's your return policy                            HIT       0.971     12.0
...

summary
------------------------------------------------------------------------------
  requests              12
  cache hits            7  (58% hit rate)
  cost without cache    $0.001480
  cost with cache       $0.000620
  saved                 $0.000860  (58%)

When this helps (and when it doesn't)

Good fit: support bots, FAQ/docs Q&A — high-repeat questions over a stable knowledge base.

Skip caching when: answers are user-specific or change minute to minute. The TTL and threshold limit the damage, but a cache hit is only as good as "this question was really the same question."

Notes

• Qdrant runs in-process for dev; one env var switches to a server.
• KB and cache are separate Qdrant collections.
• Stats (/stats) are in-process counters — fine for one worker.
• Tests cover cost math and the cache's TTL + eviction; the retrieval/LLM paths need a key.

Config

Variable	Default	What it does
OPENAI_API_KEY	—	your key
CHAT_MODEL	gpt-4o-mini	completion model
EMBEDDING_MODEL	text-embedding-3-small	embedding model
TOP_K	3	KB chunks retrieved per question
SIMILARITY_THRESHOLD	0.92	how close counts as the same question
CACHE_TTL_SECONDS	86400	how long a cached answer stays fresh
MAX_CACHE_ENTRIES	1000	cache size cap (LRU eviction)
QDRANT_PATH	:memory:	in-process store (or a folder path)