prompt-cache

A 200-line Python disk cache for LLM API calls. Wrap any client. Stop paying for the same call twice during dev iteration.

from prompt_cache import PromptCache
import openai

cache = PromptCache(".cache/prompts")
client = openai.OpenAI()

@cache.wrap
def chat(model, messages, **kwargs):
    return client.chat.completions.create(model=model, messages=messages, **kwargs)

# First call → hits the API, costs money
chat("gpt-4o-mini", [{"role": "user", "content": "Explain monads"}])

# Second identical call → returns instantly from disk, $0
chat("gpt-4o-mini", [{"role": "user", "content": "Explain monads"}])

print(cache.stats())  # {'hits': 1, 'misses': 1, 'files': 1}

Install

pip install prompt-cache

Or copy prompt_cache.py into your project. Zero dependencies beyond the Python 3.10+ stdlib.

Why

You're prototyping. You write a script that hits the API. Run, iterate, run, iterate. You just paid $0.40 to ask GPT the same thing 20 times because you were debugging your downstream parsing code.

prompt-cache makes that loop cost $0.02 instead.

Also useful for:

• Eval reproducibility — same prompt + temperature 0 + cached response = byte-identical eval runs across CI
• Demo determinism — your live demo doesn't depend on API uptime or rate limits
• Long-running pipelines — re-run from any checkpoint without re-paying for completed steps

Works with anything

The decorator caches any function. Provider-agnostic. Here are common shapes:

OpenAI / OpenAI-compatible (Groq, Together, OpenRouter, etc.)

@cache.wrap
def chat(model, messages, **kwargs):
    return client.chat.completions.create(model=model, messages=messages, **kwargs)

Anthropic

@cache.wrap
def chat(model, messages, max_tokens=1024, **kwargs):
    return client.messages.create(model=model, messages=messages, max_tokens=max_tokens, **kwargs)

Raw HTTP

@cache.wrap
def chat(model, messages, **kwargs):
    r = requests.post(API_URL, json={"model": model, "messages": messages, **kwargs}, headers=H)
    return r.json()

Manual API

If you don't want the decorator:

key = cache.key(model="gpt-4o-mini", messages=[{"role": "user", "content": "hi"}])

if cache.has(key):
    response = cache.get(key)
else:
    response = expensive_api_call(...)
    cache.set(key, response)

Ignoring noisy kwargs

Some kwargs change every call but shouldn't bust the cache (e.g. an idempotency_key you generate per call). Use key_from= to whitelist what's part of the cache key:

@cache.wrap(key_from=["model", "messages", "temperature"])
def chat(model, messages, temperature=0, idempotency_key=None, **kwargs):
    ...

Invalidation

Three options, from coarsest to finest:

cache.bust()                    # nuke every cache file
cache = PromptCache(".cache", version="v2")  # change version → old keys never hit
cache.delete(specific_key)      # remove one entry

Storage

Files live at <root>/<first-2-hex>/<rest>.cache, sharded to avoid one giant flat directory. JSON-serializable values are stored as JSON (diffable, portable); fallback to pickle for anything weirder.

Each file is a single response — easy to inspect with cat, easy to git-track if you want shared eval fixtures.

TTL

cache = PromptCache(".cache", max_age_seconds=3600)  # entries expire after 1h

Set per PromptCache instance; not per entry (yet).

Why a single file

Because LLM tooling shouldn't require you to depend on a framework. Read prompt_cache.py in five minutes. Fork it if you don't like a choice.

License

MIT — see LICENSE.

Sponsor

If prompt-cache paid for itself in API savings, sponsor on GitHub. 🙏

Built by Tubbster-Claw — automation tooling shop.