CacheBench is derived from and includes content from the following upstream datasets and papers. Each is listed with its license and the portion of CacheBench it contributes to.
| Upstream | License | CacheBench rows | How used |
|---|---|---|---|
| PAWS-X (Yang et al., EMNLP 2019, arXiv:1908.11828) | CC BY-SA 4.0 | ~60 multilingual rows (ADVERSARIAL + PARA_SAFE) | PAWS-X pair IDs retained in source field; CC BY-SA propagates → CacheBench-v1 is CC BY-SA 4.0 |
| TriviaQA (Joshi et al., ACL 2017) | Apache 2.0 | ~30 qa_factual rows (EQUIV / UNRELATED) | Question text + canonical answer |
| Quora Question Pairs (QQP) | Quora Terms of Service (restrictive) | ~80 qa_factual + qa_open rows (PARA_SAFE + UNRELATED) | Question pairs from labeled dataset |
| BANKING77 (Casanueva et al., ConvAI 2020) | CC BY 4.0 | ~70 qa_factual rows (intent-grouped EQUIV / RELATED_UNSAFE / ADVERSARIAL high-stakes) | Intent-grouped query pairs |
| BFCL v3 (Berkeley Function Calling Leaderboard, Gorilla team) | Apache 2.0 | ~30 tool rows (identical_call seed) | Function-call schemas from possible_answer/ |
| xLAM-irrelevance-7p5k (Salesforce, MadeAgents) | CC BY-NC 4.0 (non-commercial) | ~30 tool rows (irrelevance hard negatives) | Hard negative tool-call pairs |
Note on Quora QQP: Quora's ToS restricts commercial use of the dataset. If CacheBench is used commercially, the ~80 QQP-derived rows should be removed and replaced with synthetic paraphrases. The provenance is tagged in the
sourcefield of every affected row for easy filtering.
Note on xLAM-irrelevance: CC BY-NC 4.0 — non-commercial only. Same caveat as QQP: drop these 30 rows for commercial use.
| Paper | Citation | What we borrowed |
|---|---|---|
| PAWS (Zhang et al., NAACL 2019) | arXiv:1904.01130 | Adversarial paraphrase construction recipe |
| MeTMaP (Wang et al., 2024) | arXiv:2402.14480 | Metamorphic word-level perturbation operators |
| vCache (Schroeder et al., ICLR 2026) | arXiv:2502.03771 | Wilson binomial CI metric, LMArena eval setup, 5-class taxonomy inspiration |
| W5H2 (Basu, NeurIPS 2025) | arXiv:2602.18922 | Structured intent canonicalization for tool rows |
| Krites / Async Verified (Singh et al., 2026) | arXiv:2602.13165 | LLM-judge minimization protocol |
| MeanCache (Gill et al., IPDPS 2025) | arXiv:2403.02694 | Multi-turn context-chain pair construction |
| SCALM (Li et al., 2024) | arXiv:2406.00025 | LMSYS-derived pair clustering |
| LangCache-Embed (Gill et al., 2025) | arXiv:2504.02268 | LM-generated positive/negative pair listings |
| SAFE-CACHE (Afiffy et al., Nature Sci Rep 2026) | – | Adversarial robustness eval |
| ContextCache (2025) | arXiv:2506.22791 | Multi-turn cache evaluation methodology |
| GenCache (Microsoft, NeurIPS 2025) | arXiv:2511.17565 | Pattern-equivalent prompt benchmark |
| SemCacheOLAP (2026) | arXiv:2602.19811 | Intent-signature canonicalization for SQL/NL |
| MASSIVE (FitzGerald et al., ACL 2023) | – | Intent-classification dataset structure inspiration |
| AmbigQA (Min et al., EMNLP 2020) | arXiv:2004.10645 | Ambiguity-as-feature labeling |
| Dynabench (Kiela et al., NAACL 2021) | arXiv:2104.14337 | Adversarial-evolving slice protocol |
See requirements.txt for full pinned dependencies. Key libraries:
sympy(BSD)pyarrow(Apache 2.0)pandas(BSD)datasets,huggingface_hub(Apache 2.0)jsonschema(MIT)sqlglot(MIT)
If you use CacheBench, please cite:
@misc{cachebench-v1,
title={CacheBench v1: A Pair-Equivalence Benchmark for LLM Prompt Caches},
year={2026},
note={2000-row benchmark across 10 domains with 5-class equivalence labels.}
}And the upstream sources above.
If you are a copyright holder and want content removed, please open an issue. We will redact within 7 days and ship a v1.x.y patch release.