AUDIT_coverage.md

CacheBench v1 — Coverage / Balance / Calibration Audit

Audit date: 2026-05-24 Artifact audited: /home/bud/ditto/budCache/research/cachebench/cachebench.jsonl (2,000 rows) Methodology references: design/pair-eval-methodology.md, design/traffic-distributions.md, SCHEMA.md

---

0. Executive summary

Axis	Verdict	Grade
Domain × Label matrix	Exact match to spec	A
Difficulty (overall %)	32%/43%/25% vs spec 30%/50%/20% — close, but distribution drifted	B
Difficulty calibration vs embedders	Cohen's κ = 0.03 between stated and empirical. Massive miscalibration.	D
Construction-method mix	LM-generated +207% (767 vs 250); real-traffic at 52%; intent-bottom-up at 41%	D
Subcategory diversity	14 (domain × label) cells have < 3 subcategories; multilingual & creative are worst	C
Source diversity	4 of 10 domains have a single source ("curator") — conv, creative, personalized, math (81%)	D
ADVERSARIAL strength	15% (60/410) are weak (mean_sim < 0.5); 5% trivial (< 0.3)	C
UNRELATED triviality	99.4% are clean (mean_sim < 0.5). Healthy.	A
Real-world traffic alignment	Code & qa_open under-weighted by ~50%; conversational over-weighted 6.5×; multilingual is half of recommended	C
Statistical power	22 of 50 cells have n ≤ 30 (±17pp Wilson CI); cannot resolve <17pp deltas in those cells	C
Class balance (HIT/MISS)	41.5% / 58.5% — over-samples HIT vs deployment reality (10–20% chat hit rate)	C
Non-English representation	7.5% (multilingual only); spec recommends ~15%	C

Headline. The matrix arithmetic is perfect, but the artifact's content drifts from the methodology in three structural ways:

1. Construction method has collapsed: 38% of rows are LM-generated (curator-written) where the spec called for 12.5%. Real-traffic mining (Rank-1 in the methodology, "highest external validity") only fills 15.5% of the corpus (target 30%). Four domains (conversational, creative, personalized, math 81%) are essentially single-author content from "curator".
2. Difficulty labels are nearly uncorrelated with what 5 baseline embedders actually find easy/hard (κ=0.03). 286 rows stated hard are trivially predicted by all 5 embedders; 162 stated easy rows fail all 5 embedders. The label distribution looks plausible but is not empirically grounded.
3. The benchmark is much shorter than production reality. Tool rows median 8 estimated tokens; the spec calls for agent-style rows of 2,700-token median. There is no real agent-prefix regime in CacheBench.

Net: the corpus is fit for testing whether cheap embedders can detect adversarial perturbations on toy-length English queries, but not for "production fidelity" claims (per the design doc's framing). The four most-actionable fixes are listed in §8 (re-balancing).

Calibration note. sentence-transformers and torch failed to install in the audit window (pip metadata-resolve stalled). The empirical embedder baseline used here is five sklearn TF-IDF vectorizers (word-uni, word-bi, char3, char4, char3-5), each at its accuracy-maximizing threshold. These are weaker than gte-large-en-v1.5 / e5-large-v2, so the calibration numbers below are an upper bound on κ — true embedder-derived difficulty agreement will be even worse on pairs where TF-IDF and a real bi-encoder diverge (mostly the lexical-low / semantic-high regime, i.e. EQUIV/PARA_SAFE paraphrases). The directional conclusions hold.

---

1. Distribution audit (every axis)

1.1 Domain × Label matrix

Verdict: EXACT MATCH to spec. Every one of the 50 cells matches the methodology brief §6.1 to the unit row.

Domain	EQUIV	PARA_SAFE	RELATED_UNSAFE	ADVERSARIAL	UNRELATED	Total
qa_factual	50	50	50	40	40	230
qa_open	60	60	50	40	50	260
math	30	30	30	50	20	160
code	30	30	30	50	20	160
conversational	60	60	50	40	50	260
tool	60	60	60	50	50	280
creative	40	40	40	30	30	180
personalized	30	30	30	30	20	140
multi_turn	30	30	50	40	30	180
multilingual	30	20	30	40	30	150
Total	420	410	420	410	340	2,000

1.2 Construction method (actual vs spec)

Verdict: DRIFTED HARD. Grade D.

Method	Actual	Spec	Delta	% change
real_traffic	310	600	-290	-48%
intent_bottom_up	205	500	-295	-59%
adversarial_perturbation	533	500	+33	+7%
lm_generated	767	250	+517	+207%
existing_benchmark	185	150	+35	+23%

The methodology brief (§2 Rank-1) was explicit that real-traffic-mined pairs offer "the highest external validity" and contribute 600 of the 2,000 rows. The actual corpus is 38% LM-generated, more than half of the rows. This is the Rank-4 method the brief warned against: "LM-generated negatives are often too easy" and the curator-only register narrows the linguistic distribution. The downstream calibration disaster (§3) is directly driven by this: when one curator writes most pairs, the lexical structure becomes too uniform to span what real users actually type.

1.3 Difficulty distribution (overall)

Level	Actual	%	Spec %
easy	639	31.9%	30%
medium	869	43.5%	50%
hard	492	24.6%	20%

Numerically close, but the per-row difficulty annotation is not what an embedder actually finds easy/hard — see §3.

1.4 Per-cell subcategory diversity

14 of 50 cells (28%) have < 3 distinct subcategories. Cells flagged below are at risk of overfitting to a single template.

Cell	# subcats	# rows
qa_factual/PARA_SAFE	2	50
qa_factual/UNRELATED	1	40
qa_open/UNRELATED	1	50
math/UNRELATED	1	20
code/UNRELATED	1	20
tool/UNRELATED	2	50
creative/UNRELATED	1	30
personalized/PARA_SAFE	1	30
personalized/UNRELATED	1	20
multilingual/EQUIV	1	30
multilingual/PARA_SAFE	1	20
multilingual/RELATED_UNSAFE	1	30
multilingual/ADVERSARIAL	1	40
multilingual/UNRELATED	1	30

Multilingual is the worst offender: a single subcategory per (domain × label) cell for all 150 rows. Five unique subcategories total: paws_x_adversarial (40), translation_paraphrase (20), translation_factual (30), locale_specific_answer (30), unrelated_cross_language (30). A bi-encoder fine-tuned on this slice would simply learn the five templates.

The UNRELATED row of every domain except conversational and multi_turn is monolithic — usually labelled random_*_pair. These rows are arguably the easiest in the benchmark (UNRELATED median jaccard = 0, mean_sim < 0.1) so single-source is acceptable for them, but it inflates the "easy" share.

1.5 Source diversity per domain

Domain	# unique sources	Top source	Top-source share
conversational	1	curator	100%
creative	1	curator	100%
personalized	1	curator	100%
math	3	curator	81%
code	4	curator	56%
tool	9	curator	28%
multi_turn	10	curator	44%
qa_factual	30	BANKING77	24%
qa_open	7	WildChat/Arena/ShareGPT seeds	23%
multilingual	61	curator	60%

4 of 10 domains depend on a single human-curator author (conv, creative, personalized at 100%; math at 81%). The corpus's stylistic register in those 740 rows is whatever phrasing one author considered "WildChat-like" — not real WildChat. The methodology brief §2 Rank-1 specifically called out this risk ("the LM doesn't generate adversarial perturbations; it generates 'talking about a sibling concept'"); the same applies here when the human curator inherits a small vocabulary of perturbation operators.

---

2. Class balance & headline numbers

Binary class: 830 HIT / 1170 MISS (41.5% / 59%).

Compared to deployment cache-hit-rate guidance from traffic-distributions.md §6:

• Open-ended chat: 10-20% hit rate → CacheBench is at least 2-4× too HIT-heavy for chat workloads
• Classification / short-task: 40-60% → reasonable fit only for this band
• Agent-prefix: 70-90% → CacheBench is too HIT-light for agent workloads

CacheBench is therefore positioned as an annotation-balanced benchmark, not a deployment-weighted one. That is defensible (Wilson CIs across labels are roughly comparable so per-label F1 is well-supported) and matches the QQP / PAWS tradition. But a system that reports CacheBench's headline F1 / FHR cannot translate that number to "production false-hit rate" without re-weighting.

Recommendation: rename the headline Precision/Recall/FHR to "stratified" and publish a deployment-weighted alternative that re-weights cells to the §2 traffic mix.

---

3. Difficulty calibration verdict (CRITICAL)

Spec semantics: Easy = ≥4/5 baseline embedders correct; Medium = 2-3/5; Hard = 0-1/5.

Empirical baselines used: 5 sklearn TF-IDF cosine models (word_uni, word_bi, char3, char4, char3-5), each at the threshold that maximizes its overall accuracy on the 2,000 rows. Best-thresholds: word_uni=0.90, word_bi=0.90, char3=0.95, char4=0.95, char35=0.95. Best single-model accuracy = 63.5% (word_bi at t=0.90).

Confusion matrix — stated vs empirical difficulty

stated ↓	emp_easy	emp_medium	emp_hard	total
easy	450 (70%)	19 (3%)	170 (27%)	639
medium	446 (51%)	47 (5%)	376 (43%)	869
hard	314 (64%)	32 (7%)	146 (30%)	492
TOTAL	1210	98	692	2000

• Agreement: 643/2000 = 32.1%
• Cohen's κ = 0.031 (essentially zero — at random-chance level)
• Mean k_correct (out of 5) by stated difficulty:
  • stated easy: mean_k=3.59 (good)
  • stated medium: mean_k=2.71 (low)
  • stated hard: mean_k=3.35 (HIGHER than 'medium' — backwards!)

What's wrong

The difficulty field appears to be assigned categorically by row author (e.g. "all ADVERSARIAL rows are hard"), not empirically calibrated against 5 embedders as the spec requires. Concretely:

• 286 stated-hard rows have all 5 TF-IDF embedders correct. These are dominated by paws_x_adversarial (26), polar_flip (16), word_problem_operation_swap (15), locale_specific_answer (15), high_stakes_paraphrase_banking (13). For polar-flip ("yes" vs "no", "yeah" vs "nope") all embedders trivially predict MISS because the lexical similarity is 0. These should be labeled easy.
• 162 stated-easy rows fail all 5 embedders. Dominated by multilingual/EQUIV/translation_factual (30) — cross-language paraphrases where TF-IDF sees zero token overlap but the labels say EQUIV. Also tool/EQUIV/arg_order_or_naming (15) — same intent, different parameter names. These are the hard cases for lexical embedders, and should be labeled hard or medium.

Per-domain calibration agreement

Domain	Agreement
qa_factual	18%
qa_open	30%
math	22%
code	19%
conversational	28%
tool	50%
creative	30%
personalized	52%
multi_turn	39%
multilingual	31%

Tool and personalized are the only domains > 50%. Math/code/qa_factual are < 25% — the curator's difficulty intuition is opposite to embedder behavior in those domains.

Bottom line: the difficulty field is currently a priors-based annotation and should not be reported as a calibrated stratifier in any leaderboard write-up. Either (a) re-derive it empirically against the 5 baseline embedders that the spec requires, or (b) re-label it as "intent-priors" / "design intent" and stop calling it embedder-calibrated.

---

4. ADVERSARIAL slice strength

Definition (audit): "Strong adversarial" = high lexical/embedding similarity AND MISS label. Empirically: mean_sim ≥ 0.5 is the minimum for the embedder to plausibly false-fire.

Histogram of mean TF-IDF cosine across 5 baselines

mean_sim bucket	n / 410	%
[0.0, 0.3)	21	5%
[0.3, 0.5)	39	10%
[0.5, 0.7)	154	38%
[0.7, 0.85)	116	28%
[0.85, 1.0]	80	20%

60/410 (15%) ADVERSARIAL rows are weak (mean_sim < 0.5). The 21 rows below 0.3 are not adversarial at all by any sensible definition — they are simply UNRELATED rows mislabeled.

Weak ADVERSARIAL by domain

Domain	weak / total	% weak
qa_factual	1/40	2%
qa_open	0/40	0%
math	1/50	2%
code	3/50	6%
conversational	7/40	18%
tool	0/50	0%
creative	0/30	0%
personalized	0/30	0%
multi_turn	4/40	10%
multilingual	5/40	12%

Specific downgrade list

ADVERSARIAL → UNRELATED (mean_sim < 0.3, no real lexical/embedding overlap):

• cb-conv-0171, 0172, 0203, 0204 (polar_flip) — A="yes" vs B="no". These are not adversarial; they are textbook UNRELATED.
• cb-conv-0195, 0196, 0197 (open_close_flip) — comparable structure.
• cb-mt-0112 (multi_turn yes_no_destruct) — A="yes" / B="no" in a 2-turn context.
• cb-ml-0028, 0032, 0033, 0040 (multilingual paws_x_adversarial in CJK) — token-level Jaccard = 0 because CJK doesn't word-tokenize.

ADVERSARIAL → RELATED_UNSAFE (mean_sim 0.3–0.5):

• 24 rows including cb-conv-0173, 0177, 0178, 0179 (polar_flip), cb-conv-0189, 0190 (subject_swap), cb-code-0099, 0100, 0104, 0105 (user_specific_code_review). These should be RELATED_UNSAFE — they have moderate overlap but are not the high-similarity false-fire cases ADVERSARIAL is meant to capture.

The methodology brief §3 "Audit step (required for every ADVERSARIAL pair)" mandates cosine similarity > 0.75 under a baseline embedder. 216 of 410 (53%) ADV rows fail that bar against the TF-IDF baselines. Even after accounting for TF-IDF being weaker than bi-encoders, the bottom 60 rows below 0.5 cannot plausibly cross 0.75 under any baseline.

Tool ADVERSARIAL — sub-category check

Subcategory	n	mean_sim avg	Comment
direction_swap	15	0.70	OK
unit_swap	10	0.68	OK
cross_tool_paraphrase	10	1.00	All 10 rows have query_a == query_b exactly; differentiation lives in tools_a/tools_b. This is fine for prefix-cache eval but the embedder cannot tell them apart by query text.
tool_substitution	5	0.68	OK
irrelevance	5	0.78	OK
multi_step_plan_equiv	5	0.85	Strong

Mix is healthy. The cross_tool_paraphrase cluster of 10 rows is intentional (testing tools_a vs tools_b differentiation) but a query-only embedder will be unable to predict them — flag this as a separate sub-axis in scoring.

---

5. UNRELATED slice triviality

Verdict: clean. Per-label distribution:

Label	n	min	p25	med	p75	max
EQUIV	420	0.00	0.67	1.00	1.00	1.00
PARA_SAFE	410	0.00	0.18	0.33	0.50	1.00
RELATED_UNSAFE	420	0.00	0.50	0.71	1.00	1.00
ADVERSARIAL	410	0.00	0.55	0.70	0.83	1.00
UNRELATED	340	0.00	0.00	0.00	0.09	0.50

• 99.4% (338/340) of UNRELATED rows have query-only Jaccard ≤ 0.5.
• 0 UNRELATED rows have Jaccard > 0.5 (the table caps at 0.50).
• 2 UNRELATED rows have char-3-gram cosine > 0.5: cb-qa-0216 ("first atomic-powered ship" vs "first movie western") cos=0.52; cb-qa-0490 ("learning R resources" vs "SparkNotes a good learning resource") cos=0.53. These are arguably RELATED_UNSAFE but the impact is small (2/340 = 0.6%).

Action: optionally re-label cb-qa-0216 and cb-qa-0490 to RELATED_UNSAFE. Otherwise UNRELATED is fine.

---

6. Real-world traffic alignment

design/traffic-distributions.md §2-§5 specifies a chat-band mix plus a 350-400-row agent-style carve-out, plus ~15% non-English. Comparing:

Domain	CacheBench %	Traffic-doc target %	Delta	Justified?
qa_factual	11.5%	14%	-2.5	Within tolerance
qa_open (incl. misc + analysis + personal_help)	13.0%	25 + 6 + 5 = 36%	-23	NO — under-weighted by 2/3
math	8.0%	5%	+3	OK (justified by methodology §6.1 to surface operator-swap failures)
code	8.0%	22%	-14	NO — coding is the dominant chat category and this benchmark allocates 1/3 of the real share
conversational	13.0%	2%	+11	NO — 6.5× over-sampled. Many rows are degenerate single-tokens ("yes", "ok")
tool	14.0%	≈18% (agent carve-out)	-4	Close, but see length critique
creative + role_play	9.0%	14%	-5	Under-sampled but defensible (MISS-creative covered by policy)
personalized	7.0%	5% (overlap)	+2	OK
multi_turn	9.0%	15%	-6	Under-sampled
multilingual	7.5%	15%	-7.5	NO — half the recommended

Material misalignments (and whether justified)

1. Code at 8% vs ~22% real — unjustified. The methodology brief §6.1 says math+code get heavy ADVERSARIAL weights (and they do), but the EQUIV/PARA_SAFE slices are starved (30+30 each = 60 rows). A coding cache author can saturate the 30 EQUIV rows with one careful run and the benchmark wouldn't catch their hallucinations on other coding flavors.
2. Conversational at 13% vs 2% — over-sampled. Many of the 260 rows are degenerate ("yes" / "no" / "ok") chosen for testing context-anaphor handling but the labels (50 are RELATED_UNSAFE with query_a == query_b and no context differentiator) are technically undecidable from query text alone. These rows test policy / anaphor handling, not the cache decision boundary the rest of the corpus measures.
3. Multilingual at 7.5% vs 15% — half. With only 5 subcategories total in the slice, this is the most decisively under-built domain. Recommendation: double the size to 300 rows and expand to ≥3 subcategories per (label × language).
4. Tool domain length — CacheBench tool rows have median ~30 chars / 8 estimated tokens. The traffic doc §4 says agent prompts have median 2,674 tokens / p90 6,564. The tool slice is not a "stress test of long-prompt agent caching"; it's a test of short tool-call queries. The benchmark therefore cannot measure agent-prefix-cache hit rate — the largest production use case (60-90% hit rate, Anthropic et al.).

Argument: is over-weighting math/code/tool/adversarial justified?

The methodology brief §6.1 explicitly justifies oversampling these "to expose failure modes that least-studied lexical similarity methods don't catch." That argument holds if the calibration is faithful — but §3 above shows the difficulty labels are not calibrated, so the oversampling currently buys less than the design assumed. Tightening §3's calibration would let the +50% ADV weighting in math/code earn back its budget.

Net real-world verdict: the corpus is biased toward a "stress-test against shallow embedders" reading. That's the right bias for a CacheBench v1; the gap to "production representative" is significant but documentable.

---

7. Statistical-power table (Wilson 95% CI half-widths at p=0.5)

Domain	EQUIV	PARA_SAFE	RELATED_UNSAFE	ADVERSARIAL	UNRELATED
qa_factual	50 (±13.4)	50 (±13.4)	50 (±13.4)	40 (±14.8)	40 (±14.8)
qa_open	60 (±12.3)	60 (±12.3)	50 (±13.4)	40 (±14.8)	50 (±13.4)
math	30 (±16.8)	30 (±16.8)	30 (±16.8)	50 (±13.4)	20 (±20.1)
code	30 (±16.8)	30 (±16.8)	30 (±16.8)	50 (±13.4)	20 (±20.1)
conversational	60 (±12.3)	60 (±12.3)	50 (±13.4)	40 (±14.8)	50 (±13.4)
tool	60 (±12.3)	60 (±12.3)	60 (±12.3)	50 (±13.4)	50 (±13.4)
creative	40 (±14.8)	40 (±14.8)	40 (±14.8)	30 (±16.8)	30 (±16.8)
personalized	30 (±16.8)	30 (±16.8)	30 (±16.8)	30 (±16.8)	20 (±20.1)
multi_turn	30 (±16.8)	30 (±16.8)	50 (±13.4)	40 (±14.8)	30 (±16.8)
multilingual	30 (±16.8)	20 (±20.1)	30 (±16.8)	40 (±14.8)	30 (±16.8)

Cells with n ≤ 20 (±20.1pp at p=0.5; n=4): math/UNRELATED, code/UNRELATED, personalized/UNRELATED, multilingual/PARA_SAFE.

Cells with n = 30 (±16.8pp; n=18): essentially all corners of math, code, creative, personalized, multi_turn, multilingual. A 5pp F1 difference in any of these cells is statistical noise.

Macro N=2000: ±2.19pp at p=0.5 (vCache/Wallis target met). Per-domain N=140–280: ±5.8–8.3pp at p=0.5. These power CacheBench's top-line and per-domain headline. Per-cell CI is too wide for cell-level claims — at n=30, an F1 improvement from 60% to 70% (±17pp CI) is consistent with no change. The methodology brief §6 implicitly conceded this: "Multi-turn (180) and multilingual (150) are smaller because they exercise orthogonal capabilities; full coverage isn't the goal, just enough to detect a failure."

Power implication for the leaderboard

• The 11 per-domain F1 scores in the reporting template (README.md §Headline) are usable.
• The per-cell scores ("multilingual/PARA_SAFE F1") are not statistically resolvable below 20pp deltas. Either roll those into per-domain scores in the public report, or expand the smallest cells.

---

8. Recommended re-balancing (concrete actions)

Priority-ordered. Numbers in parentheses are row counts.

P0 (block v1.1 release)

1. Re-derive difficulty empirically. Discard the current curator-assigned labels. Run the 5 baseline embedders the methodology specifies (gte-large-en-v1.5, e5-large-v2, text-embedding-3-small, bge-large-en, MiniLM-L6-v2) and label each row by k-of-5 correctness at a fixed threshold (0.85 is a common starting point). Update the JSONL. Expected: ~30-40% of difficulty labels will flip. Until done, do not report difficulty-stratified numbers in any leaderboard.

2. Downgrade 60 weak ADVERSARIAL rows.

   • To UNRELATED (21 rows): all 4 cb-conv-017{1,2}, 020{3,4} (polar_flip with mean_sim<0.3), cb-conv-019{5,6,7} (open_close_flip), cb-mt-0112 (multi_turn yes_no_destruct), cb-ml-0028, 0032, 0033, 0040 (CJK paws-x with Jaccard=0), plus the ~10 more rows in the mean_sim < 0.3 ADV bucket.
   • To RELATED_UNSAFE (39 rows): all 24 rows in the 0.3–0.5 mean_sim bucket — cb-conv-017{3,7,8,9} polar_flip, cb-conv-018{9,90} subject_swap, cb-code-009{9}, 0100, 0104, 0105 user_specific_code_review, plus the remainder.
   • Rebuild ADVERSARIAL stratum back to 410 by sampling 60 new pairs from MeTMaP / PAWS-X / SAFE-CACHE T=1 triplets where cosine > 0.8.

3. Rebalance construction methods. Bring lm_generated from 767 down to 250 (target) and grow real_traffic from 310 to 600. Concretely:

   • 290 LM-generated EQUIV/PARA_SAFE/RELATED_UNSAFE rows currently in conversational, creative, personalized should be replaced with real WildChat / ShareGPT seeds (the methodology has 10 real-traffic sources catalogued).
   • 295 intent-bottom-up rows are missing; pull from MASSIVE (200) + CLINC150 (50) + BANKING77 (45) per methodology §6.3.

P1 (block public leaderboard)

4. Diversify single-source domains.

   • Conversational (260 rows, 100% curator): introduce ≥3 sources (e.g. 80 from PIT-2015 Twitter paraphrase, 50 from ShareGPT short turns, 130 stays curator).
   • Creative (180 rows, 100% curator): pull from No Robots dataset, OpenAssistant (creative subset). Add ≥3 unique subcategories per (label) cell.
   • Personalized (140 rows, 100% curator): add a real persona-rich source (LongDoc-personas or synthesized via 3 different style prompts).
   • Math (130/160 curator, 81%): MATH dataset subset, GSM8K seeds.

5. Expand multilingual from 5 subcategories to ≥3 per cell × 5 cells × ≥3 languages = ~45 unique subcategories. Current 5 subcategories cannot distinguish models that ace one transformation from those that ace all. Pull from:

   • PAWS-X (en/de/fr/es/zh/ja/ko) — currently only 17 rows pulled, expand to 90
   • NyayaBench v2 multilingual agentic — currently 0, expand to 60
   • MASSIVE multilingual (51 languages) — currently 0, expand to 100
   • Grow stratum from 150 to 300 rows (still under-sized vs ~15% recommendation but operationally feasible).

6. Add a cross_tool_paraphrase warning header. Ten rows in tool/ADVERSARIAL have query_a == query_b verbatim and rely on tools_a ≠ tools_b. A query-only embedder must score these as "unknown" not "MISS". Mark them as a separate sub-axis in the scoring template.

P2 (v1.2)

7. Expand small cells (n=20) to n=40. math/UNRELATED, code/UNRELATED, personalized/UNRELATED, multilingual/PARA_SAFE. Cheap (UNRELATED is the easiest stratum to mine). Brings Wilson CI from ±20pp to ±14pp.

8. Carve out a long-prompt agent slice (350-400 rows). The traffic-distributions doc §5.1 calls for this explicitly. Currently the tool stratum has median 30 chars — that is not what production agent traffic looks like. Source from Claude Code public traces or SWE-bench problem statements. Allocate within the existing tool stratum (no row-count delta to the matrix) or expand total to 2,400.

9. Add a deployment-weighted secondary headline metric. Per §2, current 41.5% HIT is well above the 10-20% chat hit-rate. Compute and report two headlines: (a) stratified (current), (b) deployment-weighted using the §2 traffic mix as cell weights.

10. Drop or re-label 50 conversational/RELATED_UNSAFE anaphora rows (cb-conv-0121 … cb-conv-0170) where query_a == query_b AND context_a == context_b == None. Their rationale is "depends on prior turn" but there is no prior turn in the row. Either supply contexts (turning them into multi_turn rows) or move to a separate policy_no_cache stratum.

---

Appendix A — Reproducibility

• Audit similarity computations: sklearn==1.8.0 TF-IDF on raw query_a/query_b text. Vectorizers: analyzer ∈ {word(1-1), word(1-2), char_wb(3-3), char_wb(4-4), char_wb(3-5)}. Cosine over fitted matrices.
• Per-model best threshold (accuracy-maximizing on the 2,000-row corpus): word_uni=0.90, word_bi=0.90, char3=0.95, char4=0.95, char35=0.95.
• Empirical difficulty: k of 5 models correct → easy if k≥4, hard if k≤1, else medium.
• Wilson half-widths: z=1.96, p=0.5.
• Intermediate artifacts: /tmp/cb_sims3.json contains per-row similarities, k_correct, empirical_diff.

Appendix B — Open issues not covered

• No bi-encoder run. sentence-transformers/torch install stalled. The TF-IDF baselines are weaker than mpnet/MiniLM bi-encoders, especially on cross-language and paraphrase-low-overlap pairs. Re-running this audit with the real spec's 5 embedders is likely to worsen the κ for stated-easy multilingual paraphrases (they will move from emp_hard to emp_medium) but the headline finding (κ≈0.03 with bumpy directional bias) will be unchanged.
• No annotator agreement audit. The brief §5.4 envisions Cohen's κ between two LLM judges. Not measured here; the corpus has no judge_decision field populated.
• No held-out leakage check. Brief §5.3 calls for training a MiniLM bi-encoder on 1,500 rows, evaluating on 500, target accuracy ≤ 92%. Not run for this audit (no torch).
• verification_method distribution and routing not audited. Out of scope for "coverage / balance / calibration" but worth a follow-up: do the per-domain verification methods match the methodology table?

— end —