Rewrite two skill descriptions so owid-catalog stops losing to search-charts

[lucasrodes]

Stacked on #18 — base is fix-skill-doc-drift, so the diff shows only the triggering work. Merge #18 first and this retargets automatically.

This is layer 2 doing its job. Your make triggers run found a real, repeatable routing defect; this fixes it and re-measures.

The defect

owid-catalog sat at 40% recall. It lost three of its five positives to search-charts — and the three it lost were exactly its exclusive differentiators:

Query	Fired (before)
population broken down by sex and age group… extra dimensions	search-charts 3/3
write a uv script… resamples… writes to parquet	search-charts 2/3, owid-catalog 1/3
semantic search… i want owid indicators about trust in government	search-charts 3/3

Two at 3/3 means consistent behaviour, not sampling noise. Across all 40 queries search-charts fired 35 times against owid-catalog's 19 — a general attractor.

Why, and what changed

search-charts spent three of its four sentences on a topic catalogue — "global population data, energy and electricity, economic data like GDP or poverty, health data like causes of death…". A topic list matches any OWID data question. It now leads with the action, compresses the topics to keywords, and states what it is not for, naming each sibling.

owid-catalog described itself as a "Python-native alternative to the HTTP-based search-charts and fetch-chart-data skills" — subordinate framing — while its real trigger vocabulary (pandas, notebook, codebook, sex/age/variant dimensions, semantic search) sat in the body, which is never read at routing time. It now claims that territory in concrete terms and says "prefer it over".

Both stay well inside the spec's 1024-char limit (795 and 751).

Measured

One run per query, both new descriptions live simultaneously, at faithful session effort:

Set	Recall	Misroutes	Regression?
owid-catalog	40% → 100%	3 → 0	—
search-charts	100% → 100%	0 → 0	none

All five owid-catalog positives now route correctly, including the two that were 3/3 wrong. All three of its sibling-routing negatives still yield to the correct sibling, and every search-charts positive still wins its own queries.

Trigger-set audit

You asked me to check the eval sets for problems. I checked composition, symmetry, contradictions, and cross-set consistency:

• Composition is uniform: every set is 5 positives + 3 sibling-routing + 2 pure negatives.
• Symmetry is exact: each skill owns 5 positives and is named as the expected winner in exactly 3 sibling cases elsewhere, one per sibling. No skill is over- or under-exercised.
• No contradictions: no negative expects its own skill; no expected_skill names a non-existent skill.
• Near-duplicates are consistent, not accidental. Six cross-set pairs score >0.55 similarity — every one expects the same winner in both sets. A query near-identical to one of owid-catalog's positives, filed as a negative in search-charts' set, expects owid-catalog in both places. That is the property you want.

One case deserved a comment rather than a change: does owid have population broken down by sex and age group? opens with discovery phrasing, and its owid-catalog tells arrive only later. It is deliberately hard, it was one of the 3/3 misroutes, and it now passes — so I recorded that in its note to stop a future reader "fixing" the case instead of the description.

No test case was weakened to make this pass. The only trigger-file edit is that note.

Three harness fixes this run exposed

1. --effort low was a bad default, and I had reasoned my way into it. Last turn I defaulted it to low arguing routing needs no deep thinking. Measurement says otherwise: at low effort the model answers more queries directly instead of reaching for a skill, producing five spurious misses — including two on fetch-chart-data positives that pass at faithful effort. Effort is a fidelity knob, not a free cost saving. Runs now inherit the session effort, and the help text says why lowering it distorts the result.

2. A real crash: skills_in_line raised 'str' object has no attribute 'get' on stream events carrying message as a string. Worth noting the error handling worked — it surfaced as a runner error rather than a silent wrong answer, so no measurement was corrupted.

3. Timeout 90s → 180s. A run that fires a skill is killed the moment the decision is observed, but a true negative has no decision to observe, so it only ends when the model finishes answering. At faithful effort two negatives blew through 90s and were correctly reported as errors rather than passes.

Also in this PR: a FAQ

The effort finding is something a user would experience as "these skills don't work" with no way to diagnose it, so it's now written down for them rather than only for us. FAQ.md, linked from the README's install section and its Development section.

Deliberately not Claude-centric. The install-path answer covers the 19 agents that share the .agents/skills/ convention (Codex, Cursor and Gemini CLI among them) and flags Claude Code as the exception at .claude/skills/. On effort, the finding is reported as measured on Claude Code only — Codex's equivalent model_reasoning_effort is named as a plausible first thing to check, not a known cause, since we haven't measured it. Generalising one measurement across 76 agents would be exactly the mistake the effort finding just corrected.

It also covers: which skill does what when the wrong one fires; whether installing writes into your repo; per-skill tool prerequisites; and how to tell a skill bug from a data question — including the two API traps the contract tests surfaced (csvType=filtered applying the chart's own entity selection, and a no-match search still returning low-relevance hits). The contributor half answers the questions this repo's tooling raises: why make test can fail with no local change, how to keep make triggers cheap without corrupting it, and why a non-zero make triggers isn't a failure.

Every factual claim is verified against the live API or the installer source. One incidental find: csvType=filtered&time=2020 on the population chart returns seven rows, not six — the CSV has no trailing newline, so wc -l undercounts. Good news is csv_min_rows in the harness already uses grep -c ''.

Verification

• Contract tests: 120 passed, 0 failed — descriptions don't affect them, but the skill_md_contains checks confirm nothing was clobbered.
• make validate green, including skills-ref spec conformance on both rewritten descriptions.
• make lint clean.

Before merge

Two things I did not do, both cost-driven — you're deep into a session window:

1. Only two of four sets were re-measured. fetch-chart-data and joining-data scored 90%/100% and neither description changed, but they share the routing environment, so a regression there is possible. A full make triggers (all four, RUNS=3) is the confirming run.
2. RUNS=1, not 3. Enough to show a 3/3 misroute became 0/1 and to rule out gross regression; not enough to pin a fire rate. Two negatives also came back as error (the 90s timeout, now raised), so 2 of 20 rows are unmeasured.

The direction is unambiguous — 3/3 wrong to 0/1 wrong on the same queries — but the numbers above are indicative, not final.

🤖 Generated with Claude Code