Land the work from #18 and #19 on main

AGENTS.md

@@ -5,6 +5,7 @@ This repository publishes **agent skills for working with Our World in Data** (s
 ## Structure
 
 ```
+FAQ.md                          # common user and contributor questions
 Makefile                        # entry points: make validate / test / triggers
 skills/<skill-name>/SKILL.md    # one directory per skill, and nothing else
 .claude-plugin/marketplace.json # marketplace + plugin definition

FAQ.md

@@ -0,0 +1,169 @@
+# FAQ
+
+Common questions about using and contributing to these skills. For repo
+conventions see [AGENTS.md](AGENTS.md); for how the skills are evaluated see
+[evals/README.md](evals/README.md).
+
+## Using the skills
+
+### My agent isn't using the skills at all
+
+Work through these in order — the first two are the most common.
+
+**1. Are the files where your agent looks?** Every agent reads a different
+directory, and installing to the wrong one fails silently. The
+[`skills` CLI](https://github.com/vercel-labs/skills) knows the paths for 76
+agents and picks the right one:
+
+```bash
+npx skills add owid/skills            # into the current project
+npx skills add owid/skills --global   # user-level, all projects
+npx skills list                       # what is installed where
+```
+
+Nineteen of those agents — including Codex, Cursor and Gemini CLI — share the
+project-level `.agents/skills/` convention. Claude Code is the notable exception,
+using `.claude/skills/` per project and `~/.claude/skills/` globally. If you
+installed by hand, check with `ls .agents/skills/ .claude/skills/ 2>/dev/null`.
+
+**2. Is your agent's reasoning effort turned down?** Skills are invoked through a
+tool call, and lowering reasoning effort makes agents make fewer tool calls — so
+a setting you turned down for cost can stop skills firing at all.
+
+We measured this on Claude Code while building the trigger evals: at
+`--effort low`, five queries that reliably invoke a skill at normal effort were
+answered directly instead, with no skill consulted. Nothing about the skills
+changed; only the effort did.
+
+The same class of setting exists elsewhere — Codex has `model_reasoning_effort`
+(`minimal`/`low`/`medium`/`high`/`xhigh`) in `~/.codex/config.toml`, and most
+agents expose something similar. **We have only measured the effect on Claude
+Code**, so treat the others as a plausible first thing to check rather than a
+known cause. If your agent ignores skills, raise the effort and try again before
+assuming the skill is at fault.
+
+**3. Is the task substantial enough?** Agents skip skills for work they can do
+unaided. "What's the URL for OWID's CO2 chart?" may not trigger anything, while
+"find the best OWID chart on CO2 per capita and pull the data for the G7" will.
+This is by design — it isn't a bug you need to report.
+
+**4. Ask for it by name.** `use the search-charts skill to find…` bypasses
+routing entirely, and is the quickest way to tell "the skill is missing" apart
+from "the skill wasn't selected".
+
+### The wrong skill triggered
+
+These four skills cover deliberately adjacent ground, so this happens. The rough
+division of labour:
+
+| You have… | You want… | Skill |
+|---|---|---|
+| a topic | to find a chart | `search-charts` |
+| a chart URL | its data or metadata | `fetch-chart-data` |
+| your own dataset | it joined to OWID data | `joining-data` |
+| Python, or a need for units/metadata/indicators | a DataFrame | `owid-catalog` |
+
+Naming the skill explicitly always wins. If a realistic request routes to the
+wrong skill repeatedly, that is a bug in our `description` fields and worth
+[opening an issue](https://github.com/owid/skills/issues) — please include the
+prompt you used, since that becomes a trigger-eval case.
+
+### Does installing this put files in my repo?
+
+Yes, if you install per-project: the skill directories are copied into your
+agent's skills directory inside the project. Each skill is a single `SKILL.md`
+and nothing else — about 32 KB in total — and this is deliberate. Our test
+fixtures and eval scripts live in a top-level `evals/` directory precisely so
+they are never copied into your repository, where a fixture CSV could be mistaken
+for your own data. See
+[evals/README.md](evals/README.md#why-evals-live-here-and-not-inside-the-skill-directory).
+
+To keep them out of version control, add your agent's skills directory to
+`.gitignore`, or install with `--global` instead.
+
+### Which tools do I need installed?
+
+Per skill, so you only need what you use:
+
+| Skill | Needs |
+|---|---|
+| `search-charts` | `curl`, `jq` |
+| `fetch-chart-data` | `curl`, `jq` |
+| `joining-data` | `duckdb` |
+| `owid-catalog` | `uv` (or `pip`) |
+
+On macOS, `./install-prerequisites-macos.sh` installs all four. Skills only use
+public OWID endpoints — there are no credentials to configure.
+
+### A skill gave me data I think is wrong
+
+Check whether the skill or the data is at fault. The skills are documentation
+over OWID's public API; they don't transform values. Fetch the same numbers
+directly:
+
+```bash
+curl -s "https://ourworldindata.org/grapher/life-expectancy.csv?csvType=filtered&country=USA&time=2020"
+```
+
+If that matches what the agent told you, the skill worked and any concern belongs
+with the underlying data — see the chart's own page on
+[ourworldindata.org](https://ourworldindata.org). If it doesn't match, that's our
+bug. Two known traps worth ruling out first:
+
+- **`csvType=filtered` applies the chart's own default entity selection**, not
+  "all countries". `population.csv?csvType=filtered&time=2020` returns seven rows
+  — continents and World — with no individual countries. Pass an explicit
+  `country=` filter.
+- **A no-match search still returns results.** OWID's search falls back to
+  low-relevance hits rather than returning nothing, so `nbHits` is never a
+  reliable signal that a topic is missing. Judge the titles.
+
+## Contributing
+
+### `make test` fails and I didn't change anything
+
+That is the contract tests doing their job. They check the OWID endpoints and
+response shapes each `SKILL.md` documents against what the API actually returns,
+so they can break when OWID ships a change and nobody has touched this repo.
+They also run nightly for exactly that reason.
+
+Read the failure before assuming it's a flake — it names the endpoint and the
+mismatch, and the downloaded responses are kept under `evals/results/contract/`
+so you can inspect one without re-running. Network outages also surface here,
+which is intentional.
+
+### `make triggers` costs a lot. How do I make it cheaper?
+
+It runs `queries x RUNS x skills` full agent sessions — 120 for the default
+invocation. While iterating on a description, narrow it:
+
+```bash
+make triggers SKILL=owid-catalog RUNS=1
+```
+
+Don't reach for a lower effort level to save money: as above, effort changes
+whether skills fire at all, so a cheap run measures something other than what
+your users experience. Same caution for `MODEL=` — routing is model-dependent, so
+a cheaper model measures that model's routing. Both are fine for fast iteration
+on wording, then confirm on the real model and effort before believing a number.
+
+### Why do the skills never mention their own evals?
+
+Because the `description` and `SKILL.md` body are loaded into the user's context
+when a skill triggers, and eval prose would be pure overhead there. It's enforced
+by `make validate`, not left to discipline.
+
+### `make triggers` exits non-zero. Is that a failure?
+
+Only if runs errored. Trigger accuracy is a measurement, not a pass/fail gate —
+100% routing accuracy isn't a realistic bar. The runner exits non-zero when runs
+actually failed (meaning the numbers can't be trusted) or when you set a floor
+with `--min-accuracy`. `make test` is the gate.
+
+### How do I add a skill?
+
+See [AGENTS.md](AGENTS.md). The short version: create
+`skills/<name>/SKILL.md` with `name` matching the directory, register it in
+`.claude-plugin/marketplace.json`, then run `make validate`. Keep the directory
+to just `SKILL.md` unless you genuinely need bundled `scripts/`, `references/` or
+`assets/` — everything in there ships to every user.

Makefile

@@ -40,6 +40,13 @@ validate: ## Check spec conformance, the plugin manifest and marketplace registr
 	  echo "  x a SKILL.md references eval files - drop the reference or inline the content"; \
 	  exit 1; \
 	else echo "  ok  no SKILL.md references eval files"; fi
+	@# Eval JSON is hand-authored and hand-reviewed, so it must stay readable. A
+	@# python json.dumps without ensure_ascii=False silently rewrites every em dash
+	@# and accent as a \uXXXX escape, which is unreviewable prose.
+	@if grep -rln '\\u[0-9a-fA-F]\{4\}' evals/skills/*/*.json 2>/dev/null; then \
+	  echo "  x the file(s) above contain escaped unicode - rewrite with ensure_ascii=False"; \
+	  exit 1; \
+	else echo "  ok  eval json has no escaped unicode"; fi
 	@# Registration: neither validator above knows about marketplace.json, and an
 	@# unregistered skill is installable by neither route.
 	@fail=0; \

README.md

@@ -65,6 +65,13 @@ Then put `owid-skills/skills/*` where your agent looks for skills:
 - **Per project** — `./.agents/skills/` (the shared convention read by Codex, Cursor, OpenCode, …)
 - **Per user** — your agent's own skills directory, e.g. `~/.codex/skills/`, `~/.gemini/skills/`, or `~/.claude/skills/`
 
+### Not working?
+
+If your agent doesn't seem to be using the skills, see the
+[FAQ](FAQ.md#my-agent-isnt-using-the-skills-at-all). The two usual causes are the
+files being in a directory your agent doesn't read, and your agent's reasoning
+effort being turned down far enough that it stops making tool calls.
+
 ### Prerequisites
 
 The skills use a few common command-line tools: `curl`, `jq`, `duckdb`, and `uv`. Install them with your package manager (e.g. `brew install jq duckdb uv`), or on macOS run:
@@ -79,7 +86,7 @@ Data published by Our World in Data is open: it is available under the [Creative
 
 ## Development
 
-Want to add or improve a skill? See [AGENTS.md](AGENTS.md) for repo conventions and [evals/README.md](evals/README.md) for how the skills are evaluated.
+Want to add or improve a skill? See [AGENTS.md](AGENTS.md) for repo conventions, [evals/README.md](evals/README.md) for how the skills are evaluated, and the [FAQ](FAQ.md) for questions that come up often.
 
 ```bash
 make            # list targets

evals/run-trigger-eval.py

@@ -73,12 +73,9 @@ def build_command(query: str, model: str | None, effort: str, max_budget: float
         "--verbose",
         "--allowed-tools",
         ALLOWED_TOOLS,
-        # Routing is a shallow decision the model makes before doing any work, so
-        # thinking tokens are pure waste here. Raise this for a fidelity run that
-        # should match a real session's effort.
-        "--effort",
-        effort,
     ]
+    if effort:
+        cmd += ["--effort", effort]
     if model:
         cmd += ["--model", model]
     if max_budget is not None:
@@ -106,7 +103,10 @@ def skills_in_line(line: str, candidates: list[str]) -> set[str]:
         obj = None
 
     if isinstance(obj, dict):
-        blocks = (obj.get("message") or {}).get("content")
+        message = obj.get("message")
+        # Some stream events carry `message` as a plain string, so this cannot
+        # assume a dict — `(x or {}).get(...)` blows up on a non-empty string.
+        blocks = message.get("content") if isinstance(message, dict) else None
         if isinstance(blocks, list):
             for block in blocks:
                 if not isinstance(block, dict) or block.get("type") != "tool_use":
@@ -234,12 +234,20 @@ def main() -> int:
     target.add_argument("--all", action="store_true", help="every skill with a triggers.json")
     parser.add_argument("--runs", type=int, default=3, help="runs per query (default: 3)")
     parser.add_argument("--workers", type=int, default=4, help="parallel runs (default: 4)")
-    parser.add_argument("--timeout", type=int, default=90, help="seconds per run (default: 90)")
+    parser.add_argument(
+        "--timeout", type=int, default=180,
+        help="seconds per run (default: 180). A run that fires a skill is killed immediately, "
+             "but a true negative has no decision to observe, so it only ends when the model "
+             "finishes answering — which at higher effort can take a while.",
+    )
     parser.add_argument("--threshold", type=float, default=0.5, help="fire rate counted as a trigger (default: 0.5)")
     parser.add_argument("--model", default=None, help="model for claude -p (default: your configured model)")
     parser.add_argument(
-        "--effort", default="low", choices=["low", "medium", "high", "xhigh", "max"],
-        help="reasoning effort per run (default: low — routing needs no deep thinking)",
+        "--effort", default=None, choices=["low", "medium", "high", "xhigh", "max"],
+        help="reasoning effort per run (default: inherit your session's effort). Measured, "
+             "not assumed: at low effort the model answers more queries directly instead of "
+             "reaching for a skill, which shows up as misses and understates real triggering. "
+             "Use it to cut cost only when comparing two descriptions at the same effort.",
     )
     parser.add_argument(
         "--max-budget-usd", type=float, default=None,

evals/skills/fetch-chart-data/contract.sh

@@ -33,11 +33,13 @@ if fetch "$CHART.metadata.json?$PARAMS" "$META"; then
     all_match "timespan looks like a year range" "$META" '.columns[] | select(has("timespan"))' \
         '.timespan | test("^-?[0-9]+-[0-9]+$")'
 
-    # SKILL.md types descriptionKey as string[], and tells agents to pay special
-    # attention to it. A markdown bullet string and an array of strings need
-    # different handling, so the type has to be right.
-    all_match "descriptionKey is an array of strings, as documented" "$META" \
-        '.columns[] | select(has("descriptionKey"))' '.descriptionKey | type == "array"'
+    # SKILL.md tells agents to pay special attention to descriptionKey, so its
+    # type has to be right: a markdown bullet string and an array of strings need
+    # different handling.
+    all_match "descriptionKey is a string, as documented" "$META" \
+        '.columns[] | select(has("descriptionKey"))' '.descriptionKey | type == "string"'
+    all_match "descriptionKey is a markdown bulleted list" "$META" \
+        '.columns[] | select(has("descriptionKey"))' '.descriptionKey | test("^- ")'
     note "descriptionKey type: $(jq -r '[.columns[] | select(has("descriptionKey")) | .descriptionKey | type] | unique | join(", ")' "$META")"
     note "columns: $(jq -r '.columns | keys | join(", ")' "$META")"
 fi
@@ -58,10 +60,10 @@ if fetch "$CHART.csv?$PARAMS" "$CSV_SHORT"; then
     ok "short column names contain no spaces" \
         test "$data_columns" = "${data_columns// /}"
 
-    # SKILL.md documents the first three columns as Entity, Code, Year. With the
-    # recommended useColumnShortNames=true they come back lowercased, so this
-    # check reports whether the doc matches the call the doc recommends.
-    csv_header "header is Entity,Code,Year as documented" "$CSV_SHORT" "Entity,Code,Year"
+    # useColumnShortNames=true lowercases the first three headers. SKILL.md now
+    # documents both forms and tells agents to match case-insensitively; these two
+    # checks pin down which call produces which.
+    csv_header "useColumnShortNames=true lowercases the first three headers" "$CSV_SHORT" "entity,code,year"
 fi
 
 section "CSV endpoint without useColumnShortNames"

evals/skills/owid-catalog/contract_check.py

@@ -123,9 +123,22 @@ def main() -> int:
         ("indicator results convert to a DataFrame", lambda: not indicators.to_frame().empty),
         ("an indicator result can .fetch() a single column", lambda: _nonempty(indicators[0].fetch())),
         ("an indicator result can .fetch_table()", lambda: _nonempty(indicators[0].fetch_table())),
+        # search() takes no sort argument; sorting happens on the returned
+        # ResponseSet, keyed by a field of the result.
         (
-            "sort_by='relevance' is accepted",
-            lambda: search("CO2 emissions per capita", kind="indicator", sort_by="relevance") is not None,
+            "search() rejects a sort_by argument",
+            lambda: _raises(
+                TypeError,
+                lambda: search("CO2 emissions per capita", kind="indicator", sort_by="relevance"),
+            ),
+        ),
+        (
+            "indicator results expose score, popularity and n_charts",
+            lambda: {"score", "popularity", "n_charts"} <= set(indicators.to_frame(all_fields=True).columns),
+        ),
+        *(
+            (f"indicator results re-sort by {key!r}", lambda k=key: indicators.sort_by(k, reverse=True) is not None)
+            for key in ("score", "popularity", "n_charts")
         ),
     ]
     reason = "SKIP_SLOW=1" if skip_slow else "indicator search did not return results"
@@ -144,6 +157,17 @@ def _import() -> bool:
     return True
 
 
+def _raises(exc_type: type[BaseException], fn: Callable[[], Any]) -> bool:
+    """True if fn raises exc_type. Used to pin down what the API refuses, so a
+    SKILL.md example cannot drift back into calling an argument that never
+    existed."""
+    try:
+        fn()
+    except exc_type:
+        return True
+    return False
+
+
 def _nonempty(obj: Any) -> Any:
     """Assert the table or result set has rows, and return it so callers can
     keep using the object rather than a bool."""

evals/skills/owid-catalog/triggers.json

@@ -12,7 +12,7 @@
   {
     "query": "does owid have population broken down by sex and age group? the grapher charts only seem to give me totals and i need the extra dimensions for a cohort model",
     "should_trigger": true,
-    "note": "the garden vs grapher channel distinction is the answer"
+    "note": "the garden vs grapher channel distinction is the answer. Deliberately hard: it opens with 'does owid have', which reads as chart discovery, and the owid-catalog tells (extra dimensions, cohort model) only arrive later. It misrouted to search-charts 3/3 before the descriptions were rewritten."
   },
   {
     "query": "write a uv script that pulls owid renewable energy share, resamples to 5-year means per country, and writes the result to renewables_5yr.parquet",