Fix the seven documentation drifts the contract tests found

[lucasrodes]

Stacked on #17 — base is evaluate-skills, not main, so the diff shows only the fixes. Merge #17 first and this retargets to main automatically.

#17 landed the contract tests with seven checks failing on purpose, so the first run would report the drift rather than freeze it in. This fixes all seven. Contract tests go from 104/7 to 120 passed, 0 failed.

Every change is backed by a check that passes. Where a check asserted "the API sends what SKILL.md documents" and the doc turned out to be the wrong side of the comparison, I inverted the check to assert the corrected behaviour rather than deleting it — so a field can't creep back into the docs without the API actually sending it.

The fixes

search-charts

Change	Evidence
Removed objectID from BaseSearchChartHit	returned on 0 of 200 hits sampled
Removed explorerType from SearchExplorerViewHit, and the now-unused ExplorerType enum	returned on 0 of 59 explorerView hits
Removed chartConfigId from SearchMultiDimViewHit	returned on 0 of 112 multiDimView hits
Added publishedAt / updatedAt	present on 200 of 200, ISO 8601
Added containerTitle to both non-chart types	present on every explorerView and multiDimView hit, absent on every chart hit
Added StackedDiscreteBar → stacked-discrete-bar and Dumbbell → dumbbell to the tab table; added Dumbbell to GrapherTabName	values taken from GRAPHER_TAB_CONFIG_OPTIONS in owid-grapher, not guessed

Net effect on the skill's token cost is slightly negative — the five-line ExplorerType enum was dead weight once explorerType went.

fetch-chart-data

• descriptionKey is typed string[] but is a single markdown bulleted string (- per line). Fixed the type and said so, since the skill tells agents to surface this field to users.
• Added a note that useColumnShortNames=true — which the skill recommends — lowercases the first three headers to entity,code,year, and to match column names case-insensitively. The documented Entity,Code,Year was only true for calls that skip the recommended parameter.

owid-catalog

• search(..., sort_by="relevance") raised TypeError: search() in owid-catalog 1.1.0 has no sort_by parameter. Replaced with ResponseSet.sort_by(), which does exist, and named the fields it can actually sort on (score, popularity, n_charts) — verified by inspecting the installed signature and result fields.

One thing I added beyond the six

chartConfigId on SearchMultiDimViewHit is a seventh drift, same defect class as objectID and explorerType (schema declares a required field the API never sends). I only found it because the search index shifted mid-session — see below — which finally gave me multiDimView hits to inspect. Fixing it alongside the other two seemed more coherent than leaving one instance of the same bug behind, but say the word and I'll split it out.

A real finding about eval fragility

The query energy mix returned 47 chart / 53 explorerView hits at the start of this work and 100 multiDimView hits about an hour later — same query, same parameters. The index composition is not stable.

That broke my own assertion ("this query returns at least one explorerView hit"), not the skill. So the non-chart hit-shape checks now:

• assert shared fields across all non-chart hits,
• assert each specific type's shape only when that type appears in the sample, and skip() loudly when it doesn't,
• use q=life expectancy, which currently returns all three types (chart=58, explorerView=30, multiDimView=12).

Worth knowing for layer 2 as well: assertions pinned to live-index composition will flake. Assert shapes, not populations.

Verification

fetch-chart-data: 31 passed, 0 failed
joining-data:     29 passed, 0 failed
owid-catalog:     29 passed, 0 failed
search-charts:    31 passed, 0 failed

Run it yourself with ./evals/run-contract-tests.sh.

Still not fixed (deliberately)

These were in #17's description as observations rather than failing checks, and each needs a judgement call rather than a mechanical fix:

1. joining-data tells agents to read $.columns.[0].timespan — columns is an object keyed by column name, so that path is not valid jq. It matters most for population-with-un-projections, where the first column is estimates (1950–2023) and the projection to 2100 is a second column; reading one column's timespan gives the wrong answer for exactly the recent/future-year case the skill recommends that chart for. Needs a decision on what to tell agents instead (union across columns? name the two columns explicitly?).
2. A nonsense query returns five irrelevant hits, not zero. search-charts says "If you don't get any results, try slightly different terms", but nbHits == 0 essentially never happens. The guidance needs rewriting around low relevance rather than zero results — a prose change with real behavioural consequences, so probably yours to make.
3. csvType=filtered also applies the chart's own default entity selection. population.csv?csvType=filtered&time=2020 returns 7 rows — continents and World, no countries. Both fetch-chart-data and joining-data imply country rows are present.
4. owid-catalog says indicator results are "ranked by popularity". They're actually ordered by semantic similarity (score) descending, with popularity breaking ties. I left the prose alone since it's adjacent to but not part of the sort_by fix.

Happy to do all four in a follow-up.

🤖 Generated with Claude Code