Skip to content

overture_canary: field-coverage regression gate (#416) - #417

Merged
chuofringer merged 3 commits into
mainfrom
auto/issue-416
Aug 25, 2026
Merged

overture_canary: field-coverage regression gate (#416)#417
chuofringer merged 3 commits into
mainfrom
auto/issue-416

Conversation

@chuofringer

Copy link
Copy Markdown
Owner

Summary

  • Extends scripts/overture_canary.py (Stay current with the latest Overture release automatically (rollover, staleness alarm, schema canary) #219) with a third check: field-coverage regression, alongside the existing pin-staleness and per-theme column-presence checks (unchanged).
  • Motivation (north-star): Widespread drop in brand field coverage for Brazil between releases ~90% of a random sample of well-known chains lost matched POIs (median -21%, some down to zero OvertureMaps/data#546 documented a release where a whole country's brand-matched POIs collapsed (median -21%, some chains to zero) with no schema change — the existing column-presence probe would have sailed straight through it. This closes that blind spot before the next pin bump (docs/PIN.md) adopts a damaged release blindly, protecting the honesty pillar.
  • Probe set: 5 dense metro bboxes spread across continents — Central Paris (Europe), Manhattan (North America), Tokyo/Shibuya (Asia), São Paulo/Sé (South America), Lagos Island (Africa — the continent #546 was actually about). Each ~0.1° class (PROBE_RADIUS_M = 5_500), built with geo.bbox_around/geo.bbox_filter_sql, same helpers the runtime uses.
  • Metrics per bbox, pinned vs newest release: places row count, addresses row count, and non-null rate of brand, confidence, and taxonomy.primary (the column overture._place_select_exprs actually reads for "category" — not basic_category).
  • REGRESSION_TOLERANCE = 0.20 — chosen just under #546's documented median -21% drop, so the gate catches an incident of that shape without chasing ordinary release-to-release noise.
  • MIN_ROWS = 50 floors row-count-backed metrics on the pinned side: a tiny denominator turns "one row missing" into a meaningless "100% drop".
  • Reuses the existing markdown-report + exit-1 channel unchanged — the workflow's issue-creation path needs no changes and none were made (no .github/workflows/* touched).
  • Gate is skipped (noted, not failed) when the pinned release already is newest, or when a probe can't read a release.
  • Pure logic (compare_bbox_metrics, render_coverage_report) is separate from the network probe (probe_bbox_metrics, never run from pytest — same rule the script already states) and is covered offline with synthetic numbers in tests/test_overture_canary.py.

Test plan

  • uv run pytest -q — 2375 passed (2368 baseline on main + 7 new offline tests), including existing column-check/pin-staleness tests unchanged
  • uv run ruff check . — clean
  • No .github/workflows/* files changed
  • No new dependencies
  • Offline coverage: healthy releases → clean; synthetic brand-collapse and row-count drops → flagged with correct table numbers; small-denominator metric → skipped, not flagged; pinned == newest → whole gate skipped with a note
  • CHANGELOG.md ### Added entry added
  • No schema/tool/server-surface changes → no benchmark regen needed

🤖 Generated with Claude Code

Extends the weekly canary (#219) with a coverage probe over 5 metro
bboxes, comparing pinned vs newest release on places/addresses row
counts and brand/confidence/taxonomy non-null rates. Flags a >20%
relative drop (OvertureMaps/data#546's median -21% brand collapse) via
the existing markdown-report/exit-1 channel, catching a release that
keeps every required column but silently guts the values behind them.
Script-only; comparison/report logic is pure and offline-tested.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@chuofringer

Copy link
Copy Markdown
Owner Author

Lead review (loop)

Re-ran locally: 2375 passed (+7), ruff clean; verified zero .github/ changes (script-only per the issue's hard-rule scope).

Verified in review:

  • Pure comparison/threshold/report logic is cleanly separated from the network probes (which correctly stay untested, per the script's own never-from-pytest rule).
  • The 0.20 tolerance is sized against the documented incident (#546's median −21%), MIN_ROWS floors keep tiny denominators from producing noise flags, and skip conditions append to the clean-line suffix rather than findings — so the gate can never cause a false workflow issue.
  • Probe set is continent-spread with Lagos deliberately covering #546's failure geography; taxonomy.primary confirmed as the column the runtime actually reads.
  • Existing pin-staleness + column-drift behavior untouched on the normal path.

Acceptance criteria of #416 met. Ready for owner merge (loop does not merge).

claude added 2 commits August 25, 2026 03:56
- The MIN_ROWS floor now also applies to a rate metric's pinned
  non-null count: 800 places rows with 6 branded ones turned two rows
  moving into a -33% 'drop' — a recurring false alarm in exactly the
  sparse-brand metros the probe set targets. Offline test added.
- Coverage-probe globs are built from DEFAULT_UPSTREAM_BASE directly,
  matching the column probe: a runner's PLACEROOT_DATA_PATH* pin or a
  registered override can no longer redirect the probes or raise
  UpstreamUnavailable outside the try block (which aborted the run and
  discarded findings the other checks had already collected).
- Corrected the probe-box size comment (half-width semantics, and
  longitude degrees widen away from the equator).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YCuTkstNj5KscUQ9u8cLMK
@chuofringer
chuofringer merged commit 90c619d into main Aug 25, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants