overture_canary: field-coverage regression gate (#416) - #417
Merged
Conversation
Extends the weekly canary (#219) with a coverage probe over 5 metro bboxes, comparing pinned vs newest release on places/addresses row counts and brand/confidence/taxonomy non-null rates. Flags a >20% relative drop (OvertureMaps/data#546's median -21% brand collapse) via the existing markdown-report/exit-1 channel, catching a release that keeps every required column but silently guts the values behind them. Script-only; comparison/report logic is pure and offline-tested. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Owner
Author
Lead review (loop)Re-ran locally: 2375 passed (+7), ruff clean; verified zero Verified in review:
Acceptance criteria of #416 met. Ready for owner merge (loop does not merge). |
- The MIN_ROWS floor now also applies to a rate metric's pinned non-null count: 800 places rows with 6 branded ones turned two rows moving into a -33% 'drop' — a recurring false alarm in exactly the sparse-brand metros the probe set targets. Offline test added. - Coverage-probe globs are built from DEFAULT_UPSTREAM_BASE directly, matching the column probe: a runner's PLACEROOT_DATA_PATH* pin or a registered override can no longer redirect the probes or raise UpstreamUnavailable outside the try block (which aborted the run and discarded findings the other checks had already collected). - Corrected the probe-box size comment (half-width semantics, and longitude degrees widen away from the equator). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YCuTkstNj5KscUQ9u8cLMK
# Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/overture_canary.py(Stay current with the latest Overture release automatically (rollover, staleness alarm, schema canary) #219) with a third check: field-coverage regression, alongside the existing pin-staleness and per-theme column-presence checks (unchanged).PROBE_RADIUS_M = 5_500), built withgeo.bbox_around/geo.bbox_filter_sql, same helpers the runtime uses.brand,confidence, andtaxonomy.primary(the columnoverture._place_select_exprsactually reads for "category" — notbasic_category).REGRESSION_TOLERANCE = 0.20— chosen just under #546's documented median -21% drop, so the gate catches an incident of that shape without chasing ordinary release-to-release noise.MIN_ROWS = 50floors row-count-backed metrics on the pinned side: a tiny denominator turns "one row missing" into a meaningless "100% drop"..github/workflows/*touched).compare_bbox_metrics,render_coverage_report) is separate from the network probe (probe_bbox_metrics, never run from pytest — same rule the script already states) and is covered offline with synthetic numbers intests/test_overture_canary.py.Test plan
uv run pytest -q— 2375 passed (2368 baseline on main + 7 new offline tests), including existing column-check/pin-staleness tests unchangeduv run ruff check .— clean.github/workflows/*files changed### Addedentry added🤖 Generated with Claude Code