🎉🤖 Declare datasets to the Admin API instead of upserting them

[Marigold]

Written by Claude Opus 5 — @Marigold at the wheel.

Stacked on #6694 — base is ghost-variables, not master. Needs owid/owid-grapher#7001 (same branch name, one staging server).

ETL now declares what a dataset should contain and lets Grapher reconcile it, instead of upserting rows and inferring deletions from what it didn't mention. Three calls, all addressed by catalogPath — no variable ids cross the boundary any more.

PUT /datasets/by-catalog-path/:path             dataset + its full indicator list
PUT /datasets/by-catalog-path/:path/indicators  metadata chunks (repeatable)
PUT /datasets/by-catalog-path/:path/checksum    what we published, + source checksum

Why declaring changes so much

A ghost indicator used to be identified by absence — nobody says "delete this", the signal is that nobody mentioned it. That's why ETL had to accumulate every variable id across ~95 chunk calls and hand the list over at the end.

Sending metadata for all indicators every run removes that: if nothing is ever skipped, every live indicator is named in every run, so the first call's list is the complete membership and deletions follow from it directly. No keep-list, no run id, no session state.

That also deletes calculate_checksum_metadata — the most bug-prone function in the file, carrying scar tissue from two real incidents — along with _dataset_metadata_fields and its hand-maintained "kept in sync with" invariant. Grapher compares metadata against what it last published, which is where the previous version actually lives.

What ETL keeps

Everything that needs the values, which Grapher never receives: the data checksum, the inferred type, the timespan, the distinct entity ids and years, and the data.json upload itself. Entity names and codes are resolved Grapher-side, so the payload carries ids.

Nothing is written before it's true

Grapher publishes metadata.json itself now, so the row, the R2 object and metadataChecksum land together inside one call. dataChecksum is written only by the third call, after ETL's uploads have gone through. Whoever publishes a file records its checksum — which is why there's no confirmation call and no in-progress marker.

A run that dies mid-way leaves those files looking stale and gets them re-uploaded next time. Wasteful, never wrong.

Cost we're accepting

A one-line .meta.yml edit on a huge dataset now sends all its metadata — ~470MB on who/2021-07-01/ghe/estimates (47,160 indicators in a single table). The bill lands on a handful of datasets and buys the removal of the run-state machinery.

Still open

• The byte-for-byte metadata diff has not been run. Before this merges, both implementations need to run over a large real dataset on staging with the JSON compared. A near-miss port would silently change metadata for 550,215 variables and nothing would fail. This is the gate.
• Admin API load is still unmeasured, and this adds more than 🔨🤖 Move ghost-variable cleanup to the Grapher admin API #6694 did.
• SUBSET/INSTANT runs never declare and never record a checksum — correct, but it means a filtered run no longer cleans up anything. Worth a second opinion on whether that's the behaviour we want.

Details

Files

• etl/grapher/to_db.py — prepare_indicator (payload + data checksum, no skipping) and flush_indicator_batch (send chunk, upload the data files Grapher asks for, return their checksums). upsert_dataset, upsert_metadata, upload_metadata, upsert_origins, calculate_checksum_metadata, _dataset_metadata_fields and cleanup_ghost_variables are all gone. blocked_indicators_allow_run keeps the raise-or-warn policy, which stays here because it depends on the environment and a chart-diff against production.
• apps/chart_sync/admin_api.py — put_dataset, put_indicators, put_dataset_checksum, all through requests_with_retry().
• etl/steps/__init__.py — prepares every indicator, declares, then chunks by payload bytes. _filter_indicators handles the SUBSET/INSTANT paths.
• etl/grapher/helpers.py — _dataset_metadata_payload.
• apps/backport/datasync/data_metadata.py — 295 lines → 77.
• lib/owl/owl/grapher.py — same flow.

Checked

make check and make unittest (679 passed).

[owidbot]

Quick links (staging server):

Site Dev	Site Preview	Admin	Wizard	Docs	Docs Preview

Login: ssh owid@staging-site-variable-metadata-admin-api

chart-diff: ✅

No charts for review.

data-diff: ✅ no differences

No differences found.

Automatically updated datasets matching excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included. Run locally with etl diff REMOTE data/ --include <dataset> --verbose.

Edited: 2026-08-18 11:54:30 UTC
Execution time: 11.79 seconds