🎉 Charts as first-class citizens in ETL, addressed by chart config UUID

[danyx23]

Written by Claude Opus 5 — @danyx23 at the wheel.

Summary

This is a slight modification of @pabloarosado's existing "charts as first-class citizens" work in #6174 — the collection/chart-upsert machinery and the layered etlConfig approach are his. This PR adds two commits on top that make the chart's config UUID (charts.configId) its identity, both for addressing charts and for matching them across environments, instead of the numeric auto-increment id. Companion grapher PR: owid/owid-grapher#6826 (stacked on owid/owid-grapher#6825).

New on top of #6174:

• chart_upsert: replaces the bootstrap create + etlConfig PUT with the new idempotent PUT /charts/by-config/:uuid/etlConfig upsert. A new chart's admin patch starts out empty so the ETL layer owns dimensions etc. from birth (no more rediff workaround needed for fresh charts). Retries can no longer create duplicate charts.
• The UUID is declared in the config YAML, not looked up. See below — this is the main departure from 🎉 Charts as first-class citizens in ETL: layered approach with grapher #6174's addressing model.
• chart-diff: matches source/target charts by configId first (the column exists on production today, so no runtime column guard is needed), then catalogPath, keeping id+createdAt only as a legacy fallback. Catalog-path twins generalize to cross-env twins.
• chart-sync: carries the source chart's configId when creating charts on the target (via the upsert endpoint for ETL-authored charts, and a new configId param on POST /charts for hand-authored ones), so future diffs match the rows directly. Narrative charts are now matched by their chartConfigId instead of the collision-prone numeric id.
• model: Chart.load_chart supports config_id lookups; NarrativeChart.load_by_chart_config_id added.
• etl chart-config-id: new CLI to write the UUID into a chart's config YAML — see below.

Requires the grapher endpoints from owid/owid-grapher#6826.

How chart identity differs from the base PR

In #6174 a single chart's ETL identity was resolved from the target database: chart_upsert looked the chart up by catalogPath, fell back to matching on slug, adopted whatever configId it found, and minted a fresh UUID only when nothing matched.

Here the identity is declared in the config YAML as a new required top-level field, and nothing is looked up:

chart_config_id: "0191b6c7-5595-70b2-8d30-fa03fccd7add"
topic_tags:
  - "Animal Welfare"
dimensions: []
views:
  - ...

Why the change: a lookup makes the identity environment-derived, so the same YAML can resolve to a different chart in each environment, or create a new chart wherever the lookup misses (e.g. a chart whose slug was changed in the admin). Declaring the UUID makes one YAML address one chart everywhere — local, staging and production — and makes "which chart does this step own?" answerable by reading the file.

Consequences:

• chart_config_id is required for single-chart collections (dimensions: []) and rejected on mdims, which stay identified by their catalog path.
• Bringing an existing chart into ETL means putting that chart's current charts.configId in the YAML. Creating a new chart means minting a UUIDv7 (new_chart_config_id(), formerly the private _uuid_v7).
• Validation is deliberately strict, in Collection.validate_chart_config_id(): the value must be a canonical lower-case dashed UUID. uuid.UUID() on its own also accepts {braces}, a urn:uuid: prefix, undashed hex and upper case — none of which match grapher's CHAR(36) lookup key, so they would have missed the intended chart and silently created a new one. A non-v7 UUID is accepted but logged as a warning, since every chart config UUID grapher has generated so far is a v7. The check runs from Collection.save() (covering programmatically-built collections) and again from upsert_collection_as_chart() (so no entry point bypasses it), and the multidim schema carries the same canonical-form pattern for YAML configs.
• Dropped guard: the previous slug-based lookup raised if a step's slug collided with a chart owned by a different ETL step. Duplicate slugs are now left to grapher to reject. If we want that check back it can be a build-time check over the config YAMLs, which needs no DB.

etl chart-config-id

Since the UUID now has to be in the file, there's a subcommand to put it there, rather than expecting people to invent a UUIDv7 or hand-query the DB:

etl chart-config-id new <config.yml>                                    # brand-new chart
etl chart-config-id lookup <config.yml> --slug banning-of-chick-culling # existing chart, by slug
etl chart-config-id lookup <config.yml> --chart-id 7118                 # existing chart, by numeric id

lookup queries the configured grapher DB (OWID_ENV, or --env <staging-branch> / --env <path/to/.env>). The chart is always named explicitly — by slug, or by the numeric id that the admin shows and that older automation carries — because guessing which chart is meant is precisely the failure this field exists to prevent.

Guards, since this field decides chart identity:

• Only single-chart configs are accepted (dimensions: [] and exactly one view), so an mdim — including one that fills dimensions programmatically — can't be given a field that would then be rejected at build time.
• An existing chart_config_id is never replaced without --force.
• Re-writing the same value is a no-op, so the command is safe to re-run.
• A slug matching several charts resolves to the published one, or errors telling you to pick with --chart-id.

The write goes through ruamel, so comments and formatting survive.

banning_of_chick_culling.config.yml — the one ETL-authored single chart so far — declares the UUID of the chart that already exists at /grapher/banning-of-chick-culling, so pushes keep landing on it rather than orphaning it.

Test plan

• Extended chart-diff matching tests (config-UUID twins, catalogPath twins, legacy id+createdAt fallback).
• chart_config_id validation tests: required for single charts, rejected on mdims, must be a string, must be canonical (parametrized over the four lenient forms uuid.UUID would otherwise wave through), non-v7 warning, and UUIDv7 format/time-ordering of the minted value.
• etl chart-config-id tests: accepted/rejected config shapes, overwrite guard, no-op on re-write, comment preservation, and the exactly-one-identifier rule.
• Both lookup modes exercised against staging-site-etl-charts: --slug banning-of-chick-culling and --chart-id 7118 both resolve to 0191b6c7-5595-70b2-8d30-fa03fccd7add, matching the UUID committed in the YAML.
• make check and the affected test files pass.

🤖 Generated with Claude Code

[owidbot]

Quick links (staging server):

Site Dev	Site Preview	Admin	Wizard	Docs	Docs Preview

Login: ssh owid@staging-site-etl-charts

chart-diff: ✅

No charts for review.

data-diff: ✅ no differences

No differences found.

Automatically updated datasets matching excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included. Run locally with etl diff REMOTE data/ --include <dataset> --verbose.

Edited: 2026-07-31 10:07:01 UTC
Execution time: 4.40 seconds

[danyx23]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

etl/apps/chart_sync/cli.py

Lines 203 to 206 in 5d174a2

	select(gm.Variable.datasetId)
	.distinct()
	.join(gm.ChartDimensions, gm.ChartDimensions.variableId == gm.Variable.id)
	.where(gm.ChartDimensions.chartId.in_(chart_ids), gm.Variable.datasetId.isnot(None))

Query archived datasets with target chart IDs

Written by Claude GPT-5.6 Sol — @danyx23 at the wheel.

With --archive, UUID/catalog-path twins can have different source and target numeric IDs, but this target-session query still filters with the source IDs collected above. Consequently it does not collect the datasets currently used by the actual target charts, so datasets orphaned by their synchronization are omitted from the later archive pass. Build this filter from each diff's target_chart.id when a target twin exists.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[pabloarosado]

Thanks @danyx23! While testing the scenario of creating a new chart from VSCode, I realised that the grapher chart editor was empty (showing a "No table loaded yet"). This looked like a bug, but it turned out to be an issue with the staging server. This was Fable's assessment:

The box serves the admin client from live-data/bakedSite/assets-admin/, and that bundle had been frozen since the server's creation (Jul 24, 06:38) even though the grapher checkout and server-side code were current and fresh builds kept landing in owid-grapher/dist/. So the editor was running pre-PR client code that doesn't know about the etlConfig layer. It manually copied dist/assets-admin/ into bakedSite/ yesterday, so the editor works now, but whatever re-bakes those assets on this box is still broken (staging-site-etl-chart-config re-bakes fine, most recently Jul 30 at 11:32). Until that's fixed, client-side changes you push won't reach the served admin bundle.

Apart from that staging server hiccup, etl chart creation and editing is working as expected, and the addition of UUID to the chart steps makes sense.

[pabloarosado]

Thanks for this work, Daniel. I've tested it and works as expected (except for an issue I mentioned that seems to be temporary).
I've also created this other PR to handle how chart diff and chart sync should behave. But it shouldn't block the current PR.

[danyx23]

Written by Claude Opus 5 — @danyx23 at the wheel.

Heads-up: the grapher side of this work (owid/owid-grapher#6826) removes PUT /charts/:chartId/etlConfig. ETL config pushes are now addressed only by the chart's config UUID, via PUT /charts/by-config/:chartConfigId/etlConfig.

This branch still calls the removed endpoint in one place: AdminAPI.put_chart_etl_config in apps/chart_sync/admin_api.py, used from apps/chart_sync/cli.py when syncing a chart that already exists in the target environment. etl/collection/chart_upsert.py is unaffected — it already uses upsert_chart_etl_config.

The fix is to drop put_chart_etl_config and call upsert_chart_etl_config instead, with the target chart's own config UUID, which is available right at that call site:

resp = target_api.upsert_chart_etl_config(
    chart_config_id=diff.target_chart.configId,
    grapher_config=migrated_etl_config,
    catalog_path=diff.source_chart.catalogPath,
    user_id=user_id,
)

It has to be target_chart.configId, not source_chart.configId: for a cross-environment twin matched on catalogPath the two can differ, and the by-config endpoint would then create a new chart in the target instead of updating the matched one.

Two related grapher-side changes worth knowing about:

• catalogPath is no longer treated as a chart's identity. A push carrying a different path than the one on record now simply updates it, so a renamed or moved step is an ordinary update rather than a 409. Only assigning a path that already belongs to a different chart is still rejected.
• Chart config UUIDs are normalised to lower case when stored. The case-insensitive comparison added in 9f278a9 stays correct, and should stop being needed for newly written rows.

DELETE /charts/:chartId/etlConfig is unchanged (and has no callers here — delete_chart_etl_config is defined but never used).

These two branches need to land together, so this one should be updated before the grapher PR merges.