🐛🤖 Energy mix: scope and rewrite the processing descriptions - #6674
Merged
Conversation
The EIA note was in the shared definitions block, so every indicator claimed to be extended with EIA data, when the EIA is only used to extend total energy supply. It now sits on total energy supply and the indicators derived from it. The Smil note is split in two, so the shares of primary energy including traditional biomass no longer claim to exclude it, and its wording is plainer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pabloarosado
changed the title
Contributor
|
Quick links (staging server):
Login: chart-diff: ✅No charts for review.data-diff: ❌ 1 changed · 1 identical — full report~ garden/energy/2026-06-30/energy_mix
~ energy_mix.biofuels_annual_change_pct (changed metadata)
~ energy_mix.biofuels_annual_change_twh (changed metadata)
~ energy_mix.biofuels_per_capita_kwh (changed metadata)
~ energy_mix.biofuels_share_including_biomass_pct (changed metadata)
~ energy_mix.biofuels_share_pct (changed metadata)
~ energy_mix.biofuels_twh (changed metadata)
~ energy_mix.coal_annual_change_pct (changed metadata)
~ energy_mix.coal_annual_change_twh (changed metadata)
~ energy_mix.coal_per_capita_kwh (changed metadata)
~ energy_mix.coal_share_including_biomass_pct (changed metadata)
~ energy_mix.coal_share_pct (changed metadata)
~ energy_mix.coal_twh (changed metadata)
~ energy_mix.fossil_fuels_annual_change_pct (changed metadata)
~ energy_mix.fossil_fuels_annual_change_twh (changed metadata)
~ energy_mix.fossil_fuels_per_capita_kwh (changed metadata)
~ energy_mix.fossil_fuels_share_pct (changed metadata)
~ energy_mix.fossil_fuels_twh (changed metadata)
~ energy_mix.gas_annual_change_pct (changed metadata)
~ energy_mix.gas_annual_change_twh (changed metadata)
~ energy_mix.gas_per_capita_kwh (changed metadata)
~ energy_mix.gas_share_including_biomass_pct (changed metadata)
~ energy_mix.gas_share_pct (changed metadata)
~ energy_mix.gas_twh (changed metadata)
~ energy_mix.hydro_annual_change_pct (changed metadata)
~ energy_mix.hydro_annual_change_twh (changed metadata)
~ energy_mix.hydro_per_capita_kwh (changed metadata)
~ energy_mix.hydro_share_including_biomass_pct (changed metadata)
~ energy_mix.hydro_share_pct (changed metadata)
~ energy_mix.hydro_twh (changed metadata)
~ energy_mix.low_carbon_energy_annual_change_pct (changed metadata)
~ energy_mix.low_carbon_energy_annual_change_twh (changed metadata)
~ energy_mix.low_carbon_energy_per_capita_kwh (changed metadata)
~ energy_mix.low_carbon_energy_share_pct (changed metadata)
~ energy_mix.low_carbon_energy_twh (changed metadata)
~ energy_mix.nuclear_annual_change_pct (changed metadata)
~ energy_mix.nuclear_annual_change_twh (changed metadata)
~ energy_mix.nuclear_per_capita_kwh (changed metadata)
~ energy_mix.nuclear_share_including_biomass_pct (changed metadata)
~ energy_mix.nuclear_share_pct (changed metadata)
~ energy_mix.nuclear_twh (changed metadata)
~ energy_mix.oil_annual_change_pct (changed metadata)
~ energy_mix.oil_annual_change_twh (changed metadata)
~ energy_mix.oil_per_capita_kwh (changed metadata)
~ energy_mix.oil_share_including_biomass_pct (changed metadata)
~ energy_mix.oil_share_pct (changed metadata)
~ energy_mix.oil_twh (changed metadata)
~ energy_mix.other_renewables_annual_change_pct (changed metadata)
~ energy_mix.other_renewables_annual_change_twh (changed metadata)
~ energy_mix.other_renewables_per_capita_kwh (changed metadata)
~ energy_mix.other_renewables_share_including_biomass_pct (changed metadata)
~ energy_mix.other_renewables_share_pct (changed metadata)
~ energy_mix.other_renewables_twh (changed metadata)
~ energy_mix.renewables_annual_change_pct (changed metadata)
~ energy_mix.renewables_annual_change_twh (changed metadata)
~ energy_mix.renewables_per_capita_kwh (changed metadata)
~ energy_mix.renewables_share_pct (changed metadata)
~ energy_mix.renewables_twh (changed metadata)
~ energy_mix.solar_and_wind_annual_change_pct (changed metadata)
~ energy_mix.solar_and_wind_annual_change_twh (changed metadata)
~ energy_mix.solar_and_wind_per_capita_kwh (changed metadata)
~ energy_mix.solar_and_wind_share_pct (changed metadata)
~ energy_mix.solar_and_wind_twh (changed metadata)
~ energy_mix.solar_annual_change_pct (changed metadata)
~ energy_mix.solar_annual_change_twh (changed metadata)
~ energy_mix.solar_per_capita_kwh (changed metadata)
~ energy_mix.solar_share_including_biomass_pct (changed metadata)
~ energy_mix.solar_share_pct (changed metadata)
~ energy_mix.solar_twh (changed metadata)
~ energy_mix.total_energy_supply_annual_change_pct (changed metadata)
~ energy_mix.total_energy_supply_annual_change_twh (changed metadata)
~ energy_mix.total_energy_supply_per_capita_kwh (changed metadata)
~ energy_mix.total_energy_supply_per_gdp_kwh_per_dollar (changed metadata)
~ energy_mix.total_energy_supply_twh (changed metadata)
~ energy_mix.traditional_biomass_share_including_biomass_pct (changed metadata)
~ energy_mix.traditional_biomass_twh (changed metadata)
~ energy_mix.wind_annual_change_pct (changed metadata)
~ energy_mix.wind_annual_change_twh (changed metadata)
~ energy_mix.wind_per_capita_kwh (changed metadata)
~ energy_mix.wind_share_including_biomass_pct (changed metadata)
~ energy_mix.wind_share_pct (changed metadata)
~ energy_mix.wind_twh (changed metadata)= 1 compared dataset(s) turned out identical; details in the full report Automatically updated datasets matching excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included. Run locally with Edited: 2026-08-14 18:13:11 UTC |
… applies to Every indicator inherited both notes from the shared definitions block, so by-source indicators claimed to be extended with EIA data (which only extends the total), and the shares of primary energy including traditional biomass claimed to exclude it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ot processing Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Smil's years are decadal before 1965, and only consecutive years produce an annual change, so none of the 28 annual-change indicators has a pre-1965 value. The annual change of the total keeps the EIA note, which does apply to it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Maddison's World GDP starts in 1820, so that series does not go back to 1800 like the rest. Also state the year traditional biomass starts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Charts cite an indicator's origins, so the annual-change indicators credited Smil for data they cannot contain (its World series is decadal before 1965, and only consecutive years produce an annual change), and the biomass-inclusive shares credited the EIA although they are World-only, where the total always comes from the Statistical Review or Smil. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… it measures Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 77a40f1761
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refines metadata for the garden/energy/2026-06-30/energy_mix dataset so that description_processing text and indicator origins are applied only where they are accurate, avoiding incorrect notes/citations on derived indicators.
Changes:
- Adds a small helper in the garden step to remove producer origins from derived columns when that producer contributes no values (e.g., Smil on annual-change indicators; EIA on biomass-inclusive World-only shares).
- Reworks
energy_mix.meta.ymlto split processing notes into targeted definitions and apply them per-indicator (including clearing the common note where none applies).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
etl/steps/data/garden/energy/2026-06-30/energy_mix.py |
Drops misleading origins from specific derived columns so chart citations better reflect actual contributing producers. |
etl/steps/data/garden/energy/2026-06-30/energy_mix.meta.yml |
Scopes processing descriptions by indicator type/source to avoid incorrect inherited notes. |
Suppressed comments (1)
etl/steps/data/garden/energy/2026-06-30/energy_mix.meta.yml:762
- Duplicate
description_processingkeys undertotal_energy_supply_annual_change_twhmake the YAML invalid/ambiguous; the later key will override the earlier one, and this may also break metadata parsing. Keep a singledescription_processingentry (per PR description, this annual-change indicator should only carry the EIA note, not the Smil long-run note).
description_processing: |-
{definitions.description_processing_eia}
{definitions.description_processing_smil_world}
description_processing: |-
{definitions.description_processing_eia}
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The two total annual-change indicators each had two description_processing blocks. YAML resolved the last one, so the output was right by luck, but the file was ambiguous. Also correct the docstring that said Smil's World series is decadal only before 1900: it is decadal from 1800 to 1960. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both
description_processingnotes ofgarden/energy/2026-06-30/energy_mixlived in the shareddefinitions.commonblock, so all 81 indicators inherited both, and each note was wrong for most of them:Each note is now applied only where it holds:
total_energy_supply_twh,_per_capita_kwhtotal_energy_supply_per_gdp_kwh_per_dollar*_share_including_biomass_pcttraditional_biomass_twhand its shareThe wording of the Smil note is also plainer, and the EIA note no longer says the extension covers "countries not covered by the Statistical Review": it also fills country-years where the Statistical Review reports individual fuels but no total.
Origins
Charts cite an indicator's origins, so the same problem showed up there: derived columns inherited producers that contribute nothing to them. Annual change in primary energy by source credited Smil while starting in 1965. Two sets of origins are now dropped in the garden step:
No data values change. Every remaining claim was re-checked against the built data (start years, which indicators carry EIA-derived values, the World-only and held-constant claims, and that the biomass-inclusive shares sum to 100%): 81 indicators, 0 failures.
Left open
The 13
*_share_pctindicators keep the EIA origin, which is honest today: their denominator is the EIA-extended total, so South America's shares divide an EI numerator by an EIA denominator. That is #6681, and the origin should be dropped when it is fixed.