Skip to content

🐛 Energy: rebuild the region aggregates and drop the residual "Other *" regions - #6681

Draft
pabloarosado wants to merge 49 commits into
masterfrom
data-energymix-southamerica-producers
Draft

🐛 Energy: rebuild the region aggregates and drop the residual "Other *" regions#6681
pabloarosado wants to merge 49 commits into
masterfrom
data-energymix-southamerica-producers

Conversation

@pabloarosado

@pabloarosado pabloarosado commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Written by Claude Opus 5 — @pabloarosado at the wheel.

This started as a narrow fix (South America's energy-mix shares summed to ~31% instead of 100%) but the cause turned out to be structural, so the PR now rebuilds how we make region aggregates across the energy datasets.

What was wrong

The Energy Institute's Statistical Review publishes its own regions, and they rest on residual buckets ("Other Africa", "Other South and Central America", "Rest of World") that hold energy it cannot attribute to any country. Two consequences:

  • The residual buckets do not map onto our regions. "Other South and Central America" straddles North and South America, so we deleted South America's total instead of publishing a wrong one, and it was then silently refilled from an EIA row. That left EI numerators over an EIA denominator with far wider country coverage, which is where the ~31% came from.
  • Where we did build our own aggregates from EI countries, they were understated, because the residuals were left out: EI's gas production for low-income countries is a third of the sum of those countries, and its oil consumption for lower-middle-income countries is 30% short.
  • The buckets have no fixed composition. "Other Europe" can cover a different set of countries for each indicator, so the same entity name means different things on different charts. They were never filtered out of the Statistical Review's own grapher dataset, so they were selectable on every by-region chart.

The approach

Combine all producers at the country level first, then build the region aggregates from that combined set, in the step that publishes them. This replaces the previous mix of EI's regions, EIA's regions, and our own partial aggregates.

Changes by step

garden/energy_institute/2026-06-30/statistical_review_of_world_energy

  • Residual and undefined provider regions (22 entities) are dropped from the output. They are still used internally as inputs to the region aggregates, and the meadow table keeps all of them. The defined (EI) regions and self-explanatory organizations (OECD, OPEC) stay.
  • Missing energy supply by source is filled with zeros where the reported total confirms there is no room for anything else, and missing hydro, solar and wind generation is filled from their energy supply, which the physical energy content method makes the same quantity. Both are guarded by assertions that fail if the producer's totals stop reconciling.
  • Bangladesh's placeholder zeros for 1965-1970 are removed via the corrections file. The producer reports a zero total and zero hydro for years before the country existed, but the Karnaphuli dam had been generating since 1962, and the producer itself starts reporting it in 1971. This was the only entity in the dataset with an exactly-zero total.

garden/eia/2026-05-05/international_energy

  • Adds biofuels consumption, which the producer reports but we were not reading.
  • Derives other renewables from the electricity it reports for geothermal, biomass and waste, and tide and wave. EIA gives electricity generation for these but not the energy supply behind it, so each is converted with the efficiency the IEA's physical energy content method assumes (10% geothermal, 33% for the combustible sources), which is the basis the rest of our energy data uses. Validated against EI on geothermal-dominated countries, where it lands within 3-5%.
  • Fills gaps in the biofuels series that sit between reported zeros.

garden/energy/2026-06-30/energy_mix

  • All nine sources now come from EIA where EI does not report a country, not just four. 143 countries gain a full source breakdown.
  • EIA's own total is no longer used as a denominator. It is on a different basis: it nets out electricity trade and counts renewable electricity as generated rather than as heat input, so Paraguay's hydro generation alone exceeds it, and Kenya's geothermal heat input is four times its whole renewables figure. The total is now the sum of the nine sources, which is how EI's own total behaves.
  • Region aggregates are built from the combined countries, over 1980-2024, the years both producers cover. Outside that window EI itemizes too few countries to stand for a region (4 of Africa's 57), and starting at 1980 with EI alone would show a rise that is ours rather than the world's.
  • Duplicated territories (East and West Germany against Germany, Czechoslovakia against Czechia and Slovakia) are excluded from the aggregates but keep their own series. Left in, they put the continents 10-14% above the World for 1980-1990.
  • A row either publishes its whole mix or none of it, so a partial mix can no longer read as a complete one in a stacked chart.
  • Sanity checks now cover the full year range, not just EIA's: the shares sum to 100%, the total equals the sum of the nine sources, every region is contiguous over the expected years, and the continents reconcile with the World.

garden/energy/2026-06-30/fossil_fuels

  • Same treatment: one add_aggregates call over the combined countries, replacing EI's understated regions, with the energy columns bounded to the producers' shared years (1981-2024 for production, since EI's coal production starts in 1981; 1980-2024 for consumption).
  • Europe's reserves aggregate requires Russia, which holds about nine tenths of them. EIA reports the USSR's until 1991 and Russia's only from 1997, so the plain sum showed Europe losing 89% of its gas reserves in 1992 and recovering them in 1997.

garden/energy/2026-06-30/electricity_mix

  • Drops its local handling of the Other * (EI) entities, which the Statistical Review no longer emits. shared.py is deleted along with the per-consumer filters it held.

Affected surfaces

The four energy grapher datasets and their MDims: 27 charts on energy_mix, 29 on the Statistical Review, 14 on electricity_mix, 1 on fossil_fuels (which is consumed mostly through its MDim). Region series change on all of them; country series change only where a country gains sources it did not have.

Left open

  • Nothing has been reviewed on a chart on staging yet.
  • The Statistical Review still computes its own region aggregates. Now that both consumers build their own, these are only needed for the charts that read them directly, and that inventory has not been done.
  • The expected first and last years of the aggregates are hardcoded (2024 for the last), so the next EIA release will trip the assertion, which is deliberate but will need updating.
  • 2025 exists for EI countries but not for the regions, because EIA stops at 2024. The latest year therefore shows ~90 entities against ~230 in 2024.
  • Sixteen of EIA's physical columns still start in different years by region, and gas production in 1998 puts the continents 2.5% above the World.

@pabloarosado pabloarosado changed the title 📊 Energy mix: South America shares mix two producers 🐛 Energy mix: South America shares mix two producers Aug 14, 2026
@owidbot

owidbot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Quick links (staging server):

Site Dev Site Preview Admin Wizard Docs Docs Preview

Login: ssh owid@staging-site-data-energymix-southamerica

chart-diff: ❌
  • 6/8 reviewed charts
  • Modified: 6/8
  • New: 0/0
  • Rejected: 0
  • Data changes: 0
  • Metadata changes: 0
data-diff: ❌ 12 changed · 11 identical — full report 
~ garden/energy/2026-06-30/energy_mix
    ~ energy_mix.country dim (new data, removed data)
        + New values: 248 / 12,710 (1.95%)
            2025: Antigua and Barbuda
            2025: Armenia
            2025: Aruba
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            1965: High-income countries
            1966: High-income countries
            1967: High-income countries
            (+58 more)
    ~ energy_mix.year dim (new data, removed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda: 2025
            Armenia: 2025
            Aruba: 2025
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries: 1965
            High-income countries: 1966
            High-income countries: 1967
            (+58 more)
    ~ energy_mix.biofuels_annual_change_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: NaN
            High-income countries 1967: NaN
            (+58 more)
        ~ Changed values: 435 / 12,710 (3.42%)
            Lower-middle-income countries 2012: -7.67763090133667 → 1.63904869556427
            Lower-middle-income countries 2014: -12.717665672302246 → -4.342413902282715
            Lower-middle-income countries 2020: -6.2597222328186035 → -3.5131630897521973
            (+432 more)
    ~ energy_mix.biofuels_annual_change_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 0.0
            High-income countries 1967: 0.0
            (+58 more)
        ~ Changed values: 8,511 / 12,710 (66.96%)
            High-income countries 1990: 16.73274040222168 → -4.23570442199707
            High-income countries 1983: 0.0 → 4.57493257522583
            High-income countries 1982: 0.0 → 3.6599466800689697
            (+8,508 more)
    ~ energy_mix.biofuels_per_capita_kwh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 0.0
            High-income countries 1966: 0.0
            High-income countries 1967: 0.0
            (+58 more)
        ~ Changed values: 8,504 / 12,710 (66.91%)
            High-income countries 1989: 0.0 → 17.77144432067871
            High-income countries 1988: 0.0 → 17.621654510498047
            High-income countries 1987: 0.0 → 17.37317657470703
            (+8,501 more)
    ~ energy_mix.biofuels_share_including_biomass_pct (changed metadata, new data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: NaN
            High-income countries 1967: NaN
            (+58 more)
    ~ energy_mix.biofuels_share_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 0.0
            High-income countries 1966: 0.0
            High-income countries 1967: 0.0
            (+58 more)
        ~ Changed values: 8,752 / 12,710 (68.86%)
            High-income countries 1987: 0.0 → 0.032659925520420074
            High-income countries 1988: 0.0 → 0.03228830546140671
            High-income countries 1989: 0.0 → 0.032200854271650314
            (+8,749 more)
    ~ energy_mix.biofuels_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 0.0
            High-income countries 1966: 0.0
            High-income countries 1967: 0.0
            (+58 more)
        ~ Changed values: 8,676 / 12,710 (68.26%)
            High-income countries 1989: 0.0 → 20.96844482421875
            High-income countries 1988: 0.0 → 20.64360237121582
            High-income countries 1987: 0.0 → 20.205955505371094
            (+8,673 more)
    ~ energy_mix.coal_annual_change_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: -0.4555027186870575
            High-income countries 1967: -1.7226054668426514
            (+58 more)
        ~ Changed values: 2,207 / 12,710 (17.36%)
            Lower-middle-income countries 2006: 4.2140583992004395 → -1.4972186088562012
            Upper-middle-income countries 1992: 3.3829314708709717 → -1.456376552581787
            Lower-middle-income countries 1992: 5.3374924659729 → 16.52724838256836
            (+2,204 more)
    ~ energy_mix.coal_annual_change_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: -46.841796875
            High-income countries 1967: -176.337890625
            (+58 more)
        ~ Changed values: 6,292 / 12,710 (49.50%)
            Upper-middle-income countries 1992: 308.5029296875 → -139.85479736328125
            Lower-middle-income countries 2006: 108.1845703125 → -41.34584045410156
            Lower-middle-income countries 1992: 76.0665283203125 → 242.64892578125
            (+6,289 more)
    ~ energy_mix.coal_per_capita_kwh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 10630.525390625
            High-income countries 1966: 10470.248046875
            High-income countries 1967: 10193.291015625
            (+58 more)
        ~ Changed values: 6,358 / 12,710 (50.02%)
            Lower-middle-income countries 1992: 902.912353515625 → 1028.9910888671875
            Lower-middle-income countries 1993: 913.6849365234375 → 1033.227294921875
            Lower-middle-income countries 1995: 955.5599975585938 → 1072.8614501953125
            (+6,355 more)
    ~ energy_mix.coal_share_including_biomass_pct (changed metadata, new data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: NaN
            High-income countries 1967: NaN
            (+58 more)
    ~ energy_mix.coal_share_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 34.656105041503906
            High-income countries 1966: 32.84675216674805
            High-income countries 1967: 31.01588249206543
            (+58 more)
        ~ Changed values: 6,422 / 12,710 (50.53%)
            Lower-middle-income countries 1981: 35.88575744628906 → 30.373855590820312
            Lower-middle-income countries 1980: 34.80689239501953 → 29.546924591064453
            Lower-middle-income countries 1982: 34.580841064453125 → 29.46200180053711
            (+6,419 more)
    ~ energy_mix.coal_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 10283.5380859375
            High-income countries 1966: 10236.6962890625
            High-income countries 1967: 10060.3583984375
            (+58 more)
        ~ Changed values: 6,437 / 12,710 (50.65%)
            Lower-middle-income countries 1992: 1501.2025146484375 → 1710.823974609375
            Lower-middle-income countries 1993: 1553.705078125 → 1756.9847412109375
            Lower-middle-income countries 1995: 1698.5467529296875 → 1907.0548095703125
            (+6,434 more)
    ~ energy_mix.fossil_fuels_annual_change_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 4.929871559143066
            High-income countries 1967: 4.079148292541504
            (+58 more)
        ~ Changed values: 6,732 / 12,710 (52.97%)
            Upper-middle-income countries 1992: 1.5359188318252563 → -0.8492581248283386
            Lower-middle-income countries 2006: 4.678356170654297 → 0.899867594242096
            Lower-middle-income countries 1992: 4.505779266357422 → 11.374197959899902
            (+6,729 more)
    ~ energy_mix.fossil_fuels_annual_change_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 1420.2802734375
            High-income countries 1967: 1233.125
            (+58 more)
        ~ Changed values: 6,752 / 12,710 (53.12%)
            Upper-middle-income countries 1992: 302.78173828125 → -177.63706970214844
            Lower-middle-income countries 2006: 326.009521484375 → 74.04163360595703
            Lower-middle-income countries 1992: 177.88238525390625 → 514.0588989257812
            (+6,749 more)
    ~ energy_mix.fossil_fuels_per_capita_kwh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 29781.779296875
            High-income countries 1966: 30919.662109375
            High-income countries 1967: 31878.82421875
            (+58 more)
        ~ Changed values: 6,818 / 12,710 (53.64%)
            Lower-middle-income countries 1992: 2481.4736328125 → 3027.491943359375
            Lower-middle-income countries 1980: 1507.852783203125 → 1838.3370361328125
            Lower-middle-income countries 1981: 1607.1822509765625 → 1951.73486328125
            (+6,815 more)
    ~ energy_mix.fossil_fuels_share_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 97.09026336669922
            High-income countries 1966: 96.99966430664062
            High-income countries 1967: 97.00006866455078
            (+58 more)
        ~ Changed values: 6,882 / 12,710 (54.15%)
            Lower-middle-income countries 2021: 94.43414306640625 → 93.47894287109375
            Lower-middle-income countries 2022: 93.96951293945312 → 93.03060913085938
            Lower-middle-income countries 2020: 94.29757690429688 → 93.36713409423828
            (+6,879 more)
    ~ energy_mix.fossil_fuels_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 28809.681640625
            High-income countries 1966: 30229.962890625
            High-income countries 1967: 31463.087890625
            (+58 more)
        ~ Changed values: 6,897 / 12,710 (54.26%)
            Lower-middle-income countries 1992: 4125.75439453125 → 5033.57666015625
            Lower-middle-income countries 1980: 1861.044677734375 → 2268.93994140625
            Lower-middle-income countries 1981: 2036.5147705078125 → 2473.109130859375
            (+6,894 more)
    ~ energy_mix.gas_annual_change_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 8.375866889953613
            High-income countries 1967: 6.9199604988098145
            (+58 more)
        ~ Changed values: 1,479 / 12,710 (11.64%)
            High-income countries 2001: 0.002550177276134491 → 0.053438108414411545
            Lower-middle-income countries 2008: 0.15831832587718964 → 1.462382435798645
            Lower-middle-income countries 2009: 5.974233627319336 → 0.6496405005455017
            (+1,476 more)
    ~ energy_mix.gas_annual_change_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 403.265625
            High-income countries 1967: 361.0751953125
            (+58 more)
        ~ Changed values: 6,432 / 12,710 (50.61%)
            High-income countries 2001: 0.4609375 → 9.717265129089355
            Lower-middle-income countries 2008: 3.318359375 → 34.63725662231445
            Lower-middle-income countries 2009: 125.41845703125 → 15.612077713012695
            (+6,429 more)
    ~ energy_mix.gas_per_capita_kwh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 4977.068359375
            High-income countries 1966: 5336.92578125
            High-income countries 1967: 5652.671875
            (+58 more)
        ~ Changed values: 6,498 / 12,710 (51.13%)
            Lower-middle-income countries 2024: 936.2948608398438 → 1114.96044921875
            Lower-middle-income countries 2022: 921.1389770507812 → 1096.0126953125
            Lower-middle-income countries 2023: 918.3475952148438 → 1087.68994140625
            (+6,495 more)
    ~ energy_mix.gas_share_including_biomass_pct (changed metadata, new data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: NaN
            High-income countries 1967: NaN
            (+58 more)
    ~ energy_mix.gas_share_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 16.225521087646484
            High-income countries 1966: 16.74274444580078
            High-income countries 1967: 17.199804306030273
            (+58 more)
        ~ Changed values: 6,623 / 12,710 (52.11%)
            Lower-middle-income countries 1980: 13.755606651306152 → 12.332404136657715
            Lower-middle-income countries 1982: 14.430987358093262 → 13.144712448120117
            South America 1990: 18.315061569213867 → 19.690338134765625
            (+6,620 more)
    ~ energy_mix.gas_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 4814.61376953125
            High-income countries 1966: 5217.87939453125
            High-income countries 1967: 5578.95458984375
            (+58 more)
        ~ Changed values: 6,577 / 12,710 (51.75%)
            Lower-middle-income countries 2024: 2722.32421875 → 3241.803466796875
            Lower-middle-income countries 2022: 2609.29931640625 → 3104.662109375
            Lower-middle-income countries 2023: 2635.577392578125 → 3121.5751953125
            (+6,574 more)
    ~ energy_mix.hydro_annual_change_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 5.563568115234375
            High-income countries 1967: 2.556910991668701
            (+58 more)
        ~ Changed values: 3,703 / 12,710 (29.13%)
            USSR 1972: 54.36602020263672 → -2.5377769470214844
            Europe 1973: -11.92697811126709 → 1.142989993095398
            Lower-middle-income countries 2010: -1.477232813835144 → 2.638908624649048
            (+3,700 more)
    ~ energy_mix.hydro_annual_change_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 39.428955078125
            High-income countries 1967: 19.12896728515625
            (+58 more)
        ~ Changed values: 6,006 / 12,710 (47.25%)
            Europe 1973: -66.22845458984375 → 5.52667236328125
            USSR 1972: 68.55500793457031 → -3.2001113891601562
            Lower-middle-income countries 2021: -4.308868408203125 → 10.60894775390625
            (+6,003 more)
    ~ energy_mix.hydro_per_capita_kwh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 732.6119995117188
            High-income countries 1966: 765.1966552734375
            High-income countries 1967: 777.3950805664062
            (+58 more)
        ~ Changed values: 6,069 / 12,710 (47.75%)
            Iran 2021: 0.0 → 156.31527709960938
            Lower-middle-income countries 2024: 100.9272232055664 → 168.10426330566406
            Lower-middle-income countries 2022: 105.85189819335938 → 172.8815155029297
            (+6,066 more)
    ~ energy_mix.hydro_share_including_biomass_pct (changed metadata, new data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: NaN
            High-income countries 1967: NaN
            (+58 more)
    ~ energy_mix.hydro_share_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 2.3883559703826904
            High-income countries 1966: 2.4005377292633057
            High-income countries 1967: 2.3654377460479736
            (+58 more)
        ~ Changed values: 6,127 / 12,710 (48.21%)
            Iran 2021: 0.0 → 0.4125072658061981
            USSR 1972: 2.05464243888855 → 1.2972415685653687
            Lower-middle-income countries 2024: 2.005993366241455 → 2.874573230743408
            (+6,124 more)
    ~ energy_mix.hydro_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 708.6990356445312
            High-income countries 1966: 748.1279907226562
            High-income countries 1967: 767.2569580078125
            (+58 more)
        ~ Changed values: 6,148 / 12,710 (48.37%)
            Iran 2021: 0.0 → 13.826944351196289
            Lower-middle-income countries 2024: 293.4509582519531 → 488.7715759277344
            Lower-middle-income countries 2022: 299.84539794921875 → 489.71942138671875
            (+6,145 more)
    ~ energy_mix.low_carbon_energy_annual_change_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 8.297872543334961
            High-income countries 1967: 4.065077781677246
            (+58 more)
        ~ Changed values: 4,543 / 12,710 (35.74%)
            USSR 1973: -26.171710968017578 → 8.521702766418457
            Europe 1973: -3.737684965133667 → 5.385578632354736
            Lower-middle-income countries 1997: 2.069866180419922 → -0.15199901163578033
            (+4,540 more)
    ~ energy_mix.low_carbon_energy_annual_change_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 71.6445541381836
            High-income countries 1967: 38.010643005371094
            (+58 more)
        ~ Changed values: 6,099 / 12,710 (47.99%)
            Europe 1973: -30.980369567871094 → 40.774757385253906
            USSR 1973: -58.74279022216797 → 13.0123291015625
            North America 1992: -19.271459579467773 → 5.451318264007568
            (+6,096 more)
    ~ energy_mix.low_carbon_energy_per_capita_kwh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 892.5418701171875
            High-income countries 1966: 956.3865966796875
            High-income countries 1967: 985.9214477539062
            (+58 more)
        ~ Changed values: 6,164 / 12,710 (48.50%)
            Iran 2021: 141.70846557617188 → 298.02374267578125
            Lower-middle-income countries 1992: 106.24597930908203 → 158.9773406982422
            Lower-middle-income countries 1993: 105.62039184570312 → 156.27743530273438
            (+6,161 more)
    ~ energy_mix.low_carbon_energy_share_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 2.909736156463623
            High-income countries 1966: 3.0003297328948975
            High-income countries 1967: 2.99993634223938
            (+58 more)
        ~ Changed values: 6,222 / 12,710 (48.95%)
            Iran 2021: 0.3739607334136963 → 0.786467969417572
            USSR 1972: 2.3691651821136475 → 1.6117644309997559
            Lower-middle-income countries 1996: 3.8822898864746094 → 4.7270708084106445
            (+6,219 more)
    ~ energy_mix.low_carbon_energy_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 863.40869140625
            High-income countries 1966: 935.05322265625
            High-income countries 1967: 973.0639038085938
            (+58 more)
        ~ Changed values: 6,243 / 12,710 (49.12%)
            Iran 2021: 12.534892082214355 → 26.36183738708496
            Lower-middle-income countries 1992: 176.64697265625 → 264.3193359375
            Lower-middle-income countries 1993: 179.60560607910156 → 265.74700927734375
            (+6,240 more)
    ~ energy_mix.nuclear_annual_change_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 36.649417877197266
            High-income countries 1967: 19.64522361755371
            (+58 more)
        ~ Changed values: 120 / 12,710 (0.94%)
            Upper-middle-income countries 1991: 1.206352949142456 → 0.3821001946926117
            Upper-middle-income countries 1992: -2.0285472869873047 → -5.905368804931641
            Upper-middle-income countries 1999: -1.6871532201766968 → -1.0647776126861572
            (+117 more)
    ~ energy_mix.nuclear_annual_change_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 26.29779815673828
            High-income countries 1967: 19.2626953125
            (+58 more)
        ~ Changed values: 665 / 12,710 (5.23%)
            Upper-middle-income countries 1992: -6.039794921875 → -18.30706787109375
            Upper-middle-income countries 1991: 3.548980712890625 → 1.1800291538238525
            Upper-middle-income countries 1999: -6.2890625 → -4.0210418701171875
            (+662 more)
    ~ energy_mix.nuclear_per_capita_kwh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 74.17617797851562
            High-income countries 1966: 100.28990936279297
            High-income countries 1967: 118.86566162109375
            (+58 more)
        ~ Changed values: 642 / 12,710 (5.05%)
            Upper-middle-income countries 1986: 82.6965560913086 → 89.09880828857422
            Upper-middle-income countries 1987: 90.57332611083984 → 97.1648941040039
            Upper-middle-income countries 1985: 99.51836395263672 → 106.04036712646484
            (+639 more)
    ~ energy_mix.nuclear_share_including_biomass_pct (changed metadata, new data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: NaN
            High-income countries 1967: NaN
            (+58 more)
    ~ energy_mix.nuclear_share_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 0.24181848764419556
            High-income countries 1966: 0.3146246075630188
            High-income countries 1967: 0.36168137192726135
            (+58 more)
        ~ Changed values: 440 / 12,710 (3.46%)
            Lower-middle-income countries 1992: 0.4899403154850006 → 0.3978786766529083
            Lower-middle-income countries 1980: 0.3909268379211426 → 0.3190876543521881
            Lower-middle-income countries 1981: 0.3730098605155945 → 0.30558353662490845
            (+437 more)
    ~ energy_mix.nuclear_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 71.75502014160156
            High-income countries 1966: 98.05281829833984
            High-income countries 1967: 117.31551361083984
            (+58 more)
        ~ Changed values: 662 / 12,710 (5.21%)
            Upper-middle-income countries 1986: 173.8396453857422 → 187.29806518554688
            Upper-middle-income countries 1987: 194.0313720703125 → 208.15220642089844
            Upper-middle-income countries 1985: 205.36001586914062 → 218.8184356689453
            (+659 more)
    ~ energy_mix.oil_annual_change_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 7.758845329284668
            High-income countries 1967: 7.095500946044922
            (+58 more)
        ~ Changed values: 5,972 / 12,710 (46.99%)
            Lower-middle-income countries 2018: -0.7288522124290466 → 0.2856288254261017
            Lower-middle-income countries 1991: -0.40510833263397217 → 0.17864444851875305
            Upper-middle-income countries 1992: 0.8456049561500549 → 0.04114567115902901
            (+5,969 more)
    ~ energy_mix.oil_annual_change_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 1063.8564453125
            High-income countries 1967: 1048.3876953125
            (+58 more)
        ~ Changed values: 5,992 / 12,710 (47.14%)
            Lower-middle-income countries 2018: -29.142822265625 → 14.834186553955078
            Lower-middle-income countries 1991: -6.321044921875 → 3.5678441524505615
            Upper-middle-income countries 1992: 61.6455078125 → 3.2402126789093018
            (+5,989 more)
    ~ energy_mix.oil_per_capita_kwh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 14174.1845703125
            High-income countries 1966: 15112.4892578125
            High-income countries 1967: 16032.861328125
            (+58 more)
        ~ Changed values: 6,059 / 12,710 (47.67%)
            Lower-middle-income countries 1980: 736.8550415039062 → 1023.7504272460938
            Lower-middle-income countries 1981: 773.642822265625 → 1065.979736328125
            Lower-middle-income countries 1982: 790.2518920898438 → 1078.7796630859375
            (+6,056 more)
    ~ energy_mix.oil_share_including_biomass_pct (changed metadata, new data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: NaN
            High-income countries 1967: NaN
            (+58 more)
    ~ energy_mix.oil_share_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 46.20863342285156
            High-income countries 1966: 47.4101676940918
            High-income countries 1967: 48.78437805175781
            (+58 more)
        ~ Changed values: 6,123 / 12,710 (48.17%)
            Lower-middle-income countries 2021: 28.88503646850586 → 32.98238754272461
            Lower-middle-income countries 1980: 46.41196823120117 → 52.632808685302734
            Lower-middle-income countries 2020: 30.03055763244629 → 34.030513763427734
            (+6,120 more)
    ~ energy_mix.oil_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 13711.5302734375
            High-income countries 1966: 14775.38671875
            High-income countries 1967: 15823.7744140625
            (+58 more)
        ~ Changed values: 6,138 / 12,710 (48.29%)
            Lower-middle-income countries 1980: 909.4522705078125 → 1263.548583984375
            Lower-middle-income countries 1981: 980.3088989257812 → 1350.7388916015625
            Lower-middle-income countries 1982: 1027.6112060546875 → 1402.801025390625
            (+6,135 more)
    ~ energy_mix.other_renewables_annual_change_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 7.133779048919678
            High-income countries 1967: -0.42872634530067444
            (+58 more)
        ~ Changed values: 1,830 / 12,710 (14.40%)
            Lower-middle-income countries 1993: 111.11107635498047 → -2.2004799842834473
            Lower-middle-income countries 1994: 50.87723159790039 → -5.607398986816406
            Lower-middle-income countries 1997: 29.824565887451172 → -2.689333200454712
            (+1,827 more)
    ~ energy_mix.other_renewables_annual_change_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 5.9178009033203125
            High-income countries 1967: -0.38101959228515625
            (+58 more)
        ~ Changed values: 6,058 / 12,710 (47.66%)
            Lower-middle-income countries 2015: -8.286609649658203 → 5.360385894775391
            Lower-middle-income countries 2023: -3.4091567993164062 → 1.4275403022766113
            Lower-middle-income countries 2016: -2.3048667907714844 → 1.716187834739685
            (+6,055 more)
    ~ energy_mix.other_renewables_per_capita_kwh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 85.75370025634766
            High-income countries 1966: 90.90007781982422
            High-income countries 1967: 89.66070556640625
            (+58 more)
        ~ Changed values: 6,126 / 12,710 (48.20%)
            Lower-middle-income countries 1989: 0.0 → 5.1469268798828125
            Lower-middle-income countries 1985: 0.0 → 4.855642795562744
            Lower-middle-income countries 1987: 0.0 → 4.270795822143555
            (+6,123 more)
    ~ energy_mix.other_renewables_share_including_biomass_pct (changed metadata, new data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: NaN
            High-income countries 1967: NaN
            (+58 more)
    ~ energy_mix.other_renewables_share_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 0.279561847448349
            High-income countries 1966: 0.28516730666160583
            High-income countries 1967: 0.2728172838687897
            (+58 more)
        ~ Changed values: 6,184 / 12,710 (48.65%)
            Lower-middle-income countries 1984: 0.0 → 0.1986865997314453
            Lower-middle-income countries 1985: 0.0 → 0.19269190728664398
            Lower-middle-income countries 1989: 0.0 → 0.18185438215732574
            (+6,181 more)
    ~ energy_mix.other_renewables_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 82.95464324951172
            High-income countries 1966: 88.87244415283203
            High-income countries 1967: 88.49142456054688
            (+58 more)
        ~ Changed values: 6,205 / 12,710 (48.82%)
            Lower-middle-income countries 1989: 0.0 → 7.97318172454834
            Lower-middle-income countries 1985: 0.0 → 6.815000057220459
            Lower-middle-income countries 1987: 0.0 → 6.301817893981934
            (+6,202 more)
    ~ energy_mix.renewables_annual_change_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 5.728105068206787
            High-income countries 1967: 2.2398970127105713
            (+58 more)
        ~ Changed values: 4,543 / 12,710 (35.74%)
            USSR 1972: 54.36602020263672 → -2.5377769470214844
            Europe 1973: -10.992682456970215 → 1.2254008054733276
            North America 1976: -0.596811056137085 → 2.335195541381836
            (+4,540 more)
    ~ energy_mix.renewables_annual_change_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 45.34675598144531
            High-income countries 1967: 18.747947692871094
            (+58 more)
        ~ Changed values: 6,099 / 12,710 (47.99%)
            Europe 1973: -65.34961700439453 → 6.405509948730469
            USSR 1972: 68.55500793457031 → -3.2001113891601562
            North America 1976: -3.2100419998168945 → 14.759379386901855
            (+6,096 more)
    ~ energy_mix.renewables_per_capita_kwh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 818.3656616210938
            High-income countries 1966: 856.0966796875
            High-income countries 1967: 867.0557861328125
            (+58 more)
        ~ Changed values: 6,164 / 12,710 (48.50%)
            Iran 2021: 20.552549362182617 → 176.86782836914062
            North America 2001: 1293.1580810546875 → 2148.024169921875
            North America 1992: 1462.658447265625 → 2388.666015625
            (+6,161 more)
    ~ energy_mix.renewables_share_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 2.6679177284240723
            High-income countries 1966: 2.6857049465179443
            High-income countries 1967: 2.6382551193237305
            (+58 more)
        ~ Changed values: 6,222 / 12,710 (48.95%)
            Iran 2021: 0.05423702672123909 → 0.4667443037033081
            North America 2001: 2.0392260551452637 → 3.387294054031372
            North America 1992: 2.375332832336426 → 3.8791534900665283
            (+6,219 more)
    ~ energy_mix.renewables_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 791.6536865234375
            High-income countries 1966: 837.0004272460938
            High-income countries 1967: 855.7484130859375
            (+58 more)
        ~ Changed values: 6,243 / 12,710 (49.12%)
            Iran 2021: 1.8179858922958374 → 15.644929885864258
            North America 2001: 635.4996948242188 → 1055.6085205078125
            North America 1992: 643.6588134765625 → 1051.158447265625
            (+6,240 more)
    ~ energy_mix.solar_and_wind_annual_change_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: NaN
            High-income countries 1967: NaN
            (+58 more)
        ~ Changed values: 2,057 / 12,710 (16.18%)
            South America 2006: -98.53485107421875 → 62.69596862792969
            South America 1999: 52.9411735534668 → -1.710547924041748
            South America 2009: -20.822547912597656 → 9.892806053161621
            (+2,054 more)
    ~ energy_mix.solar_and_wind_annual_change_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 0.0
            High-income countries 1967: 0.0
            (+58 more)
        ~ Changed values: 5,789 / 12,710 (45.55%)
            Low-income countries 2021: 0.0 → 1.3042018413543701
            Low-income countries 2022: 0.0 → 1.083827018737793
            Democratic Republic of Congo 2022: 0.0 → 1.0590001344680786
            (+5,786 more)
    ~ energy_mix.solar_and_wind_per_capita_kwh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 0.0
            High-income countries 1966: 0.0
            High-income countries 1967: 0.0
            (+58 more)
        ~ Changed values: 5,849 / 12,710 (46.02%)
            New Caledonia 2024: 0.0 → 1027.7188720703125
            New Caledonia 2022: 0.0 → 936.8578491210938
            New Caledonia 2023: 0.0 → 927.973388671875
            (+5,846 more)
    ~ energy_mix.solar_and_wind_share_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 0.0
            High-income countries 1966: 0.0
            High-income countries 1967: 0.0
            (+58 more)
        ~ Changed values: 5,429 / 12,710 (42.71%)
            Democratic Republic of Congo 2024: 0.0 → 7.701277732849121
            Democratic Republic of Congo 2022: 0.0 → 6.865350723266602
            Democratic Republic of Congo 2023: 0.0 → 6.857144355773926
            (+5,426 more)
    ~ energy_mix.solar_and_wind_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 0.0
            High-income countries 1966: 0.0
            High-income countries 1967: 0.0
            (+58 more)
        ~ Changed values: 6,023 / 12,710 (47.39%)
            Low-income countries 2024: 0.0 → 5.042963027954102
            Low-income countries 2023: 0.0 → 4.505652904510498
            Low-income countries 2022: 0.0 → 4.378726959228516
            (+6,020 more)
    ~ energy_mix.solar_annual_change_pct (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: NaN
            High-income countries 1967: NaN
            (+58 more)
        ~ Changed values: 1,724 / 12,710 (13.56%)
            Lower-middle-income countries 2000: 20.131359100341797 → 87.78130340576172
            Lower-middle-income countries 2005: 7.019922733306885 → 15.215200424194336
            Lower-middle-income countries 2007: 406.4071350097656 → 213.5760955810547
            (+1,721 more)
    ~ energy_mix.solar_annual_change_twh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: NaN
            High-income countries 1966: 0.0
            High-income countries 1967: 0.0
            (+58 more)
        ~ Changed values: 5,874 / 12,710 (46.22%)
            Lower-middle-income countries 2000: 0.0012200833298265934 → 0.005320083349943161
            Lower-middle-income countries 2005: 0.0014880001544952393 → 0.004487999714910984
            Lower-middle-income countries 2002: 0.002273205667734146 → 0.0033732049632817507
            (+5,871 more)
    ~ energy_mix.solar_per_capita_kwh (changed metadata, new data, changed data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)
        - Removed values: 61 / 12,710 (0.48%)
            High-income countries 1965: 0.0
            High-income countries 1966: 0.0
            High-income countries 1967: 0.0
            (+58 more)
        ~ Changed values: 5,942 / 12,710 (46.75%)
            Lower-middle-income countries 2006: 0.006205181125551462 → 0.011807641945779324
            Lower-middle-income countries 2000: 0.0036870709154754877 → 0.005763382650911808
            Lower-middle-income countries 2005: 0.010434744879603386 → 0.015632610768079758
            (+5,939 more)
    ~ energy_mix.solar_share_including_biomass_pct (changed metadata, new data)
        + New values: 248 / 12,710 (1.95%)
            Antigua and Barbuda 2025: NaN
            Armenia 2025: NaN
            Aruba 2025: NaN
            (+245 more)

... diff too long, truncated — see the full report ...

= 11 compared dataset(s) turned out identical; details in the full report

Automatically updated datasets matching excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included. Run locally with etl diff REMOTE data/ --include <dataset> --verbose.

Edited: 2026-08-21 12:56:50 UTC
Execution time: 8.87 seconds

@pabloarosado pabloarosado mentioned this pull request Aug 14, 2026
Merged
pabloarosado and others added 4 commits August 14, 2026 22:41
@pabloarosado @claude
🐛🤖 Energy mix: keep numerators and denominators from one producer
d5e12f7 
Shares are now computed before the EIA total extension, so they always
divide Statistical Review sources by the Statistical Review's own total
(South America's producer-mixed shares go blank). Aggregate sources with
an upstream-deleted component are removed instead of undercounted, EIA
entities redundant with Statistical Review coverage (East/West Germany,
Czechoslovakia, post-1984 USSR) are dropped, annual changes across a
producer boundary are removed, and two sanity checks now enforce that
shares sum to ~100% and that combined country totals match EI's World.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado @claude
🔨🤖 Energy mix: drop pandas import used for one line
0791c3d 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado @claude
✨🤖 Energy Institute: exclude residual provider regions at the source
d6e01fe 
EI's residual buckets ("Other Europe (EI)", ...) have no fixed
composition: the same name can cover a different set of countries for
each indicator, so they are removed from the Statistical Review garden
output right after serving as aggregation inputs (the meadow table
keeps them all). This replaces the per-consumer EXCLUDED_PROVIDER_REGIONS
filters (shared.py deleted) and stops the Statistical Review grapher
dataset from exposing these entities in chart entity selectors, the one
consumer that never filtered them. No published chart selects any of
them; all other values are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Energy mix: gate shares on breakdown-total reconciliation
579d5f0 
Reverts computing shares before the EIA extension (conceptually wrong:
series should be complete before deriving shares). Instead, an
entity-year gets share indicators only if its exclusive sources add up
to its total within 1%. This enforces completeness in countries
(numerator and denominator describing the same set of countries) and in
components (no materially missing source) at once, while a source an
entity genuinely does not use never disqualifies it. North America's
shares now end in 1985, when its unattributable other-renewables
remainder becomes material. A guard asserts the exact set of gated-out
entities so coverage changes in future releases fail loudly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado pabloarosado mentioned this pull request Aug 18, 2026
Draft
pabloarosado and others added 21 commits August 18, 2026 13:47
@pabloarosado @claude
🔨🤖 Revert energy mix changes, keeping provider-region exclusion
cdc35ba 
Reverts the share/aggregate/entity changes to energy_mix so the step
matches the published dataset again (verified cell-by-cell). The only
remaining difference is the removal of the now-redundant provider-region
filter, since those entities are excluded in the Statistical Review
garden step. The energy mix problem will be reworked from scratch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado @claude
🔨🤖 Energy mix: drop redundant reset_index round trip
128373e 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Energy Institute: fill missing zeros in energy supply and generation
4acebea 
The Statistical Review omits coal, oil, gas and biofuels for country-years
where they are zero, while reporting hydro, solar, wind and other
renewables explicitly. Its own total confirms the omissions: wherever the
total is informed it equals the sum of the reported sources, leaving no
room for the missing ones. EIA agrees independently (exactly zero in 170
of 178 checkable coal omissions and 194 of 203 gas ones), and every
omission ends the year a country's coal or gas era begins, confirmed for
Spain 1969, Ireland 1979, South Korea 1986 and Israel 1980.

fill_missing_total_energy_supply asserts that reconciliation and fills
3,464 source cells with zeros, skipping rows whose total is a placeholder
zero (USSR 1985-1991, Bangladesh before 1971).

fill_missing_electricity_generation asserts that hydro, solar and wind
supply is identical to their generation, which the physical energy content
method makes definitional, and fills 2,206 generation cells from it.

Three corrections found by that second assertion, each confirmed both by
the generation column and by closing the total's reconciliation gap to
zero: USSR 1972 and Other Asia Pacific 1997 wind, and Iran 2021 hydro.

Every fill is a zero; no published value is lost or altered beyond the
three corrections. Downstream, fossil fuels gains 759 computable values
and low-carbon generation 324, since completing the components makes the
sums computable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Energy mix: require every component of an aggregate source
ce114f2 
The Statistical Review garden step now fills in the zeros the producer
omits, so a component still missing in energy_mix is either genuinely
unknown or was removed upstream for not representing its region. Treating
it as zero undercounted the aggregate, so the seven fillna(0) calls are
replaced by min_count sums, matching the fossil fuels line above them.

Removes only fabricated values: North America's renewables and low-carbon
(its other renewables was deleted upstream), Bangladesh's 1965-1970
placeholder rows, and 1,834 solar_and_wind values. Of those, 1,806 were
zeros for countries with no solar or wind data at all, and 28 were South
America 1998-2025, where solar alone was published under the label 'solar
and wind' because the region's wind had been removed upstream.

No value is gained or altered.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Energy mix: build region aggregates from combined EI and EIA data
c8d64bb 
South America's shares divided Energy Institute sources by an EIA total,
because the Statistical Review cannot attribute part of the continent to
any country (its 'Other South and Central America' residual, 1,108 TWh in
2024) and its own aggregates are therefore incomplete, or removed
entirely where the residual exceeds 15% of the region.

Rather than inherit those aggregates, energy_mix now drops them and
rebuilds them from country-level data, extending each source and the total
to the countries the Statistical Review omits with EIA. This is what
fossil_fuels already does, which is why its South America was complete
while energy_mix's was empty. Seven of the nine sources have a directly
comparable EIA column, verified on overlapping country-years (median
EIA/EI ratio 1.00 for wind, 0.99 for hydro and solar, 1.01 for coal, 0.96
for nuclear, 1.07 for oil, 1.11 for gas). Other renewables and biofuels
have none, so they stay Statistical-Review-only.

South America and North America regain their totals, all nine sources and
all their shares, which now sum to 100.3% and 99.9%. Every region's shares
sum to ~100%. A new check requires the combined countries to add up to the
Statistical Review's own World total, which they do to a median of +0.08%.

The Statistical Review is prioritized per value, so no country's numbers
change: the only non-aggregate differences are the Iran and USSR hydro
corrections added earlier. EIA entities whose territory the Statistical
Review already covers (East and West Germany, Czechoslovakia, post-1984
USSR) are dropped to avoid double-counting them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado @claude
💄🤖 Energy: cut the comments back to what the code cannot say
a1ba4c7 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado @claude
✨🤖 Energy mix: reconcile the combined data against the producer's own…
e5204b6 
… totals

Three checks, all comparing against numbers the Statistical Review
publishes itself:

- The combined countries must add up to its World total (worst 0.4%).
- Each of its seven regions must be recovered by summing that region's
  members (worst 3.5%, Africa, where it itemizes only 4 of 58 countries).
- Where all nine exclusive sources are reported, their shares must add up
  to 100% (worst 1.2%). This is the check whose absence let South
  America's producer-mixed shares ship.

Both reconciliations are restricted to the years EIA covers, derived from
the data rather than hardcoded, since outside them the countries the
Statistical Review omits have none.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado @claude
🔨🤖 Energy mix: tighten the World reconciliation tolerance to 1%
22fa32d 
It was sized at 10% when the check still spanned years EIA does not cover.
Capped to EIA's window the sum lands within 0.4%, so 10% would have caught
almost nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado @claude
🔨🤖 Energy mix: derive EIA's year window where it is used
e2a8103 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Energy mix: fill the gaps in the region aggregates instead of repl…
e4f1027 
…acing them

Rebuilding every aggregate from the combined countries introduced abrupt
jumps at EIA's coverage boundaries, because the country set changed size
mid-series: Africa rose 69% from 1979 to 1980 and fell 38% into 2025, and
lower-middle-income countries rose 31% at 1980. The reconciliation checks
were restricted to EIA's window, so they skipped exactly those years.

The Statistical Review's own aggregates include the energy it cannot
attribute to any country, so they are complete and span its full range.
They are now kept wherever it publishes them, and only the ones it removes
for not being representative are rebuilt, within EIA's years alone so the
country set never changes mid-series.

Every region's worst year-on-year jump now matches the published data
exactly. South America keeps its total, all nine sources and shares summing
to 100% over 1980-2024, which is the coverage it already had, and North
America keeps its full 1965-2025 total while gaining other renewables.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pabloarosado
Revert "🐛🤖 Energy mix: fill the gaps in the region aggregates instead…
fcd4534 
… of replacing them"

This reverts commit e4f1027.
@pabloarosado @claude
✨🤖 EIA: add biofuels and estimate other renewables
6ab7d27 
The Statistical Review's regional aggregates rely on residual "Other *"
buckets that cannot always be attributed to a single OWID region, so we are
moving to aggregates built from combined EI and EIA countries. That needs all
nine sources from EIA, and two were absent.

Biofuels consumption was already in the meadow table and simply never carried
through. EIA reported an explicit zero for the ~145 countries that consume no
biofuels until 2019 and publishes only the ~70 that do from 2020, so series
ending in zero are continued at zero; Belarus and Hong Kong end on a positive
value and keep their gap.

Other renewables has no published equivalent. EIA's total energy supply for
renewables is a single lump on a different basis, sitting ~36% below the
Statistical Review's, so it can neither be split nor used as a residual. The
estimate is built from geothermal, biomass and waste, and tide and wave
generation, converted with the efficiencies the IEA's physical energy content
method assumes (10% geothermal, 33% thermal), and is flagged as an OWID
estimate in description_processing.

Both track the Statistical Review's World series to within 6-13% in recent
decades, the same direction and size as EIA's disagreement on its own total.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Energy mix: build region aggregates from combined EI and EIA count…
e9110cf 
…ries

South America's shares summed to 31%. The Statistical Review deletes its
total, because the energy it cannot attribute to any South or Central American
country is 30% of it, and EIA's garden step publishes a row named plainly
"South America" that the "(EIA)" filter missed, so that total filled the gap
while the nine sources stayed with the Statistical Review. Numerator and
denominator covered different countries.

The Statistical Review's regional aggregates depend throughout on residual
"Other *" buckets, and the one spanning South and Central America cannot be
assigned to a single OWID region. Aggregates are therefore built here, from
countries, once the two producers have been combined.

EIA now supplies all nine sources rather than a bare total, so the 143
countries it alone covers gain a full energy mix. Their total is the sum of
those sources, because EIA's own total measures something else: it counts
electricity consumed rather than generated and leaves geothermal and biomass
uninflated. Both differences are large — Iceland's EIA total sits 62% below
the Statistical Review's, Paraguay's excludes the hydro it exports — so the
published total silently mixed two definitions across countries. Nuclear needs
no adjustment; EIA already reports it as heat input, at 3.06 times generation
against the Statistical Review's flat 3.03.

Aggregates span the years EIA covers, 1980-2024. Outside them the Statistical
Review itemizes too few countries to stand for a region (4 of Africa's 57,
home to 23% of the continent), and admitting them would show a rise that is
ours rather than the world's: South America would climb 6.5% in a year purely
because Bolivia, Paraguay, Uruguay, Guyana and Suriname appear. Within the
window, aggregates also require 65% of the countries that ever report plus
each region's largest consumers, so a region cannot quietly lose a country.

The checks no longer carry a year window of their own. Restricting them to
EIA's years is what let a 69% jump in Africa pass review earlier in this
branch. They now assert over everything published: the total equals the sum of
its sources, the shares sum to 100%, each region's years are unbroken and match
what is recorded here, no region has an unexpected gap, and the continents
reconcile with the Statistical Review's World.

South America's shares now sum to 100.00%, and across the whole dataset to
between 100.00% and 100.67%. No country the Statistical Review reports changes.
Lower-middle-income countries rise 16-18%, which is the residual its aggregate
never included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Fossil fuels: build region aggregates from combined EI and EIA cou…
92c4af7 
…ntries

The step took its region rows ready-made from whichever producer had them. The
Statistical Review's rest on residual "Other *" buckets it cannot attribute to
any country, so they are understated: its gas production for the low-income
group is a third of the sum of those countries, and lower-middle-income oil
consumption a quarter short. Aggregates are now built here, from the countries,
under one set of rules.

Doing so surfaced a double count that had been harmless while nothing was
aggregated: EIA's East Germany, West Germany and Czechoslovakia cover territory
the Statistical Review already reports as Germany, Czechia and Slovakia over the
same years, so the continents summed 10-14% above the World for 1980-1990. They
are dropped, as EIA's USSR already was, and the continents now reconcile within
0.7%.

Coverage is bounded by the years both producers cover. Coal production starts in
1981 rather than 1980 because the Statistical Review begins there: in 1980 that
column would rest on EIA alone, whose USSR is dropped as a duplicate, so the
aggregate would omit the largest coal producer of the time and upper-middle-income
countries would jump 72% into 1981 as it reappears.

Europe is held to a minimum count of reporting countries rather than a fraction
for the energy columns. A fraction misjudges it there, because the denominator
counts the USSR's and Yugoslavia's successors in years where those two are still
one entity each: it reports 28 of 47 countries in 1980 and 39 by 1992, though its
territory is covered throughout. It keeps the fraction elsewhere, where a
shortfall is real rather than a matter of naming.

Reserves get no region aggregates at all. EIA reports the USSR's gas reserves
until 1991 and Russia's only from 1997, so any European sum loses 89% of them in
1992 and recovers it in 1997, and no coverage rule catches it: 39 of Europe's 47
countries report that year, and the one missing holds nine tenths of the
reserves. No chart uses them for a region — the fossil fuels collection shows
reserves for countries only, and the reserves-to-production ratio chart is
World-only and comes from the Statistical Review.

Every region-column is now unbroken. The World is unchanged. Low-income gas
production rises 245%, lower-middle-income oil consumption 33%, and Oceania's
gas production 12%, all of which is the residual those aggregates never included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
✨🤖 Fossil fuels: stop gating the coverage of EIA's own columns
341479d 
Only the energy columns are gated now. The rest are EIA's physical production,
trade and consumption, whose coverage varies by column and region however EIA
happens to report it, so any threshold on them just traded ragged start years for
gaps in the middle of a series: 0.6 left oil production in cubic metres starting
in 1989 for South America against 2004 elsewhere, 0.7 pushed Europe's whole
physical set to 1992, and a year-aware variant opened five gaps.

They are summed over whichever countries report. Sixteen of them still start at
different years for different regions, which is what EIA's own coverage looks
like, but no region series has a hole in it, and the six energy columns stay
uniform at 1980-2024 (coal production 1981-2024).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🔨🤖 Fossil fuels: keep the overlap check meaningful for both aggregations
aaeedba 
The two aggregation calls see different data, so they need different accepted
overlaps. Passing one list to both made add_aggregates report the Statistical
Review's Yugoslavia overlaps as missing, since EIA's columns do not have them,
and a check that always warns is a check nobody reads.

Verified in passing that nothing is double-counted: no predecessor shares a year
with its own successors in any column, in either call, so add_aggregates now runs
silent. The one real pair is Aruba against the rest of the Netherlands Antilles,
which EIA reports side by side from 1986.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Fossil fuels: hold the three fuels of a metric to one window
40533bd 
Every region's total fossil production for 1980 was oil plus gas only. Coal
production was clipped to 1981, where the Statistical Review's series begins,
while oil and gas ran from 1980, and the total sums whatever fuels are present:
Africa's 1980 figure came out 900 TWh short, a step in the "Total" view of every
region.

Requiring all three fuels everywhere is not an option — 4,811 country rows
legitimately have fewer, among them the United Kingdom's coal-only series before
1900 — so instead the three fuels of a metric now share one window. Region
production spans 1981-2024 and consumption 1980-2024, and a regional total is
always the sum of three fuels. The World is untouched, keeping its coal-only
totals from 1800.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Fossil fuels: require Russia for Europe's reserves aggregate
77e0e39 
Region aggregates for reserves are back, as production has them, but with the one
condition that separates the good years from the bad. EIA reports the USSR's
reserves until 1991 and Russia's only from 1997, and Russia holds about nine
tenths of Europe's, so an unconditional sum shows Europe losing 89% of its gas
reserves in 1992 and recovering them in 1997. Production has the same defect for
1996, where it publishes 5.5 trillion cubic metres before jumping tenfold.

Requiring Russia cannot be applied to the other EIA columns, where the USSR covers
1980-1991 and dropping those years would be wrong, so reserves are aggregated on
their own. Europe's oil and gas reserves now span 1997-2021; every other region
keeps 1980-2021, and coal reserves 2008-2023.

Each of the three aggregations is also given the overlaps its own columns contain,
so add_aggregates only speaks up when something is actually wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
✨🤖 Fossil fuels: let the aggregates reach the Statistical Review's la…
2fa2d25 
…test year

The aggregates were cut at EIA's last year at both ends of the window, but only
the lower bound had a reason (coal production in 1980 would rest on EIA alone,
whose USSR is dropped as a duplicate). At the recent end the cut removed a year
the Statistical Review does publish.

Europe and high-income countries now reach 2025 on consumption, where enough of
them has reported; every other region stops at 2024, as does all production. The
countries EIA has yet to publish for 2025 are worth 0.6-2.2% of Europe's oil and
gas and 5.8% of its coal — the Western Balkans, Serbia and Bosnia above all, which
the Statistical Review does not report at all — which sits within the spread
between the two producers elsewhere in this step.

The three fuels of a metric still share a start year, so a regional total is
always the sum of all three, and the years each aggregate spans are still
asserted, now as a start year per column and a permitted set of end years.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Fossil fuels: keep East Germany, West Germany and Czechoslovakia
77729fc 
They were dropped from EIA's input to stop them double-counting Germany, Czechia
and Slovakia in the region aggregates, but that removed the entities themselves:
the Statistical Review does not report them, so their energy series had nowhere
else to come from and vanished, while their physical series survived through the
other readers. An entity with tonnes but no terawatt-hours for the same years.

They are excluded from the aggregation instead, where the double count actually
arises, and keep their own series: East and West Germany 1980-1990, Czechoslovakia
production from 1900 and consumption from 1980, all matching what is published.
The continents still reconcile with the World to within 0.1%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
📜🤖 Energy: describe the processing on the data pages
17b07fc 
The energy mix's processing note still told readers that the total energy supply
for countries the Statistical Review does not cover comes from EIA's total energy
consumption, which is precisely what this branch stopped doing. It now says the
total is the sum of the sources, and why: EIA's own total counts electricity as
consumed rather than generated, and does not convert geothermal and bioenergy to
the heat needed to produce them.

Both energy mix and fossil fuels now also say that continents and income groups
are our own sums of their countries, shown only where enough of a region reports.
EIA's biofuels gains a note about the zeros the producer stopped publishing after
2019, and the note on other renewables is rewritten for a reader rather than a
maintainer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pabloarosado and others added 5 commits August 19, 2026 09:22
@pabloarosado @claude
✨🤖 Energy: end the fossil fuel regions where the energy mix ends its own
3e71235 
Both steps rebuild their regions from the same combined countries, so both should
span the same years. The energy mix bounds them by EIA's coverage; fossil fuels
briefly did not, which let Europe and high-income countries alone reach the
Statistical Review's extra year on a sum missing the countries EIA has yet to
publish. Every region in both now ends in 2024, and fossil fuel production starts
in 1981 rather than 1980 so a regional total is always the sum of all three fuels.

The note on EIA's estimated other renewables also drops the "our own estimate"
framing, and gives the reason a reader cares about: the conversion is there so the
figures sit on the same basis as our other energy datasets.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🔨🤖 Energy: drop the coverage conditions and the scaffolding around them
96bb59a 
The conditions no longer do anything. Once the aggregates are bounded by EIA's
years, every region reports at least three quarters of the countries that ever
report there, so a threshold low enough to admit them all never fires: removing
the fraction, Europe's country floor and the mandatory-producer lists changes not
one cell in either step. What catches a region losing data is the checks — the
years each aggregate spans, no gaps within them, the continents reconciling with
the World, and in the energy mix the total equalling the sum of its sources.

With the mandatory countries gone, fossil fuels needs only one call to
add_aggregates rather than three behind a nested helper. Europe's reserves still
require Russia, now as three lines after the fact rather than a separate
aggregation, and the two overlap lists collapse into one since a single call sees
every column. The single-use get_eia_year_range is inlined, the region checks fold
into sanity_check_outputs, and the three tables that were read with
reset_index=False and then reset at all ten use sites are read with the default.

Both steps produce byte-identical output, verified column by column.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Energy: keep the entities and years the rebuild was dropping by ac…
7524933 
…cident

The data diff for this branch turned up two losses that were not intended.

The energy mix was dropping East Germany, West Germany and Czechoslovakia
entirely, the same mistake already fixed in fossil fuels: they were removed from
EIA's input to stop them double-counting Germany, Czechia and Slovakia in the
aggregates, but the Statistical Review does not report them, so the entities went
with it. They are excluded from the aggregation instead and keep their own series,
1980-1990 and 1980-1992 as published.

Kenya lost its total energy supply for 1983-1993, and Saudi Arabia a year, because
EIA leaves an interior gap in their biofuels and a total needs all nine sources.
Kenya reports zero biofuels up to 1982 and zero again from 1994, so the gap is
zero as well; the fill now covers gaps enclosed by zeros, not only those that run
to the present. Cuba's two missing years still get nothing, since there a gap
separates a zero from a positive figure and the truth is unknown.

What remains is 217 cells of total energy supply, none of them a whole entity:
three Pacific territories and four French overseas departments, for which EIA
reports too few sources to build a total on the Statistical Review's basis, a few
partial years elsewhere, and the USSR from 1985, which is dropped deliberately
because the Statistical Review hands over to its successors there and EIA's own
figures sit 15% above its last year.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Energy mix: never publish part of a mix as though it were the whole
d11d689 
Seven small territories lost their total energy supply because EIA reports too few
of the nine sources for them, but they kept the sources it does report. In a
stacked chart that reads as a complete mix: Tuvalu's oil would be all of its
energy, and Reunion's three zeros would say it consumes nothing at all. Losing
their data is one thing, showing a fraction of it as the whole is another.

Every row now reports either the whole mix or none of it. Rows without a total
lose their sources, which also clears 865 stray zeros left by the nuclear and
biofuels fills on rows carrying nothing else — Angola before the Statistical
Review covers it, Afghanistan in 2025. Rows whose total is zero have the missing
sources set to zero instead, since energy cannot be negative and the mix is then
complete by arithmetic; the Statistical Review carries such rows for the years
before a country exists, Bangladesh before 1971 among them.

An assertion holds the invariant, and it earned its place immediately by finding
the Bangladesh case, which the reasoning above had missed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Statistical Review: remove Bangladesh's placeholder zeros before 1971
8ec94cc 
Bangladesh was reported with a total energy supply of exactly zero from 1965 to
1970 and with real figures from 1971, the year it became independent. The
territory was East Pakistan until then and certainly used energy, so those zeros
stand in for years the producer does not cover rather than describing them, and
publishing them made Bangladesh appear to have consumed no energy for six years.
It was the only entity given an exactly-zero total.

The same years also reported zero hydropower, which is demonstrably wrong: the
Karnaphuli dam has been generating since 1962, and the producer itself reports
0.175 TWh for 1971. Both go, as corrections rather than inline exceptions, guarded
so they fail if the producer starts reporting something other than zero.

The zeros for solar, wind and nuclear in those years stay, since they are true.
The same reasoning does not reach the USSR's successors, which was worth checking:
the producer leaves Russia, Ukraine, Kazakhstan, Croatia and Slovenia empty before
their predecessors dissolve rather than filling them with zeros, and reports
Czechia and Slovakia in full from 1965.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado pabloarosado changed the title 🐛 Energy mix: South America shares mix two producers 🐛 Energy: rebuild the region aggregates and drop the residual "Other *" regions Aug 19, 2026
pabloarosado and others added 18 commits August 19, 2026 12:06
@pabloarosado @claude
🐛🤖 Energy: reconcile every source against the producer's World total
b881b01 
The check only covered total energy supply in the energy mix, and only the
continents in both steps. It now runs over every energy column and over the
income groups as well, which is what would have caught South America's shares
resting on a foreign denominator.

The tolerance drops from 5% to 3% in the energy mix (the largest deviation
measured is 2.2%, for renewables in 1988). Biofuels needs an absolute floor
beside it: the producer leaves 10-16 TWh of Caribbean biofuels unattributed to
any country until 1999, which is 40% of the world total in 1980 and nothing at
all by 2000.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Energy: fill biofuels once both producers are in, not before
7f82a88 
The Statistical Review step filled every missing source with zero where the
producer's own total left no room for it. That premise fails for biofuels: the
producer's total omits the biofuels it had not started tracking, reporting none
at all for the United States until 1990, while the EIA measured 2-21 TWh a year
there over 1981-1989. Since the Statistical Review takes priority when the two
producers are combined, a zero we derived was overriding a measurement, for 42
country-years across the United States, Vietnam, Argentina, Pakistan and South
Korea.

Biofuels is now filled in the energy mix instead, after the combination, on the
same condition. The other eight sources keep their zeros where they are, so the
fossil fuels step is untouched, and no country loses its mix: the 1,002 pre-1980
cells that no other producer can reach are still filled.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
📜🤖 Energy mix: record what the World reconciliation actually measures
b29697a 
The comment described the tolerances before biofuels was fixed, when the gap only
ever ran one way. It now runs in both directions, and the measured worst case is
13.5% (15 TWh, in 1991), which is 0.009% of world total energy supply.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 EIA: measure tide and wave as electricity, not as heat input
ca1f323 
The derived other-renewables estimate inflated tide and wave by the 33% thermal
efficiency, treating it as a combustion source. The Statistical Review's
methodology puts it at 100%, alongside hydro, solar PV and wind: only geothermal
(10%) and the combustion of biomass (33%) have their heat input estimated.

Lesotho is the only published country affected, by 1 GWh in 2023 and 2024, since
the estimate applies solely where the Statistical Review does not report a
country. Validation against EI is unchanged: the median deviation across 1,933
overlapping country-years is 4.6%, and the geothermal-dominated countries land
within 3-7%.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 EIA: stop counting biofuels as oil consumption
20590ba 
The producer defines "petroleum and other liquids" to include biofuels, so its
oil consumption series double count them against our separate biofuels source.
Subtract them from both the energy and the barrel-denominated series, before the
region aggregates are summed. Production is left alone: the energy series already
excludes biofuels, and the barrel-denominated one keeps the producer's own name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
📜🤖 EIA: say plainly why oil production needs no correction
9202520 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🎉🤖 EIA: publish biofuels production alongside biofuels consumption
7bbb6e4 
The producer reports it and we were dropping it. It is already part of the
producer's renewables production, in energy units, and of its petroleum and other
liquids production, in barrels, so both columns say so in their metadata.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Statistical Review: stop deleting region aggregates that are complete
e679c7c 
"Other South and Central America" is a rollup of finer residual regions the
producer publishes alongside it, and that we already assign: "Other South America"
to South America, "Other Caribbean" and "Central America" to North America. Where
the rollup is exactly their sum, every country it covers is already in one of those
aggregates, so testing the rollup against them and deleting on a 15% threshold
removed data for no reason. South America lost its total energy supply, oil, coal,
hydro and wind for all 61 years on that basis.

Skip a column when the finer regions cover the rollup, and assert they still do for
total energy supply, so a vintage that stops publishing them fails loudly. The check
still fires where the difference is genuinely unassigned: electricity generation by
fuel, biodiesel consumption and coal reserves. 38 deletions become 12.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
📜🤖 Statistical Review: correct the notes on the residual regions
0d32627 
The note on "Other South and Central America" said it cannot be mapped to either
region and that both are therefore underestimated. That is only true for the
indicators where the finer residual regions are absent or fall short of it. The
claim about which indicators "Other CIS" affects was from the 2025 release and no
longer holds. Oceania's note used a region name that does not exist. Income groups
had no note at all, though they are the aggregates with the largest gap.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
✨🤖 Energy: take continents from the Statistical Review, rebuild only …
f00f67f 
…income groups

The producer assigns every one of its undisclosed residual regions to a continent,
so its continents cover the whole globe and, once the deletion rule stopped removing
complete aggregates, reach back to 1965. Rebuilding them from combined countries was
measured to move them by under 2% while costing every year before EIA begins, so it
is no longer done. It assigns no residual region to an income group, and cannot: the
countries "Other South America" stands for span three of them. So income groups are
still summed from countries, which is the only way to get them right.

Continents now span 1965-2025 instead of 1980-2024 in the energy mix, and the same
in fossil fuels except coal production (1981) and gas production (1970), which
follow the producer's own coverage. Fossil fuels keeps summing their physical
production, reserves and trade from countries, since the producer has no counterpart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Statistical Review: judge each region against its own residual reg…
e594aa9 
…ions

The skip added earlier was per column: a column counted as covered only when the
rollup equalled the sum of every finer region, so a column where the producer
publishes the South American residual but not the Caribbean one still had South
America's aggregate deleted, though nothing was missing from it. What a rollup holds
beyond a region's own residual belongs to the other region, and says nothing about
this one. Judge each region against its own components instead, which also keeps the
electricity-generation columns for South America. 12 deletions become 9, all of them
now cases where the region really is missing data: coal reserves for South America,
biodiesel and oil-fired electricity for North America.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Statistical Review: require two countries for Oceania's aggregate
4b29e75 
Oceania is the one continent with no residual "Other *" region of its own, since the
producer folds it into Asia, so its aggregate is only the countries it names, and it
names at most two for an energy indicator. A minimum of two countries informed keeps
the aggregate where New Zealand is reported and drops it where it would be Australia
alone: 17 columns go, among them oil and gas production, oil reserves and electricity
generation by fuel, while all 38 consumption columns and coal production stay.

Fossil fuels records Oceania's two missing production columns rather than rebuilding
them from countries, which would count New Zealand and Papua New Guinea twice, once
there and again inside the residual region folded into Asia. The continents-against-
World check is skipped for those two columns; the income groups still cover them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
📜🤖 Statistical Review: measure the Asia Pacific residual instead of g…
8872d24 
…uessing at it

The note assigning "Other Asia Pacific" to Asia rested on the expectation that
Papua New Guinea was the main, possibly only, Oceanian country in it. Pricing the
countries it must stand for with EIA's data puts it at 93% Asian and 7% Oceanian in
2000 and 2023 alike, with Papua New Guinea only 38% of the Oceanian part. The
conclusion is unchanged; the reasoning is now checkable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
🐛🤖 Electricity mix: stacked charts get columns that cannot misstate t…
badbef1 
…he mix

Stacked area charts were assembling standalone source columns, so an entity-year
missing one source showed part of its mix as though it were the whole: Oceania's
1985-1999 stack covered a fifth of its own total, and 128 countries had the same
defect somewhere. The standalone columns are accurate one by one and stay untouched;
the fix extends the existing chart-specific-column pattern to the two stacked charts
still on standalone columns and hardens all three, with the logic in one place:

- a gap the reported total leaves no room for is a zero confirmed by the producer,
  and is filled (Sri Lanka reports no nuclear; its other sources add up to its total);
- any other row missing a source is blanked whole;
- a row reporting every source whose sum still deviates from the total by more than
  5% is blanked too (the United Kingdom's sources before 1985 reach 77% of its total).

Both invariants are asserted on the result. The multidim by-source views now use
these columns as well, instead of the coalesced pair, whose "bioenergy may be
included prior to 2000" footnote no longer applies. The unused columns for the
retired electricity-prod-source-stacked slug are gone.

The electricity-production-by-source and share-elec-by-source chart configs still
point at the standalone columns and are repointed on staging next.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
📜🤖 Docs: technical page on regional aggregates in the energy data
b1155e9 
Documents the choices this PR settles and the evidence behind them: what the
Statistical Review's residual "Other *" regions are and where each is assigned,
why the continents come from the producer while the income groups are rebuilt
from countries, why Oceania's production is withheld, how EIA's biofuels-in-oil
accounting is corrected, how "other renewables" is derived and validated, and
the dedicated indicators behind stacked charts. Six figures generated from the
pipeline's own data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
📜🤖 Docs: rename the energy aggregates page to energy of world regions
717a766 
Matches the naming of the existing food supply of world regions publication.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
📜🤖 Docs: energy of world regions as an executable notebook
ff1bfe4 
Rewritten in the format of the other technical publications: author and date
header, data loaded from the public catalog with pinned versions, interactive
charts, and assertions in place of prose claims (the rollup decomposes exactly,
the continents sum to the producer's World total, the derived other renewables
match within tolerance). Replaces the static markdown page and its figures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@pabloarosado @claude
📜🤖 Docs: correct attribution and framing in the energy regions public…
36eda25 
…ation

The Statistical Review publishes no aggregates for OWID regions or income groups;
every such aggregate is built by us from its data. The notebook and several code
comments wrongly spoke of "the producer's aggregates" being understated, and
summarized the income-group gap as 91-99% coverage of the World total, a figure
dominated by the near-complete rich groups. Both are corrected: aggregates are
described as our own constructions, and the gap is stated where it lives (about
14% of the lower-middle-income group's energy supply, a third of its hydropower,
over half of its other renewables, and no low-income aggregate at all from EI's
individually-reported countries).

Also: the notebook is rewritten in plainer wording, the region boundaries are taken
from the Statistical Review's own Definitions sheet (which places Ukraine and
Georgia in its Europe, not its C.I.S., and never lists the countries inside an
"Other" group), and a world map shows which source covers each country (79 in both,
135 only in the EIA, 33 in neither).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pabloarosado @owidbot