Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
b5ae1b7
πŸ“Š energy: Update IRENA data
pabloarosado Jul 28, 2026
ea8024d
πŸ“ŠπŸ€– economics: Update OWID deflator (WDI release 129, covers 2025)
pabloarosado Aug 21, 2026
fe73cf8
πŸ“ŠπŸ€– energy: Update IRENA statistics, costs and patents datasets
pabloarosado Aug 21, 2026
0d44196
πŸ›πŸ€– lazard: Adapt to re-referenced WDI deflator (base year 2017 -> 2020)
pabloarosado Aug 21, 2026
ae95d5a
Merge branch 'master' of github.com:owid/etl into data-irena-update
pabloarosado Aug 21, 2026
687a134
Merge remote-tracking branch 'origin/master' into data-irena-update
pabloarosado Aug 26, 2026
0937fa7
πŸ“œπŸ€– energy: Name the linked GDP deflator series explicitly in metadata
pabloarosado Aug 26, 2026
6e9618b
πŸ”¨πŸ€– irena: Use paths.regions helpers in the patents step
pabloarosado Aug 26, 2026
24cdf94
πŸ”¨πŸ€– irena: Give producer regions readable names in the country mapping
pabloarosado Aug 26, 2026
248dcd0
πŸ”¨πŸ€– Modernize wdi_prices harmonization and the IRENA patents snapshot …
pabloarosado Aug 26, 2026
c94dbe2
πŸ’„πŸ€– irena: Restore nested list indentation in the statistics snapshot …
pabloarosado Aug 26, 2026
892bc55
πŸ”¨πŸ€– irena: Trim the power generation costs garden step
pabloarosado Aug 26, 2026
4c96936
Merge branch 'master' of github.com:owid/etl into data-irena-update
pabloarosado Aug 28, 2026
631f228
πŸ”¨πŸ€– Remove superseded IRENA, photovoltaic and deflator steps, and reor…
pabloarosado Aug 28, 2026
e216599
πŸ”¨πŸ€– Archive superseded IRENA, photovoltaic and deflator steps
pabloarosado Aug 28, 2026
b37c71f
πŸ“ŠπŸ€– lazard: Update levelized cost of energy to the 2026 report
pabloarosado Aug 28, 2026
4e7bbf7
πŸ”¨πŸ€– Archive the lazard 2025-09-17 steps
pabloarosado Aug 28, 2026
baec769
βœ¨πŸ€– lazard: Inherit the chart title from the indicator metadata
pabloarosado Aug 28, 2026
c8ffd28
πŸ”¨πŸ€– Revert the Lazard update and keep the 2025-08-29 deflator for it
pabloarosado Aug 28, 2026
c17ea55
πŸ”¨πŸ€– Regenerate the DAG archive after restoring the Lazard and 2025-08-…
pabloarosado Aug 28, 2026
8a00fcc
πŸ’„πŸ€– Fix a typo and use en dashes in year ranges
pabloarosado Aug 28, 2026
3a9588c
Merge branch 'master' of github.com:owid/etl into data-irena-update
pabloarosado Sep 1, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions dag/archive/energy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1482,3 +1482,45 @@ steps:
# archived; last active in 2a47d03292a9 on 2026-08-05
export://explorers/energy/latest/fossil_fuels:
- data://grapher/energy/2026-05-05/fossil_fuels
# archived; last active in 4387a320d17b on 2026-08-28
data://meadow/irena/2025-06-20/renewable_energy_patents:
- snapshot://irena/2025-06-20/renewable_energy_patents.csv
# archived; last active in 4387a320d17b on 2026-08-28
data://grapher/irena/2025-08-22/renewable_power_generation_costs:
- data://garden/irena/2025-08-22/renewable_power_generation_costs
# archived; last active in 4387a320d17b on 2026-08-28
data://meadow/irena/2025-07-18/renewable_energy_statistics:
- snapshot://irena/2025-07-18/renewable_energy_statistics.xlsx
# archived; last active in 4387a320d17b on 2026-08-28
data://grapher/energy/2025-08-22/photovoltaic_cost_and_capacity:
- data://garden/energy/2025-08-22/photovoltaic_cost_and_capacity
# archived; last active in 4387a320d17b on 2026-08-28
data://grapher/irena/2025-06-20/renewable_energy_patents:
- data://garden/irena/2025-06-20/renewable_energy_patents
# archived; last active in 4387a320d17b on 2026-08-28
data://garden/irena/2025-07-18/renewable_energy_statistics:
- data://garden/regions/2023-01-01/regions
- data://meadow/irena/2025-07-18/renewable_energy_statistics
- data://garden/wb/2026-07-01/income_groups
# archived; last active in 4387a320d17b on 2026-08-28
data://garden/irena/2025-06-20/renewable_energy_patents:
- data://meadow/irena/2025-06-20/renewable_energy_patents
- data://garden/regions/2023-01-01/regions
- data://garden/wb/2026-07-01/income_groups
# archived; last active in 4387a320d17b on 2026-08-28
data://grapher/irena/2025-07-18/renewable_capacity:
- data://garden/irena/2025-07-18/renewable_energy_statistics
# archived; last active in 4387a320d17b on 2026-08-28
data://grapher/irena/2025-07-18/renewable_capacity_by_technology:
- data://garden/irena/2025-07-18/renewable_energy_statistics
# archived; last active in 4387a320d17b on 2026-08-28
data://garden/irena/2025-08-22/renewable_power_generation_costs:
- data://meadow/irena/2025-08-22/renewable_power_generation_costs
- data://garden/economics/2025-08-29/owid_deflator
- data://meadow/irena/2024-11-15/renewable_power_generation_costs
# archived; last active in 4387a320d17b on 2026-08-28
data://garden/energy/2025-08-22/photovoltaic_cost_and_capacity:
- data://garden/irena/2025-07-18/renewable_energy_statistics
- data://garden/papers/2023-12-12/nemet_2009
- data://garden/irena/2025-08-22/renewable_power_generation_costs
- data://garden/papers/2023-12-12/farmer_lafond_2016
8 changes: 8 additions & 0 deletions dag/economics.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,18 @@ steps:
- snapshot://economics/2026-07-28/wealth_france.xlsx

# OWID - OWID Deflator.
# NOTE: The 2025-08-29 version is kept while Lazard's levelized cost of energy step depends on it.
data://garden/economics/2025-08-29/owid_deflator:
# World Bank - WDI (specifically indicators used for inflation adjustments).
- data://garden/wb/2025-08-29/wdi_prices:
- data://meadow/wb/2025-08-29/wdi_prices:
- snapshot://wb/2025-08-29/wdi_cpi.zip
- snapshot://wb/2025-08-29/wdi_gdp.zip
- snapshot://wb/2025-08-29/wdi_gdp_linked.zip
data://garden/economics/2026-08-21/owid_deflator:
# World Bank - WDI (specifically indicators used for inflation adjustments).
- data://garden/wb/2026-08-21/wdi_prices:
- data://meadow/wb/2026-08-21/wdi_prices:
- snapshot://wb/2026-08-21/wdi_cpi.zip
- snapshot://wb/2026-08-21/wdi_gdp.zip
- snapshot://wb/2026-08-21/wdi_gdp_linked.zip
47 changes: 23 additions & 24 deletions dag/energy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,11 @@ steps:
#
# IRENA - Renewable energy patents.
#
data://grapher/irena/2025-06-20/renewable_energy_patents:
- data://garden/irena/2025-06-20/renewable_energy_patents:
data://grapher/irena/2026-07-28/renewable_energy_patents:
- data://garden/irena/2026-07-28/renewable_energy_patents:
- data://garden/wb/2026-07-01/income_groups
- data://meadow/irena/2025-06-20/renewable_energy_patents:
- snapshot://irena/2025-06-20/renewable_energy_patents.csv
- data://meadow/irena/2026-07-28/renewable_energy_patents:
- snapshot://irena/2026-07-28/renewable_energy_patents.csv
- data://garden/regions/2023-01-01/regions
#
# Energy Institute - Statistical Review of World Energy (2026, Total Energy Supply).
Expand Down Expand Up @@ -136,38 +136,39 @@ steps:
#
# IRENA - Renewable energy statistics.
#
data://garden/irena/2025-07-18/renewable_energy_statistics:
data://garden/irena/2026-07-28/renewable_energy_statistics:
- data://garden/regions/2023-01-01/regions
- data://garden/wb/2026-07-01/income_groups
- data://meadow/irena/2025-07-18/renewable_energy_statistics:
- snapshot://irena/2025-07-18/renewable_energy_statistics.xlsx
data://grapher/irena/2025-07-18/renewable_capacity_by_technology:
- data://garden/irena/2025-07-18/renewable_energy_statistics
data://grapher/irena/2025-07-18/renewable_capacity:
- data://garden/irena/2025-07-18/renewable_energy_statistics
- data://meadow/irena/2026-07-28/renewable_energy_statistics:
- snapshot://irena/2026-07-28/renewable_energy_statistics.xlsx
data://grapher/irena/2026-07-28/renewable_capacity_by_technology:
- data://garden/irena/2026-07-28/renewable_energy_statistics
data://grapher/irena/2026-07-28/renewable_capacity:
- data://garden/irena/2026-07-28/renewable_energy_statistics
#
# IRENA - Renewable power generation costs.
#
# NOTE: This step combines two versions of the data, because the latest release does not have data prior to 2010, for some reason.
data://garden/irena/2025-08-22/renewable_power_generation_costs:
# NOTE: Older releases are kept because they contain data that newer releases dropped (see the garden step).
data://garden/irena/2026-07-28/renewable_power_generation_costs:
- data://meadow/irena/2026-07-28/renewable_power_generation_costs:
- snapshot://irena/2026-07-28/renewable_power_generation_costs.xlsx
- data://meadow/irena/2025-08-22/renewable_power_generation_costs:
- snapshot://irena/2025-08-22/renewable_power_generation_costs.xlsx
# IRENA - Renewable power generation costs.
# NOTE: The following step is kept because it contains more data (mostly prior to 2010) than the latest release.
- data://meadow/irena/2024-11-15/renewable_power_generation_costs:
- snapshot://irena/2024-11-15/renewable_power_generation_costs.xlsx
- data://garden/economics/2025-08-29/owid_deflator
data://grapher/irena/2025-08-22/renewable_power_generation_costs:
- data://garden/irena/2025-08-22/renewable_power_generation_costs
- data://garden/economics/2026-08-21/owid_deflator
data://grapher/irena/2026-07-28/renewable_power_generation_costs:
- data://garden/irena/2026-07-28/renewable_power_generation_costs
#
# Energy - Photovoltaic cost and capacity.
#
data://grapher/energy/2025-08-22/photovoltaic_cost_and_capacity:
- data://garden/energy/2025-08-22/photovoltaic_cost_and_capacity:
- data://garden/irena/2025-08-22/renewable_power_generation_costs
- data://garden/irena/2025-07-18/renewable_energy_statistics
data://grapher/energy/2026-07-28/photovoltaic_cost_and_capacity:
- data://garden/energy/2026-07-28/photovoltaic_cost_and_capacity:
- data://garden/irena/2026-07-28/renewable_power_generation_costs
- data://garden/irena/2026-07-28/renewable_energy_statistics
- data://garden/papers/2023-12-12/nemet_2009
- data://garden/papers/2023-12-12/farmer_lafond_2016
- data://garden/economics/2026-08-21/owid_deflator
#
# Lazard - Levelized Cost of Energy+
#
Expand Down Expand Up @@ -294,5 +295,3 @@ steps:
- data://garden/energy/2025-11-10/ireland_metered_consumption:
- data://meadow/energy/2025-11-10/ireland_metered_consumption:
- snapshot://energy/2025-11-10/ireland_metered_consumption.csv


Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
dataset:
update_period_days: 180

owners:
- Pablo Rosado
tables:
owid_deflator:
title: OWID Deflator
variables: {}
36 changes: 36 additions & 0 deletions etl/steps/data/garden/economics/2026-08-21/owid_deflator.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
"""OWID Deflator dataset.

This is meant to be an auxiliary dataset with indicators commonly used to do inflation adjustments.
"""

from etl.helpers import PathFinder

# Get paths and naming conventions for current step.
paths = PathFinder(__file__)


def run() -> None:
#
# Load inputs.
#
# Load WDI auxiliary dataset with prices-related indicators, and read its table.
ds_wdi = paths.load_dataset("wdi_prices")
tb_wdi = ds_wdi.read("wdi_prices")

#
# Process data.
#
# Combine tables.
tb = tb_wdi.copy()

# Improve table format.
tb = tb.format(short_name=paths.short_name)

#
# Save outputs.
#
# Create a new garden dataset with the same metadata as the snapshot.
ds_garden = paths.create_dataset(tables=[tb], default_metadata=ds_wdi.metadata)

# Save changes in the new garden dataset.
ds_garden.save()
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ tables:
short_unit: "$/W"
description_key:
- "Solar photovoltaic module prices refer to the cost of the solar panel itself, and do not include installation or other system components."
- "Prices are compiled from three sources: Nemet (2009) for 1975-2003, Farmer & Lafond (2016) for 2004-2009, and IRENA for 2010 onward."
- "Prices are compiled from three sources: Nemet (2009) for 1975–2003, Farmer & Lafond (2016) for 2004–2009, and IRENA for 2010 onward."
- "Due to limited data availability, we use the Global Price Index series reported by IRENA, based on pvXchange benchmark prices for modules sold in Europe."
- "Historical prices have been adjusted for inflation using the [World Bank's US GDP deflator](https://data.worldbank.org/indicator/NY.GDP.DEFL.ZS?locations=US)."
- "Historical prices have been adjusted for inflation using the [World Bank's US GDP deflator (linked series)](https://data.worldbank.org/indicator/NY.GDP.DEFL.ZS.AD?locations=US)."
display:
name: Solar PV module cost
shortUnit: "$"
Expand All @@ -36,7 +36,7 @@ tables:
description_processing: |-
- Historical photovoltaic cost data between 1975 and 2003 has been taken from Nemet (2009), and between 2004 and 2009 from Farmer & Lafond (2016).
- From 2010 onward, prices come from IRENA's Renewable Power Generation Costs report, based on pvXchange benchmarks for modules sold in Europe, using the 'Thin film a-Si/u-Si or Global Price Index (from Q4 2013)' series.
- Prices from Nemet (2009) and Farmer & Lafond (2016) have been converted to {LATEST_YEAR} US$ using the US GDP deflator, to account for the effects of inflation. The deflator data is available from the [World Bank World Development Indicators](https://data.worldbank.org/indicator/NY.GDP.DEFL.ZS?locations=US).
- Prices from Nemet (2009) and Farmer & Lafond (2016) have been converted to {LATEST_YEAR} US$ using the US GDP deflator (linked series), to account for the effects of inflation. The deflator data is available from the [World Bank World Development Indicators](https://data.worldbank.org/indicator/NY.GDP.DEFL.ZS.AD?locations=US).
cost_source:
title: Source for each value of cost data
unit: ''
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,24 @@
# Get paths and naming conventions for current data step.
paths = PathFinder(__file__)

# Conversion factors.
# IRENA costs are given in the latest year's USD, so we convert other costs to the same currency.
LATEST_YEAR = 2024
# Convert 2004 USD and 2013 USD to LATEST_YEAR USD using the World Bank's GDP deflator:
# https://data.worldbank.org/indicator/NY.GDP.DEFL.ZS.AD?locations=US
# Download the excel file, and divide the value for the latest year by the value of 2004 (79.0769459998626):
USD2004_TO_USDLATEST = 1.58
# Similarly, divide the value for the latest year by the value of 2013 (94.7705183765681):
USD2013_TO_USDLATEST = 1.32
LATEST_YEAR = 2025


def get_usd_conversion_factor(from_year: int) -> float:
"""Get the factor to convert constant US$ of a given year into constant US$ of LATEST_YEAR, using the US GDP
deflator (linked series).

NOTE: The factor is returned as a plain float so that the deflator's metadata (from WDI, which is used as an
auxiliary dataset) does not propagate into the indicators.
"""
ds_deflator = paths.load_dataset("owid_deflator")
tb_deflator = ds_deflator.read("owid_deflator")
us = tb_deflator[tb_deflator["country"] == "United States"].set_index("year")["gdp_deflator_linked"]
factor = float(us.loc[LATEST_YEAR] / us.loc[from_year])
assert 1.0 <= factor < 2.0, f"Unexpected US deflator factor from {from_year} to {LATEST_YEAR}."

return factor


def prepare_capacity_data(tb_nemet: Table, tb_irena_capacity: Table) -> Table:
Expand Down Expand Up @@ -85,7 +94,7 @@ def prepare_cost_data(tb_nemet: Table, tb_irena_cost: Table, tb_farmer_lafond: T
tb_nemet_cost = tb_nemet[["year", "cost"]].copy()
tb_nemet_cost["cost_source"] = "Nemet (2009)"
# Costs are given in "2004 USD/watt", so we need to convert them to the latest year USD.
tb_nemet_cost["cost"] *= USD2004_TO_USDLATEST
tb_nemet_cost["cost"] *= get_usd_conversion_factor(from_year=2004)
tb_nemet_cost["cost"].metadata.unit = f"constant {LATEST_YEAR} US$ per watt"

# Prepare solar photovoltaic cost data from Farmer & Lafond (2016).
Expand All @@ -97,16 +106,18 @@ def prepare_cost_data(tb_nemet: Table, tb_irena_cost: Table, tb_farmer_lafond: T
)
tb_farmer_lafond["cost_source"] = "Farmer & Lafond (2016)"
# Costs are given in "2013 USD/Wp", so we need to convert them to the latest year USD.
tb_farmer_lafond["cost"] *= USD2013_TO_USDLATEST
tb_farmer_lafond["cost"] *= get_usd_conversion_factor(from_year=2013)
tb_farmer_lafond["cost"].metadata.unit = f"constant {LATEST_YEAR} US$ per watt"

# Prepare solar photovoltaic cost data from IRENA.
tb_irena_cost = tb_irena_cost.drop(columns="country", errors="raise")

tb_irena_cost["cost_source"] = "IRENA"
# Costs are given in latest year "USD/W", so we do not need to correct them.
# NOTE: The series ends in 2024 (IRENA discontinued the global module price index after the Renewable Power
# Generation Costs in 2024 report), but its values are expressed in constant US$ of LATEST_YEAR.
error = "IRENA data has changed, prices may need to be deflated to the latest year."
assert tb_irena_cost["year"].max() == LATEST_YEAR, error
assert tb_irena_cost["cost"].metadata.unit == f"constant {LATEST_YEAR} US$ per watt", error

# Combine Nemet (2009) and Farmer & Lafond (2016), prioritizing the former.
combined = combine_two_overlapping_dataframes(df1=tb_nemet_cost, df2=tb_farmer_lafond, index_columns="year")
Expand Down
Loading
Loading