Geothermal integration - #39
Open
doneachh wants to merge 15 commits into
Open
Conversation
doneachh
commented
Apr 24, 2026
doneachh
left a comment
doneachh
left a comment
Owner
Author
There was a problem hiding this comment.
@SalkinWho great work 🔥🚀 just some minor remarks. Also dont forget to add your new config entries to config default and commit it as well :)
Comment on lines
+791
to
+795
| #geothermal_potential=( | ||
| # "resources/" + RDIR + "geothermal_potentials_elec_s{simpl}_{clusters}.csv" | ||
| # if "geothermal" in config["electricity"]["extendable_carriers"]["Generator"] | ||
| # else [] | ||
| #) |
Owner
Author
There was a problem hiding this comment.
if the other workflow works -> delete it
| **config["export"], | ||
| ), | ||
|
|
||
| #if "geothermal" in config["electricity"]["extendable_carriers"]["Generator"]: |
Owner
Author
There was a problem hiding this comment.
if we use the other workflow -> delete it
| sector=config["sector"], | ||
| snapshots=config["snapshots"], | ||
| input: | ||
| egs_input="/mnt/c/Users/nikla/Documents/Git/pypsa-earth-geothermal/data/egs_input_data.csv", |
Owner
Author
There was a problem hiding this comment.
would recommend to use "data/egs_input_dataset.csv"
Where do we get the data from rn? Do we download it manually and store it there?
If yes would recommend something like this:
| network=RESDIR | ||
| + "prenetworks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export.nc", | ||
| costs="resources/" + RDIR + "costs_{planning_horizons}_sec.csv", | ||
| + "prenetworks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export_import.nc", |
Owner
Author
There was a problem hiding this comment.
would recommend to add a wildcard {h2import} representing the configurated import value
| output: | ||
| RESDIR | ||
| + "postnetworks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export.nc", | ||
| + "postnetworks/elec_s{simpl}_{clusters}_ec_l{ll}_{opts}_{sopts}_{planning_horizons}_{discountrate}_{demand}_{h2export}export_import.nc", |
| # set the pipelines efficiency and the electricity required by the pipeline for compression | ||
| set_length_based_efficiency(n, "H2 pipeline", " H2", transmission_efficiency) | ||
|
|
||
| #Geothermie Generators hinzufügen |
Owner
Author
There was a problem hiding this comment.
Wenn nicht mehr benötigt -> löschen
Comment on lines
+349
to
+351
| # TODO: download geothermal potential automatically and read it in here | ||
| geothermal_pot = read_csv_nafix( | ||
| snakemake.input.geothermal_potential, |
Owner
Author
There was a problem hiding this comment.
wenn nicht mehr benötigt -> löschen
| attach_stores(n, costs, config) | ||
| attach_hydrogen_pipelines(n, costs, config, transmission_efficiency) | ||
|
|
||
| attach_extendable_generators(n, costs, geothermal_pot) |
Owner
Author
There was a problem hiding this comment.
Wenn nicht mehr benötigt -> löschen
| ports_sel = ports.loc[~ports[gcol].duplicated(keep="first")].set_index(gcol) | ||
|
|
||
| # Select the hydrogen buses based on nodes with ports | ||
| hydrogen_buses_ports = n.buses.loc[ports_sel.index + " H2"] |
Owner
Author
There was a problem hiding this comment.
For which case did we have a problem here? just bc I'm interested - solution is fine
doneachh
commented
May 4, 2026
doneachh
left a comment
doneachh
left a comment
Owner
Author
There was a problem hiding this comment.
@SalkinWho looks great! just 4 minor things and we're ready to merge 🚀
| logger = create_logger(__name__) | ||
|
|
||
|
|
||
| CRF = 0.09 # calculated with 0.08 interest rate see Franzmann et al. |
Owner
Author
There was a problem hiding this comment.
think about including them into the config / make them customizable
Owner
Author
There was a problem hiding this comment.
@SalkinWho I think you mentioned, that this CRF is based on a paper and has to be this value for our calculations. Please indicate that with a comment
| ports_sel = ports.loc[~ports[gcol].duplicated(keep="first")].set_index(gcol) | ||
|
|
||
| # Select the hydrogen buses based on nodes with ports | ||
| hydrogen_buses_ports = n.buses.loc[ports_sel.index + " H2"] |
| bus_regions_onshore, | ||
| bus_regions_offshore, | ||
| bus_regions_all, | ||
| model_borders, #muss noch entfernt werden |
Owner
Author
|
@ekatef is it possible for you to give us your thoughts about this implementation an potential further improvements? Also your thoughts on what we have to adjust to be able to merge it in PyPSA-Earth main would be much appreciated :) |
doneachh
commented
Aug 2, 2026
Owner
Author
There was a problem hiding this comment.
Hi @SalkinWho!
Unfortunately, I have no write access to your branch. Because of that, here some last review comments. After resolving them, we have to do the following:
- Merge it with the latest doneachh/geothemal_integration branch. (right now we have some merge issues, which i can't resolve as i do not have write access to your branch)
- Create a PR at PyPSA-meets-Earth (I can do that, if you want)
- Create a release note mentioning all the new features within the PR (global EGS potentials for power and heat, hydrogen imports + optimization of memory usage for global_buildings) (I can do that as well, if you want)
- At some point, adding the doi within the config and the scripts from our paper/preprint (I can do that as well, if you want)
Please resolve the last comments and merge the latest doneachh/geothermal_integration branch so that we can continue making this implementation accessible for the whole community :)
PS: Feel free to also add your calculated CAGR, fuel_shares and other demand data as well
Comment on lines
+103
to
+108
| """ | ||
| Add pipeline-based H2 import generators. | ||
|
|
||
| Pipeline imports are modeled as extendable generators with | ||
| ``p_nom_max = p_nom_max_pipeline``. | ||
| """ |
Owner
Author
There was a problem hiding this comment.
Suggested change
| """ | |
| Add pipeline-based H2 import generators. | |
| Pipeline imports are modeled as extendable generators with | |
| ``p_nom_max = p_nom_max_pipeline``. | |
| """ | |
| """ | |
| Add pipeline-based H2 import generators to the network. | |
| Pipeline imports are modeled as extendable generators attached to | |
| the H2 buses of the specified import nodes, using ``p_nom_max_pipeline`` | |
| as their maximum nominal capacity limit. | |
| Required columns | |
| ---------------- | |
| bus : str | |
| Base bus name without the H2 suffix. | |
| p_nom_max_pipeline : float | |
| Maximum pipeline-based H2 import capacity at the node. | |
| Parameters | |
| ---------- | |
| n : pypsa.Network | |
| Network to which the pipeline-based H2 import generators will be added. | |
| import_nodes : pandas.DataFrame | |
| DataFrame containing prepared H2 import node information. | |
| Returns | |
| ------- | |
| None | |
| """ |
Comment on lines
+135
to
+140
| """ | ||
| Add port-based H2 import generators. | ||
|
|
||
| Port imports are modeled as extendable generators with | ||
| ``p_nom_max = p_nom_max_port``. | ||
| """ |
Owner
Author
There was a problem hiding this comment.
Suggested change
| """ | |
| Add port-based H2 import generators. | |
| Port imports are modeled as extendable generators with | |
| ``p_nom_max = p_nom_max_port``. | |
| """ | |
| """ | |
| Add port-based H2 import generators. | |
| Port imports are modeled as extendable generators attached to | |
| the H2 buses of the specified import nodes, using ``p_nom_max_port`` | |
| as their maximum nominal capacity limit. | |
| Required columns | |
| ---------------- | |
| bus : str | |
| Base bus name without the H2 suffix. | |
| h2_bus : str | |
| Name of the H2 bus to which the generator is attached. | |
| p_nom_max_port : float | |
| Maximum port-based H2 import capacity at the node. | |
| Parameters | |
| ---------- | |
| n : pypsa.Network | |
| Network to which the port-based H2 import generators will be added. | |
| import_nodes : pandas.DataFrame | |
| DataFrame containing prepared H2 import node information. | |
| Returns | |
| ------- | |
| None | |
| """ |
Comment on lines
+231
to
+232
|
|
||
|
|
| logger = create_logger(__name__) | ||
|
|
||
|
|
||
| CRF = 0.09 # calculated with 0.08 interest rate see Franzmann et al. |
Owner
Author
There was a problem hiding this comment.
@SalkinWho I think you mentioned, that this CRF is based on a paper and has to be this value for our calculations. Please indicate that with a comment
| @@ -1,115 +1,236 @@ | |||
| # -*- coding: utf-8 -*- | |||
| # SPDX-FileCopyrightText: PyPSA-Earth Authors | |||
| # SPDX-FileCopyrightText: PyPSA-Earth Authors | |||
Owner
Author
There was a problem hiding this comment.
Suggested change
| # SPDX-FileCopyrightText: PyPSA-Earth Authors | |
| # SPDX-FileCopyrightText: PyPSA-Earth Authors |
Comment on lines
6
to
11
| This script processes global building data to calculate solar rooftop area. | ||
|
|
||
| Memory-efficient version: | ||
| The input parquet file is read in batches and each batch is spatially joined to | ||
| the onshore regions. Only the aggregated rooftop area per region is kept in RAM. | ||
| """ |
Owner
Author
There was a problem hiding this comment.
Suggested change
| This script processes global building data to calculate solar rooftop area. | |
| Memory-efficient version: | |
| The input parquet file is read in batches and each batch is spatially joined to | |
| the onshore regions. Only the aggregated rooftop area per region is kept in RAM. | |
| """ | |
| Calculate usable solar rooftop area per region from global building data. | |
| This script processes global building footprint data in Parquet format and | |
| aggregates the estimated usable rooftop area for each onshore bus region in a | |
| specified country. | |
| To process large datasets memory-efficiently: | |
| 1. The building Parquet file is read sequentially in batches. | |
| 2. Useful rooftop areas are calculated via a pre-compiled lookup table. | |
| 3. Buildings are mapped to onshore region geometries using spatial joins | |
| (point-in-polygon with a nearest-neighbor fallback for near-border points). | |
| 4. Results are accumulated incrementally per region, keeping memory consumption low. | |
| Inputs | |
| ------ | |
| snakemake.input.country_buildings : str | |
| Path to the Parquet file containing building data with columns 'area', 'x', and 'y'. | |
| snakemake.input.regions_onshore : str | |
| Path to the vector dataset (GeoJSON/GPKG/SHP) containing onshore regions. | |
| Outputs | |
| ------- | |
| snakemake.output.solar_rooftop_layout : str | |
| Path to the output CSV file containing aggregated useful rooftop area per region. | |
| Parameters | |
| ---------- | |
| snakemake.params.crs : dict | |
| Dictionary with keys 'distance_crs' (projected CRS) and 'geo_crs' (geographic CRS). | |
| snakemake.params.install_ratio : dict | |
| Mapping of building area thresholds to usable rooftop fractions. | |
| snakemake.params.solar_rooftop_enable : bool | |
| Flag indicating whether to execute the calculation or output an empty CSV. | |
| snakemake.params.tolerance : float | |
| Maximum distance in km to assign unmatched buildings to the nearest region. | |
| GLOBAL_BUILDINGS_BATCH_SIZE : env variable, optional | |
| Number of rows per Parquet batch (default: 250,000). | |
| """ |
Comment on lines
+86
to
+92
| """ | ||
| Calculates the area and centroid of buildings from Microsoft Global Buildings | ||
| quadrant files. | ||
|
|
||
| This version processes one quadrant at a time and explicitly frees memory | ||
| after each quadrant. | ||
| """ |
Owner
Author
There was a problem hiding this comment.
Suggested change
| """ | |
| Calculates the area and centroid of buildings from Microsoft Global Buildings | |
| quadrant files. | |
| This version processes one quadrant at a time and explicitly frees memory | |
| after each quadrant. | |
| """ | |
| """ | |
| Calculate the footprint area and centroid coordinates of buildings. | |
| Processes Microsoft Global Buildings quadrant files sequentially (one quadrant | |
| at a time) to keep memory usage low. Converts raw JSON geometries into spatial | |
| geometries, reprojects them to derive accurate surface area and centroid coordinates, | |
| and yields the results chunk-by-chunk. | |
| Required columns | |
| ---------------- | |
| Url : str | |
| URL or file path to the Microsoft Global Buildings JSON/GeoJSON-Lines quadrant file. | |
| Parameters | |
| ---------- | |
| df : pandas.DataFrame | |
| DataFrame containing links to building quadrant files in a 'Url' column. | |
| crs : dict | |
| Dictionary containing CRS definitions with keys 'geo_crs' (geographic CRS, | |
| e.g., EPSG:4326), 'area_crs' (equal-area projection for area calculation), | |
| and 'distance_crs' (projected CRS for centroid calculations). | |
| Yields | |
| ------ | |
| pandas.DataFrame | |
| Dataframe for a single quadrant containing: | |
| - area (int64): Building footprint area. | |
| - x (float64): Centroid longitude/x-coordinate in geographic CRS. | |
| - y (float64): Centroid latitude/y-coordinate in geographic CRS. | |
| """ |
Comment on lines
+158
to
+167
| """ | ||
| Downloads global building data for a specific country and writes the processed | ||
| result incrementally to a single parquet file. | ||
|
|
||
| Compared to the original implementation, this avoids | ||
|
|
||
| pd.concat(list(...)) | ||
|
|
||
| and therefore does not keep all building quadrants in RAM at once. | ||
| """ |
Owner
Author
There was a problem hiding this comment.
Suggested change
| """ | |
| Downloads global building data for a specific country and writes the processed | |
| result incrementally to a single parquet file. | |
| Compared to the original implementation, this avoids | |
| pd.concat(list(...)) | |
| and therefore does not keep all building quadrants in RAM at once. | |
| """ | |
| """ | |
| Download global building data for a country and write results incrementally. | |
| Fetches Microsoft Global Buildings quadrant URLs for the specified country, | |
| processes each quadrant using ``get_building_area_center``, and streams the | |
| processed building records directly into a Parquet file to avoid keeping all | |
| quadrants in RAM at once. | |
| Parameters | |
| ---------- | |
| country_code : str | |
| Country code used to filter building download URLs (e.g. 'NG', 'DE'). | |
| country_buildings_fn : str or path-like | |
| Output file path where the final Parquet file will be saved. | |
| crs : dict | |
| Dictionary containing CRS definitions with keys 'geo_crs', 'area_crs', | |
| and 'distance_crs'. | |
| update : bool, optional | |
| Whether to force updating/re-downloading the building dataset index URL | |
| file (default is False). | |
| Returns | |
| ------- | |
| None | |
| Raises | |
| ------ | |
| Exception | |
| Re-raises any error encountered during processing after cleaning up | |
| temporary output files. | |
| """ |
|
|
||
| raise | ||
|
|
||
| def get_building_area_center_old(df, crs): |
Owner
Author
There was a problem hiding this comment.
do we still need this method?
|
|
||
|
|
||
| def download_global_buildings(country_code, country_buildings_fn, crs, update=False): | ||
| def download_global_buildings_old(country_code, country_buildings_fn, crs, update=False): |
Owner
Author
There was a problem hiding this comment.
do we still need this method?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes # (if applicable)
Changes proposed in this Pull Request
Checklist
envs/environment.yamlanddoc/requirements.txt.config.default.yamlandconfig.tutorial.yaml.test/(note tests are changing the config.tutorial.yaml)doc/configtables/*.csvand line references are adjusted indoc/user-guide/configuration.mdanddoc/tutorials/electricity-model.md.doc/assets/scripts/extract_config_snippets.pyaccordingly.configs/bundle_config.yaml.doc/release-notes.mdis amended in the format of previous release notes, including reference to the requested PR.