Scenario export compatible with csv loader

[mthede]

User description

Related Issue

Closes #831

Description

WIP!

To try it out, just call ´world.export()´ after setting up a simulation with whatever tool you did.

Open issues:

• Small differences in outputs (e.g. demand bid prices in market_orders and volumes for powerplants in example_01h_eom)
• Should all values be written or only non-default values?
• Is writting dsm-unit specific attributes to each line in csv causing issues?

Checklist

• Documentation updated (docstrings, READMEs, user guides, inline comments, docs folder updates, etc.)
• New unit/integration tests added (if applicable)
• Changes noted in release notes (if any)
• Consent to release this PR's code under the GNU Affero General Public License v3.0

Additional Notes (optional)

This PR still needs work and through testing, in particular on:

• DSM unit export and compatibility
• Exchange units
• Whether re-running the simulation leads to the same results as the original simulation
• If all relevant data has been exported and no values/defaults were changed

---

PR Type

Enhancement, Bug fix, Tests, Documentation

---

Description

• Add CSV-compatible scenario export

• Serialize units, forecasts, grid data

• Fix PyPSA and cost export conventions

• Add round-trip export tests and docs

---

Diagram Walkthrough

flowchart LR
  W["World setup"]
  E["CSV exporter"]
  F["Scenario folder"]
  L["CSV loader"]
  W -- "export()" --> E
  E -- "writes config, units, time series" --> F
  F -- "reloads" --> L

Loading

File Walkthrough

	Relevant files
Enhancement	2 files exporter_csv.py Add CSV scenario export implementation                                      +762/-0  world.py Add `World.export()` scenario method                                          +24/-0
Bug fix	2 files loader_pypsa.py Align PyPSA scenario timing setup                                                +10/-6    powerplant.py Normalize startup costs in unit export                                      +3/-3
Tests	2 files test_world.py Add scenario export round-trip tests                                          +204/-0  config.yaml Add forecast export test scenario                                                +35/-0
Documentation	6 files unit_forecasts.rst Clarify forecast keep-given behavior                                          +15/-6    demand_df.csv.license Add demand fixture license metadata                                            +3/-0      demand_units.csv.license Add demand units fixture license                                                  +3/-0      forecasts_df.csv.license Add forecasts fixture license metadata                                      +3/-0      fuel_prices_df.csv.license Add fuel prices fixture license                                                    +3/-0      powerplant_units.csv.license Add powerplant fixture license metadata                                    +3/-0

---

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 5666f76)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 831 - Partially compliant Compliant requirements: Adds a World.export() entry point. Exports config, units, forecasts, demand, availability, fuel prices, exchange volumes, and grid CSV files. Adds round-trip tests for standard CSV scenarios, forecast export behavior, and PyPSA export. Adds documentation related to forecast export behavior. Non-compliant requirements: Grid data is not reliably re-loadable when the original market param_dict only contains grid_data. Requires further human verification: Whether exported and reloaded simulations produce equivalent numerical results across all supported loaders and unit types. Whether DSM and exchange unit exports are fully compatible for realistic scenarios.

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Grid Reload network_path is only written when grid_data was present and at least one other param_dict entry remains after removing it. If a market config contains only grid_data, the exporter still writes buses.csv and lines.csv, but the exported config.yaml has no param_dict.network_path, so the CSV loader will not know to reload the grid data. if has_grid_data and param_dict: param_dict["network_path"] = "." if param_dict: market_dict["param_dict"] = param_dict

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 5666f76

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Preserve exported grid path When param_dict contains only grid_data, network_path is never written, even though _export_grid creates the grid CSV files. Add network_path whenever grid_data was present so the CSV loader can find the exported network data. assume/scenario/exporter_csv.py [267-272] # Add network_path if grid_data was present -if has_grid_data and param_dict: +if has_grid_data: param_dict["network_path"] = "." if param_dict: market_dict["param_dict"] = param_dict Suggestion importance[1-10]: 8 __ Why: This identifies a real round-trip bug: when param_dict only contains grid_data, the exported config.yaml omits network_path, so the CSV loader may not find the exported grid files. The proposed change accurately fixes the condition without affecting non-grid parameters.	Medium
	Avoid ambiguous index truthiness Evaluating first_unit.forecaster.index as a boolean can raise for pandas indexes because their truth value is ambiguous. Store the index and explicitly check for None and non-empty length before reading freq. assume/scenario/exporter_csv.py [155-157] -if hasattr(first_unit.forecaster, "index") and first_unit.forecaster.index: - freq = first_unit.forecaster.index.freq +index = getattr(first_unit.forecaster, "index", None) +if index is not None and len(index) > 0: + freq = index.freq if freq: Suggestion importance[1-10]: 7 __ Why: The boolean check on first_unit.forecaster.index can fail for pandas-like indexes with ambiguous truth values. Explicitly checking index is not None and len(index) > 0 is a robust fix for a likely export-time failure.	Medium
	Guard startup cost normalization as_dict() will now fail with a division error when self.max_power is 0 or unset. Guard the normalization so exporting edge-case power plants does not crash. assume/units/powerplant.py [396-398] -"hot_start_cost": self.hot_start_cost / self.max_power, -"warm_start_cost": self.warm_start_cost / self.max_power, -"cold_start_cost": self.cold_start_cost / self.max_power, +"hot_start_cost": ( + self.hot_start_cost / self.max_power + if self.max_power + else self.hot_start_cost +), +"warm_start_cost": ( + self.warm_start_cost / self.max_power + if self.max_power + else self.warm_start_cost +), +"cold_start_cost": ( + self.cold_start_cost / self.max_power + if self.max_power + else self.cold_start_cost +), Suggestion importance[1-10]: 4 __ Why: Guarding division by self.max_power avoids crashes for edge-case zero or missing capacities, but such power plants may be invalid and returning unnormalized startup costs has somewhat unclear semantics. This is a reasonable defensive improvement with limited impact.	Low

---

Previous suggestions

Suggestions up to commit ecc4c48

Category	Suggestion	Impact
Possible issue	Preserve exported grid path network_path is currently omitted when grid_data is the only entry in param_dict, so the exported grid CSVs may not be discoverable by the CSV loader. Add network_path whenever grid_data was present, regardless of whether other parameters remain. assume/scenario/exporter_csv.py [267-269] # Add network_path if grid_data was present -if has_grid_data and param_dict: +if has_grid_data: param_dict["network_path"] = "." Suggestion importance[1-10]: 8 __ Why: This addresses a real round-trip bug: when grid_data is the only param_dict entry, network_path is omitted and the exported buses.csv/lines.csv may not be loaded back. The improved condition accurately fixes the issue without changing unrelated parameters.	Medium
	Avoid ambiguous index truthiness Avoid evaluating first_unit.forecaster.index directly because pandas.Index truthiness is ambiguous and can raise during export. Store the index and check it explicitly before reading freq. assume/scenario/exporter_csv.py [153-156] if world.units: first_unit = next(iter(world.units.values())) - if hasattr(first_unit.forecaster, "index") and first_unit.forecaster.index: - freq = first_unit.forecaster.index.freq + index = getattr(first_unit.forecaster, "index", None) + if index is not None and len(index) > 0: + freq = index.freq Suggestion importance[1-10]: 7 __ Why: This is a valid fix because evaluating first_unit.forecaster.index directly can fail for a pandas.Index due to ambiguous truthiness. The suggested code preserves the intent while making _infer_time_step_or_none safer during export.	Medium
	Guard startup cost division Dividing by self.max_power can raise ZeroDivisionError for zero-capacity units and break as_dict() callers, including scenario export. Guard the division so serialization remains safe when max_power is zero. assume/units/powerplant.py [396-398] -"hot_start_cost": self.hot_start_cost / self.max_power, -"warm_start_cost": self.warm_start_cost / self.max_power, -"cold_start_cost": self.cold_start_cost / self.max_power, +"hot_start_cost": self.hot_start_cost / self.max_power if self.max_power else self.hot_start_cost, +"warm_start_cost": self.warm_start_cost / self.max_power if self.max_power else self.warm_start_cost, +"cold_start_cost": self.cold_start_cost / self.max_power if self.max_power else self.cold_start_cost, Suggestion importance[1-10]: 5 __ Why: The suggestion is technically valid because self.max_power being zero would make as_dict() raise during serialization. Its impact is limited because zero-capacity PowerPlant units are likely an edge case, and the fallback semantics for startup costs are debatable.	Low

Suggestions up to commit db6c85a

Category	Suggestion	Impact
Possible issue	Preserve exported grid path Add network_path whenever grid_data was present, even if all other param_dict entries were filtered out. Otherwise markets whose param_dict only contains grid_data will export buses.csv and lines.csv but the reloaded config will not point the loader to them. assume/scenario/exporter_csv.py [267-272] # Add network_path if grid_data was present -if has_grid_data and param_dict: +if has_grid_data: param_dict["network_path"] = "." if param_dict: market_dict["param_dict"] = param_dict Suggestion importance[1-10]: 7 __ Why: This correctly identifies that network_path is omitted when param_dict contains only grid_data, making exported grid CSVs unusable on reload. The proposed change is accurate and fixes a real round-trip export issue.	Medium
	Avoid optional column crash Use a tolerant drop for technology. If an Exchange unit does not expose this column, the current export path raises a KeyError and prevents the whole scenario from being exported. assume/scenario/exporter_csv.py [372] -df = pd.DataFrame(data).set_index("name").drop("technology", axis=1) +df = pd.DataFrame(data).set_index("name").drop("technology", axis=1, errors="ignore") Suggestion importance[1-10]: 6 __ Why: Using errors="ignore" is a safe improvement because _unit_to_dict may not always produce a technology column for Exchange units. This prevents a possible KeyError without changing behavior when the column exists.	Low
	Use contributing series index Do not use the loop variable unit after iteration to choose the dataframe index. If the last demand unit did not contribute a series, or has a different index, the exported demand_df.csv can be wrong or fail; apply the same pattern to the other time-series exporters that use the last loop variable. assume/scenario/exporter_csv.py [657-667] series_dict = {} +index = None for unit in demand_units: if hasattr(unit.forecaster, "demand"): - # Use the forecaster's index as the datetime index + # Use the contributing forecaster's index as the datetime index series_dict[unit.id] = -unit.forecaster.demand + if index is None: + index = unit.forecaster.index.as_datetimeindex() -if series_dict: - df = pd.DataFrame( - series_dict, index=unit.forecaster.index.as_datetimeindex() - ).rename_axis("datetime") +if series_dict and index is not None: + df = pd.DataFrame(series_dict, index=index).rename_axis("datetime") df.to_csv(scenario_path / "demand_df.csv", index=True) Suggestion importance[1-10]: 6 __ Why: The suggestion correctly avoids relying on the final loop variable unit when building demand_df.csv, which can select a non-contributing or mismatched forecaster index. It is a valid robustness fix, though it only partially addresses the same pattern in other exporters.	Low

Suggestions up to commit ff4854d

Category	Suggestion	Impact
Possible issue	Preserve exported grid linkage Add network_path whenever grid_data was present, even if grid_data was the only entry in param_dict. Otherwise exported grid files can be written but the reloaded scenario will not know to load them. assume/scenario/exporter_csv.py [267-269] # Add network_path if grid_data was present -if has_grid_data and param_dict: +if has_grid_data: param_dict["network_path"] = "." Suggestion importance[1-10]: 8 __ Why: This is a real functional issue: when grid_data is the only entry in market_config.param_dict, _export_grid can write grid CSVs but _serialize_markets_config omits network_path, preventing reload from finding them. The proposed change accurately fixes the condition.	Medium
	Avoid ambiguous index checks Avoid evaluating first_unit.forecaster.index as a boolean because pandas indexes raise on ambiguous truth values. Check explicitly for None and non-empty length before reading freq. assume/scenario/exporter_csv.py [153-156] if world.units: first_unit = next(iter(world.units.values())) - if hasattr(first_unit.forecaster, "index") and first_unit.forecaster.index: - freq = first_unit.forecaster.index.freq + index = getattr(first_unit.forecaster, "index", None) + if index is not None and len(index) > 0: + freq = index.freq Suggestion importance[1-10]: 7 __ Why: The existing boolean check on first_unit.forecaster.index can fail for pandas-like indexes with ambiguous truth values. The suggested explicit None and length check is accurate and improves export robustness.	Medium
	Use matching series index Do not use the loop variable unit after the loop to choose the DataFrame index; it may refer to a different unit than the one that supplied fuel_prices. Capture the index from the first unit that contributes data so the exported CSV aligns with the actual series. assume/scenario/exporter_csv.py [667-678] all_fuel_prices = {} +index = None for unit in world.units.values(): if hasattr(unit.forecaster, "fuel_prices") and unit.forecaster.fuel_prices: for fuel, series in unit.forecaster.fuel_prices.items(): if fuel not in all_fuel_prices: all_fuel_prices[fuel] = series + if index is None: + index = unit.forecaster.index.as_datetimeindex() -if all_fuel_prices: - df = pd.DataFrame( - all_fuel_prices, index=unit.forecaster.index.as_datetimeindex() - ).rename_axis("datetime") +if all_fuel_prices and index is not None: + df = pd.DataFrame(all_fuel_prices, index=index).rename_axis("datetime") Suggestion importance[1-10]: 7 __ Why: Using the loop variable unit after iteration can select an unrelated forecaster index for fuel_prices_df.csv. Capturing the index from the first contributing unit is a reasonable fix for alignment and avoids exporting with an arbitrary last unit's index.	Medium

Suggestions up to commit 6033a24

Category	Suggestion	Impact
Possible issue	Avoid overwriting exported units Exporting steel_units and then hydrogen_units/steam_units to the same industrial_dsm_units.csv overwrites the first file, causing silent data loss. Combine all industrial DSM units and write the CSV once. assume/scenario/exporter_csv.py [387-391] -if steel_units: - _export_units_to_csv(world, steel_units, scenario_path, "industrial_dsm_units") -if hydrogen_units or steam_units: - combined = hydrogen_units + steam_units - _export_units_to_csv(world, combined, scenario_path, "industrial_dsm_units") +industrial_units = steel_units + hydrogen_units + steam_units +if industrial_units: + _export_units_to_csv( + world, industrial_units, scenario_path, "industrial_dsm_units" + ) Suggestion importance[1-10]: 8 __ Why: This is a valid correctness issue: exporting steel_units and then hydrogen_units/steam_units to the same industrial_dsm_units.csv can overwrite previously exported units. Combining all industrial DSM units before writing prevents silent data loss.	Medium
	Preserve exported grid reference When param_dict contains only grid_data, network_path is never written because param_dict becomes empty after skipping grid_data. This makes the exported buses.csv and lines.csv unreachable for the CSV loader. assume/scenario/exporter_csv.py [264-269] # Add network_path if grid_data was present -if has_grid_data and param_dict: +if has_grid_data: param_dict["network_path"] = "." if param_dict: market_dict["param_dict"] = param_dict Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies that when param_dict only contains grid_data, network_path is omitted even though grid CSVs are exported. This can break round-tripping grid data through the CSV loader.	Medium
	Avoid ambiguous index checks Checking first_unit.forecaster.index in a boolean context can raise for pandas indexes, and relying only on the first unit can miss a valid index on another unit. Iterate over units and explicitly check for a non-empty index before reading freq. assume/scenario/exporter_csv.py [150-171] -if world.units: - first_unit = next(iter(world.units.values())) - if hasattr(first_unit.forecaster, "index") and first_unit.forecaster.index: - freq = first_unit.forecaster.index.freq - if freq: - # Convert timedelta to pandas frequency string - if isinstance(freq, timedelta): - total_seconds = int(freq.total_seconds()) - if total_seconds == 3600: - return "1h" - elif total_seconds == 1800: - return "30min" - elif total_seconds == 60: - return "1min" - else: - return ( - f"{total_seconds // 3600}h" - if total_seconds % 3600 == 0 - else f"{total_seconds // 60}min" - ) - return str(freq) +for unit in world.units.values(): + index = getattr(unit.forecaster, "index", None) + if index is None or len(index) == 0: + continue + + freq = getattr(index, "freq", None) + if freq: + # Convert timedelta to pandas frequency string + if isinstance(freq, timedelta): + total_seconds = int(freq.total_seconds()) + if total_seconds == 3600: + return "1h" + elif total_seconds == 1800: + return "30min" + elif total_seconds == 60: + return "1min" + else: + return ( + f"{total_seconds // 3600}h" + if total_seconds % 3600 == 0 + else f"{total_seconds // 60}min" + ) + return str(freq) return None Suggestion importance[1-10]: 7 __ Why: The boolean check on first_unit.forecaster.index can fail for pandas-style indexes, and checking only the first unit may miss a usable index. Iterating over units with explicit None and length checks makes _infer_time_step_or_none more robust.	Medium

[github-actions]

Persistent review updated to latest commit 0493bbd

[codecov]

Codecov Report

❌ Patch coverage is 68.11224% with 125 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.57%. Comparing base (ad4cb29) to head (ecc4c48).

Files with missing lines	Patch %	Lines
assume/scenario/exporter_csv.py	67.70%	125 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #832      +/-   ##
==========================================
- Coverage   82.12%   81.57%   -0.55%     
==========================================
  Files          56       58       +2     
  Lines        9078     9521     +443     
==========================================
+ Hits         7455     7767     +312     
- Misses       1623     1754     +131     

Flag	Coverage Δ
pytest	81.57% <68.11%> (-0.55%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[maurerle]

I would rather have scenario/exporter_csv.py and a convenience function in world:

from scenario.exporter_csv import export_to_folder
def export(self, scenario_save_path, study_case):
   export_to_folder(self, scenario_save_path, study_case)

This way, we keep the world.py from handling IO

[reinecfi]

Edit: forecasts_df is actually temporarily saved in World which I missed in my first review.
Regarding forecasts:
One correct approach would be to save the forecast algorithm names on a per unit basis (as already in the PR addressed) and copy the forecasts_df.csv file to the exported files which is stored in the World.scenario_data
Following this, _get_forecast_algorithms can be safely removed.
_export_forecasts_df should thenbe adjusted to save the temporarily stored forecasts_df from world

Below I will try to explain the reasoning behind it as best as I can for those that are interested. Sorry if it is quite long, but at least its documented.
Currently, the PR deals with all three regular ways of selecting forecasts algorithms in the input (config.yaml, unit specific, forecasts_df). From my current understanding, this results in some problems in the proposed exporter. I will explain the differences and problems starting with the forecasts_df.csv as it will supersede the otheres by default.

1. (_export_forecasts_df) Per default, if in forecasts_df there exists a forecast for any forecast type (e.g. "price_EOM") this will supersede all other forecast algorithms for "price" on EOM. The proposed Exporter will naively choose the first price forecasts it gets (even if it was not from the forecasts_df.csv in the first place). Thus each unit will automatically have the same forecast for "price_EOM" after exporting (even if before they had different ones). Same applies for residual_load.
2. specifying forecasts in the individual units csv (powerplant_units.csv, ...) is the most versatile option, that will give each unit the opportunity to have its own forecasting algorithm. The Exporter currently will infer all used (and unused) forecast algorithms per unit and save them. While this might lead to a different looking csv file in the exported csv then in the input / imported one, this should give the correct behaviour as long this forecast algorithm was not superseded by forecasts_df in the scenario loading. Using forecasts_df.csv saving and this per unit forecast_algorithms will thus restore functionality
3. (_get_forecast_algorithms) For the config.yaml variation, this will currently naively take the first algorithm for a specific type (e.g. "price") and select it as default in the config. While in theory the input version of the config.yaml forecast algorithms could be found in World.scenario_data, it would be overwritten by 1. and 2. in the current exporter. It further does not support unit specific forecasts and thus is not suitable in general case.
4. There is a fourth possibility to add a forecast to a forecaster that is ignored by this exporter currently. A forecaster can get a specific forecast in the init which will be kept. As this functionality is not used in the regular loaders and mostly for testing, development and very customized usage, this case should be fine to ignore. This is especially the case, as it was "manually" added to the simulation in the first place, so it should be possible to add it "manually" again for the next usage after exporting.

[reinecfi]

Another point:
There is a lot of hasattr(unit.forecaster, ...):
Each UnitForecaster should have the attributes "forecaster_algorithms", "availability", "price", "residual_load", this could be taken as granted. Similar for the specific DemandForecaster and ExchangeForecaster with attributes "demand" and "volume_ex/import". I think in general we should assume that the unit specific fields of a corresponding forecaster exist.

[github-actions]

Persistent review updated to latest commit 6033a24

[github-actions]

Persistent review updated to latest commit ff4854d

[github-actions]

Persistent review updated to latest commit db6c85a

[Manish-Khanra]

@mthede I traced the DSM export path (_dsm_unit_to_rows) against what the CSV loader actually reads back (load_dsm_units). The two example fixtures (example_01h building, example_03 steel plant) round trip works fine, but a few unit level attributes are silently dropped and reset to defaults on reload. e.g. Building is_prosumer, set is_prosumer = Yes on building A360 in example_01h -> export -> reload comes back as No (the default). The building loses its prosumer/PV feed-in behavior.

2. dsm_unit_to_rows only writes three unit-level attrs. for attr in ["objective", "flexibility_measure", "cost_tolerance"]. But load_dsm_units treats is_prosumer, congestion_threshold and peak_load_cap as common columns too.
3. the non DSM units go through _unit_to_dict and would catch these. Only the DSM path hardcodes the list. A suggestion would be to align the DSM common attr list with the loader's common_columns. Somethings like below:

for attr in ["objective", "flexibility_measure", "cost_tolerance",
"is_prosumer", "congestion_threshold", "peak_load_cap"]:
if hasattr(unit, attr):
common_attrs[attr] = getattr(unit, attr)
4. In _dsm_unit_to_rows (assume/scenario/exporter_csv.py), the main attribute loop skips non scalar values at lines 507–510, but the kwargs expansion just above it (lines 500–505) writes every value with no such check. Some Building components hold whole profiles in kwargs (PV uses_power_profile, EV availability_profile/charging_profile), so a full time series could get into one cell and produce a broken CSV. The current fixtures only use scalar kwargs, so it's untested.

question: DSM forecaster profiles (building inflexible load, heat demand, PV, EV availability, hydrogen demand series) don't seem to be written by _export_time_series. If those came from input CSVs, will a re-run of the exported folder reconstruct them, or is DSM profile input out of scope for this PR?

[github-actions]

Persistent review updated to latest commit ecc4c48

[github-actions]

Persistent review updated to latest commit 5666f76