scenario_loader: entsoe: new scenario loader for European Countries

[maurerle]

User description

Description

Adds an ENTSO-E Transparency Platform scenario loader to build country-level ASSUME scenarios from API data (no CSV plant lists).
Thanks to @mthede who inspired this as a simpler version which does not rely on data from OEDS

This is using a similar approach as AMIRIS to use minimum and maximum prices and splits the generation capacity into useful unit sizes with linear interpolated prices.

New code

• assume/scenario/loader_entsoe.py — load_entsoe() and dev __main__
• assume/scenario/entsoe_helper/ — ENTSO-E client (cached under ~/.assume/entsoe/), instrat.pl fuel/CO₂ prices, PSR → technology mappings
• tests/test_loader_entsoe.py (18 tests)
• Docs in docs/source/scenario_loader.rst, assume.scenario.rst
• Agent handoff: docs/dev/entsoe_loader_handoff.md

Behaviour

• Demand, generation, and installed capacity from ENTSO-E; fleet split into MW blocks with merit-order fuel spreads
• Wind/solar: max_power = installed capacity, hourly availability = generation / capacity
• Other plants: full block capacity, availability = 1
• Shared CO₂ series for all fossils (instrat ETS or 70 €/tCO₂ fallback)
• Storage units aligned with example_03 (8 h energy, 0.5 initial SOC, flexABLE storage strategy)
• Fuel price sanitization (bad coal cache → fallback) so NaN bids do not break pay-as-clear merit order

Dependency: optional extra assume-framework[entsoe] (entsoe-py, yfinance). Requires ENTSOE_API_KEY.

Checklist

• Documentation updated (docstrings, READMEs, user guides, inline comments, doc folder updates etc.)
• New unit/integration tests added (if applicable)
• Changes noted in release notes (if any)
• Consent to release this PR's code under the GNU Affero General Public License v3.0

Screenshot

• Verified 2024 DE run (entsoe_DE_2024)

---

PR Type

Enhancement, Tests, Documentation

---

Description

• Add ENTSO-E country scenario loader

• Fetch cached demand, generation, capacity

• Build units with fuel and CO2 prices

• Document loader and add test coverage

---

Diagram Walkthrough

flowchart LR
  A["ENTSO-E API"] -- "load, generation, capacity" --> B["EntsoeInterface"]
  C["instrat.pl"] -- "fuel and CO2 prices" --> D["InstratFuelPrices"]
  B -- "aggregated technologies" --> E["load_entsoe"]
  D -- "hourly price series" --> E
  E -- "demand, plants, storage" --> F["ASSUME World"]

Loading

File Walkthrough

	Relevant files
Enhancement	4 files client.py Add cached ENTSO-E data interface                                                +265/-0  fuel_prices.py Add instrat fuel price fetching                                                    +205/-0  mappings.py Define ENTSO-E technology mappings                                              +173/-0  loader_entsoe.py Add ENTSO-E scenario loader                                                            +474/-0
Tests	1 files test_loader_entsoe.py Test ENTSO-E loader behavior                                                          +396/-0
Documentation	2 files assume.scenario.rst Include ENTSO-E API documentation                                                +15/-0    scenario_loader.rst Document ENTSO-E loader usage                                                        +89/-0
Dependencies	1 files pyproject.toml Add optional ENTSO-E dependencies                                                +6/-1

---

[reinecfi]

I really like the idea of this addition. For me the changes look fine in general. Only documentation in general and especially for the the amount of hardcoded values / defaults could be improved for better understanding.

[reinecfi]

This maybe overkill for now: I feel like InstratFuelPrices has some similarities to EntsoeInterface. I wonder if we want to have a standardised way / Interface / common base class to incorporate new data from outside of ASSUME via API or downloads

[reinecfi]

There is a lot of default values here that I don't really understand. As a user I would be interested if there is some reasoning behind them.

[reinecfi]

In this line we seem sure about the sign of the demand.
Below in line 142 it seems like we are not sure about the sign of the demand (otherwise we would not need abs)
DemandForecaster(index, demand=-abs(demand))

Should we add abs() to the max_power as well?

[reinecfi]

A lot of hardcoded values that are either specified directly (like max_soc) or are imported from somewhere else (like additional_cost_charge). Maybe bundle them at one place or make them configurable?

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 0dbcca0)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Demand Bounds For positive ENTSO-E load, the demand unit is created with min_power set to 0 and max_power set to a negative value. If unit validation or bidding logic expects min_power <= max_power, scenarios with any positive demand can fail or have demand bids constrained incorrectly. Make the power bounds consistent with the sign convention used for demand. "min_power": 0, "max_power": -demand.max(), "bidding_strategies": bidding_strategies["demand"], "technology": "demand", "location": location, "node": country, "price": 1e3, }, DemandForecaster(index, demand=-abs(demand)), Version Bug yf.download(... )["Close"] can return a Series for a single ticker in some yfinance versions. In that case accessing pln_eur.columns raises AttributeError, so fuel price loading fails before any fallback is applied. Check whether pln_eur is a DataFrame before reading .columns. pln_eur = yf.download("PLNEUR=X", start=start, end=end, progress=False)["Close"] if isinstance(pln_eur.columns, pd.MultiIndex): pln_eur = pln_eur["PLNEUR=X"] else: pln_eur = pln_eur.squeeze() pln_eur = pln_eur.reindex(index).ffill().bfill()

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 0dbcca0
Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Preserve cached generation frames Single-column cached generation data is squeezed into a Series, but later code expects generation.columns. Make cache squeezing configurable and keep generation caches as DataFrame objects. assume/scenario/entsoe_helper/client.py [59-152] def _read_cache( - self, path: Path, parse_dates: bool = True + self, + path: Path, + parse_dates: bool = True, + squeeze_single_column: bool = True, ) -> pd.Series | pd.DataFrame | None: if not path.is_file(): return None logger.info(f"using cached ENTSO-E data from {path}") data = pd.read_csv(path, index_col=0, parse_dates=parse_dates) - if isinstance(data, pd.DataFrame) and data.shape[1] == 1: + if squeeze_single_column and isinstance(data, pd.DataFrame) and data.shape[1] == 1: return data.squeeze() return data ... - cached = self._read_cache(cache_path) + cached = self._read_cache(cache_path, squeeze_single_column=False) if cached is not None: + if isinstance(cached, pd.Series): + cached = cached.to_frame() return self._slice_to_period( self._ensure_unique_index(self._to_naive_index(cached)), start, end, ) Suggestion importance[1-10]: 7 __ Why: The issue is real: _read_cache squeezes any single-column CSV into a Series, and cached generation data can later break code that expects generation.columns. Making squeezing configurable and disabling it for generation caches is an accurate functional fix.	Medium
	Handle series load responses EntsoePandasClient.query_load can return a pd.Series, so indexing it with "Actual Load" may raise a KeyError. Handle both Series and DataFrame responses before resampling. assume/scenario/entsoe_helper/client.py [127-130] load = _flatten_columns( self.client.query_load(country, start=start_ts, end=end_ts) ) -demand = load["Actual Load"].resample("h").mean() +if isinstance(load, pd.Series): + demand = load.resample("h").mean() +else: + demand = load["Actual Load"].resample("h").mean() Suggestion importance[1-10]: 6 __ Why: This is a valid robustness fix if EntsoePandasClient.query_load returns a pd.Series, because the current load["Actual Load"] access would fail. The impact is moderate since the common response may be a DataFrame, but handling both shapes is safer.	Low
	Collapse duplicate production columns Flattening MultiIndex columns to only level 0 can create duplicate production-type columns. Those duplicates make generation[psr_name] return a DataFrame, causing float(gen_series.max()) to fail during aggregation. assume/scenario/entsoe_helper/client.py [28-35] def _flatten_columns(data: pd.DataFrame | pd.Series) -> pd.DataFrame | pd.Series: if not isinstance(data, pd.DataFrame): return data + data = data.copy() if isinstance(data.columns, pd.MultiIndex): - data = data.copy() data.columns = data.columns.get_level_values(0) data.columns = data.columns.map(str) + if data.columns.has_duplicates: + data = data.T.groupby(level=0).sum(min_count=1).T return data Suggestion importance[1-10]: 6 __ Why: Flattening a MultiIndex to only level 0 can create duplicate generation columns, causing generation[psr_name] to return a DataFrame and break float(gen_series.max()). Collapsing duplicates is a reasonable fix, though summing duplicate columns may not always preserve the exact ENTSO-E semantics.	Low

---

Previous suggestions

Suggestions up to commit c12ccb8

Category	Suggestion	Impact
Possible issue	Fix yfinance output handling yf.download(...)["Close"] can be a Series (single ticker) or a DataFrame (MultiIndex columns), and the current code will crash when pln_eur is a Series because it has no .columns. Handle both return types explicitly and also guard against empty downloads to avoid propagating NaNs into fuel prices. assume/scenario/entsoe_helper/fuel_prices.py [104-110] -pln_eur = yf.download("PLNEUR=X", start=start, end=end, progress=False)["Close"] -if isinstance(pln_eur.columns, pd.MultiIndex): - pln_eur = pln_eur["PLNEUR=X"] +data = yf.download("PLNEUR=X", start=start, end=end, progress=False) +close = data["Close"] + +if isinstance(close, pd.DataFrame): + # MultiIndex columns when multiple tickers/fields are returned + if "PLNEUR=X" in close.columns: + close = close["PLNEUR=X"] + else: + close = close.iloc[:, 0] else: - pln_eur = pln_eur.squeeze() -pln_eur = pln_eur.reindex(index).ffill().bfill() + close = close.squeeze() + +if close.empty: + raise RuntimeError("No FX data returned for PLNEUR=X") + +pln_eur = close.reindex(index).ffill().bfill() return pln_eur Suggestion importance[1-10]: 8 __ Why: This is a real correctness bug: yf.download(...)[\"Close\"] can be a pd.Series, so accessing pln_eur.columns will raise an AttributeError and break fuel-price loading. The proposed handling correctly normalizes Series vs DataFrame and avoids silently propagating missing FX data.	Medium
	Robustly parse load responses query_load can return a Series (or a DataFrame without an "Actual Load" column depending on ENTSO-E responses), which will make load["Actual Load"] fail. Normalize the result into a single Series before resampling, falling back to the only column when appropriate and raising a clear error otherwise. assume/scenario/entsoe_helper/client.py [127-131] load = _flatten_columns( self.client.query_load(country, start=start_ts, end=end_ts) ) -demand = load["Actual Load"].resample("h").mean() + +if isinstance(load, pd.Series): + load_series = load +elif "Actual Load" in load.columns: + load_series = load["Actual Load"] +elif load.shape[1] == 1: + load_series = load.iloc[:, 0] +else: + raise AssumeException( + f"Unexpected ENTSO-E load format for {country}: columns={list(load.columns)}" + ) + +demand = load_series.resample("h").mean() demand = self._ensure_unique_index(self._to_naive_index(demand)) Suggestion importance[1-10]: 8 __ Why: EntsoePandasClient.query_load(...) can yield a pd.Series or a pd.DataFrame whose columns don’t include \"Actual Load\", so load[\"Actual Load\"] can fail at runtime. Normalizing to a single series (or raising a clear AssumeException) improves robustness without changing intended behavior.	Medium
	Guard non-datetime capacity index capacity.index.tz assumes a DatetimeIndex, but query_installed_generation_capacity can yield a non-datetime index after reshaping (e.g., when the API returns a Series), causing an attribute error. Only use the timestamp lookup when the index is a DatetimeIndex; otherwise just take the last row. assume/scenario/entsoe_helper/client.py [197-202] -index_tz = capacity.index.tz -target = pd.Timestamp(start.year, 1, 1, tz=index_tz) -if target not in capacity.index: +if isinstance(capacity.index, pd.DatetimeIndex): + index_tz = capacity.index.tz + target = pd.Timestamp(start.year, 1, 1, tz=index_tz) + if target not in capacity.index: + capacity_row = capacity.iloc[-1] + else: + capacity_row = capacity.loc[target] +else: capacity_row = capacity.iloc[-1] -else: - capacity_row = capacity.loc[target] Suggestion importance[1-10]: 7 __ Why: Accessing capacity.index.tz assumes a pd.DatetimeIndex, but the code path that coerces a pd.Series into a one-row DataFrame can leave a non-datetime index and crash. The guard makes get_installed_capacity() resilient across ENTSO-E response shapes while preserving the timestamp-lookup behavior when available.	Medium

[github-actions]

Persistent review updated to latest commit 0dbcca0