Advance run scenarios

[ekatef]

Relates to #1850 and aims to fix a duplication issue in the first place.

Changes proposed in this Pull Request

The PR is advancing update() function which aims to update the base config by setting values according to the diff config. The revision adds functionality which is replacing a block under a key instead adding one more block under the same key.

Additionally, some minor refactoring changes are introduced, such as adding a docstring and naming revision.

Checklist

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented, including updates to docstrings for meaningful functions.
• Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/user-guide/configuration.md and doc/tutorials/electricity-model.md.
• If config sections were added, renamed, or removed, update doc/assets/scripts/extract_config_snippets.py accordingly.
• Archives of the uploaded data do not have an enclosing folder and archive names correspond to the conventions of configs/bundle_config.yaml.
• A note for the release notes doc/release-notes.md is amended in the format of previous release notes, including reference to the requested PR.

[ekatef]

As discussed with @davide-f , the implementation must allow to modify only a few entries in the config. E.g. it should be possible to specify in the config diff only a few parameters for electricity block without the need to copy-paste the whole block.

Also, we need to find a way to avoid over-writing config.yaml.

[ekatef]

Sharing feedback from @tacwebservices who has been working with run_all_scenarios. There has been the need to introduce some additionally changes into Snakemake as implemented by this commit\

@tacwebservices I'm adding an explanation on that you have shared, feel free to correct me if needed.

For --unlock, the reason to use it was to avoid conflicts between different Snakemake processes. When outer process run_all_scenarios starts an inner process running a single scenario, the inner one sees the lock already there and stops, thinking someone else is using the folder. So only the first scenario would finish and the rest would not start

[ekatef]

For --nolock, it appears behaviour can differ depending on an operation system. I have experienced some weird issues for build_powerplants when having --nolock enabled.

The error looked as follows: ERROR:_helpers:An error happened in module '~/miniforge3/envs/pypsa-earth-aug/lib/python3.11/tokenize.py', function '_tokenize': ('EOF in multi-line statement', (2, 0))

[davide-f]

I've investigated this issue aiming to understand the needs. When looking at the pinned issue #1850 , I perceive the key issue is the inability to "delete" entries from the original dictionary.
We could explicitly capture that feature by enablind deleting options when the diff value is empty, such as "cutout-2013-era5: {}".
May this solve the issue? I'm attaching a code below that does that, though it requires polishing and local testing with run_all_scenarios. I've been testing with the notebook test_update.ipynb attached to this message.
Many thanks!

Moreover, please remind the release_note

Suggested change

	def update(base_dictionary, diff_dictionary):
	"""
	A function to recursively update items in a dictionary which
	intended usage is unpdating the base config according to the diff
	config
	"""
	for key_to_apply, value_to_apply in diff_dictionary.items():
	if isinstance(value_to_apply, collections.abc.Mapping) and isinstance(
	base_dictionary.get(key_to_apply), collections.abc.Mapping
	):
	base_keys = set(base_dictionary[key_to_apply])
	diff_keys = set(value_to_apply)
	if base_keys & diff_keys:
	base_dictionary[key_to_apply] = update(
	base_dictionary[key_to_apply], value_to_apply
	)
	else:
	base_dictionary[key_to_apply] = copy.deepcopy(value_to_apply)
	else:
	d[k] = v
	return d
	base_dictionary[key_to_apply] = copy.deepcopy(value_to_apply)
	return base_dictionary
	def update(base_dictionary, diff_dictionary):
	"""
	A function to recursively update items in a dictionary which
	intended usage is unpdating the base config according to the diff
	config
	"""
	for key_to_apply, value_to_apply in diff_dictionary.items():
	if isinstance(value_to_apply, collections.abc.Mapping) and not set(value_to_apply):
	# if the diff value is an empty dict, delete the key from the base dict
	del base_dictionary[key_to_apply]
	elif isinstance(value_to_apply, collections.abc.Mapping) and isinstance(
	base_dictionary.get(key_to_apply), collections.abc.Mapping
	):
	# if the diff value is a non-empty dict and the base value is also a dict, update the base dict with the diff dict
	base_dictionary[key_to_apply] = update(
	base_dictionary[key_to_apply], value_to_apply
	)
	else:
	# if the diff value is not a dict, update the base dict with the diff value
	base_dictionary[key_to_apply] = copy.deepcopy(value_to_apply)
	return base_dictionary