Generate overwrite parameter grid (simplification)

[GernotMaier]

This is based on #2248 with the following changes are applied:

• call python modules directly instead of using workflow approach using run-application
• remove hardwired threshold and energy ranges and allow to set it on the command line
• streamline code to minimize dublications
• remove simtools-generate-parameter-scan-grid (not needed anymore, as we call modules directly)

[Copilot]

Pull request overview

This PR streamlines generation of production job grids and introduces backend-neutral parameter-scan grid expansion to support NSB/proton bias-curve workflows, including optional scan metadata fields that can be consumed by the HTCondor submission generator.

Changes:

• Refactor job-grid generation to a shared generate_job_grid() helper (and update the CLI app to use it).
• Add a parameter-scan grid expander that generates overwrite YAMLs and augments job grids with optional scan fields (overwrite_model_parameters, scan_label, telescope), plus schemas/tests/docs.
• Extend the HTCondor submission generator to detect and include optional scan fields in params/submit scripts.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/unit_tests/production_configuration/test_simulation_jobs.py	Adds unit coverage for _renumber_job_rows and generate_job_grid.
tests/unit_tests/production_configuration/test_job_grid_io.py	Updates serialization tests for optional columns and schema-driven types.
tests/unit_tests/job_execution/test_parameter_scan_generator.py	New unit tests for scan config parsing, overwrite generation, and grid expansion.
tests/unit_tests/job_execution/test_htcondor_script_generator.py	Updates and expands coverage for optional queue fields in HTCondor scripts/params.
tests/unit_tests/job_execution/test_bias_curve_submissions.py	New unit tests for bias-curve scan-grid generation and validation.
tests/unit_tests/applications/test_production_generate_grid.py	Updates app test to assert generate_job_grid() usage.
tests/unit_tests/applications/test_generate_bias_curve_submissions.py	New tests for the new bias-curve submissions CLI argument parsing.
src/simtools/schemas/parameter_scan_config.schema.yml	Adds schema for parameter-scan configuration files.
src/simtools/schemas/job_grid_density.schema.yml	Extends job-grid schema with optional scan-related columns.
src/simtools/production_configuration/simulation_jobs.py	Adds generate_job_grid() and _renumber_job_rows() in shared module.
src/simtools/production_configuration/job_grid_io.py	Makes job-grid columns/schema-driven and adds optional string fields handling.
src/simtools/job_execution/parameter_scan_generator.py	New module to expand base grids into scan grids with generated overwrite YAMLs.
src/simtools/job_execution/htcondor_script_generator.py	Adds optional queue fields support in params/submit generation and job specs.
src/simtools/job_execution/bias_curve_submissions.py	New module to generate NSB/proton scan grids and configs.
src/simtools/applications/production_generate_grid.py	Switches CLI implementation to call shared generate_job_grid().
src/simtools/applications/generate_bias_curve_submissions.py	New CLI app to generate bias-curve scan grids.
pyproject.toml	Registers new CLI entry point simtools-generate-bias-curve-submissions.
docs/source/user-guide/applications/simtools-generate-bias-curve-submissions.rst	Adds user-guide page for the new application.
docs/source/user-guide/applications.md	Adds the new application to the toctree list.
docs/source/api-reference/job_execution.md	Documents new job_execution modules.
docs/changes/2248.feature.md	Adds a changelog fragment for the new functionality.

[GernotMaier]

@EshitaJoshi - I am trying to understand the default of 1. here. We need to be better what nsb_rate means - sometimes it is used as multiplier to dark, but we also have a simulation model parameter which gives the NSB rate per pixel. I am not sure what this one is.

job_grid_io.py:

def _deserialize_job_row(serialized_row):
    """Deserialize one stored row to the in-memory job-row schema."""
    job_row = {
        "primary": serialized_row["primary"],
        "cores_per_shower": int(serialized_row["cores_per_shower"]),
        "showers_per_run": int(serialized_row["showers_per_run"]),
        "nsb_rate": float(serialized_row.get("nsb_rate", 1.0)),
        "model_version": serialized_row["model_version"],
        "array_layout_name": serialized_row["array_layout_name"],
        "corsika_le_interaction": serialized_row["corsika_le_interaction"],
        "corsika_he_interaction": serialized_row["corsika_he_interaction"],
        "run_number": int(serialized_row["run_number"]),
    }

[ctao-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[GernotMaier]

@EshitaJoshi - this is a copy of your branch with some simplifications applied. Feel free to change the base branch for this merge request to see the changes.

I've checked the output grid, but didn't text the submission with HT Condor etc. Please do this testing.