Generate htcondor submit files to make bias curves

[EshitaJoshi]

Give it the apptainer image path and run with:

simtools-generate-bias-curve-submissions  --site North     --model_version 7.0.0     --array_layout_name LSTN-01     --azimuth_angle 0.0     --zenith_angle 20.0     --showers_per_run 1000     --core_scatter "20 1900 m"     --view_cone "0 deg 5 deg"     --number_of_runs 10     --apptainer_image path-to-image.sif  --label test    --output_path simtools-output/bias_curves

It will then generate bias_curves/nsb/htcondor_submit and bias_curves/proton/htcondor_submit with the .condor and .sh files respectively that can be submitted. Remember to copy the env.txt file into each htcondor_submit folder!

[GernotMaier]

Thanks @EshitaJoshi for the draft on the HTCondor / overwrite model parameter scanning. I think there is too much duplication - parameter_scan_generator.py is a variation of htcondor_script_generator.py and we would have to maintain in several places the same functionality. My suggestion is to combine this and get a job list file (as discussed yesterday), but with the overwrite model parameter as parameter:

run_number  zenith_angle  ...  overwrite_model_parameters                 scan_label
1           20 deg        ...  overwrite_threshold_220.yaml                threshold_220
2           20 deg        ...  overwrite_threshold_220.yaml                threshold_220
3           20 deg        ...  overwrite_threshold_230.yaml

To use as much of the existing code (and also what the user is familiar with), the workflow would be:

simtools-production-generate-grid --output_file base_grid.ecsv ...
simtools-generate-parameter-scan-grid \
  --job_grid_file base_grid.ecsv \
  --scan_config scan.yaml \
  --output_file scan_grid.ecsv
simtools-simulate-prod-htcondor-generator \
  --job_grid_file scan_grid.ecsv \
  --output_path htcondor_submit \
  --apptainer_image ...

This way you don't need to duplicate the basic job preparation, and the job scripts only needs to be extended to use (if available) the overwrite_model_parameters columns.

I paste below the diagnostic of the AI:

Yes. parameter_scan_generator.py is largely a second HTCondor generator beside htcondor_script_generator.py. The unique part is only the scan logic: cartesian parameter combinations and overwrite YAML generation. The submit file, params file, executable script, image resolution, log directories, and queue-from handling should stay in the existing HTCondor path.

Main concerns with the new diff:

• It bypasses the production grid flow from production_generate_grid.py, so it loses existing quantity parsing, run expansion, CORSIKA-limit clipping, shower scaling, metadata, and grid validation.
• It duplicates submit rendering from htcondor_script_generator.py.
• It uses --nshow in the generated script, while the established production path uses --showers_per_run.
• It only uses the first Apptainer image from a mapping, so it drops the existing multi-image behavior.
• It hardcodes HTCondor log paths and has its own params format, so future fixes would need to be made twice.

My suggested implementation plan:

1. Keep simtools-production-generate-grid as the source of simulation job rows.

2. Reframe parameter scan generation as “expand an existing job grid”, not “generate HTCondor submission files”. Add a small scan-grid step that:

   • reads a base job_grid.ecsv with read_job_grid;
   • reads the scan YAML;
   • generates one overwrite YAML per scan combination;
   • duplicates each base grid row for each scan combination;
   • adds optional row fields like overwrite_model_parameters and label_suffix;
   • writes a new scan_job_grid.ecsv.

3. Extend job_grid_io.py to preserve optional string fields, at least:

   • overwrite_model_parameters
   • label_suffix or scan_label

4. Extend htcondor_script_generator.py to include optional queued fields when present:

   • pass --overwrite_model_parameters "$overwrite_model_parameters" only if set;
   • append the scan label suffix to job_label;
   • keep all existing output, image-label grouping, htcondor_log_path, simulation_output, and params-file behavior.

And for full clarity, this is the prompt I've used:

Carefully analyse the diff to main and compare that to the case where we:

-  generate a grid of simulation parameters with src/simtools/applications/production_generate_grid.py
- prepare a submission file, script and condor file for HT Condor with src/simtools/applications/simulate_prod_htcondor_generator.py

Is src/simtools/job_execution/htcondor_script_generator.py and src/simtools/job_execution/parameter_scan_generator.py quite similar to each other?

Plan and make a suggestion on how to implement the parameter scan using as much of the existing functionality.

[Copilot]

Pull request overview

This PR adds a parameter-scan workflow that expands an existing production job grid into a scan grid with per-combination overwrite YAMLs, and updates the HTCondor submit generator to optionally pass --overwrite_model_parameters, --scan_label, and --telescope per queued job. On top of that, it introduces a new “bias curve” helper application that generates NSB (gamma) and proton submission bundles via an internal scan definition.

Changes:

• Added parameter_scan_generator + CLI (simtools-generate-parameter-scan-grid) to expand job grids and generate overwrite YAMLs per scan combination.
• Extended HTCondor script generation to accept optional queue fields (overwrite_model_parameters, scan_label, telescope) and forward them to simtools-simulate-prod.
• Added bias_curve_submissions + CLI (simtools-generate-bias-curve-submissions) to generate NSB/proton scan configs, workflow configs, and submission directories.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/unit_tests/job_execution/test_parameter_scan_generator.py	Unit tests for scan config parsing, combo naming, overwrite generation, and grid expansion behavior.
tests/unit_tests/job_execution/test_job_manager.py	Additional unit tests for job_manager edge cases (retry, env propagation, stream prep).
tests/unit_tests/job_execution/test_htcondor_script_generator.py	Updated/extended tests for optional HTCondor queue fields and script rendering.
tests/unit_tests/job_execution/test_bias_curve_submissions.py	Unit tests for the bias-curve workflow config, telescope resolution rules, and scan config creation.
src/simtools/production_configuration/job_grid_io.py	Adds optional string-column support for job grids (used by scan expansions).
src/simtools/job_execution/parameter_scan_generator.py	New scan-grid expander that generates overwrite YAMLs and duplicates base rows per combination.
src/simtools/job_execution/htcondor_script_generator.py	Adds optional queue fields and forwards them to simtools-simulate-prod.
src/simtools/job_execution/bias_curve_submissions.py	New bias-curve submission workflow builder (NSB + proton).
src/simtools/applications/generate_parameter_scan_grid.py	New CLI wrapper around parameter_scan_generator.expand_job_grid_with_scan.
src/simtools/applications/generate_bias_curve_submissions.py	New CLI wrapper around bias_curve_submissions.generate_bias_curve_submissions.
pyproject.toml	Registers the two new CLI entry points.
docs/source/user-guide/applications/simtools-generate-parameter-scan-grid.rst	Adds user-guide page for the new parameter-scan grid generator CLI.
docs/source/user-guide/applications/simtools-generate-bias-curve-submissions.rst	Adds user-guide page for the new bias-curve submissions CLI.
docs/source/user-guide/applications.md	Adds both new application docs pages to the toctree list.
docs/source/api-reference/job_execution.md	Adds API reference entries for the two new job_execution modules.
docs/changes/2248.feature.md	Changelog fragment describing the added bias-curve workflow feature.

[ctao-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[GernotMaier]

Very nice!! I have minor comments below plus the separation of configuration and submission. But in general, this is a nice implementation - especially as simtools-production-generate-grid is a generic base-grid generator.

I didn't test the submission step yet (as I had to do this first review offline without HT Condor). Will do this after the updates.

[GernotMaier]

Why do need to pass it directly to the HT Condor generator? This mixes a bit configuration steps and the actual backend.

[EshitaJoshi]

Will stop the application after the scan_grid.ecsv is generated. The user can then pass this on themselves, making this backend agnostic.

[GernotMaier]

Public functions should have numpy doc style description of parameters.

[GernotMaier]

Use

values = general.ensure_list(param_spec["values"])
if not values:
    raise ValueError("'values' must contain at least one scan value.")

Otherwise a "220" is expanded to [2,2,0]

[GernotMaier]

Does this break the original functionality? e.g. preparing a grid with different interaction models or a list of fixed energies?

[GernotMaier]

Missing spaces?

[GernotMaier]

Missing spaces?

[GernotMaier]

job_spec.get("overwrite_model_parameters") is a path? Would _format_param_value change the path? e.g. if there it is something like "my path"?

[GernotMaier]

Is this needed? It seems to me that params_field is always set when _get_submit_file is called.