Cleaned notebooks, added instructions for different scattering geomet…

pipelines/GEOMETRY_AND_ROI_SETUP.md

@@ -0,0 +1,118 @@
+# Set up ROIs in `XPCS_auto.ipynb`
+
+In the **Run controls** cell, set `scat_geometry` to one of the modes below.
+Bluesky run-start `scat_geometry` takes precedence over the notebook value.
+The ROI choices are:
+
+| Geometry | Default ROIs | Custom choice |
+| --- | --- | --- |
+| `saxs` | Ring or q-phi template | Create a named template or set `manual_roi_file`. |
+| `gi_saxs` | 8 × 3 q grid | Set `gisaxs_q_parallel_edges`/`gisaxs_q_z_edges` or `surface_roi_config_file`. |
+| `waxs` | Intensity bands, indexed 1, 2, ... | Set `manual_roi_file` for custom pixel ROIs. |
+| `gi_waxs` | 12 × 4 q grid | Set `giwaxs_q_parallel_edges`/`giwaxs_q_z_edges` or `surface_roi_config_file`. |
+
+GI grid edges and surface JSON box ranges use `(q_parallel, q_z)` in inverse
+ångströms. WAXS intensity-band indices are not calibrated q values.
+
+For SAXS templates, add a name and parameters to `mask_param_dict` in
+[`Auto_ROI_Mask_Generator.ipynb`](masking_and_rois/Auto_ROI_Mask_Generator.ipynb),
+set `mask_selection` to that name, and run it. It writes
+`_mask_path_/ROI_masks/general_roi_mask_<name>.json` and will not overwrite an
+existing name. Keep `roi_auto=True` to use this template. If run-start
+`analysis` names a different template, `analysis_type_auto=True` selects that
+instead. To force a notebook choice, set `analysis_type_auto=False`, choose
+`qphi_analysis=False` for rings or `True` for q-phi, and set the corresponding
+mask name. If you set `roi_auto=False`, you must set `manual_roi_file`.
+
+## Check calibration and masks first
+
+GI-SAXS and GI-WAXS require calibrated q maps. Supply these values in Bluesky
+metadata when collecting the run, or use the **Run controls** fallback for an
+older run:
+
+| Value | Run metadata | Notebook fallback |
+| --- | --- | --- |
+| Incident angle (degrees) | `inc_ang`, `incident_angle`, or `incident_angle_deg` | `surface_incident_angle_deg`; required if metadata lacks an angle |
+| In-plane angle (degrees) | `inc_phi`, `inplane_angle`, or `inplane_angle_deg` | `surface_inplane_angle_deg` (default 0) |
+| Incident beam center | Image `beam_center_x/y` | `surface_beam_center_x/y` if either center coordinate is missing; use oriented image column/row |
+| Pixel size | `x_pixel_size` (metres) | `surface_pixel_size_mm` |
+| Detector distance | `det_distance` or `detector_distance` (metres) | `surface_detector_distance_mm` |
+| Wavelength | `wavelength` or `incident_wavelength` (ångströms) | `surface_wavelength_angstrom` |
+
+The notebook always loads the oriented detector mask from
+`_mask_path_/pixel_masks/detector_mask_<detector>.json`. Run
+[`Detector_Masks.ipynb`](masking_and_rois/Detector_Masks.ipynb) only if it is
+missing or needs rebuilding; use
+[`Detector_Masks_Add_Pixels.ipynb`](masking_and_rois/Detector_Masks_Add_Pixels.ipynb)
+for known bad pixels (`pixels_to_add` uses **row, column**).
+
+For a beam shadow, put `XPCS_beam_mask_<scat_geometry>.json` in the cycle/user
+`masks/` directory, or set `beam_mask_config_file`. SAXS also accepts the old
+`XPCS_beam_mask.json`. Alternatively, set `manual_mask_file` to a saved mask
+with the oriented image shape and 1=keep/0=exclude. An `.npz` mask must
+contain `mask` and `orientation='analysis'`; for legacy `.npy`/`.pkl` masks,
+set `manual_mask_orientation='analysis'`. The default GI-SAXS beam mask
+excludes a stripe; the default GI-WAXS and WAXS beam masks are empty when
+`auto_bst_mask=True`. Check the combined-mask plot before analyzing a new setup.
+
+## Edit or create surface ROIs with JSON
+
+For `gi_saxs` or `gi_waxs`, save a JSON file and set
+`surface_roi_config_file` to its path. This example starts from the 2025-3
+rheadrick GI-SAXS ROI and q-map files in
+`/nsls2/users/jgoodrich/proposals/2025-3/rheadrick/masks/`. Save the JSON
+there too so the relative paths work:
+
+```json
+{
+  "geometry": "gi_saxs",
+  "orientation": "analysis",
+  "base_roi_file": "uid=0ae234d7_roi_mask.pkl",
+  "base_qmap_file": "uid=0ae234d7_qmap.pkl",
+  "remove_labels": [1],
+  "boxes": [
+    {"q_parallel": [0.003, 0.018], "q_z": [0.151, 0.158]}
+  ]
+}
+```
+
+`remove_labels` refers to labels in the saved ROI. Each box uses q values in
+inverse ångströms with `[lower, upper)` bounds. Adjust the example ranges for
+your run; a box must contain unmasked pixels and cannot overlap a retained
+ROI. Set `remove_labels` and `boxes` to `[]` to start with all saved regions.
+For a new layout, omit both base-file fields and list your boxes. Change
+`geometry` to `gi_waxs` for GI-WAXS.
+
+Boxes with the same `q_z` bounds share one z profile band. A box also joins a
+retained saved band when its z range contains that band's pixels and no other
+saved band.
+
+GI-SAXS intensity profiles for JSON, manual, or explicitly edged ROIs use one
+mean-intensity point per ROI at its calibrated `q_parallel` center; gaps between
+boxes are excluded. The automatic grid keeps the existing continuous q profile
+calculation.
+
+With `base_qmap_file`, the loader estimates each old ROI's q bounds from its
+saved pixels and draws those ranges on the **current** q maps. Inspect the
+resulting ROI overlay: this reprojection is an approximation and can produce
+empty or overlapping regions after a geometry change. Without
+`base_qmap_file`, it uses the saved pixel labels directly and checks their q
+values against the current calibration. The other rheadrick starting point is
+`uid=151209_roi_mask.pkl` with its matching `uid=151209_qmap.pkl`. The old
+notebooks used a hardcoded 0.46° incident angle; verify that angle against the
+run metadata before applying it in the unified notebook.
+
+For direct saved ROI use, set `manual_roi_file` instead of
+`surface_roi_config_file`. A new `.npz` file needs `roi_mask`, ordered
+`q_values` (one row per positive label), and `orientation='analysis'`.
+SAXS values are calibrated q; surface values are `(q_parallel, q_z)`; WAXS
+values are ROI indices `1, 2, ...`. Labels must be contiguous and inside the
+analysis mask. For an old `.pkl` ROI, also set
+`manual_roi_orientation='analysis'`. No ROI-generation notebook is needed
+for surface JSON recipes or default WAXS bands.
+
+Check the resolved geometry, calibration, combined mask, and ROI overlay for
+each new setup. Before committing an edited notebook, run
+`pre-commit run nbstripout --files pipelines/XPCS_auto.ipynb`. See
+[coordinate conventions](COORDINATE_CONVENTIONS.md) when moving saved pixel
+masks between detector orientations.

pipelines/XPCS_auto.ipynb

@@ -48,10 +48,10 @@
     "\n",
     "from pyCHX.chx_packages import *\n",
     "from pyCHX._xpcs_pipeline_geometry import (\n",
-    "    build_surface_rois, build_waxs_rois, combine_detector_masks, intersect_roi_with_mask,\n",
-    "    load_manual_mask, load_manual_roi,\n",
+    "    build_surface_rois, build_waxs_rois, cal_1d_qr_from_roi, combine_detector_masks,\n",
+    "    get_t_qrc_from_roi, intersect_roi_with_mask,\n",
+    "    load_manual_mask, load_manual_roi, load_surface_roi_config,\n",
     "    plot_roi_index_rates, plot_surface_roi, plot_xpcs_correlation, resolve_geometry,\n",
-    "    surface_profile_args_from_roi,\n",
     "    surface_q_maps, validate_roi,\n",
     ")\n",
     "#from pyCHX.chx_xpcs_xsvs_jupyter_V1 import *\n",
@@ -183,6 +183,7 @@
     "# carry orientation='analysis'; legacy .npy/.pkl inputs need the explicit\n",
     "# orientation setting below. ROI .npz files contain roi_mask and q_values.\n",
     "manual_roi_file = None\n",
+    "surface_roi_config_file = None  # optional JSON recipe for gi_saxs or gi_waxs ROIs\n",
     "manual_mask_file = None\n",
     "beam_mask_config_file = None  # optional mode-specific polygon JSON\n",
     "manual_roi_orientation = None\n",
@@ -1231,12 +1232,21 @@
     "    q_mask_name = scat_geometry\n",
     "    parallel_edges = gisaxs_q_parallel_edges if scat_geometry == 'gi_saxs' else giwaxs_q_parallel_edges\n",
     "    z_edges = gisaxs_q_z_edges if scat_geometry == 'gi_saxs' else giwaxs_q_z_edges\n",
-    "    if manual_roi_file is not None:\n",
+    "    use_roi_profiles = (surface_roi_config_file is not None or manual_roi_file is not None\n",
+    "                        or parallel_edges is not None or z_edges is not None)\n",
+    "    surface_rois = None\n",
+    "    if surface_roi_config_file is not None:\n",
+    "        if manual_roi_file is not None:\n",
+    "            raise ValueError('Set either surface_roi_config_file or manual_roi_file, not both')\n",
+    "        fp = str(surface_roi_config_file)\n",
+    "        roi_mask, qval_dict = load_surface_roi_config(\n",
+    "            fp, mask.shape, scat_geometry, mask, surface_maps)\n",
+    "        roi_input_source = fp\n",
+    "    elif manual_roi_file is not None:\n",
     "        fp = str(manual_roi_file)\n",
     "        roi_mask, qval_dict = load_manual_roi(\n",
     "            fp, mask.shape, scat_geometry, orientation=manual_roi_orientation,\n",
     "            mask=mask, q_maps=surface_maps)\n",
-    "        surface_rois = surface_profile_args_from_roi(roi_mask, qval_dict, surface_maps)\n",
     "        roi_input_source = fp\n",
     "    else:\n",
     "        surface_rois = build_surface_rois(qr_map, qz_map, mask,\n",
@@ -1246,7 +1256,7 @@
     "        roi_mask, qval_dict = surface_rois['roi_mask'], surface_rois['qval_dict']\n",
     "        fp = 'generated:%s:%s'%(scat_geometry, md['uid'])\n",
     "        roi_input_source = fp\n",
-    "    Qr, Qz = surface_rois['Qr'], surface_rois['Qz']\n",
+    "    Qr, Qz = (None, None) if use_roi_profiles else (surface_rois['Qr'], surface_rois['Qz'])\n",
     "    Qrs, Qzs = Qr, Qz\n",
     "    roi_masks, qval_dicts = roi_mask.copy(), qval_dict.copy()\n",
     "    ticks = None\n",
@@ -2247,7 +2257,11 @@
     "    time_edge = create_time_slice( N= Nimg, slice_num= 3, slice_width= 2, edges = None )\n",
     "    time_edge =  np.array( time_edge ) + good_start\n",
     "    print( time_edge )    \n",
-    "    qrt_pds = get_t_qrc( FD, time_edge, Qrs, Qzs, qr_map, qz_map, mask=mask, path=data_dir, uid = uidstr )    \n",
+    "    if use_roi_profiles:\n",
+    "        qrt_pds = get_t_qrc_from_roi(\n",
+    "            FD, time_edge, roi_mask, qval_dict, get_avg_imgc, path=data_dir, uid=uidstr)\n",
+    "    else:\n",
+    "        qrt_pds = get_t_qrc( FD, time_edge, Qrs, Qzs, qr_map, qz_map, mask=mask, path=data_dir, uid = uidstr )\n",
     "    plot_qrt_pds( qrt_pds, time_edge, qz_index = 0, uid = uidstr, path =  data_dir )\n",
     "\n",
     "_record_xpcs_cell_timing(\"49 - Calculate time-dependent GISAXS profiles\", _xpcs_cell_start)\n"
@@ -2358,7 +2372,11 @@
     "if scat_geometry =='gi_saxs':       \n",
     "    # ROI overlay saved by the private surface plotting adapter.        \n",
     "    ## Get 1D Curve (Q||-intensity¶)\n",
-    "    qr_1d_pds = cal_1d_qr( avg_img, Qr, Qz, qr_map, qz_map, inc_x0= None, mask=mask, setup_pargs=setup_pargs )\n",
+    "    if use_roi_profiles:\n",
+    "        qr_1d_pds = cal_1d_qr_from_roi(\n",
+    "            avg_img, roi_mask, qval_dict, path=data_dir, uid=uidstr)\n",
+    "    else:\n",
+    "        qr_1d_pds = cal_1d_qr( avg_img, Qr, Qz, qr_map, qz_map, inc_x0= None, mask=mask, setup_pargs=setup_pargs )\n",
     "    plot_qr_1d_with_ROI( qr_1d_pds, qr_center=np.unique( np.array(list( qval_dict.values() ) )[:,0] ),\n",
     "                    loglog=True, save=True, uid=uidstr, path = data_dir)\n",
     "\n",
@@ -2586,7 +2604,6 @@
      "start_time": "2023-06-20T08:53:55.221802",
      "status": "completed"
     },
-    "scrolled": true,
     "tags": []
    },
    "outputs": [],
@@ -4028,9 +4045,12 @@
     "        'inplane_angle_deg': surface_maps['inplane_angle_deg'],\n",
     "        'incident_center': surface_maps['incident_center'],\n",
     "        'reflected_center': surface_maps['reflected_center'],\n",
-    "        'q_parallel_edges': surface_rois['parallel_edges'],\n",
-    "        'q_z_edges': surface_rois['z_edges'],\n",
     "    }\n",
+    "    if scat_geometry == 'gi_saxs':\n",
+    "        md['surface_geometry']['profile_source'] = 'roi_mask' if use_roi_profiles else 'regular_q_grid'\n",
+    "    if surface_rois is not None:\n",
+    "        md['surface_geometry']['q_parallel_edges'] = surface_rois['parallel_edges']\n",
+    "        md['surface_geometry']['q_z_edges'] = surface_rois['z_edges']\n",
     "    md['surface_reflected_center_x'] = refl_x0\n",
     "    md['surface_reflected_center_y'] = refl_y0\n",
     "md['mask'] = mask\n",
@@ -4045,8 +4065,9 @@
     "    md['maximum_outlier_fraction'] = maximum_outlier_fraction\n",
     "md['setup_pargs'] = setup_pargs\n",
     "if scat_geometry == 'gi_saxs':        \n",
-    "    md['Qr'] = Qr\n",
-    "    md['Qz'] = Qz\n",
+    "    if not use_roi_profiles:\n",
+    "        md['Qr'] = Qr\n",
+    "        md['Qz'] = Qz\n",
     "    md['qval_dict'] = qval_dict\n",
     "    md['beam_center_x'] =  inc_x0\n",
     "    md['beam_center_y']=   inc_y0\n",
@@ -4096,7 +4117,7 @@
     "                [md,       roi_mask, qval_dict, avg_img,mask,pixel_mask, imgsum, bad_frame_list] ):\n",
     "        Exdt[ k ] = v\n",
     "        \n",
-    "if scat_geometry in ('gi_saxs', 'gi_waxs'):\n",
+    "if scat_geometry in ('gi_saxs', 'gi_waxs') and surface_rois is not None:\n",
     "    Exdt['q_parallel_edges'] = surface_rois['parallel_edges']\n",
     "    Exdt['q_z_edges'] = surface_rois['z_edges']\n",
     "if run_waterfall:Exdt['wat'] =  wat\n",