CHANGELOG.md

Changelog

[Unreleased]

Fixed

• Background scatter is now measured on the frame it corrects. The scatter step fits its model on the flat lamp frames (every instrument's id_scatter points at a flat), and ScienceExtraction subtracted those coefficients from science frames unchanged. Since a master flat is a sum of lamp exposures, the model is in the flux units of that stack and generally has no relation to a single science exposure: measured on XSHOOTER nir (10x400 s flats vs a 5 s science frame) the subtraction was 67x the signal at the trace centres, driving 99.97% of in-order pixels negative; on HARPS red (5x40 s vs 600 s) it under-subtracted at 0.14x. NormalizeFlatField was unaffected, as it corrects the same flat the model was measured on. Consumers now re-estimate the background on the calibrated image they are about to extract, via the new ScatterModel.refit, so the model matches that frame's own inter-order level by construction (1.000 on the XSHOOTER frame above, against 67x before). No exposure-time bookkeeping is involved and no settings changed
• HARPSPOL: the scatter step is no longer declared, having been measured to remove most of the stellar flux at every extraction_height tried (#38, #39)
• Scatter extraction_height was narrower than the order footprint on two instruments, leaving order wings in the background fit and biasing it high: HARPS shipped 20 px against 77 px order spacing (now 0.9 of the spacing, model/truth 6.1x -> 1.3x) and LICK_APF 18 px with degree 4 (now 0.6 and scatter_degree: 2, 0.48x -> 1.2x, the degree drop because 22 px order spacing leaves too few inter-order pixels for a quartic). UVES moved 0.9 -> 1.0 (1.23x -> 1.04x). Measured against the median of pixels further than 0.4x the order spacing from any trace. See scatter.md

Changed

• The scatter step returns a ScatterModel (coefficients plus the fit parameters) rather than a bare coefficient array; .scatter.npz on disk is unchanged and older files still load. A bare array passed to a consumer is still honoured, with a warning that its flux scale is unknown and it cannot be re-estimated

Removed

• HARPS wavecal_{blue,red}_pol_2D.npz and the polarimetry branch of HARPS.get_wavecal_filename. The _pol files held every order twice (90 and 52 orders against 45 and 26 real ones), interleaved, with cs_lines otherwise identical to the non-pol files and obase/bad_order not adjusted — an artifact of the era before fiber groups, when the two beams were extracted as 2N separate orders. Duplicated orders are incompatible with the 2D solution HARPS configures, which fits wavelength = P(x, order): refitting the shipped lines at degree [3, 6] gives a residual RMS of 0.0005 A from the non-pol lists against 8.4 A (blue) and 14.8 A (red) from the _pol ones, roughly half an order spacing, or ~600 km/s. Setting instrument.polarimetry: true selected them silently. HARPSPOL was already using the non-pol lists, which is correct: both beams see the same spectral orders, and its per-group trace count matches their order count

[0.9] - 2026-07-31

No code changes from 0.9b3; stable release of the 0.9 series.

[0.9b3] - 2026-07-31

Added

• Two pure-Python extraction backends implementing the same algorithm as clib/slitdec.c, for pipelines that cannot build a C extension: numba_slitdec (optional numba extra, 1.5-1.8x the C) and numpy_slitdec (numpy/scipy only, no extra to install, 1.8-2.1x). Both agree with the C to ~1e-14 relative with identical masks and iteration counts, checked by test/test_{numba,numpy}_slitdec.py
• CLI --extraction on reduce run and the per-step commands, equivalent to setting PYREDUCE_EXTRACTION
• CI job running the unit suite with the compiled extension removed and PYREDUCE_EXTRACTION=numpy, so the pure-Python path stays working

Changed

• Breaking: extraction backend selection moved from PYREDUCE_USE_CHARSLIT to PYREDUCE_EXTRACTION, one of c (default), charslit, numba, numpy. The old flag is no longer read; an unknown name raises instead of falling back. PYREDUCE_USE_DELTAS is unchanged
• The compiled extension is imported only when the c backend is selected, so import pyreduce.extract works without it; the resolved backend is logged once at INFO
• Extraction is 1.1-1.4x faster (most on tall slits): the sP band width is now measured from the zeta key ranges rather than sized from delta_x, shrinking bandsol from a 185-wide elimination to a 5-wide one on a 176-row swath, and the uncertainty pass reads the zeta tensor sequentially. Spectra, slit functions, models and masks are bit-identical; uncertainties move by <2e-15 relative from the changed summation order

[0.9b2] - 2026-07-15

Added

• Parallel science extraction via new n_jobs science setting (joblib semantics, -1 = all cores)
• CLI: --skip-existing flag, list-channels command; bad channel/instrument names fail fast listing the valid options
• Fiber selection in the Pipeline API: use_fibers(), extract(use=), from_instrument(use=), mirroring CLI --use
• Wavecal quality metrics: per-group log summary and .wavecal_quality.json export (RMS/median residuals in m/s, line counts, AIC)
• Trace: max_error setting rejects fused double-order clusters (port of IDL REDUCE MAX_ERR)
• Provenance: PR_version/PR_githash stamped into all FITS products; DATE and e_cont recorded in final headers
• Atlas: full KPNO ThAr line list; UNe NIR line list
• MOSAIC: per-bundle wavelength range guess for single-order multi-bundle wavecal initialization
• Example examples/andes_yjh_fp.py: extract and measure FP lines in an IFU frame

Changed

• reduce.py split into pyreduce/steps/ package; reduce.py keeps main() and re-exports step classes. mark_orders alias renamed to detect_traces; unused FitsIOStep removed
• MOSAIC channels renamed to the ESO INS MODE scheme (NIR: J_LR/H_LR/H_HR; VIS: arm+resolution x detector quadrant)
• wavecal_init: FFT cross-correlation against the synthesized atlas for the global offset; new wave_delta search-radius parameter
• Config errors fail early: id_* regex validation, per-channel list length checks, missing centers files raise instead of falling back silently

Fixed

• Corrupted thar.fits atlas replaced with the clean KPNO spectrum; corrupt une.fits dropped
• Shared bad pixel mask was attached by reference and mutated in-place by load_fits/combine_frames
• group_fibers: per-order range clamp no longer truncates later orders
• Traces file write errors in wavecal/freq_comb/curvature now propagate instead of being swallowed
• Wavelength solutions are written to the correct traces when wavecal results are group-keyed (MOSAIC)
• Legacy linelist .npz files migrated to the current dtype on load

[0.9b1] - 2026-06-17

Added

• HERMES instrument (Mercator telescope)
• MOSAIC LR-J / LR-H NIR channels and starsky example
• ANDES YJH sampling/resolution example (examples/andes_yjh_sampl.py)
• Explicit fibers.numbering config field (bottom_up/top_down) for fiber numbering direction
• Fallback to flat files for trace/curvature/scatter when files are set manually

Changed

• Extraction: slitdec (copied from charslit) is now the sole CFFI backend, ~10x faster
• Wavecal: disentangle bundle index from spectral order m
• ANDES YJH: env-var output suffix, trace_range, and IFU science flow
• Trace: center_weight merge for missing-fiber bundles

Fixed

• get_supported_channels for single-channel instruments
• Bundle naming in group_fibers when one bundle per m

[0.8.2] - 2026-04-08

Fixed

• Curvature step: fall back to per-trace height when extraction_height is None (matches extract.py behavior)

Changed

• Lower minimum Python version to 3.12
• CI test matrix now covers Python 3.12, 3.13, and 3.14

[0.8.1] - 2026-03-05

Added

• CRIRES+ L-band example script
• Settings fallback chain for composite channel names
• hdf2trace tool: write HDF wavelength polynomials into trace FITS

Fixed

• Fix create_custom_instrument crash (was returning None config)
• Fix charslit backend selection to check env var at call time
• Bundle centers fallback, curvature and norm_flat fixes for MOSAIC VIS

Changed

• Rewrite custom instrument example as student-friendly walkthrough
• create_custom_instrument now accepts **overrides for inline property setting

[0.8] - 2026-02-25

No code changes from 0.8a5; docs update and stable release.

[0.8a5] - 2026-02-25

Added

• ANDES_UBV instrument, IZ channel for ANDES_RIZ, per-channel settings files
• Preserve individual fiber traces alongside merged group traces
• Tests for get_y_scale covering standard, edge, and invariant cases

Fixed

• Center extraction window on trace peak (+0.5/+1 shift for odd/even height)
• Fix get_y_scale silently mutating caller's ycen array
• Fix slit function plot alignment in ProgressPlot
• Fix curvature save: match traces by (m, group) instead of index
• Recompute heights after merging even/odd fibers (use nearest neighbor)
• Fix test warnings: tempfile leak, divide-by-zero, misplaced instrument test
• Guard against empty order_centers YAML files

[0.8a4] - 2026-02-14

Added

• HARPSPOL instrument with dual-beam (beam splitter) support
• Otsu+DP beam-pair detection for multi-fiber trace pairing
• --use CLI flag to select fiber groups (e.g., --use upper,lower)
• Wavelength propagation from trace into extracted Spectrum

Changed

• Output files go into per-night subdirectory (reduced/{night}/)
• Group filenames use {prefix}_{group}.{step}.{ext} convention
• Traces without order_centers get sequential m values (was None)

Fixed

• Fix create_image_from_lines row count (use ntrace, not order range)
• Fix 2D wavecal polynomial eval: use trace index, not physical order m
• Fix multi-channel filename overwrite (include channel in science/rectify)
• Fix linelist order drift across re-runs (normalize to 0-based on save)
• Fix output_dir CLI default (None, so config's reduced/{night} is used)

[0.8a3] - 2026-02-11

Changed

• Replace MCMC-based wavecal_init with iterative peak matching algorithm
• Remove emcee and corner dependencies

Fixed

• Fix continuum trace count mismatch when norm_flat drops edge traces
• Fix extraction_height precedence: settings over trace.height
• Fix LFC plot crash on negative orders and Finalize trace selection
• Fix 2D wavecal poly eval, continuum trace selection, and numeric guards
• Fix wavecal/freq_comb trace data flow and continuum trace mismatch

[0.8a2] - 2026-02-03

Added

• Trace dataclass (trace_model.py) for unified trace data model
• Spectrum/Spectra classes (spectra.py) replacing legacy Echelle format
• Trace.wlen(x) method to evaluate wavelength polynomial
• Per-group wavelength calibration support
• LFC wavecal support for ANDES_RIZ
• docs/output_formats.md documenting new file formats

Changed

• All pipeline steps use list[Trace] interface instead of array-based
• extract() takes list[Trace], returns list[Spectrum]
• Renamed nord/iord to ntrace/idx for clarity
• Renamed Trace.fiber to Trace.group, added fiber_idx for per-fiber wavecal
• File format: traces.npz -> traces.fits (unified FITS format)
• Wavelength calibration stored in traces.fits instead of separate wavecal.npz
• Spectra use NaN masking instead of COLUMNS+MASK redundancy

Removed

• curvature_model.py (curvature now stored in Trace)
• echelle.py functionality (deprecated, use spectra.py)
• Dead code from util.py

[0.8a1] - 2026-02-03

Added

• Charslit extraction backend with degree 1-5 curvature support
• ANDES_RIZ instrument configuration and settings
• trace_by config for separate tracing of illumination groups
• HDF-to-wavecal tool and reference files for ANDES_RIZ
• ANDES spectra plotting tool

Changed

• Curvature: use literal pixel extraction_height, not fractional
• Simplify extraction_height to single full-height value
• Move fiber selection default from hardcoded to config
• Update METIS_IFU for dynamic wavelength-based channels
• Use git URL for charslit optional dependency

Removed

• PEP 723 inline metadata from examples (caused confusion with uv run)

Fixed

• Fix curve_height to use new full-height format in plot_comparison
• Fix extraction_height to give exactly N rows
• Fix CRIRES_PLUS docstring (was incorrectly HARPS)

[0.7] - 2026-01-25

First stable release of the 0.7 series!

Changed

• Rename OrderTracing class to Trace, save file from .orders.npz to .traces.npz
• Store per-trace extraction heights in traces.npz
• Compute extraction heights for fiber bundles automatically
• CLI: consistent -t/--target option across all commands
• Allow explicit paths in settings __inherits__ directive

Fixed

• Limit trace overlay to column_range in science step plot

Documentation

• Add WhatsNew.md summarizing 0.7 series improvements
• Update docs for traces.npz rename and mask convention

[0.7b4] - 2026-01-22

Added

• Per-channel settings support via settings_{channel}.json files
• Fiber grouping configuration for multi-fiber instruments (bundle_centers, fiber_groups)
• preset_slitfunc parameter for single-pass extraction using pre-computed slit function
• noise_relative parameter for trace threshold scaling
• extraction_reject threshold parameter for outlier rejection
• ANDES_YJH instrument with multi-channel support
• MOSAIC VIS quadrant support (VIS1-VIS4 channels) and NIR settings
• HARPN fiber B wavecal file
• Extraction residual panel in ProgressPlot
• tools/plot_swath_debug.py for analyzing extraction debug data

Changed

• Mask convention: Switch to numpy convention (True/1=bad) on Python side; C code uses inverted (1=good)
• Extraction convergence now based on spectrum change with global RMS for rejection
• Improved outlier rejection using 6*sigma instead of MAD in 2D extraction
• Norm_flat step saves slit function with metadata for reuse
• Refactor Pipeline trace API: add trace_raw() and organize() methods
• Remove extraction_cutoff parameter (dead code)

Fixed

• smooth_spectrum normalization (port from cr2res)
• Masked array handling in swath debug plotting
• Minimum 4 iterations enforced in curved extraction

[0.7b3] - 2026-01-11

Added

• MOSAIC instrument support with fiber group detection and curvature step
• Extraction animation controls: pause/step buttons and speed control (PYREDUCE_PLOT_ANIMATION_SPEED)
• --plot-dir and --plot-show CLI options for flexible plot output

Changed

• Rename orders to traces in rectify and slit_curve modules for consistency
• Downgrade extraction max-iterations message from ERROR to WARNING
• Only warn about missing files for steps that are actually requested

Fixed

• Handle channel mismatch gracefully in CLI
• Curvature plotting index error when peaks need int casting

[0.7b2] - 2026-01-09

Changed

• Reorganize instrument files: All instrument-related files now in per-instrument directories
  • instruments/{name}.py → instruments/{NAME}/__init__.py
  • instruments/{name}.yaml → instruments/{NAME}/config.yaml
  • settings/settings_{NAME}.json → instruments/{NAME}/settings.json
  • wavecal/{name}_*.npz → instruments/{NAME}/wavecal_*.npz
  • masks/mask_{name}_*.fits.gz → instruments/{NAME}/mask_*.fits.gz
• Base settings and schema moved to instruments/defaults/
• Wavelength atlas files moved to instruments/defaults/atlas/

Removed

• Orphan mask files for undefined instruments (elodie, sarg, hds, etc.)

[0.7b1] - 2026-01-06

Changed

• Curvature algorithm rewrite: Replace 2D model fitting with row-tracking method for better robustness
• Rename make_shear.py to slit_curve.py
• Rename curvature coefficients: tilt/shear → p1/p2 throughout codebase
• Rename arc_extraction to simple_extraction
• Split curvature extraction_height into separate extraction_height and curve_height parameters
• Save file renamed from .shear.npz to .curve.npz

Added

• discover_channels() for automatic channel detection from data files
• CLI: --target is now optional; loops over all targets if not specified
• CLI: Uses $REDUCE_DATA for base_dir, reads default input_dir from config
• Comprehensive CLI test coverage

Fixed

• Curvature step index error when traces are removed during processing
• Validate base_dir and input_dir exist with clear error messages
• File sorting to correctly loop over all nights when not specified
• CLI dynamic dependency loading for step commands

[0.7a7] - 2026-01-04

Added

• --settings option for CLI to override reduction parameters from JSON file

Changed

• Rename extraction_width to extraction_height in settings (clarifies coordinate system)
• Rename orders to traces in extract.py internal API

[0.7a6] - 2026-01-03

Added

• --file option for CLI step commands to bypass file discovery
• NEID instrument with multi-amplifier support

Changed

• Rename orders step to trace throughout codebase (CLI, API, configs)

Fixed

• test_normflat to use column_range returned by extract

[0.7a5] - 2025-12-30

Changed

• Rename arm to channel throughout codebase (API, CLI, configs)
• Remove decker from CRIRES+ channel format (J1228_det1 instead of J1228_Open_det1)
• Example scripts now use $REDUCE_DATA env var instead of hardcoded paths

[0.7a4] - 2025-12-23

Added

• reduce examples --run flag to download and execute examples directly
• PEP 723 inline metadata in examples for uv run compatibility

[0.7a3] - 2025-12-23

Added

• reduce examples command to list/download examples from GitHub matching installed version

Changed

• CLI startup 12x faster via lazy imports (1.2s -> 0.1s)

[0.7a2] - 2025-12-23

Added

• Manual API calls documentation (manual_calls.md)

Fixed

• ReadTheDocs build failing due to missing myst_parser
• Trace module reference in documentation

Changed

• Docs Makefile now uses uv
• pyproject.toml cleanup

[0.7a1] - 2025-12-22

Added

• New Pipeline API with Pipeline.from_instrument() for simplified usage
• Click-based CLI replacing argparse (uv run reduce run UVES HD132205)
• Pydantic models for instrument configuration validation
• YAML instrument configs replacing JSON
• IPython startup script for interactive development
• plot_dir option to save plots as PNG files
• Fiber bundle tracing support for multi-fiber instruments
• filter_x and filter_type options for order tracing

Changed

• Rename mode to channel terminology throughout
• Output extension changed from .ech to .fits
• Documentation converted from RST to Markdown
• Trace detection parameters renamed (opower -> degree, filter_size -> filter_y)

Fixed

• Plotting issues with non-finite values
• Use interpolate_replace_nans for masked pixels

[0.6.0] - 2025-12-22

Added

• JSON schema validation test for instrument configurations
• GitHub workflow creates GitHub Release on tag push

Fixed

• Fix test_wavecal to match WavelengthCalibration.execute() signature
• Fix spec fixture to match ScienceExtraction.run() signature
• Fix instrument schema: replace invalid 'value' keyword with standard JSON Schema 'type'

[0.6.0b5] - 2025-10-04

Fixed

• Include *.pyd files for Windows builds
• Remove redundant build test from workflow

[0.6.0b4] - 2025-10-03

Changed

• workflow builds multi-arch wheels
• minor fixes to make build pass

[0.6.0b3] - 2025-10-02

Added

• Test organization with pytest markers (unit, instrument, slow, downloads)
• ANDES instrument configuration and settings

Changed

• CI now runs fast tests (~7s) on every push to master
• Test suite optimized with slow marker for tests >3s

Fixed

• ANDES instrument configuration errors (missing decker fields, regex pattern)
• NIRSPEC FITS header errors with illegal keywords containing dots
• Pre-commit hooks now documented and working correctly

[0.6.0b2] - 2025-10-02

Changed

• lazy imports
• cosmetic fixes like version string

[0.6.0b1] - 2025-10-02

Changed

• Modern build system using uv package manager
• Minimum Python version now 3.11, default 3.13
• Migrated build system from setuptools to Hatchling (PEP 517)
• Replaced black, isort, flake8, and pyupgrade with Ruff
• Improved code formatting across entire codebase using Ruff
• Modernized GitHub Actions workflow (uv-based, Python 3.11-3.13 matrix)
• Updated documentation to reflect modern installation with uv
• Consolidated legacy build files (setup.py, requirements.txt) into pyproject.toml
• Removed automatic PyPI publishing from GitHub Actions (now manual)
• GitHub Actions tests now manual-only on master branch
• Modernized ReadTheDocs build configuration

Fixed

• Close FITS file handles after reading to prevent resource leaks (#28)
• Store wavelength calibration in double precision instead of single (#30)
• Add fallback value for E_ORIENT header keyword (#31)
• Return final wavelength linelist with flags from calibration (#33)
• Fix MaskedArray filling in normflat routine
• Fix undefined TypeFilter import in NEID instrument
• Fix Sphinx documentation build warnings
• Fix pre-commit hook configuration issues

Documentation

• Updated README badges for GitHub Actions and Python versions
• Updated installation instructions to use uv instead of pip
• Improved documentation build process
• CLAUDE.md / AGENTS.md for AI-assisted development guidance