Skip to content

FIX spectral indices to use band values as read, with radiometric scaling - #52

Merged
aradfarahani merged 1 commit into
hotfix/afri-formulafrom
hotfix/spectral-index-radiometry
Aug 21, 2026
Merged

FIX spectral indices to use band values as read, with radiometric scaling#52
aradfarahani merged 1 commit into
hotfix/afri-formulafrom
hotfix/spectral-index-radiometry

Conversation

@mkiani12

Copy link
Copy Markdown
Collaborator

Stacked on #51 — review #50 and #51 first; this PR targets hotfix/afri-formula.

Problem

All six index calculators read from FileHandler.get_normalized_bands():

(image - np.min(image)) / (np.max(image) - np.min(image))

applied independently to each band. Each band therefore receives a different affine transform, which changes the relationships between bands — and those relationships are the entire physical content of a spectral index. (NIR' - Red') / (NIR' + Red') where NIR' and Red' were rescaled by different factors is simply not NDVI.

Reproduced on example/data/:

before after
NDVI mean 0.3351 0.5103
pixels NDVI > 0.3 55.64% 59.88%
max abs difference vs rasterio reference 0.7453 0.0
same pixel, full scene vs 400×400 crop 0.8373 / 0.8692 identical
AFRI vs NDVI correlation −0.6927 +0.7117
BSI vs NDVI correlation +0.3908 −0.6507

The last two are the ones I find most telling. Karnieli et al. report AFRI and NDVI as "almost identical" under clear sky, and bare ground must oppose canopy. Both relationships held on the stored values and inverted under the rescale — the index was not merely rescaled, it was reporting the opposite of the physical truth. Those are the two indices whose formulas #51 just corrected; they only start behaving correctly here.

The extent dependence is the reproducibility failure: loading a crop changed the answer for pixels the crop shared with the full scene.

Changes

1. Indices compute on values as read. New FileHandler.get_bands() returns the arrays already held in self.bands, mirroring the existing accessor pattern. The six calculators switch to it; self.normalized_bands becomes self.source_bands.

Min–max normalization is untouched for the enhancement, HSV, PCA and SVM modules, where rescaling is appropriate — and for SVM it is genuinely required, since an RBF kernel needs comparable feature scales.

2. Radiometric scaling, because removing the rescale is not sufficient on its own. It leaves the indices computing on digital numbers, and two of them carry constants defined in reflectance units:

Index Constant Safe on raw DN?
NDVI, NDWI, UI, BSI none Yes — invariant to a gain applied to all bands equally
SAVI soil adjustment L = 0.5 No
AFRI coefficients 0.66 / 0.50 on SWIR No

Adding L = 0.5 to a DN in the thousands contributes nothing, so SAVI on unscaled input is silently not SAVI. Every index now accepts scale_factor/offset, applied as ρ = DN × s + o, with published values for the common analysis-ready products shipped as RADIOMETRIC_PRESETS:

SAVICalculator(nir_path=..., red_path=..., **RADIOMETRIC_PRESETS["landsat-c2-l2"])

Covering Landsat C2 L2 (2.75e-5, −0.2), Sentinel-2 L2A (1e-4, 0), and Sentinel-2 L2A baseline ≥ 04.00 (1e-4, −0.1, for the BOA_ADD_OFFSET introduced in that baseline). SAVI and AFRI emit a UserWarning when handed values far outside the reflectance range, so the silent-nonsense case becomes loud.

Behaviour change

Index values change for every user — deliberately, and documented prominently in docs/spectral-indices.md. The previous values were scene-dependent and not the cited indices, so preserving them would preserve the defect. Defaults are scale_factor=1.0, offset=0.0, so input already in reflectance needs nothing passed.

Tests

347 pass (334 on #51, 13 new). tests/tools/spectral_indices/radiometry_test.py covers:

  • Extent invariance — a pixel's NDVI is identical for the full array and a crop containing it, plus a companion test asserting the same comparison fails under min–max, so the first is not vacuous.
  • Gain invariance — NDVI unchanged when all bands are multiplied by 10000, the property that makes normalized differences safe on DN.
  • Physical relationships on the bundled Landsat subset — AFRI must track NDVI (> 0.5) and BSI must oppose it (< 0), with both asserted to invert under per-band rescaling. These read the real bands via rasterio and skip if unavailable; synthetic noise has no spectral structure to preserve, so this property can only be tested on genuinely correlated bands. My first attempt used random arrays and was measuring nothing.
  • Scaling arithmetic, calculator-level application, and the SAVI/AFRI warnings firing on DN while NDVI stays silent.

The six existing index test fixtures were updated from get_normalized_bands/normalized_bands to get_bands/source_bands.

…ling

All six index calculators obtained their inputs from
FileHandler.get_normalized_bands(), which applies a min-max rescale
independently to each band. Because every band received a different affine
transform, this altered the relationships between bands - and those
relationships are the entire physical content of a band ratio.

Three consequences, all reproducible on the bundled example:

  * Published thresholds did not apply. NDVI mean was 0.3351 against 0.5103
    for the standard definition, and the fraction above 0.3 read 55.64%
    instead of 59.88%.
  * A pixel's value depended on how much of the image was loaded, since the
    rescale used the loaded extent's own extrema. The same pixel gave 0.8373
    on the full scene and 0.8692 on a 400x400 crop.
  * Inter-band relationships inverted outright. AFRI's correlation with NDVI
    ran +0.71 on the values as stored and -0.69 after rescaling, against a
    source paper reporting the two as nearly identical under clear sky.

Add FileHandler.get_bands(), returning the arrays as read, and switch the six
indices to it. Min-max normalization is untouched for the enhancement, HSV,
PCA and SVM modules, where rescaling is appropriate - and for SVM it is
required, since an RBF kernel needs comparable feature scales.

Removing the rescale is necessary but not sufficient: it leaves the indices
computing on digital numbers, and two of them carry constants defined in
reflectance units. SAVI's soil adjustment L = 0.5 and AFRI's 0.66/0.50
coefficients contribute nothing when added to a DN in the thousands, so the
result is silently not the cited index. Normalized differences (NDVI, NDWI,
UI, BSI) are invariant to a common gain and remain valid on raw DN.

Add scale_factor/offset to every index, applied as rho = DN * s + o, with
published values for the common analysis-ready products exposed as
RADIOMETRIC_PRESETS (Landsat C2 L2, Sentinel-2 L2A before and after
processing baseline 04.00). SAVI and AFRI warn when handed values far outside
the reflectance range.

NDVI now reproduces a rasterio-computed reference exactly, and is identical
between a full scene and a crop.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants