Skip to content

Fix stride computation for dimensions with shape 0 in ndarray - #68

Open
cgarling wants to merge 5 commits into
mainfrom
roman-l2-check
Open

Fix stride computation for dimensions with shape 0 in ndarray#68
cgarling wants to merge 5 commits into
mainfrom
roman-l2-check

Conversation

@cgarling

@cgarling cgarling commented Aug 6, 2026

Copy link
Copy Markdown
Member

Some STScI Roman L2 .asdf files produced with romanisim contain !core/ndarray nodes with a zero-length shape (e.g. the chisq/dumo arrays, shape [0, 0]) and no explicit strides key. ASDF.jl's implicit C-order stride formula (stride[i] = itemsize * prod(shape[i+1:])) collapses to zero for any outer dimension whose product includes a zero-length axis, which then failed the constructor's "strides must be positive" check.

NumPy avoids this by treating zero-length dimensions as length 1 only within the running product when computing default C-contiguous strides (PyArray_NewFromDescr). The Python asdf package relies on this. Here I reproduce that convention, and have verified it against NumPy's actual output for several representative shapes/dtypes.

I also relax the post-materialization stride sanity check in getindex: Julia's reshape/reinterpret don't preserve stride values along size-0 or size-1 axes (no adjacent elements to space apart), so the check now only compares strides for dimensions with more than one element.

I verified this patch on my local L2 .asdf files: all arrays materialize with correct shapes/dtypes, and pixel values match Python's asdf exactly (accounting for Julia's column-major vs. NumPy's row-major indexing convention).

I added regression tests in test/test-ndarray.jl covering the implicit stride computation for zero-length dimensions and materialization of a zero-size block-backed array. The rest of the tests also pass for me locally.

The comments are somewhat verbose so I'm happy if you want to cut them down to size.

@codecov

codecov Bot commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (fd82f70) to head (3bda108).

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #68   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            1         1           
  Lines          583       590    +7     
=========================================
+ Hits           583       590    +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@icweaver
icweaver self-requested a review August 13, 2026 00:41

@icweaver icweaver left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @cgarling! I've left my light review comments for this really nice PR

Comment thread src/ASDF.jl
Comment on lines +788 to +799
# Calculate byte strides in C order. Any dimension of length zero is treated as
# length 1 within this product only (not in `shape` itself); this matches NumPy's
# convention for computing default C-contiguous strides (`PyArray_NewFromDescr`),
# relied on by the reference Python `asdf` package when constructing arrays via
# `np.ndarray(shape, dtype, data, offset, None, order)`. Without the clamp, any
# zero-length dimension collapses the strides of every outer dimension whose
# product includes it down to zero, which then fails the `strides` positivity
# check below even though no data is ever read from a zero-size array. Negative
# entries are left unclamped so the `shape` negativity check below still reports
# them with its own clear error message. STScI Roman L2 `.asdf` products contain
# zero-shape arrays (e.g. `chisq`, `dumo`) with no explicit `strides` key,
# triggering this exact failure prior to the fix.

@icweaver icweaver Aug 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Calculate byte strides in C order. Any dimension of length zero is treated as
# length 1 within this product only (not in `shape` itself); this matches NumPy's
# convention for computing default C-contiguous strides (`PyArray_NewFromDescr`),
# relied on by the reference Python `asdf` package when constructing arrays via
# `np.ndarray(shape, dtype, data, offset, None, order)`. Without the clamp, any
# zero-length dimension collapses the strides of every outer dimension whose
# product includes it down to zero, which then fails the `strides` positivity
# check below even though no data is ever read from a zero-size array. Negative
# entries are left unclamped so the `shape` negativity check below still reports
# them with its own clear error message. STScI Roman L2 `.asdf` products contain
# zero-shape arrays (e.g. `chisq`, `dumo`) with no explicit `strides` key,
# triggering this exact failure prior to the fix.
# Calculate byte strides in C order, treating any zero-length dimension as
# length 1 within this product only (not in `shape` itself). This matches NumPy's
# convention for default C-contiguous strides (`PyArray_NewFromDescr`),
# relied on by the reference Python `asdf` package. Without the clamp, a
# zero-length dimension collapses the stride of every outer dimension whose
# product includes it down to zero, which then fails the `strides` positivity
# check below even though no data is ever read from a zero-size array. STScI Roman L2 `.asdf` products contain such
# zero-shape arrays (e.g. `chisq`, `dumo`) with no explicit `strides` key.

@cgarling cgarling Aug 15, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would keep "Negative entries are left unclamped so the shape negativity check below still reports them with its own clear error message.", see comment below

Comment thread src/ASDF.jl
Comment on lines +801 to +802
clamped_shape = [s == 0 ? 1 : s for s in shape[(begin + 1):end]]
strides = reverse(cumprod([sz; reverse(clamped_shape)]))

@icweaver icweaver Aug 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
clamped_shape = [s == 0 ? 1 : s for s in shape[(begin + 1):end]]
strides = reverse(cumprod([sz; reverse(clamped_shape)]))
strides = reverse(cumprod([sz; reverse(max.(shape[(begin + 1):end], 1))]))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is type-stable and requires fewer changes to the original line here

@cgarling cgarling Aug 15, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this first but the max will also hit any negative entries in shape which I thought we should avoid, thus the comment above Negative entries are left unclamped so the shape negativity check below still reports them with its own clear error message.. If we use max negatives in shape will no longer error; lines 746-748

        if any(shape .< 0)
            throw(ArgumentError("`shape` cannot have negative elements."))
        end

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if I follow. Even when a negative shape entry is clamped within max, it is still passed unmodified to the inner constructor for NDArray afterwards, where the check you pointed to on L746-748 would throw as expected, no? That is what I am seeing on my end at least with the max.(...) version:

julia> let
           shape = Int64[-2, 3] # [2, -3], [-3], etc.
           strides = nothing
           ASDF.NDArray(ASDF.LazyBlockHeaders(), Int64(0), nothing, shape, ASDF.Datatype_float32, ASDF.host_byteorder, Int64(0), strides)
       end
ERROR: ArgumentError: `shape` cannot have negative elements.

Comment thread src/ASDF.jl Outdated
Comment thread src/ASDF.jl Outdated
Comment thread test/test-ndarray.jl Outdated
Comment thread test/test-ndarray.jl Outdated
cgarling and others added 4 commits August 15, 2026 10:58
Co-authored-by: Ian Weaver <weaveric@gmail.com>
Co-authored-by: Ian Weaver <weaveric@gmail.com>
Co-authored-by: Ian Weaver <weaveric@gmail.com>
Co-authored-by: Ian Weaver <weaveric@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants