Skip to content

Carry the track length and start frame through to Insight3 - #63

Merged
HazenBabcock merged 1 commit into
ZhuangLab:masterfrom
HazenBabcock:fix-track-export
Sep 8, 2026
Merged

HazenBabcock merged 1 commit into
ZhuangLab:masterfrom
HazenBabcock:fix-track-export

Conversation

@HazenBabcock

Copy link
Copy Markdown
Member

bin_to_lmchallenge_format.py expands a molecule that was on for several
frames back into one row per frame:

track_length = i3_block['tl'][i]
for j in range(track_length):
    fr = i3_block['fr'][i] + j

That worked against an Insight3 era file, where tracker.c wrote TLEN,
avemlist.c flagged it NOAVERAGE so it survived averaging, and
insight.h defines FRAME as "frame the peak first occured in".

hdf5ToBin(), which replaced that path in Dec 2017, wrote neither field, so
the expansion was silently a no-op:

in the HDF5 before after
track_length / tl 1, 3, 10 1, 1, 1 1, 3, 10
frame_number / fr 0, 4, 20 1, 1, 1 1, 5, 21
rows out of lmchallenge 14 3 14

Both halves have to change together, which is why #59 deliberately left tl
alone: each expanded row is given the whole of a, so restoring the track
length on its own would have reported a ten frame track's full photon count
ten times over. a is a track total in the files this package writes, and in
Insight3 era ones too since avemlist flagged AREA as TOTAL, so each row
now gets its share.

Changes

  • createFromMultiFit() maps frame_number onto fr and track_length
    onto tl, which is where every other HDF5 field is already translated.
    Insight3 frames are 1-based and HDF5 frames are 0-based, hence the +1.
    Localizations carry neither field, so that path is untouched.
  • bin_to_lmchallenge_format.py divides a by the track length.

The converter was executable-only — all module level code reading sys.argv
at import — so nothing could test it. It is now binToLMChallenge() with a
__main__ guard, like every other utility here, and returns the row count.
That accounts for most of its diff.

The new test runs the whole path, HDF5 to .bin to challenge text, and
checks that each track occupies consecutive frames from its own start frame
and that each row carries its per-frame photons. It fails on the parent
commit.

Note that a track with a gap in it is still expanded into consecutive frames.
The .bin format records only where a track started and how long it was, so
the information needed to do better is not there.


Full test suite passes, 278 tests.

bin_to_lmchallenge_format.py expands a molecule that was on for several
frames back into one row per frame:

    track_length = i3_block['tl'][i]
    for j in range(track_length):
        fr = i3_block['fr'][i] + j

That worked against an Insight3 era file. tracker.c wrote
object_data_int[TLEN] = cur->track_length, avemlist.c flagged TLEN as
NOAVERAGE so it survived averaging, and insight.h defines FRAME as "frame
the peak first occured in".

hdf5ToBin(), which replaced that path in Dec 2017, wrote neither.
createFromMultiFit() never touched 'tl', so it kept its default of 1, and
every track was written into frame 1. The expansion was silently a no-op:

                              in the HDF5      exported
    track_length / tl          1, 3, 10        1, 1, 1
    frame_number / fr          0, 4, 20        1, 1, 1
    rows out of lmchallenge          14               3

Map both fields in createFromMultiFit(), which is where every other HDF5
field is translated. Insight3 frame numbers start at 1 and HDF5 frame
numbers start at 0, hence the +1. Localizations carry neither field so
that path is unchanged.

Restoring 'tl' on its own would have made things worse, which is why
2e43489 deliberately left it alone: each expanded row was given the
whole of 'a', so a ten frame track would have reported its full photon
count ten times. 'a' is a track total in our files, and in Insight3 era
ones too since avemlist flagged AREA as TOTAL, so each row now gets its
share.

bin_to_lmchallenge_format.py was executable-only, all module level code
reading sys.argv at import, so nothing could test it. Wrapped in
binToLMChallenge() with a __main__ guard, like every other utility here,
and it returns the row count.

Note that a track with a gap in it is still expanded into consecutive
frames. The .bin format records only where a track started and how long
it was, so the information needed to do better is not there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@HazenBabcock
HazenBabcock merged commit efe9b40 into ZhuangLab:master Sep 8, 2026
5 checks passed
@HazenBabcock
HazenBabcock deleted the fix-track-export branch September 8, 2026 22:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant