Skip to content

Make the project easier to work in - #68

Merged
HazenBabcock merged 5 commits into
ZhuangLab:masterfrom
HazenBabcock:agent-friendly
Sep 8, 2026
Merged

HazenBabcock merged 5 commits into
ZhuangLab:masterfrom
HazenBabcock:agent-friendly

Conversation

@HazenBabcock

Copy link
Copy Markdown
Member

Makes the repository easier to work in, whether the reader is a person arriving
for the first time or an automated tool. Five commits, one concern each. No
analysis code changes; 280 tests green and the documentation builds without
warnings.

Line endings

Twenty-eight files were committed with CRLF while the other 579 text files used
LF, spread across ten directories with no rule to infer - admm_lasso.c is CRLF
and admm_lasso.py is not. Nothing depended on it, but any tool that rewrites a
file whole converts the endings as a side effect, so a two line change arrives as
a diff touching all 765 lines. That happened repeatedly during the recent audit
work.

The conversion is CRLF to LF and nothing else:

  • stripping every CR from each file before and after gives identical bytes
  • git diff --ignore-cr-at-eol is empty across the whole change
  • insertions and deletions are equal at 7140 each, as they must be

.gitattributes keeps it from returning. text=auto lets git detect binaries
itself, with an explicit list underneath as a backstop - the checked-in Windows
DLLs matter most, since rewriting those would break the wheel the Windows job
builds. A fresh clone now comes out with zero CRLF files.

AGENTS.md

There was nothing describing how to work on this project. The new file covers
build, tests, running an analysis, diagnostics, layout, conventions and traps,
and every count and path in it was checked against the tree.

It records the things that are only learnable by getting them wrong once:

  • re-run scons after editing C, because the Python side loads whatever is
    already in c_libraries/ and a stale library gives a plausible wrong answer
    rather than an error
  • trust pytest's exit status - several C libraries flush stdout at interpreter
    exit and push the N passed line off the end of the output
  • pytest does not run the __main__ blocks, which is how a renamed test stayed
    green across six pull requests
  • decon_storm/ has C sources that SConstruct never builds and nothing
    imports, so a definition found there is not the one that runs

The conventions section lists only what was measured: all 242 modules that
import numpy write import numpy, and as np appears nowhere; class methods
are camelCase 704 times against 6 in snake_case. Keyword argument spacing is
explicitly called out as not a convention - it is close to evenly split and
108 of 254 files use both forms, so tidying it either way would be inventing a
rule.

PSF measurement input files

This is a real documentation defect rather than a polish item. analysis.rst
said to create "a text file containing the z value of each frame in the z stack
movie". measure_psf_beads.py passes that file to makeZIndexArray(), which
asserts a shape of (N,2): a valid flag and the z offset in microns, with
flag-0 frames dropped. Following the instructions as written ends at

AssertionError: Z offsets must have shape (N,2).

with nothing to indicate what the second column is. offset_to_z.py writes the
right format but was mentioned only in the Multiplane section a hundred lines
below, and it reads a storm-control .off file so it is not a general answer.

The bead location file had a smaller version of the same problem - described
only as "a text file containing the locations of a few (isolated) beads", with
the format living in a source comment. Both now have a stated format and an
example, verified against the files the spliner diagnostic actually generates.
Multiplane's step 5 takes the same two columns and now refers to the Spliner
description instead of restating it loosely.

Also fixes beads_zoffset.text, spelled .txt two lines below and everywhere
else.

Running an analysis

AGENTS.md covered changing this code and said nothing about driving it, though
"analyze this movie with 3D-DAOSTORM in 2d fixed mode" is an ordinary request.
Running one works as documented; what is missing is the two things that go wrong
quietly.

Parameter files are always copied from an existing one, and four of the values
in them describe the microscope rather than the analysis - pixel_size,
camera_gain, camera_offset and the initial sigma. Nothing compares them
against the movie, so inheriting them from a sample gives a clean run, a full
set of localizations and the wrong units.

The other is that parameter problems are reported at the top of the output. An
unrecognized parameter already prints a warning, correctly, as the first line -
followed by several hundred lines of per-frame progress, so anything reading the
tail concludes the run was clean.

Diagnostics README

e644347e said "the rest take no arguments" when describing --psf-model.
fista_decon, spliner, spliner_2d and multiplane all accept
--no-splines. Only the missing --psf-model actually stops a script, so the
note now distinguishes required from optional.

Twenty-eight files were committed with CRLF endings while the other 579
text files in the repository used LF. Seven are C, twenty-one are Python,
and they are spread across ten directories, so there was no rule to infer
about which files were which - admm_lasso.c is CRLF and admm_lasso.py is
not.

Nothing depended on the difference. It only mattered when something wrote
to one of these files, because an editor or script that rewrites a file
whole converts the endings as a side effect, and a two line change then
arrives as a diff touching all 765 lines. That is the kind of diff nobody
can review, and it happened repeatedly while working on pupil_math.py,
matched_filter.c and admm_lasso_c.py.

The conversion is CRLF to LF and nothing else. Stripping every CR from
each file before and after gives identical bytes, and

  git diff --ignore-cr-at-eol

is empty across the whole change, which is the same statement made by
git rather than by hand. The insertion and deletion counts are equal at
7140 each, as they must be when only line endings move.

The .gitattributes then keeps it from coming back. text=auto lets git
detect binaries itself; the explicit binary list below it is a backstop
for the formats actually in the tree. The checked in Windows DLLs under
storm_analysis/c_libraries are the ones worth being careful about, since
rewriting those would break the wheel the Windows CI job builds.

Rebuilt the C libraries and ran the suite afterwards: 280 tests, no
failures.
There was nothing in the repository describing how to work on it. README.md
covers installing and using the package, and the documentation covers the
analysis, but the things you need in order to change the code safely were
only discoverable by doing it wrong once.

The file collects what actually cost time rather than what a template would
suggest. The build section says to re-run scons after editing C, because the
Python side loads whatever is already in c_libraries/ and a stale library
produces a plausible wrong answer instead of an error. The test section says
to trust the exit status, because several C libraries flush their stdout at
interpreter exit and push pytest's summary line off the end of the output.
It also records that pytest does not run the __main__ blocks, which is how a
renamed test stayed green across six pull requests.

The conventions section is short and only lists things measured across the
tree: all 242 modules that import numpy write 'import numpy', and no file
anywhere uses 'as np'; class methods are camelCase 704 times against 6 in
snake_case. Keyword argument spacing is explicitly called out as *not* a
convention - it is close to evenly split and 108 of 254 files use both forms,
so an agent tidying it in one direction would be inventing a rule.

The traps section names decon_storm/, which has C sources that SConstruct
never builds and nothing imports. That directory is why the scan behind
76080b6a first reported the wrong getPeaks: it kept the first definition
matching each name and found the dead one.

Nothing here changes code. Every path and count in the file was checked
against the tree rather than recalled, and the claim that the diagnostics
other than multiplane take no arguments was wrong when first written -
fista_decon, spliner and spliner_2d all accept --no-splines.
e644347 added a note that multiplane/configure.py needs --psf-model and
said "the rest take no arguments". That is not true: fista_decon, spliner,
spliner_2d and multiplane itself all accept --no-splines.

The distinction that matters is required versus optional, since only the
missing --psf-model stops a script from running, so say it that way.
The Spliner section said to create "a text file containing the z value of
each frame in the z stack movie". measure_psf_beads.py does not accept
that. It reads the file with numpy.loadtxt and passes the result to
makeZIndexArray(), which asserts a shape of (N,2): a valid flag in the
first column and the z offset in microns in the second, with frames
whose flag is 0 dropped from the measurement.

Following the instructions as written therefore ends at

  AssertionError: Z offsets must have shape (N,2).

with nothing to suggest what the second column should be. offset_to_z.py
writes the correct format, but it was only mentioned in the Multiplane
section a hundred lines further down, and it reads a storm-control .off
file so it is not a general answer either.

The bead location file had a smaller version of the same problem. It was
described only as "a text file containing the locations of a few
(isolated) beads", and the format - one space separated x y pair per
line, in pixels - appeared nowhere outside a comment in the source.

Both now have a stated format and a short example. Multiplane step 5
described its z offset file the same loose way and takes the same two
columns, so it now refers to the Spliner description rather than
repeating it.

Also fixes beads_zoffset.text, which is spelled beads_zoffset.txt two
lines below it and everywhere else.

The documentation builds without warnings, as it did before.
The file covered how to change this code and said nothing about how to
drive it, but "analyze this movie with 3D-DAOSTORM in 2d fixed mode" is a
perfectly ordinary request and the answer was spread across analysis.rst,
parameters.rst and the sample files.

Running one of these turns out to work fine as documented. What is missing
is not procedure, it is the two things that go wrong quietly:

Parameter files are always copied from an existing one, and four of the
parameters in them describe the microscope rather than the analysis -
pixel_size, camera_gain, camera_offset and the initial sigma. Nothing
compares them against the movie, so inheriting them from a sample file
gives a clean run, a full set of localizations, and the wrong units. That
is worth calling out where someone will read it before starting.

The other is that parameter problems are reported at the top of the
output. An unrecognized parameter prints a warning as the first line,
which is then followed by several hundred lines of per-frame progress, so
anything reading the tail of the output will conclude the run was clean.
The warning already exists and works; it is just easy to miss.

Also records that Spliner and Multiplane need a measured PSF first and
that the text files for it have formats worth reading rather than
guessing, and that multi_plane.py takes --basename instead of --movie
since it loads one movie per channel.

Entry points, the analyze() signature, the sample XML naming and the four
parameter names were checked against the tree.
@HazenBabcock
HazenBabcock merged commit b4169db into ZhuangLab:master Sep 8, 2026
5 checks passed
@HazenBabcock
HazenBabcock deleted the agent-friendly branch September 8, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant