Repository navigation
Make the project easier to work in - #68
Merged
Merged
Conversation
Twenty-eight files were committed with CRLF endings while the other 579 text files in the repository used LF. Seven are C, twenty-one are Python, and they are spread across ten directories, so there was no rule to infer about which files were which - admm_lasso.c is CRLF and admm_lasso.py is not. Nothing depended on the difference. It only mattered when something wrote to one of these files, because an editor or script that rewrites a file whole converts the endings as a side effect, and a two line change then arrives as a diff touching all 765 lines. That is the kind of diff nobody can review, and it happened repeatedly while working on pupil_math.py, matched_filter.c and admm_lasso_c.py. The conversion is CRLF to LF and nothing else. Stripping every CR from each file before and after gives identical bytes, and git diff --ignore-cr-at-eol is empty across the whole change, which is the same statement made by git rather than by hand. The insertion and deletion counts are equal at 7140 each, as they must be when only line endings move. The .gitattributes then keeps it from coming back. text=auto lets git detect binaries itself; the explicit binary list below it is a backstop for the formats actually in the tree. The checked in Windows DLLs under storm_analysis/c_libraries are the ones worth being careful about, since rewriting those would break the wheel the Windows CI job builds. Rebuilt the C libraries and ran the suite afterwards: 280 tests, no failures.
There was nothing in the repository describing how to work on it. README.md covers installing and using the package, and the documentation covers the analysis, but the things you need in order to change the code safely were only discoverable by doing it wrong once. The file collects what actually cost time rather than what a template would suggest. The build section says to re-run scons after editing C, because the Python side loads whatever is already in c_libraries/ and a stale library produces a plausible wrong answer instead of an error. The test section says to trust the exit status, because several C libraries flush their stdout at interpreter exit and push pytest's summary line off the end of the output. It also records that pytest does not run the __main__ blocks, which is how a renamed test stayed green across six pull requests. The conventions section is short and only lists things measured across the tree: all 242 modules that import numpy write 'import numpy', and no file anywhere uses 'as np'; class methods are camelCase 704 times against 6 in snake_case. Keyword argument spacing is explicitly called out as *not* a convention - it is close to evenly split and 108 of 254 files use both forms, so an agent tidying it in one direction would be inventing a rule. The traps section names decon_storm/, which has C sources that SConstruct never builds and nothing imports. That directory is why the scan behind 76080b6a first reported the wrong getPeaks: it kept the first definition matching each name and found the dead one. Nothing here changes code. Every path and count in the file was checked against the tree rather than recalled, and the claim that the diagnostics other than multiplane take no arguments was wrong when first written - fista_decon, spliner and spliner_2d all accept --no-splines.
e644347 added a note that multiplane/configure.py needs --psf-model and said "the rest take no arguments". That is not true: fista_decon, spliner, spliner_2d and multiplane itself all accept --no-splines. The distinction that matters is required versus optional, since only the missing --psf-model stops a script from running, so say it that way.
The Spliner section said to create "a text file containing the z value of each frame in the z stack movie". measure_psf_beads.py does not accept that. It reads the file with numpy.loadtxt and passes the result to makeZIndexArray(), which asserts a shape of (N,2): a valid flag in the first column and the z offset in microns in the second, with frames whose flag is 0 dropped from the measurement. Following the instructions as written therefore ends at AssertionError: Z offsets must have shape (N,2). with nothing to suggest what the second column should be. offset_to_z.py writes the correct format, but it was only mentioned in the Multiplane section a hundred lines further down, and it reads a storm-control .off file so it is not a general answer either. The bead location file had a smaller version of the same problem. It was described only as "a text file containing the locations of a few (isolated) beads", and the format - one space separated x y pair per line, in pixels - appeared nowhere outside a comment in the source. Both now have a stated format and a short example. Multiplane step 5 described its z offset file the same loose way and takes the same two columns, so it now refers to the Spliner description rather than repeating it. Also fixes beads_zoffset.text, which is spelled beads_zoffset.txt two lines below it and everywhere else. The documentation builds without warnings, as it did before.
The file covered how to change this code and said nothing about how to drive it, but "analyze this movie with 3D-DAOSTORM in 2d fixed mode" is a perfectly ordinary request and the answer was spread across analysis.rst, parameters.rst and the sample files. Running one of these turns out to work fine as documented. What is missing is not procedure, it is the two things that go wrong quietly: Parameter files are always copied from an existing one, and four of the parameters in them describe the microscope rather than the analysis - pixel_size, camera_gain, camera_offset and the initial sigma. Nothing compares them against the movie, so inheriting them from a sample file gives a clean run, a full set of localizations, and the wrong units. That is worth calling out where someone will read it before starting. The other is that parameter problems are reported at the top of the output. An unrecognized parameter prints a warning as the first line, which is then followed by several hundred lines of per-frame progress, so anything reading the tail of the output will conclude the run was clean. The warning already exists and works; it is just easy to miss. Also records that Spliner and Multiplane need a measured PSF first and that the text files for it have formats worth reading rather than guessing, and that multi_plane.py takes --basename instead of --movie since it loads one movie per channel. Entry points, the analyze() signature, the sample XML naming and the four parameter names were checked against the tree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the repository easier to work in, whether the reader is a person arriving
for the first time or an automated tool. Five commits, one concern each. No
analysis code changes; 280 tests green and the documentation builds without
warnings.
Line endings
Twenty-eight files were committed with CRLF while the other 579 text files used
LF, spread across ten directories with no rule to infer -
admm_lasso.cis CRLFand
admm_lasso.pyis not. Nothing depended on it, but any tool that rewrites afile whole converts the endings as a side effect, so a two line change arrives as
a diff touching all 765 lines. That happened repeatedly during the recent audit
work.
The conversion is CRLF to LF and nothing else:
git diff --ignore-cr-at-eolis empty across the whole change.gitattributeskeeps it from returning.text=autolets git detect binariesitself, with an explicit list underneath as a backstop - the checked-in Windows
DLLs matter most, since rewriting those would break the wheel the Windows job
builds. A fresh clone now comes out with zero CRLF files.
AGENTS.md
There was nothing describing how to work on this project. The new file covers
build, tests, running an analysis, diagnostics, layout, conventions and traps,
and every count and path in it was checked against the tree.
It records the things that are only learnable by getting them wrong once:
sconsafter editing C, because the Python side loads whatever isalready in
c_libraries/and a stale library gives a plausible wrong answerrather than an error
exit and push the
N passedline off the end of the output__main__blocks, which is how a renamed test stayedgreen across six pull requests
decon_storm/has C sources thatSConstructnever builds and nothingimports, so a definition found there is not the one that runs
The conventions section lists only what was measured: all 242 modules that
import numpy write
import numpy, andas npappears nowhere; class methodsare camelCase 704 times against 6 in snake_case. Keyword argument spacing is
explicitly called out as not a convention - it is close to evenly split and
108 of 254 files use both forms, so tidying it either way would be inventing a
rule.
PSF measurement input files
This is a real documentation defect rather than a polish item.
analysis.rstsaid to create "a text file containing the z value of each frame in the z stack
movie".
measure_psf_beads.pypasses that file tomakeZIndexArray(), whichasserts a shape of
(N,2): a valid flag and the z offset in microns, withflag-0 frames dropped. Following the instructions as written ends at
with nothing to indicate what the second column is.
offset_to_z.pywrites theright format but was mentioned only in the Multiplane section a hundred lines
below, and it reads a storm-control
.offfile so it is not a general answer.The bead location file had a smaller version of the same problem - described
only as "a text file containing the locations of a few (isolated) beads", with
the format living in a source comment. Both now have a stated format and an
example, verified against the files the spliner diagnostic actually generates.
Multiplane's step 5 takes the same two columns and now refers to the Spliner
description instead of restating it loosely.
Also fixes
beads_zoffset.text, spelled.txttwo lines below and everywhereelse.
Running an analysis
AGENTS.mdcovered changing this code and said nothing about driving it, though"analyze this movie with 3D-DAOSTORM in 2d fixed mode" is an ordinary request.
Running one works as documented; what is missing is the two things that go wrong
quietly.
Parameter files are always copied from an existing one, and four of the values
in them describe the microscope rather than the analysis -
pixel_size,camera_gain,camera_offsetand the initialsigma. Nothing compares themagainst the movie, so inheriting them from a sample gives a clean run, a full
set of localizations and the wrong units.
The other is that parameter problems are reported at the top of the output. An
unrecognized parameter already prints a warning, correctly, as the first line -
followed by several hundred lines of per-frame progress, so anything reading the
tail concludes the run was clean.
Diagnostics README
e644347esaid "the rest take no arguments" when describing--psf-model.fista_decon,spliner,spliner_2dandmultiplaneall accept--no-splines. Only the missing--psf-modelactually stops a script, so thenote now distinguishes required from optional.