gotchas.md

Gotchas

• 2026-06-24: Standard ROI-bank gamma update must use ROI-bank vector-PSF Poisson projection end to end. Do not reintroduce Gaussian sigma/blob projections or Gaussian quicklook fallbacks on the formal gamma route.
• 2026-06-24: When debugging the standard run, isolate epoch-30 ROI library construction first: old-route parity is half-FOV domain crop -> 128x128 tiled loc inference -> FiLM/SoftMoE domain condition -> p-threshold emitters -> 128x128 ROI bank. Do not advance gamma/reconstruction debugging until this stage is verified.
• 2026-06-26: Active SMLM 10-channel output must stay in old IWAE order: p, phot_mu, x_mu, y_mu, z_mu, phot_sig, x_sig, y_sig, z_sig, bg. GMM target/order is phot,x,y,z; do not silently switch back to x,y,z,phot.
• 2026-06-26: Formal old-IWAE GMMLoss metrics expose only loss_gmm, loss_bkg, and loss_total. Count and mixture-localization contributions are internal to loss_gmm; do not log them as separate training loss components.
• 2026-07-06: When aligning Neptune v0.3 fast-route training to 3052, check both scheduler step unit and scheduler parameters. 3052 uses StepLR(step_size=10, gamma=0.9) stepped once per epoch; setting only lr_step_unit=epoch while leaving lr_step_size=1000 is not true 3052 LR parity.
• 2026-07-08: For ncp center400, each channel should use its own 400x400 center crop: left full-frame x=100:500,y=400:800 and right full-frame x=700:1100,y=400:800. Keep high-quality zmap bootstrap crops, real_tiff_wake domains, ROI-bank domains, and zmap NPZ shapes aligned; otherwise physical update can fail late with conditioning patch out-of-bounds.
• 2026-07-11: Do not quote worst-case storage for a benchmark as the required capacity. Separate the streaming single-seed minimum from the fully materialized multi-seed upper bound. For the microtube-right five-method benchmark, online generation and chunked evaluation need about 50 GB minimum / 80 GB recommended, not 200-300 GB.
• 2026-07-23: When a Neptune v0.3 request asks to process a 500x500 ROI through the standard route, treat 500x500 as the per-channel training and physical-model domain. Recompute the channel-specific high-quality zmaps and train from scratch; do not reinterpret it as an inference-only crop using an existing checkpoint.
• 2026-08-02: UnityPSF astigmatism must preserve fully independent left/right network ownership. The two channel experts may copy the same initial neural weights and use the same anchor policy, but each must build its own peak zmap and independently own backbone, FiLM, heads, optimizer, physical state, and checkpoint. A UnityPSF bundle may index both checkpoints; do not silently replace them with a shared astigmatism backbone.
• 2026-08-04: UnityPSF paper figures must foreground scientific evidence objects: axial PSF manifolds, sample-wise routing, and channel-isolated physical states. Keep expert placement and checkpoint structure subordinate; avoid equal-weight software-box diagrams as the primary paper figure.
• 2026-08-04: Expert Parallel is an execution strategy, not the UnityPSF model definition. The primary architecture figure must center the unified input contract, modality/channel hard router, complete expert internals, independent channel instances, and common localization output; show GPU placement only as a subordinate inset.
• 2026-08-04: The primary UnityPSF figure must read first as one joint checkpoint and one PSF-general foundation model. Router, expert bank, channel instances, and calibration states are internal cutaways of that single model identity; do not let separate expert boxes visually imply separate user-facing models.
• 2026-08-04: UnityPSF development must be visually auditable. Every formal training milestone needs modality/channel-separated input audits, route counts, loss curves, localization overlays, reconstructions, physical-state figures, summary.json, and a static report. First freeze the real dual-modality/multichannel milestone with Origami Emitter2D main plus Astigmatism left/right; do not wait for DH data or hide a failing branch behind averaged metrics.
• 2026-08-04: Keep the UnityPSF engineering and scientific gates separate. A synthetic multi-GPU smoke may freeze routing, optimizer ownership, joint-checkpoint commit, reload, and report generation, but it is not the dual-modality scientific baseline; that baseline requires real Origami plus independent Astigmatism left/right peak-zmap/gamma states and human review of modality-separated results.
• 2026-08-04: Origami 2D is also a spatially split dual-channel acquisition. Its 1200x1200 raw frames map to left/right measurement channels, and the bead PSF_aligned.tif stores two adjacent 27x27 PSF patches. Formal UnityPSF planning must therefore instantiate Emitter2DExpert(left) and Emitter2DExpert(right) independently; do not collapse Origami back to Emitter2DExpert(main).
• 2026-08-05: The corrected Neptune v0.3-aligned routing contract supersedes the earlier per-channel-network entries above. Route only by PSF modality: one shared network and optimizer per modality jointly alternate left/right batches, while each channel keeps independent crops, condition stores, zmap/calibration files, physical state, metrics, and checkpoint channel state. Do not turn (modality, channel) back into the top-level expert key.
• 2026-08-05: UnityPSF cross-modality experiment parity is defined in photon space, not raw camera ADU. Keep density, background photons, and photon intensity sampling identical across modalities, but compute each modality's fd_deeploc_exact_recenter target from its own camera baseline, QE, and electrons-per-ADU; do not force Origami and Astigmatism to share an ADU baseline.
• 2026-08-05: UnityPSF Emitter2D formal optics must be bead-independent. Initialize its vector-PSF/LUT state from explicit Zernike anchor coefficients, then derive channel-specific field dependence from the real Origami sample through the same physical-state lifecycle used by the other experts. Historical bead-fitted zmap or aligned mean-PSF files may be used only for external validation, never as formal training dependencies.
• 2026-08-05: The inspected Origami 8000x1200x1200 TIFF is not a usable left/right spatial dual-channel acquisition, superseding the 2026-08-04 Origami entry above. Only x=[0,600) contains molecular fluorescence; x=[600,1200) is dark camera background. Formal Emitter2D training for this acquisition must use only left/main. Do not infer channel validity from frame dimensions or bead PSF layouts.
• 2026-08-05: A mixed-channel Expert Parallel smoke must balance attempted steps per modality rank, not assign the same budget to every channel instance. With one Emitter2D channel and two Astigmatism channels, use 16 Emitter2D steps versus 8+8 Astigmatism steps so FP16 GradScaler warm-up cannot consume the entire Emitter2D smoke.
• 2026-08-06: Single-focus Emitter2D intensity data cannot identify a free 21-gamma NAT wavefront. Do not rely on a norm cap alone: use the reduced order1 model, keep defocus fixed at zero, fit only low-order astigmatism field terms, and export coma, spherical, and trefoil channels as zero. This preserves the shared LUT/FiLM contract without creating sidelobes from phase-wrapped high-order solutions.
• 2026-08-06: Formal Emitter2D Vector-PSF support is intentionally 15x15. Across the reduced three-mode LUT at z=-100/0/+100 nm it retains at least 92.3% of the 63x63 diagnostic energy before the renderer renormalizes the cropped kernel; do not silently restore 25x25 or apply this smaller support to Astigmatism.
• 2026-08-07: Determine a submitted UnityPSF run's density from its logged formal_runtime_contracts, not from the current YAML or memory. Submission 4545 recorded emitter_density_um2: 0.5 for Emitter2D left and Astigmatism left/right even if the run is later remembered as density 1.0.
• 2026-08-21: Formal three-expert UnityPSF training keeps Emitter2D left/right, Astigmatism left/right, and DoubleHelix main. Expert Parallel assigns one rank per PSF modality, so this is three modality experts and five channel states across three ranks. Do not narrow the Emitter2D formal audit to left-only based on an acquisition-specific interpretation.