Abort the scan when a mask camera stops (or never starts) streaming

[boringethan]

Refs #298

Problem

SideAverageStage emits corrected side-average frames (cam_id=-1) only below a cross-camera watermark. A mask camera that never streams keeps that watermark at None; a camera that dies mid-scan freezes it. Every later side-average frame then waits in self._frames[side] until on_scan_stop. In reduced (clinical) mode those rows are the only DB record, so:

• a crash mid-scan loses all of them;
• memory grows with scan length;
• stop does one burst write.

Approach: abort, instead of averaging without the camera

This PR takes the abort route over re-computing the watermark without stale cameras (stale_lag_frames). With the abort, a stall lasts at most camera_dropout_abort_s.

• FrameClassificationStage already emits CameraStreamGap after 9 missing packets. It now also emits one CameraDropoutTimeout per camera per scan once an alerted gap has lasted camera_dropout_abort_s.
  • The duration is packet time (timestamp_s - first_missing_timestamp_s), not a frame count, so the limit holds at any trigger rate.
  • It is only checked after the 9-packet alert, so one timestamp glitch cannot fire it on a camera that blinked.
  • never_delivered distinguishes "never started" from "stopped".
• CameraDropoutWatchdogSink (diagnostics channel) calls back on the first timeout. ScanWorkflow._on_camera_dropout then trips _stop_evt, the same path a mid-scan disconnect uses.
  • The duration guard runs the normal teardown (stop trigger, disable cameras, close source). on_scan_stop flushes the buffered side average, so data up to the abort is persisted.
  • The worker then raises CameraDropoutError(side, cam_id, never_delivered, silent_s, threshold_s). It surfaces via last_scan_error and ScanRequest.on_error. last_scan_canceled stays False: this is an error, not a user cancel.
• ScanRequest.camera_dropout_abort_s defaults to 5 s. It is passed through default_pipeline(camera_dropout_abort_s=...); None turns it off.
  • run_collection_scan (calibration and contact-quality sub-scans) passes None, because those collectors judge each camera themselves.
  • The WI-00015 safety calibration's "normal scan" (calibration/safety_hardware.py) keeps the default, so a dead camera fails that stage with the dropout error.
• Docs: docs/API.md (request field and abort behaviour) and docs/SciencePipeline.md §5.1 (event table and timing rule).

Tradeoff

The operator loses the rest of the session rather than getting a scan averaged over fewer cameras. The partial record up to the abort is kept.

Out of scope

• A whole side going silent produces no packets, so no CameraStreamGap or timeout. That is Histogram stream has no failure reporting: a side that stops delivering frames mid-scan is never detected #192 (fix PR feat: detect a histogram stream that stops delivering frames #193, not merged).
• App side: if a camera sensor never starts up, it is not recorded as having gone offline. need to add this protection to averaging mode. openmotion-bloodflow-app#585 / PR fix: a camera that never starts is marked offline in averaging mode (#585) openmotion-bloodflow-app#654 (dropout warning, averaged-plot badge). The app should show CameraDropoutError from on_error as a distinct scan-aborted reason.

Tests (no hardware)

• tests/test_pipeline/test_camera_dropout.py (new, 11 tests):
  • a camera that never delivers, and one that dies mid-scan, each time out exactly once;
  • a sub-limit gap that resumes does not time out;
  • timing is in seconds, not frames (10 Hz case);
  • other sides and unmasked cameras are ignored;
  • disabled by default at the stage; reset() re-arms;
  • watchdog sink behaviour; factory pass-through.
• tests/test_scan_workflow.py:
  • the default is 5 s;
  • a timeout aborts with CameraDropoutError through on_error / last_scan_error and is not marked canceled;
  • None disables the abort;
  • the threshold reaches the classifier.
• tests/test_run_collection_scan.py: sub-scans disable the abort.

New tests fail against next and pass with the change. Targeted run of tests/test_pipeline, test_reduced_mode.py, test_run_collection_scan.py and test_scan_workflow.py: 337 passed, 1 failed. The failure, test_start_scan_aborts_synchronously_when_scan_db_readonly, fails identically on next in this container: it runs as root, so the chmod'd read-only DB stays writable.

Not verified on hardware. Bench check: start a reduced-mode scan with one mask camera unplugged or unpowered (never delivers), and one where a camera is killed mid-scan. Confirm the scan stops about 5 s after the loss, names the right side and camera, and the DB holds side-average rows up to the abort.

🤖 Generated with Claude Code

https://claude.ai/code/session_017sfVYYF9YDDyUtNHCR5abT

---

Generated by Claude Code