Skip to content

fix(pipeline): flag dark-correction intervals that span a missed dark - #304

Open
boringethan wants to merge 1 commit into
nextfrom
feature/175-missed-dark-simple
Open

boringethan wants to merge 1 commit into
nextfrom
feature/175-missed-dark-simple

Conversation

@boringethan

@boringethan boringethan commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When a scheduled dark frame never arrives (dropped frame, camera dropout, stale rejection), DarkCorrectionStage keeps the interval open until the next dark that does arrive. It then interpolates the dark baseline across twice the intended span, about 30 s of frames per missed dark, with no log, event or flag (#175).

This is the smaller replacement for draft #179, which came out more complicated than it needed to be.

Detection, one site. DarkCorrectionStage takes dark_interval (forwarded by default_pipeline, the same value the classifier schedules with). In _emit_interval, which both the mid-scan and terminal-flush paths go through, it counts the scheduled dark positions strictly inside [left_abs, right_abs] (scheduled_darks_between). An interval's left edge is always a dark, so (abs_id − 1) % dark_interval == 0 covers everything inside it and discard_count isn't needed. This catches a missed dark from any cause, not just the gap-fill path.

On a missed dark: the interval is kept, not discarded, and:

  • its light frames get quality = "wide_interval". nan_filled placeholders keep their own flag, because CalibrationWorkflow and side_avg use it to recognise synthetic rows;
  • the stencilled dark row (DarkFrameHoldStage) inherits the flag when any neighbour has it. It used to hardcode "ok";
  • a WARNING is logged, and a new MissedDarkWarning(side, cam_id, left_abs, right_abs, missed_abs_ids) diagnostics event is routed like DarkIntegrityWarning, with no sink changes;
  • "wide_interval" ranks worst in side_avg._QUALITY_RANK. A nan_filled camera is NaN and drops out of the average by itself, but a wide-interval camera contributes finite values built on a stretched baseline.

Deliberately unchanged: timestamp_repair (fill rows stay "light", so a NaN dark can never reach DarkHistory or become an interval boundary, which is the hazard that deferred this out of #114) and classify.py.

Where the flag lands: quality reaches scans.db (session_data.quality); the corrected CSV no longer has a quality column, so the warning event and log are the second surface. The batch quality arrays are <U14, so the 13-character value fits.

Tests

  • scheduled_darks_between: ordinary, first-interval, double-miss and terminal-boundary cases.
  • A dropped dark at 1201 gives one warning naming it; the interval is still emitted, real frames are flagged wide_interval and stay finite, the nan_filled placeholder keeps its flag.
  • A full interval isn't flagged. A dark missed inside the terminal interval is caught at on_scan_stop.
  • The dark-frame-hold row inherits the flag; wide_interval outranks nan_filled in the side average.
  • The factory forwards a non-default dark_interval. I checked this test fails when the forwarding is removed, which was a review finding on feat: flag corrected intervals that span a missed dark #179.

pytest tests/test_pipeline/ tests/test_contact_quality_monitor.py tests/test_calibration_workflow_compute.py tests/test_scan_database.py tests/test_console_telemetry_unit.py tests/test_scan_db_sink_empty.py tests/test_calibration_outcome.py: 427 passed, 30 skipped (pure software; no hardware modules run).

Refs #175

🤖 Generated with Claude Code

A scheduled dark that never arrives (dropped frame, camera dropout,
stale rejection) leaves no "dark" row, so DarkCorrectionStage kept the
interval open to the next real dark and interpolated the baseline across
twice the span with no log, event or flag.

The stage now takes dark_interval (forwarded by default_pipeline) and,
before correcting each closed interval, mid-scan or at the terminal
flush, counts the scheduled dark positions strictly inside it. If any
are found, the interval is still corrected, but:
- its light frames get quality "wide_interval" (nan_filled placeholders
  keep their own flag, which consumers key on);
- the stencilled dark row inherits the flag from its neighbours;
- a WARNING is logged and a MissedDarkWarning diagnostics event names
  the missed abs_ids.
"wide_interval" ranks worst in the side-average quality: unlike
nan_filled, it contributes finite but biased values.

No timestamp_repair or classifier change: gap-fill rows stay "light",
so a NaN dark can never reach DarkHistory or an interval boundary.

Refs #175

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: boringethan <ethan@openwater.cc>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant