Repository navigation
fix(pipeline): flag dark-correction intervals that span a missed dark - #304
Open
boringethan wants to merge 1 commit into
Open
boringethan wants to merge 1 commit into
boringethan wants to merge 1 commit into
Conversation
A scheduled dark that never arrives (dropped frame, camera dropout, stale rejection) leaves no "dark" row, so DarkCorrectionStage kept the interval open to the next real dark and interpolated the baseline across twice the span with no log, event or flag. The stage now takes dark_interval (forwarded by default_pipeline) and, before correcting each closed interval, mid-scan or at the terminal flush, counts the scheduled dark positions strictly inside it. If any are found, the interval is still corrected, but: - its light frames get quality "wide_interval" (nan_filled placeholders keep their own flag, which consumers key on); - the stencilled dark row inherits the flag from its neighbours; - a WARNING is logged and a MissedDarkWarning diagnostics event names the missed abs_ids. "wide_interval" ranks worst in the side-average quality: unlike nan_filled, it contributes finite but biased values. No timestamp_repair or classifier change: gap-fill rows stay "light", so a NaN dark can never reach DarkHistory or an interval boundary. Refs #175 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: boringethan <ethan@openwater.cc>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a scheduled dark frame never arrives (dropped frame, camera dropout, stale rejection),
DarkCorrectionStagekeeps the interval open until the next dark that does arrive. It then interpolates the dark baseline across twice the intended span, about 30 s of frames per missed dark, with no log, event or flag (#175).This is the smaller replacement for draft #179, which came out more complicated than it needed to be.
Detection, one site.
DarkCorrectionStagetakesdark_interval(forwarded bydefault_pipeline, the same value the classifier schedules with). In_emit_interval, which both the mid-scan and terminal-flush paths go through, it counts the scheduled dark positions strictly inside[left_abs, right_abs](scheduled_darks_between). An interval's left edge is always a dark, so(abs_id − 1) % dark_interval == 0covers everything inside it anddiscard_countisn't needed. This catches a missed dark from any cause, not just the gap-fill path.On a missed dark: the interval is kept, not discarded, and:
quality = "wide_interval".nan_filledplaceholders keep their own flag, becauseCalibrationWorkflowandside_avguse it to recognise synthetic rows;DarkFrameHoldStage) inherits the flag when any neighbour has it. It used to hardcode"ok";MissedDarkWarning(side, cam_id, left_abs, right_abs, missed_abs_ids)diagnostics event is routed likeDarkIntegrityWarning, with no sink changes;"wide_interval"ranks worst inside_avg._QUALITY_RANK. Anan_filledcamera is NaN and drops out of the average by itself, but a wide-interval camera contributes finite values built on a stretched baseline.Deliberately unchanged:
timestamp_repair(fill rows stay"light", so a NaN dark can never reachDarkHistoryor become an interval boundary, which is the hazard that deferred this out of #114) andclassify.py.Where the flag lands:
qualityreachesscans.db(session_data.quality); the corrected CSV no longer has a quality column, so the warning event and log are the second surface. The batchqualityarrays are<U14, so the 13-character value fits.Tests
scheduled_darks_between: ordinary, first-interval, double-miss and terminal-boundary cases.wide_intervaland stay finite, thenan_filledplaceholder keeps its flag.on_scan_stop.wide_intervaloutranksnan_filledin the side average.dark_interval. I checked this test fails when the forwarding is removed, which was a review finding on feat: flag corrected intervals that span a missed dark #179.pytest tests/test_pipeline/ tests/test_contact_quality_monitor.py tests/test_calibration_workflow_compute.py tests/test_scan_database.py tests/test_console_telemetry_unit.py tests/test_scan_db_sink_empty.py tests/test_calibration_outcome.py: 427 passed, 30 skipped (pure software; no hardware modules run).Refs #175
🤖 Generated with Claude Code