Repository navigation
fix(oura): score one R-R beat channel per window, not 0x80 on top of 0x60 - #2423
Conversation
ryanbr
left a comment
There was a problem hiding this comment.
Thanks @pipiche38. The report behind this is the best-evidenced one on the tracker: eleven nights of per-channel coverage, cross-checked against the app's own hrv diag lines so the numbers are the same beats the gate classified, and the observation that only 305 of channel 1's 1,373 occupied seconds are also channel 3 seconds. That last one is what makes the diagnosis land, because it rules out same-second duplicates and leaves two channels tiling the same wall time, which is a different bug needing a different remedy.
It is also the root cause of the refusals behind #2425, which merged earlier today. That one stopped a refused night being replaced by naps. This stops the refusal happening for a reason that has nothing to do with the night.
I checked the two ways the predicate could be wrong and it is right on both:
0x6Ecannot return through the new clause.NOT IN (1, 3, 4)is true for channel 2, but the outer read still carriessrcChannel <> :rrxindependently, so the known duplicate stays excluded.- The subquery cannot evaluate to NULL for a row that reaches it. Any Oura row passing the outer time and suspect predicates is itself counted by the subquery, so the SUMs are never over an empty set.
Ties going to the amplitude family matches the doc, the Kotlin SQL matches the Swift byte for byte, and grouping 0x60 with 0x44 as one decoder rather than letting them compete is right.
The witness is the part I want to single out. rrFingerprint deliberately counts every beat BOTH channels offer rather than the selected one, because a new row on the losing channel can flip the selection. Witnessing only the winner would have cached a score the next drain silently invalidates. You wrote that down at the site. I raised the same class of problem on #2414 today where it had been missed, so this one standing correct is worth naming.
One thing to change: the evidence goes one-sided
RAW_RR_INTERVALS_SQL keeps both channels for the Android export. Swift has no equivalent, and Strand/System/DebugDataDiagnostics.swift:284 reads store.rrIntervals, which now applies the selection. Two things follow on iOS:
- The
rr=count in the strap log'sNight …: grav=N hr=N rr=N resp=N skin=Nline quietly changes meaning, from beats banked to beats scored, and stops being comparable with the Android line or with every log already filed. - An iOS export can no longer reproduce the per-channel analysis that found this bug. You are on iOS. The next person to hit a channel-balance problem would be holding an export that has already hidden it.
The parity ledger reaches the same place from the other direction:
add-unpaired-function: WhoopDao.kt::rawRrIntervals/4#1
parity-governance does not run here, since nothing under Tools/ is touched, so that is a local run rather than something CI would have told you.
A Swift rawRrIntervals twin with DebugDataDiagnostics pointed at it settles both the divergence and the ledger finding at once. Everything else is ready.
|
Taking the Swift twin myself rather than sending this back, since it is a mechanical mirror of SQL you have already written and you have turned enough around today. Do not start on it. I will add Everything else here stands as reviewed. |
|
Thanks @ryanbr, and thanks for taking the Swift twin. One thing I found while checking the shape, before you push, because it changes where the fix belongs: Android's So the fix is three small changes, not two:
One side effect: on a WHOOP 5, the raw read also skips the strict single-transport selection, so On the export: an iOS I have this as a patch (the three changes, plus a Swift |
…0x60 The scoring read excluded only 0x6E (ryanbr#1071) and kept both remaining Oura beat channels. On two rings, 0x60 (ibiAmplitude) is the complete beat train and 0x80 (greenQuality) a partial second copy of the same beats over the same time: on one of them 0x60 alone covers 0.91-0.99 of the wall clock and the pair reads 1.01-1.31. The ryanbr#1118 coverage gate then refused whichever nights banked more 0x80, so HRV and Charge calibration stalled for reasons unrelated to the night. The read now keeps 0x80 alone when it holds more beats in the requested window than the amplitude family (0x60 + 0x44, one decoder, split for labelling only), and the amplitude family otherwise, ties included. A ring with only one of them keeps it. NULL rows and non-Oura codes are untouched, so WHOOP and legacy rows read as before. Same shape as the WHOOP 5 transport selection: an uncorrelated subquery over the same time/suspect predicates, evaluated before LIMIT. Android: RR_INTERVALS_SQL carries the same predicate through a shared SCORABLE_OURA_CHANNELS constant; the diagnostic export moves to its own RAW_RR_INTERVALS_SQL so it still carries both beat channels. Fingerprint comments on both platforms now say they count the superset the selection chooses from. Verified with the exact SQL against two real backups: 5 of 6 refused nights pass on the reporting ring and 4 on the second, no night that passed regresses. The nights still refused are 0x60 over-covering on its own, a separate problem. Tests: WhoopStore 616/0 (5 new in RrSourceChannelTests); Kotlin OuraOneChannelReadSqliteTest 6/0 runs the production SQL on real SQLite with the Swift fixtures and expected values verbatim; RrSourceChannelTest 14/0, Whoop5RRSqliteTest 19/0. Full Android suite: the same 8 failures as untouched upstream/main (RecoveryDrivers, StandardHrSensorFormat, StepsDetailIntegration, TodayExplainability), none in touched code. Refs ryanbr#1071, ryanbr#1118 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AvMtCweTWqqM9iyDBz88XX
bde7b94 to
fe20957
Compare
…latforms (#2435) The one-Oura-channel selection (#2423) changed what `rr=` counts in the strap log's `Night …: grav=N hr=N rr=N resp=N skin=N` line, from beats BANKED to beats scored, on both platforms: Swift `DebugDataDiagnostics` read `WhoopStore.rrIntervals`, and Kotlin `AndroidDiagnostics` read `rrIntervalsForDevice`, which routes to the same selecting SQL. The two platforms still agreed with each other and both differed from every log filed before. `WhoopStore.rawRrIntervals` is added as the twin of Kotlin `WhoopDao.rawRrIntervals` (`RAW_RR_INTERVALS_SQL`): the 0x6E duplicate and quarantined beats excluded, no scoring selection, the same ORDER BY. Both Night lines point at it. Scoring is untouched, every scoring reader still goes through `rrIntervals`, and the Kotlin function stops being one-sided in the parity ledger. Two consequences are documented where the number is produced rather than left to be discovered. A WHOOP 5's `rr=` does not go back to what it was; it steps UP. The strict single-transport selection predates #2423, so a 5/MG's `rr=` was already the scored transport in every older log, and the raw read drops that too. That is the right answer for a line whose job is to say what the strap banked, since a legacy row that cannot be spliced into a scored beat train was still banked, and one number meaning "banked on a ring, scored on a strap" would be worse. It is still a change, on a device this work otherwise leaves alone. The raw read also does not collapse a record the drain served twice, where `rrIntervals` now does (#2456). On a redrained night `rr=` counts the duplicates and the `hrv diag` line does not, so the two disagreeing is the clearest signal that a redrain happened at all, rather than a fault. Tested on the same fixture and expected values as the Kotlin `OuraOneChannelReadSqliteTest.theRawExportReadKeepsBothBeatChannels`, so the two reads cannot drift into agreeing in prose while differing in fact. Authored by @pipiche38. Refs #2424, #2423.
… WHOOP 4 rebase The branch was written against a base that predates #2423 and #2457, and both touch the ONE read it changes. A literal conflict resolution would have dropped one or both silently, so this records what the merged read has to keep. `rrIntervals` is now a three-way choice rather than two: WHOOP 5 picks the lowest scorable transport for the interval; a WHOOP 4 picks its labelled type-47 history, or its unlabelled standard-BLE rows when the window has no history; everything else takes the Oura branch from #2423, one beat channel per window, the fuller of green and the amplitude family. `OuraRedrainCollapse.withoutRedrainedRuns` still wraps the result on BOTH platforms, which the Kotlin conflict would also have removed from `WhoopRepository`. Parity, which CI would not have caught here: `parity-governance.yml` filters on `Tools/parity_*`, and this branch touches `Packages/**` and `android/**`, so that job never runs on it. Run locally instead, and the ratchet named five findings. `extractHistoricalStreams` gained a declared twin reference rather than a disposition: the Kotlin twin exists, and adding `family` only moved the Swift identity, which the inventory keys on arity. Calling that platform-specific would have written a false rationale into the registry permanently. The three Room DAO methods and their two `@Query` constants took `platform_specific` dispositions. Room needs a compile-time constant string and a generated method; the Swift store issues the same SQL inline in `Reads.swift` and `StreamStore.swift`, which is why the WHOOP 5 siblings are one-sided for the same reason. Hashes came from `parity_ledger._canonical_sha256`, not by hand. `--migrate-authority` was deliberately not used. It waives whether the BASE's stored manifest reproduces, and reaching for it to absorb debt a branch introduces is laundering; the ratchet asks for an issue-bound authority change instead. Two smaller things: `isWhoop4RRSource` gained the doc its siblings have, including that `confirmedRegistryFamily` returns nil for an unknown registry and that nil must not be read as "a WHOOP 4"; and the promote path names `RRSourceChannel.whoop4Historical` instead of force-unwrapping the optional its own pattern match already proved non-nil. The derived authority is refreshed here too, which the first push missed. The ratchet went green on the dispositions and I took that as the whole verdict, but the ledger had already said `twin-map-authority-drift|functions`: pairing `extractHistoricalStreams` and adding five functions moved the derived inventory, so the stored snapshot no longer reproduced. CI caught it as `test_checked_metadata_is_compact_v3_and_expands_losslessly`, stored 4486 against a derived 4491. Refreshed with `--refresh-derived` and NOT `--migrate-authority`: the base authority reproduces fine, only this branch's derived counts moved, and migration would have waived a check that was not the thing failing. Findings held at 292.
… WHOOP 4 rebase The branch was written against a base that predates #2423 and #2457, and both touch the ONE read it changes. A literal conflict resolution would have dropped one or both silently, so this records what the merged read has to keep. `rrIntervals` is now a three-way choice rather than two: WHOOP 5 picks the lowest scorable transport for the interval; a WHOOP 4 picks its labelled type-47 history, or its unlabelled standard-BLE rows when the window has no history; everything else takes the Oura branch from #2423, one beat channel per window, the fuller of green and the amplitude family. `OuraRedrainCollapse.withoutRedrainedRuns` still wraps the result on BOTH platforms, which the Kotlin conflict would also have removed from `WhoopRepository`. Parity, which CI would not have caught here: `parity-governance.yml` filters on `Tools/parity_*`, and this branch touches `Packages/**` and `android/**`, so that job never runs on it. Run locally instead, and the ratchet named five findings. `extractHistoricalStreams` gained a declared twin reference rather than a disposition: the Kotlin twin exists, and adding `family` only moved the Swift identity, which the inventory keys on arity. Calling that platform-specific would have written a false rationale into the registry permanently. The three Room DAO methods and their two `@Query` constants took `platform_specific` dispositions. Room needs a compile-time constant string and a generated method; the Swift store issues the same SQL inline in `Reads.swift` and `StreamStore.swift`, which is why the WHOOP 5 siblings are one-sided for the same reason. Hashes came from `parity_ledger._canonical_sha256`, not by hand. `--migrate-authority` was deliberately not used. It waives whether the BASE's stored manifest reproduces, and reaching for it to absorb debt a branch introduces is laundering; the ratchet asks for an issue-bound authority change instead. Two smaller things: `isWhoop4RRSource` gained the doc its siblings have, including that `confirmedRegistryFamily` returns nil for an unknown registry and that nil must not be read as "a WHOOP 4"; and the promote path names `RRSourceChannel.whoop4Historical` instead of force-unwrapping the optional its own pattern match already proved non-nil. The derived authority is refreshed here too, which the first push missed. The ratchet went green on the dispositions and I took that as the whole verdict, but the ledger had already said `twin-map-authority-drift|functions`: pairing `extractHistoricalStreams` and adding five functions moved the derived inventory, so the stored snapshot no longer reproduced. CI caught it as `test_checked_metadata_is_compact_v3_and_expands_losslessly`, stored 4486 against a derived 4491. Refreshed with `--refresh-derived` and NOT `--migrate-authority`: the base authority reproduces fine, only this branch's derived counts moved, and migration would have waived a check that was not the thing failing. Findings held at 292.
…0x60 (ryanbr#2423) An Oura ring measures the same heartbeats on more than one tag and banks every one of them. The scoring read excluded only 0x6E (ryanbr#1071) and kept both remaining beat channels, so a night could be read as roughly two overlapping copies of itself: on one ring 0x60 alone covers 0.91-0.99 of the wall clock over 10 of 11 nights, while 0x60 plus 0x80 reads 1.01-1.31. The ryanbr#1118 coverage gate then refused whichever nights happened to bank more 0x80, HRV went blank, and Charge calibration stalled at "3 of 4 nights" for days, for reasons that had nothing to do with those nights. Neither channel is a duplicate to exclude by name. Which one is complete is a property of the capture, not of the tag, so the read keeps green alone when it holds MORE beats in the requested window than the amplitude family, and the amplitude family alone otherwise, ties included. 0x60 and 0x44 are counted and kept together: one decoder, split for labelling only, so they are never scored against each other. A ring with only one of them keeps it, which is the guarantee the old exclusion was protecting. NULL rows and every non-Oura code pass untouched, so WHOOP 4, WHOOP 5 and legacy rows read exactly as before. An uncorrelated subquery over the same time and suspect predicates as the outer read, evaluated before LIMIT, the same shape as the WHOOP 5 transport selection beside it. Rows are filtered, never deleted, and no migration is involved: the other channel stays on disk as its cross-check. Verified against two real backups over the window the app actually reads, 30 h before to 6 h after each main session. Six refused nights on the reporting ring become five passing and one still refused; four refused nights on a second ring now pass; no night that passed before regresses on either. The nights still refused are 0x60 over-covering on its own, which is record overlap and a separate problem this does not claim to answer. One interaction worth recording, because the two changes were built in parallel and meet here for the first time. ryanbr#2457 collapses a ring record the drain served twice, and it does so AFTER this read's SQL. So the "which channel is fuller" decision is made on beats that still include re-served duplicates, and a redrain inflates channels unequally: the ryanbr#2443 capture shows one channel gaining 22% and another 7.6%. In these captures the margin is far too large to flip, 36,317 beats against 2,006, but the count the choice rests on is not exactly the count that ends up scored, and that is worth knowing before someone reads it as a contradiction. Five new Swift tests: the fuller channel wins, green wins when green is the fuller one, the choice is per requested window, a tie goes to the amplitude family, and 0x60 plus 0x44 count and survive as one. Kotlin runs the production SQL on real SQLite using the Swift fixtures and expected values verbatim, so the two cannot drift into agreeing in prose while differing in fact. Authored by @pipiche38. Refs ryanbr#2424, ryanbr#1071, ryanbr#1118. (cherry picked from commit 401b457)
…latforms (ryanbr#2435) The one-Oura-channel selection (ryanbr#2423) changed what `rr=` counts in the strap log's `Night …: grav=N hr=N rr=N resp=N skin=N` line, from beats BANKED to beats scored, on both platforms: Swift `DebugDataDiagnostics` read `WhoopStore.rrIntervals`, and Kotlin `AndroidDiagnostics` read `rrIntervalsForDevice`, which routes to the same selecting SQL. The two platforms still agreed with each other and both differed from every log filed before. `WhoopStore.rawRrIntervals` is added as the twin of Kotlin `WhoopDao.rawRrIntervals` (`RAW_RR_INTERVALS_SQL`): the 0x6E duplicate and quarantined beats excluded, no scoring selection, the same ORDER BY. Both Night lines point at it. Scoring is untouched, every scoring reader still goes through `rrIntervals`, and the Kotlin function stops being one-sided in the parity ledger. Two consequences are documented where the number is produced rather than left to be discovered. A WHOOP 5's `rr=` does not go back to what it was; it steps UP. The strict single-transport selection predates ryanbr#2423, so a 5/MG's `rr=` was already the scored transport in every older log, and the raw read drops that too. That is the right answer for a line whose job is to say what the strap banked, since a legacy row that cannot be spliced into a scored beat train was still banked, and one number meaning "banked on a ring, scored on a strap" would be worse. It is still a change, on a device this work otherwise leaves alone. The raw read also does not collapse a record the drain served twice, where `rrIntervals` now does (ryanbr#2456). On a redrained night `rr=` counts the duplicates and the `hrv diag` line does not, so the two disagreeing is the clearest signal that a redrain happened at all, rather than a fault. Tested on the same fixture and expected values as the Kotlin `OuraOneChannelReadSqliteTest.theRawExportReadKeepsBothBeatChannels`, so the two reads cannot drift into agreeing in prose while differing in fact. Authored by @pipiche38. Refs ryanbr#2424, ryanbr#2423.
…eeps the daytime beats #2423 made the scoring read keep ONE Oura beat channel for the whole requested window: green (0x80) when it holds more beats than the amplitude family (0x60 + 0x44), the amplitude family otherwise. That is right over a night, where the two channels overlap. It is wrong over a window that crosses wake, where they are disjoint in time: 0x60 is banked only overnight and 0x80 carries every daytime beat. On a real ring, 00:00-17:29 held 38,667 amplitude beats (night only) and 11,377 green (all of the day), so the whole window kept amplitude and dropped every daytime beat. The Deep Timeline's Windowed rMSSD for "Today" then drew the night and nothing after ~07:00. The choice is now made per UTC hour, counted inside the requested window with the same time/suspect predicates as before, the shape the WHOOP 4 per-hour source choice already uses. An hour where both channels fire still keeps the fuller one; an hour where only one fires keeps it. A window inside one hour reads exactly as the whole-window rule did, so every #2423 test is unchanged. Swift WhoopStore.rrIntervals and Kotlin RR_INTERVALS_SQL change together; a new twin test with one shared fixture (night hour, green-only hour, mixed hour) fails on the old read and passes on the new one. Refs #2423 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zPBbdmGTu19hgrMbnwSG1
Taken from upstream: illness alerts tied to the latest evaluated night (ryanbr#2533, ryanbr#2586), skin temperature attributed by source (ryanbr#2603), archived devices keep their day ownership at the lowest priority, Oura R-R channel chosen per UTC hour (ryanbr#2423 follow-up), backgrounded re-scores spaced 30 minutes apart, the breathing check-in using the resonance pace, segmented control accessibility (ryanbr#2376), weekdays beside vital dates (ryanbr#2383), the light-sky stillness, the Today-only Key Metrics option, the effort calibration diagnostics and the unreferenced declaration cleanup. Resolved on the fork side, deliberately: - android/ and the parity-governance workflow, ledgers and acceptance test stay deleted (parity retired 2026-07-23, Android tree removed 2026-08-14). - R-R reading keeps the fork's per-segment transport reconciler. Upstream's per-hour Oura channel choice is adopted inside it as a materialised per-hour lookup, so upstream's new RrSourceChannelTests case passes. - The legacy WHOOP 5 R-R gap note (ryanbr#2117) is not taken: it explains nights upstream's strict read withholds, and the fork's reconciler does not withhold them. Its auto-merged helpers are removed again. - Re-score admission (ryanbr#2607) stays on the fork's merged analysis queue, which already serialises passes. - Day ownership uses all devices, so upstream's archived-device priority applies. - project.yml keeps the fork's build number; tools-python.yml keeps the fork's jobs; the Key Metrics editor keeps the fork's "Detailed tiles" toggle. Analysis migration required: no. Upstream marks none of these as a recipe change. The Oura per-hour choice equals the whole-window choice over a night, where the channels overlap; the archived-device ownership affects only installs with an archived device that holds a night's heart rate. Verified: StrandTests 3811 green, StrandAnalytics 2862, WhoopStore 810, StrandImport 363, NOOPiOS and Strand builds, Tools Python tests, doc comment lint, i18n audit.
Problem
The scoring R-R read excludes only 0x6E (#1071) and keeps both remaining Oura beat channels. On two rings, 0x60 (
ibiAmplitude) is the complete beat train and 0x80 (greenQuality) a partial second copy of the same beats over the same time. On one of them, 0x60 alone covers 0.91–0.99 of the wall clock on 10 of 11 nights, and 0x60 + 0x80 reads 1.01–1.31. The #1118 coverage gate then refuses whichever nights banked more 0x80. HRV goes blank and Charge calibration stalls ("3 of 4 nights" for days) for reasons unrelated to the night.Change
WhoopStore.rrIntervals: among Oura beat channels, keep 0x80 alone when it holds more beats in the requested window than the amplitude family (0x60 + 0x44: one decoder, split for labelling only, so never scored against each other), otherwise the amplitude family alone, ties included. It's an uncorrelated subquery over the same time/suspect predicates, evaluated beforeLIMIT, the same shape as the WHOOP 5 transport selection.rrIntervalmixes two optical channels, so every beat is stored twice #1071's guarantee). NULL rows and non-Oura codes are untouched, so WHOOP 4, WHOOP 5 and legacy rows read exactly as before.RR_INTERVALS_SQL: same predicate through a sharedSCORABLE_OURA_CHANNELStwin constant. The diagnostic export (rawRrIntervalsForDevice) moves to its ownRAW_RR_INTERVALS_SQL, so an export still carries both beat channels as each other's cross-check.Verification
swift testWhoopStore: 616/0. Five new tests inRrSourceChannelTests: fuller channel wins, green wins when it is the fuller one, the choice is per requested window, a tie goes to the amplitude family, 0x60 + 0x44 are counted and kept as one.OuraOneChannelReadSqliteTest(new, 6/0) runs the production SQL on real SQLite using the Swift fixtures and expected values verbatim, plus a raw-export case.RrSourceChannelTest14/0,Whoop5RRSqliteTest19/0, and Room's compile-time query check passes.upstream/main(RecoveryDrivers, StandardHrSensorFormat, StepsDetailIntegration, TodayExplainability), none in touched code.doc_comment_lint: OK. No UI strings.Refs #2424, #1071, #1118