Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 33 additions & 7 deletions Packages/WhoopStore/Sources/WhoopStore/Reads.swift
Original file line number Diff line number Diff line change
Expand Up @@ -251,8 +251,9 @@ extension WhoopStore {
/// The streams are exactly the ones the per-day loop reads and hands to `analyzeDay`:
/// - `ppgHrSample`: the day's HR read is measured ∪ PPG-derived ([hrSamples]), so a PPG row for a second
/// with no measured HR changes the scored series while `hrSample` stays put.
/// - `rrInterval`: filtered exactly as [rrIntervals] filters at read (the 0x6E SpO2-IBI duplicate and
/// future-stamped beats are excluded), so the witness counts the beats that are actually scored.
/// - `rrInterval`: filtered as [rrIntervals] filters at read (the 0x6E SpO2-IBI duplicate and
/// future-stamped beats are excluded), but WITHOUT its one-Oura-channel selection: the witness counts
/// every beat that selection chooses from, since a new row on either channel can change which is scored.
/// - `sleepStateSample`: appended from the SAME v18 record at the same `ts` as HR, so a re-offloaded
/// record whose HR row is dropped on conflict still lands a new band row that `analyzeDay` scores.
/// Its letter is `b` (band), not `s`, which is reserved for the version prefix.
Expand Down Expand Up @@ -399,6 +400,13 @@ extension WhoopStore {
}
}

/// The Oura beat channels `rrIntervals` chooses between, as a SQL list: `greenQuality` (0x80) on one
/// side, and the amplitude family on the other, `ibiAmplitude` (0x60) + `ibiBare` (0x44). Those two
/// share one decoder and were split for labelling only, so they are scored together, never against
/// each other. `spo2Ibi` (0x6E) is not listed because `rrIntervals` excludes it outright. Twin of
/// Kotlin `SCORABLE_OURA_CHANNELS`, so the channel set cannot drift between the two scoring reads.
static let scorableOuraChannels = "(1, 3, 4)"

/// R-R intervals in EMISSION order (#823). `ord` leads the sort: ordering by `rrMs` returned a
/// second's beats sorted by VALUE, which makes successive beats similar by construction and biases
/// RMSSD — all successive differences — downward. Pre-v30 rows have `ord` NULL and SQLite sorts NULL
Expand All @@ -415,14 +423,25 @@ extension WhoopStore {
/// The predicate EXCLUDES the one channel proven redundant (`spo2Ibi`, 0x6E) rather than whitelisting
/// the one preferred (`greenQuality`, 0x80), which matters for what it does NOT drop:
/// - Outside strict WHOOP 5 policy, NULL is kept for WHOOP 4 and unlabelled legacy rows.
/// - `ibiAmplitude` (0x60/0x44) is kept. It does not fire on the Gen-3 hardware this was measured
/// on, so there is no evidence it duplicates green — and dropping a ring's ONLY beat source on an
/// untested assumption is the more expensive mistake. If a capture ever shows 0x60 and 0x80 firing
/// together, that is a second exclusion here, decided on that evidence.
/// - `ibiAmplitude` (0x60/0x44) is kept, because dropping a ring's ONLY beat source on an untested
/// assumption is the more expensive mistake.
/// 0x6E is the one excluded because it is the demonstrated duplicate AND the worse measurement of the
/// two: it is quantised to an 8 ms grid, applies no quality gate, and runs only while an SpO2
/// measurement is on — so scoring off it would make HRV coverage a function of the SpO2 duty cycle.
///
/// ONE Oura channel per window, not merely one excluded. Captures since have shown 0x60 and 0x80
/// firing TOGETHER over the same nights, on two rings. On one, 0x60 alone covers 0.91-0.99 of the
/// wall clock and 0x80 adds a partial second copy of 8-33 % of it, so the pair read 1.01-1.31 and the
/// #1118 coverage gate refused whichever nights happened to bank more 0x80. Neither channel is a
/// duplicate to exclude by name. Which one is complete is a property of the capture, not of the tag,
/// so the read keeps green alone when it holds MORE beats in the requested window than the amplitude
/// family (0x60 + 0x44, see `scorableOuraChannels`), and the amplitude family alone otherwise, ties
/// included. A ring with only one of them keeps it, which is the guarantee the exclusion above was
/// protecting. NULL rows and every
/// non-Oura code are untouched, so WHOOP and pre-v32 rows read exactly as before. Same shape as the
/// WHOOP 5 transport selection below: an uncorrelated subquery over the SAME time/suspect predicates
/// as the outer read, evaluated before LIMIT.
///
/// Rows are FILTERED, never deleted: the 0x6E stream stays on disk as the cross-check on green.
/// Every R-R consumer reads through this one function, so the `hrv diag` trace moves with the scores
/// rather than reporting a coverage nobody can reproduce.
Expand All @@ -441,11 +460,18 @@ extension WhoopStore {
// One transport for the complete requested interval. Legacy WHOOP 5 rows mix units and
// origins, so they remain stored but cannot be converted or spliced into a scored beat train.
// This subquery uses the SAME time/suspect predicates as the outer read, before LIMIT.
// Every other source takes the Oura branch: one beat channel for the interval, the fuller one
// (see the doc comment on `rrIntervals`), with NULL and non-Oura codes passing untouched.
let sourcePredicate = strictWhoop5 ? """
srcChannel = (SELECT MIN(srcChannel) FROM rrInterval
WHERE deviceId = :d AND ts >= :f AND ts <= :t AND srcChannel IN \(Self.scorableWhoop5Channels)
AND (tsSuspect IS NULL OR tsSuspect <> 1))
""" : "1"
""" : """
(srcChannel IS NULL OR srcChannel NOT IN \(Self.scorableOuraChannels) OR (srcChannel = 1) = (
SELECT SUM(srcChannel = 1) > SUM(srcChannel <> 1) FROM rrInterval
WHERE deviceId = :d AND ts >= :f AND ts <= :t AND srcChannel IN \(Self.scorableOuraChannels)
AND (tsSuspect IS NULL OR tsSuspect <> 1)))
"""
let rows = try Row.fetchAll(db, sql: """
SELECT ts, rrMs, srcChannel, ord, seq FROM rrInterval
WHERE deviceId = :d AND ts >= :f AND ts <= :t
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -212,4 +212,97 @@ final class RrSourceChannelTests: XCTestCase {
XCTAssertEqual(scored.count, trueBeats)
XCTAssertEqual(Set(scored.compactMap(\.srcChannel)), [.greenQuality])
}

// MARK: - One Oura beat channel per window

/// 0x60 and 0x80 over the SAME night: 0x60 is the complete train, 0x80 a partial second copy of the
/// same beats. Read together they over-cover the wall clock and the #1118 gate refuses the night, so
/// the read keeps the fuller channel alone. NULL rows pass untouched; 0x6E stays excluded.
/// Fixture shared byte-for-byte with Kotlin `OuraOneChannelReadSqliteTest`.
func testTwoOuraBeatChannelsOverOneNightScoreOnlyTheFullerOne() async throws {
let store = try await WhoopStore.inMemory()
try await store.upsertDevice(id: "ring", mac: nil, name: nil)
_ = try await store.insert(Streams(rr: Self.oneChannelFixture(ts: ts)), deviceId: "ring")

let read = try await store.rrIntervals(deviceId: "ring", from: ts, to: ts + 200, limit: 1000)
XCTAssertEqual(read.map(\.rrMs), [1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009,
1010, 1011, 777])
XCTAssertEqual(Set(read.compactMap(\.srcChannel)), [.ibiAmplitude])
XCTAssertEqual(read.filter { $0.srcChannel == nil }.count, 1)
}

/// Which channel is complete is a property of the capture, not the tag. Where 0x80 is the fuller
/// stream it is the one scored — the #1071 guarantee that a ring's only full source is never dropped.
func testWhenGreenIsTheFullerChannelItIsTheOneScored() async throws {
let store = try await WhoopStore.inMemory()
try await store.upsertDevice(id: "ring", mac: nil, name: nil)
var rows: [RRInterval] = []
for i in 0..<10 { rows.append(RRInterval(ts: ts + i, rrMs: 950 + i, srcChannel: .greenQuality)) }
for i in 0..<3 { rows.append(RRInterval(ts: ts + 3 * i, rrMs: 1100 + i, srcChannel: .ibiAmplitude)) }
_ = try await store.insert(Streams(rr: rows), deviceId: "ring")

let read = try await store.rrIntervals(deviceId: "ring", from: ts, to: ts + 100, limit: 1000)
XCTAssertEqual(read.map(\.rrMs), Array(950...959))
XCTAssertEqual(Set(read.compactMap(\.srcChannel)), [.greenQuality])
}

/// The choice is made over the REQUESTED window, not the device's whole history: a window where 0x80
/// is the only channel present still reads it, even though 0x60 dominates the night around it.
func testTheChannelIsChosenWithinTheRequestedWindow() async throws {
let store = try await WhoopStore.inMemory()
try await store.upsertDevice(id: "ring", mac: nil, name: nil)
_ = try await store.insert(Streams(rr: Self.oneChannelFixture(ts: ts)), deviceId: "ring")
// A window after the 0x60 train ends, holding only the late 0x80 tail.
_ = try await store.insert(Streams(rr: [
RRInterval(ts: ts + 300, rrMs: 880, srcChannel: .greenQuality),
RRInterval(ts: ts + 301, rrMs: 881, srcChannel: .greenQuality),
]), deviceId: "ring")

let read = try await store.rrIntervals(deviceId: "ring", from: ts + 250, to: ts + 350, limit: 1000)
XCTAssertEqual(read.map(\.rrMs), [880, 881])
}

/// Equal counts keep the amplitude family, so a tie has one defined answer on both platforms.
func testAChannelTieResolvesToTheAmplitudeFamily() async throws {
let store = try await WhoopStore.inMemory()
try await store.upsertDevice(id: "ring", mac: nil, name: nil)
var rows: [RRInterval] = []
for i in 0..<3 {
rows.append(RRInterval(ts: ts + i, rrMs: 900 + i, srcChannel: .greenQuality))
rows.append(RRInterval(ts: ts + i, rrMs: 1200 + i, srcChannel: .ibiAmplitude))
}
_ = try await store.insert(Streams(rr: rows), deviceId: "ring")

let read = try await store.rrIntervals(deviceId: "ring", from: ts, to: ts + 100, limit: 1000)
XCTAssertEqual(read.map(\.rrMs), [1200, 1201, 1202])
}

/// 0x60 and 0x44 are ONE family (one decoder, split for labelling only), so they are counted together
/// against green and kept together: 4 + 4 amplitude beats outweigh 6 green, though neither tag would
/// alone.
func testTheAmplitudeFamilyIsCountedAndKeptAsOne() async throws {
let store = try await WhoopStore.inMemory()
try await store.upsertDevice(id: "ring", mac: nil, name: nil)
var rows: [RRInterval] = []
for i in 0..<6 { rows.append(RRInterval(ts: ts + i, rrMs: 900 + i, srcChannel: .greenQuality)) }
for i in 0..<8 {
rows.append(RRInterval(ts: ts + i, rrMs: 1000 + i, srcChannel: i % 2 == 0 ? .ibiAmplitude : .ibiBare))
}
_ = try await store.insert(Streams(rr: rows), deviceId: "ring")

let read = try await store.rrIntervals(deviceId: "ring", from: ts, to: ts + 100, limit: 1000)
XCTAssertEqual(read.map(\.rrMs), Array(1000...1007))
XCTAssertEqual(Set(read.compactMap(\.srcChannel)), [.ibiAmplitude, .ibiBare])
}

/// 12 s of 0x60 (the full train), 4 overlapping 0x80 beats (the partial copy), 0x6E over the same
/// seconds (excluded by name), and one unlabelled row. Mirrored exactly in the Kotlin test.
static func oneChannelFixture(ts: Int) -> [RRInterval] {
var rows: [RRInterval] = []
for i in 0..<12 { rows.append(RRInterval(ts: ts + i, rrMs: 1000 + i, srcChannel: .ibiAmplitude)) }
for i in 0..<4 { rows.append(RRInterval(ts: ts + 2 * i, rrMs: 900 + i, srcChannel: .greenQuality)) }
for i in 0..<12 { rows.append(RRInterval(ts: ts + i, rrMs: 1000 + 8 * i, srcChannel: .spo2Ibi)) }
rows.append(RRInterval(ts: ts + 100, rrMs: 777))
return rows
}
}
50 changes: 43 additions & 7 deletions android/app/src/main/java/com/noop/data/WhoopDao.kt
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,9 @@ internal const val ANALYSIS_FINGERPRINT_SQL =
* which MAX(ts) alone would miss. Every arm is an index range walk over the same (deviceId, ts) key the HR
* fingerprint uses, materializing no rows.
*
* rrInterval is filtered exactly as [WhoopDao.rrIntervals] filters at read (the 0x6E SpO2-IBI duplicate and
* future-stamped beats), so the witness counts the beats that are actually scored. The literal 2 is
* rrInterval is filtered as [WhoopDao.rrIntervals] filters at read (the 0x6E SpO2-IBI duplicate and
* future-stamped beats), but WITHOUT its one-Oura-channel selection: the witness counts every beat that
* selection chooses from, since a new row on either channel can change which one is scored. The literal 2 is
* RrSourceChannel.SPO2_IBI.code, pinned to the enum by RrChannelTest, for the same reason it is a literal
* there: a Room @Query is a compile-time constant string.
*
Expand Down Expand Up @@ -84,8 +85,29 @@ internal const val DAY_STREAM_FINGERPRINT_SQL =
"'|ownerTagged' || EXISTS(SELECT 1 FROM rrInterval WHERE deviceId = :deviceId AND srcChannel IN (5, 6, 7)) || " +
"'|registry' || COALESCE((SELECT QUOTE(brand) || ':' || QUOTE(model) FROM pairedDevice WHERE id = :deviceId), 'absent')"

/** Shared production SQL used by Room and the SQLite repository contract tests. */
/** The Oura beat channels [RR_INTERVALS_SQL] chooses between, as a SQL list: GREEN_QUALITY (0x80) on
* one side, and the amplitude family on the other, IBI_AMPLITUDE (0x60) + IBI_BARE (0x44). Those two
* share one decoder and were split for labelling only, so they are scored together, never against each
* other. SPO2_IBI (0x6E) is not listed because the read excludes it outright. Twin of Swift
* `WhoopStore.scorableOuraChannels`, so the channel set cannot drift between the two scoring reads. */
internal const val SCORABLE_OURA_CHANNELS = "(1, 3, 4)"

/** Shared production SQL used by Room and the SQLite repository contract tests. The SCORING read: see
* [WhoopDao.rrIntervals] for the one-Oura-channel selection. Twin of Swift `WhoopStore.rrIntervals`. */
internal const val RR_INTERVALS_SQL =
"SELECT * FROM rrInterval WHERE deviceId = :deviceId AND ts >= :from AND ts <= :to " +
"AND (srcChannel IS NULL OR srcChannel <> 2) " +
"AND (srcChannel IS NULL OR srcChannel NOT IN " + SCORABLE_OURA_CHANNELS + " OR (srcChannel = 1) = (" +
"SELECT SUM(srcChannel = 1) > SUM(srcChannel <> 1) FROM rrInterval " +
"WHERE deviceId = :deviceId AND ts >= :from AND ts <= :to AND srcChannel IN " + SCORABLE_OURA_CHANNELS + " " +
"AND (tsSuspect IS NULL OR tsSuspect <> 1))) " +
"AND (tsSuspect IS NULL OR tsSuspect <> 1) " +
"ORDER BY ts ASC, ord ASC, rrMs ASC, seq ASC LIMIT :limit"

/** The diagnostic export's read: every beat channel except the 0x6E duplicate, with NO scoring
* selection, so an exported night still carries both Oura beat channels as each other's cross-check.
* This is what [RR_INTERVALS_SQL] was before the one-channel selection. */
internal const val RAW_RR_INTERVALS_SQL =
"SELECT * FROM rrInterval WHERE deviceId = :deviceId AND ts >= :from AND ts <= :to " +
"AND (srcChannel IS NULL OR srcChannel <> 2) " +
"AND (tsSuspect IS NULL OR tsSuspect <> 1) " +
Expand Down Expand Up @@ -583,21 +605,35 @@ interface WhoopDao : DeviceRegistryDao {
// whitelisting the one preferred (GREEN_QUALITY, 0x80), which matters for what it does NOT drop:
// - NULL is kept for WHOOP 4 and unlabelled legacy owners. The repository routes strict
// WHOOP 5 owners to whoop5RrIntervals instead.
// - IBI_AMPLITUDE (0x60/0x44) is kept. It does not fire on the Gen-3 hardware this was measured
// on, so there is no evidence it duplicates green — and dropping a ring's ONLY beat source on
// an untested assumption is the more expensive mistake. If a capture ever shows 0x60 and 0x80
// firing together, that is a second exclusion here, decided on that evidence.
// - IBI_AMPLITUDE (0x60/0x44) is kept, because dropping a ring's ONLY beat source on an
// untested assumption is the more expensive mistake.
// 0x6E is the one excluded because it is the demonstrated duplicate AND the worse measurement of
// the two: quantised to an 8 ms grid, no quality gate, and running only while an SpO2 measurement
// is on — so scoring off it would make HRV coverage a function of the SpO2 duty cycle.
//
// ONE Oura channel per window, not merely one excluded. Captures since have shown 0x60 and 0x80
// firing TOGETHER over the same nights, on two rings. On one, 0x60 alone covers 0.91-0.99 of the
// wall clock and 0x80 adds a partial second copy of 8-33 % of it, so the pair read 1.01-1.31 and
// the #1118 coverage gate refused whichever nights happened to bank more 0x80. Neither channel is
// a duplicate to exclude by name. Which one is complete is a property of the capture, not of the
// tag, so the read keeps green alone when it holds MORE beats in the requested window than the
// amplitude family (0x60 + 0x44, see SCORABLE_OURA_CHANNELS), and the amplitude family alone
// otherwise, ties included. A ring with only one of them keeps it, which is the guarantee the
// exclusion above was protecting. NULL rows and every non-Oura code are untouched, so WHOOP and
// pre-v26 rows read exactly as before. The diagnostic export reads [rawRrIntervals] instead, which
// makes no selection.
//
// Rows are FILTERED, never deleted: the 0x6E stream stays on disk as the cross-check on green.
// Legacy and non-WHOOP5 readers use this query. Strict WHOOP 5 uses the source-selected query below. The literal 2 is RrSourceChannel.SPO2_IBI
// .code — a Room @Query is a compile-time constant string and cannot reference it; RrChannelTest
// pins the two together.
@Query(RR_INTERVALS_SQL)
suspend fun rrIntervals(deviceId: String, from: Long, to: Long, limit: Int): List<RrInterval>

/** Diagnostic export only: [RAW_RR_INTERVALS_SQL], no one-channel selection. */
@Query(RAW_RR_INTERVALS_SQL)
suspend fun rawRrIntervals(deviceId: String, from: Long, to: Long, limit: Int): List<RrInterval>

/** WHOOP 5: one eligible transport over the entire requested interval, selected before LIMIT. */
@Query(WHOOP5_RR_INTERVALS_SQL)
suspend fun whoop5RrIntervals(deviceId: String, from: Long, to: Long, limit: Int): List<RrInterval>
Expand Down
6 changes: 3 additions & 3 deletions android/app/src/main/java/com/noop/data/WhoopRepository.kt
Original file line number Diff line number Diff line change
Expand Up @@ -1372,11 +1372,11 @@ class WhoopRepository(
dao.legacyWhoop5RrWithheld(deviceId, from, to)
}

/** Diagnostic export keeps all WHOOP transports and legacy values without scoring selection.
* Existing quarantine and Oura SpO2-IBI exclusions still apply. */
/** Diagnostic export keeps all WHOOP transports, both Oura beat channels and legacy values without
* scoring selection. Existing quarantine and Oura SpO2-IBI exclusions still apply. */
suspend fun rawRrIntervalsForDevice(deviceId: String, from: Long, to: Long,
limit: Int = DEFAULT_LIMIT): List<RrInterval> =
dao.rrIntervals(deviceId, from, to, limit)
dao.rawRrIntervals(deviceId, from, to, limit)

suspend fun rrIntervalsForDevice(deviceId: String, from: Long, to: Long,
limit: Int = DEFAULT_LIMIT,
Expand Down
Loading
Loading