From eca744317d7a5c88780d004e67588b030ac773db Mon Sep 17 00:00:00 2001 From: Fanboynz Date: Tue, 18 Aug 2026 21:11:25 +1200 Subject: [PATCH] =?UTF-8?q?feat(apple):=20hourly=20step=20import=20?= =?UTF-8?q?=E2=80=94=20per-hour=20iPhone=20steps=20with=2090-day=20backfil?= =?UTF-8?q?l?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-cut of #369 (by @vishk23) against current main. That branch was 385 commits behind and BOTH of its migration slots had been taken since: GRDB v32-apple-step-hour collided with v32-rr-src-channel (main is at v37) and Room 26 collided with MIGRATION_25_26 (main is at 30). Renumbered to the current tail rather than mutating shipped migrations. The daily Apple Health import flattens iPhone steps to one appleDaily.steps total, so an hour the phone spent dead or on a desk is invisible - the day just reads low. This adds an hourly table beside it: - GRDB v38-apple-step-hour + Room MIGRATION_30_31 (@Database version 31, SCHEMA_VERSION 31): additive CREATE TABLE only, appleStepHour(deviceId, ts, steps), PK (deviceId, ts) like every other per-sample table. - Store API upsertAppleStepHours/appleStepHours mirroring OuraRawStore, with an idempotent ON CONFLICT upsert keyed on the natural key. - HKStatisticsCollectionQuery at hour interval over the same sync window, same anchor and error handling as the existing daily collect(), plus a one-time 90-day backfill flagged only after rows actually land (HealthKit returns EMPTY rather than an error when step access is denied, so gating on no-throw alone would burn the one-time widen on a deny-then-grant). - appleStepHour registered in deviceScopedTables so Forget-device wipes it. - Kotlin schema twin + AppleStepHourMigrationTest, and both byte-identical schema_oracle.json copies updated (roomVersion 31, v38 migration, table entry). Fixes on top of the original: - Corrected the documented claim about what a missing hour means. Only hours with a step sample produce a row, so an absent hour is "no steps recorded" - a dead phone AND a still hour both look identical. Step data carries no separate "was recording" signal, so the doc now says a gap is evidence, not proof, and a consumer must not label it "phone off". - Refreshed the stale renumbering commentary and the "v27 store" header left over from the earlier renumber. - Added a test pinning SCHEMA_VERSION to the migration target so the two cannot drift apart (the #1410 failure mode). SCOPE: plumbing only, no UI reads the table yet - same precedent as #346/#895. Android carries the schema but no importer writes it (HealthKit has no Android analogue), exactly as the older appleDaily table already does, so .noopbak stays byte-identical. Re-review also found the Android Forget-device path incomplete: appleStepHour was registered in the Swift deviceScopedTables (whose test enumerates every deviceId-keyed table automatically) but NOT in the hand-maintained Android twin in DeviceRegistryDao. Android never writes the table, but a .noopbak restored FROM iOS carries its rows, so "delete this device's data" — and in particular "Remove Apple Health imported data", which is exactly this data — left the hourly step history behind. Added the delete + re-key queries, wired both call sites, and extended the fan-out guard test to pin appleStepHour. Validation: full Android suite 4037 tests / 489 classes green (the fan-out guard caught the gap above before this landed), including SchemaOracleTest (compares Room KSP-generated schema against the oracle) and the new migration test. test (WhoopStore) green in CI — that covers the v38 migration, the store API, the GRDB-side oracle and the automatic deviceScopedTables coverage assertion. HealthKitBridge is app-target Swift, validated by app-build. Co-authored-by: vishk23 <119831996+vishk23@users.noreply.github.com> --- .../WhoopStore/AppleStepHourStore.swift | 42 +++++++++++ .../Sources/WhoopStore/Database.swift | 21 ++++++ .../WhoopStore/DeviceRegistryStore.swift | 4 + .../AppleStepHourStoreTests.swift | 56 ++++++++++++++ .../Resources/schema_oracle.json | 33 +++++++- StrandiOS/Health/HealthKitBridge.swift | 75 +++++++++++++++++++ .../main/java/com/noop/data/DeviceRegistry.kt | 2 + .../java/com/noop/data/DeviceRegistryDao.kt | 5 ++ .../src/main/java/com/noop/data/Entities.kt | 22 ++++++ .../main/java/com/noop/data/WhoopDatabase.kt | 34 ++++++++- .../analytics/RegistryDayOwnerSourceTest.kt | 2 + .../noop/ble/SourceCoordinatorAdoptionTest.kt | 2 + .../noop/data/AppleStepHourMigrationTest.kt | 72 ++++++++++++++++++ .../java/com/noop/data/DeviceRegistryTest.kt | 6 ++ .../app/src/test/resources/schema_oracle.json | 33 +++++++- 15 files changed, 403 insertions(+), 6 deletions(-) create mode 100644 Packages/WhoopStore/Sources/WhoopStore/AppleStepHourStore.swift create mode 100644 Packages/WhoopStore/Tests/WhoopStoreTests/AppleStepHourStoreTests.swift create mode 100644 android/app/src/test/java/com/noop/data/AppleStepHourMigrationTest.kt diff --git a/Packages/WhoopStore/Sources/WhoopStore/AppleStepHourStore.swift b/Packages/WhoopStore/Sources/WhoopStore/AppleStepHourStore.swift new file mode 100644 index 0000000000..42e256a649 --- /dev/null +++ b/Packages/WhoopStore/Sources/WhoopStore/AppleStepHourStore.swift @@ -0,0 +1,42 @@ +import Foundation +import GRDB + +// MARK: - v38 store: hourly Apple Health step counts +// Mirrors OuraRawStore: an idempotent ON CONFLICT upsert keyed by the natural key (deviceId, ts), and +// a range read — all GRDB work via syncWrite/syncRead. `appleDaily.steps` already answers "how many +// steps that day"; this table answers "which HOURS were recorded", so the UI can show retroactively +// when the iPhone was off/dead/left behind (e.g. mid-hike) instead of a single flattened daily total. + +extension WhoopStore { + + /// Upsert hourly Apple Health step counts. Idempotent by (deviceId, ts): re-importing the same + /// hour overwrites its count rather than duplicating. Returns rows changed. + @discardableResult + public func upsertAppleStepHours(_ rows: [(ts: Int, steps: Int)], deviceId: String) async throws -> Int { + try syncWrite { db in + var n = 0 + for r in rows { + try db.execute(sql: """ + INSERT INTO appleStepHour (deviceId, ts, steps) + VALUES (?, ?, ?) + ON CONFLICT(deviceId, ts) DO UPDATE SET + steps = excluded.steps + """, arguments: [deviceId, r.ts, r.steps]) + n += db.changesCount + } + return n + } + } + + /// Hourly step counts for a device over `[fromTs, toTs]` inclusive, oldest hour first. + public func appleStepHours(deviceId: String, fromTs: Int, toTs: Int) async throws -> [(ts: Int, steps: Int)] { + try syncRead { db in + try Row.fetchAll(db, sql: """ + SELECT ts, steps FROM appleStepHour + WHERE deviceId = ? AND ts >= ? AND ts <= ? + ORDER BY ts ASC + """, arguments: [deviceId, fromTs, toTs]) + .map { (ts: $0["ts"], steps: $0["steps"]) } + } + } +} diff --git a/Packages/WhoopStore/Sources/WhoopStore/Database.swift b/Packages/WhoopStore/Sources/WhoopStore/Database.swift index 60d2b3d036..e9bf6cef82 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/Database.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/Database.swift @@ -847,6 +847,27 @@ extension WhoopStore { t.add(column: "avgSdnn", .double) } } + // v38-apple-step-hour: hourly Apple Health step counts. The daily `collect(.stepCount, …)` path + // in HealthKitBridge flattens a whole day to one `appleDaily.steps` total, so a dead/absent phone + // for part of a day (e.g. the phone died mid-hike) is invisible — steps just read low for the + // WHOLE day instead of showing exactly which hours had no recording. HealthKit retains hourly + // statistics historically, so this table lets a one-time backfill answer PAST days + // retroactively, not just from the day this migration ships. `ts` is the hour-BUCKET START + // (unix seconds), aligned to local-clock hour boundaries by the `HKStatisticsCollectionQuery` + // anchored at local midnight (see HealthKitBridge); `steps` is the cumulative sum within that + // hour. PK (deviceId, ts) mirrors every other per-sample table (hrSample, stepSample, …) and + // makes the hourly upsert idempotent. Additive only — a NEW table, no existing row touched, old + // readers unaffected. + migrator.registerMigration("v38-apple-step-hour") { db in + // ifNotExists: forks/sideloads that already carry this table under a different migration + // identifier converge cleanly instead of failing the migrator. + try db.create(table: "appleStepHour", options: [.ifNotExists]) { t in + t.column("deviceId", .text).notNull() // "apple-health" + t.column("ts", .integer).notNull() // hour-start unix seconds (local-hour aligned by HK) + t.column("steps", .integer).notNull() + t.primaryKey(["deviceId", "ts"]) + } + } return migrator } } diff --git a/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift b/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift index 15ac038ecf..151c887318 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift @@ -124,6 +124,10 @@ public struct DeviceRegistryStore: Sendable { // v31-deep-capture-channels: the banked 5/MG v18 auxiliary fields are deviceId-keyed per-second // rows like every stream above, so a "delete all of this device's data" must clear them too. "v18AuxSample", + // v38-apple-step-hour: the hourly Apple Health step buckets are deviceId-keyed ("apple-health"), + // so forgetting that source must clear them — otherwise an imported phone's hour-by-hour step + // history survives the delete (the same privacy defect this list exists to close). + "appleStepHour", ] /// Permanently delete every recorded sample/derived row belonging to one device, across all diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/AppleStepHourStoreTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/AppleStepHourStoreTests.swift new file mode 100644 index 0000000000..a3bedb49cb --- /dev/null +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/AppleStepHourStoreTests.swift @@ -0,0 +1,56 @@ +import XCTest +import GRDB +@testable import WhoopStore + +final class AppleStepHourStoreTests: XCTestCase { + func testUpsertIsIdempotentByNaturalKey() async throws { + let store = try await WhoopStore.inMemory() + let n1 = try await store.upsertAppleStepHours([(ts: 1_000, steps: 120)], deviceId: "apple-health") + XCTAssertEqual(n1, 1) + + // Re-import the same hour with a revised count → overwrite in place, still one row. + let n2 = try await store.upsertAppleStepHours([(ts: 1_000, steps: 250)], deviceId: "apple-health") + XCTAssertEqual(n2, 1) + + let rows = try await store.appleStepHours(deviceId: "apple-health", fromTs: 0, toTs: 10_000) + XCTAssertEqual(rows.count, 1) + XCTAssertEqual(rows.first?.ts, 1_000) + XCTAssertEqual(rows.first?.steps, 250) + } + + func testRangeReadReturnsOldestFirstWithinBoundsInclusive() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertAppleStepHours([ + (ts: 3_600, steps: 100), // hour 1 + (ts: 7_200, steps: 200), // hour 2 + (ts: 10_800, steps: 300), // hour 3 + ], deviceId: "apple-health") + + // Inclusive bounds: both edges of the range are included. + let rows = try await store.appleStepHours(deviceId: "apple-health", fromTs: 3_600, toTs: 10_800) + XCTAssertEqual(rows.map(\.ts), [3_600, 7_200, 10_800]) + XCTAssertEqual(rows.map(\.steps), [100, 200, 300]) + + // A narrower window excludes rows outside it. + let narrow = try await store.appleStepHours(deviceId: "apple-health", fromTs: 3_601, toTs: 10_799) + XCTAssertEqual(narrow.map(\.ts), [7_200]) + + // A different device sees no rows. + let other = try await store.appleStepHours(deviceId: "my-whoop", fromTs: 0, toTs: 100_000) + XCTAssertTrue(other.isEmpty) + } + + func testUpsertPartitionsByDevice() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertAppleStepHours([(ts: 1_000, steps: 50)], deviceId: "apple-health") + _ = try await store.upsertAppleStepHours([(ts: 1_000, steps: 999)], deviceId: "other-device") + + let mine = try await store.appleStepHours(deviceId: "apple-health", fromTs: 0, toTs: 10_000) + XCTAssertEqual(mine.count, 1) + XCTAssertEqual(mine.first?.steps, 50) + + let others = try await store.appleStepHours(deviceId: "other-device", fromTs: 0, toTs: 10_000) + XCTAssertEqual(others.count, 1) + XCTAssertEqual(others.first?.steps, 999) + } +} diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json b/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json index 0a3b819eb2..06735f9672 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json @@ -1,6 +1,6 @@ { "_readme": "SHARED Room<->GRDB SCHEMA ORACLE (#775). Two byte-identical copies: Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json and android/app/src/test/resources/schema_oracle.json. SchemaOracleTests.swift compares GRDB's PRAGMA table_info/index_list against it; SchemaOracleTest.kt compares Room's exported schema JSON against it. `columns` is the iOS/GRDB shape in GRDB column order (macOS is the reference implementation); every way Android differs is spelled out in an `android` / `iosAbsent` / `androidColumnOrder` override naming a key in `divergenceReasons`. Adding a column, reordering one, or changing a type/nullability on one platform only fails both suites until it is either fixed or written down here.", - "roomVersion": 30, + "roomVersion": 31, "grdbMigrations": [ "v1", "v2", @@ -38,7 +38,8 @@ "v34-sleep-staging-sparse", "v35-rr-future-quarantine", "v36-whoop-caps-no-spo2", - "v37-daily-avg-sdnn" + "v37-daily-avg-sdnn", + "v38-apple-step-hour" ], "divergenceReasons": { "android-orphan-synced-column": "REAL COLUMN DRIFT: `synced` exists on Android only. GRDB's v10 note is explicit that stepSample gets no `synced` column ('unused; see StreamStore'), and v12's ppgHrSample never had one either; the Room entities copied the flag from the older per-second entities. Nothing reads it on Android \u2014 the per-row upload flag belongs to an upload path that does not exist there \u2014 so it is dead width, not divergent data. Removing it needs a Room table rebuild.", @@ -122,6 +123,34 @@ ], "indices": [] }, + "appleStepHour": { + "platform": "both", + "columns": [ + { + "name": "deviceId", + "affinity": "TEXT", + "notNull": true, + "default": null + }, + { + "name": "ts", + "affinity": "INTEGER", + "notNull": true, + "default": null + }, + { + "name": "steps", + "affinity": "INTEGER", + "notNull": true, + "default": null + } + ], + "primaryKey": [ + "deviceId", + "ts" + ], + "indices": [] + }, "battery": { "platform": "both", "columns": [ diff --git a/StrandiOS/Health/HealthKitBridge.swift b/StrandiOS/Health/HealthKitBridge.swift index 8a6dcc6f95..13beed0b93 100644 --- a/StrandiOS/Health/HealthKitBridge.swift +++ b/StrandiOS/Health/HealthKitBridge.swift @@ -140,6 +140,11 @@ final class HealthKitBridge: ObservableObject { /// UserDefaults key holding the read set the user was last asked about. private static let readTypeSignatureKey = "noop.health.readTypeSignature" + /// UserDefaults key gating the one-time 90-day hourly-step backfill (see the hourly collection + /// below). Set once the first hourly-step HealthKit query actually returns rows; every later + /// `sync()` only walks the same window the daily collectors already use. + private static let hourlyStepsBackfilledKey = "applehealth.hourlySteps.backfilled" + /// A stable fingerprint of the read types currently requested. private static var readTypeSignature: String { quantityReadIds.map(\.rawValue).sorted().joined(separator: ",") @@ -573,6 +578,31 @@ final class HealthKitBridge: ObservableObject { CaffeineLogStore.shared.replaceImported(imported) } } + // Hourly step counts (v38-apple-step-hour). `appleDaily.steps` above flattens a whole day to + // one total, so an hour the phone spent dead/on a desk is invisible — the day just reads low. + // Walk the same window at HOURLY granularity into `appleStepHour` so a UI can show the shape + // of the day. First run ever (flag unset) widens to 90 days back: HealthKit keeps hourly + // statistics historically, so this answers PAST days retroactively rather than only from the + // day it ships; later syncs re-cover the normal start...end window like the daily collectors. + // + // Collected HERE (not earlier) so a transient HealthKit error throws into the existing catch + // below and the backfill flag is never set — a failed first run retries next sync instead of + // permanently skipping the one-time 90-day widen. + let hourlyStepsBackfilled = UserDefaults.standard.bool(forKey: Self.hourlyStepsBackfilledKey) + let hourlyStepsStart: Date = hourlyStepsBackfilled ? start + : (cal.date(byAdding: .day, value: -90, to: cal.startOfDay(for: end)) ?? start) + let hourlySteps = try await collectHourlySteps(start: hourlyStepsStart, end: end) + // Only mark the backfill done once hourly data actually lands. HealthKit returns EMPTY (not + // an error) when step read-access is denied, and users grant Health scopes incrementally — so + // gating on non-empty, not merely on no-throw, stops a deny-then-grant sequence from burning + // the one-time 90-day widen before the user ever authorises steps. A genuinely step-less + // window just re-scans next sync: cheap, bounded, self-healing. + if !hourlySteps.isEmpty { + try await store.upsertAppleStepHours(hourlySteps, deviceId: appleDeviceId) + if !hourlyStepsBackfilled { + UserDefaults.standard.set(true, forKey: Self.hourlyStepsBackfilledKey) + } + } try await writeBack(whoopStore: store) lastSync = Date() lastError = nil @@ -1007,6 +1037,51 @@ final class HealthKitBridge: ObservableObject { /// the same empty result — and treating a failed query as an authoritative zero would wipe the /// stored figure for the whole window. @discardableResult + /// Hourly cumulative step counts over `[start, end)`, mirroring `collect()`'s + /// `HKStatisticsCollectionQuery` shape but bucketed by HOUR instead of day, so a backfill can show + /// the shape of a day rather than one flattened total. `anchorDate` is local midnight — the same + /// anchor `collect()` uses for its daily buckets — so hour buckets fall on local-clock hour + /// boundaries, not UTC ones. + /// + /// WHAT A MISSING HOUR MEANS. Only hours with at least one step sample produce a row (HealthKit's + /// `sumQuantity()` is nil for an empty bucket). So an absent hour means "no steps were recorded" — + /// which covers a dead/left-behind phone AND an hour the wearer simply sat still. Step data carries + /// no separate "was recording" signal, so a consumer must not label a gap as "phone off" on its own; + /// it is evidence, not proof. Rows are never fabricated for empty hours. + /// + /// Throws on a HealthKit query error (distinguished from a genuinely-empty window, which returns + /// zero rows) so a transient failure propagates to `sync()`'s existing `catch` instead of being + /// mistaken for "no data". + private func collectHourlySteps(start: Date, end: Date) async throws -> [(ts: Int, steps: Int)] { + guard let type = HKQuantityType.quantityType(forIdentifier: .stepCount) else { return [] } + let cal = Calendar.current + let anchor = cal.startOfDay(for: start) + let predicate = NSCompoundPredicate(andPredicateWithSubpredicates: [ + HKQuery.predicateForSamples(withStart: start, end: end, options: .strictStartDate), + Self.notNoopAuthored, + ]) + return try await withCheckedThrowingContinuation { (cont: CheckedContinuation<[(ts: Int, steps: Int)], Error>) in + let q = HKStatisticsCollectionQuery(quantityType: type, quantitySamplePredicate: predicate, + options: .cumulativeSum, anchorDate: anchor, + intervalComponents: DateComponents(hour: 1)) + q.initialResultsHandler = { _, results, error in + if let error { + cont.resume(throwing: error) + return + } + var rows: [(ts: Int, steps: Int)] = [] + results?.enumerateStatistics(from: start, to: end) { stats, _ in + if let sum = stats.sumQuantity() { + let steps = Int(sum.doubleValue(for: .count()).rounded()) + rows.append((ts: Int(stats.startDate.timeIntervalSince1970), steps: steps)) + } + } + cont.resume(returning: rows) + } + store.execute(q) + } + } + private func collect(_ id: HKQuantityTypeIdentifier, unit: HKUnit, start: Date, end: Date, op: HKStatisticsOptions, sink: @escaping (String, Double) -> Void) async -> Bool { guard let type = HKQuantityType.quantityType(forIdentifier: id) else { return false } diff --git a/android/app/src/main/java/com/noop/data/DeviceRegistry.kt b/android/app/src/main/java/com/noop/data/DeviceRegistry.kt index 9b3f37be8d..200499f35f 100644 --- a/android/app/src/main/java/com/noop/data/DeviceRegistry.kt +++ b/android/app/src/main/java/com/noop/data/DeviceRegistry.kt @@ -107,6 +107,7 @@ class DeviceRegistry( dao.reKeyJournal(activeId, serialId); dao.deleteJournalFor(activeId) dao.reKeyWorkouts(activeId, serialId); dao.deleteWorkoutsFor(activeId) dao.reKeyAppleDaily(activeId, serialId); dao.deleteAppleDailyFor(activeId) + dao.reKeyAppleStepHour(activeId, serialId); dao.deleteAppleStepHoursFor(activeId) dao.reKeyMetricSeries(activeId, serialId); dao.deleteMetricSeriesFor(activeId) dao.reKeyDayOwnership(activeId, serialId); dao.deleteDayOwnershipFor(activeId) dao.reKeySleepStates(activeId, serialId); dao.deleteSleepStatesFor(activeId) @@ -191,6 +192,7 @@ class DeviceRegistry( dao.deleteJournalFor(id) dao.deleteWorkoutsFor(id) dao.deleteAppleDailyFor(id) + dao.deleteAppleStepHoursFor(id) dao.deleteMetricSeriesFor(id) dao.deleteDayOwnershipFor(id) dao.deleteScoreInputProvenanceFor(id) diff --git a/android/app/src/main/java/com/noop/data/DeviceRegistryDao.kt b/android/app/src/main/java/com/noop/data/DeviceRegistryDao.kt index 4f3d86666d..02bbd6402e 100644 --- a/android/app/src/main/java/com/noop/data/DeviceRegistryDao.kt +++ b/android/app/src/main/java/com/noop/data/DeviceRegistryDao.kt @@ -85,6 +85,10 @@ interface DeviceRegistryDao { @Query("DELETE FROM journal WHERE deviceId = :deviceId") suspend fun deleteJournalFor(deviceId: String) @Query("DELETE FROM workout WHERE deviceId = :deviceId") suspend fun deleteWorkoutsFor(deviceId: String) @Query("DELETE FROM appleDaily WHERE deviceId = :deviceId") suspend fun deleteAppleDailyFor(deviceId: String) + // v38-apple-step-hour: no Android importer writes this table, but a `.noopbak` restored FROM iOS + // carries its rows, so "delete this device's data" must clear them here too — otherwise the hourly + // step history survives a delete on Android alone (the defect this set exists to close). + @Query("DELETE FROM appleStepHour WHERE deviceId = :deviceId") suspend fun deleteAppleStepHoursFor(deviceId: String) @Query("DELETE FROM metricSeries WHERE deviceId = :deviceId") suspend fun deleteMetricSeriesFor(deviceId: String) @Query("DELETE FROM dayOwnership WHERE deviceId = :deviceId") suspend fun deleteDayOwnershipFor(deviceId: String) @Query("DELETE FROM scoreInputProvenance WHERE deviceId = :deviceId OR sourceId = :deviceId") @@ -122,6 +126,7 @@ interface DeviceRegistryDao { @Query("UPDATE OR IGNORE journal SET deviceId = :to WHERE deviceId = :from") suspend fun reKeyJournal(from: String, to: String) @Query("UPDATE OR IGNORE workout SET deviceId = :to WHERE deviceId = :from") suspend fun reKeyWorkouts(from: String, to: String) @Query("UPDATE OR IGNORE appleDaily SET deviceId = :to WHERE deviceId = :from") suspend fun reKeyAppleDaily(from: String, to: String) + @Query("UPDATE OR IGNORE appleStepHour SET deviceId = :to WHERE deviceId = :from") suspend fun reKeyAppleStepHour(from: String, to: String) @Query("UPDATE OR IGNORE metricSeries SET deviceId = :to WHERE deviceId = :from") suspend fun reKeyMetricSeries(from: String, to: String) @Query("UPDATE OR IGNORE dayOwnership SET deviceId = :to WHERE deviceId = :from") suspend fun reKeyDayOwnership(from: String, to: String) @Query("UPDATE OR IGNORE sleepStateSample SET deviceId = :to WHERE deviceId = :from") suspend fun reKeySleepStates(from: String, to: String) diff --git a/android/app/src/main/java/com/noop/data/Entities.kt b/android/app/src/main/java/com/noop/data/Entities.kt index 45e059e4c2..2c5b062adf 100644 --- a/android/app/src/main/java/com/noop/data/Entities.kt +++ b/android/app/src/main/java/com/noop/data/Entities.kt @@ -575,6 +575,28 @@ data class AppleDaily( val weightKg: Double? = null, ) +/** + * Cached hourly Apple-Health step count (v38 / MIGRATION_30_31). Swift `appleStepHour` (WhoopStore + * Database.swift `v38-apple-step-hour` migration). `ts` is the hour-BUCKET START (wall-clock unix + * seconds, local-hour aligned by the HealthKit collection query); `steps` is the cumulative step sum + * within that hour. Natural key (deviceId, ts) mirrors every other per-sample table so the hourly + * upsert is idempotent. [AppleDaily.steps] answers "how many steps that day"; this table answers + * "which HOURS were recorded", so a dead/absent phone for part of a day is visible instead of a single + * flattened daily total. + * + * Fields are declared in the SAME order as the Swift GRDB schema (deviceId, ts, steps) so the + * migration's CREATE TABLE column order matches Room's generated shape. SCHEMA-ONLY twin: the + * Apple-Health IMPORT that populates it is iOS-only (HealthKit has no Android analogue), so Android + * carries the table for `.noopbak` byte-parity but no importer writes to it — exactly as it already + * does for [AppleDaily]. + */ +@Entity(tableName = "appleStepHour", primaryKeys = ["deviceId", "ts"]) +data class AppleStepHour( + val deviceId: String, + val ts: Long, + val steps: Int, +) + /** * The RAW WHOOP 5.0 v26 optical PPG waveform, one record per second (v27 / MIGRATION_18_19, issue #156 * follow-up). Swift `ppgWaveformSample` (WhoopStore Database.swift `v27-ppg-waveform` migration). The diff --git a/android/app/src/main/java/com/noop/data/WhoopDatabase.kt b/android/app/src/main/java/com/noop/data/WhoopDatabase.kt index 996a297f95..03cba2b787 100644 --- a/android/app/src/main/java/com/noop/data/WhoopDatabase.kt +++ b/android/app/src/main/java/com/noop/data/WhoopDatabase.kt @@ -51,8 +51,9 @@ import androidx.sqlite.db.SupportSQLiteDatabase PpgWaveformSampleEntity::class, RawImuSampleEntity::class, V18AuxSampleEntity::class, + AppleStepHour::class, ], - version = 30, + version = 31, // #775: ON so Room's KSP processor writes the generated schema (every table's exact `CREATE TABLE`, // columns in declaration order with affinity/NOT NULL/default, PK and indices) as JSON. That export // is what lets a plain JVM test — no device, no Robolectric — read Android's REAL schema and compare @@ -69,7 +70,7 @@ abstract class WhoopDatabase : RoomDatabase() { const val DB_NAME = "noop_whoop.db" /** Room schema version — MUST equal the `@Database(version = …)` above. Surfaced in the backup * manifest (#1410) so an export states its schema. Bump both together on a migration. */ - const val SCHEMA_VERSION = 30 + const val SCHEMA_VERSION = 31 @Volatile private var instance: WhoopDatabase? = null @@ -845,6 +846,34 @@ abstract class WhoopDatabase : RoomDatabase() { } } + /** + * v30 -> v31: ADDITIVE, adds the `appleStepHour` table, the Android twin of the Swift WhoopStore + * `v38-apple-step-hour` migration. Hourly Apple-Health step counts: `appleDaily.steps` flattens a + * whole day to one total, so a dead/absent phone for part of a day is invisible; this per-hour + * table records exactly which hours had a recording. `ts` is the hour-bucket start (unix seconds), + * `steps` the cumulative sum within that hour, composite PK (deviceId, ts) so the hourly upsert is + * idempotent. + * + * CREATE TABLE only (no existing data touched), so already-offloaded raw streams survive. The SQL + * MUST match Room's generated schema for [AppleStepHour] exactly: deviceId TEXT NOT NULL, ts + * INTEGER NOT NULL, steps INTEGER NOT NULL (all Kotlin non-null, no SQL DEFAULT), composite + * PRIMARY KEY (deviceId, ts) in declaration order. SCHEMA-ONLY twin: the Apple-Health IMPORT that + * fills it is iOS-only (HealthKit has no Android analogue), so Android carries the table for + * `.noopbak` byte-parity but no importer writes it — as it already does for [AppleDaily]. No + * destructive fallback (see the class doc). Exposed as [APPLE_STEP_HOUR_MIGRATION_SQL] so a + * plain-JVM unit test can pin the shape without Robolectric. + */ + internal val APPLE_STEP_HOUR_MIGRATION_SQL: List = listOf( + "CREATE TABLE IF NOT EXISTS `appleStepHour` (`deviceId` TEXT NOT NULL, " + + "`ts` INTEGER NOT NULL, `steps` INTEGER NOT NULL, PRIMARY KEY(`deviceId`, `ts`))", + ) + + internal val MIGRATION_30_31 = object : Migration(30, 31) { + override fun migrate(db: SupportSQLiteDatabase) { + for (stmt in APPLE_STEP_HOUR_MIGRATION_SQL) db.execSQL(stmt) + } + } + private fun build(appContext: Context): WhoopDatabase = Room.databaseBuilder(appContext, WhoopDatabase::class.java, DB_NAME) // #1014: replace ONLY the corruption handling of the default open-helper. The @@ -863,6 +892,7 @@ abstract class WhoopDatabase : RoomDatabase() { MIGRATION_18_19, MIGRATION_19_20, MIGRATION_20_21, MIGRATION_21_22, MIGRATION_22_23, MIGRATION_23_24, MIGRATION_24_25, MIGRATION_25_26, MIGRATION_26_27, MIGRATION_27_28, MIGRATION_28_29, MIGRATION_29_30, + MIGRATION_30_31, ) // #1037: a FRESH install builds the schema straight at the current version and runs NO // migrations, so the MIGRATION_7_8 "my-whoop" registry seed never fires and the WHOOP, diff --git a/android/app/src/test/java/com/noop/analytics/RegistryDayOwnerSourceTest.kt b/android/app/src/test/java/com/noop/analytics/RegistryDayOwnerSourceTest.kt index 661dff1727..3169fce5c8 100644 --- a/android/app/src/test/java/com/noop/analytics/RegistryDayOwnerSourceTest.kt +++ b/android/app/src/test/java/com/noop/analytics/RegistryDayOwnerSourceTest.kt @@ -69,6 +69,7 @@ class RegistryDayOwnerSourceTest { override suspend fun deleteJournalFor(deviceId: String) {} override suspend fun deleteWorkoutsFor(deviceId: String) {} override suspend fun deleteAppleDailyFor(deviceId: String) {} + override suspend fun deleteAppleStepHoursFor(deviceId: String) {} override suspend fun deleteMetricSeriesFor(deviceId: String) {} override suspend fun deleteDayOwnershipFor(deviceId: String) {} override suspend fun deleteScoreInputProvenanceFor(deviceId: String) {} @@ -99,6 +100,7 @@ class RegistryDayOwnerSourceTest { override suspend fun reKeyJournal(from: String, to: String) {} override suspend fun reKeyWorkouts(from: String, to: String) {} override suspend fun reKeyAppleDaily(from: String, to: String) {} + override suspend fun reKeyAppleStepHour(from: String, to: String) {} override suspend fun reKeyMetricSeries(from: String, to: String) {} override suspend fun reKeyDayOwnership(from: String, to: String) { for ((day, row) in owners) if (row.deviceId == from) owners[day] = row.copy(deviceId = to) diff --git a/android/app/src/test/java/com/noop/ble/SourceCoordinatorAdoptionTest.kt b/android/app/src/test/java/com/noop/ble/SourceCoordinatorAdoptionTest.kt index cc23195f99..f21925b5c8 100644 --- a/android/app/src/test/java/com/noop/ble/SourceCoordinatorAdoptionTest.kt +++ b/android/app/src/test/java/com/noop/ble/SourceCoordinatorAdoptionTest.kt @@ -83,6 +83,7 @@ class SourceCoordinatorAdoptionTest { override suspend fun deleteJournalFor(deviceId: String) {} override suspend fun deleteWorkoutsFor(deviceId: String) {} override suspend fun deleteAppleDailyFor(deviceId: String) {} + override suspend fun deleteAppleStepHoursFor(deviceId: String) {} override suspend fun deleteMetricSeriesFor(deviceId: String) {} override suspend fun deleteSleepStatesFor(deviceId: String) {} override suspend fun deleteLabMarkersFor(deviceId: String) {} @@ -115,6 +116,7 @@ class SourceCoordinatorAdoptionTest { override suspend fun reKeyJournal(from: String, to: String) {} override suspend fun reKeyWorkouts(from: String, to: String) {} override suspend fun reKeyAppleDaily(from: String, to: String) {} + override suspend fun reKeyAppleStepHour(from: String, to: String) {} override suspend fun reKeyMetricSeries(from: String, to: String) {} override suspend fun reKeyDayOwnership(from: String, to: String) { for ((day, row) in owners) if (row.deviceId == from) owners[day] = row.copy(deviceId = to) diff --git a/android/app/src/test/java/com/noop/data/AppleStepHourMigrationTest.kt b/android/app/src/test/java/com/noop/data/AppleStepHourMigrationTest.kt new file mode 100644 index 0000000000..1cc6283cdd --- /dev/null +++ b/android/app/src/test/java/com/noop/data/AppleStepHourMigrationTest.kt @@ -0,0 +1,72 @@ +package com.noop.data + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * Guards the additive v30 -> v31 Room migration (the `appleStepHour` table), the Android twin of the Swift + * WhoopStore `v38-apple-step-hour` migration (originally PR #369). This environment has no Robolectric / + * Room-testing, so the migration's SQL is exposed as an internal constant + * ([WhoopDatabase.APPLE_STEP_HOUR_MIGRATION_SQL]) and pinned here to Room's generated shape for + * [AppleStepHour]: + * + * - one CREATE TABLE IF NOT EXISTS statement — deviceId TEXT NOT NULL, ts INTEGER NOT NULL, steps INTEGER + * NOT NULL, composite PRIMARY KEY (deviceId, ts) in declaration order. + * - ADDITIVE: CREATE TABLE only; no DROP/DELETE/UPDATE/INSERT/ALTER on existing data. + * + * SCHEMA-ONLY parity: the Apple-Health hourly-step IMPORT that populates this table is iOS-only (HealthKit + * has no Android analogue), so Android carries the table for .noopbak byte-parity but no importer writes it + * — the same arrangement the older `appleDaily` table already has. The column set + order + PK must match + * the GRDB migration in Packages/WhoopStore/Sources/WhoopStore/Database.swift so the two schemas agree. + */ +class AppleStepHourMigrationTest { + + @Test + fun migration_isAdditive_onlyCreateTable() { + val sql = WhoopDatabase.APPLE_STEP_HOUR_MIGRATION_SQL + assertEquals("one CREATE TABLE statement", 1, sql.size) + for (s in sql) { + val up = s.trimStart().uppercase() + assertTrue("only CREATE TABLE allowed, got: $s", up.startsWith("CREATE TABLE")) + for (banned in listOf("DROP ", "DELETE ", "UPDATE ", "INSERT ", "ALTER ")) { + assertTrue("additive migration must not contain '$banned': $s", !up.contains(banned)) + } + } + } + + @Test + fun migration_createsExactTable() { + assertEquals( + listOf( + "CREATE TABLE IF NOT EXISTS `appleStepHour` (`deviceId` TEXT NOT NULL, " + + "`ts` INTEGER NOT NULL, `steps` INTEGER NOT NULL, PRIMARY KEY(`deviceId`, `ts`))", + ), + WhoopDatabase.APPLE_STEP_HOUR_MIGRATION_SQL, + ) + } + + @Test + fun migration_versionPair_is30to31() { + assertEquals(30, WhoopDatabase.MIGRATION_30_31.startVersion) + assertEquals(31, WhoopDatabase.MIGRATION_30_31.endVersion) + } + + /** The declared schema version must track the `@Database(version = …)` the migration lands on. */ + @Test + fun schemaVersionConstant_matchesMigrationTarget() { + assertEquals(WhoopDatabase.MIGRATION_30_31.endVersion, WhoopDatabase.SCHEMA_VERSION) + } + + /** + * The entity carries the hour-bucket start ts + the cumulative step sum verbatim; the natural key + * (deviceId, ts) makes the hourly upsert idempotent, exactly like the GRDB ON CONFLICT(deviceId, ts). + */ + @Test + fun appleStepHour_entity_shape() { + val row = AppleStepHour(deviceId = "apple-health", ts = 1_780_917_200L, steps = 250) + assertEquals("apple-health", row.deviceId) + assertEquals(1_780_917_200L, row.ts) + assertEquals(250, row.steps) + } +} diff --git a/android/app/src/test/java/com/noop/data/DeviceRegistryTest.kt b/android/app/src/test/java/com/noop/data/DeviceRegistryTest.kt index b72093f32b..97fbb06a5b 100644 --- a/android/app/src/test/java/com/noop/data/DeviceRegistryTest.kt +++ b/android/app/src/test/java/com/noop/data/DeviceRegistryTest.kt @@ -100,6 +100,7 @@ class DeviceRegistryTest { override suspend fun deleteJournalFor(deviceId: String) { deletedTables += "journal" to deviceId } override suspend fun deleteWorkoutsFor(deviceId: String) { deletedTables += "workout" to deviceId } override suspend fun deleteAppleDailyFor(deviceId: String) { deletedTables += "appleDaily" to deviceId } + override suspend fun deleteAppleStepHoursFor(deviceId: String) { deletedTables += "appleStepHour" to deviceId } override suspend fun deleteMetricSeriesFor(deviceId: String) { deletedTables += "metricSeries" to deviceId } override suspend fun deleteDayOwnershipFor(deviceId: String) { deletedTables += "dayOwnership" to deviceId @@ -136,6 +137,7 @@ class DeviceRegistryTest { override suspend fun reKeyJournal(from: String, to: String) {} override suspend fun reKeyWorkouts(from: String, to: String) {} override suspend fun reKeyAppleDaily(from: String, to: String) {} + override suspend fun reKeyAppleStepHour(from: String, to: String) {} override suspend fun reKeyMetricSeries(from: String, to: String) {} override suspend fun reKeyDayOwnership(from: String, to: String) { for ((day, row) in owners) if (row.deviceId == from) owners[day] = row.copy(deviceId = to) @@ -309,6 +311,10 @@ class DeviceRegistryTest { "journal", "workout", "appleDaily", "metricSeries", "dayOwnership", "scoreInputProvenance", "sleepStateSample", "labMarker", "liveSession", "dismissedWorkout", "dismissedSleep", + // v38-apple-step-hour: hourly Apple-Health steps. No Android importer writes this table, but a + // `.noopbak` restored from iOS carries its rows — and THIS path is "Remove Apple Health + // imported data", so leaving them behind would be the plainest form of the defect. + "appleStepHour", ) assertEquals(expectedTables, dao.deletedTables.map { it.first }.toSet()) // Every delete was scoped to the requested device, not the seeded my-whoop. diff --git a/android/app/src/test/resources/schema_oracle.json b/android/app/src/test/resources/schema_oracle.json index 0a3b819eb2..06735f9672 100644 --- a/android/app/src/test/resources/schema_oracle.json +++ b/android/app/src/test/resources/schema_oracle.json @@ -1,6 +1,6 @@ { "_readme": "SHARED Room<->GRDB SCHEMA ORACLE (#775). Two byte-identical copies: Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json and android/app/src/test/resources/schema_oracle.json. SchemaOracleTests.swift compares GRDB's PRAGMA table_info/index_list against it; SchemaOracleTest.kt compares Room's exported schema JSON against it. `columns` is the iOS/GRDB shape in GRDB column order (macOS is the reference implementation); every way Android differs is spelled out in an `android` / `iosAbsent` / `androidColumnOrder` override naming a key in `divergenceReasons`. Adding a column, reordering one, or changing a type/nullability on one platform only fails both suites until it is either fixed or written down here.", - "roomVersion": 30, + "roomVersion": 31, "grdbMigrations": [ "v1", "v2", @@ -38,7 +38,8 @@ "v34-sleep-staging-sparse", "v35-rr-future-quarantine", "v36-whoop-caps-no-spo2", - "v37-daily-avg-sdnn" + "v37-daily-avg-sdnn", + "v38-apple-step-hour" ], "divergenceReasons": { "android-orphan-synced-column": "REAL COLUMN DRIFT: `synced` exists on Android only. GRDB's v10 note is explicit that stepSample gets no `synced` column ('unused; see StreamStore'), and v12's ppgHrSample never had one either; the Room entities copied the flag from the older per-second entities. Nothing reads it on Android \u2014 the per-row upload flag belongs to an upload path that does not exist there \u2014 so it is dead width, not divergent data. Removing it needs a Room table rebuild.", @@ -122,6 +123,34 @@ ], "indices": [] }, + "appleStepHour": { + "platform": "both", + "columns": [ + { + "name": "deviceId", + "affinity": "TEXT", + "notNull": true, + "default": null + }, + { + "name": "ts", + "affinity": "INTEGER", + "notNull": true, + "default": null + }, + { + "name": "steps", + "affinity": "INTEGER", + "notNull": true, + "default": null + } + ], + "primaryKey": [ + "deviceId", + "ts" + ], + "indices": [] + }, "battery": { "platform": "both", "columns": [