From a6fc0f05c8ce9e89c8214c60745d882c00e4f61f Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sat, 27 Jun 2026 20:26:21 -0400 Subject: [PATCH 001/171] docs: design spec for Oura live API import (one-time backfill lane) Off-by-default OAuth (BYO Oura app, authorization-code) one-time pull of the full Oura API v2 surface. Lossless raw archive (new ouraRaw table, migration v19) plus normalized projection into existing WhoopStore tables, incl. the sleep_phase_5_min hypnogram the file-import lane can't carry. Networking lives in the app target beside AICoach; StrandImport gains only pure API parsers and stays offline-pure. New DataSourceKind.ouraApi; honest-data doctrine preserved. --- .../2026-06-27-oura-live-api-import-design.md | 372 ++++++++++++++++++ 1 file changed, 372 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-27-oura-live-api-import-design.md diff --git a/docs/superpowers/specs/2026-06-27-oura-live-api-import-design.md b/docs/superpowers/specs/2026-06-27-oura-live-api-import-design.md new file mode 100644 index 0000000000..d805c23cfd --- /dev/null +++ b/docs/superpowers/specs/2026-06-27-oura-live-api-import-design.md @@ -0,0 +1,372 @@ +# Oura live API import — design + +- **Date:** 2026-06-27 +- **Status:** Draft for review +- **Scope:** iOS only (the Swift app). macOS, Android explicitly out (see §13). +- **Author:** design session (brainstorming → spec) +- **Supersedes/relates:** the existing offline Oura file-import lane (`WearableExportImporter` / `OuraExportParser`); the roadmap entry [`docs/DEVICE_SUPPORT_ROADMAP.md`](../../DEVICE_SUPPORT_ROADMAP.md) "Oura — Cloud API v2 — off-by-default OAuth import lane only". + +--- + +## 1. Summary + +NOOP already imports a user's own Oura data from the **account export file** (a single JSON), parsed by +[`OuraExportParser`](../../../Packages/StrandImport/Sources/StrandImport/OuraExportParser.swift) into the +normalized `WearableDailyRow` / `WearableSleepSession` models. That lane is lossy by necessity: the file +export carries stage **durations** but no hypnogram, no heart-rate/HRV time series, and only a handful of +daily metrics. + +This feature adds a **live Oura API v2 lane** that pulls the user's **full** Oura history once, over the +network, under the user's own OAuth grant — capturing the rich data the file export can't: the +`sleep_phase_5_min` hypnogram, `heart_rate`/`hrv` time series, per-night HR series, SpO2, daytime stress, +resilience, cardiovascular age, VO2 max, workouts, sessions, tags, and ring metadata, across ~18 endpoints. + +It is a **one-time, pull-everything** operation (the user's stated use case), behind an off-by-default +opt-in — NOOP's **second** network exception after the AI Coach. Storage is **lossless**: every raw Oura +payload is archived verbatim, and a curated subset is normalized into the existing on-device tables so the +dashboard, Sleep tab, and Metric Explorer light up immediately. A future "Sync again" re-pull is a button, +not new architecture. + +--- + +## 2. Goals / non-goals + +### Goals +- **G1 — Maximum fidelity.** Pull as much of the Oura v2 surface as the grant allows; never silently drop a + field. Raw payloads are retained verbatim so new metrics can be derived later without re-fetching. +- **G2 — One-time backfill.** A single "Connect Oura & Import Everything" action that authenticates, walks + all of history across all endpoints, stores, and reports an honest summary. +- **G3 — Fit, don't duplicate.** Reuse `OuraExportParser`'s field mapping and the existing `WhoopStore` + write path (`dailyMetric`, `sleepSession`, `metricSeries`, `hrSample`, `workout`) and the device registry. +- **G4 — Honest data.** Oura's own scores (readiness, sleep, activity, resilience) are stored **reference-only** + and never become NOOP's Charge/Effort/Rest, exactly as the file lane already does. +- **G5 — Preserve invariants.** Every Swift package stays network-free; `StrandImport` stays offline-pure; + untrusted JSON is treated as hostile; the WHOOP experience never regresses. +- **G6 — Re-pull seam.** Tokens (incl. refresh) persist so a later re-pull needs no re-auth; auth sits + behind an `AuthProvider` protocol so OAuth-with-backend can replace BYO-app later without touching fetch/parse. + +### Non-goals (this spec) +- **NG1 — No background sync / webhooks.** No `BGTaskScheduler`, no incremental "changed since" diffing, no + Oura webhook subscriptions (those need a public callback URL / backend). +- **NG2 — No backend.** No server component. The OAuth `client_secret` is the user's own (BYO app). +- **NG3 — No macOS / Android.** iOS Swift app only. +- **NG4 — No new scoring.** NOOP's recovery/strain/sleep math is unchanged; this only feeds it inputs. + +--- + +## 3. Background constraints (verified) + +### 3.1 Oura API v2 (verified against the live OpenAPI spec `openapi-1.35.json`, `info.version 2.0`, fetched 2026-06-27) +- **Personal Access Tokens are deprecated (Dec 2025) and can no longer be created.** OAuth2 is the only path. +- **OAuth2 is plain authorization-code, no PKCE.** Authorize `https://cloud.ouraring.com/oauth/authorize`; + token `https://api.ouraring.com/oauth/token`; bearer header `Authorization: Bearer `. The code→token + exchange requires `client_id` + `client_secret`. Refresh tokens are issued by the server-side flow. +- **Scopes (8):** `email`, `personal`, `daily`, `heartrate`, `workout`, `tag`, `session`, `spo2Daily` + (the auth page labels SpO2 as `spo2` — verify against the live consent screen at build time). +- **Pagination:** every list/time-series response carries `next_token` (repeat with `next_token=` until null). +- **Rate limit:** 5000 requests / 5-minute window → HTTP 429 on exceed. A multi-year backfill across ~18 + endpoints fits comfortably with light pacing + backoff. +- **No `changed_since`.** Incremental sync (future) re-pulls a trailing window and upserts by `id`, because + Oura re-scores prior days. Not needed for the one-time pull. +- **Membership gate:** Gen3 / Ring 4 users without an active Oura Membership get empty/4xx from the API. + Handle honestly in the UI (see §9). +- **Sandbox:** every collection endpoint has a twin at `/v2/sandbox/usercollection/...` returning static demo + data with identical schemas — used for development and as a live smoke test without a real account. + +### 3.2 NOOP invariants this feature must not break +- **Offline-by-default privacy posture** ([`docs/PRIVACY_SECURITY.md`](../../PRIVACY_SECURITY.md) §1). + The five Swift packages are network-free; the AI Coach (in the **app target**) is the single opt-in network + exception. → The Oura networking lives in the **app target**, and is documented as the **second** opt-in + exception. The data flow is **inbound** (Oura → device, the user's own data, under the user's own grant); + nothing of the user's existing NOOP data leaves the device except the OAuth handshake with Oura. +- **`StrandImport` is offline-pure** — its banner reads "STAY OFFLINE: nothing here touches the network." + → Only **pure** (no-network) Oura-API response parsers are added to `StrandImport`; the fetch happens in the + app target. This mirrors the existing split: `StrandImport` parses, `Strand/Data/WearableImporter.swift` writes. +- **Honest data** — a brand's own scores are reference-only; NOOP recomputes downstream. +- **Untrusted input is hostile** ([`docs/PRIVACY_SECURITY.md`](../../PRIVACY_SECURITY.md) §3) — byte caps, + finite/range-checked numerics, bounded collection counts (reuse `WearableJSON` coercion helpers). +- **Supply chain** — dependencies are pinned `exact:` and must match across `Packages/*/Package.swift` and + `project.yml`. This feature adds **no** third-party dependency (uses `URLSession` + `ASWebAuthenticationSession` + + `Security`, all first-party). +- **WHOOP primacy** — Oura is an additional source; it must not auto-seize day ownership from WHOOP (§8.4). + +--- + +## 4. Architecture overview + +``` +┌──────────────────────────── app target: Strand/Oura/ (NETWORK lives here, beside Strand/AI/AICoach) ────────────────────────────┐ +│ │ +│ OuraCredentials OuraAuth (AuthProvider) OuraAPIClient OuraSyncCoordinator OuraSyncWriter │ +│ client_id/secret → ASWebAuthenticationSession → URLSession + paging + → one-time backfill: → WhoopStore │ +│ from xcconfig code → token exchange rate-limit/backoff + fan out across all upserts + │ +│ (Info.plist) tokens → Keychain (AIKeyStore) sandbox/prod base URL endpoints, page each registry + │ +│ date-windowed, decode dayOwnership │ +│ via pure parsers ↓ │ +└─────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────┘ + │ Data (raw JSON) +┌──────────────── package: StrandImport (PURE, network-free) ─────────────┐ ▼ +│ OuraApiParser (new) ── reuses ──▶ OuraExportParser helpers │ ┌──────────── package: WhoopStore (GRDB/SQLite) ───────────┐ +│ + OuraHypnogram decoder (sleep_phase_5_min / 30_sec, movement_30_sec) │ │ ouraRaw (NEW, v19) · dailyMetric · sleepSession │ +│ + rich API model types (time series, sessions, resilience, …) │ │ metricSeries · hrSample · workout · pairedDevice │ +│ emits: WearableDailyRow / WearableSleepSession (+ stages) + rich models │ │ dayOwnership │ +└─────────────────────────────────────────────────────────────────────────┘ └───────────────────────────────────────────────────────────┘ +``` + +**Components (all new unless noted):** + +| Component | Location | Responsibility | Networks? | +|---|---|---|---| +| `OuraCredentials` | `Strand/Oura/` | Read `client_id` / `client_secret` / redirect from an **untracked xcconfig** surfaced via Info.plist. | no | +| `AuthProvider` (protocol) + `OuraOAuthProvider` | `Strand/Oura/` | The auth seam. `OuraOAuthProvider` runs `ASWebAuthenticationSession`, exchanges code→tokens, stores/loads/refreshes via Keychain. | yes | +| `OuraTokenStore` | `Strand/Oura/` | Keychain wrapper for access/refresh tokens + expiry, mirroring `AIKeyStore` (service `com.noop.oura`). | no | +| `OuraAPIClient` | `Strand/Oura/` | Typed `URLSession` client: per-endpoint GET, `next_token` paging, 429/5xx backoff, base-URL switch (prod vs sandbox), bounded byte reads. | yes | +| `OuraSyncCoordinator` | `Strand/Oura/` | One-time backfill orchestration: discover account span, fan out across endpoints, window requests, hand pages to the pure parser, drive progress, persist a "last full sync" marker. | yes (via client) | +| `OuraSyncWriter` | `Strand/Oura/` | Map parsed models → `WhoopStore` upserts; archive raw; register the source; never auto-own days. Mirrors `WearableImporter`. | no | +| `OuraApiParser` + `OuraHypnogram` + rich model types | `Packages/StrandImport/Sources/StrandImport/` | **Pure** decode of Oura API JSON → normalized + rich models; reuses `OuraExportParser` + `WearableJSON`. | no | +| `ouraRaw` table (migration **v19**) | `Packages/WhoopStore/.../Database.swift` | Lossless raw payload archive. | no | +| `DataSourceKind.ouraApi` | `Packages/StrandImport/.../ImportModels.swift` | New provenance case (enum only). | no | +| Oura card + connect/sync UI | `Strand/Screens/DataSourcesView.swift` (+ small new views) | Connect, progress, summary, membership-gate messaging, disconnect. | no | + +--- + +## 5. Auth design (BYO Oura app, authorization-code, one-time) + +### 5.1 Flow +1. The user registers a **free OAuth application** on Oura's developer portal once (redirect URI = + NOOP's custom scheme, e.g. `noop://oura/callback`), then provides its `client_id` + `client_secret`. + For the personal/one-time case these go into an **untracked xcconfig** (no in-app credential screen to + build); see §5.3. (A paste-credentials screen is a noted alternative — §12 Q2.) +2. Tapping **Connect** runs `ASWebAuthenticationSession` against + `https://cloud.ouraring.com/oauth/authorize?response_type=code&client_id=…&redirect_uri=…&scope=&state=`. +3. On redirect, NOOP validates `state`, extracts `code`, and POSTs to + `https://api.ouraring.com/oauth/token` with `grant_type=authorization_code`, the code, redirect, and + `client_id`/`client_secret` → `{access_token, refresh_token, expires_in}`. +4. Tokens + expiry are stored in Keychain (`OuraTokenStore`, `kSecAttrAccessibleAfterFirstUnlock`, mirroring + [`AIKeyStore`](../../../Strand/AI/AICoach.swift)). +5. The backfill (§7) runs. On 401 mid-sync, the client refreshes via `grant_type=refresh_token` once and retries. + +### 5.2 The `AuthProvider` seam +```swift +protocol AuthProvider { // Strand/Oura/ + func validAccessToken() async throws -> String // returns a fresh token, refreshing if needed + func authorize() async throws // interactive: runs ASWebAuthenticationSession + exchange + func signOut() throws // clears Keychain + var isConnected: Bool { get } +} +``` +`OuraAPIClient` depends only on `AuthProvider`. Shipping BYO-app today vs. a backend-mediated flow later is a +provider swap; fetch/parse/store are untouched. + +### 5.3 Credentials delivery (default: xcconfig) +- An untracked `Strand/Oura/OuraSecrets.xcconfig` (gitignored) defines `OURA_CLIENT_ID`, `OURA_CLIENT_SECRET`, + `OURA_REDIRECT_URI`, referenced from `project.yml` (XcodeGen) and surfaced into Info.plist; `OuraCredentials` + reads them at runtime. A committed `OuraSecrets.example.xcconfig` documents the keys. +- The redirect scheme is registered in `CFBundleURLTypes` (Info.plist via `project.yml`). +- **No secret is committed**; for a single-user build the secret never leaves that user's machine/binary. + +--- + +## 6. The Oura API client + +- **Base URL switch:** `https://api.ouraring.com/v2/usercollection/…` (prod) vs `…/v2/sandbox/usercollection/…` + (dev/tests). A single `OuraEnvironment` enum. +- **Request:** `GET` with `Authorization: Bearer`, `start_date`/`end_date` (or `start_datetime`/`end_datetime` + for `heartrate`), and `next_token` when paging. +- **Paging:** loop until `next_token == nil`, accumulating `data[]`. A per-endpoint hard page cap (e.g. 10⁴ + pages) is a runaway guard, logged if hit (never silent — §3.2). +- **Rate-limit & resilience:** on `429`, honor backoff and retry; on `5xx`/transport error, bounded + exponential backoff with jitter; on `401`, refresh-once-then-retry. A small global pacer keeps well under + 5000/5-min. +- **Bounded reads:** cap per-response bytes (reuse the 256 MB ceiling rationale from `WearableExportImporter`) + and decode with `JSONSerialization` through the existing `WearableJSON` finite/range-checked coercion so a + hostile/garbled response can't OOM or trap. +- **Decoding:** the client returns raw `Data` per page to the coordinator, which (a) hands it to `OuraSyncWriter` + for verbatim archival and (b) hands it to the pure `OuraApiParser` for normalization. The client itself does + not interpret payloads beyond `data`/`next_token` envelope extraction. + +--- + +## 7. Sync orchestration (one-time backfill) + +`OuraSyncCoordinator.runFullImport()`: +1. **Span discovery.** `GET personal_info` (also yields PII to optionally store, §12 Q3) and probe the earliest + available day (e.g. widen `start_date` back in coarse windows until a daily endpoint returns empty). Default + floor: a fixed early date (e.g. 2013-01-01, before any Oura ring shipped) so we never miss history. +2. **Fan out across endpoints** (§8.1). For each: page through `[span.start … today]` in date-windowed chunks + (e.g. 90-day windows for summaries; shorter for `heartrate`/`sleep` to bound page size), following `next_token`. +3. **Per page:** archive raw (`ouraRaw` upsert by `(endpoint, documentId)`), then parse + accumulate normalized rows. +4. **Persist** via `OuraSyncWriter` (§8): batched upserts to `dailyMetric` / `sleepSession` / `metricSeries` / + `hrSample` / `workout`; register the `pairedDevice`; set ring model from `ring_configuration`. +5. **Progress** is reported per endpoint (e.g. "sleep: 1,240 nights", "heartrate: 412k samples") to the UI. +6. **Completion marker:** store `last_full_sync` (epoch) in `cursors` (`oura:lastFullSync`) so a future re-pull + is informed and the UI can show "Last imported …". + +Ordering: fetch `ring_configuration` early (sets the device model), then `sleep` + daily summaries (the +dashboard's core), then `heartrate` (the largest), then the long tail (sessions/tags/rest_mode/etc.). A failure +in one endpoint is logged and skipped without aborting the whole import (honest partial — surfaced in the summary). + +**Future re-pull** (out of scope to build, in scope to not preclude): the same `runFullImport` with a trailing +window + refreshed token. Idempotent because every store upsert is keyed by natural key / Oura `id`. + +--- + +## 8. Data model & storage + +### 8.1 Endpoint → storage mapping + +All endpoints are archived verbatim to `ouraRaw`. The normalized columns below are the curated projection that +drives existing UI. "ref-only" = reference `metricSeries` key, never a NOOP score (G4). + +| Oura endpoint | Raw | Normalized target | Notes | +|---|---|---|---| +| `personal_info` | ✓ | (optional) profile PII, local only | §12 Q3 decides storing age/sex/height/weight. | +| `ring_configuration` | ✓ | `pairedDevice.model` (hardware_type → "Oura Ring Gen3/4/…") | Sets the source's display model. | +| `sleep` (detailed) | ✓ | `sleepSession` (+ **hypnogram** → `stagesJSON`, **movement** → `motionJSON`), daily sleep rollup → `dailyMetric` | The richest object. HR series → `hrSample`; HRV series → raw + `avgHrv`. See §8.2. | +| `daily_sleep` | ✓ | `ref_sleep_score` (ref-only) + contributor keys | Score is reference. | +| `daily_readiness` | ✓ | `dailyMetric.restingHr`, `skinTempDevC`; `ref_readiness_score` + contributor keys | Mirrors `OuraExportParser`. | +| `daily_activity` | ✓ | `dailyMetric.steps`; `metricSeries` steps/active_kcal/total_kcal/met-minutes; `ref_activity_score` | `met`/`class_5_min` series → raw. | +| `daily_spo2` | ✓ | `dailyMetric.spo2Pct`; `metricSeries` spo2, bdi | | +| `daily_stress` | ✓ | `metricSeries` stress_high_s, recovery_high_s; day_summary encoded ref | Categorical summary stored ref-only. | +| `daily_resilience` | ✓ | `metricSeries` resilience contributors (0–100); `ref_resilience_level` | Level encoded 1–5 ref-only. | +| `daily_cardiovascular_age` | ✓ | `metricSeries` vascular_age, pulse_wave_velocity | | +| `vO2_max` | ✓ | `metricSeries` vo2max | (no `dailyMetric` column; `appleDaily.vo2max` is Apple-specific). | +| `sleep_time` | ✓ | `metricSeries` optimal bedtime offsets (ref) | Recommendation enum → raw. | +| `workout` | ✓ | `workout` table (sport=activity, source, durationS, energyKcal, distanceM, start/end) | | +| `session` | ✓ | `metricSeries` session counts/day; HR/HRV/motion arrays → raw | Sessions aren't workouts; normalized surface kept minimal v1. | +| `tag` / `enhanced_tag` | ✓ | raw (annotations surfaced later) | `tag` is deprecated; prefer `enhanced_tag`. | +| `rest_mode_period` | ✓ | raw | | +| `ring_battery_level` | ✓ | raw | | +| `heartrate` (time series) | ✓ | `hrSample(deviceId, ts, bpm)` | `source` enum preserved in raw. Largest payload — see §8.3. | + +Reused write APIs (all already exist): `upsertDailyMetrics`, `upsertSleepSessions`, `upsertMetricSeries` +([`MetricsCache.swift` / `MetricSeriesStore.swift`]), `upsertWorkouts` +([`JournalWorkoutAppleCache.swift`]), `insert(_ streams:deviceId:)` for `hrSample` +([`StreamStore.swift`]), and `DeviceRegistryStore.add` / `setDayOwner` / `deleteAllData`. + +### 8.2 Sleep hypnogram (the headline win) +- `OuraHypnogram.decode(_:)` turns `sleep_phase_5_min` (or the finer `sleep_phase_30_sec` when present) into + `[{start,end,stage}]` segments, epoch-aligned from `bedtime_start`. Legend (verbatim): `1=deep, 2=light, + 3=REM, 4=awake` → NOOP stage strings `"deep"/"light"/"rem"/"wake"` (matching `WearableSleepStageInterval`). + This populates `sleepSession.stagesJSON`, which the Oura **file** lane always left empty. +- `movement_30_sec` (legend `1=no motion … 4=active` — a **different** decoder) → `sleepSession.motionJSON` + (the v18 per-epoch motion column). +- `heart_rate` `PublicSample` (interval+items+timestamp) → `hrSample` rows under `deviceId="oura-api"`. +- `hrv` `PublicSample` → archived raw + per-night `avgHrv` on the session (no good per-sample normalized home; + `rrInterval` is RR not rMSSD — noted honestly, not faked). +- Scalar mapping (`bedtime_start/end`, `total/deep/light/rem_sleep_duration`, `awake_time`, `efficiency`, + `average_heart_rate`, `lowest_heart_rate`, `average_hrv`, `average_breath`) reuses `OuraExportParser.sleepSession` + semantics (durations are seconds → minutes). + +### 8.3 Heart-rate volume +5-minute cadence ≈ 288 points/day (denser during workouts/sessions) → low-hundreds-of-thousands of rows over +years; `hrSample` is a compact `(deviceId, ts, bpm)` table and absorbs this fine (tens of MB). Raw `heartrate` +pages are also archived (lossless). Because raw HR roughly doubles HR storage, **raw heartrate archival is a +toggle** (default on for "as much as we can get"; off saves the most space) — §12 Q1. + +### 8.4 Provenance, registry, day ownership +- New `DataSourceKind.ouraApi`; partition `deviceId = "oura-api"` (distinct from the file lane's `"oura-import"`, + so live and file data coexist as separate sources). +- Register a `PairedDevice(id:"oura-api", brand:"Oura", model:, sourceKind:, capabilities:[sleep,hr,hrv,spo2,skinTemp,steps,…], status:.active)` via `DeviceRegistryStore.add`. +- **Do not auto-set `dayOwnership` to `oura-api`** — WHOOP/primary stays the default owner; Oura is an additional + comparable source the user can select. (Respects WHOOP primacy; never blends scores.) +- **Disconnect** = `DeviceRegistryStore.deleteAllData(deviceId:"oura-api")` + clear Keychain tokens + delete + `ouraRaw` rows for the source. + +### 8.5 `ouraRaw` schema (migration v19) +``` +ouraRaw: + endpoint TEXT NOT NULL -- "sleep" | "daily_readiness" | "heartrate" | … + documentId TEXT NOT NULL -- Oura `id`; for heartrate pages, a synthesized window key + day TEXT -- YYYY-MM-DD when the doc is day-keyed (nullable) + payloadJSON TEXT NOT NULL -- the raw object, verbatim (JSONEncoder .sortedKeys for stable bytes) + fetchedAt INTEGER NOT NULL -- unix seconds + PRIMARY KEY (endpoint, documentId) + INDEX (endpoint, day) +``` +Additive migration only; no existing table touched. (`DATA_MODEL.md` is stale at "v9"; the live migrator is at +v18 — this is **v19**. The doc's schema table will be updated to reflect v10–v19, including `ouraRaw`.) + +--- + +## 9. UI (iOS) + +- **Data Sources screen** ([`Strand/Screens/DataSourcesView.swift`]): add an **Oura (live)** card, sibling to + the existing wearable file-import card. Primary action **"Connect Oura & Import Everything."** +- **Connect:** runs auth (§5) then the backfill (§7) as a foreground operation with **per-endpoint progress** + and a cancel. (One-time, foreground — no background task.) +- **Summary:** honest completion line — counts per category and date span (mirrors `WearableExportImporter.summaryText`), + including any endpoints that were skipped/failed (never a silent "complete"). +- **Membership gate:** if the API returns empty/4xx for the membership reason, show a clear, non-alarming + message ("Oura returned no data — an active Oura Membership is required for API access") rather than a generic error. +- **Connected state:** show "Last imported ", a **Sync again** button (re-runs §7), and **Disconnect** (§8.4). +- The card is **off/empty until connected**; no network call happens until the user taps Connect (privacy posture). + +--- + +## 10. Security & privacy + +- **Second opt-in network lane.** Update [`docs/PRIVACY_SECURITY.md`](../../PRIVACY_SECURITY.md) §1/§4/§5 to + document the Oura lane honestly: off-by-default, user-initiated, **inbound** (the user's own Oura data → device + under the user's own OAuth app), tokens in Keychain, no NOOP server, raw data never exfiltrated. §4's "No + WHOOP account or API credentials" stays true (this is Oura, the user's own grant) but the blanket "offline by + default / one exception" framing is amended to "two opt-in exceptions: AI Coach and Oura import." +- **Token storage:** Keychain only, `kSecAttrAccessibleAfterFirstUnlock`, service `com.noop.oura`. Never logged + (the strap-log redaction policy extends: no tokens, no `code`, no `client_secret` in any log). +- **Untrusted responses:** bounded byte reads + `WearableJSON` finite/range-checked coercion + per-collection + caps (parity with `WearableExportImporter`). +- **iOS entitlements:** the iOS build permits outbound network (no macOS-style sandbox block); the redirect + scheme is registered in `CFBundleURLTypes`. No new ATS exemption expected (Oura is standard TLS) — verify. +- **macOS note:** the macOS app would need the `com.apple.security.network.client` entitlement to use this lane; + out of scope here (iOS-only), and the spec deliberately does not add it. + +--- + +## 11. Testing + +- **Pure parsers** (`StrandImportTests`, XCTest, inline-JSON fixtures like `WearableExportImporterTests`): + `OuraApiParser` per endpoint; `OuraHypnogram.decode` round-trips (`sleep_phase_5_min`/`30_sec` legends, + epoch alignment, the distinct `movement_30_sec` legend); honest-data assertions (Oura scores land only on + `ref_*` keys, never `recovery`/`strain`). +- **Client** (app-target tests): a `URLProtocol` stub feeds canned responses — paging via `next_token`, 429 + backoff, 401-refresh-once, bounded-bytes, prod/sandbox base URL. +- **Writer/integration:** in-memory `WhoopStore.inMemory()` → assert `dailyMetric`/`sleepSession`(+stages)/ + `hrSample`/`metricSeries`/`workout` rows and `pairedDevice` registration; assert day-ownership is **not** + seized from WHOOP. +- **Live smoke (manual/dev):** point the client at the `/v2/sandbox/...` tree (no account) to exercise the real + envelope/paging end-to-end. + +--- + +## 12. Open questions for your review + +- **Q1 — Raw heart-rate archival.** Default **on** (true lossless) but it roughly doubles HR storage. OK to + default on with a toggle, or default off (still keep normalized `hrSample`)? +- **Q2 — Credentials UX.** Default is an untracked **xcconfig** (no UI to build, best for one-time/personal). Want + a small **paste-credentials screen** instead so any user can connect without rebuilding? (Larger scope.) +- **Q3 — `personal_info` PII.** Store age/sex/height/weight locally (useful for some scores), or archive raw only? +- **Q4 — Module placement.** Pure parsers in **`StrandImport`** (max reuse, recommended) vs. a new pure package + **`StrandOura`** (cleaner home, more scaffolding). This refines the earlier "new networked module" framing: + networking is in the app target regardless; only the parser home is in question. + +--- + +## 13. Out of scope + +Background sync, webhooks, incremental "changed-since" diffing, a backend/token-proxy, OAuth-for-all-users +(BYO-app only for now), macOS, Android, and any change to NOOP's scoring math. All are reachable later behind +the `AuthProvider` seam and the idempotent store upserts without reworking this lane. + +--- + +## 14. Touched files (high level — detailed steps come in the implementation plan) + +**New (app target `Strand/Oura/`):** `OuraCredentials.swift`, `OuraAuth.swift` (`AuthProvider` + `OuraOAuthProvider`), +`OuraTokenStore.swift`, `OuraAPIClient.swift`, `OuraEnvironment.swift`, `OuraSyncCoordinator.swift`, `OuraSyncWriter.swift`, ++ small SwiftUI views for the card/progress/summary. +**New (package `StrandImport`):** `OuraApiParser.swift`, `OuraHypnogram.swift`, rich API model types (+ tests). +**Edited:** `Packages/WhoopStore/.../Database.swift` (migration **v19** `ouraRaw`) + a small `OuraRawStore.swift`; +`Packages/StrandImport/.../ImportModels.swift` (`DataSourceKind.ouraApi`); `Strand/Screens/DataSourcesView.swift` +(Oura card + flow); `project.yml` (xcconfig ref + `CFBundleURLTypes`); `.gitignore` (`OuraSecrets.xcconfig`); +docs: `PRIVACY_SECURITY.md`, `DEVICE_SUPPORT_ROADMAP.md` (Oura row → "in progress"), `DATA_MODEL.md` (v10–v19 incl. `ouraRaw`). From 9ea5a7d820c63c9f13e5f115bad1774eef90f054 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sat, 27 Jun 2026 20:37:12 -0400 Subject: [PATCH 002/171] =?UTF-8?q?docs:=20implementation=20plan=201/3=20?= =?UTF-8?q?=E2=80=94=20Oura=20import=20foundation=20(parsers=20+=20raw=20s?= =?UTF-8?q?tore)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TDD task breakdown for the network-free core: ouraRaw archive (migration v19) + OuraRawStore, the OuraHypnogram decoder, and the pure OuraApiParser (sleep/daily/events) reusing OuraExportParser semantics. Additive only. --- ...06-27-oura-live-api-import-1-foundation.md | 867 ++++++++++++++++++ 1 file changed, 867 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-27-oura-live-api-import-1-foundation.md diff --git a/docs/superpowers/plans/2026-06-27-oura-live-api-import-1-foundation.md b/docs/superpowers/plans/2026-06-27-oura-live-api-import-1-foundation.md new file mode 100644 index 0000000000..e3504bfac8 --- /dev/null +++ b/docs/superpowers/plans/2026-06-27-oura-live-api-import-1-foundation.md @@ -0,0 +1,867 @@ +# Oura Live API Import — Plan 1: Foundation (pure parsers + raw storage) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the network-free core of the Oura live-API lane — the lossless raw payload store and the pure JSON→model parsers (incl. the sleep hypnogram) — fully unit-tested, touching only new files plus one additive migration. + +**Architecture:** Two packages, additive only. `WhoopStore` gains a `ouraRaw` table (migration **v19**) and an `OuraRawStore` extension mirroring `MetricSeriesStore`. `StrandImport` gains pure decoders (`OuraHypnogram`, `OuraApiParser`) and small neutral model structs that reuse the existing `OuraExportParser` field semantics, `WhoopTime`, and `WearableJSON`. No networking, no edits to existing exhaustive switches, no app-target changes — so this plan can't break any existing build path. The fetch layer, auth, writer, and UI come in Plans 2–3. + +**Tech Stack:** Swift 5.9, GRDB 6.29.3 (pinned), XCTest, `Foundation`. iOS 16 / macOS 13 (both package platforms; the lane ships iOS-only but the packages stay multi-platform). + +## Global Constraints + +Every task implicitly includes these (verbatim from the spec): +- **Packages stay network-free.** No `URLSession`/`URLRequest`/`NWConnection` in `StrandImport` or `WhoopStore`. (`StrandImport` banner: "STAY OFFLINE: nothing here touches the network.") +- **Honest data.** Oura's own scores (readiness/sleep/activity/resilience) are written ONLY under reference keys (`ref_*`) or namespaced (`oura_*`) — never as a NOOP score column (`recovery`/`strain`). NOOP recomputes downstream. +- **Untrusted input is hostile.** Parse via the existing `WearableJSON` finite/range-checked coercion (`dbl`/`int`/`posInt`/`posDbl`/`str`); never trap on attacker NaN/inf/huge values. +- **Additive migrations only.** New tables/indexes; never alter or drop existing rows. The live migrator is at **v18** (`Packages/WhoopStore/Sources/WhoopStore/Database.swift`); this adds **v19**. +- **Pinned deps.** Add no new dependency. Versions across `Packages/*/Package.swift` must stay identical (`GRDB.swift exact 6.29.3`, `ZIPFoundation exact 0.9.20`). +- **Provenance string.** The live-API partition is `deviceId = "oura-api"` (distinct from the file lane's `"oura-import"`). + +--- + +### Task 1: `ouraRaw` table (migration v19) + `OuraRawStore` + +Lossless archive of every raw Oura payload, keyed by `(deviceId, endpoint, documentId)`, idempotent on re-pull. + +**Files:** +- Modify: `Packages/WhoopStore/Sources/WhoopStore/Database.swift` (add migration after the `v18-sleep-motion-state` block, before `return migrator`) +- Create: `Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift` +- Test: `Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift` + +**Interfaces:** +- Consumes: `WhoopStore.inMemory()`, the actor's `syncWrite`/`syncRead` helpers, `db.changesCount` (all existing). +- Produces: + - `struct OuraRawRow { let endpoint: String; let documentId: String; let day: String?; let payloadJSON: String; let fetchedAt: Int }` + - `WhoopStore.upsertOuraRaw(_ rows: [OuraRawRow], deviceId: String) async throws -> Int` + - `WhoopStore.ouraRaw(deviceId: String, endpoint: String) async throws -> [OuraRawRow]` + - `WhoopStore.ouraRawCount(deviceId: String, endpoint: String) async throws -> Int` + +- [ ] **Step 1: Write the failing test** + +Create `Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift`: + +```swift +import XCTest +import GRDB +@testable import WhoopStore + +final class OuraRawStoreTests: XCTestCase { + func testUpsertIsIdempotentByNaturalKey() async throws { + let store = try await WhoopStore.inMemory() + let first = OuraRawRow(endpoint: "sleep", documentId: "abc", day: "2026-01-01", + payloadJSON: #"{"id":"abc","v":1}"#, fetchedAt: 100) + let n1 = try await store.upsertOuraRaw([first], deviceId: "oura-api") + XCTAssertEqual(n1, 1) + + // Re-pull the same document id with a newer payload → overwrite in place, still one row. + let second = OuraRawRow(endpoint: "sleep", documentId: "abc", day: "2026-01-01", + payloadJSON: #"{"id":"abc","v":2}"#, fetchedAt: 200) + _ = try await store.upsertOuraRaw([second], deviceId: "oura-api") + + let rows = try await store.ouraRaw(deviceId: "oura-api", endpoint: "sleep") + XCTAssertEqual(rows.count, 1) + XCTAssertEqual(rows.first?.payloadJSON, #"{"id":"abc","v":2}"#) + XCTAssertEqual(rows.first?.fetchedAt, 200) + XCTAssertEqual(try await store.ouraRawCount(deviceId: "oura-api", endpoint: "sleep"), 1) + // A different endpoint is a different partition. + XCTAssertEqual(try await store.ouraRawCount(deviceId: "oura-api", endpoint: "daily_sleep"), 0) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `swift test --package-path Packages/WhoopStore --filter OuraRawStoreTests` +Expected: FAIL — compile error, `OuraRawRow` / `upsertOuraRaw` undefined. + +- [ ] **Step 3: Add migration v19** + +In `Packages/WhoopStore/Sources/WhoopStore/Database.swift`, insert immediately before `return migrator`: + +```swift + // v19: Oura live-API raw payload archive (lossless). One row per (deviceId, endpoint, documentId); + // payloadJSON holds the verbatim Oura object so any field can be re-derived later without re-fetching + // from the API. Additive only — a NEW table, no existing row touched, old readers unaffected. + migrator.registerMigration("v19-oura-raw") { db in + try db.create(table: "ouraRaw") { t in + t.column("deviceId", .text).notNull() + t.column("endpoint", .text).notNull() // "sleep" | "daily_readiness" | "heartrate" | … + t.column("documentId", .text).notNull() // Oura `id`; heartrate pages use a window key + t.column("day", .text) // YYYY-MM-DD when day-keyed (nullable) + t.column("payloadJSON", .text).notNull() // verbatim object + t.column("fetchedAt", .integer).notNull() // unix seconds + t.primaryKey(["deviceId", "endpoint", "documentId"]) + } + // Per-endpoint reads scan (deviceId, endpoint) then walk day in order. + try db.create(index: "idx_ouraRaw_device_endpoint_day", + on: "ouraRaw", columns: ["deviceId", "endpoint", "day"]) + } +``` + +- [ ] **Step 4: Implement `OuraRawStore`** + +Create `Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift`: + +```swift +import Foundation +import GRDB + +// MARK: - v19 store: lossless Oura live-API payload archive +// Mirrors MetricSeriesStore exactly: a Codable row struct, an idempotent ON CONFLICT upsert keyed by the +// natural key (deviceId, endpoint, documentId), and range reads — all GRDB work via syncWrite/syncRead. +// This is the "as much as we can get" backstop: the verbatim payload is kept so a future metric can be +// derived without re-hitting the Oura API. + +/// One archived Oura API payload. Natural key (deviceId, endpoint, documentId). +public struct OuraRawRow: Equatable, Codable, Sendable { + public let endpoint: String // "sleep" | "daily_readiness" | "heartrate" | … + public let documentId: String // Oura `id`; for heartrate pages, a synthesized window key + public let day: String? // YYYY-MM-DD when the document is day-keyed + public let payloadJSON: String // verbatim object + public let fetchedAt: Int // unix seconds + public init(endpoint: String, documentId: String, day: String?, payloadJSON: String, fetchedAt: Int) { + self.endpoint = endpoint; self.documentId = documentId + self.day = day; self.payloadJSON = payloadJSON; self.fetchedAt = fetchedAt + } +} + +extension WhoopStore { + + /// Upsert raw Oura payloads. Idempotent by (deviceId, endpoint, documentId): re-pulling the same + /// document overwrites its payload/day/fetchedAt rather than duplicating. Returns rows changed. + @discardableResult + public func upsertOuraRaw(_ rows: [OuraRawRow], deviceId: String) async throws -> Int { + try syncWrite { db in + var n = 0 + for r in rows { + try db.execute(sql: """ + INSERT INTO ouraRaw (deviceId, endpoint, documentId, day, payloadJSON, fetchedAt) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(deviceId, endpoint, documentId) DO UPDATE SET + day = excluded.day, + payloadJSON = excluded.payloadJSON, + fetchedAt = excluded.fetchedAt + """, arguments: [deviceId, r.endpoint, r.documentId, r.day, r.payloadJSON, r.fetchedAt]) + n += db.changesCount + } + return n + } + } + + /// Archived payloads for a device + endpoint, oldest day first (null days sort first in SQLite). + public func ouraRaw(deviceId: String, endpoint: String) async throws -> [OuraRawRow] { + try syncRead { db in + try Row.fetchAll(db, sql: """ + SELECT endpoint, documentId, day, payloadJSON, fetchedAt FROM ouraRaw + WHERE deviceId = ? AND endpoint = ? + ORDER BY day ASC + """, arguments: [deviceId, endpoint]) + .map { OuraRawRow(endpoint: $0["endpoint"], documentId: $0["documentId"], + day: $0["day"], payloadJSON: $0["payloadJSON"], fetchedAt: $0["fetchedAt"]) } + } + } + + /// Count archived payloads for a device + endpoint (diagnostics / import summary). + public func ouraRawCount(deviceId: String, endpoint: String) async throws -> Int { + try syncRead { db in + try Int.fetchOne(db, sql: + "SELECT COUNT(*) FROM ouraRaw WHERE deviceId = ? AND endpoint = ?", + arguments: [deviceId, endpoint]) ?? 0 + } + } +} +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `swift test --package-path Packages/WhoopStore --filter OuraRawStoreTests` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add Packages/WhoopStore/Sources/WhoopStore/Database.swift \ + Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift \ + Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift +git commit -m "feat(store): ouraRaw lossless payload archive (migration v19) + OuraRawStore" +``` + +--- + +### Task 2: `OuraHypnogram` decoder + +Pure decode of Oura's two compact per-epoch sleep strings. The legends DIFFER — one decoder must never serve both. + +**Files:** +- Create: `Packages/StrandImport/Sources/StrandImport/OuraHypnogram.swift` +- Test: `Packages/StrandImport/Tests/StrandImportTests/OuraHypnogramTests.swift` + +**Interfaces:** +- Consumes: `WearableSleepStageInterval` (existing, `ImportModels.swift`: `var stage: String; var start, end: Date`). +- Produces: + - `OuraHypnogram.stageName(_ ch: Character) -> String?` + - `OuraHypnogram.decode(_ phases: String, start: Date, epochSeconds: Int) -> [WearableSleepStageInterval]` + - `OuraHypnogram.movement(_ s: String) -> [Int]` + +- [ ] **Step 1: Write the failing test** + +Create `Packages/StrandImport/Tests/StrandImportTests/OuraHypnogramTests.swift`: + +```swift +import XCTest +@testable import StrandImport + +final class OuraHypnogramTests: XCTestCase { + private func d(_ iso: String) -> Date { ISO8601DateFormatter().date(from: iso)! } + + func testStageLegendMapsOuraDigits() { + XCTAssertEqual(OuraHypnogram.stageName("1"), "deep") + XCTAssertEqual(OuraHypnogram.stageName("2"), "light") + XCTAssertEqual(OuraHypnogram.stageName("3"), "rem") + XCTAssertEqual(OuraHypnogram.stageName("4"), "wake") + XCTAssertNil(OuraHypnogram.stageName("0")) + } + + func testDecodeMergesAdjacentEqualStagesAndAligns() { + let start = d("2026-01-01T23:00:00Z") + // "1123" → deep(2 epochs merged), light(1), rem(1); 5-min epochs. + let segs = OuraHypnogram.decode("1123", start: start, epochSeconds: 300) + XCTAssertEqual(segs.map(\.stage), ["deep", "light", "rem"]) + XCTAssertEqual(segs[0].start, start) + XCTAssertEqual(segs[0].end, start.addingTimeInterval(600)) // two 5-min epochs merged + XCTAssertEqual(segs[2].end, start.addingTimeInterval(1200)) // total 4 epochs = 20 min + } + + func testMovementLegendIsDistinctFromStages() { + // movement uses 1=no motion … 4=active; unknown → 0. + XCTAssertEqual(OuraHypnogram.movement("1234x"), [1, 2, 3, 4, 0]) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `swift test --package-path Packages/StrandImport --filter OuraHypnogramTests` +Expected: FAIL — `OuraHypnogram` undefined. + +- [ ] **Step 3: Implement the decoder** + +Create `Packages/StrandImport/Sources/StrandImport/OuraHypnogram.swift`: + +```swift +import Foundation + +// MARK: - Oura compact per-epoch sleep strings (DOCUMENTED Oura API v2 encodings; NOOP's own decoder) +// +// The Oura `sleep` object carries two compact strings. Their digit legends are DIFFERENT — never share a +// decoder between them: +// • sleep_phase_5_min / sleep_phase_30_sec — the hypnogram: '1'=deep, '2'=light, '3'=REM, '4'=awake. +// • movement_30_sec — motion: '1'=no motion, '2'=restless, '3'=tossing, '4'=active. + +public enum OuraHypnogram { + + /// `sleep_phase_*` digit → NOOP stage string ("deep"/"light"/"rem"/"wake", matching + /// `WearableSleepStageInterval`). Unknown digits → nil (skipped, the epoch clock still advances). + public static func stageName(_ ch: Character) -> String? { + switch ch { + case "1": return "deep" + case "2": return "light" + case "3": return "rem" + case "4": return "wake" + default: return nil + } + } + + /// Decode a `sleep_phase_5_min` (epochSeconds 300) or `sleep_phase_30_sec` (epochSeconds 30) string into + /// contiguous [{stage,start,end}] segments, epoch-aligned from `start`. Adjacent equal stages are merged. + public static func decode(_ phases: String, start: Date, epochSeconds: Int) -> [WearableSleepStageInterval] { + var out: [WearableSleepStageInterval] = [] + for (i, ch) in phases.enumerated() { + guard let stage = stageName(ch) else { continue } + let segStart = start.addingTimeInterval(Double(i * epochSeconds)) + let segEnd = segStart.addingTimeInterval(Double(epochSeconds)) + if var last = out.last, last.stage == stage, last.end == segStart { + last.end = segEnd + out[out.count - 1] = last + } else { + out.append(WearableSleepStageInterval(stage: stage, start: segStart, end: segEnd)) + } + } + return out + } + + /// Decode `movement_30_sec` into per-epoch motion magnitudes 1…4 (unknown digit → 0). 30-sec epochs. + public static func movement(_ s: String) -> [Int] { + s.map { ch in + switch ch { case "1": return 1; case "2": return 2; case "3": return 3; case "4": return 4 + default: return 0 } + } + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `swift test --package-path Packages/StrandImport --filter OuraHypnogramTests` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add Packages/StrandImport/Sources/StrandImport/OuraHypnogram.swift \ + Packages/StrandImport/Tests/StrandImportTests/OuraHypnogramTests.swift +git commit -m "feat(import): OuraHypnogram decoder (sleep_phase + movement, distinct legends)" +``` + +--- + +### Task 3: `OuraApiParser.parseSleep` + sleep models + +Parse detailed `sleep` periods into the shared `WearableSleepSession` (now WITH stages — the headline win) plus the rich extras (movement, in-sleep HR) the shared model can't hold, and fold the per-day sleep rollup. + +**Files:** +- Create: `Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift` +- Create: `Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift` +- Test: `Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift` + +**Interfaces:** +- Consumes: `OuraHypnogram` (Task 2); existing `WearableSleepSession`, `WearableDailyRow`, `WearableJSON` (`str`/`dbl`/`posDbl`/`posInt`), `WhoopTime.parseISOWithOffset(_:)` (`CSVParsing.swift:478`), `WearableExportImporter.dayString(_:)`. +- Produces: + - `struct OuraHRPoint { let ts: Int; let bpm: Int }` + - `struct OuraSleepPeriod { var session: WearableSleepSession; var movement30s: [Int]; var hr: [OuraHRPoint] }` + - `enum OuraApiParser` with `static func parseSleep(_ docs: [[String: Any]]) -> (periods: [OuraSleepPeriod], days: [WearableDailyRow])` and `static func sampleSeries(_ any: Any?) -> [OuraHRPoint]`. + +- [ ] **Step 1: Write the failing test** + +Create `Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift`: + +```swift +import XCTest +@testable import StrandImport + +final class OuraApiParserSleepTests: XCTestCase { + func testParsesPeriodWithHypnogramAndFoldsDay() throws { + let docs: [[String: Any]] = [[ + "id": "s1", "day": "2026-01-02", + "bedtime_start": "2026-01-01T23:00:00+00:00", + "bedtime_end": "2026-01-02T06:00:00+00:00", + "total_sleep_duration": 24000, "deep_sleep_duration": 6000, + "light_sleep_duration": 12000, "rem_sleep_duration": 6000, "awake_time": 1200, + "efficiency": 92, "average_heart_rate": 55, "lowest_heart_rate": 48, + "average_hrv": 65, "average_breath": 14.2, + "sleep_phase_5_min": "1234", + "movement_30_sec": "1212", + "heart_rate": ["interval": 300, "timestamp": "2026-01-01T23:00:00+00:00", + "items": [60, 58, NSNull(), 57]] + ]] + let (periods, days) = OuraApiParser.parseSleep(docs) + XCTAssertEqual(periods.count, 1) + let p = try XCTUnwrap(periods.first) + XCTAssertEqual(p.session.totalSleepMin, 400) // 24000s ÷ 60 + XCTAssertEqual(p.session.lowestHr, 48) + XCTAssertEqual(p.session.avgHrvMs, 65) + XCTAssertEqual(p.session.stages.map(\.stage), ["deep", "light", "rem", "wake"]) + XCTAssertEqual(p.movement30s, [1, 2, 1, 2]) + XCTAssertEqual(p.hr.map(\.bpm), [60, 58, 57]) // null sample dropped + XCTAssertEqual(days.count, 1) + XCTAssertEqual(days.first?.day, "2026-01-02") + XCTAssertEqual(days.first?.totalSleepMin, 400) + XCTAssertEqual(days.first?.restingHr, 48) // lowestHr ≈ resting fallback + } + + func testSkipsPeriodWithInvalidSpan() { + let (periods, _) = OuraApiParser.parseSleep([["bedtime_start": "nope", "bedtime_end": "nope"]]) + XCTAssertTrue(periods.isEmpty) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `swift test --package-path Packages/StrandImport --filter OuraApiParserSleepTests` +Expected: FAIL — `OuraApiParser` / `OuraSleepPeriod` undefined. + +- [ ] **Step 3: Add the sleep models** + +Create `Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift`: + +```swift +import Foundation + +// MARK: - Neutral models emitted by OuraApiParser (network-free; the app-target writer maps them to WhoopStore). + +/// One heart-rate sample (→ hrSample on write). `ts` is unix seconds, `bpm` is positive. +public struct OuraHRPoint: Sendable, Equatable { + public let ts: Int + public let bpm: Int + public init(ts: Int, bpm: Int) { self.ts = ts; self.bpm = bpm } +} + +/// One Oura detailed sleep period: the shared `session` (incl. decoded stages) plus the rich extras the +/// shared model can't hold. `movement30s` → sleepSession.motionJSON; `hr` → hrSample. (HRV samples stay +/// in the raw archive only — there is no rMSSD-per-sample table; the per-night average is on `session`.) +public struct OuraSleepPeriod: Sendable, Equatable { + public var session: WearableSleepSession + public var movement30s: [Int] + public var hr: [OuraHRPoint] + public init(session: WearableSleepSession, movement30s: [Int], hr: [OuraHRPoint]) { + self.session = session; self.movement30s = movement30s; self.hr = hr + } +} +``` + +- [ ] **Step 4: Implement `parseSleep`** + +Create `Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift`: + +```swift +import Foundation + +// MARK: - Oura API v2 document parser (PURE / network-free) +// +// Takes already-fetched JSON `data[]` arrays (the OuraAPIClient does the I/O in the app target) and maps +// them to NOOP's normalized models. Reuses the SAME field semantics as OuraExportParser — the account export +// and the API share field names (bedtime_start, total_sleep_duration, …) — and adds the hypnogram + +// time-series the file export never carried. Crafted-input safe via WearableJSON. + +public enum OuraApiParser { + + /// Parse `sleep` documents (detailed periods; multiple per day are possible). Returns one + /// `OuraSleepPeriod` per usable period plus a per-day sleep rollup folded onto `WearableDailyRow`. + public static func parseSleep(_ docs: [[String: Any]]) -> (periods: [OuraSleepPeriod], days: [WearableDailyRow]) { + var periods: [OuraSleepPeriod] = [] + var byDay: [String: WearableDailyRow] = [:] + + for s in docs { + guard let start = WhoopTime.parseISOWithOffset(WearableJSON.str(s, "bedtime_start")), + let end = WhoopTime.parseISOWithOffset(WearableJSON.str(s, "bedtime_end")), + end > start else { continue } + + // Durations are SECONDS in the API → minutes. + func minutes(_ k: String) -> Double? { WearableJSON.posDbl(s, k).map { $0 / 60.0 } } + + // Stages: prefer the finer 30-sec hypnogram, else the 5-min; never synthesize one. + let stages: [WearableSleepStageInterval] + if let p30 = WearableJSON.str(s, "sleep_phase_30_sec"), !p30.isEmpty { + stages = OuraHypnogram.decode(p30, start: start, epochSeconds: 30) + } else if let p5 = WearableJSON.str(s, "sleep_phase_5_min"), !p5.isEmpty { + stages = OuraHypnogram.decode(p5, start: start, epochSeconds: 300) + } else { + stages = [] + } + + let session = WearableSleepSession( + start: start, end: end, + deepMin: minutes("deep_sleep_duration"), + lightMin: minutes("light_sleep_duration"), + remMin: minutes("rem_sleep_duration"), + awakeMin: minutes("awake_time"), + totalSleepMin: minutes("total_sleep_duration"), + efficiencyPct: WearableJSON.posDbl(s, "efficiency"), + avgHr: WearableJSON.posInt(s, "average_heart_rate"), + lowestHr: WearableJSON.posInt(s, "lowest_heart_rate"), + avgHrvMs: WearableJSON.posDbl(s, "average_hrv"), + respRateBpm: WearableJSON.posDbl(s, "average_breath"), + sleepScore: nil, // the night's reference score comes from daily_sleep + stages: stages) + + let movement = WearableJSON.str(s, "movement_30_sec").map(OuraHypnogram.movement) ?? [] + let hr = sampleSeries(s["heart_rate"]) + periods.append(OuraSleepPeriod(session: session, movement30s: movement, hr: hr)) + + // Fold the night onto its calendar day (Oura's "day" = the wake day). + let key = WearableJSON.str(s, "day") ?? WearableExportImporter.dayString(end) + var row = byDay[key] ?? WearableDailyRow(day: key) + row.totalSleepMin = row.totalSleepMin ?? session.totalSleepMin + row.deepMin = row.deepMin ?? session.deepMin + row.lightMin = row.lightMin ?? session.lightMin + row.remMin = row.remMin ?? session.remMin + row.awakeMin = row.awakeMin ?? session.awakeMin + row.efficiencyPct = row.efficiencyPct ?? session.efficiencyPct + row.avgHrvMs = row.avgHrvMs ?? session.avgHrvMs + if row.restingHr == nil { row.restingHr = session.lowestHr } + byDay[key] = row + } + return (periods, Array(byDay.values)) + } + + /// Decode an Oura `PublicSample` object ({interval, items:[Double?], timestamp}) into HR points. + /// Non-finite / non-positive items are dropped (Oura uses null/0 for gaps). + static func sampleSeries(_ any: Any?) -> [OuraHRPoint] { + guard let obj = any as? [String: Any], + let interval = WearableJSON.dbl(obj, "interval"), interval > 0, + let items = obj["items"] as? [Any], + let t0 = WhoopTime.parseISOWithOffset(WearableJSON.str(obj, "timestamp")) else { return [] } + var out: [OuraHRPoint] = [] + for (i, item) in items.enumerated() { + guard let v = (item as? Double) ?? (item as? NSNumber)?.doubleValue, v.isFinite, v > 0 else { continue } + out.append(OuraHRPoint(ts: Int(t0.timeIntervalSince1970 + Double(i) * interval), bpm: Int(v))) + } + return out + } +} +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `swift test --package-path Packages/StrandImport --filter OuraApiParserSleepTests` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift \ + Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift \ + Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift +git commit -m "feat(import): OuraApiParser.parseSleep — sessions with hypnogram, movement, in-sleep HR" +``` + +--- + +### Task 4: `OuraApiParser.parseDaily` (summary endpoints) + extras + +One dispatching parser for the day-keyed summary endpoints. Maps onto `WearableDailyRow` columns where a home exists; everything else becomes `OuraDailyExtra` points (→ `metricSeries`). Oura's own scores land ONLY under `ref_*`/`oura_*` keys (honest data). + +**Files:** +- Modify: `Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift` (add `OuraDailyExtra`) +- Create: `Packages/StrandImport/Sources/StrandImport/OuraApiParser+Daily.swift` +- Test: `Packages/StrandImport/Tests/StrandImportTests/OuraApiParserDailyTests.swift` + +**Interfaces:** +- Consumes: existing `WearableDailyRow`, `WearableJSON`. +- Produces: + - `struct OuraDailyExtra { let day: String; let key: String; let value: Double }` + - `OuraApiParser.parseDaily(_ docs: [[String: Any]], endpoint: String) -> (days: [WearableDailyRow], extras: [OuraDailyExtra])` + - `OuraApiParser.resilienceLevel(_ s: String?) -> Double?` + +- [ ] **Step 1: Write the failing test** + +Create `Packages/StrandImport/Tests/StrandImportTests/OuraApiParserDailyTests.swift`: + +```swift +import XCTest +@testable import StrandImport + +final class OuraApiParserDailyTests: XCTestCase { + func testReadinessMapsRhrSkinTempAndRefScore() { + let (days, extras) = OuraApiParser.parseDaily([[ + "day": "2026-01-02", "score": 80, "temperature_deviation": -0.2, + "contributors": ["resting_heart_rate": 50, "hrv_balance": 70] + ]], endpoint: "daily_readiness") + XCTAssertEqual(days.first?.restingHr, 50) + XCTAssertEqual(days.first?.skinTempDevC, -0.2) + XCTAssertEqual(days.first?.readinessScore, 80) + XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "ref_readiness_score", value: 80))) + XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "oura_readiness_hrv_balance", value: 70))) + } + + func testActivityMapsStepsAndScoreIsReferenceOnly() { + let (days, extras) = OuraApiParser.parseDaily([[ + "day": "2026-01-02", "score": 88, "steps": 9000, + "active_calories": 500, "total_calories": 2400 + ]], endpoint: "daily_activity") + XCTAssertEqual(days.first?.steps, 9000) + XCTAssertEqual(days.first?.activeKcal, 500) + // The activity SCORE is reference-only — never a dailyMetric score column. + XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "ref_activity_score", value: 88))) + } + + func testResilienceLevelEncodedReferenceOnly() { + let (_, extras) = OuraApiParser.parseDaily([[ + "day": "2026-01-02", "level": "solid", "contributors": ["sleep_recovery": 60.0] + ]], endpoint: "daily_resilience") + XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "ref_resilience_level", value: 3))) + XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "oura_resilience_sleep_recovery", value: 60))) + } + + func testSpo2MapsAverage() { + let (days, _) = OuraApiParser.parseDaily([[ + "day": "2026-01-02", "spo2_percentage": ["average": 96.5] + ]], endpoint: "daily_spo2") + XCTAssertEqual(days.first?.spo2Pct, 96.5) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `swift test --package-path Packages/StrandImport --filter OuraApiParserDailyTests` +Expected: FAIL — `parseDaily` / `OuraDailyExtra` undefined. + +- [ ] **Step 3: Add the `OuraDailyExtra` model** + +Append to `Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift`: + +```swift +/// One extra daily scalar Oura returns that the wide `dailyMetric` columns don't hold (→ metricSeries on +/// write). `key` is the metricSeries key; the brand's OWN scores use a `ref_` prefix and its contributor +/// breakdowns an `oura_` prefix, so they are browseable but never mistaken for a NOOP score. +public struct OuraDailyExtra: Sendable, Equatable { + public let day: String + public let key: String + public let value: Double + public init(day: String, key: String, value: Double) { self.day = day; self.key = key; self.value = value } +} +``` + +- [ ] **Step 4: Implement `parseDaily`** + +Create `Packages/StrandImport/Sources/StrandImport/OuraApiParser+Daily.swift`: + +```swift +import Foundation + +public extension OuraApiParser { + + /// Parse a day-keyed summary endpoint's documents. `endpoint` selects the field mapping. Writes to + /// `WearableDailyRow` columns only where the on-device schema already has a home (RHR, skin-temp dev, + /// steps, calories, SpO2); everything else — and ALL of Oura's own scores — becomes `OuraDailyExtra`. + static func parseDaily(_ docs: [[String: Any]], endpoint: String) -> (days: [WearableDailyRow], extras: [OuraDailyExtra]) { + var byDay: [String: WearableDailyRow] = [:] + var extras: [OuraDailyExtra] = [] + func extra(_ day: String, _ key: String, _ v: Double?) { + if let v { extras.append(OuraDailyExtra(day: day, key: key, value: v)) } + } + + for doc in docs { + guard let day = WearableJSON.str(doc, "day") else { continue } + switch endpoint { + + case "daily_readiness": + var r = byDay[day] ?? WearableDailyRow(day: day) + if let c = doc["contributors"] as? [String: Any] { + r.restingHr = WearableJSON.posInt(c, "resting_heart_rate") ?? r.restingHr + for (k, _) in c { extra(day, "oura_readiness_\(k)", WearableJSON.posDbl(c, k)) } + } + r.skinTempDevC = WearableJSON.dbl(doc, "temperature_deviation") ?? r.skinTempDevC + r.readinessScore = WearableJSON.posInt(doc, "score") ?? r.readinessScore + extra(day, "ref_readiness_score", WearableJSON.posDbl(doc, "score")) + byDay[day] = r + + case "daily_sleep": + extra(day, "ref_sleep_score", WearableJSON.posDbl(doc, "score")) + if let c = doc["contributors"] as? [String: Any] { + for (k, _) in c { extra(day, "oura_sleep_\(k)", WearableJSON.posDbl(c, k)) } + } + + case "daily_activity": + var r = byDay[day] ?? WearableDailyRow(day: day) + r.steps = WearableJSON.posInt(doc, "steps") ?? r.steps + r.activeKcal = WearableJSON.posDbl(doc, "active_calories") ?? r.activeKcal + r.totalKcal = WearableJSON.posDbl(doc, "total_calories") ?? r.totalKcal + extra(day, "ref_activity_score", WearableJSON.posDbl(doc, "score")) + extra(day, "oura_equiv_walk_m", WearableJSON.posDbl(doc, "equivalent_walking_distance")) + byDay[day] = r + + case "daily_spo2": + var r = byDay[day] ?? WearableDailyRow(day: day) + if let s = doc["spo2_percentage"] as? [String: Any] { + r.spo2Pct = WearableJSON.posDbl(s, "average") ?? r.spo2Pct + extra(day, "spo2", WearableJSON.posDbl(s, "average")) + } + extra(day, "oura_breathing_disturbance_index", WearableJSON.dbl(doc, "breathing_disturbance_index")) + byDay[day] = r + + case "daily_stress": + extra(day, "oura_stress_high_s", WearableJSON.posDbl(doc, "stress_high")) + extra(day, "oura_recovery_high_s", WearableJSON.posDbl(doc, "recovery_high")) + + case "daily_resilience": + if let c = doc["contributors"] as? [String: Any] { + for (k, _) in c { extra(day, "oura_resilience_\(k)", WearableJSON.dbl(c, k)) } + } + extra(day, "ref_resilience_level", resilienceLevel(WearableJSON.str(doc, "level"))) + + case "daily_cardiovascular_age": + extra(day, "oura_vascular_age", WearableJSON.dbl(doc, "vascular_age")) + extra(day, "oura_pulse_wave_velocity", WearableJSON.dbl(doc, "pulse_wave_velocity")) + + case "vO2_max": + extra(day, "vo2max", WearableJSON.posDbl(doc, "vo2_max")) + + default: + break + } + } + return (Array(byDay.values), extras) + } + + /// Oura resilience level → an ordered reference number (1…5). Reference-only — never a NOOP score. + static func resilienceLevel(_ s: String?) -> Double? { + switch s { + case "limited": return 1 + case "adequate": return 2 + case "solid": return 3 + case "strong": return 4 + case "exceptional": return 5 + default: return nil + } + } +} +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `swift test --package-path Packages/StrandImport --filter OuraApiParserDailyTests` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift \ + Packages/StrandImport/Sources/StrandImport/OuraApiParser+Daily.swift \ + Packages/StrandImport/Tests/StrandImportTests/OuraApiParserDailyTests.swift +git commit -m "feat(import): OuraApiParser.parseDaily — summary endpoints, scores kept reference-only" +``` + +--- + +### Task 5: `OuraApiParser` events — workouts + heart-rate time series + +Parse `workout` documents (→ workout table later) and the `/heartrate` time-series page (→ hrSample later). + +**Files:** +- Modify: `Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift` (add `OuraWorkout`) +- Create: `Packages/StrandImport/Sources/StrandImport/OuraApiParser+Events.swift` +- Test: `Packages/StrandImport/Tests/StrandImportTests/OuraApiParserEventsTests.swift` + +**Interfaces:** +- Consumes: existing `WearableJSON`, `WhoopTime.parseISOWithOffset(_:)`; `OuraHRPoint` (Task 3). +- Produces: + - `struct OuraWorkout { let start: Date; let end: Date; let activity: String; let source: String; let energyKcal: Double?; let distanceM: Double? }` + - `OuraApiParser.parseWorkouts(_ docs: [[String: Any]]) -> [OuraWorkout]` + - `OuraApiParser.parseHeartRate(_ docs: [[String: Any]]) -> [OuraHRPoint]` + +- [ ] **Step 1: Write the failing test** + +Create `Packages/StrandImport/Tests/StrandImportTests/OuraApiParserEventsTests.swift`: + +```swift +import XCTest +@testable import StrandImport + +final class OuraApiParserEventsTests: XCTestCase { + func testWorkoutParse() throws { + let w = OuraApiParser.parseWorkouts([[ + "activity": "running", "source": "autodetected", + "start_datetime": "2026-01-02T07:00:00+00:00", "end_datetime": "2026-01-02T07:40:00+00:00", + "calories": 410, "distance": 6200 + ]]) + XCTAssertEqual(w.count, 1) + XCTAssertEqual(w.first?.activity, "running") + XCTAssertEqual(w.first?.source, "autodetected") + XCTAssertEqual(w.first?.energyKcal, 410) + XCTAssertEqual(w.first?.distanceM, 6200) + } + + func testHeartRatePageParseDropsNonPositive() { + let hr = OuraApiParser.parseHeartRate([ + ["timestamp": "2026-01-02T07:00:00+00:00", "bpm": 120, "source": "workout"], + ["timestamp": "2026-01-02T07:05:00+00:00", "bpm": 0, "source": "workout"] // dropped + ]) + XCTAssertEqual(hr.map(\.bpm), [120]) + XCTAssertEqual(hr.first?.ts, Int(ISO8601DateFormatter().date(from: "2026-01-02T07:00:00Z")!.timeIntervalSince1970)) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `swift test --package-path Packages/StrandImport --filter OuraApiParserEventsTests` +Expected: FAIL — `parseWorkouts` / `OuraWorkout` undefined. + +- [ ] **Step 3: Add the `OuraWorkout` model** + +Append to `Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift`: + +```swift +/// One Oura workout (→ the `workout` table on write). Times are UTC; metric fields nil when absent. +public struct OuraWorkout: Sendable, Equatable { + public let start: Date + public let end: Date + public let activity: String + public let source: String + public let energyKcal: Double? + public let distanceM: Double? + public init(start: Date, end: Date, activity: String, source: String, energyKcal: Double?, distanceM: Double?) { + self.start = start; self.end = end; self.activity = activity; self.source = source + self.energyKcal = energyKcal; self.distanceM = distanceM + } +} +``` + +- [ ] **Step 4: Implement the event parsers** + +Create `Packages/StrandImport/Sources/StrandImport/OuraApiParser+Events.swift`: + +```swift +import Foundation + +public extension OuraApiParser { + + /// Parse `workout` documents → `OuraWorkout`. Rows without a valid time span are skipped. + static func parseWorkouts(_ docs: [[String: Any]]) -> [OuraWorkout] { + var out: [OuraWorkout] = [] + for w in docs { + guard let start = WhoopTime.parseISOWithOffset(WearableJSON.str(w, "start_datetime")), + let end = WhoopTime.parseISOWithOffset(WearableJSON.str(w, "end_datetime")), + end > start else { continue } + out.append(OuraWorkout( + start: start, end: end, + activity: WearableJSON.str(w, "activity") ?? "workout", + source: WearableJSON.str(w, "source") ?? "oura", + energyKcal: WearableJSON.posDbl(w, "calories"), + distanceM: WearableJSON.posDbl(w, "distance"))) + } + return out + } + + /// Parse a `/v2/usercollection/heartrate` page's `data[]` → HR points. Each row is + /// {timestamp, bpm, source}; non-positive bpm is dropped. (The `source` enum is preserved in the raw + /// archive, not in the compact hrSample.) + static func parseHeartRate(_ docs: [[String: Any]]) -> [OuraHRPoint] { + var out: [OuraHRPoint] = [] + for h in docs { + guard let ts = WhoopTime.parseISOWithOffset(WearableJSON.str(h, "timestamp")), + let bpm = WearableJSON.posInt(h, "bpm") else { continue } + out.append(OuraHRPoint(ts: Int(ts.timeIntervalSince1970), bpm: bpm)) + } + return out + } +} +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `swift test --package-path Packages/StrandImport --filter OuraApiParserEventsTests` +Expected: PASS. + +- [ ] **Step 6: Run the full package test suites (no regressions)** + +Run: `swift test --package-path Packages/StrandImport && swift test --package-path Packages/WhoopStore` +Expected: PASS (all existing tests + the 5 new suites). + +- [ ] **Step 7: Commit** + +```bash +git add Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift \ + Packages/StrandImport/Sources/StrandImport/OuraApiParser+Events.swift \ + Packages/StrandImport/Tests/StrandImportTests/OuraApiParserEventsTests.swift +git commit -m "feat(import): OuraApiParser workouts + heartrate time-series parsing" +``` + +--- + +## Plan 1 self-review + +- **Spec coverage:** §8.5 raw store → T1. §8.2 hypnogram/movement/in-sleep HR → T2/T3. `sleep` mapping → T3. Daily summaries + reference-only scores (§8.1, G4) → T4. Workouts + heartrate (§8.1, §8.3) → T5. Sessions/tags/rest_mode/personal_info/ring_* normalization is intentionally deferred to Plan 3's writer (raw-archived here via T1's store regardless). No spec requirement for the *foundation* layer is unaddressed. +- **Network-free:** every new file imports only `Foundation`/`GRDB`. ✓ (Constraint.) +- **Honest data:** every Oura score is emitted under `ref_*`/`oura_*` only; no path writes `recovery`/`strain`. ✓ (Asserted in T4.) +- **Type consistency:** `OuraHRPoint` defined in T3, reused verbatim in T5. `OuraApiParser` opened in T3, extended in T4/T5. `OuraDailyExtra`/`OuraWorkout` appended to the same `OuraApiModels.swift`. No dangling references. +- **No placeholders:** every step has full code + exact run command + expected result. + +## What Plans 2 & 3 add (scope preview — detailed separately on request) + +- **Plan 2 — Network + auth (app target `Strand/Oura/`):** `OuraCredentials` (xcconfig→Info.plist), `OuraTokenStore` (Keychain, mirroring `AIKeyStore`), `AuthProvider` + `OuraOAuthProvider` (`ASWebAuthenticationSession` + code→token exchange, refresh), `OuraAPIClient` (`URLSession`, `next_token` paging, 429/401 handling, prod/sandbox base URL) — tested with a `URLProtocol` stub and the Oura sandbox tree. +- **Plan 3 — Orchestration + UI + wiring:** `OuraSyncWriter` (parsed models → `upsertDailyMetrics`/`upsertSleepSessions`(+stagesJSON/motionJSON)/`upsertMetricSeries`/`insert(hrSample)`/`upsertWorkouts` + `upsertOuraRaw` + `DeviceRegistryStore.add`, no day-ownership seizure), `OuraSyncCoordinator` (one-time backfill across all endpoints), `DataSourceKind.ouraApi` + exhaustive-switch updates, the Data Sources "Connect Oura & Import Everything" card/progress/summary/disconnect, `project.yml`/Info.plist/.gitignore wiring, and doc updates (`PRIVACY_SECURITY.md` second opt-in network lane, `DEVICE_SUPPORT_ROADMAP.md`, `DATA_MODEL.md`). From 0efa46a5c5e41a94a66813c8dd7db23a549f2282 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 14:54:09 -0700 Subject: [PATCH 003/171] docs: reconcile Oura spec + plan-1 to v8.5.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rebased target is now ryanbr/noop v8.5.2 (NoopApp/noop upstream went private). Study of v8.5.2 vs the v7.2.3-era design, verified against code: - ouraRaw migration is v24 (migrator is at v23, not v18); v19 already taken. - Provenance uses existing PairedDevice.sourceKind .cloudImport (non-day-owning via IntelligenceEngine:1369) — no new SourceKind; deviceId stays 'oura-api'. - motionJSON goes through persistSessionMotion(), not the sleep upsert. - Add oura-api to Repository.wearableImportSources; disconnect via store actor. - Citation shifts (parseISOWithOffset -> CSVParsing:509). Foundation design (models, 5 upsert APIs, tables, parsers) confirmed intact. --- ...06-27-oura-live-api-import-1-foundation.md | 27 ++++++++++++------- .../2026-06-27-oura-live-api-import-design.md | 13 +++++++++ 2 files changed, 30 insertions(+), 10 deletions(-) diff --git a/docs/superpowers/plans/2026-06-27-oura-live-api-import-1-foundation.md b/docs/superpowers/plans/2026-06-27-oura-live-api-import-1-foundation.md index e3504bfac8..05c0b5179b 100644 --- a/docs/superpowers/plans/2026-06-27-oura-live-api-import-1-foundation.md +++ b/docs/superpowers/plans/2026-06-27-oura-live-api-import-1-foundation.md @@ -2,9 +2,16 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. +> **⚠️ v8.5.2 reconciliation (2026-06-30).** Written against v7.2.3; this branch (`oura-cloud-import`) is rebased onto upstream **v8.5.2** (`ryanbr/noop`). Deltas verified against current code: +> 1. **Migration is `v24`, not `v19`.** The migrator is at **v23** (`v19-step-activity-class` … `v23-daily-spo2-raw`); append `v24-oura-raw` after the v23 block. (Fixed inline in Task 1.) +> 2. **Provenance (Plan 3):** register the source with `PairedDevice.sourceKind = .cloudImport` — it already exists (`PairedDevice.swift:42`) and is treated as non-day-owning (`IntelligenceEngine.swift:1369`). Do **not** invent a new SourceKind; keep `deviceId = "oura-api"`. +> 3. **`motionJSON` (Plan 3)** is written via the dedicated `persistSessionMotion(deviceId:sessionStart:motionEpochs:[Double])` (`MetricsCache.swift:234`) after the sleep upsert (not through it); map `movement30s` `[Int]→[Double]`. +> 4. Also add `"oura-api"` to `Repository.wearableImportSources` (`Repository.swift:452`); disconnect via the actor `store.deleteAllData(deviceId:)` (`DataSourcesView.swift:598`). +> 5. Citations shifted: `WhoopTime.parseISOWithOffset` → `CSVParsing.swift:509`; `WearableJSON` is in `WearableExportImporter.swift:329`. **Tasks 2–5 compile against v8.5.2 unchanged** — only Task 1's migration number moves. + **Goal:** Build the network-free core of the Oura live-API lane — the lossless raw payload store and the pure JSON→model parsers (incl. the sleep hypnogram) — fully unit-tested, touching only new files plus one additive migration. -**Architecture:** Two packages, additive only. `WhoopStore` gains a `ouraRaw` table (migration **v19**) and an `OuraRawStore` extension mirroring `MetricSeriesStore`. `StrandImport` gains pure decoders (`OuraHypnogram`, `OuraApiParser`) and small neutral model structs that reuse the existing `OuraExportParser` field semantics, `WhoopTime`, and `WearableJSON`. No networking, no edits to existing exhaustive switches, no app-target changes — so this plan can't break any existing build path. The fetch layer, auth, writer, and UI come in Plans 2–3. +**Architecture:** Two packages, additive only. `WhoopStore` gains a `ouraRaw` table (migration **v24**) and an `OuraRawStore` extension mirroring `MetricSeriesStore`. `StrandImport` gains pure decoders (`OuraHypnogram`, `OuraApiParser`) and small neutral model structs that reuse the existing `OuraExportParser` field semantics, `WhoopTime`, and `WearableJSON`. No networking, no edits to existing exhaustive switches, no app-target changes — so this plan can't break any existing build path. The fetch layer, auth, writer, and UI come in Plans 2–3. **Tech Stack:** Swift 5.9, GRDB 6.29.3 (pinned), XCTest, `Foundation`. iOS 16 / macOS 13 (both package platforms; the lane ships iOS-only but the packages stay multi-platform). @@ -14,18 +21,18 @@ Every task implicitly includes these (verbatim from the spec): - **Packages stay network-free.** No `URLSession`/`URLRequest`/`NWConnection` in `StrandImport` or `WhoopStore`. (`StrandImport` banner: "STAY OFFLINE: nothing here touches the network.") - **Honest data.** Oura's own scores (readiness/sleep/activity/resilience) are written ONLY under reference keys (`ref_*`) or namespaced (`oura_*`) — never as a NOOP score column (`recovery`/`strain`). NOOP recomputes downstream. - **Untrusted input is hostile.** Parse via the existing `WearableJSON` finite/range-checked coercion (`dbl`/`int`/`posInt`/`posDbl`/`str`); never trap on attacker NaN/inf/huge values. -- **Additive migrations only.** New tables/indexes; never alter or drop existing rows. The live migrator is at **v18** (`Packages/WhoopStore/Sources/WhoopStore/Database.swift`); this adds **v19**. +- **Additive migrations only.** New tables/indexes; never alter or drop existing rows. The live migrator is at **v23** (`Packages/WhoopStore/Sources/WhoopStore/Database.swift`); this adds **v24** (`v19`–`v23` already exist). - **Pinned deps.** Add no new dependency. Versions across `Packages/*/Package.swift` must stay identical (`GRDB.swift exact 6.29.3`, `ZIPFoundation exact 0.9.20`). - **Provenance string.** The live-API partition is `deviceId = "oura-api"` (distinct from the file lane's `"oura-import"`). --- -### Task 1: `ouraRaw` table (migration v19) + `OuraRawStore` +### Task 1: `ouraRaw` table (migration v24) + `OuraRawStore` Lossless archive of every raw Oura payload, keyed by `(deviceId, endpoint, documentId)`, idempotent on re-pull. **Files:** -- Modify: `Packages/WhoopStore/Sources/WhoopStore/Database.swift` (add migration after the `v18-sleep-motion-state` block, before `return migrator`) +- Modify: `Packages/WhoopStore/Sources/WhoopStore/Database.swift` (add migration after the `v23-daily-spo2-raw` block, before `return migrator`) - Create: `Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift` - Test: `Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift` @@ -75,15 +82,15 @@ final class OuraRawStoreTests: XCTestCase { Run: `swift test --package-path Packages/WhoopStore --filter OuraRawStoreTests` Expected: FAIL — compile error, `OuraRawRow` / `upsertOuraRaw` undefined. -- [ ] **Step 3: Add migration v19** +- [ ] **Step 3: Add migration v24** -In `Packages/WhoopStore/Sources/WhoopStore/Database.swift`, insert immediately before `return migrator`: +In `Packages/WhoopStore/Sources/WhoopStore/Database.swift`, insert immediately **after the `v23-daily-spo2-raw` block** and before `return migrator` (`v19`–`v23` already exist, so v24 is the next free slot — do NOT reuse v19): ```swift - // v19: Oura live-API raw payload archive (lossless). One row per (deviceId, endpoint, documentId); + // v24: Oura live-API raw payload archive (lossless). One row per (deviceId, endpoint, documentId); // payloadJSON holds the verbatim Oura object so any field can be re-derived later without re-fetching // from the API. Additive only — a NEW table, no existing row touched, old readers unaffected. - migrator.registerMigration("v19-oura-raw") { db in + migrator.registerMigration("v24-oura-raw") { db in try db.create(table: "ouraRaw") { t in t.column("deviceId", .text).notNull() t.column("endpoint", .text).notNull() // "sleep" | "daily_readiness" | "heartrate" | … @@ -184,7 +191,7 @@ Expected: PASS. git add Packages/WhoopStore/Sources/WhoopStore/Database.swift \ Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift \ Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift -git commit -m "feat(store): ouraRaw lossless payload archive (migration v19) + OuraRawStore" +git commit -m "feat(store): ouraRaw lossless payload archive (migration v24) + OuraRawStore" ``` --- @@ -326,7 +333,7 @@ Parse detailed `sleep` periods into the shared `WearableSleepSession` (now WITH - Test: `Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift` **Interfaces:** -- Consumes: `OuraHypnogram` (Task 2); existing `WearableSleepSession`, `WearableDailyRow`, `WearableJSON` (`str`/`dbl`/`posDbl`/`posInt`), `WhoopTime.parseISOWithOffset(_:)` (`CSVParsing.swift:478`), `WearableExportImporter.dayString(_:)`. +- Consumes: `OuraHypnogram` (Task 2); existing `WearableSleepSession`, `WearableDailyRow`, `WearableJSON` (`str`/`dbl`/`posDbl`/`posInt`, in `WearableExportImporter.swift:329`), `WhoopTime.parseISOWithOffset(_:)` (`CSVParsing.swift:509`), `WearableExportImporter.dayString(_:)`. - Produces: - `struct OuraHRPoint { let ts: Int; let bpm: Int }` - `struct OuraSleepPeriod { var session: WearableSleepSession; var movement30s: [Int]; var hr: [OuraHRPoint] }` diff --git a/docs/superpowers/specs/2026-06-27-oura-live-api-import-design.md b/docs/superpowers/specs/2026-06-27-oura-live-api-import-design.md index d805c23cfd..0f3d4b557b 100644 --- a/docs/superpowers/specs/2026-06-27-oura-live-api-import-design.md +++ b/docs/superpowers/specs/2026-06-27-oura-live-api-import-design.md @@ -8,6 +8,19 @@ --- +## 0. v8.5.2 reconciliation (2026-06-30) + +This spec was written against v7.2.3; work now targets upstream **v8.5.2** (`ryanbr/noop`, on branch `oura-cloud-import`). Corrections below are verified against current code and **supersede the inline references** further down: + +- **Migration number.** The WhoopStore migrator is at **v23** (not v18); the `ouraRaw` table (§4/§8.5/§14) is migration **v24** (not v19 — `v19-step-activity-class`…`v23-daily-spo2-raw` already exist). +- **Provenance (§8.4).** Register the cloud source as `PairedDevice.sourceKind = .cloudImport` — it already exists (`PairedDevice.swift:42`) and is treated as non-day-owning (`IntelligenceEngine.swift:1369`), so **no new SourceKind is needed**. `deviceId = "oura-api"` stands; `DataSourceKind.ouraApi` is now *optional* (add only if a StrandImport-level provenance tag is actually consumed). +- **`motionJSON` (§8.2).** Written via the dedicated `persistSessionMotion(deviceId:sessionStart:motionEpochs:[Double])` (`MetricsCache.swift:234`), **not** through `upsertSleepSessions` (`CachedSleepSession` has no `motionJSON` field). Map `movement30s` `[Int]→[Double]`. +- **New integration points.** Add `"oura-api"` to `Repository.wearableImportSources` (`Repository.swift:452`) if it should join the import-source union; disconnect via the actor `store.deleteAllData(deviceId:)` (`DataSourcesView.swift:598`), not `DeviceRegistryStore` directly (avoids a main-actor stall). +- **UI (§9).** `DataSourcesView` is entirely file-picker cards today; the live "connect/adopt" precedent is `AddDeviceWizard.swift`. Decide explicitly whether cloud Oura is an *import card* (mirrors the export card) or a *connect-device* flow — the OAuth "Connect" button is a new pattern here. +- **Confirmed intact (no change):** the `Wearable*` models + initializers, all five upsert APIs and their row structs, every target table shape, and the network-free-packages invariant. The `#862` export-parser hardening **confirmed** our field mapping (incl. the nested `spo2_percentage.average`). The foundation design holds. + +--- + ## 1. Summary NOOP already imports a user's own Oura data from the **account export file** (a single JSON), parsed by From f06688aa8e61e59f003cc97ef707938b32c87c09 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 15:39:00 -0700 Subject: [PATCH 004/171] feat(store): ouraRaw lossless payload archive (migration v24) + OuraRawStore --- .../Sources/WhoopStore/Database.swift | 17 +++++ .../Sources/WhoopStore/OuraRawStore.swift | 67 +++++++++++++++++++ .../WhoopStoreTests/OuraRawStoreTests.swift | 28 ++++++++ 3 files changed, 112 insertions(+) create mode 100644 Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift create mode 100644 Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift diff --git a/Packages/WhoopStore/Sources/WhoopStore/Database.swift b/Packages/WhoopStore/Sources/WhoopStore/Database.swift index e836a5101b..e850e36f8c 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/Database.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/Database.swift @@ -456,6 +456,23 @@ extension WhoopStore { t.add(column: "spo2Ir", .integer) } } + // v24: Oura live-API raw payload archive (lossless). One row per (deviceId, endpoint, documentId); + // payloadJSON holds the verbatim Oura object so any field can be re-derived later without re-fetching + // from the API. Additive only — a NEW table, no existing row touched, old readers unaffected. + migrator.registerMigration("v24-oura-raw") { db in + try db.create(table: "ouraRaw") { t in + t.column("deviceId", .text).notNull() + t.column("endpoint", .text).notNull() // "sleep" | "daily_readiness" | "heartrate" | … + t.column("documentId", .text).notNull() // Oura `id`; heartrate pages use a window key + t.column("day", .text) // YYYY-MM-DD when day-keyed (nullable) + t.column("payloadJSON", .text).notNull() // verbatim object + t.column("fetchedAt", .integer).notNull() // unix seconds + t.primaryKey(["deviceId", "endpoint", "documentId"]) + } + // Per-endpoint reads scan (deviceId, endpoint) then walk day in order. + try db.create(index: "idx_ouraRaw_device_endpoint_day", + on: "ouraRaw", columns: ["deviceId", "endpoint", "day"]) + } return migrator } } diff --git a/Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift b/Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift new file mode 100644 index 0000000000..1f4fc71301 --- /dev/null +++ b/Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift @@ -0,0 +1,67 @@ +import Foundation +import GRDB + +// MARK: - v19 store: lossless Oura live-API payload archive +// Mirrors MetricSeriesStore exactly: a Codable row struct, an idempotent ON CONFLICT upsert keyed by the +// natural key (deviceId, endpoint, documentId), and range reads — all GRDB work via syncWrite/syncRead. +// This is the "as much as we can get" backstop: the verbatim payload is kept so a future metric can be +// derived without re-hitting the Oura API. + +/// One archived Oura API payload. Natural key (deviceId, endpoint, documentId). +public struct OuraRawRow: Equatable, Codable, Sendable { + public let endpoint: String // "sleep" | "daily_readiness" | "heartrate" | … + public let documentId: String // Oura `id`; for heartrate pages, a synthesized window key + public let day: String? // YYYY-MM-DD when the document is day-keyed + public let payloadJSON: String // verbatim object + public let fetchedAt: Int // unix seconds + public init(endpoint: String, documentId: String, day: String?, payloadJSON: String, fetchedAt: Int) { + self.endpoint = endpoint; self.documentId = documentId + self.day = day; self.payloadJSON = payloadJSON; self.fetchedAt = fetchedAt + } +} + +extension WhoopStore { + + /// Upsert raw Oura payloads. Idempotent by (deviceId, endpoint, documentId): re-pulling the same + /// document overwrites its payload/day/fetchedAt rather than duplicating. Returns rows changed. + @discardableResult + public func upsertOuraRaw(_ rows: [OuraRawRow], deviceId: String) async throws -> Int { + try syncWrite { db in + var n = 0 + for r in rows { + try db.execute(sql: """ + INSERT INTO ouraRaw (deviceId, endpoint, documentId, day, payloadJSON, fetchedAt) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(deviceId, endpoint, documentId) DO UPDATE SET + day = excluded.day, + payloadJSON = excluded.payloadJSON, + fetchedAt = excluded.fetchedAt + """, arguments: [deviceId, r.endpoint, r.documentId, r.day, r.payloadJSON, r.fetchedAt]) + n += db.changesCount + } + return n + } + } + + /// Archived payloads for a device + endpoint, oldest day first (null days sort first in SQLite). + public func ouraRaw(deviceId: String, endpoint: String) async throws -> [OuraRawRow] { + try syncRead { db in + try Row.fetchAll(db, sql: """ + SELECT endpoint, documentId, day, payloadJSON, fetchedAt FROM ouraRaw + WHERE deviceId = ? AND endpoint = ? + ORDER BY day ASC + """, arguments: [deviceId, endpoint]) + .map { OuraRawRow(endpoint: $0["endpoint"], documentId: $0["documentId"], + day: $0["day"], payloadJSON: $0["payloadJSON"], fetchedAt: $0["fetchedAt"]) } + } + } + + /// Count archived payloads for a device + endpoint (diagnostics / import summary). + public func ouraRawCount(deviceId: String, endpoint: String) async throws -> Int { + try syncRead { db in + try Int.fetchOne(db, sql: + "SELECT COUNT(*) FROM ouraRaw WHERE deviceId = ? AND endpoint = ?", + arguments: [deviceId, endpoint]) ?? 0 + } + } +} diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift new file mode 100644 index 0000000000..38cb202db4 --- /dev/null +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift @@ -0,0 +1,28 @@ +import XCTest +import GRDB +@testable import WhoopStore + +final class OuraRawStoreTests: XCTestCase { + func testUpsertIsIdempotentByNaturalKey() async throws { + let store = try await WhoopStore.inMemory() + let first = OuraRawRow(endpoint: "sleep", documentId: "abc", day: "2026-01-01", + payloadJSON: #"{"id":"abc","v":1}"#, fetchedAt: 100) + let n1 = try await store.upsertOuraRaw([first], deviceId: "oura-api") + XCTAssertEqual(n1, 1) + + // Re-pull the same document id with a newer payload → overwrite in place, still one row. + let second = OuraRawRow(endpoint: "sleep", documentId: "abc", day: "2026-01-01", + payloadJSON: #"{"id":"abc","v":2}"#, fetchedAt: 200) + _ = try await store.upsertOuraRaw([second], deviceId: "oura-api") + + let rows = try await store.ouraRaw(deviceId: "oura-api", endpoint: "sleep") + XCTAssertEqual(rows.count, 1) + XCTAssertEqual(rows.first?.payloadJSON, #"{"id":"abc","v":2}"#) + XCTAssertEqual(rows.first?.fetchedAt, 200) + let sleepCount = try await store.ouraRawCount(deviceId: "oura-api", endpoint: "sleep") + XCTAssertEqual(sleepCount, 1) + // A different endpoint is a different partition. + let dailySleepCount = try await store.ouraRawCount(deviceId: "oura-api", endpoint: "daily_sleep") + XCTAssertEqual(dailySleepCount, 0) + } +} From 79eee33af3500e690c61de368079d9fbeb2fa4ec Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 15:51:27 -0700 Subject: [PATCH 005/171] feat(import): OuraHypnogram decoder (sleep_phase + movement, distinct legends) --- .../Sources/StrandImport/OuraHypnogram.swift | 49 +++++++++++++++++++ .../OuraHypnogramTests.swift | 29 +++++++++++ 2 files changed, 78 insertions(+) create mode 100644 Packages/StrandImport/Sources/StrandImport/OuraHypnogram.swift create mode 100644 Packages/StrandImport/Tests/StrandImportTests/OuraHypnogramTests.swift diff --git a/Packages/StrandImport/Sources/StrandImport/OuraHypnogram.swift b/Packages/StrandImport/Sources/StrandImport/OuraHypnogram.swift new file mode 100644 index 0000000000..3bb692ddb5 --- /dev/null +++ b/Packages/StrandImport/Sources/StrandImport/OuraHypnogram.swift @@ -0,0 +1,49 @@ +import Foundation + +// MARK: - Oura compact per-epoch sleep strings (DOCUMENTED Oura API v2 encodings; NOOP's own decoder) +// +// The Oura `sleep` object carries two compact strings. Their digit legends are DIFFERENT — never share a +// decoder between them: +// • sleep_phase_5_min / sleep_phase_30_sec — the hypnogram: '1'=deep, '2'=light, '3'=REM, '4'=awake. +// • movement_30_sec — motion: '1'=no motion, '2'=restless, '3'=tossing, '4'=active. + +public enum OuraHypnogram { + + /// `sleep_phase_*` digit → NOOP stage string ("deep"/"light"/"rem"/"wake", matching + /// `WearableSleepStageInterval`). Unknown digits → nil (skipped, the epoch clock still advances). + public static func stageName(_ ch: Character) -> String? { + switch ch { + case "1": return "deep" + case "2": return "light" + case "3": return "rem" + case "4": return "wake" + default: return nil + } + } + + /// Decode a `sleep_phase_5_min` (epochSeconds 300) or `sleep_phase_30_sec` (epochSeconds 30) string into + /// contiguous [{stage,start,end}] segments, epoch-aligned from `start`. Adjacent equal stages are merged. + public static func decode(_ phases: String, start: Date, epochSeconds: Int) -> [WearableSleepStageInterval] { + var out: [WearableSleepStageInterval] = [] + for (i, ch) in phases.enumerated() { + guard let stage = stageName(ch) else { continue } + let segStart = start.addingTimeInterval(Double(i * epochSeconds)) + let segEnd = segStart.addingTimeInterval(Double(epochSeconds)) + if var last = out.last, last.stage == stage, last.end == segStart { + last.end = segEnd + out[out.count - 1] = last + } else { + out.append(WearableSleepStageInterval(stage: stage, start: segStart, end: segEnd)) + } + } + return out + } + + /// Decode `movement_30_sec` into per-epoch motion magnitudes 1…4 (unknown digit → 0). 30-sec epochs. + public static func movement(_ s: String) -> [Int] { + s.map { ch in + switch ch { case "1": return 1; case "2": return 2; case "3": return 3; case "4": return 4 + default: return 0 } + } + } +} diff --git a/Packages/StrandImport/Tests/StrandImportTests/OuraHypnogramTests.swift b/Packages/StrandImport/Tests/StrandImportTests/OuraHypnogramTests.swift new file mode 100644 index 0000000000..f4614484b8 --- /dev/null +++ b/Packages/StrandImport/Tests/StrandImportTests/OuraHypnogramTests.swift @@ -0,0 +1,29 @@ +import XCTest +@testable import StrandImport + +final class OuraHypnogramTests: XCTestCase { + private func d(_ iso: String) -> Date { ISO8601DateFormatter().date(from: iso)! } + + func testStageLegendMapsOuraDigits() { + XCTAssertEqual(OuraHypnogram.stageName("1"), "deep") + XCTAssertEqual(OuraHypnogram.stageName("2"), "light") + XCTAssertEqual(OuraHypnogram.stageName("3"), "rem") + XCTAssertEqual(OuraHypnogram.stageName("4"), "wake") + XCTAssertNil(OuraHypnogram.stageName("0")) + } + + func testDecodeMergesAdjacentEqualStagesAndAligns() { + let start = d("2026-01-01T23:00:00Z") + // "1123" → deep(2 epochs merged), light(1), rem(1); 5-min epochs. + let segs = OuraHypnogram.decode("1123", start: start, epochSeconds: 300) + XCTAssertEqual(segs.map(\.stage), ["deep", "light", "rem"]) + XCTAssertEqual(segs[0].start, start) + XCTAssertEqual(segs[0].end, start.addingTimeInterval(600)) // two 5-min epochs merged + XCTAssertEqual(segs[2].end, start.addingTimeInterval(1200)) // total 4 epochs = 20 min + } + + func testMovementLegendIsDistinctFromStages() { + // movement uses 1=no motion … 4=active; unknown → 0. + XCTAssertEqual(OuraHypnogram.movement("1234x"), [1, 2, 3, 4, 0]) + } +} From a9bc1591433285c456766fb9b9cc421fa0c19f09 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 15:58:02 -0700 Subject: [PATCH 006/171] =?UTF-8?q?feat(import):=20OuraApiParser.parseSlee?= =?UTF-8?q?p=20=E2=80=94=20sessions=20with=20hypnogram,=20movement,=20in-s?= =?UTF-8?q?leep=20HR?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../Sources/StrandImport/OuraApiModels.swift | 22 +++++ .../Sources/StrandImport/OuraApiParser.swift | 85 +++++++++++++++++++ .../OuraApiParserSleepTests.swift | 38 +++++++++ 3 files changed, 145 insertions(+) create mode 100644 Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift create mode 100644 Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift create mode 100644 Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift diff --git a/Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift b/Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift new file mode 100644 index 0000000000..89cc28fd99 --- /dev/null +++ b/Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift @@ -0,0 +1,22 @@ +import Foundation + +// MARK: - Neutral models emitted by OuraApiParser (network-free; the app-target writer maps them to WhoopStore). + +/// One heart-rate sample (→ hrSample on write). `ts` is unix seconds, `bpm` is positive. +public struct OuraHRPoint: Sendable, Equatable { + public let ts: Int + public let bpm: Int + public init(ts: Int, bpm: Int) { self.ts = ts; self.bpm = bpm } +} + +/// One Oura detailed sleep period: the shared `session` (incl. decoded stages) plus the rich extras the +/// shared model can't hold. `movement30s` → sleepSession.motionJSON; `hr` → hrSample. (HRV samples stay +/// in the raw archive only — there is no rMSSD-per-sample table; the per-night average is on `session`.) +public struct OuraSleepPeriod: Sendable, Equatable { + public var session: WearableSleepSession + public var movement30s: [Int] + public var hr: [OuraHRPoint] + public init(session: WearableSleepSession, movement30s: [Int], hr: [OuraHRPoint]) { + self.session = session; self.movement30s = movement30s; self.hr = hr + } +} diff --git a/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift b/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift new file mode 100644 index 0000000000..91a365f025 --- /dev/null +++ b/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift @@ -0,0 +1,85 @@ +import Foundation + +// MARK: - Oura API v2 document parser (PURE / network-free) +// +// Takes already-fetched JSON `data[]` arrays (the OuraAPIClient does the I/O in the app target) and maps +// them to NOOP's normalized models. Reuses the SAME field semantics as OuraExportParser — the account export +// and the API share field names (bedtime_start, total_sleep_duration, …) — and adds the hypnogram + +// time-series the file export never carried. Crafted-input safe via WearableJSON. + +public enum OuraApiParser { + + /// Parse `sleep` documents (detailed periods; multiple per day are possible). Returns one + /// `OuraSleepPeriod` per usable period plus a per-day sleep rollup folded onto `WearableDailyRow`. + public static func parseSleep(_ docs: [[String: Any]]) -> (periods: [OuraSleepPeriod], days: [WearableDailyRow]) { + var periods: [OuraSleepPeriod] = [] + var byDay: [String: WearableDailyRow] = [:] + + for s in docs { + guard let start = WhoopTime.parseISOWithOffset(WearableJSON.str(s, "bedtime_start")), + let end = WhoopTime.parseISOWithOffset(WearableJSON.str(s, "bedtime_end")), + end > start else { continue } + + // Durations are SECONDS in the API → minutes. + func minutes(_ k: String) -> Double? { WearableJSON.posDbl(s, k).map { $0 / 60.0 } } + + // Stages: prefer the finer 30-sec hypnogram, else the 5-min; never synthesize one. + let stages: [WearableSleepStageInterval] + if let p30 = WearableJSON.str(s, "sleep_phase_30_sec"), !p30.isEmpty { + stages = OuraHypnogram.decode(p30, start: start, epochSeconds: 30) + } else if let p5 = WearableJSON.str(s, "sleep_phase_5_min"), !p5.isEmpty { + stages = OuraHypnogram.decode(p5, start: start, epochSeconds: 300) + } else { + stages = [] + } + + let session = WearableSleepSession( + start: start, end: end, + deepMin: minutes("deep_sleep_duration"), + lightMin: minutes("light_sleep_duration"), + remMin: minutes("rem_sleep_duration"), + awakeMin: minutes("awake_time"), + totalSleepMin: minutes("total_sleep_duration"), + efficiencyPct: WearableJSON.posDbl(s, "efficiency"), + avgHr: WearableJSON.posInt(s, "average_heart_rate"), + lowestHr: WearableJSON.posInt(s, "lowest_heart_rate"), + avgHrvMs: WearableJSON.posDbl(s, "average_hrv"), + respRateBpm: WearableJSON.posDbl(s, "average_breath"), + sleepScore: nil, // the night's reference score comes from daily_sleep + stages: stages) + + let movement = WearableJSON.str(s, "movement_30_sec").map(OuraHypnogram.movement) ?? [] + let hr = sampleSeries(s["heart_rate"]) + periods.append(OuraSleepPeriod(session: session, movement30s: movement, hr: hr)) + + // Fold the night onto its calendar day (Oura's "day" = the wake day). + let key = WearableJSON.str(s, "day") ?? WearableExportImporter.dayString(end) + var row = byDay[key] ?? WearableDailyRow(day: key) + row.totalSleepMin = row.totalSleepMin ?? session.totalSleepMin + row.deepMin = row.deepMin ?? session.deepMin + row.lightMin = row.lightMin ?? session.lightMin + row.remMin = row.remMin ?? session.remMin + row.awakeMin = row.awakeMin ?? session.awakeMin + row.efficiencyPct = row.efficiencyPct ?? session.efficiencyPct + row.avgHrvMs = row.avgHrvMs ?? session.avgHrvMs + if row.restingHr == nil { row.restingHr = session.lowestHr } + byDay[key] = row + } + return (periods, Array(byDay.values)) + } + + /// Decode an Oura `PublicSample` object ({interval, items:[Double?], timestamp}) into HR points. + /// Non-finite / non-positive items are dropped (Oura uses null/0 for gaps). + static func sampleSeries(_ any: Any?) -> [OuraHRPoint] { + guard let obj = any as? [String: Any], + let interval = WearableJSON.dbl(obj, "interval"), interval > 0, + let items = obj["items"] as? [Any], + let t0 = WhoopTime.parseISOWithOffset(WearableJSON.str(obj, "timestamp")) else { return [] } + var out: [OuraHRPoint] = [] + for (i, item) in items.enumerated() { + guard let v = (item as? Double) ?? (item as? NSNumber)?.doubleValue, v.isFinite, v > 0 else { continue } + out.append(OuraHRPoint(ts: Int(t0.timeIntervalSince1970 + Double(i) * interval), bpm: Int(v))) + } + return out + } +} diff --git a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift new file mode 100644 index 0000000000..001fca63b2 --- /dev/null +++ b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift @@ -0,0 +1,38 @@ +import XCTest +@testable import StrandImport + +final class OuraApiParserSleepTests: XCTestCase { + func testParsesPeriodWithHypnogramAndFoldsDay() throws { + let docs: [[String: Any]] = [[ + "id": "s1", "day": "2026-01-02", + "bedtime_start": "2026-01-01T23:00:00+00:00", + "bedtime_end": "2026-01-02T06:00:00+00:00", + "total_sleep_duration": 24000, "deep_sleep_duration": 6000, + "light_sleep_duration": 12000, "rem_sleep_duration": 6000, "awake_time": 1200, + "efficiency": 92, "average_heart_rate": 55, "lowest_heart_rate": 48, + "average_hrv": 65, "average_breath": 14.2, + "sleep_phase_5_min": "1234", + "movement_30_sec": "1212", + "heart_rate": ["interval": 300, "timestamp": "2026-01-01T23:00:00+00:00", + "items": [60, 58, NSNull(), 57]] + ]] + let (periods, days) = OuraApiParser.parseSleep(docs) + XCTAssertEqual(periods.count, 1) + let p = try XCTUnwrap(periods.first) + XCTAssertEqual(p.session.totalSleepMin, 400) // 24000s ÷ 60 + XCTAssertEqual(p.session.lowestHr, 48) + XCTAssertEqual(p.session.avgHrvMs, 65) + XCTAssertEqual(p.session.stages.map(\.stage), ["deep", "light", "rem", "wake"]) + XCTAssertEqual(p.movement30s, [1, 2, 1, 2]) + XCTAssertEqual(p.hr.map(\.bpm), [60, 58, 57]) // null sample dropped + XCTAssertEqual(days.count, 1) + XCTAssertEqual(days.first?.day, "2026-01-02") + XCTAssertEqual(days.first?.totalSleepMin, 400) + XCTAssertEqual(days.first?.restingHr, 48) // lowestHr ≈ resting fallback + } + + func testSkipsPeriodWithInvalidSpan() { + let (periods, _) = OuraApiParser.parseSleep([["bedtime_start": "nope", "bedtime_end": "nope"]]) + XCTAssertTrue(periods.isEmpty) + } +} From 623d642cae96ad87729e5bdc1a9d27877d96f5a0 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 16:10:40 -0700 Subject: [PATCH 007/171] fix(import): range-guard OuraApiParser.sampleSeries + fold respRateBpm (Task 3 review) --- .../Sources/StrandImport/OuraApiParser.swift | 7 +++- .../OuraApiParserSleepTests.swift | 36 +++++++++++++++++++ 2 files changed, 42 insertions(+), 1 deletion(-) diff --git a/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift b/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift index 91a365f025..58a5e390b8 100644 --- a/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift +++ b/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift @@ -62,6 +62,7 @@ public enum OuraApiParser { row.awakeMin = row.awakeMin ?? session.awakeMin row.efficiencyPct = row.efficiencyPct ?? session.efficiencyPct row.avgHrvMs = row.avgHrvMs ?? session.avgHrvMs + row.respRateBpm = row.respRateBpm ?? session.respRateBpm if row.restingHr == nil { row.restingHr = session.lowestHr } byDay[key] = row } @@ -78,7 +79,11 @@ public enum OuraApiParser { var out: [OuraHRPoint] = [] for (i, item) in items.enumerated() { guard let v = (item as? Double) ?? (item as? NSNumber)?.doubleValue, v.isFinite, v > 0 else { continue } - out.append(OuraHRPoint(ts: Int(t0.timeIntervalSince1970 + Double(i) * interval), bpm: Int(v))) + // Finite-but-huge `ts` or `bpm` (e.g. a crafted `interval`/item of 1e300) must never trap + // Int(...); skip the sample instead, mirroring WearableJSON.int's -9e18...9e18 range guard. + let ts = t0.timeIntervalSince1970 + Double(i) * interval + guard ts.isFinite, ts >= -9e18, ts <= 9e18, v <= 9e18 else { continue } + out.append(OuraHRPoint(ts: Int(ts), bpm: Int(v))) } return out } diff --git a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift index 001fca63b2..f196bd0cfc 100644 --- a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift +++ b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift @@ -25,14 +25,50 @@ final class OuraApiParserSleepTests: XCTestCase { XCTAssertEqual(p.session.stages.map(\.stage), ["deep", "light", "rem", "wake"]) XCTAssertEqual(p.movement30s, [1, 2, 1, 2]) XCTAssertEqual(p.hr.map(\.bpm), [60, 58, 57]) // null sample dropped + // ts = bedtime_start epoch + i*interval, using the ORIGINAL items index (the null at i=2 is + // skipped, so the 3rd kept sample is i=3, i.e. epoch + 900, not epoch + 600). + let bedtimeStart = try XCTUnwrap(WhoopTime.parseISOWithOffset("2026-01-01T23:00:00+00:00")) + let ts0 = Int(bedtimeStart.timeIntervalSince1970) + XCTAssertEqual(p.hr.map(\.ts), [ts0, ts0 + 300, ts0 + 900]) XCTAssertEqual(days.count, 1) XCTAssertEqual(days.first?.day, "2026-01-02") XCTAssertEqual(days.first?.totalSleepMin, 400) XCTAssertEqual(days.first?.restingHr, 48) // lowestHr ≈ resting fallback + XCTAssertEqual(days.first?.respRateBpm, 14.2) // average_breath folded onto the day rollup } func testSkipsPeriodWithInvalidSpan() { let (periods, _) = OuraApiParser.parseSleep([["bedtime_start": "nope", "bedtime_end": "nope"]]) XCTAssertTrue(periods.isEmpty) } + + func testSampleSeriesSkipsFiniteButHugeValuesWithoutCrashing() throws { + // Both `ts` and `bpm` in sampleSeries are Double->Int casts; a finite-but-huge value for EITHER + // (a crafted `interval` or a crafted `items` element) must be skipped, never trap Int(...). + + // A huge `interval` blows the 2nd sample's ts (t0 + 1*interval) past Int range; the 1st sample's + // ts (t0 + 0*interval == t0) stays in range and must still be kept. + let hugeIntervalDocs: [[String: Any]] = [[ + "day": "2026-01-02", + "bedtime_start": "2026-01-01T23:00:00+00:00", + "bedtime_end": "2026-01-02T06:00:00+00:00", + "heart_rate": ["interval": 1e300, "timestamp": "2026-01-01T23:00:00+00:00", + "items": [60, 58]] + ]] + let (hugeIntervalPeriods, _) = OuraApiParser.parseSleep(hugeIntervalDocs) + let hugeIntervalPeriod = try XCTUnwrap(hugeIntervalPeriods.first) + XCTAssertEqual(hugeIntervalPeriod.hr.map(\.bpm), [60]) + + // A huge `items` element (bpm itself) must be skipped even with a perfectly normal interval/ts. + let hugeItemDocs: [[String: Any]] = [[ + "day": "2026-01-02", + "bedtime_start": "2026-01-01T23:00:00+00:00", + "bedtime_end": "2026-01-02T06:00:00+00:00", + "heart_rate": ["interval": 300, "timestamp": "2026-01-01T23:00:00+00:00", + "items": [60, 1e300, 58]] + ]] + let (hugeItemPeriods, _) = OuraApiParser.parseSleep(hugeItemDocs) + let hugeItemPeriod = try XCTUnwrap(hugeItemPeriods.first) + XCTAssertEqual(hugeItemPeriod.hr.map(\.bpm), [60, 58]) + } } From 4d8ba2927ec17ec095c8d7142fd78cac91caf5c8 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 16:16:16 -0700 Subject: [PATCH 008/171] =?UTF-8?q?feat(import):=20OuraApiParser.parseDail?= =?UTF-8?q?y=20=E2=80=94=20summary=20endpoints,=20scores=20kept=20referenc?= =?UTF-8?q?e-only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../Sources/StrandImport/OuraApiModels.swift | 10 +++ .../StrandImport/OuraApiParser+Daily.swift | 89 +++++++++++++++++++ .../OuraApiParserDailyTests.swift | 42 +++++++++ 3 files changed, 141 insertions(+) create mode 100644 Packages/StrandImport/Sources/StrandImport/OuraApiParser+Daily.swift create mode 100644 Packages/StrandImport/Tests/StrandImportTests/OuraApiParserDailyTests.swift diff --git a/Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift b/Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift index 89cc28fd99..a15f1e2efa 100644 --- a/Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift +++ b/Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift @@ -20,3 +20,13 @@ public struct OuraSleepPeriod: Sendable, Equatable { self.session = session; self.movement30s = movement30s; self.hr = hr } } + +/// One extra daily scalar Oura returns that the wide `dailyMetric` columns don't hold (→ metricSeries on +/// write). `key` is the metricSeries key; the brand's OWN scores use a `ref_` prefix and its contributor +/// breakdowns an `oura_` prefix, so they are browseable but never mistaken for a NOOP score. +public struct OuraDailyExtra: Sendable, Equatable { + public let day: String + public let key: String + public let value: Double + public init(day: String, key: String, value: Double) { self.day = day; self.key = key; self.value = value } +} diff --git a/Packages/StrandImport/Sources/StrandImport/OuraApiParser+Daily.swift b/Packages/StrandImport/Sources/StrandImport/OuraApiParser+Daily.swift new file mode 100644 index 0000000000..7e7a75c397 --- /dev/null +++ b/Packages/StrandImport/Sources/StrandImport/OuraApiParser+Daily.swift @@ -0,0 +1,89 @@ +import Foundation + +public extension OuraApiParser { + + /// Parse a day-keyed summary endpoint's documents. `endpoint` selects the field mapping. Writes to + /// `WearableDailyRow` columns only where the on-device schema already has a home (RHR, skin-temp dev, + /// steps, calories, SpO2); everything else — and ALL of Oura's own scores — becomes `OuraDailyExtra`. + static func parseDaily(_ docs: [[String: Any]], endpoint: String) -> (days: [WearableDailyRow], extras: [OuraDailyExtra]) { + var byDay: [String: WearableDailyRow] = [:] + var extras: [OuraDailyExtra] = [] + func extra(_ day: String, _ key: String, _ v: Double?) { + if let v { extras.append(OuraDailyExtra(day: day, key: key, value: v)) } + } + + for doc in docs { + guard let day = WearableJSON.str(doc, "day") else { continue } + switch endpoint { + + case "daily_readiness": + var r = byDay[day] ?? WearableDailyRow(day: day) + if let c = doc["contributors"] as? [String: Any] { + r.restingHr = WearableJSON.posInt(c, "resting_heart_rate") ?? r.restingHr + for (k, _) in c { extra(day, "oura_readiness_\(k)", WearableJSON.posDbl(c, k)) } + } + r.skinTempDevC = WearableJSON.dbl(doc, "temperature_deviation") ?? r.skinTempDevC + r.readinessScore = WearableJSON.posInt(doc, "score") ?? r.readinessScore + extra(day, "ref_readiness_score", WearableJSON.posDbl(doc, "score")) + byDay[day] = r + + case "daily_sleep": + extra(day, "ref_sleep_score", WearableJSON.posDbl(doc, "score")) + if let c = doc["contributors"] as? [String: Any] { + for (k, _) in c { extra(day, "oura_sleep_\(k)", WearableJSON.posDbl(c, k)) } + } + + case "daily_activity": + var r = byDay[day] ?? WearableDailyRow(day: day) + r.steps = WearableJSON.posInt(doc, "steps") ?? r.steps + r.activeKcal = WearableJSON.posDbl(doc, "active_calories") ?? r.activeKcal + r.totalKcal = WearableJSON.posDbl(doc, "total_calories") ?? r.totalKcal + extra(day, "ref_activity_score", WearableJSON.posDbl(doc, "score")) + extra(day, "oura_equiv_walk_m", WearableJSON.posDbl(doc, "equivalent_walking_distance")) + byDay[day] = r + + case "daily_spo2": + var r = byDay[day] ?? WearableDailyRow(day: day) + if let s = doc["spo2_percentage"] as? [String: Any] { + r.spo2Pct = WearableJSON.posDbl(s, "average") ?? r.spo2Pct + extra(day, "spo2", WearableJSON.posDbl(s, "average")) + } + extra(day, "oura_breathing_disturbance_index", WearableJSON.dbl(doc, "breathing_disturbance_index")) + byDay[day] = r + + case "daily_stress": + extra(day, "oura_stress_high_s", WearableJSON.posDbl(doc, "stress_high")) + extra(day, "oura_recovery_high_s", WearableJSON.posDbl(doc, "recovery_high")) + + case "daily_resilience": + if let c = doc["contributors"] as? [String: Any] { + for (k, _) in c { extra(day, "oura_resilience_\(k)", WearableJSON.dbl(c, k)) } + } + extra(day, "ref_resilience_level", resilienceLevel(WearableJSON.str(doc, "level"))) + + case "daily_cardiovascular_age": + extra(day, "oura_vascular_age", WearableJSON.dbl(doc, "vascular_age")) + extra(day, "oura_pulse_wave_velocity", WearableJSON.dbl(doc, "pulse_wave_velocity")) + + case "vO2_max": + extra(day, "vo2max", WearableJSON.posDbl(doc, "vo2_max")) + + default: + break + } + } + return (Array(byDay.values), extras) + } + + /// Oura resilience level → an ordered reference number (1…5). Reference-only — never a NOOP score. + static func resilienceLevel(_ s: String?) -> Double? { + switch s { + case "limited": return 1 + case "adequate": return 2 + case "solid": return 3 + case "strong": return 4 + case "exceptional": return 5 + default: return nil + } + } +} diff --git a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserDailyTests.swift b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserDailyTests.swift new file mode 100644 index 0000000000..ca3e176b8e --- /dev/null +++ b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserDailyTests.swift @@ -0,0 +1,42 @@ +import XCTest +@testable import StrandImport + +final class OuraApiParserDailyTests: XCTestCase { + func testReadinessMapsRhrSkinTempAndRefScore() { + let (days, extras) = OuraApiParser.parseDaily([[ + "day": "2026-01-02", "score": 80, "temperature_deviation": -0.2, + "contributors": ["resting_heart_rate": 50, "hrv_balance": 70] + ]], endpoint: "daily_readiness") + XCTAssertEqual(days.first?.restingHr, 50) + XCTAssertEqual(days.first?.skinTempDevC, -0.2) + XCTAssertEqual(days.first?.readinessScore, 80) + XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "ref_readiness_score", value: 80))) + XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "oura_readiness_hrv_balance", value: 70))) + } + + func testActivityMapsStepsAndScoreIsReferenceOnly() { + let (days, extras) = OuraApiParser.parseDaily([[ + "day": "2026-01-02", "score": 88, "steps": 9000, + "active_calories": 500, "total_calories": 2400 + ]], endpoint: "daily_activity") + XCTAssertEqual(days.first?.steps, 9000) + XCTAssertEqual(days.first?.activeKcal, 500) + // The activity SCORE is reference-only — never a dailyMetric score column. + XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "ref_activity_score", value: 88))) + } + + func testResilienceLevelEncodedReferenceOnly() { + let (_, extras) = OuraApiParser.parseDaily([[ + "day": "2026-01-02", "level": "solid", "contributors": ["sleep_recovery": 60.0] + ]], endpoint: "daily_resilience") + XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "ref_resilience_level", value: 3))) + XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "oura_resilience_sleep_recovery", value: 60))) + } + + func testSpo2MapsAverage() { + let (days, _) = OuraApiParser.parseDaily([[ + "day": "2026-01-02", "spo2_percentage": ["average": 96.5] + ]], endpoint: "daily_spo2") + XCTAssertEqual(days.first?.spo2Pct, 96.5) + } +} From 05bd69e09a25077ab3a2c23b8459d9b06e742235 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 16:27:23 -0700 Subject: [PATCH 009/171] feat(import): OuraApiParser workouts + heartrate time-series parsing --- .../Sources/StrandImport/OuraApiModels.swift | 14 ++++++++ .../StrandImport/OuraApiParser+Events.swift | 34 +++++++++++++++++++ .../OuraApiParserEventsTests.swift | 26 ++++++++++++++ 3 files changed, 74 insertions(+) create mode 100644 Packages/StrandImport/Sources/StrandImport/OuraApiParser+Events.swift create mode 100644 Packages/StrandImport/Tests/StrandImportTests/OuraApiParserEventsTests.swift diff --git a/Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift b/Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift index a15f1e2efa..204f11a189 100644 --- a/Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift +++ b/Packages/StrandImport/Sources/StrandImport/OuraApiModels.swift @@ -30,3 +30,17 @@ public struct OuraDailyExtra: Sendable, Equatable { public let value: Double public init(day: String, key: String, value: Double) { self.day = day; self.key = key; self.value = value } } + +/// One Oura workout (→ the `workout` table on write). Times are UTC; metric fields nil when absent. +public struct OuraWorkout: Sendable, Equatable { + public let start: Date + public let end: Date + public let activity: String + public let source: String + public let energyKcal: Double? + public let distanceM: Double? + public init(start: Date, end: Date, activity: String, source: String, energyKcal: Double?, distanceM: Double?) { + self.start = start; self.end = end; self.activity = activity; self.source = source + self.energyKcal = energyKcal; self.distanceM = distanceM + } +} diff --git a/Packages/StrandImport/Sources/StrandImport/OuraApiParser+Events.swift b/Packages/StrandImport/Sources/StrandImport/OuraApiParser+Events.swift new file mode 100644 index 0000000000..c205da5cf6 --- /dev/null +++ b/Packages/StrandImport/Sources/StrandImport/OuraApiParser+Events.swift @@ -0,0 +1,34 @@ +import Foundation + +public extension OuraApiParser { + + /// Parse `workout` documents → `OuraWorkout`. Rows without a valid time span are skipped. + static func parseWorkouts(_ docs: [[String: Any]]) -> [OuraWorkout] { + var out: [OuraWorkout] = [] + for w in docs { + guard let start = WhoopTime.parseISOWithOffset(WearableJSON.str(w, "start_datetime")), + let end = WhoopTime.parseISOWithOffset(WearableJSON.str(w, "end_datetime")), + end > start else { continue } + out.append(OuraWorkout( + start: start, end: end, + activity: WearableJSON.str(w, "activity") ?? "workout", + source: WearableJSON.str(w, "source") ?? "oura", + energyKcal: WearableJSON.posDbl(w, "calories"), + distanceM: WearableJSON.posDbl(w, "distance"))) + } + return out + } + + /// Parse a `/v2/usercollection/heartrate` page's `data[]` → HR points. Each row is + /// {timestamp, bpm, source}; non-positive bpm is dropped. (The `source` enum is preserved in the raw + /// archive, not in the compact hrSample.) + static func parseHeartRate(_ docs: [[String: Any]]) -> [OuraHRPoint] { + var out: [OuraHRPoint] = [] + for h in docs { + guard let ts = WhoopTime.parseISOWithOffset(WearableJSON.str(h, "timestamp")), + let bpm = WearableJSON.posInt(h, "bpm") else { continue } + out.append(OuraHRPoint(ts: Int(ts.timeIntervalSince1970), bpm: bpm)) + } + return out + } +} diff --git a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserEventsTests.swift b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserEventsTests.swift new file mode 100644 index 0000000000..8ed904ed65 --- /dev/null +++ b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserEventsTests.swift @@ -0,0 +1,26 @@ +import XCTest +@testable import StrandImport + +final class OuraApiParserEventsTests: XCTestCase { + func testWorkoutParse() throws { + let w = OuraApiParser.parseWorkouts([[ + "activity": "running", "source": "autodetected", + "start_datetime": "2026-01-02T07:00:00+00:00", "end_datetime": "2026-01-02T07:40:00+00:00", + "calories": 410, "distance": 6200 + ]]) + XCTAssertEqual(w.count, 1) + XCTAssertEqual(w.first?.activity, "running") + XCTAssertEqual(w.first?.source, "autodetected") + XCTAssertEqual(w.first?.energyKcal, 410) + XCTAssertEqual(w.first?.distanceM, 6200) + } + + func testHeartRatePageParseDropsNonPositive() { + let hr = OuraApiParser.parseHeartRate([ + ["timestamp": "2026-01-02T07:00:00+00:00", "bpm": 120, "source": "workout"], + ["timestamp": "2026-01-02T07:05:00+00:00", "bpm": 0, "source": "workout"] // dropped + ]) + XCTAssertEqual(hr.map(\.bpm), [120]) + XCTAssertEqual(hr.first?.ts, Int(ISO8601DateFormatter().date(from: "2026-01-02T07:00:00Z")!.timeIntervalSince1970)) + } +} From d6814bac51ca3fb7ca788225e6393c50bf0d64e8 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 16:45:07 -0700 Subject: [PATCH 010/171] chore(store): correct OuraRawStore MARK comment v19->v24 (final-review nit) --- Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift b/Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift index 1f4fc71301..dc8596a0ab 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift @@ -1,7 +1,7 @@ import Foundation import GRDB -// MARK: - v19 store: lossless Oura live-API payload archive +// MARK: - v24 store: lossless Oura live-API payload archive // Mirrors MetricSeriesStore exactly: a Codable row struct, an idempotent ON CONFLICT upsert keyed by the // natural key (deviceId, endpoint, documentId), and range reads — all GRDB work via syncWrite/syncRead. // This is the "as much as we can get" backstop: the verbatim payload is kept so a future metric can be From eff93c49a11d684dfa483ef8cbe6886603d395d4 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 19:03:44 -0700 Subject: [PATCH 011/171] =?UTF-8?q?docs:=20implementation=20plan=202/3=20?= =?UTF-8?q?=E2=80=94=20Oura=20network=20+=20auth=20(app=20target)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Keychain token store, OuraCredentials (xcconfig->Info.plist), OAuth2 authorization-code (BYO Oura app) behind an AuthProvider seam, and the URLSession client (next_token paging, 429 backoff, 401 refresh, sandbox/prod). Networking confined to Strand/Oura/; packages stay network-free. Tested via xcodebuild StrandTests with a URLProtocol stub + pure builders/parsers. --- ...-27-oura-live-api-import-2-network-auth.md | 894 ++++++++++++++++++ 1 file changed, 894 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-27-oura-live-api-import-2-network-auth.md diff --git a/docs/superpowers/plans/2026-06-27-oura-live-api-import-2-network-auth.md b/docs/superpowers/plans/2026-06-27-oura-live-api-import-2-network-auth.md new file mode 100644 index 0000000000..4473e6c528 --- /dev/null +++ b/docs/superpowers/plans/2026-06-27-oura-live-api-import-2-network-auth.md @@ -0,0 +1,894 @@ +# Oura Live API Import — Plan 2: Network + Auth (app target) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the app-target network + auth layer for the Oura cloud import — Keychain token storage, credentials, the OAuth2 authorization-code flow (bring-your-own Oura app), and the paging/backoff `URLSession` API client — behind an `AuthProvider` seam, all in `Strand/Oura/`. + +**Architecture:** Networking lives in the **app target** (`Strand/`), beside `Strand/AI/AICoach.swift` — never in a package (the packages stay network-free; that's the privacy invariant Plan 1 preserved). We mirror three proven app-target patterns: `AIKeyStore` (Keychain), `AICoachError` (`LocalizedError`), and the injected `session: URLSession = .shared` seam with **pure parse/build helpers split out** so the request-building, response-envelope parsing, paging, and token math are unit-tested without a live socket. The interactive `ASWebAuthenticationSession` step is thin, behind `AuthProvider`, and integration-verified (it opens a browser — not headless-unit-testable). + +**Tech stack:** Swift 6.3.1 (app builds via XcodeGen `Strand.xcodeproj`), `Foundation` + `Security` (Keychain) + `AuthenticationServices` (`ASWebAuthenticationSession`, new to this repo). XCTest in `StrandTests`. Tests run via `xcodebuild`, NOT `swift test` (this is the Xcode app target, not an SPM package). + +## Global Constraints + +Every task implicitly includes these: +- **Networking only in the app target.** These files live in `Strand/Oura/`. No package gains a `URLSession`/`URLRequest`/`ASWebAuthenticationSession` import. `Strand/` sources are auto-globbed into both the `Strand` (macOS) and `NOOPiOS` targets by `project.yml` — no manifest edit needed for new `.swift` files (but the project must be regenerated: `xcodegen generate`). +- **Second opt-in network lane.** This is off-by-default and user-initiated (the AI Coach is the first exception). Data flow is inbound (Oura → device, under the user's own OAuth grant). No NOOP server. Tokens live only in the Keychain, never UserDefaults/plist/logs. +- **BYO Oura app, no PKCE.** OAuth2 authorization-code against Oura: authorize `https://cloud.ouraring.com/oauth/authorize`, token `https://api.ouraring.com/oauth/token`, bearer header `Authorization: Bearer `. `client_id`+`client_secret` come from an untracked xcconfig via Info.plist (Task 5). Scopes (all 8): `email personal daily heartrate workout tag session spo2Daily`. +- **Testability seam.** Every networked type takes `session: URLSession = .shared` in its initializer (mirroring `AICoachEngine`), so tests inject a `URLSession` backed by a `URLProtocol` stub. Pure builders/parsers are `static` and separately unit-tested (mirroring `AnthropicClient.parseModels`). +- **Provenance / consumes Plan 1.** This plan does not write to the store yet (that's Plan 3). It produces the fetched raw JSON pages + tokens that Plan 3's writer consumes. `deviceId = "oura-api"` is the partition (Plan 3). +- **Build/test path.** `xcodegen generate` then `xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/ CODE_SIGNING_ALLOWED=NO`. The first build is a full app compile (minutes); subsequent task builds are incremental. The full macOS app compiling clean under this toolchain is confirmed. + +--- + +### Task 1: `OuraError` + `OuraTokenStore` (Keychain) + +The typed error surface and secure token storage, mirroring `AICoachError` / `AIKeyStore`. + +**Files:** +- Create: `Strand/Oura/OuraError.swift` +- Create: `Strand/Oura/OuraTokenStore.swift` +- Test: `StrandTests/OuraTokenStoreTests.swift` + +**Interfaces:** +- Produces: + - `enum OuraError: LocalizedError { case notConnected, authFailed(String), badResponse(Int, String), rateLimited, network(String), decode, tokenExchangeFailed(String) }` + - `struct OuraTokens: Equatable { let accessToken: String; let refreshToken: String?; let expiresAt: Date }` with `var isExpired: Bool` (true within a 60 s skew of `expiresAt`). + - `enum OuraTokenStore` with `static func save(_ tokens: OuraTokens) -> Bool`, `static func load() -> OuraTokens?`, `static func clear()`, `static var isConnected: Bool`. + +- [ ] **Step 1: Write the failing test** + +Create `StrandTests/OuraTokenStoreTests.swift`: + +```swift +import XCTest +@testable import Strand + +final class OuraTokenStoreTests: XCTestCase { + override func setUp() { super.setUp(); OuraTokenStore.clear() } + override func tearDown() { OuraTokenStore.clear(); super.tearDown() } + + func testSaveLoadClearRoundTrip() { + XCTAssertFalse(OuraTokenStore.isConnected) + XCTAssertNil(OuraTokenStore.load()) + + let exp = Date(timeIntervalSince1970: 2_000_000_000) + let t = OuraTokens(accessToken: "acc", refreshToken: "ref", expiresAt: exp) + XCTAssertTrue(OuraTokenStore.save(t)) + XCTAssertTrue(OuraTokenStore.isConnected) + + let loaded = OuraTokenStore.load() + XCTAssertEqual(loaded, t) + + OuraTokenStore.clear() + XCTAssertNil(OuraTokenStore.load()) + XCTAssertFalse(OuraTokenStore.isConnected) + } + + func testExpiryUsesSkew() { + let almostNow = OuraTokens(accessToken: "a", refreshToken: nil, + expiresAt: Date(timeIntervalSinceNow: 30)) // <60 s away + XCTAssertTrue(almostNow.isExpired) + let later = OuraTokens(accessToken: "a", refreshToken: nil, + expiresAt: Date(timeIntervalSinceNow: 600)) + XCTAssertFalse(later.isExpired) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `xcodegen generate && xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/OuraTokenStoreTests CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20` +Expected: FAIL — `OuraTokenStore` / `OuraTokens` / `OuraError` undefined (compile error). + +- [ ] **Step 3: Implement `OuraError`** + +Create `Strand/Oura/OuraError.swift`: + +```swift +import Foundation + +/// User-facing failure reasons for the Oura cloud-import lane, mapped to clear, non-crashing messages. +/// Mirrors AICoachError's shape (LocalizedError + errorDescription). +enum OuraError: LocalizedError, Equatable { + case notConnected + case authFailed(String) + case tokenExchangeFailed(String) + case badResponse(Int, String) + case rateLimited + case network(String) + case decode + + var errorDescription: String? { + switch self { + case .notConnected: + return "Connect your Oura account first." + case .authFailed(let d): + return "Oura sign-in didn't complete: \(d)" + case .tokenExchangeFailed(let d): + return "Couldn't exchange the Oura authorization code: \(d)" + case .badResponse(let code, let detail): + let extra = detail.isEmpty ? "" : " — \(detail)" + return "Oura returned an error (\(code))\(extra)." + case .rateLimited: + return "Oura is rate-limiting requests. Waiting before retrying." + case .network(let d): + return "Network problem talking to Oura: \(d)" + case .decode: + return "Couldn't read Oura's response." + } + } +} +``` + +- [ ] **Step 4: Implement `OuraTokens` + `OuraTokenStore`** + +Create `Strand/Oura/OuraTokenStore.swift`: + +```swift +import Foundation +import Security + +/// The OAuth tokens for the Oura cloud lane. `expiresAt` is absolute (computed from `expires_in` at +/// exchange time). `isExpired` applies a 60 s skew so we refresh slightly early rather than mid-request. +struct OuraTokens: Equatable, Codable { + let accessToken: String + let refreshToken: String? + let expiresAt: Date + + var isExpired: Bool { Date() >= expiresAt.addingTimeInterval(-60) } +} + +/// Keychain Services wrapper for the Oura tokens. One generic-password item (JSON-encoded `OuraTokens`) +/// under a fixed service, so tokens never land in UserDefaults, a plist, or on disk in the clear. +/// Mirrors AIKeyStore exactly (delete-then-add, kSecAttrAccessibleAfterFirstUnlock). +enum OuraTokenStore { + private static let service = "com.noop.oura" + private static let account = "oauth-tokens" + + private static var baseQuery: [String: Any] { + [ + kSecClass as String: kSecClassGenericPassword, + kSecAttrService as String: service, + kSecAttrAccount as String: account, + ] + } + + /// Store (or replace) the tokens. Returns false if the Keychain write fails (so callers never assume + /// a token is stored when it isn't). + @discardableResult + static func save(_ tokens: OuraTokens) -> Bool { + guard let data = try? JSONEncoder().encode(tokens) else { return false } + SecItemDelete(baseQuery as CFDictionary) + var attrs = baseQuery + attrs[kSecValueData as String] = data + attrs[kSecAttrAccessible as String] = kSecAttrAccessibleAfterFirstUnlock + return SecItemAdd(attrs as CFDictionary, nil) == errSecSuccess + } + + /// Load the stored tokens, or nil if none. + static func load() -> OuraTokens? { + var query = baseQuery + query[kSecReturnData as String] = kCFBooleanTrue + query[kSecMatchLimit as String] = kSecMatchLimitOne + var item: CFTypeRef? + guard SecItemCopyMatching(query as CFDictionary, &item) == errSecSuccess, + let data = item as? Data, + let tokens = try? JSONDecoder().decode(OuraTokens.self, from: data) else { return nil } + return tokens + } + + /// Remove any stored tokens. + static func clear() { SecItemDelete(baseQuery as CFDictionary) } + + /// True when tokens are present (regardless of expiry — an expired token is still "connected", it + /// just needs a refresh). + static var isConnected: Bool { load() != nil } +} +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/OuraTokenStoreTests CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20` +Expected: PASS (`Test Suite 'OuraTokenStoreTests' passed`). If the Keychain is unavailable in the test host and `save` returns false, note it in the report — the AICoach Keychain tests establish that the host allows it; if they pass and this doesn't, investigate before proceeding. + +- [ ] **Step 6: Commit** + +```bash +git add Strand/Oura/OuraError.swift Strand/Oura/OuraTokenStore.swift StrandTests/OuraTokenStoreTests.swift +git commit -m "feat(oura): OuraError + OuraTokenStore (Keychain token storage)" +``` + +--- + +### Task 2: `OuraCredentials` + `AuthProvider` protocol + +Read the BYO-app credentials from Info.plist (injected via xcconfig in Task 5) and define the auth seam. + +**Files:** +- Create: `Strand/Oura/OuraCredentials.swift` +- Create: `Strand/Oura/AuthProvider.swift` +- Test: `StrandTests/OuraCredentialsTests.swift` + +**Interfaces:** +- Consumes: nothing from Task 1. +- Produces: + - `struct OuraCredentials: Equatable { let clientId: String; let clientSecret: String; let redirectURI: String }` with `static func from(_ info: [String: Any]) -> OuraCredentials?` (nil if any key missing/blank) and `static var fromBundle: OuraCredentials?` (reads `Bundle.main.infoDictionary`). + - `protocol AuthProvider { var isConnected: Bool { get }; func validAccessToken() async throws -> String; func authorize(presentationAnchor: ASPresentationAnchor) async throws; func signOut() }` + +- [ ] **Step 1: Write the failing test** + +Create `StrandTests/OuraCredentialsTests.swift`: + +```swift +import XCTest +@testable import Strand + +final class OuraCredentialsTests: XCTestCase { + func testParsesCompleteInfoDict() { + let info: [String: Any] = [ + "OURA_CLIENT_ID": "cid", "OURA_CLIENT_SECRET": "secret", + "OURA_REDIRECT_URI": "noop://oura/callback", + ] + let c = OuraCredentials.from(info) + XCTAssertEqual(c, OuraCredentials(clientId: "cid", clientSecret: "secret", + redirectURI: "noop://oura/callback")) + } + + func testNilWhenAnyKeyMissingOrBlank() { + XCTAssertNil(OuraCredentials.from(["OURA_CLIENT_ID": "cid", "OURA_CLIENT_SECRET": "s"])) // no redirect + XCTAssertNil(OuraCredentials.from([ + "OURA_CLIENT_ID": "", "OURA_CLIENT_SECRET": "s", "OURA_REDIRECT_URI": "noop://x", + ])) // blank id + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `xcodegen generate && xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/OuraCredentialsTests CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20` +Expected: FAIL — `OuraCredentials` undefined. + +- [ ] **Step 3: Implement `OuraCredentials`** + +Create `Strand/Oura/OuraCredentials.swift`: + +```swift +import Foundation + +/// The user's own Oura OAuth app credentials, injected at build time via an untracked xcconfig → +/// Info.plist (Task 5). Absent credentials mean the Connect flow is unavailable (surfaced in the UI). +struct OuraCredentials: Equatable { + let clientId: String + let clientSecret: String + let redirectURI: String + + /// Build from an Info-dictionary-shaped map. Returns nil unless all three keys are present and + /// non-blank (so a build without the xcconfig cleanly disables the lane rather than half-configuring). + static func from(_ info: [String: Any]) -> OuraCredentials? { + func nonBlank(_ key: String) -> String? { + guard let s = (info[key] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines), + !s.isEmpty else { return nil } + return s + } + guard let id = nonBlank("OURA_CLIENT_ID"), + let secret = nonBlank("OURA_CLIENT_SECRET"), + let redirect = nonBlank("OURA_REDIRECT_URI") else { return nil } + return OuraCredentials(clientId: id, clientSecret: secret, redirectURI: redirect) + } + + /// The live credentials from the app bundle's Info.plist, or nil if not configured. + static var fromBundle: OuraCredentials? { from(Bundle.main.infoDictionary ?? [:]) } +} +``` + +- [ ] **Step 4: Implement `AuthProvider`** + +Create `Strand/Oura/AuthProvider.swift`: + +```swift +import Foundation +import AuthenticationServices + +/// The auth seam. Today's concrete provider is `OuraOAuthProvider` (BYO-app authorization-code, Task 3); +/// a future backend-mediated provider can replace it without touching the API client or the sync layer. +protocol AuthProvider { + /// True when tokens are stored (connected), regardless of expiry. + var isConnected: Bool { get } + /// Return a currently-valid access token, refreshing transparently if the stored one is expired. + func validAccessToken() async throws -> String + /// Run the interactive authorization (opens Oura's consent page) and store the resulting tokens. + @MainActor func authorize(presentationAnchor: ASPresentationAnchor) async throws + /// Forget the stored tokens (disconnect). + func signOut() +} +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/OuraCredentialsTests CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add Strand/Oura/OuraCredentials.swift Strand/Oura/AuthProvider.swift StrandTests/OuraCredentialsTests.swift +git commit -m "feat(oura): OuraCredentials (Info.plist) + AuthProvider seam" +``` + +--- + +### Task 3: OAuth pure builders + `OuraOAuthProvider` + +The authorization-code flow: pure URL/body builders + token-response parser (unit-tested), plus the provider that runs `ASWebAuthenticationSession` and the token-exchange POST. + +**Files:** +- Create: `Strand/Oura/OuraOAuth.swift` (pure builders/parser) +- Create: `Strand/Oura/OuraOAuthProvider.swift` (`AuthProvider` impl) +- Test: `StrandTests/OuraOAuthTests.swift` + +**Interfaces:** +- Consumes: `OuraCredentials`, `OuraTokens`, `OuraTokenStore`, `OuraError`, `AuthProvider`. +- Produces: + - `enum OuraOAuth` with pure statics: `authorizeURL(credentials:state:) -> URL`, `tokenExchangeRequest(credentials:code:) -> URLRequest`, `refreshRequest(credentials:refreshToken:) -> URLRequest`, `parseTokenResponse(_ data: Data, now: Date) throws -> OuraTokens`, and `static let scopes: [String]`. + - `final class OuraOAuthProvider: AuthProvider` (init `credentials:session:`). + +- [ ] **Step 1: Write the failing test** + +Create `StrandTests/OuraOAuthTests.swift`: + +```swift +import XCTest +@testable import Strand + +final class OuraOAuthTests: XCTestCase { + private let creds = OuraCredentials(clientId: "cid", clientSecret: "sec", + redirectURI: "noop://oura/callback") + + func testAuthorizeURLHasRequiredParams() throws { + let url = OuraOAuth.authorizeURL(credentials: creds, state: "xyz") + let comps = try XCTUnwrap(URLComponents(url: url, resolvingAgainstBaseURL: false)) + XCTAssertEqual(comps.host, "cloud.ouraring.com") + XCTAssertEqual(comps.path, "/oauth/authorize") + let q = Dictionary(uniqueKeysWithValues: (comps.queryItems ?? []).map { ($0.name, $0.value) }) + XCTAssertEqual(q["response_type"], "code") + XCTAssertEqual(q["client_id"], "cid") + XCTAssertEqual(q["redirect_uri"], "noop://oura/callback") + XCTAssertEqual(q["state"], "xyz") + XCTAssertEqual(q["scope"], OuraOAuth.scopes.joined(separator: " ")) + } + + func testTokenExchangeRequestIsFormPost() throws { + let req = OuraOAuth.tokenExchangeRequest(credentials: creds, code: "the-code") + XCTAssertEqual(req.url?.absoluteString, "https://api.ouraring.com/oauth/token") + XCTAssertEqual(req.httpMethod, "POST") + XCTAssertEqual(req.value(forHTTPHeaderField: "Content-Type"), "application/x-www-form-urlencoded") + let body = String(data: req.httpBody ?? Data(), encoding: .utf8) ?? "" + XCTAssertTrue(body.contains("grant_type=authorization_code")) + XCTAssertTrue(body.contains("code=the-code")) + XCTAssertTrue(body.contains("client_id=cid")) + XCTAssertTrue(body.contains("client_secret=sec")) + } + + func testParseTokenResponseComputesExpiry() throws { + let now = Date(timeIntervalSince1970: 1_000_000) + let json = #"{"access_token":"acc","refresh_token":"ref","expires_in":86400}"#.data(using: .utf8)! + let tokens = try OuraOAuth.parseTokenResponse(json, now: now) + XCTAssertEqual(tokens.accessToken, "acc") + XCTAssertEqual(tokens.refreshToken, "ref") + XCTAssertEqual(tokens.expiresAt, now.addingTimeInterval(86400)) + } + + func testParseTokenResponseThrowsOnMissingAccessToken() { + let json = #"{"error":"invalid_grant"}"#.data(using: .utf8)! + XCTAssertThrowsError(try OuraOAuth.parseTokenResponse(json, now: Date())) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `xcodegen generate && xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/OuraOAuthTests CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20` +Expected: FAIL — `OuraOAuth` undefined. + +- [ ] **Step 3: Implement the pure builders/parser** + +Create `Strand/Oura/OuraOAuth.swift`: + +```swift +import Foundation + +/// Pure (network-free) pieces of the Oura OAuth2 authorization-code flow: URL/request builders and the +/// token-response parser. Unit-tested directly; the interactive step + the actual POST live in +/// OuraOAuthProvider. No PKCE (Oura's flow doesn't support it), so the exchange carries the client_secret. +enum OuraOAuth { + static let authorizeEndpoint = URL(string: "https://cloud.ouraring.com/oauth/authorize")! + static let tokenEndpoint = URL(string: "https://api.ouraring.com/oauth/token")! + static let scopes = ["email", "personal", "daily", "heartrate", "workout", "tag", "session", "spo2Daily"] + + /// The consent URL to open in ASWebAuthenticationSession. `state` is a caller-generated nonce echoed + /// back on redirect and verified, to defeat CSRF / stray callbacks. + static func authorizeURL(credentials: OuraCredentials, state: String) -> URL { + var comps = URLComponents(url: authorizeEndpoint, resolvingAgainstBaseURL: false)! + comps.queryItems = [ + .init(name: "response_type", value: "code"), + .init(name: "client_id", value: credentials.clientId), + .init(name: "redirect_uri", value: credentials.redirectURI), + .init(name: "scope", value: scopes.joined(separator: " ")), + .init(name: "state", value: state), + ] + return comps.url! + } + + static func tokenExchangeRequest(credentials: OuraCredentials, code: String) -> URLRequest { + form(tokenEndpoint, [ + "grant_type": "authorization_code", + "code": code, + "redirect_uri": credentials.redirectURI, + "client_id": credentials.clientId, + "client_secret": credentials.clientSecret, + ]) + } + + static func refreshRequest(credentials: OuraCredentials, refreshToken: String) -> URLRequest { + form(tokenEndpoint, [ + "grant_type": "refresh_token", + "refresh_token": refreshToken, + "client_id": credentials.clientId, + "client_secret": credentials.clientSecret, + ]) + } + + /// Parse a token endpoint response. `now` is injected so expiry is deterministic in tests. + static func parseTokenResponse(_ data: Data, now: Date) throws -> OuraTokens { + guard let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let access = obj["access_token"] as? String, !access.isEmpty else { + let detail = String(data: data, encoding: .utf8) ?? "" + throw OuraError.tokenExchangeFailed(detail) + } + let refresh = obj["refresh_token"] as? String + let expiresIn = (obj["expires_in"] as? Double) ?? (obj["expires_in"] as? Int).map(Double.init) ?? 2_592_000 + return OuraTokens(accessToken: access, refreshToken: refresh, + expiresAt: now.addingTimeInterval(expiresIn)) + } + + private static func form(_ url: URL, _ fields: [String: String]) -> URLRequest { + var req = URLRequest(url: url) + req.httpMethod = "POST" + req.setValue("application/x-www-form-urlencoded", forHTTPHeaderField: "Content-Type") + var comps = URLComponents() + comps.queryItems = fields.map { URLQueryItem(name: $0.key, value: $0.value) } + req.httpBody = comps.percentEncodedQuery?.data(using: .utf8) + return req + } +} +``` + +- [ ] **Step 4: Implement `OuraOAuthProvider`** + +Create `Strand/Oura/OuraOAuthProvider.swift`: + +```swift +import Foundation +import AuthenticationServices + +/// BYO-app OAuth2 authorization-code AuthProvider. Runs ASWebAuthenticationSession for the interactive +/// consent, exchanges the code for tokens (Keychain), and refreshes on demand. The `session` is injected +/// so the token exchange/refresh POSTs are URLProtocol-testable; the interactive step is integration-only. +final class OuraOAuthProvider: NSObject, AuthProvider, ASWebAuthenticationPresentationContextProviding { + private let credentials: OuraCredentials + private let session: URLSession + private var anchor: ASPresentationAnchor? + + init(credentials: OuraCredentials, session: URLSession = .shared) { + self.credentials = credentials + self.session = session + } + + var isConnected: Bool { OuraTokenStore.isConnected } + func signOut() { OuraTokenStore.clear() } + + func validAccessToken() async throws -> String { + guard let tokens = OuraTokenStore.load() else { throw OuraError.notConnected } + if !tokens.isExpired { return tokens.accessToken } + guard let refresh = tokens.refreshToken else { throw OuraError.notConnected } + let req = OuraOAuth.refreshRequest(credentials: credentials, refreshToken: refresh) + let refreshed = try await exchange(req) + _ = OuraTokenStore.save(refreshed) + return refreshed.accessToken + } + + @MainActor + func authorize(presentationAnchor: ASPresentationAnchor) async throws { + self.anchor = presentationAnchor + let state = UUID().uuidString + let url = OuraOAuth.authorizeURL(credentials: credentials, state: state) + let scheme = URL(string: credentials.redirectURI)?.scheme + + let callback: URL = try await withCheckedThrowingContinuation { cont in + let webSession = ASWebAuthenticationSession(url: url, callbackURLScheme: scheme) { url, err in + if let url { cont.resume(returning: url) } + else { cont.resume(throwing: OuraError.authFailed(err?.localizedDescription ?? "cancelled")) } + } + webSession.presentationContextProvider = self + webSession.prefersEphemeralWebBrowserSession = false + if !webSession.start() { cont.resume(throwing: OuraError.authFailed("couldn't start web session")) } + } + + let items = URLComponents(url: callback, resolvingAgainstBaseURL: false)?.queryItems ?? [] + let q = Dictionary(uniqueKeysWithValues: items.map { ($0.name, $0.value ?? "") }) + guard q["state"] == state else { throw OuraError.authFailed("state mismatch") } + guard let code = q["code"], !code.isEmpty else { throw OuraError.authFailed(q["error"] ?? "no code") } + + let tokens = try await exchange(OuraOAuth.tokenExchangeRequest(credentials: credentials, code: code)) + guard OuraTokenStore.save(tokens) else { throw OuraError.tokenExchangeFailed("keychain write failed") } + } + + /// POST a token request and parse the response (shared by exchange + refresh). + private func exchange(_ req: URLRequest) async throws -> OuraTokens { + let (data, resp): (Data, URLResponse) + do { (data, resp) = try await session.data(for: req) } + catch { throw OuraError.network(error.localizedDescription) } + let code = (resp as? HTTPURLResponse)?.statusCode ?? 0 + guard (200..<300).contains(code) else { + throw OuraError.tokenExchangeFailed("HTTP \(code): \(String(data: data, encoding: .utf8) ?? "")") + } + return try OuraOAuth.parseTokenResponse(data, now: Date()) + } + + func presentationAnchor(for session: ASWebAuthenticationSession) -> ASPresentationAnchor { + anchor ?? ASPresentationAnchor() + } +} +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/OuraOAuthTests CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20` +Expected: PASS (4 tests). Note in the report: `OuraOAuthProvider.authorize` (the interactive ASWebAuthenticationSession step) is compile-verified here but exercised only in live/integration testing — its pure inputs (`authorizeURL`, exchange request, response parse) and the exchange POST path are what the tests cover. + +- [ ] **Step 6: Commit** + +```bash +git add Strand/Oura/OuraOAuth.swift Strand/Oura/OuraOAuthProvider.swift StrandTests/OuraOAuthTests.swift +git commit -m "feat(oura): OAuth2 authorization-code flow (pure builders + OuraOAuthProvider)" +``` + +--- + +### Task 4: `OuraAPIClient` — paging + backoff + +The `URLSession` client: per-endpoint fetch, `next_token` paging, 429 backoff, 401 refresh-once, prod/sandbox base URL. Pure envelope parser split out for unit tests; the networked paths driven by a `URLProtocol` stub. + +**Files:** +- Create: `Strand/Oura/OuraAPIClient.swift` +- Create: `StrandTests/OuraURLProtocolStub.swift` (reusable stub) +- Test: `StrandTests/OuraAPIClientTests.swift` + +**Interfaces:** +- Consumes: `AuthProvider`, `OuraError`. +- Produces: + - `enum OuraEnvironment { case production, sandbox; var baseURL: String }` + - `struct OuraPage { let data: [[String: Any]]; let nextToken: String? }` — intentionally NOT `Equatable` (`[[String: Any]]` isn't); tests assert on `.count`/fields. + - `enum OuraAPI` pure: `static func parsePage(_ data: Data) throws -> OuraPage`. + - `final class OuraAPIClient` init `auth:environment:session:` with `func fetchAll(endpoint: String, query: [String: String]) async throws -> [[String: Any]]` (follows `next_token`, returns concatenated `data`), and `func fetchAllRaw(endpoint:query:) async throws -> [Data]` (the verbatim page bodies for Plan 3's raw archive). + +- [ ] **Step 1: Write the failing test + the URLProtocol stub** + +Create `StrandTests/OuraURLProtocolStub.swift`: + +```swift +import Foundation + +/// Test-only URLProtocol that serves canned responses by request order, so the paging loop and +/// status-code handling are driven deterministically without a live socket. +final class OuraURLProtocolStub: URLProtocol { + struct Stubbed { let status: Int; let body: Data } + /// FIFO queue of responses; each request pops the next. Set before the test acts. + nonisolated(unsafe) static var queue: [Stubbed] = [] + /// Captured request URLs, in order, for assertions. + nonisolated(unsafe) static var requestedURLs: [URL] = [] + + static func reset() { queue = []; requestedURLs = [] } + + override class func canInit(with request: URLRequest) -> Bool { true } + override class func canonicalRequest(for request: URLRequest) -> URLRequest { request } + override func startLoading() { + if let u = request.url { Self.requestedURLs.append(u) } + let s = Self.queue.isEmpty ? Stubbed(status: 500, body: Data()) : Self.queue.removeFirst() + let resp = HTTPURLResponse(url: request.url!, statusCode: s.status, httpVersion: nil, headerFields: nil)! + client?.urlProtocol(self, didReceive: resp, cacheStoragePolicy: .notAllowed) + client?.urlProtocol(self, didLoad: s.body) + client?.urlProtocolDidFinishLoading(self) + } + override func stopLoading() {} + + /// A URLSession wired to this stub. + static func session() -> URLSession { + let cfg = URLSessionConfiguration.ephemeral + cfg.protocolClasses = [OuraURLProtocolStub.self] + return URLSession(configuration: cfg) + } +} +``` + +Create `StrandTests/OuraAPIClientTests.swift`: + +```swift +import XCTest +import AuthenticationServices +@testable import Strand + +/// A stub AuthProvider that hands out a fixed token and counts refreshes. +private final class StubAuth: AuthProvider { + var token = "tok-1" + var validCalls = 0 + var isConnected: Bool { true } + func validAccessToken() async throws -> String { validCalls += 1; return token } + @MainActor func authorize(presentationAnchor: ASPresentationAnchor) async throws {} + func signOut() {} +} + +final class OuraAPIClientTests: XCTestCase { + override func setUp() { super.setUp(); OuraURLProtocolStub.reset() } + + func testParsePageExtractsDataAndNextToken() throws { + let json = #"{"data":[{"id":"a"},{"id":"b"}],"next_token":"tok2"}"#.data(using: .utf8)! + let page = try OuraAPI.parsePage(json) + XCTAssertEqual(page.data.count, 2) + XCTAssertEqual(page.nextToken, "tok2") + } + + func testFetchAllFollowsNextTokenThenStops() async throws { + OuraURLProtocolStub.queue = [ + .init(status: 200, body: #"{"data":[{"id":"a"}],"next_token":"t2"}"#.data(using: .utf8)!), + .init(status: 200, body: #"{"data":[{"id":"b"}],"next_token":null}"#.data(using: .utf8)!), + ] + let client = OuraAPIClient(auth: StubAuth(), environment: .production, + session: OuraURLProtocolStub.session()) + let rows = try await client.fetchAll(endpoint: "daily_sleep", query: ["start_date": "2026-01-01"]) + XCTAssertEqual(rows.count, 2) + // Second request carried next_token=t2. + XCTAssertTrue(OuraURLProtocolStub.requestedURLs.last?.absoluteString.contains("next_token=t2") ?? false) + } + + func test429IsRetriedAfterBackoff() async throws { + OuraURLProtocolStub.queue = [ + .init(status: 429, body: Data()), + .init(status: 200, body: #"{"data":[{"id":"a"}],"next_token":null}"#.data(using: .utf8)!), + ] + let client = OuraAPIClient(auth: StubAuth(), environment: .production, + session: OuraURLProtocolStub.session(), backoff: 0) + let rows = try await client.fetchAll(endpoint: "daily_sleep", query: [:]) + XCTAssertEqual(rows.count, 1) + } + + func testSandboxBaseURL() async throws { + OuraURLProtocolStub.queue = [.init(status: 200, + body: #"{"data":[],"next_token":null}"#.data(using: .utf8)!)] + let client = OuraAPIClient(auth: StubAuth(), environment: .sandbox, + session: OuraURLProtocolStub.session()) + _ = try await client.fetchAll(endpoint: "personal_info", query: [:]) + XCTAssertTrue(OuraURLProtocolStub.requestedURLs.first?.absoluteString.contains("/v2/sandbox/") ?? false) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `xcodegen generate && xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/OuraAPIClientTests CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20` +Expected: FAIL — `OuraAPIClient` / `OuraAPI` undefined. + +- [ ] **Step 3: Implement `OuraAPIClient`** + +Create `Strand/Oura/OuraAPIClient.swift`: + +```swift +import Foundation + +enum OuraEnvironment { + case production, sandbox + var baseURL: String { + switch self { + case .production: return "https://api.ouraring.com/v2/usercollection" + case .sandbox: return "https://api.ouraring.com/v2/sandbox/usercollection" + } + } +} + +/// One decoded page of an Oura list/time-series response. +struct OuraPage { + let data: [[String: Any]] + let nextToken: String? +} + +/// Pure (network-free) response-envelope parsing, unit-tested directly. +enum OuraAPI { + static func parsePage(_ data: Data) throws -> OuraPage { + guard let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { + throw OuraError.decode + } + let rows = (obj["data"] as? [[String: Any]]) ?? [] + return OuraPage(data: rows, nextToken: obj["next_token"] as? String) + } +} + +/// The Oura v2 API client. Injected `AuthProvider` supplies the bearer token (refreshing on expiry); +/// injected `session` makes it URLProtocol-testable. Follows `next_token` paging, retries 429 with a +/// bounded backoff, and refreshes-once on 401. Networking lives here in the app target by design. +final class OuraAPIClient { + private let auth: AuthProvider + private let environment: OuraEnvironment + private let session: URLSession + private let backoff: TimeInterval + private let maxPages = 10_000 // runaway guard (logged if hit) + + init(auth: AuthProvider, environment: OuraEnvironment = .production, + session: URLSession = .shared, backoff: TimeInterval = 2) { + self.auth = auth + self.environment = environment + self.session = session + self.backoff = backoff + } + + /// Fetch every page of `endpoint` for `query`, following `next_token`, returning concatenated rows. + func fetchAll(endpoint: String, query: [String: String]) async throws -> [[String: Any]] { + var rows: [[String: Any]] = [] + try await forEachPage(endpoint: endpoint, query: query) { page in rows.append(contentsOf: page.data) } + return rows + } + + /// Fetch every page, returning the verbatim page bodies (for Plan 3's lossless raw archive). + func fetchAllRaw(endpoint: String, query: [String: String]) async throws -> [Data] { + var bodies: [Data] = [] + try await forEachPage(endpoint: endpoint, query: query, rawSink: { bodies.append($0) }) { _ in } + return bodies + } + + private func forEachPage(endpoint: String, query: [String: String], + rawSink: ((Data) -> Void)? = nil, + _ handle: (OuraPage) -> Void) async throws { + var next: String? = nil + var pages = 0 + repeat { + var q = query + if let next { q["next_token"] = next } + let (data, _) = try await getWithRetry(endpoint: endpoint, query: q) + rawSink?(data) + let page = try OuraAPI.parsePage(data) + handle(page) + next = page.nextToken + pages += 1 + if pages >= maxPages { break } // Plan-3 note: log if this cap is ever hit. + } while next != nil + } + + /// GET with auth + 429-backoff + 401-refresh-once. Returns (body, status). + private func getWithRetry(endpoint: String, query: [String: String]) async throws -> (Data, Int) { + for attempt in 0..<3 { + let token = try await auth.validAccessToken() + let (data, code) = try await get(endpoint: endpoint, query: query, token: token) + switch code { + case 200..<300: + return (data, code) + case 429: + if backoff > 0 { try? await Task.sleep(nanoseconds: UInt64(backoff * 1_000_000_000)) } + continue + case 401 where attempt == 0: + _ = try? await auth.validAccessToken() // force a refresh, then retry once + continue + default: + throw OuraError.badResponse(code, String(data: data, encoding: .utf8) ?? "") + } + } + throw OuraError.rateLimited + } + + private func get(endpoint: String, query: [String: String], token: String) async throws -> (Data, Int) { + var comps = URLComponents(string: "\(environment.baseURL)/\(endpoint)")! + if !query.isEmpty { comps.queryItems = query.map { URLQueryItem(name: $0.key, value: $0.value) } } + var req = URLRequest(url: comps.url!) + req.setValue("Bearer \(token)", forHTTPHeaderField: "Authorization") + do { + let (data, resp) = try await session.data(for: req) + return (data, (resp as? HTTPURLResponse)?.statusCode ?? 0) + } catch { + throw OuraError.network(error.localizedDescription) + } + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/OuraAPIClientTests CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20` +Expected: PASS (4 tests). + +- [ ] **Step 5: Commit** + +```bash +git add Strand/Oura/OuraAPIClient.swift StrandTests/OuraURLProtocolStub.swift StrandTests/OuraAPIClientTests.swift +git commit -m "feat(oura): OuraAPIClient — next_token paging, 429 backoff, 401 refresh, sandbox/prod" +``` + +--- + +### Task 5: Wiring — Info.plist redirect scheme + xcconfig credentials + +Register the OAuth redirect URL scheme and wire the untracked credentials xcconfig. No unit test (build config); verify the app still generates + builds. + +**Files:** +- Modify: `project.yml` (add `CFBundleURLTypes` to the `Strand` and `NOOPiOS` targets' Info settings + reference the xcconfig) +- Create: `Strand/Oura/OuraSecrets.example.xcconfig` (committed template) +- Modify: `.gitignore` (ignore `Strand/Oura/OuraSecrets.xcconfig`) + +**Interfaces:** none (configuration). Provides the Info.plist keys `OURA_CLIENT_ID`/`OURA_CLIENT_SECRET`/`OURA_REDIRECT_URI` and the `noop` URL scheme that Tasks 2–3 read. + +- [ ] **Step 1: Add the example xcconfig** + +Create `Strand/Oura/OuraSecrets.example.xcconfig`: + +``` +// Copy to OuraSecrets.xcconfig (gitignored) and fill in your own Oura OAuth app's values. +// Register a free app at the Oura developer portal; set the redirect URI to noop://oura/callback. +OURA_CLIENT_ID = your_client_id_here +OURA_CLIENT_SECRET = your_client_secret_here +OURA_REDIRECT_URI = noop://oura/callback +``` + +- [ ] **Step 2: Ignore the real xcconfig** + +Add to `.gitignore` (append): + +``` +# Oura BYO-app OAuth credentials (never committed) +Strand/Oura/OuraSecrets.xcconfig +``` + +- [ ] **Step 3: Wire the xcconfig + URL scheme in `project.yml`** + +Under the `Strand` target (and the same under `NOOPiOS`), reference the xcconfig via `configFiles` and add the Info.plist keys. Add to the target's `settings` / `info` block: + +```yaml + # (within target: Strand) + configFiles: + Debug: Strand/Oura/OuraSecrets.xcconfig + Release: Strand/Oura/OuraSecrets.xcconfig + info: + path: Strand/Resources/Info.plist + properties: + OURA_CLIENT_ID: $(OURA_CLIENT_ID) + OURA_CLIENT_SECRET: $(OURA_CLIENT_SECRET) + OURA_REDIRECT_URI: $(OURA_REDIRECT_URI) + CFBundleURLTypes: + - CFBundleURLSchemes: [noop] +``` + +(If `configFiles` already exists on the target, merge these keys in. The exact YAML location must match the existing `project.yml` structure — read it first.) + +- [ ] **Step 4: Verify generate + build** + +Create an `OuraSecrets.xcconfig` with placeholder values (so the build resolves the variables), then: + +Run: `xcodegen generate && xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' build-for-testing CODE_SIGNING_ALLOWED=NO 2>&1 | tail -15` +Expected: `** TEST BUILD SUCCEEDED **`. Then confirm the Info.plist keys resolve: the `OuraCredentials.fromBundle` path will read them at runtime. + +- [ ] **Step 5: Commit** + +```bash +git add project.yml .gitignore Strand/Oura/OuraSecrets.example.xcconfig +git commit -m "chore(oura): wire OAuth redirect scheme + BYO-app credentials xcconfig" +``` + +--- + +## Plan 2 self-review + +- **Spec coverage:** §5 auth (BYO-app, ASWebAuthenticationSession, Keychain, AuthProvider seam) → T1/T2/T3; §6 API client (paging, 429/401, prod/sandbox, bounded) → T4; §5.3 credentials (xcconfig→Info.plist) → T2/T5; redirect scheme → T5. `fetchAllRaw` (T4) is the seam Plan 3's writer consumes for the lossless archive. +- **Constraints:** networking confined to `Strand/Oura/`; no package touched; tokens Keychain-only; `session` injected everywhere (URLProtocol-testable); no PKCE (secret in the exchange, sourced from the user's own untracked xcconfig). +- **Interactive boundary is honest:** `OuraOAuthProvider.authorize`'s ASWebAuthenticationSession step is compile-verified + integration-tested, not unit-tested; everything around it (URL/body builders, token parse, exchange POST, paging, backoff, refresh) IS unit-tested via pure functions + the URLProtocol stub. +- **Type consistency:** `OuraTokens`/`OuraCredentials`/`OuraError` defined in T1/T2 and consumed unchanged in T3/T4; `AuthProvider` defined T2, implemented T3, injected T4; `OuraPage` is intentionally NOT `Equatable` (holds `[[String: Any]]`) — tests assert on fields. +- **Build reality:** every test step uses `xcodebuild -only-testing:StrandTests/` (first build full, then incremental); `swift test` is NOT used here (app target, not a package). + +## What Plan 3 adds (next) + +`OuraSyncCoordinator` (one-time backfill across all ~18 endpoints via `OuraAPIClient.fetchAllRaw` + parse via Plan 1's `OuraApiParser`) → `OuraSyncWriter` (raw archive via `upsertOuraRaw`; normalized via `upsertDailyMetrics`/`upsertSleepSessions`(+stagesJSON via decoded stages, +`persistSessionMotion`)/`upsertMetricSeries`/`insert(hrSample)`/`upsertWorkouts`; register `PairedDevice(sourceKind: .cloudImport)`; **coalesce** per-day rows, prefer readiness RHR, map sleep-score/vo2max onto the daily columns — per the Plan-1 final-review advisories) → the "Connect Oura & Import Everything" UI on `DataSourcesView` + progress/summary/disconnect → doc updates (`PRIVACY_SECURITY.md` second opt-in lane, `DEVICE_SUPPORT_ROADMAP.md`, `DATA_MODEL.md` v24 `ouraRaw`). From 78e325098058cceccfdc21a0e47547a33e66f31a Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 19:44:45 -0700 Subject: [PATCH 012/171] feat(oura): OuraError + OuraTokenStore (Keychain token storage) --- Strand/Oura/OuraError.swift | 33 +++++++++++++++ Strand/Oura/OuraTokenStore.swift | 59 +++++++++++++++++++++++++++ StrandTests/OuraTokenStoreTests.swift | 33 +++++++++++++++ 3 files changed, 125 insertions(+) create mode 100644 Strand/Oura/OuraError.swift create mode 100644 Strand/Oura/OuraTokenStore.swift create mode 100644 StrandTests/OuraTokenStoreTests.swift diff --git a/Strand/Oura/OuraError.swift b/Strand/Oura/OuraError.swift new file mode 100644 index 0000000000..83ec7fe8a2 --- /dev/null +++ b/Strand/Oura/OuraError.swift @@ -0,0 +1,33 @@ +import Foundation + +/// User-facing failure reasons for the Oura cloud-import lane, mapped to clear, non-crashing messages. +/// Mirrors AICoachError's shape (LocalizedError + errorDescription). +enum OuraError: LocalizedError, Equatable { + case notConnected + case authFailed(String) + case tokenExchangeFailed(String) + case badResponse(Int, String) + case rateLimited + case network(String) + case decode + + var errorDescription: String? { + switch self { + case .notConnected: + return "Connect your Oura account first." + case .authFailed(let d): + return "Oura sign-in didn't complete: \(d)" + case .tokenExchangeFailed(let d): + return "Couldn't exchange the Oura authorization code: \(d)" + case .badResponse(let code, let detail): + let extra = detail.isEmpty ? "" : " — \(detail)" + return "Oura returned an error (\(code))\(extra)." + case .rateLimited: + return "Oura is rate-limiting requests. Waiting before retrying." + case .network(let d): + return "Network problem talking to Oura: \(d)" + case .decode: + return "Couldn't read Oura's response." + } + } +} diff --git a/Strand/Oura/OuraTokenStore.swift b/Strand/Oura/OuraTokenStore.swift new file mode 100644 index 0000000000..9df4058804 --- /dev/null +++ b/Strand/Oura/OuraTokenStore.swift @@ -0,0 +1,59 @@ +import Foundation +import Security + +/// The OAuth tokens for the Oura cloud lane. `expiresAt` is absolute (computed from `expires_in` at +/// exchange time). `isExpired` applies a 60 s skew so we refresh slightly early rather than mid-request. +struct OuraTokens: Equatable, Codable { + let accessToken: String + let refreshToken: String? + let expiresAt: Date + + var isExpired: Bool { Date() >= expiresAt.addingTimeInterval(-60) } +} + +/// Keychain Services wrapper for the Oura tokens. One generic-password item (JSON-encoded `OuraTokens`) +/// under a fixed service, so tokens never land in UserDefaults, a plist, or on disk in the clear. +/// Mirrors AIKeyStore exactly (delete-then-add, kSecAttrAccessibleAfterFirstUnlock). +enum OuraTokenStore { + private static let service = "com.noop.oura" + private static let account = "oauth-tokens" + + private static var baseQuery: [String: Any] { + [ + kSecClass as String: kSecClassGenericPassword, + kSecAttrService as String: service, + kSecAttrAccount as String: account, + ] + } + + /// Store (or replace) the tokens. Returns false if the Keychain write fails (so callers never assume + /// a token is stored when it isn't). + @discardableResult + static func save(_ tokens: OuraTokens) -> Bool { + guard let data = try? JSONEncoder().encode(tokens) else { return false } + SecItemDelete(baseQuery as CFDictionary) + var attrs = baseQuery + attrs[kSecValueData as String] = data + attrs[kSecAttrAccessible as String] = kSecAttrAccessibleAfterFirstUnlock + return SecItemAdd(attrs as CFDictionary, nil) == errSecSuccess + } + + /// Load the stored tokens, or nil if none. + static func load() -> OuraTokens? { + var query = baseQuery + query[kSecReturnData as String] = kCFBooleanTrue + query[kSecMatchLimit as String] = kSecMatchLimitOne + var item: CFTypeRef? + guard SecItemCopyMatching(query as CFDictionary, &item) == errSecSuccess, + let data = item as? Data, + let tokens = try? JSONDecoder().decode(OuraTokens.self, from: data) else { return nil } + return tokens + } + + /// Remove any stored tokens. + static func clear() { SecItemDelete(baseQuery as CFDictionary) } + + /// True when tokens are present (regardless of expiry — an expired token is still "connected", it + /// just needs a refresh). + static var isConnected: Bool { load() != nil } +} diff --git a/StrandTests/OuraTokenStoreTests.swift b/StrandTests/OuraTokenStoreTests.swift new file mode 100644 index 0000000000..f0a69aeafe --- /dev/null +++ b/StrandTests/OuraTokenStoreTests.swift @@ -0,0 +1,33 @@ +import XCTest +@testable import Strand + +final class OuraTokenStoreTests: XCTestCase { + override func setUp() { super.setUp(); OuraTokenStore.clear() } + override func tearDown() { OuraTokenStore.clear(); super.tearDown() } + + func testSaveLoadClearRoundTrip() { + XCTAssertFalse(OuraTokenStore.isConnected) + XCTAssertNil(OuraTokenStore.load()) + + let exp = Date(timeIntervalSince1970: 2_000_000_000) + let t = OuraTokens(accessToken: "acc", refreshToken: "ref", expiresAt: exp) + XCTAssertTrue(OuraTokenStore.save(t)) + XCTAssertTrue(OuraTokenStore.isConnected) + + let loaded = OuraTokenStore.load() + XCTAssertEqual(loaded, t) + + OuraTokenStore.clear() + XCTAssertNil(OuraTokenStore.load()) + XCTAssertFalse(OuraTokenStore.isConnected) + } + + func testExpiryUsesSkew() { + let almostNow = OuraTokens(accessToken: "a", refreshToken: nil, + expiresAt: Date(timeIntervalSinceNow: 30)) // <60 s away + XCTAssertTrue(almostNow.isExpired) + let later = OuraTokens(accessToken: "a", refreshToken: nil, + expiresAt: Date(timeIntervalSinceNow: 600)) + XCTAssertFalse(later.isExpired) + } +} From 2320ad153c08f56f5190cb2bd7694d64b581196a Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 19:49:49 -0700 Subject: [PATCH 013/171] fix(build): StrandTests TEST_HOST -> 'NOOP Staging' (macOS xcodebuild test was broken) The app PRODUCT_NAME is 'NOOP Staging' but the StrandTests TEST_HOST/BUNDLE_LOADER still pointed at NOOP.app/NOOP, so xcodebuild test failed with 'Could not find test host' for the whole macOS scheme. CI only runs build (never test) so it went unnoticed. --- project.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/project.yml b/project.yml index 6f827fbd07..38f2aca34f 100644 --- a/project.yml +++ b/project.yml @@ -142,7 +142,7 @@ targets: GENERATE_INFOPLIST_FILE: YES PRODUCT_BUNDLE_IDENTIFIER: com.noopapp.strandtests # Host the unit tests in the app so `@testable import Strand` links against the real app module. - TEST_HOST: "$(BUILT_PRODUCTS_DIR)/NOOP.app/Contents/MacOS/NOOP" + TEST_HOST: "$(BUILT_PRODUCTS_DIR)/NOOP Staging.app/Contents/MacOS/NOOP Staging" BUNDLE_LOADER: "$(TEST_HOST)" # --------------------------------------------------------------------------- From ff7013db833f273478e55bda7ad7bb09c8c6f900 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 19:55:02 -0700 Subject: [PATCH 014/171] feat(oura): OuraCredentials (Info.plist) + AuthProvider seam --- Strand/Oura/AuthProvider.swift | 15 +++++++++++++++ Strand/Oura/OuraCredentials.swift | 26 ++++++++++++++++++++++++++ StrandTests/OuraCredentialsTests.swift | 21 +++++++++++++++++++++ 3 files changed, 62 insertions(+) create mode 100644 Strand/Oura/AuthProvider.swift create mode 100644 Strand/Oura/OuraCredentials.swift create mode 100644 StrandTests/OuraCredentialsTests.swift diff --git a/Strand/Oura/AuthProvider.swift b/Strand/Oura/AuthProvider.swift new file mode 100644 index 0000000000..d42311e75f --- /dev/null +++ b/Strand/Oura/AuthProvider.swift @@ -0,0 +1,15 @@ +import Foundation +import AuthenticationServices + +/// The auth seam. Today's concrete provider is `OuraOAuthProvider` (BYO-app authorization-code, Task 3); +/// a future backend-mediated provider can replace it without touching the API client or the sync layer. +protocol AuthProvider { + /// True when tokens are stored (connected), regardless of expiry. + var isConnected: Bool { get } + /// Return a currently-valid access token, refreshing transparently if the stored one is expired. + func validAccessToken() async throws -> String + /// Run the interactive authorization (opens Oura's consent page) and store the resulting tokens. + @MainActor func authorize(presentationAnchor: ASPresentationAnchor) async throws + /// Forget the stored tokens (disconnect). + func signOut() +} diff --git a/Strand/Oura/OuraCredentials.swift b/Strand/Oura/OuraCredentials.swift new file mode 100644 index 0000000000..83dfed0216 --- /dev/null +++ b/Strand/Oura/OuraCredentials.swift @@ -0,0 +1,26 @@ +import Foundation + +/// The user's own Oura OAuth app credentials, injected at build time via an untracked xcconfig → +/// Info.plist (Task 5). Absent credentials mean the Connect flow is unavailable (surfaced in the UI). +struct OuraCredentials: Equatable { + let clientId: String + let clientSecret: String + let redirectURI: String + + /// Build from an Info-dictionary-shaped map. Returns nil unless all three keys are present and + /// non-blank (so a build without the xcconfig cleanly disables the lane rather than half-configuring). + static func from(_ info: [String: Any]) -> OuraCredentials? { + func nonBlank(_ key: String) -> String? { + guard let s = (info[key] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines), + !s.isEmpty else { return nil } + return s + } + guard let id = nonBlank("OURA_CLIENT_ID"), + let secret = nonBlank("OURA_CLIENT_SECRET"), + let redirect = nonBlank("OURA_REDIRECT_URI") else { return nil } + return OuraCredentials(clientId: id, clientSecret: secret, redirectURI: redirect) + } + + /// The live credentials from the app bundle's Info.plist, or nil if not configured. + static var fromBundle: OuraCredentials? { from(Bundle.main.infoDictionary ?? [:]) } +} diff --git a/StrandTests/OuraCredentialsTests.swift b/StrandTests/OuraCredentialsTests.swift new file mode 100644 index 0000000000..46cacdac0c --- /dev/null +++ b/StrandTests/OuraCredentialsTests.swift @@ -0,0 +1,21 @@ +import XCTest +@testable import Strand + +final class OuraCredentialsTests: XCTestCase { + func testParsesCompleteInfoDict() { + let info: [String: Any] = [ + "OURA_CLIENT_ID": "cid", "OURA_CLIENT_SECRET": "secret", + "OURA_REDIRECT_URI": "noop://oura/callback", + ] + let c = OuraCredentials.from(info) + XCTAssertEqual(c, OuraCredentials(clientId: "cid", clientSecret: "secret", + redirectURI: "noop://oura/callback")) + } + + func testNilWhenAnyKeyMissingOrBlank() { + XCTAssertNil(OuraCredentials.from(["OURA_CLIENT_ID": "cid", "OURA_CLIENT_SECRET": "s"])) // no redirect + XCTAssertNil(OuraCredentials.from([ + "OURA_CLIENT_ID": "", "OURA_CLIENT_SECRET": "s", "OURA_REDIRECT_URI": "noop://x", + ])) // blank id + } +} From f9d55dbcd2d5c198b8e00ca0e950e3182e665191 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 20:03:31 -0700 Subject: [PATCH 015/171] feat(oura): OAuth2 authorization-code flow (pure builders + OuraOAuthProvider) --- Strand/Oura/OuraOAuth.swift | 66 +++++++++++++++++++++++++++ Strand/Oura/OuraOAuthProvider.swift | 71 +++++++++++++++++++++++++++++ StrandTests/OuraOAuthTests.swift | 46 +++++++++++++++++++ 3 files changed, 183 insertions(+) create mode 100644 Strand/Oura/OuraOAuth.swift create mode 100644 Strand/Oura/OuraOAuthProvider.swift create mode 100644 StrandTests/OuraOAuthTests.swift diff --git a/Strand/Oura/OuraOAuth.swift b/Strand/Oura/OuraOAuth.swift new file mode 100644 index 0000000000..71fc1c52c4 --- /dev/null +++ b/Strand/Oura/OuraOAuth.swift @@ -0,0 +1,66 @@ +import Foundation + +/// Pure (network-free) pieces of the Oura OAuth2 authorization-code flow: URL/request builders and the +/// token-response parser. Unit-tested directly; the interactive step + the actual POST live in +/// OuraOAuthProvider. No PKCE (Oura's flow doesn't support it), so the exchange carries the client_secret. +enum OuraOAuth { + static let authorizeEndpoint = URL(string: "https://cloud.ouraring.com/oauth/authorize")! + static let tokenEndpoint = URL(string: "https://api.ouraring.com/oauth/token")! + static let scopes = ["email", "personal", "daily", "heartrate", "workout", "tag", "session", "spo2Daily"] + + /// The consent URL to open in ASWebAuthenticationSession. `state` is a caller-generated nonce echoed + /// back on redirect and verified, to defeat CSRF / stray callbacks. + static func authorizeURL(credentials: OuraCredentials, state: String) -> URL { + var comps = URLComponents(url: authorizeEndpoint, resolvingAgainstBaseURL: false)! + comps.queryItems = [ + .init(name: "response_type", value: "code"), + .init(name: "client_id", value: credentials.clientId), + .init(name: "redirect_uri", value: credentials.redirectURI), + .init(name: "scope", value: scopes.joined(separator: " ")), + .init(name: "state", value: state), + ] + return comps.url! + } + + static func tokenExchangeRequest(credentials: OuraCredentials, code: String) -> URLRequest { + form(tokenEndpoint, [ + "grant_type": "authorization_code", + "code": code, + "redirect_uri": credentials.redirectURI, + "client_id": credentials.clientId, + "client_secret": credentials.clientSecret, + ]) + } + + static func refreshRequest(credentials: OuraCredentials, refreshToken: String) -> URLRequest { + form(tokenEndpoint, [ + "grant_type": "refresh_token", + "refresh_token": refreshToken, + "client_id": credentials.clientId, + "client_secret": credentials.clientSecret, + ]) + } + + /// Parse a token endpoint response. `now` is injected so expiry is deterministic in tests. + static func parseTokenResponse(_ data: Data, now: Date) throws -> OuraTokens { + guard let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let access = obj["access_token"] as? String, !access.isEmpty else { + let detail = String(data: data, encoding: .utf8) ?? "" + throw OuraError.tokenExchangeFailed(detail) + } + let refresh = obj["refresh_token"] as? String + let expiresIn = (obj["expires_in"] as? Double) ?? (obj["expires_in"] as? Int).map(Double.init) ?? 2_592_000 + return OuraTokens(accessToken: access, refreshToken: refresh, + expiresAt: now.addingTimeInterval(expiresIn)) + } + + private static func form(_ url: URL, _ fields: [String: String]) -> URLRequest { + var req = URLRequest(url: url) + req.httpMethod = "POST" + req.setValue("application/x-www-form-urlencoded", forHTTPHeaderField: "Content-Type") + var comps = URLComponents() + comps.queryItems = fields.map { URLQueryItem(name: $0.key, value: $0.value) } + req.httpBody = comps.percentEncodedQuery?.data(using: .utf8) + return req + } +} diff --git a/Strand/Oura/OuraOAuthProvider.swift b/Strand/Oura/OuraOAuthProvider.swift new file mode 100644 index 0000000000..c69dbd8f4d --- /dev/null +++ b/Strand/Oura/OuraOAuthProvider.swift @@ -0,0 +1,71 @@ +import Foundation +import AuthenticationServices + +/// BYO-app OAuth2 authorization-code AuthProvider. Runs ASWebAuthenticationSession for the interactive +/// consent, exchanges the code for tokens (Keychain), and refreshes on demand. The `session` is injected +/// so the token exchange/refresh POSTs are URLProtocol-testable; the interactive step is integration-only. +final class OuraOAuthProvider: NSObject, AuthProvider, ASWebAuthenticationPresentationContextProviding { + private let credentials: OuraCredentials + private let session: URLSession + private var anchor: ASPresentationAnchor? + + init(credentials: OuraCredentials, session: URLSession = .shared) { + self.credentials = credentials + self.session = session + } + + var isConnected: Bool { OuraTokenStore.isConnected } + func signOut() { OuraTokenStore.clear() } + + func validAccessToken() async throws -> String { + guard let tokens = OuraTokenStore.load() else { throw OuraError.notConnected } + if !tokens.isExpired { return tokens.accessToken } + guard let refresh = tokens.refreshToken else { throw OuraError.notConnected } + let req = OuraOAuth.refreshRequest(credentials: credentials, refreshToken: refresh) + let refreshed = try await exchange(req) + _ = OuraTokenStore.save(refreshed) + return refreshed.accessToken + } + + @MainActor + func authorize(presentationAnchor: ASPresentationAnchor) async throws { + self.anchor = presentationAnchor + let state = UUID().uuidString + let url = OuraOAuth.authorizeURL(credentials: credentials, state: state) + let scheme = URL(string: credentials.redirectURI)?.scheme + + let callback: URL = try await withCheckedThrowingContinuation { cont in + let webSession = ASWebAuthenticationSession(url: url, callbackURLScheme: scheme) { url, err in + if let url { cont.resume(returning: url) } + else { cont.resume(throwing: OuraError.authFailed(err?.localizedDescription ?? "cancelled")) } + } + webSession.presentationContextProvider = self + webSession.prefersEphemeralWebBrowserSession = false + if !webSession.start() { cont.resume(throwing: OuraError.authFailed("couldn't start web session")) } + } + + let items = URLComponents(url: callback, resolvingAgainstBaseURL: false)?.queryItems ?? [] + let q = Dictionary(uniqueKeysWithValues: items.map { ($0.name, $0.value ?? "") }) + guard q["state"] == state else { throw OuraError.authFailed("state mismatch") } + guard let code = q["code"], !code.isEmpty else { throw OuraError.authFailed(q["error"] ?? "no code") } + + let tokens = try await exchange(OuraOAuth.tokenExchangeRequest(credentials: credentials, code: code)) + guard OuraTokenStore.save(tokens) else { throw OuraError.tokenExchangeFailed("keychain write failed") } + } + + /// POST a token request and parse the response (shared by exchange + refresh). + private func exchange(_ req: URLRequest) async throws -> OuraTokens { + let (data, resp): (Data, URLResponse) + do { (data, resp) = try await session.data(for: req) } + catch { throw OuraError.network(error.localizedDescription) } + let code = (resp as? HTTPURLResponse)?.statusCode ?? 0 + guard (200..<300).contains(code) else { + throw OuraError.tokenExchangeFailed("HTTP \(code): \(String(data: data, encoding: .utf8) ?? "")") + } + return try OuraOAuth.parseTokenResponse(data, now: Date()) + } + + func presentationAnchor(for session: ASWebAuthenticationSession) -> ASPresentationAnchor { + anchor ?? ASPresentationAnchor() + } +} diff --git a/StrandTests/OuraOAuthTests.swift b/StrandTests/OuraOAuthTests.swift new file mode 100644 index 0000000000..035cbc62e7 --- /dev/null +++ b/StrandTests/OuraOAuthTests.swift @@ -0,0 +1,46 @@ +import XCTest +@testable import Strand + +final class OuraOAuthTests: XCTestCase { + private let creds = OuraCredentials(clientId: "cid", clientSecret: "sec", + redirectURI: "noop://oura/callback") + + func testAuthorizeURLHasRequiredParams() throws { + let url = OuraOAuth.authorizeURL(credentials: creds, state: "xyz") + let comps = try XCTUnwrap(URLComponents(url: url, resolvingAgainstBaseURL: false)) + XCTAssertEqual(comps.host, "cloud.ouraring.com") + XCTAssertEqual(comps.path, "/oauth/authorize") + let q = Dictionary(uniqueKeysWithValues: (comps.queryItems ?? []).map { ($0.name, $0.value) }) + XCTAssertEqual(q["response_type"], "code") + XCTAssertEqual(q["client_id"], "cid") + XCTAssertEqual(q["redirect_uri"], "noop://oura/callback") + XCTAssertEqual(q["state"], "xyz") + XCTAssertEqual(q["scope"], OuraOAuth.scopes.joined(separator: " ")) + } + + func testTokenExchangeRequestIsFormPost() throws { + let req = OuraOAuth.tokenExchangeRequest(credentials: creds, code: "the-code") + XCTAssertEqual(req.url?.absoluteString, "https://api.ouraring.com/oauth/token") + XCTAssertEqual(req.httpMethod, "POST") + XCTAssertEqual(req.value(forHTTPHeaderField: "Content-Type"), "application/x-www-form-urlencoded") + let body = String(data: req.httpBody ?? Data(), encoding: .utf8) ?? "" + XCTAssertTrue(body.contains("grant_type=authorization_code")) + XCTAssertTrue(body.contains("code=the-code")) + XCTAssertTrue(body.contains("client_id=cid")) + XCTAssertTrue(body.contains("client_secret=sec")) + } + + func testParseTokenResponseComputesExpiry() throws { + let now = Date(timeIntervalSince1970: 1_000_000) + let json = #"{"access_token":"acc","refresh_token":"ref","expires_in":86400}"#.data(using: .utf8)! + let tokens = try OuraOAuth.parseTokenResponse(json, now: now) + XCTAssertEqual(tokens.accessToken, "acc") + XCTAssertEqual(tokens.refreshToken, "ref") + XCTAssertEqual(tokens.expiresAt, now.addingTimeInterval(86400)) + } + + func testParseTokenResponseThrowsOnMissingAccessToken() { + let json = #"{"error":"invalid_grant"}"#.data(using: .utf8)! + XCTAssertThrowsError(try OuraOAuth.parseTokenResponse(json, now: Date())) + } +} From 257060fbdee25f207cba7cc5ce954fa5ae01ac41 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 20:11:00 -0700 Subject: [PATCH 016/171] fix(oura): guard token save on refresh + cover expires_in default (Task 3 review) --- Strand/Oura/OuraOAuthProvider.swift | 2 +- StrandTests/OuraOAuthTests.swift | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/Strand/Oura/OuraOAuthProvider.swift b/Strand/Oura/OuraOAuthProvider.swift index c69dbd8f4d..f62b583f0b 100644 --- a/Strand/Oura/OuraOAuthProvider.swift +++ b/Strand/Oura/OuraOAuthProvider.swift @@ -23,7 +23,7 @@ final class OuraOAuthProvider: NSObject, AuthProvider, ASWebAuthenticationPresen guard let refresh = tokens.refreshToken else { throw OuraError.notConnected } let req = OuraOAuth.refreshRequest(credentials: credentials, refreshToken: refresh) let refreshed = try await exchange(req) - _ = OuraTokenStore.save(refreshed) + guard OuraTokenStore.save(refreshed) else { throw OuraError.tokenExchangeFailed("keychain write failed") } return refreshed.accessToken } diff --git a/StrandTests/OuraOAuthTests.swift b/StrandTests/OuraOAuthTests.swift index 035cbc62e7..910d0f4092 100644 --- a/StrandTests/OuraOAuthTests.swift +++ b/StrandTests/OuraOAuthTests.swift @@ -39,6 +39,15 @@ final class OuraOAuthTests: XCTestCase { XCTAssertEqual(tokens.expiresAt, now.addingTimeInterval(86400)) } + func testParseTokenResponseDefaultsExpiryWhenExpiresInAbsent() throws { + let now = Date(timeIntervalSince1970: 1_000_000) + let json = #"{"access_token":"acc"}"#.data(using: .utf8)! + let tokens = try OuraOAuth.parseTokenResponse(json, now: now) + XCTAssertEqual(tokens.accessToken, "acc") + XCTAssertNil(tokens.refreshToken) + XCTAssertEqual(tokens.expiresAt, now.addingTimeInterval(2_592_000)) + } + func testParseTokenResponseThrowsOnMissingAccessToken() { let json = #"{"error":"invalid_grant"}"#.data(using: .utf8)! XCTAssertThrowsError(try OuraOAuth.parseTokenResponse(json, now: Date())) From 135ff8b251dcb33f1765ec013d1c1ffcd3f1a2a7 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 20:14:08 -0700 Subject: [PATCH 017/171] =?UTF-8?q?feat(oura):=20OuraAPIClient=20=E2=80=94?= =?UTF-8?q?=20next=5Ftoken=20paging,=20429=20backoff,=20401=20refresh,=20s?= =?UTF-8?q?andbox/prod?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Strand/Oura/OuraAPIClient.swift | 113 ++++++++++++++++++++++++++ StrandTests/OuraAPIClientTests.swift | 57 +++++++++++++ StrandTests/OuraURLProtocolStub.swift | 32 ++++++++ 3 files changed, 202 insertions(+) create mode 100644 Strand/Oura/OuraAPIClient.swift create mode 100644 StrandTests/OuraAPIClientTests.swift create mode 100644 StrandTests/OuraURLProtocolStub.swift diff --git a/Strand/Oura/OuraAPIClient.swift b/Strand/Oura/OuraAPIClient.swift new file mode 100644 index 0000000000..a98f30e87d --- /dev/null +++ b/Strand/Oura/OuraAPIClient.swift @@ -0,0 +1,113 @@ +import Foundation + +enum OuraEnvironment { + case production, sandbox + var baseURL: String { + switch self { + case .production: return "https://api.ouraring.com/v2/usercollection" + case .sandbox: return "https://api.ouraring.com/v2/sandbox/usercollection" + } + } +} + +/// One decoded page of an Oura list/time-series response. +struct OuraPage { + let data: [[String: Any]] + let nextToken: String? +} + +/// Pure (network-free) response-envelope parsing, unit-tested directly. +enum OuraAPI { + static func parsePage(_ data: Data) throws -> OuraPage { + guard let obj = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else { + throw OuraError.decode + } + let rows = (obj["data"] as? [[String: Any]]) ?? [] + return OuraPage(data: rows, nextToken: obj["next_token"] as? String) + } +} + +/// The Oura v2 API client. Injected `AuthProvider` supplies the bearer token (refreshing on expiry); +/// injected `session` makes it URLProtocol-testable. Follows `next_token` paging, retries 429 with a +/// bounded backoff, and refreshes-once on 401. Networking lives here in the app target by design. +final class OuraAPIClient { + private let auth: AuthProvider + private let environment: OuraEnvironment + private let session: URLSession + private let backoff: TimeInterval + private let maxPages = 10_000 // runaway guard (logged if hit) + + init(auth: AuthProvider, environment: OuraEnvironment = .production, + session: URLSession = .shared, backoff: TimeInterval = 2) { + self.auth = auth + self.environment = environment + self.session = session + self.backoff = backoff + } + + /// Fetch every page of `endpoint` for `query`, following `next_token`, returning concatenated rows. + func fetchAll(endpoint: String, query: [String: String]) async throws -> [[String: Any]] { + var rows: [[String: Any]] = [] + try await forEachPage(endpoint: endpoint, query: query) { page in rows.append(contentsOf: page.data) } + return rows + } + + /// Fetch every page, returning the verbatim page bodies (for Plan 3's lossless raw archive). + func fetchAllRaw(endpoint: String, query: [String: String]) async throws -> [Data] { + var bodies: [Data] = [] + try await forEachPage(endpoint: endpoint, query: query, rawSink: { bodies.append($0) }) { _ in } + return bodies + } + + private func forEachPage(endpoint: String, query: [String: String], + rawSink: ((Data) -> Void)? = nil, + _ handle: (OuraPage) -> Void) async throws { + var next: String? = nil + var pages = 0 + repeat { + var q = query + if let next { q["next_token"] = next } + let (data, _) = try await getWithRetry(endpoint: endpoint, query: q) + rawSink?(data) + let page = try OuraAPI.parsePage(data) + handle(page) + next = page.nextToken + pages += 1 + if pages >= maxPages { break } // Plan-3 note: log if this cap is ever hit. + } while next != nil + } + + /// GET with auth + 429-backoff + 401-refresh-once. Returns (body, status). + private func getWithRetry(endpoint: String, query: [String: String]) async throws -> (Data, Int) { + for attempt in 0..<3 { + let token = try await auth.validAccessToken() + let (data, code) = try await get(endpoint: endpoint, query: query, token: token) + switch code { + case 200..<300: + return (data, code) + case 429: + if backoff > 0 { try? await Task.sleep(nanoseconds: UInt64(backoff * 1_000_000_000)) } + continue + case 401 where attempt == 0: + _ = try? await auth.validAccessToken() // force a refresh, then retry once + continue + default: + throw OuraError.badResponse(code, String(data: data, encoding: .utf8) ?? "") + } + } + throw OuraError.rateLimited + } + + private func get(endpoint: String, query: [String: String], token: String) async throws -> (Data, Int) { + var comps = URLComponents(string: "\(environment.baseURL)/\(endpoint)")! + if !query.isEmpty { comps.queryItems = query.map { URLQueryItem(name: $0.key, value: $0.value) } } + var req = URLRequest(url: comps.url!) + req.setValue("Bearer \(token)", forHTTPHeaderField: "Authorization") + do { + let (data, resp) = try await session.data(for: req) + return (data, (resp as? HTTPURLResponse)?.statusCode ?? 0) + } catch { + throw OuraError.network(error.localizedDescription) + } + } +} diff --git a/StrandTests/OuraAPIClientTests.swift b/StrandTests/OuraAPIClientTests.swift new file mode 100644 index 0000000000..2783297f82 --- /dev/null +++ b/StrandTests/OuraAPIClientTests.swift @@ -0,0 +1,57 @@ +import XCTest +import AuthenticationServices +@testable import Strand + +/// A stub AuthProvider that hands out a fixed token and counts refreshes. +private final class StubAuth: AuthProvider { + var token = "tok-1" + var validCalls = 0 + var isConnected: Bool { true } + func validAccessToken() async throws -> String { validCalls += 1; return token } + @MainActor func authorize(presentationAnchor: ASPresentationAnchor) async throws {} + func signOut() {} +} + +final class OuraAPIClientTests: XCTestCase { + override func setUp() { super.setUp(); OuraURLProtocolStub.reset() } + + func testParsePageExtractsDataAndNextToken() throws { + let json = #"{"data":[{"id":"a"},{"id":"b"}],"next_token":"tok2"}"#.data(using: .utf8)! + let page = try OuraAPI.parsePage(json) + XCTAssertEqual(page.data.count, 2) + XCTAssertEqual(page.nextToken, "tok2") + } + + func testFetchAllFollowsNextTokenThenStops() async throws { + OuraURLProtocolStub.queue = [ + .init(status: 200, body: #"{"data":[{"id":"a"}],"next_token":"t2"}"#.data(using: .utf8)!), + .init(status: 200, body: #"{"data":[{"id":"b"}],"next_token":null}"#.data(using: .utf8)!), + ] + let client = OuraAPIClient(auth: StubAuth(), environment: .production, + session: OuraURLProtocolStub.session()) + let rows = try await client.fetchAll(endpoint: "daily_sleep", query: ["start_date": "2026-01-01"]) + XCTAssertEqual(rows.count, 2) + // Second request carried next_token=t2. + XCTAssertTrue(OuraURLProtocolStub.requestedURLs.last?.absoluteString.contains("next_token=t2") ?? false) + } + + func test429IsRetriedAfterBackoff() async throws { + OuraURLProtocolStub.queue = [ + .init(status: 429, body: Data()), + .init(status: 200, body: #"{"data":[{"id":"a"}],"next_token":null}"#.data(using: .utf8)!), + ] + let client = OuraAPIClient(auth: StubAuth(), environment: .production, + session: OuraURLProtocolStub.session(), backoff: 0) + let rows = try await client.fetchAll(endpoint: "daily_sleep", query: [:]) + XCTAssertEqual(rows.count, 1) + } + + func testSandboxBaseURL() async throws { + OuraURLProtocolStub.queue = [.init(status: 200, + body: #"{"data":[],"next_token":null}"#.data(using: .utf8)!)] + let client = OuraAPIClient(auth: StubAuth(), environment: .sandbox, + session: OuraURLProtocolStub.session()) + _ = try await client.fetchAll(endpoint: "personal_info", query: [:]) + XCTAssertTrue(OuraURLProtocolStub.requestedURLs.first?.absoluteString.contains("/v2/sandbox/") ?? false) + } +} diff --git a/StrandTests/OuraURLProtocolStub.swift b/StrandTests/OuraURLProtocolStub.swift new file mode 100644 index 0000000000..8bd46945d1 --- /dev/null +++ b/StrandTests/OuraURLProtocolStub.swift @@ -0,0 +1,32 @@ +import Foundation + +/// Test-only URLProtocol that serves canned responses by request order, so the paging loop and +/// status-code handling are driven deterministically without a live socket. +final class OuraURLProtocolStub: URLProtocol { + struct Stubbed { let status: Int; let body: Data } + /// FIFO queue of responses; each request pops the next. Set before the test acts. + nonisolated(unsafe) static var queue: [Stubbed] = [] + /// Captured request URLs, in order, for assertions. + nonisolated(unsafe) static var requestedURLs: [URL] = [] + + static func reset() { queue = []; requestedURLs = [] } + + override class func canInit(with request: URLRequest) -> Bool { true } + override class func canonicalRequest(for request: URLRequest) -> URLRequest { request } + override func startLoading() { + if let u = request.url { Self.requestedURLs.append(u) } + let s = Self.queue.isEmpty ? Stubbed(status: 500, body: Data()) : Self.queue.removeFirst() + let resp = HTTPURLResponse(url: request.url!, statusCode: s.status, httpVersion: nil, headerFields: nil)! + client?.urlProtocol(self, didReceive: resp, cacheStoragePolicy: .notAllowed) + client?.urlProtocol(self, didLoad: s.body) + client?.urlProtocolDidFinishLoading(self) + } + override func stopLoading() {} + + /// A URLSession wired to this stub. + static func session() -> URLSession { + let cfg = URLSessionConfiguration.ephemeral + cfg.protocolClasses = [OuraURLProtocolStub.self] + return URLSession(configuration: cfg) + } +} From dc44bfddb68fb623d248dfd37a8f25f1b2bd979b Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 20:27:01 -0700 Subject: [PATCH 018/171] fix(oura): real force-refresh on 401 + 401/429 tests (Task 4 review) getWithRetry's 401 branch called validAccessToken(), which only refreshes when clock-expired -- a 401 for a revoked/invalidated-but-unexpired token re-served the same rejected token and the retry failed silently. Add refreshedAccessToken() to AuthProvider for an unconditional refresh; validAccessToken() now delegates to it on expiry. Cover both the 401 force-refresh path and 429 exhaustion with tests. --- Strand/Oura/AuthProvider.swift | 3 +++ Strand/Oura/OuraAPIClient.swift | 2 +- Strand/Oura/OuraOAuthProvider.swift | 10 ++++++--- StrandTests/OuraAPIClientTests.swift | 31 ++++++++++++++++++++++++++++ 4 files changed, 42 insertions(+), 4 deletions(-) diff --git a/Strand/Oura/AuthProvider.swift b/Strand/Oura/AuthProvider.swift index d42311e75f..fe99030e84 100644 --- a/Strand/Oura/AuthProvider.swift +++ b/Strand/Oura/AuthProvider.swift @@ -8,6 +8,9 @@ protocol AuthProvider { var isConnected: Bool { get } /// Return a currently-valid access token, refreshing transparently if the stored one is expired. func validAccessToken() async throws -> String + /// Force an unconditional refresh (regardless of expiry) and return the new access token. + /// Used when a token was rejected server-side (401) even though it wasn't clock-expired. + func refreshedAccessToken() async throws -> String /// Run the interactive authorization (opens Oura's consent page) and store the resulting tokens. @MainActor func authorize(presentationAnchor: ASPresentationAnchor) async throws /// Forget the stored tokens (disconnect). diff --git a/Strand/Oura/OuraAPIClient.swift b/Strand/Oura/OuraAPIClient.swift index a98f30e87d..ca2839f5f3 100644 --- a/Strand/Oura/OuraAPIClient.swift +++ b/Strand/Oura/OuraAPIClient.swift @@ -89,7 +89,7 @@ final class OuraAPIClient { if backoff > 0 { try? await Task.sleep(nanoseconds: UInt64(backoff * 1_000_000_000)) } continue case 401 where attempt == 0: - _ = try? await auth.validAccessToken() // force a refresh, then retry once + _ = try? await auth.refreshedAccessToken() // force a real refresh; the loop-top re-reads the saved token continue default: throw OuraError.badResponse(code, String(data: data, encoding: .utf8) ?? "") diff --git a/Strand/Oura/OuraOAuthProvider.swift b/Strand/Oura/OuraOAuthProvider.swift index f62b583f0b..0519637be0 100644 --- a/Strand/Oura/OuraOAuthProvider.swift +++ b/Strand/Oura/OuraOAuthProvider.swift @@ -20,9 +20,13 @@ final class OuraOAuthProvider: NSObject, AuthProvider, ASWebAuthenticationPresen func validAccessToken() async throws -> String { guard let tokens = OuraTokenStore.load() else { throw OuraError.notConnected } if !tokens.isExpired { return tokens.accessToken } - guard let refresh = tokens.refreshToken else { throw OuraError.notConnected } - let req = OuraOAuth.refreshRequest(credentials: credentials, refreshToken: refresh) - let refreshed = try await exchange(req) + return try await refreshedAccessToken() + } + + /// Unconditionally refresh, regardless of the stored token's expiry state. + func refreshedAccessToken() async throws -> String { + guard let refresh = OuraTokenStore.load()?.refreshToken else { throw OuraError.notConnected } + let refreshed = try await exchange(OuraOAuth.refreshRequest(credentials: credentials, refreshToken: refresh)) guard OuraTokenStore.save(refreshed) else { throw OuraError.tokenExchangeFailed("keychain write failed") } return refreshed.accessToken } diff --git a/StrandTests/OuraAPIClientTests.swift b/StrandTests/OuraAPIClientTests.swift index 2783297f82..7c76914ee5 100644 --- a/StrandTests/OuraAPIClientTests.swift +++ b/StrandTests/OuraAPIClientTests.swift @@ -6,8 +6,10 @@ import AuthenticationServices private final class StubAuth: AuthProvider { var token = "tok-1" var validCalls = 0 + var refreshCount = 0 var isConnected: Bool { true } func validAccessToken() async throws -> String { validCalls += 1; return token } + func refreshedAccessToken() async throws -> String { refreshCount += 1; token = "tok-refreshed"; return token } @MainActor func authorize(presentationAnchor: ASPresentationAnchor) async throws {} func signOut() {} } @@ -54,4 +56,33 @@ final class OuraAPIClientTests: XCTestCase { _ = try await client.fetchAll(endpoint: "personal_info", query: [:]) XCTAssertTrue(OuraURLProtocolStub.requestedURLs.first?.absoluteString.contains("/v2/sandbox/") ?? false) } + + func test401TriggersRefreshThenRetrySucceeds() async throws { + OuraURLProtocolStub.queue = [ + .init(status: 401, body: Data()), + .init(status: 200, body: #"{"data":[{"id":"a"}],"next_token":null}"#.data(using: .utf8)!), + ] + let stub = StubAuth() + let client = OuraAPIClient(auth: stub, environment: .production, + session: OuraURLProtocolStub.session()) + let rows = try await client.fetchAll(endpoint: "daily_sleep", query: [:]) + XCTAssertEqual(rows.count, 1) + XCTAssertEqual(stub.refreshCount, 1) + } + + func test429ExhaustionThrowsRateLimited() async throws { + OuraURLProtocolStub.queue = [ + .init(status: 429, body: Data()), + .init(status: 429, body: Data()), + .init(status: 429, body: Data()), + ] + let client = OuraAPIClient(auth: StubAuth(), environment: .production, + session: OuraURLProtocolStub.session(), backoff: 0) + do { + _ = try await client.fetchAll(endpoint: "daily_sleep", query: [:]) + XCTFail("expected rateLimited to be thrown") + } catch { + XCTAssertEqual(error as? OuraError, OuraError.rateLimited) + } + } } From 8cc4c164e0b274d6cd93e28d468607f8157cad79 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 20:38:17 -0700 Subject: [PATCH 019/171] chore(oura): wire OAuth redirect scheme + BYO-app credentials xcconfig MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wires the OURA_CLIENT_ID/SECRET/REDIRECT_URI Info.plist keys (read by OuraCredentials.fromBundle) and the noop:// CFBundleURLTypes scheme into both the Strand (macOS) and NOOPiOS targets via a new untracked Strand/Oura/OuraSecrets.xcconfig (gitignored; committed template at OuraSecrets.example.xcconfig). Also regenerates Strand/Resources/Info.plist and StrandiOS/Resources/Info.plist via xcodegen generate — both targets set GENERATE_INFOPLIST_FILE: NO, so xcodegen pre-materializes these tracked plists from project.yml's info.properties (same convention as the prior "regenerate Info.plist to match project.yml" commits). Escapes the redirect URI's "//" with the empty-macro $() trick in both xcconfigs: .xcconfig treats // as a start-of-comment anywhere on the line (not just at line-start), so a literal noop://oura/callback silently truncates to noop: at build time and would break the redirect_uri match against the app registered at the Oura developer portal. Verified: xcodegen generate + macOS build-for-testing succeeds, and the built app's Info.plist resolves all four keys correctly (not left as literal $(OURA_*)), including the full noop://oura/callback string. NOOPiOS gets the same wiring but isn't build-verified in this environment. --- .gitignore | 2 ++ Strand/Oura/OuraSecrets.example.xcconfig | 9 ++++++++ Strand/Resources/Info.plist | 15 +++++++++++++ StrandiOS/Resources/Info.plist | 15 +++++++++++++ project.yml | 28 ++++++++++++++++++++++++ 5 files changed, 69 insertions(+) create mode 100644 Strand/Oura/OuraSecrets.example.xcconfig diff --git a/.gitignore b/.gitignore index a4c70bd1d4..5097b1fa07 100644 --- a/.gitignore +++ b/.gitignore @@ -69,6 +69,8 @@ android/app/release/ # ── Misc / secrets (never commit) ─────────────────────────────── *.log .env +# Oura BYO-app OAuth credentials (never committed) +Strand/Oura/OuraSecrets.xcconfig # Deployment credentials (PAT / signing / endpoints) — never commit. `.env` above # won't match `deploy.env`, so list it (and any `deploy.env.`) explicitly. deploy.env diff --git a/Strand/Oura/OuraSecrets.example.xcconfig b/Strand/Oura/OuraSecrets.example.xcconfig new file mode 100644 index 0000000000..04f429e246 --- /dev/null +++ b/Strand/Oura/OuraSecrets.example.xcconfig @@ -0,0 +1,9 @@ +// Copy to OuraSecrets.xcconfig (gitignored) and fill in your own Oura OAuth app's values. +// Register a free app at the Oura developer portal; set the redirect URI to noop://oura/callback. +OURA_CLIENT_ID = your_client_id_here +OURA_CLIENT_SECRET = your_client_secret_here +// .xcconfig treats // as a start-of-comment ANYWHERE on the line (not just at line-start), so a +// literal "noop://oura/callback" here gets silently truncated to "noop:" — the empty macro +// expansion $() below breaks up the "//" token without changing the resolved string, which is +// still exactly noop://oura/callback at build time. Keep this trick if you edit the value. +OURA_REDIRECT_URI = noop:/$()/oura/callback diff --git a/Strand/Resources/Info.plist b/Strand/Resources/Info.plist index 31a3c7052d..8a2f2a392f 100644 --- a/Strand/Resources/Info.plist +++ b/Strand/Resources/Info.plist @@ -18,6 +18,15 @@ APPL CFBundleShortVersionString $(MARKETING_VERSION) + CFBundleURLTypes + + + CFBundleURLSchemes + + noop + + + CFBundleVersion 198 LSApplicationCategoryType @@ -33,5 +42,11 @@ NOOP connects directly to your WHOOP strap over Bluetooth to read heart rate, R-R intervals, battery, and sensor data locally on your Mac. Nothing leaves your device. NSLocationWhenInUseUsageDescription NOOP records your route during an outdoor workout to show distance, pace, and a map. The route stays on your device. Nothing leaves it. + OURA_CLIENT_ID + $(OURA_CLIENT_ID) + OURA_CLIENT_SECRET + $(OURA_CLIENT_SECRET) + OURA_REDIRECT_URI + $(OURA_REDIRECT_URI) diff --git a/StrandiOS/Resources/Info.plist b/StrandiOS/Resources/Info.plist index fa3dce93fd..b9bb28c9bb 100644 --- a/StrandiOS/Resources/Info.plist +++ b/StrandiOS/Resources/Info.plist @@ -37,6 +37,15 @@ APPL CFBundleShortVersionString $(MARKETING_VERSION) + CFBundleURLTypes + + + CFBundleURLSchemes + + noop + + + CFBundleVersion $(CURRENT_PROJECT_VERSION) LSApplicationCategoryType @@ -58,6 +67,12 @@ NOOP records your route during an outdoor workout to show distance, pace, and a map, and keeps recording while the screen is off. The route stays on your device. Nothing leaves it. NSSupportsLiveActivities + OURA_CLIENT_ID + $(OURA_CLIENT_ID) + OURA_CLIENT_SECRET + $(OURA_CLIENT_SECRET) + OURA_REDIRECT_URI + $(OURA_REDIRECT_URI) UIBackgroundModes bluetooth-central diff --git a/project.yml b/project.yml index 38f2aca34f..b5b894f131 100644 --- a/project.yml +++ b/project.yml @@ -76,6 +76,12 @@ targets: # The liquid Today is now cross-platform: its iOS-only chrome (topBarTrailing / navigationBarTitleDisplayMode # / presentationCompactAdaptation / two-param onChange / sensoryFeedback) is guarded, so the mac split-view # shell hosts LiquidTodayView too. All Liquid/* now compile into the mac target. + # Oura BYO-app OAuth credentials, sourced from the untracked Strand/Oura/OuraSecrets.xcconfig + # (gitignored; template at OuraSecrets.example.xcconfig). Resolves the $(OURA_*) references in + # the info.properties below. + configFiles: + Debug: Strand/Oura/OuraSecrets.xcconfig + Release: Strand/Oura/OuraSecrets.xcconfig info: path: Strand/Resources/Info.plist properties: @@ -93,6 +99,15 @@ targets: # HTTPS endpoints are unaffected. No arbitrary-loads blanket exemption. NSAppTransportSecurity: NSAllowsLocalNetworking: true + # Oura BYO-app OAuth credentials (values come from configFiles above); read at runtime via + # OuraCredentials.fromBundle. Absent/blank values cleanly disable the Connect-Oura lane. + OURA_CLIENT_ID: $(OURA_CLIENT_ID) + OURA_CLIENT_SECRET: $(OURA_CLIENT_SECRET) + OURA_REDIRECT_URI: $(OURA_REDIRECT_URI) + # Registers the noop:// URL scheme so ASWebAuthenticationSession's OAuth redirect + # (noop://oura/callback) reaches OuraOAuthProvider instead of bouncing off the OS. + CFBundleURLTypes: + - CFBundleURLSchemes: [noop] entitlements: path: Strand/Resources/Strand.entitlements properties: @@ -180,6 +195,10 @@ targets: - "Resources/Info.plist" - "Resources/NOOP.entitlements" - path: StrandiOSShared + # Same untracked xcconfig as the macOS target (Strand/Oura/ is shared, not per-platform). + configFiles: + Debug: Strand/Oura/OuraSecrets.xcconfig + Release: Strand/Oura/OuraSecrets.xcconfig info: path: StrandiOS/Resources/Info.plist properties: @@ -242,6 +261,15 @@ targets: # Both keys are required for the folder to actually appear in Files. UIFileSharingEnabled: true LSSupportsOpeningDocumentsInPlace: true + # Oura BYO-app OAuth credentials (values come from configFiles above); read at runtime via + # OuraCredentials.fromBundle. Absent/blank values cleanly disable the Connect-Oura lane. + OURA_CLIENT_ID: $(OURA_CLIENT_ID) + OURA_CLIENT_SECRET: $(OURA_CLIENT_SECRET) + OURA_REDIRECT_URI: $(OURA_REDIRECT_URI) + # Registers the noop:// URL scheme so ASWebAuthenticationSession's OAuth redirect + # (noop://oura/callback) reaches OuraOAuthProvider instead of bouncing off the OS. + CFBundleURLTypes: + - CFBundleURLSchemes: [noop] entitlements: path: StrandiOS/Resources/NOOP.entitlements properties: From 536a6d174320791fc1c98c255f3c52b2d76df880 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 20:55:09 -0700 Subject: [PATCH 020/171] fix(oura): form-encode token-request body (escape + / = ; was corrupting base64 secrets) --- Strand/Oura/OuraOAuth.swift | 11 ++++++++--- StrandTests/OuraOAuthTests.swift | 9 +++++++++ 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/Strand/Oura/OuraOAuth.swift b/Strand/Oura/OuraOAuth.swift index 71fc1c52c4..df6f842e15 100644 --- a/Strand/Oura/OuraOAuth.swift +++ b/Strand/Oura/OuraOAuth.swift @@ -58,9 +58,14 @@ enum OuraOAuth { var req = URLRequest(url: url) req.httpMethod = "POST" req.setValue("application/x-www-form-urlencoded", forHTTPHeaderField: "Content-Type") - var comps = URLComponents() - comps.queryItems = fields.map { URLQueryItem(name: $0.key, value: $0.value) } - req.httpBody = comps.percentEncodedQuery?.data(using: .utf8) + // x-www-form-urlencoded: escape everything but unreserved chars. Crucially '+' (else the server + // decodes it to a space) and the '&'/'=' separators — URLComponents.percentEncodedQuery leaves '+' + // unescaped, which silently corrupts base64 client_secret/code/refresh_token values. + var allowed = CharacterSet.alphanumerics + allowed.insert(charactersIn: "-._~") + func enc(_ s: String) -> String { s.addingPercentEncoding(withAllowedCharacters: allowed) ?? s } + req.httpBody = fields.map { "\(enc($0.key))=\(enc($0.value))" } + .joined(separator: "&").data(using: .utf8) return req } } diff --git a/StrandTests/OuraOAuthTests.swift b/StrandTests/OuraOAuthTests.swift index 910d0f4092..90f9e9ca73 100644 --- a/StrandTests/OuraOAuthTests.swift +++ b/StrandTests/OuraOAuthTests.swift @@ -52,4 +52,13 @@ final class OuraOAuthTests: XCTestCase { let json = #"{"error":"invalid_grant"}"#.data(using: .utf8)! XCTAssertThrowsError(try OuraOAuth.parseTokenResponse(json, now: Date())) } + + func testTokenExchangeRequestEscapesReservedCharsInSecret() { + let c = OuraCredentials(clientId: "cid", clientSecret: "aB+cd/eF12==", redirectURI: "noop://oura/callback") + let req = OuraOAuth.tokenExchangeRequest(credentials: c, code: "x+y") + let body = String(data: req.httpBody ?? Data(), encoding: .utf8) ?? "" + XCTAssertTrue(body.contains("client_secret=aB%2Bcd%2FeF12%3D%3D")) // + -> %2B, / -> %2F, = -> %3D + XCTAssertTrue(body.contains("code=x%2By")) // + escaped, not left literal + XCTAssertFalse(body.contains("aB+cd")) // no raw '+' + } } From 066fbb2382bf6f698d00ad7a0e57c1e2ef12f9ee Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 21:17:57 -0700 Subject: [PATCH 021/171] =?UTF-8?q?docs:=20implementation=20plan=203/3=20?= =?UTF-8?q?=E2=80=94=20Oura=20orchestration=20+=20writer=20+=20Connect=20U?= =?UTF-8?q?I?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit OuraSyncWriter (persist backfill to WhoopStore as .cloudImport, honest-data + coalesce + persistSessionMotion + hrSample + raw archive), OuraSyncCoordinator (one-time backfill across all endpoints, readiness-RHR precedence), deleteOuraRaw for clean disconnect, the Connect Oura card + presentation-anchor helper, scoring source + docs. Verified against v8.5.2 signatures. --- ...oura-live-api-import-3-orchestration-ui.md | 685 ++++++++++++++++++ 1 file changed, 685 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-27-oura-live-api-import-3-orchestration-ui.md diff --git a/docs/superpowers/plans/2026-06-27-oura-live-api-import-3-orchestration-ui.md b/docs/superpowers/plans/2026-06-27-oura-live-api-import-3-orchestration-ui.md new file mode 100644 index 0000000000..11dba508c9 --- /dev/null +++ b/docs/superpowers/plans/2026-06-27-oura-live-api-import-3-orchestration-ui.md @@ -0,0 +1,685 @@ +# Oura Live API Import — Plan 3: Orchestration + Writer + Connect UI + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Turn Plan 1's parsers + Plan 2's client/auth into a working, tappable feature — a one-time backfill that fetches every Oura v2 endpoint, writes the normalized subset + lossless raw into `WhoopStore` under `deviceId = "oura-api"`, and a "Connect Oura & Import Everything" card on Data Sources. + +**Architecture:** `OuraSyncCoordinator` (fetch all endpoints via `OuraAPIClient.fetchAllRaw`, parse via Plan 1's `OuraApiParser`, merge per-day rows, hand an assembled `OuraSyncResult` to the writer) → `OuraSyncWriter` (register the `PairedDevice`, then map the result to the existing `WhoopStore` upserts, archiving every raw page). All in the app target `Strand/Oura/`, beside Plan 2. The writer mirrors `Strand/Data/WearableImporter.swift`; the UI mirrors `DataSourcesView`'s existing import cards. Honest-data + no-day-ownership-seizure are structural (see constraints). + +**Tech stack:** Swift 5.0 mode (app target), `Foundation` + `AuthenticationServices` + SwiftUI. XCTest in `StrandTests`, run via `xcodebuild -scheme Strand test` (the `TEST_HOST` fix from Plan 2 makes this work). Store-writing tests use `WhoopStore.inMemory()`. + +## Global Constraints + +- **App-target only.** Files live in `Strand/Oura/` (+ one `WhoopStore` addition in Task 3, + edits in Task 5). No package gains networking. +- **Provenance = `deviceId "oura-api"`, `sourceKind .cloudImport`.** `.cloudImport` is structurally priority-2 in `DayOwnerResolver` (`IntelligenceEngine.swift:1369`: `isImport = .cloudImport || .fileImport` → priority 2, lowest), so the cloud source **can never seize a day from the active WHOOP or any live BLE device** — this is automatic from the `sourceKind` alone. **Never call `setDayOwner`.** +- **Honest data.** Oura's own scores go ONLY to `ref_*` / `oura_*` `metricSeries` keys (from the `[OuraDailyExtra]` array), never to `DailyMetric.recovery`/`.strain` (always `nil`). Mirror `WearableImporter`. +- **Coalesce, prefer readiness RHR.** A single day gets a `WearableDailyRow` from `parseSleep` AND from each `parseDaily(_, endpoint:)`. Merge them into ONE `[String: WearableDailyRow]` with first-non-nil-wins (`??`), merging the **daily endpoints (readiness) BEFORE `sleep`**, so `daily_readiness`'s true `resting_heart_rate` wins over `parseSleep`'s `lowestHr` fallback (Plan-1 final-review advisory). +- **Lossless raw.** Every fetched page → `upsertOuraRaw` regardless of whether a parser exists for that endpoint (8 of 18 endpoints are raw-only). +- **Reuse, don't reinvent.** `DailyMetric`/`CachedSleepSession`/`MetricPoint`/`WorkoutRow`/`HRSample`/`Streams` inits are exact (see each task); `WhoopStore.inMemory()`, `store.registryWriter` (public nonisolated), `repo.refresh()`, `repo.storeHandle()` all exist. +- **Build/test:** `xcodegen generate` (after new files) then `xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/ CODE_SIGNING_ALLOWED=NO`. + +--- + +### Task 1: `OuraSyncResult` + `OuraSyncWriter` + +The assembled-result model + the persistence mapper. Mirrors `WearableImporter`, adds device registration + `persistSessionMotion` + hrSample + raw archive. + +**Files:** +- Create: `Strand/Oura/OuraSyncResult.swift` +- Create: `Strand/Oura/OuraSyncWriter.swift` +- Test: `StrandTests/OuraSyncWriterTests.swift` + +**Interfaces:** +- Consumes (Plan 1, StrandImport): `WearableDailyRow`, `OuraSleepPeriod`, `OuraHRPoint`, `OuraDailyExtra`, `OuraWorkout`, `WearableSleepStageInterval`, `OuraRawRow`. (WhoopStore/WhoopProtocol) `DailyMetric`, `CachedSleepSession`, `MetricPoint`, `WorkoutRow`, `HRSample`, `Streams`, `PairedDevice`, `SourceKind`, `DeviceRegistryStore`, `WhoopStore`. +- Produces: + - `struct OuraSyncResult { var days: [WearableDailyRow]; var sleepPeriods: [OuraSleepPeriod]; var extras: [OuraDailyExtra]; var workouts: [OuraWorkout]; var heartRate: [OuraHRPoint]; var rawPages: [OuraRawRow]; var ringModel: String? }` + - `struct OuraSyncSummary: Equatable { var days: Int; var sleeps: Int; var workouts: Int; var hrSamples: Int; var metricPoints: Int; var rawPages: Int }` + - `enum OuraSyncWriter { static func persist(_ result: OuraSyncResult, into store: WhoopStore, deviceId: String = "oura-api") async throws -> OuraSyncSummary }` + +- [ ] **Step 1: Write the failing test** + +Create `StrandTests/OuraSyncWriterTests.swift`: + +```swift +import XCTest +import WhoopStore +import WhoopProtocol +@testable import StrandImport +@testable import Strand + +final class OuraSyncWriterTests: XCTestCase { + private func iso(_ s: String) -> Date { ISO8601DateFormatter().date(from: s)! } + + func testPersistWritesAllStoresAndRegistersCloudSource() async throws { + let store = try await WhoopStore.inMemory() + + var day = WearableDailyRow(day: "2026-01-02") + day.restingHr = 50; day.steps = 9000; day.totalSleepMin = 400 + + let session = WearableSleepSession( + start: iso("2026-01-01T23:00:00Z"), end: iso("2026-01-02T06:00:00Z"), + deepMin: 100, lightMin: 200, remMin: 100, awakeMin: 20, totalSleepMin: 400, + efficiencyPct: 92, avgHr: 55, lowestHr: 48, avgHrvMs: 65, respRateBpm: 14, + sleepScore: nil, stages: [WearableSleepStageInterval(stage: "deep", + start: iso("2026-01-01T23:00:00Z"), end: iso("2026-01-01T23:05:00Z"))]) + let period = OuraSleepPeriod(session: session, movement30s: [1, 2, 1], + hr: [OuraHRPoint(ts: Int(iso("2026-01-01T23:00:00Z").timeIntervalSince1970), bpm: 60)]) + + let result = OuraSyncResult( + days: [day], sleepPeriods: [period], + extras: [OuraDailyExtra(day: "2026-01-02", key: "ref_readiness_score", value: 80), + OuraDailyExtra(day: "2026-01-02", key: "vo2max", value: 44)], + workouts: [OuraWorkout(start: iso("2026-01-02T07:00:00Z"), end: iso("2026-01-02T07:40:00Z"), + activity: "running", source: "autodetected", energyKcal: 410, distanceM: 6200)], + heartRate: [OuraHRPoint(ts: Int(iso("2026-01-02T07:00:00Z").timeIntervalSince1970), bpm: 120)], + rawPages: [OuraRawRow(endpoint: "sleep", documentId: "s1", day: "2026-01-02", + payloadJSON: "{}", fetchedAt: 100)], + ringModel: "Oura Ring Gen3") + + let summary = try await OuraSyncWriter.persist(result, into: store) + + XCTAssertEqual(summary.days, 1) + XCTAssertEqual(summary.sleeps, 1) + XCTAssertEqual(summary.workouts, 1) + XCTAssertGreaterThan(summary.hrSamples, 0) // in-sleep + whole-day HR + XCTAssertGreaterThan(summary.metricPoints, 0) + XCTAssertEqual(summary.rawPages, 1) + + // Cloud source registered as .cloudImport (so it never seizes a WHOOP day). + let devices = try DeviceRegistryStore(dbQueue: store.registryWriter).all() + let oura = devices.first { $0.id == "oura-api" } + XCTAssertEqual(oura?.sourceKind, .cloudImport) + XCTAssertEqual(oura?.model, "Oura Ring Gen3") + + // Honest data: the reference score is a metricSeries key, never a DailyMetric score column. + let refs = try await store.metricSeries(deviceId: "oura-api", key: "ref_readiness_score", + from: "2026-01-01", to: "2026-01-03") + XCTAssertEqual(refs.first?.value, 80) + let vo2 = try await store.metricSeries(deviceId: "oura-api", key: "vo2max", + from: "2026-01-01", to: "2026-01-03") + XCTAssertEqual(vo2.first?.value, 44) // vo2max parity via extras + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `xcodegen generate && xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/OuraSyncWriterTests CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20` +Expected: FAIL — `OuraSyncResult`/`OuraSyncWriter` undefined. + +- [ ] **Step 3: Implement `OuraSyncResult`** + +Create `Strand/Oura/OuraSyncResult.swift`: + +```swift +import Foundation +import StrandImport + +/// The fully-assembled, parsed output of one Oura backfill, ready to persist. The coordinator (Task 2) +/// builds this (merging per-day rows across endpoints); the writer (below) maps it to WhoopStore. +struct OuraSyncResult { + var days: [WearableDailyRow] // already coalesced per calendar day + var sleepPeriods: [OuraSleepPeriod] + var extras: [OuraDailyExtra] // every ref_*/oura_*/vo2max scalar → metricSeries + var workouts: [OuraWorkout] + var heartRate: [OuraHRPoint] // from the /heartrate endpoint (whole-day) + var rawPages: [OuraRawRow] // every fetched page, verbatim + var ringModel: String? // from ring_configuration, for the PairedDevice model + + init(days: [WearableDailyRow] = [], sleepPeriods: [OuraSleepPeriod] = [], extras: [OuraDailyExtra] = [], + workouts: [OuraWorkout] = [], heartRate: [OuraHRPoint] = [], rawPages: [OuraRawRow] = [], + ringModel: String? = nil) { + self.days = days; self.sleepPeriods = sleepPeriods; self.extras = extras + self.workouts = workouts; self.heartRate = heartRate; self.rawPages = rawPages; self.ringModel = ringModel + } +} + +/// Counts written, for the honest import summary. +struct OuraSyncSummary: Equatable { + var days = 0, sleeps = 0, workouts = 0, hrSamples = 0, metricPoints = 0, rawPages = 0 +} +``` + +- [ ] **Step 4: Implement `OuraSyncWriter`** + +Create `Strand/Oura/OuraSyncWriter.swift`: + +```swift +import Foundation +import WhoopStore +import WhoopProtocol +import StrandImport + +/// Maps an assembled `OuraSyncResult` into the on-device WhoopStore under `deviceId = "oura-api"` — the +/// cloud sibling of `Strand/Data/WearableImporter.swift`. HONEST DATA: Oura's own scores go only to +/// ref_*/oura_* metricSeries keys; `DailyMetric.recovery`/`.strain` stay nil. The source registers as +/// `.cloudImport`, which is structurally priority-2 in DayOwnerResolver, so it never seizes a WHOOP day. +enum OuraSyncWriter { + + @discardableResult + static func persist(_ result: OuraSyncResult, into store: WhoopStore, + deviceId: String = "oura-api") async throws -> OuraSyncSummary { + var summary = OuraSyncSummary() + + // 0. Register the cloud source (the one thing WearableImporter skips). Idempotent by id. + let now = Int(Date().timeIntervalSince1970) + let device = PairedDevice( + id: deviceId, brand: "Oura", model: result.ringModel ?? "Oura (cloud)", + sourceKind: .cloudImport, capabilities: [], status: .active, + addedAt: now, lastSeenAt: now) + try DeviceRegistryStore(dbQueue: store.registryWriter).add(device) + + // 1. Raw archive — verbatim, every page, lossless. + if !result.rawPages.isEmpty { + summary.rawPages = try await store.upsertOuraRaw(result.rawPages, deviceId: deviceId) + } + + // 2. Days → DailyMetric. recovery/strain ALWAYS nil (never Oura's readiness). Mirrors WearableImporter. + let metrics = result.days.map { d in + DailyMetric(day: d.day, totalSleepMin: d.totalSleepMin, efficiency: d.efficiencyPct, + deepMin: d.deepMin, remMin: d.remMin, lightMin: d.lightMin, disturbances: nil, + restingHr: d.restingHr, avgHrv: d.avgHrvMs, recovery: nil, strain: nil, + exerciseCount: nil, spo2Pct: d.spo2Pct, skinTempDevC: d.skinTempDevC, + respRateBpm: d.respRateBpm, steps: d.steps, activeKcalEst: d.activeKcal) + } + summary.days = try await store.upsertDailyMetrics(metrics, deviceId: deviceId) + + // 3. Sleep sessions → CachedSleepSession (+ stagesJSON), then per-session motion via the dedicated setter. + var sessions: [CachedSleepSession] = [] + for p in result.sleepPeriods { + let startTs = Int(p.session.start.timeIntervalSince1970) + let endTs = Int(p.session.end.timeIntervalSince1970) + sessions.append(CachedSleepSession( + startTs: startTs, endTs: endTs, efficiency: p.session.efficiencyPct, + restingHr: p.session.lowestHr ?? p.session.avgHr, avgHrv: p.session.avgHrvMs, + stagesJSON: stagesJSON(p.session.stages))) + } + summary.sleeps = try await store.upsertSleepSessions(sessions, deviceId: deviceId) + for p in result.sleepPeriods where !p.movement30s.isEmpty { // motionJSON is NOT on the sleep upsert + _ = try await store.persistSessionMotion( + deviceId: deviceId, sessionStart: Int(p.session.start.timeIntervalSince1970), + motionEpochs: p.movement30s.map(Double.init)) + } + + // 4. HR → hrSample via the Streams insert path (in-sleep HR + the whole-day /heartrate series). + var hr: [HRSample] = result.heartRate.map { HRSample(ts: $0.ts, bpm: $0.bpm) } + for p in result.sleepPeriods { hr.append(contentsOf: p.hr.map { HRSample(ts: $0.ts, bpm: $0.bpm) }) } + if !hr.isEmpty { + let counts = try await store.insert(Streams(hr: hr), deviceId: deviceId) + summary.hrSamples = counts.hr + } + + // 5. Workouts → WorkoutRow. + let workouts = result.workouts.map { w in + WorkoutRow(startTs: Int(w.start.timeIntervalSince1970), endTs: Int(w.end.timeIntervalSince1970), + sport: w.activity, source: w.source, durationS: w.end.timeIntervalSince(w.start), + energyKcal: w.energyKcal, avgHr: nil, maxHr: nil, strain: nil, + distanceM: w.distanceM, zonesJSON: nil, notes: nil) + } + summary.workouts = try await store.upsertWorkouts(workouts, deviceId: deviceId) + + // 6. Extras → metricSeries (ref_*/oura_*/vo2max — parity with the file-import lane, sourced from + // the extras array so vo2max/ref_sleep_score are never lost even though DailyMetric lacks them). + let points = result.extras.map { MetricPoint(day: $0.day, key: $0.key, value: $0.value) } + summary.metricPoints = try await store.upsertMetricSeries(points, deviceId: deviceId) + + return summary + } + + /// Serialize decoded stages to the `[{start,end,stage}]` JSON the cache stores (same shape as WearableImporter). + private static func stagesJSON(_ stages: [WearableSleepStageInterval]) -> String? { + guard !stages.isEmpty else { return nil } + let segs = stages.map { ["start": Int($0.start.timeIntervalSince1970), + "end": Int($0.end.timeIntervalSince1970), "stage": $0.stage] as [String: Any] } + return (try? JSONSerialization.data(withJSONObject: segs)).flatMap { String(data: $0, encoding: .utf8) } + } +} +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/OuraSyncWriterTests CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20` +Expected: PASS. (If `PairedDevice`'s `status`/`capabilities` types differ — e.g. `DeviceStatus` has no `.active`, or `capabilities` isn't `Set` — read `Packages/WhoopStore/Sources/WhoopStore/PairedDevice.swift` and adjust the enum case / empty-set literal to match; keep behavior identical. Report any such adjustment.) + +- [ ] **Step 6: Commit** + +```bash +git add Strand/Oura/OuraSyncResult.swift Strand/Oura/OuraSyncWriter.swift StrandTests/OuraSyncWriterTests.swift +git commit -m "feat(oura): OuraSyncWriter — persist backfill to WhoopStore as .cloudImport source" +``` + +--- + +### Task 2: `OuraSyncCoordinator` + +Fetch every endpoint, parse, merge per-day rows (readiness before sleep), assemble the `OuraSyncResult`, persist, report progress. + +**Files:** +- Create: `Strand/Oura/OuraSyncCoordinator.swift` +- Test: `StrandTests/OuraSyncCoordinatorTests.swift` + +**Interfaces:** +- Consumes: `OuraAPIClient` (Plan 2), `OuraApiParser` (Plan 1), `OuraSyncWriter`/`OuraSyncResult` (Task 1), `WhoopStore`, `OuraError`. +- Produces: + - `protocol OuraPageFetching { func fetchAllRaw(endpoint: String, query: [String: String]) async throws -> [Data] }` (+ `extension OuraAPIClient: OuraPageFetching {}`) + - `struct OuraSyncProgress: Equatable { let endpoint: String; let pages: Int }` + - `final class OuraSyncCoordinator` with `init(fetcher: OuraPageFetching, store: WhoopStore)` and `func runFullImport(startDate: String = "2013-01-01", today: String, onProgress: @escaping (OuraSyncProgress) -> Void = { _ in }) async throws -> OuraSyncSummary`. + +- [ ] **Step 1: Write the failing test** + +Create `StrandTests/OuraSyncCoordinatorTests.swift`: + +```swift +import XCTest +import WhoopStore +@testable import StrandImport +@testable import Strand + +/// A fetcher that serves canned page bodies per endpoint. +private final class StubFetcher: OuraPageFetching { + var pages: [String: [Data]] = [:] + func fetchAllRaw(endpoint: String, query: [String: String]) async throws -> [Data] { + pages[endpoint] ?? [] + } +} + +final class OuraSyncCoordinatorTests: XCTestCase { + func testBackfillFetchesParsesAndWrites() async throws { + let store = try await WhoopStore.inMemory() + let fetcher = StubFetcher() + fetcher.pages["daily_readiness"] = [#"{"data":[{"day":"2026-01-02","score":80,"contributors":{"resting_heart_rate":50}}],"next_token":null}"#.data(using: .utf8)!] + fetcher.pages["daily_activity"] = [#"{"data":[{"day":"2026-01-02","steps":9000}],"next_token":null}"#.data(using: .utf8)!] + fetcher.pages["sleep"] = [#"{"data":[{"id":"s1","day":"2026-01-02","bedtime_start":"2026-01-01T23:00:00+00:00","bedtime_end":"2026-01-02T06:00:00+00:00","total_sleep_duration":24000,"lowest_heart_rate":48,"sleep_phase_5_min":"1234"}],"next_token":null}"#.data(using: .utf8)!] + + var progress: [String] = [] + let coord = OuraSyncCoordinator(fetcher: fetcher, store: store) + let summary = try await coord.runFullImport(today: "2026-02-01") { progress.append($0.endpoint) } + + XCTAssertEqual(summary.days, 1) + XCTAssertEqual(summary.sleeps, 1) + XCTAssertTrue(summary.rawPages >= 3) + XCTAssertTrue(progress.contains("sleep")) + + // Coalesce + RHR precedence: daily_readiness's true RHR (50) wins over sleep's lowestHr (48). + let days = try await store.dailyMetrics(deviceId: "oura-api", from: "2026-01-01", to: "2026-01-03") + XCTAssertEqual(days.first?.restingHr, 50) + XCTAssertEqual(days.first?.steps, 9000) + XCTAssertEqual(days.first?.totalSleepMin, 400) + } +} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `xcodegen generate && xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/OuraSyncCoordinatorTests CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20` +Expected: FAIL — `OuraSyncCoordinator`/`OuraPageFetching` undefined. (Note: confirm `WhoopStore.dailyMetrics(deviceId:from:to:)` is the read accessor name — if it differs, read `MetricsCache.swift` and use the actual reader.) + +- [ ] **Step 3: Implement `OuraSyncCoordinator`** + +Create `Strand/Oura/OuraSyncCoordinator.swift`: + +```swift +import Foundation +import WhoopStore +import StrandImport + +/// Abstracts the page fetch so the coordinator is testable without a live network (OuraAPIClient conforms). +protocol OuraPageFetching { + func fetchAllRaw(endpoint: String, query: [String: String]) async throws -> [Data] +} +extension OuraAPIClient: OuraPageFetching {} + +struct OuraSyncProgress: Equatable { let endpoint: String; let pages: Int } + +/// One-time backfill across every Oura v2 endpoint → an assembled OuraSyncResult → OuraSyncWriter. Merges +/// the per-day WearableDailyRows across endpoints (daily endpoints BEFORE sleep, so readiness RHR wins). +final class OuraSyncCoordinator { + private let fetcher: OuraPageFetching + private let store: WhoopStore + init(fetcher: OuraPageFetching, store: WhoopStore) { self.fetcher = fetcher; self.store = store } + + /// Daily-summary endpoints handed to `parseDaily(_, endpoint:)`. Ordered so `daily_readiness` merges + /// before `sleep` (RHR precedence). `daily_sleep` etc. contribute extras only. + private static let dailyEndpoints = ["daily_readiness", "daily_activity", "daily_spo2", "daily_sleep", + "daily_stress", "daily_resilience", "daily_cardiovascular_age", "vO2_max"] + /// Endpoints with no Plan-1 parser — archived raw only. + private static let rawOnlyEndpoints = ["personal_info", "ring_configuration", "sleep_time", "session", + "tag", "enhanced_tag", "rest_mode_period", "ring_battery_level"] + + func runFullImport(startDate: String = "2013-01-01", today: String, + onProgress: @escaping (OuraSyncProgress) -> Void = { _ in }) async throws -> OuraSyncSummary { + var result = OuraSyncResult() + var byDay: [String: WearableDailyRow] = [:] + + func fetchRaw(_ endpoint: String, dateParam: String?) async throws -> [Data] { + var q: [String: String] = [:] + if let dateParam { q[dateParam] = startDate; q[dateParam == "start_datetime" ? "end_datetime" : "end_date"] = today } + let pages = try await fetcher.fetchAllRaw(endpoint: endpoint, query: q) + var idx = 0 + for body in pages { + result.rawPages.append(OuraRawRow(endpoint: endpoint, documentId: "\(endpoint)-\(startDate)-\(idx)", + day: nil, payloadJSON: String(data: body, encoding: .utf8) ?? "", + fetchedAt: Int(Date().timeIntervalSince1970))) + idx += 1 + } + onProgress(OuraSyncProgress(endpoint: endpoint, pages: pages.count)) + return pages + } + + func docs(_ pages: [Data]) -> [[String: Any]] { + pages.flatMap { (try? JSONSerialization.jsonObject(with: $0) as? [String: Any])?["data"] as? [[String: Any]] ?? [] } + } + func merge(_ rows: [WearableDailyRow]) { + for r in rows { + var m = byDay[r.day] ?? WearableDailyRow(day: r.day) + m.restingHr = m.restingHr ?? r.restingHr; m.avgHrvMs = m.avgHrvMs ?? r.avgHrvMs + m.skinTempDevC = m.skinTempDevC ?? r.skinTempDevC; m.spo2Pct = m.spo2Pct ?? r.spo2Pct + m.steps = m.steps ?? r.steps; m.activeKcal = m.activeKcal ?? r.activeKcal + m.totalKcal = m.totalKcal ?? r.totalKcal; m.respRateBpm = m.respRateBpm ?? r.respRateBpm + m.totalSleepMin = m.totalSleepMin ?? r.totalSleepMin; m.deepMin = m.deepMin ?? r.deepMin + m.lightMin = m.lightMin ?? r.lightMin; m.remMin = m.remMin ?? r.remMin + m.awakeMin = m.awakeMin ?? r.awakeMin; m.efficiencyPct = m.efficiencyPct ?? r.efficiencyPct + byDay[r.day] = m + } + } + + // Daily endpoints first (readiness RHR precedence), extras accumulate. + for endpoint in Self.dailyEndpoints { + let pages = try await fetchRaw(endpoint, dateParam: "start_date") + let (days, extras) = OuraApiParser.parseDaily(docs(pages), endpoint: endpoint) + merge(days); result.extras.append(contentsOf: extras) + } + // Sleep last (its lowestHr fallback only fills days readiness didn't cover). + let sleepPages = try await fetchRaw("sleep", dateParam: "start_date") + let (periods, sleepDays) = OuraApiParser.parseSleep(docs(sleepPages)) + result.sleepPeriods = periods; merge(sleepDays) + // Workouts + heart-rate. + let workoutPages = try await fetchRaw("workout", dateParam: "start_date") + result.workouts = OuraApiParser.parseWorkouts(docs(workoutPages)) + let hrPages = try await fetchRaw("heartrate", dateParam: "start_datetime") + result.heartRate = OuraApiParser.parseHeartRate(docs(hrPages)) + // Raw-only endpoints (no parser). + for endpoint in Self.rawOnlyEndpoints { + _ = try await fetchRaw(endpoint, dateParam: Self.rawOnlyEndpoints.contains(endpoint) && + ["personal_info", "ring_configuration"].contains(endpoint) ? nil : "start_date") + } + + result.days = Array(byDay.values) + return try await OuraSyncWriter.persist(result, into: store) + } +} +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/OuraSyncCoordinatorTests CODE_SIGNING_ALLOWED=NO 2>&1 | tail -20` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add Strand/Oura/OuraSyncCoordinator.swift StrandTests/OuraSyncCoordinatorTests.swift +git commit -m "feat(oura): OuraSyncCoordinator — one-time backfill across all endpoints, coalesced days" +``` + +--- + +### Task 3: `deleteOuraRaw` for clean disconnect + +`store.deleteAllData(deviceId:)` does NOT clear `ouraRaw` (it's not in `deviceScopedTables`). Add a targeted delete so Disconnect leaves nothing behind. + +**Files:** +- Modify: `Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift` +- Test: `Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift` (extend) + +**Interfaces:** +- Produces: `WhoopStore.deleteOuraRaw(deviceId: String) async throws -> Int` + +- [ ] **Step 1: Write the failing test** (append to `OuraRawStoreTests.swift`) + +```swift + func testDeleteOuraRawRemovesAllForDevice() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertOuraRaw([ + OuraRawRow(endpoint: "sleep", documentId: "a", day: "2026-01-01", payloadJSON: "{}", fetchedAt: 1), + OuraRawRow(endpoint: "workout", documentId: "b", day: "2026-01-01", payloadJSON: "{}", fetchedAt: 1), + ], deviceId: "oura-api") + let deleted = try await store.deleteOuraRaw(deviceId: "oura-api") + XCTAssertEqual(deleted, 2) + XCTAssertEqual(try await store.ouraRawCount(deviceId: "oura-api", endpoint: "sleep"), 0) + } +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `swift test --package-path Packages/WhoopStore --filter OuraRawStoreTests` +Expected: FAIL — `deleteOuraRaw` undefined. + +- [ ] **Step 3: Implement** (add to the `extension WhoopStore` in `OuraRawStore.swift`) + +```swift + /// Remove every archived Oura payload for a device (used by Disconnect — `deleteAllData` does not + /// cover `ouraRaw`). Returns rows deleted. + @discardableResult + public func deleteOuraRaw(deviceId: String) async throws -> Int { + try syncWrite { db in + try db.execute(sql: "DELETE FROM ouraRaw WHERE deviceId = ?", arguments: [deviceId]) + return db.changesCount + } + } +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `swift test --package-path Packages/WhoopStore --filter OuraRawStoreTests` +Expected: PASS (this is a package test — fast `swift test`, no xcodebuild). + +- [ ] **Step 5: Commit** + +```bash +git add Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift +git commit -m "feat(store): deleteOuraRaw(deviceId:) for full Oura cloud-source disconnect" +``` + +--- + +### Task 4: `ASPresentationAnchor` helper + Connect UI + +The cross-platform key-window accessor + the "Connect Oura & Import Everything" card. The card is compile-verified + manually tested (SwiftUI + interactive OAuth aren't headless-unit-testable); the anchor helper's logic is the only new unit-testable piece and is trivial, so this task's gate is a clean build. + +**Files:** +- Create: `Strand/Oura/OuraPresentationAnchor.swift` +- Create: `Strand/Oura/OuraConnectModel.swift` (the `@MainActor` view-model driving connect → backfill → summary) +- Modify: `Strand/Screens/DataSourcesView.swift` (add the Oura card + wire the model) + +**Interfaces:** +- Consumes: `OuraOAuthProvider`, `OuraCredentials`, `OuraAPIClient`, `OuraSyncCoordinator`, `OuraTokenStore`, `Repository`. +- Produces: `func ouraPresentationAnchor() -> ASPresentationAnchor`; `@MainActor final class OuraConnectModel: ObservableObject`. + +- [ ] **Step 1: Implement the anchor helper** + +Create `Strand/Oura/OuraPresentationAnchor.swift` (no unit test — platform window lookup, modeled on `Strand/System/DisplayScreenshot.swift`): + +```swift +import AuthenticationServices +#if os(iOS) +import UIKit +#elseif os(macOS) +import AppKit +#endif + +/// The app's key window as an ASPresentationAnchor for ASWebAuthenticationSession. Modeled on +/// DisplayScreenshot's window lookup. Falls back to a bare anchor (OuraOAuthProvider tolerates it). +@MainActor func ouraPresentationAnchor() -> ASPresentationAnchor { + #if os(iOS) + let scenes = UIApplication.shared.connectedScenes.compactMap { $0 as? UIWindowScene } + let scene = scenes.first { $0.activationState == .foregroundActive } ?? scenes.first + return scene?.keyWindow ?? scene?.windows.first ?? ASPresentationAnchor() + #elseif os(macOS) + return NSApplication.shared.keyWindow ?? NSApplication.shared.windows.first ?? ASPresentationAnchor() + #else + return ASPresentationAnchor() + #endif +} +``` + +- [ ] **Step 2: Implement the connect view-model** + +Create `Strand/Oura/OuraConnectModel.swift`: + +```swift +import Foundation +import Combine +import AuthenticationServices + +/// Drives the Data Sources "Connect Oura" card: connect (OAuth) → one-time backfill → honest summary → +/// disconnect. @MainActor so all @Published mutations are main-thread; the network/backfill hops off-main. +@MainActor +final class OuraConnectModel: ObservableObject { + @Published var busy = false + @Published var statusText: String? + @Published var isConnected = OuraTokenStore.isConnected + + private let repo: Repository + init(repo: Repository) { self.repo = repo } + + /// True when the BYO-app credentials are present (else the lane is disabled with guidance). + var isConfigured: Bool { OuraCredentials.fromBundle != nil } + + func connectAndImport() { + guard let creds = OuraCredentials.fromBundle else { + statusText = "Add your Oura app's client_id/secret to OuraSecrets.xcconfig first."; return + } + busy = true; statusText = "Connecting to Oura…" + Task { + do { + let provider = OuraOAuthProvider(credentials: creds) + try await provider.authorize(presentationAnchor: ouraPresentationAnchor()) + isConnected = true + guard let store = await repo.storeHandle() else { fail("No local store."); return } + let client = OuraAPIClient(auth: provider, environment: .production) + let coord = OuraSyncCoordinator(fetcher: client, store: store) + let today = Self.dayFormatter.string(from: Date()) + statusText = "Importing your Oura history…" + let s = try await coord.runFullImport(today: today) { [weak self] p in + Task { @MainActor in self?.statusText = "Importing \(p.endpoint)…" } + } + await repo.refresh() + statusText = "Imported \(s.days) days · \(s.sleeps) sleeps · \(s.workouts) workouts · \(s.hrSamples) HR samples" + } catch { fail((error as? LocalizedError)?.errorDescription ?? error.localizedDescription) } + busy = false + } + } + + func disconnect() { + busy = true + Task { + OuraOAuthProvider(credentials: OuraCredentials.fromBundle ?? .init(clientId: "", clientSecret: "", redirectURI: "")).signOut() + if let store = await repo.storeHandle() { + try? await store.deleteAllData(deviceId: "oura-api") + try? await store.deleteOuraRaw(deviceId: "oura-api") + await repo.refresh() + } + isConnected = false; statusText = "Disconnected."; busy = false + } + } + + private func fail(_ m: String) { statusText = m; busy = false } + private static let dayFormatter: DateFormatter = { + let f = DateFormatter(); f.calendar = Calendar(identifier: .gregorian) + f.locale = Locale(identifier: "en_US_POSIX"); f.timeZone = TimeZone(identifier: "UTC") + f.dateFormat = "yyyy-MM-dd"; return f + }() +} +``` + +- [ ] **Step 3: Add the Oura card to `DataSourcesView`** + +Read `Strand/Screens/DataSourcesView.swift` around the `wearableCard` (~:265) and the shared `card(...)` builder (~:811). Add a `@StateObject private var oura = OuraConnectModel(repo: repo)` (or wire via the existing model), and a card mirroring `wearableCard`, placed next to it: + +```swift +private var ouraCloudCard: some View { + card(title: "Oura (cloud)", icon: "circle.circle", tint: StrandPalette.metricPurple, + subtitle: "Connect your Oura account and import your full history over the API.") { + VStack(alignment: .leading, spacing: 8) { + if oura.isConnected { + HStack { + Button { oura.connectAndImport() } label: { Label("Sync again", systemImage: "arrow.clockwise") } + Button(role: .destructive) { oura.disconnect() } label: { Label("Disconnect", systemImage: "xmark.circle") } + }.disabled(oura.busy) + } else { + Button { oura.connectAndImport() } label: { + Label(oura.busy ? "Working…" : "Connect Oura & Import Everything", systemImage: "link") + }.disabled(oura.busy || !oura.isConfigured) + if !oura.isConfigured { + Text("Add your Oura app credentials to OuraSecrets.xcconfig to enable this.") + .font(.caption).foregroundStyle(.secondary) + } + } + if let s = oura.statusText { Text(s).font(.caption).foregroundStyle(.secondary) } + } + } +} +``` +Then render `ouraCloudCard` in the same list/section as `wearableCard`. + +- [ ] **Step 4: Verify the build (compile-gate; no headless test for SwiftUI/interactive OAuth)** + +Run: `xcodegen generate && xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' build-for-testing CODE_SIGNING_ALLOWED=NO 2>&1 | tail -15` +Expected: `** TEST BUILD SUCCEEDED **`. Manually note in the report: the interactive OAuth + backfill flow is live-tested by the user with real credentials, not in CI. + +- [ ] **Step 5: Commit** + +```bash +git add Strand/Oura/OuraPresentationAnchor.swift Strand/Oura/OuraConnectModel.swift Strand/Screens/DataSourcesView.swift +git commit -m "feat(oura): Connect Oura & Import Everything card + presentation-anchor helper" +``` + +--- + +### Task 5: Scoring source + documentation + +Add `"oura-api"` to the import-source list so cloud-only days can be scored, and update the honest-data docs. + +**Files:** +- Modify: `Strand/Data/Repository.swift` (`wearableImportSources`) +- Modify: `docs/PRIVACY_SECURITY.md`, `docs/DEVICE_SUPPORT_ROADMAP.md`, `docs/DATA_MODEL.md` + +**Interfaces:** none. + +- [ ] **Step 1: Add the scoring source** + +In `Strand/Data/Repository.swift` (~:452): +```swift +static let wearableImportSources = ["oura-import", "fitbit-import", "garmin-import", "oura-api", healthConnectSource] +``` + +- [ ] **Step 2: Docs** + +- `docs/PRIVACY_SECURITY.md` §1: amend "exactly one opt-in exception (the AI Coach)" → "two opt-in exceptions: the AI Coach and the Oura cloud import." Add a short §1.1b describing the Oura lane: off-by-default, user-initiated, inbound (the user's own Oura data under their own OAuth app), tokens in Keychain, no NOOP server, raw data never exfiltrated. +- `docs/DEVICE_SUPPORT_ROADMAP.md`: change the Oura row from "📋 Researched, not built | Cloud API v2 — off-by-default OAuth import lane only" to "🔬 Built (cloud import) | Cloud API v2 — off-by-default OAuth, one-time backfill; local BLE ring also supported." +- `docs/DATA_MODEL.md`: note the current migrator version (v24) and add the `ouraRaw` table (deviceId, endpoint, documentId, day, payloadJSON, fetchedAt) to the metric-caches / new-tables section. + +- [ ] **Step 3: Verify + commit** + +Run: `xcodegen generate && xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' build-for-testing CODE_SIGNING_ALLOWED=NO 2>&1 | tail -8` +Expected: `** TEST BUILD SUCCEEDED **`. + +```bash +git add Strand/Data/Repository.swift docs/PRIVACY_SECURITY.md docs/DEVICE_SUPPORT_ROADMAP.md docs/DATA_MODEL.md +git commit -m "chore(oura): score cloud-import days + document the second opt-in network lane" +``` + +--- + +## Plan 3 self-review + +- **Spec coverage:** §7 orchestration → Task 2; §8 writer/mapping (raw + normalized + registry + coalesce + motion + hrSample) → Task 1; §8.4 disconnect (deleteOuraRaw) → Task 3; §9 Connect UI → Task 4; §8.4 scoring + §10 docs → Task 5. The Plan-1 final-review advisories (coalesce, readiness-RHR precedence, sleep-score/vo2max parity via extras) are baked into Tasks 1–2 and the constraints. +- **Honest data / ownership are structural:** `recovery`/`strain` hardcoded nil; scores only via extras → `ref_*` keys; `.cloudImport` = priority-2 → never seizes a WHOOP day; `setDayOwner` never called. +- **Testability:** writer + coordinator use `WhoopStore.inMemory()` + a stub fetcher (no network); `deleteOuraRaw` is a fast package test; the UI/OAuth (inherently interactive) is compile-gated + user-live-tested, called out honestly. +- **Type consistency:** every `WhoopStore` init/method is the exact signature from the v8.5.2 study; `OuraSyncResult`/`OuraSyncSummary` defined Task 1, consumed Task 2; `OuraPageFetching` defined Task 2, `OuraAPIClient` conformance added there. +- **Risk notes for the implementer (verify against code, adjust minimally, report):** `PairedDevice.status`/`capabilities` enum types; the `dailyMetrics(deviceId:from:to:)` read-accessor name; `StrandPalette`/`card(...)` exact signatures in `DataSourcesView`. + +## After Plan 3 + +The feature is complete and tappable. To use it live, the user registers a free Oura OAuth app (redirect `noop://oura/callback`), drops `client_id`/`secret` into `Strand/Oura/OuraSecrets.xcconfig`, builds, and taps **Connect Oura & Import Everything**. Remaining deferred Minors (from the Plan-1/2 final reviews) — exponential backoff/jitter/pacer in the coordinator, `maxPages` logging, `spo2Daily` scope verification via the sandbox, richer normalization of the 8 raw-only endpoints — are follow-ups, not blockers. From a5a8e7727207dfe47dac9948806e4e7d6f6c01a2 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 21:24:48 -0700 Subject: [PATCH 022/171] =?UTF-8?q?feat(oura):=20OuraSyncWriter=20?= =?UTF-8?q?=E2=80=94=20persist=20backfill=20to=20WhoopStore=20as=20.cloudI?= =?UTF-8?q?mport=20source?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Strand/Oura/OuraSyncResult.swift | 27 ++++++++ Strand/Oura/OuraSyncWriter.swift | 89 +++++++++++++++++++++++++++ StrandTests/OuraSyncWriterTests.swift | 59 ++++++++++++++++++ 3 files changed, 175 insertions(+) create mode 100644 Strand/Oura/OuraSyncResult.swift create mode 100644 Strand/Oura/OuraSyncWriter.swift create mode 100644 StrandTests/OuraSyncWriterTests.swift diff --git a/Strand/Oura/OuraSyncResult.swift b/Strand/Oura/OuraSyncResult.swift new file mode 100644 index 0000000000..c9e15800b0 --- /dev/null +++ b/Strand/Oura/OuraSyncResult.swift @@ -0,0 +1,27 @@ +import Foundation +import StrandImport +import WhoopStore // OuraRawRow lives in WhoopStore (OuraRawStore.swift), not StrandImport + +/// The fully-assembled, parsed output of one Oura backfill, ready to persist. The coordinator (Task 2) +/// builds this (merging per-day rows across endpoints); the writer (below) maps it to WhoopStore. +struct OuraSyncResult { + var days: [WearableDailyRow] // already coalesced per calendar day + var sleepPeriods: [OuraSleepPeriod] + var extras: [OuraDailyExtra] // every ref_*/oura_*/vo2max scalar → metricSeries + var workouts: [OuraWorkout] + var heartRate: [OuraHRPoint] // from the /heartrate endpoint (whole-day) + var rawPages: [OuraRawRow] // every fetched page, verbatim + var ringModel: String? // from ring_configuration, for the PairedDevice model + + init(days: [WearableDailyRow] = [], sleepPeriods: [OuraSleepPeriod] = [], extras: [OuraDailyExtra] = [], + workouts: [OuraWorkout] = [], heartRate: [OuraHRPoint] = [], rawPages: [OuraRawRow] = [], + ringModel: String? = nil) { + self.days = days; self.sleepPeriods = sleepPeriods; self.extras = extras + self.workouts = workouts; self.heartRate = heartRate; self.rawPages = rawPages; self.ringModel = ringModel + } +} + +/// Counts written, for the honest import summary. +struct OuraSyncSummary: Equatable { + var days = 0, sleeps = 0, workouts = 0, hrSamples = 0, metricPoints = 0, rawPages = 0 +} diff --git a/Strand/Oura/OuraSyncWriter.swift b/Strand/Oura/OuraSyncWriter.swift new file mode 100644 index 0000000000..1c728af5d9 --- /dev/null +++ b/Strand/Oura/OuraSyncWriter.swift @@ -0,0 +1,89 @@ +import Foundation +import WhoopStore +import WhoopProtocol +import StrandImport + +/// Maps an assembled `OuraSyncResult` into the on-device WhoopStore under `deviceId = "oura-api"` — the +/// cloud sibling of `Strand/Data/WearableImporter.swift`. HONEST DATA: Oura's own scores go only to +/// ref_*/oura_* metricSeries keys; `DailyMetric.recovery`/`.strain` stay nil. The source registers as +/// `.cloudImport`, which is structurally priority-2 in DayOwnerResolver, so it never seizes a WHOOP day. +enum OuraSyncWriter { + + @discardableResult + static func persist(_ result: OuraSyncResult, into store: WhoopStore, + deviceId: String = "oura-api") async throws -> OuraSyncSummary { + var summary = OuraSyncSummary() + + // 0. Register the cloud source (the one thing WearableImporter skips). Idempotent by id. + let now = Int(Date().timeIntervalSince1970) + let device = PairedDevice( + id: deviceId, brand: "Oura", model: result.ringModel ?? "Oura (cloud)", + sourceKind: .cloudImport, capabilities: [], status: .active, + addedAt: now, lastSeenAt: now) + try DeviceRegistryStore(dbQueue: store.registryWriter).add(device) + + // 1. Raw archive — verbatim, every page, lossless. + if !result.rawPages.isEmpty { + summary.rawPages = try await store.upsertOuraRaw(result.rawPages, deviceId: deviceId) + } + + // 2. Days → DailyMetric. recovery/strain ALWAYS nil (never Oura's readiness). Mirrors WearableImporter. + let metrics = result.days.map { d in + DailyMetric(day: d.day, totalSleepMin: d.totalSleepMin, efficiency: d.efficiencyPct, + deepMin: d.deepMin, remMin: d.remMin, lightMin: d.lightMin, disturbances: nil, + restingHr: d.restingHr, avgHrv: d.avgHrvMs, recovery: nil, strain: nil, + exerciseCount: nil, spo2Pct: d.spo2Pct, skinTempDevC: d.skinTempDevC, + respRateBpm: d.respRateBpm, steps: d.steps, activeKcalEst: d.activeKcal) + } + summary.days = try await store.upsertDailyMetrics(metrics, deviceId: deviceId) + + // 3. Sleep sessions → CachedSleepSession (+ stagesJSON), then per-session motion via the dedicated setter. + var sessions: [CachedSleepSession] = [] + for p in result.sleepPeriods { + let startTs = Int(p.session.start.timeIntervalSince1970) + let endTs = Int(p.session.end.timeIntervalSince1970) + sessions.append(CachedSleepSession( + startTs: startTs, endTs: endTs, efficiency: p.session.efficiencyPct, + restingHr: p.session.lowestHr ?? p.session.avgHr, avgHrv: p.session.avgHrvMs, + stagesJSON: stagesJSON(p.session.stages))) + } + summary.sleeps = try await store.upsertSleepSessions(sessions, deviceId: deviceId) + for p in result.sleepPeriods where !p.movement30s.isEmpty { // motionJSON is NOT on the sleep upsert + _ = try await store.persistSessionMotion( + deviceId: deviceId, sessionStart: Int(p.session.start.timeIntervalSince1970), + motionEpochs: p.movement30s.map(Double.init)) + } + + // 4. HR → hrSample via the Streams insert path (in-sleep HR + the whole-day /heartrate series). + var hr: [HRSample] = result.heartRate.map { HRSample(ts: $0.ts, bpm: $0.bpm) } + for p in result.sleepPeriods { hr.append(contentsOf: p.hr.map { HRSample(ts: $0.ts, bpm: $0.bpm) }) } + if !hr.isEmpty { + let counts = try await store.insert(Streams(hr: hr), deviceId: deviceId) + summary.hrSamples = counts.hr + } + + // 5. Workouts → WorkoutRow. + let workouts = result.workouts.map { w in + WorkoutRow(startTs: Int(w.start.timeIntervalSince1970), endTs: Int(w.end.timeIntervalSince1970), + sport: w.activity, source: w.source, durationS: w.end.timeIntervalSince(w.start), + energyKcal: w.energyKcal, avgHr: nil, maxHr: nil, strain: nil, + distanceM: w.distanceM, zonesJSON: nil, notes: nil) + } + summary.workouts = try await store.upsertWorkouts(workouts, deviceId: deviceId) + + // 6. Extras → metricSeries (ref_*/oura_*/vo2max — parity with the file-import lane, sourced from + // the extras array so vo2max/ref_sleep_score are never lost even though DailyMetric lacks them). + let points = result.extras.map { MetricPoint(day: $0.day, key: $0.key, value: $0.value) } + summary.metricPoints = try await store.upsertMetricSeries(points, deviceId: deviceId) + + return summary + } + + /// Serialize decoded stages to the `[{start,end,stage}]` JSON the cache stores (same shape as WearableImporter). + private static func stagesJSON(_ stages: [WearableSleepStageInterval]) -> String? { + guard !stages.isEmpty else { return nil } + let segs = stages.map { ["start": Int($0.start.timeIntervalSince1970), + "end": Int($0.end.timeIntervalSince1970), "stage": $0.stage] as [String: Any] } + return (try? JSONSerialization.data(withJSONObject: segs)).flatMap { String(data: $0, encoding: .utf8) } + } +} diff --git a/StrandTests/OuraSyncWriterTests.swift b/StrandTests/OuraSyncWriterTests.swift new file mode 100644 index 0000000000..7f459bb1dc --- /dev/null +++ b/StrandTests/OuraSyncWriterTests.swift @@ -0,0 +1,59 @@ +import XCTest +import WhoopStore +import WhoopProtocol +@testable import StrandImport +@testable import Strand + +final class OuraSyncWriterTests: XCTestCase { + private func iso(_ s: String) -> Date { ISO8601DateFormatter().date(from: s)! } + + func testPersistWritesAllStoresAndRegistersCloudSource() async throws { + let store = try await WhoopStore.inMemory() + + var day = WearableDailyRow(day: "2026-01-02") + day.restingHr = 50; day.steps = 9000; day.totalSleepMin = 400 + + let session = WearableSleepSession( + start: iso("2026-01-01T23:00:00Z"), end: iso("2026-01-02T06:00:00Z"), + deepMin: 100, lightMin: 200, remMin: 100, awakeMin: 20, totalSleepMin: 400, + efficiencyPct: 92, avgHr: 55, lowestHr: 48, avgHrvMs: 65, respRateBpm: 14, + sleepScore: nil, stages: [WearableSleepStageInterval(stage: "deep", + start: iso("2026-01-01T23:00:00Z"), end: iso("2026-01-01T23:05:00Z"))]) + let period = OuraSleepPeriod(session: session, movement30s: [1, 2, 1], + hr: [OuraHRPoint(ts: Int(iso("2026-01-01T23:00:00Z").timeIntervalSince1970), bpm: 60)]) + + let result = OuraSyncResult( + days: [day], sleepPeriods: [period], + extras: [OuraDailyExtra(day: "2026-01-02", key: "ref_readiness_score", value: 80), + OuraDailyExtra(day: "2026-01-02", key: "vo2max", value: 44)], + workouts: [OuraWorkout(start: iso("2026-01-02T07:00:00Z"), end: iso("2026-01-02T07:40:00Z"), + activity: "running", source: "autodetected", energyKcal: 410, distanceM: 6200)], + heartRate: [OuraHRPoint(ts: Int(iso("2026-01-02T07:00:00Z").timeIntervalSince1970), bpm: 120)], + rawPages: [OuraRawRow(endpoint: "sleep", documentId: "s1", day: "2026-01-02", + payloadJSON: "{}", fetchedAt: 100)], + ringModel: "Oura Ring Gen3") + + let summary = try await OuraSyncWriter.persist(result, into: store) + + XCTAssertEqual(summary.days, 1) + XCTAssertEqual(summary.sleeps, 1) + XCTAssertEqual(summary.workouts, 1) + XCTAssertGreaterThan(summary.hrSamples, 0) // in-sleep + whole-day HR + XCTAssertGreaterThan(summary.metricPoints, 0) + XCTAssertEqual(summary.rawPages, 1) + + // Cloud source registered as .cloudImport (so it never seizes a WHOOP day). + let devices = try DeviceRegistryStore(dbQueue: store.registryWriter).all() + let oura = devices.first { $0.id == "oura-api" } + XCTAssertEqual(oura?.sourceKind, .cloudImport) + XCTAssertEqual(oura?.model, "Oura Ring Gen3") + + // Honest data: the reference score is a metricSeries key, never a DailyMetric score column. + let refs = try await store.metricSeries(deviceId: "oura-api", key: "ref_readiness_score", + from: "2026-01-01", to: "2026-01-03") + XCTAssertEqual(refs.first?.value, 80) + let vo2 = try await store.metricSeries(deviceId: "oura-api", key: "vo2max", + from: "2026-01-01", to: "2026-01-03") + XCTAssertEqual(vo2.first?.value, 44) // vo2max parity via extras + } +} From d4bce9fd818ac4c54c062a4d2f22fe0707ede6df Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 21:41:56 -0700 Subject: [PATCH 023/171] fix(oura): register cloud source .paired not .active (single-active invariant) + harden test OuraSyncWriter.persist registered the Oura PairedDevice with status: .active, creating a second active row alongside the live WHOOP. DeviceRegistryStore.add does not enforce the at-most-one-active invariant (only setActive() does), and activeDeviceId() feeds day-owner priority-0 plus BLEManager's live-sample deviceId routing, so this was a real data-integrity violation. Every other PairedDevice construction in the app already uses .paired; an import source must never be the active device. Also hardens OuraSyncWriterTests: exact hrSamples count (2, not just >0), asserts the registered source is .paired, and reads the day back via dailyMetrics() to prove recovery/strain stay nil (honest data). --- Strand/Oura/OuraSyncWriter.swift | 2 +- StrandTests/OuraSyncWriterTests.swift | 11 +++++++++-- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/Strand/Oura/OuraSyncWriter.swift b/Strand/Oura/OuraSyncWriter.swift index 1c728af5d9..d0d61a2582 100644 --- a/Strand/Oura/OuraSyncWriter.swift +++ b/Strand/Oura/OuraSyncWriter.swift @@ -18,7 +18,7 @@ enum OuraSyncWriter { let now = Int(Date().timeIntervalSince1970) let device = PairedDevice( id: deviceId, brand: "Oura", model: result.ringModel ?? "Oura (cloud)", - sourceKind: .cloudImport, capabilities: [], status: .active, + sourceKind: .cloudImport, capabilities: [], status: .paired, addedAt: now, lastSeenAt: now) try DeviceRegistryStore(dbQueue: store.registryWriter).add(device) diff --git a/StrandTests/OuraSyncWriterTests.swift b/StrandTests/OuraSyncWriterTests.swift index 7f459bb1dc..cdff54b88f 100644 --- a/StrandTests/OuraSyncWriterTests.swift +++ b/StrandTests/OuraSyncWriterTests.swift @@ -38,15 +38,17 @@ final class OuraSyncWriterTests: XCTestCase { XCTAssertEqual(summary.days, 1) XCTAssertEqual(summary.sleeps, 1) XCTAssertEqual(summary.workouts, 1) - XCTAssertGreaterThan(summary.hrSamples, 0) // in-sleep + whole-day HR + XCTAssertEqual(summary.hrSamples, 2) // in-sleep 60bpm + whole-day 120bpm, distinct ts XCTAssertGreaterThan(summary.metricPoints, 0) XCTAssertEqual(summary.rawPages, 1) - // Cloud source registered as .cloudImport (so it never seizes a WHOOP day). + // Cloud source registered as .cloudImport, .paired (never .active — single-active invariant; + // an import source must never seize the live-device slot alongside a paired WHOOP). let devices = try DeviceRegistryStore(dbQueue: store.registryWriter).all() let oura = devices.first { $0.id == "oura-api" } XCTAssertEqual(oura?.sourceKind, .cloudImport) XCTAssertEqual(oura?.model, "Oura Ring Gen3") + XCTAssertEqual(oura?.status, .paired) // Honest data: the reference score is a metricSeries key, never a DailyMetric score column. let refs = try await store.metricSeries(deviceId: "oura-api", key: "ref_readiness_score", @@ -55,5 +57,10 @@ final class OuraSyncWriterTests: XCTestCase { let vo2 = try await store.metricSeries(deviceId: "oura-api", key: "vo2max", from: "2026-01-01", to: "2026-01-03") XCTAssertEqual(vo2.first?.value, 44) // vo2max parity via extras + + // Honest data, read back through the day accessor: recovery/strain stay nil (never Oura's readiness). + let days = try await store.dailyMetrics(deviceId: "oura-api", from: "2026-01-01", to: "2026-01-03") + XCTAssertNil(days.first?.recovery) + XCTAssertNil(days.first?.strain) } } From 272303a2444cf092ef136f230d1203a98dca1a2c Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 21:46:44 -0700 Subject: [PATCH 024/171] =?UTF-8?q?feat(oura):=20OuraSyncCoordinator=20?= =?UTF-8?q?=E2=80=94=20one-time=20backfill=20across=20all=20endpoints,=20c?= =?UTF-8?q?oalesced=20days?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Strand/Oura/OuraSyncCoordinator.swift | 89 ++++++++++++++++++++++ StrandTests/OuraSyncCoordinatorTests.swift | 37 +++++++++ 2 files changed, 126 insertions(+) create mode 100644 Strand/Oura/OuraSyncCoordinator.swift create mode 100644 StrandTests/OuraSyncCoordinatorTests.swift diff --git a/Strand/Oura/OuraSyncCoordinator.swift b/Strand/Oura/OuraSyncCoordinator.swift new file mode 100644 index 0000000000..8f818d9aad --- /dev/null +++ b/Strand/Oura/OuraSyncCoordinator.swift @@ -0,0 +1,89 @@ +import Foundation +import WhoopStore +import StrandImport + +/// Abstracts the page fetch so the coordinator is testable without a live network (OuraAPIClient conforms). +protocol OuraPageFetching { + func fetchAllRaw(endpoint: String, query: [String: String]) async throws -> [Data] +} +extension OuraAPIClient: OuraPageFetching {} + +struct OuraSyncProgress: Equatable { let endpoint: String; let pages: Int } + +/// One-time backfill across every Oura v2 endpoint → an assembled OuraSyncResult → OuraSyncWriter. Merges +/// the per-day WearableDailyRows across endpoints (daily endpoints BEFORE sleep, so readiness RHR wins). +final class OuraSyncCoordinator { + private let fetcher: OuraPageFetching + private let store: WhoopStore + init(fetcher: OuraPageFetching, store: WhoopStore) { self.fetcher = fetcher; self.store = store } + + /// Daily-summary endpoints handed to `parseDaily(_, endpoint:)`. Ordered so `daily_readiness` merges + /// before `sleep` (RHR precedence). `daily_sleep` etc. contribute extras only. + private static let dailyEndpoints = ["daily_readiness", "daily_activity", "daily_spo2", "daily_sleep", + "daily_stress", "daily_resilience", "daily_cardiovascular_age", "vO2_max"] + /// Endpoints with no Plan-1 parser — archived raw only. + private static let rawOnlyEndpoints = ["personal_info", "ring_configuration", "sleep_time", "session", + "tag", "enhanced_tag", "rest_mode_period", "ring_battery_level"] + + func runFullImport(startDate: String = "2013-01-01", today: String, + onProgress: @escaping (OuraSyncProgress) -> Void = { _ in }) async throws -> OuraSyncSummary { + var result = OuraSyncResult() + var byDay: [String: WearableDailyRow] = [:] + + func fetchRaw(_ endpoint: String, dateParam: String?) async throws -> [Data] { + var q: [String: String] = [:] + if let dateParam { q[dateParam] = startDate; q[dateParam == "start_datetime" ? "end_datetime" : "end_date"] = today } + let pages = try await fetcher.fetchAllRaw(endpoint: endpoint, query: q) + var idx = 0 + for body in pages { + result.rawPages.append(OuraRawRow(endpoint: endpoint, documentId: "\(endpoint)-\(startDate)-\(idx)", + day: nil, payloadJSON: String(data: body, encoding: .utf8) ?? "", + fetchedAt: Int(Date().timeIntervalSince1970))) + idx += 1 + } + onProgress(OuraSyncProgress(endpoint: endpoint, pages: pages.count)) + return pages + } + + func docs(_ pages: [Data]) -> [[String: Any]] { + pages.flatMap { (try? JSONSerialization.jsonObject(with: $0) as? [String: Any])?["data"] as? [[String: Any]] ?? [] } + } + func merge(_ rows: [WearableDailyRow]) { + for r in rows { + var m = byDay[r.day] ?? WearableDailyRow(day: r.day) + m.restingHr = m.restingHr ?? r.restingHr; m.avgHrvMs = m.avgHrvMs ?? r.avgHrvMs + m.skinTempDevC = m.skinTempDevC ?? r.skinTempDevC; m.spo2Pct = m.spo2Pct ?? r.spo2Pct + m.steps = m.steps ?? r.steps; m.activeKcal = m.activeKcal ?? r.activeKcal + m.totalKcal = m.totalKcal ?? r.totalKcal; m.respRateBpm = m.respRateBpm ?? r.respRateBpm + m.totalSleepMin = m.totalSleepMin ?? r.totalSleepMin; m.deepMin = m.deepMin ?? r.deepMin + m.lightMin = m.lightMin ?? r.lightMin; m.remMin = m.remMin ?? r.remMin + m.awakeMin = m.awakeMin ?? r.awakeMin; m.efficiencyPct = m.efficiencyPct ?? r.efficiencyPct + byDay[r.day] = m + } + } + + // Daily endpoints first (readiness RHR precedence), extras accumulate. + for endpoint in Self.dailyEndpoints { + let pages = try await fetchRaw(endpoint, dateParam: "start_date") + let (days, extras) = OuraApiParser.parseDaily(docs(pages), endpoint: endpoint) + merge(days); result.extras.append(contentsOf: extras) + } + // Sleep last (its lowestHr fallback only fills days readiness didn't cover). + let sleepPages = try await fetchRaw("sleep", dateParam: "start_date") + let (periods, sleepDays) = OuraApiParser.parseSleep(docs(sleepPages)) + result.sleepPeriods = periods; merge(sleepDays) + // Workouts + heart-rate. + let workoutPages = try await fetchRaw("workout", dateParam: "start_date") + result.workouts = OuraApiParser.parseWorkouts(docs(workoutPages)) + let hrPages = try await fetchRaw("heartrate", dateParam: "start_datetime") + result.heartRate = OuraApiParser.parseHeartRate(docs(hrPages)) + // Raw-only endpoints (no parser). + for endpoint in Self.rawOnlyEndpoints { + _ = try await fetchRaw(endpoint, dateParam: Self.rawOnlyEndpoints.contains(endpoint) && + ["personal_info", "ring_configuration"].contains(endpoint) ? nil : "start_date") + } + + result.days = Array(byDay.values) + return try await OuraSyncWriter.persist(result, into: store) + } +} diff --git a/StrandTests/OuraSyncCoordinatorTests.swift b/StrandTests/OuraSyncCoordinatorTests.swift new file mode 100644 index 0000000000..a84a124bf5 --- /dev/null +++ b/StrandTests/OuraSyncCoordinatorTests.swift @@ -0,0 +1,37 @@ +import XCTest +import WhoopStore +@testable import StrandImport +@testable import Strand + +/// A fetcher that serves canned page bodies per endpoint. +private final class StubFetcher: OuraPageFetching { + var pages: [String: [Data]] = [:] + func fetchAllRaw(endpoint: String, query: [String: String]) async throws -> [Data] { + pages[endpoint] ?? [] + } +} + +final class OuraSyncCoordinatorTests: XCTestCase { + func testBackfillFetchesParsesAndWrites() async throws { + let store = try await WhoopStore.inMemory() + let fetcher = StubFetcher() + fetcher.pages["daily_readiness"] = [#"{"data":[{"day":"2026-01-02","score":80,"contributors":{"resting_heart_rate":50}}],"next_token":null}"#.data(using: .utf8)!] + fetcher.pages["daily_activity"] = [#"{"data":[{"day":"2026-01-02","steps":9000}],"next_token":null}"#.data(using: .utf8)!] + fetcher.pages["sleep"] = [#"{"data":[{"id":"s1","day":"2026-01-02","bedtime_start":"2026-01-01T23:00:00+00:00","bedtime_end":"2026-01-02T06:00:00+00:00","total_sleep_duration":24000,"lowest_heart_rate":48,"sleep_phase_5_min":"1234"}],"next_token":null}"#.data(using: .utf8)!] + + var progress: [String] = [] + let coord = OuraSyncCoordinator(fetcher: fetcher, store: store) + let summary = try await coord.runFullImport(today: "2026-02-01") { progress.append($0.endpoint) } + + XCTAssertEqual(summary.days, 1) + XCTAssertEqual(summary.sleeps, 1) + XCTAssertTrue(summary.rawPages >= 3) + XCTAssertTrue(progress.contains("sleep")) + + // Coalesce + RHR precedence: daily_readiness's true RHR (50) wins over sleep's lowestHr (48). + let days = try await store.dailyMetrics(deviceId: "oura-api", from: "2026-01-01", to: "2026-01-03") + XCTAssertEqual(days.first?.restingHr, 50) + XCTAssertEqual(days.first?.steps, 9000) + XCTAssertEqual(days.first?.totalSleepMin, 400) + } +} From b10ba896cbf905abb38cd6be4412f8a5c501681d Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 21:58:35 -0700 Subject: [PATCH 025/171] fix(oura): assert endpoint date-param wiring + drop tautological clause (Task 2 review) --- Strand/Oura/OuraSyncCoordinator.swift | 3 +-- StrandTests/OuraSyncCoordinatorTests.swift | 21 ++++++++++++++++++++- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/Strand/Oura/OuraSyncCoordinator.swift b/Strand/Oura/OuraSyncCoordinator.swift index 8f818d9aad..ed37426574 100644 --- a/Strand/Oura/OuraSyncCoordinator.swift +++ b/Strand/Oura/OuraSyncCoordinator.swift @@ -79,8 +79,7 @@ final class OuraSyncCoordinator { result.heartRate = OuraApiParser.parseHeartRate(docs(hrPages)) // Raw-only endpoints (no parser). for endpoint in Self.rawOnlyEndpoints { - _ = try await fetchRaw(endpoint, dateParam: Self.rawOnlyEndpoints.contains(endpoint) && - ["personal_info", "ring_configuration"].contains(endpoint) ? nil : "start_date") + _ = try await fetchRaw(endpoint, dateParam: ["personal_info", "ring_configuration"].contains(endpoint) ? nil : "start_date") } result.days = Array(byDay.values) diff --git a/StrandTests/OuraSyncCoordinatorTests.swift b/StrandTests/OuraSyncCoordinatorTests.swift index a84a124bf5..5afe1079fe 100644 --- a/StrandTests/OuraSyncCoordinatorTests.swift +++ b/StrandTests/OuraSyncCoordinatorTests.swift @@ -6,8 +6,10 @@ import WhoopStore /// A fetcher that serves canned page bodies per endpoint. private final class StubFetcher: OuraPageFetching { var pages: [String: [Data]] = [:] + var capturedQueries: [String: [String: String]] = [:] func fetchAllRaw(endpoint: String, query: [String: String]) async throws -> [Data] { - pages[endpoint] ?? [] + capturedQueries[endpoint] = query + return pages[endpoint] ?? [] } } @@ -34,4 +36,21 @@ final class OuraSyncCoordinatorTests: XCTestCase { XCTAssertEqual(days.first?.steps, 9000) XCTAssertEqual(days.first?.totalSleepMin, 400) } + + func testEndpointDateParamsAreWiredCorrectly() async throws { + let store = try await WhoopStore.inMemory() + let fetcher = StubFetcher() + let coord = OuraSyncCoordinator(fetcher: fetcher, store: store) + _ = try await coord.runFullImport(today: "2026-02-01") + + XCTAssertNotNil(fetcher.capturedQueries["heartrate"]?["start_datetime"]) + XCTAssertNotNil(fetcher.capturedQueries["heartrate"]?["end_datetime"]) + XCTAssertNil(fetcher.capturedQueries["heartrate"]?["start_date"]) + + XCTAssertEqual(fetcher.capturedQueries["personal_info"], [:]) + XCTAssertEqual(fetcher.capturedQueries["ring_configuration"], [:]) + + XCTAssertNotNil(fetcher.capturedQueries["daily_readiness"]?["start_date"]) + XCTAssertNotNil(fetcher.capturedQueries["daily_readiness"]?["end_date"]) + } } From 1f13ec50fe33409b6a46e4fdbd1151d3aacb8ffb Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 22:00:40 -0700 Subject: [PATCH 026/171] feat(store): deleteOuraRaw(deviceId:) for full Oura cloud-source disconnect --- .../WhoopStore/Sources/WhoopStore/OuraRawStore.swift | 10 ++++++++++ .../Tests/WhoopStoreTests/OuraRawStoreTests.swift | 12 ++++++++++++ 2 files changed, 22 insertions(+) diff --git a/Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift b/Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift index dc8596a0ab..23fefba9a4 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift @@ -64,4 +64,14 @@ extension WhoopStore { arguments: [deviceId, endpoint]) ?? 0 } } + + /// Remove every archived Oura payload for a device (used by Disconnect — `deleteAllData` does not + /// cover `ouraRaw`). Returns rows deleted. + @discardableResult + public func deleteOuraRaw(deviceId: String) async throws -> Int { + try syncWrite { db in + try db.execute(sql: "DELETE FROM ouraRaw WHERE deviceId = ?", arguments: [deviceId]) + return db.changesCount + } + } } diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift index 38cb202db4..633a03c6e1 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/OuraRawStoreTests.swift @@ -25,4 +25,16 @@ final class OuraRawStoreTests: XCTestCase { let dailySleepCount = try await store.ouraRawCount(deviceId: "oura-api", endpoint: "daily_sleep") XCTAssertEqual(dailySleepCount, 0) } + + func testDeleteOuraRawRemovesAllForDevice() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertOuraRaw([ + OuraRawRow(endpoint: "sleep", documentId: "a", day: "2026-01-01", payloadJSON: "{}", fetchedAt: 1), + OuraRawRow(endpoint: "workout", documentId: "b", day: "2026-01-01", payloadJSON: "{}", fetchedAt: 1), + ], deviceId: "oura-api") + let deleted = try await store.deleteOuraRaw(deviceId: "oura-api") + XCTAssertEqual(deleted, 2) + let remainingCount = try await store.ouraRawCount(deviceId: "oura-api", endpoint: "sleep") + XCTAssertEqual(remainingCount, 0) + } } From 5bb7bbd96ea95c64606d1163f495d56e170eb631 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 9 Jul 2026 22:10:50 -0700 Subject: [PATCH 027/171] feat(oura): Connect Oura & Import Everything card + presentation-anchor helper --- Strand/Oura/OuraConnectModel.swift | 68 ++++++++++++++++++++++++ Strand/Oura/OuraPresentationAnchor.swift | 20 +++++++ Strand/Screens/DataSourcesView.swift | 41 ++++++++++++-- 3 files changed, 125 insertions(+), 4 deletions(-) create mode 100644 Strand/Oura/OuraConnectModel.swift create mode 100644 Strand/Oura/OuraPresentationAnchor.swift diff --git a/Strand/Oura/OuraConnectModel.swift b/Strand/Oura/OuraConnectModel.swift new file mode 100644 index 0000000000..878de2ef6d --- /dev/null +++ b/Strand/Oura/OuraConnectModel.swift @@ -0,0 +1,68 @@ +import Foundation +import Combine +import AuthenticationServices + +/// Drives the Data Sources "Connect Oura" card: connect (OAuth) → one-time backfill → honest summary → +/// disconnect. @MainActor so all @Published mutations are main-thread; the network/backfill hops off-main. +/// +/// `repo: Repository` is taken as a PARAMETER on each action rather than stored at construction. `Repository` +/// only reaches `DataSourcesView` via `@EnvironmentObject`, which SwiftUI populates after `init()` runs — +/// storing it in a property initializer (`@StateObject private var oura = OuraConnectModel(repo: repo)`) +/// either fails to compile (a property initializer can't reference another instance member) or, moved into +/// a custom `init()`, crashes at runtime ("No ObservableObject of type Repository found") because the +/// environment isn't resolved yet. This mirrors how the sibling `wearableCard`/`importWearable(url:)` on +/// `DataSourcesView` read `repo` lazily inside their action closures instead of at construction time. +@MainActor +final class OuraConnectModel: ObservableObject { + @Published var busy = false + @Published var statusText: String? + @Published var isConnected = OuraTokenStore.isConnected + + /// True when the BYO-app credentials are present (else the lane is disabled with guidance). + var isConfigured: Bool { OuraCredentials.fromBundle != nil } + + func connectAndImport(repo: Repository) { + guard let creds = OuraCredentials.fromBundle else { + statusText = "Add your Oura app's client_id/secret to OuraSecrets.xcconfig first."; return + } + busy = true; statusText = "Connecting to Oura…" + Task { + do { + let provider = OuraOAuthProvider(credentials: creds) + try await provider.authorize(presentationAnchor: ouraPresentationAnchor()) + isConnected = true + guard let store = await repo.storeHandle() else { fail("No local store."); return } + let client = OuraAPIClient(auth: provider, environment: .production) + let coord = OuraSyncCoordinator(fetcher: client, store: store) + let today = Self.dayFormatter.string(from: Date()) + statusText = "Importing your Oura history…" + let s = try await coord.runFullImport(today: today) { [weak self] p in + Task { @MainActor in self?.statusText = "Importing \(p.endpoint)…" } + } + await repo.refresh() + statusText = "Imported \(s.days) days · \(s.sleeps) sleeps · \(s.workouts) workouts · \(s.hrSamples) HR samples" + } catch { fail((error as? LocalizedError)?.errorDescription ?? error.localizedDescription) } + busy = false + } + } + + func disconnect(repo: Repository) { + busy = true + Task { + OuraOAuthProvider(credentials: OuraCredentials.fromBundle ?? .init(clientId: "", clientSecret: "", redirectURI: "")).signOut() + if let store = await repo.storeHandle() { + try? await store.deleteAllData(deviceId: "oura-api") + try? await store.deleteOuraRaw(deviceId: "oura-api") + await repo.refresh() + } + isConnected = false; statusText = "Disconnected."; busy = false + } + } + + private func fail(_ m: String) { statusText = m; busy = false } + private static let dayFormatter: DateFormatter = { + let f = DateFormatter(); f.calendar = Calendar(identifier: .gregorian) + f.locale = Locale(identifier: "en_US_POSIX"); f.timeZone = TimeZone(identifier: "UTC") + f.dateFormat = "yyyy-MM-dd"; return f + }() +} diff --git a/Strand/Oura/OuraPresentationAnchor.swift b/Strand/Oura/OuraPresentationAnchor.swift new file mode 100644 index 0000000000..337964fd63 --- /dev/null +++ b/Strand/Oura/OuraPresentationAnchor.swift @@ -0,0 +1,20 @@ +import AuthenticationServices +#if os(iOS) +import UIKit +#elseif os(macOS) +import AppKit +#endif + +/// The app's key window as an ASPresentationAnchor for ASWebAuthenticationSession. Modeled on +/// DisplayScreenshot's window lookup. Falls back to a bare anchor (OuraOAuthProvider tolerates it). +@MainActor func ouraPresentationAnchor() -> ASPresentationAnchor { + #if os(iOS) + let scenes = UIApplication.shared.connectedScenes.compactMap { $0 as? UIWindowScene } + let scene = scenes.first { $0.activationState == .foregroundActive } ?? scenes.first + return scene?.keyWindow ?? scene?.windows.first ?? ASPresentationAnchor() + #elseif os(macOS) + return NSApplication.shared.keyWindow ?? NSApplication.shared.windows.first ?? ASPresentationAnchor() + #else + return ASPresentationAnchor() + #endif +} diff --git a/Strand/Screens/DataSourcesView.swift b/Strand/Screens/DataSourcesView.swift index 045b30edd5..b1d5b874cc 100644 --- a/Strand/Screens/DataSourcesView.swift +++ b/Strand/Screens/DataSourcesView.swift @@ -32,6 +32,12 @@ struct DataSourcesView: View { @State private var wearableImporting = false @State private var wearableSummary: String? @State private var wearableFailed = false + // Oura (cloud) import: connect via OAuth and backfill the full history over the Oura API, as an + // alternative to the manual "Oura / Fitbit / Garmin export" file above. `OuraConnectModel` takes + // `repo: Repository` as a call-time parameter (not at construction) — `repo` is an + // `@EnvironmentObject`, unavailable until after this view's `init()` runs, so storing it at + // `@StateObject` construction time would either fail to compile or crash at runtime. + @StateObject private var oura = OuraConnectModel() // "Remove Apple Health imported data" (ah-delete #616): a destructive escape hatch that purges every // row stored under the "apple-health" source via DeviceRegistryStore.deleteAllData. Two-step (a // confirmation alert) since it can't be undone. Local to this screen; no live strap data is touched. @@ -70,8 +76,8 @@ struct DataSourcesView: View { ScreenScaffold(title: "Data Sources", subtitle: "Everything stays on \(Platform.deviceNounPhrase). Bring your history in once, then it's yours.", onRefresh: { await repo.refresh() }, - // PERF: a nine-card import/source column (WHOOP, Apple Health, Xiaomi, nutrition, - // lifting, activity files, wearables, broadcast-out, live strap). The LazyVStack + // PERF: a ten-card import/source column (WHOOP, Apple Health, Xiaomi, nutrition, + // lifting, activity files, wearables, Oura cloud, broadcast-out, live strap). The LazyVStack // path is byte-identical layout. The cards stay in their inner VStack(sectionSpacing) // for pixel-identical spacing, so the lazy win is partial until they're promoted to // direct children. NOTE: this screen still observes `LiveState` for the broadcaster @@ -86,8 +92,9 @@ struct DataSourcesView: View { liftingCard.staggeredAppear(index: 4) activityFileCard.staggeredAppear(index: 5) wearableCard.staggeredAppear(index: 6) - broadcastHrCard.staggeredAppear(index: 7) - liveCard.staggeredAppear(index: 8) + ouraCloudCard.staggeredAppear(index: 7) + broadcastHrCard.staggeredAppear(index: 8) + liveCard.staggeredAppear(index: 9) } } .onAppear { @@ -281,6 +288,32 @@ struct DataSourcesView: View { } } + /// Oura (cloud): OAuth connect + one-time API backfill, as an alternative to the manual export file + /// above. `oura.connectAndImport(repo:)`/`disconnect(repo:)` take `repo` at call time (see the + /// `@StateObject` declaration's note) rather than storing it in `OuraConnectModel` at construction. + private var ouraCloudCard: some View { + card(title: "Oura (cloud)", icon: "circle.circle", tint: StrandPalette.metricPurple, + subtitle: "Connect your Oura account and import your full history over the API.") { + VStack(alignment: .leading, spacing: 8) { + if oura.isConnected { + HStack { + Button { oura.connectAndImport(repo: repo) } label: { Label("Sync again", systemImage: "arrow.clockwise") } + Button(role: .destructive) { oura.disconnect(repo: repo) } label: { Label("Disconnect", systemImage: "xmark.circle") } + }.disabled(oura.busy) + } else { + Button { oura.connectAndImport(repo: repo) } label: { + Label(oura.busy ? "Working…" : "Connect Oura & Import Everything", systemImage: "link") + }.disabled(oura.busy || !oura.isConfigured) + if !oura.isConfigured { + Text("Add your Oura app credentials to OuraSecrets.xcconfig to enable this.") + .font(.caption).foregroundStyle(.secondary) + } + } + if let s = oura.statusText { Text(s).font(.caption).foregroundStyle(.secondary) } + } + } + } + private func presentImporter(_ target: ImportTarget) { importTarget = target #if os(iOS) From cbc009b00d3d740363fc7f2db6b4e3c6af7f782a Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 10:14:03 -0700 Subject: [PATCH 028/171] fix(oura): NoopButtonStyle + localized strings on Oura card; honest disconnect (Task 4 review) --- Strand/Oura/OuraConnectModel.swift | 13 ++++++++++--- Strand/Screens/DataSourcesView.swift | 10 +++++++--- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/Strand/Oura/OuraConnectModel.swift b/Strand/Oura/OuraConnectModel.swift index 878de2ef6d..b41706b310 100644 --- a/Strand/Oura/OuraConnectModel.swift +++ b/Strand/Oura/OuraConnectModel.swift @@ -51,11 +51,18 @@ final class OuraConnectModel: ObservableObject { Task { OuraOAuthProvider(credentials: OuraCredentials.fromBundle ?? .init(clientId: "", clientSecret: "", redirectURI: "")).signOut() if let store = await repo.storeHandle() { - try? await store.deleteAllData(deviceId: "oura-api") - try? await store.deleteOuraRaw(deviceId: "oura-api") + do { + try await store.deleteAllData(deviceId: "oura-api") + try await store.deleteOuraRaw(deviceId: "oura-api") + statusText = "Disconnected." + } catch { + statusText = "Disconnected, but couldn't fully clear local Oura data: \(error.localizedDescription)" + } await repo.refresh() + } else { + statusText = "Disconnected." } - isConnected = false; statusText = "Disconnected."; busy = false + isConnected = false; busy = false } } diff --git a/Strand/Screens/DataSourcesView.swift b/Strand/Screens/DataSourcesView.swift index b1d5b874cc..88448928b1 100644 --- a/Strand/Screens/DataSourcesView.swift +++ b/Strand/Screens/DataSourcesView.swift @@ -292,18 +292,22 @@ struct DataSourcesView: View { /// above. `oura.connectAndImport(repo:)`/`disconnect(repo:)` take `repo` at call time (see the /// `@StateObject` declaration's note) rather than storing it in `OuraConnectModel` at construction. private var ouraCloudCard: some View { - card(title: "Oura (cloud)", icon: "circle.circle", tint: StrandPalette.metricPurple, - subtitle: "Connect your Oura account and import your full history over the API.") { + card(title: String(localized: "Oura (cloud)"), icon: "circle.circle", tint: StrandPalette.metricPurple, + subtitle: String(localized: "Connect your Oura account and import your full history over the API.")) { VStack(alignment: .leading, spacing: 8) { if oura.isConnected { HStack { Button { oura.connectAndImport(repo: repo) } label: { Label("Sync again", systemImage: "arrow.clockwise") } + .buttonStyle(NoopButtonStyle(.primary)) Button(role: .destructive) { oura.disconnect(repo: repo) } label: { Label("Disconnect", systemImage: "xmark.circle") } + .buttonStyle(NoopButtonStyle(.destructive)) }.disabled(oura.busy) } else { Button { oura.connectAndImport(repo: repo) } label: { Label(oura.busy ? "Working…" : "Connect Oura & Import Everything", systemImage: "link") - }.disabled(oura.busy || !oura.isConfigured) + } + .buttonStyle(NoopButtonStyle(.primary)) + .disabled(oura.busy || !oura.isConfigured) if !oura.isConfigured { Text("Add your Oura app credentials to OuraSecrets.xcconfig to enable this.") .font(.caption).foregroundStyle(.secondary) From 8ae3e2ed1f5c3a984a47e8ebeb885cfcaac13620 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 10:38:20 -0700 Subject: [PATCH 029/171] chore(oura): score cloud-import days + document the second opt-in network lane --- Strand/Data/Repository.swift | 2 +- docs/DATA_MODEL.md | 55 ++++++++++++- docs/DEVICE_SUPPORT_ROADMAP.md | 2 +- docs/PRIVACY_SECURITY.md | 143 +++++++++++++++++++++++---------- 4 files changed, 154 insertions(+), 48 deletions(-) diff --git a/Strand/Data/Repository.swift b/Strand/Data/Repository.swift index 6653685b40..713669a0b0 100644 --- a/Strand/Data/Repository.swift +++ b/Strand/Data/Repository.swift @@ -449,7 +449,7 @@ final class Repository: ObservableObject { /// stream, so the source-only fold in IntelligenceEngine scores them from the daily aggregate vs the /// person's own baseline. Matches `WearableBrand.sourceId` plus Health Connect (Android imports HC's /// daily metrics under the strap source, but a sideloaded/standalone HC source id is covered too). - static let wearableImportSources = ["oura-import", "fitbit-import", "garmin-import", healthConnectSource] + static let wearableImportSources = ["oura-import", "fitbit-import", "garmin-import", "oura-api", healthConnectSource] /// `yyyy-MM-dd` in the device's local zone, matching how `DailyMetric.day` is stored. private static let dayKeyFormatter: DateFormatter = { diff --git a/docs/DATA_MODEL.md b/docs/DATA_MODEL.md index 23d4db19f2..93f93a2807 100644 --- a/docs/DATA_MODEL.md +++ b/docs/DATA_MODEL.md @@ -60,14 +60,20 @@ single `DatabaseQueue` and applies these PRAGMAs before any query runs: | `busyMode` | `.timeout(5)` | 5-second busy timeout under write contention. | `WhoopStore` is an `actor`: all GRDB calls run on the actor's serial executor (off the main -thread) through the `syncRead` / `syncWrite` helpers. The reported schema version is -`WhoopStoreInfo.schemaVersion = 9`. +thread) through the `syncRead` / `syncWrite` helpers. `WhoopStoreInfo.schemaVersion` is a +separate, manually-maintained constant (currently `18`) that has lagged the real migration +history for a while and should not be read as the schema's true version. The migrator itself +(`makeMigrator()`, below) is the source of truth for what tables/columns exist, and has run +through **v24** (`v24-oura-raw` — the Oura raw-payload archive, the newest addition; see +below). --- ## Schema at a glance -The schema falls into four groups: +The schema falls into five groups (this section predates, and undercounts, everything added +after v9 — see the schema-version note above; the Oura raw archive below is the one +post-v9 addition currently documented here): | Group | Tables | Origin | | --- | --- | --- | @@ -76,6 +82,7 @@ The schema falls into four groups: | **Raw outbox** (transient) | `rawBatch` | Compressed raw BLE frames, prunable | | **Bookkeeping** | `cursors` | Highwater / read cursors | | **Metric caches** | `sleepSession`, `dailyMetric`, `journal`, `workout`, `appleDaily`, `metricSeries` | Derived metrics + CSV / Apple-Health imports | +| **Oura raw archive** (durable, v24) | `ouraRaw` | Verbatim Oura API payloads behind the opt-in cloud import — see below | All timestamp columns named `ts`, `startTs`, `endTs`, `capturedAt`, etc. are **unix seconds** (integers). Day-keyed cache tables use a `day` text column in `YYYY-MM-DD` form and compare it @@ -441,12 +448,54 @@ those reads index-only. Accessors: `upsertMetricSeries(...)`, `metricSeries(...) --- +## Oura raw-payload archive + +This section documents the one table added after this document's v9 baseline (see the +schema-version note above): the lossless backstop behind the opt-in Oura cloud import +(off by default; user-initiated OAuth backfill — `docs/PRIVACY_SECURITY.md` §1.1b). It is +**not** a metric cache like the tables above — it stores verbatim API responses, not decoded +values, so any field Oura returns can be re-derived later without re-fetching. + +### `ouraRaw` *(v24)* + +One row per archived Oura API document (`OuraRawStore.swift`, `struct OuraRawRow`). +Natural key `(deviceId, endpoint, documentId)`. Written by the Oura sync coordinator in the +app target (`Strand/Oura/`); migration `v24-oura-raw` +(`Packages/WhoopStore/Sources/WhoopStore/Database.swift`) — additive only, a new table, no +existing row touched. + +| Column | Type | Notes | +| --- | --- | --- | +| `deviceId` | TEXT NOT NULL | Part of PK. `"oura-api"` for the live cloud-import lane. | +| `endpoint` | TEXT NOT NULL | Part of PK. Oura endpoint name, e.g. `"sleep"`, `"daily_readiness"`, `"heartrate"`. | +| `documentId` | TEXT NOT NULL | Part of PK. Oura's own `id` for the document; heartrate pages synthesize a window key. | +| `day` | TEXT | `YYYY-MM-DD`, nullable — set when the document is day-keyed. | +| `payloadJSON` | TEXT NOT NULL | Verbatim Oura response object. | +| `fetchedAt` | INTEGER NOT NULL | Unix seconds. | + +**Primary key:** `(deviceId, endpoint, documentId)`. `upsertOuraRaw(...)` is idempotent on this +key via `ON CONFLICT(...) DO UPDATE` — a re-pulled document overwrites its +`day`/`payloadJSON`/`fetchedAt` rather than duplicating. + +**Index** — `idx_ouraRaw_device_endpoint_day` on `(deviceId, endpoint, day)`, so per-endpoint +reads (`ouraRaw(deviceId:endpoint:)`) scan `(deviceId, endpoint)` and walk `day` in order +without a table scan. + +**Not covered by `deleteAllData(deviceId:)`.** Unlike the metric-cache tables above, `ouraRaw` +is not in `DeviceRegistryStore.deviceScopedTables`, so the general per-device wipe skips it by +construction. Disconnecting Oura calls the dedicated `deleteOuraRaw(deviceId:)` alongside +`deleteAllData(deviceId:)` (`Strand/Oura/OuraConnectModel.swift`) so the raw archive is purged +too, not left behind. + +--- + ## Index summary | Index | Table | Columns | Purpose | | --- | --- | --- | --- | | *(implicit PK)* | every table above | (its natural key) | Dedupe + primary lookup. | | `idx_metricSeries_device_key_day` | `metricSeries` | `deviceId, key, day` | Index-only per-metric range reads. | +| `idx_ouraRaw_device_endpoint_day` | `ouraRaw` | `deviceId, endpoint, day` | Index-only per-endpoint range reads. | Every other table relies on its primary-key index; the decoded-stream and date-range reads are all served by the `(deviceId, ts)` / `(deviceId, day)` / `(deviceId, startTs)` primary keys. diff --git a/docs/DEVICE_SUPPORT_ROADMAP.md b/docs/DEVICE_SUPPORT_ROADMAP.md index c28d840600..b1560e0f32 100644 --- a/docs/DEVICE_SUPPORT_ROADMAP.md +++ b/docs/DEVICE_SUPPORT_ROADMAP.md @@ -14,7 +14,7 @@ source stands and the protocol facts we've verified, so the next build can pick | **Polar deep streams** (ECG / PPG / ACC / PPI) | 🔬 Protocol verified, decoder not built | PMD service (below) — alpha, hardware-gated | | **Garmin** (sleep / HRV / Body Battery / SpO₂ / FIT) | 📋 Researched, not built | Local BLE re-derive (Gadgetbridge-informed, **never** GPLv3 copy) | | **Amazfit / Zepp** (incl. Helio deep) | 📋 Researched, not built | Encrypted Huami BLE — needs a one-time **user-pasted** vendor key (NOOP never logs into the vendor cloud) | -| **Oura** | 📋 Researched, not built | Cloud API v2 — off-by-default OAuth **import** lane only | +| **Oura** | 🔬 Built (cloud import) | Cloud API v2 — off-by-default OAuth, one-time backfill; local BLE ring also supported | | **Fitbit / Google** | 📋 Researched, not built | Build against **Google Health** API (Fitbit Web API sunsets Sept 2026) — off-by-default import | ## Polar Measurement Data (PMD) — verified protocol diff --git a/docs/PRIVACY_SECURITY.md b/docs/PRIVACY_SECURITY.md index 1232f3c519..c078b9d03d 100644 --- a/docs/PRIVACY_SECURITY.md +++ b/docs/PRIVACY_SECURITY.md @@ -26,37 +26,47 @@ local SQLite — has no network layer at all: no phone-home, no analytics, no ac no login, no cloud sync, and no telemetry. Everything NOOP computes about you lives in a single SQLite file on your own device. -There is exactly **one** opt-in exception: the **AI Coach** (§1.1a). It is off until you -turn it on with your own API key; when you ask it a question it sends a short text -summary of your recent metrics to the provider you choose. Nothing else in the app ever -touches the network, and your raw data never does. - -Data enters NOOP two ways, and leaves it (other than the optional AI Coach) only when **you** +There are exactly **two** opt-in exceptions: the **AI Coach** (§1.1a) and the **Oura cloud +import** (§1.1b). The AI Coach is off until you turn it on with your own API key; when you +ask it a question it sends a short text summary of your recent metrics to the provider you +choose. The Oura cloud import is off until you connect your own Oura account; instead of +sending data out, it pulls your own Oura data **in** over OAuth, using your own Oura +developer app, and never sends any of your existing NOOP data out. Nothing else in the app +ever touches the network, and your raw biometric streams never leave the device through +either exception. + +Data enters NOOP three ways, and leaves it (other than the optional AI Coach) only when **you** deliberately export it to another store on the **same device**: | Path | Transport | Direction | |------|-----------|-----------| | Live collection | Bluetooth LE, strap → device | Read-only from the strap | | File import (Apple Health, WHOOP CSV, nutrition CSV) | User-selected files on disk | Read-only from disk | +| Oura cloud import (opt-in, §1.1b) | HTTPS OAuth + REST, `api.ouraring.com` → device | Read-only from your own Oura account | | Apple Health export, incl. iOS "Export for Shortcuts" | On-device, user-initiated | NOOP → your Apple Health, on your device only (§1.3) | -The only **network** path is the opt-in AI Coach; the biometric pipeline produces no network -traffic of any kind. The Apple Health export above is an **on-device** hand-off, not a network -upload — see §1.3. +The only **network** paths are the opt-in AI Coach and the opt-in Oura cloud import; the +biometric pipeline produces no network traffic of any kind. The Apple Health export above is +an **on-device** hand-off, not a network upload — see §1.3. -### 1.1 Network code: only the optional AI Coach +### 1.1 Network code: only the two optional exceptions The biometric pipeline and all five Swift packages (`WhoopProtocol`, `WhoopStore`, `StrandAnalytics`, `StrandImport`, `StrandDesign`) contain **no** use of `URLSession`, `URLRequest`, `NWConnection`, `dataTask`, or any -other networking API. These Swift packages are **shared by the macOS and iOS apps** -(iOS is build-from-source only — no App Store / TestFlight — and was folded into the -main tree in v1.94), so the Swift-side privacy behaviour described here applies equally -to both. Android is a separate codebase using Room for storage and Kotlin for the BLE / -import / Coach paths. The **only** networking anywhere in the app is the AI Coach -(`Strand/AI/AICoach.swift` on the Swift side — macOS and iOS — `com.noop.ai.AiCoach` on -Android), described in §1.1a. The package manifests reference dependency *download* URLs -that Swift Package Manager resolves at build time, never at runtime: +other networking API — still true after the Oura cloud import (§1.1b) landed: its OAuth +and REST calls live entirely in the app target, `Strand/Oura/`, and `StrandImport` +gained only pure, network-free parsers for Oura's payload shapes. These Swift packages +are **shared by the macOS and iOS apps** (iOS is build-from-source only — no App Store / +TestFlight — and was folded into the main tree in v1.94), so the Swift-side privacy +behaviour described here applies equally to both. Android is a separate codebase using +Room for storage and Kotlin for the BLE / import / Coach paths; its own Oura support is +the local BLE ring-pairing lane, not a network API, so it has no equivalent to §1.1b. The +**only** networking anywhere in the app is the AI Coach (`Strand/AI/AICoach.swift` on the +Swift side — macOS and iOS — `com.noop.ai.AiCoach` on Android), described in §1.1a, and +the Oura cloud import (`Strand/Oura/`, Swift-only — macOS and iOS), described in §1.1b. +The package manifests reference dependency *download* URLs that Swift Package Manager +resolves at build time, never at runtime: ``` Packages/WhoopStore/Package.swift → https://github.com/groue/GRDB.swift.git @@ -68,8 +78,9 @@ importers. Neither opens a socket. ### 1.1a The AI Coach (optional, off by default, bring your own key) -The AI Coach lets you ask questions about your data in plain language. It is the one -feature that uses the network, and only on your terms: +The AI Coach lets you ask questions about your data in plain language. It is one of the +two features that use the network (the other is the Oura cloud import, §1.1b), and only +on your terms: - **Off until you enable it.** You enter your own API key for the provider you choose (Anthropic, OpenAI, or a local / self-hosted OpenAI-compatible LLM such as Ollama or @@ -86,20 +97,58 @@ feature that uses the network, and only on your terms: provider you picked, under your own account. NOOP runs no server in between and keeps no copy. -If you never enable the AI Coach, NOOP makes zero network connections. - -### 1.2 The macOS sandbox (and what it means for the AI Coach) - -On macOS the App Sandbox is the backstop. The app ships with a deliberately minimal -entitlement set (`Strand/Resources/Strand.entitlements`): +If you never enable the AI Coach and never connect Oura (§1.1b), NOOP makes zero network +connections. + +### 1.1b The Oura cloud import (optional, off by default, bring your own OAuth app) + +The Oura cloud import pulls your own historical Oura data into NOOP over Oura's official +API — a one-time backfill you trigger yourself, not an ongoing background sync: + +- **Off until you connect it.** Nothing happens until you register your own free Oura + developer app and drop its client ID/secret into an untracked + `Strand/Oura/OuraSecrets.xcconfig` (absent/blank credentials cleanly disable the lane — + `OuraCredentials.fromBundle` — and the Data Sources card says so instead of connecting), + then tap **"Connect Oura & Import Everything"** in Data Sources. No credentials + configured, no network calls, ever. +- **What is sent.** An OAuth authorization-code handshake — you sign into Oura's own + consent page (`cloud.ouraring.com`) through Apple's system `ASWebAuthenticationSession`, + not an in-app WebView NOOP controls — followed by bearer-token `GET` requests to + `api.ouraring.com/v2/usercollection/*` carrying only your access token and the + endpoint/date-range parameters needed to page through your history. No NOOP data rides + along with these requests beyond the token itself. +- **What comes back.** Your own Oura data — sleep, readiness, activity, workouts, heart + rate, and the other endpoints your granted scopes cover — flowing **in**, once, to seed + your local database. Oura's own readiness/sleep scores are kept for reference only + (`ref_*`/`oura_*` metric keys); NOOP's own Charge/Effort/Rest are never derived from + them and are never sent anywhere. +- **What is NOT sent.** None of your existing NOOP data — no WHOOP streams, no other + imports, no computed scores — ever leaves the device via this lane. It is inbound-only. +- **Your app, your grant, revocable at Oura.** You register your own OAuth app at Oura's + developer portal; NOOP runs no server in between. Revoke access any time from your Oura + account settings, or tap **Disconnect** in NOOP, which signs out locally and deletes the + stored tokens plus every row this lane wrote — including the raw archive (`ouraRaw` + table, see `docs/DATA_MODEL.md`). +- **Tokens in the Keychain, not a plist.** The access/refresh tokens are stored via + `OuraTokenStore` as a single Keychain item (`kSecAttrAccessibleAfterFirstUnlock`), the + same pattern as the AI Coach's API key (`AIKeyStore`) — never UserDefaults, never on + disk in the clear. + +If you never connect Oura, NOOP makes zero calls to `ouraring.com`. + +### 1.2 The macOS sandbox (and what it means for the AI Coach and the Oura import) + +On macOS the App Sandbox is the backstop. The app ships with a minimal entitlement set +(`Strand/Resources/Strand.entitlements`): ```xml com.apple.security.app-sandbox com.apple.security.device.bluetooth com.apple.security.files.user-selected.read-write +com.apple.security.network.client ``` -That is the entire entitlement file. Three keys: +That is the entire entitlement file. Four keys: - **`app-sandbox`** — the process runs inside the macOS App Sandbox container. - **`device.bluetooth`** — permits BLE access to talk to the strap. The matching @@ -109,24 +158,25 @@ That is the entire entitlement file. Three keys: your device."* - **`files.user-selected.read-write`** — lets the app read import files the user explicitly picks (and write the database in its own container). +- **`network.client`** — outbound socket access. Added for the AI Coach on a + signed/sandboxed build, where the sandbox otherwise refuses any socket the app tries + to open (#128); the Oura cloud import (§1.1b) now relies on the same entitlement. The + ad-hoc distributed build applies **no** entitlements at all (unsigned build + ad-hoc + re-sign), so this key only matters for a signed/sandboxed build. The entitlement only + permits the socket the sandbox would otherwise refuse — it doesn't make either feature + call out on its own; both stay off until you deliberately turn the Coach on or tap + Connect Oura. Notably **absent**: -- `com.apple.security.network.client` — **no outbound network entitlement.** The macOS - sandbox will refuse any socket the app tries to open, **including the AI Coach's**. So - on the sandboxed macOS build the AI Coach cannot reach the network as currently - shipped — the whole macOS app, Coach included, is offline. (Android has no equivalent - sandbox restriction, so the AI Coach's call works there with your own key. The iOS app - shares the same Swift Coach code; whether its Coach can reach the network depends on - that build's own entitlements.) Turning the macOS Coach on would mean adding this - entitlement; until that's a deliberate choice, it stays out and macOS stays fully - offline. - `com.apple.security.network.server` — no inbound listener. - No `files.downloads`, `files.documents`, or any broad filesystem entitlement — the app cannot wander the disk; it sees only what the user hands it through the open panel, plus its own sandbox container. -This is the structural guarantee behind "offline by design" on macOS: the privacy +This is the structural guarantee behind "offline by design" on macOS: the sandbox +permits exactly the two deliberate, opt-in exceptions above and nothing else — no +undeclared entitlement could smuggle out a connection the user didn't ask for. The property is enforced by the OS, not merely by convention. > **Note on Hardened Runtime.** `project.yml` currently sets @@ -439,14 +489,20 @@ dedicated source id `nutrition-csv`, alongside your other metrics and entirely o ## 4. What NOOP does *not* collect or transmit -- **No accounts, no login.** Nothing to sign into; no credentials - stored. +- **No NOOP account, no NOOP login.** Nothing to sign into with NOOP itself; NOOP + issues no credentials of its own. The one exception is opt-in: the Oura cloud import + (§1.1b) has *you* sign into *your own* Oura account, at Oura's own login page, over + OAuth — NOOP never sees your Oura password, only the resulting tokens, kept in the + Keychain. - **No telemetry / analytics / crash reporting.** No third-party SDKs of that kind. -- **No cloud, no sync, no remote backup.** Your data never leaves the machine via - NOOP. +- **No cloud, no sync, no remote backup of NOOP's own data.** Your NOOP data never + leaves the machine via NOOP. The Oura cloud import (§1.1b) is the one path that + reaches a cloud service, and it is **inbound only** — it pulls your own Oura data in; + it does not sync or back up any NOOP data out. - **No advertising identifiers, no tracking.** - **No WHOOP account or API credentials.** NOOP talks only to the strap over local - BLE; it does not authenticate against, or pull from, any WHOOP server. + BLE; it does not authenticate against, or pull from, any WHOOP server. (Oura is the + one account-based exception — see above and §1.1b.) --- @@ -454,7 +510,8 @@ dedicated source id `nutrition-csv`, alongside your other metrics and entirely o | Surface | Risk | Mitigation | Where | |---------|------|------------|-------| -| Process | Data exfiltration / network egress | Only the opt-in AI Coach networks (your key, to your chosen provider, a text summary — §1.1a) — nothing else makes a network call, and nothing is sent until you ask. The Coach's call works on Android (and on iOS if that build's entitlements allow it), but is **blocked by the macOS App Sandbox as shipped** (no `network.client` entitlement — §1.2), so macOS stays fully offline | `Strand/AI/AICoach.swift`, `android/.../ai/AiCoach.kt` | +| Process | Data exfiltration / network egress | Only two opt-in features network: the AI Coach (your key, to your chosen provider, a text summary — §1.1a) and the Oura cloud import (your own OAuth app, to `api.ouraring.com`, inbound-only — §1.1b). Nothing else makes a network call, and nothing is sent until you ask. Both work on Android/iOS where shipped (Android's Oura support is local-BLE only, no cloud lane); on macOS both cross the sandbox on the same `network.client` entitlement, present since #128 (§1.2) | `Strand/AI/AICoach.swift`, `Strand/Oura/`, `android/.../ai/AiCoach.kt` | +| Oura cloud import | OAuth token / scope leakage, cross-account data mixing | Off by default; tokens Keychain-only (`kSecAttrAccessibleAfterFirstUnlock`, never UserDefaults/plist); fixed OAuth scopes set at build time; raw + normalized rows partitioned under `deviceId = "oura-api"`; Oura's own scores kept reference-only (`ref_*`/`oura_*` metricSeries keys, never NOOP's Charge/Effort/Rest); `.cloudImport` is structurally priority-2 so it never seizes a WHOOP day; Disconnect purges tokens + every `oura-api` row incl. the raw archive | `Strand/Oura/OuraTokenStore.swift`, `Strand/Oura/OuraConnectModel.swift`, `Packages/WhoopStore/Sources/WhoopStore/OuraRawStore.swift` | | Filesystem | Broad disk access | Only `files.user-selected.read-write`; data stays in the sandbox container | `Strand.entitlements`, `Strand/Collect/StorePaths.swift` | | BLE frames | Malformed / adversarial packets | CRC8 + CRC32 (+ CRC16 for v5) gating; reject on failure | `WhoopProtocol/Framing.swift`, `Strand/BLE/FrameRouter.swift` | | BLE frames | Out-of-bounds reads from short/lying length | `nil`-returning bounds-checked readers; slice clamping; min-length guards | `WhoopProtocol/Interpreter.swift` | From 46708648f25389f437b5c1de9fb8fb988803a714 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 11:04:38 -0700 Subject: [PATCH 030/171] fix(oura): guard make-active for import sources; archive registry row on disconnect; ouraRaw doc accuracy (final review) --- .../Sources/WhoopStore/PairedDevice.swift | 6 +++++ Strand/Oura/OuraConnectModel.swift | 8 ++++++ Strand/Screens/DevicesView.swift | 27 ++++++++++++++----- docs/DATA_MODEL.md | 18 +++++++------ 4 files changed, 44 insertions(+), 15 deletions(-) diff --git a/Packages/WhoopStore/Sources/WhoopStore/PairedDevice.swift b/Packages/WhoopStore/Sources/WhoopStore/PairedDevice.swift index 08080e6c9d..00a215e6a7 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/PairedDevice.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/PairedDevice.swift @@ -34,6 +34,12 @@ public struct PairedDevice: Equatable, Sendable, Identifiable { if model.localizedCaseInsensitiveContains(brand) { return model } return "\(brand) \(model)" } + + /// True for a data-partition source (a cloud API pull or a file/CSV import) rather than a live, + /// connectable device. UI that offers "make active" must exclude these: activating one would demote + /// whatever live device (WHOOP, a BLE strap, Apple Watch) currently drives BLE routing + day-owner + /// priority 0 — an import source has no live connection to hand that role to. + public var isImportSource: Bool { sourceKind == .cloudImport || sourceKind == .fileImport } } public enum DeviceStatus: String, Sendable, CaseIterable { case active, paired, archived } diff --git a/Strand/Oura/OuraConnectModel.swift b/Strand/Oura/OuraConnectModel.swift index b41706b310..a57260cf2a 100644 --- a/Strand/Oura/OuraConnectModel.swift +++ b/Strand/Oura/OuraConnectModel.swift @@ -1,6 +1,7 @@ import Foundation import Combine import AuthenticationServices +import WhoopStore /// Drives the Data Sources "Connect Oura" card: connect (OAuth) → one-time backfill → honest summary → /// disconnect. @MainActor so all @Published mutations are main-thread; the network/backfill hops off-main. @@ -54,6 +55,13 @@ final class OuraConnectModel: ObservableObject { do { try await store.deleteAllData(deviceId: "oura-api") try await store.deleteOuraRaw(deviceId: "oura-api") + // M-1: `deleteAllData` intentionally leaves the `pairedDevice` registry row intact + // (archiving/removing it is a separate op — DeviceRegistryStore.swift) so it doesn't + // just purge data, else a data-less "Oura (cloud)" card lingers in DevicesView. Archive + // it via the same off-actor raw-write pattern OuraSyncWriter.persist uses to register + // the row on connect (`registryWriter` is `nonisolated`; `DeviceRegistryStore` is a + // thin synchronous, Sendable wrapper — no separate actor hop needed). + try DeviceRegistryStore(dbQueue: store.registryWriter).archive("oura-api") statusText = "Disconnected." } catch { statusText = "Disconnected, but couldn't fully clear local Oura data: \(error.localizedDescription)" diff --git a/Strand/Screens/DevicesView.swift b/Strand/Screens/DevicesView.swift index 773bcfb084..34a79ede68 100644 --- a/Strand/Screens/DevicesView.swift +++ b/Strand/Screens/DevicesView.swift @@ -61,6 +61,10 @@ private struct DevicesContent: View { private var activeDevices: [PairedDevice] { registry.devices.filter { $0.status != .archived } } private var removedDevices: [PairedDevice] { registry.devices.filter { $0.status == .archived } } + /// I-1: `activeDevices` minus import sources (cloud/file) — the candidates actually eligible for + /// "make active". An import source is a data partition, not a live device; offering it here would let + /// activating it demote the live WHOOP driving BLE routing + day-owner priority 0. + private var activatableDevices: [PairedDevice] { activeDevices.filter { !$0.isImportSource } } /// #987: the active+connected strap's clock state, from the SAME pure ConnectionReadout parsers the /// Test Centre Connection panel binds (one source of truth). nil (no row at all) until the WHOOP path @@ -181,7 +185,9 @@ private struct DevicesContent: View { .confirmationDialog("Pick a new active strap", isPresented: $pickNewActive, titleVisibility: .visible) { - ForEach(activeDevices) { device in + // I-1: import sources (Oura cloud import, file imports) are excluded — they're data + // partitions, not live devices, and must never be offered as an active-strap candidate. + ForEach(activatableDevices) { device in Button(device.displayName) { registry.setActive(device.id) } } Button("Leave none active", role: .cancel) { } @@ -258,8 +264,9 @@ private struct DevicesContent: View { registry.archive(device.id) removeTarget = nil if wasActive { - // Other paired devices left → ask which becomes active; otherwise no active device remains. - if !activeDevices.isEmpty { + // Other ACTIVATABLE devices left → ask which becomes active; otherwise (none left, or only + // import sources remain) no active device remains and there's nothing to offer (I-1). + if !activatableDevices.isEmpty { pickNewActive = true } } @@ -423,16 +430,20 @@ private struct DeviceCard: View { /// The card's primary tap action, or nil when there isn't one. A paired-but-not-active band → make it /// active; a removed band → re-add it as active. The active band and any card without those callbacks - /// have no whole-card tap (their controls live entirely in the ⋮ menu). + /// have no whole-card tap (their controls live entirely in the ⋮ menu). I-1: an import source (Oura + /// cloud import, file imports) never offers activation — it's a data partition, not a live device; + /// making it "active" would demote whatever live device drives BLE routing + day-owner priority 0. private var primaryAction: (() -> Void)? { + if device.isImportSource { return nil } if device.status == .archived { return onReAdd } if !isActive { return onMakeActive } return nil } /// Short accent hint mirroring the primary tap, shown in the footer row. nil when the card has no - /// whole-card action (active band / menu-only removed band). + /// whole-card action (active band / menu-only removed band / I-1 import source). private var primaryActionHint: String? { + if device.isImportSource { return nil } if device.status == .archived { return onReAdd == nil ? nil : String(localized: "Make active") } if !isActive { return String(localized: "Make active") } return nil @@ -477,7 +488,9 @@ private struct DeviceCard: View { private var actionsMenu: some View { Menu { if device.status == .archived { - if let onReAdd { + // I-1: a removed import source (e.g. Oura cloud import, archived on Disconnect) never + // offers "Make active" reactivation — it's a data partition, not a live device. + if let onReAdd, !device.isImportSource { Button { onReAdd() } label: { Label("Make active", systemImage: "bolt.fill") } } Button { onRename() } label: { Label("Rename", systemImage: "pencil") } @@ -488,7 +501,7 @@ private struct DeviceCard: View { } } } else { - if !isActive { + if !isActive && !device.isImportSource { Button { onMakeActive() } label: { Label("Make active", systemImage: "bolt.fill") } } Button { onRename() } label: { Label("Rename", systemImage: "pencil") } diff --git a/docs/DATA_MODEL.md b/docs/DATA_MODEL.md index 93f93a2807..35f85ec5f1 100644 --- a/docs/DATA_MODEL.md +++ b/docs/DATA_MODEL.md @@ -458,9 +458,10 @@ values, so any field Oura returns can be re-derived later without re-fetching. ### `ouraRaw` *(v24)* -One row per archived Oura API document (`OuraRawStore.swift`, `struct OuraRawRow`). -Natural key `(deviceId, endpoint, documentId)`. Written by the Oura sync coordinator in the -app target (`Strand/Oura/`); migration `v24-oura-raw` +One row per fetched PAGE of an Oura API endpoint response — not one row per Oura document; a +single page's `data` array can carry many documents (`OuraRawStore.swift`, `struct OuraRawRow`). +Written by `OuraSyncCoordinator.fetchRaw(_:dateParam:)` in the app target (`Strand/Oura/`). +Natural key `(deviceId, endpoint, documentId)`. Migration `v24-oura-raw` (`Packages/WhoopStore/Sources/WhoopStore/Database.swift`) — additive only, a new table, no existing row touched. @@ -468,14 +469,15 @@ existing row touched. | --- | --- | --- | | `deviceId` | TEXT NOT NULL | Part of PK. `"oura-api"` for the live cloud-import lane. | | `endpoint` | TEXT NOT NULL | Part of PK. Oura endpoint name, e.g. `"sleep"`, `"daily_readiness"`, `"heartrate"`. | -| `documentId` | TEXT NOT NULL | Part of PK. Oura's own `id` for the document; heartrate pages synthesize a window key. | -| `day` | TEXT | `YYYY-MM-DD`, nullable — set when the document is day-keyed. | -| `payloadJSON` | TEXT NOT NULL | Verbatim Oura response object. | +| `documentId` | TEXT NOT NULL | Part of PK. A SYNTHESIZED page key, `"--"` (`startDate` is the backfill window's start date, `pageIndex` the fetched page's 0-based position) — never Oura's own document `id`, for any endpoint. | +| `day` | TEXT | `YYYY-MM-DD`, nullable. Currently always NULL — the coordinator never sets it. Reserved for a future per-document (rather than per-page) keying scheme. | +| `payloadJSON` | TEXT NOT NULL | Verbatim JSON body of the fetched page (the raw HTTP response, including its `data` array of documents) — losslessness holds at the page level, not the individual-document level. | | `fetchedAt` | INTEGER NOT NULL | Unix seconds. | **Primary key:** `(deviceId, endpoint, documentId)`. `upsertOuraRaw(...)` is idempotent on this -key via `ON CONFLICT(...) DO UPDATE` — a re-pulled document overwrites its -`day`/`payloadJSON`/`fetchedAt` rather than duplicating. +key via `ON CONFLICT(...) DO UPDATE` — re-pulling the SAME window (same `startDate`, so the same +`pageIndex` synthesizes the same `documentId`) overwrites that page's `day`/`payloadJSON`/ +`fetchedAt` in place rather than duplicating. **Index** — `idx_ouraRaw_device_endpoint_day` on `(deviceId, endpoint, day)`, so per-endpoint reads (`ouraRaw(deviceId:endpoint:)`) scan `(deviceId, endpoint)` and walk `day` in order From c47ea6b7acfd23557e461f324db5a56a3d85f270 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 14:16:22 -0700 Subject: [PATCH 031/171] =?UTF-8?q?docs:=20design=20spec=20for=20noop-clou?= =?UTF-8?q?d=20=E2=80=94=20personal=20biometrics=20MCP=20server=20on=20Fly?= =?UTF-8?q?.io?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phased: (1) .noopbak mirror + read/corroborate tools + morning report (zero app-side code via iOS Shortcut upload), (1.5) OAuth2.1 shim for claude.ai/ ChatGPT + optional in-app sync button, (2) server-side staged writes (propose/confirm; Routines skip client approvals so gating is server-side), (3) sync-back under a noop-cloud source namespace honoring userEdited/ recompute rules. Grounded in verified 2026-07-10 research (MCP 2025-11-25 spec + 7/28 RC, connector auth matrix, Fly single-machine SQLite, no LiteFS) and the repo backup/provenance study. Personal fork + sidecar only — never upstream. --- .../specs/2026-07-10-noop-cloud-mcp-design.md | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-10-noop-cloud-mcp-design.md diff --git a/docs/superpowers/specs/2026-07-10-noop-cloud-mcp-design.md b/docs/superpowers/specs/2026-07-10-noop-cloud-mcp-design.md new file mode 100644 index 0000000000..2d58061df1 --- /dev/null +++ b/docs/superpowers/specs/2026-07-10-noop-cloud-mcp-design.md @@ -0,0 +1,113 @@ +# noop-cloud — personal biometrics MCP server (Fly.io) — design + +- **Date:** 2026-07-10 +- **Status:** Draft for review +- **Scope:** A new, standalone, self-hostable server project (sibling repo `noop-cloud/`, its own git) + minimal opt-in app-side hooks in this fork. Phased: read mirror → gated writes → sync-back. +- **Relation to upstream:** `CLAUDE.md` marks "no server / no cloud sync" a hard upstream constraint — this is deliberately a **personal-fork + sidecar** project, never a PR to `ryanbr/noop`. The server ships as an agnostic container anyone can self-host; nothing phones home by default. + +--- + +## 1. Summary + +The user's NOOP data (WHOOP live + Oura cloud import + Apple Health) lives in the iPhone app's SQLite. He wants an **agentic layer over it**: ask Claude/ChatGPT deep questions, get a plain-English **morning report**, have AI **find and fix mislogged data** (dedupe workouts, correct sleep bounds), **corroborate sources** (WHOOP vs Oura vs Apple baselines), and let his **other agents** read/write workout context — all without a Mac dependency. + +`noop-cloud` is a small remote **MCP server on Fly.io** holding a **mirror** of the phone's database. The phone ships its existing `.noopbak` backup to the server (Phase 1: an iOS Shortcut posts it — zero app code; Phase 1.5: an in-app "Sync to my server" button). Claude (web/iOS/Code/API), ChatGPT, and scheduled agents connect over Streamable HTTP. Writes are **staged server-side** (`propose_edit` → `confirm_edit`) because scheduled agents skip client approval UIs. Sync-back to the phone is Phase 3, under a dedicated source namespace so the app's recompute can never clobber (or be clobbered by) server edits. + +## 2. Goals / non-goals + +**Goals** +- **G1 — Mirror + query.** The full phone DB queryable via MCP tools from Claude + ChatGPT + agents, no Mac. +- **G2 — Morning report.** A scheduled agent produces a plain-English daily briefing from the mirror. +- **G3 — Gated cleanup writes.** AI proposes fixes (workouts, sleep bounds, bad points); nothing mutates without explicit confirmation; every mutation journaled + reversible. +- **G4 — Cross-source corroboration.** First-class tools to compare WHOOP/Oura/Apple per-day and surface discrepancies/baselines. +- **G5 — Self-hostable.** One container + one volume; secrets via env; no third-party data processors. Shareable with the community as "run your own." +- **G6 — Sync-back seam (Phase 3).** Server-side corrections eventually visible in the phone app without corruption. + +**Non-goals** +- No multi-user/hosted service. No upstream PR. No replacement of NOOP's scoring. No live BLE anything (phone stays the collector). No LiteFS/replication (single writer by design). + +## 3. Verified constraints the design is built on (2026-07-10) + +**MCP / clients** +- Current spec **2025-11-25**: remote transport = **Streamable HTTP** (SSE transport deprecated). A **stateless 2026-07-28 revision** is locked (removes `Mcp-Session-Id`/initialize); official guidance: existing servers keep working. → Build on **TypeScript SDK 1.29.x pinned** (Tier-1, ships `StreamableHTTPServerTransport` + OAuth plumbing); migrate post-7/28 deliberately. +- **Auth matrix:** ChatGPT developer-mode connectors = **OAuth 2.1 + PKCE (CIMD/DCR) only — static keys rejected**. claude.ai web custom connectors (individual) = OAuth or no-auth in practice. **Claude Code** = static bearer via `--header` works today; also auto-OAuth. **Messages API MCP connector** (beta `mcp-client-2025-11-20`) = any `https://` URL + static `authorization_token`, per-tool allowlists, tools-only. +- **Claude Code Routines** (cloud, cron ≥1h) include the user's connectors and **execute write tools without permission prompts** → server-side write gating is mandatory, client approval UI is not a defense. +- ChatGPT Deep Research requires exactly-shaped **`search`/`fetch`** tools; general connectors don't. + +**Fly.io** +- One `shared-cpu-1x` machine + one volume (`/data`), SQLite **WAL**, `auto_stop_machines="stop"` + `auto_start_machines=true` (wake on request). Cost ≈ $0.20–$2.20/mo. Native scheduled machines lack time-of-day precision → morning cron runs **outside** (GitHub Actions cron or a Claude Routine) and doubles as the wake-up. **No LiteFS** (unmaintained ~15 mo; Fly warns against autostop combos). + +**App-side facts (from code study)** +- **`.noopbak`** = ZIP{`noop-backup.sqlite` (+optional `settings.json`)}; export **checkpoints WAL** + `PRAGMA quick_check` first; restore is a **whole-DB destructive swap** with origin/integrity gates and rollback; fully available on iOS (share sheet / Files). +- **BackupSync** = one-way timestamped snapshots into a user folder (iCloud Drive capable) + on-launch daily catch-up. No merge logic exists anywhere. +- **No change tracking**: no `updatedAt` columns; the v5 `synced` flags are documented dead scaffolding from a removed upload feature (which failed once on ts-highwater ordering — lesson recorded). Delta sync = new columns; whole-DB ship is the honest Phase-1 transport. +- **Edit provenance is thin**: only `sleepSession.userEdited` (+`startTsAdjusted`) survives recompute, and only for bounds/stages — `efficiency/restingHr/avgHrv` are overwritten regardless. `dailyMetric`/`workout`/`metricSeries` have **zero** protection and `analyzeRecent` blindly re-upserts the trailing ~21 days every ~15 foreground minutes. **Sleep-delete tombstones live in UserDefaults, not the DB** — backups don't carry deletion state. +- Multi-source reconciliation already exists (per-source `deviceId` namespaces + `Repository.resolvedSeries`/`dayOwnership`) — the correct integration point for server-origin data (mirror the `"oura-api"` pattern; never mutate another source's rows in place). + +## 4. Architecture + +``` +iPhone (collector) Fly.io app "noop-cloud" (1 machine, 1 volume) +┌───────────────────────┐ HTTPS ┌──────────────────────────────────────────────┐ +│ NOOP app │ POST │ /ingest ← bearer(rw). .noopbak → verify → │ +│ ├ BackupSync writes │ .noopbak │ unzip → quick_check → atomic swap │ +│ │ .noopbak snapshot ├─────────────▶│ /data/mirror.sqlite (RO for tools) │ +│ └ (P1: iOS Shortcut │ │ /mcp ← Streamable HTTP MCP │ +│ posts it; P1.5: │ │ read tools · corroborate · search/fetch │ +│ in-app button) │ │ write tools → staging (propose/confirm) │ +└───────────▲───────────┘ │ /data/server.sqlite: edits journal, staging, │ + │ Phase 3: pull server │ tokens, ingest log │ + │ edits (userEdited rules) │ auth: OAuth2.1 shim (PKCE+CIMD/DCR) + 2 static │ + └──────────────────────────│ bearer tokens (ro / rw) │ + └──────▲───────────────▲───────────────────────┘ + Claude web/iOS/Code/API │ │ ChatGPT (OAuth) + Routines/cron (ro token)┘ │ other agents (MCP) +``` + +Two SQLite files on the volume: **`mirror.sqlite`** (verbatim phone DB, replaced atomically per ingest, opened read-only by tools) and **`server.sqlite`** (server-owned: staged edits, confirmed-edit journal, ingest history, token hashes). Server edits are **never** written into the mirror — the journal is the source of truth for Phase-3 sync-back, and survives every mirror swap. + +## 5. Phases + +**Phase 1 — Mirror + read tools + morning report (MVP; no app code changes)** +- Fly app (TS SDK 1.29.x, Express, better-sqlite3): `/ingest` + `/mcp` + `/healthz`. +- Upload path: the app's existing **BackupSync** snapshot + an **iOS Shortcut** ("Send NOOP to cloud": share/automation → POST file with `Authorization: Bearer `), so Phase 1 requires zero app-side code. Manual cadence is fine (data freshness tool reports staleness honestly). +- Read tools (schema from the repo study; multi-source aware): `health_snapshot`, `metric_series` (any `deviceId` source incl. `oura-api`), `sleep_summary`, `workout_summary`, `data_freshness` (incl. mirror age), **`compare_sources`** (per-day WHOOP/Oura/Apple side-by-side with deltas — G4's workhorse), plus ChatGPT-shaped `search`/`fetch`. Prompts: `morning_report`, `corroborate_sources`, `find_messy_data`. +- Auth v1: two static bearer tokens (ro/rw) → Claude Code + Messages API + cron work day 1. +- Morning report: **GitHub Actions cron** (free, exact-time) → Messages API with the MCP connector (ro token, tool-allowlisted) → delivers via ntfy/email push; a Claude Routine is the zero-infra alternative (ro token only — routines skip write approvals). +- **Exit criteria:** user asks Claude (phone or desktop) "how's my week?" against live-ish data with no Mac involved; morning report arrives on schedule. + +**Phase 1.5 — Chat-UI auth + in-app upload** +- OAuth 2.1 shim (SDK `mcpAuthRouter` + proxy provider; GitHub/Google as the single-user login; PKCE + CIMD, DCR fallback) so **claude.ai web/iOS and ChatGPT** connect properly. Static tokens remain for headless. +- Optional in-app **"Sync to my server"** (Settings; URL+token fields; POSTs the checkpointed `.noopbak` via the `OuraAPIClient` URLSession pattern; manual button first, on-launch catch-up like BackupSync later). Off by default, personal-fork only. + +**Phase 2 — Gated writes + corroboration cleanup** +- `server.sqlite` staging: `propose_edit(kind, payload, rationale)` → human-readable diff → `confirm_edit(id)` / `reject_edit(id)`; `list_pending`, `edit_journal`, `undo_edit`. Kinds: `fix_workout`, `delete_workout`, `add_workout`, `adjust_sleep_bounds`, `delete_metric_point`, `set_baseline_note`. +- Tool annotations (`readOnlyHint`/`destructiveHint`) so both chat UIs render approval framing; **enforcement stays server-side**: the ro token cannot call write tools at all; `confirm_edit` requires the rw credential and is never auto-invoked by prompts we ship. +- Confirmed edits apply to a **server overlay** consulted by read tools (mirror stays pristine) — the overlay + journal are exactly what Phase 3 ships to the phone. + +**Phase 3 — Sync-back to the phone (design constraints locked now, detail then)** +- Transport: app pulls `GET /edits?since=` (journal is append-only + idempotent by edit id). +- Apply rules (from the code study): sleep-bound edits → `applySleepEdit`-equivalent with `userEdited=1` (only path that survives recompute); workout/daily/metric corrections → written under a **new source namespace `"noop-cloud"`** + registered in the source resolver (mirror the `oura-api` integration), never in-place mutations of another source's rows; deletions → require a DB-backed tombstone mechanism (the UserDefaults one doesn't sync) — designed in the Phase-3 plan; every applied edit ack'd back so the journal tracks phone-applied state. + +## 6. Auth & security + +- HTTPS only (Fly-terminated). Tokens: ≥32-byte random, stored hashed in `server.sqlite`, `ro` + `rw` scopes; rotation = insert new/revoke old. OAuth shim auto-approves exactly one upstream identity (the user's GitHub/Google), everything else 403. +- PHI posture: single-tenant container, no analytics/telemetry, no third-party processors; Fly volume snapshots (default retention) are the backup; `/ingest` size-capped + zip-validated (`quick_check`, magic bytes, origin gate — mirroring `DataBackup.restore`'s own checks); rate-limited. +- Threat notes: a leaked **ro** token exposes read access (rotate; consider IP allowlist later); a leaked **rw** token is the crown jewel — it's entered exactly twice (Shortcut + Claude Code config). Routines/cron get **ro only**, always. + +## 7. Testing + +- Server: vitest — ingest pipeline (fixture `.noopbak` incl. corrupt/oversized/foreign-origin rejects), every tool against a fixture mirror (multi-source days), staging state machine (propose→confirm/reject/undo), auth middleware (scope enforcement, token hashing), `search`/`fetch` shape conformance. +- Contract smoke: MCP Inspector against local server; Claude Code `claude mcp add --transport http` end-to-end. +- Phase 3 app-side: XCTest against `WhoopStore.inMemory()` proving recompute-survival of applied edits (the `userEdited` + namespace rules). + +## 8. Open questions (answers assumed → flag if wrong) + +- **Q1 Report delivery:** assumed **ntfy.sh push** (free, no account, self-hostable) with email as fallback. OK? +- **Q2 Upload cadence P1:** manual Shortcut tap (or iOS automation e.g. daily at 6am while charging). OK to start manual? +- **Q3 Server repo home:** new sibling repo `/Users/vk/VKDEV/NOOP/noop-cloud` (own git, MIT/PolyForm — pick at creation). OK? +- **Q4 OAuth upstream identity:** GitHub login assumed (user has GH). OK? + +## 9. Out of scope + +Multi-user hosting, HIPAA formalities, WHOOP/Oura credential proxying (Oura import keeps running on-phone), replacing NOOP scoring, Android app changes, upstreaming any of this. From e9501221be70c9819b96e77a2e15775f033e2674 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 19:17:33 -0700 Subject: [PATCH 032/171] iOS: write sleep stages, 1-min heart rate, and workouts back to Apple Health writeBack previously wrote only four noon-stamped daily vitals and never used the sleepAnalysis share permission it requested. Now each feature saves independently, guarded on its own type's share status: - Sleep sessions as one inBed sample plus a category sample per stage segment (deep/REM/core/awake), keyed noop::sleep:; sessions whose stagesJSON carries no timing degrade to one asleepUnspecified block. Stage parsing/clamping is a pure function in StrandImport (HealthWriteback.stageIntervals) with swift-test coverage. - Continuous heart rate as 1-minute hrBuckets means, forward-only UserDefaults cursor plus a 48h rewrite window for late strap offloads. - Strap/manual workouts via HKWorkoutBuilder with energy and distance samples; apple-health-sourced rows excluded so imports never echo back. - Nightly vitals re-stamped at the day's wake time (unchanged keys, so they replace their noon ancestors). Auth resume now checks the pre-update legacy write set and fires one re-request so existing grants surface a single sheet listing only the new types instead of getting stuck at .unknown. Verified: StrandImport swift test (191 passing), NOOPiOS and Strand schemes compile (no default CI covers app targets). --- .../StrandImport/HealthWriteback.swift | 74 ++++ .../HealthWritebackTests.swift | 90 +++++ Strand/Screens/AppleHealthView.swift | 2 +- StrandiOS/Health/HealthKitBridge.swift | 320 +++++++++++++++++- ...0-apple-health-highres-writeback-design.md | 61 ++++ 5 files changed, 537 insertions(+), 10 deletions(-) create mode 100644 Packages/StrandImport/Sources/StrandImport/HealthWriteback.swift create mode 100644 Packages/StrandImport/Tests/StrandImportTests/HealthWritebackTests.swift create mode 100644 docs/superpowers/specs/2026-07-10-apple-health-highres-writeback-design.md diff --git a/Packages/StrandImport/Sources/StrandImport/HealthWriteback.swift b/Packages/StrandImport/Sources/StrandImport/HealthWriteback.swift new file mode 100644 index 0000000000..fd230fa0a7 --- /dev/null +++ b/Packages/StrandImport/Sources/StrandImport/HealthWriteback.swift @@ -0,0 +1,74 @@ +import Foundation + +/// Pure mapping from NOOP's persisted sleep-stage JSON to the interval list the iOS HealthKit +/// write-back turns into `sleepAnalysis` category samples. Lives here (not in the app target) so +/// the parsing/clamping logic is covered by `swift test` — HealthKit itself can't be unit-tested. +public enum HealthWriteback { + + /// A HealthKit-agnostic sleep stage. The bridge maps these onto `HKCategoryValueSleepAnalysis` + /// (`awake → .awake`, `light → .asleepCore`, `deep → .asleepDeep`, `rem → .asleepREM`). + public enum StageKind: String, Equatable { + case awake, light, deep, rem + } + + public struct StageInterval: Equatable { + public let start: Int // unix seconds + public let end: Int // unix seconds, > start + public let kind: StageKind + public init(start: Int, end: Int, kind: StageKind) { + self.start = start; self.end = end; self.kind = kind + } + } + + /// Decode a session's `stagesJSON` into clamped stage intervals for HealthKit. + /// + /// Only the on-device SleepStager segment shape (`[{"start","end","stage"}]`, unix seconds) + /// carries timing, so only it yields intervals. The two aggregate shapes NOOP has also + /// persisted (`{"deep":min,…}` and `[{"stage","min"}]` — see `WhoopCsvExporter.stageMinutes`) + /// have no placement information; fabricating positions for them would write fiction into + /// Health, so they return `[]` and the caller falls back to one `.asleepUnspecified` block. + /// + /// Normalization: the stager labels awake as `"wake"`, importers as `"awake"` — both map to + /// `.awake`. Unknown labels are dropped. Segments are clamped to `[sessionStart, sessionEnd]` + /// and zero/negative-length segments (before or after clamping) are dropped. + public static func stageIntervals(stagesJSON: String?, + sessionStart: Int, + sessionEnd: Int) -> [StageInterval] { + guard sessionEnd > sessionStart, + let stagesJSON, let data = stagesJSON.data(using: .utf8), + let segments = (try? JSONSerialization.jsonObject(with: data)) as? [[String: Any]] + else { return [] } + var out: [StageInterval] = [] + for seg in segments { + // Aggregate shape ([{"stage","min"}]) has no start/end — bail to the no-timing path + // for the WHOLE session rather than emit a partial mix. + guard let rawStart = intValue(seg["start"]), let rawEnd = intValue(seg["end"]) else { + return [] + } + guard let stage = (seg["stage"] as? String)?.lowercased() else { continue } + let kind: StageKind? + switch stage { + case "wake", "awake": kind = .awake + case "light": kind = .light + case "deep": kind = .deep + case "rem": kind = .rem + default: kind = nil + } + guard let kind else { continue } + let start = max(rawStart, sessionStart) + let end = min(rawEnd, sessionEnd) + guard end > start else { continue } + out.append(StageInterval(start: start, end: end, kind: kind)) + } + return out.sorted { $0.start < $1.start } + } + + /// JSON numbers arrive as Int, Double, or NSNumber depending on the writer; accept all. + private static func intValue(_ any: Any?) -> Int? { + switch any { + case let i as Int: return i + case let d as Double: return Int(d) + default: return nil + } + } +} diff --git a/Packages/StrandImport/Tests/StrandImportTests/HealthWritebackTests.swift b/Packages/StrandImport/Tests/StrandImportTests/HealthWritebackTests.swift new file mode 100644 index 0000000000..cbb5cce899 --- /dev/null +++ b/Packages/StrandImport/Tests/StrandImportTests/HealthWritebackTests.swift @@ -0,0 +1,90 @@ +import XCTest +@testable import StrandImport + +final class HealthWritebackTests: XCTestCase { + + private let start = 1_700_000_000 + private var end: Int { start + 8 * 3600 } + + private func intervals(_ json: String?) -> [HealthWriteback.StageInterval] { + HealthWriteback.stageIntervals(stagesJSON: json, sessionStart: start, sessionEnd: end) + } + + func testSegmentShapeParsesAndSorts() { + let json = """ + [{"start": \(start + 3600), "end": \(start + 5400), "stage": "deep"}, + {"start": \(start), "end": \(start + 3600), "stage": "light"}, + {"start": \(start + 5400), "end": \(start + 7200), "stage": "rem"}] + """ + let out = intervals(json) + XCTAssertEqual(out, [ + .init(start: start, end: start + 3600, kind: .light), + .init(start: start + 3600, end: start + 5400, kind: .deep), + .init(start: start + 5400, end: start + 7200, kind: .rem), + ]) + } + + func testWakeAndAwakeBothMapToAwake() { + let json = """ + [{"start": \(start), "end": \(start + 60), "stage": "wake"}, + {"start": \(start + 60), "end": \(start + 120), "stage": "awake"}] + """ + XCTAssertEqual(intervals(json).map(\.kind), [.awake, .awake]) + } + + func testSegmentsClampedToSessionBounds() { + let json = """ + [{"start": \(start - 600), "end": \(start + 600), "stage": "light"}, + {"start": \(end - 600), "end": \(end + 600), "stage": "rem"}] + """ + let out = intervals(json) + XCTAssertEqual(out.first, .init(start: start, end: start + 600, kind: .light)) + XCTAssertEqual(out.last, .init(start: end - 600, end: end, kind: .rem)) + } + + func testZeroAndNegativeLengthSegmentsDropped() { + let json = """ + [{"start": \(start), "end": \(start), "stage": "deep"}, + {"start": \(start + 100), "end": \(start + 50), "stage": "rem"}, + {"start": \(end + 100), "end": \(end + 200), "stage": "light"}] + """ + XCTAssertEqual(intervals(json), []) + } + + func testUnknownStageLabelDropped() { + let json = """ + [{"start": \(start), "end": \(start + 60), "stage": "hyperdrive"}, + {"start": \(start + 60), "end": \(start + 120), "stage": "deep"}] + """ + XCTAssertEqual(intervals(json).map(\.kind), [.deep]) + } + + func testAggregateMinuteShapesReturnEmpty() { + // Dict shape (Android/demo seeds) and per-stage-minutes array carry no timing. + XCTAssertEqual(intervals(#"{"deep": 90, "rem": 60, "light": 210}"#), []) + XCTAssertEqual(intervals(#"[{"stage": "deep", "min": 90}, {"stage": "rem", "min": 60}]"#), []) + } + + func testMixedSegmentAndAggregateBailsEntirely() { + // One segment missing start/end means the shape is untrustworthy — no partial mix. + let json = """ + [{"start": \(start), "end": \(start + 3600), "stage": "light"}, + {"stage": "deep", "min": 90}] + """ + XCTAssertEqual(intervals(json), []) + } + + func testNilGarbageAndInvertedSessionReturnEmpty() { + XCTAssertEqual(intervals(nil), []) + XCTAssertEqual(intervals("not json"), []) + XCTAssertEqual(HealthWriteback.stageIntervals(stagesJSON: "[]", + sessionStart: end, sessionEnd: start), []) + } + + func testDoubleTimestampsAccepted() { + let json = """ + [{"start": \(start).0, "end": \(start + 60).5, "stage": "deep"}] + """ + XCTAssertEqual(intervals(json), [.init(start: start, end: start + 60, kind: .deep)]) + } +} diff --git a/Strand/Screens/AppleHealthView.swift b/Strand/Screens/AppleHealthView.swift index 64f524990f..fe1ea2f500 100644 --- a/Strand/Screens/AppleHealthView.swift +++ b/Strand/Screens/AppleHealthView.swift @@ -411,7 +411,7 @@ struct AppleHealthView: View { .fixedSize(horizontal: false, vertical: true) case .unknown, .denied: - Text("Read your heart rate, HRV, blood oxygen, respiratory rate, sleep, steps and energy straight from Apple Health, and write NOOP's strap-derived metrics back. Everything stays on \(Platform.deviceNounPhrase).") + Text("Read your heart rate, HRV, blood oxygen, respiratory rate, sleep, steps and energy straight from Apple Health, and write NOOP's strap data back: sleep with full stages, continuous heart rate, workouts, and nightly vitals. Everything stays on \(Platform.deviceNounPhrase).") .font(StrandFont.caption) .foregroundStyle(StrandPalette.textTertiary) .fixedSize(horizontal: false, vertical: true) diff --git a/StrandiOS/Health/HealthKitBridge.swift b/StrandiOS/Health/HealthKitBridge.swift index 02ac07f621..a3fd7996e9 100644 --- a/StrandiOS/Health/HealthKitBridge.swift +++ b/StrandiOS/Health/HealthKitBridge.swift @@ -69,6 +69,20 @@ final class HealthKitBridge: ObservableObject { } private var writeTypes: Set { + var s = Set() + for id in HealthKitBridge.quantityWriteIds + HealthKitBridge.highResQuantityWriteIds { + if let t = HKObjectType.quantityType(forIdentifier: id) { s.insert(t) } + } + if let sleep = HKObjectType.categoryType(forIdentifier: .sleepAnalysis) { s.insert(sleep) } + s.insert(HKObjectType.workoutType()) + return s + } + + /// The write set as it existed before the high-res write-back (4 nightly vitals + sleep). A + /// returning user granted THIS set; `refreshAuthIfPreviouslyGranted` must resume off it — checking + /// the full `writeTypes` would leave every pre-existing grant stuck at `.unknown` after the update + /// because the new types are still `.notDetermined`. + private var legacyCoreWriteTypes: Set { var s = Set() for id in HealthKitBridge.quantityWriteIds { if let t = HKObjectType.quantityType(forIdentifier: id) { s.insert(t) } } if let sleep = HKObjectType.categoryType(forIdentifier: .sleepAnalysis) { s.insert(sleep) } @@ -89,6 +103,12 @@ final class HealthKitBridge: ObservableObject { private static let quantityWriteIds: [HKQuantityTypeIdentifier] = [ .restingHeartRate, .heartRateVariabilitySDNN, .oxygenSaturation, .respiratoryRate ] + // High-res write-back shares: the continuous 1-minute HR stream, and the energy/distance samples + // attached to written workouts. Kept out of `quantityWriteIds` so `legacyCoreWriteTypes` (the + // auth-resume set) stays exactly what pre-update users granted. + private static let highResQuantityWriteIds: [HKQuantityTypeIdentifier] = [ + .heartRate, .activeEnergyBurned, .distanceWalkingRunning, .distanceCycling + ] // MARK: - Authorization @@ -135,13 +155,23 @@ final class HealthKitBridge: ObservableObject { /// status, so no system permission sheet is shown. func refreshAuthIfPreviouslyGranted() { guard auth == .unknown, HKHealthStore.isHealthDataAvailable() else { return } - let granted = writeTypes.allSatisfy { store.authorizationStatus(for: $0) == .sharingAuthorized } + let granted = legacyCoreWriteTypes.allSatisfy { store.authorizationStatus(for: $0) == .sharingAuthorized } if granted { auth = .authorized // A returning user who already granted access should get the live stream re-armed for this // process. enableLiveDelivery is idempotent (HealthKit dedups observers + background // delivery per type), so calling it here as well as after a fresh requestAuthorization is safe. enableLiveDelivery() + // The high-res write-back added share types (HR stream, workouts, energy/distance) that a + // pre-update grant has as `.notDetermined`. Re-request once: HealthKit shows a single sheet + // listing ONLY the new types, and each write feature independently guards on its own type's + // share status, so declining any checkbox just skips that feature. + // Raw request, NOT requestAuthorization(): that method reclassifies a thrown error as + // `.denied`, which must never demote a bridge that just resumed a valid legacy grant. + let newTypesPending = writeTypes.contains { store.authorizationStatus(for: $0) == .notDetermined } + if newTypesPending { + Task { try? await store.requestAuthorization(toShare: writeTypes, read: readTypes) } + } } } @@ -412,10 +442,15 @@ final class HealthKitBridge: ObservableObject { // MARK: - Write back (NOOP → Health) - /// Write NOOP's strap-derived daily metrics (resting HR, HRV, SpO₂, respiratory rate) into Apple - /// Health so they appear across the user's Health ecosystem. + /// Write NOOP's strap-derived data into Apple Health: sleep sessions with full stage segments, + /// the continuous 1-minute heart-rate stream, strap/manual workouts, and the nightly vitals + /// (resting HR, HRV, SpO₂, respiratory rate) stamped at that day's wake time. /// - /// Dedup model: each emitted sample carries a deterministic `HKMetadataKeyExternalUUID` derived + /// Each feature saves independently and guards on ITS OWN type's share status, so one declined + /// Health checkbox (or a save error) skips that feature without sinking the rest; the first error + /// is rethrown at the end so `sync` still surfaces it in `lastError` without advancing `lastSync`. + /// + /// Dedup model (vitals): each emitted sample carries a deterministic `HKMetadataKeyExternalUUID` /// from `noopDeviceId + metric + day`. Before saving, we delete any of *our* prior samples that /// carry the same key (scoped to `HKSource.default()` so we never touch another app's data) and /// then save the fresh batch. HealthKit assigns a new UUID per save, so the previous strategy @@ -424,10 +459,48 @@ final class HealthKitBridge: ObservableObject { /// Throws on save failure so the caller can decide whether to advance `lastSync`. private func writeBack(whoopStore: WhoopStore, days: Int = 14) async throws { guard auth == .authorized else { return } + let now = Date() + guard let fromDate = Calendar.current.date(byAdding: .day, value: -days, to: now) else { return } + let fromTs = Int(fromDate.timeIntervalSince1970) + let nowTs = Int(now.timeIntervalSince1970) + + // Sleep sessions drive both the sleep write and the vitals' wake-time stamps: computed + // sessions (deviceId + "-noop") first, imported rows override on startTs collision — the + // same source precedence as the dailies union below and IntelligenceEngine's sleep reads. + let computedSleeps = (try? await whoopStore.sleepSessions(deviceId: computedDeviceId, from: fromTs, to: nowTs, limit: 200)) ?? [] + let importedSleeps = (try? await whoopStore.sleepSessions(deviceId: noopDeviceId, from: fromTs, to: nowTs, limit: 200)) ?? [] + var sleepsByStart: [Int: CachedSleepSession] = [:] + for s in computedSleeps { sleepsByStart[s.startTs] = s } + for s in importedSleeps { sleepsByStart[s.startTs] = s } + let sessions = sleepsByStart.keys.sorted().map { sleepsByStart[$0]! } + + var firstError: Error? + func attempt(_ op: () async throws -> Void) async { + do { try await op() } catch { if firstError == nil { firstError = error } } + } + await attempt { try await writeVitals(whoopStore: whoopStore, days: days, sessions: sessions) } + await attempt { try await writeSleep(sessions: sessions) } + await attempt { try await writeHeartRate(whoopStore: whoopStore, fromTs: fromTs, nowTs: nowTs) } + await attempt { try await writeWorkouts(whoopStore: whoopStore, fromTs: fromTs, toTs: nowTs) } + if let firstError { throw firstError } + } + + /// The nightly vitals write (the original write-back), now stamped at the day's wake time when + /// that day has a sleep session — a real timestamp inside the night the value describes, instead + /// of a fabricated noon. Keys are unchanged, so re-stamped samples replace their noon ancestors. + private func writeVitals(whoopStore: WhoopStore, days: Int, sessions: [CachedSleepSession]) async throws { let cal = Calendar.current let to = HealthKitBridge.dayString(Date()) guard let fromDate = cal.date(byAdding: .day, value: -days, to: Date()) else { return } let from = HealthKitBridge.dayString(fromDate) + + // day (of wake) → wake instant. Ascending session order means the latest wake of a day wins, + // matching collectSleep's end-date day attribution. + var wakeByDay: [String: Date] = [:] + for s in sessions where s.endTs > s.effectiveStartTs { + let wake = Date(timeIntervalSince1970: TimeInterval(s.endTs)) + wakeByDay[HealthKitBridge.dayString(wake)] = wake + } // Read NOOP's COMPUTED dailies (deviceId + "-noop"), which is the only place a strap-only // user's recovery/HRV/RHR/SpO₂/resp lives, then union with any imported `noopDeviceId` rows so // a user who ALSO imported a WHOOP export still gets the imported values. Imported overrides @@ -442,7 +515,8 @@ final class HealthKitBridge: ObservableObject { struct Candidate { let type: HKQuantityType; let key: String; let sample: HKQuantitySample } var candidates: [Candidate] = [] func add(_ id: HKQuantityTypeIdentifier, _ unit: HKUnit, _ value: Double, _ day: String, _ at: Date) { - guard let type = HKQuantityType.quantityType(forIdentifier: id) else { return } + guard let type = HKQuantityType.quantityType(forIdentifier: id), + store.authorizationStatus(for: type) == .sharingAuthorized else { return } let key = "noop:\(noopDeviceId):\(id.rawValue):\(day)" let sample = HKQuantitySample( type: type, @@ -456,17 +530,18 @@ final class HealthKitBridge: ObservableObject { for row in rows { guard let date = HealthKitBridge.date(from: row.day) else { continue } let noon = cal.date(bySettingHour: 12, minute: 0, second: 0, of: date) ?? date + let at = wakeByDay[row.day] ?? noon if let rhr = row.restingHr { - add(.restingHeartRate, HKUnit.count().unitDivided(by: .minute()), Double(rhr), row.day, noon) + add(.restingHeartRate, HKUnit.count().unitDivided(by: .minute()), Double(rhr), row.day, at) } if let hrv = row.avgHrv { - add(.heartRateVariabilitySDNN, .secondUnit(with: .milli), hrv, row.day, noon) + add(.heartRateVariabilitySDNN, .secondUnit(with: .milli), hrv, row.day, at) } if let spo2 = row.spo2Pct { - add(.oxygenSaturation, .percent(), spo2 / 100, row.day, noon) + add(.oxygenSaturation, .percent(), spo2 / 100, row.day, at) } if let rr = row.respRateBpm { - add(.respiratoryRate, HKUnit.count().unitDivided(by: .minute()), rr, row.day, noon) + add(.respiratoryRate, HKUnit.count().unitDivided(by: .minute()), rr, row.day, at) } } guard !candidates.isEmpty else { return } @@ -487,6 +562,233 @@ final class HealthKitBridge: ObservableObject { try await self.store.save(candidates.map { $0.sample }) } + /// Write each sleep session as one `.inBed` sample spanning `effectiveStartTs → endTs` plus one + /// category sample per stage segment (`deep → .asleepDeep`, `rem → .asleepREM`, + /// `light → .asleepCore`, `wake → .awake`) — the same shape Oura and Apple Watch write, so + /// Health renders the full hypnogram. Sessions whose `stagesJSON` carries no timing (the legacy + /// aggregate-minutes shapes) get one honest `.asleepUnspecified` block instead of fabricated + /// stage placement. + /// + /// Dedup: every sample of a session carries `HKMetadataKeyExternalUUID = + /// noop::sleep:` (the immutable detected onset, so a user-edited session + /// keeps its identity); delete-then-write scoped to our own `HKSource`, like the vitals. + private func writeSleep(sessions: [CachedSleepSession]) async throws { + guard let type = HKObjectType.categoryType(forIdentifier: .sleepAnalysis), + store.authorizationStatus(for: type) == .sharingAuthorized else { return } + var samples: [HKCategorySample] = [] + var keys: [String] = [] + for s in sessions { + let startTs = s.effectiveStartTs + guard s.endTs > startTs else { continue } + let start = Date(timeIntervalSince1970: TimeInterval(startTs)) + let end = Date(timeIntervalSince1970: TimeInterval(s.endTs)) + let key = "noop:\(noopDeviceId):sleep:\(s.startTs)" + keys.append(key) + let meta = [HKMetadataKeyExternalUUID: key] + samples.append(HKCategorySample(type: type, value: HKCategoryValueSleepAnalysis.inBed.rawValue, + start: start, end: end, metadata: meta)) + let stages = HealthWriteback.stageIntervals(stagesJSON: s.stagesJSON, + sessionStart: startTs, sessionEnd: s.endTs) + if stages.isEmpty { + samples.append(HKCategorySample(type: type, + value: HKCategoryValueSleepAnalysis.asleepUnspecified.rawValue, + start: start, end: end, metadata: meta)) + } else { + for seg in stages { + let value: HKCategoryValueSleepAnalysis + switch seg.kind { + case .awake: value = .awake + case .light: value = .asleepCore + case .deep: value = .asleepDeep + case .rem: value = .asleepREM + } + samples.append(HKCategorySample( + type: type, value: value.rawValue, + start: Date(timeIntervalSince1970: TimeInterval(seg.start)), + end: Date(timeIntervalSince1970: TimeInterval(seg.end)), + metadata: meta)) + } + } + } + guard !samples.isEmpty else { return } + let pred = NSCompoundPredicate(andPredicateWithSubpredicates: [ + HKQuery.predicateForObjects(from: HKSource.default()), + HKQuery.predicateForObjects(withMetadataKey: HKMetadataKeyExternalUUID, allowedValues: keys), + ]) + _ = try? await store.deleteObjects(of: type, predicate: pred) + try await store.save(samples) + } + + /// UserDefaults key for the HR write cursor (the newest bucket ts we've written). Per-strap so a + /// device switch restarts the backfill for the new strap instead of resuming mid-stream. + private var hrWriteCursorKey: String { "hkHRWriteCursor.v1.\(noopDeviceId)" } + + /// Write the strap's continuous heart rate as 1-minute mean samples — the same `hrBuckets` SQL + /// the charts read (measured-first, PPG fallback), so Health sees exactly what NOOP plots. Raw + /// ~1 Hz is deliberately downsampled: a fully-worn day is ~86k samples, which bloats the Health + /// store; 1/min matches Apple Watch's background cadence. + /// + /// Dedup: forward-only cursor plus a 48 h rewrite window. Each run deletes OUR OWN prior HR + /// samples in `[windowStart, now]` (source-scoped, date-range predicate — far cheaper than per- + /// sample external-UUID keys at this volume) and rewrites the window, so a strap offload that + /// backfills a recent night reconciles. Offloads older than 48 h behind the cursor are missed + /// until the cursor is cleared — accepted trade-off for not re-walking 14 days every sync. + private func writeHeartRate(whoopStore: WhoopStore, fromTs: Int, nowTs: Int) async throws { + guard let type = HKQuantityType.quantityType(forIdentifier: .heartRate), + store.authorizationStatus(for: type) == .sharingAuthorized else { return } + let cursor = UserDefaults.standard.integer(forKey: hrWriteCursorKey) + let windowStart = cursor > 0 ? max(fromTs, cursor - 48 * 3600) : fromTs + let buckets = (try? await whoopStore.hrBuckets(deviceId: noopDeviceId, from: windowStart, + to: nowTs, bucketSeconds: 60)) ?? [] + guard !buckets.isEmpty else { return } + + let pred = NSCompoundPredicate(andPredicateWithSubpredicates: [ + HKQuery.predicateForObjects(from: HKSource.default()), + HKQuery.predicateForSamples(withStart: Date(timeIntervalSince1970: TimeInterval(windowStart)), + end: Date(timeIntervalSince1970: TimeInterval(nowTs) + 60), + options: []), + ]) + _ = try? await store.deleteObjects(of: type, predicate: pred) + + let unit = HKUnit.count().unitDivided(by: .minute()) + var samples: [HKQuantitySample] = [] + samples.reserveCapacity(buckets.count) + for b in buckets { + let start = Date(timeIntervalSince1970: TimeInterval(b.ts)) + // Span the bucket, clamped so a bucket at the window edge can't end in the future + // (HealthKit rejects future-dated samples). + let end = Date(timeIntervalSince1970: TimeInterval(min(b.ts + 60, nowTs))) + samples.append(HKQuantitySample(type: type, + quantity: .init(unit: unit, doubleValue: b.bpm), + start: start, end: max(start, end))) + } + // First run backfills ~20k samples (14 d × 1440/day); chunk the saves so no single HealthKit + // transaction is oversized. Cursor only advances past what actually saved. + var lastSaved = cursor + var pending = samples[...] + var pendingTs = buckets.map(\.ts)[...] + while !pending.isEmpty { + let chunk = Array(pending.prefix(5000)) + let chunkTs = Array(pendingTs.prefix(5000)) + pending = pending.dropFirst(chunk.count) + pendingTs = pendingTs.dropFirst(chunk.count) + try await store.save(chunk) + lastSaved = max(lastSaved, chunkTs.last ?? lastSaved) + UserDefaults.standard.set(lastSaved, forKey: hrWriteCursorKey) + } + } + + /// Write strap-detected and manual workouts into Health via `HKWorkoutBuilder`, with an + /// `activeEnergyBurned` sample when the row has energy and a distance sample for distance + /// sports. Workouts whose source is `apple-health` are EXCLUDED — those were imported FROM + /// Health, and writing them back would duplicate the user's own Apple Watch/gym-app workouts. + /// + /// Dedup: `HKMetadataKeyExternalUUID = noop::workout:` in the workout + /// metadata; delete-then-write scoped to our own source, like sleep and the vitals. + private func writeWorkouts(whoopStore: WhoopStore, fromTs: Int, toTs: Int) async throws { + guard store.authorizationStatus(for: .workoutType()) == .sharingAuthorized else { return } + let mine = (try? await whoopStore.workouts(deviceId: noopDeviceId, from: fromTs, to: toTs, limit: 500)) ?? [] + let computed = (try? await whoopStore.workouts(deviceId: computedDeviceId, from: fromTs, to: toTs, limit: 500)) ?? [] + var byKey: [String: WorkoutRow] = [:] + for w in computed + mine where w.source != HealthKitBridge.appleWorkoutSource { + byKey["\(w.startTs):\(w.sport)"] = w + } + let rows = byKey.values.sorted { $0.startTs < $1.startTs } + guard !rows.isEmpty else { return } + + func key(_ row: WorkoutRow) -> String { "noop:\(noopDeviceId):workout:\(row.startTs)" } + let pred = NSCompoundPredicate(andPredicateWithSubpredicates: [ + HKQuery.predicateForObjects(from: HKSource.default()), + HKQuery.predicateForObjects(withMetadataKey: HKMetadataKeyExternalUUID, + allowedValues: rows.map(key)), + ]) + _ = try? await store.deleteObjects(of: .workoutType(), predicate: pred) + + for row in rows { + let start = Date(timeIntervalSince1970: TimeInterval(row.startTs)) + let end = Date(timeIntervalSince1970: TimeInterval(row.endTs)) + guard end > start else { continue } + let config = HKWorkoutConfiguration() + config.activityType = Self.activityType(forSport: row.sport) + let builder = HKWorkoutBuilder(healthStore: store, configuration: config, device: .local()) + do { + try await builder.beginCollection(at: start) + try await builder.addMetadata([HKMetadataKeyExternalUUID: key(row)]) + var extras: [HKSample] = [] + if let kcal = row.energyKcal, kcal > 0, + let t = HKQuantityType.quantityType(forIdentifier: .activeEnergyBurned), + store.authorizationStatus(for: t) == .sharingAuthorized { + extras.append(HKQuantitySample(type: t, quantity: .init(unit: .kilocalorie(), doubleValue: kcal), + start: start, end: end)) + } + if let meters = row.distanceM, meters > 0, + let id = Self.distanceTypeId(forSport: row.sport), + let t = HKQuantityType.quantityType(forIdentifier: id), + store.authorizationStatus(for: t) == .sharingAuthorized { + extras.append(HKQuantitySample(type: t, quantity: .init(unit: .meter(), doubleValue: meters), + start: start, end: end)) + } + if !extras.isEmpty { try await builder.addSamples(extras) } + try await builder.endCollection(at: end) + _ = try await builder.finishWorkout() + } catch { + builder.discardWorkout() + throw error + } + } + } + + /// Reverse of `sportName`: NOOP's sport label → the `HKWorkoutActivityType` written to Health. + /// Labels the forward map collapses (e.g. boxing/kickboxing → "Boxing") reverse to the first + /// member; unknown labels fall back to `.other`, never dropped. + private static func activityType(forSport sport: String) -> HKWorkoutActivityType { + if sport == LiftingImporter.sport { return .traditionalStrengthTraining } + switch sport.lowercased() { + case "running": return .running + case "walking": return .walking + case "hiking": return .hiking + case "cycling": return .cycling + case "hiit": return .highIntensityIntervalTraining + case "core training": return .coreTraining + case "yoga": return .yoga + case "pilates": return .pilates + case "rowing": return .rowing + case "elliptical": return .elliptical + case "stairs": return .stairClimbing + case "jump rope": return .jumpRope + case "boxing": return .boxing + case "basketball": return .basketball + case "soccer": return .soccer + case "football": return .americanFootball + case "baseball": return .baseball + case "badminton": return .badminton + case "tennis": return .tennis + case "table tennis": return .tableTennis + case "volleyball": return .volleyball + case "squash": return .squash + case "martial arts": return .martialArts + case "dancing": return .dance + case "golf": return .golf + case "climbing": return .climbing + case "skiing": return .downhillSkiing + case "snowboarding": return .snowboarding + case "swimming": return .swimming + case "surfing": return .surfingSports + case "paddling": return .paddleSports + default: return .other + } + } + + /// Which distance quantity a sport's `distanceM` maps to; nil for sports whose Health distance + /// type NOOP doesn't request share access for (e.g. swimming). + private static func distanceTypeId(forSport sport: String) -> HKQuantityTypeIdentifier? { + switch sport.lowercased() { + case "running", "walking", "hiking": return .distanceWalkingRunning + case "cycling": return .distanceCycling + default: return nil + } + } + private struct DayAgg { var restingHr: Double?; var avgHr: Double?; var maxHr: Double?; var hrv: Double? var spo2: Double?; var respRate: Double?; var steps: Double? diff --git a/docs/superpowers/specs/2026-07-10-apple-health-highres-writeback-design.md b/docs/superpowers/specs/2026-07-10-apple-health-highres-writeback-design.md new file mode 100644 index 0000000000..aefb23a6c1 --- /dev/null +++ b/docs/superpowers/specs/2026-07-10-apple-health-highres-writeback-design.md @@ -0,0 +1,61 @@ +# Apple Health high-resolution write-back — design + +**Date:** 2026-07-10 +**Scope:** iOS only (`StrandiOS/Health/HealthKitBridge.swift` + a pure helper in +`Packages/StrandImport`). Android Health Connect write-back is explicitly deferred — no analytics +or stored-data change, so the byte-parity contract is not triggered. + +## Problem + +`HealthKitBridge.writeBack` writes only four daily quantity samples (resting HR, HRV SDNN, SpO₂, +respiratory rate), each stamped at a fabricated noon timestamp. It requests share permission for +`sleepAnalysis` but never writes a sleep sample, so a user coming from Oura (which writes full +sleep sessions with stages) sees an empty Sleep section in Health. No heart-rate stream, no +workouts. + +## What gets written + +| Feature | NOOP source | HealthKit output | Idempotency key | +|---|---|---|---| +| Sleep sessions | `WhoopStore.sleepSessions` — computed (`deviceId+"-noop"`) unioned with imported (`deviceId`), imported wins on `startTs` | One `.inBed` sample spanning `effectiveStartTs→endTs`, plus one `HKCategorySample` per stage segment: `deep→.asleepDeep`, `rem→.asleepREM`, `light→.asleepCore`, `wake`/`awake`→`.awake`. Sessions without `stagesJSON` get one `.asleepUnspecified` block. | `noop::sleep:` on every sample of the session | +| Heart rate | `WhoopStore.hrBuckets(bucketSeconds: 60)` under the strap device id (measured-first with PPG fallback, same read the charts use) | 1-minute `heartRate` samples, 24/7. Raw 1 Hz deliberately downsampled: ~86–170k samples/day would bloat the Health store; 1/min matches Apple Watch background cadence and beats Oura's 5-min. | Forward-only cursor in UserDefaults (`hkHRWriteCursor.v1.`) + a 48 h rewrite window: each run deletes NOOP-authored HR in the window (source-scoped, date-range predicate) and rewrites, so late strap offloads inside 48 h reconcile. First run backfills the full 14-day window. | +| Workouts | `WhoopStore.workouts` for both device ids, **excluding** `source == "apple-health"` (those were imported FROM Health — writing them back would duplicate) | `HKWorkoutBuilder` workout with reverse-mapped `HKWorkoutActivityType`, an `activeEnergyBurned` sample when `energyKcal` is present, a distance sample for distance sports when `distanceM` is present. | `noop::workout:` in workout metadata; delete-then-write on `workoutType()` | +| Nightly vitals (existing) | unchanged | unchanged types, but stamped at the day's sleep-session end (wake time) when one exists instead of fabricated noon | unchanged (`noop:::`), so re-stamping replaces cleanly | + +## Deliberately excluded + +- **Raw 1 Hz HR** — Health-store bloat; 1-min mean via the existing `hrBuckets` SQL is the ceiling + that stays usable. +- **Skin temperature** — NOOP stores only a deviation from baseline (`skinTempDevC`), not absolute + °C; `appleSleepingWristTemperature` is not third-party-writable and writing a fabricated absolute + `bodyTemperature` would be dishonest data. +- **Daily steps / active-kcal estimates** — approximate HR-only estimates that would double-count + against the phone/watch in Activity totals. Honest energy goes on the workouts. + +## Authorization migration + +`writeTypes` grows by `heartRate`, `activeEnergyBurned`, `distanceWalkingRunning`, +`distanceCycling`, and `workoutType()`. Two consequences handled explicitly: + +1. `refreshAuthIfPreviouslyGranted` (resume without prompting) checks `allSatisfy(.sharingAuthorized)` + over `writeTypes` — a returning user who granted the old set would never resume. Fix: resume when + the **legacy core set** (4 vitals + sleep) is authorized; if the new types are still + `.notDetermined`, fire one `requestAuthorization()` so HealthKit shows a single sheet listing only + the new types. +2. Every write feature guards on its own type's `.sharingAuthorized` and saves independently — a + denied checkbox for one type can no longer sink the whole write-back. First error is kept for + `lastError`; remaining features still run. + +## Feedback-loop safety + +The read path (`collect*`, `fetchTouchedDayWindow` observers) already filters +`notNoopAuthored` (`NOT source == HKSource.default()`), so NOOP's own writes neither re-import nor +wake the observers. Verified against the current predicates. + +## Testing + +- Stage-segment parsing/clamping/normalization is a pure function in `Packages/StrandImport` + (`HealthWriteback.stageIntervals`) with `swift test` coverage: valid JSON, nil/garbage JSON, + `wake` vs `awake` vocab, segments clamped to session bounds, zero-length segments dropped. +- HealthKit itself is not unit-testable; app-target changes are verified by compiling `NOOPiOS` + locally (no default CI covers app targets per CLAUDE.md). From 1f58099067453d79800e12d70354db5c1b6f9c5b Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 19:33:49 -0700 Subject: [PATCH 033/171] fix(oura): request both spo2 scope spellings + skip failing endpoints instead of aborting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live test hit Oura's documented scope-name ambiguity: a token minted with only 'spo2Daily' (the OpenAPI spec's name) gets 401 on daily_spo2; the auth page calls it 'spo2'. Request both (authorize ignores unknown scopes — verified live). Also implement spec §7 honest-partial: one endpoint's failure no longer aborts the whole backfill — it's recorded in OuraSyncSummary.skippedEndpoints and surfaced in the UI with a retry hint. --- Strand/Oura/OuraConnectModel.swift | 6 +++- Strand/Oura/OuraOAuth.swift | 6 +++- Strand/Oura/OuraSyncCoordinator.swift | 41 +++++++++++++++------- Strand/Oura/OuraSyncResult.swift | 5 ++- StrandTests/OuraSyncCoordinatorTests.swift | 15 +++++++- 5 files changed, 57 insertions(+), 16 deletions(-) diff --git a/Strand/Oura/OuraConnectModel.swift b/Strand/Oura/OuraConnectModel.swift index a57260cf2a..911d5aad19 100644 --- a/Strand/Oura/OuraConnectModel.swift +++ b/Strand/Oura/OuraConnectModel.swift @@ -41,7 +41,11 @@ final class OuraConnectModel: ObservableObject { Task { @MainActor in self?.statusText = "Importing \(p.endpoint)…" } } await repo.refresh() - statusText = "Imported \(s.days) days · \(s.sleeps) sleeps · \(s.workouts) workouts · \(s.hrSamples) HR samples" + var line = "Imported \(s.days) days · \(s.sleeps) sleeps · \(s.workouts) workouts · \(s.hrSamples) HR samples" + if !s.skippedEndpoints.isEmpty { + line += " · skipped: \(s.skippedEndpoints.joined(separator: ", ")) — tap Sync again to retry" + } + statusText = line } catch { fail((error as? LocalizedError)?.errorDescription ?? error.localizedDescription) } busy = false } diff --git a/Strand/Oura/OuraOAuth.swift b/Strand/Oura/OuraOAuth.swift index df6f842e15..a2f135f2e9 100644 --- a/Strand/Oura/OuraOAuth.swift +++ b/Strand/Oura/OuraOAuth.swift @@ -6,7 +6,11 @@ import Foundation enum OuraOAuth { static let authorizeEndpoint = URL(string: "https://cloud.ouraring.com/oauth/authorize")! static let tokenEndpoint = URL(string: "https://api.ouraring.com/oauth/token")! - static let scopes = ["email", "personal", "daily", "heartrate", "workout", "tag", "session", "spo2Daily"] + // Oura's docs disagree on the SpO2 scope name: the OpenAPI spec says "spo2Daily", the auth page says + // "spo2", and live testing showed a token minted with only "spo2Daily" gets 401 on daily_spo2. Oura's + // authorize endpoint silently ignores scope names it doesn't recognize (verified: the "spo2Daily"-only + // request completed), so requesting BOTH spellings is safe and future-proof in either direction. + static let scopes = ["email", "personal", "daily", "heartrate", "workout", "tag", "session", "spo2", "spo2Daily"] /// The consent URL to open in ASWebAuthenticationSession. `state` is a caller-generated nonce echoed /// back on redirect and verified, to defeat CSRF / stray callbacks. diff --git a/Strand/Oura/OuraSyncCoordinator.swift b/Strand/Oura/OuraSyncCoordinator.swift index ed37426574..9bd7b0e46c 100644 --- a/Strand/Oura/OuraSyncCoordinator.swift +++ b/Strand/Oura/OuraSyncCoordinator.swift @@ -62,27 +62,44 @@ final class OuraSyncCoordinator { } } + // One failing endpoint must not abort the whole backfill (spec §7 — honest partial). A fetch + // error (e.g. a scope 401 like the live spo2 naming mismatch) records the endpoint as skipped + // and the run continues; the UI surfaces the skips from the summary. + var skipped: [String] = [] + // Daily endpoints first (readiness RHR precedence), extras accumulate. for endpoint in Self.dailyEndpoints { - let pages = try await fetchRaw(endpoint, dateParam: "start_date") - let (days, extras) = OuraApiParser.parseDaily(docs(pages), endpoint: endpoint) - merge(days); result.extras.append(contentsOf: extras) + do { + let pages = try await fetchRaw(endpoint, dateParam: "start_date") + let (days, extras) = OuraApiParser.parseDaily(docs(pages), endpoint: endpoint) + merge(days); result.extras.append(contentsOf: extras) + } catch { skipped.append(endpoint) } } // Sleep last (its lowestHr fallback only fills days readiness didn't cover). - let sleepPages = try await fetchRaw("sleep", dateParam: "start_date") - let (periods, sleepDays) = OuraApiParser.parseSleep(docs(sleepPages)) - result.sleepPeriods = periods; merge(sleepDays) + do { + let sleepPages = try await fetchRaw("sleep", dateParam: "start_date") + let (periods, sleepDays) = OuraApiParser.parseSleep(docs(sleepPages)) + result.sleepPeriods = periods; merge(sleepDays) + } catch { skipped.append("sleep") } // Workouts + heart-rate. - let workoutPages = try await fetchRaw("workout", dateParam: "start_date") - result.workouts = OuraApiParser.parseWorkouts(docs(workoutPages)) - let hrPages = try await fetchRaw("heartrate", dateParam: "start_datetime") - result.heartRate = OuraApiParser.parseHeartRate(docs(hrPages)) + do { + let workoutPages = try await fetchRaw("workout", dateParam: "start_date") + result.workouts = OuraApiParser.parseWorkouts(docs(workoutPages)) + } catch { skipped.append("workout") } + do { + let hrPages = try await fetchRaw("heartrate", dateParam: "start_datetime") + result.heartRate = OuraApiParser.parseHeartRate(docs(hrPages)) + } catch { skipped.append("heartrate") } // Raw-only endpoints (no parser). for endpoint in Self.rawOnlyEndpoints { - _ = try await fetchRaw(endpoint, dateParam: ["personal_info", "ring_configuration"].contains(endpoint) ? nil : "start_date") + do { + _ = try await fetchRaw(endpoint, dateParam: ["personal_info", "ring_configuration"].contains(endpoint) ? nil : "start_date") + } catch { skipped.append(endpoint) } } result.days = Array(byDay.values) - return try await OuraSyncWriter.persist(result, into: store) + var summary = try await OuraSyncWriter.persist(result, into: store) + summary.skippedEndpoints = skipped + return summary } } diff --git a/Strand/Oura/OuraSyncResult.swift b/Strand/Oura/OuraSyncResult.swift index c9e15800b0..b4ce94a25e 100644 --- a/Strand/Oura/OuraSyncResult.swift +++ b/Strand/Oura/OuraSyncResult.swift @@ -21,7 +21,10 @@ struct OuraSyncResult { } } -/// Counts written, for the honest import summary. +/// Counts written, for the honest import summary. `skippedEndpoints` lists endpoints whose fetch failed +/// (e.g. a scope 401) — the backfill continues past them (honest partial, spec §7) and the UI surfaces +/// the skips so a partial import never silently looks complete. struct OuraSyncSummary: Equatable { var days = 0, sleeps = 0, workouts = 0, hrSamples = 0, metricPoints = 0, rawPages = 0 + var skippedEndpoints: [String] = [] } diff --git a/StrandTests/OuraSyncCoordinatorTests.swift b/StrandTests/OuraSyncCoordinatorTests.swift index 5afe1079fe..9036cd2d9d 100644 --- a/StrandTests/OuraSyncCoordinatorTests.swift +++ b/StrandTests/OuraSyncCoordinatorTests.swift @@ -3,12 +3,14 @@ import WhoopStore @testable import StrandImport @testable import Strand -/// A fetcher that serves canned page bodies per endpoint. +/// A fetcher that serves canned page bodies per endpoint (and can fail chosen endpoints, like a scope 401). private final class StubFetcher: OuraPageFetching { var pages: [String: [Data]] = [:] var capturedQueries: [String: [String: String]] = [:] + var failEndpoints: Set = [] func fetchAllRaw(endpoint: String, query: [String: String]) async throws -> [Data] { capturedQueries[endpoint] = query + if failEndpoints.contains(endpoint) { throw OuraError.badResponse(401, "token is not authorized") } return pages[endpoint] ?? [] } } @@ -53,4 +55,15 @@ final class OuraSyncCoordinatorTests: XCTestCase { XCTAssertNotNil(fetcher.capturedQueries["daily_readiness"]?["start_date"]) XCTAssertNotNil(fetcher.capturedQueries["daily_readiness"]?["end_date"]) } + + func testFailingEndpointIsSkippedNotFatal() async throws { + let store = try await WhoopStore.inMemory() + let fetcher = StubFetcher() + fetcher.pages["daily_readiness"] = [#"{"data":[{"day":"2026-01-02","score":80,"contributors":{"resting_heart_rate":50}}],"next_token":null}"#.data(using: .utf8)!] + fetcher.failEndpoints = ["daily_spo2"] // the live failure mode: scope 401 on one endpoint + let coord = OuraSyncCoordinator(fetcher: fetcher, store: store) + let summary = try await coord.runFullImport(today: "2026-02-01") + XCTAssertEqual(summary.days, 1) // the rest of the backfill still lands + XCTAssertEqual(summary.skippedEndpoints, ["daily_spo2"]) // and the skip is reported honestly + } } From 2b20af91546b883c0a716dad5db890c3ef866cae Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 19:38:50 -0700 Subject: [PATCH 034/171] =?UTF-8?q?fix(oura):=20omit=20the=20OAuth=20scope?= =?UTF-8?q?=20param=20=E2=80=94=20Oura=20grants=20all=20app-configured=20s?= =?UTF-8?q?copes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Oura's auth docs: scope is optional and blank = request ALL available scopes. This sidesteps both live traps: the spo2/spo2Daily naming mismatch AND the three portal-only scopes the public docs never list (Stress, Heart Health, Ring Configuration — docs still say 8 scopes, the dev portal shows 11). The user still chooses on Oura's consent screen; the import's skip-and-continue handles any scope they decline. --- Strand/Oura/OuraOAuth.swift | 14 ++++++++------ StrandTests/OuraOAuthTests.swift | 4 +++- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/Strand/Oura/OuraOAuth.swift b/Strand/Oura/OuraOAuth.swift index a2f135f2e9..507dbda72b 100644 --- a/Strand/Oura/OuraOAuth.swift +++ b/Strand/Oura/OuraOAuth.swift @@ -6,11 +6,13 @@ import Foundation enum OuraOAuth { static let authorizeEndpoint = URL(string: "https://cloud.ouraring.com/oauth/authorize")! static let tokenEndpoint = URL(string: "https://api.ouraring.com/oauth/token")! - // Oura's docs disagree on the SpO2 scope name: the OpenAPI spec says "spo2Daily", the auth page says - // "spo2", and live testing showed a token minted with only "spo2Daily" gets 401 on daily_spo2. Oura's - // authorize endpoint silently ignores scope names it doesn't recognize (verified: the "spo2Daily"-only - // request completed), so requesting BOTH spellings is safe and future-proof in either direction. - static let scopes = ["email", "personal", "daily", "heartrate", "workout", "tag", "session", "spo2", "spo2Daily"] + // No hardcoded scope list. Oura's own docs: the `scope` param is OPTIONAL and "if left blank, the + // application will request ALL available scopes" — which is exactly the goal, and sidesteps two + // documentation traps hit live: the SpO2 scope's name differs between the OpenAPI spec ("spo2Daily") + // and the auth docs ("spo2"), and the dev portal grants scopes the public docs don't list at all + // (Stress, Heart Health, Ring Configuration — the docs still say "8 scopes", the portal shows 11). + // Omitting `scope` requests everything the app registration is configured for; the user still picks + // on Oura's consent screen. /// The consent URL to open in ASWebAuthenticationSession. `state` is a caller-generated nonce echoed /// back on redirect and verified, to defeat CSRF / stray callbacks. @@ -20,7 +22,7 @@ enum OuraOAuth { .init(name: "response_type", value: "code"), .init(name: "client_id", value: credentials.clientId), .init(name: "redirect_uri", value: credentials.redirectURI), - .init(name: "scope", value: scopes.joined(separator: " ")), + // `scope` deliberately omitted — see the note above: blank = all available scopes. .init(name: "state", value: state), ] return comps.url! diff --git a/StrandTests/OuraOAuthTests.swift b/StrandTests/OuraOAuthTests.swift index 90f9e9ca73..fa394f5211 100644 --- a/StrandTests/OuraOAuthTests.swift +++ b/StrandTests/OuraOAuthTests.swift @@ -15,7 +15,9 @@ final class OuraOAuthTests: XCTestCase { XCTAssertEqual(q["client_id"], "cid") XCTAssertEqual(q["redirect_uri"], "noop://oura/callback") XCTAssertEqual(q["state"], "xyz") - XCTAssertEqual(q["scope"], OuraOAuth.scopes.joined(separator: " ")) + // `scope` is deliberately omitted: Oura grants ALL app-configured scopes when it's blank — + // covering the portal-only scopes (Stress, Heart Health, Ring Configuration) the docs don't name. + XCTAssertNil(q["scope"]) } func testTokenExchangeRequestIsFormPost() throws { From c020fea906f622822ffc54f730c00fd2d2e32781 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 19:54:29 -0700 Subject: [PATCH 035/171] Predictive low-battery alert: warn at ~24h of estimated runtime left The fixed 15% low alert (#368) gives ~16h of warning on a WHOOP 4.0 but ~1.8 days on a 5.0/MG. BatteryEstimator (#713) already fits a personal runtime estimate from the banked SoC series; connect the two: - BatteryEstimator.runtimeAlert (Swift + Kotlin twins, same fixtures): fire when remainingHours <= 24 and not confirmed charging, re-arm only at >= 36 so slope jitter can't re-fire; suppressed-while-charging does not consume the once-per-discharge gate. - iOS/macOS: BatteryNotifier.onRuntimeEstimate posts "~22h left on your WHOOP" with the live figure; evaluated in AppModel's existing battery hook where the just-banked estimate is fresh. - Android: BatteryAlertNotifier.onRuntimeEstimate (id 4205) fed from WhoopConnectionService, re-fitting from the persisted battery table only when the SoC actually changes (~8-min strap cadence); same samples/rated inputs as the Today badge. - The 15% alert stays as the safety net (covers the nil-estimate case); everything remains behind the existing "Battery alerts" setting. Verified: StrandAnalytics swift test (980 passing, 7 new), Strand and NOOPiOS schemes compile. Android JVM tests are behavior-identical mirrors of the Swift fixtures; no Android toolchain on this machine, so Kotlin is signature-checked against the existing call-site patterns but not compiled here. --- .../StrandAnalytics/BatteryEstimator.swift | 28 ++++++++ .../BatteryRuntimeAlertTests.swift | 57 ++++++++++++++++ Strand/App/AppModel.swift | 6 ++ Strand/System/BatteryNotifier.swift | 22 +++++++ .../com/noop/analytics/BatteryEstimator.kt | 30 +++++++++ .../com/noop/ble/WhoopConnectionService.kt | 28 ++++++++ .../com/noop/notif/BatteryAlertNotifier.kt | 39 +++++++++++ .../src/main/java/com/noop/ui/MainActivity.kt | 12 ++++ .../noop/analytics/BatteryRuntimeAlertTest.kt | 65 +++++++++++++++++++ ...6-07-10-predictive-battery-alert-design.md | 53 +++++++++++++++ 10 files changed, 340 insertions(+) create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BatteryRuntimeAlertTests.swift create mode 100644 android/app/src/test/java/com/noop/analytics/BatteryRuntimeAlertTest.kt create mode 100644 docs/superpowers/specs/2026-07-10-predictive-battery-alert-design.md diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/BatteryEstimator.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/BatteryEstimator.swift index 36ed112f6b..a83c0d0684 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/BatteryEstimator.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/BatteryEstimator.swift @@ -246,4 +246,32 @@ public enum BatteryEstimator { if hours < 48 { return "~\(Int(hours.rounded()))h" } return "~\(String(format: "%.1f", hours / 24)) days" } + + // MARK: - Predictive low-battery alert policy + + /// Fire the runtime alert when the estimate drops to this many hours of remaining life. A fixed + /// SoC threshold gives wildly different lead time per strap generation (15% is ~16 h on a 4.0 but + /// ~1.8 days on a 5.0/MG); a runtime threshold means the same "charge it tonight" warning for both. + public static let runtimeAlertHours: Double = 24 + /// Re-arm only when the estimate recovers to this. The 12 h hysteresis band means jitter in the + /// fitted slope around the alert line can't re-fire; only a genuine charge opens the gate again. + public static let runtimeRearmHours: Double = 36 + + /// Crossing-with-hysteresis decision for the predictive alert, mirroring + /// `BatteryNotifier.BatteryAlertPolicy.evaluate`'s shape: PERSISTED `alerted` gate in, fire + /// decision plus next gate state out. Pure so it's `swift test`-covered here (the SoC policy + /// lives in the app target where only xcodebuild tests reach it). + /// + /// `charging == nil` means unknown — the alert still fires (only a confirmed `true` suppresses + /// it), the same null-tolerant rule as the SoC low alert: the strap reports its charge bit only + /// every ~8 min, and a strap that never reports one should still warn. + public static func runtimeAlert(remainingHours: Double, + charging: Bool?, + alerted: Bool) -> (fire: Bool, newAlerted: Bool) { + var armedOff = alerted + if remainingHours >= runtimeRearmHours { armedOff = false } + let fire = !armedOff && remainingHours <= runtimeAlertHours && charging != true + if fire { armedOff = true } + return (fire, armedOff) + } } diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BatteryRuntimeAlertTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BatteryRuntimeAlertTests.swift new file mode 100644 index 0000000000..cee4480cab --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BatteryRuntimeAlertTests.swift @@ -0,0 +1,57 @@ +import XCTest +@testable import StrandAnalytics + +final class BatteryRuntimeAlertTests: XCTestCase { + + func testFiresAtThresholdAndArms() { + let r = BatteryEstimator.runtimeAlert(remainingHours: 24, charging: false, alerted: false) + XCTAssertTrue(r.fire) + XCTAssertTrue(r.newAlerted) + } + + func testAboveThresholdNoFire() { + let r = BatteryEstimator.runtimeAlert(remainingHours: 25, charging: false, alerted: false) + XCTAssertFalse(r.fire) + XCTAssertFalse(r.newAlerted) + } + + func testJitterInsideBandCannotRefire() { + // Fired at 24h, estimate bounces 23 → 26 → 22: still armed-off, no second alert. + var alerted = BatteryEstimator.runtimeAlert(remainingHours: 23, charging: nil, alerted: false).newAlerted + XCTAssertTrue(alerted) + for hours in [26.0, 22.0, 30.0, 24.0] { + let r = BatteryEstimator.runtimeAlert(remainingHours: hours, charging: nil, alerted: alerted) + XCTAssertFalse(r.fire, "re-fired at \(hours)h inside the hysteresis band") + alerted = r.newAlerted + } + } + + func testRearmsOnGenuineRecoveryThenFiresNextCycle() { + // Charge recovers the estimate past 36h → gate re-arms; the next drop to 24h fires again. + var alerted = BatteryEstimator.runtimeAlert(remainingHours: 20, charging: nil, alerted: false).newAlerted + let recovered = BatteryEstimator.runtimeAlert(remainingHours: 100, charging: nil, alerted: alerted) + XCTAssertFalse(recovered.fire) + XCTAssertFalse(recovered.newAlerted) + alerted = recovered.newAlerted + XCTAssertTrue(BatteryEstimator.runtimeAlert(remainingHours: 24, charging: nil, alerted: alerted).fire) + } + + func testConfirmedChargingSuppressesButUnknownFires() { + XCTAssertFalse(BatteryEstimator.runtimeAlert(remainingHours: 10, charging: true, alerted: false).fire) + XCTAssertTrue(BatteryEstimator.runtimeAlert(remainingHours: 10, charging: nil, alerted: false).fire) + } + + func testChargingSuppressionDoesNotArmGate() { + // Suppressed while charging must NOT consume the once-per-cycle gate: unplug below the + // threshold and the alert should still fire. + let plugged = BatteryEstimator.runtimeAlert(remainingHours: 10, charging: true, alerted: false) + XCTAssertFalse(plugged.newAlerted) + XCTAssertTrue(BatteryEstimator.runtimeAlert(remainingHours: 10, charging: false, + alerted: plugged.newAlerted).fire) + } + + func testRearmBoundaryIsInclusive() { + XCTAssertFalse(BatteryEstimator.runtimeAlert(remainingHours: 36, charging: nil, alerted: true).newAlerted) + XCTAssertTrue(BatteryEstimator.runtimeAlert(remainingHours: 35.9, charging: nil, alerted: true).newAlerted) + } +} diff --git a/Strand/App/AppModel.swift b/Strand/App/AppModel.swift index 89c36d7940..7e2ff75fb3 100644 --- a/Strand/App/AppModel.swift +++ b/Strand/App/AppModel.swift @@ -257,6 +257,12 @@ final class AppModel: ObservableObject { BatteryNotifier.onBatteryUpdate(pct: Int(pct.rounded()), charging: self.live.charging, enabled: self.behavior.batteryAlerts) + // Predictive runtime alert: the same reading just banked into the SoC buffer, so + // batteryEstimate is fresh here. Nil estimate (no readings yet) is a no-op — the 15% + // alert above remains the safety net. + BatteryNotifier.onRuntimeEstimate(remainingHours: self.live.batteryEstimate?.remainingHours, + charging: self.live.charging, + enabled: self.behavior.batteryAlerts) } // HR-zone haptic coaching watches the smoothed bpm. $bpm.sink { [weak self] hr in self?.coachZone(hr) }.store(in: &hrCancellables) diff --git a/Strand/System/BatteryNotifier.swift b/Strand/System/BatteryNotifier.swift index 0c543584a1..7ce20fe527 100644 --- a/Strand/System/BatteryNotifier.swift +++ b/Strand/System/BatteryNotifier.swift @@ -1,5 +1,6 @@ import Foundation import UserNotifications +import StrandAnalytics /// Surfaces the strap battery state as a user notification — a LOW warning when the cell falls to /// the threshold so the user can top up before tonight's sleep, and a CHARGED note when it reaches @@ -9,6 +10,7 @@ import UserNotifications enum BatteryNotifier { private static let lowAlertedKey = "behavior.batteryLowAlerted" private static let fullAlertedKey = "behavior.batteryFullAlerted" + private static let runtimeAlertedKey = "behavior.batteryRuntimeAlerted" /// Pure crossing-with-hysteresis policy, identical on macOS/iOS and Android (#368). The two /// `*Alerted` flags are PERSISTED, so they survive process death — and the 25% re-arm band means @@ -93,6 +95,26 @@ enum BatteryNotifier { } } + /// Predictive twin of `onBatteryUpdate`: run the runtime estimate against + /// `BatteryEstimator.runtimeAlert` (fire ≤24 h, re-arm ≥36 h — see the policy for why a runtime + /// threshold beats a fixed SoC one) and post at most one notification per discharge cycle. The + /// 15% SoC alert stays as the safety net for straps with no usable estimate (`estimate == nil` + /// is a no-op here). Same gating discipline as #368: the persisted flag advances even when + /// delivery is deferred, and the whole thing no-ops when the "Battery alerts" setting is off. + static func onRuntimeEstimate(remainingHours: Double?, charging: Bool?, enabled: Bool) { + guard enabled, let remainingHours else { return } + let d = UserDefaults.standard + let result = BatteryEstimator.runtimeAlert(remainingHours: remainingHours, + charging: charging, + alerted: d.bool(forKey: runtimeAlertedKey)) + d.set(result.newAlerted, forKey: runtimeAlertedKey) + if result.fire { + post(identifier: "battery-runtime", + title: String(localized: "Strap battery low"), + body: String(localized: "\(BatteryEstimator.label(hours: remainingHours)) left on your WHOOP — recharge tonight.")) + } + } + private static func post(identifier: String, title: String, body: String) { let center = UNUserNotificationCenter.current() // Authorization is requested once via requestAuthorization() when alerts are enabled; here diff --git a/android/app/src/main/java/com/noop/analytics/BatteryEstimator.kt b/android/app/src/main/java/com/noop/analytics/BatteryEstimator.kt index 4e5fdb4f00..642317aad0 100644 --- a/android/app/src/main/java/com/noop/analytics/BatteryEstimator.kt +++ b/android/app/src/main/java/com/noop/analytics/BatteryEstimator.kt @@ -248,4 +248,34 @@ object BatteryEstimator { fun label(hours: Double): String = if (hours < 48) "~${hours.roundToLong()}h" else "~${String.format(Locale.US, "%.1f", hours / 24)} days" + + // ---- Predictive low-battery alert policy ---- + + /** Fire the runtime alert when the estimate drops to this many hours of remaining life. A fixed + * SoC threshold gives wildly different lead time per strap generation (15% is ~16 h on a 4.0 but + * ~1.8 days on a 5.0/MG); a runtime threshold means the same "charge it tonight" warning for both. */ + const val runtimeAlertHours = 24.0 + + /** Re-arm only when the estimate recovers to this. The 12 h hysteresis band means jitter in the + * fitted slope around the alert line can't re-fire; only a genuine charge opens the gate again. */ + const val runtimeRearmHours = 36.0 + + data class RuntimeAlertDecision(val fire: Boolean, val newAlerted: Boolean) + + /** + * Crossing-with-hysteresis decision for the predictive alert, mirroring BatteryAlertPolicy's + * shape: PERSISTED `alerted` gate in, fire decision plus next gate state out. Behaviour-identical + * twin of the Swift `BatteryEstimator.runtimeAlert` (same fixtures, same numbers). + * + * `charging == null` means unknown — the alert still fires (only a confirmed `true` suppresses + * it), the same null-tolerant rule as the SoC low alert: the strap reports its charge bit only + * every ~8 min, and a strap that never reports one should still warn. + */ + fun runtimeAlert(remainingHours: Double, charging: Boolean?, alerted: Boolean): RuntimeAlertDecision { + var armedOff = alerted + if (remainingHours >= runtimeRearmHours) armedOff = false + val fire = !armedOff && remainingHours <= runtimeAlertHours && charging != true + if (fire) armedOff = true + return RuntimeAlertDecision(fire, armedOff) + } } diff --git a/android/app/src/main/java/com/noop/ble/WhoopConnectionService.kt b/android/app/src/main/java/com/noop/ble/WhoopConnectionService.kt index 4e54effa38..c6963f3639 100644 --- a/android/app/src/main/java/com/noop/ble/WhoopConnectionService.kt +++ b/android/app/src/main/java/com/noop/ble/WhoopConnectionService.kt @@ -21,6 +21,7 @@ import com.noop.R import com.noop.alarm.SleepWindowWatcher import com.noop.alarm.SmartAlarmScheduler import com.noop.alarm.SmartAlarmStore +import com.noop.analytics.BatteryEstimator import com.noop.analytics.IllnessWatch import com.noop.analytics.RestScorer import com.noop.data.DailyMetric @@ -103,6 +104,11 @@ class WhoopConnectionService : Service() { * process restarts and the AppViewModel call site. */ private var lastIllnessAlert: String? = null + /** Last battery % the predictive runtime alert was evaluated at. The live-state flow emits far + * more often than the strap's ~8-min battery cadence; gating the Room read + estimator fit on an + * actual SoC change keeps the predictive path as cheap as the SoC-only alert beside it. */ + private var lastRuntimeEvalPct: Int? = null + /** Smart-alarm light-sleep watcher (#207). Feeds the live HR while we're inside the wake window * and, on a lighter-phase reading, advances the GUARANTEED alarm earlier. It can only ever move * the alarm earlier within the window — the hard deadline scheduled via AlarmManager is the floor @@ -240,6 +246,28 @@ class WhoopConnectionService : Service() { currPct = state.batteryPct?.roundToInt(), charging = state.charging, ) + // Predictive runtime alert (iOS/macOS twin: BatteryNotifier.onRuntimeEstimate): + // re-fit the "~X left" estimate from the persisted SoC series and warn at ≤24 h of + // runtime, whatever the strap generation. Evaluated only when the battery % actually + // changes (~8-min strap cadence), so the Room read + slope fit never rides every + // live-state emission. Same samples/rated inputs as the Today badge, so the alert can + // never disagree with the number on screen. + val runtimePct = state.batteryPct?.roundToInt() + if (runtimePct != null && runtimePct != lastRuntimeEvalPct) { + lastRuntimeEvalPct = runtimePct + runCatching { + val nowS = System.currentTimeMillis() / 1000 + val samples = repo.batterySamples("my-whoop", nowS - 14L * 86_400, nowS, limit = 2_000) + .mapNotNull { s -> s.soc?.let { s.ts to it } } + val rated = if (state.whoop5Detected) BatteryEstimator.ratedLifeHoursWhoop5 + else BatteryEstimator.ratedLifeHoursWhoop4 + BatteryAlertNotifier.onRuntimeEstimate( + this@WhoopConnectionService, + remainingHours = BatteryEstimator.estimate(samples, rated)?.hoursRemaining, + charging = state.charging, + ) + } + } // Feed the home-screen widget from the same stream — this service is its heartbeat // while the app UI is closed. Throttled + no-op without a placed widget (the store // checks both); runCatching so a Glance hiccup never tears down the connection. diff --git a/android/app/src/main/java/com/noop/notif/BatteryAlertNotifier.kt b/android/app/src/main/java/com/noop/notif/BatteryAlertNotifier.kt index 3cb520deb3..16e61abf6d 100644 --- a/android/app/src/main/java/com/noop/notif/BatteryAlertNotifier.kt +++ b/android/app/src/main/java/com/noop/notif/BatteryAlertNotifier.kt @@ -77,6 +77,45 @@ object BatteryAlertNotifier { private const val CHANNEL_ID = "noop_battery_alert" private const val NOTIF_ID_LOW = 4203 private const val NOTIF_ID_FULL = 4204 + private const val NOTIF_ID_RUNTIME = 4205 + + /** + * Predictive twin of [onBatteryUpdate]: run the runtime estimate against + * [com.noop.analytics.BatteryEstimator.runtimeAlert] (fire ≤24 h, re-arm ≥36 h — a runtime + * threshold gives the same warning lead time on a 4.0 and a 5.0/MG, which a fixed SoC line + * can't) and post at most one notification per discharge cycle. The 15% SoC alert stays as the + * safety net for straps with no usable estimate (null skips here). Same gating discipline as + * #368: persisted flag advances even when delivery is deferred; no-ops when battery alerts are + * off. iOS/macOS twin: BatteryNotifier.onRuntimeEstimate. + */ + @SuppressLint("MissingPermission") // guarded by areNotificationsEnabled() + runCatching + fun onRuntimeEstimate(context: Context, remainingHours: Double?, charging: Boolean?) { + if (remainingHours == null) return + if (!NoopPrefs.batteryAlerts(context)) return + runCatching { + val decision = com.noop.analytics.BatteryEstimator.runtimeAlert( + remainingHours = remainingHours, + charging = charging, + alerted = NoopPrefs.batteryRuntimeAlerted(context), + ) + // ALWAYS persist the updated gate — re-arming must stick even when nothing fired. + NoopPrefs.setBatteryRuntimeAlerted(context, decision.newAlerted) + if (!decision.fire) return + if (!NotificationManagerCompat.from(context).areNotificationsEnabled()) return + ensureChannel(context) + val label = com.noop.analytics.BatteryEstimator.label(remainingHours) + val n = NotificationCompat.Builder(context, CHANNEL_ID) + .setSmallIcon(R.drawable.ic_stat_heart) + .setContentTitle("Strap battery low") + .setContentText("$label left on your WHOOP — recharge tonight.") + .setContentIntent(openAppIntent(context)) + .setAutoCancel(true) + .setCategory(NotificationCompat.CATEGORY_STATUS) + .setPriority(NotificationCompat.PRIORITY_HIGH) + .build() + NotificationManagerCompat.from(context).notify(NOTIF_ID_RUNTIME, n) + } + } @SuppressLint("MissingPermission") // guarded by areNotificationsEnabled() + runCatching fun onBatteryUpdate(context: Context, currPct: Int?, charging: Boolean?) { diff --git a/android/app/src/main/java/com/noop/ui/MainActivity.kt b/android/app/src/main/java/com/noop/ui/MainActivity.kt index 8a56d0f3b5..8b7ddc60d3 100644 --- a/android/app/src/main/java/com/noop/ui/MainActivity.kt +++ b/android/app/src/main/java/com/noop/ui/MainActivity.kt @@ -584,6 +584,18 @@ object NoopPrefs { of(context).edit().putBoolean(KEY_BATTERY_FULL_ALERTED, alerted).apply() } + /** Persisted once-per-discharge gate behind BatteryEstimator.runtimeAlert (the predictive + * "~X left" alert; fires ≤24 h, re-arms ≥36 h). Same survive-process-death contract as the + * SoC flags above. */ + const val KEY_BATTERY_RUNTIME_ALERTED = "noop.batteryRuntimeAlerted" + + fun batteryRuntimeAlerted(context: Context): Boolean = + of(context).getBoolean(KEY_BATTERY_RUNTIME_ALERTED, false) + + fun setBatteryRuntimeAlerted(context: Context, alerted: Boolean) { + of(context).edit().putBoolean(KEY_BATTERY_RUNTIME_ALERTED, alerted).apply() + } + /** Scheduled report notifications (#517), opt-in, default OFF, no AI. Two independent toggles: * - [KEY_REPORT_MORNING]: a morning recap (Charge + Rest) posted once after a fresh night is * processed. It is NOT alarm-precise, it lands when the next sync + analytics pass completes, diff --git a/android/app/src/test/java/com/noop/analytics/BatteryRuntimeAlertTest.kt b/android/app/src/test/java/com/noop/analytics/BatteryRuntimeAlertTest.kt new file mode 100644 index 0000000000..90d227178e --- /dev/null +++ b/android/app/src/test/java/com/noop/analytics/BatteryRuntimeAlertTest.kt @@ -0,0 +1,65 @@ +package com.noop.analytics + +import org.junit.Assert.assertFalse +import org.junit.Assert.assertTrue +import org.junit.Test + +/** Behaviour-identical twin of the Swift BatteryRuntimeAlertTests — same fixtures, same numbers. */ +class BatteryRuntimeAlertTest { + + @Test + fun firesAtThresholdAndArms() { + val r = BatteryEstimator.runtimeAlert(remainingHours = 24.0, charging = false, alerted = false) + assertTrue(r.fire) + assertTrue(r.newAlerted) + } + + @Test + fun aboveThresholdNoFire() { + val r = BatteryEstimator.runtimeAlert(remainingHours = 25.0, charging = false, alerted = false) + assertFalse(r.fire) + assertFalse(r.newAlerted) + } + + @Test + fun jitterInsideBandCannotRefire() { + // Fired at 23h, estimate bounces 26 → 22 → 30 → 24: still armed-off, no second alert. + var alerted = BatteryEstimator.runtimeAlert(23.0, charging = null, alerted = false).newAlerted + assertTrue(alerted) + for (hours in listOf(26.0, 22.0, 30.0, 24.0)) { + val r = BatteryEstimator.runtimeAlert(hours, charging = null, alerted = alerted) + assertFalse("re-fired at ${hours}h inside the hysteresis band", r.fire) + alerted = r.newAlerted + } + } + + @Test + fun rearmsOnGenuineRecoveryThenFiresNextCycle() { + var alerted = BatteryEstimator.runtimeAlert(20.0, charging = null, alerted = false).newAlerted + val recovered = BatteryEstimator.runtimeAlert(100.0, charging = null, alerted = alerted) + assertFalse(recovered.fire) + assertFalse(recovered.newAlerted) + assertTrue(BatteryEstimator.runtimeAlert(24.0, charging = null, alerted = recovered.newAlerted).fire) + } + + @Test + fun confirmedChargingSuppressesButUnknownFires() { + assertFalse(BatteryEstimator.runtimeAlert(10.0, charging = true, alerted = false).fire) + assertTrue(BatteryEstimator.runtimeAlert(10.0, charging = null, alerted = false).fire) + } + + @Test + fun chargingSuppressionDoesNotArmGate() { + // Suppressed while charging must NOT consume the once-per-cycle gate: unplug below the + // threshold and the alert should still fire. + val plugged = BatteryEstimator.runtimeAlert(10.0, charging = true, alerted = false) + assertFalse(plugged.newAlerted) + assertTrue(BatteryEstimator.runtimeAlert(10.0, charging = false, alerted = plugged.newAlerted).fire) + } + + @Test + fun rearmBoundaryIsInclusive() { + assertFalse(BatteryEstimator.runtimeAlert(36.0, charging = null, alerted = true).newAlerted) + assertTrue(BatteryEstimator.runtimeAlert(35.9, charging = null, alerted = true).newAlerted) + } +} diff --git a/docs/superpowers/specs/2026-07-10-predictive-battery-alert-design.md b/docs/superpowers/specs/2026-07-10-predictive-battery-alert-design.md new file mode 100644 index 0000000000..5250fe7bf2 --- /dev/null +++ b/docs/superpowers/specs/2026-07-10-predictive-battery-alert-design.md @@ -0,0 +1,53 @@ +# Predictive low-battery notification — design + +**Date:** 2026-07-10 +**Scope:** macOS + iOS shared layer (`Strand/System/BatteryNotifier.swift`, `Strand/App/AppModel.swift`) +plus a pure policy in `Packages/StrandAnalytics`. Android twin deferred until this is upstreamed — +`BatteryAlertPolicy` is documented behavior-identical with the Kotlin twin (#368), so an upstream PR +must carry the Kotlin `runtimeAlert` policy + tests to honor the parity contract. + +## Problem + +The existing low-battery alert (#368) fires at a fixed 15% state of charge. Runtime at 15% differs +by strap generation — ~16 h on a WHOOP 4.0 but ~1.8 days on a 5.0/MG — so the warning's lead time is +an accident of hardware. Meanwhile `BatteryEstimator` (#713) already produces a personalized +"~X left" runtime estimate from the strap's banked discharge history. The two are not connected. + +## Behavior + +- On every battery reading (`LiveState.onBatteryUpdate` → `AppModel` hook), evaluate the current + `LiveState.batteryEstimate` against a new pure policy: + - **Fire** when `remainingHours <= 24`, the strap is not confirmed charging (`charging != true`, + matching the low alert's null-tolerant rule), and the runtime gate is armed. + - **Re-arm** only when `remainingHours >= 36` — estimate jitter around the 24 h line cannot + re-fire; only a genuine charge recovers 12+ hours of headroom. + - Both estimate sources fire (`measured` and `rated`): the estimator's own gates (`minSpanHours`, + `minDropPct`, near-full anchoring) already reject junk fits, and a rated-fallback estimate is + still generation-correct. +- Notification: id `battery-runtime`, title "Strap battery low", body built from + `BatteryEstimator.label(hours:)` — e.g. "~22h left on your WHOOP — recharge tonight." +- The `runtimeAlerted` gate is persisted (`behavior.batteryRuntimeAlerted` in UserDefaults), like + `lowAlerted`/`fullAlerted`, so the once-per-discharge-cycle guarantee survives process death, and + it advances even when delivery is deferred (authorization declined), mirroring #368. +- **Unchanged:** the 15% low alert (safety net — covers the nil-estimate case), the 100% full note, + the existing "Battery alerts" setting gating everything (no new setting), and + `BatteryNotifier.requestAuthorization()`. +- A user near the boundary may receive the runtime alert (≤24 h) and later the 15% alert in the same + discharge — accepted: they carry different messages (plan vs act), and deduplicating them would + couple the two gates. + +## Where the code goes + +- `BatteryEstimator.runtimeAlert(remainingHours:charging:alerted:)` in `StrandAnalytics` — pure + `(fire, newAlerted)` decision with the 24/36 constants (`runtimeAlertHours`, `runtimeRearmHours`). + `swift test` covered: fire, jitter no-refire, re-arm on recovery, charging suppression (`nil` + charging still fires), nil-estimate no-op at the caller. +- `BatteryNotifier.onRuntimeEstimate(remainingHours:charging:enabled:)` — reads/writes the persisted + flag, posts via the existing `post` helper. +- `AppModel`'s existing `live.onBatteryUpdate` closure additionally calls `onRuntimeEstimate` with + `live.batteryEstimate?.remainingHours`. + +## Testing + +Policy: `swift test` in StrandAnalytics. Notifier/UI glue: compile both app schemes (no default CI +covers app targets); on-hardware check is watching the alert land when the strap next runs down. From 5ea90660a30bd1917ed441cd112fe9bd2c62c030 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 20:17:24 -0700 Subject: [PATCH 036/171] fix(oura): window heartrate backfill into 30-day ISO-Z chunks (endpoint range cap) Live import skipped heartrate: the endpoint rejects spans >30 days with HTTP 400 (probe-verified), and an unencoded +00:00 offset 422s ('+' decodes to a space). Fetch in 30-day windows of yyyy-MM-dd'T'HH:mm:ss'Z' datetimes floored at the earliest day the dailies returned; window-scoped raw keys keep the archive idempotent; boundary dupes dedupe on hrSample (deviceId, ts). --- Strand/Oura/OuraSyncCoordinator.swift | 49 ++++++++++++++++++---- StrandTests/OuraSyncCoordinatorTests.swift | 11 ++++- 2 files changed, 51 insertions(+), 9 deletions(-) diff --git a/Strand/Oura/OuraSyncCoordinator.swift b/Strand/Oura/OuraSyncCoordinator.swift index 9bd7b0e46c..7276a5d1f9 100644 --- a/Strand/Oura/OuraSyncCoordinator.swift +++ b/Strand/Oura/OuraSyncCoordinator.swift @@ -30,13 +30,14 @@ final class OuraSyncCoordinator { var result = OuraSyncResult() var byDay: [String: WearableDailyRow] = [:] - func fetchRaw(_ endpoint: String, dateParam: String?) async throws -> [Data] { - var q: [String: String] = [:] - if let dateParam { q[dateParam] = startDate; q[dateParam == "start_datetime" ? "end_datetime" : "end_date"] = today } - let pages = try await fetcher.fetchAllRaw(endpoint: endpoint, query: q) + func fetchRawQuery(_ endpoint: String, query: [String: String]) async throws -> [Data] { + let pages = try await fetcher.fetchAllRaw(endpoint: endpoint, query: query) + // Window-scoped raw key so multi-window endpoints (heartrate) never collide, and a re-pull + // of the same window stays idempotent (same key → upsert overwrite). + let windowKey = query["start_datetime"] ?? query["start_date"] ?? startDate var idx = 0 for body in pages { - result.rawPages.append(OuraRawRow(endpoint: endpoint, documentId: "\(endpoint)-\(startDate)-\(idx)", + result.rawPages.append(OuraRawRow(endpoint: endpoint, documentId: "\(endpoint)-\(windowKey)-\(idx)", day: nil, payloadJSON: String(data: body, encoding: .utf8) ?? "", fetchedAt: Int(Date().timeIntervalSince1970))) idx += 1 @@ -44,6 +45,11 @@ final class OuraSyncCoordinator { onProgress(OuraSyncProgress(endpoint: endpoint, pages: pages.count)) return pages } + func fetchRaw(_ endpoint: String, dateParam: String?) async throws -> [Data] { + var q: [String: String] = [:] + if let dateParam { q[dateParam] = startDate; q[dateParam == "start_datetime" ? "end_datetime" : "end_date"] = today } + return try await fetchRawQuery(endpoint, query: q) + } func docs(_ pages: [Data]) -> [[String: Any]] { pages.flatMap { (try? JSONSerialization.jsonObject(with: $0) as? [String: Any])?["data"] as? [[String: Any]] ?? [] } @@ -86,9 +92,26 @@ final class OuraSyncCoordinator { let workoutPages = try await fetchRaw("workout", dateParam: "start_date") result.workouts = OuraApiParser.parseWorkouts(docs(workoutPages)) } catch { skipped.append("workout") } + // Heart-rate is the one endpoint with a hard ≤30-day range cap (probe-verified: a wider span is + // HTTP 400 "Timerange ... has to be less than or equal to 30 days"). Chunk it into 30-day windows + // of full ISO 'Z' datetimes (an unencoded '+00:00' offset 422s — '+' decodes to a space in the + // query). The floor is the earliest day the dailies actually returned, so we never page a decade + // of empty pre-account windows; boundary-duplicate samples dedupe on hrSample's (deviceId, ts) PK. do { - let hrPages = try await fetchRaw("heartrate", dateParam: "start_datetime") - result.heartRate = OuraApiParser.parseHeartRate(docs(hrPages)) + let floorDay = byDay.keys.min() ?? startDate + var cursor = Self.dayFmt.date(from: floorDay) ?? Date(timeIntervalSince1970: 0) + let endDate = (Self.dayFmt.date(from: today) ?? Date()).addingTimeInterval(86_400) + var hrDocs: [[String: Any]] = [] + while cursor < endDate { + let next = min(cursor.addingTimeInterval(30 * 86_400), endDate) + let pages = try await fetchRawQuery("heartrate", query: [ + "start_datetime": Self.isoFmt.string(from: cursor), + "end_datetime": Self.isoFmt.string(from: next), + ]) + hrDocs.append(contentsOf: docs(pages)) + cursor = next + } + result.heartRate = OuraApiParser.parseHeartRate(hrDocs) } catch { skipped.append("heartrate") } // Raw-only endpoints (no parser). for endpoint in Self.rawOnlyEndpoints { @@ -102,4 +125,16 @@ final class OuraSyncCoordinator { summary.skippedEndpoints = skipped return summary } + + // UTC day / ISO-'Z' datetime formatters for the windowed heartrate fetch. + private static let dayFmt: DateFormatter = { + let f = DateFormatter(); f.calendar = Calendar(identifier: .gregorian) + f.locale = Locale(identifier: "en_US_POSIX"); f.timeZone = TimeZone(identifier: "UTC") + f.dateFormat = "yyyy-MM-dd"; return f + }() + private static let isoFmt: DateFormatter = { + let f = DateFormatter(); f.calendar = Calendar(identifier: .gregorian) + f.locale = Locale(identifier: "en_US_POSIX"); f.timeZone = TimeZone(identifier: "UTC") + f.dateFormat = "yyyy-MM-dd'T'HH:mm:ss'Z'"; return f + }() } diff --git a/StrandTests/OuraSyncCoordinatorTests.swift b/StrandTests/OuraSyncCoordinatorTests.swift index 9036cd2d9d..6beef37da8 100644 --- a/StrandTests/OuraSyncCoordinatorTests.swift +++ b/StrandTests/OuraSyncCoordinatorTests.swift @@ -7,9 +7,11 @@ import WhoopStore private final class StubFetcher: OuraPageFetching { var pages: [String: [Data]] = [:] var capturedQueries: [String: [String: String]] = [:] + var callCounts: [String: Int] = [:] var failEndpoints: Set = [] func fetchAllRaw(endpoint: String, query: [String: String]) async throws -> [Data] { capturedQueries[endpoint] = query + callCounts[endpoint, default: 0] += 1 if failEndpoints.contains(endpoint) { throw OuraError.badResponse(401, "token is not authorized") } return pages[endpoint] ?? [] } @@ -42,11 +44,16 @@ final class OuraSyncCoordinatorTests: XCTestCase { func testEndpointDateParamsAreWiredCorrectly() async throws { let store = try await WhoopStore.inMemory() let fetcher = StubFetcher() + // A daily page sets the heartrate window floor (2026-01-02); today 2026-02-01 → exactly two + // 30-day windows. Heartrate has a hard ≤30-day range cap upstream, so windowing is load-bearing. + fetcher.pages["daily_readiness"] = [#"{"data":[{"day":"2026-01-02","score":80}],"next_token":null}"#.data(using: .utf8)!] let coord = OuraSyncCoordinator(fetcher: fetcher, store: store) _ = try await coord.runFullImport(today: "2026-02-01") - XCTAssertNotNil(fetcher.capturedQueries["heartrate"]?["start_datetime"]) - XCTAssertNotNil(fetcher.capturedQueries["heartrate"]?["end_datetime"]) + // Windowed, full ISO-'Z' datetimes (bare dates parse upstream but a +00:00 offset 422s), ≤30d each. + XCTAssertEqual(fetcher.callCounts["heartrate"], 2) + XCTAssertEqual(fetcher.capturedQueries["heartrate"]?["start_datetime"], "2026-02-01T00:00:00Z") + XCTAssertEqual(fetcher.capturedQueries["heartrate"]?["end_datetime"], "2026-02-02T00:00:00Z") XCTAssertNil(fetcher.capturedQueries["heartrate"]?["start_date"]) XCTAssertEqual(fetcher.capturedQueries["personal_info"], [:]) From 7850db1bf9d2f3df2a1a20e8592ddb23b9b60c45 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 20:24:10 -0700 Subject: [PATCH 037/171] =?UTF-8?q?feat(oura):=20import=20resilience=20?= =?UTF-8?q?=E2=80=94=20keep=20screen=20awake,=20windowed=20HR=20progress,?= =?UTF-8?q?=20honest=20save=20phase?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An auto-lock mid-backfill suspends the foreground URLSession flow and freezes the run with no error, and the long heartrate+persist stretch showed a static status. busy now pins isIdleTimerDisabled on iOS; heartrate reports 'window i/N'; the write phase reports 'Saving to your database…'. --- Strand/Oura/OuraConnectModel.swift | 23 +++++++++++++++++++---- Strand/Oura/OuraSyncCoordinator.swift | 14 +++++++++++++- 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/Strand/Oura/OuraConnectModel.swift b/Strand/Oura/OuraConnectModel.swift index 911d5aad19..50fc738fe6 100644 --- a/Strand/Oura/OuraConnectModel.swift +++ b/Strand/Oura/OuraConnectModel.swift @@ -2,6 +2,9 @@ import Foundation import Combine import AuthenticationServices import WhoopStore +#if os(iOS) +import UIKit +#endif /// Drives the Data Sources "Connect Oura" card: connect (OAuth) → one-time backfill → honest summary → /// disconnect. @MainActor so all @Published mutations are main-thread; the network/backfill hops off-main. @@ -26,7 +29,7 @@ final class OuraConnectModel: ObservableObject { guard let creds = OuraCredentials.fromBundle else { statusText = "Add your Oura app's client_id/secret to OuraSecrets.xcconfig first."; return } - busy = true; statusText = "Connecting to Oura…" + setBusy(true); statusText = "Connecting to Oura…" Task { do { let provider = OuraOAuthProvider(credentials: creds) @@ -38,7 +41,10 @@ final class OuraConnectModel: ObservableObject { let today = Self.dayFormatter.string(from: Date()) statusText = "Importing your Oura history…" let s = try await coord.runFullImport(today: today) { [weak self] p in - Task { @MainActor in self?.statusText = "Importing \(p.endpoint)…" } + Task { @MainActor in + if p.endpoint == "saving" { self?.statusText = "Saving to your database…" } + else { self?.statusText = "Importing \(p.endpoint)\(p.detail.map { " (\($0))" } ?? "")…" } + } } await repo.refresh() var line = "Imported \(s.days) days · \(s.sleeps) sleeps · \(s.workouts) workouts · \(s.hrSamples) HR samples" @@ -47,7 +53,7 @@ final class OuraConnectModel: ObservableObject { } statusText = line } catch { fail((error as? LocalizedError)?.errorDescription ?? error.localizedDescription) } - busy = false + setBusy(false) } } @@ -78,7 +84,16 @@ final class OuraConnectModel: ObservableObject { } } - private func fail(_ m: String) { statusText = m; busy = false } + private func fail(_ m: String) { statusText = m; setBusy(false) } + + /// Busy also pins the screen awake on iOS: the backfill is a foreground URLSession flow, and an + /// auto-lock mid-import suspends the app and freezes the run with no error. Always reset on exit. + private func setBusy(_ b: Bool) { + busy = b + #if os(iOS) + UIApplication.shared.isIdleTimerDisabled = b + #endif + } private static let dayFormatter: DateFormatter = { let f = DateFormatter(); f.calendar = Calendar(identifier: .gregorian) f.locale = Locale(identifier: "en_US_POSIX"); f.timeZone = TimeZone(identifier: "UTC") diff --git a/Strand/Oura/OuraSyncCoordinator.swift b/Strand/Oura/OuraSyncCoordinator.swift index 7276a5d1f9..11a8bf1218 100644 --- a/Strand/Oura/OuraSyncCoordinator.swift +++ b/Strand/Oura/OuraSyncCoordinator.swift @@ -8,7 +8,12 @@ protocol OuraPageFetching { } extension OuraAPIClient: OuraPageFetching {} -struct OuraSyncProgress: Equatable { let endpoint: String; let pages: Int } +struct OuraSyncProgress: Equatable { + let endpoint: String + let pages: Int + /// Optional human detail (e.g. "window 4/17" for the chunked heartrate fetch). + var detail: String? = nil +} /// One-time backfill across every Oura v2 endpoint → an assembled OuraSyncResult → OuraSyncWriter. Merges /// the per-day WearableDailyRows across endpoints (daily endpoints BEFORE sleep, so readiness RHR wins). @@ -101,8 +106,12 @@ final class OuraSyncCoordinator { let floorDay = byDay.keys.min() ?? startDate var cursor = Self.dayFmt.date(from: floorDay) ?? Date(timeIntervalSince1970: 0) let endDate = (Self.dayFmt.date(from: today) ?? Date()).addingTimeInterval(86_400) + let totalWindows = max(1, Int(ceil(endDate.timeIntervalSince(cursor) / (30 * 86_400)))) + var window = 0 var hrDocs: [[String: Any]] = [] while cursor < endDate { + window += 1 + onProgress(OuraSyncProgress(endpoint: "heartrate", pages: 0, detail: "window \(window)/\(totalWindows)")) let next = min(cursor.addingTimeInterval(30 * 86_400), endDate) let pages = try await fetchRawQuery("heartrate", query: [ "start_datetime": Self.isoFmt.string(from: cursor), @@ -121,6 +130,9 @@ final class OuraSyncCoordinator { } result.days = Array(byDay.values) + // The write phase can take a while on a first full import — surface it honestly instead of + // leaving the last endpoint's "Importing …" text on screen. + onProgress(OuraSyncProgress(endpoint: "saving", pages: 0)) var summary = try await OuraSyncWriter.persist(result, into: store) summary.skippedEndpoints = skipped return summary From c3c9286d3240dde182ccdcd1d7afc205ff69914b Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 21:20:39 -0700 Subject: [PATCH 038/171] =?UTF-8?q?fix(oura):=20persist=20heart-rate=20per?= =?UTF-8?q?=20window=20=E2=80=94=20interrupted=20imports=20keep=20complete?= =?UTF-8?q?d=20progress?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The whole-day HR series was assembled in memory across all ~29 windows and written in a single end-of-run persist. A screen-lock/suspend before that persist lost the ENTIRE series (verified on-device: 0 hrSample rows after a run that reached window 25/29). Now the light data (days/sleeps/workouts) persists first as a durable checkpoint, then each HR window is written to hrSample + ouraRaw as it arrives. An interruption keeps every completed window; a re-run re-fetches from the floor and idempotently overwrites on the (deviceId, ts) / (endpoint, documentId) keys. New test drives a mid-backfill fetch failure and asserts window 1's HR is already in the store — durability through interruption, the exact bug that bit the live import. --- Strand/Oura/OuraSyncCoordinator.swift | 61 +++++++++++++--------- StrandTests/OuraSyncCoordinatorTests.swift | 25 +++++++++ 2 files changed, 62 insertions(+), 24 deletions(-) diff --git a/Strand/Oura/OuraSyncCoordinator.swift b/Strand/Oura/OuraSyncCoordinator.swift index 11a8bf1218..04682a64a1 100644 --- a/Strand/Oura/OuraSyncCoordinator.swift +++ b/Strand/Oura/OuraSyncCoordinator.swift @@ -1,5 +1,6 @@ import Foundation import WhoopStore +import WhoopProtocol import StrandImport /// Abstracts the page fetch so the coordinator is testable without a live network (OuraAPIClient conforms). @@ -32,6 +33,7 @@ final class OuraSyncCoordinator { func runFullImport(startDate: String = "2013-01-01", today: String, onProgress: @escaping (OuraSyncProgress) -> Void = { _ in }) async throws -> OuraSyncSummary { + let deviceId = "oura-api" var result = OuraSyncResult() var byDay: [String: WearableDailyRow] = [:] @@ -92,48 +94,59 @@ final class OuraSyncCoordinator { let (periods, sleepDays) = OuraApiParser.parseSleep(docs(sleepPages)) result.sleepPeriods = periods; merge(sleepDays) } catch { skipped.append("sleep") } - // Workouts + heart-rate. + // Workouts. do { let workoutPages = try await fetchRaw("workout", dateParam: "start_date") result.workouts = OuraApiParser.parseWorkouts(docs(workoutPages)) } catch { skipped.append("workout") } - // Heart-rate is the one endpoint with a hard ≤30-day range cap (probe-verified: a wider span is - // HTTP 400 "Timerange ... has to be less than or equal to 30 days"). Chunk it into 30-day windows - // of full ISO 'Z' datetimes (an unencoded '+00:00' offset 422s — '+' decodes to a space in the - // query). The floor is the earliest day the dailies actually returned, so we never page a decade - // of empty pre-account windows; boundary-duplicate samples dedupe on hrSample's (deviceId, ts) PK. + // Raw-only endpoints (no parser). + for endpoint in Self.rawOnlyEndpoints { + do { + _ = try await fetchRaw(endpoint, dateParam: ["personal_info", "ring_configuration"].contains(endpoint) ? nil : "start_date") + } catch { skipped.append(endpoint) } + } + + // Persist the light data (days/sleeps/workouts/extras/raw + source registration) FIRST, as a + // durable checkpoint. It's small and fetched one call apiece; landing it in SQLite before the long + // heart-rate backfill means an interruption during HR can never cost it. + result.days = Array(byDay.values) + onProgress(OuraSyncProgress(endpoint: "saving", pages: 0)) + var summary = try await OuraSyncWriter.persist(result, into: store, deviceId: deviceId) + + // Whole-day heart-rate: the ONE endpoint with a hard ≤30-day range cap (probe-verified: a wider + // span is HTTP 400 "Timerange ... has to be less than or equal to 30 days"; an unencoded '+00:00' + // offset 422s since '+' decodes to a space). Fetched in 30-day windows of full ISO-'Z' datetimes, + // floored at the earliest day the dailies returned (never page years of empty pre-account time). + // CRITICAL: each window is PERSISTED before advancing, so a lock or interruption keeps every + // completed window instead of losing the whole series (the prior one-shot end-persist did exactly + // that). Re-runs re-fetch from the floor; hrSample dedupes on (deviceId, ts) and ouraRaw on + // (endpoint, documentId), so replaying an already-saved window is a harmless overwrite. do { let floorDay = byDay.keys.min() ?? startDate var cursor = Self.dayFmt.date(from: floorDay) ?? Date(timeIntervalSince1970: 0) let endDate = (Self.dayFmt.date(from: today) ?? Date()).addingTimeInterval(86_400) let totalWindows = max(1, Int(ceil(endDate.timeIntervalSince(cursor) / (30 * 86_400)))) + let fetchedAt = Int(Date().timeIntervalSince1970) var window = 0 - var hrDocs: [[String: Any]] = [] while cursor < endDate { window += 1 onProgress(OuraSyncProgress(endpoint: "heartrate", pages: 0, detail: "window \(window)/\(totalWindows)")) let next = min(cursor.addingTimeInterval(30 * 86_400), endDate) - let pages = try await fetchRawQuery("heartrate", query: [ - "start_datetime": Self.isoFmt.string(from: cursor), - "end_datetime": Self.isoFmt.string(from: next), - ]) - hrDocs.append(contentsOf: docs(pages)) + let startISO = Self.isoFmt.string(from: cursor) + let pages = try await fetcher.fetchAllRaw(endpoint: "heartrate", + query: ["start_datetime": startISO, "end_datetime": Self.isoFmt.string(from: next)]) + // Persist THIS window before fetching the next — the durability guarantee. + let rawRows = pages.enumerated().map { idx, body in + OuraRawRow(endpoint: "heartrate", documentId: "heartrate-\(startISO)-\(idx)", day: nil, + payloadJSON: String(data: body, encoding: .utf8) ?? "", fetchedAt: fetchedAt) + } + if !rawRows.isEmpty { summary.rawPages += try await store.upsertOuraRaw(rawRows, deviceId: deviceId) } + let hr = OuraApiParser.parseHeartRate(docs(pages)).map { HRSample(ts: $0.ts, bpm: $0.bpm) } + if !hr.isEmpty { summary.hrSamples += try await store.insert(Streams(hr: hr), deviceId: deviceId).hr } cursor = next } - result.heartRate = OuraApiParser.parseHeartRate(hrDocs) } catch { skipped.append("heartrate") } - // Raw-only endpoints (no parser). - for endpoint in Self.rawOnlyEndpoints { - do { - _ = try await fetchRaw(endpoint, dateParam: ["personal_info", "ring_configuration"].contains(endpoint) ? nil : "start_date") - } catch { skipped.append(endpoint) } - } - result.days = Array(byDay.values) - // The write phase can take a while on a first full import — surface it honestly instead of - // leaving the last endpoint's "Importing …" text on screen. - onProgress(OuraSyncProgress(endpoint: "saving", pages: 0)) - var summary = try await OuraSyncWriter.persist(result, into: store) summary.skippedEndpoints = skipped return summary } diff --git a/StrandTests/OuraSyncCoordinatorTests.swift b/StrandTests/OuraSyncCoordinatorTests.swift index 6beef37da8..d948e4dbaf 100644 --- a/StrandTests/OuraSyncCoordinatorTests.swift +++ b/StrandTests/OuraSyncCoordinatorTests.swift @@ -9,10 +9,14 @@ private final class StubFetcher: OuraPageFetching { var capturedQueries: [String: [String: String]] = [:] var callCounts: [String: Int] = [:] var failEndpoints: Set = [] + var failAfterCall: [String: Int] = [:] // endpoint → allow this many calls, then throw (mimics a mid-run suspend) func fetchAllRaw(endpoint: String, query: [String: String]) async throws -> [Data] { capturedQueries[endpoint] = query callCounts[endpoint, default: 0] += 1 if failEndpoints.contains(endpoint) { throw OuraError.badResponse(401, "token is not authorized") } + if let limit = failAfterCall[endpoint], (callCounts[endpoint] ?? 0) > limit { + throw OuraError.badResponse(500, "simulated interruption") + } return pages[endpoint] ?? [] } } @@ -63,6 +67,27 @@ final class OuraSyncCoordinatorTests: XCTestCase { XCTAssertNotNil(fetcher.capturedQueries["daily_readiness"]?["end_date"]) } + func testHeartRateWindowPersistsBeforeAnInterruptionSoProgressSurvives() async throws { + let store = try await WhoopStore.inMemory() + let fetcher = StubFetcher() + // A daily day in early Jan floors the HR windows; today mid-March → several 30-day windows. + fetcher.pages["daily_readiness"] = [#"{"data":[{"day":"2026-01-02","score":80}],"next_token":null}"#.data(using: .utf8)!] + // Each heartrate window returns one sample, but the fetcher dies AFTER the first window — + // exactly the screen-lock/suspend failure that used to lose the entire series. + fetcher.pages["heartrate"] = [#"{"data":[{"bpm":58,"timestamp":"2026-01-05T12:00:00.000Z"}],"next_token":null}"#.data(using: .utf8)!] + fetcher.failAfterCall["heartrate"] = 1 + + let coord = OuraSyncCoordinator(fetcher: fetcher, store: store) + let summary = try await coord.runFullImport(today: "2026-03-15") + + XCTAssertTrue(summary.skippedEndpoints.contains("heartrate")) // interruption reported honestly + XCTAssertEqual(summary.days, 1) // light data landed before HR began + XCTAssertGreaterThan(summary.hrSamples, 0) // window 1's HR was durably saved BEFORE the failure + // Ground truth in the store, not just the summary: the first window's sample is really persisted. + let hr = try await store.hrSamples(deviceId: "oura-api", from: 0, to: Int.max, limit: 1000) + XCTAssertFalse(hr.isEmpty) + } + func testFailingEndpointIsSkippedNotFatal() async throws { let store = try await WhoopStore.inMemory() let fetcher = StubFetcher() From 85993fe32435f256567ac867c44bed0280cd6fd8 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 22:19:31 -0700 Subject: [PATCH 039/171] docs: noop-cloud Phase-1 MVP implementation plan (12 tasks, TDD) Server-only Phase 1: /ingest (.noopbak validate + atomic mirror swap), stateless Streamable-HTTP /mcp, read tools (health_snapshot, metric_series, sleep/workout summary, data_freshness, compare_sources) + ChatGPT search/fetch + 3 prompts, two static ro/rw bearer tokens, Fly single-machine+volume+WAL+autostop, morning report via GitHub Actions cron -> Anthropic MCP connector -> ntfy. Defers OAuth/writes/ sync-back. Sibling repo noop-cloud/ (own git), MCP SDK pinned 1.29.0. --- .../2026-07-10-noop-cloud-1-mvp-server.md | 1720 +++++++++++++++++ 1 file changed, 1720 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md diff --git a/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md b/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md new file mode 100644 index 0000000000..00db1ddcf1 --- /dev/null +++ b/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md @@ -0,0 +1,1720 @@ +# noop-cloud Phase-1 MVP Server — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A self-hostable remote MCP server that mirrors the NOOP phone database (uploaded as a `.noopbak`) and exposes read-only biometrics tools to Claude/ChatGPT/agents over Streamable HTTP, deployed on a single Fly.io machine, with a scheduled morning report. + +**Architecture:** One Node/TypeScript process behind Fly's HTTPS proxy. `POST /ingest` (rw bearer) accepts a `.noopbak` ZIP, validates it, and atomically swaps it into `/data/mirror.sqlite`. `POST /mcp` (ro bearer) is a **stateless** Streamable HTTP MCP endpoint whose tools open `mirror.sqlite` read-only. `/data/server.sqlite` holds only an ingest log in Phase 1. A GitHub Actions cron calls the Anthropic Messages API (MCP connector, ro token) and pushes the report to ntfy. No writes, no OAuth, no sync-back in this phase. + +**Tech Stack:** TypeScript 5.6, Node 20, Express 4, `@modelcontextprotocol/sdk` **pinned 1.29.0**, `better-sqlite3` (native, pinned), `adm-zip`, `zod`, `vitest`. Fly.io (1 machine + 1 volume, WAL SQLite, autostop). Docker deploy. + +## Global Constraints + +- **MCP SDK pinned to exactly `1.29.0`** (`"@modelcontextprotocol/sdk": "1.29.0"`, no `^`). Do NOT use the 2.0.0-beta line. Rationale: the 2026-07-28 stateless spec revision is still an RC; 1.29.x is the supported stable line. +- **`better-sqlite3` pinned exact** (`"better-sqlite3": "11.10.0"`, no `^`) — native binding; a floating range risks an ABI mismatch on the Fly builder. `npm rebuild better-sqlite3` is the fix if the prebuilt binding mismatches Node 20. +- **Stateless MCP transport**: every `/mcp` request constructs a fresh `McpServer` + `StreamableHTTPServerTransport({ sessionIdGenerator: undefined })`. No session store — a Fly autostop between requests must not break a session. +- **The mirror is opened READ-ONLY always** (`new Database(path, { readonly: true, fileMustExist: true })`). Server code never writes to `mirror.sqlite`. Server-owned state lives only in `server.sqlite`. +- **`deviceId` is a *source* discriminator, not a physical device.** The same calendar day legitimately has rows under multiple `deviceId`s. Source families for Phase 1: WHOOP = any id that is NOT `apple-health` and NOT `oura-api` (straps use ids like `my-whoop`, plus derived `-noop`); Oura = `oura-api`; Apple = `apple-health`. Never assume one row per day. +- **Two static bearer tokens** from env: `RO_TOKEN` (read; `/mcp`) and `RW_TOKEN` (ingest; `/ingest`). Compare with `crypto.timingSafeEqual`. `rw` implies `ro`. Tokens are ≥32 bytes; never logged. +- **Ingest hard limit**: reject bodies over `MAX_INGEST_BYTES` (default 250 MB). Validate the inner SQLite by magic bytes (`SQLite format 3\0`) + `PRAGMA quick_check`. Reject a foreign DB (no `grdb_migrations` table). +- **No secrets in git.** `.env` is gitignored; `.env.example` documents keys. Fly secrets via `fly secrets set`. +- **License/voice:** new repo, MIT license, neutral third-person README. This is a standalone sibling repo at `/Users/vk/VKDEV/NOOP/noop-cloud` — NOT inside the NOOP tree, its own git. + +--- + +### Task 1: Repo scaffold, config, and `/healthz` + +**Files:** +- Create: `noop-cloud/package.json` +- Create: `noop-cloud/tsconfig.json` +- Create: `noop-cloud/vitest.config.ts` +- Create: `noop-cloud/.gitignore` +- Create: `noop-cloud/.env.example` +- Create: `noop-cloud/src/config.ts` +- Create: `noop-cloud/src/server.ts` +- Test: `noop-cloud/test/healthz.test.ts` + +**Interfaces:** +- Produces: `loadConfig(): Config` where `Config = { port: number; dataDir: string; mirrorPath: string; serverDbPath: string; roToken: string; rwToken: string; maxIngestBytes: number }`. +- Produces: `createApp(cfg: Config): express.Express` — the Express app (routes added in later tasks), with `GET /healthz` returning `{ ok: true }`. + +- [ ] **Step 1: Initialize the repo and pinned dependencies** + +```bash +mkdir -p /Users/vk/VKDEV/NOOP/noop-cloud && cd /Users/vk/VKDEV/NOOP/noop-cloud +git init -q +``` + +Create `package.json`: + +```json +{ + "name": "noop-cloud", + "version": "0.1.0", + "private": true, + "type": "module", + "license": "MIT", + "engines": { "node": ">=20" }, + "scripts": { + "build": "tsc -p tsconfig.json", + "start": "node dist/server.js", + "dev": "tsx watch src/server.ts", + "test": "vitest run", + "test:watch": "vitest" + }, + "dependencies": { + "@modelcontextprotocol/sdk": "1.29.0", + "adm-zip": "0.5.16", + "better-sqlite3": "11.10.0", + "express": "^4.21.2", + "zod": "^3.23.8" + }, + "devDependencies": { + "@types/adm-zip": "^0.5.5", + "@types/better-sqlite3": "^7.6.11", + "@types/express": "^4.17.21", + "@types/node": "^20.14.0", + "tsx": "^4.19.0", + "typescript": "^5.6.0", + "vitest": "^2.1.0" + } +} +``` + +- [ ] **Step 2: Add tsconfig, vitest config, gitignore, env template** + +`tsconfig.json`: + +```json +{ + "compilerOptions": { + "target": "ES2022", + "module": "NodeNext", + "moduleResolution": "NodeNext", + "outDir": "dist", + "rootDir": "src", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "resolveJsonModule": true, + "declaration": false, + "sourceMap": true + }, + "include": ["src"] +} +``` + +`vitest.config.ts`: + +```typescript +import { defineConfig } from "vitest/config"; +export default defineConfig({ test: { include: ["test/**/*.test.ts"], testTimeout: 20000 } }); +``` + +`.gitignore`: + +``` +node_modules/ +dist/ +.env +data/ +*.sqlite +*.sqlite-wal +*.sqlite-shm +test/.tmp/ +``` + +`.env.example`: + +``` +PORT=8080 +DATA_DIR=./data +RO_TOKEN=replace-with-32+-byte-random-read-token +RW_TOKEN=replace-with-32+-byte-random-write-token +MAX_INGEST_BYTES=262144000 +``` + +- [ ] **Step 3: Write the failing test for config + `/healthz`** + +`test/healthz.test.ts`: + +```typescript +import { describe, it, expect } from "vitest"; +import request from "node:http"; +import { loadConfig } from "../src/config.js"; +import { createApp } from "../src/server.js"; + +describe("scaffold", () => { + it("loadConfig throws when RO_TOKEN missing", () => { + const prev = process.env.RO_TOKEN; + delete process.env.RO_TOKEN; + expect(() => loadConfig()).toThrow(/RO_TOKEN/); + if (prev !== undefined) process.env.RO_TOKEN = prev; + }); + + it("GET /healthz returns ok", async () => { + process.env.RO_TOKEN = "ro-token-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"; + process.env.RW_TOKEN = "rw-token-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"; + const app = createApp(loadConfig()); + const server = app.listen(0); + const port = (server.address() as any).port; + const body = await new Promise((resolve, reject) => { + request.get(`http://127.0.0.1:${port}/healthz`, (res) => { + let d = ""; res.on("data", (c) => (d += c)); res.on("end", () => resolve(d)); + }).on("error", reject); + }); + server.close(); + expect(JSON.parse(body)).toEqual({ ok: true }); + }); +}); +``` + +- [ ] **Step 4: Run the test to verify it fails** + +Run: `npm install && npm test` +Expected: FAIL — `Cannot find module '../src/config.js'`. + +- [ ] **Step 5: Implement `src/config.ts`** + +```typescript +import path from "node:path"; + +export interface Config { + port: number; + dataDir: string; + mirrorPath: string; + serverDbPath: string; + roToken: string; + rwToken: string; + maxIngestBytes: number; +} + +function required(name: string): string { + const v = process.env[name]; + if (!v || v.length < 16) throw new Error(`${name} must be set (>=16 chars)`); + return v; +} + +export function loadConfig(): Config { + const dataDir = process.env.DATA_DIR ?? "./data"; + return { + port: Number(process.env.PORT ?? 8080), + dataDir, + mirrorPath: path.join(dataDir, "mirror.sqlite"), + serverDbPath: path.join(dataDir, "server.sqlite"), + roToken: required("RO_TOKEN"), + rwToken: required("RW_TOKEN"), + maxIngestBytes: Number(process.env.MAX_INGEST_BYTES ?? 262_144_000), + }; +} +``` + +- [ ] **Step 6: Implement `src/server.ts`** + +```typescript +import express from "express"; +import fs from "node:fs"; +import { Config, loadConfig } from "./config.js"; + +export function createApp(cfg: Config): express.Express { + fs.mkdirSync(cfg.dataDir, { recursive: true }); + const app = express(); + app.get("/healthz", (_req, res) => res.json({ ok: true })); + return app; +} + +// Entry point (ignored by tests, which import createApp directly). +if (process.argv[1] && process.argv[1].endsWith("server.js")) { + const cfg = loadConfig(); + createApp(cfg).listen(cfg.port, () => console.log(`noop-cloud on :${cfg.port}`)); +} +``` + +- [ ] **Step 7: Run tests + build to verify pass** + +Run: `npm test && npm run build` +Expected: PASS (2 tests); `dist/` compiles with no errors. + +- [ ] **Step 8: Commit** + +```bash +git add -A && git commit -m "chore: scaffold noop-cloud (config + /healthz + pinned deps)" +``` + +--- + +### Task 2: Bearer auth middleware (ro/rw scopes) + +**Files:** +- Create: `noop-cloud/src/auth.ts` +- Test: `noop-cloud/test/auth.test.ts` + +**Interfaces:** +- Consumes: `Config` (`roToken`, `rwToken`) from Task 1. +- Produces: `requireScope(cfg: Config, scope: "ro" | "rw"): express.RequestHandler`. `rw` token satisfies both scopes; `ro` token satisfies only `ro`. Missing/malformed/incorrect → 401 JSON `{ error: "unauthorized" }`. Uses `crypto.timingSafeEqual`. + +- [ ] **Step 1: Write the failing test** + +`test/auth.test.ts`: + +```typescript +import { describe, it, expect } from "vitest"; +import express from "express"; +import http from "node:http"; +import { requireScope } from "../src/auth.js"; + +const cfg = { roToken: "ro".padEnd(40, "x"), rwToken: "rw".padEnd(40, "y") } as any; + +function call(handlerScope: "ro" | "rw", header?: string): Promise { + const app = express(); + app.get("/p", requireScope(cfg, handlerScope), (_r, res) => res.json({ ok: true })); + const server = app.listen(0); + const port = (server.address() as any).port; + return new Promise((resolve) => { + http.get({ port, path: "/p", headers: header ? { authorization: header } : {} }, (res) => { + server.close(); resolve(res.statusCode!); + }); + }); +} + +describe("requireScope", () => { + it("401 with no header", async () => expect(await call("ro")).toBe(401)); + it("401 with wrong token", async () => expect(await call("ro", "Bearer nope")).toBe(401)); + it("ro token satisfies ro scope", async () => expect(await call("ro", `Bearer ${cfg.roToken}`)).toBe(200)); + it("rw token satisfies ro scope", async () => expect(await call("ro", `Bearer ${cfg.rwToken}`)).toBe(200)); + it("ro token does NOT satisfy rw scope", async () => expect(await call("rw", `Bearer ${cfg.roToken}`)).toBe(401)); + it("rw token satisfies rw scope", async () => expect(await call("rw", `Bearer ${cfg.rwToken}`)).toBe(200)); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- auth` +Expected: FAIL — `Cannot find module '../src/auth.js'`. + +- [ ] **Step 3: Implement `src/auth.ts`** + +```typescript +import crypto from "node:crypto"; +import type { RequestHandler } from "express"; +import type { Config } from "./config.js"; + +function safeEqual(a: string, b: string): boolean { + const ab = Buffer.from(a), bb = Buffer.from(b); + if (ab.length !== bb.length) return false; + return crypto.timingSafeEqual(ab, bb); +} + +export function requireScope(cfg: Config, scope: "ro" | "rw"): RequestHandler { + return (req, res, next) => { + const h = req.header("authorization") ?? ""; + const m = /^Bearer (.+)$/.exec(h); + if (!m) return res.status(401).json({ error: "unauthorized" }); + const token = m[1]; + const isRw = safeEqual(token, cfg.rwToken); + const isRo = safeEqual(token, cfg.roToken); + const ok = scope === "rw" ? isRw : isRo || isRw; + if (!ok) return res.status(401).json({ error: "unauthorized" }); + next(); + }; +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `npm test -- auth` +Expected: PASS (6 tests). + +- [ ] **Step 5: Commit** + +```bash +git add -A && git commit -m "feat: bearer auth middleware with ro/rw scopes" +``` + +--- + +### Task 3: Test fixtures — a GRDB-shaped mirror + a `.noopbak` + +**Files:** +- Create: `noop-cloud/test/fixtures/make-fixture.ts` +- Test: `noop-cloud/test/fixtures/make-fixture.test.ts` + +**Interfaces:** +- Produces: `buildMirrorSqlite(path: string): void` — writes a SQLite file with the mirror schema and multi-source rows (WHOOP `my-whoop`, `oura-api`, `apple-health`) across days `2026-06-10..2026-06-13`, including a `grdb_migrations` table (origin marker). +- Produces: `buildNoopbak(path: string): void` — a ZIP with entry `noop-backup.sqlite` (the mirror bytes) first. Also `buildNoopbakFrom(sqlitePath, zipPath)`. +- Schema (subset sufficient for Phase-1 tools), tables + keys: `grdb_migrations(identifier TEXT)`, `hrSample(deviceId,ts,bpm)`, `dailyMetric(deviceId,day,totalSleepMin,efficiency,restingHr,avgHrv,recovery,strain,spo2Pct,skinTempDevC,respRateBpm,steps,activeKcalEst)`, `sleepSession(deviceId,startTs,endTs,efficiency,restingHr,avgHrv,stagesJSON,userEdited,startTsAdjusted)`, `workout(deviceId,startTs,sport,endTs,source,durationS,energyKcal,distanceM)`, `metricSeries(deviceId,day,key,value)`, `appleDaily(deviceId,day,steps,restingHr)`, `pairedDevice(id,brand,model,sourceKind)`. + +- [ ] **Step 1: Write the failing test** + +`test/fixtures/make-fixture.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; +import path from "node:path"; +import Database from "better-sqlite3"; +import AdmZip from "adm-zip"; +import { buildMirrorSqlite, buildNoopbak } from "./make-fixture.js"; + +const tmp = path.join(process.cwd(), "test/.tmp"); +beforeAll(() => fs.mkdirSync(tmp, { recursive: true })); + +describe("fixtures", () => { + it("mirror has multi-source dailyMetric rows and grdb_migrations", () => { + const p = path.join(tmp, "m.sqlite"); + buildMirrorSqlite(p); + const db = new Database(p, { readonly: true }); + const sources = db.prepare("SELECT DISTINCT deviceId FROM dailyMetric ORDER BY deviceId").all().map((r: any) => r.deviceId); + expect(sources).toContain("oura-api"); + expect(sources).toContain("apple-health"); + expect(sources.some((s: string) => s !== "oura-api" && s !== "apple-health")).toBe(true); + expect(db.prepare("SELECT count(*) c FROM grdb_migrations").get() as any).toHaveProperty("c"); + db.close(); + }); + + it("noopbak is a zip whose FIRST entry is noop-backup.sqlite", () => { + const p = path.join(tmp, "b.noopbak"); + buildNoopbak(p); + const entries = new AdmZip(p).getEntries(); + expect(entries[0].entryName).toBe("noop-backup.sqlite"); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- make-fixture` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement `test/fixtures/make-fixture.ts`** + +```typescript +import Database from "better-sqlite3"; +import AdmZip from "adm-zip"; +import fs from "node:fs"; + +const DAYS = ["2026-06-10", "2026-06-11", "2026-06-12", "2026-06-13"]; +const tsOf = (day: string, h = 3) => Math.floor(new Date(`${day}T0${h}:00:00Z`).getTime() / 1000); + +export function buildMirrorSqlite(target: string): void { + if (fs.existsSync(target)) fs.rmSync(target); + const db = new Database(target); + db.pragma("journal_mode = WAL"); + db.exec(` + CREATE TABLE grdb_migrations (identifier TEXT PRIMARY KEY); + CREATE TABLE hrSample (deviceId TEXT, ts INTEGER, bpm INTEGER, PRIMARY KEY(deviceId, ts)); + CREATE TABLE dailyMetric (deviceId TEXT, day TEXT, totalSleepMin INTEGER, efficiency REAL, + restingHr REAL, avgHrv REAL, recovery REAL, strain REAL, spo2Pct REAL, skinTempDevC REAL, + respRateBpm REAL, steps INTEGER, activeKcalEst REAL, PRIMARY KEY(deviceId, day)); + CREATE TABLE sleepSession (deviceId TEXT, startTs INTEGER, endTs INTEGER, efficiency REAL, + restingHr REAL, avgHrv REAL, stagesJSON TEXT, userEdited INTEGER DEFAULT 0, + startTsAdjusted INTEGER, PRIMARY KEY(deviceId, startTs)); + CREATE TABLE workout (deviceId TEXT, startTs INTEGER, sport TEXT, endTs INTEGER, source TEXT, + durationS REAL, energyKcal REAL, distanceM REAL, PRIMARY KEY(deviceId, startTs, sport)); + CREATE TABLE metricSeries (deviceId TEXT, day TEXT, key TEXT, value REAL, PRIMARY KEY(deviceId, day, key)); + CREATE TABLE appleDaily (deviceId TEXT, day TEXT, steps INTEGER, restingHr REAL, PRIMARY KEY(deviceId, day)); + CREATE TABLE pairedDevice (id TEXT PRIMARY KEY, brand TEXT, model TEXT, sourceKind TEXT); + `); + db.prepare("INSERT INTO grdb_migrations VALUES (?)").run("v25-oura-raw"); + const pd = db.prepare("INSERT INTO pairedDevice VALUES (?,?,?,?)"); + pd.run("my-whoop", "WHOOP", "5.0", "strap"); + pd.run("oura-api", "Oura", "Oura (cloud)", "cloudImport"); + pd.run("apple-health", "Apple", "Apple Health", "appleHealth"); + + const dm = db.prepare(`INSERT INTO dailyMetric + (deviceId,day,totalSleepMin,efficiency,restingHr,avgHrv,recovery,strain,spo2Pct,skinTempDevC,respRateBpm,steps,activeKcalEst) + VALUES (@deviceId,@day,@totalSleepMin,@efficiency,@restingHr,@avgHrv,@recovery,@strain,@spo2Pct,@skinTempDevC,@respRateBpm,@steps,@activeKcalEst)`); + const base = (deviceId: string, day: string, o: Partial) => ({ + deviceId, day, totalSleepMin: 420, efficiency: 90, restingHr: 52, avgHrv: 65, recovery: null, + strain: null, spo2Pct: 97, skinTempDevC: -0.1, respRateBpm: 14.5, steps: 8000, activeKcalEst: 500, ...o, + }); + for (const day of DAYS) { + // WHOOP is the scored source: recovery/strain present. + dm.run(base("my-whoop", day, { restingHr: 51, avgHrv: 70, recovery: 66, strain: 12.5 })); + // Oura cloud: honest data, recovery/strain null; slightly different RHR/HRV. + dm.run(base("oura-api", day, { restingHr: 53, avgHrv: 62 })); + // Apple: steps only-ish. + dm.run(base("apple-health", day, { restingHr: 54, avgHrv: null, steps: 8200 })); + } + const ss = db.prepare(`INSERT INTO sleepSession + (deviceId,startTs,endTs,efficiency,restingHr,avgHrv,stagesJSON,userEdited,startTsAdjusted) + VALUES (?,?,?,?,?,?,?,?,?)`); + for (const day of DAYS) { + ss.run("my-whoop", tsOf(day, 3), tsOf(day, 3) + 25200, 90, 51, 70, null, 0, null); + ss.run("oura-api", tsOf(day, 3) + 60, tsOf(day, 3) + 25000, 88, 53, 62, null, 0, null); + } + const wo = db.prepare(`INSERT INTO workout (deviceId,startTs,sport,endTs,source,durationS,energyKcal,distanceM) + VALUES (?,?,?,?,?,?,?,?)`); + wo.run("my-whoop", tsOf("2026-06-12", 18), "running", tsOf("2026-06-12", 18) + 1800, "whoop", 1800, 320, 5000); + wo.run("oura-api", tsOf("2026-06-11", 17), "walking", tsOf("2026-06-11", 17) + 2400, "oura", 2400, 180, 3000); + + const ms = db.prepare("INSERT INTO metricSeries VALUES (?,?,?,?)"); + for (const day of DAYS) { + ms.run("oura-api", day, "ref_sleep_score", 82); + ms.run("oura-api", day, "oura_readiness", 78); + } + const hr = db.prepare("INSERT INTO hrSample VALUES (?,?,?)"); + for (const day of DAYS) for (let i = 0; i < 5; i++) hr.run("oura-api", tsOf(day, 3) + i * 300, 55 + i); + db.close(); +} + +export function buildNoopbakFrom(sqlitePath: string, zipPath: string): void { + if (fs.existsSync(zipPath)) fs.rmSync(zipPath); + const zip = new AdmZip(); + zip.addFile("noop-backup.sqlite", fs.readFileSync(sqlitePath)); // DB entry FIRST + zip.writeZip(zipPath); +} + +export function buildNoopbak(zipPath: string): void { + const tmpSqlite = zipPath + ".src.sqlite"; + buildMirrorSqlite(tmpSqlite); + buildNoopbakFrom(tmpSqlite, zipPath); + fs.rmSync(tmpSqlite); + for (const ext of ["-wal", "-shm"]) if (fs.existsSync(tmpSqlite + ext)) fs.rmSync(tmpSqlite + ext); +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `npm test -- make-fixture` +Expected: PASS (2 tests). + +- [ ] **Step 5: Commit** + +```bash +git add -A && git commit -m "test: fixture builders for mirror sqlite + .noopbak" +``` + +--- + +### Task 4: Mirror read layer + +**Files:** +- Create: `noop-cloud/src/mirror.ts` +- Test: `noop-cloud/test/mirror.test.ts` + +**Interfaces:** +- Consumes: fixtures from Task 3. +- Produces: `class Mirror` opened read-only, methods: + - `sourceFamily(deviceId: string): "whoop" | "oura" | "apple"` (module fn `sourceFamily`). + - `sources(): { deviceId: string; family: string; brand: string | null; latestDay: string | null }[]` + - `dailyMetrics(opts: { deviceId?: string; from: string; to: string }): DailyMetricRow[]` + - `sleepSummary(opts: { from: string; to: string }): SleepRow[]` + - `workoutSummary(opts: { from: string; to: string }): WorkoutRow[]` + - `metricSeries(opts: { deviceId?: string; key?: string; from: string; to: string }): MetricPointRow[]` + - `latestDataDay(): string | null` (max day across `dailyMetric`) + - `hrCoverageDays(deviceId: string): number` + - `close(): void` +- Produces types `DailyMetricRow`, `SleepRow`, `WorkoutRow`, `MetricPointRow` (plain interfaces mirroring the columns; `deviceId` + `family` included on each). + +- [ ] **Step 1: Write the failing test** + +`test/mirror.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; +import { buildMirrorSqlite } from "./fixtures/make-fixture.js"; +import { Mirror, sourceFamily } from "../src/mirror.js"; + +const p = path.join(process.cwd(), "test/.tmp/mirror-read.sqlite"); +beforeAll(() => { fs.mkdirSync(path.dirname(p), { recursive: true }); buildMirrorSqlite(p); }); + +describe("Mirror", () => { + it("classifies source families", () => { + expect(sourceFamily("oura-api")).toBe("oura"); + expect(sourceFamily("apple-health")).toBe("apple"); + expect(sourceFamily("my-whoop")).toBe("whoop"); + }); + it("lists sources with latest day", () => { + const m = new Mirror(p); + const s = m.sources(); + expect(s.find((x) => x.deviceId === "oura-api")?.latestDay).toBe("2026-06-13"); + m.close(); + }); + it("filters dailyMetrics by source and range", () => { + const m = new Mirror(p); + const rows = m.dailyMetrics({ deviceId: "oura-api", from: "2026-06-11", to: "2026-06-12" }); + expect(rows.length).toBe(2); + expect(rows[0].family).toBe("oura"); + m.close(); + }); + it("reports latest data day across sources", () => { + const m = new Mirror(p); expect(m.latestDataDay()).toBe("2026-06-13"); m.close(); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- mirror` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement `src/mirror.ts`** + +```typescript +import Database from "better-sqlite3"; + +export type Family = "whoop" | "oura" | "apple"; +export function sourceFamily(deviceId: string): Family { + if (deviceId === "oura-api") return "oura"; + if (deviceId === "apple-health") return "apple"; + return "whoop"; +} + +export interface DailyMetricRow { + deviceId: string; family: Family; day: string; + totalSleepMin: number | null; efficiency: number | null; restingHr: number | null; + avgHrv: number | null; recovery: number | null; strain: number | null; spo2Pct: number | null; + skinTempDevC: number | null; respRateBpm: number | null; steps: number | null; activeKcalEst: number | null; +} +export interface SleepRow { deviceId: string; family: Family; startTs: number; endTs: number; efficiency: number | null; restingHr: number | null; avgHrv: number | null; userEdited: number; } +export interface WorkoutRow { deviceId: string; family: Family; startTs: number; endTs: number; sport: string; source: string | null; durationS: number | null; energyKcal: number | null; distanceM: number | null; } +export interface MetricPointRow { deviceId: string; family: Family; day: string; key: string; value: number; } + +const withFamily = (r: T) => ({ ...r, family: sourceFamily(r.deviceId) }); + +export class Mirror { + private db: Database.Database; + constructor(path: string) { this.db = new Database(path, { readonly: true, fileMustExist: true }); } + close(): void { this.db.close(); } + + sources() { + const rows = this.db.prepare(` + SELECT d.deviceId, MAX(d.day) AS latestDay, p.brand AS brand + FROM dailyMetric d LEFT JOIN pairedDevice p ON p.id = d.deviceId + GROUP BY d.deviceId ORDER BY d.deviceId`).all() as any[]; + return rows.map((r) => ({ deviceId: r.deviceId, family: sourceFamily(r.deviceId), brand: r.brand ?? null, latestDay: r.latestDay ?? null })); + } + dailyMetrics(opts: { deviceId?: string; from: string; to: string }): DailyMetricRow[] { + const where = ["day >= ? AND day <= ?"]; const args: any[] = [opts.from, opts.to]; + if (opts.deviceId) { where.push("deviceId = ?"); args.push(opts.deviceId); } + return (this.db.prepare(`SELECT * FROM dailyMetric WHERE ${where.join(" AND ")} ORDER BY day, deviceId`).all(...args) as any[]).map(withFamily); + } + sleepSummary(opts: { from: string; to: string }): SleepRow[] { + const lo = Math.floor(new Date(`${opts.from}T00:00:00Z`).getTime() / 1000); + const hi = Math.floor(new Date(`${opts.to}T23:59:59Z`).getTime() / 1000); + return (this.db.prepare(`SELECT deviceId,startTs,endTs,efficiency,restingHr,avgHrv,userEdited FROM sleepSession WHERE startTs >= ? AND startTs <= ? ORDER BY startTs`).all(lo, hi) as any[]).map(withFamily); + } + workoutSummary(opts: { from: string; to: string }): WorkoutRow[] { + const lo = Math.floor(new Date(`${opts.from}T00:00:00Z`).getTime() / 1000); + const hi = Math.floor(new Date(`${opts.to}T23:59:59Z`).getTime() / 1000); + return (this.db.prepare(`SELECT deviceId,startTs,endTs,sport,source,durationS,energyKcal,distanceM FROM workout WHERE startTs >= ? AND startTs <= ? ORDER BY startTs`).all(lo, hi) as any[]).map(withFamily); + } + metricSeries(opts: { deviceId?: string; key?: string; from: string; to: string }): MetricPointRow[] { + const where = ["day >= ? AND day <= ?"]; const args: any[] = [opts.from, opts.to]; + if (opts.deviceId) { where.push("deviceId = ?"); args.push(opts.deviceId); } + if (opts.key) { where.push("key = ?"); args.push(opts.key); } + return (this.db.prepare(`SELECT deviceId,day,key,value FROM metricSeries WHERE ${where.join(" AND ")} ORDER BY day, key`).all(...args) as any[]).map(withFamily); + } + latestDataDay(): string | null { return (this.db.prepare("SELECT MAX(day) AS d FROM dailyMetric").get() as any)?.d ?? null; } + hrCoverageDays(deviceId: string): number { return (this.db.prepare("SELECT COUNT(DISTINCT date(ts,'unixepoch')) AS c FROM hrSample WHERE deviceId = ?").get(deviceId) as any).c; } +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `npm test -- mirror` +Expected: PASS (4 tests). + +- [ ] **Step 5: Commit** + +```bash +git add -A && git commit -m "feat: read-only mirror query layer (multi-source aware)" +``` + +--- + +### Task 5: Ingest pipeline and `POST /ingest` + +**Files:** +- Create: `noop-cloud/src/ingest.ts` +- Modify: `noop-cloud/src/server.ts` (add `/ingest` route + server.sqlite ingest log) +- Test: `noop-cloud/test/ingest.test.ts` + +**Interfaces:** +- Consumes: `Config`, `requireScope` (rw), fixtures. +- Produces: `ingestNoopbak(buf: Buffer, cfg: Config): { ok: true; bytes: number; latestDay: string | null }` — validates and atomically swaps into `cfg.mirrorPath`. Throws `IngestError` (with `.code`) on: `too_large`, `bad_zip`, `no_sqlite_entry`, `bad_magic`, `quick_check_failed`, `foreign_db`. +- Produces: `openServerDb(cfg): Database` creating `ingestLog(id INTEGER PK, receivedAt INTEGER, bytes INTEGER, latestDay TEXT)`. +- Produces: `latestIngest(cfg): { receivedAt: number; bytes: number; latestDay: string | null } | null`. +- Route: `POST /ingest` (rw), `express.raw({ type: "*/*", limit })`, returns 200 `{ ok, bytes, latestDay }` or 4xx `{ error: code }`. + +- [ ] **Step 1: Write the failing test** + +`test/ingest.test.ts`: + +```typescript +import { describe, it, expect, beforeEach } from "vitest"; +import fs from "node:fs"; import path from "node:path"; import Database from "better-sqlite3"; import AdmZip from "adm-zip"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; +import { ingestNoopbak, latestIngest, IngestError } from "../src/ingest.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/ingest"); +const cfg = () => ({ dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000 } as any); +beforeEach(() => { fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); }); + +describe("ingest", () => { + it("accepts a valid .noopbak and swaps the mirror", () => { + const zip = path.join(dataDir, "in.noopbak"); buildNoopbak(zip); + const r = ingestNoopbak(fs.readFileSync(zip), cfg()); + expect(r.ok).toBe(true); expect(r.latestDay).toBe("2026-06-13"); + const db = new Database(cfg().mirrorPath, { readonly: true }); + expect((db.prepare("SELECT COUNT(*) c FROM dailyMetric").get() as any).c).toBeGreaterThan(0); db.close(); + expect(latestIngest(cfg())?.latestDay).toBe("2026-06-13"); + }); + it("rejects oversized", () => { + const c = cfg(); c.maxIngestBytes = 10; + expect(() => ingestNoopbak(Buffer.alloc(100), c)).toThrow(IngestError); + }); + it("rejects a zip with no sqlite entry", () => { + const z = new AdmZip(); z.addFile("notes.txt", Buffer.from("hi")); + try { ingestNoopbak(z.toBuffer(), cfg()); expect.fail("should throw"); } + catch (e: any) { expect(e.code).toBe("no_sqlite_entry"); } + }); + it("rejects a foreign sqlite (no grdb_migrations)", () => { + const foreign = path.join(dataDir, "f.sqlite"); const db = new Database(foreign); db.exec("CREATE TABLE x(a)"); db.close(); + const z = new AdmZip(); z.addFile("noop-backup.sqlite", fs.readFileSync(foreign)); + try { ingestNoopbak(z.toBuffer(), cfg()); expect.fail("should throw"); } + catch (e: any) { expect(e.code).toBe("foreign_db"); } + }); + it("does not corrupt an existing mirror when the new upload is invalid", () => { + const zip = path.join(dataDir, "ok.noopbak"); buildNoopbak(zip); ingestNoopbak(fs.readFileSync(zip), cfg()); + try { ingestNoopbak(Buffer.from("not a zip"), cfg()); } catch { /* expected */ } + const db = new Database(cfg().mirrorPath, { readonly: true }); + expect((db.prepare("SELECT COUNT(*) c FROM dailyMetric").get() as any).c).toBeGreaterThan(0); db.close(); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- ingest` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement `src/ingest.ts`** + +```typescript +import fs from "node:fs"; import path from "node:path"; import crypto from "node:crypto"; +import Database from "better-sqlite3"; import AdmZip from "adm-zip"; +import type { Config } from "./config.js"; + +export class IngestError extends Error { constructor(public code: string, msg?: string) { super(msg ?? code); } } +const SQLITE_MAGIC = Buffer.from("SQLite format 3", "binary"); + +export function openServerDb(cfg: Pick): Database.Database { + const db = new Database(cfg.serverDbPath); + db.pragma("journal_mode = WAL"); + db.exec("CREATE TABLE IF NOT EXISTS ingestLog (id INTEGER PRIMARY KEY AUTOINCREMENT, receivedAt INTEGER, bytes INTEGER, latestDay TEXT)"); + return db; +} +export function latestIngest(cfg: Pick) { + const db = openServerDb(cfg); + const r = db.prepare("SELECT receivedAt, bytes, latestDay FROM ingestLog ORDER BY id DESC LIMIT 1").get() as any; + db.close(); + return r ?? null; +} + +export function ingestNoopbak(buf: Buffer, cfg: Pick): { ok: true; bytes: number; latestDay: string | null } { + if (buf.length > cfg.maxIngestBytes) throw new IngestError("too_large", `body ${buf.length} > ${cfg.maxIngestBytes}`); + let zip: AdmZip; + try { zip = new AdmZip(buf); } catch { throw new IngestError("bad_zip"); } + const entry = zip.getEntries().find((e) => e.entryName.endsWith(".sqlite")); + if (!entry) throw new IngestError("no_sqlite_entry"); + const sqliteBytes = entry.getData(); + if (!sqliteBytes.subarray(0, SQLITE_MAGIC.length).equals(SQLITE_MAGIC)) throw new IngestError("bad_magic"); + + // Stage to a temp file, validate, then atomically rename into place. + fs.mkdirSync(cfg.dataDir, { recursive: true }); + const staged = path.join(cfg.dataDir, `.staged-${crypto.randomBytes(6).toString("hex")}.sqlite`); + fs.writeFileSync(staged, sqliteBytes); + try { + const sdb = new Database(staged, { readonly: true }); + try { + const qc = sdb.pragma("quick_check", { simple: true }); + if (qc !== "ok") throw new IngestError("quick_check_failed", String(qc)); + const hasGrdb = sdb.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name='grdb_migrations'").get(); + if (!hasGrdb) throw new IngestError("foreign_db"); + var latestDay = (sdb.prepare("SELECT MAX(day) d FROM dailyMetric").get() as any)?.d ?? null; + } finally { sdb.close(); } + // Atomic swap: rename staged -> mirror (same filesystem). Remove stale WAL/SHM sidecars. + for (const ext of ["-wal", "-shm"]) { const s = cfg.mirrorPath + ext; if (fs.existsSync(s)) fs.rmSync(s); } + fs.renameSync(staged, cfg.mirrorPath); + } catch (e) { + if (fs.existsSync(staged)) fs.rmSync(staged); + throw e; + } + const db = openServerDb(cfg); + db.prepare("INSERT INTO ingestLog (receivedAt, bytes, latestDay) VALUES (?,?,?)").run(Math.floor(Date.now() / 1000), buf.length, latestDay); + db.close(); + return { ok: true, bytes: buf.length, latestDay }; +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `npm test -- ingest` +Expected: PASS (5 tests). + +- [ ] **Step 5: Wire the `/ingest` route in `src/server.ts`** + +Replace `createApp` in `src/server.ts` with: + +```typescript +import express from "express"; +import fs from "node:fs"; +import { Config, loadConfig } from "./config.js"; +import { requireScope } from "./auth.js"; +import { ingestNoopbak, IngestError } from "./ingest.js"; + +export function createApp(cfg: Config): express.Express { + fs.mkdirSync(cfg.dataDir, { recursive: true }); + const app = express(); + app.get("/healthz", (_req, res) => res.json({ ok: true })); + + app.post("/ingest", requireScope(cfg, "rw"), + express.raw({ type: "*/*", limit: cfg.maxIngestBytes }), + (req, res) => { + try { + const out = ingestNoopbak(req.body as Buffer, cfg); + res.json(out); + } catch (e) { + if (e instanceof IngestError) return res.status(400).json({ error: e.code }); + console.error("ingest error", e); + res.status(500).json({ error: "ingest_failed" }); + } + }); + + return app; +} +``` + +(keep the `if (process.argv[1]...)` entry block from Task 1) + +- [ ] **Step 6: Add an HTTP-level ingest test** + +Append to `test/ingest.test.ts`: + +```typescript +import http from "node:http"; import { createApp } from "../src/server.js"; +it("POST /ingest requires rw and swaps the mirror", async () => { + process.env.RO_TOKEN = "ro".padEnd(40, "x"); process.env.RW_TOKEN = "rw".padEnd(40, "y"); + const c = cfg(); c.roToken = process.env.RO_TOKEN; c.rwToken = process.env.RW_TOKEN; c.port = 0; + const zip = path.join(dataDir, "http.noopbak"); buildNoopbak(zip); const body = fs.readFileSync(zip); + const app = createApp(c); const server = app.listen(0); const port = (server.address() as any).port; + const status = await new Promise((resolve) => { + const r = http.request({ port, path: "/ingest", method: "POST", headers: { authorization: `Bearer ${c.rwToken}`, "content-type": "application/octet-stream" } }, (res) => { res.resume(); res.on("end", () => resolve(res.statusCode!)); }); + r.end(body); + }); + server.close(); expect(status).toBe(200); +}); +``` + +- [ ] **Step 7: Run to verify pass** + +Run: `npm test -- ingest && npm run build` +Expected: PASS (6 tests); build clean. + +- [ ] **Step 8: Commit** + +```bash +git add -A && git commit -m "feat: /ingest — validate + atomic-swap .noopbak into mirror.sqlite" +``` + +--- + +### Task 6: MCP stateless transport, tool registry, and prompts + +**Files:** +- Create: `noop-cloud/src/mcp.ts` +- Create: `noop-cloud/src/tools/index.ts` +- Modify: `noop-cloud/src/server.ts` (mount `/mcp`, ro scope) +- Test: `noop-cloud/test/mcp.test.ts` + +**Interfaces:** +- Consumes: `Config`, `Mirror`, `requireScope`. +- Produces: `buildMcpServer(cfg: Config): McpServer` — a fresh server with all tools + prompts registered. Tools open a `Mirror` per call and `close()` it in a `finally`. +- Produces: `registerTools(server: McpServer, cfg: Config): void` (Task 7-10 add tool files, each exporting `register(server, cfg)`; `registerTools` calls them). +- Produces: prompts `morning_report`, `corroborate_sources`, `find_messy_data` (each returns a `messages` array seeding the agent). +- Route: `POST /mcp` (ro) constructs a fresh transport (`sessionIdGenerator: undefined`) + server per request; `DELETE`/`GET /mcp` → 405. +- **Note:** verify exact import paths against the installed `@modelcontextprotocol/sdk@1.29.0` (`node -e "console.log(require.resolve('@modelcontextprotocol/sdk/server/mcp.js'))"`). If a path differs, adjust — the API shape (`McpServer`, `registerTool`, `registerPrompt`, `StreamableHTTPServerTransport`, `server.connect`, `transport.handleRequest`) is stable in 1.29.x. + +- [ ] **Step 1: Write the failing test (MCP list-tools over HTTP)** + +`test/mcp.test.ts`: + +```typescript +import { describe, it, expect } from "vitest"; +import fs from "node:fs"; import path from "node:path"; import http from "node:http"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; +import { ingestNoopbak } from "../src/ingest.js"; +import { createApp } from "../src/server.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/mcp"); +function cfg() { return { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000, roToken: "ro".padEnd(40, "x"), rwToken: "rw".padEnd(40, "y"), port: 0 } as any; } + +function mcpCall(port: number, token: string, body: object): Promise<{ status: number; json: any }> { + return new Promise((resolve) => { + const payload = JSON.stringify(body); + const r = http.request({ port, path: "/mcp", method: "POST", headers: { authorization: `Bearer ${token}`, "content-type": "application/json", accept: "application/json, text/event-stream" } }, (res) => { + let d = ""; res.on("data", (c) => (d += c)); res.on("end", () => { + // Streamable HTTP may reply as SSE (`data: {json}`) or plain JSON. + const line = d.split("\n").find((l) => l.startsWith("data:")) ?? d; + const txt = line.replace(/^data:\s*/, ""); + resolve({ status: res.statusCode!, json: txt ? JSON.parse(txt) : null }); + }); + }); + r.end(payload); + }); +} + +describe("/mcp", () => { + it("401 without ro token", async () => { + fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); + const zip = path.join(dataDir, "b.noopbak"); buildNoopbak(zip); ingestNoopbak(fs.readFileSync(zip), cfg()); + const app = createApp(cfg()); const server = app.listen(0); const port = (server.address() as any).port; + const { status } = await mcpCall(port, "wrong", { jsonrpc: "2.0", id: 1, method: "tools/list" }); + server.close(); expect(status).toBe(401); + }); + it("lists registered tools", async () => { + const app = createApp(cfg()); const server = app.listen(0); const port = (server.address() as any).port; + const { json } = await mcpCall(port, cfg().roToken, { jsonrpc: "2.0", id: 1, method: "tools/list" }); + server.close(); + const names = (json.result?.tools ?? []).map((t: any) => t.name); + expect(names).toContain("data_freshness"); + expect(names).toContain("health_snapshot"); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- mcp` +Expected: FAIL — `src/mcp.js` missing. + +- [ ] **Step 3: Implement `src/tools/index.ts` (registry stub)** + +```typescript +import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; +import type { Config } from "../config.js"; +import { registerCoreTools } from "./core.js"; + +export function registerTools(server: McpServer, cfg: Config): void { + registerCoreTools(server, cfg); // health_snapshot + data_freshness (Task 7) + // Task 8: registerQueryTools(server, cfg) + // Task 9: registerCompareSources(server, cfg) + // Task 10: registerSearchFetch(server, cfg) +} +``` + +- [ ] **Step 4: Implement `src/mcp.ts`** + +```typescript +import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; +import type { Config } from "./config.js"; +import { registerTools } from "./tools/index.js"; + +export function buildMcpServer(cfg: Config): McpServer { + const server = new McpServer({ name: "noop-cloud", version: "0.1.0" }); + + server.registerPrompt("morning_report", { + title: "Morning report", + description: "Draft a plain-English morning briefing from the last few days of biometrics.", + }, () => ({ + messages: [{ role: "user", content: { type: "text", text: + "Using the noop-cloud tools, write my morning report. Call data_freshness first and state the mirror age. " + + "Then summarize the last 3 days: recovery/strain (WHOOP), sleep (duration, efficiency), resting HR and HRV trend, " + + "and any workouts. Use compare_sources to flag where WHOOP, Oura, and Apple disagree by more than ~10%. " + + "Keep it under 200 words, concrete, no medical advice." } }], + })); + + server.registerPrompt("corroborate_sources", { + title: "Corroborate sources", + description: "Cross-check WHOOP vs Oura vs Apple for a date range and surface disagreements.", + }, () => ({ + messages: [{ role: "user", content: { type: "text", text: + "Call compare_sources for the last 14 days. List days where resting HR, HRV, or sleep duration differ by more than 10% " + + "across sources, and say which source looks like the outlier. Note any per-source baseline offsets." } }], + })); + + server.registerPrompt("find_messy_data", { + title: "Find messy data", + description: "Scan recent data for likely-mislogged workouts or implausible values.", + }, () => ({ + messages: [{ role: "user", content: { type: "text", text: + "Call workout_summary and metric_series for the last 30 days. Flag workouts with implausible duration/distance/energy, " + + "duplicate workouts across sources on the same day, and daily values that break trend. Do not change anything — just list findings." } }], + })); + + registerTools(server, cfg); + return server; +} +``` + +- [ ] **Step 5: Mount `/mcp` in `src/server.ts`** + +Add to `createApp` (before `return app`): + +```typescript +import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js"; +import { buildMcpServer } from "./mcp.js"; + +// ... inside createApp, after /ingest: +app.post("/mcp", requireScope(cfg, "ro"), express.json({ limit: "4mb" }), async (req, res) => { + const server = buildMcpServer(cfg); + const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: undefined }); + res.on("close", () => { transport.close(); server.close(); }); + try { + await server.connect(transport); + await transport.handleRequest(req, res, req.body); + } catch (e) { + console.error("mcp error", e); + if (!res.headersSent) res.status(500).json({ jsonrpc: "2.0", error: { code: -32603, message: "internal error" }, id: null }); + } +}); +app.all("/mcp", (_req, res) => res.status(405).json({ error: "method_not_allowed" })); +``` + +- [ ] **Step 6: Run to verify pass** + +Run: `npm test -- mcp && npm run build` +Expected: PASS (2 tests) once Task 7's `core.ts` exists. If `core.js` is not yet created, temporarily stub `registerCoreTools` to a no-op that registers a `data_freshness` returning `{}`; Task 7 fills it in. (Prefer implementing Task 7 Step 3 before running this.) + +- [ ] **Step 7: Commit** + +```bash +git add -A && git commit -m "feat: stateless Streamable HTTP /mcp endpoint + prompts + tool registry" +``` + +--- + +### Task 7: Core tools — `health_snapshot` + `data_freshness` + +**Files:** +- Create: `noop-cloud/src/tools/core.ts` +- Test: `noop-cloud/test/tools-core.test.ts` + +**Interfaces:** +- Consumes: `Mirror`, `latestIngest`, `Config`. +- Produces: `registerCoreTools(server: McpServer, cfg: Config): void`. + - `data_freshness` (no args) → `{ mirrorAgeSeconds, lastIngestAt, latestDataDay, sources: [{deviceId, family, latestDay}] }`. + - `health_snapshot` (args `{ days?: number }`, default 3) → per-day, per-family roll-up of recovery/strain/sleep/restingHr/avgHrv for the most recent `days`. +- Each tool handler returns `{ content: [{ type: "text", text: }], structuredContent: }`. + +- [ ] **Step 1: Write the failing test** + +`test/tools-core.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; +import { ingestNoopbak } from "../src/ingest.js"; +import { dataFreshness, healthSnapshot } from "../src/tools/core.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/tools-core"); +const cfg = { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000 } as any; +beforeAll(() => { fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); const z = path.join(dataDir, "b.noopbak"); buildNoopbak(z); ingestNoopbak(fs.readFileSync(z), cfg); }); + +describe("core tools", () => { + it("data_freshness reports sources + latest day", () => { + const r = dataFreshness(cfg); + expect(r.latestDataDay).toBe("2026-06-13"); + expect(r.sources.map((s) => s.deviceId)).toContain("oura-api"); + expect(r.mirrorAgeSeconds).toBeGreaterThanOrEqual(0); + }); + it("health_snapshot rolls up recent days by family", () => { + const r = healthSnapshot(cfg, { days: 2 }); + expect(r.days.length).toBe(2); + const last = r.days[r.days.length - 1]; + expect(last.day).toBe("2026-06-13"); + expect(last.whoop?.recovery).toBe(66); + expect(last.oura?.restingHr).toBe(53); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- tools-core` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement `src/tools/core.ts`** + +```typescript +import { z } from "zod"; +import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; +import type { Config } from "../config.js"; +import { Mirror, Family } from "../mirror.js"; +import { latestIngest } from "../ingest.js"; + +export function dataFreshness(cfg: Config) { + const m = new Mirror(cfg.mirrorPath); + try { + const li = latestIngest(cfg); + const now = Math.floor(Date.now() / 1000); + return { + mirrorAgeSeconds: li ? now - li.receivedAt : null, + lastIngestAt: li ? new Date(li.receivedAt * 1000).toISOString() : null, + latestDataDay: m.latestDataDay(), + sources: m.sources().map((s) => ({ deviceId: s.deviceId, family: s.family, latestDay: s.latestDay })), + }; + } finally { m.close(); } +} + +export function healthSnapshot(cfg: Config, args: { days?: number }) { + const days = Math.max(1, Math.min(args.days ?? 3, 31)); + const m = new Mirror(cfg.mirrorPath); + try { + const latest = m.latestDataDay(); + if (!latest) return { days: [] as any[] }; + const to = latest; + const from = new Date(new Date(`${to}T00:00:00Z`).getTime() - (days - 1) * 86_400_000).toISOString().slice(0, 10); + const rows = m.dailyMetrics({ from, to }); + const byDay = new Map(); + for (const r of rows) { + const d = byDay.get(r.day) ?? { day: r.day }; + const fam: Family = r.family; + d[fam] = { restingHr: r.restingHr, avgHrv: r.avgHrv, totalSleepMin: r.totalSleepMin, efficiency: r.efficiency, recovery: r.recovery, strain: r.strain, steps: r.steps }; + byDay.set(r.day, d); + } + return { from, to, days: [...byDay.values()].sort((a, b) => a.day.localeCompare(b.day)) }; + } finally { m.close(); } +} + +const asTool = (obj: unknown) => ({ content: [{ type: "text" as const, text: JSON.stringify(obj, null, 2) }], structuredContent: obj as Record }); + +export function registerCoreTools(server: McpServer, cfg: Config): void { + server.registerTool("data_freshness", { + title: "Data freshness", + description: "How stale the mirror is and which sources it holds. Call this first.", + inputSchema: {}, + annotations: { readOnlyHint: true, openWorldHint: false }, + }, async () => asTool(dataFreshness(cfg))); + + server.registerTool("health_snapshot", { + title: "Health snapshot", + description: "Recent per-day roll-up (recovery, strain, sleep, resting HR, HRV) grouped by source family.", + inputSchema: { days: z.number().int().min(1).max(31).optional().describe("How many recent days (default 3).") }, + annotations: { readOnlyHint: true, openWorldHint: false }, + }, async (args) => asTool(healthSnapshot(cfg, args))); +} +``` + +- [ ] **Step 4: Run to verify pass (core + mcp)** + +Run: `npm test -- tools-core && npm test -- mcp && npm run build` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add -A && git commit -m "feat: core MCP tools health_snapshot + data_freshness" +``` + +--- + +### Task 8: Query tools — `metric_series`, `sleep_summary`, `workout_summary` + +**Files:** +- Create: `noop-cloud/src/tools/query.ts` +- Modify: `noop-cloud/src/tools/index.ts` (call `registerQueryTools`) +- Test: `noop-cloud/test/tools-query.test.ts` + +**Interfaces:** +- Produces: `registerQueryTools(server, cfg)` plus testable fns `metricSeries(cfg, args)`, `sleepSummary(cfg, args)`, `workoutSummary(cfg, args)`. +- `metric_series` args `{ from, to, deviceId?, key? }` → `{ points: MetricPointRow[] }` (accepts any source incl. `oura-api`; keys like `ref_sleep_score`, `oura_readiness`). +- `sleep_summary` args `{ from, to }` → sessions with computed `durationMin`, per family. +- `workout_summary` args `{ from, to }` → workouts with `startIso`, per family. +- Date args are `YYYY-MM-DD`; validate with a zod regex. + +- [ ] **Step 1: Write the failing test** + +`test/tools-query.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; import { ingestNoopbak } from "../src/ingest.js"; +import { metricSeries, sleepSummary, workoutSummary } from "../src/tools/query.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/tools-query"); +const cfg = { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000 } as any; +beforeAll(() => { fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); const z = path.join(dataDir, "b.noopbak"); buildNoopbak(z); ingestNoopbak(fs.readFileSync(z), cfg); }); + +describe("query tools", () => { + it("metric_series returns Oura reference keys", () => { + const r = metricSeries(cfg, { from: "2026-06-10", to: "2026-06-13", deviceId: "oura-api", key: "oura_readiness" }); + expect(r.points.length).toBe(4); + expect(r.points[0].value).toBe(78); + }); + it("sleep_summary computes durationMin per family", () => { + const r = sleepSummary(cfg, { from: "2026-06-10", to: "2026-06-13" }); + expect(r.sessions.length).toBeGreaterThan(0); + expect(r.sessions[0].durationMin).toBeGreaterThan(0); + }); + it("workout_summary returns a running workout", () => { + const r = workoutSummary(cfg, { from: "2026-06-10", to: "2026-06-13" }); + expect(r.workouts.some((w) => w.sport === "running")).toBe(true); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- tools-query` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement `src/tools/query.ts`** + +```typescript +import { z } from "zod"; +import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; +import type { Config } from "../config.js"; +import { Mirror } from "../mirror.js"; + +const DAY = z.string().regex(/^\d{4}-\d{2}-\d{2}$/, "YYYY-MM-DD"); +const range = { from: DAY, to: DAY }; +const asTool = (obj: unknown) => ({ content: [{ type: "text" as const, text: JSON.stringify(obj, null, 2) }], structuredContent: obj as Record }); + +export function metricSeries(cfg: Config, args: { from: string; to: string; deviceId?: string; key?: string }) { + const m = new Mirror(cfg.mirrorPath); try { return { points: m.metricSeries(args) }; } finally { m.close(); } +} +export function sleepSummary(cfg: Config, args: { from: string; to: string }) { + const m = new Mirror(cfg.mirrorPath); + try { return { sessions: m.sleepSummary(args).map((s) => ({ ...s, startIso: new Date(s.startTs * 1000).toISOString(), durationMin: Math.round((s.endTs - s.startTs) / 60) })) }; } + finally { m.close(); } +} +export function workoutSummary(cfg: Config, args: { from: string; to: string }) { + const m = new Mirror(cfg.mirrorPath); + try { return { workouts: m.workoutSummary(args).map((w) => ({ ...w, startIso: new Date(w.startTs * 1000).toISOString(), durationMin: w.durationS ? Math.round(w.durationS / 60) : null })) }; } + finally { m.close(); } +} + +export function registerQueryTools(server: McpServer, cfg: Config): void { + server.registerTool("metric_series", { + title: "Metric series", + description: "Long-format metric points for a date range. Any source (incl. oura-api) and key (e.g. ref_sleep_score, oura_readiness).", + inputSchema: { ...range, deviceId: z.string().optional(), key: z.string().optional() }, + annotations: { readOnlyHint: true, openWorldHint: false }, + }, async (a) => asTool(metricSeries(cfg, a))); + + server.registerTool("sleep_summary", { + title: "Sleep summary", + description: "Sleep sessions in a date range with duration and efficiency, per source family.", + inputSchema: { ...range }, + annotations: { readOnlyHint: true, openWorldHint: false }, + }, async (a) => asTool(sleepSummary(cfg, a))); + + server.registerTool("workout_summary", { + title: "Workout summary", + description: "Workouts in a date range (sport, duration, energy, distance), per source family.", + inputSchema: { ...range }, + annotations: { readOnlyHint: true, openWorldHint: false }, + }, async (a) => asTool(workoutSummary(cfg, a))); +} +``` + +- [ ] **Step 4: Wire into `src/tools/index.ts`** + +Add `import { registerQueryTools } from "./query.js";` and call `registerQueryTools(server, cfg);` inside `registerTools`. + +- [ ] **Step 5: Run to verify pass** + +Run: `npm test -- tools-query && npm run build` +Expected: PASS (3 tests). + +- [ ] **Step 6: Commit** + +```bash +git add -A && git commit -m "feat: query tools metric_series + sleep_summary + workout_summary" +``` + +--- + +### Task 9: `compare_sources` (cross-source corroboration — G4) + +**Files:** +- Create: `noop-cloud/src/tools/compare.ts` +- Modify: `noop-cloud/src/tools/index.ts` +- Test: `noop-cloud/test/tools-compare.test.ts` + +**Interfaces:** +- Produces: `compareSources(cfg, args)` + `registerCompareSources(server, cfg)`. +- Args `{ from, to, metrics?: string[] }` (default metrics `["restingHr","avgHrv","totalSleepMin"]`). +- Returns `{ from, to, days: [{ day, metrics: { [metric]: { whoop?, oura?, apple?, spreadPct } } }] }` where `spreadPct = (max-min)/mean*100` over present family values, rounded; `null` families omitted. + +- [ ] **Step 1: Write the failing test** + +`test/tools-compare.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; import { ingestNoopbak } from "../src/ingest.js"; +import { compareSources } from "../src/tools/compare.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/tools-compare"); +const cfg = { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000 } as any; +beforeAll(() => { fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); const z = path.join(dataDir, "b.noopbak"); buildNoopbak(z); ingestNoopbak(fs.readFileSync(z), cfg); }); + +describe("compare_sources", () => { + it("puts WHOOP/Oura/Apple restingHr side by side with a spread", () => { + const r = compareSources(cfg, { from: "2026-06-13", to: "2026-06-13", metrics: ["restingHr"] }); + const day = r.days[0]; + expect(day.metrics.restingHr.whoop).toBe(51); + expect(day.metrics.restingHr.oura).toBe(53); + expect(day.metrics.restingHr.apple).toBe(54); + expect(day.metrics.restingHr.spreadPct).toBeGreaterThan(0); + }); + it("omits families with a null value (Apple avgHrv)", () => { + const r = compareSources(cfg, { from: "2026-06-13", to: "2026-06-13", metrics: ["avgHrv"] }); + expect(r.days[0].metrics.avgHrv.apple).toBeUndefined(); + expect(r.days[0].metrics.avgHrv.whoop).toBe(70); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- tools-compare` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement `src/tools/compare.ts`** + +```typescript +import { z } from "zod"; +import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; +import type { Config } from "../config.js"; +import { Mirror, DailyMetricRow } from "../mirror.js"; + +const DAY = z.string().regex(/^\d{4}-\d{2}-\d{2}$/, "YYYY-MM-DD"); +const DEFAULT_METRICS = ["restingHr", "avgHrv", "totalSleepMin"] as const; +const asTool = (obj: unknown) => ({ content: [{ type: "text" as const, text: JSON.stringify(obj, null, 2) }], structuredContent: obj as Record }); + +export function compareSources(cfg: Config, args: { from: string; to: string; metrics?: string[] }) { + const metrics = (args.metrics && args.metrics.length ? args.metrics : [...DEFAULT_METRICS]); + const m = new Mirror(cfg.mirrorPath); + try { + const rows = m.dailyMetrics({ from: args.from, to: args.to }); + const byDay = new Map(); + for (const r of rows) { const a = byDay.get(r.day) ?? []; a.push(r); byDay.set(r.day, a); } + const days = [...byDay.entries()].sort((a, b) => a[0].localeCompare(b[0])).map(([day, drows]) => { + const metricsOut: Record = {}; + for (const metric of metrics) { + const cell: Record = {}; + for (const r of drows) { const v = (r as any)[metric]; if (v !== null && v !== undefined) cell[r.family] = v; } + const vals = Object.values(cell); + const spreadPct = vals.length >= 2 ? Math.round(((Math.max(...vals) - Math.min(...vals)) / (vals.reduce((s, x) => s + x, 0) / vals.length)) * 1000) / 10 : 0; + metricsOut[metric] = { ...cell, spreadPct }; + } + return { day, metrics: metricsOut }; + }); + return { from: args.from, to: args.to, days }; + } finally { m.close(); } +} + +export function registerCompareSources(server: McpServer, cfg: Config): void { + server.registerTool("compare_sources", { + title: "Compare sources", + description: "Per-day WHOOP vs Oura vs Apple side by side for chosen metrics, with a spread %. The corroboration workhorse.", + inputSchema: { from: DAY, to: DAY, metrics: z.array(z.string()).optional().describe("dailyMetric columns, e.g. restingHr, avgHrv, totalSleepMin, steps.") }, + annotations: { readOnlyHint: true, openWorldHint: false }, + }, async (a) => asTool(compareSources(cfg, a))); +} +``` + +- [ ] **Step 4: Wire into `src/tools/index.ts`** — import + call `registerCompareSources(server, cfg)`. + +- [ ] **Step 5: Run to verify pass** + +Run: `npm test -- tools-compare && npm run build` +Expected: PASS (2 tests). + +- [ ] **Step 6: Commit** + +```bash +git add -A && git commit -m "feat: compare_sources — per-day WHOOP/Oura/Apple corroboration" +``` + +--- + +### Task 10: ChatGPT-shaped `search` + `fetch` + +**Files:** +- Create: `noop-cloud/src/tools/search-fetch.ts` +- Modify: `noop-cloud/src/tools/index.ts` +- Test: `noop-cloud/test/tools-search.test.ts` + +**Interfaces:** +- Produces: `search(cfg, { query })` → `{ results: [{ id, title, url }] }` and `fetch(cfg, { id })` → `{ id, title, text, url, metadata }`, plus `registerSearchFetch(server, cfg)`. +- **Contract (ChatGPT Deep Research):** these two names + shapes are fixed. `id` format = `day:YYYY-MM-DD`. `search` matches a `YYYY-MM-DD` substring in the query, else returns the most recent days. `fetch` returns a text digest of that day across sources. `url` is a stable `noop-cloud://day/` placeholder (no external fetch). + +- [ ] **Step 1: Write the failing test** + +`test/tools-search.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; import { ingestNoopbak } from "../src/ingest.js"; +import { search, fetch as fetchDay } from "../src/tools/search-fetch.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/tools-search"); +const cfg = { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000 } as any; +beforeAll(() => { fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); const z = path.join(dataDir, "b.noopbak"); buildNoopbak(z); ingestNoopbak(fs.readFileSync(z), cfg); }); + +describe("search/fetch", () => { + it("search returns day results with id/title/url", () => { + const r = search(cfg, { query: "2026-06-12" }); + expect(r.results[0].id).toBe("day:2026-06-12"); + expect(r.results[0]).toHaveProperty("title"); + expect(r.results[0]).toHaveProperty("url"); + }); + it("fetch returns a text digest for a day", () => { + const r = fetchDay(cfg, { id: "day:2026-06-13" }); + expect(r.id).toBe("day:2026-06-13"); + expect(r.text).toMatch(/resting/i); + expect(r.text.length).toBeGreaterThan(20); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- tools-search` +Expected: FAIL — module not found. + +- [ ] **Step 3: Implement `src/tools/search-fetch.ts`** + +```typescript +import { z } from "zod"; +import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; +import type { Config } from "../config.js"; +import { Mirror } from "../mirror.js"; + +const urlFor = (day: string) => `noop-cloud://day/${day}`; + +export function search(cfg: Config, args: { query: string }) { + const m = new Mirror(cfg.mirrorPath); + try { + const to = m.latestDataDay(); if (!to) return { results: [] }; + const from = new Date(new Date(`${to}T00:00:00Z`).getTime() - 30 * 86_400_000).toISOString().slice(0, 10); + const daysSet = new Set(m.dailyMetrics({ from, to }).map((r) => r.day)); + let days = [...daysSet].sort().reverse(); + const m2 = /\d{4}-\d{2}-\d{2}/.exec(args.query); + if (m2) days = days.filter((d) => d === m2[0]); + return { results: days.slice(0, 20).map((d) => ({ id: `day:${d}`, title: `Biometrics for ${d}`, url: urlFor(d) })) }; + } finally { m.close(); } +} + +export function fetch(cfg: Config, args: { id: string }) { + const day = args.id.replace(/^day:/, ""); + const m = new Mirror(cfg.mirrorPath); + try { + const rows = m.dailyMetrics({ from: day, to: day }); + const lines = rows.map((r) => `${r.family}: resting HR ${r.restingHr ?? "—"}, HRV ${r.avgHrv ?? "—"}, sleep ${r.totalSleepMin ?? "—"} min, recovery ${r.recovery ?? "—"}, strain ${r.strain ?? "—"}, steps ${r.steps ?? "—"}`); + const text = rows.length ? `Biometrics for ${day}\n${lines.join("\n")}` : `No data for ${day}.`; + return { id: args.id, title: `Biometrics for ${day}`, text, url: urlFor(day), metadata: { day, sources: rows.map((r) => r.family) } }; + } finally { m.close(); } +} + +export function registerSearchFetch(server: McpServer, cfg: Config): void { + server.registerTool("search", { + title: "Search", + description: "ChatGPT Deep Research search: find day-records. Returns {id,title,url}. Use a YYYY-MM-DD in the query to target a day.", + inputSchema: { query: z.string() }, + annotations: { readOnlyHint: true, openWorldHint: false }, + }, async (a) => { const r = search(cfg, a); return { content: [{ type: "text", text: JSON.stringify(r) }], structuredContent: r }; }); + + server.registerTool("fetch", { + title: "Fetch", + description: "ChatGPT Deep Research fetch: full text of a record by id (e.g. day:2026-06-13).", + inputSchema: { id: z.string() }, + annotations: { readOnlyHint: true, openWorldHint: false }, + }, async (a) => { const r = fetch(cfg, a); return { content: [{ type: "text", text: JSON.stringify(r) }], structuredContent: r }; }); +} +``` + +- [ ] **Step 4: Wire into `src/tools/index.ts`** — import + call `registerSearchFetch(server, cfg)`. + +- [ ] **Step 5: Run all tests + build** + +Run: `npm test && npm run build` +Expected: PASS (all suites). + +- [ ] **Step 6: Commit** + +```bash +git add -A && git commit -m "feat: ChatGPT-shaped search + fetch tools" +``` + +--- + +### Task 11: Fly.io deploy config + README + +**Files:** +- Create: `noop-cloud/Dockerfile` +- Create: `noop-cloud/.dockerignore` +- Create: `noop-cloud/fly.toml` +- Create: `noop-cloud/README.md` + +**Interfaces:** none (deploy artifacts). Deliverable: `docker build` succeeds locally; `fly.toml` declares one machine + one volume at `/data` + autostop. + +- [ ] **Step 1: Write the `Dockerfile`** + +```dockerfile +FROM node:20-slim AS build +WORKDIR /app +RUN apt-get update && apt-get install -y python3 build-essential && rm -rf /var/lib/apt/lists/* +COPY package.json package-lock.json* ./ +RUN npm install +COPY tsconfig.json ./ +COPY src ./src +RUN npm run build + +FROM node:20-slim +WORKDIR /app +ENV NODE_ENV=production +COPY package.json package-lock.json* ./ +RUN apt-get update && apt-get install -y python3 build-essential && npm install --omit=dev && apt-get purge -y python3 build-essential && apt-get autoremove -y && rm -rf /var/lib/apt/lists/* +COPY --from=build /app/dist ./dist +ENV DATA_DIR=/data +EXPOSE 8080 +CMD ["node", "dist/server.js"] +``` + +- [ ] **Step 2: Write `.dockerignore`** + +``` +node_modules +dist +.env +data +test +.git +``` + +- [ ] **Step 3: Write `fly.toml`** (replace `` at deploy time) + +```toml +app = "noop-cloud" +primary_region = "iad" + +[build] + +[env] + DATA_DIR = "/data" + PORT = "8080" + +[[mounts]] + source = "noop_data" + destination = "/data" + +[http_service] + internal_port = 8080 + force_https = true + auto_stop_machines = "stop" + auto_start_machines = true + min_machines_running = 0 + +[[vm]] + size = "shared-cpu-1x" + memory = "512mb" + +[checks.health] + type = "http" + port = 8080 + method = "get" + path = "/healthz" + interval = "30s" + timeout = "5s" +``` + +- [ ] **Step 4: Write `README.md`** (self-host guide) + +````markdown +# noop-cloud + +A self-hostable remote MCP server that mirrors a NOOP health database and exposes +read-only biometrics tools to Claude, ChatGPT, and agents over Streamable HTTP. + +## Deploy (Fly.io) + +```bash +fly launch --no-deploy --name # edit fly.toml app name +fly volumes create noop_data --size 1 --region iad +fly secrets set RO_TOKEN=$(openssl rand -hex 32) RW_TOKEN=$(openssl rand -hex 32) +fly deploy +``` + +Save the two tokens. `RW_TOKEN` uploads data; `RO_TOKEN` is for read clients (Claude Code, cron). + +## Upload data (iOS Shortcut, no app changes) + +In NOOP: **Settings → Backup & Sync → Back up now** to write a `.noopbak`. Then an iOS Shortcut: +"Get File → Get Contents of URL": `POST https://.fly.dev/ingest`, header +`Authorization: Bearer `, request body = the file. + +## Connect Claude Code + +```bash +claude mcp add --transport http noop-cloud https://.fly.dev/mcp \ + --header "Authorization: Bearer " +``` + +Then ask: "call data_freshness, then health_snapshot for the last 7 days." + +## Tools + +`data_freshness`, `health_snapshot`, `metric_series`, `sleep_summary`, `workout_summary`, +`compare_sources`, plus ChatGPT Deep Research `search`/`fetch`. Prompts: `morning_report`, +`corroborate_sources`, `find_messy_data`. All read-only in this phase. +```` + +- [ ] **Step 5: Verify the image builds** + +Run: `cd /Users/vk/VKDEV/NOOP/noop-cloud && docker build -t noop-cloud .` +Expected: build succeeds (better-sqlite3 compiles under python3/build-essential). + +- [ ] **Step 6: Commit** + +```bash +git add -A && git commit -m "chore: Fly.io deploy config (Dockerfile, fly.toml) + README" +``` + +--- + +### Task 12: Morning report — GitHub Actions cron + local dry-run + +**Files:** +- Create: `noop-cloud/scripts/report.mjs` +- Create: `noop-cloud/.github/workflows/morning-report.yml` +- Modify: `noop-cloud/README.md` (report section) + +**Interfaces:** +- Produces: `scripts/report.mjs` — reads env `ANTHROPIC_API_KEY`, `NOOP_CLOUD_URL`, `NOOP_RO_TOKEN`, `NTFY_TOPIC`; calls the Anthropic Messages API with the MCP connector (ro token, tool-allowlist), extracts the text, POSTs it to `https://ntfy.sh/`. Exits non-zero on API error. `--dry-run` prints instead of pushing. +- Cron workflow runs it daily and on manual dispatch. + +- [ ] **Step 1: Write `scripts/report.mjs`** + +```javascript +// Node 20+. Generates the morning report via the Anthropic Messages API MCP connector, pushes to ntfy. +const { ANTHROPIC_API_KEY, NOOP_CLOUD_URL, NOOP_RO_TOKEN, NTFY_TOPIC } = process.env; +const dryRun = process.argv.includes("--dry-run"); +if (!ANTHROPIC_API_KEY || !NOOP_CLOUD_URL || !NOOP_RO_TOKEN) { console.error("missing env"); process.exit(2); } + +const res = await fetch("https://api.anthropic.com/v1/messages", { + method: "POST", + headers: { + "x-api-key": ANTHROPIC_API_KEY, + "anthropic-version": "2023-06-01", + "anthropic-beta": "mcp-client-2025-11-20", + "content-type": "application/json", + }, + body: JSON.stringify({ + model: "claude-sonnet-5", + max_tokens: 1024, + messages: [{ role: "user", content: "Write my morning health report. Call data_freshness first and state the mirror age; if it is older than 36 hours, say so plainly. Then summarize the last 3 days (recovery/strain, sleep, resting HR + HRV trend, workouts) and flag any source disagreements via compare_sources. Under 200 words. No medical advice." }], + mcp_servers: [{ + type: "url", url: `${NOOP_CLOUD_URL.replace(/\/$/, "")}/mcp`, name: "noop-cloud", + authorization_token: NOOP_RO_TOKEN, + tool_configuration: { enabled: true, allowed_tools: ["data_freshness", "health_snapshot", "sleep_summary", "workout_summary", "compare_sources", "metric_series"] }, + }], + }), +}); +if (!res.ok) { console.error("anthropic error", res.status, await res.text()); process.exit(1); } +const data = await res.json(); +const text = (data.content ?? []).filter((b) => b.type === "text").map((b) => b.text).join("\n").trim() || "(empty report)"; + +if (dryRun || !NTFY_TOPIC) { console.log(text); process.exit(0); } +const push = await fetch(`https://ntfy.sh/${NTFY_TOPIC}`, { method: "POST", headers: { Title: "NOOP morning report", Priority: "default" }, body: text }); +if (!push.ok) { console.error("ntfy error", push.status); process.exit(1); } +console.log("pushed report to ntfy"); +``` + +- [ ] **Step 2: Write `.github/workflows/morning-report.yml`** + +```yaml +name: morning-report +on: + schedule: + - cron: "0 13 * * *" # 13:00 UTC daily — adjust to your morning; also wakes the Fly machine + workflow_dispatch: {} +jobs: + report: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: { node-version: "20" } + - run: node scripts/report.mjs + env: + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + NOOP_CLOUD_URL: ${{ secrets.NOOP_CLOUD_URL }} + NOOP_RO_TOKEN: ${{ secrets.NOOP_RO_TOKEN }} + NTFY_TOPIC: ${{ secrets.NTFY_TOPIC }} +``` + +- [ ] **Step 3: Document + local dry-run test** + +Add to `README.md`: + +````markdown +## Morning report + +A GitHub Actions cron (`.github/workflows/morning-report.yml`) calls the Anthropic Messages API +with this server as an MCP connector (read-only token, tool-allowlisted) and pushes the result to +[ntfy](https://ntfy.sh). Set repo secrets `ANTHROPIC_API_KEY`, `NOOP_CLOUD_URL`, `NOOP_RO_TOKEN`, +`NTFY_TOPIC` and subscribe to the topic in the ntfy app. + +Local dry-run (prints, no push): + +```bash +ANTHROPIC_API_KEY=… NOOP_CLOUD_URL=https://.fly.dev NOOP_RO_TOKEN=… node scripts/report.mjs --dry-run +``` +```` + +Run the dry-run against the deployed server (or a local `npm run dev` with an ingested fixture) and confirm a coherent report prints. + +- [ ] **Step 4: Commit** + +```bash +git add -A && git commit -m "feat: morning report via GitHub Actions cron + Anthropic MCP connector + ntfy" +``` + +--- + +## Self-Review + +**Spec coverage (§5 Phase 1):** +- Fly app `/ingest` + `/mcp` + `/healthz` → Tasks 1, 5, 6. ✅ +- Upload via iOS Shortcut (no app code) → Task 11 README documents the Shortcut against `/ingest`. ✅ +- Read tools `health_snapshot`, `metric_series`, `sleep_summary`, `workout_summary`, `data_freshness`, `compare_sources` → Tasks 7–9. ✅ +- ChatGPT `search`/`fetch` → Task 10. ✅ +- Prompts `morning_report`, `corroborate_sources`, `find_messy_data` → Task 6. ✅ +- Auth: two static bearer tokens ro/rw, server-side → Task 2; `/ingest` rw, `/mcp` ro → Tasks 5–6. ✅ +- Morning report: GitHub Actions cron → Messages API MCP connector (ro, allowlisted) → ntfy → Task 12. ✅ +- Fly single-machine + volume + WAL + autostop → Task 11 `fly.toml`. ✅ +- Two SQLite files (`mirror.sqlite` RO, `server.sqlite` for ingest log) → Tasks 4, 5. ✅ +- `/ingest` validation (size cap, magic bytes, quick_check, foreign-db reject, atomic swap, no-corruption-on-failure) → Task 5. ✅ (spec §6 security posture) +- Multi-source awareness (deviceId families incl. oura-api) → Task 4 `sourceFamily` + every tool. ✅ + +**Deferred to later phases (correctly absent):** OAuth 2.1 shim (Phase 1.5), in-app upload button (1.5), write/staging tools + `server.sqlite` staging tables (Phase 2), sync-back `/edits` (Phase 3). Token-hashing-in-DB is deferred (env tokens for MVP) — noted in Task 2 as a Phase-1.5 upgrade. + +**Placeholder scan:** No `TBD`/`handle edge cases`/prose-only steps — every code step carries runnable code. The one explicit verification note (Task 6 Step: confirm SDK import paths) is a real check, not a placeholder. + +**Type consistency:** `Config` (Task 1) is consumed unchanged everywhere. `Mirror` methods (Task 4) match tool call sites (Tasks 7–10). `sourceFamily`/`Family` used consistently. `ingestNoopbak`/`latestIngest`/`IngestError` (Task 5) match their test + `data_freshness` call sites. Tool handler return shape `{ content:[{type:"text",text}], structuredContent }` is uniform. Registry `register(server, cfg)` naming is consistent across Tasks 7–10 and `tools/index.ts`. + +**Known follow-ups to flag during execution:** (1) verify the exact `better-sqlite3` prebuilt binding matches the Fly Node-20 image (else `npm rebuild`); (2) confirm `@modelcontextprotocol/sdk@1.29.0` streamable-HTTP stateless option name (`sessionIdGenerator: undefined`) against the installed package before relying on the `/mcp` test; (3) `claude-sonnet-5` model id in `report.mjs` — confirm current id at build time. + +--- + +## Execution Handoff + +Plan complete and saved to `docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md`. Two execution options: + +**1. Subagent-Driven (recommended)** — I dispatch a fresh subagent per task, review between tasks, fast iteration. + +**2. Inline Execution** — Execute tasks in this session using executing-plans, batch execution with checkpoints. + +Which approach? From 72f73df14c7f7c023fa480b69af7051c4a5cee12 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 22:23:37 -0700 Subject: [PATCH 040/171] docs(noop-cloud plan): pin better-sqlite3 12.11.1 (Node 26 ABI verified) --- docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md b/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md index 00db1ddcf1..a5a485f30a 100644 --- a/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md +++ b/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md @@ -11,7 +11,7 @@ ## Global Constraints - **MCP SDK pinned to exactly `1.29.0`** (`"@modelcontextprotocol/sdk": "1.29.0"`, no `^`). Do NOT use the 2.0.0-beta line. Rationale: the 2026-07-28 stateless spec revision is still an RC; 1.29.x is the supported stable line. -- **`better-sqlite3` pinned exact** (`"better-sqlite3": "11.10.0"`, no `^`) — native binding; a floating range risks an ABI mismatch on the Fly builder. `npm rebuild better-sqlite3` is the fix if the prebuilt binding mismatches Node 20. +- **`better-sqlite3` pinned exact** (`"better-sqlite3": "12.11.1"`, no `^`) — native binding; a floating range risks an ABI mismatch on the Fly builder. Verified locally: 12.11.1 installs + loads on this machine's **Node 26 (ABI 147)**, and 12.x still ships a **Node-20 prebuilt** for the `node:20-slim` Docker image. `npm rebuild better-sqlite3` is the fix if a prebuilt binding ever mismatches the runtime Node. - **Stateless MCP transport**: every `/mcp` request constructs a fresh `McpServer` + `StreamableHTTPServerTransport({ sessionIdGenerator: undefined })`. No session store — a Fly autostop between requests must not break a session. - **The mirror is opened READ-ONLY always** (`new Database(path, { readonly: true, fileMustExist: true })`). Server code never writes to `mirror.sqlite`. Server-owned state lives only in `server.sqlite`. - **`deviceId` is a *source* discriminator, not a physical device.** The same calendar day legitimately has rows under multiple `deviceId`s. Source families for Phase 1: WHOOP = any id that is NOT `apple-health` and NOT `oura-api` (straps use ids like `my-whoop`, plus derived `-noop`); Oura = `oura-api`; Apple = `apple-health`. Never assume one row per day. @@ -65,7 +65,7 @@ Create `package.json`: "dependencies": { "@modelcontextprotocol/sdk": "1.29.0", "adm-zip": "0.5.16", - "better-sqlite3": "11.10.0", + "better-sqlite3": "12.11.1", "express": "^4.21.2", "zod": "^3.23.8" }, From 1d9e1819fe0bad53aa88ab717ed3786361644dec Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Fri, 10 Jul 2026 23:44:00 -0700 Subject: [PATCH 041/171] Fix CI: tracked OuraConfig.xcconfig wrapper for the optional Oura secrets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit App-build CI has failed since the Oura cloud import landed on main: project.yml pointed configFiles at the gitignored Strand/Oura/ OuraSecrets.xcconfig, so xcodegen's spec validation failed on any checkout without local credentials. configFiles now reference the tracked OuraConfig.xcconfig, which optionally includes OuraSecrets.xcconfig (#include? — skipped when absent). Without secrets the OURA_* build settings stay undefined, the Info.plist keys resolve to empty, and OuraCredentials.fromBundle returns nil — the Oura Cloud lane stays inert. Same fix as on the oura-cloud-import PR branch. --- Strand/Oura/OuraConfig.xcconfig | 5 +++++ project.yml | 17 +++++++++-------- 2 files changed, 14 insertions(+), 8 deletions(-) create mode 100644 Strand/Oura/OuraConfig.xcconfig diff --git a/Strand/Oura/OuraConfig.xcconfig b/Strand/Oura/OuraConfig.xcconfig new file mode 100644 index 0000000000..79c65a4daf --- /dev/null +++ b/Strand/Oura/OuraConfig.xcconfig @@ -0,0 +1,5 @@ +// Tracked wrapper so a clean checkout (no local credentials) still generates and builds. +// OuraSecrets.xcconfig is gitignored; when absent, the optional include is skipped, the +// OURA_* build settings stay undefined, the Info.plist keys resolve to empty strings, and +// OuraCredentials.fromBundle returns nil — the whole Oura Cloud lane is inert. +#include? "OuraSecrets.xcconfig" diff --git a/project.yml b/project.yml index 7556dc1eee..73fd703ec2 100644 --- a/project.yml +++ b/project.yml @@ -76,12 +76,13 @@ targets: # The liquid Today is now cross-platform: its iOS-only chrome (topBarTrailing / navigationBarTitleDisplayMode # / presentationCompactAdaptation / two-param onChange / sensoryFeedback) is guarded, so the mac split-view # shell hosts LiquidTodayView too. All Liquid/* now compile into the mac target. - # Oura BYO-app OAuth credentials, sourced from the untracked Strand/Oura/OuraSecrets.xcconfig - # (gitignored; template at OuraSecrets.example.xcconfig). Resolves the $(OURA_*) references in - # the info.properties below. + # Oura BYO-app OAuth credentials. OuraConfig.xcconfig is a tracked wrapper that optionally + # includes the untracked, gitignored Strand/Oura/OuraSecrets.xcconfig (template at + # OuraSecrets.example.xcconfig) — absent secrets leave the $(OURA_*) references in the + # info.properties below empty, which disables the Oura Cloud lane at runtime. configFiles: - Debug: Strand/Oura/OuraSecrets.xcconfig - Release: Strand/Oura/OuraSecrets.xcconfig + Debug: Strand/Oura/OuraConfig.xcconfig + Release: Strand/Oura/OuraConfig.xcconfig info: path: Strand/Resources/Info.plist properties: @@ -196,10 +197,10 @@ targets: - "Resources/Info.plist" - "Resources/NOOP.entitlements" - path: StrandiOSShared - # Same untracked xcconfig as the macOS target (Strand/Oura/ is shared, not per-platform). + # Same tracked wrapper as the macOS target (Strand/Oura/ is shared, not per-platform). configFiles: - Debug: Strand/Oura/OuraSecrets.xcconfig - Release: Strand/Oura/OuraSecrets.xcconfig + Debug: Strand/Oura/OuraConfig.xcconfig + Release: Strand/Oura/OuraConfig.xcconfig info: path: StrandiOS/Resources/Info.plist properties: From b340fba3c715f995219bbf49dcc087d9a65fec13 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sat, 11 Jul 2026 19:08:10 -0700 Subject: [PATCH 042/171] docs(noop-cloud plan): service-level health check (review finding) --- .../plans/2026-07-10-noop-cloud-1-mvp-server.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md b/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md index a5a485f30a..cb5ce591ba 100644 --- a/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md +++ b/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md @@ -394,7 +394,7 @@ import AdmZip from "adm-zip"; import fs from "node:fs"; const DAYS = ["2026-06-10", "2026-06-11", "2026-06-12", "2026-06-13"]; -const tsOf = (day: string, h = 3) => Math.floor(new Date(`${day}T0${h}:00:00Z`).getTime() / 1000); +const tsOf = (day: string, h = 3) => Math.floor(new Date(`${day}T${String(h).padStart(2, "0")}:00:00Z`).getTime() / 1000); export function buildMirrorSqlite(target: string): void { if (fs.existsSync(target)) fs.rmSync(target); @@ -1521,13 +1521,12 @@ primary_region = "iad" size = "shared-cpu-1x" memory = "512mb" -[checks.health] - type = "http" - port = 8080 - method = "get" - path = "/healthz" - interval = "30s" - timeout = "5s" + [[http_service.checks]] + grace_period = "10s" + interval = "30s" + method = "GET" + timeout = "5s" + path = "/healthz" ``` - [ ] **Step 4: Write `README.md`** (self-host guide) From 0b1881583eea8f359340353357b2518462844e27 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sat, 11 Jul 2026 19:11:15 -0700 Subject: [PATCH 043/171] fix(android/i18n): German strings for the compact widget GermanLocalizationTest.germanResourcesResolveForEveryLocalizedKey fails on main since the compact widget landed (#77): widget_compact_name and widget_compact_description exist in values/ but not values-de/. Adds both, reusing the app's established German terms (Ruhe/Ladung/Anstrengung, Live-Herzfrequenz, Band-Akku) from widget_description. --- android/app/src/main/res/values-de/strings.xml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/android/app/src/main/res/values-de/strings.xml b/android/app/src/main/res/values-de/strings.xml index 1085235c03..77cdcc9bc1 100644 --- a/android/app/src/main/res/values-de/strings.xml +++ b/android/app/src/main/res/values-de/strings.xml @@ -3,6 +3,8 @@ Ruhe, Ladung und Anstrengung von heute, mit Live-Herzfrequenz und Band-Akku auf einen Blick. + NOOP Kompakt + Kompaktes Widget für Ruhe, Ladung und Anstrengung, mit Live-Herzfrequenz und Band-Akku. Das NOOP-Widget konnte nicht gezeichnet werden. Es erholt sich beim nächsten Update. Heute From c244b559dce929d31213669ed0c25b4a44657163 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sat, 11 Jul 2026 19:19:05 -0700 Subject: [PATCH 044/171] docs(noop-cloud plan): report script max_tokens + stop_reason guard (review finding) --- docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md b/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md index cb5ce591ba..cda0b4ecef 100644 --- a/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md +++ b/docs/superpowers/plans/2026-07-10-noop-cloud-1-mvp-server.md @@ -1612,7 +1612,7 @@ const res = await fetch("https://api.anthropic.com/v1/messages", { }, body: JSON.stringify({ model: "claude-sonnet-5", - max_tokens: 1024, + max_tokens: 4096, messages: [{ role: "user", content: "Write my morning health report. Call data_freshness first and state the mirror age; if it is older than 36 hours, say so plainly. Then summarize the last 3 days (recovery/strain, sleep, resting HR + HRV trend, workouts) and flag any source disagreements via compare_sources. Under 200 words. No medical advice." }], mcp_servers: [{ type: "url", url: `${NOOP_CLOUD_URL.replace(/\/$/, "")}/mcp`, name: "noop-cloud", @@ -1623,6 +1623,7 @@ const res = await fetch("https://api.anthropic.com/v1/messages", { }); if (!res.ok) { console.error("anthropic error", res.status, await res.text()); process.exit(1); } const data = await res.json(); +if (data.stop_reason && data.stop_reason !== "end_turn") { console.error("anthropic stop_reason", data.stop_reason); process.exit(1); } const text = (data.content ?? []).filter((b) => b.type === "text").map((b) => b.text).join("\n").trim() || "(empty report)"; if (dryRun || !NTFY_TOPIC) { console.log(text); process.exit(0); } From 091ebc51f8ea5438f9c2c4aa86222af2f069ef71 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 11:20:01 -0700 Subject: [PATCH 045/171] =?UTF-8?q?docs:=20noop-cloud=20Phase-2=20plan=20?= =?UTF-8?q?=E2=80=94=20gated=20writes=20+=20append-only=20edit=20journal?= =?UTF-8?q?=20(8=20tasks,=20TDD)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit propose_edit (before-snapshot + human diff) / confirm/reject/undo (rw-only, scope-threaded tool visibility) / overlay-aware reads with provenance / GET /edits?since= for Phase-3 phone pull. Mirror stays pristine by construction. Co-Authored-By: Claude Fable 5 --- .../2026-07-11-noop-cloud-2-gated-writes.md | 1137 +++++++++++++++++ 1 file changed, 1137 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-11-noop-cloud-2-gated-writes.md diff --git a/docs/superpowers/plans/2026-07-11-noop-cloud-2-gated-writes.md b/docs/superpowers/plans/2026-07-11-noop-cloud-2-gated-writes.md new file mode 100644 index 0000000000..99f893be2f --- /dev/null +++ b/docs/superpowers/plans/2026-07-11-noop-cloud-2-gated-writes.md @@ -0,0 +1,1137 @@ +# noop-cloud Phase 2 — Gated Writes + Edit Journal Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** AI can *propose* data corrections (with a human-readable diff and a snapshot of the original), a human *confirms* with the rw credential, confirmed edits land in an append-only journal + overlay that read tools reflect — and the pristine mirror is never touched. + +**Architecture:** All write state lives in `server.sqlite` (two new tables: `proposal`, `editJournal`). `propose_edit` validates the payload per edit-kind, captures a **before-snapshot** from the mirror, renders a human diff, and stages a proposal. `confirm_edit` (rw-only, invisible to ro callers) moves it to the append-only journal; `undo_edit` appends a reversal entry (the journal never mutates). Read tools consult a computed **overlay** derived from active journal entries. A new `GET /edits?since=` endpoint exposes the sequenced journal for Phase-3 phone pull. Scope is threaded per request: the stateless `/mcp` route detects ro-vs-rw from the bearer token and builds the MCP server with only the tools that scope may see. + +**Tech Stack:** unchanged — TypeScript 5.6, Express 4, `@modelcontextprotocol/sdk` pinned **1.29.0**, `better-sqlite3` pinned **12.11.1**, zod, vitest. Repo `/Users/vk/VKDEV/NOOP/noop-cloud` (Phase 1 deployed at `https://vk-noop-cloud.fly.dev`; 41 tests green). + +## Global Constraints + +- **The mirror is never written.** Overlay + journal live only in `server.sqlite`. Any SQL against `mirror.sqlite` stays read-only (`{ readonly: true, fileMustExist: true }`). +- **Scope model (spec §5/§6):** `propose_edit`, `list_pending`, `edit_journal` available to **ro** (routines may stage, never apply). `confirm_edit`, `reject_edit`, `undo_edit` **registered only for rw callers** — invisible in ro `tools/list`, not merely rejected. rw implies ro. +- **Journal is append-only and idempotent by edit id.** Undo appends a reversal row (`kind='undo'`) and marks the original's `undoneBySeq`; rows are never updated otherwise, never deleted. `seq` (AUTOINCREMENT) is the Phase-3 pull cursor. +- **Every proposal carries `beforeJSON`** — the affected row(s) as they exist at propose time — so "what the data originally said" is permanently auditable. +- **Edit kinds (exactly these six):** `fix_workout`, `delete_workout`, `add_workout`, `adjust_sleep_bounds`, `delete_metric_point`, `set_baseline_note`. `add_workout` rows carry `deviceId: "noop-cloud"` (the server's own source namespace — spec §5 Phase 3 rule). +- **Annotations:** propose/list/journal → `readOnlyHint` false/true as appropriate but `destructiveHint: false`; `confirm_edit` and `undo_edit` → `destructiveHint: true`. (These drive client approval UI; enforcement stays server-side.) +- **Guard pattern (house style):** tools that read the mirror keep the `!fs.existsSync(cfg.mirrorPath)` → `notIngested: true` guard. Proposing an edit against a missing mirror returns a structured error, not a throw. +- **Overlay honesty:** overlay changes what `sleep_summary` / `workout_summary` / `metric_series` return (with `edited: true` / `added: true` provenance), but `dailyMetric`-derived numbers (`health_snapshot`, `compare_sources`, `totalSleepMin`) still come from the phone's own rollups — corrections reach those only after Phase 3 applies the edit on-phone and the next upload returns. Tool descriptions must say this plainly. +- **No secrets in git**; same pins; deploy via `fly deploy --ha=false` at the end. + +--- + +### Task 1: `serverdb.ts` (shared server-DB opener + staging schema) and `staging.ts` (state machine) + +**Files:** +- Create: `noop-cloud/src/serverdb.ts` +- Create: `noop-cloud/src/staging.ts` +- Modify: `noop-cloud/src/ingest.ts` (import `openServerDb` from serverdb.js; delete its local copy) +- Test: `noop-cloud/test/staging.test.ts` + +**Interfaces:** +- Consumes: `Config` (`serverDbPath`). +- Produces (serverdb.ts): `openServerDb(cfg: Pick): Database.Database` — WAL, `CREATE TABLE IF NOT EXISTS` for `ingestLog` (unchanged shape), `proposal`, `editJournal`. +- Produces (staging.ts): + - `interface ProposalRow { id: string; kind: string; payloadJSON: string; rationale: string; beforeJSON: string | null; diffText: string; status: "pending"|"confirmed"|"rejected"; createdAt: number; resolvedAt: number | null }` + - `interface JournalRow { seq: number; editId: string; kind: string; payloadJSON: string; beforeJSON: string | null; rationale: string | null; appliedAt: number; undoneBySeq: number | null }` + - `createProposal(cfg, p: Omit): ProposalRow` + - `getProposal(cfg, id: string): ProposalRow | null` + - `listPending(cfg): ProposalRow[]` + - `resolveProposal(cfg, id: string, status: "confirmed"|"rejected"): ProposalRow | null` (only from `pending`; returns null if absent or already resolved) + - `appendJournal(cfg, e: { editId: string; kind: string; payloadJSON: string; beforeJSON: string | null; rationale: string | null }): number` — returns `seq`; UNIQUE(editId) makes double-apply throw (caller treats as idempotency signal) + - `journalSince(cfg, since: number): JournalRow[]` (seq > since, ascending) + - `activeEdits(cfg): JournalRow[]` — `undoneBySeq IS NULL AND kind != 'undo'`, ascending + - `markUndone(cfg, targetSeq: number, bySeq: number): void` + +- [ ] **Step 1: Write the failing test** + +`test/staging.test.ts`: + +```typescript +import { describe, it, expect, beforeEach } from "vitest"; +import fs from "node:fs"; import path from "node:path"; +import { createProposal, getProposal, listPending, resolveProposal, appendJournal, journalSince, activeEdits, markUndone } from "../src/staging.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/staging"); +const cfg = { serverDbPath: path.join(dataDir, "server.sqlite") } as any; +beforeEach(() => { fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); }); + +const draft = (id: string) => ({ id, kind: "delete_workout", payloadJSON: JSON.stringify({ deviceId: "my-whoop", startTs: 100, sport: "running" }), rationale: "duplicate", beforeJSON: JSON.stringify({ sport: "running" }), diffText: "- running workout @100" }); + +describe("staging state machine", () => { + it("propose → pending → confirm; not listed after", () => { + createProposal(cfg, draft("edit_a")); + expect(listPending(cfg).map((p) => p.id)).toEqual(["edit_a"]); + const r = resolveProposal(cfg, "edit_a", "confirmed"); + expect(r?.status).toBe("confirmed"); + expect(listPending(cfg)).toEqual([]); + }); + it("resolve is single-shot (second resolve returns null)", () => { + createProposal(cfg, draft("edit_b")); + expect(resolveProposal(cfg, "edit_b", "rejected")?.status).toBe("rejected"); + expect(resolveProposal(cfg, "edit_b", "confirmed")).toBeNull(); + }); + it("journal is append-only, sequenced, idempotent by editId", () => { + const s1 = appendJournal(cfg, { editId: "edit_c", kind: "delete_workout", payloadJSON: "{}", beforeJSON: null, rationale: null }); + const s2 = appendJournal(cfg, { editId: "edit_d", kind: "set_baseline_note", payloadJSON: "{}", beforeJSON: null, rationale: null }); + expect(s2).toBe(s1 + 1); + expect(() => appendJournal(cfg, { editId: "edit_c", kind: "delete_workout", payloadJSON: "{}", beforeJSON: null, rationale: null })).toThrow(); + expect(journalSince(cfg, s1).map((j) => j.seq)).toEqual([s2]); + }); + it("undo marks original inactive; undo rows never count as active", () => { + const s1 = appendJournal(cfg, { editId: "edit_e", kind: "delete_workout", payloadJSON: "{}", beforeJSON: null, rationale: null }); + const s2 = appendJournal(cfg, { editId: "undo_of_e", kind: "undo", payloadJSON: JSON.stringify({ targetSeq: s1 }), beforeJSON: null, rationale: null }); + markUndone(cfg, s1, s2); + expect(activeEdits(cfg)).toEqual([]); + expect(journalSince(cfg, 0).length).toBe(2); // history intact + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- staging` +Expected: FAIL — `Cannot find module '../src/staging.js'`. + +- [ ] **Step 3: Implement `src/serverdb.ts`** + +```typescript +import Database from "better-sqlite3"; +import type { Config } from "./config.js"; + +/** Single opener for the server-owned DB (never the mirror). All CREATEs are idempotent. */ +export function openServerDb(cfg: Pick): Database.Database { + const db = new Database(cfg.serverDbPath); + db.pragma("journal_mode = WAL"); + db.exec(` + CREATE TABLE IF NOT EXISTS ingestLog (id INTEGER PRIMARY KEY AUTOINCREMENT, receivedAt INTEGER, bytes INTEGER, latestDay TEXT); + CREATE TABLE IF NOT EXISTS proposal ( + id TEXT PRIMARY KEY, kind TEXT NOT NULL, payloadJSON TEXT NOT NULL, rationale TEXT NOT NULL, + beforeJSON TEXT, diffText TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending','confirmed','rejected')), + createdAt INTEGER NOT NULL, resolvedAt INTEGER); + CREATE TABLE IF NOT EXISTS editJournal ( + seq INTEGER PRIMARY KEY AUTOINCREMENT, editId TEXT NOT NULL UNIQUE, kind TEXT NOT NULL, + payloadJSON TEXT NOT NULL, beforeJSON TEXT, rationale TEXT, + appliedAt INTEGER NOT NULL, undoneBySeq INTEGER); + `); + return db; +} +``` + +- [ ] **Step 4: Implement `src/staging.ts`** + +```typescript +import type { Config } from "./config.js"; +import { openServerDb } from "./serverdb.js"; + +type C = Pick; + +export interface ProposalRow { + id: string; kind: string; payloadJSON: string; rationale: string; + beforeJSON: string | null; diffText: string; + status: "pending" | "confirmed" | "rejected"; createdAt: number; resolvedAt: number | null; +} +export interface JournalRow { + seq: number; editId: string; kind: string; payloadJSON: string; + beforeJSON: string | null; rationale: string | null; appliedAt: number; undoneBySeq: number | null; +} + +function withDb(cfg: C, fn: (db: ReturnType) => T): T { + const db = openServerDb(cfg); + try { return fn(db); } finally { db.close(); } +} + +export function createProposal(cfg: C, p: Omit): ProposalRow { + return withDb(cfg, (db) => { + const createdAt = Math.floor(Date.now() / 1000); + db.prepare(`INSERT INTO proposal (id, kind, payloadJSON, rationale, beforeJSON, diffText, status, createdAt) + VALUES (?,?,?,?,?,?, 'pending', ?)`) + .run(p.id, p.kind, p.payloadJSON, p.rationale, p.beforeJSON, p.diffText, createdAt); + return { ...p, status: "pending", createdAt, resolvedAt: null }; + }); +} +export function getProposal(cfg: C, id: string): ProposalRow | null { + return withDb(cfg, (db) => (db.prepare("SELECT * FROM proposal WHERE id = ?").get(id) as ProposalRow | undefined) ?? null); +} +export function listPending(cfg: C): ProposalRow[] { + return withDb(cfg, (db) => db.prepare("SELECT * FROM proposal WHERE status = 'pending' ORDER BY createdAt, id").all() as ProposalRow[]); +} +export function resolveProposal(cfg: C, id: string, status: "confirmed" | "rejected"): ProposalRow | null { + return withDb(cfg, (db) => { + const r = db.prepare("UPDATE proposal SET status = ?, resolvedAt = ? WHERE id = ? AND status = 'pending'") + .run(status, Math.floor(Date.now() / 1000), id); + if (r.changes === 0) return null; + return db.prepare("SELECT * FROM proposal WHERE id = ?").get(id) as ProposalRow; + }); +} +export function appendJournal(cfg: C, e: { editId: string; kind: string; payloadJSON: string; beforeJSON: string | null; rationale: string | null }): number { + return withDb(cfg, (db) => { + const r = db.prepare(`INSERT INTO editJournal (editId, kind, payloadJSON, beforeJSON, rationale, appliedAt) + VALUES (?,?,?,?,?,?)`) + .run(e.editId, e.kind, e.payloadJSON, e.beforeJSON, e.rationale, Math.floor(Date.now() / 1000)); + return Number(r.lastInsertRowid); + }); +} +export function journalSince(cfg: C, since: number): JournalRow[] { + return withDb(cfg, (db) => db.prepare("SELECT * FROM editJournal WHERE seq > ? ORDER BY seq").all(since) as JournalRow[]); +} +export function activeEdits(cfg: C): JournalRow[] { + return withDb(cfg, (db) => db.prepare("SELECT * FROM editJournal WHERE undoneBySeq IS NULL AND kind != 'undo' ORDER BY seq").all() as JournalRow[]); +} +export function markUndone(cfg: C, targetSeq: number, bySeq: number): void { + withDb(cfg, (db) => db.prepare("UPDATE editJournal SET undoneBySeq = ? WHERE seq = ? AND undoneBySeq IS NULL").run(bySeq, targetSeq)); +} +``` + +- [ ] **Step 5: Point `src/ingest.ts` at the shared opener** + +In `src/ingest.ts`: delete its local `openServerDb` function, add `import { openServerDb } from "./serverdb.js";`, keep `latestIngest` and the `ingestLog` insert working exactly as before (the table shape is unchanged). Re-export for compatibility: `export { openServerDb } from "./serverdb.js";` + +- [ ] **Step 6: Run to verify pass** + +Run: `npm test` — Expected: staging 4/4 new + all 41 existing = 45 passing. `npm run build` clean. + +- [ ] **Step 7: Commit** + +```bash +git add -A && git commit -m "feat: staging store — proposals + append-only edit journal in server.sqlite" +``` + +--- + +### Task 2: Edit kinds — payload validation, before-snapshot, human diff + +**Files:** +- Create: `noop-cloud/src/edits/kinds.ts` +- Create: `noop-cloud/src/edits/diff.ts` +- Test: `noop-cloud/test/edits-diff.test.ts` + +**Interfaces:** +- Consumes: `Mirror` (Task 4 of Phase 1), `Config`. +- Produces (kinds.ts): + - `const EDIT_KINDS = ["fix_workout","delete_workout","add_workout","adjust_sleep_bounds","delete_metric_point","set_baseline_note"] as const; type EditKind = typeof EDIT_KINDS[number];` + - `payloadSchema(kind: EditKind): z.ZodTypeAny` with shapes: + - `fix_workout`: `{ deviceId: string, startTs: number, sport: string, patch: { sport?: string, startTs?: number, endTs?: number, energyKcal?: number|null, distanceM?: number|null } }` (patch must be non-empty) + - `delete_workout`: `{ deviceId, startTs, sport }` + - `add_workout`: `{ startTs: number, endTs: number, sport: string, energyKcal?: number, distanceM?: number, notes?: string }` (deviceId is NOT accepted — forced to `"noop-cloud"`) + - `adjust_sleep_bounds`: `{ deviceId, startTs, newStartTs?: number, newEndTs?: number }` (at least one of newStartTs/newEndTs) + - `delete_metric_point`: `{ deviceId, day: string (YYYY-MM-DD regex), key: string }` + - `set_baseline_note`: `{ note: string (1..500), deviceId?: string }` +- Produces (diff.ts): + - `class EditTargetError extends Error { constructor(public code: "target_not_found"|"not_ingested", msg?: string) }` + - `captureBefore(cfg: Config, kind: EditKind, payload: any): object | null` — reads the mirror; returns the affected row for fix/delete/adjust kinds (throws `EditTargetError("target_not_found")` if absent; throws `EditTargetError("not_ingested")` if no mirror); returns `null` for `add_workout`/`set_baseline_note`. + - `renderDiff(kind: EditKind, payload: any, before: object | null): string` — short human-readable text, e.g. `sleep 2026-06-13 03:00→10:00 ⇒ end 06:00 (was 10:00)`. + +- [ ] **Step 1: Write the failing test** + +`test/edits-diff.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; +import { ingestNoopbak } from "../src/ingest.js"; +import { payloadSchema, EDIT_KINDS } from "../src/edits/kinds.js"; +import { captureBefore, renderDiff, EditTargetError } from "../src/edits/diff.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/edits-diff"); +const cfg = { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000 } as any; +const RUN_TS = Math.floor(new Date("2026-06-12T18:00:00Z").getTime() / 1000); // fixture running workout + +beforeAll(() => { fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); const z = path.join(dataDir, "b.noopbak"); buildNoopbak(z); ingestNoopbak(fs.readFileSync(z), cfg); }); + +describe("edit kinds", () => { + it("validates payloads per kind", () => { + expect(payloadSchema("delete_workout").safeParse({ deviceId: "my-whoop", startTs: RUN_TS, sport: "running" }).success).toBe(true); + expect(payloadSchema("adjust_sleep_bounds").safeParse({ deviceId: "my-whoop", startTs: 1 }).success).toBe(false); // needs a new bound + expect(payloadSchema("add_workout").safeParse({ startTs: 1, endTs: 2, sport: "yoga", deviceId: "my-whoop" }).success).toBe(false); // deviceId not accepted + expect(EDIT_KINDS.length).toBe(6); + }); + it("captureBefore snapshots the real row and errors on a missing target", () => { + const before = captureBefore(cfg, "delete_workout", { deviceId: "my-whoop", startTs: RUN_TS, sport: "running" }) as any; + expect(before.sport).toBe("running"); + expect(() => captureBefore(cfg, "delete_workout", { deviceId: "my-whoop", startTs: 1, sport: "nope" })).toThrow(EditTargetError); + }); + it("renderDiff is human-readable and mentions old and new values", () => { + const before = captureBefore(cfg, "adjust_sleep_bounds", { deviceId: "my-whoop", startTs: Math.floor(new Date("2026-06-13T03:00:00Z").getTime() / 1000), newEndTs: Math.floor(new Date("2026-06-13T06:00:00Z").getTime() / 1000) }); + const d = renderDiff("adjust_sleep_bounds", { deviceId: "my-whoop", startTs: Math.floor(new Date("2026-06-13T03:00:00Z").getTime() / 1000), newEndTs: Math.floor(new Date("2026-06-13T06:00:00Z").getTime() / 1000) }, before); + expect(d).toContain("06:00"); + expect(d.toLowerCase()).toContain("end"); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- edits-diff` — Expected: FAIL, module not found. + +- [ ] **Step 3: Implement `src/edits/kinds.ts`** + +```typescript +import { z } from "zod"; + +export const EDIT_KINDS = ["fix_workout", "delete_workout", "add_workout", "adjust_sleep_bounds", "delete_metric_point", "set_baseline_note"] as const; +export type EditKind = (typeof EDIT_KINDS)[number]; + +const DAY = z.string().regex(/^\d{4}-\d{2}-\d{2}$/, "YYYY-MM-DD"); +const workoutKey = { deviceId: z.string().min(1), startTs: z.number().int(), sport: z.string().min(1) }; + +const schemas: Record = { + fix_workout: z.object({ + ...workoutKey, + patch: z.object({ + sport: z.string().min(1).optional(), startTs: z.number().int().optional(), endTs: z.number().int().optional(), + energyKcal: z.number().nullable().optional(), distanceM: z.number().nullable().optional(), + }).refine((p) => Object.keys(p).length > 0, "patch must not be empty"), + }).strict(), + delete_workout: z.object(workoutKey).strict(), + add_workout: z.object({ + startTs: z.number().int(), endTs: z.number().int(), sport: z.string().min(1), + energyKcal: z.number().optional(), distanceM: z.number().optional(), notes: z.string().max(500).optional(), + }).strict().refine((p) => p.endTs > p.startTs, "endTs must be after startTs"), + adjust_sleep_bounds: z.object({ + deviceId: z.string().min(1), startTs: z.number().int(), + newStartTs: z.number().int().optional(), newEndTs: z.number().int().optional(), + }).strict().refine((p) => p.newStartTs !== undefined || p.newEndTs !== undefined, "need newStartTs and/or newEndTs"), + delete_metric_point: z.object({ deviceId: z.string().min(1), day: DAY, key: z.string().min(1) }).strict(), + set_baseline_note: z.object({ note: z.string().min(1).max(500), deviceId: z.string().min(1).optional() }).strict(), +}; + +export function payloadSchema(kind: EditKind): z.ZodTypeAny { return schemas[kind]; } +``` + +- [ ] **Step 4: Implement `src/edits/diff.ts`** + +```typescript +import fs from "node:fs"; +import Database from "better-sqlite3"; +import type { Config } from "../config.js"; +import type { EditKind } from "./kinds.js"; + +export class EditTargetError extends Error { + constructor(public code: "target_not_found" | "not_ingested", msg?: string) { super(msg ?? code); } +} + +const iso = (ts: number) => new Date(ts * 1000).toISOString().slice(0, 16).replace("T", " ") + "Z"; + +/** Read-only peek at the mirror for the row an edit targets. null for kinds with no target. */ +export function captureBefore(cfg: Pick, kind: EditKind, payload: any): object | null { + if (kind === "add_workout" || kind === "set_baseline_note") return null; + if (!fs.existsSync(cfg.mirrorPath)) throw new EditTargetError("not_ingested"); + const db = new Database(cfg.mirrorPath, { readonly: true, fileMustExist: true }); + try { + let row: unknown; + if (kind === "fix_workout" || kind === "delete_workout") { + row = db.prepare("SELECT * FROM workout WHERE deviceId=? AND startTs=? AND sport=?").get(payload.deviceId, payload.startTs, payload.sport); + } else if (kind === "adjust_sleep_bounds") { + row = db.prepare("SELECT * FROM sleepSession WHERE deviceId=? AND startTs=?").get(payload.deviceId, payload.startTs); + } else if (kind === "delete_metric_point") { + row = db.prepare("SELECT * FROM metricSeries WHERE deviceId=? AND day=? AND key=?").get(payload.deviceId, payload.day, payload.key); + } + if (!row) throw new EditTargetError("target_not_found", `${kind}: no matching row in the mirror`); + return row as object; + } finally { db.close(); } +} + +export function renderDiff(kind: EditKind, payload: any, before: any): string { + switch (kind) { + case "delete_workout": + return `DELETE workout ${before.sport} @ ${iso(before.startTs)} (${before.durationS ? Math.round(before.durationS / 60) + " min" : "?"}, source ${before.source ?? "?"}, ${payload.deviceId})`; + case "fix_workout": { + const parts = Object.entries(payload.patch).map(([k, v]) => `${k}: ${JSON.stringify((before as any)[k])} ⇒ ${JSON.stringify(v)}`); + return `FIX workout ${before.sport} @ ${iso(before.startTs)} (${payload.deviceId}): ${parts.join(", ")}`; + } + case "add_workout": + return `ADD workout ${payload.sport} ${iso(payload.startTs)} → ${iso(payload.endTs)} (source noop-cloud${payload.energyKcal ? `, ${payload.energyKcal} kcal` : ""})`; + case "adjust_sleep_bounds": { + const bits: string[] = []; + if (payload.newStartTs !== undefined) bits.push(`start ${iso(before.startTs)} ⇒ ${iso(payload.newStartTs)}`); + if (payload.newEndTs !== undefined) bits.push(`end ${iso(before.endTs)} ⇒ ${iso(payload.newEndTs)}`); + return `ADJUST sleep @ ${iso(before.startTs)} (${payload.deviceId}): ${bits.join(", ")}`; + } + case "delete_metric_point": + return `DELETE metric ${payload.key}=${before.value} on ${payload.day} (${payload.deviceId})`; + case "set_baseline_note": + return `NOTE${payload.deviceId ? ` [${payload.deviceId}]` : ""}: ${payload.note}`; + } +} +``` + +- [ ] **Step 5: Run to verify pass** + +Run: `npm test -- edits-diff` — Expected: 3/3 PASS. Full: `npm test` → 48. Build clean. + +- [ ] **Step 6: Commit** + +```bash +git add -A && git commit -m "feat: edit kinds — zod payloads, before-snapshot capture, human diffs" +``` + +--- + +### Task 3: Overlay computation + +**Files:** +- Create: `noop-cloud/src/edits/overlay.ts` +- Test: `noop-cloud/test/overlay.test.ts` + +**Interfaces:** +- Consumes: `activeEdits` (Task 1), payload shapes (Task 2). +- Produces: + - `interface Overlay { sleepBounds: Map; deletedWorkouts: Set; patchedWorkouts: Map; addedWorkouts: { editId: string; deviceId: "noop-cloud"; startTs: number; endTs: number; sport: string; energyKcal: number | null; distanceM: number | null; notes: string | null }[]; deletedMetricPoints: Set; baselineNotes: { note: string; deviceId: string | null; at: number }[] }` + - Keys: workouts `"${deviceId}|${startTs}|${sport}"`, sleep `"${deviceId}|${startTs}"`, metric points `"${deviceId}|${day}|${key}"`. Export `workoutKeyOf(deviceId,startTs,sport)`, `sleepKeyOf(deviceId,startTs)`, `pointKeyOf(deviceId,day,key)`. + - `computeOverlay(cfg: Pick): Overlay` — folds `activeEdits` in seq order (later edits to the same target win). + +- [ ] **Step 1: Write the failing test** + +`test/overlay.test.ts`: + +```typescript +import { describe, it, expect, beforeEach } from "vitest"; +import fs from "node:fs"; import path from "node:path"; +import { appendJournal, markUndone } from "../src/staging.js"; +import { computeOverlay, workoutKeyOf, sleepKeyOf } from "../src/edits/overlay.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/overlay"); +const cfg = { serverDbPath: path.join(dataDir, "server.sqlite") } as any; +beforeEach(() => { fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); }); + +describe("overlay", () => { + it("folds active edits into lookup structures", () => { + appendJournal(cfg, { editId: "e1", kind: "delete_workout", payloadJSON: JSON.stringify({ deviceId: "my-whoop", startTs: 100, sport: "running" }), beforeJSON: null, rationale: null }); + appendJournal(cfg, { editId: "e2", kind: "adjust_sleep_bounds", payloadJSON: JSON.stringify({ deviceId: "my-whoop", startTs: 200, newEndTs: 999 }), beforeJSON: null, rationale: null }); + appendJournal(cfg, { editId: "e3", kind: "add_workout", payloadJSON: JSON.stringify({ startTs: 300, endTs: 400, sport: "yoga" }), beforeJSON: null, rationale: null }); + const o = computeOverlay(cfg); + expect(o.deletedWorkouts.has(workoutKeyOf("my-whoop", 100, "running"))).toBe(true); + expect(o.sleepBounds.get(sleepKeyOf("my-whoop", 200))?.newEndTs).toBe(999); + expect(o.addedWorkouts[0]).toMatchObject({ deviceId: "noop-cloud", sport: "yoga" }); + }); + it("undone edits leave the overlay", () => { + const s = appendJournal(cfg, { editId: "e4", kind: "delete_workout", payloadJSON: JSON.stringify({ deviceId: "d", startTs: 1, sport: "s" }), beforeJSON: null, rationale: null }); + const u = appendJournal(cfg, { editId: "u4", kind: "undo", payloadJSON: JSON.stringify({ targetSeq: s }), beforeJSON: null, rationale: null }); + markUndone(cfg, s, u); + const o = computeOverlay(cfg); + expect(o.deletedWorkouts.size).toBe(0); + }); + it("later sleep adjustments to the same session win", () => { + appendJournal(cfg, { editId: "e5", kind: "adjust_sleep_bounds", payloadJSON: JSON.stringify({ deviceId: "d", startTs: 5, newEndTs: 10 }), beforeJSON: null, rationale: null }); + appendJournal(cfg, { editId: "e6", kind: "adjust_sleep_bounds", payloadJSON: JSON.stringify({ deviceId: "d", startTs: 5, newEndTs: 20 }), beforeJSON: null, rationale: null }); + expect(computeOverlay(cfg).sleepBounds.get(sleepKeyOf("d", 5))?.newEndTs).toBe(20); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- overlay` — Expected: FAIL, module not found. + +- [ ] **Step 3: Implement `src/edits/overlay.ts`** + +```typescript +import type { Config } from "../config.js"; +import { activeEdits } from "../staging.js"; + +export const workoutKeyOf = (deviceId: string, startTs: number, sport: string) => `${deviceId}|${startTs}|${sport}`; +export const sleepKeyOf = (deviceId: string, startTs: number) => `${deviceId}|${startTs}`; +export const pointKeyOf = (deviceId: string, day: string, key: string) => `${deviceId}|${day}|${key}`; + +export interface Overlay { + sleepBounds: Map; + deletedWorkouts: Set; + patchedWorkouts: Map; + addedWorkouts: { editId: string; deviceId: "noop-cloud"; startTs: number; endTs: number; sport: string; energyKcal: number | null; distanceM: number | null; notes: string | null }[]; + deletedMetricPoints: Set; + baselineNotes: { note: string; deviceId: string | null; at: number }[]; +} + +export function computeOverlay(cfg: Pick): Overlay { + const o: Overlay = { sleepBounds: new Map(), deletedWorkouts: new Set(), patchedWorkouts: new Map(), addedWorkouts: [], deletedMetricPoints: new Set(), baselineNotes: [] }; + for (const e of activeEdits(cfg)) { + const p = JSON.parse(e.payloadJSON); + switch (e.kind) { + case "delete_workout": o.deletedWorkouts.add(workoutKeyOf(p.deviceId, p.startTs, p.sport)); break; + case "fix_workout": { + const k = workoutKeyOf(p.deviceId, p.startTs, p.sport); + const prev = o.patchedWorkouts.get(k)?.patch ?? {}; + o.patchedWorkouts.set(k, { patch: { ...prev, ...p.patch }, editId: e.editId }); + break; + } + case "add_workout": + o.addedWorkouts.push({ editId: e.editId, deviceId: "noop-cloud", startTs: p.startTs, endTs: p.endTs, sport: p.sport, energyKcal: p.energyKcal ?? null, distanceM: p.distanceM ?? null, notes: p.notes ?? null }); + break; + case "adjust_sleep_bounds": { + const k = sleepKeyOf(p.deviceId, p.startTs); + const prev = o.sleepBounds.get(k) ?? { editId: e.editId }; + o.sleepBounds.set(k, { ...prev, ...(p.newStartTs !== undefined ? { newStartTs: p.newStartTs } : {}), ...(p.newEndTs !== undefined ? { newEndTs: p.newEndTs } : {}), editId: e.editId }); + break; + } + case "delete_metric_point": o.deletedMetricPoints.add(pointKeyOf(p.deviceId, p.day, p.key)); break; + case "set_baseline_note": o.baselineNotes.push({ note: p.note, deviceId: p.deviceId ?? null, at: e.appliedAt }); break; + } + } + return o; +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `npm test -- overlay` — 3/3 PASS. Full suite 51. Build clean. + +- [ ] **Step 5: Commit** + +```bash +git add -A && git commit -m "feat: overlay — fold active journal edits into read-side lookups" +``` + +--- + +### Task 4: Scope threading + `propose_edit` / `list_pending` / `edit_journal` + +**Files:** +- Create: `noop-cloud/src/tools/writes.ts` +- Modify: `noop-cloud/src/auth.ts` (export `tokenScope`) +- Modify: `noop-cloud/src/mcp.ts` (`buildMcpServer(cfg, scope)`) +- Modify: `noop-cloud/src/server.ts` (detect scope in the /mcp route; pass through) +- Modify: `noop-cloud/src/tools/index.ts` (`registerTools(server, cfg, scope)` → call `registerWriteTools`) +- Test: `noop-cloud/test/tools-writes.test.ts` + +**Interfaces:** +- Produces (auth.ts): `tokenScope(cfg: Config, authHeader: string | undefined): "rw" | "ro" | null` (timing-safe; rw checked first). +- Produces (mcp.ts): `buildMcpServer(cfg: Config, scope: "ro" | "rw"): McpServer` — existing callers in tests must pass a scope; the /mcp route passes the detected one. +- Produces (writes.ts): `registerWriteTools(server: McpServer, cfg: Config, scope: "ro" | "rw"): void` registering for ALL scopes: `propose_edit`, `list_pending`, `edit_journal`; and ONLY when `scope === "rw"`: `confirm_edit`, `reject_edit`, `undo_edit` (Task 5 implements those bodies; this task registers propose/list/journal only and leaves a `// Task 5:` marker). + - `propose_edit(kind, payload, rationale)`: validate kind ∈ EDIT_KINDS and payload via `payloadSchema`; `captureBefore` (structured errors `{ error: "not_ingested" | "target_not_found" }` as tool results, not throws); `renderDiff`; `createProposal` with id `"edit_" + crypto.randomBytes(5).toString("hex")`; returns `{ id, kind, diff, rationale, status: "pending", hint: "confirm_edit requires the read-write credential" }`. + - `list_pending()`: `{ pending: [{ id, kind, diff, rationale, createdAt }] }`. + - `edit_journal({ since?: number })`: `{ edits: JournalRow[], latestSeq: number }` (since default 0). + - Annotations: propose `{ readOnlyHint: false, destructiveHint: false }`; list/journal `{ readOnlyHint: true }`. + +- [ ] **Step 1: Write the failing test** + +`test/tools-writes.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; import http from "node:http"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; +import { ingestNoopbak } from "../src/ingest.js"; +import { createApp } from "../src/server.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/writes"); +function cfg() { return { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000, roToken: "ro".padEnd(40, "x"), rwToken: "rw".padEnd(40, "y"), port: 0 } as any; } +const RUN_TS = Math.floor(new Date("2026-06-12T18:00:00Z").getTime() / 1000); + +function mcp(port: number, token: string, body: object): Promise { + return new Promise((resolve) => { + const payload = JSON.stringify(body); + const r = http.request({ port, path: "/mcp", method: "POST", headers: { authorization: `Bearer ${token}`, "content-type": "application/json", accept: "application/json, text/event-stream" } }, (res) => { + let d = ""; res.on("data", (c) => (d += c)); res.on("end", () => { + const line = d.split("\n").find((l) => l.startsWith("data:")) ?? d; + resolve(JSON.parse(line.replace(/^data:\s*/, ""))); + }); + }); + r.end(payload); + }); +} + +beforeAll(() => { fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); const z = path.join(dataDir, "b.noopbak"); buildNoopbak(z); ingestNoopbak(fs.readFileSync(z), cfg()); }); + +describe("write tools + scope", () => { + it("ro caller sees propose/list/journal but NOT confirm/reject/undo; rw sees all", async () => { + const app = createApp(cfg()); const server = app.listen(0); const port = (server.address() as any).port; + const ro = await mcp(port, cfg().roToken, { jsonrpc: "2.0", id: 1, method: "tools/list" }); + const rw = await mcp(port, cfg().rwToken, { jsonrpc: "2.0", id: 2, method: "tools/list" }); + server.close(); + const roNames = ro.result.tools.map((t: any) => t.name); + const rwNames = rw.result.tools.map((t: any) => t.name); + expect(roNames).toContain("propose_edit"); + expect(roNames).not.toContain("confirm_edit"); + expect(rwNames).toContain("confirm_edit"); + expect(rwNames).toContain("undo_edit"); + }); + it("propose_edit validates, snapshots, and stages; list_pending shows it", async () => { + const app = createApp(cfg()); const server = app.listen(0); const port = (server.address() as any).port; + const p = await mcp(port, cfg().roToken, { jsonrpc: "2.0", id: 3, method: "tools/call", params: { name: "propose_edit", arguments: { kind: "delete_workout", payload: { deviceId: "my-whoop", startTs: RUN_TS, sport: "running" }, rationale: "test dup" } } }); + const out = p.result.structuredContent; + expect(out.status).toBe("pending"); + expect(out.diff).toContain("running"); + const l = await mcp(port, cfg().roToken, { jsonrpc: "2.0", id: 4, method: "tools/call", params: { name: "list_pending", arguments: {} } }); + server.close(); + expect(l.result.structuredContent.pending.map((x: any) => x.id)).toContain(out.id); + }); + it("propose_edit on a missing target returns a structured error", async () => { + const app = createApp(cfg()); const server = app.listen(0); const port = (server.address() as any).port; + const p = await mcp(port, cfg().roToken, { jsonrpc: "2.0", id: 5, method: "tools/call", params: { name: "propose_edit", arguments: { kind: "delete_workout", payload: { deviceId: "my-whoop", startTs: 1, sport: "ghost" }, rationale: "x" } } }); + server.close(); + expect(p.result.structuredContent.error).toBe("target_not_found"); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- tools-writes` — Expected: FAIL. + +- [ ] **Step 3: `src/auth.ts` — add tokenScope (keep everything else)** + +```typescript +export function tokenScope(cfg: Config, authHeader: string | undefined): "rw" | "ro" | null { + const m = /^Bearer (.+)$/.exec(authHeader ?? ""); + if (!m) return null; + if (safeEqual(m[1], cfg.rwToken)) return "rw"; + if (safeEqual(m[1], cfg.roToken)) return "ro"; + return null; +} +``` + +- [ ] **Step 4: `src/mcp.ts` — thread scope** + +Change signature to `export function buildMcpServer(cfg: Config, scope: "ro" | "rw"): McpServer` and the registry call to `registerTools(server, cfg, scope);`. Prompts unchanged. + +- [ ] **Step 5: `src/server.ts` — detect scope in the /mcp route** + +```typescript +// inside createApp, replace the /mcp POST handler's first line: +app.post("/mcp", requireScope(cfg, "ro"), express.json({ limit: "4mb" }), async (req, res) => { + const scope = tokenScope(cfg, req.header("authorization")) ?? "ro"; // requireScope already vetted it + const server = buildMcpServer(cfg, scope); + // ... rest unchanged +``` +(add `tokenScope` to the auth import) + +- [ ] **Step 6: `src/tools/index.ts` — signature + write tools** + +```typescript +import { registerWriteTools } from "./writes.js"; +export function registerTools(server: McpServer, cfg: Config, scope: "ro" | "rw"): void { + registerCoreTools(server, cfg); + registerQueryTools(server, cfg); + registerCompareSources(server, cfg); + registerSearchFetch(server, cfg); + registerWriteTools(server, cfg, scope); +} +``` + +- [ ] **Step 7: Implement `src/tools/writes.ts` (propose/list/journal; Task 5 adds the rest)** + +```typescript +import crypto from "node:crypto"; +import { z } from "zod"; +import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; +import type { Config } from "../config.js"; +import { EDIT_KINDS, EditKind, payloadSchema } from "../edits/kinds.js"; +import { captureBefore, renderDiff, EditTargetError } from "../edits/diff.js"; +import { createProposal, listPending, journalSince } from "../staging.js"; + +const asTool = (obj: unknown) => ({ content: [{ type: "text" as const, text: JSON.stringify(obj, null, 2) }], structuredContent: obj as Record }); + +export function registerWriteTools(server: McpServer, cfg: Config, scope: "ro" | "rw"): void { + server.registerTool("propose_edit", { + title: "Propose a data edit", + description: "Stage a correction (nothing is applied until a human confirms with the read-write credential). Kinds: " + EDIT_KINDS.join(", ") + ". Returns a human-readable diff and the proposal id.", + inputSchema: { + kind: z.enum(EDIT_KINDS), + payload: z.record(z.unknown()).describe("Kind-specific payload; see the kind's schema."), + rationale: z.string().min(3).max(500).describe("Why this edit is correct — recorded in the audit journal."), + }, + annotations: { readOnlyHint: false, destructiveHint: false, openWorldHint: false }, + }, async (a) => { + const parsed = payloadSchema(a.kind as EditKind).safeParse(a.payload); + if (!parsed.success) return asTool({ error: "invalid_payload", detail: parsed.error.issues.map((i) => i.message).join("; ") }); + let before: object | null; + try { before = captureBefore(cfg, a.kind as EditKind, parsed.data); } + catch (e) { return asTool({ error: e instanceof EditTargetError ? e.code : "capture_failed" }); } + const id = "edit_" + crypto.randomBytes(5).toString("hex"); + const diff = renderDiff(a.kind as EditKind, parsed.data, before); + createProposal(cfg, { id, kind: a.kind, payloadJSON: JSON.stringify(parsed.data), rationale: a.rationale, beforeJSON: before ? JSON.stringify(before) : null, diffText: diff }); + return asTool({ id, kind: a.kind, diff, rationale: a.rationale, status: "pending", hint: "confirm_edit requires the read-write credential" }); + }); + + server.registerTool("list_pending", { + title: "List pending proposals", + description: "Staged edits awaiting confirm/reject.", + inputSchema: {}, + annotations: { readOnlyHint: true, openWorldHint: false }, + }, async () => asTool({ pending: listPending(cfg).map((p) => ({ id: p.id, kind: p.kind, diff: p.diffText, rationale: p.rationale, createdAt: p.createdAt })) })); + + server.registerTool("edit_journal", { + title: "Edit journal", + description: "Append-only audit log of every confirmed edit (and undos). since = last seq you have.", + inputSchema: { since: z.number().int().min(0).optional() }, + annotations: { readOnlyHint: true, openWorldHint: false }, + }, async (a) => { + const edits = journalSince(cfg, a.since ?? 0); + return asTool({ edits, latestSeq: edits.length ? edits[edits.length - 1].seq : (a.since ?? 0) }); + }); + + if (scope === "rw") { + registerResolutionTools(server, cfg); // Task 5 + } +} + +// Task 5 implements: confirm_edit, reject_edit, undo_edit +export function registerResolutionTools(_server: McpServer, _cfg: Config): void {} +``` + +- [ ] **Step 8: Fix existing test callers of buildMcpServer** — `test/mcp.test.ts` calls go through `createApp` (no direct buildMcpServer callers), so no change expected; if any test imports `buildMcpServer` directly, pass `"ro"`. + +- [ ] **Step 9: Run to verify pass** + +Run: `npm test` — Expected: 54 passing (51 + 3 new). Build clean. + +- [ ] **Step 10: Commit** + +```bash +git add -A && git commit -m "feat: scope-threaded MCP + propose_edit/list_pending/edit_journal" +``` + +--- + +### Task 5: `confirm_edit` / `reject_edit` / `undo_edit` (rw-only) + +**Files:** +- Modify: `noop-cloud/src/tools/writes.ts` (implement `registerResolutionTools`) +- Test: `noop-cloud/test/tools-resolve.test.ts` + +**Interfaces:** +- Consumes: `resolveProposal`, `getProposal`, `appendJournal`, `journalSince`, `markUndone` (Task 1). +- Produces tools (rw-registered only): + - `confirm_edit({ id })`: proposal must be pending → `resolveProposal(confirmed)` → `appendJournal({ editId: id, kind, payloadJSON, beforeJSON, rationale })` → `{ id, seq, applied: true, diff }`. Absent/already-resolved → `{ error: "not_pending" }`. If `appendJournal` throws UNIQUE (editId already journaled) → `{ id, applied: true, note: "already applied" }` (idempotent). + - `reject_edit({ id })`: pending → rejected → `{ id, rejected: true }`; else `{ error: "not_pending" }`. + - `undo_edit({ seq })`: journal row must exist, `kind != 'undo'`, `undoneBySeq IS NULL` → append `{ editId: "undo_" + , kind: "undo", payloadJSON: {targetSeq} }` → `markUndone` → `{ undoneSeq: seq, bySeq }`. Else `{ error: "not_undoable" }`. + - Annotations: confirm + undo `{ readOnlyHint: false, destructiveHint: true }`; reject `{ readOnlyHint: false, destructiveHint: false }`. + +- [ ] **Step 1: Write the failing test** + +`test/tools-resolve.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; import http from "node:http"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; +import { ingestNoopbak } from "../src/ingest.js"; +import { createApp } from "../src/server.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/resolve"); +function cfg() { return { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000, roToken: "ro".padEnd(40, "x"), rwToken: "rw".padEnd(40, "y"), port: 0 } as any; } +const RUN_TS = Math.floor(new Date("2026-06-12T18:00:00Z").getTime() / 1000); + +function mcp(port: number, token: string, id: number, name: string, args: object): Promise { + return new Promise((resolve) => { + const payload = JSON.stringify({ jsonrpc: "2.0", id, method: "tools/call", params: { name, arguments: args } }); + const r = http.request({ port, path: "/mcp", method: "POST", headers: { authorization: `Bearer ${token}`, "content-type": "application/json", accept: "application/json, text/event-stream" } }, (res) => { + let d = ""; res.on("data", (c) => (d += c)); res.on("end", () => { + const line = d.split("\n").find((l) => l.startsWith("data:")) ?? d; + resolve(JSON.parse(line.replace(/^data:\s*/, "")).result.structuredContent); + }); + }); + r.end(payload); + }); +} + +let port = 0; let server: any; +beforeAll(() => { + fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); + const z = path.join(dataDir, "b.noopbak"); buildNoopbak(z); ingestNoopbak(fs.readFileSync(z), cfg()); + const app = createApp(cfg()); server = app.listen(0); port = (server.address() as any).port; +}); + +describe("confirm/reject/undo", () => { + it("full lifecycle: propose → confirm → journal → undo → journal grows, overlay empties", async () => { + const p = await mcp(port, cfg().roToken, 1, "propose_edit", { kind: "delete_workout", payload: { deviceId: "my-whoop", startTs: RUN_TS, sport: "running" }, rationale: "dup" }); + const c = await mcp(port, cfg().rwToken, 2, "confirm_edit", { id: p.id }); + expect(c.applied).toBe(true); expect(c.seq).toBeGreaterThan(0); + const c2 = await mcp(port, cfg().rwToken, 3, "confirm_edit", { id: p.id }); + expect(c2.error).toBe("not_pending"); // single-shot resolution + const j1 = await mcp(port, cfg().rwToken, 4, "edit_journal", {}); + expect(j1.edits.length).toBe(1); + const u = await mcp(port, cfg().rwToken, 5, "undo_edit", { seq: c.seq }); + expect(u.undoneSeq).toBe(c.seq); + const j2 = await mcp(port, cfg().rwToken, 6, "edit_journal", {}); + expect(j2.edits.length).toBe(2); // append-only: undo is a new entry + const u2 = await mcp(port, cfg().rwToken, 7, "undo_edit", { seq: c.seq }); + expect(u2.error).toBe("not_undoable"); + }); + it("reject leaves no journal entry", async () => { + const p = await mcp(port, cfg().roToken, 8, "propose_edit", { kind: "set_baseline_note", payload: { note: "oura reads low" }, rationale: "baseline" }); + const r = await mcp(port, cfg().rwToken, 9, "reject_edit", { id: p.id }); + expect(r.rejected).toBe(true); + const j = await mcp(port, cfg().rwToken, 10, "edit_journal", {}); + expect(j.edits.filter((e: any) => e.editId === p.id).length).toBe(0); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- tools-resolve` — Expected: FAIL (confirm_edit not implemented/registered). + +- [ ] **Step 3: Implement `registerResolutionTools` in `src/tools/writes.ts`** + +```typescript +import { getProposal, resolveProposal, appendJournal, markUndone, journalSince } from "../staging.js"; // extend the existing import + +export function registerResolutionTools(server: McpServer, cfg: Config): void { + server.registerTool("confirm_edit", { + title: "Confirm a proposed edit", + description: "Apply a pending proposal to the edit journal + overlay. Requires the read-write credential. The mirror itself is never modified; corrections reach the phone in Phase 3.", + inputSchema: { id: z.string().min(1) }, + annotations: { readOnlyHint: false, destructiveHint: true, openWorldHint: false }, + }, async (a) => { + const resolved = resolveProposal(cfg, a.id, "confirmed"); + if (!resolved) { + const existing = getProposal(cfg, a.id); + if (existing?.status === "confirmed") return asTool({ id: a.id, applied: true, note: "already applied" }); + return asTool({ error: "not_pending" }); + } + try { + const seq = appendJournal(cfg, { editId: resolved.id, kind: resolved.kind, payloadJSON: resolved.payloadJSON, beforeJSON: resolved.beforeJSON, rationale: resolved.rationale }); + return asTool({ id: resolved.id, seq, applied: true, diff: resolved.diffText }); + } catch { + return asTool({ id: resolved.id, applied: true, note: "already applied" }); // UNIQUE(editId) — idempotent + } + }); + + server.registerTool("reject_edit", { + title: "Reject a proposed edit", + description: "Discard a pending proposal (no journal entry, no data change).", + inputSchema: { id: z.string().min(1) }, + annotations: { readOnlyHint: false, destructiveHint: false, openWorldHint: false }, + }, async (a) => { + const r = resolveProposal(cfg, a.id, "rejected"); + return asTool(r ? { id: a.id, rejected: true } : { error: "not_pending" }); + }); + + server.registerTool("undo_edit", { + title: "Undo a confirmed edit", + description: "Reverse a journal entry by appending an undo record (history is never deleted).", + inputSchema: { seq: z.number().int().min(1) }, + annotations: { readOnlyHint: false, destructiveHint: true, openWorldHint: false }, + }, async (a) => { + const target = journalSince(cfg, a.seq - 1).find((e) => e.seq === a.seq); + if (!target || target.kind === "undo" || target.undoneBySeq !== null) return asTool({ error: "not_undoable" }); + const bySeq = appendJournal(cfg, { editId: "undo_" + crypto.randomBytes(5).toString("hex"), kind: "undo", payloadJSON: JSON.stringify({ targetSeq: a.seq }), beforeJSON: null, rationale: null }); + markUndone(cfg, a.seq, bySeq); + return asTool({ undoneSeq: a.seq, bySeq }); + }); +} +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `npm test` — Expected: 56 passing. Build clean. + +- [ ] **Step 5: Commit** + +```bash +git add -A && git commit -m "feat: confirm/reject/undo — rw-only resolution over the append-only journal" +``` + +--- + +### Task 6: Overlay-aware read tools + +**Files:** +- Modify: `noop-cloud/src/tools/query.ts` (sleep_summary bounds; workout_summary delete/patch/add; metric_series point deletes) +- Modify: `noop-cloud/src/tools/core.ts` (data_freshness gains `pendingEdits`, `journalSeq`, `baselineNotes`) +- Test: `noop-cloud/test/tools-overlay-reads.test.ts` + +**Interfaces:** +- Consumes: `computeOverlay`, key helpers (Task 3), `listPending`, `journalSince` (Task 1). +- Produces (behavioral contract): + - `sleep_summary`: a session with an overlay bound adjustment returns adjusted `startTs`/`endTs`, recomputed `durationMin`, plus `edited: true, editId`. Unedited sessions unchanged (no `edited` key). + - `workout_summary`: deleted workouts absent; patched workouts carry patch fields + `edited: true, editId`; added workouts appear with `deviceId: "noop-cloud"`, `family: "cloud"`... **No** — keep `sourceFamily` untouched (Phase-1 contract): added rows get `family: "whoop"`? Wrong either way — added rows carry `deviceId: "noop-cloud"` and `family` computed by `sourceFamily("noop-cloud")` which yields "whoop" today. To avoid misclassification, `workout_summary` sets `family: "cloud" as any` for added rows explicitly and marks `added: true, editId`. (Phase 3 will register the namespace properly; the tool-level label is honest today.) + - `metric_series`: deleted points absent. + - `data_freshness`: adds `pendingEdits: number`, `journalSeq: number` (latest seq, 0 if none), `baselineNotes: [{note, deviceId, at}]`. + - Tool descriptions for sleep_summary/workout_summary/metric_series append: "Reflects confirmed server-side edits; dailyMetric-derived numbers (health_snapshot/compare_sources) update only after the phone applies edits (Phase 3) and re-uploads." + +- [ ] **Step 1: Write the failing test** + +`test/tools-overlay-reads.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; +import { ingestNoopbak } from "../src/ingest.js"; +import { appendJournal } from "../src/staging.js"; +import { sleepSummary, workoutSummary, metricSeries } from "../src/tools/query.js"; +import { dataFreshness } from "../src/tools/core.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/overlay-reads"); +const cfg = { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000 } as any; +const SLEEP_TS = Math.floor(new Date("2026-06-13T03:00:00Z").getTime() / 1000); +const RUN_TS = Math.floor(new Date("2026-06-12T18:00:00Z").getTime() / 1000); +const NEW_END = Math.floor(new Date("2026-06-13T06:00:00Z").getTime() / 1000); + +beforeAll(() => { + fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); + const z = path.join(dataDir, "b.noopbak"); buildNoopbak(z); ingestNoopbak(fs.readFileSync(z), cfg); + appendJournal(cfg, { editId: "e_sleep", kind: "adjust_sleep_bounds", payloadJSON: JSON.stringify({ deviceId: "my-whoop", startTs: SLEEP_TS, newEndTs: NEW_END }), beforeJSON: null, rationale: null }); + appendJournal(cfg, { editId: "e_del", kind: "delete_workout", payloadJSON: JSON.stringify({ deviceId: "my-whoop", startTs: RUN_TS, sport: "running" }), beforeJSON: null, rationale: null }); + appendJournal(cfg, { editId: "e_add", kind: "add_workout", payloadJSON: JSON.stringify({ startTs: RUN_TS + 7200, endTs: RUN_TS + 9000, sport: "yoga" }), beforeJSON: null, rationale: null }); + appendJournal(cfg, { editId: "e_pt", kind: "delete_metric_point", payloadJSON: JSON.stringify({ deviceId: "oura-api", day: "2026-06-13", key: "oura_readiness" }), beforeJSON: null, rationale: null }); + appendJournal(cfg, { editId: "e_note", kind: "set_baseline_note", payloadJSON: JSON.stringify({ note: "oura RHR reads ~2bpm high", deviceId: "oura-api" }), beforeJSON: null, rationale: null }); +}); + +describe("overlay-aware reads", () => { + it("sleep_summary reflects adjusted end + recomputed duration + provenance", () => { + const s = sleepSummary(cfg, { from: "2026-06-13", to: "2026-06-13" }).sessions.find((x: any) => x.deviceId === "my-whoop"); + expect(s.endTs).toBe(NEW_END); + expect(s.durationMin).toBe(Math.round((NEW_END - SLEEP_TS) / 60)); + expect(s.edited).toBe(true); + }); + it("workout_summary hides deleted, shows added with provenance", () => { + const w = workoutSummary(cfg, { from: "2026-06-11", to: "2026-06-13" }); + expect(w.workouts.find((x: any) => x.sport === "running")).toBeUndefined(); + const added = w.workouts.find((x: any) => x.sport === "yoga"); + expect(added.deviceId).toBe("noop-cloud"); + expect(added.added).toBe(true); + }); + it("metric_series drops deleted points", () => { + const r = metricSeries(cfg, { deviceId: "oura-api", key: "oura_readiness", from: "2026-06-10", to: "2026-06-13" }); + expect(r.points.length).toBe(3); // fixture had 4 days + }); + it("data_freshness reports journal state + notes", () => { + const f = dataFreshness(cfg) as any; + expect(f.journalSeq).toBeGreaterThanOrEqual(5); + expect(f.baselineNotes[0].note).toContain("RHR"); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- overlay-reads` — Expected: FAIL (endTs unadjusted, running present, 4 points, no journalSeq). + +- [ ] **Step 3: Apply overlay in `src/tools/query.ts`** + +```typescript +import { computeOverlay, workoutKeyOf, sleepKeyOf, pointKeyOf } from "../edits/overlay.js"; + +export function sleepSummary(cfg: Config, args: { from: string; to: string }) { + if (!fs.existsSync(cfg.mirrorPath)) return { sessions: [], notIngested: true }; + const overlay = computeOverlay(cfg); + const m = new Mirror(cfg.mirrorPath); + try { + const sessions = m.sleepSummary(args).map((s) => { + const adj = overlay.sleepBounds.get(sleepKeyOf(s.deviceId, s.startTs)); + const startTs = adj?.newStartTs ?? s.startTs; + const endTs = adj?.newEndTs ?? s.endTs; + return { ...s, startTs, endTs, startIso: new Date(startTs * 1000).toISOString(), durationMin: Math.round((endTs - startTs) / 60), ...(adj ? { edited: true, editId: adj.editId } : {}) }; + }); + return { sessions }; + } finally { m.close(); } +} + +export function workoutSummary(cfg: Config, args: { from: string; to: string }) { + if (!fs.existsSync(cfg.mirrorPath)) return { workouts: [], notIngested: true }; + const overlay = computeOverlay(cfg); + const lo = Math.floor(new Date(`${args.from}T00:00:00Z`).getTime() / 1000); + const hi = Math.floor(new Date(`${args.to}T23:59:59Z`).getTime() / 1000); + const m = new Mirror(cfg.mirrorPath); + try { + const base = m.workoutSummary(args) + .filter((w) => !overlay.deletedWorkouts.has(workoutKeyOf(w.deviceId, w.startTs, w.sport))) + .map((w) => { + const patch = overlay.patchedWorkouts.get(workoutKeyOf(w.deviceId, w.startTs, w.sport)); + const merged = patch ? { ...w, ...patch.patch, edited: true, editId: patch.editId } : w; + const durationS = (merged as any).endTs && merged.startTs ? (merged as any).endTs - merged.startTs : merged.durationS; + return { ...merged, startIso: new Date(merged.startTs * 1000).toISOString(), durationMin: durationS != null ? Math.round(durationS / 60) : null }; + }); + const added = overlay.addedWorkouts + .filter((w) => w.startTs >= lo && w.startTs <= hi) + .map((w) => ({ deviceId: w.deviceId, family: "cloud" as any, startTs: w.startTs, endTs: w.endTs, sport: w.sport, source: "noop-cloud", durationS: w.endTs - w.startTs, energyKcal: w.energyKcal, distanceM: w.distanceM, startIso: new Date(w.startTs * 1000).toISOString(), durationMin: Math.round((w.endTs - w.startTs) / 60), added: true, editId: w.editId })); + return { workouts: [...base, ...added].sort((a, b) => a.startTs - b.startTs) }; + } finally { m.close(); } +} + +export function metricSeries(cfg: Config, args: { from: string; to: string; deviceId?: string; key?: string }) { + if (!fs.existsSync(cfg.mirrorPath)) return { points: [], notIngested: true }; + const overlay = computeOverlay(cfg); + const m = new Mirror(cfg.mirrorPath); + try { return { points: m.metricSeries(args).filter((p) => !overlay.deletedMetricPoints.has(pointKeyOf(p.deviceId, p.day, p.key))) }; } + finally { m.close(); } +} +``` +Append to the three registered tool descriptions: `" Reflects confirmed server-side edits; dailyMetric-derived numbers update only after Phase-3 phone sync."` + +- [ ] **Step 4: `src/tools/core.ts` — data_freshness additions** + +```typescript +import { listPending, journalSince } from "../staging.js"; +import { computeOverlay } from "../edits/overlay.js"; +// inside dataFreshness's returned object (both the notIngested early-return and the normal path): +// pendingEdits: listPending(cfg).length, +// journalSeq: (() => { const j = journalSince(cfg, 0); return j.length ? j[j.length - 1].seq : 0; })(), +// baselineNotes: computeOverlay(cfg).baselineNotes, +``` +(server.sqlite exists independently of the mirror, so these fields are correct in both paths.) + +- [ ] **Step 5: Run to verify pass** + +Run: `npm test` — Expected: 60 passing. Build clean. + +- [ ] **Step 6: Commit** + +```bash +git add -A && git commit -m "feat: overlay-aware reads — edits reflected with provenance markers" +``` + +--- + +### Task 7: `GET /edits?since=` (Phase-3 pull endpoint) + +**Files:** +- Modify: `noop-cloud/src/server.ts` (new route, ro scope, before the error middleware) +- Test: `noop-cloud/test/edits-endpoint.test.ts` + +**Interfaces:** +- Produces: `GET /edits?since=` (ro): `{ edits: JournalRow[], latestSeq: number }` — the FULL journal including `undo` rows (the phone replays reversals too). `since` optional/default 0; non-numeric → 400 `{ error: "bad_since" }`. + +- [ ] **Step 1: Write the failing test** + +`test/edits-endpoint.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; import http from "node:http"; +import { appendJournal } from "../src/staging.js"; +import { createApp } from "../src/server.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/edits-ep"); +function cfg() { return { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000, roToken: "ro".padEnd(40, "x"), rwToken: "rw".padEnd(40, "y"), port: 0 } as any; } + +function get(port: number, pathq: string, token?: string): Promise<{ status: number; json: any }> { + return new Promise((resolve) => { + http.get({ port, path: pathq, headers: token ? { authorization: `Bearer ${token}` } : {} }, (res) => { + let d = ""; res.on("data", (c) => (d += c)); res.on("end", () => resolve({ status: res.statusCode!, json: d ? JSON.parse(d) : null })); + }); + }); +} + +beforeAll(() => { + fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); + appendJournal(cfg(), { editId: "a", kind: "set_baseline_note", payloadJSON: "{}", beforeJSON: null, rationale: null }); + appendJournal(cfg(), { editId: "b", kind: "undo", payloadJSON: JSON.stringify({ targetSeq: 1 }), beforeJSON: null, rationale: null }); +}); + +describe("GET /edits", () => { + it("requires auth", async () => { + const app = createApp(cfg()); const s = app.listen(0); const port = (s.address() as any).port; + expect((await get(port, "/edits")).status).toBe(401); s.close(); + }); + it("returns the sequenced journal including undo rows, filtered by since", async () => { + const app = createApp(cfg()); const s = app.listen(0); const port = (s.address() as any).port; + const all = await get(port, "/edits", cfg().roToken); + expect(all.json.edits.length).toBe(2); + expect(all.json.latestSeq).toBe(2); + const tail = await get(port, "/edits?since=1", cfg().roToken); + expect(tail.json.edits.map((e: any) => e.editId)).toEqual(["b"]); + const bad = await get(port, "/edits?since=zzz", cfg().roToken); + expect(bad.status).toBe(400); s.close(); + }); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `npm test -- edits-endpoint` — Expected: 404s / FAIL. + +- [ ] **Step 3: Add the route in `src/server.ts`** (after `/mcp`, before the error middleware) + +```typescript +import { journalSince } from "./staging.js"; + +app.get("/edits", requireScope(cfg, "ro"), (req, res) => { + const raw = req.query.since; + const since = raw === undefined ? 0 : Number(raw); + if (!Number.isInteger(since) || since < 0) return res.status(400).json({ error: "bad_since" }); + const edits = journalSince(cfg, since); + res.json({ edits, latestSeq: edits.length ? edits[edits.length - 1].seq : since }); +}); +``` + +- [ ] **Step 4: Run to verify pass** + +Run: `npm test` — Expected: 62 passing. Build clean. + +- [ ] **Step 5: Commit** + +```bash +git add -A && git commit -m "feat: GET /edits — sequenced journal pull for Phase-3 phone sync" +``` + +--- + +### Task 8: README, deploy, live smoke + +**Files:** +- Modify: `noop-cloud/README.md` (add "Editing your data (Phase 2)" section) +- Deploy: `fly deploy --ha=false` + +- [ ] **Step 1: README section** (append after Tools) + +````markdown +## Editing your data (Phase 2) + +The AI can *propose* corrections; nothing changes until you confirm with the read-write credential. +The mirror (your uploaded data) is never modified — confirmed edits live in an append-only journal +and an overlay that read tools reflect (marked `edited: true` / `added: true`). Every proposal +records the original row (`before`) and your rationale; `undo_edit` reverses by appending, so the +audit trail is complete forever. + +- Read-only callers (routines, cron, shared agents) can `propose_edit`, `list_pending`, `edit_journal`. +- `confirm_edit` / `reject_edit` / `undo_edit` exist **only** for read-write callers — invisible otherwise. +- `GET /edits?since=` (read token) streams the journal for downstream sync. +- Honesty note: `health_snapshot` / `compare_sources` aggregate the phone's own daily rollups; those + numbers update after the phone applies your edits (Phase 3) and re-uploads. +```` + +- [ ] **Step 2: Full suite + build one last time** + +Run: `npm test && npm run build` — Expected: 62 passing, clean. + +- [ ] **Step 3: Commit + deploy** + +```bash +git add -A && git commit -m "docs: Phase-2 gated-writes README section" +fly deploy --ha=false --yes +``` +Expected: image builds on Fly's remote builder; machine updates; `https://vk-noop-cloud.fly.dev/healthz` → `{ok:true}`. + +- [ ] **Step 4: Live smoke (curl, using .env tokens)** + +```bash +source .env; BASE=https://vk-noop-cloud.fly.dev +# ro sees propose but not confirm +curl -s -X POST "$BASE/mcp" -H "Authorization: Bearer $RO_TOKEN" -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | grep -c confirm_edit # expect 0 +curl -s -X POST "$BASE/mcp" -H "Authorization: Bearer $RW_TOKEN" -H 'content-type: application/json' -H 'accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}' | grep -c confirm_edit # expect >=1 +# journal endpoint +curl -s "$BASE/edits" -H "Authorization: Bearer $RO_TOKEN" +``` + +- [ ] **Step 5: Ledger the deploy** in `/Users/vk/VKDEV/NOOP/noop/.superpowers/sdd/progress.md`. + +--- + +## Self-Review + +**Spec coverage (spec §5 Phase 2 + §6):** staging tables ✅ (T1); propose→diff→confirm/reject ✅ (T2/T4/T5); list_pending/edit_journal/undo ✅ (T4/T5); six edit kinds ✅ (T2); ro-can-stage/rw-applies with tool *invisibility* ✅ (T4/T5 via scope threading); destructive/readOnly annotations ✅; overlay consulted by read tools, mirror pristine ✅ (T6); journal append-only + idempotent + sequenced for Phase 3 ✅ (T1/T5/T7); `/edits?since=` ✅ (T7); deploy ✅ (T8). Honesty constraint (dailyMetric-derived numbers) documented in tool descriptions + README ✅ (T6/T8). + +**Placeholder scan:** none — every code step carries complete code. Task 4's `registerResolutionTools` no-op is an explicit, named Task-5 seam, implemented in Task 5 with full code. + +**Type consistency:** `ProposalRow`/`JournalRow` (T1) used by T4/T5/T7; overlay keys + `computeOverlay` (T3) consumed in T6 with the same helpers; `buildMcpServer(cfg, scope)` signature change is propagated (mcp.ts, server.ts, tools/index.ts) in one task (T4); `asTool` shape matches Phase 1's envelope. + +**Known deltas from Phase 1 the implementers must respect:** suite count grows 41→45→48→51→54→56→60→62; `openServerDb` moves to serverdb.ts with a compatibility re-export from ingest.ts. + +--- + +## Execution Handoff + +Plan complete and saved to `docs/superpowers/plans/2026-07-11-noop-cloud-2-gated-writes.md`. Two execution options: + +**1. Subagent-Driven (recommended)** — fresh subagent per task, review between tasks. + +**2. Inline Execution** — executing-plans in this session with checkpoints. From de8e33704a3f0d2fffff61cddd29f1bda0d913e1 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 12:10:34 -0700 Subject: [PATCH 046/171] =?UTF-8?q?docs:=20noop-cloud=20Phase-2b=20plan=20?= =?UTF-8?q?=E2=80=94=20granular=20reads=20(hr=5Fseries,=20sleep=5Fdetail)?= =?UTF-8?q?=20+=20granular=20edit=20kinds=20(edit=5Fsleep=5Fstages,=20dele?= =?UTF-8?q?te=5Fhr=5Frange)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The user's full-loop requirement: read per-second HR + hypnogram evidence, edit stages/HR/duration through the same propose-confirm-journal-undo audit rail. Supersedes Phase-2 Task 8 (single deploy at 2b end). Co-Authored-By: Claude Fable 5 --- .../2026-07-11-noop-cloud-2b-granular.md | 358 ++++++++++++++++++ 1 file changed, 358 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-11-noop-cloud-2b-granular.md diff --git a/docs/superpowers/plans/2026-07-11-noop-cloud-2b-granular.md b/docs/superpowers/plans/2026-07-11-noop-cloud-2b-granular.md new file mode 100644 index 0000000000..27dc983b00 --- /dev/null +++ b/docs/superpowers/plans/2026-07-11-noop-cloud-2b-granular.md @@ -0,0 +1,358 @@ +# noop-cloud Phase 2b — Granular Reads + Granular Edit Kinds + +> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Extends the Phase-2 plan (2026-07-11-noop-cloud-2-gated-writes.md); execute AFTER its Task 7. Phase 2's Task 8 (README/deploy) is SUPERSEDED by this plan's Task 4 — one deploy covers both. + +**Goal:** The AI can read the raw evidence (per-second heart rate, stage-by-stage hypnogram, movement) and edit at the same granularity (rewrite a night's stages, delete artifact HR ranges) — so "you were actually asleep until 6am" is provable from data and fixable end-to-end. + +**Architecture:** Two new read tools over existing mirror tables (`hrSample`, `sleepSession.stagesJSON`), downsample-windowed to keep responses LLM-sized. Two new edit kinds flowing through the existing propose→confirm→journal→overlay rail; `hr_series` and `sleep_detail` are overlay-aware from day one. + +**Tech Stack:** unchanged (SDK 1.29.0, better-sqlite3 12.11.1). Repo `/Users/vk/VKDEV/NOOP/noop-cloud`. + +## Global Constraints + +- Mirror read-only always; all edit state in server.sqlite; journal append-only (unchanged from Phase 2). +- `hr_series`: max span **7 days** per call (reject wider with `{ error: "span_too_wide" }`); optional `bucketSeconds` (60..3600) returns per-bucket `{ts, avg, min, max, n}`; raw mode caps at **5000 samples** (`truncated: true` + hint to bucket when hit). Multi-source: optional `deviceId`, else all sources, each sample carries `deviceId`/`family`. +- `sleep_detail`: one session by `(deviceId, startTs)`; returns bounds (overlay-adjusted, `edited` marker), `stages` parsed from stagesJSON (`[{start,end,stage}]`, overlay-replaced when an `edit_sleep_stages` edit is active), plus `hrDuringSleep` (bucketed 5-min from hrSample within bounds, same deviceId's family fallback: exact deviceId first, else any source overlapping the window — labeled per sample). +- New edit kinds (total becomes **eight**): `edit_sleep_stages` payload `{ deviceId, startTs, stages: [{start,end,stage}] (1..96 segments, contiguous non-overlapping ascending, stage ∈ awake|light|deep|rem) }`; `delete_hr_range` payload `{ deviceId, fromTs, toTs (toTs>fromTs, span ≤ 6h) }`. Both `.strict()`, both captureBefore (sleep row must exist / count of HR samples in range must be > 0 → else `target_not_found`), human diffs show old→new (stage minutes summary; sample count dropped). +- Overlay: `stageEdits: Map`; `deletedHrRanges: {deviceId, fromTs, toTs, editId}[]`. `hr_series` excludes samples inside deleted ranges (matching deviceId); `sleep_detail` reflects both stage edits and HR deletions; `metric_series`/others unchanged. +- Tool annotations `readOnlyHint: true` for both reads; guard pattern (`notIngested`) on both. +- Fixture additions must keep every existing test green (fixtures gain stagesJSON on one session + denser HR; adjust existing counts ONLY by appending new data that no existing assertion pins — existing hrSample rows are 5/day under oura-api and `hrCoverageDays` asserts 4 days: keep additions inside the same 4 days). + +--- + +### Task 1: Fixture enrichment + `hr_series` / `sleep_detail` read tools + +**Files:** +- Modify: `noop-cloud/test/fixtures/make-fixture.ts` (one WHOOP sleep session gets a real stagesJSON; add my-whoop HR samples across the 2026-06-13 night, 5-min cadence 03:00–10:00) +- Create: `noop-cloud/src/tools/granular.ts` +- Modify: `noop-cloud/src/tools/index.ts` (register) +- Test: `noop-cloud/test/tools-granular.test.ts` + +**Interfaces:** +- Produces: `hrSeries(cfg, args: { from: string(ISO datetime or YYYY-MM-DD), to: string, deviceId?: string, bucketSeconds?: number })` → `{ samples: [{ts, bpm, deviceId, family}] , truncated?: true }` or `{ buckets: [{ts, avg, min, max, n, deviceId, family}] }`; `{ error: "span_too_wide" }` past 7d; notIngested guard. +- Produces: `sleepDetail(cfg, args: { deviceId: string, startTs: number })` → `{ session: {deviceId, family, startTs, endTs, durationMin, efficiency, restingHr, avgHrv, edited?}, stages: [{start,end,stage}] | null, stagesEdited?: true, hrDuringSleep: [{ts,bpm,deviceId}], notFound?: true }`. +- Produces: `registerGranularTools(server, cfg)` (both `readOnlyHint: true`). +- Mirror gains two methods (Modify `src/mirror.ts`): `hrSamplesRange(opts: {fromTs: number, toTs: number, deviceId?: string, limit: number}): {deviceId: string, ts: number, bpm: number}[]` and `sleepSessionAt(deviceId: string, startTs: number): (SleepRow & { stagesJSON: string | null }) | null`. + +- [ ] **Step 1: Failing test** — `test/tools-granular.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; +import { ingestNoopbak } from "../src/ingest.js"; +import { hrSeries, sleepDetail } from "../src/tools/granular.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/granular"); +const cfg = { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000 } as any; +const SLEEP_TS = Math.floor(new Date("2026-06-13T03:00:00Z").getTime() / 1000); + +beforeAll(() => { fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); const z = path.join(dataDir, "b.noopbak"); buildNoopbak(z); ingestNoopbak(fs.readFileSync(z), cfg); }); + +describe("granular reads", () => { + it("hr_series raw returns my-whoop night samples with family", () => { + const r = hrSeries(cfg, { from: "2026-06-13T03:00:00Z", to: "2026-06-13T10:00:00Z", deviceId: "my-whoop" }) as any; + expect(r.samples.length).toBeGreaterThanOrEqual(80); // 5-min cadence over 7h + expect(r.samples[0].family).toBe("whoop"); + }); + it("hr_series buckets aggregate", () => { + const r = hrSeries(cfg, { from: "2026-06-13T03:00:00Z", to: "2026-06-13T10:00:00Z", deviceId: "my-whoop", bucketSeconds: 3600 }) as any; + expect(r.buckets.length).toBeGreaterThanOrEqual(6); + expect(r.buckets[0]).toHaveProperty("avg"); + }); + it("hr_series rejects >7d span", () => { + expect((hrSeries(cfg, { from: "2026-06-01", to: "2026-06-13" }) as any).error).toBe("span_too_wide"); + }); + it("sleep_detail returns stages + in-sleep HR", () => { + const r = sleepDetail(cfg, { deviceId: "my-whoop", startTs: SLEEP_TS }) as any; + expect(r.session.durationMin).toBeGreaterThan(0); + expect(r.stages.length).toBeGreaterThanOrEqual(3); + expect(r.hrDuringSleep.length).toBeGreaterThanOrEqual(80); + }); + it("sleep_detail notFound for a ghost session", () => { + expect((sleepDetail(cfg, { deviceId: "my-whoop", startTs: 1 }) as any).notFound).toBe(true); + }); +}); +``` + +- [ ] **Step 2: Run to verify FAIL** (`npm test -- tools-granular`). + +- [ ] **Step 3: Fixture additions** — in `buildMirrorSqlite` after the existing hr inserts: + +```typescript +// Granular night data (2026-06-13, my-whoop): stages on the sleep row + 5-min HR across the night. +const night = tsOf("2026-06-13", 3); +const stages = JSON.stringify([ + { start: night, end: night + 3600, stage: "light" }, + { start: night + 3600, end: night + 10800, stage: "deep" }, + { start: night + 10800, end: night + 18000, stage: "rem" }, + { start: night + 18000, end: night + 25200, stage: "light" }, +]); +db.prepare("UPDATE sleepSession SET stagesJSON = ? WHERE deviceId = 'my-whoop' AND startTs = ?").run(stages, night); +for (let t = night; t <= night + 25200; t += 300) hr.run("my-whoop", t, 46 + Math.round(8 * Math.abs(Math.sin(t / 3000)))); +``` + +- [ ] **Step 4: Mirror methods** — append to `src/mirror.ts` class: + +```typescript +hrSamplesRange(opts: { fromTs: number; toTs: number; deviceId?: string; limit: number }): { deviceId: string; ts: number; bpm: number }[] { + const where = ["ts >= ? AND ts <= ?"]; const args: any[] = [opts.fromTs, opts.toTs]; + if (opts.deviceId) { where.push("deviceId = ?"); args.push(opts.deviceId); } + args.push(opts.limit); + return this.db.prepare(`SELECT deviceId, ts, bpm FROM hrSample WHERE ${where.join(" AND ")} ORDER BY ts LIMIT ?`).all(...args) as any[]; +} +sleepSessionAt(deviceId: string, startTs: number) { + return (this.db.prepare("SELECT deviceId, startTs, endTs, efficiency, restingHr, avgHrv, userEdited, stagesJSON FROM sleepSession WHERE deviceId = ? AND startTs = ?").get(deviceId, startTs) as any) ?? null; +} +``` + +- [ ] **Step 5: Implement `src/tools/granular.ts`** + +```typescript +import fs from "node:fs"; +import { z } from "zod"; +import type { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; +import type { Config } from "../config.js"; +import { Mirror, sourceFamily } from "../mirror.js"; +import { computeOverlay, sleepKeyOf } from "../edits/overlay.js"; + +const asTool = (obj: unknown) => ({ content: [{ type: "text" as const, text: JSON.stringify(obj, null, 2) }], structuredContent: obj as Record }); +const toTs = (s: string, endOfDay = false) => /^\d{4}-\d{2}-\d{2}$/.test(s) + ? Math.floor(new Date(`${s}T${endOfDay ? "23:59:59" : "00:00:00"}Z`).getTime() / 1000) + : Math.floor(new Date(s).getTime() / 1000); +const MAX_SPAN_S = 7 * 86_400, RAW_CAP = 5000; + +function dropDeleted(samples: { deviceId: string; ts: number; bpm: number }[], ranges: { deviceId: string; fromTs: number; toTs: number }[]) { + if (!ranges.length) return samples; + return samples.filter((s) => !ranges.some((r) => r.deviceId === s.deviceId && s.ts >= r.fromTs && s.ts <= r.toTs)); +} + +export function hrSeries(cfg: Config, args: { from: string; to: string; deviceId?: string; bucketSeconds?: number }) { + if (!fs.existsSync(cfg.mirrorPath)) return { samples: [], notIngested: true }; + const fromTs = toTs(args.from), toT = toTs(args.to, true); + if (!Number.isFinite(fromTs) || !Number.isFinite(toT)) return { error: "bad_range" }; + if (toT - fromTs > MAX_SPAN_S) return { error: "span_too_wide", maxDays: 7 }; + const ranges = computeOverlay(cfg).deletedHrRanges; + const m = new Mirror(cfg.mirrorPath); + try { + const raw = dropDeleted(m.hrSamplesRange({ fromTs, toTs: toT, deviceId: args.deviceId, limit: 200_000 }), ranges); + if (args.bucketSeconds) { + const b = Math.min(3600, Math.max(60, args.bucketSeconds)); + const byBucket = new Map(); + for (const s of raw) { + const key = `${s.deviceId}|${Math.floor(s.ts / b) * b}`; + const cur = byBucket.get(key) ?? { ts: Math.floor(s.ts / b) * b, deviceId: s.deviceId, sum: 0, min: s.bpm, max: s.bpm, n: 0 }; + cur.sum += s.bpm; cur.min = Math.min(cur.min, s.bpm); cur.max = Math.max(cur.max, s.bpm); cur.n += 1; + byBucket.set(key, cur); + } + const buckets = [...byBucket.values()].sort((a, b2) => a.ts - b2.ts) + .map((x) => ({ ts: x.ts, avg: Math.round((x.sum / x.n) * 10) / 10, min: x.min, max: x.max, n: x.n, deviceId: x.deviceId, family: sourceFamily(x.deviceId) })); + return { buckets }; + } + const capped = raw.slice(0, RAW_CAP); + return { samples: capped.map((s) => ({ ...s, family: sourceFamily(s.deviceId) })), ...(raw.length > RAW_CAP ? { truncated: true, hint: "pass bucketSeconds to aggregate" } : {}) }; + } finally { m.close(); } +} + +export function sleepDetail(cfg: Config, args: { deviceId: string; startTs: number }) { + if (!fs.existsSync(cfg.mirrorPath)) return { notIngested: true }; + const overlay = computeOverlay(cfg); + const m = new Mirror(cfg.mirrorPath); + try { + const row = m.sleepSessionAt(args.deviceId, args.startTs); + if (!row) return { notFound: true }; + const adj = overlay.sleepBounds.get(sleepKeyOf(args.deviceId, args.startTs)); + const startTs = adj?.newStartTs ?? row.startTs, endTs = adj?.newEndTs ?? row.endTs; + const stageEdit = overlay.stageEdits.get(sleepKeyOf(args.deviceId, args.startTs)); + const stages = stageEdit ? stageEdit.stages : (row.stagesJSON ? JSON.parse(row.stagesJSON) : null); + const hr = dropDeleted(m.hrSamplesRange({ fromTs: startTs, toTs: endTs, deviceId: args.deviceId, limit: RAW_CAP }), overlay.deletedHrRanges); + const hrAny = hr.length ? hr : dropDeleted(m.hrSamplesRange({ fromTs: startTs, toTs: endTs, limit: RAW_CAP }), overlay.deletedHrRanges); + return { + session: { deviceId: row.deviceId, family: sourceFamily(row.deviceId), startTs, endTs, durationMin: Math.round((endTs - startTs) / 60), efficiency: row.efficiency, restingHr: row.restingHr, avgHrv: row.avgHrv, ...(adj ? { edited: true, editId: adj.editId } : {}) }, + stages, ...(stageEdit ? { stagesEdited: true, editId: stageEdit.editId } : {}), + hrDuringSleep: hrAny.map((s) => ({ ts: s.ts, bpm: s.bpm, deviceId: s.deviceId })), + }; + } finally { m.close(); } +} + +export function registerGranularTools(server: McpServer, cfg: Config): void { + server.registerTool("hr_series", { + title: "Heart-rate series", + description: "Raw or bucketed heart-rate samples for a time range (max 7 days). Use to inspect what actually happened (e.g. verify sleep/wake from HR). Reflects confirmed HR deletions.", + inputSchema: { from: z.string(), to: z.string(), deviceId: z.string().optional(), bucketSeconds: z.number().int().min(60).max(3600).optional() }, + annotations: { readOnlyHint: true, openWorldHint: false }, + }, async (a) => asTool(hrSeries(cfg, a))); + + server.registerTool("sleep_detail", { + title: "Sleep night detail", + description: "One night's full evidence: bounds, stage-by-stage hypnogram, and in-sleep heart-rate. Reflects confirmed stage edits, bound adjustments, and HR deletions.", + inputSchema: { deviceId: z.string(), startTs: z.number().int() }, + annotations: { readOnlyHint: true, openWorldHint: false }, + }, async (a) => asTool(sleepDetail(cfg, a))); +} +``` +NOTE: this imports `overlay.deletedHrRanges` / `overlay.stageEdits`, which exist only after Task 2. For THIS task, add both fields to the `Overlay` interface + `computeOverlay` as empty structures (`stageEdits: new Map(), deletedHrRanges: []`) with a `// filled by edit kinds in 2b Task 2` comment — keeps this task self-contained and green. + +- [ ] **Step 6: register in `src/tools/index.ts`** (`registerGranularTools(server, cfg);`). + +- [ ] **Step 7: Full suite** — existing tests must stay green (fixture additions are within pinned bounds: my-whoop hrSamples are new rows; `hrCoverageDays("oura-api")` untouched). Expected: prior count + 5. + +- [ ] **Step 8: Commit** — `feat: granular reads — hr_series + sleep_detail (overlay-aware)` + +--- + +### Task 2: Edit kinds `edit_sleep_stages` + `delete_hr_range` + +**Files:** +- Modify: `noop-cloud/src/edits/kinds.ts` (two new kinds; EDIT_KINDS becomes 8) +- Modify: `noop-cloud/src/edits/diff.ts` (captureBefore + renderDiff for both) +- Modify: `noop-cloud/src/edits/overlay.ts` (fill `stageEdits` + `deletedHrRanges`) +- Test: `noop-cloud/test/edits-granular.test.ts` + +**Interfaces:** +- `edit_sleep_stages` schema: `{ deviceId: min1, startTs: int, stages: array(1..96 of {start:int,end:int,stage:enum(awake|light|deep|rem)}) }` `.strict()` + refine: ascending, contiguous (`stages[i].end === stages[i+1].start`), every `end > start`. +- `delete_hr_range` schema: `{ deviceId: min1, fromTs: int, toTs: int }` `.strict()` + refine `toTs > fromTs && toTs - fromTs <= 21_600`. +- captureBefore: `edit_sleep_stages` → the sleepSession row (target_not_found if absent); `delete_hr_range` → `{ count }` from `SELECT COUNT(*) FROM hrSample WHERE deviceId=? AND ts BETWEEN ? AND ?` (target_not_found if 0). +- renderDiff: stages → `RESTAGE sleep @ : light 60m/deep 120m/rem 120m/light 120m ⇒ ` (compute per-stage minutes from old stagesJSON if present, else "no stages"); hr → `DELETE ${count} HR samples ${iso(fromTs)} → ${iso(toTs)} (${deviceId})`. +- overlay: `edit_sleep_stages` → `stageEdits.set(sleepKeyOf(deviceId,startTs), {stages, editId})` (later wins); `delete_hr_range` → push `{deviceId, fromTs, toTs, editId}`. + +- [ ] **Step 1: Failing test** — `test/edits-granular.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; +import { ingestNoopbak } from "../src/ingest.js"; +import { appendJournal } from "../src/staging.js"; +import { payloadSchema, EDIT_KINDS } from "../src/edits/kinds.js"; +import { captureBefore, renderDiff } from "../src/edits/diff.js"; +import { computeOverlay, sleepKeyOf } from "../src/edits/overlay.js"; +import { hrSeries, sleepDetail } from "../src/tools/granular.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/edits-granular"); +const cfg = { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000 } as any; +const NIGHT = Math.floor(new Date("2026-06-13T03:00:00Z").getTime() / 1000); +const NEW_STAGES = [{ start: NIGHT, end: NIGHT + 10800, stage: "deep" }, { start: NIGHT + 10800, end: NIGHT + 25200, stage: "light" }]; + +beforeAll(() => { fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); const z = path.join(dataDir, "b.noopbak"); buildNoopbak(z); ingestNoopbak(fs.readFileSync(z), cfg); }); + +describe("granular edit kinds", () => { + it("EDIT_KINDS is 8 and schemas validate", () => { + expect(EDIT_KINDS.length).toBe(8); + expect(payloadSchema("edit_sleep_stages").safeParse({ deviceId: "my-whoop", startTs: NIGHT, stages: NEW_STAGES }).success).toBe(true); + expect(payloadSchema("edit_sleep_stages").safeParse({ deviceId: "d", startTs: 1, stages: [{ start: 5, end: 4, stage: "deep" }] }).success).toBe(false); + expect(payloadSchema("delete_hr_range").safeParse({ deviceId: "my-whoop", fromTs: NIGHT, toTs: NIGHT + 30000 }).success).toBe(false); // >6h + }); + it("captureBefore + renderDiff for both kinds", () => { + const b1 = captureBefore(cfg, "edit_sleep_stages", { deviceId: "my-whoop", startTs: NIGHT, stages: NEW_STAGES }) as any; + expect(b1.stagesJSON).toBeTruthy(); + expect(renderDiff("edit_sleep_stages", { deviceId: "my-whoop", startTs: NIGHT, stages: NEW_STAGES }, b1)).toContain("deep"); + const b2 = captureBefore(cfg, "delete_hr_range", { deviceId: "my-whoop", fromTs: NIGHT, toTs: NIGHT + 3600 }) as any; + expect(b2.count).toBeGreaterThan(0); + expect(renderDiff("delete_hr_range", { deviceId: "my-whoop", fromTs: NIGHT, toTs: NIGHT + 3600 }, b2)).toMatch(/DELETE \d+ HR/); + }); + it("journal entries flow into overlay and granular reads", () => { + appendJournal(cfg, { editId: "g1", kind: "edit_sleep_stages", payloadJSON: JSON.stringify({ deviceId: "my-whoop", startTs: NIGHT, stages: NEW_STAGES }), beforeJSON: null, rationale: null }); + appendJournal(cfg, { editId: "g2", kind: "delete_hr_range", payloadJSON: JSON.stringify({ deviceId: "my-whoop", fromTs: NIGHT, toTs: NIGHT + 3599 }), beforeJSON: null, rationale: null }); + const o = computeOverlay(cfg); + expect(o.stageEdits.get(sleepKeyOf("my-whoop", NIGHT))?.stages.length).toBe(2); + expect(o.deletedHrRanges.length).toBe(1); + const d = sleepDetail(cfg, { deviceId: "my-whoop", startTs: NIGHT }) as any; + expect(d.stagesEdited).toBe(true); + expect(d.stages.length).toBe(2); + const hr = hrSeries(cfg, { from: "2026-06-13T03:00:00Z", to: "2026-06-13T10:00:00Z", deviceId: "my-whoop" }) as any; + const inRange = hr.samples.filter((s: any) => s.ts >= NIGHT && s.ts <= NIGHT + 3599); + expect(inRange.length).toBe(0); // deleted range excluded + }); +}); +``` + +- [ ] **Step 2: FAIL run.** +- [ ] **Step 3: kinds.ts** — extend `EDIT_KINDS`, add both schemas with the refines above (stage enum via `z.enum(["awake","light","deep","rem"])`; contiguity refine loops pairs). +- [ ] **Step 4: diff.ts** — `captureBefore` cases (`edit_sleep_stages` → sleepSession row; `delete_hr_range` → `{count}` via COUNT query, throw target_not_found on 0); `renderDiff` cases with stage-minute summaries: helper `stageSummary(stages)` → `"light 60m/deep 120m/…"` (merge repeated stages by summing). +- [ ] **Step 5: overlay.ts** — fill the two structures in the switch (replacing the Task-1 placeholders' comment). +- [ ] **Step 6: Full suite green (prior + 3); build clean.** +- [ ] **Step 7: Commit** — `feat: granular edit kinds — edit_sleep_stages + delete_hr_range` + +--- + +### Task 3: End-to-end lifecycle test through the live MCP surface + +**Files:** +- Test: `noop-cloud/test/tools-granular-e2e.test.ts` + +**Interfaces:** none new — this pins the full loop the feature exists for. + +- [ ] **Step 1: Write the test** (propose edit_sleep_stages via ro → confirm via rw → sleep_detail reflects → undo via rw → sleep_detail reverts; same HTTP harness as test/tools-resolve.test.ts — copy its `mcp()` helper verbatim): + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; import http from "node:http"; +import { buildNoopbak } from "./fixtures/make-fixture.js"; +import { ingestNoopbak } from "../src/ingest.js"; +import { createApp } from "../src/server.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/granular-e2e"); +function cfg() { return { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000, roToken: "ro".padEnd(40, "x"), rwToken: "rw".padEnd(40, "y"), port: 0 } as any; } +const NIGHT = Math.floor(new Date("2026-06-13T03:00:00Z").getTime() / 1000); + +function mcp(port: number, token: string, id: number, name: string, args: object): Promise { + return new Promise((resolve) => { + const payload = JSON.stringify({ jsonrpc: "2.0", id, method: "tools/call", params: { name, arguments: args } }); + const r = http.request({ port, path: "/mcp", method: "POST", headers: { authorization: `Bearer ${token}`, "content-type": "application/json", accept: "application/json, text/event-stream" } }, (res) => { + let d = ""; res.on("data", (c) => (d += c)); res.on("end", () => { + const line = d.split("\n").find((l) => l.startsWith("data:")) ?? d; + resolve(JSON.parse(line.replace(/^data:\s*/, "")).result.structuredContent); + }); + }); + r.end(payload); + }); +} + +let port = 0; let server: any; +beforeAll(() => { + fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); + const z = path.join(dataDir, "b.noopbak"); buildNoopbak(z); ingestNoopbak(fs.readFileSync(z), cfg()); + const app = createApp(cfg()); server = app.listen(0); port = (server.address() as any).port; +}); + +describe("granular edit full loop over MCP", () => { + it("propose(ro) → confirm(rw) → sleep_detail reflects → undo(rw) → reverts", async () => { + const stages = [{ start: NIGHT, end: NIGHT + 12600, stage: "deep" }, { start: NIGHT + 12600, end: NIGHT + 25200, stage: "light" }]; + const p = await mcp(port, cfg().roToken, 1, "propose_edit", { kind: "edit_sleep_stages", payload: { deviceId: "my-whoop", startTs: NIGHT, stages }, rationale: "HR shows deep sleep until 6:30, band misread movement" }); + expect(p.status).toBe("pending"); + expect(p.diff).toContain("deep"); + const c = await mcp(port, cfg().rwToken, 2, "confirm_edit", { id: p.id }); + expect(c.applied).toBe(true); + const d1 = await mcp(port, cfg().roToken, 3, "sleep_detail", { deviceId: "my-whoop", startTs: NIGHT }); + expect(d1.stagesEdited).toBe(true); + expect(d1.stages.length).toBe(2); + const u = await mcp(port, cfg().rwToken, 4, "undo_edit", { seq: c.seq }); + expect(u.undoneSeq).toBe(c.seq); + const d2 = await mcp(port, cfg().roToken, 5, "sleep_detail", { deviceId: "my-whoop", startTs: NIGHT }); + expect(d2.stagesEdited).toBeUndefined(); + expect(d2.stages.length).toBe(4); // fixture's original hypnogram + }); +}); +``` + +- [ ] **Step 2: Run (expect PASS if Tasks 1-2 are correct — this is an integration pin, not TDD-new-code). Full suite green.** +- [ ] **Step 3: Commit** — `test: granular edit full loop (propose→confirm→read→undo) over live MCP` + +--- + +### Task 4: README + deploy + live smoke (supersedes Phase-2 Task 8) + +- [ ] **Step 1:** README: add Phase-2 "Editing your data" section from the Phase-2 plan's Task 8 text, PLUS a "Granular evidence & edits" paragraph naming hr_series / sleep_detail / edit_sleep_stages / delete_hr_range and the 3am-vs-6am example. +- [ ] **Step 2:** `npm test && npm run build` — all green. +- [ ] **Step 3:** Commit `docs: Phase-2 + granular editing README`; then `fly deploy --ha=false --yes`. +- [ ] **Step 4: Live smoke** (`.env` tokens): ro tools/list contains hr_series+sleep_detail+propose_edit, NOT confirm_edit; rw contains confirm_edit; `GET /edits` 200; propose+confirm+undo one edit_sleep_stages round-trip against the fixture data currently on the server; `data_freshness` shows journalSeq ≥ 2. +- [ ] **Step 5:** Ledger the deploy in `/Users/vk/VKDEV/NOOP/noop/.superpowers/sdd/progress.md`. + +--- + +## Self-Review + +**Coverage vs the user's ask:** read any day at sensor granularity (hr_series 7d-windowed, sleep_detail full night) ✅; reason from evidence (both tools' descriptions steer that use) ✅; edit duration/stages/HR — adjust_sleep_bounds (P2) + edit_sleep_stages + delete_hr_range ✅; full audit loop (same propose/confirm/journal/undo rail) ✅; write-back seam (journal entries carry kind+payload+before for Phase-3 apply; stages align with the app's userEdited protection) ✅. +**Placeholder scan:** Task 1 Step 5's overlay placeholder is explicit and filled by Task 2 (named seam, same pattern as P2 T4→T5). No TBDs. +**Type consistency:** `Overlay.stageEdits`/`deletedHrRanges` names consistent across Tasks 1/2; `hrSamplesRange`/`sleepSessionAt` signatures match call sites; sleepKeyOf reused; EDIT_KINDS length assertions updated (6→8) — NOTE for implementers: `test/edits-diff.test.ts` asserts `EDIT_KINDS.length).toBe(6)` from Phase 2 Task 2 — Task 2 of THIS plan must update that assertion to 8 in the same commit. From 59e873a32acca25b6762b2ee23612d5a04945927 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 13:34:43 -0700 Subject: [PATCH 047/171] docs(plan): durationS override after spread (final-review finding backport) --- docs/superpowers/plans/2026-07-11-noop-cloud-2-gated-writes.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-07-11-noop-cloud-2-gated-writes.md b/docs/superpowers/plans/2026-07-11-noop-cloud-2-gated-writes.md index 99f893be2f..3b35a938e3 100644 --- a/docs/superpowers/plans/2026-07-11-noop-cloud-2-gated-writes.md +++ b/docs/superpowers/plans/2026-07-11-noop-cloud-2-gated-writes.md @@ -938,7 +938,8 @@ export function workoutSummary(cfg: Config, args: { from: string; to: string }) const patch = overlay.patchedWorkouts.get(workoutKeyOf(w.deviceId, w.startTs, w.sport)); const merged = patch ? { ...w, ...patch.patch, edited: true, editId: patch.editId } : w; const durationS = (merged as any).endTs && merged.startTs ? (merged as any).endTs - merged.startTs : merged.durationS; - return { ...merged, startIso: new Date(merged.startTs * 1000).toISOString(), durationMin: durationS != null ? Math.round(durationS / 60) : null }; + // durationS AFTER the spread so the recomputed value overrides the mirror's stale field. + return { ...merged, durationS, startIso: new Date(merged.startTs * 1000).toISOString(), durationMin: durationS != null ? Math.round(durationS / 60) : null }; }); const added = overlay.addedWorkouts .filter((w) => w.startTs >= lo && w.startTs <= hi) From f4bdc31eb55026b3abb7fb93abcdfb94dc8f6bd7 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 13:38:50 -0700 Subject: [PATCH 048/171] =?UTF-8?q?docs:=20noop-cloud=20Phase-3=20plan=20?= =?UTF-8?q?=E2=80=94=20phone=20sync-back=20+=20Apple=20Health=20re-export?= =?UTF-8?q?=20(6=20tasks)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pull-only /edits sync into a CLOUD_SYNC-gated Strand/CloudSync module: recompute-safe apply (applySleepEdit/updateSleepStages userEdited paths, noop-cloud namespace, v26 cloudTombstone + resurrection guards), ack-back, cursor in the cursors table, HK re-export via merged #249 write-back. --- .../2026-07-12-noop-cloud-3-phone-sync.md | 296 ++++++++++++++++++ 1 file changed, 296 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-12-noop-cloud-3-phone-sync.md diff --git a/docs/superpowers/plans/2026-07-12-noop-cloud-3-phone-sync.md b/docs/superpowers/plans/2026-07-12-noop-cloud-3-phone-sync.md new file mode 100644 index 0000000000..99fad928b8 --- /dev/null +++ b/docs/superpowers/plans/2026-07-12-noop-cloud-3-phone-sync.md @@ -0,0 +1,296 @@ +# noop-cloud Phase 3 — Phone Sync-Back + Apple Health Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development. Tasks 1 is in `/Users/vk/VKDEV/NOOP/noop-cloud` (TypeScript); Tasks 2–6 are in `/Users/vk/VKDEV/NOOP/noop` (Swift, personal fork lane). Steps use checkbox syntax. + +**Goal:** Confirmed cloud edits (sleep bounds/stages, workout fixes/deletes/adds, HR-range deletions, metric-point deletions) land on VK's iPhone — applied through recompute-safe store APIs — and corrected sleep/workouts/HR re-export to Apple Health via the merged write-back; every applied edit is acked back so the server journal tracks phone state. + +**Architecture:** The phone PULLS `GET /edits?since=` (never pushed to). A gated `Strand/CloudSync/` module (compiled only under `CLOUD_SYNC`, same untracked-xcconfig mechanism as `OURA_CLOUD_IMPORT`) fetches active journal rows, applies them via a `CloudEditApplier` that uses `applySleepEdit`/`updateSleepStages` (userEdited=1 → survives recompute), physical deletes + a new `cloudTombstone` table (migration v26) so recompute/re-import can't resurrect deletions, and the `"noop-cloud"` source namespace for adds/corrected copies. After a batch that touched sleep/workouts/HR, the module triggers the existing HealthKit write-back for the affected window. Cursor lives in the store's `cursors` table; acks POST back with rw. + +**Tech Stack:** Swift (app target `Strand/CloudSync/`, package `Packages/WhoopStore`), XCTest (StrandTests + WhoopStore package tests), TypeScript/vitest for the server ack endpoint. Device deploy via the established personal-signing pipeline. + +## Global Constraints + +- **Fork-only, flag-gated.** Every new app-target file starts with `#if CLOUD_SYNC` and the header comment pattern used by `Strand/Oura/*` ("Compiled ONLY when the CLOUD_SYNC compilation condition is set…"). The flag is added to `SWIFT_ACTIVE_COMPILATION_CONDITIONS` in the untracked `Strand/Oura/OuraSecrets.xcconfig` (rename-scope stays; the example file documents it). A default build contains none of this code. +- **Never mutate another source's rows in place.** Sleep edits go through `applySleepEdit` / `updateSleepStages` ONLY (they set/require `userEdited=1`). `fix_workout` = tombstone original + corrected copy under `deviceId "noop-cloud"`. `add_workout` under `"noop-cloud"`. Deletions = physical delete + `cloudTombstone` row; resurrection guards live in the store's upsert/insert paths. +- **Journal replay semantics:** apply rows in `seq` order where `kind != "undo" AND undoneBySeq == null` and seq > cursor. A row that arrives ALREADY undone is skipped (never applied). An edit undone AFTER the phone applied it is **out of MVP scope**: the applier detects it (acked seq now shows `undoneBySeq != null`) and surfaces "N server undos need manual attention" in the sync status — documented, not silently ignored. +- **`set_baseline_note` is server-only** — the applier records it as acked-no-op. +- **Idempotency:** re-applying an already-applied edit must be harmless (applySleepEdit is last-write-wins on the same values; deletes of absent rows are 0-row no-ops; adds under noop-cloud upsert on the same key). The cursor only advances after a successful batch + ack. +- **Migrations:** new migration is **`v26-cloud-tombstone`**, registered AFTER `v25-oura-raw`, additive only, with a migration test. Never edit existing migrations. +- **HealthKit:** iOS-only (`#if os(iOS)`); reuse the merged write-back (`HealthKitBridge.writeBack` path) rather than new HK write code — Task 5 first READS `StrandiOS/Health/HealthKitBridge.swift` to determine the public trigger + dedup semantics and adapts (expose a minimal public entry if needed; do not duplicate its sample-building logic). +- **Both app targets must compile** (`Strand` macOS + `NOOPiOS`) — CLOUD_SYNC files are shared; HealthKit calls guarded `#if os(iOS) && canImport(HealthKit)`. CI won't catch app-target breaks: build both locally. +- **Server change (Task 1) is additive**: `POST /edits/ack {seqs:[...]}` (rw) sets `ackedAt` on journal rows (new nullable column via `ALTER TABLE` guarded `IF NOT EXISTS`-style check); `GET /edits` gains `ackedAt` passthrough. SDK/pins unchanged; deploy at the end of Task 1. +- **Commits:** conventional, no Claude attribution (repo rule). + +--- + +### Task 1 (noop-cloud repo): `POST /edits/ack` + `ackedAt` + +**Files:** +- Modify: `noop-cloud/src/serverdb.ts` (add `ackedAt INTEGER` to editJournal CREATE; plus a startup `ALTER TABLE` catch for existing DBs) +- Modify: `noop-cloud/src/staging.ts` (`ackEdits(cfg, seqs: number[]): number`; include `ackedAt` in `JournalRow`) +- Modify: `noop-cloud/src/server.ts` (route) +- Test: `noop-cloud/test/edits-ack.test.ts` + +**Interfaces:** +- Produces: `POST /edits/ack` (rw scope), JSON body `{ seqs: number[] }` (1..500 ints) → `{ acked: number }` (count of rows updated where `ackedAt IS NULL`); invalid body → 400 `{ error: "bad_seqs" }`. +- `JournalRow` gains `ackedAt: number | null`; `GET /edits` responses now carry it (additive field). + +- [ ] **Step 1: Failing test** `test/edits-ack.test.ts`: + +```typescript +import { describe, it, expect, beforeAll } from "vitest"; +import fs from "node:fs"; import path from "node:path"; import http from "node:http"; +import { appendJournal, journalSince } from "../src/staging.js"; +import { createApp } from "../src/server.js"; + +const dataDir = path.join(process.cwd(), "test/.tmp/edits-ack"); +function cfg() { return { dataDir, mirrorPath: path.join(dataDir, "mirror.sqlite"), serverDbPath: path.join(dataDir, "server.sqlite"), maxIngestBytes: 262_144_000, roToken: "ro".padEnd(40, "x"), rwToken: "rw".padEnd(40, "y"), port: 0 } as any; } +function post(port: number, token: string, body: object): Promise<{ status: number; json: any }> { + return new Promise((resolve) => { + const payload = JSON.stringify(body); + const r = http.request({ port, path: "/edits/ack", method: "POST", headers: { authorization: `Bearer ${token}`, "content-type": "application/json" } }, (res) => { + let d = ""; res.on("data", (c) => (d += c)); res.on("end", () => resolve({ status: res.statusCode!, json: d ? JSON.parse(d) : null })); + }); + r.end(payload); + }); +} +beforeAll(() => { + fs.rmSync(dataDir, { recursive: true, force: true }); fs.mkdirSync(dataDir, { recursive: true }); + appendJournal(cfg(), { editId: "a1", kind: "set_baseline_note", payloadJSON: "{}", beforeJSON: null, rationale: null }); + appendJournal(cfg(), { editId: "a2", kind: "set_baseline_note", payloadJSON: "{}", beforeJSON: null, rationale: null }); +}); +describe("POST /edits/ack", () => { + it("requires rw (ro → 401)", async () => { + const app = createApp(cfg()); const s = app.listen(0); const port = (s.address() as any).port; + expect((await post(port, cfg().roToken, { seqs: [1] })).status).toBe(401); s.close(); + }); + it("acks rows once and exposes ackedAt via GET /edits", async () => { + const app = createApp(cfg()); const s = app.listen(0); const port = (s.address() as any).port; + const r1 = await post(port, cfg().rwToken, { seqs: [1, 2] }); + expect(r1.json.acked).toBe(2); + const r2 = await post(port, cfg().rwToken, { seqs: [1, 2] }); + expect(r2.json.acked).toBe(0); // already acked + s.close(); + const rows = journalSince(cfg(), 0); + expect(rows.every((e: any) => e.ackedAt !== null)).toBe(true); + }); + it("400 on garbage", async () => { + const app = createApp(cfg()); const s = app.listen(0); const port = (s.address() as any).port; + expect((await post(port, cfg().rwToken, { seqs: "x" })).status).toBe(400); s.close(); + }); +}); +``` + +- [ ] **Step 2: FAIL run** (`npm test -- edits-ack`). +- [ ] **Step 3: serverdb.ts** — add `ackedAt INTEGER` to the editJournal CREATE, and after the `db.exec`, a guarded migration for pre-existing DBs: + +```typescript +const cols = db.prepare("PRAGMA table_info(editJournal)").all() as any[]; +if (!cols.some((c) => c.name === "ackedAt")) db.exec("ALTER TABLE editJournal ADD COLUMN ackedAt INTEGER"); +``` + +- [ ] **Step 4: staging.ts** — `ackedAt: number | null` on `JournalRow`; and: + +```typescript +export function ackEdits(cfg: C, seqs: number[]): number { + return withDb(cfg, (db) => { + const stmt = db.prepare("UPDATE editJournal SET ackedAt = ? WHERE seq = ? AND ackedAt IS NULL"); + const now = Math.floor(Date.now() / 1000); + let n = 0; for (const s of seqs) n += stmt.run(now, s).changes; + return n; + }); +} +``` + +- [ ] **Step 5: server.ts route** (after GET /edits, before error middleware): + +```typescript +app.post("/edits/ack", requireScope(cfg, "rw"), express.json({ limit: "64kb" }), (req, res) => { + const seqs = (req.body as any)?.seqs; + if (!Array.isArray(seqs) || seqs.length === 0 || seqs.length > 500 || !seqs.every((s) => Number.isInteger(s) && s > 0)) { + return res.status(400).json({ error: "bad_seqs" }); + } + res.json({ acked: ackEdits(cfg, seqs) }); +}); +``` +(import `ackEdits` alongside `journalSince`) + +- [ ] **Step 6: PASS + full suite (expect 77) + build.** +- [ ] **Step 7: Commit `feat: POST /edits/ack — phone-applied tracking` and deploy: `fly deploy --ha=false --yes`; smoke: ack with rw → `{acked:N}`, ro → 401.** + +--- + +### Task 2 (NOOP repo): migration v26 + `CloudTombstoneStore` + resurrection guards + +**Files:** +- Modify: `Packages/WhoopStore/Sources/WhoopStore/Database.swift` (migration `v26-cloud-tombstone`) +- Create: `Packages/WhoopStore/Sources/WhoopStore/CloudTombstoneStore.swift` +- Modify: `Packages/WhoopStore/Sources/WhoopStore/JournalWorkoutAppleCache.swift` (`upsertWorkouts` skips tombstoned) +- Modify: `Packages/WhoopStore/Sources/WhoopStore/MetricSeriesStore.swift` (`upsertMetricSeries` skips tombstoned) +- Modify: `Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift` (`insert` filters HR samples inside tombstoned ranges) +- Create: `Packages/WhoopStore/Sources/WhoopStore/CloudEditDeletes.swift` (`deleteHrRange`, `deleteMetricPoint` physical deletes) +- Test: `Packages/WhoopStore/Tests/WhoopStoreTests/CloudTombstoneTests.swift` + +**Interfaces:** +- Migration `v26-cloud-tombstone`: + +```swift +migrator.registerMigration("v26-cloud-tombstone") { db in + try db.create(table: "cloudTombstone") { t in + t.autoIncrementedPrimaryKey("id") + t.column("kind", .text).notNull() // "workout" | "hrRange" | "metricPoint" + t.column("deviceId", .text).notNull() + t.column("startTs", .integer) // workout: startTs; hrRange: fromTs + t.column("endTs", .integer) // hrRange: toTs + t.column("sport", .text) // workout + t.column("day", .text) // metricPoint + t.column("key", .text) // metricPoint + t.column("editSeq", .integer).notNull() // server journal seq (idempotency) + t.column("createdAt", .integer).notNull() + t.uniqueKey(["kind", "editSeq"]) + } + try db.create(index: "idx_cloudTombstone_kind_device", on: "cloudTombstone", columns: ["kind", "deviceId"]) +} +``` +- `CloudTombstoneStore` (actor-friendly `WhoopStore` extension, same style as `OuraRawStore`): + - `addTombstone(kind: String, deviceId: String, startTs: Int?, endTs: Int?, sport: String?, day: String?, key: String?, editSeq: Int) async throws` (INSERT OR IGNORE on the unique key) + - `workoutTombstones(deviceId: String) async throws -> [(startTs: Int, sport: String)]` + - `hrTombstoneRanges(deviceId: String) async throws -> [(fromTs: Int, toTs: Int)]` + - `metricPointTombstones(deviceId: String) async throws -> [(day: String, key: String)]` +- Guards (all inside the store's existing write paths, same DB connection): + - `upsertWorkouts`: before insert, drop input rows matching a `cloudTombstone` row `kind='workout' AND deviceId=? AND startTs=? AND sport=?` (single pre-query into a Set). + - `upsertMetricSeries`: same for `kind='metricPoint'` on `(day,key)`. + - `StreamStore.insert`: load `hrRange` tombstones for the deviceId once per call; filter `streams.hr` samples with `fromTs <= ts <= toTs`. +- `CloudEditDeletes`: `deleteHrRange(deviceId: String, fromTs: Int, toTs: Int) async throws -> Int`; `deleteMetricPoint(deviceId: String, day: String, key: String) async throws -> Int` (plain DELETEs returning changes). + +- [ ] **Step 1: Failing test** `CloudTombstoneTests.swift`: + +```swift +import XCTest +@testable import WhoopStore +import WhoopProtocol + +final class CloudTombstoneTests: XCTestCase { + func testWorkoutTombstoneBlocksResurrection() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertWorkouts([WorkoutRow(startTs: 100, endTs: 200, sport: "running", source: "whoop", durationS: 100, energyKcal: nil, avgHr: nil, maxHr: nil, strain: nil, distanceM: nil, zonesJSON: nil, notes: nil)], deviceId: "d1") + try await store.addTombstone(kind: "workout", deviceId: "d1", startTs: 100, endTs: nil, sport: "running", day: nil, key: nil, editSeq: 7) + _ = try await store.deleteWorkouts(deviceId: "d1", sport: "running", from: 100, to: 100) + // Re-import attempts to resurrect — the guard must drop it. + let n = try await store.upsertWorkouts([WorkoutRow(startTs: 100, endTs: 200, sport: "running", source: "whoop", durationS: 100, energyKcal: nil, avgHr: nil, maxHr: nil, strain: nil, distanceM: nil, zonesJSON: nil, notes: nil)], deviceId: "d1") + XCTAssertEqual(n, 0) + // Idempotent tombstone insert (same kind+editSeq). + try await store.addTombstone(kind: "workout", deviceId: "d1", startTs: 100, endTs: nil, sport: "running", day: nil, key: nil, editSeq: 7) + } + func testHrRangeTombstoneFiltersInserts() async throws { + let store = try await WhoopStore.inMemory() + try await store.addTombstone(kind: "hrRange", deviceId: "d1", startTs: 1000, endTs: 2000, sport: nil, day: nil, key: nil, editSeq: 8) + try await store.insert(Streams(hr: [HRSample(ts: 1500, bpm: 60), HRSample(ts: 2500, bpm: 61)]), deviceId: "d1") + let survivors = try await store.hrSamples(deviceId: "d1", from: 0, to: 10_000, limit: 10) + XCTAssertEqual(survivors.map(\.ts), [2500]) + } + func testDeleteHelpersAndMetricGuard() async throws { + let store = try await WhoopStore.inMemory() + try await store.insert(Streams(hr: [HRSample(ts: 1200, bpm: 60)]), deviceId: "d1") + XCTAssertEqual(try await store.deleteHrRange(deviceId: "d1", fromTs: 1000, toTs: 2000), 1) + _ = try await store.upsertMetricSeries([MetricPoint(day: "2026-07-01", key: "ref_sleep_score", value: 80)], deviceId: "d1") + try await store.addTombstone(kind: "metricPoint", deviceId: "d1", startTs: nil, endTs: nil, sport: nil, day: "2026-07-01", key: "ref_sleep_score", editSeq: 9) + XCTAssertEqual(try await store.deleteMetricPoint(deviceId: "d1", day: "2026-07-01", key: "ref_sleep_score"), 1) + let n = try await store.upsertMetricSeries([MetricPoint(day: "2026-07-01", key: "ref_sleep_score", value: 81)], deviceId: "d1") + XCTAssertEqual(n, 0) // guard drops the tombstoned point + } +} +``` +(Adjust `WorkoutRow`/`MetricPoint`/`Streams` initializers to the real signatures — read the source files first; the test intent is contractual, exact initializers are whatever the package defines.) + +- [ ] **Step 2: FAIL** — `swift test --package-path Packages/WhoopStore --filter CloudTombstoneTests`. +- [ ] **Step 3: implement migration + store + guards + deletes** per Interfaces (read each modified file first; guards are small pre-filters, mirror existing code style; keep guard queries indexed). +- [ ] **Step 4: PASS filter + FULL package suite** (`swift test --package-path Packages/WhoopStore`) — zero regressions. +- [ ] **Step 5: Commit** `feat(store): cloudTombstone (v26) + resurrection guards + cloud delete helpers` + +--- + +### Task 3 (NOOP repo): `CloudSyncClient` + settings (gated app-target module) + +**Files:** +- Create: `Strand/CloudSync/CloudSyncSettings.swift` (Keychain URL+token, same pattern as `OuraTokenStore`) +- Create: `Strand/CloudSync/CloudEdit.swift` (Decodable journal row + `/edits` response) +- Create: `Strand/CloudSync/CloudSyncClient.swift` (fetch since-cursor, ack) +- Test: `StrandTests/CloudSyncClientTests.swift` (URLProtocol stub, same pattern as `OuraURLProtocolStub`) + +**Interfaces:** +- All files `#if CLOUD_SYNC` … `#endif`, header comment per house pattern. +- `CloudSyncSettings`: `static var serverURL: String?` / `static var token: String?` (Keychain service `"com.noop.cloudsync"`, account `"server-url"` / `"rw-token"`); `static var isConfigured: Bool`. +- `CloudEdit: Decodable, Equatable`: `seq: Int, editId: String, kind: String, payloadJSON: String, beforeJSON: String?, rationale: String?, appliedAt: Int, undoneBySeq: Int?, ackedAt: Int?`. +- `CloudSyncClient` (init `(baseURL: URL, token: String, session: URLSession = .shared)`): + - `fetchEdits(since: Int) async throws -> (edits: [CloudEdit], latestSeq: Int)` — GET `/edits?since=`, Bearer auth, decodes `{edits, latestSeq}`. + - `ack(seqs: [Int]) async throws -> Int` — POST `/edits/ack`, returns `acked`. + - Non-2xx → `CloudSyncError.badResponse(status, bodyPrefix)`. +- Tests: stubbed 200 happy paths (fixture JSON), 401 → error, malformed JSON → error. + +- [ ] **Step 1: failing tests → Step 2: implement → Step 3: PASS via** `xcodebuild -project Strand.xcodeproj -scheme Strand -destination 'platform=macOS' test -only-testing:StrandTests/CloudSyncClientTests CODE_SIGNING_ALLOWED=NO` (after adding `CLOUD_SYNC` to the untracked xcconfig — see Task 6 Step 1; for tests, StrandTests shares the app xcconfig so the flag applies). +- [ ] **Step 4: Commit** `feat(cloudsync): gated client + keychain settings (fork lane)` + +--- + +### Task 4 (NOOP repo): `CloudEditApplier` — journal → store, recompute-safe + +**Files:** +- Create: `Strand/CloudSync/CloudEditApplier.swift` +- Test: `StrandTests/CloudEditApplierTests.swift` (against `WhoopStore.inMemory()`) + +**Interfaces:** +- `struct CloudApplySummary: Equatable { var applied: Int; var skipped: Int; var needsAttention: Int; var touchedSleep: Bool; var touchedWorkouts: Bool; var touchedHr: Bool; var appliedSeqs: [Int] }` +- `enum CloudEditApplier { static func apply(_ edits: [CloudEdit], store: WhoopStore) async -> CloudApplySummary }` + - Processes in seq order. Skips (`skipped += 1`): `kind == "undo"`, `undoneBySeq != nil`, `set_baseline_note`, unknown kinds (forward-compat), malformed payloads (decode failure — log, skip, DO NOT crash). + - `adjust_sleep_bounds` → decode `{deviceId, startTs, newStartTs?, newEndTs?}`; fetch the session's current bounds via the store read; call `applySleepEdit(deviceId:detectedStartTs:newStartTs:newEndTs:)` (missing new value = keep current). 0-row result (session absent) → `needsAttention`. + - `edit_sleep_stages` → `{deviceId, startTs, stages[]}` re-encoded to the app's stagesJSON shape `[{start,end,stage}]` → `updateSleepStages(deviceId:detectedStartTs:stagesJSON:)`. NOTE: `updateSleepStages` only touches `userEdited=1` rows — when the target row has `userEdited=0`, first promote it via `applySleepEdit` with its CURRENT bounds (sets the flag), then update stages. 0 rows → `needsAttention`. + - `delete_workout` → `{deviceId, startTs, sport}` → `deleteWorkouts(deviceId:sport:from:startTs to:startTs)` + `addTombstone(kind:"workout"…, editSeq: seq)`. + - `fix_workout` → tombstone + delete original (as above), then upsert the corrected copy (original fields overlaid with `patch`) under `deviceId "noop-cloud"`; requires `beforeJSON` (the server snapshot) for the original's full fields — if `beforeJSON` nil → `needsAttention`. + - `add_workout` → upsert under `"noop-cloud"`. + - `delete_hr_range` → `deleteHrRange` + `addTombstone(kind:"hrRange"…)`; `delete_metric_point` → `deleteMetricPoint` + tombstone. + - First successful apply that used `"noop-cloud"` registers the pairedDevice row once (id `"noop-cloud"`, brand `"noop-cloud"`, model `"Cloud edits"`, `.cloudImport`, `.paired`) — same registration style as `OuraSyncWriter.persist` step 0. + - Sets `touchedSleep/touchedWorkouts/touchedHr` per kind; `appliedSeqs` collects for ack. +- Tests (in-memory store): one test per kind family — sleep bounds survive a re-`upsertSleepSessions` (recompute simulation asserts userEdited protection engaged); stage edit on a userEdited=0 row (promotion path); fix_workout produces tombstone + noop-cloud copy; malformed payload skips without throwing; undone row skipped; summary flags correct. + +- [ ] **Steps: failing tests → implement → PASS (`-only-testing:StrandTests/CloudEditApplierTests`) → commit** `feat(cloudsync): CloudEditApplier — recompute-safe journal application` + +--- + +### Task 5 (NOOP repo): sync coordinator + Apple Health re-export + UI card + +**Files:** +- Create: `Strand/CloudSync/CloudSyncCoordinator.swift` (fetch→apply→ack→cursor; HK trigger) +- Create: `Strand/CloudSync/CloudSyncModel.swift` (@MainActor ObservableObject; mirrors `OuraConnectModel` shape: busy/statusText; actions `pullNow(repo:)`, `saveSettings(url:token:)`) +- Modify: `Strand/Screens/DataSourcesView.swift` (add `cloudSyncCard` under `#if CLOUD_SYNC`, staggered below the Oura card; fields: server URL, token (SecureField), "Pull edits now" button, status line) +- Modify: `StrandiOS/Health/HealthKitBridge.swift` (ONLY IF NEEDED: expose a public `func rerunWriteBack(whoopStore:days:)` wrapper around the existing private write-back — read the file first; if a public trigger already exists, use it) +- Test: `StrandTests/CloudSyncCoordinatorTests.swift` + +**Interfaces:** +- `CloudSyncCoordinator.pull(store: WhoopStore, client: CloudSyncClient) async throws -> CloudApplySummary`: + 1. `since = store.cursor("cloud_edits") ?? 0` (the cursors table — `setCursor`/`cursor` exist on the store). + 2. `fetchEdits(since:)` → `CloudEditApplier.apply` → if `appliedSeqs` non-empty: `ack(seqs:)` then `setCursor("cloud_edits", latestSeq)`. Cursor advances to `latestSeq` even when everything was skipped-but-acked… **No**: cursor advances to `latestSeq` only after ack succeeds; if ack throws, cursor stays (re-apply is idempotent by design). + 3. Returns summary (UI renders "Applied 3 · skipped 1 · 0 need attention"). +- HK trigger (in `CloudSyncModel.pullNow`, iOS-only guard): when `summary.touchedSleep || touchedWorkouts || touchedHr`, call the write-back entry (days: 30) inside a `do/catch` that appends "· Apple Health re-export failed: " to status rather than failing the pull. +- Coordinator tests use a stub `CloudSyncClient` seam (protocol `CloudEditFetching` with fetch/ack, client conforms — mirror `OuraPageFetching` pattern): cursor advances only after ack; ack failure keeps cursor; needsAttention surfaces in summary. + +- [ ] **Steps: read HealthKitBridge.swift first (record the chosen trigger + its dedup semantics in the report) → failing tests → implement → PASS coordinator tests + BOTH app builds (`Strand` macOS CODE_SIGNING_ALLOWED=NO; `NOOPiOS` device destination signed) → commit** `feat(cloudsync): coordinator + Data Sources card + Apple Health re-export hook` + +--- + +### Task 6 (controller, not subagent): flag, device deploy, live E2E + +- [ ] **Step 1:** Add `CLOUD_SYNC` to the untracked xcconfig (`Strand/Oura/OuraSecrets.xcconfig`): append ` CLOUD_SYNC` to its `SWIFT_ACTIVE_COMPILATION_CONDITIONS` line (create the line `SWIFT_ACTIVE_COMPILATION_CONDITIONS = $(inherited) OURA_CLOUD_IMPORT CLOUD_SYNC` if absent). Mirror documentation line in `OuraSecrets.example.xcconfig` (committed). +- [ ] **Step 2:** `xcodegen generate`; build + install NOOPiOS on the iPhone (established pipeline); launch. +- [ ] **Step 3:** On-device: Data Sources → Cloud Sync card → paste `https://vk-noop-cloud.fly.dev` + rw token (from noop-cloud/.env) → **Pull edits now** (server journal currently holds the smoke-test rows: one edit_sleep_stages + its undo → expect "applied 0 · skipped 2" — both rows are undone/undo). +- [ ] **Step 4:** Real E2E: via Claude Code (noop-cloud MCP): propose+confirm a small `set_baseline_note` AND an `adjust_sleep_bounds` against a REAL night from VK's data (after his real .noopbak upload) — or against fixture data if the mirror still holds fixtures (then bounds target the fixture night, which won't exist on the phone → exercises needsAttention honestly). Pull on phone; verify NOOP UI reflects the sleep change (if real data) and Apple Health shows the corrected night (Health app). Ack verified: `GET /edits` shows `ackedAt` set. +- [ ] **Step 5:** Ledger + memory update. + +--- + +## Self-Review + +**Spec coverage (design §5 Phase 3):** pull transport `GET /edits?since=` ✅ (T1 ack + existing endpoint; T3 client); apply rules — sleep via `userEdited=1` paths ✅ (T4, incl. the updateSleepStages promotion subtlety), corrections under `"noop-cloud"` namespace + registration ✅ (T4), DB-backed tombstones replacing the UserDefaults gap ✅ (T2), ack-back ✅ (T1/T5). Apple Health via merged write-back ✅ (T5). Fork-only gating ✅ (Global Constraints + T6). +**Placeholder scan:** T2 test notes initializer signatures must be read from source — that is an explicit instruction to verify, not a TBD; all other steps carry code or exact commands. T5 deliberately defers HealthKitBridge internals to a read-first step because the file is #249's merged code (unknown to this plan) — the task's Interface pins the REQUIRED behavior (public trigger, days window, error containment). +**Type consistency:** `CloudEdit` fields = server `JournalRow` + `ackedAt` (T1); `CloudApplySummary` consumed by T5; store APIs referenced (applySleepEdit/updateSleepStages/deleteWorkouts/upsertWorkouts/upsertMetricSeries/insert/cursor/setCursor) verified to exist by controller scout (grep) — signatures re-checked in-task. +**Known risk:** `updateSleepStages` stage-string vocabulary (app uses its stagesJSON `stage` strings — verify in-task that "awake|light|deep|rem" match the app's legend; if the app uses different tokens (e.g. "wake"), map in the applier and record it). From 1d47014c181a403d2d91a489f7bd6a67c4f76c2a Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 14:02:12 -0700 Subject: [PATCH 049/171] feat(store): cloudTombstone (v26) + resurrection guards + cloud delete helpers Adds the local record of server-side journal deletes (workout / HR range / metric point) so a later sync, backfill, or live-BLE write can't silently resurrect data the user deleted via the cloud journal. - Migration v26-cloud-tombstone: cloudTombstone table, idempotent on (kind, editSeq), indexed on (kind, deviceId). - CloudTombstoneStore: addTombstone + workoutTombstones/hrTombstoneRanges/ metricPointTombstones reads, plus same-connection helpers guards call directly inside an existing syncWrite. - Resurrection guards: upsertWorkouts and upsertMetricSeries drop tombstoned rows before insert; StreamStore.insert filters HR samples inside a tombstoned range (one indexed pre-query per call, not per row). - CloudEditDeletes: deleteHrRange / deleteMetricPoint physical deletes. - DeviceRegistryStore.deviceScopedTables gains cloudTombstone so "delete all of this device's data" clears it too (caught by the existing testDeviceScopedTablesCoversEveryDeviceIdKeyedTable invariant test). 259/259 WhoopStore package tests pass (256 pre-existing + 3 new). --- .../Sources/WhoopStore/CloudEditDeletes.swift | 36 ++++++++ .../WhoopStore/CloudTombstoneStore.swift | 87 +++++++++++++++++++ .../Sources/WhoopStore/Database.swift | 24 +++++ .../WhoopStore/DeviceRegistryStore.swift | 5 ++ .../WhoopStore/JournalWorkoutAppleCache.swift | 7 ++ .../WhoopStore/MetricSeriesStore.swift | 7 ++ .../Sources/WhoopStore/StreamStore.swift | 19 +++- .../WhoopStoreTests/CloudTombstoneTests.swift | 37 ++++++++ 8 files changed, 220 insertions(+), 2 deletions(-) create mode 100644 Packages/WhoopStore/Sources/WhoopStore/CloudEditDeletes.swift create mode 100644 Packages/WhoopStore/Sources/WhoopStore/CloudTombstoneStore.swift create mode 100644 Packages/WhoopStore/Tests/WhoopStoreTests/CloudTombstoneTests.swift diff --git a/Packages/WhoopStore/Sources/WhoopStore/CloudEditDeletes.swift b/Packages/WhoopStore/Sources/WhoopStore/CloudEditDeletes.swift new file mode 100644 index 0000000000..7e6e4b9d52 --- /dev/null +++ b/Packages/WhoopStore/Sources/WhoopStore/CloudEditDeletes.swift @@ -0,0 +1,36 @@ +import Foundation +import GRDB + +// MARK: - Cloud edit deletes: physical DELETEs paired with a CloudTombstoneStore.addTombstone call. +// A server-side journal delete must remove the local row(s) AND record a tombstone (else the next +// sync/backfill/live-BLE write resurrects them) — a caller applies both, in either order. Plain +// DELETEs, no upsert semantics; each returns the number of rows actually removed. + +extension WhoopStore { + + /// Physically delete HR samples for a device with `ts` in `[fromTs, toTs]` inclusive. + /// Returns rows deleted. Pair with `addTombstone(kind: "hrRange", ...)` so a later re-sync + /// (backfill replay, or the strap re-streaming an already-deleted window) can't resurrect them. + @discardableResult + public func deleteHrRange(deviceId: String, fromTs: Int, toTs: Int) async throws -> Int { + try syncWrite { db in + try db.execute(sql: """ + DELETE FROM hrSample WHERE deviceId = ? AND ts >= ? AND ts <= ? + """, arguments: [deviceId, fromTs, toTs]) + return db.changesCount + } + } + + /// Physically delete one metric point by natural key (deviceId, day, key). Returns rows deleted. + /// Pair with `addTombstone(kind: "metricPoint", ...)` so a later re-derive/re-upsert can't + /// resurrect it. + @discardableResult + public func deleteMetricPoint(deviceId: String, day: String, key: String) async throws -> Int { + try syncWrite { db in + try db.execute(sql: """ + DELETE FROM metricSeries WHERE deviceId = ? AND day = ? AND key = ? + """, arguments: [deviceId, day, key]) + return db.changesCount + } + } +} diff --git a/Packages/WhoopStore/Sources/WhoopStore/CloudTombstoneStore.swift b/Packages/WhoopStore/Sources/WhoopStore/CloudTombstoneStore.swift new file mode 100644 index 0000000000..ceb30eff4f --- /dev/null +++ b/Packages/WhoopStore/Sources/WhoopStore/CloudTombstoneStore.swift @@ -0,0 +1,87 @@ +import Foundation +import GRDB + +// MARK: - v26 store: cloud edit tombstones (resurrection guards) +// A server-side journal delete (workout / HR range / metric point) is applied locally as a +// physical DELETE (see CloudEditDeletes) plus a tombstone row here, so a LATER re-sync/backfill/ +// live-BLE write can never silently resurrect the same data. Mirrors OuraRawStore/MetricSeriesStore: +// idempotent inserts, range/set reads, all GRDB work via the actor's syncWrite/syncRead helpers. +// +// The three `*TombstoneRows` helpers take an already-open `Database` connection rather than going +// through `syncRead`, so a guard running INSIDE another store's `syncWrite` block (upsertWorkouts, +// upsertMetricSeries, StreamStore.insert) queries on the SAME connection/transaction instead of +// opening a second one. The public `workoutTombstones`/etc. accessors below simply wrap them in +// `syncRead` for callers outside a write. + +extension WhoopStore { + + /// Workout tombstones for a device: `(startTs, sport)` pairs a resurrection guard must drop. + /// Shared by the public `workoutTombstones` read and the `upsertWorkouts` guard. + static func workoutTombstoneRows(_ db: Database, deviceId: String) throws -> [(startTs: Int, sport: String)] { + try Row.fetchAll(db, sql: """ + SELECT startTs, sport FROM cloudTombstone WHERE kind = 'workout' AND deviceId = ? + """, arguments: [deviceId]) + .map { (startTs: $0["startTs"], sport: $0["sport"]) } + } + + /// HR-range tombstones for a device: `(fromTs, toTs)` ranges a live/backfill insert must filter. + /// Shared by the public `hrTombstoneRanges` read and the `StreamStore.insert` guard. + static func hrTombstoneRangeRows(_ db: Database, deviceId: String) throws -> [(fromTs: Int, toTs: Int)] { + try Row.fetchAll(db, sql: """ + SELECT startTs, endTs FROM cloudTombstone WHERE kind = 'hrRange' AND deviceId = ? + """, arguments: [deviceId]) + .map { (fromTs: $0["startTs"], toTs: $0["endTs"]) } + } + + /// Metric-point tombstones for a device: `(day, key)` pairs a resurrection guard must drop. + /// Shared by the public `metricPointTombstones` read and the `upsertMetricSeries` guard. + static func metricPointTombstoneRows(_ db: Database, deviceId: String) throws -> [(day: String, key: String)] { + try Row.fetchAll(db, sql: """ + SELECT day, key FROM cloudTombstone WHERE kind = 'metricPoint' AND deviceId = ? + """, arguments: [deviceId]) + .map { (day: $0["day"], key: $0["key"]) } + } + + /// Record a cloud-edit tombstone. Idempotent by (kind, editSeq) — INSERT OR IGNORE, so replaying + /// the same journal entry (e.g. after a reconnect) is a no-op rather than a duplicate row. + /// `startTs`/`endTs`/`sport` apply to "workout"/"hrRange" kinds; `day`/`key` to "metricPoint" — + /// the caller passes nil for whichever fields don't apply to `kind`. + public func addTombstone(kind: String, deviceId: String, startTs: Int?, endTs: Int?, + sport: String?, day: String?, key: String?, editSeq: Int) async throws { + let now = Int(Date().timeIntervalSince1970) + try syncWrite { db in + try db.execute(sql: """ + INSERT OR IGNORE INTO cloudTombstone + (kind, deviceId, startTs, endTs, sport, day, key, editSeq, createdAt) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + """, arguments: [kind, deviceId, startTs, endTs, sport, day, key, editSeq, now]) + } + } + + /// Workout tombstones for a device (unordered). See `workoutTombstoneRows` for the guard-side use. + public func workoutTombstones(deviceId: String) async throws -> [(startTs: Int, sport: String)] { + try syncRead { db in try WhoopStore.workoutTombstoneRows(db, deviceId: deviceId) } + } + + /// HR-range tombstones for a device (unordered). See `hrTombstoneRangeRows` for the guard-side use. + public func hrTombstoneRanges(deviceId: String) async throws -> [(fromTs: Int, toTs: Int)] { + try syncRead { db in try WhoopStore.hrTombstoneRangeRows(db, deviceId: deviceId) } + } + + /// Metric-point tombstones for a device (unordered). See `metricPointTombstoneRows` for the guard-side use. + public func metricPointTombstones(deviceId: String) async throws -> [(day: String, key: String)] { + try syncRead { db in try WhoopStore.metricPointTombstoneRows(db, deviceId: deviceId) } + } +} + +/// Hashable natural key for the `upsertWorkouts` resurrection guard's pre-loaded tombstone Set. +struct WorkoutTombstoneKey: Hashable { + let startTs: Int + let sport: String +} + +/// Hashable natural key for the `upsertMetricSeries` resurrection guard's pre-loaded tombstone Set. +struct MetricPointTombstoneKey: Hashable { + let day: String + let key: String +} diff --git a/Packages/WhoopStore/Sources/WhoopStore/Database.swift b/Packages/WhoopStore/Sources/WhoopStore/Database.swift index 58d8e0272b..65e37909d9 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/Database.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/Database.swift @@ -500,6 +500,30 @@ extension WhoopStore { try db.create(index: "idx_ouraRaw_device_endpoint_day", on: "ouraRaw", columns: ["deviceId", "endpoint", "day"]) } + + // v26: cloud edit tombstones (Phase 3 cloud journal apply). When a server-side journal edit + // deletes a workout / HR range / metric point, NOOP physically deletes the local row(s) AND + // records a tombstone here — otherwise the NEXT sync/backfill/live-BLE write would silently + // resurrect the same row from the strap or a re-pulled cloud page. `editSeq` is the cloud + // journal's own sequence number; the (kind, editSeq) unique key makes replaying the same + // journal entry idempotent (INSERT OR IGNORE, never a duplicate tombstone). Additive only — + // a NEW table, no existing row touched. + migrator.registerMigration("v26-cloud-tombstone") { db in + try db.create(table: "cloudTombstone") { t in + t.autoIncrementedPrimaryKey("id") + t.column("kind", .text).notNull() // "workout" | "hrRange" | "metricPoint" + t.column("deviceId", .text).notNull() + t.column("startTs", .integer) // workout: startTs; hrRange: fromTs + t.column("endTs", .integer) // hrRange: toTs + t.column("sport", .text) // workout + t.column("day", .text) // metricPoint + t.column("key", .text) // metricPoint + t.column("editSeq", .integer).notNull() // server journal seq (idempotency) + t.column("createdAt", .integer).notNull() + t.uniqueKey(["kind", "editSeq"]) + } + try db.create(index: "idx_cloudTombstone_kind_device", on: "cloudTombstone", columns: ["kind", "deviceId"]) + } return migrator } } diff --git a/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift b/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift index 2bd7e1edcc..b8ffbfa11c 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift @@ -90,6 +90,11 @@ public struct DeviceRegistryStore: Sendable { // v25-oura-raw: the opt-in Oura cloud-import raw archive is deviceId-keyed too, so "delete this // device's data" must clear it — else an imported Oura source's payloads would survive deletion. "ouraRaw", + // v26-cloud-tombstone: cloud-edit tombstones are deviceId-keyed too, so "delete this device's + // data" must clear them — else a deleted device would leave orphaned edit-history metadata + // behind. A later full re-sync from the cloud journal replays deletes from scratch and + // re-creates any tombstones still needed. + "cloudTombstone", ] /// Permanently delete every recorded sample/derived row belonging to one device, across all diff --git a/Packages/WhoopStore/Sources/WhoopStore/JournalWorkoutAppleCache.swift b/Packages/WhoopStore/Sources/WhoopStore/JournalWorkoutAppleCache.swift index 217716b229..183cf9ff88 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/JournalWorkoutAppleCache.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/JournalWorkoutAppleCache.swift @@ -138,11 +138,18 @@ extension WhoopStore { } /// Upsert workouts. Natural key (deviceId, startTs, sport). Returns rows changed. + /// + /// Resurrection guard: a cloud-journal delete tombstones a workout's (startTs, sport) — see + /// `CloudTombstoneStore`. A row matching a tombstone is dropped BEFORE the insert, so a later + /// re-import (backfill replay, cloud re-pull) can't resurrect a workout the user deleted. @discardableResult public func upsertWorkouts(_ rows: [WorkoutRow], deviceId: String) async throws -> Int { try syncWrite { db in + let tombstoned = Set(try WhoopStore.workoutTombstoneRows(db, deviceId: deviceId) + .map { WorkoutTombstoneKey(startTs: $0.startTs, sport: $0.sport) }) var n = 0 for r in rows { + if tombstoned.contains(WorkoutTombstoneKey(startTs: r.startTs, sport: r.sport)) { continue } try db.execute(sql: """ INSERT INTO workout (deviceId, startTs, endTs, sport, source, durationS, energyKcal, diff --git a/Packages/WhoopStore/Sources/WhoopStore/MetricSeriesStore.swift b/Packages/WhoopStore/Sources/WhoopStore/MetricSeriesStore.swift index 2c951d105f..e0875c5d9b 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/MetricSeriesStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/MetricSeriesStore.swift @@ -27,11 +27,18 @@ extension WhoopStore { /// Upsert metric points. Natural key (deviceId, day, key). Returns rows changed. /// Idempotent: re-upserting the same (deviceId, day, key) updates `value` in place rather than /// creating a duplicate. + /// + /// Resurrection guard: a cloud-journal delete tombstones a metric point's (day, key) — see + /// `CloudTombstoneStore`. A row matching a tombstone is dropped BEFORE the insert, so a later + /// re-derive/re-upsert can't resurrect a point the user deleted. @discardableResult public func upsertMetricSeries(_ rows: [MetricPoint], deviceId: String) async throws -> Int { try syncWrite { db in + let tombstoned = Set(try WhoopStore.metricPointTombstoneRows(db, deviceId: deviceId) + .map { MetricPointTombstoneKey(day: $0.day, key: $0.key) }) var n = 0 for r in rows { + if tombstoned.contains(MetricPointTombstoneKey(day: r.day, key: r.key)) { continue } try db.execute(sql: """ INSERT INTO metricSeries (deviceId, day, key, value) diff --git a/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift b/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift index c79522aa18..77ca2655ad 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift @@ -45,12 +45,27 @@ extension WhoopStore { // over potentially millions of historical rows). cachedStatement persists the compiled // statement on the connection across insert() calls too. Each loop is guarded so empty // streams (the common live case) compile nothing. - if !streams.hr.isEmpty { + // + // Resurrection guard: a cloud-journal delete tombstones an HR range — see + // `CloudTombstoneStore`. Samples inside a tombstoned range are dropped BEFORE the insert, + // so a re-sync (backfill replay, or the strap re-streaming an already-deleted window) + // can't resurrect them. This runs on the LIVE BLE path, so it's one SELECT per call + // (not per row), and only when there's HR data to filter. + var hrRows = streams.hr + if !hrRows.isEmpty { + let tombstones = try WhoopStore.hrTombstoneRangeRows(db, deviceId: deviceId) + if !tombstones.isEmpty { + hrRows = hrRows.filter { s in + !tombstones.contains { $0.fromTs <= s.ts && s.ts <= $0.toTs } + } + } + } + if !hrRows.isEmpty { let stmt = try db.cachedStatement(sql: """ INSERT INTO hrSample (deviceId, ts, bpm) VALUES (?, ?, ?) ON CONFLICT(deviceId, ts) DO NOTHING """) - for s in streams.hr { + for s in hrRows { try stmt.execute(arguments: [deviceId, s.ts, s.bpm]) hr += db.changesCount } diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/CloudTombstoneTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/CloudTombstoneTests.swift new file mode 100644 index 0000000000..3f706aea87 --- /dev/null +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/CloudTombstoneTests.swift @@ -0,0 +1,37 @@ +import XCTest +@testable import WhoopStore +import WhoopProtocol + +final class CloudTombstoneTests: XCTestCase { + func testWorkoutTombstoneBlocksResurrection() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertWorkouts([WorkoutRow(startTs: 100, endTs: 200, sport: "running", source: "whoop", durationS: 100, energyKcal: nil, avgHr: nil, maxHr: nil, strain: nil, distanceM: nil, zonesJSON: nil, notes: nil)], deviceId: "d1") + try await store.addTombstone(kind: "workout", deviceId: "d1", startTs: 100, endTs: nil, sport: "running", day: nil, key: nil, editSeq: 7) + _ = try await store.deleteWorkouts(deviceId: "d1", sport: "running", from: 100, to: 100) + // Re-import attempts to resurrect — the guard must drop it. + let n = try await store.upsertWorkouts([WorkoutRow(startTs: 100, endTs: 200, sport: "running", source: "whoop", durationS: 100, energyKcal: nil, avgHr: nil, maxHr: nil, strain: nil, distanceM: nil, zonesJSON: nil, notes: nil)], deviceId: "d1") + XCTAssertEqual(n, 0) + // Idempotent tombstone insert (same kind+editSeq). + try await store.addTombstone(kind: "workout", deviceId: "d1", startTs: 100, endTs: nil, sport: "running", day: nil, key: nil, editSeq: 7) + } + func testHrRangeTombstoneFiltersInserts() async throws { + let store = try await WhoopStore.inMemory() + try await store.addTombstone(kind: "hrRange", deviceId: "d1", startTs: 1000, endTs: 2000, sport: nil, day: nil, key: nil, editSeq: 8) + try await store.insert(Streams(hr: [HRSample(ts: 1500, bpm: 60), HRSample(ts: 2500, bpm: 61)]), deviceId: "d1") + let survivors = try await store.hrSamples(deviceId: "d1", from: 0, to: 10_000, limit: 10) + XCTAssertEqual(survivors.map(\.ts), [2500]) + } + func testDeleteHelpersAndMetricGuard() async throws { + let store = try await WhoopStore.inMemory() + try await store.insert(Streams(hr: [HRSample(ts: 1200, bpm: 60)]), deviceId: "d1") + // Hoisted out of XCTAssertEqual's autoclosure: this toolchain rejects `await` inside it. + let deletedHr = try await store.deleteHrRange(deviceId: "d1", fromTs: 1000, toTs: 2000) + XCTAssertEqual(deletedHr, 1) + _ = try await store.upsertMetricSeries([MetricPoint(day: "2026-07-01", key: "ref_sleep_score", value: 80)], deviceId: "d1") + try await store.addTombstone(kind: "metricPoint", deviceId: "d1", startTs: nil, endTs: nil, sport: nil, day: "2026-07-01", key: "ref_sleep_score", editSeq: 9) + let deletedMetric = try await store.deleteMetricPoint(deviceId: "d1", day: "2026-07-01", key: "ref_sleep_score") + XCTAssertEqual(deletedMetric, 1) + let n = try await store.upsertMetricSeries([MetricPoint(day: "2026-07-01", key: "ref_sleep_score", value: 81)], deviceId: "d1") + XCTAssertEqual(n, 0) // guard drops the tombstoned point + } +} From 99afb13cf929cc92f70f142feb1d4deec3bb4dcb Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 14:21:07 -0700 Subject: [PATCH 050/171] fix(store): tombstones survive device-data wipes + guard LabMarker reprojection Two resurrection-guard gaps in the v26 cloudTombstone work: 1. LabMarkerStore.reprojectCells wrote metricSeries directly, bypassing the metricPoint tombstone guard every other writer (upsertMetricSeries) respects. A lab-marker edit could resurrect a metric point the cloud journal had deleted. Now pre-loads metricPointTombstoneRows for the constant labBookSourceId (the deviceId the projection actually writes under) and skips tombstoned (day, key) cells before the insert. 2. DeviceRegistryStore.deviceScopedTables included cloudTombstone, so "Delete this device's data" erased the guard's own memory, defeating it on the next BLE backfill or cloud re-sync. Moved cloudTombstone to a new deviceScopedTableExemptions allowlist that the coverage invariant test now also honors, and added CloudTombstoneStore.deleteCloudTombstones( deviceId:) (mirrors deleteOuraRaw) for a future full-disconnect flow that does mean to forget a device's cloud-edit history. --- .../WhoopStore/CloudTombstoneStore.swift | 11 ++++ .../WhoopStore/DeviceRegistryStore.swift | 23 ++++++-- .../Sources/WhoopStore/LabMarkerStore.swift | 10 ++++ .../WhoopStoreTests/CloudTombstoneTests.swift | 56 +++++++++++++++++++ .../DeviceRegistryStoreTests.swift | 14 +++-- 5 files changed, 103 insertions(+), 11 deletions(-) diff --git a/Packages/WhoopStore/Sources/WhoopStore/CloudTombstoneStore.swift b/Packages/WhoopStore/Sources/WhoopStore/CloudTombstoneStore.swift index ceb30eff4f..ba02476921 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/CloudTombstoneStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/CloudTombstoneStore.swift @@ -72,6 +72,17 @@ extension WhoopStore { public func metricPointTombstones(deviceId: String) async throws -> [(day: String, key: String)] { try syncRead { db in try WhoopStore.metricPointTombstoneRows(db, deviceId: deviceId) } } + + /// Remove every cloud-edit tombstone for a device (used by a future full-disconnect flow — + /// `deleteAllData` deliberately does NOT cover `cloudTombstone`, see + /// `DeviceRegistryStore.deviceScopedTableExemptions`). Mirrors `deleteOuraRaw`. Returns rows deleted. + @discardableResult + public func deleteCloudTombstones(deviceId: String) async throws -> Int { + try syncWrite { db in + try db.execute(sql: "DELETE FROM cloudTombstone WHERE deviceId = ?", arguments: [deviceId]) + return db.changesCount + } + } } /// Hashable natural key for the `upsertWorkouts` resurrection guard's pre-loaded tombstone Set. diff --git a/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift b/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift index b8ffbfa11c..a997ae7a24 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift @@ -90,18 +90,29 @@ public struct DeviceRegistryStore: Sendable { // v25-oura-raw: the opt-in Oura cloud-import raw archive is deviceId-keyed too, so "delete this // device's data" must clear it — else an imported Oura source's payloads would survive deletion. "ouraRaw", - // v26-cloud-tombstone: cloud-edit tombstones are deviceId-keyed too, so "delete this device's - // data" must clear them — else a deleted device would leave orphaned edit-history metadata - // behind. A later full re-sync from the cloud journal replays deletes from scratch and - // re-creates any tombstones still needed. - "cloudTombstone", ] + /// deviceId-keyed tables deliberately EXCLUDED from `deviceScopedTables` — a normal "delete this + /// device's data" wipe must NOT touch them, even though they carry a `deviceId` column. + /// + /// - `cloudTombstone` (v26-cloud-tombstone) is the guard's memory of cloud-journal deletes (a + /// workout / HR range / metric point the user removed on the cloud side). If a device-data wipe + /// cleared it, the very next BLE backfill or cloud re-sync could silently resurrect data the user + /// already deleted — tombstones must outlive per-device data wipes. A dedicated + /// `CloudTombstoneStore.deleteCloudTombstones(deviceId:)` exists for sites that DO mean to forget + /// a device's cloud-edit history (a full disconnect), never `deleteAllData`. + /// + /// `DeviceRegistryStoreTests.testDeviceScopedTablesCoversEveryDeviceIdKeyedTable` treats this list + /// as the only legitimate reason a deviceId-keyed table can be absent from `deviceScopedTables`; + /// any other uncovered table still fails that test. + static let deviceScopedTableExemptions = ["cloudTombstone"] + /// Permanently delete every recorded sample/derived row belonging to one device, across all /// `deviceId`-keyed tables, in a single transaction (all-or-nothing). The `pairedDevice` registry /// row is left intact — the caller archives/removes that separately. Tables are deleted defensively /// with `DELETE FROM WHERE deviceId = ?`; a missing table would throw, but every table here - /// is created unconditionally by the migrator, so the set is stable. + /// is created unconditionally by the migrator, so the set is stable. `cloudTombstone` is + /// deliberately NOT among these tables — see `deviceScopedTableExemptions`. public func deleteAllData(deviceId: String) throws { try dbQueue.write { db in for table in Self.deviceScopedTables { diff --git a/Packages/WhoopStore/Sources/WhoopStore/LabMarkerStore.swift b/Packages/WhoopStore/Sources/WhoopStore/LabMarkerStore.swift index ef6e26f18e..4789d07dbe 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/LabMarkerStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/LabMarkerStore.swift @@ -214,8 +214,18 @@ extension WhoopStore { /// from the CURRENT `labMarker` rows. Latest-numeric-per-day wins; if no numeric /// reading remains for a cell, its projected day is deleted (so a removed/last /// reading never leaves a stale projected value behind). + /// + /// Resurrection guard: a cloud-journal delete tombstones a metric point's (day, key) — see + /// `CloudTombstoneStore`. Every cell this projects lands in `metricSeries` under the constant + /// `labBookSourceId` (not the marker's own `deviceId`), so tombstones are loaded for THAT id — a + /// tombstoned cell is skipped entirely, so a later marker upsert/delete can't resurrect a point + /// the user deleted on the cloud side. private func reprojectCells(_ db: Database, cells: Set) throws { + let tombstoned = Set(try WhoopStore.metricPointTombstoneRows(db, deviceId: WhoopStore.labBookSourceId) + .map { MetricPointTombstoneKey(day: $0.day, key: $0.key) }) for cell in cells { + if tombstoned.contains(MetricPointTombstoneKey(day: cell.day, key: cell.markerKey)) { continue } + // Latest NUMERIC reading for this (markerKey, day): greatest takenAt with a // non-null value. Matches LabBookProjection.project(.latest) on numeric rows. let latest = try Double.fetchOne(db, sql: """ diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/CloudTombstoneTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/CloudTombstoneTests.swift index 3f706aea87..cc78b2a2e1 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/CloudTombstoneTests.swift +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/CloudTombstoneTests.swift @@ -34,4 +34,60 @@ final class CloudTombstoneTests: XCTestCase { let n = try await store.upsertMetricSeries([MetricPoint(day: "2026-07-01", key: "ref_sleep_score", value: 81)], deviceId: "d1") XCTAssertEqual(n, 0) // guard drops the tombstoned point } + + // Finding 1: LabMarkerStore.reprojectCells wrote metricSeries directly, bypassing the + // metricPoint tombstone guard every other writer (upsertMetricSeries) respects. A lab-marker + // upsert that reprojects a tombstoned (day, key) cell must NOT resurrect it. + func testLabMarkerReprojectionRespectsMetricPointTombstone() async throws { + let store = try await WhoopStore.inMemory() + // The lab-book projection always lands under the constant `labBookSourceId`, not the + // marker's own deviceId (see LabMarkerStore) — the tombstone must be recorded against that + // same id to guard the cell that reprojectCells actually writes. + try await store.addTombstone(kind: "metricPoint", deviceId: WhoopStore.labBookSourceId, + startTs: nil, endTs: nil, sport: nil, + day: "2026-01-10", key: "ldl", editSeq: 11) + // Upserting a numeric reading for that exact (day, key) triggers reprojectCells — the guard + // must drop the cell instead of resurrecting it into metricSeries. + _ = try await store.upsertLabMarkers([ + LabMarkerRow(id: "a", deviceId: "my-whoop", markerKey: "ldl", category: "bloodPanel", + day: "2026-01-10", takenAt: 1_736_500_000, value: 3.4, valueText: nil, + unit: "mmol/L", source: "manual", note: nil, referenceText: nil), + ]) + let proj = try await store.metricSeries(deviceId: WhoopStore.labBookSourceId, + key: "ldl", from: "2026-01-01", to: "2026-12-31") + XCTAssertEqual(proj.count, 0, "tombstoned cell must stay absent from metricSeries") + // The marker row itself is untouched — only the metricSeries projection is guarded. + let markers = try await store.labMarkers(deviceId: "my-whoop", markerKey: "ldl") + XCTAssertEqual(markers.count, 1) + } + + // Finding 2: cloudTombstone must survive "Delete this device's data" (DeviceRegistryStore + // .deleteAllData, reachable from DevicesView's keep-paired wipe) or the guard's memory is erased + // and a later BLE backfill/cloud re-sync can resurrect data the user deleted on the cloud side. + func testDeviceDataWipePreservesTombstoneUntilExplicitlyDeleted() async throws { + let store = try await WhoopStore.inMemory() + let workout = WorkoutRow(startTs: 100, endTs: 200, sport: "running", source: "whoop", + durationS: 100, energyKcal: nil, avgHr: nil, maxHr: nil, strain: nil, + distanceM: nil, zonesJSON: nil, notes: nil) + _ = try await store.upsertWorkouts([workout], deviceId: "d1") + try await store.addTombstone(kind: "workout", deviceId: "d1", startTs: 100, endTs: nil, + sport: "running", day: nil, key: nil, editSeq: 21) + + try await store.deleteAllData(deviceId: "d1") + + let workoutsAfterWipe = try await store.workouts(deviceId: "d1", from: 0, to: 1000, limit: 10) + XCTAssertEqual(workoutsAfterWipe.count, 0, "device-data wipe clears the workout") + let tombstonesAfterWipe = try await store.workoutTombstones(deviceId: "d1") + XCTAssertEqual(tombstonesAfterWipe.count, 1, "device-data wipe must NOT erase the tombstone") + + // A later re-sync/backfill re-importing the same workout must still be blocked. + let resurrected = try await store.upsertWorkouts([workout], deviceId: "d1") + XCTAssertEqual(resurrected, 0, "tombstone still blocks resurrection after a device-data wipe") + + // A dedicated full-disconnect flow (not deleteAllData) can forget the tombstone. + let deletedCount = try await store.deleteCloudTombstones(deviceId: "d1") + XCTAssertEqual(deletedCount, 1) + let tombstonesAfterDelete = try await store.workoutTombstones(deviceId: "d1") + XCTAssertEqual(tombstonesAfterDelete.count, 0) + } } diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/DeviceRegistryStoreTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/DeviceRegistryStoreTests.swift index f9dfb7c724..8cb6beac8d 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/DeviceRegistryStoreTests.swift +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/DeviceRegistryStoreTests.swift @@ -117,9 +117,11 @@ final class DeviceRegistryStoreTests: XCTestCase { } // Regression guard (audit finding): every table with a `deviceId` column MUST appear in - // `deviceScopedTables`, or `deleteAllData` silently leaves that device's rows behind — a privacy - // defect for a delete-means-gone app. Enumerate the live schema and fail if any deviceId-keyed table - // is uncovered, so a future migration that adds one can't reintroduce the gap. + // `deviceScopedTables` OR the explicit `deviceScopedTableExemptions` allowlist, or `deleteAllData` + // silently leaves that device's rows behind — a privacy defect for a delete-means-gone app. + // Enumerate the live schema and fail if any deviceId-keyed table is uncovered by either list, so a + // future migration that adds one can't reintroduce the gap (and can't silently opt out of the wipe + // either — a genuine exemption must be named in `deviceScopedTableExemptions` with a reason). func testDeviceScopedTablesCoversEveryDeviceIdKeyedTable() throws { let dbq = try makeDB() let uncovered = try dbq.read { db -> [String] in @@ -131,14 +133,16 @@ final class DeviceRegistryStoreTests: XCTestCase { for table in tables { let cols = try Row.fetchAll(db, sql: "PRAGMA table_info(\(table))") let hasDeviceId = cols.contains { ($0["name"] as String?) == "deviceId" } - if hasDeviceId && !DeviceRegistryStore.deviceScopedTables.contains(table) { + let covered = DeviceRegistryStore.deviceScopedTables.contains(table) + || DeviceRegistryStore.deviceScopedTableExemptions.contains(table) + if hasDeviceId && !covered { missing.append(table) } } return missing } XCTAssertTrue(uncovered.isEmpty, - "deviceId-keyed tables missing from deviceScopedTables (deleteAllData would skip them): \(uncovered)") + "deviceId-keyed tables missing from deviceScopedTables/deviceScopedTableExemptions (deleteAllData would skip them): \(uncovered)") } func testDayOwnershipUpsertAndRead() throws { From dc08f77c64f3bab83e45a5867f9550b0e1710578 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 14:35:24 -0700 Subject: [PATCH 051/171] feat(cloudsync): gated client + keychain settings (fork lane) --- Strand/CloudSync/CloudEdit.swift | 27 ++++ Strand/CloudSync/CloudSyncClient.swift | 103 +++++++++++++++ Strand/CloudSync/CloudSyncSettings.swift | 77 +++++++++++ StrandTests/CloudSyncClientTests.swift | 155 +++++++++++++++++++++++ 4 files changed, 362 insertions(+) create mode 100644 Strand/CloudSync/CloudEdit.swift create mode 100644 Strand/CloudSync/CloudSyncClient.swift create mode 100644 Strand/CloudSync/CloudSyncSettings.swift create mode 100644 StrandTests/CloudSyncClientTests.swift diff --git a/Strand/CloudSync/CloudEdit.swift b/Strand/CloudSync/CloudEdit.swift new file mode 100644 index 0000000000..fc8d19735c --- /dev/null +++ b/Strand/CloudSync/CloudEdit.swift @@ -0,0 +1,27 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +#if CLOUD_SYNC +import Foundation + +/// One row of the noop-cloud server's append-only edit journal. Field names match the server's +/// JSON keys exactly (camelCase), so no CodingKeys translation is needed. +struct CloudEdit: Decodable, Equatable { + let seq: Int + let editId: String + let kind: String + let payloadJSON: String + let beforeJSON: String? + let rationale: String? + let appliedAt: Int + let undoneBySeq: Int? + let ackedAt: Int? +} + +/// The `GET /edits` response envelope: the page of journal rows since the requested cursor, plus +/// the server's latest known seq (so the caller can detect it's fully caught up). +struct CloudEditsResponse: Decodable { + let edits: [CloudEdit] + let latestSeq: Int +} +#endif // CLOUD_SYNC diff --git a/Strand/CloudSync/CloudSyncClient.swift b/Strand/CloudSync/CloudSyncClient.swift new file mode 100644 index 0000000000..434853408c --- /dev/null +++ b/Strand/CloudSync/CloudSyncClient.swift @@ -0,0 +1,103 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +#if CLOUD_SYNC +import Foundation + +/// User-facing failure reasons for the cloud-sync lane, mapped to clear, non-crashing messages. +/// Mirrors OuraError's shape (LocalizedError + errorDescription). +enum CloudSyncError: LocalizedError, Equatable { + case badResponse(Int, String) + case network(String) + case decode + + var errorDescription: String? { + switch self { + case .badResponse(let code, let detail): + let extra = detail.isEmpty ? "" : " — \(detail)" + return "The cloud sync server returned an error (\(code))\(extra)." + case .network(let d): + return "Network problem talking to the cloud sync server: \(d)" + case .decode: + return "Couldn't read the cloud sync server's response." + } + } +} + +/// The noop-cloud edit-journal client. Injected `session` makes it URLProtocol-testable. +/// Networking lives here in the app target by design (mirrors Strand/Oura/OuraAPIClient.swift). +final class CloudSyncClient { + private let baseURL: URL + private let token: String + private let session: URLSession + + init(baseURL: URL, token: String, session: URLSession = .shared) { + self.baseURL = baseURL + self.token = token + self.session = session + } + + /// GET /edits?since=, Bearer-authed. Returns the page of rows plus the server's latest seq. + func fetchEdits(since: Int) async throws -> (edits: [CloudEdit], latestSeq: Int) { + var req = URLRequest(url: url(path: "edits", query: ["since": String(since)])) + req.setValue("Bearer \(token)", forHTTPHeaderField: "Authorization") + let (data, status) = try await send(req) + guard (200..<300).contains(status) else { + throw CloudSyncError.badResponse(status, bodyPrefix(data)) + } + guard let decoded = try? JSONDecoder().decode(CloudEditsResponse.self, from: data) else { + throw CloudSyncError.decode + } + return (decoded.edits, decoded.latestSeq) + } + + /// POST /edits/ack {"seqs":[...]}, Bearer-authed. Returns the count the server actually acked. + func ack(seqs: [Int]) async throws -> Int { + var req = URLRequest(url: url(path: "edits/ack")) + req.httpMethod = "POST" + req.setValue("Bearer \(token)", forHTTPHeaderField: "Authorization") + req.setValue("application/json", forHTTPHeaderField: "Content-Type") + req.httpBody = try JSONEncoder().encode(AckRequest(seqs: seqs)) + let (data, status) = try await send(req) + guard (200..<300).contains(status) else { + throw CloudSyncError.badResponse(status, bodyPrefix(data)) + } + guard let decoded = try? JSONDecoder().decode(AckResponse.self, from: data) else { + throw CloudSyncError.decode + } + return decoded.acked + } + + private func url(path: String, query: [String: String] = [:]) -> URL { + var comps = URLComponents(string: "\(baseURL.absoluteString)/\(path)")! + if !query.isEmpty { + comps.queryItems = query.map { URLQueryItem(name: $0.key, value: $0.value) } + } + return comps.url! + } + + private func send(_ req: URLRequest) async throws -> (Data, Int) { + do { + let (data, resp) = try await session.data(for: req) + return (data, (resp as? HTTPURLResponse)?.statusCode ?? 0) + } catch { + throw CloudSyncError.network(error.localizedDescription) + } + } + + /// Truncates the response body to a bounded prefix for error messages (never surface an + /// unbounded server body through an error description). + private func bodyPrefix(_ data: Data) -> String { + let s = String(data: data, encoding: .utf8) ?? "" + return s.count > 200 ? String(s.prefix(200)) : s + } +} + +private struct AckRequest: Encodable { + let seqs: [Int] +} + +private struct AckResponse: Decodable { + let acked: Int +} +#endif // CLOUD_SYNC diff --git a/Strand/CloudSync/CloudSyncSettings.swift b/Strand/CloudSync/CloudSyncSettings.swift new file mode 100644 index 0000000000..e58caa33bb --- /dev/null +++ b/Strand/CloudSync/CloudSyncSettings.swift @@ -0,0 +1,77 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +#if CLOUD_SYNC +import Foundation +import Security + +/// Keychain Services wrapper for the noop-cloud server URL + read-write token. One service, two +/// generic-password accounts, plain UTF-8 string values — neither ever lands in UserDefaults, a +/// plist, or on disk in the clear. Mirrors OuraTokenStore's get/set/delete shape; value encoding +/// (trim, empty-means-clear) mirrors AICoach's AIKeyStore. +enum CloudSyncSettings { + private static let service = "com.noop.cloudsync" + private static let serverURLAccount = "server-url" + private static let tokenAccount = "rw-token" + + private static func baseQuery(account: String) -> [String: Any] { + [ + kSecClass as String: kSecClassGenericPassword, + kSecAttrService as String: service, + kSecAttrAccount as String: account, + ] + } + + /// The noop-cloud server base URL (e.g. "https://vk-noop-cloud.fly.dev"), or nil if unset. + static var serverURL: String? { + get { load(account: serverURLAccount) } + set { save(newValue, account: serverURLAccount) } + } + + /// The read-write bearer token for the noop-cloud server, or nil if unset. + static var token: String? { + get { load(account: tokenAccount) } + set { save(newValue, account: tokenAccount) } + } + + /// True once both a server URL and a token are stored. + static var isConfigured: Bool { serverURL != nil && token != nil } + + /// Remove both stored values. + static func clear() { + SecItemDelete(baseQuery(account: serverURLAccount) as CFDictionary) + SecItemDelete(baseQuery(account: tokenAccount) as CFDictionary) + } + + /// Store (or replace) `value` for `account`. Empty/whitespace input clears the item instead + /// (so an accidental blank Save can't leave a dead-but-present Keychain entry). + @discardableResult + private static func save(_ value: String?, account: String) -> Bool { + let trimmed = (value ?? "").trimmingCharacters(in: .whitespacesAndNewlines) + let query = baseQuery(account: account) + + // Delete any existing item first so we always insert a single, fresh value (or none). + SecItemDelete(query as CFDictionary) + guard !trimmed.isEmpty else { return true } + guard let data = trimmed.data(using: .utf8) else { return false } + + var attrs = query + attrs[kSecValueData as String] = data + attrs[kSecAttrAccessible as String] = kSecAttrAccessibleAfterFirstUnlock + return SecItemAdd(attrs as CFDictionary, nil) == errSecSuccess + } + + /// Load the stored value for `account`, or nil if none. + private static func load(account: String) -> String? { + var query = baseQuery(account: account) + query[kSecReturnData as String] = kCFBooleanTrue + query[kSecMatchLimit as String] = kSecMatchLimitOne + var item: CFTypeRef? + guard SecItemCopyMatching(query as CFDictionary, &item) == errSecSuccess, + let data = item as? Data, + let str = String(data: data, encoding: .utf8), + !str.isEmpty else { return nil } + return str + } +} +#endif // CLOUD_SYNC diff --git a/StrandTests/CloudSyncClientTests.swift b/StrandTests/CloudSyncClientTests.swift new file mode 100644 index 0000000000..81d9b16c76 --- /dev/null +++ b/StrandTests/CloudSyncClientTests.swift @@ -0,0 +1,155 @@ +// Tests the CLOUD_SYNC-gated cloud sync client + settings; compiled only when the flag is set +// (StrandTests shares the app's OuraConfig.xcconfig, so flag + creds arrive together). +#if CLOUD_SYNC +import XCTest +@testable import Strand + +final class CloudSyncClientTests: XCTestCase { + override func setUp() { super.setUp(); OuraURLProtocolStub.reset() } + + private let baseURL = URL(string: "https://cloud.example.com")! + + private func makeClient() -> CloudSyncClient { + CloudSyncClient(baseURL: baseURL, token: "tok-1", session: OuraURLProtocolStub.session()) + } + + /// One journal row. `payloadJSON`'s embedded quotes are hand-escaped at the JSON level (`\"`) + /// inside a raw string literal, so no Swift-level escaping is needed. + private let editFixtureJSON = + #"{"seq":3,"editId":"e-1","kind":"adjust_sleep_bounds","payloadJSON":"{\"deviceId\":\"d1\"}","beforeJSON":null,"rationale":"user requested","appliedAt":1752300000,"undoneBySeq":null,"ackedAt":null}"# + + // MARK: - fetchEdits + + func testFetchEditsDecodesEditsAndLatestSeq() async throws { + let body = (#"{"edits":["# + editFixtureJSON + #"],"latestSeq":3}"#).data(using: .utf8)! + OuraURLProtocolStub.queue = [.init(status: 200, body: body)] + let client = makeClient() + + let result = try await client.fetchEdits(since: 0) + + XCTAssertEqual(result.latestSeq, 3) + XCTAssertEqual(result.edits.count, 1) + let edit = result.edits[0] + XCTAssertEqual(edit.seq, 3) + XCTAssertEqual(edit.editId, "e-1") + XCTAssertEqual(edit.kind, "adjust_sleep_bounds") + XCTAssertEqual(edit.payloadJSON, #"{"deviceId":"d1"}"#) + XCTAssertNil(edit.beforeJSON) + XCTAssertEqual(edit.rationale, "user requested") + XCTAssertEqual(edit.appliedAt, 1_752_300_000) + XCTAssertNil(edit.undoneBySeq) + XCTAssertNil(edit.ackedAt) + } + + func testFetchEditsSendsSinceInQueryString() async throws { + OuraURLProtocolStub.queue = [.init(status: 200, + body: #"{"edits":[],"latestSeq":0}"#.data(using: .utf8)!)] + let client = makeClient() + + _ = try await client.fetchEdits(since: 42) + + let url = OuraURLProtocolStub.requestedURLs.last?.absoluteString ?? "" + XCTAssertTrue(url.contains("/edits")) + XCTAssertTrue(url.contains("since=42")) + } + + func testFetchEdits401ThrowsBadResponse() async throws { + OuraURLProtocolStub.queue = [.init(status: 401, body: "nope".data(using: .utf8)!)] + let client = makeClient() + + do { + _ = try await client.fetchEdits(since: 0) + XCTFail("expected badResponse to be thrown") + } catch { + XCTAssertEqual(error as? CloudSyncError, CloudSyncError.badResponse(401, "nope")) + } + } + + func testFetchEditsMalformedJSONThrowsDecode() async throws { + OuraURLProtocolStub.queue = [.init(status: 200, body: "not json".data(using: .utf8)!)] + let client = makeClient() + + do { + _ = try await client.fetchEdits(since: 0) + XCTFail("expected decode to be thrown") + } catch { + XCTAssertEqual(error as? CloudSyncError, CloudSyncError.decode) + } + } + + // MARK: - ack + + func testAckReturnsAckedCount() async throws { + OuraURLProtocolStub.queue = [.init(status: 200, body: #"{"acked":2}"#.data(using: .utf8)!)] + let client = makeClient() + + let acked = try await client.ack(seqs: [1, 2]) + + XCTAssertEqual(acked, 2) + let url = OuraURLProtocolStub.requestedURLs.last?.absoluteString ?? "" + XCTAssertTrue(url.contains("/edits/ack")) + } + + func testAck401ThrowsBadResponse() async throws { + OuraURLProtocolStub.queue = [.init(status: 401, body: "denied".data(using: .utf8)!)] + let client = makeClient() + + do { + _ = try await client.ack(seqs: [1]) + XCTFail("expected badResponse to be thrown") + } catch { + XCTAssertEqual(error as? CloudSyncError, CloudSyncError.badResponse(401, "denied")) + } + } + + func testAckMalformedJSONThrowsDecode() async throws { + OuraURLProtocolStub.queue = [.init(status: 200, body: "{".data(using: .utf8)!)] + let client = makeClient() + + do { + _ = try await client.ack(seqs: [1]) + XCTFail("expected decode to be thrown") + } catch { + XCTAssertEqual(error as? CloudSyncError, CloudSyncError.decode) + } + } +} + +final class CloudSyncSettingsTests: XCTestCase { + override func setUp() { super.setUp(); CloudSyncSettings.clear() } + override func tearDown() { CloudSyncSettings.clear(); super.tearDown() } + + func testSaveLoadClearRoundTrip() { + XCTAssertFalse(CloudSyncSettings.isConfigured) + XCTAssertNil(CloudSyncSettings.serverURL) + XCTAssertNil(CloudSyncSettings.token) + + CloudSyncSettings.serverURL = "https://vk-noop-cloud.fly.dev" + CloudSyncSettings.token = "rw-secret" + XCTAssertEqual(CloudSyncSettings.serverURL, "https://vk-noop-cloud.fly.dev") + XCTAssertEqual(CloudSyncSettings.token, "rw-secret") + XCTAssertTrue(CloudSyncSettings.isConfigured) + + CloudSyncSettings.clear() + XCTAssertNil(CloudSyncSettings.serverURL) + XCTAssertNil(CloudSyncSettings.token) + XCTAssertFalse(CloudSyncSettings.isConfigured) + } + + func testIsConfiguredRequiresBothValues() { + CloudSyncSettings.serverURL = "https://vk-noop-cloud.fly.dev" + XCTAssertFalse(CloudSyncSettings.isConfigured) + + CloudSyncSettings.token = "rw-secret" + XCTAssertTrue(CloudSyncSettings.isConfigured) + } + + func testEmptyStringClearsValue() { + CloudSyncSettings.serverURL = "https://vk-noop-cloud.fly.dev" + XCTAssertNotNil(CloudSyncSettings.serverURL) + + CloudSyncSettings.serverURL = " " + XCTAssertNil(CloudSyncSettings.serverURL) + } +} +#endif // CLOUD_SYNC From 0629a5fd4475b1062eb0344459e19c4440fb92eb Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 16:36:51 -0700 Subject: [PATCH 052/171] =?UTF-8?q?feat(cloudsync):=20CloudEditApplier=20?= =?UTF-8?q?=E2=80=94=20recompute-safe=20journal=20application?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applies confirmed noop-cloud edit-journal rows to WhoopStore through the recompute-safe applySleepEdit/updateSleepStages entry points (userEdited=1 survives a later strap re-sync/recompute), pairs physical deletes with cloudTombstone rows to block resurrection, and lands corrections/adds with no natural device identity under the synthetic "noop-cloud" source. Maps the server's sleep-stage vocabulary (awake/light/deep/rem) to the on-device stagesJSON legend (wake/light/deep/rem) — only "awake" differs. Covers all 8 journal kinds, the undo/undone/set_baseline_note/unknown-kind skip rules, malformed-payload resilience, and seq-ordered processing. --- Strand/CloudSync/CloudEditApplier.swift | 372 ++++++++++++++++++++++++ StrandTests/CloudEditApplierTests.swift | 302 +++++++++++++++++++ 2 files changed, 674 insertions(+) create mode 100644 Strand/CloudSync/CloudEditApplier.swift create mode 100644 StrandTests/CloudEditApplierTests.swift diff --git a/Strand/CloudSync/CloudEditApplier.swift b/Strand/CloudSync/CloudEditApplier.swift new file mode 100644 index 0000000000..f4baaa675c --- /dev/null +++ b/Strand/CloudSync/CloudEditApplier.swift @@ -0,0 +1,372 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +#if CLOUD_SYNC +import Foundation +import WhoopStore + +/// Outcome of applying one batch of `CloudEdit` journal rows to the local store. +/// +/// `appliedSeqs` collects EVERY processed seq — applied, skipped, AND needsAttention alike, never +/// just the successful ones: the sync cursor must advance past whatever this device has SEEN, not +/// just what it acted on, or an edit this device can never act on (an unknown forward-compat kind, a +/// permanently malformed payload) would wedge the cursor forever. `needsAttention` is a SUBSET of +/// processed edits that made no store change and need a human look (see `CloudEditApplier`'s per-kind +/// rules below); it is reported separately so the sync status can say so, but its seq is still acked. +struct CloudApplySummary: Equatable { + var applied = 0 + var skipped = 0 + var needsAttention = 0 + var touchedSleep = false + var touchedWorkouts = false + var touchedHr = false + var appliedSeqs: [Int] = [] +} + +/// Applies confirmed noop-cloud edit-journal rows to the on-device `WhoopStore` through the SAME +/// recompute-safe entry points a human on-device correction uses — `applySleepEdit`/ +/// `updateSleepStages` (which set `userEdited = 1`), never a raw UPDATE — so a later strap re-sync or +/// nightly recompute can't silently revert a cloud correction (`upsertSleepSessions`'s ON CONFLICT +/// preserves an edited row's bounds/stages exactly because that flag is set). Deletions pair a +/// physical DELETE with a `cloudTombstone` row (`CloudTombstoneStore`) so a later re-import/backfill +/// can't resurrect them either. Corrections/additions with no natural "my-whoop" identity (a +/// manually-added workout, a fixed workout's replacement copy) land under the synthetic +/// `"noop-cloud"` source, registered in the device registry once per batch, the first time it's +/// actually needed (mirrors `OuraSyncWriter.persist`'s step 0). +/// +/// Never throws: a malformed payload, an unknown (forward-compat) kind, or a store-level failure are +/// all folded into the summary (`skipped`/`needsAttention`) rather than propagated — one bad journal +/// row must never abort the whole batch. +enum CloudEditApplier { + /// The synthetic paired-device id every "no natural home" write (an add, a fix's replacement + /// copy) lands under — distinct from any real strap/import source id. + static let cloudDeviceId = "noop-cloud" + + static func apply(_ edits: [CloudEdit], store: WhoopStore) async -> CloudApplySummary { + var summary = CloudApplySummary() + var registeredCloudDevice = false + + for edit in edits.sorted(by: { $0.seq < $1.seq }) { + // Every processed row is acked, whatever happened to it — see the type doc above. + summary.appliedSeqs.append(edit.seq) + + // An undo marker, an edit some LATER row already undid, or the server-only baseline note + // all need no on-device action. + if edit.kind == "undo" || edit.undoneBySeq != nil || edit.kind == "set_baseline_note" { + summary.skipped += 1 + continue + } + + let payload = Data(edit.payloadJSON.utf8) + switch edit.kind { + case "adjust_sleep_bounds": + await applyAdjustSleepBounds(payload, store: store, summary: &summary) + case "edit_sleep_stages": + await applyEditSleepStages(payload, store: store, summary: &summary) + case "delete_workout": + await applyDeleteWorkout(payload, seq: edit.seq, store: store, summary: &summary) + case "fix_workout": + await applyFixWorkout(payload, beforeJSON: edit.beforeJSON, seq: edit.seq, store: store, + summary: &summary, registeredCloudDevice: ®isteredCloudDevice) + case "add_workout": + await applyAddWorkout(payload, store: store, summary: &summary, + registeredCloudDevice: ®isteredCloudDevice) + case "delete_hr_range": + await applyDeleteHrRange(payload, seq: edit.seq, store: store, summary: &summary) + case "delete_metric_point": + await applyDeleteMetricPoint(payload, seq: edit.seq, store: store, summary: &summary) + default: + summary.skipped += 1 // unknown kind: forward-compat, not an error + } + } + return summary + } + + // MARK: - adjust_sleep_bounds + + private struct AdjustSleepBoundsPayload: Decodable { + let deviceId: String + let startTs: Int + let newStartTs: Int? + let newEndTs: Int? + } + + private static func applyAdjustSleepBounds(_ data: Data, store: WhoopStore, + summary: inout CloudApplySummary) async { + guard let p = try? JSONDecoder().decode(AdjustSleepBoundsPayload.self, from: data) else { + summary.skipped += 1 + return + } + do { + guard let current = try await store.sleepSessions(deviceId: p.deviceId, from: p.startTs, + to: p.startTs, limit: 1).first else { + summary.needsAttention += 1 // session absent — nothing to adjust + return + } + _ = try await store.applySleepEdit( + deviceId: p.deviceId, detectedStartTs: p.startTs, + newStartTs: p.newStartTs ?? current.effectiveStartTs, + newEndTs: p.newEndTs ?? current.endTs) + summary.applied += 1 + summary.touchedSleep = true + } catch { + summary.needsAttention += 1 + } + } + + // MARK: - edit_sleep_stages + + private struct SleepStagePayload: Decodable { + let start: Int + let end: Int + let stage: String + } + + private struct EditSleepStagesPayload: Decodable { + let deviceId: String + let startTs: Int + let stages: [SleepStagePayload] + } + + /// The noop-cloud server's stage vocabulary is "awake"/"light"/"deep"/"rem" (its `sleepStage` zod + /// enum, `noop-cloud/src/edits/kinds.ts`). The on-device stagesJSON legend uses "wake" for the + /// same concept — see `WearableSleepStageInterval`'s doc comment ("Normalized stage name written + /// into the stage JSON: 'deep' / 'light' / 'rem' / 'wake'") and `SleepStagerV2`'s own + /// `labels[i] == "awake" ? "wake" : labels[i]` conversion when it writes a detected session's + /// stages. "light"/"deep"/"rem" already match verbatim; only "awake" needs remapping so a + /// re-encoded edit reads identically to every other stage source (e.g. + /// `WearableImporter.efficiency`'s `stage != "wake"` check, `WhoopCsvExporter`'s + /// `("wake" == awake)` doc note). + private static func mapServerStage(_ stage: String) -> String { + stage == "awake" ? "wake" : stage + } + + /// Re-encode to the on-device `[{start,end,stage}]` shape (same JSON layout `OuraSyncWriter` + /// writes), mapping the stage vocabulary. nil for an empty list — a stage edit must carry at + /// least one segment; an empty array is treated as a malformed payload upstream. + private static func encodeStagesJSON(_ stages: [SleepStagePayload]) -> String? { + guard !stages.isEmpty else { return nil } + let segs = stages.map { + ["start": $0.start, "end": $0.end, "stage": mapServerStage($0.stage)] as [String: Any] + } + return (try? JSONSerialization.data(withJSONObject: segs)).flatMap { String(data: $0, encoding: .utf8) } + } + + private static func applyEditSleepStages(_ data: Data, store: WhoopStore, + summary: inout CloudApplySummary) async { + guard let p = try? JSONDecoder().decode(EditSleepStagesPayload.self, from: data), + let stagesJSON = encodeStagesJSON(p.stages) else { + summary.skipped += 1 + return + } + do { + guard let current = try await store.sleepSessions(deviceId: p.deviceId, from: p.startTs, + to: p.startTs, limit: 1).first else { + summary.needsAttention += 1 // session absent — nothing to restage + return + } + // updateSleepStages only touches userEdited=1 rows; promote first (CURRENT bounds, + // stages left untouched by this call) so an un-edited night can still receive a stage + // correction — see the brief's explicit two-step note. + if !current.userEdited { + _ = try await store.applySleepEdit(deviceId: p.deviceId, detectedStartTs: p.startTs, + newStartTs: current.effectiveStartTs, + newEndTs: current.endTs) + } + _ = try await store.updateSleepStages(deviceId: p.deviceId, detectedStartTs: p.startTs, + stagesJSON: stagesJSON) + summary.applied += 1 + summary.touchedSleep = true + } catch { + summary.needsAttention += 1 + } + } + + // MARK: - delete_workout + + private struct WorkoutKeyPayload: Decodable { + let deviceId: String + let startTs: Int + let sport: String + } + + private static func applyDeleteWorkout(_ data: Data, seq: Int, store: WhoopStore, + summary: inout CloudApplySummary) async { + guard let p = try? JSONDecoder().decode(WorkoutKeyPayload.self, from: data) else { + summary.skipped += 1 + return + } + do { + _ = try await store.deleteWorkouts(deviceId: p.deviceId, sport: p.sport, from: p.startTs, to: p.startTs) + try await store.addTombstone(kind: "workout", deviceId: p.deviceId, startTs: p.startTs, endTs: nil, + sport: p.sport, day: nil, key: nil, editSeq: seq) + summary.applied += 1 + summary.touchedWorkouts = true + } catch { + summary.needsAttention += 1 + } + } + + // MARK: - fix_workout + + private struct FixWorkoutPatch: Decodable { + let sport: String? + let startTs: Int? + let endTs: Int? + let energyKcal: Double? + let distanceM: Double? + } + + private struct FixWorkoutPayload: Decodable { + let deviceId: String + let startTs: Int + let sport: String + let patch: FixWorkoutPatch + } + + /// Lenient snapshot of the server mirror's workout row (the `beforeJSON` the journal carries for + /// `fix_workout`). Every field optional: the mirror's exact row shape isn't this applier's + /// contract to enforce, and only `endTs` (when `patch` doesn't supply its own) is actually + /// load-bearing — everything else has a sane fallback or is itself optional on `WorkoutRow`. + private struct FixWorkoutBefore: Decodable { + let source: String? + let endTs: Int? + let energyKcal: Double? + let distanceM: Double? + } + + private static func applyFixWorkout(_ data: Data, beforeJSON: String?, seq: Int, store: WhoopStore, + summary: inout CloudApplySummary, + registeredCloudDevice: inout Bool) async { + guard let p = try? JSONDecoder().decode(FixWorkoutPayload.self, from: data) else { + summary.skipped += 1 + return + } + guard let beforeJSON, + let before = try? JSONDecoder().decode(FixWorkoutBefore.self, from: Data(beforeJSON.utf8)) else { + summary.needsAttention += 1 // no server snapshot to reconstruct the corrected copy from + return + } + let finalStartTs = p.patch.startTs ?? p.startTs + guard let finalEndTs = p.patch.endTs ?? before.endTs else { + summary.needsAttention += 1 // neither the patch nor the snapshot carries an end time + return + } + do { + _ = try await store.deleteWorkouts(deviceId: p.deviceId, sport: p.sport, from: p.startTs, to: p.startTs) + try await store.addTombstone(kind: "workout", deviceId: p.deviceId, startTs: p.startTs, endTs: nil, + sport: p.sport, day: nil, key: nil, editSeq: seq) + + try await registerCloudDeviceIfNeeded(store: store, registeredCloudDevice: ®isteredCloudDevice) + let corrected = WorkoutRow( + startTs: finalStartTs, endTs: finalEndTs, sport: p.patch.sport ?? p.sport, + source: before.source ?? "manual", durationS: Double(finalEndTs - finalStartTs), + energyKcal: p.patch.energyKcal ?? before.energyKcal, avgHr: nil, maxHr: nil, strain: nil, + distanceM: p.patch.distanceM ?? before.distanceM, zonesJSON: nil, notes: nil) + _ = try await store.upsertWorkouts([corrected], deviceId: cloudDeviceId) + + summary.applied += 1 + summary.touchedWorkouts = true + } catch { + summary.needsAttention += 1 + } + } + + // MARK: - add_workout + + private struct AddWorkoutPayload: Decodable { + let startTs: Int + let endTs: Int + let sport: String + let energyKcal: Double? + let distanceM: Double? + let notes: String? + } + + private static func applyAddWorkout(_ data: Data, store: WhoopStore, summary: inout CloudApplySummary, + registeredCloudDevice: inout Bool) async { + guard let p = try? JSONDecoder().decode(AddWorkoutPayload.self, from: data) else { + summary.skipped += 1 + return + } + do { + try await registerCloudDeviceIfNeeded(store: store, registeredCloudDevice: ®isteredCloudDevice) + let row = WorkoutRow(startTs: p.startTs, endTs: p.endTs, sport: p.sport, source: "manual", + durationS: Double(p.endTs - p.startTs), energyKcal: p.energyKcal, + avgHr: nil, maxHr: nil, strain: nil, distanceM: p.distanceM, + zonesJSON: nil, notes: p.notes) + _ = try await store.upsertWorkouts([row], deviceId: cloudDeviceId) + summary.applied += 1 + summary.touchedWorkouts = true + } catch { + summary.needsAttention += 1 + } + } + + // MARK: - delete_hr_range + + private struct DeleteHrRangePayload: Decodable { + let deviceId: String + let fromTs: Int + let toTs: Int + } + + private static func applyDeleteHrRange(_ data: Data, seq: Int, store: WhoopStore, + summary: inout CloudApplySummary) async { + guard let p = try? JSONDecoder().decode(DeleteHrRangePayload.self, from: data) else { + summary.skipped += 1 + return + } + do { + _ = try await store.deleteHrRange(deviceId: p.deviceId, fromTs: p.fromTs, toTs: p.toTs) + try await store.addTombstone(kind: "hrRange", deviceId: p.deviceId, startTs: p.fromTs, endTs: p.toTs, + sport: nil, day: nil, key: nil, editSeq: seq) + summary.applied += 1 + summary.touchedHr = true + } catch { + summary.needsAttention += 1 + } + } + + // MARK: - delete_metric_point + + private struct DeleteMetricPointPayload: Decodable { + let deviceId: String + let day: String + let key: String + } + + private static func applyDeleteMetricPoint(_ data: Data, seq: Int, store: WhoopStore, + summary: inout CloudApplySummary) async { + guard let p = try? JSONDecoder().decode(DeleteMetricPointPayload.self, from: data) else { + summary.skipped += 1 + return + } + do { + _ = try await store.deleteMetricPoint(deviceId: p.deviceId, day: p.day, key: p.key) + try await store.addTombstone(kind: "metricPoint", deviceId: p.deviceId, startTs: nil, endTs: nil, + sport: nil, day: p.day, key: p.key, editSeq: seq) + summary.applied += 1 + // No sleep/workouts/hr flag applies to a generic metric point. + } catch { + summary.needsAttention += 1 + } + } + + // MARK: - noop-cloud device registration + + /// Registers the synthetic `"noop-cloud"` paired-device row the first time this batch actually + /// writes under it — mirrors `OuraSyncWriter.persist`'s step 0. `DeviceRegistryStore.add` is an + /// upsert (`ON CONFLICT(id) DO UPDATE`), so re-registering across separate batches/runs is + /// harmless; `registeredCloudDevice` only avoids repeat writes WITHIN one `apply` call. + private static func registerCloudDeviceIfNeeded(store: WhoopStore, registeredCloudDevice: inout Bool) async throws { + guard !registeredCloudDevice else { return } + let now = Int(Date().timeIntervalSince1970) + let device = PairedDevice(id: cloudDeviceId, brand: cloudDeviceId, model: "Cloud edits", + sourceKind: .cloudImport, capabilities: [], status: .paired, + addedAt: now, lastSeenAt: now) + try DeviceRegistryStore(dbQueue: store.registryWriter).add(device) + registeredCloudDevice = true + } +} +#endif // CLOUD_SYNC diff --git a/StrandTests/CloudEditApplierTests.swift b/StrandTests/CloudEditApplierTests.swift new file mode 100644 index 0000000000..8ba24a97ad --- /dev/null +++ b/StrandTests/CloudEditApplierTests.swift @@ -0,0 +1,302 @@ +// Tests the CLOUD_SYNC-gated journal → store applier; compiled only when the flag is set +// (StrandTests shares the app's OuraConfig.xcconfig, so flag + creds arrive together). +#if CLOUD_SYNC +import XCTest +import WhoopStore +import WhoopProtocol +@testable import Strand + +final class CloudEditApplierTests: XCTestCase { + + // MARK: - Fixture helpers + + private func json(_ obj: [String: Any]) -> String { + String(data: try! JSONSerialization.data(withJSONObject: obj), encoding: .utf8)! + } + + private func edit(seq: Int, kind: String, payload: [String: Any], beforeJSON: String? = nil, + undoneBySeq: Int? = nil) -> CloudEdit { + CloudEdit(seq: seq, editId: "e\(seq)", kind: kind, payloadJSON: json(payload), beforeJSON: beforeJSON, + rationale: nil, appliedAt: 1_752_300_000, undoneBySeq: undoneBySeq, ackedAt: nil) + } + + // MARK: - adjust_sleep_bounds + + func testAdjustSleepBoundsSurvivesRecompute() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertSleepSessions([ + CachedSleepSession(startTs: 1000, endTs: 29000, efficiency: 90, restingHr: 50, avgHrv: 60, stagesJSON: nil), + ], deviceId: "my-whoop") + + let e = edit(seq: 1, kind: "adjust_sleep_bounds", + payload: ["deviceId": "my-whoop", "startTs": 1000, "newEndTs": 30000]) + let summary = await CloudEditApplier.apply([e], store: store) + + XCTAssertEqual(summary.applied, 1) + XCTAssertEqual(summary.skipped, 0) + XCTAssertEqual(summary.needsAttention, 0) + XCTAssertTrue(summary.touchedSleep) + XCTAssertEqual(summary.appliedSeqs, [1]) + + let edited = try await store.sleepSessions(deviceId: "my-whoop", from: 1000, to: 1000, limit: 1).first + XCTAssertEqual(edited?.endTs, 30000) + XCTAssertTrue(edited?.userEdited ?? false) + + // Recompute simulation: a later strap/import upsert tries to overwrite with DETECTED (un-edited) + // values — the userEdited flag set by applySleepEdit must survive it. + _ = try await store.upsertSleepSessions([ + CachedSleepSession(startTs: 1000, endTs: 29000, efficiency: 91, restingHr: 51, avgHrv: 61, stagesJSON: nil), + ], deviceId: "my-whoop") + let survived = try await store.sleepSessions(deviceId: "my-whoop", from: 1000, to: 1000, limit: 1).first + XCTAssertEqual(survived?.endTs, 30000, "userEdited protection must survive a recompute upsert") + XCTAssertTrue(survived?.userEdited ?? false) + } + + func testAdjustSleepBoundsNeedsAttentionWhenSessionAbsent() async throws { + let store = try await WhoopStore.inMemory() + let e = edit(seq: 5, kind: "adjust_sleep_bounds", + payload: ["deviceId": "my-whoop", "startTs": 9999, "newEndTs": 30000]) + let summary = await CloudEditApplier.apply([e], store: store) + XCTAssertEqual(summary.applied, 0) + XCTAssertEqual(summary.needsAttention, 1) + XCTAssertEqual(summary.appliedSeqs, [5]) + } + + // MARK: - edit_sleep_stages + + func testEditSleepStagesPromotesUnEditedRowAndMapsStageVocabulary() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertSleepSessions([ + CachedSleepSession(startTs: 2000, endTs: 5000, efficiency: 90, restingHr: 50, avgHrv: 60, stagesJSON: nil), + ], deviceId: "my-whoop") + let before = try await store.sleepSessions(deviceId: "my-whoop", from: 2000, to: 2000, limit: 1).first + XCTAssertFalse(before?.userEdited ?? true, "fixture must start un-edited to exercise the promotion path") + + // Server vocabulary is "awake"/"light"/"deep"/"rem" — "awake" must map to the app's "wake" token. + let stages: [[String: Any]] = [ + ["start": 2000, "end": 3000, "stage": "awake"], + ["start": 3000, "end": 5000, "stage": "deep"], + ] + let e = edit(seq: 9, kind: "edit_sleep_stages", + payload: ["deviceId": "my-whoop", "startTs": 2000, "stages": stages]) + let summary = await CloudEditApplier.apply([e], store: store) + + XCTAssertEqual(summary.applied, 1) + XCTAssertEqual(summary.needsAttention, 0) + XCTAssertTrue(summary.touchedSleep) + + let after = try await store.sleepSessions(deviceId: "my-whoop", from: 2000, to: 2000, limit: 1).first + XCTAssertTrue(after?.userEdited ?? false, "promotion must set userEdited") + XCTAssertEqual(after?.endTs, 5000, "promotion must keep the CURRENT bounds, not reset them") + + let stagesJSON = try XCTUnwrap(after?.stagesJSON) + let decodedAny = try JSONSerialization.jsonObject(with: Data(stagesJSON.utf8)) + let decoded = try XCTUnwrap(decodedAny as? [[String: Any]]) + XCTAssertEqual(decoded.first?["stage"] as? String, "wake", "server 'awake' must map to the app's 'wake' token") + XCTAssertEqual(decoded.last?["stage"] as? String, "deep", "non-awake stages pass through verbatim") + } + + func testEditSleepStagesNeedsAttentionWhenSessionAbsent() async throws { + let store = try await WhoopStore.inMemory() + let stages: [[String: Any]] = [["start": 2000, "end": 3000, "stage": "light"]] + let e = edit(seq: 3, kind: "edit_sleep_stages", + payload: ["deviceId": "my-whoop", "startTs": 2000, "stages": stages]) + let summary = await CloudEditApplier.apply([e], store: store) + XCTAssertEqual(summary.needsAttention, 1) + XCTAssertEqual(summary.applied, 0) + } + + // MARK: - delete_workout + + func testDeleteWorkoutTombstonesAndBlocksResurrection() async throws { + let store = try await WhoopStore.inMemory() + let workout = WorkoutRow(startTs: 100, endTs: 700, sport: "running", source: "whoop", durationS: 600, + energyKcal: 400, avgHr: nil, maxHr: nil, strain: nil, distanceM: nil, + zonesJSON: nil, notes: nil) + _ = try await store.upsertWorkouts([workout], deviceId: "my-whoop") + + let e = edit(seq: 4, kind: "delete_workout", payload: ["deviceId": "my-whoop", "startTs": 100, "sport": "running"]) + let summary = await CloudEditApplier.apply([e], store: store) + + XCTAssertEqual(summary.applied, 1) + XCTAssertTrue(summary.touchedWorkouts) + let remaining = try await store.workouts(deviceId: "my-whoop", from: 0, to: 1000, limit: 10) + XCTAssertEqual(remaining.count, 0) + + // Resurrection guard: a later re-import of the exact same workout must be dropped. + let resurrected = try await store.upsertWorkouts([workout], deviceId: "my-whoop") + XCTAssertEqual(resurrected, 0) + } + + // MARK: - fix_workout + + func testFixWorkoutTombstonesOriginalAndUpsertsCloudCopy() async throws { + let store = try await WhoopStore.inMemory() + let original = WorkoutRow(startTs: 100, endTs: 700, sport: "running", source: "whoop", durationS: 600, + energyKcal: 400, avgHr: 140, maxHr: 170, strain: 8.5, distanceM: 5000, + zonesJSON: nil, notes: nil) + _ = try await store.upsertWorkouts([original], deviceId: "my-whoop") + + let beforeJSON = json(["deviceId": "my-whoop", "startTs": 100, "endTs": 700, "sport": "running", + "source": "whoop", "durationS": 600, "energyKcal": 400, "distanceM": 5000]) + let e = edit(seq: 6, kind: "fix_workout", + payload: ["deviceId": "my-whoop", "startTs": 100, "sport": "running", + "patch": ["energyKcal": 450]], + beforeJSON: beforeJSON) + let summary = await CloudEditApplier.apply([e], store: store) + + XCTAssertEqual(summary.applied, 1) + XCTAssertTrue(summary.touchedWorkouts) + + let originalGone = try await store.workouts(deviceId: "my-whoop", from: 0, to: 1000, limit: 10) + XCTAssertEqual(originalGone.count, 0) + let tombstones = try await store.workoutTombstones(deviceId: "my-whoop") + XCTAssertEqual(tombstones.count, 1) + + let copies = try await store.workouts(deviceId: "noop-cloud", from: 0, to: 1000, limit: 10) + XCTAssertEqual(copies.count, 1) + XCTAssertEqual(copies.first?.energyKcal, 450, "patch overlay must win over the snapshot") + XCTAssertEqual(copies.first?.distanceM, 5000, "un-patched fields carry over from the snapshot") + XCTAssertEqual(copies.first?.source, "whoop", "original detection source carries over") + + let devices = try DeviceRegistryStore(dbQueue: store.registryWriter).all() + let cloudDevice = devices.first { $0.id == "noop-cloud" } + XCTAssertEqual(cloudDevice?.brand, "noop-cloud") + XCTAssertEqual(cloudDevice?.model, "Cloud edits") + XCTAssertEqual(cloudDevice?.sourceKind, .cloudImport) + XCTAssertEqual(cloudDevice?.status, .paired) + } + + func testFixWorkoutNeedsAttentionWhenBeforeJSONMissing() async throws { + let store = try await WhoopStore.inMemory() + let e = edit(seq: 7, kind: "fix_workout", + payload: ["deviceId": "my-whoop", "startTs": 100, "sport": "running", + "patch": ["energyKcal": 450]]) + let summary = await CloudEditApplier.apply([e], store: store) + XCTAssertEqual(summary.needsAttention, 1) + XCTAssertEqual(summary.applied, 0) + } + + // MARK: - add_workout + + func testAddWorkoutUpsertsUnderCloudDevice() async throws { + let store = try await WhoopStore.inMemory() + let e = edit(seq: 8, kind: "add_workout", + payload: ["startTs": 1000, "endTs": 2800, "sport": "cycling", "energyKcal": 300, + "notes": "morning ride"]) + let summary = await CloudEditApplier.apply([e], store: store) + + XCTAssertEqual(summary.applied, 1) + XCTAssertTrue(summary.touchedWorkouts) + let added = try await store.workouts(deviceId: "noop-cloud", from: 0, to: 3000, limit: 10) + XCTAssertEqual(added.count, 1) + XCTAssertEqual(added.first?.durationS, 1800, "durationS must be derived as endTs - startTs") + XCTAssertEqual(added.first?.sport, "cycling") + XCTAssertEqual(added.first?.notes, "morning ride") + } + + // MARK: - delete_hr_range + + func testDeleteHrRangeTombstonesAndBlocksResurrection() async throws { + let store = try await WhoopStore.inMemory() + try await store.insert(Streams(hr: [HRSample(ts: 1500, bpm: 60), HRSample(ts: 2500, bpm: 61)]), deviceId: "my-whoop") + + let e = edit(seq: 10, kind: "delete_hr_range", payload: ["deviceId": "my-whoop", "fromTs": 1000, "toTs": 2000]) + let summary = await CloudEditApplier.apply([e], store: store) + + XCTAssertEqual(summary.applied, 1) + XCTAssertTrue(summary.touchedHr) + let survivors = try await store.hrSamples(deviceId: "my-whoop", from: 0, to: 10_000, limit: 10) + XCTAssertEqual(survivors.map(\.ts), [2500]) + + // Resurrection guard: re-streaming into the deleted range must be filtered. + try await store.insert(Streams(hr: [HRSample(ts: 1500, bpm: 60)]), deviceId: "my-whoop") + let stillGone = try await store.hrSamples(deviceId: "my-whoop", from: 0, to: 10_000, limit: 10) + XCTAssertEqual(stillGone.map(\.ts), [2500]) + } + + // MARK: - delete_metric_point + + func testDeleteMetricPointTombstonesAndBlocksResurrection() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertMetricSeries([MetricPoint(day: "2026-07-01", key: "vo2max", value: 44)], deviceId: "my-whoop") + + let e = edit(seq: 11, kind: "delete_metric_point", + payload: ["deviceId": "my-whoop", "day": "2026-07-01", "key": "vo2max"]) + let summary = await CloudEditApplier.apply([e], store: store) + + XCTAssertEqual(summary.applied, 1) + XCTAssertFalse(summary.touchedSleep) + XCTAssertFalse(summary.touchedWorkouts) + XCTAssertFalse(summary.touchedHr) + + let remaining = try await store.metricSeries(deviceId: "my-whoop", key: "vo2max", from: "2026-01-01", to: "2026-12-31") + XCTAssertEqual(remaining.count, 0) + + let resurrected = try await store.upsertMetricSeries([MetricPoint(day: "2026-07-01", key: "vo2max", value: 45)], deviceId: "my-whoop") + XCTAssertEqual(resurrected, 0) + } + + // MARK: - Skip rules + + func testMalformedPayloadSkipsWithoutThrowing() async throws { + let store = try await WhoopStore.inMemory() + let e = CloudEdit(seq: 12, editId: "bad", kind: "adjust_sleep_bounds", payloadJSON: "not json at all", + beforeJSON: nil, rationale: nil, appliedAt: 1_752_300_000, undoneBySeq: nil, ackedAt: nil) + let summary = await CloudEditApplier.apply([e], store: store) + XCTAssertEqual(summary.skipped, 1) + XCTAssertEqual(summary.applied, 0) + XCTAssertEqual(summary.needsAttention, 0) + XCTAssertEqual(summary.appliedSeqs, [12], "a malformed row is still acked so the cursor can advance") + } + + func testUndoneRowSkipped() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertSleepSessions([ + CachedSleepSession(startTs: 1000, endTs: 29000, efficiency: 90, restingHr: 50, avgHrv: 60, stagesJSON: nil), + ], deviceId: "my-whoop") + let e = edit(seq: 13, kind: "adjust_sleep_bounds", + payload: ["deviceId": "my-whoop", "startTs": 1000, "newEndTs": 99999], undoneBySeq: 14) + let summary = await CloudEditApplier.apply([e], store: store) + + XCTAssertEqual(summary.skipped, 1) + XCTAssertEqual(summary.applied, 0) + let untouched = try await store.sleepSessions(deviceId: "my-whoop", from: 1000, to: 1000, limit: 1).first + XCTAssertEqual(untouched?.endTs, 29000, "an edit undone by a later row must never be applied") + } + + func testUndoAndSetBaselineNoteAndUnknownKindAllSkip() async throws { + let store = try await WhoopStore.inMemory() + let edits = [ + CloudEdit(seq: 20, editId: "u1", kind: "undo", payloadJSON: #"{"targetSeq":1}"#, beforeJSON: nil, + rationale: nil, appliedAt: 1, undoneBySeq: nil, ackedAt: nil), + edit(seq: 21, kind: "set_baseline_note", payload: ["note": "felt great"]), + edit(seq: 22, kind: "some_future_kind", payload: [:]), + ] + let summary = await CloudEditApplier.apply(edits, store: store) + XCTAssertEqual(summary.skipped, 3) + XCTAssertEqual(summary.applied, 0) + XCTAssertEqual(summary.appliedSeqs, [20, 21, 22]) + } + + // MARK: - Ordering / summary bookkeeping + + func testProcessesInSeqOrderRegardlessOfArrayOrder() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertSleepSessions([ + CachedSleepSession(startTs: 1000, endTs: 29000, efficiency: 90, restingHr: 50, avgHrv: 60, stagesJSON: nil), + ], deviceId: "my-whoop") + // Fed out of array order (seq 2 before seq 1) — the HIGHER seq must win, proving the internal sort. + let edits = [ + edit(seq: 2, kind: "adjust_sleep_bounds", payload: ["deviceId": "my-whoop", "startTs": 1000, "newEndTs": 40000]), + edit(seq: 1, kind: "adjust_sleep_bounds", payload: ["deviceId": "my-whoop", "startTs": 1000, "newEndTs": 30000]), + ] + let summary = await CloudEditApplier.apply(edits, store: store) + XCTAssertEqual(summary.applied, 2) + XCTAssertEqual(summary.appliedSeqs, [1, 2], "appliedSeqs must reflect seq order, not array order") + + let finalState = try await store.sleepSessions(deviceId: "my-whoop", from: 1000, to: 1000, limit: 1).first + XCTAssertEqual(finalState?.endTs, 40000, "the higher-seq edit, applied last, must win") + } +} +#endif // CLOUD_SYNC From 9a20e78449c66a4eac69e73fbdf0ffd268f5e02b Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 17:06:50 -0700 Subject: [PATCH 053/171] fix(cloudsync): SleepEditGuard on cloud bounds + destructive-last ordering + fix-after-delete guard Three correctness fixes in CloudEditApplier, the noop-cloud edit-journal -> WhoopStore applier: - adjust_sleep_bounds and the edit_sleep_stages promotion path now route their bounds through SleepEditGuard.clampedEditWindow (the same #940 belt-and-braces Repository.editSleepTimes uses) before calling applySleepEdit, so an inverted or future-dated window from the server is refused (needsAttention) instead of persisted. - Every destructive multi-step apply (delete_workout, delete_hr_range, delete_metric_point, fix_workout) now writes its cloudTombstone BEFORE the physical delete, so a mid-chain throw degrades to "an extra row is visible" rather than losing the tombstone and letting a later re-import/backfill resurrect deleted data. - fix_workout checks the original still exists before rebuilding it under the noop-cloud device: a same-batch delete_workout for the same key now counts as skipped (delete already won) instead of resurrecting a corrected copy; a genuinely stale server mirror with no tombstone still surfaces as needsAttention. Adds WhoopStore.workoutExists(deviceId:startTs:sport:) for the existence check. --- .../WhoopStore/JournalWorkoutAppleCache.swift | 11 ++ .../JournalWorkoutAppleCacheTests.swift | 23 ++++ Strand/CloudSync/CloudEditApplier.swift | 66 ++++++++-- StrandTests/CloudEditApplierTests.swift | 113 ++++++++++++++++++ 4 files changed, 204 insertions(+), 9 deletions(-) diff --git a/Packages/WhoopStore/Sources/WhoopStore/JournalWorkoutAppleCache.swift b/Packages/WhoopStore/Sources/WhoopStore/JournalWorkoutAppleCache.swift index 183cf9ff88..ffae03a3e4 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/JournalWorkoutAppleCache.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/JournalWorkoutAppleCache.swift @@ -256,6 +256,17 @@ extension WhoopStore { } } + /// True when a workout exists at the exact natural key (deviceId, startTs, sport). A cheap + /// existence check for a resurrection/staleness guard (`CloudEditApplier`'s `fix_workout` + /// handler) that only needs a boolean, not a fully decoded `WorkoutRow`. + public func workoutExists(deviceId: String, startTs: Int, sport: String) async throws -> Bool { + try syncRead { db in + try Row.fetchOne(db, sql: """ + SELECT 1 FROM workout WHERE deviceId = ? AND startTs = ? AND sport = ? LIMIT 1 + """, arguments: [deviceId, startTs, sport]) != nil + } + } + /// Apple-Health daily aggregates for days in [from, to] (lexicographic compare), oldest first. public func appleDaily(deviceId: String, from: String, to: String) async throws -> [AppleDaily] { try syncRead { db in diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/JournalWorkoutAppleCacheTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/JournalWorkoutAppleCacheTests.swift index 98e390d403..3fc7251483 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/JournalWorkoutAppleCacheTests.swift +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/JournalWorkoutAppleCacheTests.swift @@ -259,6 +259,29 @@ final class JournalWorkoutAppleCacheTests: XCTestCase { XCTAssertEqual(other.count, 1, "other device untouched") } + func testWorkoutExists() async throws { + let store = try await WhoopStore.inMemory() + let w = WorkoutRow(startTs: 100, endTs: 700, sport: "running", source: "whoop", durationS: 600, + energyKcal: nil, avgHr: nil, maxHr: nil, strain: nil, distanceM: nil, + zonesJSON: nil, notes: nil) + try await store.upsertWorkouts([w], deviceId: "devA") + + let exists = try await store.workoutExists(deviceId: "devA", startTs: 100, sport: "running") + XCTAssertTrue(exists) + + // A mismatch on any single key column must miss. + let wrongSport = try await store.workoutExists(deviceId: "devA", startTs: 100, sport: "cycling") + XCTAssertFalse(wrongSport) + let wrongStart = try await store.workoutExists(deviceId: "devA", startTs: 200, sport: "running") + XCTAssertFalse(wrongStart) + let wrongDevice = try await store.workoutExists(deviceId: "devB", startTs: 100, sport: "running") + XCTAssertFalse(wrongDevice) + + _ = try await store.deleteWorkouts(deviceId: "devA", sport: "running", from: 100, to: 100) + let afterDelete = try await store.workoutExists(deviceId: "devA", startTs: 100, sport: "running") + XCTAssertFalse(afterDelete, "a deleted workout must no longer report as existing") + } + // MARK: - appleDaily func testAppleDailyUpsertReadAndIdempotency() async throws { diff --git a/Strand/CloudSync/CloudEditApplier.swift b/Strand/CloudSync/CloudEditApplier.swift index f4baaa675c..6895024a97 100644 --- a/Strand/CloudSync/CloudEditApplier.swift +++ b/Strand/CloudSync/CloudEditApplier.swift @@ -4,6 +4,7 @@ #if CLOUD_SYNC import Foundation import WhoopStore +import StrandAnalytics /// Outcome of applying one batch of `CloudEdit` journal rows to the local store. /// @@ -103,10 +104,19 @@ enum CloudEditApplier { summary.needsAttention += 1 // session absent — nothing to adjust return } + // #940 belt-and-braces, mirrored from Repository.editSleepTimes: never persist a future- + // ending or inverted window, whatever the server payload says. A malicious/buggy server + // mirror or a stale queued edit must not be able to write a phantom night the display + // merge can't render — refuse (needsAttention) rather than clamp-and-apply silently. + guard let window = SleepEditGuard.clampedEditWindow( + start: p.newStartTs ?? current.effectiveStartTs, end: p.newEndTs ?? current.endTs, + now: Int(Date().timeIntervalSince1970)) else { + summary.needsAttention += 1 + return + } _ = try await store.applySleepEdit( deviceId: p.deviceId, detectedStartTs: p.startTs, - newStartTs: p.newStartTs ?? current.effectiveStartTs, - newEndTs: p.newEndTs ?? current.endTs) + newStartTs: window.start, newEndTs: window.end) summary.applied += 1 summary.touchedSleep = true } catch { @@ -167,11 +177,19 @@ enum CloudEditApplier { } // updateSleepStages only touches userEdited=1 rows; promote first (CURRENT bounds, // stages left untouched by this call) so an un-edited night can still receive a stage - // correction — see the brief's explicit two-step note. + // correction — see the brief's explicit two-step note. This promotion also calls + // applySleepEdit with a bounds pair, so it gets the same #940 guard as adjust_sleep_bounds + // above: a stored row with corrupt (inverted/future) bounds — e.g. a bad import that never + // went through the guard — must not be silently promoted into a userEdited phantom night. if !current.userEdited { + guard let window = SleepEditGuard.clampedEditWindow( + start: current.effectiveStartTs, end: current.endTs, + now: Int(Date().timeIntervalSince1970)) else { + summary.needsAttention += 1 + return + } _ = try await store.applySleepEdit(deviceId: p.deviceId, detectedStartTs: p.startTs, - newStartTs: current.effectiveStartTs, - newEndTs: current.endTs) + newStartTs: window.start, newEndTs: window.end) } _ = try await store.updateSleepStages(deviceId: p.deviceId, detectedStartTs: p.startTs, stagesJSON: stagesJSON) @@ -197,9 +215,15 @@ enum CloudEditApplier { return } do { - _ = try await store.deleteWorkouts(deviceId: p.deviceId, sport: p.sport, from: p.startTs, to: p.startTs) + // Destructive step LAST: tombstone before the physical delete, so a mid-chain throw + // degrades to "the row is still visible but future re-imports/backfills are blocked" — + // never to "the delete happened but nothing recorded it, so a later re-import silently + // resurrects the workout the user just deleted." A crash between these two calls is the + // worst case this ordering must survive, and it does: re-processing the same journal row + // (addTombstone is idempotent by (kind, editSeq)) simply finishes the delete. try await store.addTombstone(kind: "workout", deviceId: p.deviceId, startTs: p.startTs, endTs: nil, sport: p.sport, day: nil, key: nil, editSeq: seq) + _ = try await store.deleteWorkouts(deviceId: p.deviceId, sport: p.sport, from: p.startTs, to: p.startTs) summary.applied += 1 summary.touchedWorkouts = true } catch { @@ -253,7 +277,27 @@ enum CloudEditApplier { return } do { - _ = try await store.deleteWorkouts(deviceId: p.deviceId, sport: p.sport, from: p.startTs, to: p.startTs) + // Same-batch race guard: a delete_workout for this exact key can sort BEFORE this row + // (lower seq) in the SAME batch and have already removed the original. Without this check + // the upsert below would resurrect it under noop-cloud, silently undoing a delete the user + // just made. + guard try await store.workoutExists(deviceId: p.deviceId, startTs: p.startTs, sport: p.sport) else { + let alreadyDeleted = try await store.workoutTombstones(deviceId: p.deviceId) + .contains { $0.startTs == p.startTs && $0.sport == p.sport } + if alreadyDeleted { + summary.skipped += 1 // an earlier/same-batch delete already won — nothing to fix + } else { + summary.needsAttention += 1 // gone with no tombstone — stale server mirror + } + return + } + + // Destructive step LAST: tombstone the original, write the corrected copy, THEN delete the + // original — so a mid-chain throw degrades to "an extra/duplicate row is visible", never to + // "the original is gone and no correction exists" (see the ordering rationale on + // applyDeleteWorkout above). The tombstone keys the ORIGINAL deviceId, so it can never + // block the noop-cloud upsert below (upsertWorkouts' resurrection guard only checks + // tombstones for the deviceId it is writing INTO). try await store.addTombstone(kind: "workout", deviceId: p.deviceId, startTs: p.startTs, endTs: nil, sport: p.sport, day: nil, key: nil, editSeq: seq) @@ -265,6 +309,8 @@ enum CloudEditApplier { distanceM: p.patch.distanceM ?? before.distanceM, zonesJSON: nil, notes: nil) _ = try await store.upsertWorkouts([corrected], deviceId: cloudDeviceId) + _ = try await store.deleteWorkouts(deviceId: p.deviceId, sport: p.sport, from: p.startTs, to: p.startTs) + summary.applied += 1 summary.touchedWorkouts = true } catch { @@ -318,9 +364,10 @@ enum CloudEditApplier { return } do { - _ = try await store.deleteHrRange(deviceId: p.deviceId, fromTs: p.fromTs, toTs: p.toTs) + // Destructive step LAST — see the ordering rationale on applyDeleteWorkout above. try await store.addTombstone(kind: "hrRange", deviceId: p.deviceId, startTs: p.fromTs, endTs: p.toTs, sport: nil, day: nil, key: nil, editSeq: seq) + _ = try await store.deleteHrRange(deviceId: p.deviceId, fromTs: p.fromTs, toTs: p.toTs) summary.applied += 1 summary.touchedHr = true } catch { @@ -343,9 +390,10 @@ enum CloudEditApplier { return } do { - _ = try await store.deleteMetricPoint(deviceId: p.deviceId, day: p.day, key: p.key) + // Destructive step LAST — see the ordering rationale on applyDeleteWorkout above. try await store.addTombstone(kind: "metricPoint", deviceId: p.deviceId, startTs: nil, endTs: nil, sport: nil, day: p.day, key: p.key, editSeq: seq) + _ = try await store.deleteMetricPoint(deviceId: p.deviceId, day: p.day, key: p.key) summary.applied += 1 // No sleep/workouts/hr flag applies to a generic metric point. } catch { diff --git a/StrandTests/CloudEditApplierTests.swift b/StrandTests/CloudEditApplierTests.swift index 8ba24a97ad..15469a57dc 100644 --- a/StrandTests/CloudEditApplierTests.swift +++ b/StrandTests/CloudEditApplierTests.swift @@ -62,6 +62,28 @@ final class CloudEditApplierTests: XCTestCase { XCTAssertEqual(summary.appliedSeqs, [5]) } + func testAdjustSleepBoundsRejectsInvertedWindow() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertSleepSessions([ + CachedSleepSession(startTs: 1000, endTs: 29000, efficiency: 90, restingHr: 50, avgHrv: 60, stagesJSON: nil), + ], deviceId: "my-whoop") + + // newEndTs (500) is BEFORE the session's own startTs (1000) — an inverted window that + // SleepEditGuard.clampedEditWindow (#940 belt-and-braces) must refuse outright, exactly as + // Repository.editSleepTimes refuses one from a misbehaving client UI. + let e = edit(seq: 17, kind: "adjust_sleep_bounds", + payload: ["deviceId": "my-whoop", "startTs": 1000, "newEndTs": 500]) + let summary = await CloudEditApplier.apply([e], store: store) + + XCTAssertEqual(summary.applied, 0) + XCTAssertEqual(summary.needsAttention, 1, "an inverted window must be refused, not silently clamped") + XCTAssertEqual(summary.appliedSeqs, [17]) + + let untouched = try await store.sleepSessions(deviceId: "my-whoop", from: 1000, to: 1000, limit: 1).first + XCTAssertEqual(untouched?.endTs, 29000, "a rejected edit must leave the stored bounds unchanged") + XCTAssertFalse(untouched?.userEdited ?? true, "a rejected edit must not set userEdited") + } + // MARK: - edit_sleep_stages func testEditSleepStagesPromotesUnEditedRowAndMapsStageVocabulary() async throws { @@ -106,6 +128,30 @@ final class CloudEditApplierTests: XCTestCase { XCTAssertEqual(summary.applied, 0) } + func testEditSleepStagesPromotionRejectsInvertedCurrentBounds() async throws { + let store = try await WhoopStore.inMemory() + // Seed a corrupted (inverted) session directly via upsertSleepSessions, which does NOT route + // through SleepEditGuard (only applySleepEdit does) — simulates a bad import/server mirror + // reaching the store with bounds no on-device edit path could ever produce. + _ = try await store.upsertSleepSessions([ + CachedSleepSession(startTs: 5000, endTs: 4000, efficiency: 90, restingHr: 50, avgHrv: 60, stagesJSON: nil), + ], deviceId: "my-whoop") + let before = try await store.sleepSessions(deviceId: "my-whoop", from: 5000, to: 5000, limit: 1).first + XCTAssertFalse(before?.userEdited ?? true, "fixture must start un-edited to exercise the promotion path") + + let stages: [[String: Any]] = [["start": 4000, "end": 5000, "stage": "light"]] + let e = edit(seq: 18, kind: "edit_sleep_stages", + payload: ["deviceId": "my-whoop", "startTs": 5000, "stages": stages]) + let summary = await CloudEditApplier.apply([e], store: store) + + XCTAssertEqual(summary.needsAttention, 1, "promoting an inverted stored window must be refused") + XCTAssertEqual(summary.applied, 0) + + let untouched = try await store.sleepSessions(deviceId: "my-whoop", from: 5000, to: 5000, limit: 1).first + XCTAssertEqual(untouched?.endTs, 4000, "rejected promotion must leave stored bounds unchanged") + XCTAssertFalse(untouched?.userEdited ?? true, "rejected promotion must not set userEdited") + } + // MARK: - delete_workout func testDeleteWorkoutTombstonesAndBlocksResurrection() async throws { @@ -177,6 +223,73 @@ final class CloudEditApplierTests: XCTestCase { XCTAssertEqual(summary.applied, 0) } + func testFixWorkoutNeedsAttentionWhenNoUsableEndTs() async throws { + // beforeJSON IS present (unlike testFixWorkoutNeedsAttentionWhenBeforeJSONMissing above) but + // carries no endTs, and the patch doesn't supply one either — the previously-untested branch + // of the `finalEndTs` guard. + let store = try await WhoopStore.inMemory() + let beforeJSON = json(["deviceId": "my-whoop", "startTs": 100, "sport": "running", "source": "whoop"]) + let e = edit(seq: 15, kind: "fix_workout", + payload: ["deviceId": "my-whoop", "startTs": 100, "sport": "running", + "patch": ["energyKcal": 450]], + beforeJSON: beforeJSON) + let summary = await CloudEditApplier.apply([e], store: store) + XCTAssertEqual(summary.needsAttention, 1) + XCTAssertEqual(summary.applied, 0) + XCTAssertEqual(summary.skipped, 0) + } + + func testFixWorkoutSkippedWhenSameBatchDeleteAlreadyWon() async throws { + // delete_workout(seq1) then fix_workout(seq2) targeting the SAME (deviceId, startTs, sport) in + // one batch: the delete must win outright — the fix must not resurrect a noop-cloud copy of a + // workout the same batch just deleted. + let store = try await WhoopStore.inMemory() + let workout = WorkoutRow(startTs: 100, endTs: 700, sport: "running", source: "whoop", durationS: 600, + energyKcal: 400, avgHr: nil, maxHr: nil, strain: nil, distanceM: nil, + zonesJSON: nil, notes: nil) + _ = try await store.upsertWorkouts([workout], deviceId: "my-whoop") + + let beforeJSON = json(["deviceId": "my-whoop", "startTs": 100, "endTs": 700, "sport": "running", + "source": "whoop", "durationS": 600, "energyKcal": 400, "distanceM": 5000]) + let edits = [ + edit(seq: 1, kind: "delete_workout", payload: ["deviceId": "my-whoop", "startTs": 100, "sport": "running"]), + edit(seq: 2, kind: "fix_workout", + payload: ["deviceId": "my-whoop", "startTs": 100, "sport": "running", "patch": ["energyKcal": 450]], + beforeJSON: beforeJSON), + ] + let summary = await CloudEditApplier.apply(edits, store: store) + + XCTAssertEqual(summary.applied, 1, "only the delete applies") + XCTAssertEqual(summary.skipped, 1, "the same-batch fix must be skipped, not applied") + XCTAssertEqual(summary.needsAttention, 0) + XCTAssertEqual(summary.appliedSeqs, [1, 2]) + + let copies = try await store.workouts(deviceId: "noop-cloud", from: 0, to: 1000, limit: 10) + XCTAssertEqual(copies.count, 0, "the delete already won — fix must not resurrect a noop-cloud copy") + let originalGone = try await store.workouts(deviceId: "my-whoop", from: 0, to: 1000, limit: 10) + XCTAssertEqual(originalGone.count, 0) + } + + func testFixWorkoutNeedsAttentionWhenOriginalAbsentWithNoTombstone() async throws { + // The original is gone but there is no tombstone for it (no same-batch delete either) — a + // stale server mirror, not a resolved delete. This must surface for a human look, not silently + // fabricate a noop-cloud copy of a workout the store never actually held. + let store = try await WhoopStore.inMemory() + let beforeJSON = json(["deviceId": "my-whoop", "startTs": 100, "endTs": 700, "sport": "running", + "source": "whoop", "durationS": 600, "energyKcal": 400, "distanceM": 5000]) + let e = edit(seq: 16, kind: "fix_workout", + payload: ["deviceId": "my-whoop", "startTs": 100, "sport": "running", + "patch": ["energyKcal": 450]], + beforeJSON: beforeJSON) + let summary = await CloudEditApplier.apply([e], store: store) + + XCTAssertEqual(summary.needsAttention, 1) + XCTAssertEqual(summary.applied, 0) + XCTAssertEqual(summary.skipped, 0) + let copies = try await store.workouts(deviceId: "noop-cloud", from: 0, to: 1000, limit: 10) + XCTAssertEqual(copies.count, 0, "no noop-cloud copy must be created for a stale server mirror") + } + // MARK: - add_workout func testAddWorkoutUpsertsUnderCloudDevice() async throws { From 47a00eeb91eebdfa89ecb8f9c7c0a9ebbeaf8ed8 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 17:25:04 -0700 Subject: [PATCH 054/171] feat(cloudsync): coordinator + Data Sources card + Apple Health re-export hook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CloudSyncCoordinator.pull(store:client:) drives the pull side of the fork lane: since = store's "cloud_edits" cursor (0 default) -> fetchEdits -> CloudEditApplier.apply -> ack every processed seq -> advance the cursor to the server's latest seq ONLY after ack succeeds. An ack throw leaves the cursor untouched (re-apply on retry is idempotent by design); an empty page is a no-op (nothing to ack, cursor already correct). CloudEditFetching is the testable seam (CloudSyncClient conforms), mirroring OuraPageFetching. CloudSyncModel (mirrors OuraConnectModel) drives the new Data Sources "Cloud Sync" card: saveSettings(url:token:) rejects an unparseable server URL up front (CloudSyncClient's request-URL builder force-unwraps its URLComponents, so garbage saved here would otherwise crash at the first pull instead of failing the save with a clear message). pullNow(repo:) pulls, then on iOS re-exports into Apple Health when the pull touched sleep/workouts/HR, and always refreshes the repo afterward (CloudEditApplier writes land in the store before ack, so even a pull that ultimately throws can leave real local changes the UI needs to see). HealthKitBridge gets a minimal rerunWriteBack(whoopStore:days:) wrapper around the existing private writeBack — its only caller was sync(days:), which also re-pulls all of Apple Health; the cloud-sync re-export needs just the write side. Dedup is already per-feature in writeBack: vitals and sleep delete-then-rewrite by a deterministic external-UUID key (idempotent regardless of days), heart rate is cursor+48h-window gated (so a rerun's `days:` mostly only widens the FIRST-EVER backfill), and workouts delete-then-rewrite the full requested window by key. Coordinator tests (StrandTests/CloudSyncCoordinatorTests.swift, stub CloudEditFetching + WhoopStore.inMemory()): cursor advances only after a successful ack, an ack failure leaves the cursor untouched, a needsAttention row still gets acked/advances the cursor (it was processed, just not applied), and an empty fetch touches neither ack nor the cursor. --- Strand/CloudSync/CloudSyncCoordinator.swift | 46 ++++++++ Strand/CloudSync/CloudSyncModel.swift | 103 +++++++++++++++++ Strand/Screens/DataSourcesView.swift | 85 +++++++++++++- StrandTests/CloudSyncCoordinatorTests.swift | 121 ++++++++++++++++++++ StrandiOS/Health/HealthKitBridge.swift | 12 ++ 5 files changed, 365 insertions(+), 2 deletions(-) create mode 100644 Strand/CloudSync/CloudSyncCoordinator.swift create mode 100644 Strand/CloudSync/CloudSyncModel.swift create mode 100644 StrandTests/CloudSyncCoordinatorTests.swift diff --git a/Strand/CloudSync/CloudSyncCoordinator.swift b/Strand/CloudSync/CloudSyncCoordinator.swift new file mode 100644 index 0000000000..ba282e5715 --- /dev/null +++ b/Strand/CloudSync/CloudSyncCoordinator.swift @@ -0,0 +1,46 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +#if CLOUD_SYNC +import Foundation +import WhoopStore + +/// Abstracts the edit-journal fetch/ack calls so the coordinator is testable without a live network +/// (CloudSyncClient conforms below) — mirrors OuraSyncCoordinator's `OuraPageFetching` seam. +protocol CloudEditFetching { + func fetchEdits(since: Int) async throws -> (edits: [CloudEdit], latestSeq: Int) + func ack(seqs: [Int]) async throws -> Int +} +extension CloudSyncClient: CloudEditFetching {} + +/// Pulls confirmed noop-cloud edit-journal rows since this device's last cursor, applies them through +/// `CloudEditApplier`, and acknowledges what it processed. A pure coordination step — no state of its +/// own, so (like `CloudEditApplier`) it's a namespace of static functions rather than an instance. +enum CloudSyncCoordinator { + /// The `cursors` table name this lane's pull position is stored under. + static let cursorName = "cloud_edits" + + /// since = this device's stored cursor (0 if never synced) → fetch the page after it → apply + /// every row through `CloudEditApplier` → ack every seq the applier processed (applied, skipped, + /// AND needsAttention alike — see `CloudApplySummary.appliedSeqs`'s doc) → ONLY THEN advance the + /// cursor to the server's latest seq. + /// + /// The ack-then-cursor ordering is load-bearing: if `ack` throws (a transient network failure), + /// the cursor must stay at `since` so the NEXT pull re-fetches and re-applies the same batch — + /// safe only because `CloudEditApplier`'s writes are idempotent upserts/tombstones, never raw + /// appends. Advancing the cursor first would risk acknowledging-by-implication a batch the server + /// never actually heard back from us about, silently losing it forever. + /// + /// An empty page (nothing new since `since`) is a no-op: `appliedSeqs` is empty, so neither `ack` + /// nor the cursor write happens — there is nothing to acknowledge and the cursor is already correct. + static func pull(store: WhoopStore, client: any CloudEditFetching) async throws -> CloudApplySummary { + let since = try await store.cursor(cursorName) ?? 0 + let (edits, latestSeq) = try await client.fetchEdits(since: since) + let summary = await CloudEditApplier.apply(edits, store: store) + guard !summary.appliedSeqs.isEmpty else { return summary } + _ = try await client.ack(seqs: summary.appliedSeqs) + try await store.setCursor(cursorName, latestSeq) + return summary + } +} +#endif // CLOUD_SYNC diff --git a/Strand/CloudSync/CloudSyncModel.swift b/Strand/CloudSync/CloudSyncModel.swift new file mode 100644 index 0000000000..d1da32d6d8 --- /dev/null +++ b/Strand/CloudSync/CloudSyncModel.swift @@ -0,0 +1,103 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +#if CLOUD_SYNC +import Foundation +import WhoopStore +#if os(iOS) +import UIKit +#endif + +/// Drives the Data Sources "Cloud Sync" card: save the noop-cloud server URL + token → pull confirmed +/// edits → apply them locally → (iOS) re-export whatever changed into Apple Health. @MainActor so every +/// `@Published` mutation is main-thread; the network/apply/write-back work hops off-main. +/// +/// `repo: Repository` is taken as a call-time PARAMETER (not stored at construction), for the same +/// reason as `OuraConnectModel`: `Repository` only reaches `DataSourcesView` via `@EnvironmentObject`, +/// which SwiftUI populates after `init()` runs. +@MainActor +final class CloudSyncModel: ObservableObject { + @Published var busy = false + @Published var statusText: String? + /// True once both a server URL and a token are stored in the Keychain (`CloudSyncSettings`). + @Published var isConfigured = CloudSyncSettings.isConfigured + + /// Validates `url` parses as an absolute URL (scheme + host) via `URLComponents` before saving. + /// `CloudSyncClient.url(path:query:)` force-unwraps its `URLComponents`-built request URL, so a + /// garbage base URL saved here would crash later at the first pull instead of failing this save + /// with a clear, recoverable message (flagged in the Task-3 review of `CloudSyncClient`). + func saveSettings(url: String, token: String) { + let trimmedURL = url.trimmingCharacters(in: .whitespacesAndNewlines) + let trimmedToken = token.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmedURL.isEmpty, !trimmedToken.isEmpty else { + statusText = "Enter both a server URL and a token." + return + } + guard let comps = URLComponents(string: trimmedURL), let scheme = comps.scheme, !scheme.isEmpty, + let host = comps.host, !host.isEmpty else { + statusText = "That doesn't look like a valid server URL (needs e.g. https://host)." + return + } + CloudSyncSettings.serverURL = trimmedURL + CloudSyncSettings.token = trimmedToken + isConfigured = CloudSyncSettings.isConfigured + statusText = "Saved." + } + + /// Pull confirmed edits since this device's cursor, apply them, and — on iOS — re-export whatever + /// changed into Apple Health. `repo.refresh()` runs unconditionally at the end (not just on + /// success): `CloudEditApplier` writes land in the local store BEFORE `ack`, so even a pull that + /// ultimately throws (e.g. the ack call itself failing) can leave real local changes the UI needs + /// to pick up. + func pullNow(repo: Repository) { + guard let urlString = CloudSyncSettings.serverURL, let token = CloudSyncSettings.token, + let url = URL(string: urlString) else { + statusText = "Add your noop-cloud server URL and token first." + return + } + setBusy(true); statusText = "Pulling edits…" + Task { + var line: String + if let store = await repo.storeHandle() { + let client = CloudSyncClient(baseURL: url, token: token) + do { + let summary = try await CloudSyncCoordinator.pull(store: store, client: client) + line = "Applied \(summary.applied) · skipped \(summary.skipped) · \(summary.needsAttention) need attention" + #if os(iOS) && canImport(HealthKit) + // Only re-export what could plausibly have changed on Health's side. A fresh bridge + // starts at auth == .unknown (it isn't the app's long-lived HealthKitBridge instance), + // so prime it from the existing grant — refreshAuthIfPreviouslyGranted only READS + // share status, it never prompts — before attempting the write-back. + if summary.touchedSleep || summary.touchedWorkouts || summary.touchedHr { + do { + let bridge = HealthKitBridge(repo: repo, appleDeviceId: "apple-health", noopDeviceId: "my-whoop") + bridge.refreshAuthIfPreviouslyGranted() + try await bridge.rerunWriteBack(whoopStore: store, days: 30) + } catch { + line += " · Apple Health re-export failed: \(error.localizedDescription)" + } + } + #endif + } catch { + line = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription + } + } else { + line = "No local store." + } + statusText = line + await repo.refresh() + setBusy(false) + } + } + + /// Busy also pins the screen awake on iOS, mirroring `OuraConnectModel.setBusy`: the pull + + /// Apple Health re-export is a foreground async flow, and an auto-lock mid-run suspends the app + /// and freezes it with no error. Always reset on exit. + private func setBusy(_ b: Bool) { + busy = b + #if os(iOS) + UIApplication.shared.isIdleTimerDisabled = b + #endif + } +} +#endif // CLOUD_SYNC diff --git a/Strand/Screens/DataSourcesView.swift b/Strand/Screens/DataSourcesView.swift index 800c80e405..7aa2435219 100644 --- a/Strand/Screens/DataSourcesView.swift +++ b/Strand/Screens/DataSourcesView.swift @@ -42,6 +42,16 @@ struct DataSourcesView: View { // `@StateObject` construction time would either fail to compile or crash at runtime. @StateObject private var oura = OuraConnectModel() #endif + #if CLOUD_SYNC + // Cloud Sync (compiled in ONLY with CLOUD_SYNC): pulls edits confirmed on the user's own + // noop-cloud server and applies them locally, re-exporting into Apple Health on iOS. Same + // call-time-`repo:` shape as `OuraConnectModel` above, for the same environment-object-timing + // reason. The URL/token fields are local drafts — `CloudSyncModel` only holds the SAVED values + // (in the Keychain via `CloudSyncSettings`), never the in-progress text. + @StateObject private var cloudSync = CloudSyncModel() + @State private var cloudSyncURLDraft = CloudSyncSettings.serverURL ?? "" + @State private var cloudSyncTokenDraft = "" + #endif // "Remove Apple Health imported data" (ah-delete #616): a destructive escape hatch that purges every // row stored under the "apple-health" source via DeviceRegistryStore.deleteAllData. Two-step (a // confirmation alert) since it can't be undone. Local to this screen; no live strap data is touched. @@ -99,8 +109,11 @@ struct DataSourcesView: View { #if OURA_CLOUD_IMPORT ouraCloudCard.staggeredAppear(index: 7) #endif - broadcastHrCard.staggeredAppear(index: 8) - liveCard.staggeredAppear(index: 9) + #if CLOUD_SYNC + cloudSyncCard.staggeredAppear(index: 8) + #endif + broadcastHrCard.staggeredAppear(index: 9) + liveCard.staggeredAppear(index: 10) } } .onAppear { @@ -328,6 +341,74 @@ struct DataSourcesView: View { } #endif // OURA_CLOUD_IMPORT + #if CLOUD_SYNC + /// Cloud Sync: pulls edit-journal rows YOU confirmed on your own noop-cloud server (a personal-fork + /// companion, not a shipped/supported service) and applies them to the local store, re-exporting + /// into Apple Health on iOS when the pull touched sleep/workouts/HR. `cloudSync.saveSettings(url: + /// token:)`/`pullNow(repo:)` take `repo` at call time — see the `@StateObject` declaration's note. + private var cloudSyncCard: some View { + card(title: String(localized: "Cloud Sync"), icon: "arrow.triangle.2.circlepath.circle", + tint: StrandPalette.metricRose, + subtitle: String(localized: "Pull edits you confirmed on your own noop-cloud server and apply them here. A personal-fork lane — off unless you've set up your own server.")) { + VStack(alignment: .leading, spacing: NoopMetrics.space3) { + VStack(alignment: .leading, spacing: 6) { + Text("Server URL").strandOverline() + TextField("https://your-server.example.com", text: $cloudSyncURLDraft) + .textFieldStyle(.plain) + .font(StrandFont.body) + .foregroundStyle(StrandPalette.textPrimary) + .padding(.horizontal, 12) + .padding(.vertical, 9) + .background(StrandPalette.surfaceInset, in: RoundedRectangle(cornerRadius: 10, style: .continuous)) + .overlay(RoundedRectangle(cornerRadius: 10, style: .continuous) + .strokeBorder(StrandPalette.hairline, lineWidth: 1)) + .disableAutocorrection(true) + #if os(iOS) + .textInputAutocapitalization(.never) + .keyboardType(.URL) + #endif + .accessibilityLabel("noop-cloud server URL") + } + VStack(alignment: .leading, spacing: 6) { + Text("Token").strandOverline() + SecureField("Paste your server's read-write token", text: $cloudSyncTokenDraft) + .textFieldStyle(.plain) + .font(StrandFont.body) + .foregroundStyle(StrandPalette.textPrimary) + .padding(.horizontal, 12) + .padding(.vertical, 9) + .background(StrandPalette.surfaceInset, in: RoundedRectangle(cornerRadius: 10, style: .continuous)) + .overlay(RoundedRectangle(cornerRadius: 10, style: .continuous) + .strokeBorder(StrandPalette.hairline, lineWidth: 1)) + .accessibilityLabel("noop-cloud token") + } + HStack(spacing: NoopMetrics.space3) { + Button { + cloudSync.saveSettings(url: cloudSyncURLDraft, token: cloudSyncTokenDraft) + } label: { + Label("Save", systemImage: "checkmark.circle") + } + .buttonStyle(NoopButtonStyle(.secondary)) + .disabled(cloudSyncURLDraft.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty + || cloudSyncTokenDraft.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty) + + Button { + cloudSync.pullNow(repo: repo) + } label: { + Label(cloudSync.busy ? "Pulling…" : "Pull edits now", systemImage: "arrow.down.circle") + } + .buttonStyle(NoopButtonStyle(.primary)) + .disabled(!cloudSync.isConfigured || cloudSync.busy) + if cloudSync.busy { ProgressView().controlSize(.small) } + } + if let s = cloudSync.statusText { + Text(s).font(StrandFont.subhead).foregroundStyle(StrandPalette.textSecondary) + } + } + } + } + #endif // CLOUD_SYNC + private func presentImporter(_ target: ImportTarget) { importTarget = target #if os(iOS) diff --git a/StrandTests/CloudSyncCoordinatorTests.swift b/StrandTests/CloudSyncCoordinatorTests.swift new file mode 100644 index 0000000000..d5d545f78b --- /dev/null +++ b/StrandTests/CloudSyncCoordinatorTests.swift @@ -0,0 +1,121 @@ +// Tests the CLOUD_SYNC-gated cloud sync coordinator; compiled only when the flag is set +// (StrandTests shares the app's OuraConfig.xcconfig, so flag + creds arrive together). +#if CLOUD_SYNC +import XCTest +import WhoopStore +@testable import Strand + +/// A fetcher that serves canned edits + a latest seq, and can simulate an ack failure. Mirrors +/// OuraSyncCoordinatorTests's StubFetcher. +private final class StubCloudEditFetcher: CloudEditFetching { + var edits: [CloudEdit] = [] + var latestSeq = 0 + var failAck = false + private(set) var fetchSinceCalls: [Int] = [] + private(set) var ackedSeqs: [[Int]] = [] + + func fetchEdits(since: Int) async throws -> (edits: [CloudEdit], latestSeq: Int) { + fetchSinceCalls.append(since) + return (edits, latestSeq) + } + func ack(seqs: [Int]) async throws -> Int { + if failAck { throw StubAckError() } + ackedSeqs.append(seqs) + return seqs.count + } +} + +private struct StubAckError: Error {} + +/// Builds a minimal CloudEdit for a given kind + payload, with the rest of the envelope defaulted. +private func makeEdit(seq: Int, kind: String, payloadJSON: String, beforeJSON: String? = nil) -> CloudEdit { + CloudEdit(seq: seq, editId: "e\(seq)", kind: kind, payloadJSON: payloadJSON, beforeJSON: beforeJSON, + rationale: nil, appliedAt: 0, undoneBySeq: nil, ackedAt: nil) +} + +final class CloudSyncCoordinatorTests: XCTestCase { + func testAppliesEditsAcksAndAdvancesCursorOnlyAfterAck() async throws { + let store = try await WhoopStore.inMemory() + let fetcher = StubCloudEditFetcher() + // delete_metric_point applies cleanly even with nothing to delete (tombstone insert + a + // no-op DELETE both succeed), so this is a deterministic "applied" outcome. + fetcher.edits = [makeEdit(seq: 1, kind: "delete_metric_point", + payloadJSON: #"{"deviceId":"my-whoop","day":"2026-01-01","key":"steps"}"#)] + fetcher.latestSeq = 1 + + let before = try await store.cursor("cloud_edits") + XCTAssertNil(before) + + let summary = try await CloudSyncCoordinator.pull(store: store, client: fetcher) + + XCTAssertEqual(summary.applied, 1) + XCTAssertEqual(summary.appliedSeqs, [1]) + XCTAssertEqual(fetcher.ackedSeqs, [[1]]) + let after = try await store.cursor("cloud_edits") + XCTAssertEqual(after, 1) + + // A second pull starts from the advanced cursor, not from 0 again. + fetcher.edits = [] + fetcher.latestSeq = 1 + _ = try await CloudSyncCoordinator.pull(store: store, client: fetcher) + XCTAssertEqual(fetcher.fetchSinceCalls, [0, 1]) + } + + func testAckFailureKeepsCursorUnchanged() async throws { + let store = try await WhoopStore.inMemory() + let fetcher = StubCloudEditFetcher() + fetcher.edits = [makeEdit(seq: 5, kind: "delete_metric_point", + payloadJSON: #"{"deviceId":"my-whoop","day":"2026-01-01","key":"steps"}"#)] + fetcher.latestSeq = 5 + fetcher.failAck = true + + do { + _ = try await CloudSyncCoordinator.pull(store: store, client: fetcher) + XCTFail("expected the ack failure to propagate") + } catch { + // expected — ack threw, so pull() must throw rather than silently swallow it. + } + + // Cursor must stay unset: re-applying the same batch on retry is idempotent, but only if the + // cursor never advanced past an edit that was never actually acked. + let cursor = try await store.cursor("cloud_edits") + XCTAssertNil(cursor) + } + + func testNeedsAttentionSurfacesInSummaryAndStillAdvancesCursor() async throws { + let store = try await WhoopStore.inMemory() + let fetcher = StubCloudEditFetcher() + // adjust_sleep_bounds against a session that doesn't exist in the (empty) store — + // CloudEditApplier can't find it, so this lands in needsAttention, not applied. + fetcher.edits = [makeEdit(seq: 9, kind: "adjust_sleep_bounds", + payloadJSON: #"{"deviceId":"my-whoop","startTs":1700000000,"newStartTs":null,"newEndTs":null}"#)] + fetcher.latestSeq = 9 + + let summary = try await CloudSyncCoordinator.pull(store: store, client: fetcher) + + XCTAssertEqual(summary.needsAttention, 1) + XCTAssertEqual(summary.applied, 0) + // needsAttention rows are still "processed" — acked and cursor-advanced like any other seq. + XCTAssertEqual(fetcher.ackedSeqs, [[9]]) + let cursor = try await store.cursor("cloud_edits") + XCTAssertEqual(cursor, 9) + } + + func testEmptyFetchNoOps() async throws { + let store = try await WhoopStore.inMemory() + let fetcher = StubCloudEditFetcher() + fetcher.edits = [] + fetcher.latestSeq = 42 // server is ahead, but there's nothing THIS device hasn't seen + + let summary = try await CloudSyncCoordinator.pull(store: store, client: fetcher) + + XCTAssertEqual(summary.applied, 0) + XCTAssertEqual(summary.skipped, 0) + XCTAssertEqual(summary.needsAttention, 0) + XCTAssertTrue(summary.appliedSeqs.isEmpty) + XCTAssertTrue(fetcher.ackedSeqs.isEmpty) // no ack call at all + let cursor = try await store.cursor("cloud_edits") + XCTAssertNil(cursor) // nothing processed, so the cursor is left alone + } +} +#endif // CLOUD_SYNC diff --git a/StrandiOS/Health/HealthKitBridge.swift b/StrandiOS/Health/HealthKitBridge.swift index a3fd7996e9..fb1a5317f8 100644 --- a/StrandiOS/Health/HealthKitBridge.swift +++ b/StrandiOS/Health/HealthKitBridge.swift @@ -442,6 +442,18 @@ final class HealthKitBridge: ObservableObject { // MARK: - Write back (NOOP → Health) + /// Public re-entry point for the merged write-back, for callers OTHER than `sync(days:)` — namely + /// `CloudSyncModel.pullNow` (Strand/CloudSync/CloudSyncModel.swift), which needs to re-export + /// Apple Health after a cloud edit touches sleep/workouts/HR without also re-running the read-side + /// HealthKit pull `sync` does. A thin, otherwise-unchanged wrapper around the private `writeBack` + /// below — same auth guard, same dedup semantics per feature (see `writeBack`'s doc comment). + /// Guarded the same way `writeBack` itself is (`auth == .authorized`); a caller built on a fresh + /// `HealthKitBridge` instance should call `refreshAuthIfPreviouslyGranted()` first so a previously + /// granted user's re-export doesn't silently no-op against a default `.unknown` auth state. + func rerunWriteBack(whoopStore: WhoopStore, days: Int) async throws { + try await writeBack(whoopStore: whoopStore, days: days) + } + /// Write NOOP's strap-derived data into Apple Health: sleep sessions with full stage segments, /// the continuous 1-minute heart-rate stream, strap/manual workouts, and the nightly vitals /// (resting HR, HRV, SpO₂, respiratory rate) stamped at that day's wake time. From 03afea8868640fe9f51300e7b823b533014a439d Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 17:37:04 -0700 Subject: [PATCH 055/171] docs(cloudsync): document the optional CLOUD_SYNC compilation flag in the example xcconfig --- Strand/Oura/OuraSecrets.example.xcconfig | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Strand/Oura/OuraSecrets.example.xcconfig b/Strand/Oura/OuraSecrets.example.xcconfig index 4151f469e7..7dca49b1ad 100644 --- a/Strand/Oura/OuraSecrets.example.xcconfig +++ b/Strand/Oura/OuraSecrets.example.xcconfig @@ -12,3 +12,7 @@ OURA_CLIENT_SECRET = your_client_secret_here // expansion $() below breaks up the "//" token without changing the resolved string, which is // still exactly noop://oura/callback at build time. Keep this trick if you edit the value. OURA_REDIRECT_URI = noop:/$()/oura/callback + +// Optional second personal-fork lane: add CLOUD_SYNC to the same line to compile the +// Strand/CloudSync module (pull confirmed noop-cloud edits + Apple Health re-export). +// SWIFT_ACTIVE_COMPILATION_CONDITIONS = $(inherited) OURA_CLOUD_IMPORT CLOUD_SYNC From f5d3a397deaaecff7d3bd83d0f6c7731bf3c4229 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 18:20:00 -0700 Subject: [PATCH 056/171] =?UTF-8?q?feat(cloudsync):=20zero-touch=20?= =?UTF-8?q?=E2=80=94=20bundle=20credentials,=20in-app=20upload,=20on-launc?= =?UTF-8?q?h=20auto-sync?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 3.5: the CloudSync lane (Phase 3) needed a manual Keychain save and a manual "Pull edits now" tap every time. This closes both gaps. CloudSyncSettings gains effectiveURL/effectiveToken: Keychain wins, else the bundle-injected CLOUDSYNC_URL/CLOUDSYNC_TOKEN Info.plist keys (trimmed, blank = absent) — same untracked-xcconfig chain as OURA_CLIENT_ID (OuraCredentials.swift). The precedence itself is a pure effectiveValue(keychain:infoKey:info:) function, directly unit-testable the same way OuraCredentials.from(_:) is, since Bundle.main.infoDictionary can't be swapped out in a test. isConfigured and isBundleConfigured both key off the effective values, so a bundle-only install (no Keychain write ever happening) is a first-class configured state. New CloudSyncUploader.upload(store:client:) produces a checkpointed, integrity-verified .noopbak in Caches (reusing DataBackup.writeBackup, the same export FolderBackup/BackupSync use) and POSTs it to /ingest via the new CloudSyncClient.ingest(fileURL:) — Bearer + application/octet-stream, streamed from disk with URLSession.upload(for:fromFile:) since the DB can be 100-300MB. The export step is injected through an Exporter typealias (default = DataBackup.writeBackup) so CloudSyncUploaderTests can supply canned bytes without touching StorePaths.defaultDatabasePath() — a WhoopStore.inMemory() test store has no backing file for the real export to read. CloudSyncModel.syncNow(repo:) uploads then pulls in one round trip, reusing a factored-out reExportToAppleHealthIfNeeded helper (previously inline in pullNow) so both surface the same Apple Health re-export note. autoSyncIfDue(repo:) is the on-launch catch-up: no-ops when CloudSyncSettings.isConfigured is false, otherwise fires syncNow when isAutoSyncDue(lastRun:now:) — a pure, nonisolated 20h-gate function — says it's been long enough since the last SUCCESS (UserDefaults "cloudsync.lastAutoSync", stamped only on success so a failed attempt doesn't block the next retry). pullNow now also reads the effective credentials, so it stays usable on a bundle-only install too. Wired at the existing FolderBackup.catchUpIfDue launch seam in both RootView.swift (macOS) and RootTabView.swift (iOS) — CloudSyncModel().autoSyncIfDue is a plain MainActor call that spawns its own Task internally (mirrors pullNow/syncNow), so it needs no Task.detached wrapper of its own to stay off the launch-critical path. DataSourcesView's Cloud Sync card collapses to a one-line "Configured from build" summary + Sync now + status when the build carries bundle credentials and no Keychain override exists; "Override..." reveals the existing URL/token fields for a manual, per-device server (which then wins on every future launch). Design tokens only. project.yml gains CLOUDSYNC_URL/CLOUDSYNC_TOKEN info.properties next to the OURA_* keys in both the Strand and NOOPiOS target blocks (and the generated Info.plist files that follow from it); OuraSecrets.example.xcconfig documents the two new optional keys. project.yml's pre-existing personal-signing hunks (DEVELOPMENT_TEAM, APP_GROUP_ID, bundle ids, NOOPWatch embed) are untouched and stay uncommitted. Tests: CloudSyncUploaderTests (happy path asserts the Bearer header + octet-stream content type via an extended OuraURLProtocolStub that now captures full requests, non-2xx, export-failure-never-posts, temp-file cleanup), CloudSyncSettingsTests gains the bundle-fallback precedence cases, CloudSyncModelAutoSyncGateTests covers the 20h gate. --- Strand/App/RootView.swift | 10 ++ Strand/CloudSync/CloudSyncClient.swift | 36 +++++ Strand/CloudSync/CloudSyncModel.swift | 118 +++++++++++++-- Strand/CloudSync/CloudSyncSettings.swift | 51 ++++++- Strand/CloudSync/CloudSyncUploader.swift | 75 ++++++++++ Strand/Oura/OuraSecrets.example.xcconfig | 11 +- Strand/Resources/Info.plist | 4 + Strand/Screens/DataSourcesView.swift | 178 +++++++++++++++-------- StrandTests/CloudSyncClientTests.swift | 52 +++++++ StrandTests/CloudSyncUploaderTests.swift | 123 ++++++++++++++++ StrandTests/OuraURLProtocolStub.swift | 8 +- StrandiOS/App/RootTabView.swift | 5 + StrandiOS/Resources/Info.plist | 4 + project.yml | 10 ++ 14 files changed, 603 insertions(+), 82 deletions(-) create mode 100644 Strand/CloudSync/CloudSyncUploader.swift create mode 100644 StrandTests/CloudSyncUploaderTests.swift diff --git a/Strand/App/RootView.swift b/Strand/App/RootView.swift index d5213c8d7f..3a88f17ec6 100644 --- a/Strand/App/RootView.swift +++ b/Strand/App/RootView.swift @@ -301,6 +301,16 @@ struct RootView: View { Task.detached(priority: .utility) { await FolderBackup.catchUpIfDue(checkpoint: { await backupRepo.checkpointForBackup() }) } + #if CLOUD_SYNC + // Cloud Sync: zero-touch on-launch catch-up (Phase 3.5) — bundle-injected build credentials + // (or a manual Keychain save) need no further setup; `CloudSyncSettings.isConfigured` gates + // silently otherwise. `autoSyncIfDue` is a plain MainActor call that spawns its own `Task` + // internally (mirrors `pullNow`/`syncNow`'s own shape), so — unlike `FolderBackup` above, + // whose async functions the CALLER wraps in `Task.detached` — this needs no detached wrapper + // of its own to stay off the launch-critical path. The 20h gate lives in + // `CloudSyncModel.isAutoSyncDue`. + CloudSyncModel().autoSyncIfDue(repo: repo) + #endif } // Honour a cross-screen request to open a top-level destination (e.g. Live's "Manage devices"), // then clear it so the same tap can fire again later. Devices maps to the `.devices` sidebar item. diff --git a/Strand/CloudSync/CloudSyncClient.swift b/Strand/CloudSync/CloudSyncClient.swift index 434853408c..03aa7abc03 100644 --- a/Strand/CloudSync/CloudSyncClient.swift +++ b/Strand/CloudSync/CloudSyncClient.swift @@ -68,6 +68,37 @@ final class CloudSyncClient { return decoded.acked } + // MARK: - Ingest (Phase 3.5: upload this device's own .noopbak) + + /// POST /ingest, Bearer-authed, with the file at `fileURL` streamed straight from disk as a raw + /// `application/octet-stream` body via `URLSession.upload(for:fromFile:)` — the whole-DB `.noopbak` + /// can be 100-300MB, so it must never be loaded into memory as `Data` the way `send(_:)`'s + /// `session.data(for:)` would. Parses `{ok,bytes,latestDay}`; only `bytes`/`latestDay` are modelled + /// (Decodable ignores the unmodelled `ok` key — it's redundant with the 2xx status check anyway). + func ingest(fileURL: URL) async throws -> (bytes: Int, latestDay: String?) { + var req = URLRequest(url: url(path: "ingest")) + req.httpMethod = "POST" + req.setValue("Bearer \(token)", forHTTPHeaderField: "Authorization") + req.setValue("application/octet-stream", forHTTPHeaderField: "Content-Type") + let (data, status) = try await sendUpload(req, fromFile: fileURL) + guard (200..<300).contains(status) else { + throw CloudSyncError.badResponse(status, bodyPrefix(data)) + } + guard let decoded = try? JSONDecoder().decode(IngestResponse.self, from: data) else { + throw CloudSyncError.decode + } + return (decoded.bytes, decoded.latestDay) + } + + private func sendUpload(_ req: URLRequest, fromFile fileURL: URL) async throws -> (Data, Int) { + do { + let (data, resp) = try await session.upload(for: req, fromFile: fileURL) + return (data, (resp as? HTTPURLResponse)?.statusCode ?? 0) + } catch { + throw CloudSyncError.network(error.localizedDescription) + } + } + private func url(path: String, query: [String: String] = [:]) -> URL { var comps = URLComponents(string: "\(baseURL.absoluteString)/\(path)")! if !query.isEmpty { @@ -100,4 +131,9 @@ private struct AckRequest: Encodable { private struct AckResponse: Decodable { let acked: Int } + +private struct IngestResponse: Decodable { + let bytes: Int + let latestDay: String? +} #endif // CLOUD_SYNC diff --git a/Strand/CloudSync/CloudSyncModel.swift b/Strand/CloudSync/CloudSyncModel.swift index d1da32d6d8..7cd31af8b1 100644 --- a/Strand/CloudSync/CloudSyncModel.swift +++ b/Strand/CloudSync/CloudSyncModel.swift @@ -48,9 +48,11 @@ final class CloudSyncModel: ObservableObject { /// changed into Apple Health. `repo.refresh()` runs unconditionally at the end (not just on /// success): `CloudEditApplier` writes land in the local store BEFORE `ack`, so even a pull that /// ultimately throws (e.g. the ack call itself failing) can leave real local changes the UI needs - /// to pick up. + /// to pick up. Uses the EFFECTIVE credentials (`CloudSyncSettings.effectiveURL`/`effectiveToken`) + /// so a bundle-only configuration (Phase 3.5, no Keychain save) works here too, not just in + /// `syncNow`. func pullNow(repo: Repository) { - guard let urlString = CloudSyncSettings.serverURL, let token = CloudSyncSettings.token, + guard let urlString = CloudSyncSettings.effectiveURL, let token = CloudSyncSettings.effectiveToken, let url = URL(string: urlString) else { statusText = "Add your noop-cloud server URL and token first." return @@ -64,19 +66,7 @@ final class CloudSyncModel: ObservableObject { let summary = try await CloudSyncCoordinator.pull(store: store, client: client) line = "Applied \(summary.applied) · skipped \(summary.skipped) · \(summary.needsAttention) need attention" #if os(iOS) && canImport(HealthKit) - // Only re-export what could plausibly have changed on Health's side. A fresh bridge - // starts at auth == .unknown (it isn't the app's long-lived HealthKitBridge instance), - // so prime it from the existing grant — refreshAuthIfPreviouslyGranted only READS - // share status, it never prompts — before attempting the write-back. - if summary.touchedSleep || summary.touchedWorkouts || summary.touchedHr { - do { - let bridge = HealthKitBridge(repo: repo, appleDeviceId: "apple-health", noopDeviceId: "my-whoop") - bridge.refreshAuthIfPreviouslyGranted() - try await bridge.rerunWriteBack(whoopStore: store, days: 30) - } catch { - line += " · Apple Health re-export failed: \(error.localizedDescription)" - } - } + line += await reExportToAppleHealthIfNeeded(repo: repo, store: store, summary: summary) #endif } catch { line = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription @@ -90,6 +80,104 @@ final class CloudSyncModel: ObservableObject { } } + /// Upload this device's own `.noopbak` to the noop-cloud server, then pull + apply any confirmed + /// edits — the two halves of "zero-touch" in one round trip: this device's own data reaches the + /// server, and any edits confirmed elsewhere reach this device. Stamps + /// `UserDefaults["cloudsync.lastAutoSync"]` on SUCCESS ONLY, so a failed attempt (server + /// unreachable, network down) never blocks the next 20h-gated retry. Used by both the + /// user-initiated "Sync now" button and `autoSyncIfDue`'s on-launch catch-up — a manual tap also + /// resets the on-launch catch-up clock, so it doesn't immediately re-fire on the next launch. + func syncNow(repo: Repository) { + guard let urlString = CloudSyncSettings.effectiveURL, let token = CloudSyncSettings.effectiveToken, + let url = URL(string: urlString) else { + statusText = "Add your noop-cloud server URL and token first." + return + } + setBusy(true); statusText = "Uploading…" + Task { + var line: String + var succeeded = false + if let store = await repo.storeHandle() { + let client = CloudSyncClient(baseURL: url, token: token) + do { + let uploaded = try await CloudSyncUploader.upload(store: store, client: client) + let mb = Double(uploaded.bytes) / 1_048_576.0 + statusText = String(format: "Uploaded %.1f MB · pulling edits…", mb) + + let summary = try await CloudSyncCoordinator.pull(store: store, client: client) + var parts = [String(format: "Uploaded %.1f MB", mb), + "Applied \(summary.applied)", "skipped \(summary.skipped)"] + if summary.needsAttention > 0 { parts.append("\(summary.needsAttention) attention") } + line = parts.joined(separator: " · ") + succeeded = true + #if os(iOS) && canImport(HealthKit) + line += await reExportToAppleHealthIfNeeded(repo: repo, store: store, summary: summary) + #endif + } catch { + line = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription + } + } else { + line = "No local store." + } + statusText = line + if succeeded { + UserDefaults.standard.set(Date().timeIntervalSince1970, forKey: Self.lastAutoSyncKey) + } + await repo.refresh() + setBusy(false) + } + } + + /// UserDefaults key for the last successful `syncNow` (manual or automatic) — read by + /// `autoSyncIfDue`, written by `syncNow` on success only. + private static let lastAutoSyncKey = "cloudsync.lastAutoSync" + + /// Pure 20h gate, `(lastRun, now) -> Bool`, so the threshold is unit-testable without UserDefaults, + /// a store, or a network (see `CloudSyncModelAutoSyncGateTests`). `lastRun == 0` (never synced) is + /// always due — a fresh install syncs on its very first launch, matching "zero manual taps". + /// `nonisolated`: touches no actor-isolated state, so it's callable synchronously from a plain + /// (non-`@MainActor`, non-`async`) XCTest method — without this, it inherits the enclosing class's + /// `@MainActor` and a test would need `await MainActor.run { … }` just to call a pure function. + nonisolated static func isAutoSyncDue(lastRun: TimeInterval, now: TimeInterval, + intervalS: TimeInterval = 20 * 3600) -> Bool { + now - lastRun >= intervalS + } + + /// On-launch daily catch-up (Phase 3.5: zero-touch): silently no-ops when the lane isn't configured + /// — `CloudSyncSettings.isConfigured` already covers both a bundle-injected build and a manual + /// Keychain save — otherwise fires `syncNow` when it's been >20h since the last success. Call-time + /// `repo:` parameter, same reason as every other method here (see the `@StateObject` declaration's + /// note). A plain (non-`async`) MainActor call, mirroring `pullNow`/`syncNow`'s own shape: it + /// spawns its OWN `Task` internally and returns immediately, so a caller on the launch-critical path + /// (`RootView`/`RootTabView`'s `.task`) is never blocked waiting on the network. + func autoSyncIfDue(repo: Repository) { + guard CloudSyncSettings.isConfigured else { return } + let last = UserDefaults.standard.double(forKey: Self.lastAutoSyncKey) + guard Self.isAutoSyncDue(lastRun: last, now: Date().timeIntervalSince1970) else { return } + syncNow(repo: repo) + } + + #if os(iOS) && canImport(HealthKit) + /// Re-export into Apple Health when a pulled batch touched sleep/workouts/HR — shared by `pullNow` + /// and `syncNow` so both surface the same "Apple Health re-export failed" note. Only re-exports + /// what could plausibly have changed on Health's side. A fresh bridge starts at auth == .unknown + /// (it isn't the app's long-lived HealthKitBridge instance), so prime it from the existing grant — + /// `refreshAuthIfPreviouslyGranted` only READS share status, it never prompts — before attempting + /// the write-back. Returns a status-line SUFFIX: empty when nothing ran or it succeeded. + private func reExportToAppleHealthIfNeeded(repo: Repository, store: WhoopStore, + summary: CloudApplySummary) async -> String { + guard summary.touchedSleep || summary.touchedWorkouts || summary.touchedHr else { return "" } + do { + let bridge = HealthKitBridge(repo: repo, appleDeviceId: "apple-health", noopDeviceId: "my-whoop") + bridge.refreshAuthIfPreviouslyGranted() + try await bridge.rerunWriteBack(whoopStore: store, days: 30) + return "" + } catch { + return " · Apple Health re-export failed: \(error.localizedDescription)" + } + } + #endif + /// Busy also pins the screen awake on iOS, mirroring `OuraConnectModel.setBusy`: the pull + /// Apple Health re-export is a foreground async flow, and an auto-lock mid-run suspends the app /// and freezes it with no error. Always reset on exit. diff --git a/Strand/CloudSync/CloudSyncSettings.swift b/Strand/CloudSync/CloudSyncSettings.swift index e58caa33bb..b1d34cbbbd 100644 --- a/Strand/CloudSync/CloudSyncSettings.swift +++ b/Strand/CloudSync/CloudSyncSettings.swift @@ -34,8 +34,10 @@ enum CloudSyncSettings { set { save(newValue, account: tokenAccount) } } - /// True once both a server URL and a token are stored. - static var isConfigured: Bool { serverURL != nil && token != nil } + /// True once both an EFFECTIVE server URL and token are available — a Keychain value, a + /// bundle-injected one, or a mix of both. Phase 3.5 (zero-touch): a bundle-only install counts as + /// configured with no Keychain write ever happening. + static var isConfigured: Bool { effectiveURL != nil && effectiveToken != nil } /// Remove both stored values. static func clear() { @@ -43,6 +45,51 @@ enum CloudSyncSettings { SecItemDelete(baseQuery(account: tokenAccount) as CFDictionary) } + // MARK: - Bundle fallback (Phase 3.5: zero-touch) + // + // Mirrors Strand/Oura/OuraCredentials.swift's shape: a pure, dict-driven function that's directly + // unit-testable (`effectiveValue`), plus a thin `Bundle.main`-reading wrapper (`effectiveURL`/ + // `effectiveToken`) that can't be redirected in a test but has no logic of its own to get wrong. + + /// Read `key` from an Info-dictionary-shaped map, trimmed; blank/absent both read as nil so a + /// build without injected credentials cleanly falls through to "unconfigured" rather than saving a + /// whitespace server URL. + private static func bundleValue(_ key: String, from info: [String: Any]) -> String? { + guard let s = (info[key] as? String)?.trimmingCharacters(in: .whitespacesAndNewlines), + !s.isEmpty else { return nil } + return s + } + + /// Keychain-wins-over-bundle precedence, as a pure function of its inputs — the seam that makes + /// the precedence rule unit-testable without a real Info.plist (see `CloudSyncSettingsTests`). + static func effectiveValue(keychain: String?, infoKey: String, info: [String: Any]) -> String? { + keychain ?? bundleValue(infoKey, from: info) + } + + /// The noop-cloud server URL a manual Save in the Data Sources card wrote to the Keychain, or the + /// build-injected `CLOUDSYNC_URL` (from the untracked `OuraSecrets.xcconfig` → project.yml → + /// Info.plist chain — see `OuraCredentials.swift`'s doc comment for the identical pattern) when no + /// Keychain override exists. nil disables the whole cloud-sync lane. + static var effectiveURL: String? { + effectiveValue(keychain: serverURL, infoKey: "CLOUDSYNC_URL", info: Bundle.main.infoDictionary ?? [:]) + } + + /// The read-write token, same Keychain-wins-over-bundle precedence as `effectiveURL`. + static var effectiveToken: String? { + effectiveValue(keychain: token, infoKey: "CLOUDSYNC_TOKEN", info: Bundle.main.infoDictionary ?? [:]) + } + + /// True when the lane is configured ENTIRELY from the bundle-injected build credentials, with no + /// Keychain override at all. Drives the Data Sources card: bundle-configured shows a one-line + /// "Configured from build" note instead of the URL/token fields; any Keychain value (even a + /// partial one) falls through to the manual fields so a user's own in-progress edit is never + /// silently hidden behind the collapsed summary. + static var isBundleConfigured: Bool { + guard serverURL == nil, token == nil else { return false } + let info = Bundle.main.infoDictionary ?? [:] + return bundleValue("CLOUDSYNC_URL", from: info) != nil && bundleValue("CLOUDSYNC_TOKEN", from: info) != nil + } + /// Store (or replace) `value` for `account`. Empty/whitespace input clears the item instead /// (so an accidental blank Save can't leave a dead-but-present Keychain entry). @discardableResult diff --git a/Strand/CloudSync/CloudSyncUploader.swift b/Strand/CloudSync/CloudSyncUploader.swift new file mode 100644 index 0000000000..a5fdefdd66 --- /dev/null +++ b/Strand/CloudSync/CloudSyncUploader.swift @@ -0,0 +1,75 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +#if CLOUD_SYNC +import Foundation +import WhoopStore + +/// Abstracts the raw-bytes ingest POST so `CloudSyncUploader` is testable without a live network — +/// mirrors `CloudSyncCoordinator`'s `CloudEditFetching` seam. `CloudSyncClient` conforms below. +protocol CloudIngesting { + func ingest(fileURL: URL) async throws -> (bytes: Int, latestDay: String?) +} +extension CloudSyncClient: CloudIngesting {} + +/// The export half's user-facing failure. The network half of an upload throws `CloudSyncError` (same +/// typed error every other CloudSync network call throws); this covers only "never got as far as +/// having bytes to send". +enum CloudSyncUploadError: LocalizedError, Equatable { + case exportFailed(String) + + var errorDescription: String? { + switch self { + case .exportFailed(let detail): + return "Couldn't prepare the backup to upload. \(detail)" + } + } +} + +/// Produces this device's own checkpointed, integrity-verified `.noopbak` and POSTs it to the +/// noop-cloud server's `/ingest` endpoint — the "upload" half of Phase 3.5's zero-touch sync (the +/// "pull" half is `CloudSyncCoordinator`). A pure coordination step with no state of its own, like +/// `CloudSyncCoordinator`, so it's a namespace of static functions rather than an instance. +enum CloudSyncUploader { + /// Produces a `.noopbak` at `dest` from `store`, returning `DataBackup.BackupResult` so a real + /// export failure's message survives. Injectable so a test can supply canned bytes without + /// touching the app's real on-disk database: `WhoopStore.inMemory()` test stores have no backing + /// file at all (see `WhoopStore.inMemory()`'s doc comment — a `DatabaseQueue`, not a file-backed + /// `DatabasePool`), and the production default below is hardcoded to + /// `StorePaths.defaultDatabasePath()` regardless of which `WhoopStore` instance is passed in. That + /// fixed path is correct for production — there is only ever one real on-disk database, and + /// `FolderBackup.backupNow`/`DataBackup.runExport` resolve it the exact same way — but it makes the + /// default exporter untestable against a throwaway store, hence the seam. + typealias Exporter = (WhoopStore, URL) async -> DataBackup.BackupResult + + /// The real export: checkpoint `store`'s WAL (so the single `.sqlite` file is whole), then reuse + /// the SAME checkpointed, `PRAGMA quick_check`-verified export `BackupSync`/`FolderBackup` use — an + /// auto-uploaded snapshot is byte-identical to a manual "Export backup". + static let defaultExporter: Exporter = { store, dest in + await DataBackup.writeBackup(checkpoint: { (try? await store.checkpointWAL()) != nil }, to: dest) + } + + /// Export the live store to a disposable temp file in Caches (never Documents — nothing here is + /// meant to persist or be user-visible) and POST it to `/ingest`. The temp file is removed in + /// `defer`, whatever happens: success, an export failure, or a network failure. The DB can be + /// 100-300MB, so `CloudSyncClient.ingest` streams it from this file via + /// `URLSession.upload(for:fromFile:)` rather than loading it into memory. + static func upload(store: WhoopStore, client: any CloudIngesting, + exporter: Exporter = defaultExporter) async throws -> (bytes: Int, latestDay: String?) { + let cachesDir = FileManager.default.urls(for: .cachesDirectory, in: .userDomainMask)[0] + let tempURL = cachesDir.appendingPathComponent("cloudsync-upload-\(UUID().uuidString).noopbak") + defer { try? FileManager.default.removeItem(at: tempURL) } + + switch await exporter(store, tempURL) { + case .exported: + return try await client.ingest(fileURL: tempURL) + case .failure(let message): + throw CloudSyncUploadError.exportFailed(message) + case .cancelled, .imported: + // The checkpointed export path (no picker, no import flow) never actually returns these — + // handled explicitly so the switch stays exhaustive without a silently-wrong `default`. + throw CloudSyncUploadError.exportFailed("The export step returned an unexpected result.") + } + } +} +#endif // CLOUD_SYNC diff --git a/Strand/Oura/OuraSecrets.example.xcconfig b/Strand/Oura/OuraSecrets.example.xcconfig index 7dca49b1ad..b87ed68cdb 100644 --- a/Strand/Oura/OuraSecrets.example.xcconfig +++ b/Strand/Oura/OuraSecrets.example.xcconfig @@ -14,5 +14,14 @@ OURA_CLIENT_SECRET = your_client_secret_here OURA_REDIRECT_URI = noop:/$()/oura/callback // Optional second personal-fork lane: add CLOUD_SYNC to the same line to compile the -// Strand/CloudSync module (pull confirmed noop-cloud edits + Apple Health re-export). +// Strand/CloudSync module (upload this device's own backup, pull confirmed noop-cloud edits, apply +// them, and re-export into Apple Health). // SWIFT_ACTIVE_COMPILATION_CONDITIONS = $(inherited) OURA_CLOUD_IMPORT CLOUD_SYNC +// +// Zero-touch (Phase 3.5): with CLOUD_SYNC set, these two keys pre-configure the lane at build time — +// no manual URL/token entry in the app, no manual tap to start syncing (an on-launch catch-up fires +// automatically at most once per ~20h). Read at runtime via CloudSyncSettings.effectiveURL/ +// effectiveToken; absent/blank leaves the lane on manual Keychain configuration only (Data Sources → +// Cloud Sync → Override…). A Keychain value saved through that manual override always wins over these. +// CLOUDSYNC_URL = https://your-noop-cloud-server.example.com +// CLOUDSYNC_TOKEN = your_read_write_token_here diff --git a/Strand/Resources/Info.plist b/Strand/Resources/Info.plist index 8a2f2a392f..d3e20eae45 100644 --- a/Strand/Resources/Info.plist +++ b/Strand/Resources/Info.plist @@ -29,6 +29,10 @@ CFBundleVersion 198 + CLOUDSYNC_TOKEN + $(CLOUDSYNC_TOKEN) + CLOUDSYNC_URL + $(CLOUDSYNC_URL) LSApplicationCategoryType public.app-category.healthcare-fitness LSMinimumSystemVersion diff --git a/Strand/Screens/DataSourcesView.swift b/Strand/Screens/DataSourcesView.swift index 7aa2435219..1117b75345 100644 --- a/Strand/Screens/DataSourcesView.swift +++ b/Strand/Screens/DataSourcesView.swift @@ -43,14 +43,18 @@ struct DataSourcesView: View { @StateObject private var oura = OuraConnectModel() #endif #if CLOUD_SYNC - // Cloud Sync (compiled in ONLY with CLOUD_SYNC): pulls edits confirmed on the user's own - // noop-cloud server and applies them locally, re-exporting into Apple Health on iOS. Same - // call-time-`repo:` shape as `OuraConnectModel` above, for the same environment-object-timing - // reason. The URL/token fields are local drafts — `CloudSyncModel` only holds the SAVED values - // (in the Keychain via `CloudSyncSettings`), never the in-progress text. + // Cloud Sync (compiled in ONLY with CLOUD_SYNC): uploads this device's own backup and pulls edits + // confirmed on the user's own noop-cloud server, applying them locally and re-exporting into Apple + // Health on iOS. Same call-time-`repo:` shape as `OuraConnectModel` above, for the same + // environment-object-timing reason. The URL/token fields are local drafts — `CloudSyncModel` only + // holds the SAVED values (in the Keychain via `CloudSyncSettings`), never the in-progress text. @StateObject private var cloudSync = CloudSyncModel() @State private var cloudSyncURLDraft = CloudSyncSettings.serverURL ?? "" @State private var cloudSyncTokenDraft = "" + // Phase 3.5 (zero-touch): a build carrying bundle-injected CLOUDSYNC_URL/CLOUDSYNC_TOKEN collapses + // the card to a one-line summary by default; this reveals the manual fields for a per-device + // override. Starts false even on a bundle-configured build — there is nothing to override yet. + @State private var cloudSyncShowOverride = false #endif // "Remove Apple Health imported data" (ah-delete #616): a destructive escape hatch that purges every // row stored under the "apple-health" source via DeviceRegistryStore.deleteAllData. Two-step (a @@ -342,68 +346,116 @@ struct DataSourcesView: View { #endif // OURA_CLOUD_IMPORT #if CLOUD_SYNC - /// Cloud Sync: pulls edit-journal rows YOU confirmed on your own noop-cloud server (a personal-fork - /// companion, not a shipped/supported service) and applies them to the local store, re-exporting - /// into Apple Health on iOS when the pull touched sleep/workouts/HR. `cloudSync.saveSettings(url: - /// token:)`/`pullNow(repo:)` take `repo` at call time — see the `@StateObject` declaration's note. + /// Cloud Sync: uploads this device's own `.noopbak` and pulls + applies edit-journal rows YOU + /// confirmed on your own noop-cloud server (a personal-fork companion, not a shipped/supported + /// service), re-exporting into Apple Health on iOS when the pull touched sleep/workouts/HR. + /// `cloudSync.syncNow(repo:)`/`saveSettings(url:token:)` take `repo`/args at call time — see the + /// `@StateObject` declaration's note. Phase 3.5 (zero-touch): a build carrying bundle-injected + /// `CLOUDSYNC_URL`/`CLOUDSYNC_TOKEN` with no Keychain override collapses to a one-line "Configured + /// from build" summary instead of the URL/token fields; "Override…" reveals them for a manual, + /// per-device server. private var cloudSyncCard: some View { card(title: String(localized: "Cloud Sync"), icon: "arrow.triangle.2.circlepath.circle", tint: StrandPalette.metricRose, - subtitle: String(localized: "Pull edits you confirmed on your own noop-cloud server and apply them here. A personal-fork lane — off unless you've set up your own server.")) { - VStack(alignment: .leading, spacing: NoopMetrics.space3) { - VStack(alignment: .leading, spacing: 6) { - Text("Server URL").strandOverline() - TextField("https://your-server.example.com", text: $cloudSyncURLDraft) - .textFieldStyle(.plain) - .font(StrandFont.body) - .foregroundStyle(StrandPalette.textPrimary) - .padding(.horizontal, 12) - .padding(.vertical, 9) - .background(StrandPalette.surfaceInset, in: RoundedRectangle(cornerRadius: 10, style: .continuous)) - .overlay(RoundedRectangle(cornerRadius: 10, style: .continuous) - .strokeBorder(StrandPalette.hairline, lineWidth: 1)) - .disableAutocorrection(true) - #if os(iOS) - .textInputAutocapitalization(.never) - .keyboardType(.URL) - #endif - .accessibilityLabel("noop-cloud server URL") - } - VStack(alignment: .leading, spacing: 6) { - Text("Token").strandOverline() - SecureField("Paste your server's read-write token", text: $cloudSyncTokenDraft) - .textFieldStyle(.plain) - .font(StrandFont.body) - .foregroundStyle(StrandPalette.textPrimary) - .padding(.horizontal, 12) - .padding(.vertical, 9) - .background(StrandPalette.surfaceInset, in: RoundedRectangle(cornerRadius: 10, style: .continuous)) - .overlay(RoundedRectangle(cornerRadius: 10, style: .continuous) - .strokeBorder(StrandPalette.hairline, lineWidth: 1)) - .accessibilityLabel("noop-cloud token") - } - HStack(spacing: NoopMetrics.space3) { - Button { - cloudSync.saveSettings(url: cloudSyncURLDraft, token: cloudSyncTokenDraft) - } label: { - Label("Save", systemImage: "checkmark.circle") - } - .buttonStyle(NoopButtonStyle(.secondary)) - .disabled(cloudSyncURLDraft.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty - || cloudSyncTokenDraft.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty) - - Button { - cloudSync.pullNow(repo: repo) - } label: { - Label(cloudSync.busy ? "Pulling…" : "Pull edits now", systemImage: "arrow.down.circle") - } - .buttonStyle(NoopButtonStyle(.primary)) - .disabled(!cloudSync.isConfigured || cloudSync.busy) - if cloudSync.busy { ProgressView().controlSize(.small) } - } - if let s = cloudSync.statusText { - Text(s).font(StrandFont.subhead).foregroundStyle(StrandPalette.textSecondary) + subtitle: String(localized: "Upload this device's data to your own noop-cloud server and pull back edits you confirmed elsewhere. A personal-fork lane — off unless you've set up your own server.")) { + if CloudSyncSettings.isBundleConfigured && !cloudSyncShowOverride { + bundleConfiguredCloudSync + } else { + manualCloudSyncFields + } + } + } + + /// Collapsed state for a build with bundle-injected credentials and no Keychain override: nothing + /// to fill in, just a confirmation + the primary action. "Override…" reveals `manualCloudSyncFields` + /// for a rare per-device custom server — the Keychain value it saves takes over on every future + /// launch (`CloudSyncSettings.effectiveURL` prefers the Keychain), and `isBundleConfigured` reads + /// false as soon as it exists, so the card never falls back to this collapsed view on its own + /// after that. + private var bundleConfiguredCloudSync: some View { + VStack(alignment: .leading, spacing: NoopMetrics.space3) { + HStack(spacing: 6) { + Image(systemName: "checkmark.circle.fill").foregroundStyle(StrandPalette.statusPositive) + Text("Configured from build").font(StrandFont.subhead).foregroundStyle(StrandPalette.textSecondary) + } + cloudSyncSyncRow + Button("Override…") { cloudSyncShowOverride = true } + .buttonStyle(.plain) + .font(StrandFont.caption) + .foregroundStyle(StrandPalette.textSecondary) + } + } + + /// Manual entry — the Server URL / Token fields + Save, also reachable via "Override…" on a + /// bundle-configured build. + private var manualCloudSyncFields: some View { + VStack(alignment: .leading, spacing: NoopMetrics.space3) { + VStack(alignment: .leading, spacing: 6) { + Text("Server URL").strandOverline() + TextField("https://your-server.example.com", text: $cloudSyncURLDraft) + .textFieldStyle(.plain) + .font(StrandFont.body) + .foregroundStyle(StrandPalette.textPrimary) + .padding(.horizontal, 12) + .padding(.vertical, 9) + .background(StrandPalette.surfaceInset, in: RoundedRectangle(cornerRadius: 10, style: .continuous)) + .overlay(RoundedRectangle(cornerRadius: 10, style: .continuous) + .strokeBorder(StrandPalette.hairline, lineWidth: 1)) + .disableAutocorrection(true) + #if os(iOS) + .textInputAutocapitalization(.never) + .keyboardType(.URL) + #endif + .accessibilityLabel("noop-cloud server URL") + } + VStack(alignment: .leading, spacing: 6) { + Text("Token").strandOverline() + SecureField("Paste your server's read-write token", text: $cloudSyncTokenDraft) + .textFieldStyle(.plain) + .font(StrandFont.body) + .foregroundStyle(StrandPalette.textPrimary) + .padding(.horizontal, 12) + .padding(.vertical, 9) + .background(StrandPalette.surfaceInset, in: RoundedRectangle(cornerRadius: 10, style: .continuous)) + .overlay(RoundedRectangle(cornerRadius: 10, style: .continuous) + .strokeBorder(StrandPalette.hairline, lineWidth: 1)) + .accessibilityLabel("noop-cloud token") + } + Button { + cloudSync.saveSettings(url: cloudSyncURLDraft, token: cloudSyncTokenDraft) + } label: { + Label("Save", systemImage: "checkmark.circle") + } + .buttonStyle(NoopButtonStyle(.secondary)) + .disabled(cloudSyncURLDraft.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty + || cloudSyncTokenDraft.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty) + cloudSyncSyncRow + if CloudSyncSettings.isBundleConfigured { + Button("Use build configuration instead") { cloudSyncShowOverride = false } + .buttonStyle(.plain) + .font(StrandFont.caption) + .foregroundStyle(StrandPalette.textSecondary) + } + } + } + + /// The "Sync now" button (+ busy spinner) and status line — identical whichever fields are + /// showing. Its own `VStack` (not a bare `Group`) so it's always a full, independent sibling in + /// whichever outer `VStack` hosts it, never flattened sideways into a parent `HStack`. + private var cloudSyncSyncRow: some View { + VStack(alignment: .leading, spacing: NoopMetrics.space3) { + HStack(spacing: NoopMetrics.space3) { + Button { + cloudSync.syncNow(repo: repo) + } label: { + Label(cloudSync.busy ? "Syncing…" : "Sync now", systemImage: "arrow.triangle.2.circlepath") } + .buttonStyle(NoopButtonStyle(.primary)) + .disabled(!cloudSync.isConfigured || cloudSync.busy) + if cloudSync.busy { ProgressView().controlSize(.small) } + } + if let s = cloudSync.statusText { + Text(s).font(StrandFont.subhead).foregroundStyle(StrandPalette.textSecondary) } } } diff --git a/StrandTests/CloudSyncClientTests.swift b/StrandTests/CloudSyncClientTests.swift index 81d9b16c76..e2e17743e2 100644 --- a/StrandTests/CloudSyncClientTests.swift +++ b/StrandTests/CloudSyncClientTests.swift @@ -151,5 +151,57 @@ final class CloudSyncSettingsTests: XCTestCase { CloudSyncSettings.serverURL = " " XCTAssertNil(CloudSyncSettings.serverURL) } + + // MARK: - Bundle fallback (Phase 3.5: zero-touch — bundle-injected CLOUDSYNC_URL/CLOUDSYNC_TOKEN) + + /// Pure precedence logic, driven with synthetic dicts — mirrors how `OuraCredentialsTests` tests + /// `OuraCredentials.from(_:)` directly rather than `fromBundle` (which reads the real + /// `Bundle.main.infoDictionary` and can't be redirected in a unit test). + func testEffectiveValuePrefersKeychainOverBundle() { + let v = CloudSyncSettings.effectiveValue(keychain: "from-keychain", infoKey: "CLOUDSYNC_URL", + info: ["CLOUDSYNC_URL": "from-bundle"]) + XCTAssertEqual(v, "from-keychain") + } + + func testEffectiveValueFallsBackToBundleWhenKeychainNil() { + let v = CloudSyncSettings.effectiveValue(keychain: nil, infoKey: "CLOUDSYNC_URL", + info: ["CLOUDSYNC_URL": "from-bundle"]) + XCTAssertEqual(v, "from-bundle") + } + + func testEffectiveValueTrimsBundleValueAndTreatsBlankAsAbsent() { + XCTAssertNil(CloudSyncSettings.effectiveValue(keychain: nil, infoKey: "CLOUDSYNC_URL", + info: ["CLOUDSYNC_URL": " "])) + XCTAssertEqual(CloudSyncSettings.effectiveValue(keychain: nil, infoKey: "CLOUDSYNC_URL", + info: ["CLOUDSYNC_URL": " https://x "]), "https://x") + } + + func testEffectiveValueNilWhenNeitherKeychainNorBundlePresent() { + XCTAssertNil(CloudSyncSettings.effectiveValue(keychain: nil, infoKey: "CLOUDSYNC_URL", info: [:])) + } + + func testEffectiveURLAndTokenReturnKeychainValuesWhenSet() { + CloudSyncSettings.serverURL = "https://from-keychain.example.com" + CloudSyncSettings.token = "keychain-token" + XCTAssertEqual(CloudSyncSettings.effectiveURL, "https://from-keychain.example.com") + XCTAssertEqual(CloudSyncSettings.effectiveToken, "keychain-token") + XCTAssertTrue(CloudSyncSettings.isConfigured) + } + + /// No Keychain override present: `isBundleConfigured` must be false regardless of whatever the + /// test runner's own bundle happens to carry for CLOUDSYNC_URL/CLOUDSYNC_TOKEN (never set in the + /// StrandTests bundle in practice), because a real Keychain override always wins the UI's + /// collapsed-vs-manual-fields decision. + func testIsBundleConfiguredFalseWhenKeychainOverridePresent() { + CloudSyncSettings.serverURL = "https://from-keychain.example.com" + CloudSyncSettings.token = "keychain-token" + XCTAssertFalse(CloudSyncSettings.isBundleConfigured) + } + + func testIsBundleConfiguredFalseWithNoKeychainAndNoBundleCreds() { + // Fresh state (clear()'d in setUp): no Keychain values, and the StrandTests bundle carries no + // CLOUDSYNC_* Info.plist keys, so this must read false, not silently true. + XCTAssertFalse(CloudSyncSettings.isBundleConfigured) + } } #endif // CLOUD_SYNC diff --git a/StrandTests/CloudSyncUploaderTests.swift b/StrandTests/CloudSyncUploaderTests.swift new file mode 100644 index 0000000000..cc88eed2d0 --- /dev/null +++ b/StrandTests/CloudSyncUploaderTests.swift @@ -0,0 +1,123 @@ +// Tests the CLOUD_SYNC-gated cloud sync uploader (Phase 3.5: zero-touch); compiled only when the flag +// is set (StrandTests shares the app's OuraConfig.xcconfig, so flag + creds arrive together). +#if CLOUD_SYNC +import XCTest +import WhoopStore +@testable import Strand + +final class CloudSyncUploaderTests: XCTestCase { + override func setUp() { super.setUp(); OuraURLProtocolStub.reset() } + + private let baseURL = URL(string: "https://cloud.example.com")! + + private func makeClient() -> CloudSyncClient { + CloudSyncClient(baseURL: baseURL, token: "tok-upload", session: OuraURLProtocolStub.session()) + } + + /// A fake exporter that writes canned bytes to `dest` without touching the real on-disk database or + /// `StorePaths.defaultDatabasePath()` — see `CloudSyncUploader.Exporter`'s doc comment for why the + /// real `DataBackup.writeBackup` can't run against a `WhoopStore.inMemory()` test store. `capture` + /// lets a test observe exactly which temp URL the uploader chose (to assert on its location and + /// that it's gone afterward). + private func fakeExporter(bytes: Data = Data("fake-noopbak-bytes".utf8), + capture: @escaping (URL) -> Void = { _ in }) -> CloudSyncUploader.Exporter { + { _, dest in + capture(dest) + try? bytes.write(to: dest) + return .exported(dest) + } + } + + // MARK: - Happy path + + func testUploadHappyPathSendsBearerAndOctetStreamAndParsesResponse() async throws { + OuraURLProtocolStub.queue = [.init(status: 200, + body: #"{"ok":true,"bytes":12345,"latestDay":"2026-07-11"}"#.data(using: .utf8)!)] + let store = try await WhoopStore.inMemory() + let client = makeClient() + var capturedDest: URL? + + let result = try await CloudSyncUploader.upload(store: store, client: client, + exporter: fakeExporter(capture: { capturedDest = $0 })) + + XCTAssertEqual(result.bytes, 12345) + XCTAssertEqual(result.latestDay, "2026-07-11") + + let req = try XCTUnwrap(OuraURLProtocolStub.requestedRequests.last) + XCTAssertEqual(req.httpMethod, "POST") + XCTAssertEqual(req.value(forHTTPHeaderField: "Authorization"), "Bearer tok-upload") + XCTAssertEqual(req.value(forHTTPHeaderField: "Content-Type"), "application/octet-stream") + XCTAssertTrue(req.url?.absoluteString.contains("/ingest") ?? false) + + // The temp file lived in Caches (never Documents) and was cleaned up afterward. + let dest = try XCTUnwrap(capturedDest) + let cachesDir = FileManager.default.urls(for: .cachesDirectory, in: .userDomainMask)[0] + XCTAssertTrue(dest.path.hasPrefix(cachesDir.path)) + XCTAssertFalse(FileManager.default.fileExists(atPath: dest.path)) + } + + // MARK: - Network failure + + func testUploadNon2xxThrowsBadResponse() async throws { + OuraURLProtocolStub.queue = [.init(status: 401, body: "nope".data(using: .utf8)!)] + let store = try await WhoopStore.inMemory() + let client = makeClient() + + do { + _ = try await CloudSyncUploader.upload(store: store, client: client, exporter: fakeExporter()) + XCTFail("expected badResponse to be thrown") + } catch { + XCTAssertEqual(error as? CloudSyncError, CloudSyncError.badResponse(401, "nope")) + } + } + + func testUploadCleansUpTempFileEvenOnNetworkFailure() async throws { + OuraURLProtocolStub.queue = [.init(status: 500, body: Data())] + let store = try await WhoopStore.inMemory() + let client = makeClient() + var capturedDest: URL? + + do { + _ = try await CloudSyncUploader.upload(store: store, client: client, + exporter: fakeExporter(capture: { capturedDest = $0 })) + XCTFail("expected badResponse to be thrown") + } catch { + // expected — asserted below via the cleaned-up temp file + } + let dest = try XCTUnwrap(capturedDest) + XCTAssertFalse(FileManager.default.fileExists(atPath: dest.path)) + } + + // MARK: - Export failure (never reaches the network) + + func testUploadExportFailurePropagatesAsTypedErrorAndNeverPosts() async throws { + let store = try await WhoopStore.inMemory() + let client = makeClient() + let failingExporter: CloudSyncUploader.Exporter = { _, _ in .failure("disk full") } + + do { + _ = try await CloudSyncUploader.upload(store: store, client: client, exporter: failingExporter) + XCTFail("expected exportFailed to be thrown") + } catch { + XCTAssertEqual(error as? CloudSyncUploadError, CloudSyncUploadError.exportFailed("disk full")) + } + XCTAssertTrue(OuraURLProtocolStub.requestedRequests.isEmpty) // never reached the network + } +} + +/// The 20h auto-sync gate, extracted as a pure `(lastRun, now) -> Bool` function so the threshold is +/// unit-testable without UserDefaults, a store, or a network. Mirrors `BackupSyncTests`' coverage of +/// `FolderBackup`'s pure day-gate helpers. +final class CloudSyncModelAutoSyncGateTests: XCTestCase { + func testIsAutoSyncDueNeverRunBefore() { + XCTAssertTrue(CloudSyncModel.isAutoSyncDue(lastRun: 0, now: 1_752_300_000)) + } + + func testIsAutoSyncDueRespectsTwentyHourThreshold() { + let now: TimeInterval = 1_752_300_000 + XCTAssertFalse(CloudSyncModel.isAutoSyncDue(lastRun: now - 19 * 3600, now: now)) // too soon + XCTAssertTrue(CloudSyncModel.isAutoSyncDue(lastRun: now - 20 * 3600, now: now)) // exactly due (>=) + XCTAssertTrue(CloudSyncModel.isAutoSyncDue(lastRun: now - 21 * 3600, now: now)) // overdue + } +} +#endif // CLOUD_SYNC diff --git a/StrandTests/OuraURLProtocolStub.swift b/StrandTests/OuraURLProtocolStub.swift index 8bd46945d1..0db122b5c7 100644 --- a/StrandTests/OuraURLProtocolStub.swift +++ b/StrandTests/OuraURLProtocolStub.swift @@ -8,13 +8,19 @@ final class OuraURLProtocolStub: URLProtocol { nonisolated(unsafe) static var queue: [Stubbed] = [] /// Captured request URLs, in order, for assertions. nonisolated(unsafe) static var requestedURLs: [URL] = [] + /// Captured full requests (headers included), in order — for assertions that need more than the + /// URL, e.g. a bearer `Authorization` header or a `Content-Type` (CloudSyncUploaderTests). Upload + /// tasks built via `URLSession.upload(for:fromFile:)` still carry their headers on `request` even + /// though the body itself is streamed from disk rather than exposed here. + nonisolated(unsafe) static var requestedRequests: [URLRequest] = [] - static func reset() { queue = []; requestedURLs = [] } + static func reset() { queue = []; requestedURLs = []; requestedRequests = [] } override class func canInit(with request: URLRequest) -> Bool { true } override class func canonicalRequest(for request: URLRequest) -> URLRequest { request } override func startLoading() { if let u = request.url { Self.requestedURLs.append(u) } + Self.requestedRequests.append(request) let s = Self.queue.isEmpty ? Stubbed(status: 500, body: Data()) : Self.queue.removeFirst() let resp = HTTPURLResponse(url: request.url!, statusCode: s.status, httpVersion: nil, headerFields: nil)! client?.urlProtocol(self, didReceive: resp, cacheStoragePolicy: .notAllowed) diff --git a/StrandiOS/App/RootTabView.swift b/StrandiOS/App/RootTabView.swift index cf0e9e718e..b5766a51cb 100644 --- a/StrandiOS/App/RootTabView.swift +++ b/StrandiOS/App/RootTabView.swift @@ -119,6 +119,11 @@ struct RootTabView: View { Task.detached(priority: .utility) { await FolderBackup.catchUpIfDue(checkpoint: { await backupRepo.checkpointForBackup() }) } + #if CLOUD_SYNC + // Cloud Sync: zero-touch on-launch catch-up (see RootView's identical wiring). No detached + // wrapper needed — `autoSyncIfDue` spawns its own `Task` internally and returns immediately. + CloudSyncModel().autoSyncIfDue(repo: repo) + #endif } // Quick-action sheet presents with the calm easing (~0.42s) per the README sheet spec — // the easing is applied where `quickAction` is set (see `presentQuickAction`), keeping the diff --git a/StrandiOS/Resources/Info.plist b/StrandiOS/Resources/Info.plist index b9bb28c9bb..707181767f 100644 --- a/StrandiOS/Resources/Info.plist +++ b/StrandiOS/Resources/Info.plist @@ -48,6 +48,10 @@ CFBundleVersion $(CURRENT_PROJECT_VERSION) + CLOUDSYNC_TOKEN + $(CLOUDSYNC_TOKEN) + CLOUDSYNC_URL + $(CLOUDSYNC_URL) LSApplicationCategoryType public.app-category.healthcare-fitness LSSupportsOpeningDocumentsInPlace diff --git a/project.yml b/project.yml index 78c29e356d..316f2e7653 100644 --- a/project.yml +++ b/project.yml @@ -105,6 +105,11 @@ targets: OURA_CLIENT_ID: $(OURA_CLIENT_ID) OURA_CLIENT_SECRET: $(OURA_CLIENT_SECRET) OURA_REDIRECT_URI: $(OURA_REDIRECT_URI) + # Cloud Sync bundle credentials (Phase 3.5 zero-touch), same untracked-xcconfig chain as the + # OURA_* keys above; read at runtime via CloudSyncSettings.effectiveURL/effectiveToken. Absent/ + # blank values leave the lane on manual Keychain configuration only. + CLOUDSYNC_URL: $(CLOUDSYNC_URL) + CLOUDSYNC_TOKEN: $(CLOUDSYNC_TOKEN) # Registers the noop:// URL scheme so ASWebAuthenticationSession's OAuth redirect # (noop://oura/callback) reaches OuraOAuthProvider instead of bouncing off the OS. CFBundleURLTypes: @@ -273,6 +278,11 @@ targets: OURA_CLIENT_ID: $(OURA_CLIENT_ID) OURA_CLIENT_SECRET: $(OURA_CLIENT_SECRET) OURA_REDIRECT_URI: $(OURA_REDIRECT_URI) + # Cloud Sync bundle credentials (Phase 3.5 zero-touch), same untracked-xcconfig chain as the + # OURA_* keys above; read at runtime via CloudSyncSettings.effectiveURL/effectiveToken. Absent/ + # blank values leave the lane on manual Keychain configuration only. + CLOUDSYNC_URL: $(CLOUDSYNC_URL) + CLOUDSYNC_TOKEN: $(CLOUDSYNC_TOKEN) # Registers the noop:// URL scheme so ASWebAuthenticationSession's OAuth redirect # (noop://oura/callback) reaches OuraOAuthProvider instead of bouncing off the OS. CFBundleURLTypes: From 8dc6aafc2cd7788d3935f0e4b850970a9417821c Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 18:50:17 -0700 Subject: [PATCH 057/171] fix(cloudsync): cross-instance sync gate, persisted auto-sync status, bundle-independent tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three Important review findings on the Phase 3.5 zero-touch cloud sync lane: - Reentrancy: autoSyncIfDue runs on throwaway CloudSyncModel instances (RootView/RootTabView launch .task) while DataSourcesView holds its own @StateObject, so the per-instance `busy` flag couldn't stop a manual "Sync now" from racing the on-launch auto-sync. Added a process-wide `CloudSyncGate` actor; syncNow and autoSyncIfDue now each spawn their own Task into a shared runGatedSync helper (kept in one place rather than duplicated, since nesting the guard — autoSyncIfDue acquiring it and then calling syncNow — would self-deadlock). - Visibility: auto-sync had no UI trace when it ran on a throwaway instance. performSync now persists " · " to UserDefaults on every completion (success or failure); the Data Sources card shows it as "Last sync: …" whenever its own statusText is nil. - Tests: 4 CloudSyncSettingsTests assertions (3 test methods) failed now that a personal OuraSecrets.xcconfig carries real CLOUDSYNC_URL/CLOUDSYNC_TOKEN values reachable via Bundle.main. Added isConfigured(info:) / isBundleConfigured(info:) pure-seam variants (mirroring the existing effectiveValue pattern) and rewrote the three tests against synthetic info dicts instead of the live bundle. Fold-in: effectiveURL now runs the merged value through the same URLComponents scheme+host check saveSettings already applies, so a malformed bundle CLOUDSYNC_URL reads as absent instead of reaching CloudSyncClient's force-unwrapped request URL. isConfigured(info:) was updated to share that check too, closing a consistency gap where it could read true while effectiveURL read nil for the same malformed value. --- Strand/CloudSync/CloudSyncModel.swift | 164 +++++++++++++++++------ Strand/CloudSync/CloudSyncSettings.swift | 51 ++++++- Strand/Screens/DataSourcesView.swift | 7 + StrandTests/CloudSyncClientTests.swift | 57 ++++++-- 4 files changed, 227 insertions(+), 52 deletions(-) diff --git a/Strand/CloudSync/CloudSyncModel.swift b/Strand/CloudSync/CloudSyncModel.swift index 7cd31af8b1..ad10d463d2 100644 --- a/Strand/CloudSync/CloudSyncModel.swift +++ b/Strand/CloudSync/CloudSyncModel.swift @@ -8,6 +8,32 @@ import WhoopStore import UIKit #endif +/// Process-wide reentrancy guard for the cloud-sync lane. `busy` on a `CloudSyncModel` INSTANCE only +/// guards within that one instance — but `autoSyncIfDue` runs on a throwaway `CloudSyncModel()` +/// created fresh in `RootView`/`RootTabView`'s launch `.task` (a new instance every launch), while +/// `DataSourcesView` holds its OWN long-lived `@StateObject` instance. Two different instances means +/// two different `busy` flags: a manual "Sync now" tap on the Data Sources card can't see that the +/// launch-time auto-sync (on its own throwaway instance) is already mid-upload, and vice versa — both +/// would happily run `CloudSyncUploader.upload` concurrently against the same on-disk store and the +/// same server. An `actor` serializes access to its own state automatically, making it the natural +/// process-wide singleton shape here — `shared` is the one gate for the whole process. +actor CloudSyncGate { + static let shared = CloudSyncGate() + private var inFlight = false + + /// Claims the gate if it's free. Returns false (and claims nothing) if a sync is already running. + func begin() -> Bool { + if inFlight { return false } + inFlight = true + return true + } + + /// Releases the gate. Must be called exactly once for every `begin()` that returned true. + func end() { + inFlight = false + } +} + /// Drives the Data Sources "Cloud Sync" card: save the noop-cloud server URL + token → pull confirmed /// edits → apply them locally → (iOS) re-export whatever changed into Apple Health. @MainActor so every /// `@Published` mutation is main-thread; the network/apply/write-back work hops off-main. @@ -84,54 +110,110 @@ final class CloudSyncModel: ObservableObject { /// edits — the two halves of "zero-touch" in one round trip: this device's own data reaches the /// server, and any edits confirmed elsewhere reach this device. Stamps /// `UserDefaults["cloudsync.lastAutoSync"]` on SUCCESS ONLY, so a failed attempt (server - /// unreachable, network down) never blocks the next 20h-gated retry. Used by both the - /// user-initiated "Sync now" button and `autoSyncIfDue`'s on-launch catch-up — a manual tap also - /// resets the on-launch catch-up clock, so it doesn't immediately re-fire on the next launch. + /// unreachable, network down) never blocks the next 20h-gated retry. Used by the user-initiated + /// "Sync now" button; `autoSyncIfDue`'s on-launch catch-up runs the identical `performSync` body + /// under its own gated `Task` rather than calling this (see `runGatedSync`'s doc comment) — but a + /// manual tap still resets the SAME on-launch catch-up clock (`Self.lastAutoSyncKey`, written + /// inside `performSync`), so it doesn't immediately re-fire on the next launch. func syncNow(repo: Repository) { guard let urlString = CloudSyncSettings.effectiveURL, let token = CloudSyncSettings.effectiveToken, let url = URL(string: urlString) else { statusText = "Add your noop-cloud server URL and token first." return } + Task { await runGatedSync(repo: repo, url: url, token: token) } + } + + /// Acquires the process-wide `CloudSyncGate`, runs `performSync`, and always releases it — + /// written out as one linear sequence rather than `defer` (which can't `await`). This is safe + /// specifically because `performSync` never throws and never returns early once it starts: every + /// internal failure is caught inside it and folded into its own status-line string (see its + /// body), so falling off the end of this function really does mean "released on every exit path." + /// Shared by `syncNow` and `autoSyncIfDue` — each spawns its own `Task` calling this — rather than + /// having `autoSyncIfDue` acquire the gate and then call `syncNow` (which would try to acquire it + /// again and immediately see it as already held by the very call chain it's part of, bailing with + /// "Sync already in progress" on every auto-sync). + private func runGatedSync(repo: Repository, url: URL, token: String) async { + guard await CloudSyncGate.shared.begin() else { + statusText = "Sync already in progress" + return + } + await performSync(repo: repo, url: url, token: token) + await CloudSyncGate.shared.end() + } + + /// The actual upload + pull work, gated by `runGatedSync` — never called directly. Byte-identical + /// to `syncNow`'s pre-gate Task body, plus persisting the outcome for the Data Sources card to + /// read back (Finding 2: `CloudSyncModel.persistLastStatus`, below). + private func performSync(repo: Repository, url: URL, token: String) async { setBusy(true); statusText = "Uploading…" - Task { - var line: String - var succeeded = false - if let store = await repo.storeHandle() { - let client = CloudSyncClient(baseURL: url, token: token) - do { - let uploaded = try await CloudSyncUploader.upload(store: store, client: client) - let mb = Double(uploaded.bytes) / 1_048_576.0 - statusText = String(format: "Uploaded %.1f MB · pulling edits…", mb) + var line: String + var succeeded = false + if let store = await repo.storeHandle() { + let client = CloudSyncClient(baseURL: url, token: token) + do { + let uploaded = try await CloudSyncUploader.upload(store: store, client: client) + let mb = Double(uploaded.bytes) / 1_048_576.0 + statusText = String(format: "Uploaded %.1f MB · pulling edits…", mb) - let summary = try await CloudSyncCoordinator.pull(store: store, client: client) - var parts = [String(format: "Uploaded %.1f MB", mb), - "Applied \(summary.applied)", "skipped \(summary.skipped)"] - if summary.needsAttention > 0 { parts.append("\(summary.needsAttention) attention") } - line = parts.joined(separator: " · ") - succeeded = true - #if os(iOS) && canImport(HealthKit) - line += await reExportToAppleHealthIfNeeded(repo: repo, store: store, summary: summary) - #endif - } catch { - line = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription - } - } else { - line = "No local store." + let summary = try await CloudSyncCoordinator.pull(store: store, client: client) + var parts = [String(format: "Uploaded %.1f MB", mb), + "Applied \(summary.applied)", "skipped \(summary.skipped)"] + if summary.needsAttention > 0 { parts.append("\(summary.needsAttention) attention") } + line = parts.joined(separator: " · ") + succeeded = true + #if os(iOS) && canImport(HealthKit) + line += await reExportToAppleHealthIfNeeded(repo: repo, store: store, summary: summary) + #endif + } catch { + line = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription } - statusText = line - if succeeded { - UserDefaults.standard.set(Date().timeIntervalSince1970, forKey: Self.lastAutoSyncKey) - } - await repo.refresh() - setBusy(false) + } else { + line = "No local store." + } + statusText = line + if succeeded { + UserDefaults.standard.set(Date().timeIntervalSince1970, forKey: Self.lastAutoSyncKey) } + Self.persistLastStatus(line) + await repo.refresh() + setBusy(false) } /// UserDefaults key for the last successful `syncNow` (manual or automatic) — read by - /// `autoSyncIfDue`, written by `syncNow` on success only. + /// `autoSyncIfDue`, written by `performSync` on success only. private static let lastAutoSyncKey = "cloudsync.lastAutoSync" + /// UserDefaults key for the outcome of the last sync attempt, success OR failure, persisted by + /// `performSync` on EVERY completion (Finding 2). Read directly by the Data Sources card so an + /// auto-sync that ran on a throwaway `CloudSyncModel` instance (see `CloudSyncGate`'s doc comment) + /// is still visible on the card's own instance, whose `statusText` never saw it happen. + private static let lastStatusKey = "cloudsync.lastStatus" + + /// The last sync's outcome, formatted for display (" · "), or nil if no sync + /// has ever completed on this device. A plain synchronous UserDefaults read — deliberately not + /// reactive/`@Published`: `performSync` always ends with a `repo.refresh()`, which already + /// re-renders the screens that observe `repo`, so the next render picks up a fresh value with no + /// extra observation machinery. + static var lastPersistedStatus: String? { + UserDefaults.standard.string(forKey: lastStatusKey) + } + + private static let shortTimeFormatter: DateFormatter = { + let f = DateFormatter() + f.dateStyle = .none + f.timeStyle = .short + return f + }() + + /// Persist `line` (the human-readable outcome `performSync` just computed) plus a short local + /// time, so a later launch's Data Sources card can show "Last sync: …" even when ITS OWN + /// `statusText` is nil — e.g. right after launch, before that screen's own `CloudSyncModel` + /// `@StateObject` has run a sync itself. + private static func persistLastStatus(_ line: String) { + UserDefaults.standard.set("\(line) · \(shortTimeFormatter.string(from: Date()))", forKey: lastStatusKey) + } + /// Pure 20h gate, `(lastRun, now) -> Bool`, so the threshold is unit-testable without UserDefaults, /// a store, or a network (see `CloudSyncModelAutoSyncGateTests`). `lastRun == 0` (never synced) is /// always due — a fresh install syncs on its very first launch, matching "zero manual taps". @@ -145,16 +227,20 @@ final class CloudSyncModel: ObservableObject { /// On-launch daily catch-up (Phase 3.5: zero-touch): silently no-ops when the lane isn't configured /// — `CloudSyncSettings.isConfigured` already covers both a bundle-injected build and a manual - /// Keychain save — otherwise fires `syncNow` when it's been >20h since the last success. Call-time - /// `repo:` parameter, same reason as every other method here (see the `@StateObject` declaration's - /// note). A plain (non-`async`) MainActor call, mirroring `pullNow`/`syncNow`'s own shape: it - /// spawns its OWN `Task` internally and returns immediately, so a caller on the launch-critical path - /// (`RootView`/`RootTabView`'s `.task`) is never blocked waiting on the network. + /// Keychain save — otherwise runs the same gated sync as `syncNow` when it's been >20h since the + /// last success. Call-time `repo:` parameter, same reason as every other method here (see the + /// `@StateObject` declaration's note). A plain (non-`async`) MainActor call, mirroring + /// `pullNow`/`syncNow`'s own shape: it spawns its OWN `Task` internally and returns immediately, so + /// a caller on the launch-critical path (`RootView`/`RootTabView`'s `.task`) is never blocked + /// waiting on the network. Deliberately does NOT call `syncNow(repo:)` — see `runGatedSync`'s doc + /// comment for why that would self-deadlock the gate. func autoSyncIfDue(repo: Repository) { guard CloudSyncSettings.isConfigured else { return } let last = UserDefaults.standard.double(forKey: Self.lastAutoSyncKey) guard Self.isAutoSyncDue(lastRun: last, now: Date().timeIntervalSince1970) else { return } - syncNow(repo: repo) + guard let urlString = CloudSyncSettings.effectiveURL, let token = CloudSyncSettings.effectiveToken, + let url = URL(string: urlString) else { return } + Task { await runGatedSync(repo: repo, url: url, token: token) } } #if os(iOS) && canImport(HealthKit) diff --git a/Strand/CloudSync/CloudSyncSettings.swift b/Strand/CloudSync/CloudSyncSettings.swift index b1d34cbbbd..4eeaca163e 100644 --- a/Strand/CloudSync/CloudSyncSettings.swift +++ b/Strand/CloudSync/CloudSyncSettings.swift @@ -37,7 +37,26 @@ enum CloudSyncSettings { /// True once both an EFFECTIVE server URL and token are available — a Keychain value, a /// bundle-injected one, or a mix of both. Phase 3.5 (zero-touch): a bundle-only install counts as /// configured with no Keychain write ever happening. - static var isConfigured: Bool { effectiveURL != nil && effectiveToken != nil } + static var isConfigured: Bool { isConfigured(info: Bundle.main.infoDictionary ?? [:]) } + + /// Testable variant of `isConfigured`: identical Keychain-wins-over-bundle precedence, but takes + /// an explicit info dict instead of reading live `Bundle.main`. Keychain state never needed a seam + /// — a test can already set/clear it directly via `serverURL`/`token`/`clear()` — only the bundle + /// side was unswappable. Pass `info: [:]` to assert "Keychain-only" behaviour deterministically, + /// regardless of whatever CLOUDSYNC_URL/CLOUDSYNC_TOKEN this test PROCESS's real bundle happens to + /// carry: once a personal `OuraSecrets.xcconfig` carries real cloud-sync credentials, the live, + /// no-arg `isConfigured` reads true even with the Keychain cleared, which the original + /// bundle-unaware `CloudSyncSettingsTests` didn't anticipate (see that file's history). + /// + /// Also runs the URL through `isValidServerURL`, same as `effectiveURL` — this must stay in sync + /// with `effectiveURL`'s own validation (both derive the URL half from the same `effectiveValue` + /// call), or `isConfigured` could read true for a malformed bundle URL that `effectiveURL` itself + /// then reads as nil, enabling a "Sync now" button that immediately fails. + static func isConfigured(info: [String: Any]) -> Bool { + guard let url = effectiveValue(keychain: serverURL, infoKey: "CLOUDSYNC_URL", info: info), + isValidServerURL(url) else { return false } + return effectiveValue(keychain: token, infoKey: "CLOUDSYNC_TOKEN", info: info) != nil + } /// Remove both stored values. static func clear() { @@ -69,9 +88,17 @@ enum CloudSyncSettings { /// The noop-cloud server URL a manual Save in the Data Sources card wrote to the Keychain, or the /// build-injected `CLOUDSYNC_URL` (from the untracked `OuraSecrets.xcconfig` → project.yml → /// Info.plist chain — see `OuraCredentials.swift`'s doc comment for the identical pattern) when no - /// Keychain override exists. nil disables the whole cloud-sync lane. + /// Keychain override exists. nil disables the whole cloud-sync lane. Runs the merged value through + /// `isValidServerURL` before returning it — a Keychain value already passed that exact check at + /// `CloudSyncModel.saveSettings` time (so this is a no-op there), but a bundle-injected URL from + /// Info.plist is never validated anywhere else, so a malformed `CLOUDSYNC_URL` xcconfig value is + /// caught here instead of reaching `CloudSyncClient.url(path:query:)`'s force-unwrapped + /// `URLComponents`-built request URL and crashing at the first sync. static var effectiveURL: String? { - effectiveValue(keychain: serverURL, infoKey: "CLOUDSYNC_URL", info: Bundle.main.infoDictionary ?? [:]) + guard let raw = effectiveValue(keychain: serverURL, infoKey: "CLOUDSYNC_URL", + info: Bundle.main.infoDictionary ?? [:]), + isValidServerURL(raw) else { return nil } + return raw } /// The read-write token, same Keychain-wins-over-bundle precedence as `effectiveURL`. @@ -79,14 +106,28 @@ enum CloudSyncSettings { effectiveValue(keychain: token, infoKey: "CLOUDSYNC_TOKEN", info: Bundle.main.infoDictionary ?? [:]) } + /// True when `s` parses as an absolute URL with a scheme and host — the same shape check + /// `CloudSyncModel.saveSettings` runs before a manually-entered URL is ever written to the + /// Keychain. Internal (not `private`) so it's directly unit-testable — pure, no `Bundle`/Keychain + /// dependency at all. + static func isValidServerURL(_ s: String) -> Bool { + guard let comps = URLComponents(string: s), let scheme = comps.scheme, !scheme.isEmpty, + let host = comps.host, !host.isEmpty else { return false } + return true + } + /// True when the lane is configured ENTIRELY from the bundle-injected build credentials, with no /// Keychain override at all. Drives the Data Sources card: bundle-configured shows a one-line /// "Configured from build" note instead of the URL/token fields; any Keychain value (even a /// partial one) falls through to the manual fields so a user's own in-progress edit is never /// silently hidden behind the collapsed summary. - static var isBundleConfigured: Bool { + static var isBundleConfigured: Bool { isBundleConfigured(info: Bundle.main.infoDictionary ?? [:]) } + + /// Testable variant of `isBundleConfigured` — same shape as `isConfigured(info:)` above and for + /// the same reason (see its doc comment): pass `info: [:]` to assert "no bundle creds" + /// deterministically regardless of this test process's real `Bundle.main`. + static func isBundleConfigured(info: [String: Any]) -> Bool { guard serverURL == nil, token == nil else { return false } - let info = Bundle.main.infoDictionary ?? [:] return bundleValue("CLOUDSYNC_URL", from: info) != nil && bundleValue("CLOUDSYNC_TOKEN", from: info) != nil } diff --git a/Strand/Screens/DataSourcesView.swift b/Strand/Screens/DataSourcesView.swift index 1117b75345..b6bafef1db 100644 --- a/Strand/Screens/DataSourcesView.swift +++ b/Strand/Screens/DataSourcesView.swift @@ -442,6 +442,11 @@ struct DataSourcesView: View { /// The "Sync now" button (+ busy spinner) and status line — identical whichever fields are /// showing. Its own `VStack` (not a bare `Group`) so it's always a full, independent sibling in /// whichever outer `VStack` hosts it, never flattened sideways into a parent `HStack`. + /// + /// Falls back to the PERSISTED status (Finding 2) when this card's own `cloudSync.statusText` is + /// nil — e.g. right after launch, when the zero-touch auto-sync ran on its own throwaway + /// `CloudSyncModel` instance (see `CloudSyncGate`'s doc comment) and this screen's `@StateObject` + /// never saw it happen. A plain `UserDefaults` read on render, no cross-instance observation. private var cloudSyncSyncRow: some View { VStack(alignment: .leading, spacing: NoopMetrics.space3) { HStack(spacing: NoopMetrics.space3) { @@ -456,6 +461,8 @@ struct DataSourcesView: View { } if let s = cloudSync.statusText { Text(s).font(StrandFont.subhead).foregroundStyle(StrandPalette.textSecondary) + } else if let last = CloudSyncModel.lastPersistedStatus { + Text("Last sync: \(last)").font(StrandFont.subhead).foregroundStyle(StrandPalette.textSecondary) } } } diff --git a/StrandTests/CloudSyncClientTests.swift b/StrandTests/CloudSyncClientTests.swift index e2e17743e2..fc4a2aed52 100644 --- a/StrandTests/CloudSyncClientTests.swift +++ b/StrandTests/CloudSyncClientTests.swift @@ -119,8 +119,14 @@ final class CloudSyncSettingsTests: XCTestCase { override func setUp() { super.setUp(); CloudSyncSettings.clear() } override func tearDown() { CloudSyncSettings.clear(); super.tearDown() } + /// Uses `isConfigured(info:)` — the pure, dict-driven seam — with an explicit EMPTY info dict + /// rather than the live, no-arg `isConfigured` (which falls back to `Bundle.main`). This machine's + /// `OuraSecrets.xcconfig` carries real CLOUDSYNC_URL/CLOUDSYNC_TOKEN values, so the live property + /// would read true even with the Keychain cleared, defeating the "Keychain-only" round trip this + /// test is actually checking. `serverURL`/`token`/`clear()` (the Keychain side) don't need a seam + /// — they're already fully controllable in a test. func testSaveLoadClearRoundTrip() { - XCTAssertFalse(CloudSyncSettings.isConfigured) + XCTAssertFalse(CloudSyncSettings.isConfigured(info: [:])) XCTAssertNil(CloudSyncSettings.serverURL) XCTAssertNil(CloudSyncSettings.token) @@ -128,20 +134,22 @@ final class CloudSyncSettingsTests: XCTestCase { CloudSyncSettings.token = "rw-secret" XCTAssertEqual(CloudSyncSettings.serverURL, "https://vk-noop-cloud.fly.dev") XCTAssertEqual(CloudSyncSettings.token, "rw-secret") - XCTAssertTrue(CloudSyncSettings.isConfigured) + XCTAssertTrue(CloudSyncSettings.isConfigured(info: [:])) CloudSyncSettings.clear() XCTAssertNil(CloudSyncSettings.serverURL) XCTAssertNil(CloudSyncSettings.token) - XCTAssertFalse(CloudSyncSettings.isConfigured) + XCTAssertFalse(CloudSyncSettings.isConfigured(info: [:])) } + /// Same pure-seam reasoning as `testSaveLoadClearRoundTrip` above: `info: [:]` so a real bundle + /// CLOUDSYNC_TOKEN on this machine can't make the URL-only state read as already-configured. func testIsConfiguredRequiresBothValues() { CloudSyncSettings.serverURL = "https://vk-noop-cloud.fly.dev" - XCTAssertFalse(CloudSyncSettings.isConfigured) + XCTAssertFalse(CloudSyncSettings.isConfigured(info: [:])) CloudSyncSettings.token = "rw-secret" - XCTAssertTrue(CloudSyncSettings.isConfigured) + XCTAssertTrue(CloudSyncSettings.isConfigured(info: [:])) } func testEmptyStringClearsValue() { @@ -198,10 +206,43 @@ final class CloudSyncSettingsTests: XCTestCase { XCTAssertFalse(CloudSyncSettings.isBundleConfigured) } + /// Uses `isBundleConfigured(info:)` with an explicit EMPTY info dict — the pure seam — rather than + /// the live, no-arg `isBundleConfigured`. This machine's `OuraSecrets.xcconfig` carries real + /// CLOUDSYNC_URL/CLOUDSYNC_TOKEN values, which reach `Bundle.main.infoDictionary` at test-run + /// time, so the live property now genuinely reads true here — asserting against it directly would + /// make this test fail specifically BECAUSE the zero-touch bundle credentials are configured + /// correctly. `info: [:]` tests the "no bundle creds" branch deterministically instead. func testIsBundleConfiguredFalseWithNoKeychainAndNoBundleCreds() { - // Fresh state (clear()'d in setUp): no Keychain values, and the StrandTests bundle carries no - // CLOUDSYNC_* Info.plist keys, so this must read false, not silently true. - XCTAssertFalse(CloudSyncSettings.isBundleConfigured) + // Fresh state (clear()'d in setUp): no Keychain values, and a synthetic empty info dict, so + // this must read false, not silently true. + XCTAssertFalse(CloudSyncSettings.isBundleConfigured(info: [:])) + } + + // MARK: - Bundle URL validation (fold-in: a malformed CLOUDSYNC_URL is treated as absent) + + /// Pure, no `Bundle`/Keychain dependency — the same shape check `CloudSyncModel.saveSettings` + /// runs before a manually-entered URL is ever written to the Keychain. + func testIsValidServerURLAcceptsAbsoluteURLWithSchemeAndHost() { + XCTAssertTrue(CloudSyncSettings.isValidServerURL("https://vk-noop-cloud.fly.dev")) + XCTAssertTrue(CloudSyncSettings.isValidServerURL("http://localhost:8080")) + } + + func testIsValidServerURLRejectsMissingSchemeOrHost() { + XCTAssertFalse(CloudSyncSettings.isValidServerURL("not a url")) + XCTAssertFalse(CloudSyncSettings.isValidServerURL("")) + XCTAssertFalse(CloudSyncSettings.isValidServerURL("justapath")) + XCTAssertFalse(CloudSyncSettings.isValidServerURL("vk-noop-cloud.fly.dev")) // no scheme + XCTAssertFalse(CloudSyncSettings.isValidServerURL("https://")) // no host + } + + /// A malformed bundle URL must be treated as absent by BOTH `effectiveURL` and `isConfigured` — + /// not just the former. `isConfigured(info:)` derives its URL half from the same `effectiveValue` + /// call as `effectiveURL`, so this pins the two together: a build never ends up with the "Sync + /// now" button enabled (`isConfigured` true) while `effectiveURL` itself reads nil and the button + /// would immediately fail with "Add your noop-cloud server URL and token first." + func testIsConfiguredFalseWhenBundleURLIsMalformedEvenWithATokenPresent() { + XCTAssertFalse(CloudSyncSettings.isConfigured(info: ["CLOUDSYNC_URL": "not a url", + "CLOUDSYNC_TOKEN": "some-token"])) } } #endif // CLOUD_SYNC From b9f531a5c3433e03bbfc143fd10dee2d79ecae6d Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 19:27:08 -0700 Subject: [PATCH 058/171] fix(cloudsync): refuse empty-store uploads, no auto-sync under XCTest, idle-timer pin --- .../Sources/WhoopStore/MetricsCache.swift | 16 ++++ .../WhoopStoreTests/MetricsCacheTests.swift | 18 +++++ Strand/CloudSync/CloudSyncModel.swift | 19 +++++ Strand/CloudSync/CloudSyncUploader.swift | 17 ++++ StrandTests/CloudSyncUploaderTests.swift | 81 ++++++++++++++++++- 5 files changed, 147 insertions(+), 4 deletions(-) diff --git a/Packages/WhoopStore/Sources/WhoopStore/MetricsCache.swift b/Packages/WhoopStore/Sources/WhoopStore/MetricsCache.swift index 2fe0c67bc6..d6a5d9f025 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/MetricsCache.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/MetricsCache.swift @@ -476,4 +476,20 @@ extension WhoopStore { } } } + + /// Cheap existence check across EVERY device/source (no `deviceId` filter, unlike the reads + /// above): does the store have any cached daily-metric row at all? Every ingest path that puts + /// real data into this store — `WhoopImporter`, `OuraSyncWriter`, `AppleHealthImport`, + /// `XiaomiImporter`, `ShortcutHealthImport`, `HealthKitBridge`, and the on-device + /// `IntelligenceEngine` recompute that runs after a live BLE offload — upserts into `dailyMetric` + /// as part of that ingest. So a genuinely fresh/never-populated store (no BLE pairing, no import + /// ever run — the shape of the empty macOS test-host database that got auto-uploaded and replaced + /// the production mirror) has zero rows here. Used by `CloudSyncUploader.upload` to refuse + /// exporting such a store. `SELECT 1 … LIMIT 1` rather than `COUNT(*)`: this only needs existence, + /// so SQLite can stop at the first matching row instead of scanning/counting the whole table. + public func hasAnyDailyMetrics() async throws -> Bool { + try syncRead { db in + try Int.fetchOne(db, sql: "SELECT 1 FROM dailyMetric LIMIT 1") != nil + } + } } diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/MetricsCacheTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/MetricsCacheTests.swift index 2f41d3062c..c8c72d02e7 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/MetricsCacheTests.swift +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/MetricsCacheTests.swift @@ -364,6 +364,24 @@ final class MetricsCacheTests: XCTestCase { XCTAssertEqual(rows.map { $0.day }, ["2026-05-20"]) } + // MARK: - empty-store guard (incident: an empty test-host DB must never look "has data") + + func testHasAnyDailyMetricsFalseWhenEmptyTrueAfterUpsert() async throws { + let store = try await WhoopStore.inMemory() + // A freshly-created store (no BLE pairing, no import ever run) — the exact shape of the + // production incident's empty macOS test-host database — must read false. + let beforeAnyData = try await store.hasAnyDailyMetrics() + XCTAssertFalse(beforeAnyData) + + try await store.upsertDailyMetrics([ + DailyMetric(day: "2026-05-23", totalSleepMin: nil, efficiency: nil, deepMin: nil, remMin: nil, + lightMin: nil, disturbances: nil, restingHr: nil, avgHrv: nil, recovery: nil, + strain: nil, exerciseCount: nil), + ], deviceId: "devA") + let afterUpsert = try await store.hasAnyDailyMetrics() + XCTAssertTrue(afterUpsert) + } + // MARK: - windowed computed-daily delete (#277 local-day re-bucketing migration) func testDeleteDailyMetricsInRangeKeepsImportedAndOutOfRange() async throws { diff --git a/Strand/CloudSync/CloudSyncModel.swift b/Strand/CloudSync/CloudSyncModel.swift index ad10d463d2..1f98aaebc7 100644 --- a/Strand/CloudSync/CloudSyncModel.swift +++ b/Strand/CloudSync/CloudSyncModel.swift @@ -225,6 +225,24 @@ final class CloudSyncModel: ObservableObject { now - lastRun >= intervalS } + /// True when this process is running inside an XCTest test runner. Guards `autoSyncIfDue` against + /// the incident where the macOS TEST HOST — `StrandTests` runs inside the full `Staging.app` via + /// `TEST_HOST`, not a bare test bundle — executed `RootView`'s launch `.task` for real during a + /// test run. Bundle credentials were present, so `CloudSyncModel().autoSyncIfDue(...)` fired and + /// auto-uploaded the Mac's (empty) test database, replacing the production mirror. Two checks, + /// either sufficient on its own: `XCTestConfigurationFilePath` is set by Xcode/`xcodebuild test` + /// for the process actually running the tests; `NSClassFromString("XCTestCase") != nil` is + /// belt-and-suspenders for the TEST_HOST shape, where XCTest is merely linked into the process. + /// Only checked in `autoSyncIfDue` — a manual "Sync now" tap (`syncNow`) stays allowed under test, + /// same as in production, because a human (or an explicit UI-driven test) triggered it on purpose. + /// `nonisolated`, same reason as `isAutoSyncDue`: `CloudSyncModel` is `@MainActor`, and that + /// isolation applies to `static` members by default — without this a plain synchronous XCTest + /// method couldn't call it directly. + nonisolated static var isRunningUnderXCTest: Bool { + ProcessInfo.processInfo.environment["XCTestConfigurationFilePath"] != nil + || NSClassFromString("XCTestCase") != nil + } + /// On-launch daily catch-up (Phase 3.5: zero-touch): silently no-ops when the lane isn't configured /// — `CloudSyncSettings.isConfigured` already covers both a bundle-injected build and a manual /// Keychain save — otherwise runs the same gated sync as `syncNow` when it's been >20h since the @@ -235,6 +253,7 @@ final class CloudSyncModel: ObservableObject { /// waiting on the network. Deliberately does NOT call `syncNow(repo:)` — see `runGatedSync`'s doc /// comment for why that would self-deadlock the gate. func autoSyncIfDue(repo: Repository) { + guard !Self.isRunningUnderXCTest else { return } guard CloudSyncSettings.isConfigured else { return } let last = UserDefaults.standard.double(forKey: Self.lastAutoSyncKey) guard Self.isAutoSyncDue(lastRun: last, now: Date().timeIntervalSince1970) else { return } diff --git a/Strand/CloudSync/CloudSyncUploader.swift b/Strand/CloudSync/CloudSyncUploader.swift index a5fdefdd66..cadb15981c 100644 --- a/Strand/CloudSync/CloudSyncUploader.swift +++ b/Strand/CloudSync/CloudSyncUploader.swift @@ -17,11 +17,21 @@ extension CloudSyncClient: CloudIngesting {} /// having bytes to send". enum CloudSyncUploadError: LocalizedError, Equatable { case exportFailed(String) + /// The store has no real data yet — refused BEFORE export ever runs. Distinct from + /// `exportFailed`: nothing went wrong, there was simply nothing to upload. Guards against the + /// incident where the macOS TEST HOST (`StrandTests` running inside the full `Staging.app` via + /// `TEST_HOST`) executed the launch-time auto-sync `.task` with bundle credentials present, and + /// auto-uploaded the Mac's empty database, replacing the production mirror. This check protects + /// EVERY upload path — fresh installs, a never-paired Mac container, and any future race — not + /// just the test-host case (see `CloudSyncModel.isRunningUnderXCTest` for that separate guard). + case emptyStore var errorDescription: String? { switch self { case .exportFailed(let detail): return "Couldn't prepare the backup to upload. \(detail)" + case .emptyStore: + return "Nothing to upload yet — the local database is empty." } } } @@ -56,6 +66,13 @@ enum CloudSyncUploader { /// `URLSession.upload(for:fromFile:)` rather than loading it into memory. static func upload(store: WhoopStore, client: any CloudIngesting, exporter: Exporter = defaultExporter) async throws -> (bytes: Int, latestDay: String?) { + // Refuse an empty/trivial store BEFORE touching export or the network at all — see + // `CloudSyncUploadError.emptyStore`'s doc comment for the incident this guards against. + // `dailyMetric` is written by every ingest path (BLE-derived recompute, WHOOP/Apple + // Health/Oura/Xiaomi imports), so a genuinely fresh/never-populated store has zero rows here. + guard try await store.hasAnyDailyMetrics() else { + throw CloudSyncUploadError.emptyStore + } let cachesDir = FileManager.default.urls(for: .cachesDirectory, in: .userDomainMask)[0] let tempURL = cachesDir.appendingPathComponent("cloudsync-upload-\(UUID().uuidString).noopbak") defer { try? FileManager.default.removeItem(at: tempURL) } diff --git a/StrandTests/CloudSyncUploaderTests.swift b/StrandTests/CloudSyncUploaderTests.swift index cc88eed2d0..21ec711779 100644 --- a/StrandTests/CloudSyncUploaderTests.swift +++ b/StrandTests/CloudSyncUploaderTests.swift @@ -5,6 +5,18 @@ import XCTest import WhoopStore @testable import Strand +/// A spy conforming to `CloudIngesting` that records how many times `ingest` was invoked — used by +/// the empty-store guard test to assert the network is never reached, without needing a real +/// `CloudSyncClient`/`OuraURLProtocolStub` at all (mirrors `StubCloudEditFetcher` in +/// `CloudSyncCoordinatorTests.swift`). +private final class SpyCloudIngester: CloudIngesting { + private(set) var ingestCallCount = 0 + func ingest(fileURL: URL) async throws -> (bytes: Int, latestDay: String?) { + ingestCallCount += 1 + return (0, nil) + } +} + final class CloudSyncUploaderTests: XCTestCase { override func setUp() { super.setUp(); OuraURLProtocolStub.reset() } @@ -28,12 +40,26 @@ final class CloudSyncUploaderTests: XCTestCase { } } + /// An in-memory store seeded with one `dailyMetric` row, so it clears the empty-store guard added + /// to `CloudSyncUploader.upload` (see the "Empty store guard" section below, which tests that guard + /// directly against an UNSEEDED store) and these pre-existing tests keep exercising what they + /// actually test — network/export behavior beyond that guard — instead of being short-circuited by it. + private func nonEmptyStore() async throws -> WhoopStore { + let store = try await WhoopStore.inMemory() + try await store.upsertDailyMetrics([ + DailyMetric(day: "2026-07-11", totalSleepMin: nil, efficiency: nil, deepMin: nil, remMin: nil, + lightMin: nil, disturbances: nil, restingHr: nil, avgHrv: nil, recovery: nil, + strain: nil, exerciseCount: nil), + ], deviceId: "devA") + return store + } + // MARK: - Happy path func testUploadHappyPathSendsBearerAndOctetStreamAndParsesResponse() async throws { OuraURLProtocolStub.queue = [.init(status: 200, body: #"{"ok":true,"bytes":12345,"latestDay":"2026-07-11"}"#.data(using: .utf8)!)] - let store = try await WhoopStore.inMemory() + let store = try await nonEmptyStore() let client = makeClient() var capturedDest: URL? @@ -60,7 +86,7 @@ final class CloudSyncUploaderTests: XCTestCase { func testUploadNon2xxThrowsBadResponse() async throws { OuraURLProtocolStub.queue = [.init(status: 401, body: "nope".data(using: .utf8)!)] - let store = try await WhoopStore.inMemory() + let store = try await nonEmptyStore() let client = makeClient() do { @@ -73,7 +99,7 @@ final class CloudSyncUploaderTests: XCTestCase { func testUploadCleansUpTempFileEvenOnNetworkFailure() async throws { OuraURLProtocolStub.queue = [.init(status: 500, body: Data())] - let store = try await WhoopStore.inMemory() + let store = try await nonEmptyStore() let client = makeClient() var capturedDest: URL? @@ -91,7 +117,7 @@ final class CloudSyncUploaderTests: XCTestCase { // MARK: - Export failure (never reaches the network) func testUploadExportFailurePropagatesAsTypedErrorAndNeverPosts() async throws { - let store = try await WhoopStore.inMemory() + let store = try await nonEmptyStore() let client = makeClient() let failingExporter: CloudSyncUploader.Exporter = { _, _ in .failure("disk full") } @@ -103,6 +129,43 @@ final class CloudSyncUploaderTests: XCTestCase { } XCTAssertTrue(OuraURLProtocolStub.requestedRequests.isEmpty) // never reached the network } + + // MARK: - Empty store guard (incident: a test-host's empty DB must never upload) + + /// Reproduces the production incident: an empty store (a fresh `WhoopStore.inMemory()` has zero + /// `dailyMetric` rows, same shape as the macOS TEST HOST's never-populated database) must be + /// refused BEFORE export, BEFORE the network — proven here with no real `CloudSyncClient` at all, + /// just a `SpyCloudIngester` that would record a call if `upload` ever reached it. + func testUploadRefusesEmptyStoreAndNeverExportsOrIngests() async throws { + let store = try await WhoopStore.inMemory() + let ingester = SpyCloudIngester() + var exporterCallCount = 0 + let countingExporter: CloudSyncUploader.Exporter = { _, dest in + exporterCallCount += 1 + return .exported(dest) + } + + do { + _ = try await CloudSyncUploader.upload(store: store, client: ingester, exporter: countingExporter) + XCTFail("expected emptyStore to be thrown") + } catch { + XCTAssertEqual(error as? CloudSyncUploadError, CloudSyncUploadError.emptyStore) + } + XCTAssertEqual(exporterCallCount, 0, "the export step must never run for an empty store") + XCTAssertEqual(ingester.ingestCallCount, 0, "ingest must never be called for an empty store") + } + + /// The mirror-image case: once the store has at least one `dailyMetric` row, upload proceeds + /// exactly as before (happy-path regression guard for the new empty-store check). + func testUploadProceedsWhenStoreHasData() async throws { + OuraURLProtocolStub.queue = [.init(status: 200, + body: #"{"ok":true,"bytes":99,"latestDay":"2026-07-11"}"#.data(using: .utf8)!)] + let store = try await nonEmptyStore() + let client = makeClient() + + let result = try await CloudSyncUploader.upload(store: store, client: client, exporter: fakeExporter()) + XCTAssertEqual(result.bytes, 99) + } } /// The 20h auto-sync gate, extracted as a pure `(lastRun, now) -> Bool` function so the threshold is @@ -119,5 +182,15 @@ final class CloudSyncModelAutoSyncGateTests: XCTestCase { XCTAssertTrue(CloudSyncModel.isAutoSyncDue(lastRun: now - 20 * 3600, now: now)) // exactly due (>=) XCTAssertTrue(CloudSyncModel.isAutoSyncDue(lastRun: now - 21 * 3600, now: now)) // overdue } + + /// Incident guard: `autoSyncIfDue` must never fire inside an XCTest runner (the macOS TEST HOST + /// runs `StrandTests` inside the full `Staging.app` via `TEST_HOST`, so `RootView`'s launch `.task` + /// executes for real — see `CloudSyncModel.isRunningUnderXCTest`'s doc comment). This is + /// inherently env-dependent and can only be pinned from the TRUE side: every `StrandTests` run, + /// including this one, IS an XCTest run, so the helper reading true here is exactly what "detects + /// the test runner" means. There is no way to assert the false branch from inside XCTest itself. + func testIsRunningUnderXCTestIsTrueInThisTestRunner() { + XCTAssertTrue(CloudSyncModel.isRunningUnderXCTest) + } } #endif // CLOUD_SYNC From 1637b52fd24a80010b342fb9da3139b21d3792d0 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 21:59:05 -0700 Subject: [PATCH 059/171] =?UTF-8?q?fix(oura):=20stop=20storing=20the=20rea?= =?UTF-8?q?diness=20resting=5Fheart=5Frate=20SCORE=20as=20bpm=20=E2=80=94?= =?UTF-8?q?=20real=20RHR=20comes=20from=20sleep=20lowest=5Fheart=5Frate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../StrandImport/OuraApiParser+Daily.swift | 9 +++++++-- .../OuraApiParserDailyTests.swift | 19 +++++++++++++++++-- Strand/Oura/OuraSyncCoordinator.swift | 12 +++++++----- StrandTests/OuraSyncCoordinatorTests.swift | 5 +++-- 4 files changed, 34 insertions(+), 11 deletions(-) diff --git a/Packages/StrandImport/Sources/StrandImport/OuraApiParser+Daily.swift b/Packages/StrandImport/Sources/StrandImport/OuraApiParser+Daily.swift index 7e7a75c397..f7e4d12ac4 100644 --- a/Packages/StrandImport/Sources/StrandImport/OuraApiParser+Daily.swift +++ b/Packages/StrandImport/Sources/StrandImport/OuraApiParser+Daily.swift @@ -3,8 +3,11 @@ import Foundation public extension OuraApiParser { /// Parse a day-keyed summary endpoint's documents. `endpoint` selects the field mapping. Writes to - /// `WearableDailyRow` columns only where the on-device schema already has a home (RHR, skin-temp dev, + /// `WearableDailyRow` columns only where the on-device schema already has a home (skin-temp dev, /// steps, calories, SpO2); everything else — and ALL of Oura's own scores — becomes `OuraDailyExtra`. + /// Resting HR is deliberately NOT written here: `contributors.resting_heart_rate` on `daily_readiness` + /// is a 0-100 readiness contributor SCORE, not bpm. The real resting HR comes from sleep's + /// `lowest_heart_rate` (`OuraApiParser.parseSleep`). static func parseDaily(_ docs: [[String: Any]], endpoint: String) -> (days: [WearableDailyRow], extras: [OuraDailyExtra]) { var byDay: [String: WearableDailyRow] = [:] var extras: [OuraDailyExtra] = [] @@ -19,7 +22,9 @@ public extension OuraApiParser { case "daily_readiness": var r = byDay[day] ?? WearableDailyRow(day: day) if let c = doc["contributors"] as? [String: Any] { - r.restingHr = WearableJSON.posInt(c, "resting_heart_rate") ?? r.restingHr + // `resting_heart_rate` here is a 0-100 readiness contributor SCORE, not bpm — do not + // assign it to r.restingHr (that caused scores like 99/100/1 to be stored as "RHR"). + // It still surfaces honestly below as the `oura_readiness_resting_heart_rate` extra. for (k, _) in c { extra(day, "oura_readiness_\(k)", WearableJSON.posDbl(c, k)) } } r.skinTempDevC = WearableJSON.dbl(doc, "temperature_deviation") ?? r.skinTempDevC diff --git a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserDailyTests.swift b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserDailyTests.swift index ca3e176b8e..a3ec71ba60 100644 --- a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserDailyTests.swift +++ b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserDailyTests.swift @@ -2,18 +2,33 @@ import XCTest @testable import StrandImport final class OuraApiParserDailyTests: XCTestCase { - func testReadinessMapsRhrSkinTempAndRefScore() { + func testReadinessMapsSkinTempAndRefScoreButNotRhr() { let (days, extras) = OuraApiParser.parseDaily([[ "day": "2026-01-02", "score": 80, "temperature_deviation": -0.2, "contributors": ["resting_heart_rate": 50, "hrv_balance": 70] ]], endpoint: "daily_readiness") - XCTAssertEqual(days.first?.restingHr, 50) + // contributors.resting_heart_rate is a 0-100 readiness SCORE, not bpm — must not land on restingHr. + XCTAssertNil(days.first?.restingHr) XCTAssertEqual(days.first?.skinTempDevC, -0.2) XCTAssertEqual(days.first?.readinessScore, 80) XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "ref_readiness_score", value: 80))) XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "oura_readiness_hrv_balance", value: 70))) } + /// Regression test: `contributors.resting_heart_rate` is a 0-100 readiness contributor SCORE, not bpm. + /// A prior bug stored it directly as the user's resting HR (scores like 99/100/1/10/43 as "RHR" across + /// ~493 imported days). It must stay nil here — the real resting HR comes from sleep's + /// `lowest_heart_rate` (see `OuraApiParser.parseSleep`) — while the raw score still surfaces honestly + /// as an extra under its own name. + func testReadinessContributorRhrScoreDoesNotBecomeRestingHr() { + let (days, extras) = OuraApiParser.parseDaily([[ + "day": "2026-01-02", "score": 80, + "contributors": ["resting_heart_rate": 99] + ]], endpoint: "daily_readiness") + XCTAssertNil(days.first?.restingHr) + XCTAssertTrue(extras.contains(OuraDailyExtra(day: "2026-01-02", key: "oura_readiness_resting_heart_rate", value: 99))) + } + func testActivityMapsStepsAndScoreIsReferenceOnly() { let (days, extras) = OuraApiParser.parseDaily([[ "day": "2026-01-02", "score": 88, "steps": 9000, diff --git a/Strand/Oura/OuraSyncCoordinator.swift b/Strand/Oura/OuraSyncCoordinator.swift index 3dcba6cd6b..3a5af92100 100644 --- a/Strand/Oura/OuraSyncCoordinator.swift +++ b/Strand/Oura/OuraSyncCoordinator.swift @@ -21,14 +21,16 @@ struct OuraSyncProgress: Equatable { } /// One-time backfill across every Oura v2 endpoint → an assembled OuraSyncResult → OuraSyncWriter. Merges -/// the per-day WearableDailyRows across endpoints (daily endpoints BEFORE sleep, so readiness RHR wins). +/// the per-day WearableDailyRows across endpoints (daily endpoints BEFORE sleep). Resting HR comes +/// exclusively from sleep's `lowest_heart_rate` — no daily endpoint sets it (see `OuraApiParser.parseDaily`). final class OuraSyncCoordinator { private let fetcher: OuraPageFetching private let store: WhoopStore init(fetcher: OuraPageFetching, store: WhoopStore) { self.fetcher = fetcher; self.store = store } - /// Daily-summary endpoints handed to `parseDaily(_, endpoint:)`. Ordered so `daily_readiness` merges - /// before `sleep` (RHR precedence). `daily_sleep` etc. contribute extras only. + /// Daily-summary endpoints handed to `parseDaily(_, endpoint:)`. Merge order doesn't affect resting HR + /// (only `sleep`, merged separately below, ever sets it); `daily_readiness`, `daily_sleep` etc. + /// contribute their other fields / extras only. private static let dailyEndpoints = ["daily_readiness", "daily_activity", "daily_spo2", "daily_sleep", "daily_stress", "daily_resilience", "daily_cardiovascular_age", "vO2_max"] /// Endpoints with no Plan-1 parser — archived raw only. @@ -84,7 +86,7 @@ final class OuraSyncCoordinator { // and the run continues; the UI surfaces the skips from the summary. var skipped: [String] = [] - // Daily endpoints first (readiness RHR precedence), extras accumulate. + // Daily endpoints first, extras accumulate. None of them set resting HR (see parseDaily). for endpoint in Self.dailyEndpoints { do { let pages = try await fetchRaw(endpoint, dateParam: "start_date") @@ -92,7 +94,7 @@ final class OuraSyncCoordinator { merge(days); result.extras.append(contentsOf: extras) } catch { skipped.append(endpoint) } } - // Sleep last (its lowestHr fallback only fills days readiness didn't cover). + // Sleep last: its lowestHr is the sole resting-HR source (no daily endpoint sets one). do { let sleepPages = try await fetchRaw("sleep", dateParam: "start_date") let (periods, sleepDays) = OuraApiParser.parseSleep(docs(sleepPages)) diff --git a/StrandTests/OuraSyncCoordinatorTests.swift b/StrandTests/OuraSyncCoordinatorTests.swift index 0d4e4d3137..f14121ef77 100644 --- a/StrandTests/OuraSyncCoordinatorTests.swift +++ b/StrandTests/OuraSyncCoordinatorTests.swift @@ -41,9 +41,10 @@ final class OuraSyncCoordinatorTests: XCTestCase { XCTAssertTrue(summary.rawPages >= 3) XCTAssertTrue(progress.contains("sleep")) - // Coalesce + RHR precedence: daily_readiness's true RHR (50) wins over sleep's lowestHr (48). + // Coalesce: daily_readiness's contributors.resting_heart_rate (50) is a 0-100 readiness SCORE, not + // bpm, so it must NOT become restingHr — sleep's lowest_heart_rate (48) is the real resting HR. let days = try await store.dailyMetrics(deviceId: "oura-api", from: "2026-01-01", to: "2026-01-03") - XCTAssertEqual(days.first?.restingHr, 50) + XCTAssertEqual(days.first?.restingHr, 48) XCTAssertEqual(days.first?.steps, 9000) XCTAssertEqual(days.first?.totalSleepMin, 400) } From 3383dda1f855cc58f70edd83ead8bd5fcb0aeb12 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 22:12:04 -0700 Subject: [PATCH 060/171] =?UTF-8?q?fix(oura):=20export=20importer=20?= =?UTF-8?q?=E2=80=94=20stop=20storing=20the=20readiness=20resting=5Fheart?= =?UTF-8?q?=5Frate=20SCORE=20as=20bpm=20(twin=20of=201637b52f)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../StrandImport/OuraExportParser.swift | 14 ++-- .../OuraExportParserTests.swift | 69 +++++++++++++++++++ .../WearableExportImporterTests.swift | 21 ++++-- 3 files changed, 91 insertions(+), 13 deletions(-) create mode 100644 Packages/StrandImport/Tests/StrandImportTests/OuraExportParserTests.swift diff --git a/Packages/StrandImport/Sources/StrandImport/OuraExportParser.swift b/Packages/StrandImport/Sources/StrandImport/OuraExportParser.swift index 3a0b5010ec..aae7464ca2 100644 --- a/Packages/StrandImport/Sources/StrandImport/OuraExportParser.swift +++ b/Packages/StrandImport/Sources/StrandImport/OuraExportParser.swift @@ -14,7 +14,9 @@ import Foundation // a per-segment hypnogram, so the session carries the breakdown without a stage // timeline (we never fake one). // daily_readiness : day, score (Oura's OWN readiness, REFERENCE only, never NOOP Charge), -// temperature_deviation (°C from baseline), contributors.resting_heart_rate. +// temperature_deviation (°C from baseline). `contributors.resting_heart_rate` +// is a 0-100 readiness contributor SCORE, not bpm — never read as resting HR; +// the real RHR comes only from a sleep period's lowest_heart_rate. // daily_activity : day, steps, active_calories, total_calories, equivalent_walking_distance (m). // daily_sleep : day, score (sleep score, reference only). // daily_spo2 : day, spo2_percentage.average (%); note the value is NESTED under that key, not @@ -72,14 +74,14 @@ enum OuraExportParser { byDay[key] = row } - // Daily readiness → RHR + temperature deviation + reference readiness score. + // Daily readiness → temperature deviation + reference readiness score. + // `contributors.resting_heart_rate` (and a flattened `resting_heart_rate` alongside it) is a + // 0-100 readiness contributor SCORE, not bpm — it must never land on row.restingHr (the API + // lane had the same bug; the old code here also clobbered the sleep-derived value above). + // The real resting HR comes only from the sleep loop's `lowest_heart_rate`. for r in categoryArray(root, "daily_readiness") ?? categoryArray(root, "readiness") ?? [] { guard let key = WearableJSON.str(r, "day") else { continue } var row = day(key) - if let contrib = r["contributors"] as? [String: Any] { - row.restingHr = WearableJSON.posInt(contrib, "resting_heart_rate") ?? row.restingHr - } - row.restingHr = WearableJSON.posInt(r, "resting_heart_rate") ?? row.restingHr row.skinTempDevC = WearableJSON.dbl(r, "temperature_deviation") ?? row.skinTempDevC row.readinessScore = WearableJSON.posInt(r, "score") ?? row.readinessScore byDay[key] = row diff --git a/Packages/StrandImport/Tests/StrandImportTests/OuraExportParserTests.swift b/Packages/StrandImport/Tests/StrandImportTests/OuraExportParserTests.swift new file mode 100644 index 0000000000..3eac88a4ba --- /dev/null +++ b/Packages/StrandImport/Tests/StrandImportTests/OuraExportParserTests.swift @@ -0,0 +1,69 @@ +import XCTest +@testable import StrandImport + +/// Pins the account-export ("Export Data" file) lane's resting-HR semantics — the file-import twin of +/// `OuraApiParserDailyTests`. Oura's daily_readiness `contributors.resting_heart_rate` is a 0-100 +/// readiness contributor SCORE, not bpm; the real resting HR comes only from a sleep period's +/// `lowest_heart_rate`. +final class OuraExportParserTests: XCTestCase { + + private func bytes(_ s: String) -> Data { s.data(using: .utf8)! } + + /// Regression test (twin of the API-lane fix): a prior bug stored the readiness contributor score + /// directly as the user's resting HR. With no sleep period in the export, restingHr must stay nil — + /// never the 0-100 score — while the readiness score itself stays reference-only. + func testReadinessContributorRhrScoreDoesNotBecomeRestingHr() { + let json = """ + { "daily_readiness": [ { "day": "2026-01-02", "score": 80, + "contributors": { "resting_heart_rate": 99 } } ] } + """ + let (days, _) = OuraExportParser.parse(["export.json": bytes(json)]) + XCTAssertEqual(days.count, 1) + XCTAssertNil(days.first?.restingHr) + XCTAssertEqual(days.first?.readinessScore, 80) + } + + /// A flat `resting_heart_rate` key on the readiness record is the same contributor score in a + /// flattened shape — it must not land on restingHr either. + func testFlatReadinessRestingHeartRateKeyDoesNotBecomeRestingHr() { + let json = """ + { "daily_readiness": [ { "day": "2026-01-02", "score": 80, "resting_heart_rate": 97 } ] } + """ + let (days, _) = OuraExportParser.parse(["export.json": bytes(json)]) + XCTAssertNil(days.first?.restingHr) + XCTAssertEqual(days.first?.readinessScore, 80) + } + + /// Sleep's `lowest_heart_rate` is the sole resting-HR source. The old code let the readiness loop + /// overwrite the sleep-derived 48 bpm with the contributor score (96 here) whenever both categories + /// were present — regardless of file iteration order, since the overwrite didn't check for nil. + func testSleepLowestHeartRateIsTheSoleRestingHrSource() { + let json = """ + { + "sleep": [ + { "day": "2026-01-02", "bedtime_start": "2026-01-01T23:00:00+00:00", + "bedtime_end": "2026-01-02T06:30:00+00:00", "total_sleep_duration": 25200, + "lowest_heart_rate": 48 } ], + "daily_readiness": [ + { "day": "2026-01-02", "score": 80, + "contributors": { "resting_heart_rate": 96 } } ] + } + """ + let (days, _) = OuraExportParser.parse(["export.json": bytes(json)]) + let day = days.first { $0.day == "2026-01-02" } + XCTAssertEqual(day?.restingHr, 48) + XCTAssertEqual(day?.readinessScore, 80) + } + + /// The `readiness` category alias (older API-shaped exports) runs through the same loop and must + /// obey the same rule. + func testReadinessAliasCategoryDoesNotWriteRestingHr() { + let json = """ + { "readiness": [ { "day": "2026-01-03", "score": 74, + "contributors": { "resting_heart_rate": 100 } } ] } + """ + let (days, _) = OuraExportParser.parse(["export.json": bytes(json)]) + XCTAssertNil(days.first?.restingHr) + XCTAssertEqual(days.first?.readinessScore, 74) + } +} diff --git a/Packages/StrandImport/Tests/StrandImportTests/WearableExportImporterTests.swift b/Packages/StrandImport/Tests/StrandImportTests/WearableExportImporterTests.swift index e08beb8c50..3483cdeb62 100644 --- a/Packages/StrandImport/Tests/StrandImportTests/WearableExportImporterTests.swift +++ b/Packages/StrandImport/Tests/StrandImportTests/WearableExportImporterTests.swift @@ -12,8 +12,9 @@ final class WearableExportImporterTests: XCTestCase { // MARK: - Oura func testOuraSleepReadinessActivityFold() { - // Sleep period: 7h asleep (25200s) with stage durations + HRV/RHR; readiness gives RHR + temp - // deviation + a readiness SCORE (reference); activity gives steps + calories. + // Sleep period: 7h asleep (25200s) with stage durations + HRV/RHR; readiness gives temp + // deviation + a readiness SCORE (reference) — its contributors.resting_heart_rate is a 0-100 + // SCORE, not bpm, and must never become the day's RHR; activity gives steps + calories. let json = """ { "sleep": [ @@ -35,7 +36,7 @@ final class WearableExportImporterTests: XCTestCase { ], "daily_readiness": [ { "day": "2026-06-01", "score": 81, "temperature_deviation": -0.2, - "contributors": { "resting_heart_rate": 49 } } + "contributors": { "resting_heart_rate": 96 } } ], "daily_activity": [ { "day": "2026-06-01", "steps": 8421, "active_calories": 520, "total_calories": 2450 } @@ -59,7 +60,7 @@ final class WearableExportImporterTests: XCTestCase { XCTAssertEqual(r.days.count, 1) let d = r.days[0] XCTAssertEqual(d.day, "2026-06-01") - XCTAssertEqual(d.restingHr, 49) // readiness contributor wins + XCTAssertEqual(d.restingHr, 48) // sleep lowest_heart_rate, never the 96 score XCTAssertEqual(d.skinTempDevC!, -0.2, accuracy: 1e-6) XCTAssertEqual(d.steps, 8421) XCTAssertEqual(d.activeKcal!, 520, accuracy: 1e-6) @@ -155,9 +156,15 @@ final class WearableExportImporterTests: XCTestCase { func testOuraJSONStillWinsOverCSVForSameDay() { // A mixed export (JSON + CSV) for the same day: the richer JSON value must win, CSV fills gaps. + // The JSON's RHR comes from the sleep period's lowest_heart_rate; the readiness + // contributors.resting_heart_rate is a 0-100 SCORE and never competes at all. let json = """ - { "daily_readiness": [ { "day": "2026-06-01", "score": 81, - "contributors": { "resting_heart_rate": 49 } } ] } + { "sleep": [ + { "day": "2026-06-01", "bedtime_start": "2026-05-31T23:15:00+00:00", + "bedtime_end": "2026-06-01T06:30:00+00:00", "total_sleep_duration": 25200, + "lowest_heart_rate": 49 } ], + "daily_readiness": [ { "day": "2026-06-01", "score": 81, + "contributors": { "resting_heart_rate": 96 } } ] } """ let csv = """ date,Average Resting Heart Rate,Steps @@ -166,7 +173,7 @@ final class WearableExportImporterTests: XCTestCase { let files: [String: Data] = ["oura.json": bytes(json), "oura_daily.csv": bytes(csv)] let r = WearableExportImporter.parse(brand: .oura, files: files) let d = r.days.first { $0.day == "2026-06-01" }! - XCTAssertEqual(d.restingHr, 49) // JSON readiness RHR wins over the CSV's 77 + XCTAssertEqual(d.restingHr, 49) // JSON sleep lowest HR wins over the CSV's 77 XCTAssertEqual(d.steps, 8421) // CSV fills the step gap JSON lacked } From d47525ea17c63c9aec276dc409a1abefa8071055 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 22:15:57 -0700 Subject: [PATCH 061/171] =?UTF-8?q?feat(apple):=20hourly=20step=20import?= =?UTF-8?q?=20(v27)=20=E2=80=94=20per-hour=20iPhone=20steps=20with=2090-da?= =?UTF-8?q?y=20backfill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Apple Health importer flattened steps to one daily total, so a phone that died mid-hike just read as a low/zero day instead of showing which hours had no data. HealthKit retains hourly statistics historically, so this adds a per-hour store (appleStepHour, PK deviceId+ts) and wires an hourly HKStatisticsCollectionQuery alongside the existing daily collector, with a one-time 90-day backfill on first run so past days answer retroactively too. --- .../WhoopStore/AppleStepHourStore.swift | 42 ++++++++++++++ .../Sources/WhoopStore/Database.swift | 20 +++++++ .../WhoopStore/DeviceRegistryStore.swift | 4 ++ .../AppleStepHourStoreTests.swift | 56 +++++++++++++++++++ StrandiOS/Health/HealthKitBridge.swift | 52 +++++++++++++++++ 5 files changed, 174 insertions(+) create mode 100644 Packages/WhoopStore/Sources/WhoopStore/AppleStepHourStore.swift create mode 100644 Packages/WhoopStore/Tests/WhoopStoreTests/AppleStepHourStoreTests.swift diff --git a/Packages/WhoopStore/Sources/WhoopStore/AppleStepHourStore.swift b/Packages/WhoopStore/Sources/WhoopStore/AppleStepHourStore.swift new file mode 100644 index 0000000000..13cf745a86 --- /dev/null +++ b/Packages/WhoopStore/Sources/WhoopStore/AppleStepHourStore.swift @@ -0,0 +1,42 @@ +import Foundation +import GRDB + +// MARK: - v27 store: hourly Apple Health step counts +// Mirrors OuraRawStore: an idempotent ON CONFLICT upsert keyed by the natural key (deviceId, ts), and +// a range read — all GRDB work via syncWrite/syncRead. `appleDaily.steps` already answers "how many +// steps that day"; this table answers "which HOURS were recorded", so the UI can show retroactively +// when the iPhone was off/dead/left behind (e.g. mid-hike) instead of a single flattened daily total. + +extension WhoopStore { + + /// Upsert hourly Apple Health step counts. Idempotent by (deviceId, ts): re-importing the same + /// hour overwrites its count rather than duplicating. Returns rows changed. + @discardableResult + public func upsertAppleStepHours(_ rows: [(ts: Int, steps: Int)], deviceId: String) async throws -> Int { + try syncWrite { db in + var n = 0 + for r in rows { + try db.execute(sql: """ + INSERT INTO appleStepHour (deviceId, ts, steps) + VALUES (?, ?, ?) + ON CONFLICT(deviceId, ts) DO UPDATE SET + steps = excluded.steps + """, arguments: [deviceId, r.ts, r.steps]) + n += db.changesCount + } + return n + } + } + + /// Hourly step counts for a device over `[fromTs, toTs]` inclusive, oldest hour first. + public func appleStepHours(deviceId: String, fromTs: Int, toTs: Int) async throws -> [(ts: Int, steps: Int)] { + try syncRead { db in + try Row.fetchAll(db, sql: """ + SELECT ts, steps FROM appleStepHour + WHERE deviceId = ? AND ts >= ? AND ts <= ? + ORDER BY ts ASC + """, arguments: [deviceId, fromTs, toTs]) + .map { (ts: $0["ts"], steps: $0["steps"]) } + } + } +} diff --git a/Packages/WhoopStore/Sources/WhoopStore/Database.swift b/Packages/WhoopStore/Sources/WhoopStore/Database.swift index 65e37909d9..b533d4870c 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/Database.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/Database.swift @@ -524,6 +524,26 @@ extension WhoopStore { } try db.create(index: "idx_cloudTombstone_kind_device", on: "cloudTombstone", columns: ["kind", "deviceId"]) } + + // v27: hourly Apple Health step counts. The daily `collect(.stepCount, …)` path in + // HealthKitBridge flattens a whole day to one `appleDaily.steps` total, so a dead/absent phone + // for part of a day (e.g. the phone died mid-hike) is invisible — steps just read low for the + // WHOLE day instead of showing exactly which hours had no recording. HealthKit retains hourly + // statistics historically, so this table lets a one-time backfill answer PAST days + // retroactively, not just from the day this migration ships. `ts` is the hour-BUCKET START + // (unix seconds), aligned to local-clock hour boundaries by the `HKStatisticsCollectionQuery` + // anchored at local midnight (see HealthKitBridge); `steps` is the cumulative sum within that + // hour. PK (deviceId, ts) mirrors every other per-sample table (hrSample, stepSample, …) and + // makes the hourly upsert idempotent. Additive only — a NEW table, no existing row touched, old + // readers unaffected. + migrator.registerMigration("v27-apple-step-hour") { db in + try db.create(table: "appleStepHour") { t in + t.column("deviceId", .text).notNull() // "apple-health" + t.column("ts", .integer).notNull() // hour-start unix seconds (local-hour aligned by HK) + t.column("steps", .integer).notNull() + t.primaryKey(["deviceId", "ts"]) + } + } return migrator } } diff --git a/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift b/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift index a997ae7a24..b488624ac5 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift @@ -90,6 +90,10 @@ public struct DeviceRegistryStore: Sendable { // v25-oura-raw: the opt-in Oura cloud-import raw archive is deviceId-keyed too, so "delete this // device's data" must clear it — else an imported Oura source's payloads would survive deletion. "ouraRaw", + // v27-apple-step-hour: per-hour Apple Health step counts are deviceId-keyed (apple-health) and + // are real per-device recordings like appleDaily/stepSample, not tombstone-class bookkeeping — + // a device-data wipe / "Remove Apple Health data" must clear them too. + "appleStepHour", ] /// deviceId-keyed tables deliberately EXCLUDED from `deviceScopedTables` — a normal "delete this diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/AppleStepHourStoreTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/AppleStepHourStoreTests.swift new file mode 100644 index 0000000000..a3bedb49cb --- /dev/null +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/AppleStepHourStoreTests.swift @@ -0,0 +1,56 @@ +import XCTest +import GRDB +@testable import WhoopStore + +final class AppleStepHourStoreTests: XCTestCase { + func testUpsertIsIdempotentByNaturalKey() async throws { + let store = try await WhoopStore.inMemory() + let n1 = try await store.upsertAppleStepHours([(ts: 1_000, steps: 120)], deviceId: "apple-health") + XCTAssertEqual(n1, 1) + + // Re-import the same hour with a revised count → overwrite in place, still one row. + let n2 = try await store.upsertAppleStepHours([(ts: 1_000, steps: 250)], deviceId: "apple-health") + XCTAssertEqual(n2, 1) + + let rows = try await store.appleStepHours(deviceId: "apple-health", fromTs: 0, toTs: 10_000) + XCTAssertEqual(rows.count, 1) + XCTAssertEqual(rows.first?.ts, 1_000) + XCTAssertEqual(rows.first?.steps, 250) + } + + func testRangeReadReturnsOldestFirstWithinBoundsInclusive() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertAppleStepHours([ + (ts: 3_600, steps: 100), // hour 1 + (ts: 7_200, steps: 200), // hour 2 + (ts: 10_800, steps: 300), // hour 3 + ], deviceId: "apple-health") + + // Inclusive bounds: both edges of the range are included. + let rows = try await store.appleStepHours(deviceId: "apple-health", fromTs: 3_600, toTs: 10_800) + XCTAssertEqual(rows.map(\.ts), [3_600, 7_200, 10_800]) + XCTAssertEqual(rows.map(\.steps), [100, 200, 300]) + + // A narrower window excludes rows outside it. + let narrow = try await store.appleStepHours(deviceId: "apple-health", fromTs: 3_601, toTs: 10_799) + XCTAssertEqual(narrow.map(\.ts), [7_200]) + + // A different device sees no rows. + let other = try await store.appleStepHours(deviceId: "my-whoop", fromTs: 0, toTs: 100_000) + XCTAssertTrue(other.isEmpty) + } + + func testUpsertPartitionsByDevice() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertAppleStepHours([(ts: 1_000, steps: 50)], deviceId: "apple-health") + _ = try await store.upsertAppleStepHours([(ts: 1_000, steps: 999)], deviceId: "other-device") + + let mine = try await store.appleStepHours(deviceId: "apple-health", fromTs: 0, toTs: 10_000) + XCTAssertEqual(mine.count, 1) + XCTAssertEqual(mine.first?.steps, 50) + + let others = try await store.appleStepHours(deviceId: "other-device", fromTs: 0, toTs: 10_000) + XCTAssertEqual(others.count, 1) + XCTAssertEqual(others.first?.steps, 999) + } +} diff --git a/StrandiOS/Health/HealthKitBridge.swift b/StrandiOS/Health/HealthKitBridge.swift index fb1a5317f8..7455f61df4 100644 --- a/StrandiOS/Health/HealthKitBridge.swift +++ b/StrandiOS/Health/HealthKitBridge.swift @@ -298,6 +298,11 @@ final class HealthKitBridge: ObservableObject { // MARK: - Read → store + /// UserDefaults key gating the one-time 90-day hourly-step backfill (see the hourly collection + /// below). Set once the first hourly-step HealthKit query completes; every later `sync()` only + /// walks the same window the daily collectors already use. + private static let hourlyStepsBackfilledKey = "applehealth.hourlySteps.backfilled" + /// Pull the last `days` of Apple Health into the on-device store under the `apple-health` source, /// then write NOOP's own computed metrics back into Health. Safe to call repeatedly (idempotent /// upserts keyed by day). @@ -369,6 +374,17 @@ final class HealthKitBridge: ObservableObject { byDay[day] = a } + // Hourly step counts: `appleDaily.steps` above flattens a whole day to one total, so a + // dead/absent phone for part of a day (e.g. the phone died mid-hike) is invisible. Walk the + // same window at HOURLY granularity into `appleStepHour` so a UI can show exactly which hours + // were recorded. First run ever (UserDefaults flag unset) widens the window to 90 days back — + // HealthKit retains hourly statistics historically, so this answers PAST days retroactively; + // every later sync only re-covers the normal start...end window like the daily collectors above. + let hourlyStepsBackfilled = UserDefaults.standard.bool(forKey: HealthKitBridge.hourlyStepsBackfilledKey) + let hourlyStepsStart: Date = hourlyStepsBackfilled ? start + : (cal.date(byAdding: .day, value: -90, to: cal.startOfDay(for: end)) ?? start) + let hourlySteps = await collectHourlySteps(start: hourlyStepsStart, end: end) + // Build + upsert the store rows under the apple-health source. let appleRows = byDay.map { (day, a) in AppleDaily(day: day, steps: a.steps.map { Int($0) }, @@ -432,6 +448,10 @@ final class HealthKitBridge: ObservableObject { try await store.upsertDailyMetrics(dmRows, deviceId: appleDeviceId) try await store.upsertMetricSeries(points, deviceId: appleDeviceId) if !workoutRows.isEmpty { try await store.upsertWorkouts(workoutRows, deviceId: appleDeviceId) } + if !hourlySteps.isEmpty { try await store.upsertAppleStepHours(hourlySteps, deviceId: appleDeviceId) } + if !hourlyStepsBackfilled { + UserDefaults.standard.set(true, forKey: HealthKitBridge.hourlyStepsBackfilledKey) + } try await writeBack(whoopStore: store) lastSync = Date() lastError = nil @@ -882,6 +902,38 @@ final class HealthKitBridge: ObservableObject { } } + /// Hourly cumulative step counts over `[start, end)`, mirroring `collect()`'s + /// `HKStatisticsCollectionQuery` shape but bucketed by HOUR instead of day so a backfill can show + /// exactly which hours the phone was recording. `anchorDate` is local midnight — same anchor + /// `collect()` uses for its daily buckets — so hour buckets fall on local-clock hour boundaries, + /// not UTC ones. Silently returns fewer (or zero) rows on a HealthKit query error, same as + /// `collect()`; the next foreground/background catch-up re-walks the window. + private func collectHourlySteps(start: Date, end: Date) async -> [(ts: Int, steps: Int)] { + guard let type = HKQuantityType.quantityType(forIdentifier: .stepCount) else { return [] } + let cal = Calendar.current + let anchor = cal.startOfDay(for: start) + let predicate = NSCompoundPredicate(andPredicateWithSubpredicates: [ + HKQuery.predicateForSamples(withStart: start, end: end, options: .strictStartDate), + Self.notNoopAuthored, + ]) + return await withCheckedContinuation { (cont: CheckedContinuation<[(ts: Int, steps: Int)], Never>) in + let q = HKStatisticsCollectionQuery(quantityType: type, quantitySamplePredicate: predicate, + options: .cumulativeSum, anchorDate: anchor, + intervalComponents: DateComponents(hour: 1)) + q.initialResultsHandler = { _, results, _ in + var rows: [(ts: Int, steps: Int)] = [] + results?.enumerateStatistics(from: start, to: end) { stats, _ in + if let sum = stats.sumQuantity() { + let steps = Int(sum.doubleValue(for: .count()).rounded()) + rows.append((ts: Int(stats.startDate.timeIntervalSince1970), steps: steps)) + } + } + cont.resume(returning: rows) + } + store.execute(q) + } + } + // MARK: - Workouts (#835) /// Read the workouts the user logged in Apple Health over `[start, end)` and map each to a From a7c47acb8a1cdf384c64d897262604cd62c68af8 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 22:30:43 -0700 Subject: [PATCH 062/171] =?UTF-8?q?fix(oura):=20android=20export=20importe?= =?UTF-8?q?r=20=E2=80=94=20stop=20storing=20the=20readiness=20resting=5Fhe?= =?UTF-8?q?art=5Frate=20SCORE=20as=20bpm=20(kotlin=20twin=20of=203383dda1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../com/noop/ingest/WearableExportImporter.kt | 11 ++- .../noop/ingest/WearableExportImporterTest.kt | 67 ++++++++++++++++++- 2 files changed, 73 insertions(+), 5 deletions(-) diff --git a/android/app/src/main/java/com/noop/ingest/WearableExportImporter.kt b/android/app/src/main/java/com/noop/ingest/WearableExportImporter.kt index 6e32b4fe16..7116d8c683 100644 --- a/android/app/src/main/java/com/noop/ingest/WearableExportImporter.kt +++ b/android/app/src/main/java/com/noop/ingest/WearableExportImporter.kt @@ -26,7 +26,9 @@ import java.util.zip.ZipInputStream * * • Oura : the Account -> Export Data per-category files (CSV, `;`-delimited; or an older JSON * variant): sleep periods (durations/HRV/RHR/breath; a `deleted` type is skipped), daily - * readiness (RHR, temperature deviation, score), daily activity (steps/calories/distance), + * readiness (temperature deviation, score — its `contributors.resting_heart_rate` is a + * 0-100 SCORE, not bpm, never read as resting HR; the real RHR comes only from a sleep + * period's lowest_heart_rate), daily activity (steps/calories/distance), * daily SpO2 (`spo2_percentage.average`), VO2max (`vo2_max`). Field names verified against a * REAL Oura export schema (issue #862). The many health types NOOP doesn't model * (bloodglucose, contraception, medication, ring config, raw sample streams, ...) are skipped @@ -258,13 +260,16 @@ object WearableExportImporter { if (d.restingHr == null) d.restingHr = session.lowestHr } } + // Daily readiness → temperature deviation + reference readiness score. + // `contributors.resting_heart_rate` (and a flattened `resting_heart_rate` alongside it) is a + // 0-100 readiness contributor SCORE, not bpm — it must never land on d.restingHr (the old code + // clobbered the sleep-derived value above). The real resting HR comes only from the sleep + // loop's `lowest_heart_rate`. Twin of the Swift OuraExportParser fix. (categoryArray(root, "daily_readiness") ?: categoryArray(root, "readiness"))?.let { arr -> for (i in 0 until arr.length()) { val r = arr.optJSONObject(i) ?: continue val key = r.strOpt("day") ?: continue val d = day(key) - r.optJSONObject("contributors")?.let { d.restingHr = it.posInt("resting_heart_rate") ?: d.restingHr } - d.restingHr = r.posInt("resting_heart_rate") ?: d.restingHr d.skinTempDevC = r.dblOpt("temperature_deviation") ?: d.skinTempDevC d.readinessScore = r.posInt("score") ?: d.readinessScore } diff --git a/android/app/src/test/java/com/noop/ingest/WearableExportImporterTest.kt b/android/app/src/test/java/com/noop/ingest/WearableExportImporterTest.kt index cdf601facf..a1a82d08a2 100644 --- a/android/app/src/test/java/com/noop/ingest/WearableExportImporterTest.kt +++ b/android/app/src/test/java/com/noop/ingest/WearableExportImporterTest.kt @@ -30,7 +30,7 @@ class WearableExportImporterTest { "average_hrv": 65 } ], "daily_readiness": [ { "day": "2026-06-01", "score": 81, "temperature_deviation": -0.2, - "contributors": { "resting_heart_rate": 49 } } ], + "contributors": { "resting_heart_rate": 96 } } ], "daily_activity": [ { "day": "2026-06-01", "steps": 8421, "active_calories": 520 } ] } @@ -50,7 +50,7 @@ class WearableExportImporterTest { assertEquals(1, p.days.size) val d = p.days[0] assertEquals("2026-06-01", d.day) - assertEquals(49, d.restingHr) // readiness contributor wins + assertEquals(48, d.restingHr) // sleep lowest_heart_rate, never the 96 score assertEquals(-0.2, d.skinTempDevC!!, 1e-6) assertEquals(8421, d.steps) assertEquals(520.0, d.activeKcal!!, 1e-6) @@ -59,6 +59,69 @@ class WearableExportImporterTest { assertEquals(81, d.readinessScore) } + @Test + fun readinessContributorRhrScoreDoesNotBecomeRestingHr() { + // Regression (twin of the Swift OuraExportParserTests): Oura's daily_readiness + // `contributors.resting_heart_rate` is a 0-100 readiness contributor SCORE, not bpm. A prior + // bug stored it directly as the day's resting HR. With no sleep period in the export, + // restingHr must stay null — never the score — while the score itself stays reference-only. + val json = """ + { "daily_readiness": [ { "day": "2026-01-02", "score": 80, + "contributors": { "resting_heart_rate": 99 } } ] } + """ + val p = WearableExportImporter.parseOura(mapOf("export.json" to bytes(json))) + assertEquals(1, p.days.size) + assertNull(p.days[0].restingHr) + assertEquals(80, p.days[0].readinessScore) + } + + @Test + fun flatReadinessRestingHeartRateKeyDoesNotBecomeRestingHr() { + // A flat `resting_heart_rate` on the readiness record is the same contributor score in a + // flattened shape — it must not land on restingHr either. + val json = """ + { "daily_readiness": [ { "day": "2026-01-02", "score": 80, "resting_heart_rate": 97 } ] } + """ + val p = WearableExportImporter.parseOura(mapOf("export.json" to bytes(json))) + assertNull(p.days[0].restingHr) + assertEquals(80, p.days[0].readinessScore) + } + + @Test + fun sleepLowestHeartRateIsTheSoleRestingHrSource() { + // Sleep's `lowest_heart_rate` is the sole resting-HR source. The old code let the readiness + // loop overwrite the sleep-derived 48 bpm with the contributor score (96 here) whenever both + // categories were present. + val json = """ + { + "sleep": [ + { "day": "2026-01-02", "bedtime_start": "2026-01-01T23:00:00+00:00", + "bedtime_end": "2026-01-02T06:30:00+00:00", "total_sleep_duration": 25200, + "lowest_heart_rate": 48 } ], + "daily_readiness": [ + { "day": "2026-01-02", "score": 80, + "contributors": { "resting_heart_rate": 96 } } ] + } + """ + val p = WearableExportImporter.parseOura(mapOf("export.json" to bytes(json))) + val d = p.days.first { it.day == "2026-01-02" } + assertEquals(48, d.restingHr) + assertEquals(80, d.readinessScore) + } + + @Test + fun readinessAliasCategoryDoesNotWriteRestingHr() { + // The `readiness` category alias (older API-shaped exports) runs through the same loop and + // must obey the same rule. + val json = """ + { "readiness": [ { "day": "2026-01-03", "score": 74, + "contributors": { "resting_heart_rate": 100 } } ] } + """ + val p = WearableExportImporter.parseOura(mapOf("export.json" to bytes(json))) + assertNull(p.days[0].restingHr) + assertEquals(74, p.days[0].readinessScore) + } + @Test fun importedNightRespReachesDailyRow() { // #17: the night's resp rate (Oura `average_breath`) lives on the SESSION; it must also fold onto From b07e4e50854547b6d6940cc5fa8c9265ac1b9460 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 22:34:14 -0700 Subject: [PATCH 063/171] =?UTF-8?q?fix(apple):=20hourly=20backfill=20flag?= =?UTF-8?q?=20only=20after=20a=20successful=20collection=20=E2=80=94=20HK?= =?UTF-8?q?=20errors=20must=20retry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- StrandiOS/Health/HealthKitBridge.swift | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/StrandiOS/Health/HealthKitBridge.swift b/StrandiOS/Health/HealthKitBridge.swift index 7455f61df4..4128cc1b2d 100644 --- a/StrandiOS/Health/HealthKitBridge.swift +++ b/StrandiOS/Health/HealthKitBridge.swift @@ -383,7 +383,6 @@ final class HealthKitBridge: ObservableObject { let hourlyStepsBackfilled = UserDefaults.standard.bool(forKey: HealthKitBridge.hourlyStepsBackfilledKey) let hourlyStepsStart: Date = hourlyStepsBackfilled ? start : (cal.date(byAdding: .day, value: -90, to: cal.startOfDay(for: end)) ?? start) - let hourlySteps = await collectHourlySteps(start: hourlyStepsStart, end: end) // Build + upsert the store rows under the apple-health source. let appleRows = byDay.map { (day, a) in @@ -444,6 +443,10 @@ final class HealthKitBridge: ObservableObject { // lastSync — a false "success", and the next delta sync skipped the window. (Reimplemented // from @vulnix0x4's PR #375.) do { + // Collect hourly steps here (not before) so a transient HealthKit error throws to the + // catch block and prevents the backfill flag from permanently locking out the 90-day window. + let hourlySteps = try await collectHourlySteps(start: hourlyStepsStart, end: end) + try await store.upsertAppleDaily(appleRows, deviceId: appleDeviceId) try await store.upsertDailyMetrics(dmRows, deviceId: appleDeviceId) try await store.upsertMetricSeries(points, deviceId: appleDeviceId) @@ -906,9 +909,10 @@ final class HealthKitBridge: ObservableObject { /// `HKStatisticsCollectionQuery` shape but bucketed by HOUR instead of day so a backfill can show /// exactly which hours the phone was recording. `anchorDate` is local midnight — same anchor /// `collect()` uses for its daily buckets — so hour buckets fall on local-clock hour boundaries, - /// not UTC ones. Silently returns fewer (or zero) rows on a HealthKit query error, same as - /// `collect()`; the next foreground/background catch-up re-walks the window. - private func collectHourlySteps(start: Date, end: Date) async -> [(ts: Int, steps: Int)] { + /// not UTC ones. Throws on a HealthKit query error (distinguished from a genuinely-empty 90-day + /// window, which returns zero rows); a transient error will propagate to the sync's existing + /// `catch` block, preventing the backfill flag from permanently locking out the 90-day window. + private func collectHourlySteps(start: Date, end: Date) async throws -> [(ts: Int, steps: Int)] { guard let type = HKQuantityType.quantityType(forIdentifier: .stepCount) else { return [] } let cal = Calendar.current let anchor = cal.startOfDay(for: start) @@ -916,11 +920,15 @@ final class HealthKitBridge: ObservableObject { HKQuery.predicateForSamples(withStart: start, end: end, options: .strictStartDate), Self.notNoopAuthored, ]) - return await withCheckedContinuation { (cont: CheckedContinuation<[(ts: Int, steps: Int)], Never>) in + return try await withCheckedThrowingContinuation { (cont: CheckedContinuation<[(ts: Int, steps: Int)], Error>) in let q = HKStatisticsCollectionQuery(quantityType: type, quantitySamplePredicate: predicate, options: .cumulativeSum, anchorDate: anchor, intervalComponents: DateComponents(hour: 1)) - q.initialResultsHandler = { _, results, _ in + q.initialResultsHandler = { _, results, error in + if let error { + cont.resume(throwing: error) + return + } var rows: [(ts: Int, steps: Int)] = [] results?.enumerateStatistics(from: start, to: end) { stats, _ in if let sum = stats.sumQuantity() { From 492adacb861187373e80892a8e4e915cb62a3e07 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 23:40:17 -0700 Subject: [PATCH 064/171] docs(sleep): design spec for #364 night-stitch (present a wake-split night as one) --- .../2026-07-12-sleep-night-stitch-design.md | 129 ++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-12-sleep-night-stitch-design.md diff --git a/docs/superpowers/specs/2026-07-12-sleep-night-stitch-design.md b/docs/superpowers/specs/2026-07-12-sleep-night-stitch-design.md new file mode 100644 index 0000000000..918628344f --- /dev/null +++ b/docs/superpowers/specs/2026-07-12-sleep-night-stitch-design.md @@ -0,0 +1,129 @@ +# Sleep night-stitch: present a wake-split night as one night — design + +**Date:** 2026-07-12 +**Issue:** [#364](https://github.com/ryanbr/noop/issues/364) +**Scope:** Both platforms. Swift core in `Packages/StrandAnalytics` (`SleepStageTotals`) + two +app-target consumers (`StrandiOS/Health/HealthKitBridge.swift`, `Strand/Screens/SleepView.swift`); +Kotlin twins in `android/…/analytics/SleepStageTotals.kt` + `HealthConnectWriter.kt` + +`ui/SleepScreen.kt`. Detector and stored sessions are **untouched** — this is presentation only. + +## Problem + +When the sleep detector sees a sustained mid-night wake it ends the session and starts a new one, so +one night reaches storage as two `SleepSession` rows. A 16-minute wake at ~3 AM produces a main block +plus a second fragment. The daily totals are already correct — `SleepStageTotals.mainNightGroupIndices` +(#561/#525/#861) bridges fragments separated by a gap `< gapBridgeMaxMin` (60 min, plus a 90-min +night-tail bridge for overnight-onset fragments) into one **main-night group** for scoring. But two +consumers still read the raw split sessions: + +- **Apple Health write-back** (`HealthKitBridge.writeSleep`) iterates raw sessions → one `.inBed` + HealthKit entry per fragment → the reported two-entry export. +- **Sleep screen** (`SleepView` / Android `SleepScreen`) surfaces the second fragment as its own card + instead of a wake segment inside the one night. + +Prior art (WHOOP, Oura, Apple Watch) renders one night with the wake folded in as awake time; only +naps / genuinely separate sleeps split. + +## Approach + +One shared grouping/merge primitive in the analytics package; four thin consumers. The +correctness-critical logic — which fragments are one night, and the merged stage timeline with the +wake gap — lives where `swift test` / JVM tests cover it. Consumers render/emit only. + +Rejected: (B) group inline in each consumer — duplicates the bridge across four app-target surfaces +with no CI, guaranteed drift; (C) merge at the detector/storage layer — breaks sleep-time edits +(#318), union dedup, and the Apple Health round-trip. Ryan ruled out (C) explicitly on #364. + +## The primitive + +`mainNightGroupIndices` today builds bridged groups then scores and returns only the **winning** main +night. The consumers need **every** night grouped (main night merged, naps standalone, prior nights +merged), so the grouping pass ([`SleepStageTotals.swift:333-356`](../../../Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStageTotals.swift)) is extracted into a reusable, pure function: + +``` +groupBridgedNights(sessions) -> [DisplayedNight] +``` + +where each `DisplayedNight` carries only **pure grouping/time data — no stage decoding** (so the +grouping layer gains no dependency on the stage vocabulary or the HealthKit decoder): +- `fragmentIndices: [Int]` — indices into the input, ascending; +- `spanStart / spanEnd` — earliest `effectiveStartTs` → latest `endTs` across the group; +- `gapWakes: [(start, end)]` — the inter-fragment seams, one per gap (`start = prevFragment.end`, + `end = nextFragment.start`). These, folded in as `wake`, are the *only* new stage content a merged + night introduces; each fragment's own decoded stages are unchanged. + +Grouping uses the **same** `gapBridgeMaxMin` + night-tail overnight rule the scorer already applies — +no new threshold — so display/export group *consistently with the corrected daily totals*. Naps and +genuinely separate sleeps do not bridge, so they remain their own `DisplayedNight` (own card / own +HealthKit entry). Pure and deterministic; shared verbatim by the Kotlin twin for cross-platform +parity. + +The existing `mainNightGroupIndices` is refactored to call `groupBridgedNights` and then score+pick, +so there is exactly one grouping implementation and the #561 tests still pin it. + +Assembling a merged night's timeline is then "concatenate each fragment's existing decoded stages, +then splice in `gapWakes`" over `[spanStart, spanEnd]`. For the **write path** this assembly is a pure +helper in `Packages/StrandImport` `HealthWriteback` (reusing the tested `stageIntervals` per-fragment +decoder), so it gets `swift test` coverage. The **display path** does the same interleave in its +hypnogram builder (UI layer, compile-checked). A group whose fragments all lack decoded stages yields +one `asleepUnspecified` span over `[spanStart, spanEnd]` plus the `gapWakes`, exactly as `writeSleep` +degrades today. + +## The four consumers + +1. **`writeSleep`** (`HealthKitBridge`) — iterate `groupBridgedNights(sessions)` instead of raw + sessions; one `.inBed` sample over `spanStart→spanEnd` plus one `HKCategorySample` per + `mergedStages` segment (`wake→.awake`, `light→.asleepCore`, `deep→.asleepDeep`, `rem→.asleepREM`). +2. **`SleepView`** (macOS+iOS, shared) — list `DisplayedNight`s; the secondary fragment renders as a + wake segment within the night's hypnogram instead of a separate card. The delicate surface: nap + cards, sleep-time edit/undo (#318), and per-session selection all live here and must keep working — + naps stay their own `DisplayedNight`, and edit/undo still act on the underlying stored session. +3. **`HealthConnectWriter`** (Android) — twin of #1 (`SleepSessionRecord` + stage records). +4. **`SleepScreen`** (Android Compose) — twin of #2. + +## Correctness subtlety: HealthKit dedup across the 2→1 transition + +The idempotency key is `noop::sleep:` per session. A night previously written as +**two** entries must not orphan the second when it becomes **one**. So: + +- The merged entry's samples are keyed by the group's **representative** `startTs` — the earliest + fragment's immutable detected onset (not `effectiveStartTs`, so a user edit keeps identity). +- The delete-then-write predicate's `allowedValues` must include **every** fragment's per-`startTs` + key across all groups (not just the representatives), so a prior per-fragment write is fully + cleared before the merged write. + +Same care on the Health Connect side (its records are keyed analogously). + +## Verification + +- **Core primitive** — `swift test` (StrandAnalytics) + JVM test (Android `SleepStageTotals`), both in + default/fork CI. Cases: short-wake bridge merges two fragments into one night with a `wake` gap + interval at the seam; a nap stays a separate `DisplayedNight`; night-tail overnight bridge; an edited + `effectiveStartTs` moves `spanStart`; unspecified-stage fragments; a three-fragment night. Assert the + refactored `mainNightGroupIndices` still returns byte-identical groups (the #561 regression pins). +- **`HealthWriteback` merged-timeline assembly** — the pure write-path helper (concatenate decoded + fragment stages + splice `gapWakes`, degrade to `asleepUnspecified`) gets `swift test` coverage: the + seam-`wake` interval sits exactly at `[prevEnd, nextStart]`, stages stay clamped to `[spanStart, + spanEnd]`, and the multi-fragment delete-key set is complete. +- **Consumers** — app-target Swift (`NOOPiOS` + `Strand` macOS) compiled via on-demand `app-build.yml`; + Android via fork `android.yml` (the #368 play). HealthKit / Health Connect themselves are not + unit-testable. + +## Deliberately excluded + +- **Proposal 2 — motion-aware wake classification** (restless-in-bed vs out-of-bed from + `gravitySample` / `stepSample`). The gravity stream is too sparse on WHOOP 4.0 to trust (same + sparse-motion limit behind the 4.0 sleep-staging work), and per the repo's derived-biosignal + validation standard an unproven per-record signal ships as instrumentation or behind a default-off + toggle, never as the default. Deferred to an honestly-labelled 5.0/MG follow-up; the 60-min bridge + already decides "same night" for v1. +- **Detector / stored sessions** — unchanged. Sessions stay split in storage; grouping is read-time. +- **Daily totals / scoring** — already correct via the existing bridge; not touched beyond the + no-op-preserving `mainNightGroupIndices` refactor. + +## Cross-platform parity + +`groupBridgedNights` and its `DisplayedNight` output are byte-identical Swift↔Kotlin (same bridge +constants, same tie/order rules, same gap-as-wake construction). The two writers emit the same +one-span-per-night shape; the two sleep screens render the same folded hypnogram. UI is feature-level, +not pixel-level (SwiftUI Charts vs Compose Canvas), per the parity contract. From d991861660a44cd909663573703d22bc3619391e Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 23:51:07 -0700 Subject: [PATCH 065/171] docs(sleep): implementation plan for #364 night-stitch --- .../plans/2026-07-12-sleep-night-stitch.md | 575 ++++++++++++++++++ 1 file changed, 575 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-12-sleep-night-stitch.md diff --git a/docs/superpowers/plans/2026-07-12-sleep-night-stitch.md b/docs/superpowers/plans/2026-07-12-sleep-night-stitch.md new file mode 100644 index 0000000000..8fbd9aa3a2 --- /dev/null +++ b/docs/superpowers/plans/2026-07-12-sleep-night-stitch.md @@ -0,0 +1,575 @@ +# Sleep night-stitch (#364) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A night split by a short mid-night wake presents as ONE night — one `.inBed` span per bridged night in the Apple Health / Health Connect exports with the gap as a `wake` stage, and the gap rendered as a wake segment (and counted as awake minutes) in both sleep screens. + +**Architecture:** One new pure primitive in the analytics layer (`bridgedNightGroups` — the existing #561 bridge loop extracted so it returns EVERY bridged group plus each group's inter-fragment gaps), a pure merged-plan builder per write path (`HealthWriteback.mergedSleepPlan` in StrandImport; `HealthExportPlan.sleepSessions` grows grouping in Kotlin), and four thin consumers. Detector and stored sessions untouched; grouping is read-time. + +**Tech Stack:** Swift 5.9 packages (StrandAnalytics, StrandImport; `swift test` on macOS), app-target Swift (HealthKit, SwiftUI — compile via `xcodebuild`, no default CI), Kotlin/JVM (JUnit4 via `testFullDebugUnitTest` — no local toolchain, verified on fork Android CI), Health Connect SDK. + +## Global Constraints + +- Spec: `docs/superpowers/specs/2026-07-12-sleep-night-stitch-design.md`; maintainer direction on issue #364. +- Reuse `gapBridgeMaxMin` (60) + `nightTailBridgeMaxMin` (90) + `isOvernightOnset` — NO new threshold. +- The `mainNightGroupIndices` refactor must be byte-identical (the #561/#861 golden tests pin it). +- Cross-platform parity: `bridgedNightGroups` Swift↔Kotlin identical semantics; same fixtures + expected values in both test suites. +- HealthKit dedup: merged samples keyed by the group's EARLIEST fragment's immutable `startTs`; the delete predicate must carry EVERY fragment's key. Health Connect: same rule via `clientRecordId`, plus explicit `deleteRecords` for absorbed fragment ids (HC does not auto-remove). +- Spans use `effectiveStartTs` (edited onset); keys use immutable `startTs`. Kotlin `SleepInput` must gain that split (it currently conflates them). +- `Packages/` may not import AppKit/UIKit/HealthKit; StrandImport does NOT depend on StrandAnalytics (grouping happens in the caller; the plan builders take pre-grouped fragments / compute their own seams). +- No Claude attribution anywhere. Commit style: `feat(sleep): …` / `fix(sleep): …` lowercase. +- Work happens on branch `feat/sleep-night-stitch-364` off `upstream/main` in a worktree (VK's `main` is ~72 ahead of upstream; the PR must be clean). + +--- + +### Task 1: Swift analytics primitive — `bridgedNightGroups` (+ refactor `mainNightGroupIndices`) + +**Files:** +- Modify: `Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStageTotals.swift` (insert after `bridgeAdjacent`, ~line 306; refactor `mainNightGroupIndices` ~line 325) +- Test: `Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BridgedNightGroupsTests.swift` (create) + +**Interfaces:** +- Consumes: existing `NightBlock`, `gapBridgeMaxMin`, `nightTailBridgeMaxMin`, `isOvernightOnset`. +- Produces (later tasks rely on these exact names): + +```swift +public struct BridgedNightGroup: Equatable { + /// Indices into the ORIGINAL input array, ascending by effective onset. + public let indices: [Int] + /// Inter-fragment wake seams inside this group, ascending: (prev bridged end, next fragment start). + public let gaps: [GapSpan] + public struct GapSpan: Equatable { + public let start: Int, end: Int + public init(start: Int, end: Int) { self.start = start; self.end = end } + } + public init(indices: [Int], gaps: [GapSpan]) { self.indices = indices; self.gaps = gaps } +} + +/// EVERY bridged night group over `blocks` (not just the main night): the same two-tier bridge +/// `mainNightGroupIndices` applies (#561/#861), minus the winner pick. Groups ordered by start. +public static func bridgedNightGroups(_ blocks: [NightBlock], offsetSec: Int) -> [BridgedNightGroup] +``` + +- [ ] **Step 1: Write the failing tests** (`BridgedNightGroupsTests.swift`) + +```swift +import XCTest +@testable import StrandAnalytics + +/// Pins the all-groups bridge (#364): the SAME two-tier gap bridge the main-night selector applies, +/// returned for EVERY group so the Health write-back and the sleep screens can present each bridged +/// night as one night with its wake seams explicit. +final class BridgedNightGroupsTests: XCTestCase { + private typealias B = SleepStageTotals.NightBlock + // 2026-01-02 00:00 UTC for readable fixtures. + private let t0 = 1_767_312_000 + + func testShortWakeBridgesTwoFragmentsWithOneGap() { + // 23:00→02:00, 16-min wake, 02:16→06:00 — the #364 example shape. + let a = B(start: t0 - 3_600, end: t0 + 2 * 3_600) + let b = B(start: t0 + 2 * 3_600 + 16 * 60, end: t0 + 6 * 3_600) + let groups = SleepStageTotals.bridgedNightGroups([a, b], offsetSec: 0) + XCTAssertEqual(groups.count, 1) + XCTAssertEqual(groups[0].indices, [0, 1]) + XCTAssertEqual(groups[0].gaps, [.init(start: a.end, end: b.start)]) + } + + func testDaytimeNapStaysItsOwnGroup() { + let night = B(start: t0 - 3_600, end: t0 + 6 * 3_600) // 23:00→06:00 + let nap = B(start: t0 + 14 * 3_600, end: t0 + 15 * 3_600) // 14:00→15:00 + let groups = SleepStageTotals.bridgedNightGroups([night, nap], offsetSec: 0) + XCTAssertEqual(groups.map(\.indices), [[0], [1]]) + XCTAssertTrue(groups.allSatisfy { $0.gaps.isEmpty }) + } + + func testNightTailBridgesOvernightGapOverSixtyMinutes() { + // 75-min gap, second fragment onset 04:15 (overnight band) → bridged by the #861 night-tail rule. + let a = B(start: t0 - 3_600, end: t0 + 3 * 3_600) + let b = B(start: t0 + 3 * 3_600 + 75 * 60, end: t0 + 7 * 3_600) + let groups = SleepStageTotals.bridgedNightGroups([a, b], offsetSec: 0) + XCTAssertEqual(groups.map(\.indices), [[0, 1]]) + XCTAssertEqual(groups[0].gaps, [.init(start: a.end, end: b.start)]) + } + + func testThreeFragmentNightYieldsTwoGaps() { + let a = B(start: t0, end: t0 + 3_600) + let b = B(start: t0 + 3_600 + 600, end: t0 + 2 * 3_600) + let c = B(start: t0 + 2 * 3_600 + 900, end: t0 + 3 * 3_600) + let groups = SleepStageTotals.bridgedNightGroups([a, b, c], offsetSec: 0) + XCTAssertEqual(groups.map(\.indices), [[0, 1, 2]]) + XCTAssertEqual(groups[0].gaps, [.init(start: a.end, end: b.start), + .init(start: b.end, end: c.start)]) + } + + func testContainedFragmentProducesNoGap() { + // b sits inside a (gap < 0): bridged, no seam emitted. + let a = B(start: t0, end: t0 + 4 * 3_600) + let b = B(start: t0 + 3_600, end: t0 + 2 * 3_600) + let groups = SleepStageTotals.bridgedNightGroups([a, b], offsetSec: 0) + XCTAssertEqual(groups.map(\.indices), [[0, 1]]) + XCTAssertTrue(groups[0].gaps.isEmpty) + } + + func testUnsortedInputIndicesReferOriginalPositions() { + let late = B(start: t0 + 2 * 3_600 + 16 * 60, end: t0 + 6 * 3_600) + let early = B(start: t0 - 3_600, end: t0 + 2 * 3_600) + let groups = SleepStageTotals.bridgedNightGroups([late, early], offsetSec: 0) + XCTAssertEqual(groups.map(\.indices), [[0, 1]]) // ascending original indices [1(early)…] sorted → [0,1]? NO: + // indices are into the ORIGINAL array: early is index 1, late is index 0 → group indices sorted ascending [0, 1]. + XCTAssertEqual(groups[0].gaps, [.init(start: early.end, end: late.start)]) + } + + func testEmptyInputYieldsNoGroups() { + XCTAssertTrue(SleepStageTotals.bridgedNightGroups([], offsetSec: 0).isEmpty) + } + + /// The refactor guard: the main-night pick over the new all-groups pass must stay byte-identical + /// to the existing golden behaviour (the #561 suite pins the rest). + func testMainNightGroupIndicesUnchangedForBiphasicPlusNap() { + let a = B(start: t0 - 3_600, end: t0 + 2 * 3_600) + let b = B(start: t0 + 2 * 3_600 + 16 * 60, end: t0 + 6 * 3_600) + let nap = B(start: t0 + 14 * 3_600, end: t0 + 15 * 3_600) + XCTAssertEqual(SleepStageTotals.mainNightGroupIndices([a, b, nap], offsetSec: 0), [0, 1]) + } +} +``` + +- [ ] **Step 2: Run to verify they fail** — `swift test --package-path Packages/StrandAnalytics --filter BridgedNightGroupsTests` → FAIL: `bridgedNightGroups` unresolved. + +- [ ] **Step 3: Implement** — insert `BridgedNightGroup` + `bridgedNightGroups` after `bridgeAdjacent` (~line 306). Extract the existing loop from `mainNightGroupIndices` verbatim, adding gap capture: + +```swift + /// One bridged night group over the whole input: the fragments (as ORIGINAL indices) plus the + /// inter-fragment wake seams. Produced by `bridgedNightGroups` for the write-back / display + /// consumers that must present a briefly-interrupted night as ONE night (#364). + public struct BridgedNightGroup: Equatable { + public let indices: [Int] + public let gaps: [GapSpan] + public struct GapSpan: Equatable { + public let start: Int, end: Int + public init(start: Int, end: Int) { self.start = start; self.end = end } + } + public init(indices: [Int], gaps: [GapSpan]) { self.indices = indices; self.gaps = gaps } + } + + /// EVERY bridged group over `blocks` — the same two-tier bridge `mainNightGroupIndices` applies + /// (#561 short-wake + #861 overnight night-tail), WITHOUT the winner pick, so the Health + /// write-back and the sleep screens group ALL nights exactly as the day totals do. Each group's + /// `gaps` are the seams between consecutive bridged fragments (empty when fragments touch or + /// overlap) — folded in as `wake` by the consumers. Groups ordered by start; indices are into + /// the ORIGINAL array, ascending. Pure + deterministic; Kotlin twin `bridgedNightGroups`. (#364) + public static func bridgedNightGroups(_ blocks: [NightBlock], offsetSec: Int) -> [BridgedNightGroup] { + guard !blocks.isEmpty else { return [] } + let order = blocks.indices.sorted { blocks[$0].start < blocks[$1].start } + let bridgeS = gapBridgeMaxMin * 60 + let nightTailBridgeS = nightTailBridgeMaxMin * 60 + var bridged: [NightBlock] = [] + var groups: [[Int]] = [] + var gaps: [[BridgedNightGroup.GapSpan]] = [] + for idx in order { + let b = blocks[idx] + if let last = bridged.last { + let gap = b.start - last.end + let bridges = gap >= 0 + && (gap < bridgeS + || (gap < nightTailBridgeS && isOvernightOnset(b.start, offsetSec: offsetSec))) + if bridges { + if gap > 0 { + gaps[gaps.count - 1].append(.init(start: last.end, end: b.start)) + } + bridged[bridged.count - 1] = NightBlock(start: last.start, end: max(last.end, b.end)) + groups[groups.count - 1].append(idx) + continue + } + // A CONTAINED fragment (negative gap) still belongs to the night it sits inside. + if gap < 0 { + bridged[bridged.count - 1] = NightBlock(start: last.start, end: max(last.end, b.end)) + groups[groups.count - 1].append(idx) + continue + } + } + bridged.append(b) + groups.append([idx]) + gaps.append([]) + } + return zip(groups, gaps).map { BridgedNightGroup(indices: $0.sorted(), gaps: $1) } + } +``` + + **CAREFUL — byte-identity check before refactoring `mainNightGroupIndices`:** the existing loop treats `gap < 0` via `gap >= 0 &&` → a contained fragment does NOT bridge there and starts its own bridged span. Verify against the #561 golden tests what the current behaviour is for overlapping fragments; if the existing suite pins "negative gap → separate group", DROP the `gap < 0` containment branch above and emit a `GapSpan` only when `gap > 0` (matching `testContainedFragmentProducesNoGap` would then need `[[0],[1]]` — adjust that one test to pin the ACTUAL legacy semantics). The refactored `mainNightGroupIndices` becomes: + +```swift + public static func mainNightGroupIndices(_ blocks: [NightBlock], offsetSec: Int, + habitualMidsleepSec: Int? = nil) -> [Int]? { + guard !blocks.isEmpty else { return nil } + let all = bridgedNightGroups(blocks, offsetSec: offsetSec) + let bridgedSpans = all.map { g -> NightBlock in + let s = g.indices.map { blocks[$0].start }.min() ?? 0 + let e = g.indices.map { blocks[$0].end }.max() ?? 0 + return NightBlock(start: s, end: e) + } + guard let winner = mainNightIndex(bridgedSpans, offsetSec: offsetSec, + habitualMidsleepSec: habitualMidsleepSec) else { return nil } + return all[winner].indices + } +``` + + Only perform this refactor if the FULL StrandAnalytics suite stays green; otherwise keep the original body and have `bridgedNightGroups` be a sibling (duplicated loop, one comment cross-referencing) — byte-identity of the pick outranks DRY here. + +- [ ] **Step 4: Run** — `swift test --package-path Packages/StrandAnalytics` → ALL green (new + #561/#861 goldens). +- [ ] **Step 5: Commit** — `feat(sleep): bridgedNightGroups — the #561 bridge over EVERY night, with wake seams (#364)` + +--- + +### Task 2: Kotlin analytics twin — `bridgedNightGroups` + +**Files:** +- Modify: `android/app/src/main/java/com/noop/analytics/SleepStageTotals.kt` (after `bridgeAdjacent`, ~line 246) +- Test: `android/app/src/test/java/com/noop/analytics/SleepStageTotalsTest.kt` (append; file exists) + +**Interfaces (produces):** + +```kotlin +data class BridgedNightGroup(val indices: List, val gaps: List>) +fun bridgedNightGroups(blocks: List, offsetSec: Long): List +``` + +- [ ] **Step 1: Transcribe the implementation** — same loop as Swift (Long timestamps), same doc comment marking it the Swift twin, same containment decision as resolved in Task 1 Step 3. Refactor Kotlin `mainNightGroupIndices` the same way ONLY if the Kotlin golden tests are also confirmed green on CI at the end; otherwise sibling. +- [ ] **Step 2: Transcribe the tests** — same seven cases, same fixture timestamps (`1_767_312_000L`), same expected indices/gaps, JUnit4 `assertEquals`. Local red/green is impossible (no JVM toolchain) — parity with the Swift twin (which DID fail-then-pass) is the evidence; fork Android CI at Task 8 is the executable check. Note this deviation in the PR. +- [ ] **Step 3: Commit** — `feat(sleep): kotlin bridgedNightGroups twin (#364)` + +--- + +### Task 3: StrandImport — `HealthWriteback.mergedSleepPlan` + +**Files:** +- Modify: `Packages/StrandImport/Sources/StrandImport/HealthWriteback.swift` +- Test: `Packages/StrandImport/Tests/StrandImportTests/HealthWritebackTests.swift` (exists as the stageIntervals suite — append; if the file is named differently, `grep -rln stageIntervals Packages/StrandImport/Tests`) + +**Interfaces:** +- Consumes: existing `stageIntervals`, `StageInterval`, `StageKind`. +- Produces: + +```swift +public enum HealthWriteback { + // StageKind gains: case unspecified (bridge maps it to .asleepUnspecified) + + public struct SleepFragment: Equatable { + public let startTs: Int // immutable detected key + public let effectiveStartTs: Int // edited onset when present + public let endTs: Int + public let stagesJSON: String? + public init(startTs: Int, effectiveStartTs: Int, endTs: Int, stagesJSON: String?) + } + public struct MergedSleepEntry: Equatable { + public let keyStartTs: Int // group representative = earliest fragment's startTs + public let spanStart: Int // earliest effectiveStartTs + public let spanEnd: Int // latest endTs + public let intervals: [StageInterval] // fragments' stages (unspecified when undecodable) + wake seams + public let allKeyStartTs: [Int] // every fragment's startTs — the delete-key set + } + /// `groups`: fragments already grouped by `SleepStageTotals.bridgedNightGroups` (each inner + /// array ascending by effectiveStartTs). + public static func mergedSleepPlan(groups: [[SleepFragment]]) -> [MergedSleepEntry] +} +``` + +- [ ] **Step 1: Write the failing tests** (append to the writeback suite) + +```swift + // MARK: - mergedSleepPlan (#364 night-stitch) + + private func frag(_ start: Int, _ end: Int, stages: String? = nil, eff: Int? = nil) -> HealthWriteback.SleepFragment { + .init(startTs: start, effectiveStartTs: eff ?? start, endTs: end, stagesJSON: stages) + } + private func stagesJSON(_ segs: [(Int, Int, String)]) -> String { + "[" + segs.map { "{\"start\":\($0.0),\"end\":\($0.1),\"stage\":\"\($0.2)\"}" }.joined(separator: ",") + "]" + } + + func testMergedPlanFoldsTwoFragmentsWithWakeSeam() { + let t = 1_767_312_000 + let a = frag(t, t + 7_200, stages: stagesJSON([(t, t + 7_200, "light")])) + let b = frag(t + 7_200 + 960, t + 14_400, stages: stagesJSON([(t + 7_200 + 960, t + 14_400, "deep")])) + let plan = HealthWriteback.mergedSleepPlan(groups: [[a, b]]) + XCTAssertEqual(plan.count, 1) + let e = plan[0] + XCTAssertEqual(e.keyStartTs, t) + XCTAssertEqual(e.spanStart, t) + XCTAssertEqual(e.spanEnd, t + 14_400) + XCTAssertEqual(e.allKeyStartTs, [t, t + 7_200 + 960]) + // The seam sits EXACTLY on [prev.end, next.effectiveStart] as .awake. + XCTAssertTrue(e.intervals.contains(.init(start: t + 7_200, end: t + 7_200 + 960, kind: .awake))) + XCTAssertEqual(e.intervals.first?.kind, .light) + XCTAssertEqual(e.intervals.last?.kind, .deep) + } + + func testMergedPlanSingleFragmentMatchesLegacyShape() { + let t = 1_767_312_000 + let a = frag(t, t + 7_200, stages: stagesJSON([(t, t + 3_600, "light"), (t + 3_600, t + 7_200, "rem")])) + let e = HealthWriteback.mergedSleepPlan(groups: [[a]])[0] + XCTAssertEqual(e.keyStartTs, t) + XCTAssertEqual(e.allKeyStartTs, [t]) + XCTAssertEqual(e.intervals, HealthWriteback.stageIntervals(stagesJSON: a.stagesJSON, + sessionStart: t, sessionEnd: t + 7_200)) + } + + func testMergedPlanUnstagedFragmentsDegradeToUnspecifiedPlusSeam() { + let t = 1_767_312_000 + let a = frag(t, t + 7_200) // no stagesJSON + let b = frag(t + 7_200 + 960, t + 14_400) // no stagesJSON + let e = HealthWriteback.mergedSleepPlan(groups: [[a, b]])[0] + XCTAssertEqual(e.intervals, [ + .init(start: t, end: t + 7_200, kind: .unspecified), + .init(start: t + 7_200, end: t + 7_200 + 960, kind: .awake), + .init(start: t + 7_200 + 960, end: t + 14_400, kind: .unspecified), + ]) + } + + func testMergedPlanEditedOnsetMovesSpanButNotKey() { + let t = 1_767_312_000 + let a = frag(t, t + 7_200, eff: t + 600) // user moved bedtime 10 min later + let e = HealthWriteback.mergedSleepPlan(groups: [[a]])[0] + XCTAssertEqual(e.keyStartTs, t) // immutable key + XCTAssertEqual(e.spanStart, t + 600) // edited onset drives the span + } +``` + +- [ ] **Step 2: Run to verify failure** — `swift test --package-path Packages/StrandImport --filter HealthWriteback` → FAIL (`SleepFragment` unresolved). +- [ ] **Step 3: Implement** — add `case unspecified` to `StageKind`; add the two structs; implement: + +```swift + /// Fold one bridged night group (#364) into a single write-back entry: the span, the merged + /// stage timeline (each fragment's decoded stages, or one honest `.unspecified` block for a + /// fragment with no timing, with every inter-fragment seam as `.awake`), the representative + /// immutable key, and the COMPLETE per-fragment key set so a night previously exported as two + /// entries deletes both before the merged write. + public static func mergedSleepPlan(groups: [[SleepFragment]]) -> [MergedSleepEntry] { + var out: [MergedSleepEntry] = [] + for group in groups where !group.isEmpty { + let frags = group.sorted { $0.effectiveStartTs < $1.effectiveStartTs } + var intervals: [StageInterval] = [] + var prevEnd: Int? = nil + for f in frags { + guard f.endTs > f.effectiveStartTs else { continue } + if let p = prevEnd, f.effectiveStartTs > p { + intervals.append(StageInterval(start: p, end: f.effectiveStartTs, kind: .awake)) + } + let stages = stageIntervals(stagesJSON: f.stagesJSON, + sessionStart: f.effectiveStartTs, sessionEnd: f.endTs) + if stages.isEmpty { + intervals.append(StageInterval(start: f.effectiveStartTs, end: f.endTs, kind: .unspecified)) + } else { + intervals.append(contentsOf: stages) + } + prevEnd = max(prevEnd ?? f.endTs, f.endTs) + } + guard let first = frags.first, let spanEnd = frags.map(\.endTs).max(), spanEnd > first.effectiveStartTs + else { continue } + out.append(MergedSleepEntry( + keyStartTs: frags.map(\.startTs).min() ?? first.startTs, + spanStart: first.effectiveStartTs, + spanEnd: spanEnd, + intervals: intervals, + allKeyStartTs: frags.map(\.startTs).sorted())) + } + return out + } +``` + +- [ ] **Step 4: Run** — full `swift test --package-path Packages/StrandImport` green (existing stageIntervals suite must not regress from the `StageKind` addition — it's additive). +- [ ] **Step 5: Commit** — `feat(sleep): mergedSleepPlan — one write-back entry per bridged night, seams as wake (#364)` + +--- + +### Task 4: iOS consumer — `HealthKitBridge.writeSleep` groups by bridged night + +**Files:** +- Modify: `StrandiOS/Health/HealthKitBridge.swift:600-655` (`writeSleep`) + +**Interfaces:** consumes `SleepStageTotals.bridgedNightGroups` + `HealthWriteback.mergedSleepPlan`. Check the file's imports; add `import StrandAnalytics` if missing (the app target links it — SleepView already imports it). + +- [ ] **Step 1: Rewrite `writeSleep`** (doc comment updated to describe the one-entry-per-night shape and the #364 rationale): + +```swift + private func writeSleep(sessions: [CachedSleepSession]) async throws { + guard let type = HKObjectType.categoryType(forIdentifier: .sleepAnalysis), + store.authorizationStatus(for: type) == .sharingAuthorized else { return } + // Group fragments into bridged nights (#364): a night split by a brief mid-night wake exports + // as ONE .inBed span with the gap as an in-hypnogram wake segment — matching what the daily + // totals already score (#561) and what Oura / Apple Watch write. Naps never bridge. + let blocks = sessions.map { SleepStageTotals.NightBlock(start: $0.effectiveStartTs, end: $0.endTs) } + let offsetSec = TimeZone.current.secondsFromGMT() + let groups = SleepStageTotals.bridgedNightGroups(blocks, offsetSec: offsetSec).map { g in + g.indices.map { i -> HealthWriteback.SleepFragment in + let s = sessions[i] + return .init(startTs: s.startTs, effectiveStartTs: s.effectiveStartTs, + endTs: s.endTs, stagesJSON: s.stagesJSON) + } + } + var samples: [HKCategorySample] = [] + var keys: [String] = [] + for entry in HealthWriteback.mergedSleepPlan(groups: groups) { + let key = "noop:\(noopDeviceId):sleep:\(entry.keyStartTs)" + let meta = [HKMetadataKeyExternalUUID: key] + // Delete keys cover EVERY fragment, so a night previously written as two entries fully + // clears when it becomes one (the absorbed fragment's old key would otherwise orphan). + keys.append(contentsOf: entry.allKeyStartTs.map { "noop:\(noopDeviceId):sleep:\($0)" }) + samples.append(HKCategorySample(type: type, value: HKCategoryValueSleepAnalysis.inBed.rawValue, + start: Date(timeIntervalSince1970: TimeInterval(entry.spanStart)), + end: Date(timeIntervalSince1970: TimeInterval(entry.spanEnd)), + metadata: meta)) + for seg in entry.intervals { + let value: HKCategoryValueSleepAnalysis + switch seg.kind { + case .awake: value = .awake + case .light: value = .asleepCore + case .deep: value = .asleepDeep + case .rem: value = .asleepREM + case .unspecified: value = .asleepUnspecified + } + samples.append(HKCategorySample( + type: type, value: value.rawValue, + start: Date(timeIntervalSince1970: TimeInterval(seg.start)), + end: Date(timeIntervalSince1970: TimeInterval(seg.end)), + metadata: meta)) + } + } + guard !samples.isEmpty else { return } + let pred = NSCompoundPredicate(andPredicateWithSubpredicates: [ + HKQuery.predicateForObjects(from: HKSource.default()), + HKQuery.predicateForObjects(withMetadataKey: HKMetadataKeyExternalUUID, allowedValues: keys), + ]) + _ = try? await store.deleteObjects(of: type, predicate: pred) + try await store.save(samples) + } +``` + +- [ ] **Step 2: Compile** — `xcodegen generate && xcodebuild -project Strand.xcodeproj -scheme NOOPiOS -destination 'generic/platform=iOS' CODE_SIGNING_ALLOWED=NO build` (HealthKitBridge is iOS-only). Expected: BUILD SUCCEEDED. +- [ ] **Step 3: Commit** — `fix(health): export a wake-split night as ONE Apple Health entry with the gap as awake (#364)` + +--- + +### Task 5: macOS/iOS display — gap-wake segments + awake minutes in `SleepView.mergeDay` + +**Files:** +- Modify: `Strand/Screens/SleepView.swift:1981-2019` (`mergeDay`) + +- [ ] **Step 1: Splice the seams** — inside `mergeDay`, after the per-fragment loop and before `guard stages.asleep > 0`: + +```swift + // #364: the inter-fragment wake seams belong to the night — draw them as wake segments and + // count them as awake minutes, so the hero hypnogram has no hole and the card's minutes match + // the day totals (which already add the gap via interFragmentAwakeSeconds, #777/#705). + let ordered = Array(group) + for (prev, next) in zip(ordered, ordered.dropFirst()) { + let gapStart = prev.endTs, gapEnd = next.effectiveStartTs + guard gapEnd > gapStart else { continue } + stages.awake += Double(gapEnd - gapStart) / 60.0 + segs.append(SleepInterval(stage: .awake, start: TimeInterval(gapStart), end: TimeInterval(gapEnd))) + } +``` + + **Pin the exact `SleepInterval` init at implementation time** (`grep -n "struct SleepInterval" Strand/`): match its stage enum case (`.awake` vs `.wake`) and start/end types (TimeInterval vs Int) to the real definition — the snippet above assumes `TimeInterval`; adjust to whatever `decodeSegments` appends. `ordered` must use the SAME `group` (post-stub-drop) the fragment loop used. Gap uses `prev.endTs` (not running max) because group fragments are ascending and non-overlapping in practice; if Task 1 resolved containment as "bridged", guard with `guard gapEnd > gapStart` (already present) so contained fragments emit nothing. +- [ ] **Step 2: Check the in-bed subtitle** — find `stageCard`'s subtitle source (~line 635). If it renders `night.stages.total`, the gap now counts (correct — matches engine). If it renders the onset→wake span, it already included the gap. Either way NO further edit; just confirm which and note it in the PR body. +- [ ] **Step 3: Compile BOTH apps** — `xcodebuild -scheme Strand -destination 'platform=macOS' CODE_SIGNING_ALLOWED=NO build` AND the NOOPiOS build from Task 4 Step 2 (SleepView is shared). +- [ ] **Step 4: Commit** — `feat(sleep): draw the night's wake seams in the hero hypnogram + count them as awake (#364)` + +--- + +### Task 6: Android write path — grouping in `HealthExportPlan` + absorbed-id deletion in `HealthConnectWriter` + +**Files:** +- Modify: `android/app/src/main/java/com/noop/ingest/HealthExportPlan.kt` (sleep section, lines 77-128) +- Modify: `android/app/src/main/java/com/noop/ingest/HealthConnectWriter.kt` (`writeSleep`, ~lines 199-227) +- Test: `android/app/src/test/java/com/noop/ingest/HealthExportPlanTest.kt` (append) + +**Interfaces:** + +```kotlin +// SleepInput gains the immutable key / effective-onset split (iOS parity): +data class SleepInput(val keyStartTs: Long, val startTs: Long, val endTs: Long, val stagesJSON: String?) +data class SleepPlan( + val clientId: String, // "noop-sleep-" + val startSec: Long, val endSec: Long, + val stages: List, // fragments' AWAKE/SLEEPING plans + seams as StagePlan(asleep=false) + val absorbedClientIds: List, // non-representative fragments' old ids → writer deletes +) +fun sleepSessions(sessions: List, nowSec: Long, offsetSec: Long): List +``` + +- [ ] **Step 1: Write the JVM tests** (append to `HealthExportPlanTest.kt`; same fixture numbers as Task 3): + +```kotlin + @Test + fun sleepPlanFoldsBridgedFragmentsWithAwakeSeam() { + val t = 1_767_312_000L + val a = HealthExportPlan.SleepInput(t, t, t + 7_200, + """[{"start":$t,"end":${t + 7_200},"stage":"light"}]""") + val b = HealthExportPlan.SleepInput(t + 8_160, t + 8_160, t + 14_400, + """[{"start":${t + 8_160},"end":${t + 14_400},"stage":"deep"}]""") + val plans = HealthExportPlan.sleepSessions(listOf(a, b), nowSec = t + 86_400, offsetSec = 0) + assertEquals(1, plans.size) + val p = plans[0] + assertEquals("noop-sleep-$t", p.clientId) + assertEquals(t, p.startSec); assertEquals(t + 14_400, p.endSec) + assertEquals(listOf("noop-sleep-${t + 8_160}"), p.absorbedClientIds) + assertTrue(p.stages.contains(HealthExportPlan.StagePlan(t + 7_200, t + 8_160, asleep = false))) + } + + @Test + fun napStaysItsOwnPlanWithNoAbsorbedIds() { + val t = 1_767_312_000L + val night = HealthExportPlan.SleepInput(t, t, t + 21_600, null) + val nap = HealthExportPlan.SleepInput(t + 50_400, t + 50_400, t + 54_000, null) + val plans = HealthExportPlan.sleepSessions(listOf(night, nap), nowSec = t + 86_400, offsetSec = 0) + assertEquals(2, plans.size) + assertTrue(plans.all { it.absorbedClientIds.isEmpty() }) + } + + @Test + fun editedOnsetMovesSpanButClientIdStaysImmutable() { + val t = 1_767_312_000L + val edited = HealthExportPlan.SleepInput(t, t + 600, t + 7_200, null) + val p = HealthExportPlan.sleepSessions(listOf(edited), nowSec = t + 86_400, offsetSec = 0)[0] + assertEquals("noop-sleep-$t", p.clientId) + assertEquals(t + 600, p.startSec) + } +``` + +- [ ] **Step 2: Implement** — group via `com.noop.analytics.SleepStageTotals.bridgedNightGroups` (map inputs to `NightBlock(startTs, endTs)` — effective onset), fold each group: seams → `StagePlan(gapStart, gapEnd, asleep = false)`; unstaged fragments contribute no stages (HC convention unchanged); clientId from the earliest fragment's `keyStartTs`; absorbedClientIds = the rest. Update `writeSleep` call site: map `SleepInput(it.startTs, it.effectiveStartTs ?: it.startTs, it.endTs, it.stagesJSON)` — pin the actual Room entity's adjusted-onset field name (`grep -n "effectiveStart\|startTsAdjusted" android/app/src/main/java/com/noop/data/` at implementation). After `insertChunked`, delete absorbed ids: + +```kotlin + val absorbed = plans.flatMap { it.absorbedClientIds } + if (absorbed.isNotEmpty()) { + client.deleteRecords(SleepSessionRecord::class, recordIdsList = emptyList(), clientRecordIdsList = absorbed) + } +``` + +- [ ] **Step 3: Commit** — `fix(health): android — one Health Connect session per bridged night, absorbed fragments deleted (#364)` + +--- + +### Task 7: Android display — seams in `selectNight` + +**Files:** +- Modify: `android/app/src/main/java/com/noop/ui/SleepScreen.kt:2827-2851` (`groupSegments` / `sumGroupStages` / `groupInBedMin`) + +- [ ] **Step 1: Splice seams into `groupSegments`** — after the flatMap, insert a `PersistedSegment(stage = "wake", start = prev.endTs, end = next.effectiveStartTs)` between consecutive `heroGroup` fragments (guard `end > start`; pin `PersistedSegment`'s exact constructor by grepping its definition). Add the same seam minutes to `sumGroupStages`'s awake figure and to `groupInBedMin` (gaps now count — matching the engine's `interFragmentAwakeSeconds` day totals and iOS's span-based in-bed). +- [ ] **Step 2: Commit** — `feat(sleep): android — wake seams in the hero hypnogram + gap-inclusive minutes (#364)` + +--- + +### Task 8: Verification sweep + +- [ ] `swift test` — StrandAnalytics, StrandImport (and StrandDesign/WhoopStore if touched — they shouldn't be). +- [ ] `xcodegen generate && xcodebuild` — `Strand` (macOS) AND `NOOPiOS` (iOS), both `CODE_SIGNING_ALLOWED=NO build`. +- [ ] Push branch to fork; dispatch `android.yml` (assemble + JVM tests — covers Tasks 2/6 red-free) and `app-build.yml` (both app targets on clean `project.yml`). +- [ ] `swift-packages.yml` runs automatically on the PR. + +### Task 9: PR + +- [ ] One PR to `ryanbr/noop` from `vishk23:feat/sleep-night-stitch-364`, title `fix(sleep): present a wake-split night as one night — export + hypnogram, both platforms (#364)`. Body: Ryan's recommended shape restated (display-time via the existing bridge, `gapBridgeMaxMin` reused, one `.inBed`/`SleepSessionRecord` per bridged night, seams as wake), the dedup 2→1 rule, the in-bed/awake minutes now matching the engine's #777/#705 totals, Proposal 2 deferred, verification section (swift tests, both app builds, fork CI links). No attribution. + +### Task 10: Sync local main + +- [ ] `git merge feat/sleep-night-stitch-364` into VK's `main` (keeps his device builds current); run `swift test` on the two packages post-merge. From 7a7e8139f45e42ff196ba959c8e5ef12c9e5995e Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 23:55:37 -0700 Subject: [PATCH 066/171] =?UTF-8?q?feat(sleep):=20bridgedNightGroups=20?= =?UTF-8?q?=E2=80=94=20the=20#561=20bridge=20over=20EVERY=20night,=20with?= =?UTF-8?q?=20wake=20seams=20(#364)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../StrandAnalytics/SleepStageTotals.swift | 82 ++++++++++---- .../BridgedNightGroupsTests.swift | 103 ++++++++++++++++++ 2 files changed, 163 insertions(+), 22 deletions(-) create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BridgedNightGroupsTests.swift diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStageTotals.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStageTotals.swift index 86e0904170..28cda1fca7 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStageTotals.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStageTotals.swift @@ -305,26 +305,29 @@ public enum SleepStageTotals { return out } - /// The indices (into the ORIGINAL `blocks`) of the MAIN-NIGHT GROUP: the main night plus any adjacent - /// fragments bridged into it. A biphasic / briefly-interrupted main sleep that reaches the selector still - /// split into two blocks (a wake gap shorter than `gapBridgeMaxMin` between them) is scored as ONE night - /// rather than two competing fragments, then the winning bridged group's fragments are ALL returned so the - /// caller can SUM their stages for the day's headline figure. (#561) - /// - /// Pipeline: - /// 1. `bridgeAdjacent` merges blocks whose gap is in `[0, gapBridgeMaxMin*60)` into bridged spans, in - /// `start` order, recording which original indices fell into each bridged group. - /// 2. `mainNightIndex` scores the BRIDGED spans (so a two-fragment night's combined span out-scores a - /// lone nap) and picks the winning bridged group. - /// 3. The original indices of that winning group are returned, ascending. - /// - /// Returns nil only for an empty list. A day with no bridgeable gap collapses to the single-block group the - /// bare `mainNightIndex` would pick — byte-identical to the old behaviour for the common case. Pure + - /// deterministic; shares `bridgeAdjacent` + `mainNightIndex` so the bridged pick stays cross-platform - /// stable. (#561) - public static func mainNightGroupIndices(_ blocks: [NightBlock], offsetSec: Int, - habitualMidsleepSec: Int? = nil) -> [Int]? { - guard !blocks.isEmpty else { return nil } + /// One bridged night group over the whole input: the fragments (as ORIGINAL indices, ascending) + /// plus the group's inter-fragment wake seams. Produced by `bridgedNightGroups` for the consumers + /// that must present a briefly-interrupted night as ONE night — the Health write-back and the + /// sleep screens — so they group exactly as the day totals do and can fold each seam in as an + /// explicit `wake` segment. (#364) + public struct BridgedNightGroup: Equatable { + public let indices: [Int] + public let gaps: [GapSpan] + /// One inter-fragment seam: (previous bridged end, next fragment start), end > start always. + public struct GapSpan: Equatable { + public let start: Int, end: Int + public init(start: Int, end: Int) { self.start = start; self.end = end } + } + public init(indices: [Int], gaps: [GapSpan]) { self.indices = indices; self.gaps = gaps } + } + + /// EVERY bridged group over `blocks` — the same two-tier bridge `mainNightGroupIndices` applies + /// (#561 short-wake, plus the #861 overnight night-tail widening), WITHOUT the winner pick. A + /// negative gap (a block starting inside the previous span) does not bridge — pinned legacy + /// semantics, `gap >= 0` — and never fabricates a seam. Groups ordered by start; pure and + /// deterministic; Kotlin twin `bridgedNightGroups`. (#364) + public static func bridgedNightGroups(_ blocks: [NightBlock], offsetSec: Int) -> [BridgedNightGroup] { + guard !blocks.isEmpty else { return [] } // Sort indices by onset so bridging sees neighbours, exactly as `bridgeAdjacent` sorts the blocks. let order = blocks.indices.sorted { blocks[$0].start < blocks[$1].start } let bridgeS = gapBridgeMaxMin * 60 @@ -332,6 +335,7 @@ public enum SleepStageTotals { // Build the bridged spans AND the original indices that compose each one, in one pass over `order`. var bridged: [NightBlock] = [] var groups: [[Int]] = [] + var gaps: [[BridgedNightGroup.GapSpan]] = [] for idx in order { let b = blocks[idx] if let last = bridged.last { @@ -346,6 +350,9 @@ public enum SleepStageTotals { && (gap < bridgeS || (gap < nightTailBridgeS && isOvernightOnset(b.start, offsetSec: offsetSec))) if bridges { + if gap > 0 { + gaps[gaps.count - 1].append(.init(start: last.end, end: b.start)) + } bridged[bridged.count - 1] = NightBlock(start: last.start, end: max(last.end, b.end)) groups[groups.count - 1].append(idx) continue @@ -353,10 +360,41 @@ public enum SleepStageTotals { } bridged.append(b) groups.append([idx]) + gaps.append([]) } - guard let winner = mainNightIndex(bridged, offsetSec: offsetSec, + return zip(groups, gaps).map { BridgedNightGroup(indices: $0.sorted(), gaps: $1) } + } + + /// The indices (into the ORIGINAL `blocks`) of the MAIN-NIGHT GROUP: the main night plus any adjacent + /// fragments bridged into it. A biphasic / briefly-interrupted main sleep that reaches the selector still + /// split into two blocks (a wake gap shorter than `gapBridgeMaxMin` between them) is scored as ONE night + /// rather than two competing fragments, then the winning bridged group's fragments are ALL returned so the + /// caller can SUM their stages for the day's headline figure. (#561) + /// + /// Pipeline: + /// 1. `bridgedNightGroups` merges blocks whose gap is in `[0, gapBridgeMaxMin*60)` (or the #861 + /// night-tail window) into bridged groups, in `start` order. + /// 2. `mainNightIndex` scores the BRIDGED spans (so a two-fragment night's combined span out-scores a + /// lone nap) and picks the winning bridged group. + /// 3. The original indices of that winning group are returned, ascending. + /// + /// Returns nil only for an empty list. A day with no bridgeable gap collapses to the single-block group the + /// bare `mainNightIndex` would pick — byte-identical to the old behaviour for the common case. Pure + + /// deterministic; shares the `bridgedNightGroups` pass + `mainNightIndex` so the bridged pick stays + /// cross-platform stable. (#561) + public static func mainNightGroupIndices(_ blocks: [NightBlock], offsetSec: Int, + habitualMidsleepSec: Int? = nil) -> [Int]? { + guard !blocks.isEmpty else { return nil } + let all = bridgedNightGroups(blocks, offsetSec: offsetSec) + // Rebuild each group's bridged span for scoring: sorted-ascending fragments make the span + // (first start, running-max end) — identical to the span the one-pass loop accumulated. + let bridgedSpans = all.map { g -> NightBlock in + NightBlock(start: g.indices.map { blocks[$0].start }.min() ?? 0, + end: g.indices.map { blocks[$0].end }.max() ?? 0) + } + guard let winner = mainNightIndex(bridgedSpans, offsetSec: offsetSec, habitualMidsleepSec: habitualMidsleepSec) else { return nil } - return groups[winner].sorted() + return all[winner].indices } /// Index of the day's MAIN night among `blocks`, by the LEARNED-TIMING SCORE (replaces the old hard diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BridgedNightGroupsTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BridgedNightGroupsTests.swift new file mode 100644 index 0000000000..2396335fd4 --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BridgedNightGroupsTests.swift @@ -0,0 +1,103 @@ +import XCTest +@testable import StrandAnalytics + +/// Pins the all-groups bridge (#364): the SAME two-tier gap bridge the main-night selector applies +/// (#561 short-wake + #861 overnight night-tail), returned for EVERY group with each group's +/// inter-fragment wake seams explicit — so the Health write-back and the sleep screens can present +/// a briefly-interrupted night as ONE night without re-deriving (or diverging from) the bridge. +final class BridgedNightGroupsTests: XCTestCase { + private typealias B = SleepStageTotals.NightBlock + private typealias Gap = SleepStageTotals.BridgedNightGroup.GapSpan + /// 2026-01-02 00:00:00 UTC — fixtures read as local clock times with offsetSec 0. + private let t0 = 1_767_312_000 + + func testShortWakeBridgesTwoFragmentsWithOneGap() { + // The #364 example shape: 23:00→02:00, a 16-minute wake, 02:16→06:00. + let a = B(start: t0 - 3_600, end: t0 + 2 * 3_600) + let b = B(start: t0 + 2 * 3_600 + 16 * 60, end: t0 + 6 * 3_600) + let groups = SleepStageTotals.bridgedNightGroups([a, b], offsetSec: 0) + XCTAssertEqual(groups.count, 1) + XCTAssertEqual(groups[0].indices, [0, 1]) + XCTAssertEqual(groups[0].gaps, [Gap(start: a.end, end: b.start)]) + } + + func testDaytimeNapStaysItsOwnGroup() { + let night = B(start: t0 - 3_600, end: t0 + 6 * 3_600) // 23:00→06:00 + let nap = B(start: t0 + 14 * 3_600, end: t0 + 15 * 3_600) // 14:00→15:00 + let groups = SleepStageTotals.bridgedNightGroups([night, nap], offsetSec: 0) + XCTAssertEqual(groups.map(\.indices), [[0], [1]]) + XCTAssertTrue(groups.allSatisfy { $0.gaps.isEmpty }) + } + + func testNightTailBridgesOvernightGapOverSixtyMinutes() { + // 75-min gap with the second fragment's onset at 04:15 local (overnight band): folded in by + // the #861 night-tail bridge, so a longer real mid-night wake still reads as one night. + let a = B(start: t0 - 3_600, end: t0 + 3 * 3_600) + let b = B(start: t0 + 3 * 3_600 + 75 * 60, end: t0 + 7 * 3_600) + let groups = SleepStageTotals.bridgedNightGroups([a, b], offsetSec: 0) + XCTAssertEqual(groups.map(\.indices), [[0, 1]]) + XCTAssertEqual(groups[0].gaps, [Gap(start: a.end, end: b.start)]) + } + + func testDaytimeGapOverSixtyMinutesDoesNotBridge() { + // Same 75-min gap but the second fragment begins at 13:15 local (daytime): the night-tail + // widening must NOT apply, so the blocks stay separate groups (a real afternoon nap). + let a = B(start: t0 + 9 * 3_600, end: t0 + 12 * 3_600) + let b = B(start: t0 + 12 * 3_600 + 75 * 60, end: t0 + 14 * 3_600) + let groups = SleepStageTotals.bridgedNightGroups([a, b], offsetSec: 0) + XCTAssertEqual(groups.map(\.indices), [[0], [1]]) + } + + func testThreeFragmentNightYieldsTwoGaps() { + let a = B(start: t0, end: t0 + 3_600) + let b = B(start: t0 + 3_600 + 600, end: t0 + 2 * 3_600) + let c = B(start: t0 + 2 * 3_600 + 900, end: t0 + 3 * 3_600) + let groups = SleepStageTotals.bridgedNightGroups([a, b, c], offsetSec: 0) + XCTAssertEqual(groups.map(\.indices), [[0, 1, 2]]) + XCTAssertEqual(groups[0].gaps, [Gap(start: a.end, end: b.start), + Gap(start: b.end, end: c.start)]) + } + + func testOverlappingFragmentStartsItsOwnGroup() { + // Legacy semantics pinned: a negative gap (fragment starting inside the previous span) does + // NOT bridge — `mainNightGroupIndices` has always required gap >= 0, and the refactor must + // not change the pick. No seam is fabricated either. + let a = B(start: t0, end: t0 + 4 * 3_600) + let b = B(start: t0 + 3_600, end: t0 + 2 * 3_600) + let groups = SleepStageTotals.bridgedNightGroups([a, b], offsetSec: 0) + XCTAssertEqual(groups.map(\.indices), [[0], [1]]) + XCTAssertTrue(groups.allSatisfy { $0.gaps.isEmpty }) + } + + func testZeroGapBridgesWithoutSeam() { + // Touching fragments (gap == 0) bridge, and no zero-length seam is emitted. + let a = B(start: t0, end: t0 + 3_600) + let b = B(start: t0 + 3_600, end: t0 + 2 * 3_600) + let groups = SleepStageTotals.bridgedNightGroups([a, b], offsetSec: 0) + XCTAssertEqual(groups.map(\.indices), [[0, 1]]) + XCTAssertTrue(groups[0].gaps.isEmpty) + } + + func testUnsortedInputIndicesReferToOriginalPositions() { + // Input order reversed: indices still point into the ORIGINAL array (late = 0, early = 1), + // returned ascending within the group. + let late = B(start: t0 + 2 * 3_600 + 16 * 60, end: t0 + 6 * 3_600) + let early = B(start: t0 - 3_600, end: t0 + 2 * 3_600) + let groups = SleepStageTotals.bridgedNightGroups([late, early], offsetSec: 0) + XCTAssertEqual(groups.map(\.indices), [[0, 1]]) + XCTAssertEqual(groups[0].gaps, [Gap(start: early.end, end: late.start)]) + } + + func testEmptyInputYieldsNoGroups() { + XCTAssertTrue(SleepStageTotals.bridgedNightGroups([], offsetSec: 0).isEmpty) + } + + /// The refactor guard: the main-night pick over the all-groups pass stays byte-identical for a + /// biphasic night + nap day (the #561/#861 golden suite pins the rest of the behaviour). + func testMainNightGroupIndicesUnchangedForBiphasicPlusNap() { + let a = B(start: t0 - 3_600, end: t0 + 2 * 3_600) + let b = B(start: t0 + 2 * 3_600 + 16 * 60, end: t0 + 6 * 3_600) + let nap = B(start: t0 + 14 * 3_600, end: t0 + 15 * 3_600) + XCTAssertEqual(SleepStageTotals.mainNightGroupIndices([a, b, nap], offsetSec: 0), [0, 1]) + } +} From 5eabf28cfe97b77e22211c22538acf1d16c4c8a5 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 12 Jul 2026 23:57:50 -0700 Subject: [PATCH 067/171] feat(sleep): kotlin bridgedNightGroups twin (#364) --- .../com/noop/analytics/SleepStageTotals.kt | 68 ++++++---- .../noop/analytics/BridgedNightGroupsTest.kt | 119 ++++++++++++++++++ 2 files changed, 166 insertions(+), 21 deletions(-) create mode 100644 android/app/src/test/java/com/noop/analytics/BridgedNightGroupsTest.kt diff --git a/android/app/src/main/java/com/noop/analytics/SleepStageTotals.kt b/android/app/src/main/java/com/noop/analytics/SleepStageTotals.kt index 1f88715130..234dd7c75b 100644 --- a/android/app/src/main/java/com/noop/analytics/SleepStageTotals.kt +++ b/android/app/src/main/java/com/noop/analytics/SleepStageTotals.kt @@ -248,25 +248,20 @@ object SleepStageTotals { return out } - /** The indices (into the ORIGINAL [blocks]) of the MAIN-NIGHT GROUP: the main night plus any adjacent - * fragments bridged into it. A biphasic / briefly-interrupted main sleep that reaches the selector still - * split into two blocks (a wake gap shorter than [GAP_BRIDGE_MAX_MIN] between them) is scored as ONE - * night rather than two competing fragments, then the winning bridged group's fragments are ALL returned - * so the caller can SUM their stages for the day's headline figure. (#561) - * - * Pipeline: - * 1. [bridgeAdjacent]-style merge of blocks whose gap is in `[0, GAP_BRIDGE_MAX_MIN*60)` into bridged - * spans, in `start` order, recording which original indices fell into each bridged group; - * 2. [mainNightIndex] scores the BRIDGED spans (so a two-fragment night's combined span out-scores a - * lone nap) and picks the winning bridged group; - * 3. the original indices of that winning group are returned, ascending. - * - * Null only for an empty list. A day with no bridgeable gap collapses to the single-block group the bare - * [mainNightIndex] would pick — byte-identical to the old behaviour for the common case. Pure + - * deterministic; shares the bridge logic + [mainNightIndex] so the pick stays cross-platform stable. - * Mirrors Swift `mainNightGroupIndices`. (#561) */ - fun mainNightGroupIndices(blocks: List, offsetSec: Long, habitualMidsleepSec: Long? = null): List? { - if (blocks.isEmpty()) return null + /** One bridged night group over the whole input: the fragments (as ORIGINAL indices, ascending) plus + * the group's inter-fragment wake seams (start, end) pairs. Produced by [bridgedNightGroups] for the + * consumers that must present a briefly-interrupted night as ONE night — the Health Connect export + * and the Sleep screen — so they group exactly as the day totals do and can fold each seam in as an + * explicit wake segment. Mirrors Swift `BridgedNightGroup`. (#364) */ + data class BridgedNightGroup(val indices: List, val gaps: List>) + + /** EVERY bridged group over [blocks] — the same two-tier bridge [mainNightGroupIndices] applies (#561 + * short-wake, plus the #861 overnight night-tail widening), WITHOUT the winner pick. A negative gap + * (a block starting inside the previous span) does not bridge — pinned legacy semantics, `gap >= 0` — + * and never fabricates a seam. Groups ordered by start; pure and deterministic. Mirrors Swift + * `bridgedNightGroups`. (#364) */ + fun bridgedNightGroups(blocks: List, offsetSec: Long): List { + if (blocks.isEmpty()) return emptyList() // Sort indices by onset so bridging sees neighbours, exactly as `bridgeAdjacent` sorts the blocks. val order = blocks.indices.sortedBy { blocks[it].start } val bridgeS = GAP_BRIDGE_MAX_MIN * 60L @@ -274,6 +269,7 @@ object SleepStageTotals { // Build the bridged spans AND the original indices that compose each one, in one pass over `order`. val bridged = ArrayList() val groups = ArrayList>() + val gaps = ArrayList>>() for (idx in order) { val b = blocks[idx] val last = bridged.lastOrNull() @@ -289,6 +285,7 @@ object SleepStageTotals { (gap < bridgeS || (gap < nightTailBridgeS && isOvernightOnset(b.start, offsetSec))) if (bridges) { + if (gap > 0) gaps[gaps.size - 1].add(last.end to b.start) bridged[bridged.size - 1] = NightBlock(last.start, maxOf(last.end, b.end)) groups[groups.size - 1].add(idx) continue @@ -296,9 +293,38 @@ object SleepStageTotals { } bridged.add(b) groups.add(mutableListOf(idx)) + gaps.add(mutableListOf()) } - val winner = mainNightIndex(bridged, offsetSec, habitualMidsleepSec) ?: return null - return groups[winner].sorted() + return groups.zip(gaps).map { (g, gp) -> BridgedNightGroup(g.sorted(), gp) } + } + + /** The indices (into the ORIGINAL [blocks]) of the MAIN-NIGHT GROUP: the main night plus any adjacent + * fragments bridged into it. A biphasic / briefly-interrupted main sleep that reaches the selector still + * split into two blocks (a wake gap shorter than [GAP_BRIDGE_MAX_MIN] between them) is scored as ONE + * night rather than two competing fragments, then the winning bridged group's fragments are ALL returned + * so the caller can SUM their stages for the day's headline figure. (#561) + * + * Pipeline: + * 1. [bridgedNightGroups] merges blocks whose gap is in `[0, GAP_BRIDGE_MAX_MIN*60)` (or the #861 + * night-tail window) into bridged groups, in `start` order; + * 2. [mainNightIndex] scores the BRIDGED spans (so a two-fragment night's combined span out-scores a + * lone nap) and picks the winning bridged group; + * 3. the original indices of that winning group are returned, ascending. + * + * Null only for an empty list. A day with no bridgeable gap collapses to the single-block group the bare + * [mainNightIndex] would pick — byte-identical to the old behaviour for the common case. Pure + + * deterministic; shares the [bridgedNightGroups] pass + [mainNightIndex] so the pick stays + * cross-platform stable. Mirrors Swift `mainNightGroupIndices`. (#561) */ + fun mainNightGroupIndices(blocks: List, offsetSec: Long, habitualMidsleepSec: Long? = null): List? { + if (blocks.isEmpty()) return null + val all = bridgedNightGroups(blocks, offsetSec) + // Rebuild each group's bridged span for scoring: sorted-ascending fragments make the span + // (first start, running-max end) — identical to the span the one-pass loop accumulated. + val bridgedSpans = all.map { g -> + NightBlock(g.indices.minOf { blocks[it].start }, g.indices.maxOf { blocks[it].end }) + } + val winner = mainNightIndex(bridgedSpans, offsetSec, habitualMidsleepSec) ?: return null + return all[winner].indices } /** Index of the day's MAIN night among [blocks], by the LEARNED-TIMING SCORE (replaces the old hard diff --git a/android/app/src/test/java/com/noop/analytics/BridgedNightGroupsTest.kt b/android/app/src/test/java/com/noop/analytics/BridgedNightGroupsTest.kt new file mode 100644 index 0000000000..4dbc60c86d --- /dev/null +++ b/android/app/src/test/java/com/noop/analytics/BridgedNightGroupsTest.kt @@ -0,0 +1,119 @@ +package com.noop.analytics + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * Pins the all-groups bridge (#364): the SAME two-tier gap bridge the main-night selector applies + * (#561 short-wake + #861 overnight night-tail), returned for EVERY group with each group's + * inter-fragment wake seams explicit — so the Health Connect export and the Sleep screen can present + * a briefly-interrupted night as ONE night without re-deriving (or diverging from) the bridge. + * + * Faithful Kotlin mirror of BridgedNightGroupsTests.swift: same fixtures, same expected values. + */ +class BridgedNightGroupsTest { + + /** 2026-01-02 00:00:00 UTC — fixtures read as local clock times with offsetSec 0. */ + private val t0 = 1_767_312_000L + private fun b(start: Long, end: Long) = SleepStageTotals.NightBlock(start, end) + + @Test + fun shortWakeBridgesTwoFragmentsWithOneGap() { + // The #364 example shape: 23:00→02:00, a 16-minute wake, 02:16→06:00. + val a = b(t0 - 3_600, t0 + 2 * 3_600) + val c = b(t0 + 2 * 3_600 + 16 * 60, t0 + 6 * 3_600) + val groups = SleepStageTotals.bridgedNightGroups(listOf(a, c), 0L) + assertEquals(1, groups.size) + assertEquals(listOf(0, 1), groups[0].indices) + assertEquals(listOf(a.end to c.start), groups[0].gaps) + } + + @Test + fun daytimeNapStaysItsOwnGroup() { + val night = b(t0 - 3_600, t0 + 6 * 3_600) // 23:00→06:00 + val nap = b(t0 + 14 * 3_600, t0 + 15 * 3_600) // 14:00→15:00 + val groups = SleepStageTotals.bridgedNightGroups(listOf(night, nap), 0L) + assertEquals(listOf(listOf(0), listOf(1)), groups.map { it.indices }) + assertTrue(groups.all { it.gaps.isEmpty() }) + } + + @Test + fun nightTailBridgesOvernightGapOverSixtyMinutes() { + // 75-min gap with the second fragment's onset at 04:15 local (overnight band): folded in by + // the #861 night-tail bridge, so a longer real mid-night wake still reads as one night. + val a = b(t0 - 3_600, t0 + 3 * 3_600) + val c = b(t0 + 3 * 3_600 + 75 * 60, t0 + 7 * 3_600) + val groups = SleepStageTotals.bridgedNightGroups(listOf(a, c), 0L) + assertEquals(listOf(listOf(0, 1)), groups.map { it.indices }) + assertEquals(listOf(a.end to c.start), groups[0].gaps) + } + + @Test + fun daytimeGapOverSixtyMinutesDoesNotBridge() { + // Same 75-min gap but the second fragment begins at 13:15 local (daytime): the night-tail + // widening must NOT apply, so the blocks stay separate groups (a real afternoon nap). + val a = b(t0 + 9 * 3_600, t0 + 12 * 3_600) + val c = b(t0 + 12 * 3_600 + 75 * 60, t0 + 14 * 3_600) + val groups = SleepStageTotals.bridgedNightGroups(listOf(a, c), 0L) + assertEquals(listOf(listOf(0), listOf(1)), groups.map { it.indices }) + } + + @Test + fun threeFragmentNightYieldsTwoGaps() { + val a = b(t0, t0 + 3_600) + val c = b(t0 + 3_600 + 600, t0 + 2 * 3_600) + val d = b(t0 + 2 * 3_600 + 900, t0 + 3 * 3_600) + val groups = SleepStageTotals.bridgedNightGroups(listOf(a, c, d), 0L) + assertEquals(listOf(listOf(0, 1, 2)), groups.map { it.indices }) + assertEquals(listOf(a.end to c.start, c.end to d.start), groups[0].gaps) + } + + @Test + fun overlappingFragmentStartsItsOwnGroup() { + // Legacy semantics pinned: a negative gap (fragment starting inside the previous span) does + // NOT bridge — `mainNightGroupIndices` has always required gap >= 0, and the refactor must + // not change the pick. No seam is fabricated either. + val a = b(t0, t0 + 4 * 3_600) + val c = b(t0 + 3_600, t0 + 2 * 3_600) + val groups = SleepStageTotals.bridgedNightGroups(listOf(a, c), 0L) + assertEquals(listOf(listOf(0), listOf(1)), groups.map { it.indices }) + assertTrue(groups.all { it.gaps.isEmpty() }) + } + + @Test + fun zeroGapBridgesWithoutSeam() { + // Touching fragments (gap == 0) bridge, and no zero-length seam is emitted. + val a = b(t0, t0 + 3_600) + val c = b(t0 + 3_600, t0 + 2 * 3_600) + val groups = SleepStageTotals.bridgedNightGroups(listOf(a, c), 0L) + assertEquals(listOf(listOf(0, 1)), groups.map { it.indices }) + assertTrue(groups[0].gaps.isEmpty()) + } + + @Test + fun unsortedInputIndicesReferToOriginalPositions() { + // Input order reversed: indices still point into the ORIGINAL list (late = 0, early = 1), + // returned ascending within the group. + val late = b(t0 + 2 * 3_600 + 16 * 60, t0 + 6 * 3_600) + val early = b(t0 - 3_600, t0 + 2 * 3_600) + val groups = SleepStageTotals.bridgedNightGroups(listOf(late, early), 0L) + assertEquals(listOf(listOf(0, 1)), groups.map { it.indices }) + assertEquals(listOf(early.end to late.start), groups[0].gaps) + } + + @Test + fun emptyInputYieldsNoGroups() { + assertTrue(SleepStageTotals.bridgedNightGroups(emptyList(), 0L).isEmpty()) + } + + /** The refactor guard: the main-night pick over the all-groups pass stays byte-identical for a + * biphasic night + nap day (MainNightConsistencyTest pins the rest of the behaviour). */ + @Test + fun mainNightGroupIndicesUnchangedForBiphasicPlusNap() { + val a = b(t0 - 3_600, t0 + 2 * 3_600) + val c = b(t0 + 2 * 3_600 + 16 * 60, t0 + 6 * 3_600) + val nap = b(t0 + 14 * 3_600, t0 + 15 * 3_600) + assertEquals(listOf(0, 1), SleepStageTotals.mainNightGroupIndices(listOf(a, c, nap), 0L)) + } +} From bdd2c86a6495c3f9b4e3fb667ad5f56e00249f7c Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 00:01:31 -0700 Subject: [PATCH 068/171] =?UTF-8?q?feat(sleep):=20mergedSleepPlan=20?= =?UTF-8?q?=E2=80=94=20one=20write-back=20entry=20per=20bridged=20night,?= =?UTF-8?q?=20seams=20as=20wake=20(#364)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../StrandImport/HealthWriteback.swift | 87 ++++++++++++++++- .../HealthWritebackTests.swift | 96 +++++++++++++++++++ 2 files changed, 181 insertions(+), 2 deletions(-) diff --git a/Packages/StrandImport/Sources/StrandImport/HealthWriteback.swift b/Packages/StrandImport/Sources/StrandImport/HealthWriteback.swift index fd230fa0a7..8538fe9198 100644 --- a/Packages/StrandImport/Sources/StrandImport/HealthWriteback.swift +++ b/Packages/StrandImport/Sources/StrandImport/HealthWriteback.swift @@ -6,9 +6,11 @@ import Foundation public enum HealthWriteback { /// A HealthKit-agnostic sleep stage. The bridge maps these onto `HKCategoryValueSleepAnalysis` - /// (`awake → .awake`, `light → .asleepCore`, `deep → .asleepDeep`, `rem → .asleepREM`). + /// (`awake → .awake`, `light → .asleepCore`, `deep → .asleepDeep`, `rem → .asleepREM`, + /// `unspecified → .asleepUnspecified` — the honest block for a fragment whose `stagesJSON` + /// carries no timing). public enum StageKind: String, Equatable { - case awake, light, deep, rem + case awake, light, deep, rem, unspecified } public struct StageInterval: Equatable { @@ -71,4 +73,85 @@ public enum HealthWriteback { default: return nil } } + + // MARK: - Merged nights (#364) + + /// One stored sleep fragment, as the write-back sees it: the immutable detected key (`startTs`), + /// the edited onset when present (`effectiveStartTs` — drives the span, never the key, #318), + /// the wake, and the persisted stage JSON. + public struct SleepFragment: Equatable { + public let startTs: Int + public let effectiveStartTs: Int + public let endTs: Int + public let stagesJSON: String? + public init(startTs: Int, effectiveStartTs: Int, endTs: Int, stagesJSON: String?) { + self.startTs = startTs; self.effectiveStartTs = effectiveStartTs + self.endTs = endTs; self.stagesJSON = stagesJSON + } + } + + /// One bridged night, folded for the write-back: the `.inBed` span, the merged stage timeline, + /// the representative dedup key, and the COMPLETE per-fragment key set for deletion. + public struct MergedSleepEntry: Equatable { + /// The group's dedup key: the EARLIEST fragment's immutable detected `startTs` (a user edit + /// moves the span via `effectiveStartTs`, never this key, so the entry keeps its identity). + public let keyStartTs: Int + /// Earliest edited onset → latest wake across the group: the one `.inBed` span. + public let spanStart: Int + public let spanEnd: Int + /// The fragments' stage intervals in time order — a fragment with no decodable timing + /// contributes one honest `.unspecified` block over its own window — with every + /// inter-fragment seam as an explicit `.awake` interval. + public let intervals: [StageInterval] + /// EVERY fragment's immutable `startTs`, ascending. The delete predicate must carry all of + /// them: a night previously exported as two entries would otherwise orphan the absorbed + /// fragment's old entry when it becomes one. + public let allKeyStartTs: [Int] + public init(keyStartTs: Int, spanStart: Int, spanEnd: Int, + intervals: [StageInterval], allKeyStartTs: [Int]) { + self.keyStartTs = keyStartTs; self.spanStart = spanStart; self.spanEnd = spanEnd + self.intervals = intervals; self.allKeyStartTs = allKeyStartTs + } + } + + /// Fold bridged night groups (#364) into write-back entries — one `.inBed` span per night with + /// the mid-night wake seams as explicit `.awake` intervals, matching what the daily totals + /// already score (#561/#777) and what Oura / Apple Watch write into Health. `groups` comes from + /// `SleepStageTotals.bridgedNightGroups` (this package deliberately doesn't depend on the + /// analytics package, so the grouping happens in the caller). Fragments are re-sorted by + /// effective onset defensively; a zero/negative-length fragment contributes no interval but + /// keeps its key in the delete set; a group with no positive span is skipped entirely. + public static func mergedSleepPlan(groups: [[SleepFragment]]) -> [MergedSleepEntry] { + var out: [MergedSleepEntry] = [] + for group in groups where !group.isEmpty { + let frags = group.sorted { $0.effectiveStartTs < $1.effectiveStartTs } + var intervals: [StageInterval] = [] + var prevEnd: Int? = nil + var spanStart: Int? = nil + for f in frags { + guard f.endTs > f.effectiveStartTs else { continue } + if spanStart == nil { spanStart = f.effectiveStartTs } + if let p = prevEnd, f.effectiveStartTs > p { + intervals.append(StageInterval(start: p, end: f.effectiveStartTs, kind: .awake)) + } + let stages = stageIntervals(stagesJSON: f.stagesJSON, + sessionStart: f.effectiveStartTs, sessionEnd: f.endTs) + if stages.isEmpty { + intervals.append(StageInterval(start: f.effectiveStartTs, end: f.endTs, + kind: .unspecified)) + } else { + intervals.append(contentsOf: stages) + } + prevEnd = max(prevEnd ?? f.endTs, f.endTs) + } + guard let start = spanStart, let end = prevEnd, end > start else { continue } + out.append(MergedSleepEntry( + keyStartTs: frags.map(\.startTs).min() ?? 0, + spanStart: start, + spanEnd: end, + intervals: intervals, + allKeyStartTs: frags.map(\.startTs).sorted())) + } + return out + } } diff --git a/Packages/StrandImport/Tests/StrandImportTests/HealthWritebackTests.swift b/Packages/StrandImport/Tests/StrandImportTests/HealthWritebackTests.swift index cbb5cce899..40f08c789b 100644 --- a/Packages/StrandImport/Tests/StrandImportTests/HealthWritebackTests.swift +++ b/Packages/StrandImport/Tests/StrandImportTests/HealthWritebackTests.swift @@ -87,4 +87,100 @@ final class HealthWritebackTests: XCTestCase { """ XCTAssertEqual(intervals(json), [.init(start: start, end: start + 60, kind: .deep)]) } + + // MARK: - mergedSleepPlan (#364 night-stitch) + // + // One write-back entry per BRIDGED night: the caller (HealthKitBridge) groups fragments with + // SleepStageTotals.bridgedNightGroups; this pure layer folds each group into one span + one + // merged stage timeline with every inter-fragment seam as an explicit `wake`, keyed by the + // earliest fragment's immutable startTs, carrying the COMPLETE per-fragment key set so a night + // previously exported as two entries deletes both before the merged write. + + private func frag(_ start: Int, _ end: Int, stages: String? = nil, + eff: Int? = nil) -> HealthWriteback.SleepFragment { + .init(startTs: start, effectiveStartTs: eff ?? start, endTs: end, stagesJSON: stages) + } + private func stagesJSON(_ segs: [(Int, Int, String)]) -> String { + "[" + segs.map { "{\"start\":\($0.0),\"end\":\($0.1),\"stage\":\"\($0.2)\"}" } + .joined(separator: ",") + "]" + } + + func testMergedPlanFoldsTwoFragmentsWithWakeSeam() { + let t = 1_767_312_000 + let a = frag(t, t + 7_200, stages: stagesJSON([(t, t + 7_200, "light")])) + let b = frag(t + 7_200 + 960, t + 14_400, + stages: stagesJSON([(t + 7_200 + 960, t + 14_400, "deep")])) + let plan = HealthWriteback.mergedSleepPlan(groups: [[a, b]]) + XCTAssertEqual(plan.count, 1) + let e = plan[0] + XCTAssertEqual(e.keyStartTs, t) + XCTAssertEqual(e.spanStart, t) + XCTAssertEqual(e.spanEnd, t + 14_400) + XCTAssertEqual(e.allKeyStartTs, [t, t + 7_200 + 960]) + // The seam sits EXACTLY on [prev.end, next.effectiveStart] as .awake. + XCTAssertEqual(e.intervals, [ + .init(start: t, end: t + 7_200, kind: .light), + .init(start: t + 7_200, end: t + 7_200 + 960, kind: .awake), + .init(start: t + 7_200 + 960, end: t + 14_400, kind: .deep), + ]) + } + + func testMergedPlanSingleFragmentMatchesLegacyShape() { + let t = 1_767_312_000 + let a = frag(t, t + 7_200, + stages: stagesJSON([(t, t + 3_600, "light"), (t + 3_600, t + 7_200, "rem")])) + let e = HealthWriteback.mergedSleepPlan(groups: [[a]])[0] + XCTAssertEqual(e.keyStartTs, t) + XCTAssertEqual(e.spanStart, t) + XCTAssertEqual(e.spanEnd, t + 7_200) + XCTAssertEqual(e.allKeyStartTs, [t]) + XCTAssertEqual(e.intervals, HealthWriteback.stageIntervals( + stagesJSON: a.stagesJSON, sessionStart: t, sessionEnd: t + 7_200)) + } + + func testMergedPlanUnstagedFragmentsDegradeToUnspecifiedPlusSeam() { + let t = 1_767_312_000 + let a = frag(t, t + 7_200) // no stagesJSON → honest unspecified block + let b = frag(t + 7_200 + 960, t + 14_400) + let e = HealthWriteback.mergedSleepPlan(groups: [[a, b]])[0] + XCTAssertEqual(e.intervals, [ + .init(start: t, end: t + 7_200, kind: .unspecified), + .init(start: t + 7_200, end: t + 7_200 + 960, kind: .awake), + .init(start: t + 7_200 + 960, end: t + 14_400, kind: .unspecified), + ]) + } + + func testMergedPlanMixedStagedAndUnstagedFragments() { + let t = 1_767_312_000 + let a = frag(t, t + 7_200, stages: stagesJSON([(t, t + 7_200, "light")])) + let b = frag(t + 7_200 + 960, t + 14_400) // second fragment carries no timing + let e = HealthWriteback.mergedSleepPlan(groups: [[a, b]])[0] + XCTAssertEqual(e.intervals, [ + .init(start: t, end: t + 7_200, kind: .light), + .init(start: t + 7_200, end: t + 7_200 + 960, kind: .awake), + .init(start: t + 7_200 + 960, end: t + 14_400, kind: .unspecified), + ]) + } + + func testMergedPlanEditedOnsetMovesSpanButNotKey() { + let t = 1_767_312_000 + let a = frag(t, t + 7_200, eff: t + 600) // user moved bedtime 10 min later + let e = HealthWriteback.mergedSleepPlan(groups: [[a]])[0] + XCTAssertEqual(e.keyStartTs, t) // immutable key + XCTAssertEqual(e.spanStart, t + 600) // edited onset drives the span + XCTAssertEqual(e.intervals, [.init(start: t + 600, end: t + 7_200, kind: .unspecified)]) + } + + func testMergedPlanSkipsDegenerateFragmentsAndEmptyGroups() { + let t = 1_767_312_000 + let bad = frag(t, t) // zero-length → contributes no interval + let good = frag(t + 600, t + 7_200) + let plan = HealthWriteback.mergedSleepPlan(groups: [[], [bad, good]]) + XCTAssertEqual(plan.count, 1) + XCTAssertEqual(plan[0].spanStart, t + 600) + // The degenerate fragment still contributes its key to the delete set (its old export, if + // any, must clear), but produces no interval. + XCTAssertEqual(plan[0].allKeyStartTs, [t, t + 600]) + XCTAssertEqual(plan[0].intervals, [.init(start: t + 600, end: t + 7_200, kind: .unspecified)]) + } } From bf1a141ff3bce9e1444fe22674274eda6b6743a8 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 00:08:09 -0700 Subject: [PATCH 069/171] fix(health): export a wake-split night as ONE Apple Health entry with the gap as awake (#364) --- StrandiOS/Health/HealthKitBridge.swift | 79 ++++++++++++++------------ 1 file changed, 42 insertions(+), 37 deletions(-) diff --git a/StrandiOS/Health/HealthKitBridge.swift b/StrandiOS/Health/HealthKitBridge.swift index a3fd7996e9..40bdb52137 100644 --- a/StrandiOS/Health/HealthKitBridge.swift +++ b/StrandiOS/Health/HealthKitBridge.swift @@ -2,6 +2,7 @@ import Foundation import HealthKit import WhoopStore +import StrandAnalytics import StrandImport /// Two-way Apple Health bridge for the iOS app. @@ -562,52 +563,56 @@ final class HealthKitBridge: ObservableObject { try await self.store.save(candidates.map { $0.sample }) } - /// Write each sleep session as one `.inBed` sample spanning `effectiveStartTs → endTs` plus one - /// category sample per stage segment (`deep → .asleepDeep`, `rem → .asleepREM`, - /// `light → .asleepCore`, `wake → .awake`) — the same shape Oura and Apple Watch write, so - /// Health renders the full hypnogram. Sessions whose `stagesJSON` carries no timing (the legacy - /// aggregate-minutes shapes) get one honest `.asleepUnspecified` block instead of fabricated - /// stage placement. + /// Write each BRIDGED NIGHT (#364) as one `.inBed` sample plus one category sample per stage + /// segment (`deep → .asleepDeep`, `rem → .asleepREM`, `light → .asleepCore`, `wake → .awake`) — + /// the same shape Oura and Apple Watch write, so Health renders the full hypnogram. A night the + /// detector split on a brief mid-night wake exports as ONE entry whose gap is an explicit + /// `.awake` segment (grouped by `SleepStageTotals.bridgedNightGroups`, the SAME bridge the daily + /// totals score with, #561); naps never bridge and stay their own entries. Fragments whose + /// `stagesJSON` carries no timing (the legacy aggregate-minutes shapes) get one honest + /// `.asleepUnspecified` block instead of fabricated stage placement. /// - /// Dedup: every sample of a session carries `HKMetadataKeyExternalUUID = - /// noop::sleep:` (the immutable detected onset, so a user-edited session - /// keeps its identity); delete-then-write scoped to our own `HKSource`, like the vitals. + /// Dedup: every sample of a night carries `HKMetadataKeyExternalUUID = + /// noop::sleep:` keyed by the group's EARLIEST fragment's immutable detected + /// onset (a user edit moves the span, never the key). The delete predicate carries EVERY + /// fragment's key, so a night previously written as two entries fully clears when it becomes + /// one; delete-then-write scoped to our own `HKSource`, like the vitals. private func writeSleep(sessions: [CachedSleepSession]) async throws { guard let type = HKObjectType.categoryType(forIdentifier: .sleepAnalysis), store.authorizationStatus(for: type) == .sharingAuthorized else { return } + let blocks = sessions.map { SleepStageTotals.NightBlock(start: $0.effectiveStartTs, end: $0.endTs) } + let groups = SleepStageTotals.bridgedNightGroups(blocks, offsetSec: TimeZone.current.secondsFromGMT()) + .map { g in + g.indices.map { i -> HealthWriteback.SleepFragment in + let s = sessions[i] + return .init(startTs: s.startTs, effectiveStartTs: s.effectiveStartTs, + endTs: s.endTs, stagesJSON: s.stagesJSON) + } + } var samples: [HKCategorySample] = [] var keys: [String] = [] - for s in sessions { - let startTs = s.effectiveStartTs - guard s.endTs > startTs else { continue } - let start = Date(timeIntervalSince1970: TimeInterval(startTs)) - let end = Date(timeIntervalSince1970: TimeInterval(s.endTs)) - let key = "noop:\(noopDeviceId):sleep:\(s.startTs)" - keys.append(key) + for entry in HealthWriteback.mergedSleepPlan(groups: groups) { + let key = "noop:\(noopDeviceId):sleep:\(entry.keyStartTs)" let meta = [HKMetadataKeyExternalUUID: key] + keys.append(contentsOf: entry.allKeyStartTs.map { "noop:\(noopDeviceId):sleep:\($0)" }) samples.append(HKCategorySample(type: type, value: HKCategoryValueSleepAnalysis.inBed.rawValue, - start: start, end: end, metadata: meta)) - let stages = HealthWriteback.stageIntervals(stagesJSON: s.stagesJSON, - sessionStart: startTs, sessionEnd: s.endTs) - if stages.isEmpty { - samples.append(HKCategorySample(type: type, - value: HKCategoryValueSleepAnalysis.asleepUnspecified.rawValue, - start: start, end: end, metadata: meta)) - } else { - for seg in stages { - let value: HKCategoryValueSleepAnalysis - switch seg.kind { - case .awake: value = .awake - case .light: value = .asleepCore - case .deep: value = .asleepDeep - case .rem: value = .asleepREM - } - samples.append(HKCategorySample( - type: type, value: value.rawValue, - start: Date(timeIntervalSince1970: TimeInterval(seg.start)), - end: Date(timeIntervalSince1970: TimeInterval(seg.end)), - metadata: meta)) + start: Date(timeIntervalSince1970: TimeInterval(entry.spanStart)), + end: Date(timeIntervalSince1970: TimeInterval(entry.spanEnd)), + metadata: meta)) + for seg in entry.intervals { + let value: HKCategoryValueSleepAnalysis + switch seg.kind { + case .awake: value = .awake + case .light: value = .asleepCore + case .deep: value = .asleepDeep + case .rem: value = .asleepREM + case .unspecified: value = .asleepUnspecified } + samples.append(HKCategorySample( + type: type, value: value.rawValue, + start: Date(timeIntervalSince1970: TimeInterval(seg.start)), + end: Date(timeIntervalSince1970: TimeInterval(seg.end)), + metadata: meta)) } } guard !samples.isEmpty else { return } From 2c9e10c84095d91d0f1fd540d1856e0cce6d7ef2 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 00:10:47 -0700 Subject: [PATCH 070/171] feat(sleep): night-relative fragment rebase + wake seams in the hero hypnogram (#364) --- Strand/Screens/SleepView.swift | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/Strand/Screens/SleepView.swift b/Strand/Screens/SleepView.swift index 2c271607a0..117b45368d 100644 --- a/Strand/Screens/SleepView.swift +++ b/Strand/Screens/SleepView.swift @@ -2000,8 +2000,14 @@ struct SleepView: View { if let seg = decodeSegments(frag.stagesJSON, sessionStart: frag.effectiveStartTs), seg.stages.total > 0 { stages.awake += seg.stages.awake; stages.light += seg.stages.light stages.deep += seg.stages.deep; stages.rem += seg.stages.rem + // decodeSegments yields intervals relative to THAT fragment's onset; rebase them to + // NIGHT-relative (seconds from the displayed onset) so a later fragment's bars land + // after the first fragment instead of overlapping it — the same coordinate space + // sleepHRChart maps HR into (rel = ts - nightStart). No-op for the first fragment + // of a night (shift 0), so single-block nights are byte-identical. (#364) + let shift = TimeInterval(frag.effectiveStartTs - onset) for iv in seg.intervals { - segs.append(SleepInterval(stage: iv.stage, start: iv.start, end: iv.end)) + segs.append(SleepInterval(stage: iv.stage, start: iv.start + shift, end: iv.end + shift)) } } else if let st = decodeStages(frag.stagesJSON), st.total > 0 { stages.awake += st.awake; stages.light += st.light @@ -2009,6 +2015,20 @@ struct SleepView: View { } if let m = motionByStart[frag.startTs] { motion.append(contentsOf: m) } } + // #364: the inter-fragment wake seams belong to the night — draw each as a wake SEGMENT so + // the hero hypnogram has no hole where the user was up, matching what the Health export now + // writes. The stage MINUTES deliberately stay fragment-only (the seam is not added to + // `stages.awake`): the hero's in-bed/efficiency accounting sums fragment windows, mirroring + // the Android groupInBedMin rule, so asleep ≤ in-bed stays coherent. Seams are + // night-relative like the rebased segments above. + let orderedFrags = Array(group) + for (prev, next) in zip(orderedFrags, orderedFrags.dropFirst()) { + let gapStart = prev.endTs, gapEnd = next.effectiveStartTs + guard gapEnd > gapStart else { continue } + segs.append(SleepInterval(stage: .awake, + start: TimeInterval(gapStart - onset), + end: TimeInterval(gapEnd - onset))) + } guard stages.asleep > 0 else { return nil } let eff = stages.total > 0 ? stages.asleep / stages.total : nil let synth = CachedSleepSession(startTs: onset, endTs: wake, efficiency: eff, From d86cf28ce39563c229609d6a5501afa0d273905a Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 00:10:47 -0700 Subject: [PATCH 071/171] =?UTF-8?q?fix(health):=20android=20=E2=80=94=20on?= =?UTF-8?q?e=20Health=20Connect=20session=20per=20bridged=20night,=20absor?= =?UTF-8?q?bed=20fragments=20deleted=20(#364)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../com/noop/ingest/HealthConnectWriter.kt | 20 +++++- .../java/com/noop/ingest/HealthExportPlan.kt | 48 +++++++++++-- .../com/noop/ingest/HealthExportPlanTest.kt | 67 ++++++++++++++++--- 3 files changed, 115 insertions(+), 20 deletions(-) diff --git a/android/app/src/main/java/com/noop/ingest/HealthConnectWriter.kt b/android/app/src/main/java/com/noop/ingest/HealthConnectWriter.kt index b0d8c20428..0f56663459 100644 --- a/android/app/src/main/java/com/noop/ingest/HealthConnectWriter.kt +++ b/android/app/src/main/java/com/noop/ingest/HealthConnectWriter.kt @@ -194,14 +194,19 @@ object HealthConnectWriter { * split yet — only the validated asleep-vs-awake distinction is shared so we don't over-claim the * stager's precision). Uses the MERGED view (imported wins, on-device-computed gap-fills) so a * strap-only user's locally-computed nights (stored under the "-noop" computed id) are included. - * The clientRecordId keys off the immutable detected startTs so a re-write upserts in place. + * #364 — fragments are grouped into BRIDGED NIGHTS (the same #561 bridge the daily totals score + * with), so a night split by a brief mid-night wake exports as ONE record whose gap is an AWAKE + * stage; the clientRecordId keys off the group's earliest fragment's immutable detected startTs, + * and the absorbed fragments' old per-fragment records are DELETED (HC upserts by id but never + * removes an id we stop writing). */ private suspend fun writeSleep(client: HealthConnectClient, repo: WhoopRepository, deviceId: String): Int { val now = System.currentTimeMillis() / 1000 val floor = now - WINDOW_DAYS * 86_400 val sessions = repo.sleepSessionsMerged(deviceId, from = floor, to = now) - .map { HealthExportPlan.SleepInput(it.startTs, it.endTs, it.stagesJSON) } - val plans = HealthExportPlan.sleepSessions(sessions, now) + .map { HealthExportPlan.SleepInput(it.startTs, it.effectiveStartTs, it.endTs, it.stagesJSON) } + val offsetSec = (java.util.TimeZone.getDefault().getOffset(now * 1000) / 1000).toLong() + val plans = HealthExportPlan.sleepSessions(sessions, now, offsetSec) if (plans.isEmpty()) return 0 val zone = ZoneId.systemDefault() @@ -223,6 +228,15 @@ object HealthConnectWriter { metadata = Metadata(clientRecordId = p.clientId, clientRecordVersion = p.endSec), ) } + // Clear absorbed fragments' old records BEFORE the merged upsert, so a night previously + // exported as two entries never lingers as one merged + one stale fragment. (#364) + val absorbed = plans.flatMap { it.absorbedClientIds } + if (absorbed.isNotEmpty()) { + runCatching { + client.deleteRecords(SleepSessionRecord::class, + recordIdsList = emptyList(), clientRecordIdsList = absorbed) + } + } return insertChunked(client, records) } diff --git a/android/app/src/main/java/com/noop/ingest/HealthExportPlan.kt b/android/app/src/main/java/com/noop/ingest/HealthExportPlan.kt index 5890bd5028..d50828be9f 100644 --- a/android/app/src/main/java/com/noop/ingest/HealthExportPlan.kt +++ b/android/app/src/main/java/com/noop/ingest/HealthExportPlan.kt @@ -76,22 +76,56 @@ object HealthExportPlan { // ---- Sleep sessions: AWAKE vs SLEEPING only (fine stages deferred until stager validated) ---- - data class SleepInput(val startTs: Long, val endTs: Long, val stagesJSON: String?) + /** One stored fragment as the export sees it: [keyStartTs] is the immutable detected onset (the + * dedup identity — a user edit must never change it), [startTs] the EFFECTIVE onset that drives + * the exported span (`startTsAdjusted ?: startTs`, iOS parity #318). */ + data class SleepInput(val keyStartTs: Long, val startTs: Long, val endTs: Long, val stagesJSON: String?) data class StagePlan(val startSec: Long, val endSec: Long, val asleep: Boolean) data class SleepPlan( val clientId: String, val startSec: Long, val endSec: Long, val stages: List, + /** The non-representative fragments' old per-fragment ids (`noop-sleep-`). Health + * Connect upserts by clientRecordId but never removes an id we stop writing, so a night that + * previously exported as two records would orphan the second when it becomes one — the + * writer deletes these explicitly. Empty for a single-fragment night. (#364) */ + val absorbedClientIds: List = emptyList(), ) - /** Finalized sessions (endTs <= [nowSec]) only; never the currently-open night. */ - fun sleepSessions(sessions: List, nowSec: Long): List { + /** Finalized sessions (endTs <= [nowSec]) only; never the currently-open night. Fragments are + * grouped into BRIDGED NIGHTS (#364) via [SleepStageTotals.bridgedNightGroups] — the SAME + * two-tier bridge the daily totals score with (#561/#861) — so a night the detector split on a + * brief mid-night wake exports as ONE session whose gap is an explicit AWAKE stage; naps never + * bridge and stay their own records. The clientRecordId keys off the group's EARLIEST fragment's + * immutable detected onset. [offsetSec] is seconds EAST of UTC (the night-tail bridge reads the + * local clock). */ + fun sleepSessions(sessions: List, nowSec: Long, offsetSec: Long): List { + val finalized = sessions.filter { it.endTs > it.startTs && it.endTs <= nowSec } + if (finalized.isEmpty()) return emptyList() + val blocks = finalized.map { com.noop.analytics.SleepStageTotals.NightBlock(it.startTs, it.endTs) } val out = ArrayList() - for (s in sessions) { - if (s.endTs <= s.startTs) continue - if (s.endTs > nowSec) continue - out.add(SleepPlan("noop-sleep-${s.startTs}", s.startTs, s.endTs, parseStages(s.stagesJSON))) + for (group in com.noop.analytics.SleepStageTotals.bridgedNightGroups(blocks, offsetSec)) { + val frags = group.indices.map { finalized[it] }.sortedBy { it.startTs } + val stages = ArrayList() + var prevEnd: Long? = null + for (f in frags) { + val p = prevEnd + // The inter-fragment seam is time the user was demonstrably awake — export it as an + // explicit AWAKE stage so the merged night carries the wake instead of a silent hole. + if (p != null && f.startTs > p) stages.add(StagePlan(p, f.startTs, asleep = false)) + stages.addAll(parseStages(f.stagesJSON)) + prevEnd = maxOf(prevEnd ?: f.endTs, f.endTs) + } + val rep = frags.minOf { it.keyStartTs } + out.add(SleepPlan( + clientId = "noop-sleep-$rep", + startSec = frags.first().startTs, + endSec = frags.maxOf { it.endTs }, + stages = stages, + absorbedClientIds = frags.map { it.keyStartTs }.filter { it != rep } + .sorted().map { "noop-sleep-$it" }, + )) } return out } diff --git a/android/app/src/test/java/com/noop/ingest/HealthExportPlanTest.kt b/android/app/src/test/java/com/noop/ingest/HealthExportPlanTest.kt index 054f2900b7..6c36df8075 100644 --- a/android/app/src/test/java/com/noop/ingest/HealthExportPlanTest.kt +++ b/android/app/src/test/java/com/noop/ingest/HealthExportPlanTest.kt @@ -83,20 +83,25 @@ class HealthExportPlanTest { // ---- Sleep session tests ---- + /** Unedited fragment: key == effective onset (the common case). */ + private fun input(start: Long, end: Long, json: String? = null) = + HealthExportPlan.SleepInput(keyStartTs = start, startTs = start, endTs = end, stagesJSON = json) + @Test fun sleep_excludesUnfinalizedSessions() { - val sessions = listOf(HealthExportPlan.SleepInput(startTs = 100, endTs = 900, stagesJSON = null)) - val out = HealthExportPlan.sleepSessions(sessions, nowSec = 500L) // ends in the future + val sessions = listOf(input(100, 900)) + val out = HealthExportPlan.sleepSessions(sessions, nowSec = 500L, offsetSec = 0L) // ends in the future assertTrue(out.isEmpty()) } @Test fun sleep_emitsFinalizedSessionWithClientId() { - val sessions = listOf(HealthExportPlan.SleepInput(100, 900, null)) - val out = HealthExportPlan.sleepSessions(sessions, nowSec = 1000L) + val sessions = listOf(input(100, 900)) + val out = HealthExportPlan.sleepSessions(sessions, nowSec = 1000L, offsetSec = 0L) assertEquals(1, out.size) assertEquals("noop-sleep-100", out[0].clientId) assertEquals(100L, out[0].startSec) assertEquals(900L, out[0].endSec) assertTrue(out[0].stages.isEmpty()) // null stagesJSON -> session bounds only + assertTrue(out[0].absorbedClientIds.isEmpty()) } @Test fun sleep_mapsWakeVsAsleepAndCoalesces() { @@ -108,7 +113,7 @@ class HealthExportPlanTest { {"start":500,"end":600,"stage":"rem"}] """.trimIndent() val out = HealthExportPlan.sleepSessions( - listOf(HealthExportPlan.SleepInput(100, 600, json)), nowSec = 1000L) + listOf(input(100, 600, json)), nowSec = 1000L, offsetSec = 0L) val stages = out[0].stages // light+deep coalesce -> asleep[100,300); wake+awake coalesce -> awake[300,500); rem -> asleep[500,600) assertEquals(3, stages.size) @@ -119,7 +124,7 @@ class HealthExportPlanTest { @Test fun sleep_malformedJsonYieldsNoStagesButKeepsSession() { val out = HealthExportPlan.sleepSessions( - listOf(HealthExportPlan.SleepInput(100, 900, "not json")), nowSec = 1000L) + listOf(input(100, 900, "not json")), nowSec = 1000L, offsetSec = 0L) assertEquals(1, out.size) assertTrue(out[0].stages.isEmpty()) } @@ -127,7 +132,7 @@ class HealthExportPlanTest { @Test fun sleep_includesSessionEndingExactlyAtNow() { // endTs == nowSec is finalized (the session has ended); locks in the boundary contract. val out = HealthExportPlan.sleepSessions( - listOf(HealthExportPlan.SleepInput(100, 1000, null)), nowSec = 1000L) + listOf(input(100, 1000)), nowSec = 1000L, offsetSec = 0L) assertEquals(1, out.size) } @@ -140,7 +145,7 @@ class HealthExportPlanTest { {"start":300,"end":400,"stage":"deep"}] """.trimIndent() val out = HealthExportPlan.sleepSessions( - listOf(HealthExportPlan.SleepInput(100, 400, json)), nowSec = 1000L) + listOf(input(100, 400, json)), nowSec = 1000L, offsetSec = 0L) val stages = out[0].stages // light[100,200) and deep[300,400) both asleep but NOT contiguous (gap 200..300) -> 2 segments. assertEquals(2, stages.size) @@ -151,11 +156,53 @@ class HealthExportPlanTest { @Test fun sleep_skipsInvertedOrZeroLengthSession() { val out = HealthExportPlan.sleepSessions( listOf( - HealthExportPlan.SleepInput(500, 500, null), // zero-length - HealthExportPlan.SleepInput(900, 400, null), // inverted + input(500, 500), // zero-length + input(900, 400), // inverted ), nowSec = 10_000L, + offsetSec = 0L, ) assertTrue(out.isEmpty()) } + + // ---- #364: bridged-night grouping (twin of the Swift mergedSleepPlan tests) ---- + + @Test fun sleep_foldsBridgedFragmentsWithAwakeSeam() { + // 2026-01-02 00:00 UTC. Two fragments split by a 16-minute wake -> ONE record whose seam is + // an explicit AWAKE stage; the clientRecordId keys off the earliest fragment; the absorbed + // fragment's old per-fragment id is surfaced for deletion. + val t = 1_767_312_000L + val a = input(t - 3_600, t + 2 * 3_600, """[{"start":${t - 3_600},"end":${t + 2 * 3_600},"stage":"light"}]""") + val b = input(t + 2 * 3_600 + 960, t + 6 * 3_600, + """[{"start":${t + 2 * 3_600 + 960},"end":${t + 6 * 3_600},"stage":"deep"}]""") + val plans = HealthExportPlan.sleepSessions(listOf(a, b), nowSec = t + 86_400, offsetSec = 0L) + assertEquals(1, plans.size) + val p = plans[0] + assertEquals("noop-sleep-${t - 3_600}", p.clientId) + assertEquals(t - 3_600, p.startSec) + assertEquals(t + 6 * 3_600, p.endSec) + assertEquals(listOf("noop-sleep-${t + 2 * 3_600 + 960}"), p.absorbedClientIds) + assertTrue(p.stages.contains( + HealthExportPlan.StagePlan(t + 2 * 3_600, t + 2 * 3_600 + 960, asleep = false))) + } + + @Test fun sleep_napStaysItsOwnPlanWithNoAbsorbedIds() { + val t = 1_767_312_000L + val night = input(t - 3_600, t + 6 * 3_600) + val nap = input(t + 14 * 3_600, t + 15 * 3_600) // 14:00 local, hours away -> own record + val plans = HealthExportPlan.sleepSessions(listOf(night, nap), nowSec = t + 86_400, offsetSec = 0L) + assertEquals(2, plans.size) + assertTrue(plans.all { it.absorbedClientIds.isEmpty() }) + assertEquals("noop-sleep-${t - 3_600}", plans[0].clientId) + assertEquals("noop-sleep-${t + 14 * 3_600}", plans[1].clientId) + } + + @Test fun sleep_editedOnsetMovesSpanButClientIdStaysImmutable() { + val t = 1_767_312_000L + val edited = HealthExportPlan.SleepInput( + keyStartTs = t, startTs = t + 600, endTs = t + 7_200, stagesJSON = null) + val p = HealthExportPlan.sleepSessions(listOf(edited), nowSec = t + 86_400, offsetSec = 0L)[0] + assertEquals("noop-sleep-$t", p.clientId) // immutable detected key + assertEquals(t + 600, p.startSec) // edited onset drives the span + } } From e0e3fa201787cda7c8e708fd18769f7d7d8663cf Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 00:13:01 -0700 Subject: [PATCH 072/171] =?UTF-8?q?feat(sleep):=20android=20=E2=80=94=20wa?= =?UTF-8?q?ke=20seams=20in=20the=20full-night=20hypnogram=20timeline=20(#3?= =?UTF-8?q?64)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../app/src/main/java/com/noop/ui/SleepScreen.kt | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/android/app/src/main/java/com/noop/ui/SleepScreen.kt b/android/app/src/main/java/com/noop/ui/SleepScreen.kt index d69853a7c1..a8d9993bcc 100644 --- a/android/app/src/main/java/com/noop/ui/SleepScreen.kt +++ b/android/app/src/main/java/com/noop/ui/SleepScreen.kt @@ -2824,13 +2824,25 @@ internal fun selectNight( // #259: clamp each fragment's timeline to its effective onset too (matching the stage-minute clamp // and the iOS decodeSegments clamp), so the hypnogram starts where the edited bedtime does instead of // drawing pre-onset bars that contradict the clamped asleep total. No-op for non-edited nights. - val groupSegments = if (heroGroup.size > 1) { + val groupSegmentsRaw = if (heroGroup.size > 1) { heroGroup.flatMap { parsePersistedSegments(SleepStageTotals.clampStagesToOnset(it.stagesJSON, it.effectiveStartTs)).orEmpty() } .sortedBy { it.start } .takeIf { it.size >= 2 } } else null + // #364: draw each inter-fragment wake seam as an explicit wake segment in the full-night + // hypnogram, so the merged night has no silent hole where the user was up — matching what the + // Health Connect export now writes. TIMELINE only: the seam is deliberately NOT added to the + // stage MINUTES (sumGroupStages) or groupInBedMin below — those keep the fragment-only + // accounting (asleep ≤ in-bed, see the #561 comment), and iOS mergeDay applies the same split. + val groupSegments = groupSegmentsRaw?.let { segs -> + val seams = heroGroup.zipWithNext().mapNotNull { (prev, next) -> + if (next.effectiveStartTs > prev.endTs) + PersistedSegment(prev.endTs, next.effectiveStartTs, "wake") else null + } + (segs + seams).sortedBy { it.start } + } val groupStages = if (heroGroup.size > 1) sumGroupStages(heroGroup) else null - val segments = (groupSegments ?: parsePersistedSegments(SleepStageTotals.clampStagesToOnset(session.stagesJSON, session.effectiveStartTs))) + val segments = (groupSegmentsRaw ?: parsePersistedSegments(SleepStageTotals.clampStagesToOnset(session.stagesJSON, session.effectiveStartTs))) ?.map { seg -> seg.stage to ((seg.end - seg.start) / 60f) } // #407: lay the GROUP's per-epoch motion fragment-by-fragment in `heroGroup` order (the same order // `groupSegments` lays the stage timeline), reading the already-chosen group's stored series. The From d1435f6db74e2ebe5b608cb31f8c7cf422c928e0 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 00:30:54 -0700 Subject: [PATCH 073/171] fix(import): select Oura's main sleep session for the daily rollup The per-day fold in OuraApiParser.parseSleep used first-write-wins: whichever session the API listed first for a `day` claimed every daily field, so a short nap/fragment could beat the real night. Audited across a real 24-month dataset: 36 days where totalSleepMin was 0.5-25 min while the true main sleep (250-780 min) existed adjacent, including two impossible daily restingHr values (92 bpm from a 32-min wake-dominated micro-session where the real night was 277 min at 57 bpm; 90 bpm amid 41-48 bpm neighbors) plus five near-misses at 89-100 bpm. A sudden 90+ bpm daily resting-HR spike is the visible symptom of this bug. Select the day's MAIN session instead: a `long_sleep` session always outranks any other type; among same-rank candidates, the longer total_sleep_duration wins. The winner supplies every rollup field as a unit, so fields are never mixed across two sessions on the same day. dailyMetric is upserted keyed by (deviceId, day) (WhoopStore.upsertDailyMetrics), so re-running Oura sync heals every already-imported day in place - no migration needed, the same way the earlier RHR-score corruption healed. --- .../Sources/StrandImport/OuraApiParser.swift | 46 +++++++--- .../OuraApiParserSleepTests.swift | 89 +++++++++++++++++++ 2 files changed, 123 insertions(+), 12 deletions(-) diff --git a/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift b/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift index 58a5e390b8..aeb3b5ffd7 100644 --- a/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift +++ b/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift @@ -14,6 +14,10 @@ public enum OuraApiParser { public static func parseSleep(_ docs: [[String: Any]]) -> (periods: [OuraSleepPeriod], days: [WearableDailyRow]) { var periods: [OuraSleepPeriod] = [] var byDay: [String: WearableDailyRow] = [:] + // The (rank, totalSleepMin) of the session CURRENTLY backing each day's rollup. Oura does not + // guarantee session order within `docs`, so a later, higher-priority session must still be able + // to displace an earlier one that already claimed the day — see the fold below. + var dayWinner: [String: (rank: Int, totalMin: Double)] = [:] for s in docs { guard let start = WhoopTime.parseISOWithOffset(WearableJSON.str(s, "bedtime_start")), @@ -52,19 +56,37 @@ public enum OuraApiParser { let hr = sampleSeries(s["heart_rate"]) periods.append(OuraSleepPeriod(session: session, movement30s: movement, hr: hr)) - // Fold the night onto its calendar day (Oura's "day" = the wake day). + // Fold the night onto its calendar day (Oura's "day" = the wake day). Oura can list MULTIPLE + // sessions for one day — a short nap/fragment alongside the real night — and first-write-wins + // let whichever the API happened to return first claim every daily field, including a fragment + // beating the true night (audited across a real 24-month dataset: 36 days where a sub-30-min + // fragment's near-zero totalSleepMin and skewed restingHr won over an adjacent 250-780 min main + // sleep, producing daily "resting HR" spikes to 90+ bpm from wake-dominated micro-sessions). + // Select the day's MAIN session instead: Oura marks the primary sleep `type == "long_sleep"`, + // which always outranks any other type; among same-rank candidates, the longer + // total_sleep_duration wins. The winner supplies every rollup field as a UNIT — fields are + // never mixed across two different sessions on the same day. let key = WearableJSON.str(s, "day") ?? WearableExportImporter.dayString(end) - var row = byDay[key] ?? WearableDailyRow(day: key) - row.totalSleepMin = row.totalSleepMin ?? session.totalSleepMin - row.deepMin = row.deepMin ?? session.deepMin - row.lightMin = row.lightMin ?? session.lightMin - row.remMin = row.remMin ?? session.remMin - row.awakeMin = row.awakeMin ?? session.awakeMin - row.efficiencyPct = row.efficiencyPct ?? session.efficiencyPct - row.avgHrvMs = row.avgHrvMs ?? session.avgHrvMs - row.respRateBpm = row.respRateBpm ?? session.respRateBpm - if row.restingHr == nil { row.restingHr = session.lowestHr } - byDay[key] = row + let isLongSleep = WearableJSON.str(s, "type")?.lowercased() == "long_sleep" + let candidate = (rank: isLongSleep ? 1 : 0, totalMin: session.totalSleepMin ?? 0) + let incumbent = dayWinner[key] + let winsDay = incumbent.map { + candidate.rank > $0.rank || (candidate.rank == $0.rank && candidate.totalMin > $0.totalMin) + } ?? true + if winsDay { + dayWinner[key] = candidate + var row = byDay[key] ?? WearableDailyRow(day: key) + row.totalSleepMin = session.totalSleepMin + row.deepMin = session.deepMin + row.lightMin = session.lightMin + row.remMin = session.remMin + row.awakeMin = session.awakeMin + row.efficiencyPct = session.efficiencyPct + row.avgHrvMs = session.avgHrvMs + row.respRateBpm = session.respRateBpm + row.restingHr = session.lowestHr + byDay[key] = row + } } return (periods, Array(byDay.values)) } diff --git a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift index f196bd0cfc..05fd134b82 100644 --- a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift +++ b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift @@ -71,4 +71,93 @@ final class OuraApiParserSleepTests: XCTestCase { let hugeItemPeriod = try XCTUnwrap(hugeItemPeriods.first) XCTAssertEqual(hugeItemPeriod.hr.map(\.bpm), [60, 58]) } + + // MARK: - Daily rollup main-session selection (audit: first-write-wins let a nap/fragment beat the + // real night on 36 real days, incl. two 90+ bpm daily restingHr spikes from a wake-dominated micro- + // session). The rollup must pick the `long_sleep` session regardless of doc order, never mix fields + // from two different sessions, and still fall back honestly when no long_sleep exists that day. + + func testDailyRollupPrefersLongSleepOverEarlierFragment() throws { + // A 2-min fragment is listed BEFORE the real night's long_sleep session, same day. First-write-wins + // would let the fragment's near-zero duration and its own (spiked) restingHr win the day. + let docs: [[String: Any]] = [ + [ + "day": "2026-02-01", "type": "short_sleep", + "bedtime_start": "2026-02-01T04:00:00+00:00", + "bedtime_end": "2026-02-01T04:02:00+00:00", + "total_sleep_duration": 120, "lowest_heart_rate": 92, + ], + [ + "day": "2026-02-01", "type": "long_sleep", + "bedtime_start": "2026-01-31T23:00:00+00:00", + "bedtime_end": "2026-02-01T07:00:00+00:00", + "total_sleep_duration": 27_000, "lowest_heart_rate": 57, + ], + ] + let (periods, days) = OuraApiParser.parseSleep(docs) + XCTAssertEqual(periods.count, 2) // both sessions still show up as Sleep-tab periods + XCTAssertEqual(days.count, 1) + let day = try XCTUnwrap(days.first) + XCTAssertEqual(day.totalSleepMin, 450) // 27000s ÷ 60 — the long_sleep's duration, not the fragment's + XCTAssertEqual(day.restingHr, 57) // the long_sleep's lowestHr, not the fragment's 92 + } + + func testDailyRollupPrefersLongSleepRegardlessOfOrder() throws { + // Same fixture, long_sleep listed FIRST — the fix must not depend on doc order either way. + let docs: [[String: Any]] = [ + [ + "day": "2026-02-01", "type": "long_sleep", + "bedtime_start": "2026-01-31T23:00:00+00:00", + "bedtime_end": "2026-02-01T07:00:00+00:00", + "total_sleep_duration": 27_000, "lowest_heart_rate": 57, + ], + [ + "day": "2026-02-01", "type": "short_sleep", + "bedtime_start": "2026-02-01T04:00:00+00:00", + "bedtime_end": "2026-02-01T04:02:00+00:00", + "total_sleep_duration": 120, "lowest_heart_rate": 92, + ], + ] + let (_, days) = OuraApiParser.parseSleep(docs) + let day = try XCTUnwrap(days.first) + XCTAssertEqual(day.totalSleepMin, 450) + XCTAssertEqual(day.restingHr, 57) + } + + func testDailyRollupKeepsFragmentValuesWhenNoLongSleepThatDay() throws { + // A fragment-only day (a nap with no main night recorded) keeps the fragment's own values — + // honest, not zeroed or dropped. + let docs: [[String: Any]] = [[ + "day": "2026-02-02", "type": "short_sleep", + "bedtime_start": "2026-02-02T13:00:00+00:00", + "bedtime_end": "2026-02-02T13:20:00+00:00", + "total_sleep_duration": 900, "lowest_heart_rate": 64, + ]] + let (_, days) = OuraApiParser.parseSleep(docs) + let day = try XCTUnwrap(days.first) + XCTAssertEqual(day.totalSleepMin, 15) // 900s ÷ 60 — the only session that day + XCTAssertEqual(day.restingHr, 64) + } + + func testDailyRollupPicksLongerOfTwoLongSleeps() throws { + // Equal rank (both long_sleep) → the longer total_sleep_duration wins, not doc order. + let docs: [[String: Any]] = [ + [ + "day": "2026-02-03", "type": "long_sleep", + "bedtime_start": "2026-02-02T22:00:00+00:00", + "bedtime_end": "2026-02-03T02:00:00+00:00", + "total_sleep_duration": 12_000, "lowest_heart_rate": 60, + ], + [ + "day": "2026-02-03", "type": "long_sleep", + "bedtime_start": "2026-02-03T03:00:00+00:00", + "bedtime_end": "2026-02-03T07:00:00+00:00", + "total_sleep_duration": 13_000, "lowest_heart_rate": 55, + ], + ] + let (_, days) = OuraApiParser.parseSleep(docs) + let day = try XCTUnwrap(days.first) + XCTAssertEqual(day.totalSleepMin, 13_000 / 60.0) + XCTAssertEqual(day.restingHr, 55) + } } From 9d867e4de3ada154cede49f55466ad099988dbba Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 00:34:46 -0700 Subject: [PATCH 074/171] fix(import): store Oura efficiency as a 0-1 fraction, matching the native column MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Oura API importer wrote Oura's native 0-100 integer `efficiency` straight into sleepSession.efficiency / dailyMetric.efficiency, but NOOP's own sleep pipeline (StrandAnalytics) stores that same shared column as a 0-1 FRACTION everywhere it computes it (asleep / in-bed — AnalyticsEngine.swift, SleepStageTotals.swift). Real WHOOP-era rows sit at 0.36-0.94; real imported Oura rows sat at 83-96 - same column, two scales. The UI already carries defensive normalization shims for this exact inconsistency (SleepView.swift's `if e > 1.5 { e /= 100 }`, WatchSessionBridge.swift's matching comment), which is how the mismatch was traced back to the importer. Normalize at the parse boundary in OuraApiParser.swift: divide by 100 where `efficiency` is read off the session. The day rollup derives its efficiencyPct from the same session value, so both the per-session and daily rows come out as fractions with a single change. Add a data-heal migration (v26) for rows already written with the old scale: an UPDATE-only pass, no schema change, dividing efficiency by 100 wherever deviceId = 'oura-api' AND efficiency > 1.5 - a threshold no genuine fraction exceeds and no genuine Oura percent falls under, so it's idempotent and can't touch an already-correct row. Scoped to deviceId so WHOOP-native and other-brand rows are never touched. No Android migration twin: android/'s `com.noop.oura` package is the BLE ring driver, not a REST/API cloud importer - there is nothing to heal there. --- .../Sources/StrandImport/OuraApiParser.swift | 8 +++- .../OuraApiParserSleepTests.swift | 18 +++++++++ .../Sources/WhoopStore/Database.swift | 23 +++++++++++ .../WhoopStoreTests/MigrationTests.swift | 40 +++++++++++++++++++ 4 files changed, 88 insertions(+), 1 deletion(-) diff --git a/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift b/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift index 58a5e390b8..81f8500ba4 100644 --- a/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift +++ b/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift @@ -40,7 +40,13 @@ public enum OuraApiParser { remMin: minutes("rem_sleep_duration"), awakeMin: minutes("awake_time"), totalSleepMin: minutes("total_sleep_duration"), - efficiencyPct: WearableJSON.posDbl(s, "efficiency"), + // Oura's `efficiency` is a 0-100 integer percent. NOOP's own sleep pipeline + // (StrandAnalytics) stores `efficiency` as a 0-1 FRACTION everywhere it's computed + // (asleep ÷ in-bed — see AnalyticsEngine.swift / SleepStageTotals.swift) and both + // sleepSession.efficiency and dailyMetric.efficiency are that same shared column, so + // normalize to the fraction convention here, at the parse boundary, rather than + // writing Oura's native percent straight through. + efficiencyPct: WearableJSON.posDbl(s, "efficiency").map { $0 / 100.0 }, avgHr: WearableJSON.posInt(s, "average_heart_rate"), lowestHr: WearableJSON.posInt(s, "lowest_heart_rate"), avgHrvMs: WearableJSON.posDbl(s, "average_hrv"), diff --git a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift index f196bd0cfc..c650e5e072 100644 --- a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift +++ b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift @@ -71,4 +71,22 @@ final class OuraApiParserSleepTests: XCTestCase { let hugeItemPeriod = try XCTUnwrap(hugeItemPeriods.first) XCTAssertEqual(hugeItemPeriod.hr.map(\.bpm), [60, 58]) } + + /// Oura's `efficiency` is a 0-100 integer percent. NOOP's own sleep pipeline stores + /// sleepSession.efficiency / dailyMetric.efficiency as a 0-1 FRACTION everywhere else it's + /// written (StrandAnalytics: asleep ÷ in-bed), so the importer must normalize at the parse + /// boundary rather than passing Oura's native percent straight through — both the individual + /// session and the day rollup it folds onto. + func testEfficiencyIsNormalizedToZeroToOneFraction() throws { + let docs: [[String: Any]] = [[ + "day": "2026-03-01", + "bedtime_start": "2026-02-28T23:00:00+00:00", + "bedtime_end": "2026-03-01T07:00:00+00:00", + "total_sleep_duration": 27_000, "efficiency": 92, + ]] + let (periods, days) = OuraApiParser.parseSleep(docs) + let p = try XCTUnwrap(periods.first) + XCTAssertEqual(p.session.efficiencyPct, 0.92) + XCTAssertEqual(days.first?.efficiencyPct, 0.92) + } } diff --git a/Packages/WhoopStore/Sources/WhoopStore/Database.swift b/Packages/WhoopStore/Sources/WhoopStore/Database.swift index 58d8e0272b..0db9ea648a 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/Database.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/Database.swift @@ -500,6 +500,29 @@ extension WhoopStore { try db.create(index: "idx_ouraRaw_device_endpoint_day", on: "ouraRaw", columns: ["deviceId", "endpoint", "day"]) } + + // v26 (Oura efficiency unit heal): the Oura API importer (OuraApiParser.swift) wrote Oura's + // native 0-100 integer `efficiency` straight into sleepSession.efficiency / dailyMetric.efficiency, + // but NOOP's own sleep pipeline (StrandAnalytics) stores that shared column as a 0-1 FRACTION + // everywhere it computes it (asleep ÷ in-bed) — same column, two scales for oura-api rows written + // before the importer fix. UPDATE-only, NO schema change: divides `efficiency` by 100 for every + // `deviceId = 'oura-api'` row where it's > 1.5 — a threshold no genuine fraction can exceed (max + // 1.0) and no genuine Oura percent can fall under (Oura's lowest real efficiency is well above + // 1.5%), so the predicate can't touch an already-correct row. Idempotent: a second run finds + // nothing left above the threshold. Scoped to deviceId so WHOOP-native and other-brand rows are + // never touched. No Android migration twin: this only heals data written by the Swift-only, + // OURA_CLOUD_IMPORT-gated cloud importer — android/ has no equivalent Oura REST/API import path + // (its `com.noop.oura` package is the BLE ring driver, unrelated). + migrator.registerMigration("v26-oura-efficiency-heal") { db in + try db.execute(sql: """ + UPDATE sleepSession SET efficiency = efficiency / 100.0 + WHERE deviceId = 'oura-api' AND efficiency > 1.5 + """) + try db.execute(sql: """ + UPDATE dailyMetric SET efficiency = efficiency / 100.0 + WHERE deviceId = 'oura-api' AND efficiency > 1.5 + """) + } return migrator } } diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/MigrationTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/MigrationTests.swift index 22ae1a3f1b..93822f9c50 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/MigrationTests.swift +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/MigrationTests.swift @@ -96,4 +96,44 @@ final class MigrationTests: XCTestCase { let cols = try await store.columnNamesForTest(table: "pairedDevice") XCTAssertTrue(cols.contains("peripheralId"), "pairedDevice missing v16 peripheralId column") } + + /// v26 heals `deviceId = 'oura-api'` rows written with Oura's native 0-100 `efficiency` (before the + /// OuraApiParser fix) back to NOOP's 0-1 fraction convention. UPDATE-only: seed rows at the v25 + /// schema (i.e. BEFORE v26 has run), then apply the rest of the migrator and confirm the heal. + func testV26HealsOuraEfficiencyPercentToFraction() async throws { + let dbQueue = try DatabaseQueue() + try WhoopStore.makeMigrator().migrate(dbQueue, upTo: "v25-oura-raw") + try await dbQueue.write { db in + // oura-api, bad (percent) → must be healed to a fraction. + try db.execute(sql: """ + INSERT INTO sleepSession (deviceId, startTs, endTs, efficiency) VALUES ('oura-api', 100, 200, 90) + """) + try db.execute(sql: """ + INSERT INTO dailyMetric (deviceId, day, efficiency) VALUES ('oura-api', '2026-01-01', 90) + """) + // oura-api, already a fraction → must be left unchanged (idempotent predicate: efficiency > 1.5). + try db.execute(sql: """ + INSERT INTO sleepSession (deviceId, startTs, endTs, efficiency) VALUES ('oura-api', 300, 400, 0.9) + """) + try db.execute(sql: """ + INSERT INTO dailyMetric (deviceId, day, efficiency) VALUES ('oura-api', '2026-01-02', 0.9) + """) + // A non-Oura row above the threshold must be left alone — the heal is deviceId-scoped. + try db.execute(sql: """ + INSERT INTO sleepSession (deviceId, startTs, endTs, efficiency) VALUES ('my-whoop', 500, 600, 90) + """) + } + + // Apply the FULL migrator: GRDB resumes from the already-applied v25, so only v26 runs here. + try WhoopStore.makeMigrator().migrate(dbQueue) + + try await dbQueue.read { db in + XCTAssertEqual(try Double.fetchOne(db, sql: "SELECT efficiency FROM sleepSession WHERE startTs = 100"), 0.9) + XCTAssertEqual(try Double.fetchOne(db, sql: "SELECT efficiency FROM sleepSession WHERE startTs = 300"), 0.9) + XCTAssertEqual(try Double.fetchOne(db, sql: "SELECT efficiency FROM sleepSession WHERE startTs = 500"), 90, + "a non-oura-api row must not be touched by the deviceId-scoped heal") + XCTAssertEqual(try Double.fetchOne(db, sql: "SELECT efficiency FROM dailyMetric WHERE day = '2026-01-01'"), 0.9) + XCTAssertEqual(try Double.fetchOne(db, sql: "SELECT efficiency FROM dailyMetric WHERE day = '2026-01-02'"), 0.9) + } + } } From f1b062310ca4ebaeeedbb50ab618e80f081b121a Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 01:22:24 -0700 Subject: [PATCH 075/171] fix(import): normalize WHOOP CSV efficiency to the native fraction too The WHOOP CSV importer had the same unit bug as the Oura API importer: "Sleep efficiency %" (0-100) written straight into the 0-1 fraction columns, on both platforms. Fix at the same write boundary as the Day Strain rescale, with the inverse on the CSV exporters so a NOOP->NOOP round-trip stays lossless (percent rounded to 4 decimals at emission -- num() prints shortest-round-trip Doubles, and raw fraction*100 leaks FP dust like 92.30000000000001 into the cell). The heal migration drops its deviceId scoping (now v26-efficiency-heal): both known percent writers are corrected by the same >1.5 predicate, so CSV-imported rows under arbitrary strap deviceIds heal alongside oura-api rows. Android gets the identical importer/exporter boundary fix (byte-parity helpers); healing Android's historical Room rows is left to a maintainer-owned Room migration, called out in the PR. --- .../StrandImport/WhoopCsvExporter.swift | 6 +++-- .../StrandImport/WhoopExportImporter.swift | 20 +++++++++++++++++ .../WhoopCsvExporterTests.swift | 20 +++++++++++++---- .../Sources/WhoopStore/Database.swift | 22 ++++++++++--------- .../WhoopStoreTests/MigrationTests.swift | 22 +++++++++++++------ Strand/Data/WhoopImporter.swift | 5 +++-- .../java/com/noop/ingest/WhoopCsvExporter.kt | 11 ++++++++-- .../java/com/noop/ingest/WhoopCsvImporter.kt | 14 +++++++++--- .../com/noop/ingest/WhoopCsvExporterTest.kt | 4 ++-- 9 files changed, 92 insertions(+), 32 deletions(-) diff --git a/Packages/StrandImport/Sources/StrandImport/WhoopCsvExporter.swift b/Packages/StrandImport/Sources/StrandImport/WhoopCsvExporter.swift index 4201382761..423c0fde08 100644 --- a/Packages/StrandImport/Sources/StrandImport/WhoopCsvExporter.swift +++ b/Packages/StrandImport/Sources/StrandImport/WhoopCsvExporter.swift @@ -160,7 +160,9 @@ public enum WhoopCsvExporter { // disturbance COUNT exported a wrong unit that round-tripped on reimport — PR #97 // review, tigercraft4.) num(s["awake_min"]), - num(d.efficiency), num(s["sleep_consistency"]), num(s["sleep_need_min"]), + // "Sleep efficiency %" is WHOOP's 0–100 column → convert the stored 0–1 fraction up, + // mirroring the Day Strain conversion above (importer scales it back down). + num(WhoopExportImporter.whoopEfficiencyPctFromFraction(d.efficiency)), num(s["sleep_consistency"]), num(s["sleep_need_min"]), num(s["sleep_debt_min"]), field(sourceByDay[d.day]), ] @@ -195,7 +197,7 @@ public enum WhoopCsvExporter { "false", "", "", num(stages.asleep), num(inBedMin), num(stages.light), num(stages.deep), num(stages.rem), num(stages.awake), - num(s.efficiency), "", "", "", + num(WhoopExportImporter.whoopEfficiencyPctFromFraction(s.efficiency)), "", "", "", field(sourceBySession(s)), ] out += cols.joined(separator: ",") + "\r\n" diff --git a/Packages/StrandImport/Sources/StrandImport/WhoopExportImporter.swift b/Packages/StrandImport/Sources/StrandImport/WhoopExportImporter.swift index 8b13ac532e..a229a995cd 100644 --- a/Packages/StrandImport/Sources/StrandImport/WhoopExportImporter.swift +++ b/Packages/StrandImport/Sources/StrandImport/WhoopExportImporter.swift @@ -46,6 +46,26 @@ public struct WhoopExportImporter { return effort / dayStrainToEffortScale } + /// WHOOP CSVs carry "Sleep efficiency %" on a 0–100 scale; NOOP's `efficiency` columns store the + /// 0–1 fraction the native pipeline writes (`AnalyticsEngine`: actual-sleep ÷ in-bed). Convert at + /// the WRITE boundary (WhoopImporter → store), NOT at parse time, so the verbatim parsed value + /// (`sleepEfficiencyPct`) and the CSV round-trip contract are preserved — the same shape as the + /// Day Strain ⇄ Effort pair above. Keep byte-identical to the Android importer (WhoopCsvImporter.kt). + public static func fractionFromImportedEfficiencyPct(_ pct: Double?) -> Double? { + guard let pct else { return nil } + return pct / 100.0 + } + + /// Inverse: the stored 0–1 fraction back onto the CSV's 0–100 "Sleep efficiency %" column, so an + /// exported CSV is WHOOP-compatible and a NOOP export → NOOP import round-trip is lossless to + /// 4 decimal places of a percent (1e-6 of the fraction). The rounding matters: `num()` prints + /// shortest-round-trip Doubles, and a raw `fraction * 100` carries FP dust (0.923 × 100 = + /// 92.30000000000001) straight into the CSV cell. + public static func whoopEfficiencyPctFromFraction(_ fraction: Double?) -> Double? { + guard let fraction else { return nil } + return (fraction * 100.0 * 10_000).rounded() / 10_000 + } + // Recognised CSV filenames (lowercased). private static let cyclesName = "physiological_cycles.csv" private static let sleepsName = "sleeps.csv" diff --git a/Packages/StrandImport/Tests/StrandImportTests/WhoopCsvExporterTests.swift b/Packages/StrandImport/Tests/StrandImportTests/WhoopCsvExporterTests.swift index c6f683db05..38dffeae7c 100644 --- a/Packages/StrandImport/Tests/StrandImportTests/WhoopCsvExporterTests.swift +++ b/Packages/StrandImport/Tests/StrandImportTests/WhoopCsvExporterTests.swift @@ -9,7 +9,8 @@ import WhoopStore final class WhoopCsvExporterTests: XCTestCase { func testCyclesRoundTripThroughRealImporter() throws { - let day = DailyMetric(day: "2026-06-01", totalSleepMin: 420, efficiency: 92.3, deepMin: 95, + // efficiency is stored as the native 0-1 fraction; the exporter lifts it onto the CSV's % scale. + let day = DailyMetric(day: "2026-06-01", totalSleepMin: 420, efficiency: 0.923, deepMin: 95, remMin: 115, lightMin: 210, disturbances: nil, restingHr: 52, avgHrv: 68.4, recovery: 72, strain: 12.5, exerciseCount: nil, spo2Pct: 96.0, skinTempDevC: 33.1, respRateBpm: 14.2) @@ -74,10 +75,10 @@ final class WhoopCsvExporterTests: XCTestCase { func testSleepsRoundTripAllStageShapes() { // macOS-import minutes dict, Android [{stage,min}], and the stager's segment array. let dictNight = CachedSleepSession(startTs: 1_750_000_000, endTs: 1_750_027_300, - efficiency: 92.3, restingHr: nil, avgHrv: nil, + efficiency: 0.923, restingHr: nil, avgHrv: nil, stagesJSON: #"{"light":210,"deep":95,"rem":115,"awake":35}"#) let arrayNight = CachedSleepSession(startTs: 1_760_000_000, endTs: 1_760_025_500, - efficiency: 90, restingHr: nil, avgHrv: nil, + efficiency: 0.9, restingHr: nil, avgHrv: nil, stagesJSON: #"[{"stage":"light","min":200.0},{"stage":"deep","min":80.0}]"#) let segNight = CachedSleepSession(startTs: 2_000_000_000, endTs: 2_000_007_200, efficiency: nil, restingHr: nil, avgHrv: nil, @@ -118,6 +119,17 @@ final class WhoopCsvExporterTests: XCTestCase { XCTAssertEqual(back[1].notes, "one, big \"mug\"") } + func testEfficiencyPctFractionPairIsLosslessAndNilSafe() { + // Store fraction → CSV % → store fraction round-trips exactly; nil passes through both ways. + XCTAssertEqual(WhoopExportImporter.whoopEfficiencyPctFromFraction(0.923) ?? -1, 92.3, accuracy: 1e-9) + XCTAssertEqual(WhoopExportImporter.fractionFromImportedEfficiencyPct(92.3) ?? -1, 0.923, accuracy: 1e-9) + let roundTripped = WhoopExportImporter.fractionFromImportedEfficiencyPct( + WhoopExportImporter.whoopEfficiencyPctFromFraction(0.6569)) + XCTAssertEqual(roundTripped ?? -1, 0.6569, accuracy: 1e-9) + XCTAssertNil(WhoopExportImporter.whoopEfficiencyPctFromFraction(nil)) + XCTAssertNil(WhoopExportImporter.fractionFromImportedEfficiencyPct(nil)) + } + func testNumbersAreLocaleProof() { XCTAssertEqual(WhoopCsvExporter.num(72.0), "72") XCTAssertEqual(WhoopCsvExporter.num(68.4), "68.4") @@ -132,7 +144,7 @@ final class WhoopCsvExporterTests: XCTestCase { defer { try? FileManager.default.removeItem(at: dir) } let zipURL = dir.appendingPathComponent("noop-export.zip") - let day = DailyMetric(day: "2026-06-01", totalSleepMin: 420, efficiency: 92.3, deepMin: 95, + let day = DailyMetric(day: "2026-06-01", totalSleepMin: 420, efficiency: 0.923, deepMin: 95, remMin: 115, lightMin: 210, disturbances: 35, restingHr: 52, avgHrv: 68.4, recovery: 72, strain: 12.5, exerciseCount: nil, spo2Pct: 96.0, skinTempDevC: 33.1, respRateBpm: 14.2) diff --git a/Packages/WhoopStore/Sources/WhoopStore/Database.swift b/Packages/WhoopStore/Sources/WhoopStore/Database.swift index 0db9ea648a..8873b5c615 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/Database.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/Database.swift @@ -506,21 +506,23 @@ extension WhoopStore { // but NOOP's own sleep pipeline (StrandAnalytics) stores that shared column as a 0-1 FRACTION // everywhere it computes it (asleep ÷ in-bed) — same column, two scales for oura-api rows written // before the importer fix. UPDATE-only, NO schema change: divides `efficiency` by 100 for every - // `deviceId = 'oura-api'` row where it's > 1.5 — a threshold no genuine fraction can exceed (max - // 1.0) and no genuine Oura percent can fall under (Oura's lowest real efficiency is well above - // 1.5%), so the predicate can't touch an already-correct row. Idempotent: a second run finds - // nothing left above the threshold. Scoped to deviceId so WHOOP-native and other-brand rows are - // never touched. No Android migration twin: this only heals data written by the Swift-only, - // OURA_CLOUD_IMPORT-gated cloud importer — android/ has no equivalent Oura REST/API import path - // (its `com.noop.oura` package is the BLE ring driver, unrelated). - migrator.registerMigration("v26-oura-efficiency-heal") { db in + // row where it's > 1.5 — a threshold no genuine fraction can exceed (the column's convention + // caps at 1.0: `AnalyticsEngine` writes actual-sleep ÷ in-bed) and no genuine percent-scale + // leftover can fall under (no real night is ≤1.5% efficient), so the predicate can't touch an + // already-correct row and a second run finds nothing left: idempotent. Deliberately NOT + // deviceId-scoped: both known percent writers are healed by the same predicate — the Oura API + // importer ('oura-api' rows) and the WHOOP CSV importer (rows under whatever strap deviceId the + // user imported into). No Android Room migration twin in this PR: the Kotlin CSV importer gets + // the same write-boundary fix, but healing Android's historical rows needs a Room migration a + // maintainer should own (schema-version bump + column-order pinning). + migrator.registerMigration("v26-efficiency-heal") { db in try db.execute(sql: """ UPDATE sleepSession SET efficiency = efficiency / 100.0 - WHERE deviceId = 'oura-api' AND efficiency > 1.5 + WHERE efficiency > 1.5 """) try db.execute(sql: """ UPDATE dailyMetric SET efficiency = efficiency / 100.0 - WHERE deviceId = 'oura-api' AND efficiency > 1.5 + WHERE efficiency > 1.5 """) } return migrator diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/MigrationTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/MigrationTests.swift index 93822f9c50..e805315d16 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/MigrationTests.swift +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/MigrationTests.swift @@ -97,10 +97,11 @@ final class MigrationTests: XCTestCase { XCTAssertTrue(cols.contains("peripheralId"), "pairedDevice missing v16 peripheralId column") } - /// v26 heals `deviceId = 'oura-api'` rows written with Oura's native 0-100 `efficiency` (before the - /// OuraApiParser fix) back to NOOP's 0-1 fraction convention. UPDATE-only: seed rows at the v25 - /// schema (i.e. BEFORE v26 has run), then apply the rest of the migrator and confirm the heal. - func testV26HealsOuraEfficiencyPercentToFraction() async throws { + /// v26 heals `efficiency` values stored on the 0-100 percent scale (written by the pre-fix Oura API + /// importer AND the pre-fix WHOOP CSV importer, under any deviceId) back to NOOP's 0-1 fraction + /// convention. UPDATE-only: seed rows at the v25 schema (i.e. BEFORE v26 has run), then apply the + /// rest of the migrator and confirm the heal. + func testV26HealsEfficiencyPercentToFraction() async throws { let dbQueue = try DatabaseQueue() try WhoopStore.makeMigrator().migrate(dbQueue, upTo: "v25-oura-raw") try await dbQueue.write { db in @@ -118,10 +119,15 @@ final class MigrationTests: XCTestCase { try db.execute(sql: """ INSERT INTO dailyMetric (deviceId, day, efficiency) VALUES ('oura-api', '2026-01-02', 0.9) """) - // A non-Oura row above the threshold must be left alone — the heal is deviceId-scoped. + // A WHOOP-CSV-imported percent row (arbitrary strap deviceId) must ALSO be healed — the + // pre-fix WhoopImporter wrote "Sleep efficiency %" straight through, same bug class. try db.execute(sql: """ INSERT INTO sleepSession (deviceId, startTs, endTs, efficiency) VALUES ('my-whoop', 500, 600, 90) """) + // A native fraction row under a strap deviceId must be left unchanged. + try db.execute(sql: """ + INSERT INTO sleepSession (deviceId, startTs, endTs, efficiency) VALUES ('my-whoop', 700, 800, 0.66) + """) } // Apply the FULL migrator: GRDB resumes from the already-applied v25, so only v26 runs here. @@ -130,8 +136,10 @@ final class MigrationTests: XCTestCase { try await dbQueue.read { db in XCTAssertEqual(try Double.fetchOne(db, sql: "SELECT efficiency FROM sleepSession WHERE startTs = 100"), 0.9) XCTAssertEqual(try Double.fetchOne(db, sql: "SELECT efficiency FROM sleepSession WHERE startTs = 300"), 0.9) - XCTAssertEqual(try Double.fetchOne(db, sql: "SELECT efficiency FROM sleepSession WHERE startTs = 500"), 90, - "a non-oura-api row must not be touched by the deviceId-scoped heal") + XCTAssertEqual(try Double.fetchOne(db, sql: "SELECT efficiency FROM sleepSession WHERE startTs = 500"), 0.9, + "a CSV-imported percent row must be healed regardless of deviceId") + XCTAssertEqual(try Double.fetchOne(db, sql: "SELECT efficiency FROM sleepSession WHERE startTs = 700"), 0.66, + "a native fraction row must never be touched") XCTAssertEqual(try Double.fetchOne(db, sql: "SELECT efficiency FROM dailyMetric WHERE day = '2026-01-01'"), 0.9) XCTAssertEqual(try Double.fetchOne(db, sql: "SELECT efficiency FROM dailyMetric WHERE day = '2026-01-02'"), 0.9) } diff --git a/Strand/Data/WhoopImporter.swift b/Strand/Data/WhoopImporter.swift index ecdabed24c..a921913ca4 100644 --- a/Strand/Data/WhoopImporter.swift +++ b/Strand/Data/WhoopImporter.swift @@ -23,7 +23,8 @@ enum WhoopImporter { metrics.append(DailyMetric( day: day, totalSleepMin: c.asleepDurationMin, - efficiency: c.sleepEfficiencyPct, + // CSV "Sleep efficiency %" (0–100) → the store's native 0–1 fraction at the boundary. + efficiency: WhoopExportImporter.fractionFromImportedEfficiencyPct(c.sleepEfficiencyPct), deepMin: c.deepSleepDurationMin, remMin: c.remDurationMin, lightMin: c.lightSleepDurationMin, @@ -54,7 +55,7 @@ enum WhoopImporter { sessions.append(CachedSleepSession( startTs: Int(onset.timeIntervalSince1970), endTs: Int(wake.timeIntervalSince1970), - efficiency: s.sleepEfficiencyPct, + efficiency: WhoopExportImporter.fractionFromImportedEfficiencyPct(s.sleepEfficiencyPct), restingHr: nil, avgHrv: nil, stagesJSON: json)) } diff --git a/android/app/src/main/java/com/noop/ingest/WhoopCsvExporter.kt b/android/app/src/main/java/com/noop/ingest/WhoopCsvExporter.kt index 5868efd27f..cfec7766fd 100644 --- a/android/app/src/main/java/com/noop/ingest/WhoopCsvExporter.kt +++ b/android/app/src/main/java/com/noop/ingest/WhoopCsvExporter.kt @@ -17,6 +17,7 @@ import java.util.zip.ZipEntry import java.util.zip.ZipOutputStream import kotlin.math.abs import kotlin.math.floor +import kotlin.math.round /** * Serializes NOOP's own cached rows back into WHOOP's 4-CSV export shape so NOOP's OWN importer @@ -188,7 +189,12 @@ object WhoopCsvExporter { // the cell empty. (Writing the disturbance COUNT here exported a wrong unit // that round-tripped on reimport — PR #97 review, tigercraft4. Swift parity.) "", - num(d.efficiency), num(s["sleep_consistency"]), num(s["sleep_need_min"]), + // "Sleep efficiency %" is WHOOP's 0–100 column → lift the stored 0–1 fraction, + // rounded to 4 decimals of a percent so num()'s shortest-round-trip Double + // printing can't leak FP dust into the cell. Keep byte-identical to Swift + // (WhoopExportImporter.whoopEfficiencyPctFromFraction). + num(d.efficiency?.let { round(it * 100.0 * 10_000) / 10_000 }), + num(s["sleep_consistency"]), num(s["sleep_need_min"]), num(s["sleep_debt_min"]), csvField(sourceByDay[d.day]), ).joinToString(","), ).append("\r\n") @@ -228,7 +234,8 @@ object WhoopCsvExporter { "false", "", "", num(stages.asleep), num(inBedMin), num(stages.light), num(stages.deep), num(stages.rem), num(stages.awake), - num(s.efficiency), "", "", "", csvField(sourceBySession(s)), + num(s.efficiency?.let { round(it * 100.0 * 10_000) / 10_000 }), "", "", "", + csvField(sourceBySession(s)), ).joinToString(","), ).append("\r\n") } diff --git a/android/app/src/main/java/com/noop/ingest/WhoopCsvImporter.kt b/android/app/src/main/java/com/noop/ingest/WhoopCsvImporter.kt index be72b8c517..9dafad8cbc 100644 --- a/android/app/src/main/java/com/noop/ingest/WhoopCsvImporter.kt +++ b/android/app/src/main/java/com/noop/ingest/WhoopCsvImporter.kt @@ -68,6 +68,14 @@ object WhoopCsvImporter { */ private const val DAY_STRAIN_TO_EFFORT_SCALE = 100.0 / 21.0 + /** + * WHOOP CSVs carry "Sleep efficiency %" on a 0–100 scale, but NOOP's `efficiency` columns store + * the 0–1 fraction the native pipeline writes (AnalyticsEngine: actual-sleep ÷ in-bed). Divide at + * the WRITE boundary — the verbatim parsed value stays untouched, same shape as the Day Strain + * rescale above. Keep byte-identical to Swift (WhoopExportImporter.fractionFromImportedEfficiencyPct). + */ + private fun efficiencyFractionFromPct(pct: Double?): Double? = pct?.let { it / 100.0 } + private val DAY_FMT: DateTimeFormatter = DateTimeFormatter.ofPattern("yyyy-MM-dd") /** @@ -353,7 +361,7 @@ object WhoopCsvImporter { deviceId = deviceId, day = day, totalSleepMin = asleepMin, - efficiency = efficiency, + efficiency = efficiencyFractionFromPct(efficiency), deepMin = deepMin, remMin = remMin, lightMin = lightMin, @@ -443,7 +451,7 @@ object WhoopCsvImporter { deviceId = deviceId, startTs = startTs, endTs = if (endTs >= startTs) endTs else startTs, - efficiency = efficiency, + efficiency = efficiencyFractionFromPct(efficiency), restingHr = null, // resting HR is a cycles/recovery field, not in sleeps.csv avgHrv = null, // HRV likewise is a recovery field, not in sleeps.csv stagesJSON = stagesJson(lightMin, deepMin, remMin, awakeMin), @@ -467,7 +475,7 @@ object WhoopCsvImporter { deviceId = deviceId, day = epochSecondsToDay(dayTs, tz), totalSleepMin = asleepMin, - efficiency = efficiency, + efficiency = efficiencyFractionFromPct(efficiency), deepMin = deepMin, remMin = remMin, lightMin = lightMin, diff --git a/android/app/src/test/java/com/noop/ingest/WhoopCsvExporterTest.kt b/android/app/src/test/java/com/noop/ingest/WhoopCsvExporterTest.kt index c950ed52a7..494cdb86fa 100644 --- a/android/app/src/test/java/com/noop/ingest/WhoopCsvExporterTest.kt +++ b/android/app/src/test/java/com/noop/ingest/WhoopCsvExporterTest.kt @@ -26,7 +26,7 @@ class WhoopCsvExporterTest { fun cyclesRoundTripThroughRealParser() { val daily = listOf( DailyMetric( - deviceId = "my-whoop", day = "2026-06-01", totalSleepMin = 420.0, efficiency = 92.3, + deviceId = "my-whoop", day = "2026-06-01", totalSleepMin = 420.0, efficiency = 0.923, deepMin = 95.0, remMin = 115.0, lightMin = 210.0, disturbances = 35, restingHr = 52, avgHrv = 68.4, recovery = 72.0, strain = 12.5, exerciseCount = null, spo2Pct = 96.0, skinTempDevC = 33.1, respRateBpm = 14.2, @@ -101,7 +101,7 @@ class WhoopCsvExporterTest { // Android-import shape [{stage,min}] — minutes survive exactly. val imported = SleepSession( deviceId = "my-whoop", startTs = 1_750_000_000L, endTs = 1_750_030_000L, - efficiency = 91.0, restingHr = null, avgHrv = null, + efficiency = 0.91, restingHr = null, avgHrv = null, stagesJSON = """[{"stage":"light","min":210.0},{"stage":"deep","min":95.0},""" + """{"stage":"rem","min":115.0},{"stage":"awake","min":35.0}]""", ) From 03f2e465eed71a456e489071fc3f6420e4d05aaf Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 01:24:19 -0700 Subject: [PATCH 076/171] fix(import): skip Oura 'deleted' sleep periods in the API parser The CSV-side OuraExportParser already skips type == "deleted" rows (a night the user removed in the Oura app, #862); the API parser had no type check at all, so a deleted period still became a session and -- with its full bedtime span -- could even win the day's rollup over the real night. Skip it before it becomes a period, same rule both parsers. --- .../Sources/StrandImport/OuraApiParser.swift | 5 +++++ .../OuraApiParserSleepTests.swift | 22 +++++++++++++++++++ 2 files changed, 27 insertions(+) diff --git a/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift b/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift index aeb3b5ffd7..e5741412e4 100644 --- a/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift +++ b/Packages/StrandImport/Sources/StrandImport/OuraApiParser.swift @@ -24,6 +24,11 @@ public enum OuraApiParser { let end = WhoopTime.parseISOWithOffset(WearableJSON.str(s, "bedtime_end")), end > start else { continue } + // A `deleted` sleep period (Oura's `type` enum) is a night the user removed in the Oura + // app — skip it entirely so it neither becomes a session nor competes for the day's + // rollup, matching OuraExportParser's CSV-side rule (#862). + if WearableJSON.str(s, "type")?.lowercased() == "deleted" { continue } + // Durations are SECONDS in the API → minutes. func minutes(_ k: String) -> Double? { WearableJSON.posDbl(s, k).map { $0 / 60.0 } } diff --git a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift index 05fd134b82..396c10bcd5 100644 --- a/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift +++ b/Packages/StrandImport/Tests/StrandImportTests/OuraApiParserSleepTests.swift @@ -42,6 +42,28 @@ final class OuraApiParserSleepTests: XCTestCase { XCTAssertTrue(periods.isEmpty) } + func testSkipsDeletedPeriodEntirely() throws { + // A user-removed night (`type == "deleted"`) must neither become a session nor compete for + // the day's rollup — same rule as OuraExportParser's CSV side (#862). The deleted doc here + // would otherwise WIN the day on duration. + let docs: [[String: Any]] = [ + ["id": "dead", "day": "2026-01-02", "type": "deleted", + "bedtime_start": "2026-01-01T22:00:00+00:00", + "bedtime_end": "2026-01-02T07:00:00+00:00", + "total_sleep_duration": 30000, "lowest_heart_rate": 44, "efficiency": 95], + ["id": "real", "day": "2026-01-02", "type": "long_sleep", + "bedtime_start": "2026-01-02T00:00:00+00:00", + "bedtime_end": "2026-01-02T06:00:00+00:00", + "total_sleep_duration": 19800, "lowest_heart_rate": 50, "efficiency": 90], + ] + let (periods, days) = OuraApiParser.parseSleep(docs) + XCTAssertEqual(periods.count, 1) + XCTAssertEqual(try XCTUnwrap(periods.first).session.lowestHr, 50) + XCTAssertEqual(days.count, 1) + XCTAssertEqual(days.first?.totalSleepMin, 330) // 19800s ÷ 60, from the real night + XCTAssertEqual(days.first?.restingHr, 50) + } + func testSampleSeriesSkipsFiniteButHugeValuesWithoutCrashing() throws { // Both `ts` and `bpm` in sampleSeries are Double->Int casts; a finite-but-huge value for EITHER // (a crafted `interval` or a crafted `items` element) must be skipped, never trap Int(...). From d85c386f8cafd08c6ccd5b3fdf142515910c2832 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 12:45:56 -0700 Subject: [PATCH 077/171] fix(cloudsync): recompute daily rollups after cloud edits apply Applied edits rewrite stored sessions, but the daily rollups the UI headlines read are only rebuilt by the analyze cycle, which a pull never triggered -- 14 restaged nights applied cleanly while the Sleep/Today totals kept showing pre-edit numbers (verified against the on-device DB: sessions userEdited=1 with corrected stagesJSON, dailyMetric stale). performSync now runs an injected analyzeRecent+refresh whenever the applied-edits cursor is ahead of a new cloud_edits_recomputed cursor -- cursor-vs-cursor, not applied>0, so a build that gains this hook after edits already landed still heals on its first sync. The Apple Health re-export is forced after a recompute so exported vitals read the fresh dailies. Wired at the Sync-now button and both launch seams. --- Strand/App/RootView.swift | 10 ++++++- Strand/CloudSync/CloudSyncModel.swift | 39 ++++++++++++++++++++++++--- Strand/Screens/DataSourcesView.swift | 7 +++++ StrandiOS/App/RootTabView.swift | 10 ++++++- 4 files changed, 61 insertions(+), 5 deletions(-) diff --git a/Strand/App/RootView.swift b/Strand/App/RootView.swift index 3a88f17ec6..d952ad27f2 100644 --- a/Strand/App/RootView.swift +++ b/Strand/App/RootView.swift @@ -189,6 +189,7 @@ struct RootView: View { // status pill is isolated into SidebarStatus so HR/frame ticks don't re-render the whole // NavigationSplitView shell + sidebar list. @EnvironmentObject var repo: Repository + @EnvironmentObject var model: AppModel /// Cross-screen navigation requests (e.g. Live → "Manage devices"). Observed here so a screen can /// switch the sidebar selection without owning it — see `NavRouter`. @EnvironmentObject var router: NavRouter @@ -309,7 +310,14 @@ struct RootView: View { // whose async functions the CALLER wraps in `Task.detached` — this needs no detached wrapper // of its own to stay off the launch-critical path. The 20h gate lives in // `CloudSyncModel.isAutoSyncDue`. - CloudSyncModel().autoSyncIfDue(repo: repo) + let cloudSync = CloudSyncModel() + let intelligence = model.intelligence + let repository = repo + cloudSync.postApplyRefresh = { + await intelligence.analyzeRecent() + await repository.refresh() + } + cloudSync.autoSyncIfDue(repo: repo) #endif } // Honour a cross-screen request to open a top-level destination (e.g. Live's "Manage devices"), diff --git a/Strand/CloudSync/CloudSyncModel.swift b/Strand/CloudSync/CloudSyncModel.swift index 1f98aaebc7..64c8883a05 100644 --- a/Strand/CloudSync/CloudSyncModel.swift +++ b/Strand/CloudSync/CloudSyncModel.swift @@ -48,6 +48,20 @@ final class CloudSyncModel: ObservableObject { /// True once both a server URL and a token are stored in the Keychain (`CloudSyncSettings`). @Published var isConfigured = CloudSyncSettings.isConfigured + /// Recompute-and-refresh hook the wiring point supplies (IntelligenceEngine.analyzeRecent + + /// Repository.refresh). Cloud edits rewrite stored sessions/workouts, but the daily rollups the + /// UI headlines read (`dailyMetric.totalSleepMin`, efficiency, Rest/Charge) are only rebuilt by + /// the analyze cycle — which a pull does NOT trigger, so applied edits left the Sleep/Today + /// numbers stale until the next strap-data analysis (found live: 14 restaged nights applied while + /// the headline kept the pre-edit total). `performSync` invokes this whenever the applied-edits + /// cursor is ahead of `recomputedCursorName`. + var postApplyRefresh: (() async -> Void)? + + /// Cursor (same `cursors` table as `CloudSyncCoordinator.cursorName`) recording the highest + /// journal seq the rollups were recomputed for. Cursor-vs-cursor — not `summary.applied > 0` — + /// so a build that gains this hook AFTER edits were already applied still heals on its first sync. + static let recomputedCursorName = "cloud_edits_recomputed" + /// Validates `url` parses as an absolute URL (scheme + host) via `URLComponents` before saving. /// `CloudSyncClient.url(path:query:)` force-unwraps its `URLComponents`-built request URL, so a /// garbage base URL saved here would crash later at the first pull instead of failing this save @@ -162,8 +176,26 @@ final class CloudSyncModel: ObservableObject { if summary.needsAttention > 0 { parts.append("\(summary.needsAttention) attention") } line = parts.joined(separator: " · ") succeeded = true + // Rebuild the daily rollups before the Apple Health re-export, so the vitals it + // exports (which read dailyMetric) reflect the just-applied edits too. + var didRecompute = false + if let refresh = postApplyRefresh { + let applied = (try? await store.cursor(CloudSyncCoordinator.cursorName)) ?? 0 + let recomputed = (try? await store.cursor(Self.recomputedCursorName)) ?? 0 + if applied > recomputed { + statusText = "Recomputing days touched by edits…" + await refresh() + try? await store.setCursor(Self.recomputedCursorName, applied) + didRecompute = true + parts.append("recomputed") + line = parts.joined(separator: " · ") + } + } #if os(iOS) && canImport(HealthKit) - line += await reExportToAppleHealthIfNeeded(repo: repo, store: store, summary: summary) + // force after a recompute: the vitals the write-back exports read dailyMetric, so a + // rollup rebuild needs a re-export even when THIS pull itself applied nothing. + line += await reExportToAppleHealthIfNeeded(repo: repo, store: store, summary: summary, + force: didRecompute) #endif } catch { line = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription @@ -270,8 +302,9 @@ final class CloudSyncModel: ObservableObject { /// `refreshAuthIfPreviouslyGranted` only READS share status, it never prompts — before attempting /// the write-back. Returns a status-line SUFFIX: empty when nothing ran or it succeeded. private func reExportToAppleHealthIfNeeded(repo: Repository, store: WhoopStore, - summary: CloudApplySummary) async -> String { - guard summary.touchedSleep || summary.touchedWorkouts || summary.touchedHr else { return "" } + summary: CloudApplySummary, + force: Bool = false) async -> String { + guard force || summary.touchedSleep || summary.touchedWorkouts || summary.touchedHr else { return "" } do { let bridge = HealthKitBridge(repo: repo, appleDeviceId: "apple-health", noopDeviceId: "my-whoop") bridge.refreshAuthIfPreviouslyGranted() diff --git a/Strand/Screens/DataSourcesView.swift b/Strand/Screens/DataSourcesView.swift index b6bafef1db..3cb7351601 100644 --- a/Strand/Screens/DataSourcesView.swift +++ b/Strand/Screens/DataSourcesView.swift @@ -451,6 +451,13 @@ struct DataSourcesView: View { VStack(alignment: .leading, spacing: NoopMetrics.space3) { HStack(spacing: NoopMetrics.space3) { Button { + // Recompute rollups after applied edits — see CloudSyncModel.postApplyRefresh. + let intelligence = model.intelligence + let repository = repo + cloudSync.postApplyRefresh = { + await intelligence.analyzeRecent() + await repository.refresh() + } cloudSync.syncNow(repo: repo) } label: { Label(cloudSync.busy ? "Syncing…" : "Sync now", systemImage: "arrow.triangle.2.circlepath") diff --git a/StrandiOS/App/RootTabView.swift b/StrandiOS/App/RootTabView.swift index b5766a51cb..65ebce874c 100644 --- a/StrandiOS/App/RootTabView.swift +++ b/StrandiOS/App/RootTabView.swift @@ -7,6 +7,7 @@ import StrandDesign /// "More" list. Every screen is the same `StrandDesign`-built view the macOS app uses. struct RootTabView: View { @EnvironmentObject private var repo: Repository + @EnvironmentObject private var model: AppModel /// Cross-screen navigation requests (e.g. Live → "Manage devices"). Devices isn't a tab — it lives /// behind the More list — so a request presents it as a sheet, matching the quick-action screens. @EnvironmentObject private var router: NavRouter @@ -122,7 +123,14 @@ struct RootTabView: View { #if CLOUD_SYNC // Cloud Sync: zero-touch on-launch catch-up (see RootView's identical wiring). No detached // wrapper needed — `autoSyncIfDue` spawns its own `Task` internally and returns immediately. - CloudSyncModel().autoSyncIfDue(repo: repo) + let cloudSync = CloudSyncModel() + let intelligence = model.intelligence + let repository = repo + cloudSync.postApplyRefresh = { + await intelligence.analyzeRecent() + await repository.refresh() + } + cloudSync.autoSyncIfDue(repo: repo) #endif } // Quick-action sheet presents with the calm easing (~0.42s) per the README sheet spec — From 1cdaf6a05af482858f38e56d394575deae46161b Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 12:47:08 -0700 Subject: [PATCH 078/171] chore: remove personal signing remap accidentally committed in bf2e90d4 The com.vkchitti bundle-id/team remap must live only in the working tree (git add -A during a merge-conflict resolution swept it in). Restores the upstream identifiers in the committed tree; the remap returns as an uncommitted working-tree change. --- NOOPWatch/Info.plist | 2 +- project.yml | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/NOOPWatch/Info.plist b/NOOPWatch/Info.plist index aced8c59f9..200e610718 100644 --- a/NOOPWatch/Info.plist +++ b/NOOPWatch/Info.plist @@ -29,6 +29,6 @@ WKApplication WKCompanionAppBundleIdentifier - com.vkchitti.noop + com.noopapp.noop diff --git a/project.yml b/project.yml index 1d547e4d55..316f2e7653 100644 --- a/project.yml +++ b/project.yml @@ -21,13 +21,13 @@ settings: # String Catalog on build (the English (US) base entries). SWIFT_EMIT_LOC_STRINGS: YES SWIFT_STRICT_CONCURRENCY: minimal - DEVELOPMENT_TEAM: "R78VP2V5AQ" + DEVELOPMENT_TEAM: "" CODE_SIGN_STYLE: Automatic # Single source of truth for the App Group id shared by the iOS app + its widget / Live Activity # extension. Referenced as $(APP_GROUP_ID) by both targets' entitlements and Info.plist, and read # at runtime via the AppGroupIdentifier Info.plist key (WidgetSnapshot.suiteName) so the value is # never hard-coded in Swift. Change it in this one place when building under your own Apple ID. - APP_GROUP_ID: group.com.vkchitti.noop + APP_GROUP_ID: group.com.noopapp.noop.staging # Build for both Apple Silicon and Intel so the release runs on every Mac. ARCHS: "$(ARCHS_STANDARD)" configs: @@ -296,7 +296,7 @@ targets: - $(APP_GROUP_ID) settings: base: - PRODUCT_BUNDLE_IDENTIFIER: com.vkchitti.noop + PRODUCT_BUNDLE_IDENTIFIER: com.noopapp.noop PRODUCT_NAME: "NOOP Staging" ASSETCATALOG_COMPILER_APPICON_NAME: AppIcon # Ship every *.appiconset (not just the primary AppIcon) so the alternate "AppIcon-Navy" @@ -318,7 +318,7 @@ targets: # installs by riding inside its iOS companion; without this it would never reach a paired watch. # Build-from-source delivers it for certain; sideload install of the embedded watch app under a free # Apple ID is unproven (an Apple limitation), so the docs steer watch users to a source build for now. - # NOOPWatch embed disabled locally (personal sideload) + - target: NOOPWatch # WidgetKit + Live Activity extension — the iOS analogue of the macOS menu-bar # extra. Reads a tiny snapshot the app publishes into the shared App Group. @@ -355,7 +355,7 @@ targets: - $(APP_GROUP_ID) settings: base: - PRODUCT_BUNDLE_IDENTIFIER: com.vkchitti.noop.widgets + PRODUCT_BUNDLE_IDENTIFIER: com.noopapp.noop.widgets PRODUCT_NAME: NOOPWidgets GENERATE_INFOPLIST_FILE: NO TARGETED_DEVICE_FAMILY: "1,2" @@ -395,7 +395,7 @@ targets: # Modern single-target watchOS app — no separate WatchKit extension. WKApplication: true # Pairs this watch app to the iPhone companion (must match NOOPiOS PRODUCT_BUNDLE_IDENTIFIER). - WKCompanionAppBundleIdentifier: com.vkchitti.noop + WKCompanionAppBundleIdentifier: com.noopapp.noop NSHealthShareUsageDescription: "NOOP reads your heart rate from the Watch sensor to show a live readout on your wrist. It stays on your device." NSHealthUpdateUsageDescription: "NOOP records a basic workout session on your Watch for a higher-fidelity heart rate. The session stays on your device." entitlements: @@ -407,7 +407,7 @@ targets: - $(APP_GROUP_ID) settings: base: - PRODUCT_BUNDLE_IDENTIFIER: com.vkchitti.noop.watch + PRODUCT_BUNDLE_IDENTIFIER: com.noopapp.noop.watch # The bundle FILE is NOOPWatch.app so it doesn't collide with the iPhone app's NOOP.app when it is # embedded at NOOP.app/Watch/NOOPWatch.app. The user still sees "NOOP" on the wrist via the # CFBundleName / CFBundleDisplayName above. @@ -457,7 +457,7 @@ targets: - $(APP_GROUP_ID) settings: base: - PRODUCT_BUNDLE_IDENTIFIER: com.vkchitti.noop.watch.complications + PRODUCT_BUNDLE_IDENTIFIER: com.noopapp.noop.watch.complications PRODUCT_NAME: NOOPWatchComplications GENERATE_INFOPLIST_FILE: NO TARGETED_DEVICE_FAMILY: "4" From 1ee1892c8263d074214c3e1dc35e4a2a142eb65a Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 13:02:46 -0700 Subject: [PATCH 079/171] fix(cloudsync): stage-lock cloud stage edits against selfHealEditedStages selfHealEditedStages re-derives stages from raw for every userEdited night and rewrites whenever the derivation differs from what's stored. That equality check assumes stored stages CAME from the stager -- a deliberate cloud stage correction always differs, so the first analyzeRecent after apply silently reverted all 14 restaged nights (verified on-device: stagesJSON back to the stager output, byte-level). CloudEditApplier now records a stagelock cursor per corrected session; the healer skips locked rows. Cursors table, no schema change; a backup restored without the locks self-heals via journal re-pull (re-apply + re-lock). --- Strand/CloudSync/CloudEditApplier.swift | 7 +++++++ Strand/Data/Repository.swift | 6 ++++++ 2 files changed, 13 insertions(+) diff --git a/Strand/CloudSync/CloudEditApplier.swift b/Strand/CloudSync/CloudEditApplier.swift index 6895024a97..fa5e91549c 100644 --- a/Strand/CloudSync/CloudEditApplier.swift +++ b/Strand/CloudSync/CloudEditApplier.swift @@ -193,6 +193,13 @@ enum CloudEditApplier { } _ = try await store.updateSleepStages(deviceId: p.deviceId, detectedStartTs: p.startTs, stagesJSON: stagesJSON) + // Stage-lock: a cloud stage edit is an AUTHORITATIVE correction, not a placeholder to be + // re-derived. Without this, `Repository.selfHealEditedStages` (run by every analyzeRecent) + // restages the night from raw and — since a deliberate correction always differs from the + // stager's output — silently reverts it (found live: 14 applied restages clobbered by the + // first recompute). The lock lives in the `cursors` table (no schema change); a backup + // restored without it self-heals, because the journal re-pull re-applies + re-locks. + try? await store.setCursor("stagelock:\(p.deviceId):\(p.startTs)", 1) summary.applied += 1 summary.touchedSleep = true } catch { diff --git a/Strand/Data/Repository.swift b/Strand/Data/Repository.swift index a16ea2ec5d..7c839e0144 100644 --- a/Strand/Data/Repository.swift +++ b/Strand/Data/Repository.swift @@ -1280,6 +1280,12 @@ final class Repository: ObservableObject { guard !edited.isEmpty else { return [] } var healed = false for row in edited { + // Stage-locked rows carry an AUTHORITATIVE stage correction (cloud edit_sleep_stages — + // see CloudEditApplier's lock write). Re-deriving from raw would revert the correction + // on every analyze pass, since a deliberate edit always differs from the stager's output. + if ((try? await store.cursor("stagelock:\(computedDeviceId):\(row.startTs)")) ?? 0) == 1 { + continue + } // Re-derive over the LOCKED corrected window (effective onset → wake). Skip when the raw // isn't dense yet, or when the result already matches what's stored (steady state , no write). guard let newJSON = await restageFromRaw(start: row.effectiveStartTs, end: row.endTs), From 05e2b9eda6479bcb7a4cc16c909bf0158f730d60 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 13:46:41 -0700 Subject: [PATCH 080/171] fix(cloudsync): pull-first upload-last ordering + skip unchanged uploads MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit performSync uploaded before pulling, so an edit applied during a sync only reached the server's mirror on the NEXT sync's upload, one cycle late. Reorder to pull + apply edits + recompute + re-export first, then upload — the upload now ships the just-corrected state immediately. Add WhoopStore.contentToken(), a cheap composite fingerprint (count+max per data table, cursors table excluded) of what an upload would ship. performSync compares it against the token saved after the last successful upload and skips the re-export + POST when nothing changed, avoiding a redundant 100-300MB round trip on every sync. --- .../Sources/WhoopStore/ContentToken.swift | 48 ++++++++++++ .../WhoopStoreTests/ContentTokenTests.swift | 75 +++++++++++++++++++ Strand/CloudSync/CloudSyncModel.swift | 69 ++++++++++++++--- 3 files changed, 180 insertions(+), 12 deletions(-) create mode 100644 Packages/WhoopStore/Sources/WhoopStore/ContentToken.swift create mode 100644 Packages/WhoopStore/Tests/WhoopStoreTests/ContentTokenTests.swift diff --git a/Packages/WhoopStore/Sources/WhoopStore/ContentToken.swift b/Packages/WhoopStore/Sources/WhoopStore/ContentToken.swift new file mode 100644 index 0000000000..8cc0f82a30 --- /dev/null +++ b/Packages/WhoopStore/Sources/WhoopStore/ContentToken.swift @@ -0,0 +1,48 @@ +import Foundation +import GRDB + +extension WhoopStore { + /// Cheap composite fingerprint of the store's DATA tables — deliberately excluding `cursors` (see + /// below) — that changes if and only if something an upload would actually ship has changed. + /// `CloudSyncModel.performSync` (Cloud Sync v2) compares this against the token it saved after the + /// last successful upload FROM THIS DEVICE (`UserDefaults["cloudsync.lastUploadToken"]`) to skip a + /// redundant re-export + POST of an unchanged 100-300MB `.noopbak`. + /// + /// Format: `"hr::|sleep::|daily::|workout:|series:"` + /// — opaque, comparable only to itself, never parsed back apart. The `hr`/`sleep`/`daily` segments + /// mirror `hrFingerprint`'s (count, max) change-detector shape (Reads.swift): a SQL `COUNT(*)` plus a + /// `COALESCE(MAX(…), default)` aggregate over one table, computed entirely in SQLite with no row + /// materialized into Swift. `workout` and `metricSeries` use `COUNT` alone: `workout` rows are + /// rewritten IN PLACE by `fix_workout`/`delete_workout` (no column is a safe append-only max to key + /// on) and `metricSeries.day` is written far out of order by design — for both, a changed COUNT + /// already catches every insert/delete/tombstone, and an in-place correction that leaves a table's + /// own row count unchanged (e.g. `fix_workout`'s patch) still moves the `hrSample`/`dailyMetric` + /// segments in the same edit, so the composite token still changes. + /// + /// Deliberately EXCLUDES `cursors`: that table is sync BOOKKEEPING (`cloud_edits`, + /// `cloud_edits_recomputed`, `stagelock:…`, the `highwater:`/`read:` cursors) rather than user data. + /// A cursor write must never dirty the token — a pull that applies zero edits can still advance + /// `cloud_edits`, and an unrelated stream's highwater bump happens on nearly every sync; if either + /// counted, the token would read "changed" on almost every sync regardless of whether any actual + /// data moved, defeating the whole point of skipping the upload. (The saved comparison value itself, + /// `cloudsync.lastUploadToken`, lives in `UserDefaults` on `CloudSyncModel`, not in this store at + /// all — so writing it can never feed back into this computation either.) + public func contentToken() async throws -> String { + try syncRead { db in + let hrCount = try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM hrSample") ?? 0 + let hrMaxTs = try Int.fetchOne(db, sql: "SELECT COALESCE(MAX(ts), 0) FROM hrSample") ?? 0 + + let sleepCount = try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM sleepSession") ?? 0 + let sleepMaxEndTs = try Int.fetchOne(db, sql: "SELECT COALESCE(MAX(endTs), 0) FROM sleepSession") ?? 0 + + let dailyCount = try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM dailyMetric") ?? 0 + let dailyMaxDay = try String.fetchOne(db, sql: "SELECT COALESCE(MAX(day), '') FROM dailyMetric") ?? "" + + let workoutCount = try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM workout") ?? 0 + let seriesCount = try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM metricSeries") ?? 0 + + return "hr:\(hrCount):\(hrMaxTs)|sleep:\(sleepCount):\(sleepMaxEndTs)|daily:\(dailyCount):\(dailyMaxDay)" + + "|workout:\(workoutCount)|series:\(seriesCount)" + } + } +} diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/ContentTokenTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/ContentTokenTests.swift new file mode 100644 index 0000000000..762187c449 --- /dev/null +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/ContentTokenTests.swift @@ -0,0 +1,75 @@ +import XCTest +import WhoopProtocol +@testable import WhoopStore + +final class ContentTokenTests: XCTestCase { + func testTokenIsStableAndDeterministicOnAnEmptyStore() async throws { + let store = try await WhoopStore.inMemory() + let a = try await store.contentToken() + let b = try await store.contentToken() + XCTAssertEqual(a, b) + XCTAssertEqual(a, "hr:0:0|sleep:0:0|daily:0:|workout:0|series:0") + } + + func testInsertingHrSampleChangesToken() async throws { + let store = try await WhoopStore.inMemory() + let before = try await store.contentToken() + _ = try await store.insert(Streams(hr: [HRSample(ts: 100, bpm: 60)]), deviceId: "dev1") + let after = try await store.contentToken() + XCTAssertNotEqual(before, after) + } + + func testInsertingSleepSessionChangesToken() async throws { + let store = try await WhoopStore.inMemory() + let before = try await store.contentToken() + _ = try await store.upsertSleepSessions( + [CachedSleepSession(startTs: 1000, endTs: 5000, efficiency: nil, restingHr: nil, + avgHrv: nil, stagesJSON: nil)], deviceId: "dev1") + let after = try await store.contentToken() + XCTAssertNotEqual(before, after) + } + + func testInsertingDailyMetricChangesToken() async throws { + let store = try await WhoopStore.inMemory() + let before = try await store.contentToken() + _ = try await store.upsertDailyMetrics( + [DailyMetric(day: "2026-05-23", totalSleepMin: 420, efficiency: 0.9, deepMin: nil, + remMin: nil, lightMin: nil, disturbances: nil, restingHr: nil, avgHrv: nil, + recovery: nil, strain: nil, exerciseCount: nil)], deviceId: "dev1") + let after = try await store.contentToken() + XCTAssertNotEqual(before, after) + } + + func testInsertingWorkoutChangesToken() async throws { + let store = try await WhoopStore.inMemory() + let before = try await store.contentToken() + _ = try await store.upsertWorkouts( + [WorkoutRow(startTs: 100, endTs: 200, sport: "running", source: "whoop", durationS: 100, + energyKcal: nil, avgHr: nil, maxHr: nil, strain: nil, distanceM: nil, + zonesJSON: nil, notes: nil)], deviceId: "dev1") + let after = try await store.contentToken() + XCTAssertNotEqual(before, after) + } + + func testInsertingMetricSeriesPointChangesToken() async throws { + let store = try await WhoopStore.inMemory() + let before = try await store.contentToken() + _ = try await store.upsertMetricSeries( + [MetricPoint(day: "2026-05-20", key: "restingHr", value: 54)], deviceId: "dev1") + let after = try await store.contentToken() + XCTAssertNotEqual(before, after) + } + + // The whole point of this token: sync bookkeeping (cursors) must never look like a data change, + // or every pull/highwater bump would force a full re-upload regardless of whether anything a + // backup actually carries moved. + func testCursorWriteDoesNotChangeToken() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.insert(Streams(hr: [HRSample(ts: 100, bpm: 60)]), deviceId: "dev1") + let before = try await store.contentToken() + try await store.setCursor("cloud_edits", 42) + try await store.setCursor("highwater:hr", 999) + let after = try await store.contentToken() + XCTAssertEqual(before, after) + } +} diff --git a/Strand/CloudSync/CloudSyncModel.swift b/Strand/CloudSync/CloudSyncModel.swift index 64c8883a05..071f9b80f6 100644 --- a/Strand/CloudSync/CloudSyncModel.swift +++ b/Strand/CloudSync/CloudSyncModel.swift @@ -120,9 +120,10 @@ final class CloudSyncModel: ObservableObject { } } - /// Upload this device's own `.noopbak` to the noop-cloud server, then pull + apply any confirmed - /// edits — the two halves of "zero-touch" in one round trip: this device's own data reaches the - /// server, and any edits confirmed elsewhere reach this device. Stamps + /// Pull + apply any confirmed edits, then upload this device's own `.noopbak` to the noop-cloud + /// server (skipped when nothing has changed since the last upload — see `performSync`'s doc + /// comment) — the two halves of "zero-touch" in one round trip: any edits confirmed elsewhere reach + /// this device, and this device's own (possibly just-corrected) data reaches the server. Stamps /// `UserDefaults["cloudsync.lastAutoSync"]` on SUCCESS ONLY, so a failed attempt (server /// unreachable, network down) never blocks the next 20h-gated retry. Used by the user-initiated /// "Sync now" button; `autoSyncIfDue`'s on-launch catch-up runs the identical `performSync` body @@ -156,26 +157,37 @@ final class CloudSyncModel: ObservableObject { await CloudSyncGate.shared.end() } - /// The actual upload + pull work, gated by `runGatedSync` — never called directly. Byte-identical + /// The actual pull + upload work, gated by `runGatedSync` — never called directly. Byte-identical /// to `syncNow`'s pre-gate Task body, plus persisting the outcome for the Data Sources card to /// read back (Finding 2: `CloudSyncModel.persistLastStatus`, below). + /// + /// PULL-FIRST, UPLOAD-LAST (Cloud Sync v2 — reordered from the original upload-then-pull): pulling + /// and applying confirmed edits BEFORE this device uploads its own `.noopbak` means the upload + /// captures the just-corrected local state, so the server's mirror reflects THIS sync's edits + /// immediately — the original upload-first order sent the PRE-edit-application snapshot, so an + /// edit applied this sync only reached the server's mirror on the NEXT sync's upload, one full + /// cycle late. One consequence: a pull failure now aborts the whole sync before the upload ever + /// runs (previously the upload, which ran first, could still succeed even when the following pull + /// failed). That trade-off is intentional — an upload that ran anyway would still ship the stale + /// pre-edit state, exactly the problem this reorder fixes. + /// + /// SKIP-UNCHANGED UPLOAD: the upload only actually runs when `WhoopStore.contentToken()` (computed + /// AFTER pull/recompute, so a just-applied edit already counts) differs from the token saved after + /// the last successful upload FROM THIS DEVICE. A full re-export can be hundreds of MB; a device + /// that synced an hour ago with no new samples and no applied edits has nothing new to ship. private func performSync(repo: Repository, url: URL, token: String) async { - setBusy(true); statusText = "Uploading…" + setBusy(true); statusText = "Pulling edits…" var line: String var succeeded = false if let store = await repo.storeHandle() { let client = CloudSyncClient(baseURL: url, token: token) do { - let uploaded = try await CloudSyncUploader.upload(store: store, client: client) - let mb = Double(uploaded.bytes) / 1_048_576.0 - statusText = String(format: "Uploaded %.1f MB · pulling edits…", mb) - let summary = try await CloudSyncCoordinator.pull(store: store, client: client) - var parts = [String(format: "Uploaded %.1f MB", mb), - "Applied \(summary.applied)", "skipped \(summary.skipped)"] + var parts = ["Applied \(summary.applied)", "skipped \(summary.skipped)"] if summary.needsAttention > 0 { parts.append("\(summary.needsAttention) attention") } line = parts.joined(separator: " · ") - succeeded = true + statusText = line + // Rebuild the daily rollups before the Apple Health re-export, so the vitals it // exports (which read dailyMetric) reflect the just-applied edits too. var didRecompute = false @@ -197,6 +209,29 @@ final class CloudSyncModel: ObservableObject { line += await reExportToAppleHealthIfNeeded(repo: repo, store: store, summary: summary, force: didRecompute) #endif + + // Skip-unchanged upload (Cloud Sync v2): computed AFTER pull/recompute above, so an + // edit this device just applied already makes the token differ from the last upload's + // — the corrected state uploads THIS sync, matching the pull-first reorder's whole + // point (see this method's doc comment). `try?`: a token-computation failure must not + // abort an otherwise-successful sync — it just falls through to "upload anyway", the + // same safe default as a first-ever sync (no saved token to compare against). + let freshToken = try? await store.contentToken() + let lastUploadToken = UserDefaults.standard.string(forKey: Self.lastUploadTokenKey) + if let freshToken, freshToken == lastUploadToken { + parts.append("Up to date (nothing new to upload)") + line = parts.joined(separator: " · ") + } else { + statusText = "Uploading…" + let uploaded = try await CloudSyncUploader.upload(store: store, client: client) + let mb = Double(uploaded.bytes) / 1_048_576.0 + parts.append(String(format: "Uploaded %.1f MB", mb)) + line = parts.joined(separator: " · ") + if let freshToken { + UserDefaults.standard.set(freshToken, forKey: Self.lastUploadTokenKey) + } + } + succeeded = true } catch { line = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription } @@ -216,6 +251,16 @@ final class CloudSyncModel: ObservableObject { /// `autoSyncIfDue`, written by `performSync` on success only. private static let lastAutoSyncKey = "cloudsync.lastAutoSync" + /// UserDefaults key for the `WhoopStore.contentToken()` value as of the last successful upload + /// FROM THIS DEVICE (Cloud Sync v2 skip-unchanged upload) — written by `performSync` only when an + /// upload actually ran and succeeded, never on a skip (a skip means the saved token is STILL + /// accurate) or a failure (a failed upload never shipped that token's content, so the NEXT sync + /// must still compare against whatever WAS last actually uploaded). Deliberately a `UserDefaults` + /// key, like `lastAutoSyncKey`/`lastStatusKey` above, not a `WhoopStore` cursor: it is per-device + /// sync bookkeeping about an UPLOAD this device performed, not data the store itself owns, and + /// `cursors` only stores `Int` values anyway (this token is a composite string). + private static let lastUploadTokenKey = "cloudsync.lastUploadToken" + /// UserDefaults key for the outcome of the last sync attempt, success OR failure, persisted by /// `performSync` on EVERY completion (Finding 2). Read directly by the Data Sources card so an /// auto-sync that ran on a throwaway `CloudSyncModel` instance (see `CloudSyncGate`'s doc comment) From 785021e6b7cb84bd5c4b065df3f86a8594b60c3a Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 13:49:22 -0700 Subject: [PATCH 081/171] feat(cloudsync): background refresh via BGAppRefreshTask (iOS) Add a CLOUD_SYNC + iOS-gated BGAppRefreshTask so a backgrounded app gets a chance to pull confirmed edits and re-upload changed data without waiting for the next foreground launch. CloudSyncBackgroundRefresh registers the handler at launch, submits the next request when the app backgrounds, and runs the sync through the existing CloudSyncGate/runGatedSync so it can never overlap a foreground sync. CloudSyncModel.backgroundSyncIfDue(repo:) reuses runGatedSync under a new, shorter 4h staleness gate (vs. the 20h on-launch catch-up) since a background wake is opportunistic and, with skip-unchanged upload, usually cheap. Registers com.noopapp.noop.cloudsync.refresh in BGTaskSchedulerPermittedIdentifiers alongside the existing debug-export identifier. --- .../CloudSyncBackgroundRefresh.swift | 117 ++++++++++++++++++ Strand/CloudSync/CloudSyncModel.swift | 38 ++++++ StrandiOS/App/StrandiOSApp.swift | 14 +++ StrandiOS/Resources/Info.plist | 1 + project.yml | 6 +- 5 files changed, 175 insertions(+), 1 deletion(-) create mode 100644 Strand/CloudSync/CloudSyncBackgroundRefresh.swift diff --git a/Strand/CloudSync/CloudSyncBackgroundRefresh.swift b/Strand/CloudSync/CloudSyncBackgroundRefresh.swift new file mode 100644 index 0000000000..18da96be60 --- /dev/null +++ b/Strand/CloudSync/CloudSyncBackgroundRefresh.swift @@ -0,0 +1,117 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +#if CLOUD_SYNC +#if os(iOS) +import Foundation +import BackgroundTasks + +/// Registers and schedules the CLOUD_SYNC background-refresh `BGAppRefreshTask` (Cloud Sync v2) — the +/// iOS-only counterpart to `CloudSyncModel.autoSyncIfDue`'s on-launch catch-up, giving a backgrounded +/// app a chance to pick up confirmed edits (and re-upload changed data) without waiting for the next +/// foreground launch. Mirrors `ScheduledDebugExport`'s BGTask plumbing +/// (Strand/System/ScheduledDebugExport.swift) closely; the difference is this handler needs a real +/// `Repository`/`IntelligenceEngine` to run an actual sync (wired from the `AppModel` the app entry +/// point supplies), where that one only ever writes a local file. +/// +/// HONEST about platform limits, same posture as `ScheduledDebugExport`: iOS decides IF and WHEN a +/// submitted request actually runs — this is a best-effort extra chance to catch up sooner than the +/// guaranteed 20h on-launch catch-up, never a promise of timely background delivery. +@MainActor +enum CloudSyncBackgroundRefresh { + /// Must also appear in the iOS target's `BGTaskSchedulerPermittedIdentifiers` (project.yml) and be + /// registered at launch (`register(model:)`, called from `StrandiOSApp.init()`) for `schedule()` to + /// have any effect — see `ScheduledDebugExport.bgTaskIdentifier`'s doc comment for the identical + /// contract. + static let bgTaskIdentifier = "com.noopapp.noop.cloudsync.refresh" + + /// Register the BGTask handler. MUST be called from the app's launch (before launch finishes), + /// same constraint as `ScheduledDebugExport.register()`. Safe to leave uncalled, or to have + /// `schedule()`'s submit fail silently: the sync this handler runs is exactly what `autoSyncIfDue` + /// already covers on the next foreground launch anyway — this is a strictly additive, best-effort + /// earlier chance, never a dependency for correctness. + static func register(model: AppModel) { + BGTaskScheduler.shared.register(forTaskWithIdentifier: bgTaskIdentifier, using: nil) { task in + // The launch handler runs on a queue the system chooses, NOT necessarily the main actor — + // hop explicitly before touching `CloudSyncModel` (`@MainActor`) or `AppModel`. + Task { @MainActor in + handle(task, model: model) + } + } + } + + /// Submit a background-refresh request for *no earlier than* + /// `CloudSyncModel.backgroundSyncIntervalS` from now. Called when the app backgrounds + /// (`StrandiOSApp`'s `scenePhase` handler) — deliberately NOT at launch/register time, so a user + /// who keeps the app foregrounded for hours doesn't burn the window before ever backgrounding. + /// `try?` swallows the "identifier not permitted/registered" error so a build that hasn't wired + /// Info.plist yet still behaves (mirrors `ScheduledDebugExport.submitBackgroundRequest`). + static func schedule() { + let request = BGAppRefreshTaskRequest(identifier: bgTaskIdentifier) + request.earliestBeginDate = Date(timeIntervalSinceNow: CloudSyncModel.backgroundSyncIntervalS) + try? BGTaskScheduler.shared.submit(request) + } + + /// The delivered task's actual work: wire `CloudSyncModel` exactly like `RootTabView`'s on-launch + /// catch-up does (same `postApplyRefresh` closure, same `model.intelligence`/`model.repo`), run the + /// 4h-gated background sync, then unconditionally reschedule the next request and report + /// completion. Rescheduling happens FIRST, before the sync even starts, so the next window is + /// guaranteed to be requested whatever happens below — BGAppRefresh is single-shot, and nothing + /// else will ever submit the next one. + private static func handle(_ task: BGTask, model: AppModel) { + schedule() + + let cloudSync = CloudSyncModel() + let intelligence = model.intelligence + let repository = model.repo + cloudSync.postApplyRefresh = { + await intelligence.analyzeRecent() + await repository.refresh() + } + + // Exactly one `task.setTaskCompleted` call per delivered task, whichever finishes first: the + // sync itself, or iOS's expiration warning. `BGTaskCompletion` is a plain (non-actor-isolated) + // class precisely so the expiration handler below — invoked by the system on a thread of ITS + // choosing, not guaranteed to be the main actor — can call it directly with no `await`. + let completion = BGTaskCompletion(task) + let work = Task { + _ = await cloudSync.backgroundSyncIfDue(repo: repository) + completion.complete(success: true) + } + task.expirationHandler = { + // Cooperative cancellation: `backgroundSyncIfDue`'s network calls observe this and unwind + // via `URLError.cancelled`/`CancellationError`, which `performSync`'s `catch` already folds + // into a status line like any other failure — never crash-safe territory, just a sync that + // reports "cancelled" instead of a result. `Task.cancel()` is safe to call from any thread. + work.cancel() + completion.complete(success: false) + } + } +} + +/// Guarantees `BGTask.setTaskCompleted` is called exactly once for a given task, however many of the +/// (up to two) call sites race to finish first. Deliberately NOT `@MainActor` or an `actor`: iOS invokes +/// `BGTask.expirationHandler` on a thread of its own choosing (undocumented, not guaranteed to be the +/// main actor), and that closure is synchronous (`() -> Void`, no `await` available) — so the guard +/// itself has to be usable synchronously from an arbitrary thread. `NSLock` provides that directly; the +/// sync `Task`'s own completion (which runs on the main actor, inherited from `handle`) calls the SAME +/// synchronous, lock-guarded method with no isolation mismatch either way. +private final class BGTaskCompletion { + private let task: BGTask + private let lock = NSLock() + private var completed = false + + init(_ task: BGTask) { + self.task = task + } + + func complete(success: Bool) { + lock.lock() + defer { lock.unlock() } + guard !completed else { return } + completed = true + task.setTaskCompleted(success: success) + } +} +#endif // os(iOS) +#endif // CLOUD_SYNC diff --git a/Strand/CloudSync/CloudSyncModel.swift b/Strand/CloudSync/CloudSyncModel.swift index 071f9b80f6..e8d9d10e5a 100644 --- a/Strand/CloudSync/CloudSyncModel.swift +++ b/Strand/CloudSync/CloudSyncModel.swift @@ -339,6 +339,44 @@ final class CloudSyncModel: ObservableObject { Task { await runGatedSync(repo: repo, url: url, token: token) } } + #if os(iOS) + /// Staleness gate for the BACKGROUND refresh path (`backgroundSyncIfDue`, below) — SHORTER than + /// `isAutoSyncDue`'s 20h default because a background wake (BGAppRefreshTask or a silent push) is + /// opportunistic and comparatively cheap: iOS decides IF and WHEN it actually runs at all, and + /// skip-unchanged upload (see `performSync`'s doc comment) means most invocations that DO run ship + /// no bytes. There's no reason to make a user wait as long as the guaranteed on-launch catch-up does. + static let backgroundSyncIntervalS: TimeInterval = 4 * 3600 + + /// Background-refresh entry point — called from the `BGAppRefreshTask` handler + /// (`CloudSyncBackgroundRefresh`) and the silent-push handler (`CloudSyncAppDelegate`), both iOS + /// only. Unlike `autoSyncIfDue` (fire-and-forget: spawns its own `Task` and returns immediately so + /// it never blocks the launch-critical path), this AWAITS the sync to completion — both callers + /// need to know when it's done before they report back to iOS (`task.setTaskCompleted`, or the + /// push handler's `fetchCompletionHandler`). Returns `true` when a sync was actually ATTEMPTED + /// (whatever its outcome), `false` when a guard sent it home early (not configured, not due, bad + /// URL, or running under XCTest) — the push handler maps this straight to `.newData`/`.noData`. + /// + /// Reuses `runGatedSync`, the SAME `CloudSyncGate` every other entry point shares, so a background + /// wake that overlaps a foreground "Sync now" (or another background wake — a redelivered push, a + /// BGAppRefreshTask firing moments after a push already ran one) can't start a second sync; it just + /// sees the gate held and returns. Shares `Self.lastAutoSyncKey` with `autoSyncIfDue` too — a + /// successful background sync resets the SAME clock the 20h on-launch gate reads, so a launch right + /// after a successful background sync doesn't immediately re-fire (mirrors `syncNow`'s doc comment + /// on the identical interaction). + @discardableResult + func backgroundSyncIfDue(repo: Repository) async -> Bool { + guard !Self.isRunningUnderXCTest else { return false } + guard CloudSyncSettings.isConfigured else { return false } + let last = UserDefaults.standard.double(forKey: Self.lastAutoSyncKey) + guard Self.isAutoSyncDue(lastRun: last, now: Date().timeIntervalSince1970, + intervalS: Self.backgroundSyncIntervalS) else { return false } + guard let urlString = CloudSyncSettings.effectiveURL, let token = CloudSyncSettings.effectiveToken, + let url = URL(string: urlString) else { return false } + await runGatedSync(repo: repo, url: url, token: token) + return true + } + #endif + #if os(iOS) && canImport(HealthKit) /// Re-export into Apple Health when a pulled batch touched sleep/workouts/HR — shared by `pullNow` /// and `syncNow` so both surface the same "Apple Health re-export failed" note. Only re-exports diff --git a/StrandiOS/App/StrandiOSApp.swift b/StrandiOS/App/StrandiOSApp.swift index 074f9ac33a..fa803abb50 100644 --- a/StrandiOS/App/StrandiOSApp.swift +++ b/StrandiOS/App/StrandiOSApp.swift @@ -51,6 +51,14 @@ struct StrandiOSApp: App { // before the first scene so any early-fired notification is presented. UNUserNotificationCenter.current().delegate = NotificationPresenter.shared let model = AppModel() + #if CLOUD_SYNC + // Cloud Sync v2: register the background-refresh BGTask handler BEFORE launch finishes — same + // constraint as ScheduledDebugExport.register() above, and for the same reason (iOS only + // delivers a background task whose identifier was registered at launch AND listed in + // BGTaskSchedulerPermittedIdentifiers). `schedule()` itself is NOT called here — only when the + // app backgrounds (below) — so this alone doesn't consume the earliest-begin-date window. + CloudSyncBackgroundRefresh.register(model: model) + #endif _model = StateObject(wrappedValue: model) _health = StateObject(wrappedValue: HealthKitBridge( repo: model.repo, @@ -206,6 +214,12 @@ struct StrandiOSApp: App { // into Apple Health. Gated inside writeIfEnabled on the opt-in default (OFF) — a // no-op until the user turns on Shortcuts Export. Task { await ShortcutHealthExport.writeIfEnabled(repo: model.repo) } + #if CLOUD_SYNC + // Cloud Sync v2: submit the next background-refresh request on the way OUT, not at + // launch — see `CloudSyncBackgroundRefresh.schedule()`'s doc comment for why launch-time + // scheduling would waste the earliest-begin-date window on a long foreground session. + CloudSyncBackgroundRefresh.schedule() + #endif } } } diff --git a/StrandiOS/Resources/Info.plist b/StrandiOS/Resources/Info.plist index 707181767f..360c49a7f2 100644 --- a/StrandiOS/Resources/Info.plist +++ b/StrandiOS/Resources/Info.plist @@ -7,6 +7,7 @@ BGTaskSchedulerPermittedIdentifiers com.noopapp.noop.debugexport + com.noopapp.noop.cloudsync.refresh CFBundleDevelopmentRegion $(DEVELOPMENT_LANGUAGE) diff --git a/project.yml b/project.yml index 316f2e7653..af392592b8 100644 --- a/project.yml +++ b/project.yml @@ -251,13 +251,17 @@ targets: # backgrounded app near the user's chosen time to write the overnight strap-log drop. BGAppRefresh # needs the "fetch" background mode; the task id below must also be permitted (and registered at # launch in StrandiOSApp.init) or the submit fails gracefully and only the foreground/"Run now" - # paths run. + # paths run. Cloud Sync v2's own BGAppRefreshTaskRequest (CloudSyncBackgroundRefresh) shares this + # SAME "fetch" mode — one background mode covers every BGAppRefreshTask identifier in the app. - fetch # #510: permit the scheduled debug auto-export's BGAppRefreshTask identifier. MUST match # ScheduledDebugExport.bgTaskIdentifier EXACTLY and be registered at launch (StrandiOSApp.init); # otherwise iOS refuses to schedule or deliver the background drop. + # Cloud Sync v2 adds its own identifier alongside it — MUST match + # CloudSyncBackgroundRefresh.bgTaskIdentifier EXACTLY, same registration constraint. BGTaskSchedulerPermittedIdentifiers: - com.noopapp.noop.debugexport + - com.noopapp.noop.cloudsync.refresh NSBluetoothAlwaysUsageDescription: "NOOP connects directly to your WHOOP strap over Bluetooth to read heart rate, R-R intervals, battery, and sensor data locally on your iPhone. Nothing leaves your device." NSLocationWhenInUseUsageDescription: "NOOP records your route during an outdoor workout to show distance, pace, and a map, and keeps recording while the screen is off. The route stays on your device. Nothing leaves it." NSHealthShareUsageDescription: "NOOP reads your own Apple Health data on-device to compute recovery, strain, and sleep. Nothing leaves your device." From 6a4abe0a0f58e289592baae4e85c61056114fb65 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 13:50:15 -0700 Subject: [PATCH 082/171] feat(cloudsync): APNs silent-push client (code-complete, no credentials yet) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the client-side plumbing for a server-triggered silent push wake, ahead of the server endpoint and signing entitlement: - CloudSyncClient.registerDevice(token:) POSTs the hex-encoded APNs device token to /register-device. A 404 is treated as benign (the server endpoint ships in parallel with this client). - CloudSyncAppDelegate is the app's first UIApplicationDelegate (via UIApplicationDelegateAdaptor): requests registration on launch when CloudSync is configured, hands the token to registerDevice on success, and on a content-available push runs the same gated background sync as CloudSyncBackgroundRefresh, reporting .newData/.noData back to iOS. - project.yml adds the "remote-notification" UIBackgroundMode. Deliberately NOT adding the aps-environment entitlement to NOOP.entitlements in this branch — no push can actually be delivered until that's added, which needs the integrator's signing setup. --- Strand/CloudSync/CloudSyncAppDelegate.swift | 93 +++++++++++++++++++++ Strand/CloudSync/CloudSyncClient.swift | 31 +++++++ StrandiOS/App/StrandiOSApp.swift | 19 +++++ StrandiOS/Resources/Info.plist | 1 + project.yml | 6 ++ 5 files changed, 150 insertions(+) create mode 100644 Strand/CloudSync/CloudSyncAppDelegate.swift diff --git a/Strand/CloudSync/CloudSyncAppDelegate.swift b/Strand/CloudSync/CloudSyncAppDelegate.swift new file mode 100644 index 0000000000..a610ee69b4 --- /dev/null +++ b/Strand/CloudSync/CloudSyncAppDelegate.swift @@ -0,0 +1,93 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +#if CLOUD_SYNC +#if os(iOS) +import Foundation +import UIKit + +/// Minimal `UIApplicationDelegate` shim (Cloud Sync v2) — `StrandiOSApp` is otherwise a pure SwiftUI +/// `App` with no AppDelegate at all (confirmed: no `@UIApplicationDelegateAdaptor` existed before this). +/// APNs device-token registration and silent (content-available) push delivery are UIKit AppDelegate +/// callbacks with no SwiftUI equivalent — `.onReceive`/`.task` can't observe them — so this is the +/// ONLY reason an AppDelegate exists in this app. CLOUD_SYNC + iOS gated: a default build never links +/// any of this push-notification plumbing at all. +/// +/// CODE-COMPLETE, CREDENTIALS LATER: `NOOP.entitlements` does NOT carry `aps-environment` in this +/// branch (deliberately — the integrator adds it when APNs credentials/signing are ready), so +/// `registerForRemoteNotifications()` will fail every launch until that lands. +/// `didFailToRegisterForRemoteNotificationsWithError` below is expected to fire on every run of this +/// build for exactly that reason. +@MainActor +final class CloudSyncAppDelegate: NSObject, UIApplicationDelegate { + /// Set by `StrandiOSApp.init()` (alongside the `registerForRemoteNotifications()` call) before any + /// delegate callback below can plausibly fire, so `didReceiveRemoteNotification` can run the same + /// gated background sync `CloudSyncBackgroundRefresh` uses — this class has no other way to reach + /// the shared `AppModel`/`Repository`/`IntelligenceEngine` (a `UIApplicationDelegateAdaptor` + /// instance is constructed by SwiftUI, not by `StrandiOSApp.init()`, so it can't be handed the + /// model at init time the way `CloudSyncBackgroundRefresh.register(model:)` is). + static var model: AppModel? + + /// APNs handed us a device token: hex-encode it (APNs' own wire format) and hand it to the + /// server's `/register-device` endpoint. Best-effort and silent either way — see + /// `CloudSyncClient.registerDevice`'s doc comment for why the server 404 case specifically is + /// EXPECTED (it ships in parallel with this client) rather than a real failure. + func application(_ application: UIApplication, + didRegisterForRemoteNotificationsWithDeviceToken deviceToken: Data) { + guard let urlString = CloudSyncSettings.effectiveURL, let token = CloudSyncSettings.effectiveToken, + let url = URL(string: urlString) else { return } + let hexToken = deviceToken.map { String(format: "%02x", $0) }.joined() + let client = CloudSyncClient(baseURL: url, token: token) + Task { + do { + try await client.registerDevice(token: hexToken) + } catch CloudSyncError.badResponse(404, _) { + // Expected during rollout: the server's /register-device endpoint ships in parallel + // with this client. Benign — a fresh install, a token refresh (APNs rotates tokens + // occasionally), or simply the next relaunch after the server catches up will retry + // this same call, so there's nothing to recover here. + } catch { + // Any other failure (network down, non-404 error): log-not-crash, matching every other + // CloudSync network call's posture — push registration is a nice-to-have, never + // something worth surfacing to the user or aborting launch over. + } + } + } + + /// Expected on EVERY launch of this branch (see the type's doc comment: no `aps-environment` + /// entitlement yet). Silent — there is no user-facing push-status UI to update, and the + /// BGAppRefreshTask + on-launch catch-up paths cover the same ground without APNs at all. + func application(_ application: UIApplication, didFailToRegisterForRemoteNotificationsWithError error: Error) { + } + + /// A silent (content-available) push arrived: run the SAME gated background sync the + /// `BGAppRefreshTask` handler uses (`CloudSyncBackgroundRefresh.handle`), then report the fetch + /// result iOS expects. A push is a stronger signal than the 4h background-refresh gate ("a + /// specific edit is now confirmed and waiting"), but it still goes through + /// `backgroundSyncIfDue`'s SAME `CloudSyncGate`/staleness gate rather than an unconditional sync — + /// a burst of pushes (or a redelivered one) must not be able to fire overlapping syncs any more + /// than two backgrounded refreshes could. `.newData`/`.noData` is a coarse signal (this handler has + /// no per-byte insight into what changed): `.newData` whenever a sync was actually ATTEMPTED, + /// `.noData` when a guard sent it home early (e.g. the 4h gate hasn't elapsed, or CloudSync isn't + /// configured on this device). + func application(_ application: UIApplication, didReceiveRemoteNotification userInfo: [AnyHashable: Any], + fetchCompletionHandler completionHandler: @escaping (UIBackgroundFetchResult) -> Void) { + guard let model = Self.model else { + completionHandler(.noData) + return + } + let cloudSync = CloudSyncModel() + let intelligence = model.intelligence + let repository = model.repo + cloudSync.postApplyRefresh = { + await intelligence.analyzeRecent() + await repository.refresh() + } + Task { + let ran = await cloudSync.backgroundSyncIfDue(repo: repository) + completionHandler(ran ? .newData : .noData) + } + } +} +#endif // os(iOS) +#endif // CLOUD_SYNC diff --git a/Strand/CloudSync/CloudSyncClient.swift b/Strand/CloudSync/CloudSyncClient.swift index 03aa7abc03..b3d28aca3e 100644 --- a/Strand/CloudSync/CloudSyncClient.swift +++ b/Strand/CloudSync/CloudSyncClient.swift @@ -90,6 +90,32 @@ final class CloudSyncClient { return (decoded.bytes, decoded.latestDay) } + // MARK: - Device registration (Cloud Sync v2: APNs silent-push wake) + + /// POST /register-device, Bearer-authed, `{"token":"","platform":"ios"}` — hands the server + /// this device's APNs device token so it can wake this device with a silent (content-available) + /// push the moment a NEW edit is confirmed elsewhere, instead of only ever reaching it on the next + /// background-refresh/launch poll. `token` is the hex-encoded APNs device token (the caller, + /// `CloudSyncAppDelegate`, does the hex-encoding — this method just ships whatever string it's + /// given). + /// + /// The server endpoint ships in parallel with this client (code-complete ahead of it), so a 404 + /// here is EXPECTED for a while — see `CloudSyncAppDelegate`'s call site for why that's treated as + /// benign rather than a real failure. Same uniform error contract as every other call on this + /// client (`CloudSyncError.badResponse` for any non-2xx status): no status-code special-casing + /// happens in here, that judgment call belongs to the caller. + func registerDevice(token: String) async throws { + var req = URLRequest(url: url(path: "register-device")) + req.httpMethod = "POST" + req.setValue("Bearer \(self.token)", forHTTPHeaderField: "Authorization") + req.setValue("application/json", forHTTPHeaderField: "Content-Type") + req.httpBody = try JSONEncoder().encode(RegisterDeviceRequest(token: token, platform: "ios")) + let (data, status) = try await send(req) + guard (200..<300).contains(status) else { + throw CloudSyncError.badResponse(status, bodyPrefix(data)) + } + } + private func sendUpload(_ req: URLRequest, fromFile fileURL: URL) async throws -> (Data, Int) { do { let (data, resp) = try await session.upload(for: req, fromFile: fileURL) @@ -132,6 +158,11 @@ private struct AckResponse: Decodable { let acked: Int } +private struct RegisterDeviceRequest: Encodable { + let token: String + let platform: String +} + private struct IngestResponse: Decodable { let bytes: Int let latestDay: String? diff --git a/StrandiOS/App/StrandiOSApp.swift b/StrandiOS/App/StrandiOSApp.swift index fa803abb50..9c3b19014c 100644 --- a/StrandiOS/App/StrandiOSApp.swift +++ b/StrandiOS/App/StrandiOSApp.swift @@ -14,6 +14,12 @@ import UserNotifications /// `RootTabView` so the iOS app keeps the same gating without depending on the macOS-only shell. @main struct StrandiOSApp: App { + #if CLOUD_SYNC + /// The ONLY reason this app has an AppDelegate at all — APNs device-token registration + silent + /// (content-available) push delivery are UIKit callbacks with no SwiftUI equivalent. See + /// `CloudSyncAppDelegate`'s doc comment. CLOUD_SYNC-gated so a default build links none of this. + @UIApplicationDelegateAdaptor(CloudSyncAppDelegate.self) private var cloudSyncAppDelegate + #endif @StateObject private var model: AppModel @StateObject private var health: HealthKitBridge /// The phone→watch link. Built + activated here so the watch app actually receives snapshots on a @@ -58,6 +64,19 @@ struct StrandiOSApp: App { // BGTaskSchedulerPermittedIdentifiers). `schedule()` itself is NOT called here — only when the // app backgrounds (below) — so this alone doesn't consume the earliest-begin-date window. CloudSyncBackgroundRefresh.register(model: model) + // Hand the AppDelegate shim the model it needs for a push-triggered background sync (it has no + // other way to reach the shared Repository/IntelligenceEngine — see + // `CloudSyncAppDelegate.model`'s doc comment). Set BEFORE requesting registration below, so the + // model is never nil by the time a delegate callback could possibly fire. + CloudSyncAppDelegate.model = model + // APNs registration itself: request only when the lane is actually configured — an unconfigured + // device has no server to hand a token to. `NOOP.entitlements` doesn't carry `aps-environment` + // in this branch yet, so this will call `didFailToRegisterForRemoteNotificationsWithError` on + // every launch until the integrator adds it (see `CloudSyncAppDelegate`'s doc comment) — that's + // expected, not a bug. + if CloudSyncSettings.isConfigured { + UIApplication.shared.registerForRemoteNotifications() + } #endif _model = StateObject(wrappedValue: model) _health = StateObject(wrappedValue: HealthKitBridge( diff --git a/StrandiOS/Resources/Info.plist b/StrandiOS/Resources/Info.plist index 360c49a7f2..8b1479da7f 100644 --- a/StrandiOS/Resources/Info.plist +++ b/StrandiOS/Resources/Info.plist @@ -83,6 +83,7 @@ bluetooth-central location fetch + remote-notification UIFileSharingEnabled diff --git a/project.yml b/project.yml index af392592b8..7174a7ab74 100644 --- a/project.yml +++ b/project.yml @@ -254,6 +254,12 @@ targets: # paths run. Cloud Sync v2's own BGAppRefreshTaskRequest (CloudSyncBackgroundRefresh) shares this # SAME "fetch" mode — one background mode covers every BGAppRefreshTask identifier in the app. - fetch + # Cloud Sync v2: required for iOS to wake the backgrounded app on a silent (content-available) + # APNs push (CloudSyncAppDelegate.didReceiveRemoteNotification). Code-complete ahead of the + # server/entitlement: `NOOP.entitlements` doesn't carry `aps-environment` in this branch yet + # (see CloudSyncAppDelegate's doc comment), so no push can actually arrive until that lands — + # this mode alone is inert without it, but harmless to declare early. + - remote-notification # #510: permit the scheduled debug auto-export's BGAppRefreshTask identifier. MUST match # ScheduledDebugExport.bgTaskIdentifier EXACTLY and be registered at launch (StrandiOSApp.init); # otherwise iOS refuses to schedule or deliver the background drop. From e1c59d2aae87f471cbe69b96d082f16eb8344d03 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 15:49:29 -0700 Subject: [PATCH 083/171] fix(cloudsync): gate APNs registration on effective credentials, not Keychain-only isConfigured A zero-touch bundle-credentialed device (the normal personal setup) has isConfigured == false, so registerForRemoteNotifications never ran and the device never appeared in the server's push registry. Gate on effectiveURL/effectiveToken, matching every other Phase-3.5 call site. --- StrandiOS/App/StrandiOSApp.swift | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/StrandiOS/App/StrandiOSApp.swift b/StrandiOS/App/StrandiOSApp.swift index 9c3b19014c..6b8894a093 100644 --- a/StrandiOS/App/StrandiOSApp.swift +++ b/StrandiOS/App/StrandiOSApp.swift @@ -70,11 +70,11 @@ struct StrandiOSApp: App { // model is never nil by the time a delegate callback could possibly fire. CloudSyncAppDelegate.model = model // APNs registration itself: request only when the lane is actually configured — an unconfigured - // device has no server to hand a token to. `NOOP.entitlements` doesn't carry `aps-environment` - // in this branch yet, so this will call `didFailToRegisterForRemoteNotificationsWithError` on - // every launch until the integrator adds it (see `CloudSyncAppDelegate`'s doc comment) — that's - // expected, not a bug. - if CloudSyncSettings.isConfigured { + // device has no server to hand a token to. Gate on the EFFECTIVE credentials (Keychain OR the + // Phase-3.5 bundle injection), NOT `isConfigured`: `isConfigured` is Keychain-only, so a + // zero-touch bundle-credentialed device (the normal personal setup) would never register — + // found live: devices stayed 0 after the entitlement landed because this gate never opened. + if CloudSyncSettings.effectiveURL != nil, CloudSyncSettings.effectiveToken != nil { UIApplication.shared.registerForRemoteNotifications() } #endif From 3e24164ac1df4a6c9ed29d8ce98bf84bf2cc4570 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 15:58:24 -0700 Subject: [PATCH 084/171] fix(cloudsync): push registration at didFinishLaunching; ungated push sync; effective-creds gates Three live findings from the first APNs bring-up: - registerForRemoteNotifications from StrandiOSApp.init is silently ignored (pre-launch); moved to CloudSyncAppDelegate's didFinishLaunching with a diagnosis breadcrumb. - backgroundSyncIfDue gated on Keychain-only isConfigured, so a bundle-credentialed device never background-synced. - a silent push routed through the 4h staleness gate, making request_sync a no-op within 4h of any sync; pushes now run an unconditional CloudSyncGate-guarded syncFromPush. --- Strand/CloudSync/CloudSyncAppDelegate.swift | 29 ++++++++++++++++++++- Strand/CloudSync/CloudSyncModel.swift | 17 +++++++++++- StrandiOS/App/StrandiOSApp.swift | 13 ++++----- 3 files changed, 49 insertions(+), 10 deletions(-) diff --git a/Strand/CloudSync/CloudSyncAppDelegate.swift b/Strand/CloudSync/CloudSyncAppDelegate.swift index a610ee69b4..75ac65f1b6 100644 --- a/Strand/CloudSync/CloudSyncAppDelegate.swift +++ b/Strand/CloudSync/CloudSyncAppDelegate.swift @@ -37,27 +37,51 @@ final class CloudSyncAppDelegate: NSObject, UIApplicationDelegate { guard let urlString = CloudSyncSettings.effectiveURL, let token = CloudSyncSettings.effectiveToken, let url = URL(string: urlString) else { return } let hexToken = deviceToken.map { String(format: "%02x", $0) }.joined() + // Breadcrumb (readable via the app container's preferences plist): which leg of + // token→POST last ran, so a silent registration failure is diagnosable without a debugger. + UserDefaults.standard.set("token-received \(Date())", forKey: Self.registrationBreadcrumbKey) let client = CloudSyncClient(baseURL: url, token: token) Task { do { try await client.registerDevice(token: hexToken) + UserDefaults.standard.set("registered \(Date())", forKey: Self.registrationBreadcrumbKey) } catch CloudSyncError.badResponse(404, _) { // Expected during rollout: the server's /register-device endpoint ships in parallel // with this client. Benign — a fresh install, a token refresh (APNs rotates tokens // occasionally), or simply the next relaunch after the server catches up will retry // this same call, so there's nothing to recover here. + UserDefaults.standard.set("post-404 \(Date())", forKey: Self.registrationBreadcrumbKey) } catch { // Any other failure (network down, non-404 error): log-not-crash, matching every other // CloudSync network call's posture — push registration is a nice-to-have, never // something worth surfacing to the user or aborting launch over. + UserDefaults.standard.set("post-error: \(error) \(Date())", forKey: Self.registrationBreadcrumbKey) } } } + /// UserDefaults key for the registration breadcrumb above — diagnosis-only, never read by code. + static let registrationBreadcrumbKey = "cloudsync.push.lastRegistration" + + /// Request APNs registration HERE — the canonical didFinishLaunching site — not from + /// `StrandiOSApp.init()`: a request made during the SwiftUI `App` struct's init runs BEFORE the + /// app finishes launching, and UIKit silently ignores it (found live: no token callback and no + /// failure callback ever fired; the breadcrumb below never advanced past absent). + func application(_ application: UIApplication, + didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]? = nil) -> Bool { + if CloudSyncSettings.effectiveURL != nil, CloudSyncSettings.effectiveToken != nil { + UserDefaults.standard.set("requested \(Date())", forKey: Self.registrationBreadcrumbKey) + application.registerForRemoteNotifications() + } + return true + } + /// Expected on EVERY launch of this branch (see the type's doc comment: no `aps-environment` /// entitlement yet). Silent — there is no user-facing push-status UI to update, and the /// BGAppRefreshTask + on-launch catch-up paths cover the same ground without APNs at all. func application(_ application: UIApplication, didFailToRegisterForRemoteNotificationsWithError error: Error) { + UserDefaults.standard.set("os-fail: \(error.localizedDescription) \(Date())", + forKey: Self.registrationBreadcrumbKey) } /// A silent (content-available) push arrived: run the SAME gated background sync the @@ -84,7 +108,10 @@ final class CloudSyncAppDelegate: NSObject, UIApplicationDelegate { await repository.refresh() } Task { - let ran = await cloudSync.backgroundSyncIfDue(repo: repository) + // Unconditional (CloudSyncGate-guarded) — see `syncFromPush`'s doc comment: the 4h + // staleness gate belongs to the OS-initiated BGAppRefresh path, not to an explicit + // server-side request for fresh data. + let ran = await cloudSync.syncFromPush(repo: repository) completionHandler(ran ? .newData : .noData) } } diff --git a/Strand/CloudSync/CloudSyncModel.swift b/Strand/CloudSync/CloudSyncModel.swift index e8d9d10e5a..3a96dbafb6 100644 --- a/Strand/CloudSync/CloudSyncModel.swift +++ b/Strand/CloudSync/CloudSyncModel.swift @@ -366,7 +366,8 @@ final class CloudSyncModel: ObservableObject { @discardableResult func backgroundSyncIfDue(repo: Repository) async -> Bool { guard !Self.isRunningUnderXCTest else { return false } - guard CloudSyncSettings.isConfigured else { return false } + // Effective (Keychain OR bundle) credentials are the configuration check — `isConfigured` + // is Keychain-only and would keep a zero-touch bundle-credentialed device gated forever. let last = UserDefaults.standard.double(forKey: Self.lastAutoSyncKey) guard Self.isAutoSyncDue(lastRun: last, now: Date().timeIntervalSince1970, intervalS: Self.backgroundSyncIntervalS) else { return false } @@ -375,6 +376,20 @@ final class CloudSyncModel: ObservableObject { await runGatedSync(repo: repo, url: url, token: token) return true } + + /// Push-triggered sync (`request_sync` → APNs silent push → here): UNCONDITIONAL, no staleness + /// gate — the push already carries intent ("an agent wants fresh data NOW"; the server throttles + /// request_sync to one push per 120s), so routing it through `backgroundSyncIfDue`'s 4h gate + /// would silently ignore exactly the syncs the feature exists to perform. Overlap safety is + /// `runGatedSync`'s CloudSyncGate, same as every other entry point. + @discardableResult + func syncFromPush(repo: Repository) async -> Bool { + guard !Self.isRunningUnderXCTest else { return false } + guard let urlString = CloudSyncSettings.effectiveURL, let token = CloudSyncSettings.effectiveToken, + let url = URL(string: urlString) else { return false } + await runGatedSync(repo: repo, url: url, token: token) + return true + } #endif #if os(iOS) && canImport(HealthKit) diff --git a/StrandiOS/App/StrandiOSApp.swift b/StrandiOS/App/StrandiOSApp.swift index 6b8894a093..1e2ef1af3d 100644 --- a/StrandiOS/App/StrandiOSApp.swift +++ b/StrandiOS/App/StrandiOSApp.swift @@ -69,14 +69,11 @@ struct StrandiOSApp: App { // `CloudSyncAppDelegate.model`'s doc comment). Set BEFORE requesting registration below, so the // model is never nil by the time a delegate callback could possibly fire. CloudSyncAppDelegate.model = model - // APNs registration itself: request only when the lane is actually configured — an unconfigured - // device has no server to hand a token to. Gate on the EFFECTIVE credentials (Keychain OR the - // Phase-3.5 bundle injection), NOT `isConfigured`: `isConfigured` is Keychain-only, so a - // zero-touch bundle-credentialed device (the normal personal setup) would never register — - // found live: devices stayed 0 after the entitlement landed because this gate never opened. - if CloudSyncSettings.effectiveURL != nil, CloudSyncSettings.effectiveToken != nil { - UIApplication.shared.registerForRemoteNotifications() - } + // APNs registration itself lives in `CloudSyncAppDelegate.didFinishLaunching` — NOT here: + // a registerForRemoteNotifications() call made during this init runs before the app finishes + // launching, and UIKit silently ignores it (no token callback, no failure callback — found + // live). The delegate requests it at the canonical launch point, gated on the same EFFECTIVE + // (Keychain-or-bundle) credentials. #endif _model = StateObject(wrappedValue: model) _health = StateObject(wrappedValue: HealthKitBridge( From 0279c487cfbd0d74303207e8f3baaa74aaa01d9d Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 09:30:44 -0700 Subject: [PATCH 085/171] =?UTF-8?q?feat(scoring):=20wire=20real=20sleep=20?= =?UTF-8?q?regularity=20+=20population-anchored=20need=20into=20Rest=20(Wa?= =?UTF-8?q?ve=200=20=C2=B7=20SL1/T1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rest was scored on a flat neutral-0.5 consistency and a fixed 8h need: despite the "caller supplies it from history" contract, IntelligenceEngine never threaded either into AnalyticsEngine.analyzeDay (it only threaded habitualMidsleepSec). - T1: new AnalyticsEngine.Rest.personalizedNeedHours(nightlyHours:age:) — upper-quartile of the user's unrestricted nights, floored at the age-appropriate population TARGET (8h adult / 9h <18) so a chronic under-sleeper's need can only adjust UP for genuine long sleepers, never collapse toward their own deficit. Cold-start returns the population default. - SL1: thread real VitalityEngine.sleepConsistency + that personal need into the per-day analyzeDay call, computed once from the trailing per-night durations (recent 28 nights for regularity, full history for the need quantile). Extends computeHabitualMidsleep's existing read to also yield nightly durations — one read, midsleep result byte-identical. - The nightly durations are TST (span x efficiency), NOT in-bed span: the need/regularity estimate must match the tstSeconds Rest scores against. Real-data validation on VK's noop-cloud history showed in-bed span over-counts need ~0.85h (his correct need ~8.0h TST). Both degrade honestly on thin history (nil consistency -> neutral; need -> default). Validation: RestWiringImpactTests archetype table + real-data replay on ~90d of VK's nights (need 8.0h TST, consistency 0.57); RestNeedTests; full StrandAnalytics suite green; macOS build green. --- .../StrandAnalytics/AnalyticsEngine.swift | 40 ++++++++++++++ .../StrandAnalyticsTests/RestNeedTests.swift | 53 ++++++++++++++++++ .../RestWiringImpactTests.swift | 54 +++++++++++++++++++ Strand/Data/IntelligenceEngine.swift | 41 ++++++++++++-- 4 files changed, 184 insertions(+), 4 deletions(-) create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RestNeedTests.swift create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RestWiringImpactTests.swift diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift index c918efd697..ac257395e1 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift @@ -813,6 +813,46 @@ public enum AnalyticsEngine { public static let wRestorative: Double = 0.20 public static let wConsistency: Double = 0.10 + /// Minimum trailing nights before a personal sleep-need estimate is trusted; below this the + /// population default is used (cold-start honesty — never learn a need from a few nights). + public static let minNeedNights: Int = 7 + /// Hard cap on personalized need (h): beyond typical adult need even for genuine long sleepers. + public static let maxNeedHours: Double = 9.5 + + /// Population TARGET nightly sleep need (hours) for an age (NSF/AASM recommended-range + /// midpoint). Used as a FLOOR: the personal estimate can only adjust it UP for genuine long + /// sleepers, NEVER below the population target — so a chronic under-sleeper's need can't drift + /// toward their own deficit (which would falsely erase their sleep debt and read a short night + /// as "Strong"). Deliberately the target, not the minimum: flooring at the low end of the + /// range would understate a sleep-deprived (vs genuinely short-sleeping) user's deficit. + public static func populationNeedFloorHours(age: Int?) -> Double { + guard let age, age > 0 else { return 8.0 } // unknown → adult target + switch age { + case ..<18: return 9.0 // children/teens need more (NSF 8–12; target ~9) + default: return 8.0 // adults (18–64) & older (65+): NSF 7–9, target ~8 (the app default) + } + } + + /// Personalized nightly sleep need (hours), population-ANCHORED and age-floored (T1). The + /// personal component is the user's UPPER-QUARTILE nightly duration over the trailing window — + /// what they sleep on their less-restricted nights, NOT their average (which a chronic deficit + /// drags down) — floored at the age-appropriate population TARGET and capped at `maxNeedHours`. + /// So it only ever ADJUSTS UP for genuine long sleepers, never below the target. Fewer + /// than `minNeedNights` scorable nights → the population default (cold-start). Zero/negative + /// entries (no-data days) are dropped and do not count toward the minimum. + public static func personalizedNeedHours(nightlyHours: [Double], age: Int?) -> Double { + let floor = populationNeedFloorHours(age: age) + let xs = nightlyHours.filter { $0 > 0 }.sorted() + guard xs.count >= minNeedNights else { + return min(max(defaultNeedHours, floor), maxNeedHours) + } + // Linear-interpolated 75th percentile — the "unrestricted" nights. + let pos = 0.75 * Double(xs.count - 1) + let lo = Int(pos), hi = min(lo + 1, xs.count - 1) + let q = xs[lo] + (pos - Double(lo)) * (xs[hi] - xs[lo]) + return min(max(q, floor), maxNeedHours) + } + /// Build the composite. `tstSeconds` = total sleep time, `restorativeSeconds` = deep+REM /// seconds, `deepSeconds` = deep-stage seconds (nil → no deep-adequacy adjustment, pooled /// behaviour). Returns a value in [0,100]. diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RestNeedTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RestNeedTests.swift new file mode 100644 index 0000000000..f5a54c8ad7 --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RestNeedTests.swift @@ -0,0 +1,53 @@ +import XCTest +@testable import StrandAnalytics + +/// `AnalyticsEngine.Rest.personalizedNeedHours` (Wave 0 · T1) — population-anchored, age-floored +/// sleep need. The load-bearing property is the anti-self-referential guard: a chronic +/// under-sleeper's need must NOT drift down toward their own deficit (which a plain trailing mean +/// would do, inverting the honesty philosophy for exactly the users with the most sleep debt); it +/// stays anchored at the age-appropriate population floor. +final class RestNeedTests: XCTestCase { + private typealias Rest = AnalyticsEngine.Rest + + func testChronicUnderSleeperStaysAtFloor() { + // 14 nights of ~5.5 h. A self-referential mean returns ~5.5; the anchor must hold >= 7.0. + let nights = Array(repeating: 5.5, count: 14) + XCTAssertGreaterThanOrEqual(Rest.personalizedNeedHours(nightlyHours: nights, age: 30), 7.0) + } + + func testNormalSleeperReflectsUnrestrictedNights() { + let nights = [7.2, 7.8, 8.1, 7.5, 8.4, 7.9, 8.0, 7.6, 8.2, 7.7] + let need = Rest.personalizedNeedHours(nightlyHours: nights, age: 35) + XCTAssertGreaterThanOrEqual(need, 7.0) + XCTAssertLessThanOrEqual(need, 9.5) + // Upper-quartile, not the mean: reflects what they sleep on their longer nights. + XCTAssertGreaterThanOrEqual(need, 7.8) + } + + func testColdStartReturnsPopulationDefault() { + // Fewer than minNeedNights → the population default (8.0), floored. + XCTAssertEqual(Rest.personalizedNeedHours(nightlyHours: [7.0, 8.0, 6.5], age: 30), + Rest.defaultNeedHours, accuracy: 0.001) + } + + func testCapsLongSleeper() { + let nights = Array(repeating: 11.0, count: 10) + XCTAssertLessThanOrEqual(Rest.personalizedNeedHours(nightlyHours: nights, age: 40), 9.5) + } + + func testMinorHasHigherFloor() { + let nights = Array(repeating: 6.0, count: 10) // an under-sleeping teen + XCTAssertGreaterThanOrEqual(Rest.personalizedNeedHours(nightlyHours: nights, age: 15), 8.0) + } + + func testNilAgeUsesAdultFloor() { + let nights = Array(repeating: 5.0, count: 10) + XCTAssertGreaterThanOrEqual(Rest.personalizedNeedHours(nightlyHours: nights, age: nil), 7.0) + } + + func testZeroAndNegativeNightsIgnored() { + // no-data days (0) must not drag the estimate down or count toward the minimum. + let nights = [0.0, -1.0, 7.5, 8.0, 7.8, 8.2, 7.6, 8.1, 7.9, 8.3, 0.0] + XCTAssertGreaterThanOrEqual(Rest.personalizedNeedHours(nightlyHours: nights, age: 30), 7.5) + } +} diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RestWiringImpactTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RestWiringImpactTests.swift new file mode 100644 index 0000000000..dcb0d2a0e4 --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RestWiringImpactTests.swift @@ -0,0 +1,54 @@ +import XCTest +@testable import StrandAnalytics + +/// Behavioral validation of SL1/T1: score the SAME night for different sleeper HISTORIES and show +/// how Rest moves once real regularity + personalized need replace neutral-0.5 / fixed-8h. Prints an +/// old→new table for eyeballing magnitude/direction, and pins the load-bearing directions. +final class RestWiringImpactTests: XCTestCase { + private typealias Rest = AnalyticsEngine.Rest + + // One fixed scored night so only (need, consistency) vary: 7h TST, 90% eff, 1h deep, 1.5h REM. + private let tst = 7.0 * 3600, inbed = 7.0 * 3600 / 0.9, eff = 0.90 + private let deep = 1.0 * 3600, rem = 1.5 * 3600 + + private func restOld() -> Double { + Rest.composite(tstSeconds: tst, inBedSeconds: inbed, efficiency: eff, + restorativeSeconds: deep + rem, needHours: Rest.defaultNeedHours, + consistency: nil, deepSeconds: deep) // nil → neutral 0.5 + } + private func restNew(history: [Double], age: Int?) -> (rest: Double, need: Double, cons: Double) { + let need = Rest.personalizedNeedHours(nightlyHours: history, age: age) + let cons = VitalityEngine.sleepConsistency(nightlyHours: Array(history.suffix(28))) + let rest = Rest.composite(tstSeconds: tst, inBedSeconds: inbed, efficiency: eff, + restorativeSeconds: deep + rem, needHours: need, + consistency: cons, deepSeconds: deep) + return (rest, need, cons ?? -1) + } + + func testArchetypeImpactTable() { + let regular = Array(repeating: 8.0, count: 14) + let irregular = (0..<14).map { $0 % 2 == 0 ? 5.0 : 10.0 } // mean 7.5, high variance + let chronicShort = Array(repeating: 5.5, count: 14) + let longSleeper = Array(repeating: 9.2, count: 14) + + let old = restOld() + let reg = restNew(history: regular, age: 30) + let irr = restNew(history: irregular, age: 30) + let shortS = restNew(history: chronicShort, age: 30) + let longS = restNew(history: longSleeper, age: 30) + + print(String(format: "OLD (neutral 0.5, need 8h): Rest %5.1f", old)) + print(String(format: "regular good sleeper: need %.1f cons %.2f Rest %5.1f", reg.need, reg.cons, reg.rest)) + print(String(format: "irregular sleeper: need %.1f cons %.2f Rest %5.1f", irr.need, irr.cons, irr.rest)) + print(String(format: "chronic short (5.5h): need %.1f cons %.2f Rest %5.1f", shortS.need, shortS.cons, shortS.rest)) + print(String(format: "long sleeper (9.2h): need %.1f cons %.2f Rest %5.1f", longS.need, longS.cons, longS.rest)) + + // Load-bearing directions (SL1): a REGULAR history lifts Rest above an IRREGULAR one for the + // identical night — regularity now actually matters. + XCTAssertGreaterThan(reg.rest, irr.rest) + // T1: chronic-short need is floored (never collapses toward the 5.5h deficit). + XCTAssertGreaterThanOrEqual(shortS.need, 7.0) + // T1: a genuine long sleeper's need exceeds the old fixed 8h (more demanding duration term). + XCTAssertGreaterThan(longS.need, 8.0) + } +} diff --git a/Strand/Data/IntelligenceEngine.swift b/Strand/Data/IntelligenceEngine.swift index e9c750074b..77b532cf43 100644 --- a/Strand/Data/IntelligenceEngine.swift +++ b/Strand/Data/IntelligenceEngine.swift @@ -460,10 +460,20 @@ final class IntelligenceEngine: ObservableObject { // Returns nil under `habitualMinDays` of history → cold-start: every `analyzeDay`/`sleepEditedDaily` // call below stays on the overnight-band bonus. The same value threads into both seams so analytics // and the Sleep tab resolve to the identical block. (#547) - let habitualMidsleepSec = await Self.computeHabitualMidsleep( + let (habitualMidsleepSec, nightlyHours) = await Self.computeHabitualSleep( store: store, importedId: deviceId, computedId: deviceId + "-noop", windowStart: nowLocalMidnight - maxDays * 86_400 - 30 * 3_600, windowEnd: now, offsetSec: tzOffset) + // Wave 0 (SL1/T1): personal sleep REGULARITY + population-anchored NEED, computed ONCE from the + // trailing per-night durations and threaded to every analyzeDay below (mirrors the midsleep + // learner just above — one personal trait per run, applied to the whole re-scored history so + // Rest stops running on a flat neutral-0.5 consistency and a fixed 8 h need). Recent 28-night + // window for regularity (a recent-behaviour signal); full history for the need's upper-quartile + // "unrestricted nights" estimate. Both degrade honestly on thin history (consistency → nil → + // neutral term; need → population default), so cold-start is unchanged. + let sleepConsistency = VitalityEngine.sleepConsistency(nightlyHours: Array(nightlyHours.suffix(28))) + let sleepNeedHours = AnalyticsEngine.Rest.personalizedNeedHours(nightlyHours: nightlyHours, + age: profile.age) // ── FIX 1 (main-actor jank): run the ENTIRE per-day enumeration OFF the main actor ─────────── // Every `await store.…` read inside this loop has its continuation RESUME on the main actor @@ -651,6 +661,8 @@ final class IntelligenceEngine: ObservableObject { spo2: spo2, // #93 profile: up, baselines: baselines1, maxHROverride: maxHR, tzOffsetSeconds: tzOffset, wristOff: wristOff, + sleepNeedHours: sleepNeedHours, + sleepConsistency: sleepConsistency, habitualMidsleepSec: habitualMidsleepSec, bandSleepState: bandSleepState, // #690: thread the V2 toggle into the NORMAL staging path so @@ -1706,10 +1718,15 @@ final class IntelligenceEngine: ObservableObject { return samples } - private static func computeHabitualMidsleep( + /// Habitual midsleep (local seconds) AND the trailing per-night sleep DURATIONS (hours, + /// chronological) from the stored sessions over the window — the longest block per LOCAL day, so + /// naps drop out. One read serves both the main-night midsleep learner (#547) and the personal + /// sleep-need + regularity that thread into `analyzeDay` (Wave 0 · SL1/T1). The midsleep result is + /// byte-identical to before; the nightly-hours output is the Swift-side extension. + private static func computeHabitualSleep( store: WhoopStore, importedId: String, computedId: String, windowStart: Int, windowEnd: Int, offsetSec: Int - ) async -> Int? { + ) async -> (midsleepSec: Int?, nightlyHours: [Double]) { let imported = (try? await store.sleepSessions(deviceId: importedId, from: windowStart, to: windowEnd, limit: 4000)) ?? [] let computed = (try? await store.sleepSessions(deviceId: computedId, from: windowStart, @@ -1721,14 +1738,30 @@ final class IntelligenceEngine: ObservableObject { // covers an imported night and its computed twin (the longest capture wins, exactly what the // per-day length rule chose anyway). let merged = SleepSessionDedup.dedupe(imported + computed).kept + // Longest block per LOCAL day (naps drop out), chosen by in-bed SPAN — reused for BOTH the + // midsleep learner and the per-night durations (Wave 0 · SL1/T1), so the two can never read a + // different history. For the DURATIONS we keep TST (span × efficiency), NOT the in-bed span: + // the need/regularity estimate must be in the same asleep-time units as the `tstSeconds` Rest + // scores against, or need reads systematically high (validated on real data — an in-bed span + // over-counts ~0.85 h vs TST). Efficiency is 0..1 (post the v26 unit-heal); a rare nil main + // night falls back to a typical 0.9. + var longestByDay: [String: (span: Int, tstHours: Double)] = [:] let blocks = merged.compactMap { s -> SleepStageTotals.HistoryBlock? in let start = s.effectiveStartTs, end = s.endTs guard end > start else { return nil } let mid = start + (end - start) / 2 let dayKey = AnalyticsEngine.dayString(mid, offsetSec: offsetSec) + let span = end - start + if span > (longestByDay[dayKey]?.span ?? 0) { + let eff = s.efficiency.flatMap { (0.0 < $0 && $0 <= 1.0) ? $0 : nil } ?? 0.9 + longestByDay[dayKey] = (span, Double(span) / 3600.0 * eff) + } return SleepStageTotals.HistoryBlock(start: start, end: end, dayKey: dayKey) } - return SleepStageTotals.habitualMidsleepSec(blocks, offsetSec: offsetSec) + let midsleep = SleepStageTotals.habitualMidsleepSec(blocks, offsetSec: offsetSec) + // Chronological (day-key string sort == date order) so a recent-window suffix is well-defined. + let nightlyHours = longestByDay.keys.sorted().compactMap { longestByDay[$0]?.tstHours } + return (midsleep, nightlyHours) } /// Floor a unix-seconds timestamp to 00:00:00 of its UTC calendar day. Mirrors the Android From cc9d01e727780e9f0f52ea41d10d0d2d70a344ad Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 09:30:44 -0700 Subject: [PATCH 086/171] =?UTF-8?q?fix(scoring):=20z-score=20readiness=20H?= =?UTF-8?q?RV=20in=20the=20log=20domain=20(Wave=200=20=C2=B7=20RD1,=20narr?= =?UTF-8?q?ow)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ReadinessEngine scored its HRV signal as a raw-ms z-score, but RMSSD is right-skewed, so a symmetric z over-weights the long upper tail and misstates tail rarity. lnRMSSD is closer to normal (Plews/Altini; the app's own HRVReadiness already works in the log domain). Add a logDomain path to the shared zSignal builder, used ONLY by HRV — RHR/resp are ~normal and stay linear. Evidence stays in ms, but the reported baseline is the geometric mean, not an arithmetic mean a few big-recovery nights inflate. Deliberately NARROW: domain-correctness only. The full HRVReadiness SWC-tier engine stays behind its experimental flag until validated on real data (its header notes n=1; roadmap anti-#194). Tests: RD1LogDomainTests pins the geometric-mean baseline; ReadinessEngineTests unchanged; suite green. --- .../StrandAnalytics/ReadinessEngine.swift | 20 +++++++++--- .../RD1LogDomainTests.swift | 31 +++++++++++++++++++ 2 files changed, 46 insertions(+), 5 deletions(-) create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RD1LogDomainTests.swift diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/ReadinessEngine.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/ReadinessEngine.swift index 958082ba0d..a6373d3567 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/ReadinessEngine.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/ReadinessEngine.swift @@ -139,6 +139,7 @@ public enum ReadinessEngine { unit: "ms", decimals: 0, higherIsBetter: true, + logDomain: true, // RD1: lnRMSSD — HRV is right-skewed goodText: "above your baseline - well recovered", neutralText: "in your normal range", watchText: "a touch below baseline", @@ -217,13 +218,21 @@ public enum ReadinessEngine { /// Build a z-score signal for a metric where the baseline is the trailing window. private static func zSignal(value: Double?, baseline: [Double], key: String, label: String, unit: String, decimals: Int, - higherIsBetter: Bool, + higherIsBetter: Bool, logDomain: Bool = false, goodText: String, neutralText: String, watchText: String, badText: String) -> Signal? { - guard let v = value, baseline.count >= minBaseline, - let m = mean(baseline), let sd = sampleSD(baseline), sd > 0 else { return nil } + guard let v = value, baseline.count >= minBaseline else { return nil } + // RD1: right-skewed metrics (HRV/RMSSD) are z-scored in the LOG domain — lnRMSSD is closer to + // normal, so a symmetric z is statistically valid, whereas a raw-ms z over-weights the long + // upper tail and misstates tail rarity (Plews/Altini; the app's own HRVReadiness works this + // way). RHR/resp are ~normal and stay linear. Evidence stays in the metric's own units, but the + // baseline shown is then the GEOMETRIC mean (exp of the log-mean) — a typical night, not an + // outlier-inflated arithmetic mean. + let tv = logDomain ? log(max(v, 1.0)) : v + let tb = logDomain ? baseline.map { log(max($0, 1.0)) } : baseline + guard let m = mean(tb), let sd = sampleSD(tb), sd > 0 else { return nil } // Orient z so positive always means "better". - let z = (higherIsBetter ? (v - m) : (m - v)) / sd + let z = (higherIsBetter ? (tv - m) : (m - tv)) / sd let flag: Flag let text: String switch z { @@ -233,7 +242,8 @@ public enum ReadinessEngine { default: flag = .bad; text = badText } return Signal(key: key, label: label, - evidence: evidence(value: v, baseline: m, unit: unit, decimals: decimals), + evidence: evidence(value: v, baseline: logDomain ? exp(m) : m, + unit: unit, decimals: decimals), detail: text, flag: flag) } diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RD1LogDomainTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RD1LogDomainTests.swift new file mode 100644 index 0000000000..43bdaab198 --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RD1LogDomainTests.swift @@ -0,0 +1,31 @@ +import XCTest +@testable import StrandAnalytics +import WhoopStore + +/// RD1 — the readiness HRV signal is z-scored in the LOG domain (lnRMSSD), not raw ms. RMSSD is +/// right-skewed, so a symmetric z on raw ms over-weights the long upper tail; lnRMSSD is closer to +/// normal (Plews/Altini; the app's own HRVReadiness works this way). Observable consequence: the +/// baseline the signal reports is the GEOMETRIC mean (a typical night), not an arithmetic mean that +/// a few big-recovery nights inflate. RHR stays linear (it is ~normal). +final class RD1LogDomainTests: XCTestCase { + private func d(_ i: Int, hrv: Double?, rhr: Int?) -> DailyMetric { + DailyMetric(day: String(format: "2024-03-%02d", i), totalSleepMin: nil, efficiency: nil, + deepMin: nil, remMin: nil, lightMin: nil, disturbances: nil, restingHr: rhr, + avgHrv: hrv, recovery: nil, strain: 10, exerciseCount: nil, + spo2Pct: nil, skinTempDevC: nil, respRateBpm: nil) + } + + func testHrvBaselineIsGeometricNotArithmetic() { + // 20 nights at 42 ms + 8 big-recovery nights at 85 ms: arithmetic mean ≈ 54, geometric ≈ 51. + var days: [DailyMetric] = [] + for i in 1...20 { days.append(d(i, hrv: 42, rhr: i % 2 == 0 ? 53 : 51)) } + for i in 21...28 { days.append(d(i, hrv: 85, rhr: i % 2 == 0 ? 53 : 51)) } + days.append(d(29, hrv: 50, rhr: 52)) // today: a typical night + + let hrv = ReadinessEngine.evaluate(days: days).signals.first { $0.key == "hrv" } + // Log domain baselines against the geometric mean (51). Raw-ms z would report "50 vs 54 ms". + XCTAssertEqual(hrv?.evidence, "50 vs 51 ms") + // 50 sits at the typical night → neutral, read against a representative (not tail-inflated) baseline. + XCTAssertEqual(hrv?.flag, .neutral) + } +} From 921260972c9e4928ebf01e2b01c54b9bfb323e62 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 10:36:12 -0700 Subject: [PATCH 087/171] =?UTF-8?q?feat(scoring):=20readiness=20read=20car?= =?UTF-8?q?ries=20a=20baseline-density=20confidence=20(Wave=200=20=C2=B7?= =?UTF-8?q?=20RD-confidence)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ReadinessEngine scored off as few as 7 baseline nights with no confidence shown, while Charge already carries a ScoreConfidence tier. Add ScoreConfidence.readiness(hasRead:baselineNights: fullWindow:) + a `confidence` field on Readiness, from the HRV baseline density (the primary readiness driver): calibrating (no read), building (7-29 of the 30-night window), solid (full window). Analytics enabler; surfacing it on the Today card is a separate design pass. Tests: RDConfidenceTests (tiers + engine building/solid/calibrating); ReadinessEngineTests unchanged; full suite green; macOS Strand build green. --- .../StrandAnalytics/ReadinessEngine.swift | 17 +++++++- .../StrandAnalytics/ScoreConfidence.swift | 9 ++++ .../RDConfidenceTests.swift | 43 +++++++++++++++++++ 3 files changed, 67 insertions(+), 2 deletions(-) create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RDConfidenceTests.swift diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/ReadinessEngine.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/ReadinessEngine.swift index a6373d3567..cf26854953 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/ReadinessEngine.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/ReadinessEngine.swift @@ -57,10 +57,15 @@ public enum ReadinessEngine { public let acwr: Double? /// Foster training monotony over the last week (nil if not enough strain history). public let monotony: Double? + /// How much history backs this read (HRV/RHR baseline density) — so the card can show + /// calibrating / building / solid instead of a confident number off a 7-night baseline. + public let confidence: ScoreConfidence public init(level: Level, headline: String, summary: String, - signals: [Signal], acwr: Double?, monotony: Double?) { + signals: [Signal], acwr: Double?, monotony: Double?, + confidence: ScoreConfidence = .calibrating) { self.level = level; self.headline = headline; self.summary = summary self.signals = signals; self.acwr = acwr; self.monotony = monotony + self.confidence = confidence } } @@ -209,8 +214,16 @@ public enum ReadinessEngine { let (level, headline, summary) = synthesize(signals: signals, hasHistory: !history.isEmpty || acwr != nil) + // RD-confidence: surface how much history backs the read (HRV baseline density, the primary + // readiness driver). A read off a 7-night baseline must not look as certain as one off the full + // 30-night window. Insufficient reads carry .calibrating. + let hrvBaselineNights = history.suffix(baselineWindow).compactMap { $0.avgHrv }.count + let confidence = ScoreConfidence.readiness(hasRead: level != .insufficient, + baselineNights: hrvBaselineNights, + fullWindow: baselineWindow) return Readiness(level: level, headline: headline, summary: summary, - signals: signals, acwr: acwr, monotony: monotony) + signals: signals, acwr: acwr, monotony: monotony, + confidence: confidence) } // MARK: Signal builders diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/ScoreConfidence.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/ScoreConfidence.swift index bed4ed4a76..e3a95437a3 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/ScoreConfidence.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/ScoreConfidence.swift @@ -34,6 +34,15 @@ public enum ScoreConfidence: String, Equatable, Sendable, Codable { return b.trusted ? .solid : .building } + /// Readiness confidence from the HRV/RHR baseline density backing the read (readiness is HRV-led). + /// - calibrating: no read (insufficient history — the readiness level is `.insufficient`). + /// - solid: a read exists AND the full baseline window is present. + /// - building: a read exists but the baseline is shorter than the full window (e.g. 7–29 of 30). + public static func readiness(hasRead: Bool, baselineNights: Int, fullWindow: Int) -> ScoreConfidence { + guard hasRead else { return .calibrating } + return baselineNights >= fullWindow ? .solid : .building + } + /// Effort (strain) confidence. /// - calibrating: no score (no usable HR window) → absent. /// - solid: a score exists AND the HR window is dense (≥ solidReadings samples). diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RDConfidenceTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RDConfidenceTests.swift new file mode 100644 index 0000000000..1fd5f7d27f --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RDConfidenceTests.swift @@ -0,0 +1,43 @@ +import XCTest +@testable import StrandAnalytics +import WhoopStore + +/// RD-confidence — the readiness read now carries a `ScoreConfidence` from its baseline density, so +/// the card can say calibrating / building / solid instead of a confident number off a 7-night +/// baseline (Charge already does this; readiness didn't). +final class RDConfidenceTests: XCTestCase { + + func testReadinessConfidenceTiers() { + XCTAssertEqual(ScoreConfidence.readiness(hasRead: false, baselineNights: 40, fullWindow: 30), .calibrating) + XCTAssertEqual(ScoreConfidence.readiness(hasRead: true, baselineNights: 12, fullWindow: 30), .building) + XCTAssertEqual(ScoreConfidence.readiness(hasRead: true, baselineNights: 30, fullWindow: 30), .solid) + } + + private func d(_ i: Int, hrv: Double?, rhr: Int?) -> DailyMetric { + DailyMetric(day: String(format: "2024-03-%02d", i), totalSleepMin: nil, efficiency: nil, + deepMin: nil, remMin: nil, lightMin: nil, disturbances: nil, restingHr: rhr, + avgHrv: hrv, recovery: nil, strain: 10, exerciseCount: nil, + spo2Pct: nil, skinTempDevC: nil, respRateBpm: nil) + } + + func testEmptyHistoryReadsCalibrating() { + XCTAssertEqual(ReadinessEngine.evaluate(days: []).confidence, .calibrating) + } + + func testThinBaselineReadsBuilding() { + // 10 baseline nights + today: a read exists but baseline < the 30-night window → building. + var days: [DailyMetric] = [] + for i in 1...10 { days.append(d(i, hrv: i % 2 == 0 ? 62 : 58, rhr: 52)) } + days.append(d(11, hrv: 60, rhr: 52)) + let r = ReadinessEngine.evaluate(days: days) + XCTAssertNotEqual(r.level, .insufficient) + XCTAssertEqual(r.confidence, .building) + } + + func testFullBaselineReadsSolid() { + var days: [DailyMetric] = [] + for i in 1...30 { days.append(d(i, hrv: i % 2 == 0 ? 62 : 58, rhr: 52)) } + days.append(d(31, hrv: 60, rhr: 52)) + XCTAssertEqual(ReadinessEngine.evaluate(days: days).confidence, .solid) + } +} From daf8268d3caf50c32ddae125fc502a57ffa1688c Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 10:57:26 -0700 Subject: [PATCH 088/171] feat(tz): record phone timezone per local day + ship it on every upload Every timestamp in the DB is epoch-UTC and nothing recorded which zone the phone was in when a day was lived, so any wall-clock reading of a historical night is guesswork once the user crosses zones (Pacific <-> Eastern). A hard-coded-offset assumption already corrupted an analysis pass. - v28-phone-timezone migration: new additive table phoneTimezone(day PK, tzId NOT NULL), one row per local calendar day. Numbered past upstream's v26 and local v27. - PhoneTimezoneStore: idempotent upsert keyed by day + a per-day read, mirroring AppleStepHourStore. - IntelligenceEngine.analyzeRecent (the daily hook) stamps today's local day with TimeZone.current.identifier. Fire-and-forget; never blocks scoring. - CloudSyncClient.ingest adds the X-Phone-Timezone header so the server knows the current zone even before the per-day table lands in a given mirror. Tests: PhoneTimezoneStoreTests (table shape, idempotent upsert, distinct days, missing-day nil). Full WhoopStore suite green (278). NOOPiOS simulator build succeeds with CLOUD_SYNC active. Android twin deferred (Swift-only contributor); a Room migration adding phoneTimezone is needed before an Android release. --- .../Sources/WhoopStore/Database.swift | 18 ++++++++ .../WhoopStore/PhoneTimezoneStore.swift | 37 +++++++++++++++ .../PhoneTimezoneStoreTests.swift | 45 +++++++++++++++++++ Strand/CloudSync/CloudSyncClient.swift | 6 +++ Strand/Data/IntelligenceEngine.swift | 11 +++++ 5 files changed, 117 insertions(+) create mode 100644 Packages/WhoopStore/Sources/WhoopStore/PhoneTimezoneStore.swift create mode 100644 Packages/WhoopStore/Tests/WhoopStoreTests/PhoneTimezoneStoreTests.swift diff --git a/Packages/WhoopStore/Sources/WhoopStore/Database.swift b/Packages/WhoopStore/Sources/WhoopStore/Database.swift index 34d9ed4602..b10e977861 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/Database.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/Database.swift @@ -569,6 +569,24 @@ extension WhoopStore { WHERE efficiency > 1.5 """) } + + // v28: the phone's IANA timezone, recorded one row per LOCAL day. Every timestamp in the DB is + // epoch-UTC and nothing records which zone the phone was in when a day was lived, so any + // wall-clock interpretation of a historical night (bedtime, "3am", day boundaries) is guesswork + // once the traveller crosses zones (e.g. Pacific ↔ Eastern). This table stamps each local day + // with `TimeZone.current.identifier` so downstream analysis can localise that specific day's + // samples instead of assuming one fixed offset. `day` is the local calendar day ("YYYY-MM-DD") + // and PK, so a same-day re-record overwrites in place (idempotent upsert). Additive only — a NEW + // table, no existing row touched, old readers unaffected. The value also rides every upload as + // the `X-Phone-Timezone` header on POST /ingest so the server knows the CURRENT zone even before + // this table lands in a given mirror. Android Room twin deferred (Swift-only contributor); a Room + // migration adding `phoneTimezone` is needed before an Android release. + migrator.registerMigration("v28-phone-timezone") { db in + try db.create(table: "phoneTimezone") { t in + t.column("day", .text).primaryKey() // local calendar day "YYYY-MM-DD" + t.column("tzId", .text).notNull() // IANA identifier, e.g. "America/Los_Angeles" + } + } return migrator } } diff --git a/Packages/WhoopStore/Sources/WhoopStore/PhoneTimezoneStore.swift b/Packages/WhoopStore/Sources/WhoopStore/PhoneTimezoneStore.swift new file mode 100644 index 0000000000..d314319d7a --- /dev/null +++ b/Packages/WhoopStore/Sources/WhoopStore/PhoneTimezoneStore.swift @@ -0,0 +1,37 @@ +import Foundation +import GRDB + +// MARK: - v28 store: the phone's IANA timezone, one row per local day +// Every timestamp in the DB is epoch-UTC and nothing else records which zone the phone was in when a +// day was lived, so a wall-clock reading of a historical night is guesswork once the user crosses time +// zones. This table stamps each local calendar day with `TimeZone.current.identifier` so downstream +// analysis can localise that specific day. Idempotent upsert keyed by `day`: recording the same day +// again overwrites the identifier in place rather than duplicating. + +extension WhoopStore { + + /// Upsert the phone's IANA timezone identifier for a local calendar day. Idempotent by `day`: + /// re-recording the same day overwrites its identifier rather than inserting a duplicate. Returns + /// rows changed. + @discardableResult + public func upsertPhoneTimezone(day: String, tzId: String) async throws -> Int { + try syncWrite { db in + try db.execute(sql: """ + INSERT INTO phoneTimezone (day, tzId) + VALUES (?, ?) + ON CONFLICT(day) DO UPDATE SET + tzId = excluded.tzId + """, arguments: [day, tzId]) + return db.changesCount + } + } + + /// The phone's recorded IANA timezone identifier for a local calendar day, or nil if none stored. + public func phoneTimezone(day: String) async throws -> String? { + try syncRead { db in + try String.fetchOne(db, sql: """ + SELECT tzId FROM phoneTimezone WHERE day = ? + """, arguments: [day]) + } + } +} diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/PhoneTimezoneStoreTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/PhoneTimezoneStoreTests.swift new file mode 100644 index 0000000000..3222094889 --- /dev/null +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/PhoneTimezoneStoreTests.swift @@ -0,0 +1,45 @@ +import XCTest +import GRDB +@testable import WhoopStore + +final class PhoneTimezoneStoreTests: XCTestCase { + + /// v28 creates the phoneTimezone table with `day` as the sole primary key. + func testV28CreatesPhoneTimezoneTable() async throws { + let store = try await WhoopStore.inMemory() + let tables = try await store.tableNames() + XCTAssertTrue(tables.contains("phoneTimezone"), "missing v28 phoneTimezone table") + let cols = try await store.primaryKeyColumns("phoneTimezone") + XCTAssertEqual(cols, ["day"]) + } + + func testUpsertIsIdempotentByDay() async throws { + let store = try await WhoopStore.inMemory() + let n1 = try await store.upsertPhoneTimezone(day: "2026-07-14", tzId: "America/Los_Angeles") + XCTAssertEqual(n1, 1) + + // Same day, phone now on Eastern → overwrite in place, still one row. + let n2 = try await store.upsertPhoneTimezone(day: "2026-07-14", tzId: "America/New_York") + XCTAssertEqual(n2, 1) + + let tz = try await store.phoneTimezone(day: "2026-07-14") + XCTAssertEqual(tz, "America/New_York") + } + + func testDistinctDaysCoexist() async throws { + let store = try await WhoopStore.inMemory() + _ = try await store.upsertPhoneTimezone(day: "2026-07-13", tzId: "America/Los_Angeles") + _ = try await store.upsertPhoneTimezone(day: "2026-07-14", tzId: "America/New_York") + + let d13 = try await store.phoneTimezone(day: "2026-07-13") + let d14 = try await store.phoneTimezone(day: "2026-07-14") + XCTAssertEqual(d13, "America/Los_Angeles") + XCTAssertEqual(d14, "America/New_York") + } + + func testMissingDayReturnsNil() async throws { + let store = try await WhoopStore.inMemory() + let tz = try await store.phoneTimezone(day: "1999-01-01") + XCTAssertNil(tz) + } +} diff --git a/Strand/CloudSync/CloudSyncClient.swift b/Strand/CloudSync/CloudSyncClient.swift index b3d28aca3e..6225054618 100644 --- a/Strand/CloudSync/CloudSyncClient.swift +++ b/Strand/CloudSync/CloudSyncClient.swift @@ -80,6 +80,12 @@ final class CloudSyncClient { req.httpMethod = "POST" req.setValue("Bearer \(token)", forHTTPHeaderField: "Authorization") req.setValue("application/octet-stream", forHTTPHeaderField: "Content-Type") + // #tz-upload: the upload body is the whole SQLite DB (epoch-UTC timestamps only), so the server + // has no way to know which zone this phone is in right now. Ship the CURRENT IANA identifier as a + // header the server stores on the ingestLog row (and surfaces via data_freshness) — this answers + // "what zone is the phone in as of this upload" even before the per-day `phoneTimezone` table + // lands in a given mirror. + req.setValue(TimeZone.current.identifier, forHTTPHeaderField: "X-Phone-Timezone") let (data, status) = try await sendUpload(req, fromFile: fileURL) guard (200..<300).contains(status) else { throw CloudSyncError.badResponse(status, bodyPrefix(data)) diff --git a/Strand/Data/IntelligenceEngine.swift b/Strand/Data/IntelligenceEngine.swift index e9c750074b..72d05901d3 100644 --- a/Strand/Data/IntelligenceEngine.swift +++ b/Strand/Data/IntelligenceEngine.swift @@ -397,6 +397,17 @@ final class IntelligenceEngine: ObservableObject { // boundary inside the window is a negligible edge case for an hour-of-day band.) let tzOffset = TimeZone.current.secondsFromGMT() + // #tz-upload: stamp TODAY's local calendar day with the phone's IANA timezone so historical + // nights stay localisable once the user crosses zones (Pacific ↔ Eastern). Every timestamp in + // the DB is epoch-UTC and `tzOffset` above is only an integer offset — it can't distinguish + // "America/Los_Angeles" from any other −07:00 zone or survive a DST transition. analyzeRecent is + // the daily hook (idle tick + every post-sync rescore), and the upsert is keyed by day so + // repeated same-day passes overwrite one row rather than churning. Fire-and-forget: a failure + // here must never block scoring. The same identifier also rides each upload as the + // `X-Phone-Timezone` header (CloudSyncClient.ingest). + let todayLocal = AnalyticsEngine.dayString(Self.midnightLocal(now, offsetSec: tzOffset), offsetSec: tzOffset) + try? await store.upsertPhoneTimezone(day: todayLocal, tzId: TimeZone.current.identifier) + // ── Pass 1: analyse each offloaded night against the IMPORTED-ONLY baseline. For a BLE-only // user the imported daily rows are empty, so the HRV baseline isn't usable yet and recovery is // null here , but each night's avgHrv/restingHr are computed baseline-INDEPENDENTLY, so we From e8b4b7f468de861f7eaf71c3b84abe22a4946d68 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 11:22:32 -0700 Subject: [PATCH 089/171] fix(sleep): headline "Asleep" reflects the whole bridged night, not just the main session MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A night stored as a first-sleep fragment + a brief walk + the main session (e.g. the real 2026-07-14 night: 12:16 → 1:22 fragment, ~6-min gap, 1:29 → 7:32 main with the main's onset user-edited later via startTsAdjusted) showed "Asleep 1:29 AM" — the main session's start — hiding the true 12:16 bedtime. The Apple Health write-back already spans 12:16 → 7:32 (bridged night groups, #364), so the headline disagreed with it. Root cause: the fragment DOES bridge into the main-night group (6-min gap ≪ 60-min gapBridgeMaxMin), but the display onset walk (SleepView.nightOnsetTs → isPreOnsetAwakeStub) mis-classified the fragment as a spurious pre-onset lead. The #259 relative "minor lead" test compared the fragment's ~34 asleep min against 15% of the ~340-min main (≈51 min) and skipped it, so the shown onset jumped to the 1:29 main. On a long main sleep the 15% bar inflates past a genuine short first sleep. Fix: add an absolute floor (preOnsetStubMinorAsleepFloorMin = 20 min) under the #259 relative test — a leading fragment carrying a real sleep episode (≥ 20 asleep min) is never a spurious lead, whatever the main block's size. Strict relaxation: it can only un-skip a real first sleep, never newly skip one. The displayed onset now equals mainNightSpan.start (the canonical bed/wake window, #294), so the Sleep hero, Coupled, Today's HR band, and the Health write-back all agree on the 12:16 bedtime. The #736 sleepless-stub skip and the #259 tiny-stray-lead (≤ 10 min) behavior are unchanged. Tests: StrandAnalytics BridgedNightGroupsTests pins the real night bridging into one group (fragment not filtered). StrandTests SleepOnsetStubTests pins the fixed onset walk plus floor boundary + #736/#259 guards. swift test (1021 green) + macOS StrandTests (SleepOnsetStubTests 18 green) + NOOPiOS simulator build succeeded. Android twin needs the same group-onset headline logic (deferred; Swift-only). --- .../BridgedNightGroupsTests.swift | 21 ++++++++++++ Strand/Screens/SleepView.swift | 20 +++++++++-- StrandTests/SleepOnsetStubTests.swift | 33 +++++++++++++++++++ 3 files changed, 71 insertions(+), 3 deletions(-) diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BridgedNightGroupsTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BridgedNightGroupsTests.swift index 2396335fd4..6957798cf6 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BridgedNightGroupsTests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BridgedNightGroupsTests.swift @@ -100,4 +100,25 @@ final class BridgedNightGroupsTests: XCTestCase { let nap = B(start: t0 + 14 * 3_600, end: t0 + 15 * 3_600) XCTAssertEqual(SleepStageTotals.mainNightGroupIndices([a, b, nap], offsetSec: 0), [0, 1]) } + + /// REAL NIGHT (2026-07-14, PDT): a 12:16 first-sleep fragment (67 min) then a ~6-min walk then the + /// main 1:29 → 7:32 sleep, stored as two rows on `my-whoop-noop` with the main's onset user-edited + /// later via `startTsAdjusted` (so its EFFECTIVE start is 1:29). The 6-min effective gap is far under + /// `gapBridgeMaxMin`, so the two fragments MUST bridge into ONE group — proving the fragment is NOT + /// filtered out of grouping. The bug lives downstream (the display onset walk), not here: the group + /// spans the whole night, exactly as the Health write-back folds it (#364). offsetSec = -25200 (PDT). + func testRealNightFirstSleepFragmentBridgesIntoMainGroup() { + let off = -25_200 + let fragment = B(start: 1_784_013_364, end: 1_784_017_375) // 12:16 → 1:22 (67 min) + let mainEffective = B(start: 1_784_017_740, end: 1_784_039_558) // 1:29 (edited onset) → 7:32 + let groups = SleepStageTotals.bridgedNightGroups([fragment, mainEffective], offsetSec: off) + XCTAssertEqual(groups.map(\.indices), [[0, 1]], "the 6-min walk must bridge, not split the night") + XCTAssertEqual(groups[0].gaps, [Gap(start: fragment.end, end: mainEffective.start)]) + // The whole bridged group is the main night, and its span opens at the 12:16 fragment. + XCTAssertEqual(SleepStageTotals.mainNightGroupIndices([fragment, mainEffective], offsetSec: off), + [0, 1]) + let span = groups[0].indices + XCTAssertEqual([fragment, mainEffective][span[0]].start, 1_784_013_364, + "earliest onset of the bridged night is the 12:16 first-sleep fragment") + } } diff --git a/Strand/Screens/SleepView.swift b/Strand/Screens/SleepView.swift index 117b45368d..ea0d4af736 100644 --- a/Strand/Screens/SleepView.swift +++ b/Strand/Screens/SleepView.swift @@ -1954,15 +1954,29 @@ struct SleepView: View { /// a lead carrying a few minutes more than 3. Mirrors Android PRE_ONSET_STUB_MINOR_FRAC. (#259) static let preOnsetStubMinorFrac: Double = 0.15 + /// Absolute floor (ASLEEP minutes) under the #259 relative "minor lead" test: a leading fragment that + /// carries at least this much real sleep is a genuine first sleep — a real sleep episode — and is NEVER + /// a spurious pre-onset lead, however large the main block is. Without it a long main sleep inflates the + /// 15% relative bar (a 6h night → ~54 min) so a genuine ~34-min first sleep was swallowed and the shown + /// bedtime jumped hours late, hiding the real onset the bridged night (and the Health write-back, #364) + /// already spans. 20 min ≈ the shortest standalone sleep episode; below it a handful of asleep minutes + /// beside a long night is a stray lead. Mirrors Android PRE_ONSET_STUB_MINOR_ASLEEP_FLOOR_MIN. + /// (bridged-night headline: a real 2026-07-14 12:16 first sleep hidden behind the 1:29 main block) + static let preOnsetStubMinorAsleepFloorMin: Double = 20 + /// Pure stub test on a fragment's span + asleep minutes, so the rule is unit-testable without decoding /// JSON or building a view. Spurious when BRIEF and EITHER essentially sleepless OR minor relative to the /// main block (`refAsleepMin`, the group's largest asleep span): asleep below `preOnsetStubMinorFrac` of - /// it. `refAsleepMin` defaults to 0 (relative test off) so existing callers/tests are byte-identical. - /// (#736 / #259) + /// it AND below the absolute `preOnsetStubMinorAsleepFloorMin` real-sleep-episode floor. `refAsleepMin` + /// defaults to 0 (relative test off) so existing callers/tests are byte-identical. (#736 / #259) static func isPreOnsetAwakeStub(spanMin: Double, asleepMin: Double, refAsleepMin: Double = 0) -> Bool { guard spanMin <= preOnsetStubMaxMin else { return false } if asleepMin <= preOnsetStubAsleepMaxMin { return true } - return refAsleepMin > 0 && asleepMin < preOnsetStubMinorFrac * refAsleepMin + // #259 relative "minor lead" test, floored: a real sleep episode (>= the floor) is never a stray + // lead, so a long main block can't inflate the 15% bar past a genuine short first sleep. + return refAsleepMin > 0 + && asleepMin < preOnsetStubMinorFrac * refAsleepMin + && asleepMin < preOnsetStubMinorAsleepFloorMin } /// The index into an ascending-by-onset group whose fragment supplies the DISPLAYED bedtime: the first diff --git a/StrandTests/SleepOnsetStubTests.swift b/StrandTests/SleepOnsetStubTests.swift index 4530f8b99f..b445515088 100644 --- a/StrandTests/SleepOnsetStubTests.swift +++ b/StrandTests/SleepOnsetStubTests.swift @@ -102,4 +102,37 @@ final class SleepOnsetStubTests: XCTestCase { func testAllStubGroupFallsBackToZero() { XCTAssertEqual(SleepView.nightOnsetIndex(spansMin: [5, 10], asleepsMin: [0, 0]), 0) } + + // MARK: - Real-night regression: a genuine short first sleep is NOT a spurious lead + + /// THE BUG (real 2026-07-14 night): a 67-min first-sleep fragment (12:16 → 1:22, ~34 asleep min) then a + /// ~6-min walk then the main 1:29 → 7:32 sleep (~340 asleep). The two bridge into ONE night, and the + /// Health write-back spans 12:16 → 7:32. But the #259 relative "minor lead" test compared 34 asleep min + /// against 15% of the ~340-min main (≈51 min) and wrongly classified the real first sleep as a spurious + /// lead, so the displayed onset jumped to the 1:29 main and HID the true 12:16 bedtime. The absolute + /// asleep floor keeps a real sleep episode as the onset: index 0, the 12:16 fragment. + func testRealFirstSleepFragmentIsTheDisplayedOnset() { + // fragment: 66.8 span / 34 asleep. main: 363.6 span / 340 asleep. + XCTAssertEqual(SleepView.nightOnsetIndex(spansMin: [66.8, 363.6], asleepsMin: [34, 340]), 0) + } + + /// The per-fragment rule: a real 34-min sleep episode beside a 340-min main is NOT a stub, even though + /// it is under 15% of the main (the old relative test's flaw for long nights). The floor is what keeps it. + func testRealFirstSleepFragmentIsNotStub() { + XCTAssertFalse(SleepView.isPreOnsetAwakeStub(spanMin: 66.8, asleepMin: 34, refAsleepMin: 340)) + } + + /// Floor boundary: a fragment carrying at least `preOnsetStubMinorAsleepFloorMin` asleep minutes is a + /// real sleep episode and never a spurious lead, whatever the main block's size. + func testAtMinorAsleepFloorIsNotStub() { + XCTAssertFalse(SleepView.isPreOnsetAwakeStub(spanMin: 40, + asleepMin: SleepView.preOnsetStubMinorAsleepFloorMin, + refAsleepMin: 1000)) + } + + /// The floor does NOT reopen #736/#259: a tiny stray sleep lead (10 asleep, under the floor AND under + /// 15% of a big main) is still a spurious lead, so those goldens above stay green. + func testTinyStrayLeadStillStubUnderFloor() { + XCTAssertTrue(SleepView.isPreOnsetAwakeStub(spanMin: 30, asleepMin: 10, refAsleepMin: 400)) + } } From 2466aa328c7035d4f4c0fc8d21dbec1af587f081 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 11:24:02 -0700 Subject: [PATCH 090/171] =?UTF-8?q?fix(sleep):=20motion-corroborated=20wak?= =?UTF-8?q?e=20=E2=80=94=20stop=20calling=20WAKE=20on=20elevated-but-flat?= =?UTF-8?q?=20HR=20while=20motionless?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both stagers (V1 and the default SleepStagerV2) and the HR-led session confirmation (confirmSleepWithHR) call "wake" primarily off HR / HR-variability with no motion or posture cross-check. On a night whose resting HR is held elevated WITHOUT the wearer getting up (a supplement protocol, a fever, a hot room, alcohol), that logic reads hot-but-motionless sleep as WAKE — two confirmed nights (2026-07-13: 194 min WAKE vs ~67; 2026-07-14: onset 1:41 vs ~1:29 + a 44-min WAKE block) both required manual relabeling. Prior art PR #402 (merged upstream) does not prevent this: it is default-OFF, a post-pass on an already-staged hypnogram (never touches detectSleep / confirmSleepWithHR or the stager's own HR-led emission), and it corroborates on dense STEP ticks whose density gate declines on the nights that need it. This change is ON BY DEFAULT and acts where the mis-scoring is produced, corroborating with the gravity posture/jerk signal both stagers already consume. Rule: elevated HR alone is insufficient to call wake. An epoch/run at the night's quiescent MOTION floor with UNCHANGED posture cannot be scored WAKE on cardiac evidence alone. - SleepStagerV2 (default): on a motion-quiescent epoch (no observed movement; peak jerk at/below the night-relative wake-gate floor) clamp the AWAKE cardiac term to <= 0 — keep wake-suppressing (low, flat HR) evidence, drop the wake-promoting half. Byte-identical on a still low-HR epoch and on the frozen parity golden (its only WAKE phase has real gravity motion; its still phases are low-HR); genuine motion + the jerk gate still drive wake. - confirmSleepWithHR: when a run is deeply motion-quiescent (>=90% of dense-gravity minutes posture-stable), widen the HR sleep band 1.05 -> 1.30 so a supplement- elevated but motionless run isn't rejected; the band keeps a floor so genuine all-night in-bed wakefulness is still dropped. No gravity evidence -> strict band (unchanged). - adaptiveOvernightHRBaseline: personalised sleep band from recent overnight medians (self-calibrating supplement/fitness era) with a floor; threaded through detectSleep as an optional (default nil = byte-identical). Live cross-night wiring at IntelligenceEngine is a follow-up. Out-of-bed detection is untouched: sustained walking cadence / posture change with an HR jump still breaks runs and scores WAKE. Tests: 6 new fixtures replicating both nights (warm-flat-HR still night -> sleep; walking gap within a still night -> stays WAKE + still remainder not carpeted; motionless-elevated run confirmed with gravity but genuinely-high still run still rejected; adaptive baseline). Full StrandAnalytics suite 1028/1028 green; all goldens incl. the Swift<->Kotlin frozen hypnogram byte-identical. Swift only. Kotlin analytics twin (com.noop.analytics SleepStagerV2 + confirmSleepWithHR) transcription deferred to preserve the parity contract. --- .../Sources/StrandAnalytics/SleepStager.swift | 117 +++++++++- .../StrandAnalytics/SleepStagerV2.swift | 21 +- .../MotionCorroboratedWakeTests.swift | 200 ++++++++++++++++++ docs/draft-upstream-issue-hr-wake.md | 83 ++++++++ 4 files changed, 412 insertions(+), 9 deletions(-) create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/MotionCorroboratedWakeTests.swift create mode 100644 docs/draft-upstream-issue-hr-wake.md diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift index 966c8344ca..cef0f97645 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift @@ -500,8 +500,22 @@ public enum SleepStager { return HRVAnalyzer.median(vals) } - static func confirmSleepWithHR(_ p: Period, hr: [HRSample], baseline: Double?) -> Bool { - guard let baseline = baseline else { return true } + /// HR-confirmation with MOTION CORROBORATION (motion-corroborated wake). A run is confirmed as sleep when + /// its MEDIAN HR sits within the sleep band relative to `baseline`. The band is normally `hrSleepBaselineMult` + /// (×1.05); on a run that is DEEPLY MOTION-QUIESCENT (`runIsDeeplyQuiescent` — the wrist barely moved and its + /// posture did not change across the whole span) the band widens to `quiescentHRSleepMult`, because a raised + /// but FLAT overnight HR with a motionless wrist is a supplement / fever / hot-room / alcohol artefact, not + /// wakefulness — elevated-HR-alone must not reject a still run. The wider band STILL keeps a floor: a + /// genuinely awake, high-HR run (median above `quiescentHRSleepMult × baseline`) is rejected even when still, + /// so all-night in-bed wakefulness is not scored asleep. `sleepHRBaseline` (directive b) overrides `baseline` + /// with the wearer's personalised overnight band (`adaptiveOvernightHRBaseline`) when the caller supplies + /// one, so a supplement / fitness era self-calibrates instead of drifting against a fixed day-median. + /// `grav` defaults to empty (motion unprovable → strict band, byte-identical to the pre-corroboration gate); + /// the detection call site passes the night's gravity. Mirrors Kotlin `confirmSleepWithHR`. + static func confirmSleepWithHR(_ p: Period, hr: [HRSample], baseline: Double?, + grav: [GravitySample] = [], sleepHRBaseline: Double? = nil) -> Bool { + let effBaseline = sleepHRBaseline ?? baseline + guard let effBaseline = effBaseline else { return true } let seg = rowsBetween(hr, start: p.start, end: p.end) { $0.ts } if seg.count < hrRefineMinSamples { return true } // Confirm on the run's MEDIAN, not its mean. A real sleep night carries brief arousal / wake HR @@ -511,9 +525,88 @@ public enum SleepStager { // use the same robust statistic; a genuinely elevated (awake) run still has a high median and fails. // Median ≤ mean for the right-skewed HR of a real night, so this only ever RELAXES the gate — every // run the mean already accepted still passes, and runs the mean wrongly dropped are recovered. - // Mirrors Kotlin `confirmSleepWithHR`. let medHR = HRVAnalyzer.median(seg.map { Double($0.bpm) }) - return medHR <= baseline * hrSleepBaselineMult + let mult = runIsDeeplyQuiescent(p, grav: grav) ? quiescentHRSleepMult : hrSleepBaselineMult + return medHR <= effBaseline * mult + } + + // MARK: - Motion-corroborated wake (elevated-but-flat-HR nights) + + // Both stagers call "wake" primarily off HR / HR-variability, and the HR-led session confirmation + // (`confirmSleepWithHR`) rejects a still run whose median HR sits above the sleep band. On a night with + // pharmacologically- or metabolically-elevated resting HR that keeps HR up WITHOUT the wearer getting up + // (a supplement protocol, a fever, a hot room, alcohol), that HR-led logic misreads hot-but-motionless + // sleep as wake. The corroboration rule is: elevated-HR ALONE is insufficient — a run/epoch at the night's + // quiescent MOTION floor with UNCHANGED posture cannot be called wake on cardiac evidence alone. The + // per-epoch half lives in the stagers (`SleepStagerV2.motionQuiescent`); this is the session-detection half. + + /// The relaxed sleep-band multiplier applied to `confirmSleepWithHR` when the run is DEEPLY motion-quiescent. + /// Wider than `hrSleepBaselineMult` (1.05) so a supplement-elevated but motionless overnight run is not + /// rejected, yet bounded (the floor) so a genuinely awake still run above this band is still dropped. + public static let quiescentHRSleepMult: Double = 1.30 + + /// Per-minute gravity posture variance (g²) at/below which a minute counts as posture-STABLE — the wrist + /// orientation barely moved within the minute. Mirrors the stage-level posture threshold; a minute with too + /// few gravity samples to compute a variance is conservatively NOT counted as stable (silence ≠ stillness). + public static let quiescentPostureVarG2: Double = 0.05 + + /// Fraction of a run's minutes that must be posture-STABLE for the run to be DEEPLY motion-quiescent. Set + /// well above the Stage-0 stillness bar (`stillFraction` 0.70) so only a genuinely motionless run — not an + /// ordinary restless-but-in-bed night — earns the widened HR band; a night with real turn-overs/fidgets + /// across >10% of its minutes falls back to the strict band. + public static let quiescentStableFrac: Double = 0.90 + + /// Minimum posture-stable minutes with COMPUTABLE variance a run needs before the deeply-quiescent verdict + /// is trusted at all — a run with almost no dense-gravity minutes can't prove stillness, so it defers to + /// the strict HR band rather than being waved through. + public static let quiescentMinStableMinutes: Int = 20 + + /// True when `[p.start, p.end)` is DEEPLY motion-quiescent: at least `quiescentStableFrac` of the minutes + /// that carry enough gravity to judge are posture-stable (per-minute variance < `quiescentPostureVarG2`), + /// over at least `quiescentMinStableMinutes` such minutes. Empty/sparse gravity → false (motion unprovable, + /// defer to the strict HR band). Pure + deterministic. + static func runIsDeeplyQuiescent(_ p: Period, grav: [GravitySample]) -> Bool { + if grav.isEmpty || p.end <= p.start { return false } + var byMinute: [Int: [GravitySample]] = [:] + for g in grav where g.ts >= p.start && g.ts < p.end { byMinute[g.ts / 60, default: []].append(g) } + var judged = 0, stable = 0 + for (_, samples) in byMinute { + guard let v = posturVarianceG2(samples) else { continue } // too few samples this minute + judged += 1 + if v < quiescentPostureVarG2 { stable += 1 } + } + guard judged >= quiescentMinStableMinutes else { return false } + return Double(stable) / Double(judged) >= quiescentStableFrac + } + + /// Trace of the covariance of a minute's gravity vectors (Σ over x/y/z of the mean squared deviation from + /// the minute's own mean vector). ~0 when the wrist orientation is fixed within the minute; spikes on a + /// turn-over. nil below 2 samples (a single sample has zero variance by construction and would read as a + /// false "stable"). Shared shape with the stage-level posture check and the Kotlin twin. + static func posturVarianceG2(_ samples: [GravitySample]) -> Double? { + guard samples.count >= 2 else { return nil } + let n = Double(samples.count) + var sx = 0.0, sy = 0.0, sz = 0.0 + for s in samples { sx += s.x; sy += s.y; sz += s.z } + let mx = sx / n, my = sy / n, mz = sz / n + var sumSq = 0.0 + for s in samples { + let dx = s.x - mx, dy = s.y - my, dz = s.z - mz + sumSq += dx * dx + dy * dy + dz * dz + } + return sumSq / n + } + + /// Personalised overnight HR baseline (directive b): the median of recent nights' overnight median HRs, so + /// the sleep band self-calibrates to the wearer's current era (a supplement protocol, a fitness change) + /// instead of a fixed day-median that a sustained shift silently drifts against. Returns nil with no history + /// (the caller keeps the day-median). A FLOOR keeps the band from collapsing so a genuinely wakeful era is + /// not scored asleep: the returned value is never below `adaptiveBaselineFloor` bpm. Pure + deterministic. + public static let adaptiveBaselineFloor: Double = 40.0 + public static func adaptiveOvernightHRBaseline(recentOvernightMedians: [Double]) -> Double? { + let vals = recentOvernightMedians.filter { $0.isFinite && $0 > 0 } + guard !vals.isEmpty else { return nil } + return max(adaptiveBaselineFloor, HRVAnalyzer.median(vals)) } /// True when the run's CENTER, shifted to LOCAL time by tzOffsetSeconds, lands in the @@ -689,6 +782,10 @@ public enum SleepStager { /// byte-identical default (the frozen-golden tests stay green). The live call site threads /// `PuffinExperiment.experimentalSleepV2Enabled` so the Settings toggle now affects normal detected /// nights, not just the self-heal restage path. + /// `sleepHRBaseline` (motion-corroborated wake, directive b): the wearer's PERSONALISED overnight HR band + /// (`adaptiveOvernightHRBaseline`), used by `confirmSleepWithHR` in place of the day-median so a supplement / + /// fitness era self-calibrates the sleep band. Default nil keeps the day-median (byte-identical to before); + /// the live call site can thread a value derived from the trailing sleep history. public static func detectSleep(hr: [HRSample] = [], rr: [RRInterval] = [], resp: [RespSample] = [], @@ -697,6 +794,7 @@ public enum SleepStager { wristOff: [(start: Int, end: Int)] = [], bandSleepState: [(ts: Int, state: Int)] = [], useSleepStagerV2: Bool = false, + sleepHRBaseline: Double? = nil, traceSink: ((String) -> Void)? = nil) -> [SleepSession] { // Sleep & Rest test mode only: when a trace is requested we MUST run the live ladder, not a // memoized result, so each gate verdict is emitted for THIS night. The trace is side-effect- @@ -707,7 +805,7 @@ public enum SleepStager { return detectSleepUncached(hr: hr, rr: rr, resp: resp, gravity: gravity, tzOffsetSeconds: tzOffsetSeconds, wristOff: wristOff, bandSleepState: bandSleepState, useSleepStagerV2: useSleepStagerV2, - traceSink: traceSink) + sleepHRBaseline: sleepHRBaseline, traceSink: traceSink) } // v7.0.2 perf (#707): the single heaviest analytics call — it sorts the dense full-day gravity // stream (~tens of thousands of samples for a worn day), builds the gravity-delta/still spine, and @@ -725,12 +823,13 @@ public enum SleepStager { tz: tzOffsetSeconds, wristOff: StreamFingerprint.of(wristOff, ts: { $0.start }, quant: { $0.end }), band: StreamFingerprint.of(bandSleepState, ts: { $0.ts }, quant: { $0.state }), - v2: useSleepStagerV2) + v2: useSleepStagerV2, + sleepHRBaseline: sleepHRBaseline) return detectSleepCache.value(key) { detectSleepUncached(hr: hr, rr: rr, resp: resp, gravity: gravity, tzOffsetSeconds: tzOffsetSeconds, wristOff: wristOff, bandSleepState: bandSleepState, useSleepStagerV2: useSleepStagerV2, - traceSink: nil) + sleepHRBaseline: sleepHRBaseline, traceSink: nil) } } @@ -740,6 +839,7 @@ public enum SleepStager { let tz: Int let wristOff: StreamFingerprint; let band: StreamFingerprint let v2: Bool + let sleepHRBaseline: Double? } /// ≈ the number of distinct days in a scoring window; FIFO-evicted, holds only small session arrays. private static let detectSleepCache = AnalyticsMemoCache(capacity: 40) @@ -753,6 +853,7 @@ public enum SleepStager { wristOff: [(start: Int, end: Int)], bandSleepState: [(ts: Int, state: Int)], useSleepStagerV2: Bool, + sleepHRBaseline: Double? = nil, traceSink: ((String) -> Void)? = nil) -> [SleepSession] { let grav = gravity.sorted { $0.ts < $1.ts } if grav.count < 2 { return [] } @@ -824,7 +925,7 @@ public enum SleepStager { detail: "spanMin=\(spanMin) maxMainSleepSpanMin=\(maxMainSleepSpanS / 60)")) continue } - if !confirmSleepWithHR(p, hr: hrS, baseline: baseline) { + if !confirmSleepWithHR(p, hr: hrS, baseline: baseline, grav: grav, sleepHRBaseline: sleepHRBaseline) { traceSink?(GateTrace.runLine(index: runIndex, startTs: p.start, endTs: p.end, verdict: .dropped, gate: "hrConfirm", detail: "hrSleepBaselineMult=\(hrSleepBaselineMult) baseline=\(baseline.map { Int($0) } ?? -1)")) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift index 1269ddd6da..33689e673a 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift @@ -157,6 +157,20 @@ public enum SleepStagerV2 { static let jerkFloorGateMult = 35.0 // wake-boost when an epoch's peak jerk exceeds floor × this static let motionGateBoost = 4.0 + /// Motion-corroborated wake (elevated-but-flat-HR nights). An epoch is MOTION-QUIESCENT when it shows no + /// observed movement (`moveFrac == 0`) AND its peak per-second jerk sits at/below the night's own quiescent + /// floor × `jerkFloorGateMult` — i.e. the wrist did not move this epoch, on the same night-relative scale + /// the wake jerk-gate uses. On such epochs the AWAKE emission keeps any wake-SUPPRESSING cardiac evidence + /// (a low, flat HR) but discards the wake-PROMOTING half: a raised HR / HR-variability with the wrist + /// motionless is a supplement / fever / hot-room / alcohol artefact, not wakefulness, and must not vote the + /// epoch awake on its own. This never invents wake and never removes negative (pro-sleep) cardiac evidence, + /// so a genuinely still low-HR sleep epoch is byte-identical; only a still epoch whose ELEVATED HR was about + /// to push it awake is held. Motion (`zmvv`) and the jerk gate — which by construction cannot fire on a + /// quiescent epoch (`jerkMax ≤ floor × gateMult`) — still drive wake on any epoch that actually moved. + static func motionQuiescent(_ f: Epoch) -> Bool { + f.moveFrac <= 0.0 && f.jerkMax <= f.jerkScale * jerkFloorGateMult + } + /// Weight of the RSA respiration-regularity term (regular → deep, irregular → REM). static let respWeight = 0.6 @@ -447,11 +461,16 @@ public enum SleepStagerV2 { for f in feats { let zhrv = zhr(f.hr), zhvv = zhv(f.hrVar), zmvv = zmv(f.moveFrac) let gate = deepGateSlope * max(0.0, fpct(f.hrFlat11) - deepGateThresh) + // Cardiac contribution to the AWAKE emission. On a motion-quiescent epoch the wrist did not move, + // so a raised HR / HR-variability alone must NOT promote wake — clamp the cardiac term to ≤ 0, + // keeping only its wake-SUPPRESSING (pro-sleep) half. Non-quiescent epochs are unchanged. + let awakeCardiac0 = 0.5 * dz(zhvv) + 0.6 * dz(zhrv) + let awakeCardiac = motionQuiescent(f) ? min(0.0, awakeCardiac0) : awakeCardiac0 var em: [String: Double] = [ "deep": -0.8 * zhvv + 0.5 * zhrv - 0.1 * zmvv - gate + baseLogPrior["deep"]!, "rem": 0.8 * zhvv - 0.4 * zmvv + 0.4 * zhrv + baseLogPrior["rem"]!, "light": baseLogPrior["light"]!, - "awake": 1.0 * zmvv + 0.5 * dz(zhvv) + 0.6 * dz(zhrv) + baseLogPrior["awake"]!, + "awake": 1.0 * zmvv + awakeCardiac + baseLogPrior["awake"]!, ] let pr = cyclePrior(f.clock) for s in stageNames { em[s]! += pr[s]! } diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/MotionCorroboratedWakeTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/MotionCorroboratedWakeTests.swift new file mode 100644 index 0000000000..bb1a3fbe36 --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/MotionCorroboratedWakeTests.swift @@ -0,0 +1,200 @@ +import XCTest +@testable import StrandAnalytics +import WhoopProtocol + +/// Motion-corroborated wake (elevated-but-flat-HR nights). Both stagers and the HR-led session confirmation +/// call "wake" primarily off HR / HR-variability; on a night whose resting HR is held elevated WITHOUT the +/// wearer getting up (a supplement protocol, a fever, a hot room, alcohol) that logic misreads hot-but- +/// motionless sleep as wake. These fixtures replicate the two confirmed mis-scored nights and pin the rule: +/// elevated-HR ALONE cannot call wake when the wrist is at the night's quiescent motion floor with unchanged +/// posture, while genuine out-of-bed motion (walking cadence / sustained posture change) still breaks sleep. +final class MotionCorroboratedWakeTests: XCTestCase { + + // MARK: - fixtures + + /// A perfectly still gravity stream (fixed orientation) at 1 Hz — the quiescent sleep floor. + private func stillGravity(start: Int, durationS: Int, x: Double = 0, y: Double = 0, z: Double = 1.0) + -> [GravitySample] { + (0.. [GravitySample] { + (0.. GravitySample in + let minute = i / 60 + let isBurstMinute = minute % everyMin == 0 && minute > 0 + let inBurst = isBurstMinute && (i % 60) < 4 // ~4 s of turn-over inside the burst minute + return inBurst ? GravitySample(ts: start + i, x: 0.35, y: 0.30, z: 0.87) + : GravitySample(ts: start + i, x: 0, y: 0, z: 1.0) + } + } + + /// A sustained WALKING gravity stream — every second a different orientation with a large step-cadence + /// jerk (out of bed, pacing). Breaks sleep at both the detection and stage level. + private func walkingGravity(start: Int, durationS: Int) -> [GravitySample] { + (0.. GravitySample in + let ph = Double(i % 4) + return GravitySample(ts: start + i, + x: 0.5 * sin(ph), y: 0.5 * cos(ph), z: 0.7) + } + } + + /// Elevated-but-FLAT HR at 1 Hz (supplement era: resting HR held ~`base`, natural would be ~48). + private func elevatedFlatHR(start: Int, durationS: Int, base: Int) -> [HRSample] { + (0.. [RRInterval] { + (0.. RRInterval in + let rsa = Int(40.0 * sin(2.0 * Double.pi * Double(i) / 4.0)) + return RRInterval(ts: start + i, rrMs: meanMs + rsa) + } + } + + private func wakeMinutes(_ segs: [StageSegment]) -> Double { + segs.filter { $0.stage == "wake" }.reduce(0.0) { $0 + Double($1.end - $1.start) } / 60.0 + } + + // MARK: - V2 stage level: the primary default-path fix + + /// The 7/13 pattern: a ~6 h warm-flat-HR still night (turn-over bursts on a ~90-min rhythm, no walking) + /// must NOT be carpeted in WAKE. Before the fix the elevated HR drove the V2 awake emission on motionless + /// epochs; now a motion-quiescent epoch keeps only wake-SUPPRESSING cardiac evidence, so the still spans + /// score as sleep and only genuine motion could read wake. + func testWarmFlatHRStillNightScoresAsleepNotWake_V2() { + let start = 1_749_517_200 + let dur = 6 * 60 * 60 + let segs = SleepStagerV2.stageSession( + start: start, end: start + dur, + grav: stillWithTurnovers(start: start, durationS: dur, everyMin: 90), + hr: elevatedFlatHR(start: start, durationS: dur, base: 58), + rr: regularRR(start: start, durationS: dur, meanMs: 1030), + resp: []) + // The vast majority of the night is motionless sleep; wake must be a small minority, not the ~50% + // the pre-fix HR-led path produced. + XCTAssertLessThan(wakeMinutes(segs), 60.0, + "a warm-but-still night must score mostly asleep, not WAKE (got \(wakeMinutes(segs)) min)") + } + + /// The still low-HR baseline case must be byte-identical to before the fix (the change only ever removes + /// wake-PROMOTING cardiac evidence on quiescent epochs; a low, flat HR contributes none). This guards + /// against the correction accidentally shifting a normal night. + func testStillLowHRNightUnchangedByCorrection_V2() { + let start = 1_749_517_200 + let dur = 3 * 60 * 60 + let grav = stillGravity(start: start, durationS: dur) + let hr = elevatedFlatHR(start: start, durationS: dur, base: 48) // natural resting + let rr = regularRR(start: start, durationS: dur) + // Re-derive with the correction disabled by construction: a low-HR still night has no positive cardiac + // awake term to clamp, so the corrected output already equals the uncorrected recipe — assert it stages + // real sleep (deep/light present, negligible wake). + let segs = SleepStagerV2.stageSession(start: start, end: start + dur, grav: grav, hr: hr, rr: rr, resp: []) + XCTAssertLessThan(wakeMinutes(segs), 15.0, "a still low-HR night should be essentially all sleep") + } + + /// Genuine out-of-bed WALKING within an otherwise-still night must still be scored WAKE by V2 — the + /// correction must not swallow a real awakening. V2's wake gate is night-RELATIVE (thresholds scale off the + /// night's own quiescent jerk floor), so the walking block is measured against the still baseline around it, + /// its motion is positively observed, and the epochs are not quiescent — the full awake emission applies. + func testWalkingGapWithinStillNightScoresWake_V2() { + let start = 1_749_517_200 + let dur = 5 * 60 * 60 + let walkStart = 2 * 60 * 60 // a 30-min out-of-bed block two hours in + let walkEnd = walkStart + 30 * 60 + var grav: [GravitySample] = [] + for i in 0..= walkStart && i < walkEnd { + let ph = Double(i % 4) + grav.append(GravitySample(ts: start + i, x: 0.5 * sin(ph), y: 0.5 * cos(ph), z: 0.7)) + } else { + grav.append(GravitySample(ts: start + i, x: 0, y: 0, z: 1.0)) + } + } + // Elevated during the walk (out of bed), flat-elevated otherwise (supplement). + let hr = (0.. HRSample in + HRSample(ts: start + i, bpm: (i >= walkStart && i < walkEnd) ? 92 : 58) + } + let segs = SleepStagerV2.stageSession( + start: start, end: start + dur, grav: grav, + hr: hr, rr: regularRR(start: start, durationS: dur, meanMs: 1030), resp: []) + // The walking block must read WAKE; the still remainder must not be carpeted in wake. + let walkWake = segs.filter { $0.stage == "wake" && $0.end > start + walkStart && $0.start < start + walkEnd } + .reduce(0.0) { $0 + Double(min($1.end, start + walkEnd) - max($1.start, start + walkStart)) } / 60.0 + XCTAssertGreaterThan(walkWake, 15.0, + "the walking block must remain WAKE (got \(walkWake) min of \(30))") + XCTAssertLessThan(wakeMinutes(segs), 60.0, + "the still remainder must not be scored WAKE despite elevated HR") + } + + // MARK: - the motion-quiescent predicate + + func testMotionQuiescentPredicate() { + // A still epoch (no move, jerk at floor) is quiescent; an epoch with a jerk spike above the gate is not. + let still = SleepStagerV2.Epoch(start: 0, hr: 58, hrVar: 1, hrFlat11: 1, moveFrac: 0.0, jerkMax: 0.001, + respReg: nil, clock: 0.5, jerkScale: 0.001) + XCTAssertTrue(SleepStagerV2.motionQuiescent(still)) + let moved = SleepStagerV2.Epoch(start: 0, hr: 58, hrVar: 1, hrFlat11: 1, moveFrac: 0.3, jerkMax: 0.2, + respReg: nil, clock: 0.5, jerkScale: 0.001) + XCTAssertFalse(SleepStagerV2.motionQuiescent(moved)) + } + + // MARK: - detection level: confirmSleepWithHR motion corroboration + + /// A supplement night whose whole in-bed run is motionless but HR-elevated (median ~58 vs baseline 48, + /// i.e. above the strict ×1.05 = 50.4 bar) must NOT be rejected by the HR gate when gravity proves the + /// wrist was still — the widened quiescent band keeps it. Without gravity evidence the strict bar stands. + func testMotionlessElevatedRunConfirmedWithGravity() { + let start = 1_000_000 + let dur = 90 * 60 + let p = SleepStager.Period(stage: "sleep", start: start, end: start + dur) + let hr = elevatedFlatHR(start: start, durationS: dur, base: 58) + let stillGrav = stillGravity(start: start, durationS: dur) + // With gravity proving stillness → confirmed (widened band). + XCTAssertTrue(SleepStager.confirmSleepWithHR(p, hr: hr, baseline: 48.0, grav: stillGrav), + "a motionless but HR-elevated run must be confirmed as sleep when stillness is proven") + // Without gravity → strict band → rejected (the pre-existing behaviour is preserved). + XCTAssertFalse(SleepStager.confirmSleepWithHR(p, hr: hr, baseline: 48.0), + "with no motion evidence the strict HR band still rejects an elevated run") + } + + /// The floor holds: a genuinely awake, motionless run whose median HR is far above the band (72 vs + /// baseline 48 → above even the widened ×1.30 = 62.4 bar) is STILL rejected even with stillness proven, + /// so all-night in-bed wakefulness is not scored asleep. + func testGenuinelyHighHRStillRunStillRejected() { + let start = 1_000_000 + let dur = 90 * 60 + let p = SleepStager.Period(stage: "sleep", start: start, end: start + dur) + let hr = elevatedFlatHR(start: start, durationS: dur, base: 72) + let stillGrav = stillGravity(start: start, durationS: dur) + XCTAssertFalse(SleepStager.confirmSleepWithHR(p, hr: hr, baseline: 48.0, grav: stillGrav), + "a run whose HR is above even the widened quiescent band must still be rejected") + } + + func testRunIsDeeplyQuiescent() { + let start = 1_000_000 + let dur = 60 * 60 + let p = SleepStager.Period(stage: "sleep", start: start, end: start + dur) + XCTAssertTrue(SleepStager.runIsDeeplyQuiescent(p, grav: stillGravity(start: start, durationS: dur))) + XCTAssertTrue(SleepStager.runIsDeeplyQuiescent(p, grav: stillWithTurnovers(start: start, durationS: dur, everyMin: 90)), + "occasional turn-overs still leave >90% of minutes posture-stable") + XCTAssertFalse(SleepStager.runIsDeeplyQuiescent(p, grav: walkingGravity(start: start, durationS: dur)), + "a walking run is not posture-stable") + XCTAssertFalse(SleepStager.runIsDeeplyQuiescent(p, grav: []), + "no gravity → stillness unprovable → not deeply quiescent") + } + + // MARK: - adaptive overnight HR baseline (directive b) + + func testAdaptiveOvernightHRBaseline() throws { + // Self-calibrates to recent overnight medians (a supplement era's ~58 vs a fixed day-median). + XCTAssertEqual(try XCTUnwrap(SleepStager.adaptiveOvernightHRBaseline(recentOvernightMedians: [56, 58, 60, 57, 59])), + 58.0, accuracy: 1e-9) + // Floor holds so a wakeful era can't collapse the band. + XCTAssertEqual(try XCTUnwrap(SleepStager.adaptiveOvernightHRBaseline(recentOvernightMedians: [30, 32, 31])), + SleepStager.adaptiveBaselineFloor, accuracy: 1e-9) + // No history → nil (caller keeps the day-median). + XCTAssertNil(SleepStager.adaptiveOvernightHRBaseline(recentOvernightMedians: [])) + } +} diff --git a/docs/draft-upstream-issue-hr-wake.md b/docs/draft-upstream-issue-hr-wake.md new file mode 100644 index 0000000000..b425d6ae17 --- /dev/null +++ b/docs/draft-upstream-issue-hr-wake.md @@ -0,0 +1,83 @@ +# Sleep scoring over-calls WAKE when resting HR is elevated but the wrist is motionless (supplement / fever / alcohol nights) + +## Summary + +Both stagers (`SleepStager` V1 and the default `SleepStagerV2`) call **wake** primarily off HR and +HR-variability, and the session-detection HR gate (`SleepStager.confirmSleepWithHR`) rejects a still +in-bed run whose median HR sits above the sleep band. Neither path consults **motion** or **posture** +before making that call. On a night whose resting HR is held elevated **without the wearer getting up** +— a supplement protocol, a fever, a hot room, alcohol — the HR-led logic reads hot-but-motionless +sleep as wake. This over-calls WAKE (tanking efficiency, Rest, and the debt ledger) and mis-places +sleep onset, and it has required manual relabeling on multiple confirmed nights. + +The motion-aware wake refinement in **#402** (merged 2026-07-13) does not prevent this, for three +compounding reasons detailed below. + +## Evidence — two confirmed nights (both manually corrected) + +**Night A (2026-07-13).** A supplement protocol held overnight resting HR at ~55–60 (vs a natural +~47–51) and skin temp +0.4…+1.2 °C. The scorer produced **194 min WAKE** across a ~6.4 h in-bed span +(efficiency ≈ 0.50). Minute-level review of the largest WAKE blocks found **zero walking cadence**: +each was a brief turn-over burst (a single-minute posture-variance spike) bracketed by minutes of +complete stillness (per-minute posture variance < 0.05, no step ticks), recurring on a ~90-min rhythm +consistent with hot-but-atonic REM being scored as wake. Correct WAKE was ~67 min. + +**Night B (2026-07-14, PDT).** Onset was scored **1:41 AM vs a sensor truth of ~1:29** (wrist still +from 1:29:00, HR at 50 by 1:31), and a **44-min solid WAKE block 1:41–2:27** was emitted where motion +shows stillness with occasional turn-overs and HR 55–65. Sleep end (7:32:38) was correct. + +The discriminating pattern, in both nights: +- **Sleeping:** per-minute posture variance < 0.01–0.05 with **zero** walk ticks; HR flat 50–65. +- **Genuinely awake / out of bed:** sustained walking cadence (walk ticks > 0 held for minutes, e.g. + 35–53 steps/min) **or** sustained high posture variance **with** an HR jump (79–100 bpm, 46–109 + steps/min). + +Elevated-but-**flat** HR with a motionless wrist is not in the "awake" column. + +## Why #402 does not prevent this + +1. **Default-OFF.** The refinement ships behind `PuffinExperiment.motionAwareWakeEnabled`, default off. + A wearer on a supplement protocol gets no protection unless they know to toggle an Experimental flag. +2. **Wrong layer.** It is a **post-pass over an already-staged hypnogram** — it only reclassifies WAKE + *stage segments inside an already-accepted session* to `light`. It never participates in + `detectSleep` / `confirmSleepWithHR`, so it cannot fix Night B's late **onset** or protect a session + the HR gate rejected outright. And on Night A the over-called WAKE is produced by the **stager's own + HR-led emission model**, upstream of any post-pass tuning. +3. **Requires dense STEP data.** Its density self-gate (`isMotionDense`) needs ≥80% of minutes to carry + `StepSample` records with an `activityClass`. Its corroboration keys on **walk-class step ticks**. + When that stream is sparse or the activity class is unattributed, the gate declines and the pass is a + no-op — exactly on the nights that need it. + +## Proposed fix — motion-corroborated wake, on by default, at both layers + +The rule: **elevated HR alone is insufficient to call wake.** An epoch or run at the night's quiescent +**motion** floor with **unchanged posture** cannot be scored WAKE on cardiac evidence alone. Corroborate +with the **gravity posture/jerk** signal both stagers already consume (always present), not step ticks. + +- **(a) Stage level (default `SleepStagerV2`).** On a **motion-quiescent** epoch (no observed movement; + peak per-second jerk at/below the night's own quiescent floor × the wake-gate multiplier), clamp the + cardiac contribution to the AWAKE emission to ≤ 0 — keep any wake-*suppressing* (low, flat HR) evidence, + discard the wake-*promoting* half. Genuine motion (the movement z-score and the night-relative jerk + gate, which by construction cannot fire on a quiescent epoch) still drives wake. A still low-HR sleep + epoch is byte-identical; only a still epoch whose *elevated* HR was about to push it awake is held. + +- **(b) Detection level (`confirmSleepWithHR`).** When a run is **deeply motion-quiescent** (≥ ~90% of its + dense-gravity minutes are posture-stable, per-minute variance < 0.05), widen the HR sleep band from + ×1.05 to ×1.30, so a supplement-elevated but motionless overnight run is not rejected. The band still + keeps a **floor**: a run whose median HR is above even the widened band is rejected even when still, so + genuine all-night in-bed wakefulness is not scored asleep. With no gravity evidence the strict band + stands (unchanged). + +- **(c) Personalised adaptive HR baseline.** Derive the sleep band from the wearer's **recent overnight + medians** (`adaptiveOvernightHRBaseline`) rather than a fixed day-median, so a sustained supplement / + fitness era self-calibrates; keep a floor so a wakeful era can't collapse the band. + +Out-of-bed detection is untouched: sustained walking cadence and sustained posture change with an HR jump +still break runs and score WAKE. + +## Notes + +- Swift lands first (`Packages/StrandAnalytics`); the Kotlin analytics twin + (`com.noop.analytics`) transcription is a deliberate follow-up to preserve the parity contract. +- All thresholds are named constants fixed a-priori from the two reference nights, not fit to labels, + consistent with the repo's rule for a derived physiological signal. From 85e1c637cfb7349b4f4da75f4134af5f388d4dc5 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 11:50:48 -0700 Subject: [PATCH 091/171] fix(scoring): parasympathetic-saturation guard eases the Charge low-HRV penalty MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Charge (recovery) score is HRV-dominant (wHRV = 0.55): a low nightly HRV drives the score down. That mis-reads one benign pattern — parasympathetic (vagal) saturation — where in a very fit or very relaxed state HRV reads LOW while resting HR reads LOW too and the two decouple. That is a saturated rest state, not fatigue, yet the raw z-score tanked Charge on those nights. Add a conservative, additive guard in RecoveryScorer. When the HRV term points "bad" (below baseline) AND the resting-HR term points "good" (below baseline) — the decoupling signature — ease the low-HRV penalty by shrinking its (negative) z toward 0. Strength ramps from satEnterZ (0.5 sigma) to satFullZ (1.5 sigma) on the WEAKER of the two arms, capped at satMaxDampFraction (0.5), so at least half the penalty always survives, the penalty is never removed or sign-flipped, and a low-HRV + HIGH-resting-HR fatigue night is left exactly as before. With no resting-HR term the guard cannot fire (nothing corroborates the low HRV). The trace emits a "charge saturation active" line and the eased HRV term only when the guard fires; the ChargeDrivers HRV verdict explains the easing. Both route through recovery() so the breakdown and headline can never diverge. Adds RecoverySaturationGuardTests pinning both directions (saturation night not tanked; real fatigue still red) plus the helper, trace, and driver wiring. 1044 tests pass. --- .../StrandAnalytics/ChargeDrivers.swift | 22 ++- .../RecoveryScorer+Trace.swift | 24 ++- .../StrandAnalytics/RecoveryScorer.swift | 106 +++++++++- .../RecoverySaturationGuardTests.swift | 183 ++++++++++++++++++ 4 files changed, 321 insertions(+), 14 deletions(-) create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RecoverySaturationGuardTests.swift diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/ChargeDrivers.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/ChargeDrivers.swift index e79910aae3..acce836edb 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/ChargeDrivers.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/ChargeDrivers.swift @@ -154,6 +154,14 @@ extension RecoveryScorer { var drivers: [ChargeDriver] = [] + // Was the low-HRV penalty eased by the parasympathetic-saturation guard on THIS night? + // Recompute the SAME HRV / resting-HR coupling the score used so the HRV verdict can honestly + // explain a lighter-than-usual penalty (its deltaPoints already reflect the easing, since + // points(...) routes through recovery(...), which applies the guard). + let hrvZFull = zScore(hrv, mean: hrvBaseline.baseline, spread: hrvBaseline.spread) + let rhrZFull: Double? = rhrBaseline.map { zScore($0.baseline, mean: rhr, spread: $0.spread) } + let hrvSaturationEased = parasympatheticSaturation(hrvZ: hrvZFull, rhrZ: rhrZFull).active + // ── HRV (dominant driver; always present once the score exists) ────────── // Higher HRV vs baseline supports recovery. Neutral = HRV at the baseline mean. drivers.append(ChargeDriver( @@ -164,7 +172,8 @@ extension RecoveryScorer { skinTempDev: skinTempDev)), valueText: "\(Int(hrv.rounded())) ms", baselineText: "\(Int(hrvBaseline.baseline.rounded())) ms baseline", - verdict: hrvVerdict(value: hrv, baseline: hrvBaseline.baseline))) + verdict: hrvVerdict(value: hrv, baseline: hrvBaseline.baseline, + saturationEased: hrvSaturationEased))) // ── Resting HR (lower vs baseline supports recovery) ───────────────────── // Neutral = resting HR at the baseline mean. @@ -232,9 +241,16 @@ extension RecoveryScorer { // MARK: - Plain-English verdicts (no fabricated numbers; direction only) - static func hrvVerdict(value: Double, baseline: Double) -> String { + static func hrvVerdict(value: Double, baseline: Double, saturationEased: Bool = false) -> String { if value > baseline { return "above baseline, supporting recovery" } - if value < baseline { return "below baseline, limiting recovery" } + if value < baseline { + // Parasympathetic saturation: resting HR is also low and decoupled from HRV, so the low + // HRV reads as a benign saturated vagal state, not fatigue, and its penalty is eased. + // Say so instead of the plain fatigue read. (Only reached when the guard actually fired.) + return saturationEased + ? "below baseline, but low resting HR signals parasympathetic saturation, so the penalty is eased" + : "below baseline, limiting recovery" + } return "at baseline" } diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/RecoveryScorer+Trace.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/RecoveryScorer+Trace.swift index e7184ed350..44452d1cf6 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/RecoveryScorer+Trace.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/RecoveryScorer+Trace.swift @@ -71,17 +71,29 @@ extension RecoveryScorer { // pairs in the SAME order recovery(...) appends them so the renormalization below matches. var terms: [(name: String, z: Double, w: Double)] = [] + // Resting-HR z, computed up front (like recovery()) so the saturation guard can read the + // HRV<->RHR coupling before the HRV term is built. nil when there is no RHR baseline. + let rhrZForGuard: Double? = rhrBaseline.map { zScore($0.baseline, mean: rhr, spread: $0.spread) } + // HRV term: higher is better. (Always present once usable; the cold-start guard above returned.) + // The low-HRV penalty is eased when resting HR corroborates parasympathetic saturation, using + // the SAME guard recovery() applies — so the trace's HRV term matches the scored one exactly. // L9: every WEIGHT / SCALE / centre constant goes through r2() too (not just the z-scores), so a // future non-round weight (e.g. 0.333) renders identically on Swift and Kotlin and the parity // fixture cannot silently desync. The values render the same as before today. - let hrvZ = zScore(hrv, mean: hrvBaseline.baseline, spread: hrvBaseline.spread) - terms.append(("hrv", hrvZ, wHRV)) - lines.append("charge term hrv z=\(r2(hrvZ)) w=\(r2(wHRV)) (higher HRV is better)") + let hrvZRaw = zScore(hrv, mean: hrvBaseline.baseline, spread: hrvBaseline.spread) + let sat = parasympatheticSaturation(hrvZ: hrvZRaw, rhrZ: rhrZForGuard) + terms.append(("hrv", sat.effectiveHrvZ, wHRV)) + lines.append("charge term hrv z=\(r2(sat.effectiveHrvZ)) w=\(r2(wHRV)) (higher HRV is better)") + // Name the eased penalty ONLY when the guard fired, so a "Charge looks high for low HRV" + // report shows exactly why: low HRV + low resting HR = benign vagal saturation, penalty eased. + if sat.active { + lines.append("charge saturation active hrvZraw=\(r2(hrvZRaw)) rhrZ=\(r2(rhrZForGuard ?? 0)) " + + "damp=\(r2(sat.dampFraction)) (low HRV + low resting HR: parasympathetic saturation, HRV penalty eased)") + } - // RHR term: lower is better -> (mu - x) / sigma. - if let b = rhrBaseline { - let z = zScore(b.baseline, mean: rhr, spread: b.spread) + // RHR term: lower is better -> (mu - x) / sigma. (Reuses the z computed for the guard.) + if let z = rhrZForGuard { terms.append(("rhr", z, wRHR)) lines.append("charge term rhr z=\(r2(z)) w=\(r2(wRHR)) (lower RHR is better)") } else { diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/RecoveryScorer.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/RecoveryScorer.swift index 305d6c28d9..9d9dd12da3 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/RecoveryScorer.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/RecoveryScorer.swift @@ -95,6 +95,94 @@ public enum RecoveryScorer { /// Sleep-performance scale (±2 z spans the normal range). public static let sleepPerfScale: Double = 0.12 + // MARK: - Parasympathetic-saturation guard (Charge) + // + // A low nightly HRV is normally a fatigue signal, so the HRV term (wHRV = 0.55, the dominant + // driver) pulls Charge DOWN. But there is a known BENIGN pattern the raw z-score mis-reads: + // parasympathetic (vagal) saturation. In a very fit or very relaxed state the RMSSD-HRV response + // saturates and reads LOW while resting HR reads LOW too, and the two DECOUPLE. That is not poor + // recovery — it is a saturated rest state — yet the dominant HRV penalty tanks Charge anyway. + // + // The discriminator is the SIGN of the resting-HR term versus the HRV term. recovery() builds + // the HRV term as (hrv - mu)/sigma (NEGATIVE when HRV is below baseline = the penalty) and the + // RHR term as (mu - rhr)/sigma (POSITIVE when resting HR is below baseline = recovery-good). In a + // normally-coupled autonomic response a low HRV travels WITH a HIGH resting HR (sympathetic + // drive), so both terms point the SAME way (down). When instead the HRV term points "bad" while + // the resting-HR term points "good", the expected HRV<->RHR coupling has broken — the signature + // of vagal saturation, not fatigue. In that regime the guard EASES (never removes) the low-HRV + // penalty; real fatigue (low HRV + HIGH resting HR) is left exactly as it was. + + /// Personal-sigma units BOTH the low-HRV arm and the low-resting-HR arm must clear before the + /// guard engages. 0.5 sigma keeps it OFF marginal / noise nights: a night must be clearly low on + /// HRV AND clearly low on resting HR (each at least half a personal standard deviation into its + /// saturation direction) before any easing applies. + public static let satEnterZ: Double = 0.5 + + /// Personal-sigma units at which the saturation signal reaches FULL strength. Between satEnterZ + /// and this the easing ramps linearly from 0 to satMaxDampFraction; at/above it the easing is + /// capped. 1.5 sigma means "both HRV and resting HR a clear 1.5 SD below baseline" is treated as + /// unambiguous saturation. + public static let satFullZ: Double = 1.5 + + /// Maximum fraction of the low-HRV penalty the guard will ever credit back. 0.5 = at most HALF + /// the penalty is eased, so a genuinely low-HRV night is NEVER scored as if HRV sat at baseline: + /// at least half of the low-HRV penalty always survives. This is the core conservatism knob. + public static let satMaxDampFraction: Double = 0.5 + + /// Outcome of the parasympathetic-saturation check for one night. + struct ParasympatheticSaturation: Equatable, Sendable { + /// Effective HRV z after easing. Equals the input hrvZ when the guard is inactive; when + /// active it is shrunk toward 0 (never past it), lightening the low-HRV penalty. + let effectiveHrvZ: Double + /// True iff the saturation regime was detected AND the penalty was actually eased. + let active: Bool + /// Fraction of the HRV penalty credited back, in [0, satMaxDampFraction]. 0 when inactive. + let dampFraction: Double + } + + /// Detect parasympathetic saturation from tonight's HRV and resting-HR z-scores and, when the + /// signature is present, ease the low-HRV penalty by shrinking its (negative) z toward 0. + /// + /// - Parameters: + /// - hrvZ: the HRV term as recovery() builds it, (hrv - mu)/sigma. Higher is better; a + /// NEGATIVE value means HRV is below baseline (the penalty direction). + /// - rhrZ: the resting-HR term as recovery() builds it, (mu - rhr)/sigma. Higher is better; a + /// POSITIVE value means resting HR is below baseline (the recovery-good direction). Pass nil + /// when there is no resting-HR baseline (no RHR term) — with nothing to corroborate the low + /// HRV, the guard refuses to guess and never fires. + /// - Returns: the (possibly eased) effective HRV z plus whether the guard fired. + static func parasympatheticSaturation(hrvZ: Double, rhrZ: Double?) -> ParasympatheticSaturation { + // Without a resting-HR term there is nothing to corroborate the low HRV: benign saturation + // and real fatigue are indistinguishable from HRV alone, so score the night as-is. + guard let rhrZ = rhrZ else { + return ParasympatheticSaturation(effectiveHrvZ: hrvZ, active: false, dampFraction: 0) + } + // Saturation SIGNATURE, in personal-sigma units: + // hrvLow > 0 <=> HRV below baseline (the penalty we might ease) + // rhrLow > 0 <=> resting HR below baseline (the corroborating "also low" signal) + // Both must clear satEnterZ, so only a clear low-HRV + low-RHR decoupling is ever eased and a + // low-HRV + HIGH-RHR fatigue night (rhrLow < 0) can never qualify. + let hrvLow = -hrvZ + let rhrLow = rhrZ + guard hrvLow >= satEnterZ, rhrLow >= satEnterZ else { + return ParasympatheticSaturation(effectiveHrvZ: hrvZ, active: false, dampFraction: 0) + } + // Strength is driven by the WEAKER of the two arms (min), normalized from the entry gate to + // satFullZ and clamped to [0, 1]. Using the weaker arm is deliberately conservative: full + // easing requires HRV clearly low AND resting HR clearly low; a very low HRV paired with only + // a mildly low resting HR is limited by the resting-HR arm. + let couplingStrength = min(hrvLow, rhrLow) + let s = max(0.0, min(1.0, (couplingStrength - satEnterZ) / (satFullZ - satEnterZ))) + let dampFraction = satMaxDampFraction * s + // Ease by shrinking the (negative) HRV z toward 0. (1 - dampFraction) stays in + // [1 - satMaxDampFraction, 1], so the penalty is only ever REDUCED — never removed, never + // sign-flipped into a bonus. + let effectiveHrvZ = hrvZ * (1.0 - dampFraction) + return ParasympatheticSaturation(effectiveHrvZ: effectiveHrvZ, + active: dampFraction > 0, + dampFraction: dampFraction) + } + /// Rolling-mean HR window (seconds) for the resting-HR estimate. public static let restingHRWindowS: Int = 5 * 60 @@ -320,13 +408,21 @@ public enum RecoveryScorer { var terms: [(z: Double, w: Double)] = [] - // HRV term: higher is better. + // Resting-HR z, computed up front: the parasympathetic-saturation guard needs to inspect the + // HRV<->RHR coupling BEFORE the dominant HRV term is folded in. nil when there is no RHR + // baseline (no RHR term), in which case the guard cannot fire. + let rhrZ: Double? = rhrBaseline.map { zScore($0.mean, mean: rhr, spread: $0.spread) } + + // HRV term: higher is better. When resting HR corroborates a benign parasympathetic-saturation + // pattern (HRV low AND resting HR low, decoupled), the low-HRV penalty is eased — never on a + // low-HRV + HIGH-resting-HR fatigue night. See `parasympatheticSaturation`. if let b = hrvBaseline { - terms.append((zScore(hrv, mean: b.mean, spread: b.spread), wHRV)) + let hrvZ = zScore(hrv, mean: b.mean, spread: b.spread) + terms.append((parasympatheticSaturation(hrvZ: hrvZ, rhrZ: rhrZ).effectiveHrvZ, wHRV)) } - // RHR term: lower is better → (μ − x) / σ. - if let b = rhrBaseline { - terms.append((zScore(b.mean, mean: rhr, spread: b.spread), wRHR)) + // RHR term: lower is better → (μ − x) / σ. Unchanged by the guard (reuses the z above). + if let z = rhrZ { + terms.append((z, wRHR)) } // Resp term: lower is better, optional. if let r = resp, let b = respBaseline { diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RecoverySaturationGuardTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RecoverySaturationGuardTests.swift new file mode 100644 index 0000000000..f4e0bd3ce3 --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RecoverySaturationGuardTests.swift @@ -0,0 +1,183 @@ +import XCTest +import Foundation +@testable import StrandAnalytics + +/// Parasympathetic-saturation guard on Charge (recovery). A low nightly HRV normally drives Charge +/// DOWN via the dominant HRV term, but in a very fit / very relaxed state HRV can read LOW while +/// resting HR reads LOW too and the two decouple — a benign saturated vagal state, not fatigue. +/// The guard eases (never removes) the low-HRV penalty ONLY in that pattern. These tests pin BOTH +/// directions: a saturation night is NOT tanked, a real-fatigue night (low HRV + HIGH resting HR) +/// still IS. No em-dashes. +final class RecoverySaturationGuardTests: XCTestCase { + + /// A usable (trusted) baseline with a given mean and Gaussian sigma (spread is internal + /// abs-dev units; zScore multiplies by 1.253 to recover sigma). + private func baseline(mean: Double, sigma: Double, nValid: Int = 14) -> BaselineState { + BaselineState(baseline: mean, spread: sigma / 1.253, nValid: nValid, + nightsSinceUpdate: 0, status: nValid >= 14 ? .trusted : .provisional) + } + + // MARK: - The guard helper in isolation + + func testGuardFiresOnSaturationSignatureAndEasesButNeverRemovesPenalty() { + // HRV clearly below baseline (penalty direction) AND resting HR clearly below baseline + // (recovery-good direction): the saturation signature. The penalty is eased (effective z + // closer to 0) but stays negative — never inverted, never fully removed. + let s = RecoveryScorer.parasympatheticSaturation(hrvZ: -1.5, rhrZ: 1.5) + XCTAssertTrue(s.active) + XCTAssertGreaterThan(s.effectiveHrvZ, -1.5) // eased toward 0 + XCTAssertLessThan(s.effectiveHrvZ, 0.0) // still a penalty (never inverted) + XCTAssertGreaterThan(s.dampFraction, 0.0) + XCTAssertLessThanOrEqual(s.dampFraction, RecoveryScorer.satMaxDampFraction) // capped + } + + func testGuardIsSilentOnRealFatigue() { + // Low HRV + HIGH resting HR (rhrZ negative): normal sympathetic coupling, genuine fatigue. + // The guard must NOT fire — the low-HRV penalty passes through untouched. + let s = RecoveryScorer.parasympatheticSaturation(hrvZ: -1.5, rhrZ: -1.5) + XCTAssertFalse(s.active) + XCTAssertEqual(s.effectiveHrvZ, -1.5, accuracy: 1e-12) + XCTAssertEqual(s.dampFraction, 0.0, accuracy: 1e-12) + } + + func testGuardIsSilentWhenHRVIsNotLow() { + // HRV at or above baseline is not a penalty to ease, even with a very low resting HR. + XCTAssertFalse(RecoveryScorer.parasympatheticSaturation(hrvZ: 1.5, rhrZ: 1.5).active) + XCTAssertFalse(RecoveryScorer.parasympatheticSaturation(hrvZ: 0.0, rhrZ: 1.5).active) + } + + func testGuardIsSilentWithoutARestingHRSignal() { + // No RHR term (nil): nothing corroborates the low HRV, so real fatigue and benign + // saturation are indistinguishable from HRV alone. Refuse to guess -> no easing. + let s = RecoveryScorer.parasympatheticSaturation(hrvZ: -1.5, rhrZ: nil) + XCTAssertFalse(s.active) + XCTAssertEqual(s.effectiveHrvZ, -1.5, accuracy: 1e-12) + } + + func testGuardIsSilentOnMarginalDivergenceBelowTheGate() { + // Both arms present but shallow (below satEnterZ): marginal noise, not a clear pattern. + let below = RecoveryScorer.satEnterZ - 0.05 + let s = RecoveryScorer.parasympatheticSaturation(hrvZ: -below, rhrZ: below) + XCTAssertFalse(s.active) + XCTAssertEqual(s.effectiveHrvZ, -below, accuracy: 1e-12) + } + + func testDampingIsMonotonicInCorroborationAndCapped() { + // Stronger corroboration (both arms deeper below baseline) eases more, up to the cap. + let weak = RecoveryScorer.parasympatheticSaturation(hrvZ: -0.8, rhrZ: 0.8).dampFraction + let strong = RecoveryScorer.parasympatheticSaturation(hrvZ: -1.4, rhrZ: 1.4).dampFraction + let saturated = RecoveryScorer.parasympatheticSaturation(hrvZ: -3.0, rhrZ: 3.0).dampFraction + XCTAssertGreaterThan(strong, weak) + XCTAssertEqual(saturated, RecoveryScorer.satMaxDampFraction, accuracy: 1e-12) // never exceeds cap + // The weaker arm caps the strength (conservative): a very low HRV with only a mildly low + // resting HR is limited by the resting-HR arm. + let limited = RecoveryScorer.parasympatheticSaturation(hrvZ: -3.0, rhrZ: 0.8) + let bothStrong = RecoveryScorer.parasympatheticSaturation(hrvZ: -3.0, rhrZ: 3.0) + XCTAssertLessThan(limited.dampFraction, bothStrong.dampFraction) + } + + // MARK: - End to end through recovery(): both directions + + func testSaturationNightIsNotTankedButRealFatigueStillIs() { + let hrvB = baseline(mean: 50, sigma: 6.265) // sigma == 6.265 (spread * 1.253) + let rhrB = baseline(mean: 55, sigma: 5.0) + + // Same LOW HRV (41 ms, ~1.44 sigma below) for both nights; only resting HR differs. + let saturation = RecoveryScorer.recovery( + hrv: 41, rhr: 48, resp: nil, // resting HR ~1.4 sigma BELOW baseline (good) + hrvBaseline: hrvB, rhrBaseline: rhrB, respBaseline: nil, sleepPerf: nil)! + let fatigue = RecoveryScorer.recovery( + hrv: 41, rhr: 62, resp: nil, // resting HR ~1.4 sigma ABOVE baseline (bad) + hrvBaseline: hrvB, rhrBaseline: rhrB, respBaseline: nil, sleepPerf: nil)! + + // Real fatigue: still tanked into the red band. + XCTAssertLessThan(fatigue, RecoveryScorer.bandRedMax, "low HRV + high resting HR must stay red") + // Saturation: lifted OUT of red, and clearly above the fatigue night. + XCTAssertGreaterThanOrEqual(saturation, RecoveryScorer.bandRedMax, "saturation must not read red") + XCTAssertGreaterThan(saturation - fatigue, 20.0, "the guard must meaningfully separate the two") + } + + func testGuardLiftsSaturationAboveTheUndampedComposite() { + // Isolate the guard's effect (not just the low-RHR term's own contribution): reconstruct + // the score the SAME night would get with the RAW, undamped HRV z, and show recovery() is + // strictly higher because the guard eased the penalty. + let hrvB = baseline(mean: 50, sigma: 6.265) + let rhrB = baseline(mean: 55, sigma: 5.0) + + let hrvZ = RecoveryScorer.zScore(41, mean: hrvB.baseline, spread: hrvB.spread) + let rhrZ = RecoveryScorer.zScore(rhrB.baseline, mean: 48, spread: rhrB.spread) // (mu - x)/sigma + let wsum = RecoveryScorer.wHRV + RecoveryScorer.wRHR + let undampedZ = (RecoveryScorer.wHRV * hrvZ + RecoveryScorer.wRHR * rhrZ) / wsum + let undamped = 100.0 / (1.0 + exp(-RecoveryScorer.logisticK * (undampedZ - RecoveryScorer.logisticZ0))) + + let guarded = RecoveryScorer.recovery( + hrv: 41, rhr: 48, resp: nil, + hrvBaseline: hrvB, rhrBaseline: rhrB, respBaseline: nil, sleepPerf: nil)! + + XCTAssertGreaterThan(guarded, undamped + 5.0, "the guard must add real points on a saturation night") + } + + func testHighRHRHighHRVGoodNightIsUnchangedByGuard() { + // A genuinely great night (HRV up, resting HR down) must be untouched: the guard only + // engages when HRV is BELOW baseline. Recompute the plain composite and require equality. + let hrvB = baseline(mean: 50, sigma: 6.265) + let rhrB = baseline(mean: 55, sigma: 5.0) + let hrvZ = RecoveryScorer.zScore(62, mean: hrvB.baseline, spread: hrvB.spread) // HRV above + let rhrZ = RecoveryScorer.zScore(rhrB.baseline, mean: 49, spread: rhrB.spread) + let wsum = RecoveryScorer.wHRV + RecoveryScorer.wRHR + let z = (RecoveryScorer.wHRV * hrvZ + RecoveryScorer.wRHR * rhrZ) / wsum + let expected = 100.0 / (1.0 + exp(-RecoveryScorer.logisticK * (z - RecoveryScorer.logisticZ0))) + let actual = RecoveryScorer.recovery( + hrv: 62, rhr: 49, resp: nil, + hrvBaseline: hrvB, rhrBaseline: rhrB, respBaseline: nil, sleepPerf: nil)! + XCTAssertEqual(actual, expected, accuracy: 1e-9, "HRV-above-baseline night must bypass the guard") + } + + // MARK: - Explainability: trace + driver breakdown + + func testTraceEmitsSaturationLineAndDampedHRVTermOnlyWhenActive() { + let hrvB = baseline(mean: 50, sigma: 6.265) + let rhrB = baseline(mean: 55, sigma: 5.0) + + // Saturation night: the trace names the eased penalty and its score still equals recovery(). + let (satScore, satLines) = RecoveryScorer.recoveryTrace( + hrv: 41, rhr: 48, resp: nil, + hrvBaseline: hrvB, rhrBaseline: rhrB, respBaseline: nil, sleepPerf: nil) + let plain = RecoveryScorer.recovery( + hrv: 41, rhr: 48, resp: nil, + hrvBaseline: hrvB, rhrBaseline: rhrB, respBaseline: nil, sleepPerf: nil) + XCTAssertEqual(satScore, plain) + XCTAssertTrue(satLines.contains { $0.contains("charge saturation active") }) + + // Real-fatigue night: NO saturation line (guard silent). + let (_, fatLines) = RecoveryScorer.recoveryTrace( + hrv: 41, rhr: 62, resp: nil, + hrvBaseline: hrvB, rhrBaseline: rhrB, respBaseline: nil, sleepPerf: nil) + XCTAssertFalse(fatLines.contains { $0.contains("charge saturation") }) + for l in satLines + fatLines { XCTAssertFalse(l.contains("\u{2014}"), "em-dash in: \(l)") } + } + + func testChargeDriversHRVPenaltyEasedAndVerdictExplainsSaturation() { + let hrvB = baseline(mean: 50, sigma: 6.265) + let rhrB = baseline(mean: 55, sigma: 5.0) + + // Saturation night: the HRV row is still a penalty (below baseline) but its verdict + // explains the parasympathetic-saturation easing rather than the plain "limiting recovery". + let sat = RecoveryScorer.chargeDrivers( + hrv: 41, rhr: 48, resp: nil, + hrvBaseline: hrvB, rhrBaseline: rhrB, respBaseline: nil, sleepPerf: nil) + let satHRV = sat.first { $0.label == "Heart rate variability" }! + XCTAssertTrue(satHRV.verdict.contains("saturation"), "verdict must explain the eased penalty") + XCTAssertFalse(satHRV.verdict.contains("\u{2014}")) + + // Real-fatigue night: plain limiting verdict, penalty NOT eased. + let fat = RecoveryScorer.chargeDrivers( + hrv: 41, rhr: 62, resp: nil, + hrvBaseline: hrvB, rhrBaseline: rhrB, respBaseline: nil, sleepPerf: nil) + let fatHRV = fat.first { $0.label == "Heart rate variability" }! + XCTAssertTrue(fatHRV.verdict.contains("limiting recovery")) + XCTAssertLessThan(fatHRV.deltaPoints, 0) + // The eased saturation penalty is less negative than the raw fatigue penalty for the same HRV. + XCTAssertGreaterThan(satHRV.deltaPoints, fatHRV.deltaPoints) + } +} From 3ba14afc5e2114ff85529356de3227b5d6a85438 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 12:11:18 -0700 Subject: [PATCH 092/171] docs: sleep-staging motion corroboration, headline onset floor, phone timezone --- docs/ANALYTICS.md | 18 +++++++++++++ docs/DATA_MODEL.md | 39 ++++++++++++++++++++++++++++ docs/draft-upstream-issue-hr-wake.md | 2 ++ 3 files changed, 59 insertions(+) diff --git a/docs/ANALYTICS.md b/docs/ANALYTICS.md index 72e24a956c..df30a56cf6 100644 --- a/docs/ANALYTICS.md +++ b/docs/ANALYTICS.md @@ -309,6 +309,24 @@ Consecutive same-stage epochs are merged into `StageSegment`s tiling `[start, en - `SleepSession` — `start`, `end`, `efficiency` (AASM `asleep / in-bed`, where `asleep = in-bed − wake`), `stages`, per-session `restingHR` (lowest 5-min rolling-mean HR) and `avgHRV` (mean RMSSD over 5-min tumbling windows). - `hypnogramMetrics(_:)` — AASM-style roll-up: TIB / TST / SPT / SOL / REM latency / WASO / efficiency / disturbances, plus deep/REM/light minutes and percentages. +### Motion-corroborated wake — elevated-but-motionless HR is not wake (default ON) + +Both stagers (`SleepStager` V1 and the default `SleepStagerV2`) and the HR-led session confirmation (`confirmSleepWithHR`) previously called **wake** primarily off HR / HR-variability with no motion or posture cross-check. On a night whose resting HR is held elevated **without the wearer getting up** — a supplement protocol, a fever, a hot room, alcohol — that logic scored hot-but-motionless sleep as wake, over-calling WASO, mis-placing onset, and tanking efficiency and Rest. Two confirmed nights required manual relabeling (2026-07-13: 194 min WAKE vs ~67; 2026-07-14: onset 1:41 vs ~1:29 plus a 44-min WAKE block). + +The rule: **elevated HR alone is insufficient to call wake.** An epoch or run at the night's quiescent **motion** floor with **unchanged posture** cannot be scored WAKE on cardiac evidence alone; corroboration comes from the **gravity posture/jerk** signal both stagers already consume (always present), not step ticks. This acts where the mis-scoring is produced, and is **on by default** — distinct from the prior default-OFF post-pass over an already-staged hypnogram (upstream #402). + +- **`SleepStagerV2` (default).** On a motion-quiescent epoch (no observed movement; peak jerk at/below the night-relative wake-gate floor) the AWAKE cardiac term is clamped to **≤ 0** — wake-*suppressing* (low, flat HR) evidence is kept, the wake-*promoting* half is dropped. A still low-HR epoch is byte-identical; genuine motion and the night-relative jerk gate still drive wake. +- **`confirmSleepWithHR` (V1 detection).** When a run is deeply motion-quiescent (≥ ~90% of its dense-gravity minutes posture-stable), the HR sleep band widens from **×1.05 → ×1.30** so a supplement-elevated but motionless run is not rejected. The band keeps a **floor** (genuine all-night in-bed wakefulness is still dropped); with no gravity evidence the strict ×1.05 band stands. +- **`adaptiveOvernightHRBaseline`.** A personalised sleep band derived from recent overnight medians (self-calibrating across a supplement/fitness era), with a floor. Threaded through `detectSleep` as an optional argument that defaults to `nil` (byte-identical when unset); live cross-night wiring in `IntelligenceEngine` is a follow-up. + +Source: `SleepStager.swift` (`confirmSleepWithHR`, `adaptiveOvernightHRBaseline`) and `SleepStagerV2.swift` (motion-quiescent clamp), both in `Packages/StrandAnalytics`. Filed upstream as [ryanbr/noop#462](https://github.com/ryanbr/noop/issues/462). The Kotlin analytics twin (`com.noop.analytics`) is a deliberate follow-up (Swift-only contributor) — the parity contract requires the two stagers to stay byte-identical once transcribed. + +### Displayed sleep onset — the headline "Asleep at" spans the whole bridged night + +The Sleep screen headline ("Asleep at …") reports the onset of the **whole bridged night**, not the main session's start. A night stored as a short first-sleep fragment + a brief walk + the main session bridges into one group when the gap is under `gapBridgeMaxMin` (60 min). The display onset walk (`SleepView.nightOnsetTs` → `isPreOnsetAwakeStub`) previously mis-classified such a fragment as a spurious pre-onset lead: the #259 relative "minor lead" test compared the fragment's asleep minutes against 15% of the main block, so on a long main sleep a genuine short first sleep was skipped and the headline jumped forward to the main session's start. + +Fix: an **absolute floor** `preOnsetStubMinorAsleepFloorMin = 20` (min) under the #259 relative test — a leading fragment carrying **≥ 20 asleep minutes** is never treated as a spurious lead, whatever the main block's size. The relaxation is strict (it can only un-skip a real first sleep, never newly skip one), so the displayed onset now equals the bridged night's first sleep and agrees with the Apple Health write-back span (bridged night groups, #294/#364). The #736 sleepless-stub skip and the #259 tiny-stray-lead (≤ 10 min) behavior are unchanged. The constant lives in `Strand/Screens/SleepView.swift`; a Kotlin twin of the constant and the onset-walk rule is deferred (Swift-only contributor). + --- ## The **Rest** score composite — *"how restorative was your sleep?"* diff --git a/docs/DATA_MODEL.md b/docs/DATA_MODEL.md index cc1256abb2..0abe4f25d4 100644 --- a/docs/DATA_MODEL.md +++ b/docs/DATA_MODEL.md @@ -491,6 +491,45 @@ too, not left behind. --- +## Phone timezone archive + +Every timestamp in the DB is **epoch-UTC**, and nothing else records which zone the phone was in +when a day was lived, so any wall-clock reading of a historical night (bedtime, "3am", day +boundaries) is guesswork once the wearer crosses zones (Pacific ↔ Eastern) — a hard-coded-offset +assumption already corrupted an analysis pass. This table stamps each **local calendar day** with +the phone's IANA timezone so downstream analysis can localise that specific day's samples. + +### `phoneTimezone` *(v28)* + +One row per local calendar day. Written by `PhoneTimezoneStore.upsertPhoneTimezone(day:tzId:)` +(`Packages/WhoopStore/Sources/WhoopStore/PhoneTimezoneStore.swift`), stamped from +`TimeZone.current.identifier` by `IntelligenceEngine.analyzeRecent` (the daily hook) as a +fire-and-forget write that never blocks scoring. Migration `v28-phone-timezone` +(`Packages/WhoopStore/Sources/WhoopStore/Database.swift`) — additive only, a new table, no +existing row touched, numbered past upstream's v26 and local v27. + +| Column | Type | Notes | +| --- | --- | --- | +| `day` | TEXT NOT NULL | Primary key. Local calendar day, `YYYY-MM-DD`. | +| `tzId` | TEXT NOT NULL | IANA identifier, e.g. `"America/Los_Angeles"`. | + +**Primary key:** `day`. `upsertPhoneTimezone(...)` is idempotent on it via `ON CONFLICT(day) DO +UPDATE` — re-recording the same day overwrites `tzId` in place rather than duplicating. +`phoneTimezone(day:)` reads a single day back (nil when none stored). + +**Also rides every upload as a header.** When Cloud Sync is compiled in (`CLOUD_SYNC`), +`CloudSyncClient.ingest` (`Strand/CloudSync/CloudSyncClient.swift`) sets `X-Phone-Timezone: +TimeZone.current.identifier` on `POST /ingest`, so the server records the phone's **current** zone +even before this per-day table lands in a given mirror. Default builds are offline and send no such +header (`docs/PRIVACY_SECURITY.md` §1). The noop-cloud server stores the header on its `ingestLog` +row (surfaced via `data_freshness.phoneTz`) and joins this table to attach a per-night `tzId` on +`sleep_summary`. + +**Android twin deferred** (Swift-only contributor): a Room migration adding `phoneTimezone` is +needed before an Android release, per the cross-platform migration-parity contract. + +--- + ## Index summary | Index | Table | Columns | Purpose | diff --git a/docs/draft-upstream-issue-hr-wake.md b/docs/draft-upstream-issue-hr-wake.md index b425d6ae17..5c57ecbb29 100644 --- a/docs/draft-upstream-issue-hr-wake.md +++ b/docs/draft-upstream-issue-hr-wake.md @@ -1,3 +1,5 @@ +> **Filed as https://github.com/ryanbr/noop/issues/462 on 2026-07-14.** The fix shipped in this fork (default ON); see `docs/ANALYTICS.md` → *Motion-corroborated wake*. This file is kept as the filed issue's source. + # Sleep scoring over-calls WAKE when resting HR is elevated but the wrist is motionless (supplement / fever / alcohol nights) ## Summary From fe3fb931107b9150c026c051dd70222dbcdf236a Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 20:01:10 -0700 Subject: [PATCH 093/171] feat(stress): add baseline-relative daytime-stress mode + high-stress minutes DaytimeStress.analyze gains an additive `mode` parameter (default .dayRelative, byte-identical to before) alongside a new .baselineRelative case that z-scores each waking hour against a PERSONAL rolling baseline (Oura-style) instead of the day's own calm-hour reference. Reuses Baselines' existing Winsorized-EWMA machinery via two new MetricCfg entries (daytime_hr, daytime_rmssd). A missing personal RMSSD baseline (e.g. an imported Oura-era day with no R-R history) gracefully falls back to HR-only scoring, honestly flagged via Result.hrOnlyFallback. Also adds Result.highStressMinutes: an Oura-comparable "time in high stress" total (scored waking hours at/above highBandFloor, in minutes), computed uniformly in both modes. Extracted Baselines.sigma(_:) (the abs-dev-to-Gaussian-sigma conversion) out of deviation() so DaytimeStress's baseline-relative path shares the exact same conversion rather than duplicating the 1.253 constant. StressView's call site is unchanged (mode defaults to .dayRelative), so nothing the Stress screen currently shows changes. --- .../Sources/StrandAnalytics/Baselines.swift | 38 +++- .../StrandAnalytics/DaytimeStress.swift | 176 +++++++++++++++--- .../StrandAnalyticsTests/BaselinesTests.swift | 22 +++ .../DaytimeStressTests.swift | 131 +++++++++++++ 4 files changed, 333 insertions(+), 34 deletions(-) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift index fd0e2c4a7f..0247a89616 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift @@ -166,6 +166,24 @@ public enum Baselines { halfLifeB: 14.0, halfLifeS: 21.0), "strain": MetricCfg(minVal: 0.0, maxVal: 100.0, floorSpread: 5.0, halfLifeB: 14.0, halfLifeS: 21.0), + + // Daytime/waking-hours configs (DaytimeStress baseline-relative mode, added alongside + // the day-relative default). Distinct from "resting_hr"/"hrv" above, which each fold ONE + // NIGHTLY value (sleep). These fold ONE DAYTIME aggregate per day (e.g. that day's own + // calm-hour HR / RMSSD reference — see DaytimeStress.calmReference) into a cross-day + // PERSONAL baseline, so a caller can hand the resulting BaselineState to + // `DaytimeStress.analyze(mode: .baselineRelative(hr:rmssd:))` and z-score each waking + // hour against "how MY days usually run" rather than "how today's own calm hours ran". + // TUNING SEAM: minVal/maxVal/floorSpread/half-life below are a documented first pass — + // daytime HR/RMSSD are not yet validated against Oura's own baseline windows. A + // validation pass against real Oura export data may refine these. floorSpread is wider + // than the nightly "hrv" config for RMSSD: daytime RMSSD carries more incidental noise + // (posture changes, talking, movement between "calm" hours) than overnight recumbent + // HRV, so its floor tolerates more day-to-day spread before flagging a real deviation. + "daytime_hr": MetricCfg(minVal: 35.0, maxVal: 160.0, floorSpread: 3.0, + halfLifeB: 14.0, halfLifeS: 21.0), + "daytime_rmssd": MetricCfg(minVal: 5.0, maxVal: 250.0, floorSpread: 7.0, + halfLifeB: 14.0, halfLifeS: 21.0), ] /// Convenience accessors for the standard configs. @@ -174,6 +192,10 @@ public enum Baselines { public static var respCfg: MetricCfg { metricCfg["resp"]! } /// Baseline config for the RecoveryScorer Activity-Balance / previous-day-Effort term. public static var strainCfg: MetricCfg { metricCfg["strain"]! } + /// Personal daytime/waking HR baseline config — see the `daytime_hr` comment above. + public static var daytimeHRCfg: MetricCfg { metricCfg["daytime_hr"]! } + /// Personal daytime/waking RMSSD baseline config — see the `daytime_rmssd` comment above. + public static var daytimeRMSSDCfg: MetricCfg { metricCfg["daytime_rmssd"]! } /// Convert a half-life in nights to an EWMA smoothing factor. static func lambda(halfLife: Double) -> Double { @@ -352,12 +374,20 @@ public enum Baselines { // MARK: - Deviation + /// Convert a state's EWMA-abs-dev spread to an approximate Gaussian σ: 1.253 × spread, + /// floored so a caller never divides by ~0. 1.253 is E[|X−μ|] = σ·√(2/π) inverted + /// (E[|X−μ|] ≈ σ/1.253 for a Gaussian). Shared by `deviation()` below and by any other + /// z-scoring caller (e.g. `DaytimeStress`'s baseline-relative mode) so the conversion has + /// exactly one definition. + public static func sigma(_ state: BaselineState) -> Double { + max(1.253 * state.spread, 1e-9) + } + /// Compute z / delta / ratio / in-normal-range for a value vs a baseline. - /// z uses (value − baseline) / (1.253 × spread); 1.253 converts EWMA-abs-dev - /// to an approximate Gaussian σ (E[|X−μ|] = σ·√(2/π) ≈ σ/1.253). + /// z uses (value − baseline) / sigma(state); see `sigma(_:)` for the 1.253 conversion. public static func deviation(_ value: Double, state: BaselineState) -> Deviation { - let sigma = max(1.253 * state.spread, 1e-9) - let z = (value - state.baseline) / sigma + let sig = sigma(state) + let z = (value - state.baseline) / sig let delta = value - state.baseline let ratio = state.baseline != 0 ? (value / state.baseline - 1.0) : 0.0 return Deviation(z: z, delta: delta, ratio: ratio, inNormalRange: abs(z) <= 1.0) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift index 0d09e2814f..48669d7042 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift @@ -42,6 +42,36 @@ public enum DaytimeStress { public static let wakingStartHour: Int = 6 public static let wakingEndHour: Int = 22 + // MARK: - Scoring mode + + /// WHERE each hour's "calm" reference point + spread come from. Every other step — + /// bucketing, the waking-hour filter, the squash curve, sustained-high, high-stress-minutes + /// — is identical between modes; only the reference differs. + public enum ScoringMode: Equatable, Sendable { + /// DEFAULT — unchanged from before this mode existed. Each hour is z-scored against + /// THIS DAY's own calm-hour reference (`calmReference`): the lower quartile of the + /// day's own waking-hour mean HR, the upper quartile of its own waking-hour RMSSD. No + /// personal history needed — the day is its own baseline. Byte-identical output to the + /// pre-existing single-mode `analyze` for the same hr/rr/tzOffsetSeconds. + case dayRelative + + /// Oura-style — each hour is z-scored against the PERSONAL rolling baseline for daytime + /// HR (and, when available, daytime RMSSD): the SAME Winsorized-EWMA machinery + /// (`Baselines.update` / `Baselines.foldHistory`) that backs the nightly HRV / + /// resting-HR baselines elsewhere in the app, using `Baselines.metricCfg["daytime_hr"]` + /// / `["daytime_rmssd"]` (see `Baselines.daytimeHRCfg` / `daytimeRMSSDCfg`). A caller + /// builds `hr` (and, when it has the history, `rmssd`) by folding this person's past + /// daytime aggregates — e.g. day N's `calmReference` output — the same way the nightly + /// baselines are folded from past nights. + /// + /// `rmssd` is `nil` when no personal RMSSD baseline exists yet — e.g. an imported, + /// Oura-era day with no R-R stream, so there is no history to fold one from. The + /// stressor then honestly falls back to HR-only scoring for the whole read and flags + /// `Result.hrOnlyFallback`; this mirrors the per-hour graceful-nil already in + /// `rawScore`, just at the whole-baseline grain instead of the single-hour grain. + case baselineRelative(hr: BaselineState, rmssd: BaselineState?) + } + // MARK: - Output /// One hour of the daytime timeline. `level` is the shared 0–3 stress proxy, or nil @@ -82,14 +112,30 @@ public enum DaytimeStress { public let dayMean: Double? /// Peak scored hour (highest `level`), or nil. public let peak: HourPoint? + /// ADDITIVE — total minutes across SCORED waking hours at/above `highBandFloor`, the + /// Oura-comparable "time in high stress" figure. Each scored hour is one `bucketSeconds` + /// bucket, so this is `(# high-band scored hours) * bucketSeconds / 60`. Compare against + /// Oura's `stress_high_s / 60` — NOOP's timeline is hourly-grain vs Oura's ~5-minute + /// grain, so treat this as a coarse approximation, not a precise match. 0 for `.empty` + /// and for any day with no scored hours. + public let highStressMinutes: Int + /// ADDITIVE — true when `.baselineRelative` mode was requested but had no personal RMSSD + /// baseline to score against (e.g. an imported Oura-era day with no R-R history), so the + /// whole read honestly fell back to HR-only scoring. Always false in `.dayRelative` mode + /// (there, a missing RMSSD is already handled per-hour by `rawScore`, not flagged + /// day-wide) and false for `.empty`. + public let hrOnlyFallback: Bool public init(hours: [HourPoint], sustainedHigh: Bool, sustainedRun: Int, - dayMean: Double?, peak: HourPoint?) { + dayMean: Double?, peak: HourPoint?, + highStressMinutes: Int = 0, hrOnlyFallback: Bool = false) { self.hours = hours self.sustainedHigh = sustainedHigh self.sustainedRun = sustainedRun self.dayMean = dayMean self.peak = peak + self.highStressMinutes = highStressMinutes + self.hrOnlyFallback = hrOnlyFallback } /// The scored hours only (level non-nil), in time order. @@ -97,7 +143,8 @@ public enum DaytimeStress { /// Empty read — used when the day had no usable intraday HR at all. public static let empty = Result(hours: [], sustainedHigh: false, sustainedRun: 0, - dayMean: nil, peak: nil) + dayMean: nil, peak: nil, + highStressMinutes: 0, hrOnlyFallback: false) } // MARK: - Shared stress math (identical formula to the daily StressModel) @@ -144,25 +191,52 @@ public enum DaytimeStress { /// - rr: the day's `[RRInterval]`. /// - tzOffsetSeconds: seconds east of UTC, for placing each bucket on the LOCAL /// clock (so "waking hours" and the hour labels are local). Defaults to UTC. + /// - mode: `.dayRelative` (DEFAULT — unchanged existing behaviour) or + /// `.baselineRelative` (Oura-style, vs a personal rolling baseline). ADDITIVE and + /// opt-in: existing callers that don't pass `mode` keep the exact prior behaviour. /// /// Returns `.empty` when there isn't a single hour with enough HR to score. public static func analyze(hr: [HRSample], rr: [RRInterval], - tzOffsetSeconds: Int = 0) -> Result { + tzOffsetSeconds: Int = 0, + mode: ScoringMode = .dayRelative) -> Result { // v7.0.2 perf (#707): buckets the day's full HR + R-R streams into per-hour aggregates and runs an // RMSSD per hour — invoked from the Stress view, so a `body` re-evaluation re-buckets the whole day. - // Memoize on the streams' fingerprint + tz offset; result is a small `Result`, raw arrays not held. + // Memoize on the streams' fingerprint + tz offset + scoring mode; result is a small `Result`, raw + // arrays not held. The mode key folds in only (baseline, spread) for baseline-relative — the two + // fields that can change the score — not the whole BaselineState (nValid/status/etc. never do). + let modeKey: ModeKey + switch mode { + case .dayRelative: + modeKey = .dayRelative + case .baselineRelative(let hrBaseline, let rmssdBaseline): + modeKey = .baselineRelative(hrBaseline: hrBaseline.baseline, hrSpread: hrBaseline.spread, + rmssdBaseline: rmssdBaseline?.baseline, rmssdSpread: rmssdBaseline?.spread) + } let key = StressKey( hr: StreamFingerprint.of(hr, ts: { $0.ts }, quant: { Int($0.bpm) }), rr: StreamFingerprint.of(rr, ts: { $0.ts }, quant: { Int($0.rrMs) }), - tz: tzOffsetSeconds) - return analyzeCache.value(key) { analyzeUncached(hr: hr, rr: rr, tzOffsetSeconds: tzOffsetSeconds) } + tz: tzOffsetSeconds, mode: modeKey) + return analyzeCache.value(key) { + analyzeUncached(hr: hr, rr: rr, tzOffsetSeconds: tzOffsetSeconds, mode: mode) + } + } + + private struct StressKey: Hashable { + let hr: StreamFingerprint; let rr: StreamFingerprint; let tz: Int; let mode: ModeKey + } + + /// Hashable fingerprint of `ScoringMode` for the memo cache — `BaselineState` itself isn't + /// `Hashable`, and only its `baseline`/`spread` can change the score, so those are what get + /// folded in (see the `analyze` doc above). + private enum ModeKey: Hashable { + case dayRelative + case baselineRelative(hrBaseline: Double, hrSpread: Double, rmssdBaseline: Double?, rmssdSpread: Double?) } - private struct StressKey: Hashable { let hr: StreamFingerprint; let rr: StreamFingerprint; let tz: Int } private static let analyzeCache = AnalyticsMemoCache(capacity: 8) private static func analyzeUncached(hr: [HRSample], rr: [RRInterval], - tzOffsetSeconds: Int) -> Result { + tzOffsetSeconds: Int, mode: ScoringMode) -> Result { guard !hr.isEmpty else { return .empty } // 1) Bucket HR + R-R into LOCAL hour-of-day buckets, keyed by the bucket start @@ -194,25 +268,57 @@ public enum DaytimeStress { aggs.append(HourAgg(bucket: b, meanHR: mHR, rmssd: rrRes.rmssd, nHR: hrs.count)) } - // 3) The day's OWN quiet reference: centre on the CALM end (the lower quartile of - // hourly mean HR, the upper quartile of hourly RMSSD), and spread from the - // across-hour SD. This makes a flat day read ~baseline and a spiky day surface - // its tense hours — without any cross-day history. Falls back to the plain mean - // when there are too few scored hours for a quartile. - // - // Built from the WAKING hours only — the same hours scored in step 4. Sleep is the - // calmest, lowest-HR / highest-HRV stretch of the day, and the analysis window - // always begins at local midnight, so the current day routinely carries several - // hours of it. Letting those night hours into the reference drags the "calm" anchor - // far beneath every waking hour, inflating an ordinary calm day toward HIGH and - // falsely tripping the sustained-high Breathe nudge. - let referenceAggs = aggs.filter { isWakingHour($0.bucket) } - let hrMeans = referenceAggs.compactMap { $0.meanHR } - let rmssdVals = referenceAggs.compactMap { $0.rmssd } - let refHR = calmReference(hrMeans, calmIsLow: true) // calm HR is LOW - let refRMSSD = calmReference(rmssdVals, calmIsLow: false) // calm HRV is HIGH - let sdHR = std(hrMeans, mean: mean(hrMeans)) - let sdRMSSD = std(rmssdVals, mean: mean(rmssdVals)) + // 3) The reference point + spread for each signal — WHERE they come from depends on + // `mode`. Every other step (bucketing above, the waking-hour filter, the squash + // curve, sustained-high, high-stress-minutes below) is identical between modes. + let refHR: Double? + let sdHR: Double + let refRMSSD: Double? + let sdRMSSD: Double + let hrOnlyFallback: Bool + switch mode { + case .dayRelative: + // The day's OWN quiet reference: centre on the CALM end (the lower quartile of + // hourly mean HR, the upper quartile of hourly RMSSD), and spread from the + // across-hour SD. This makes a flat day read ~baseline and a spiky day surface + // its tense hours — without any cross-day history. Falls back to the plain mean + // when there are too few scored hours for a quartile. + // + // Built from the WAKING hours only — the same hours scored in step 4. Sleep is the + // calmest, lowest-HR / highest-HRV stretch of the day, and the analysis window + // always begins at local midnight, so the current day routinely carries several + // hours of it. Letting those night hours into the reference drags the "calm" anchor + // far beneath every waking hour, inflating an ordinary calm day toward HIGH and + // falsely tripping the sustained-high Breathe nudge. + let referenceAggs = aggs.filter { isWakingHour($0.bucket) } + let hrMeans = referenceAggs.compactMap { $0.meanHR } + let rmssdVals = referenceAggs.compactMap { $0.rmssd } + refHR = calmReference(hrMeans, calmIsLow: true) // calm HR is LOW + refRMSSD = calmReference(rmssdVals, calmIsLow: false) // calm HRV is HIGH + sdHR = std(hrMeans, mean: mean(hrMeans)) + sdRMSSD = std(rmssdVals, mean: mean(rmssdVals)) + hrOnlyFallback = false + + case .baselineRelative(let hrBaseline, let rmssdBaseline): + // The PERSONAL cross-day baseline, folded by the caller from past daytime + // aggregates via `Baselines.update`/`foldHistory` (see the `ScoringMode` doc). + // `sigma` is the SAME abs-dev-to-Gaussian-σ conversion `Baselines.deviation` uses. + refHR = hrBaseline.baseline + sdHR = Baselines.sigma(hrBaseline) + if let rmssdBaseline { + refRMSSD = rmssdBaseline.baseline + sdRMSSD = Baselines.sigma(rmssdBaseline) + hrOnlyFallback = false + } else { + // No personal RMSSD baseline exists (e.g. an Oura-era day with no R-R history to + // fold one from). `rawScore` already treats a nil meanRMSSD as "skip this term", + // so passing nil here gracefully degrades to HR-only scoring — flagged honestly + // in the output rather than silently. + refRMSSD = nil + sdRMSSD = 0 + hrOnlyFallback = true + } + } // 4) Score each waking-hour bucket on the shared 0–3 curve. var points: [HourPoint] = [] @@ -235,10 +341,13 @@ public enum DaytimeStress { let scored = points.compactMap { p -> (HourPoint, Double)? in p.level.map { (p, $0) } } guard !scored.isEmpty else { // No scorable waking hour — still return the (unscored) timeline so the UI can - // show "not enough data" rather than nothing. + // show "not enough data" rather than nothing. `hrOnlyFallback` is a MODE property + // (whether a personal RMSSD baseline existed to score against), so it's still worth + // reporting even though nothing ended up scored. return points.isEmpty ? .empty : Result(hours: points, sustainedHigh: false, sustainedRun: 0, - dayMean: nil, peak: nil) + dayMean: nil, peak: nil, + highStressMinutes: 0, hrOnlyFallback: hrOnlyFallback) } // 5) Sustained-high flag: walk back from the latest SCORED hour while each is HIGH. @@ -251,8 +360,15 @@ public enum DaytimeStress { let dayMean = mean(scored.map { $0.1 }) let peak = scored.max { $0.1 < $1.1 }?.0 + // 6) Oura-comparable "time in high stress": each scored hour at/above `highBandFloor` + // is one full `bucketSeconds` bucket, converted to minutes. Uses the SAME threshold + // `StressBand.high` (StressView) and the sustained-high check above already use, so + // all three stay in lockstep by construction. + let highStressMinutes = scored.filter { $0.1 >= highBandFloor }.count * (bucketSeconds / 60) + return Result(hours: points, sustainedHigh: sustained, sustainedRun: run, - dayMean: dayMean, peak: peak) + dayMean: dayMean, peak: peak, + highStressMinutes: highStressMinutes, hrOnlyFallback: hrOnlyFallback) } // MARK: - Helpers diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BaselinesTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BaselinesTests.swift index 5847831eaf..418bd97420 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BaselinesTests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/BaselinesTests.swift @@ -328,4 +328,26 @@ final class BaselinesTests: XCTestCase { let hardDay = Baselines.deviation(85.0, state: s) XCTAssertGreaterThan(hardDay.z, 0, "a much-harder-than-normal day must read ABOVE the effort baseline") } + + // MARK: - sigma() + daytime configs (DaytimeStress baseline-relative mode) + + /// `sigma(_:)` is a pure extraction of `deviation()`'s internal conversion — must match it + /// exactly and stay internally consistent (deviation at baseline + sigma is z == 1.0). + func testSigmaMatchesDeviationsInternalConversion() { + let s = Baselines.foldHistory(Array(repeating: 50.0, count: 14), cfg: Baselines.hrvCfg) + let sigma = Baselines.sigma(s) + XCTAssertEqual(sigma, max(1.253 * s.spread, 1e-9), accuracy: 1e-9) + let dev = Baselines.deviation(s.baseline + sigma, state: s) + XCTAssertEqual(dev.z, 1.0, accuracy: 1e-6) + } + + /// The new daytime_hr / daytime_rmssd configs exist, are reachable via both the dictionary + /// and the convenience accessor, and are distinct from the nightly resting_hr/hrv configs + /// they sit alongside (different bounds/floor — daytime HR runs warmer than nocturnal RHR). + func testDaytimeConfigsExistAndAreDistinctFromNightlyConfigs() { + XCTAssertEqual(Baselines.metricCfg["daytime_hr"], Baselines.daytimeHRCfg) + XCTAssertEqual(Baselines.metricCfg["daytime_rmssd"], Baselines.daytimeRMSSDCfg) + XCTAssertNotEqual(Baselines.daytimeHRCfg, Baselines.restingHRCfg) + XCTAssertNotEqual(Baselines.daytimeRMSSDCfg, Baselines.hrvCfg) + } } diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeStressTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeStressTests.swift index dd72a66aa2..95b80da4f7 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeStressTests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeStressTests.swift @@ -132,4 +132,135 @@ final class DaytimeStressTests: XCTestCase { let base = hour * 3_600 return (0.. NOT sustained + let r = DaytimeStress.analyze(hr: hr, rr: []) + + XCTAssertFalse(r.sustainedHigh, "the trailing hour is calm, so sustained-high must not fire") + let expectedHighHours = r.scored.filter { $0.level! >= DaytimeStress.highBandFloor }.count + XCTAssertGreaterThan(expectedHighHours, 0, "the isolated morning spike should read as high band") + XCTAssertEqual(r.highStressMinutes, expectedHighHours * (DaytimeStress.bucketSeconds / 60)) + XCTAssertFalse(r.hrOnlyFallback, "day-relative mode never sets the baseline-relative fallback flag") + } + + // MARK: - Baseline-relative mode (Oura-style, vs a PERSONAL rolling baseline) + + func testBaselineRelativeModeRecoversMultipleInjectedElevations() { + // Personal daytime-HR baseline: 20 constant "days" at 60 bpm converges the EWMA center + // to exactly 60 with spread pinned at the daytime_hr floor (no cross-day variability). + let hrBaseline = Baselines.foldHistory(Array(repeating: 60.0, count: 20), cfg: Baselines.daytimeHRCfg) + XCTAssertEqual(hrBaseline.baseline, 60.0, accuracy: 1e-6) + XCTAssertEqual(hrBaseline.spread, Baselines.daytimeHRCfg.floorSpread, accuracy: 1e-9) + + // FOUR distinct injected HR elevations across the SAME day's waking hours — the repo's + // derived-signal rule (CLAUDE.md "validate against the artifact, not one match") requires + // recovering MULTIPLE injected values, not a single high-vs-low pair. + let levels: [(hour: Int, bpm: Int)] = [(8, 60), (10, 64), (13, 68), (16, 72)] + var hr: [HRSample] = [] + for (h, bpm) in levels { hr += hourHR(h, bpm: bpm) } + + let r = DaytimeStress.analyze(hr: hr, rr: [], mode: .baselineRelative(hr: hrBaseline, rmssd: nil)) + let scores = levels.map { pair in r.scored.first { $0.hour == pair.hour }!.level! } + + // Strictly increasing with the injected elevation — all four levels recovered, in order. + for i in 1..= DaytimeStress.highBandFloor }.count * (DaytimeStress.bucketSeconds / 60)) + for p in r.scored { XCTAssertGreaterThanOrEqual(p.level!, DaytimeStress.highBandFloor) } + } + + func testBaselineRelativeNilRMSSDFallsBackToHROnlyAndFlagsDegraded() { + // An imported, Oura-era day: no personal RMSSD baseline exists yet (rmssd: nil) and no + // R-R stream is available either. The read must still complete honestly, never crash. + let hrBaseline = Baselines.foldHistory(Array(repeating: 60.0, count: 20), cfg: Baselines.daytimeHRCfg) + var hr: [HRSample] = [] + for h in [9, 14] { hr += hourHR(h, bpm: 80) } + let r = DaytimeStress.analyze(hr: hr, rr: [], mode: .baselineRelative(hr: hrBaseline, rmssd: nil)) + + XCTAssertTrue(r.hrOnlyFallback) + XCTAssertFalse(r.scored.isEmpty, "HR-only scoring must still produce a timeline") + for p in r.scored { XCTAssertNotNil(p.level) } + } + + func testBaselineRelativeUsesRMSSDBaselineWhenAvailable() { + // Personal baselines: HR steady at 65 bpm, RMSSD steady at 40 ms (both spread-floored). + let hrBaseline = Baselines.foldHistory(Array(repeating: 65.0, count: 20), cfg: Baselines.daytimeHRCfg) + let rmssdBaseline = Baselines.foldHistory(Array(repeating: 40.0, count: 20), cfg: Baselines.daytimeRMSSDCfg) + + var hr: [HRSample] = [] + var rr: [RRInterval] = [] + for h in [9, 14] { hr += hourHR(h, bpm: 65) } // HR AT baseline in both hours — isolates RMSSD + rr += hourRRVariable(9, rrMs: 900, jitter: 40) // normal variability + rr += hourRRVariable(14, rrMs: 900, jitter: 2) // suppressed HRV -> more stressed + + let r = DaytimeStress.analyze(hr: hr, rr: rr, + mode: .baselineRelative(hr: hrBaseline, rmssd: rmssdBaseline)) + XCTAssertFalse(r.hrOnlyFallback, "an RMSSD baseline was supplied — no fallback") + let normal = r.scored.first { $0.hour == 9 }!.level! + let suppressed = r.scored.first { $0.hour == 14 }!.level! + XCTAssertGreaterThan(suppressed, normal, + "suppressed RMSSD vs. the personal baseline should read MORE stressed than normal variability") + } } From f85f058cd0b6fbc7428838945ab203d25d49eb59 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 13 Jul 2026 20:09:04 -0700 Subject: [PATCH 094/171] fix(stress): calibrate baseline-relative HIGH threshold to validated Oura tuning MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A 26-day Oura-reference correlation found a pooled/rolling personal daytime-HR baseline (10th-percentile HR pooled across days, ~65 bpm) with a fixed ~15 bpm margin correlates best with Oura's own stress signal (r~0.6, HR-only) — better than scaling the threshold by this person's own day-to-day spread, and much better than a per-day baseline (r 0.43-0.53), confirming the cross-day rolling-EWMA design over a day-local one. Replaces .baselineRelative's HR sdHR (previously Baselines.sigma(hrBaseline), i.e. derived from personal variability) with a value solved so that exactly "baseline + baselineRelativeHighMarginBPM" lands on highBandFloor on the shared squash curve (DaytimeStress.marginToSigma). The RMSSD term is unchanged (still Baselines.sigma-scaled) since it wasn't part of the validated comparison — flagged as an open tuning seam pending an HR+HRV validation pass, which is expected to beat the HR-only r~0.6 ceiling. Also documents how this hourly, daytime-baseline mode relates to the two other baseline-relative surfaces already on the Stress screen: the daily StressModel score (nightly resting-HR/HRV vs a local trailing-30-day mean/SD, not routed through Baselines.swift) and the today-only Advanced HRV card (StressIndex/HRVFreqDomain, no baseline at all) — different grain and different signal, so this is a complementary lens, not a parallel reimplementation of either. Test fixtures updated to the new margin-calibrated sd (a 65 bpm baseline, elevations expressed in terms of the validated 15 bpm margin so the expected band crossings are exact), plus a direct test of the margin-to-sigma identity. --- .../Sources/StrandAnalytics/Baselines.swift | 20 +++--- .../StrandAnalytics/DaytimeStress.swift | 61 +++++++++++++++++-- .../DaytimeStressTests.swift | 48 ++++++++++----- 3 files changed, 102 insertions(+), 27 deletions(-) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift index 0247a89616..ce86a117f7 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift @@ -169,17 +169,21 @@ public enum Baselines { // Daytime/waking-hours configs (DaytimeStress baseline-relative mode, added alongside // the day-relative default). Distinct from "resting_hr"/"hrv" above, which each fold ONE - // NIGHTLY value (sleep). These fold ONE DAYTIME aggregate per day (e.g. that day's own - // calm-hour HR / RMSSD reference — see DaytimeStress.calmReference) into a cross-day + // NIGHTLY value (sleep). These fold ONE DAYTIME aggregate per day into a cross-day // PERSONAL baseline, so a caller can hand the resulting BaselineState to // `DaytimeStress.analyze(mode: .baselineRelative(hr:rmssd:))` and z-score each waking // hour against "how MY days usually run" rather than "how today's own calm hours ran". - // TUNING SEAM: minVal/maxVal/floorSpread/half-life below are a documented first pass — - // daytime HR/RMSSD are not yet validated against Oura's own baseline windows. A - // validation pass against real Oura export data may refine these. floorSpread is wider - // than the nightly "hrv" config for RMSSD: daytime RMSSD carries more incidental noise - // (posture changes, talking, movement between "calm" hours) than overnight recumbent - // HRV, so its floor tolerates more day-to-day spread before flagging a real deviation. + // + // VALIDATED (26-day Oura-reference correlation, HR-only, r≈0.6): the per-day value to + // fold for "daytime_hr" is that day's 10th-percentile daytime HR (~65 bpm pooled across + // the reference set) — NOT the day-relative calmReference's 25th-percentile quartile. + // The "high stress" cutoff itself is a validated fixed bpm margin over this baseline, + // NOT this config's floorSpread — see DaytimeStress.baselineRelativeHighMarginBPM. + // TUNING SEAM: minVal/maxVal/half-life below (and all of "daytime_rmssd" — RMSSD wasn't + // part of the validated HR-only comparison) are a documented first pass, refinable once + // an HR+HRV comparison exists. floorSpread is wider than the nightly "hrv" config for + // RMSSD: daytime RMSSD carries more incidental noise (posture changes, talking, movement + // between "calm" hours) than overnight recumbent HRV. "daytime_hr": MetricCfg(minVal: 35.0, maxVal: 160.0, floorSpread: 3.0, halfLifeB: 14.0, halfLifeS: 21.0), "daytime_rmssd": MetricCfg(minVal: 5.0, maxVal: 250.0, floorSpread: 7.0, diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift index 48669d7042..fe20bb1d90 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift @@ -42,11 +42,35 @@ public enum DaytimeStress { public static let wakingStartHour: Int = 6 public static let wakingEndHour: Int = 22 + /// VALIDATED (26-day Oura-reference correlation, HR-only): a personal daytime-HR elevation + /// of ~15 bpm over a POOLED/ROLLING baseline — the 10th-percentile daytime HR pooled across + /// days, ~65 bpm in the reference set — is where elevated HR starts reading as + /// Oura-comparable "high" stress (r≈0.6 against Oura's own stress signal). A PER-DAY + /// (day-relative) baseline scored WORSE in the same comparison (r 0.43–0.53): an all-day + /// elevated day pulls its own floor up and masks the stress, which is exactly why + /// `.baselineRelative` leans on `Baselines`' cross-day rolling EWMA instead of a day-local + /// reference. TUNING SEAM: this is HR-only; HR+HRV (WHOOP-era, RMSSD included) is expected + /// to beat this r≈0.6 ceiling — re-validate this margin once that comparison exists. See + /// `marginToSigma` for how it's translated onto the shared 0–3 squash curve. + public static let baselineRelativeHighMarginBPM: Double = 15.0 + // MARK: - Scoring mode /// WHERE each hour's "calm" reference point + spread come from. Every other step — /// bucketing, the waking-hour filter, the squash curve, sustained-high, high-stress-minutes /// — is identical between modes; only the reference differs. + /// + /// Relationship to the rest of the Stress screen (StressView.swift): the DAILY 0–3 score + /// (`StressModel`) already compares last night's NIGHTLY resting-HR/HRV to a plain trailing + /// 30-day mean/SD, computed locally in StressView (not via `Baselines`) — that's a + /// once-a-day number from SLEEP vitals. The Advanced HRV card (`StressIndex`, + /// `HRVFreqDomain`) is a today-only descriptive lens with no baseline at all. `.baselineRelative` + /// is neither: it's an HOURLY breakdown of TODAY from DAYTIME/waking-hours HR+RMSSD against a + /// PERSONAL cross-day rolling baseline. Daytime HR runs warmer than nocturnal resting HR + /// (posture, thermic effect), so it needs its OWN baseline (`daytime_hr`/`daytime_rmssd`, + /// below) rather than reusing the nightly `resting_hr`/`hrv` configs — reusing the nightly + /// ones would systematically over-read stress. The three surfaces are complementary lenses + /// on the same underlying autonomic signal, not competing implementations of one baseline. public enum ScoringMode: Equatable, Sendable { /// DEFAULT — unchanged from before this mode existed. Each hour is z-scored against /// THIS DAY's own calm-hour reference (`calmReference`): the lower quartile of the @@ -61,14 +85,23 @@ public enum DaytimeStress { /// resting-HR baselines elsewhere in the app, using `Baselines.metricCfg["daytime_hr"]` /// / `["daytime_rmssd"]` (see `Baselines.daytimeHRCfg` / `daytimeRMSSDCfg`). A caller /// builds `hr` (and, when it has the history, `rmssd`) by folding this person's past - /// daytime aggregates — e.g. day N's `calmReference` output — the same way the nightly + /// daytime aggregates — VALIDATED as each day's 10th-percentile daytime HR (a pooled + /// "how low does my HR run when I'm calm and awake" floor), the same way the nightly /// baselines are folded from past nights. /// + /// The HR reference point is `hr.baseline`, but the HIGH-band threshold does NOT scale + /// with this person's own day-to-day `hr.spread` — see `baselineRelativeHighMarginBPM`: + /// the validated model is a roughly FIXED bpm margin over the personal floor, not a + /// variability-scaled one. `hr.spread` still rides along on the passed-in state for + /// other consumers; this mode simply doesn't read it for the HR term. + /// /// `rmssd` is `nil` when no personal RMSSD baseline exists yet — e.g. an imported, /// Oura-era day with no R-R stream, so there is no history to fold one from. The /// stressor then honestly falls back to HR-only scoring for the whole read and flags /// `Result.hrOnlyFallback`; this mirrors the per-hour graceful-nil already in - /// `rawScore`, just at the whole-baseline grain instead of the single-hour grain. + /// `rawScore`, just at the whole-baseline grain instead of the single-hour grain. The + /// RMSSD term (when present) DOES still scale by `rmssd.spread` via `Baselines.sigma` — + /// only the HR term has a validated fixed-margin figure so far. case baselineRelative(hr: BaselineState, rmssd: BaselineState?) } @@ -182,6 +215,19 @@ public enum DaytimeStress { return min(max(s, 0), 3) } + /// Solve for the z-score spread `sd` such that a raw elevation of exactly `marginBPM` (or any + /// unit — this is unit-agnostic) squashes to exactly `band` on the shared 0–3 curve: + /// `band = 3 / (1 + e^(−marginBPM/sd))`. Used to translate `baselineRelativeHighMarginBPM`'s + /// validated bpm figure into the `sd` the shared `squash` curve expects, so "baseline + + /// margin" lands exactly on `band` by construction rather than by a second, separate + /// threshold check. Defensive fallback (never divides by zero/negative-log) if `band` is + /// ever configured at or outside the curve's open range (0, 3). + static func marginToSigma(marginBPM: Double, atBand band: Double) -> Double { + let ratio = 3.0 / band - 1.0 + guard ratio > 0, marginBPM > 0 else { return max(marginBPM, 1e-9) } + return marginBPM / (-log(ratio)) + } + // MARK: - Public API /// Build the daytime stress timeline from a day's banked HR + R-R. @@ -302,11 +348,18 @@ public enum DaytimeStress { case .baselineRelative(let hrBaseline, let rmssdBaseline): // The PERSONAL cross-day baseline, folded by the caller from past daytime // aggregates via `Baselines.update`/`foldHistory` (see the `ScoringMode` doc). - // `sigma` is the SAME abs-dev-to-Gaussian-σ conversion `Baselines.deviation` uses. refHR = hrBaseline.baseline - sdHR = Baselines.sigma(hrBaseline) + // VALIDATED tuning, not `Baselines.sigma(hrBaseline)`: the correlation study behind + // `baselineRelativeHighMarginBPM` found a roughly FIXED bpm margin over the personal + // floor — not one scaled by this person's own day-to-day spread — best matched + // Oura's stress signal. `marginToSigma` solves for the sd that makes exactly + // `refHR + baselineRelativeHighMarginBPM` land on `highBandFloor` on the shared + // squash curve, so the validated margin IS the "high" cutoff by construction. + sdHR = Self.marginToSigma(marginBPM: baselineRelativeHighMarginBPM, atBand: highBandFloor) if let rmssdBaseline { refRMSSD = rmssdBaseline.baseline + // No independently validated RMSSD margin yet (see the constant's doc) — this + // term still scales by the person's own spread via the shared σ conversion. sdRMSSD = Baselines.sigma(rmssdBaseline) hrOnlyFallback = false } else { diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeStressTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeStressTests.swift index 95b80da4f7..c48f6221ee 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeStressTests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeStressTests.swift @@ -176,18 +176,33 @@ final class DaytimeStressTests: XCTestCase { } // MARK: - Baseline-relative mode (Oura-style, vs a PERSONAL rolling baseline) + // + // Fixtures below use a 65 bpm personal HR baseline (matching the ~65 bpm pooled + // 10th-percentile figure from the validated 26-day Oura-reference correlation — see + // `DaytimeStress.baselineRelativeHighMarginBPM`) and elevations measured from it in terms of + // that validated ~15 bpm margin, so the expected band crossings are exact, not approximate. + + func testMarginToSigmaLandsExactlyOnBand() { + // The validated 15 bpm margin over baseline must land EXACTLY on highBandFloor (2.0) on + // the shared squash curve — the core identity `.baselineRelative` scoring relies on. + let sd = DaytimeStress.marginToSigma(marginBPM: DaytimeStress.baselineRelativeHighMarginBPM, + atBand: DaytimeStress.highBandFloor) + XCTAssertEqual(DaytimeStress.squash(DaytimeStress.baselineRelativeHighMarginBPM / sd), + DaytimeStress.highBandFloor, accuracy: 1e-9) + } func testBaselineRelativeModeRecoversMultipleInjectedElevations() { - // Personal daytime-HR baseline: 20 constant "days" at 60 bpm converges the EWMA center - // to exactly 60 with spread pinned at the daytime_hr floor (no cross-day variability). - let hrBaseline = Baselines.foldHistory(Array(repeating: 60.0, count: 20), cfg: Baselines.daytimeHRCfg) - XCTAssertEqual(hrBaseline.baseline, 60.0, accuracy: 1e-6) - XCTAssertEqual(hrBaseline.spread, Baselines.daytimeHRCfg.floorSpread, accuracy: 1e-9) + // Personal daytime-HR baseline: 20 constant "days" at 65 bpm converges the EWMA center + // to exactly 65 (spread is folded but NOT used for the HR high-band threshold — see + // baselineRelativeHighMarginBPM). + let hrBaseline = Baselines.foldHistory(Array(repeating: 65.0, count: 20), cfg: Baselines.daytimeHRCfg) + XCTAssertEqual(hrBaseline.baseline, 65.0, accuracy: 1e-6) // FOUR distinct injected HR elevations across the SAME day's waking hours — the repo's // derived-signal rule (CLAUDE.md "validate against the artifact, not one match") requires - // recovering MULTIPLE injected values, not a single high-vs-low pair. - let levels: [(hour: Int, bpm: Int)] = [(8, 60), (10, 64), (13, 68), (16, 72)] + // recovering MULTIPLE injected values, not a single high-vs-low pair. 65 (at baseline), + // 72 (+7, mild), 80 (+15, exactly the validated margin), 95 (+30, well past it). + let levels: [(hour: Int, bpm: Int)] = [(8, 65), (10, 72), (13, 80), (16, 95)] var hr: [HRSample] = [] for (h, bpm) in levels { hr += hourHR(h, bpm: bpm) } @@ -199,16 +214,19 @@ final class DaytimeStressTests: XCTestCase { XCTAssertGreaterThan(scores[i], scores[i - 1], "hour \(levels[i].hour) (\(levels[i].bpm) bpm) should score higher than hour \(levels[i - 1].hour) (\(levels[i - 1].bpm) bpm)") } - // The at-baseline hour reads at the 1.5 midpoint; the most-elevated hour clears HIGH. + // The at-baseline hour reads at the 1.5 midpoint; +15 bpm (the validated margin) lands + // exactly on highBandFloor; the most-elevated hour clears well past it. XCTAssertEqual(scores[0], 1.5, accuracy: 0.05) - XCTAssertGreaterThanOrEqual(scores.last!, DaytimeStress.highBandFloor) + XCTAssertEqual(scores[2], DaytimeStress.highBandFloor, accuracy: 0.01, + "the validated +15 bpm margin should land exactly on highBandFloor") + XCTAssertGreaterThan(scores.last!, DaytimeStress.highBandFloor) XCTAssertTrue(r.hrOnlyFallback, "rmssd: nil must flag the HR-only fallback") } func testBaselineRelativeCalmDayAtPersonalBaselineReadsLowNotHigh() { - let hrBaseline = Baselines.foldHistory(Array(repeating: 60.0, count: 20), cfg: Baselines.daytimeHRCfg) + let hrBaseline = Baselines.foldHistory(Array(repeating: 65.0, count: 20), cfg: Baselines.daytimeHRCfg) var hr: [HRSample] = [] - for h in [8, 10, 13, 16] { hr += hourHR(h, bpm: 60) } // every hour sits exactly at baseline + for h in [8, 10, 13, 16] { hr += hourHR(h, bpm: 65) } // every hour sits exactly at baseline let r = DaytimeStress.analyze(hr: hr, rr: [], mode: .baselineRelative(hr: hrBaseline, rmssd: nil)) for p in r.scored { @@ -220,9 +238,9 @@ final class DaytimeStressTests: XCTestCase { } func testBaselineRelativeElevatedDayProducesHighStressMinutes() { - let hrBaseline = Baselines.foldHistory(Array(repeating: 60.0, count: 20), cfg: Baselines.daytimeHRCfg) + let hrBaseline = Baselines.foldHistory(Array(repeating: 65.0, count: 20), cfg: Baselines.daytimeHRCfg) var hr: [HRSample] = [] - for h in 8...16 { hr += hourHR(h, bpm: 95) } // every waking hour well above the personal baseline + for h in 8...16 { hr += hourHR(h, bpm: 95) } // +30 bpm — twice the validated high-band margin let r = DaytimeStress.analyze(hr: hr, rr: [], mode: .baselineRelative(hr: hrBaseline, rmssd: nil)) XCTAssertGreaterThan(r.highStressMinutes, 0) @@ -234,9 +252,9 @@ final class DaytimeStressTests: XCTestCase { func testBaselineRelativeNilRMSSDFallsBackToHROnlyAndFlagsDegraded() { // An imported, Oura-era day: no personal RMSSD baseline exists yet (rmssd: nil) and no // R-R stream is available either. The read must still complete honestly, never crash. - let hrBaseline = Baselines.foldHistory(Array(repeating: 60.0, count: 20), cfg: Baselines.daytimeHRCfg) + let hrBaseline = Baselines.foldHistory(Array(repeating: 65.0, count: 20), cfg: Baselines.daytimeHRCfg) var hr: [HRSample] = [] - for h in [9, 14] { hr += hourHR(h, bpm: 80) } + for h in [9, 14] { hr += hourHR(h, bpm: 80) } // right at the validated +15 bpm margin let r = DaytimeStress.analyze(hr: hr, rr: [], mode: .baselineRelative(hr: hrBaseline, rmssd: nil)) XCTAssertTrue(r.hrOnlyFallback) From cab176d79a41fbbc207cbd98a97e232e8d3bbb0b Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 12:03:45 -0700 Subject: [PATCH 095/171] feat(stress): fold personal daytime baselines + score StressView's timeline baseline-relative MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The .baselineRelative daytime-stress mode existed but was never fed real history, so StressView still scored every intraday hour against the day's OWN calm hours (.dayRelative). This adds the missing caller-side folding path and wires StressView to use it. DaytimeBaselines.swift (new): - dayDaytimeAggregate(hr:rr:tzOffsetSeconds:): one day's daytime aggregates, computed with the SCORER's own hourly bucketing / waking window / minHourHRSamples gate so the folded value is exactly the quantity the mode later z-scores against. HR = P10 of waking-hour mean HRs (the validated ~65 bpm calm floor); RMSSD = P50 (median/typical), deliberately NOT the symmetric calm ceiling: the RMSSD term is spread-scaled (Baselines.sigma), so a median anchor keeps a typical hour neutral instead of stacking a false stress term on top of the HR term. - foldDaytimeBaselines(days:): replays the trailing per-day aggregates through Baselines.foldHistory (the same Winsorized-EWMA path as the nightly baselines) with daytimeHRCfg/daytimeRMSSDCfg. The RMSSD baseline is withheld (nil -> HR-only) until >= minNightsSeed daytime-RR days exist. - scoringMode(history:): the single graceful-degradation gate — .baselineRelative once the personal HR baseline is usable, else .dayRelative (cold start / all-sparse history stays byte-identical to before). StressView: builds the baselines from the trailing 30 local days (today excluded, unworn days skipped without an R-R read) after the existing HR-count guard, then passes the resolved mode to DaytimeStress.analyze. The timeline's explanatory copy is now mode-aware (personal baseline vs. today's own calm hours). Purely additive — a user without enough worn history keeps exactly today's behaviour. Tests: DaytimeBaselinesTests (11) cover the P10/P50 aggregate definitions, the gate + waking mirroring, the fold, the degradation gate, and the fold->score round trip including the anti-stacking property. swift test: 1054 pass; the macOS app target compiles. --- .../StrandAnalytics/DaytimeBaselines.swift | 145 ++++++++++++ .../DaytimeBaselinesTests.swift | 207 ++++++++++++++++++ Strand/Screens/StressView.swift | 62 +++++- 3 files changed, 412 insertions(+), 2 deletions(-) create mode 100644 Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeBaselines.swift create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeBaselinesTests.swift diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeBaselines.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeBaselines.swift new file mode 100644 index 0000000000..1ebf8da771 --- /dev/null +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeBaselines.swift @@ -0,0 +1,145 @@ +import Foundation +import WhoopProtocol + +// DaytimeBaselines.swift — the CALLER-side folding path that turns a person's past DAYTIME history +// into the personal rolling baselines `DaytimeStress.analyze(mode: .baselineRelative(hr:rmssd:))` +// scores today's waking hours against. +// +// `.baselineRelative` (see DaytimeStress.ScoringMode) is deliberately caller-fed: the scorer takes a +// finished `BaselineState` and never fetches history itself. This file is that missing caller — it +// computes ONE daytime aggregate per past day and folds the ordered series through the SAME +// Winsorized-EWMA machinery (`Baselines.foldHistory`) the nightly resting-HR / HRV baselines use, with +// the daytime-specific `Baselines.daytimeHRCfg` / `daytimeRMSSDCfg` configs. +// +// The per-day aggregate is defined in terms of EXACTLY the hourly means the scorer references: it reuses +// `DaytimeStress`'s own `floorDiv` / `bucketSeconds` / `isWakingHour` / `minHourHRSamples` / +// `HRVAnalyzer` so "the value we fold into the baseline" and "the value we later z-score against that +// baseline" are the same quantity, never two drifting definitions. + +public extension DaytimeStress { + + // MARK: - Per-day daytime aggregate + + /// Percentile of a day's waking-hour mean HRs used as that day's daytime-HR aggregate: the 10th, + /// i.e. the CALM FLOOR ("how low does my HR run when I'm calm and awake"). VALIDATED — this is the + /// ~65 bpm pooled figure from the 26-day Oura-reference correlation behind + /// `baselineRelativeHighMarginBPM`. It is intentionally the floor (not the median) because the HR + /// term pairs it with a FIXED bpm margin (a large effective spread via `marginToSigma`), so anchoring + /// at the floor makes "floor + margin" the exact HIGH cutoff. + static let daytimeHRAggregatePercentile: Double = 0.10 + + /// Percentile of a day's waking-hour RMSSDs used as that day's daytime-RMSSD aggregate: the 50th + /// (MEDIAN / typical), NOT the symmetric calm-ceiling percentile. The asymmetry with HR above is + /// deliberate and load-bearing: the RMSSD term is scaled by the person's OWN spread + /// (`Baselines.sigma`), not a fixed margin, so `(baseline − hourRMSSD) / σ` is only meaningful when + /// `baseline` is the TYPICAL daytime RMSSD — then a typical hour reads neutral and only genuine HRV + /// suppression reads stressed. Anchoring at a calm-ceiling percentile (the direct mirror of HR's + /// floor) would make an ordinary median hour sit ~1–2σ "below baseline" and stack a large false + /// stress term on top of the HR term. TUNING SEAM: like every RMSSD figure in this mode, the median + /// choice is a first pass pending an HR+HRV validation pass (the validated study was HR-only). + static let daytimeRMSSDAggregatePercentile: Double = 0.50 + + /// One local day's waking HR + R-R streams, the input to the daytime-baseline fold. `hr`/`rr` carry + /// wall-clock `ts`; `tzOffsetSeconds` (seconds east of UTC for THAT day) places them on the local + /// clock exactly as `analyze`'s `tzOffsetSeconds` does, so DST-varying days can each carry their own + /// offset. + /// + /// Not `Sendable`: `HRSample` / `RRInterval` (WhoopProtocol) aren't `Sendable`, and this is a plain + /// pure-function input built and consumed within one context — it is never sent across actors. + struct DaytimeDayStreams: Equatable { + public let hr: [HRSample] + public let rr: [RRInterval] + public let tzOffsetSeconds: Int + public init(hr: [HRSample], rr: [RRInterval], tzOffsetSeconds: Int) { + self.hr = hr; self.rr = rr; self.tzOffsetSeconds = tzOffsetSeconds + } + } + + /// One day's daytime aggregates, computed with the SCORER's own hourly bucketing so the folded value + /// matches what `.baselineRelative` later references: + /// - `hr`: the `daytimeHRAggregatePercentile` (P10) of the day's WAKING-hour mean HRs, where each + /// hour's mean HR is gated at `minHourHRSamples` exactly like the scorer (a sparse/imported day + /// whose hours never clear the gate yields `nil` — it contributes no daytime-HR floor, honestly). + /// - `rmssd`: the `daytimeRMSSDAggregatePercentile` (P50) of the day's WAKING-hour RMSSDs, each run + /// through the same `HRVAnalyzer.analyze(rawRR:)` cleaner the scorer uses (so ectopic beats can't + /// fabricate variability); `nil` when no waking hour had enough clean R-R (e.g. an HR-only day). + /// Either field is `nil` independently. Bucketing keys off the HR buckets (like the scorer), so an + /// hour with R-R but no HR contributes neither. + static func dayDaytimeAggregate(hr: [HRSample], rr: [RRInterval], + tzOffsetSeconds: Int) -> (hr: Double?, rmssd: Double?) { + guard !hr.isEmpty else { return (nil, nil) } + + // Bucket HR + R-R into LOCAL hour-of-day buckets, byte-for-byte the scorer's step 1. + var hrByBucket: [Int: [Double]] = [:] + for s in hr { + let local = s.ts + tzOffsetSeconds + let bucket = floorDiv(local, bucketSeconds) * bucketSeconds + hrByBucket[bucket, default: []].append(Double(s.bpm)) + } + var rrByBucket: [Int: [Double]] = [:] + for s in rr { + let local = s.ts + tzOffsetSeconds + let bucket = floorDiv(local, bucketSeconds) * bucketSeconds + rrByBucket[bucket, default: []].append(Double(s.rrMs)) + } + + // Per WAKING hour: the gated mean HR + the cleaned RMSSD, mirroring the scorer's step 2 + waking + // filter. Keyed off HR buckets so an R-R-only hour is ignored exactly as the scorer ignores it. + var wakingMeanHRs: [Double] = [] + var wakingRMSSDs: [Double] = [] + for (bucket, hrs) in hrByBucket where isWakingHour(bucket) { + if hrs.count >= minHourHRSamples, let m = mean(hrs) { wakingMeanHRs.append(m) } + if let rmssd = HRVAnalyzer.analyze(rawRR: rrByBucket[bucket] ?? []).rmssd { + wakingRMSSDs.append(rmssd) + } + } + + let hrAgg = wakingMeanHRs.isEmpty + ? nil : quantile(wakingMeanHRs.sorted(), daytimeHRAggregatePercentile) + let rmssdAgg = wakingRMSSDs.isEmpty + ? nil : quantile(wakingRMSSDs.sorted(), daytimeRMSSDAggregatePercentile) + return (hrAgg, rmssdAgg) + } + + // MARK: - Fold the trailing history into personal baselines + + /// Fold a person's trailing per-day daytime streams (OLDEST → NEWEST, TODAY EXCLUDED — today is the + /// day being scored, not part of its own baseline) into the personal baselines `.baselineRelative` + /// consumes. + /// + /// Each day contributes one `dayDaytimeAggregate`; the ordered series is replayed through + /// `Baselines.foldHistory` (the identical Winsorized-EWMA path as the nightly baselines) with + /// `daytimeHRCfg` / `daytimeRMSSDCfg`. `nil` aggregates (sparse/HR-only days) become skip-and-hold + /// nights, exactly like a missing nightly value. + /// + /// The HR baseline is always returned (it may be `.calibrating`; the caller checks `.usable` to + /// decide whether to use `.baselineRelative` at all). The RMSSD baseline is returned ONLY when it is + /// `.usable` (≥ `Baselines.minNightsSeed` days actually carried a daytime RMSSD); otherwise `nil`, so + /// the scorer honestly runs HR-only rather than z-scoring against a 1–2-day, untrustworthy HRV + /// baseline — the whole-baseline grain of the per-hour graceful-nil already in `rawScore`. + static func foldDaytimeBaselines(days: [DaytimeDayStreams]) -> (hr: BaselineState, rmssd: BaselineState?) { + var hrAggs: [Double?] = [] + var rmssdAggs: [Double?] = [] + hrAggs.reserveCapacity(days.count) + rmssdAggs.reserveCapacity(days.count) + for d in days { + let agg = dayDaytimeAggregate(hr: d.hr, rr: d.rr, tzOffsetSeconds: d.tzOffsetSeconds) + hrAggs.append(agg.hr) + rmssdAggs.append(agg.rmssd) + } + let hrState = Baselines.foldHistory(hrAggs, cfg: Baselines.daytimeHRCfg) + let rmssdState = Baselines.foldHistory(rmssdAggs, cfg: Baselines.daytimeRMSSDCfg) + return (hrState, rmssdState.usable ? rmssdState : nil) + } + + /// The scoring mode to hand `analyze` for TODAY, decided from the trailing daytime `history`: + /// `.baselineRelative` once the personal HR baseline is `.usable` (≥ `Baselines.minNightsSeed` days + /// of real daytime HR aggregates — the Oura-style, validated-r≈0.6 path), else `.dayRelative` (the + /// unchanged default). This is the single graceful-degradation gate: a cold start, or a trailing + /// window that is all sparse/imported days, keeps EXACTLY today's pre-existing day-relative behaviour. + static func scoringMode(history days: [DaytimeDayStreams]) -> ScoringMode { + let baselines = foldDaytimeBaselines(days: days) + guard baselines.hr.usable else { return .dayRelative } + return .baselineRelative(hr: baselines.hr, rmssd: baselines.rmssd) + } +} diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeBaselinesTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeBaselinesTests.swift new file mode 100644 index 0000000000..3d7cd51cd8 --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeBaselinesTests.swift @@ -0,0 +1,207 @@ +import XCTest +@testable import StrandAnalytics +import WhoopProtocol + +/// Coverage for the CALLER-side daytime-baseline fold (DaytimeBaselines.swift) that feeds +/// `DaytimeStress.analyze(mode: .baselineRelative(hr:rmssd:))`: the per-day aggregate definition, the +/// trailing-history fold, the graceful-degradation `scoringMode` gate, and the fold→score round trip. +final class DaytimeBaselinesTests: XCTestCase { + + // MARK: - Fixtures (multi-day: day `d` is offset by d·86 400 s; local hour-of-day is preserved) + + /// One local day of HR: every hour in `hours` filled with `n` 1 Hz samples at a per-hour bpm. + /// `bpms` maps to `hours` in order. tz offset 0, so local hour-of-day == wall-clock hour. + private func dayHR(_ dayIndex: Int, hours: [Int], bpms: [Int], + n: Int = DaytimeStress.minHourHRSamples) -> [HRSample] { + let dayBase = dayIndex * 86_400 + var out: [HRSample] = [] + for (h, bpm) in zip(hours, bpms) { + let base = dayBase + h * 3_600 + out += (0.. [HRSample] { + dayHR(dayIndex, hours: Array(8...17), bpms: Array(repeating: bpm, count: 10)) + } + + /// R-R for one local hour with a controllable beat-to-beat jitter (drives RMSSD). + private func hourRR(_ dayIndex: Int, hour: Int, rrMs: Int, jitter: Int, n: Int = 120) -> [RRInterval] { + let base = dayIndex * 86_400 + hour * 3_600 + return (0.. [RRInterval] { + (8...17).flatMap { hourRR(dayIndex, hour: $0, rrMs: rrMs, jitter: jitter) } + } + + // MARK: - Per-day aggregate + + func testDayHRAggregateIsTheTenthPercentileOfWakingHourMeanHRs() { + // Ten waking hours (08…17) with a spread of constant per-hour HRs; each hour's MEAN HR is just + // its bpm. P10 of [60,62,64,66,68,70,72,74,76,78] via the shared linear-interp quantile is + // 60 + 0.9·(62−60) = 61.8 — the calm floor, well below the day's median. + let bpms = [60, 62, 64, 66, 68, 70, 72, 74, 76, 78] + let hr = dayHR(0, hours: Array(8...17), bpms: bpms) + let agg = DaytimeStress.dayDaytimeAggregate(hr: hr, rr: [], tzOffsetSeconds: 0) + XCTAssertNotNil(agg.hr) + XCTAssertEqual(agg.hr!, 61.8, accuracy: 1e-6) + XCTAssertNil(agg.rmssd, "no R-R → no daytime RMSSD aggregate") + } + + func testDayHRAggregateAppliesTheSameMinSamplesGateAsTheScorer() { + // An under-gate sparse hour at a very low bpm must NOT drag the P10 floor down — the scorer would + // never score it, so the aggregate must not reference it either. + let dense = dayHR(0, hours: Array(8...17), bpms: [60, 62, 64, 66, 68, 70, 72, 74, 76, 78]) + let sparse = dayHR(0, hours: [7], bpms: [40], n: DaytimeStress.minHourHRSamples - 1) + let withSparse = DaytimeStress.dayDaytimeAggregate(hr: dense + sparse, rr: [], tzOffsetSeconds: 0) + let denseOnly = DaytimeStress.dayDaytimeAggregate(hr: dense, rr: [], tzOffsetSeconds: 0) + XCTAssertEqual(withSparse.hr!, denseOnly.hr!, accuracy: 1e-9, + "a below-gate hour leaked into the daytime-HR floor") + } + + func testDayHRAggregateExcludesNonWakingHours() { + // A very low 03:00 hour (outside 06:00–22:00) must not become the calm floor. + let waking = dayHR(0, hours: Array(8...17), bpms: [60, 62, 64, 66, 68, 70, 72, 74, 76, 78]) + let night = dayHR(0, hours: [3], bpms: [45]) + let withNight = DaytimeStress.dayDaytimeAggregate(hr: waking + night, rr: [], tzOffsetSeconds: 0) + XCTAssertEqual(withNight.hr!, 61.8, accuracy: 1e-6, + "a non-waking hour leaked into the daytime-HR floor") + } + + func testDayRMSSDAggregateIsPresentWithRRAndTracksVariability() { + // A relaxed day (high jitter → high RMSSD) vs a suppressed day (low jitter → low RMSSD): both + // produce a non-nil median RMSSD aggregate, and the relaxed day's is clearly higher. + let hr = flatDayHR(0, bpm: 65) + let relaxed = DaytimeStress.dayDaytimeAggregate(hr: hr, rr: flatDayRR(0, rrMs: 900, jitter: 45), + tzOffsetSeconds: 0) + let suppressed = DaytimeStress.dayDaytimeAggregate(hr: hr, rr: flatDayRR(0, rrMs: 900, jitter: 5), + tzOffsetSeconds: 0) + XCTAssertNotNil(relaxed.rmssd) + XCTAssertNotNil(suppressed.rmssd) + XCTAssertGreaterThan(relaxed.rmssd!, suppressed.rmssd!, + "the higher-variability day should carry a higher median daytime RMSSD") + } + + // MARK: - Fold trailing history + + func testFoldConvergesHRBaselineToThePersonalDaytimeFloor() { + // Twelve identical days whose P10 daytime HR is exactly 65 → the EWMA center converges to 65 and + // the baseline is usable (12 ≥ minNightsSeed). No R-R anywhere → RMSSD baseline is nil. + let days = (0..<12).map { DaytimeStress.DaytimeDayStreams(hr: flatDayHR($0, bpm: 65), rr: [], + tzOffsetSeconds: 0) } + let b = DaytimeStress.foldDaytimeBaselines(days: days) + XCTAssertEqual(b.hr.baseline, 65.0, accuracy: 1e-6) + XCTAssertTrue(b.hr.usable) + XCTAssertNil(b.rmssd, "no daytime R-R history → HR-only baseline") + } + + func testFoldReturnsRMSSDBaselineOnlyWhenEnoughDaytimeRRDays() { + // Two days of R-R is below minNightsSeed → RMSSD baseline withheld (nil → HR-only). Eight days + // clears the seed → a usable RMSSD baseline is returned. + let twoDays = (0..<2).map { + DaytimeStress.DaytimeDayStreams(hr: flatDayHR($0, bpm: 65), + rr: flatDayRR($0, rrMs: 900, jitter: 40), tzOffsetSeconds: 0) + } + XCTAssertNil(DaytimeStress.foldDaytimeBaselines(days: twoDays).rmssd, + "an untrustworthy 2-day RMSSD baseline must be withheld") + + let eightDays = (0..<8).map { + DaytimeStress.DaytimeDayStreams(hr: flatDayHR($0, bpm: 65), + rr: flatDayRR($0, rrMs: 900, jitter: 40), tzOffsetSeconds: 0) + } + let rmssd = DaytimeStress.foldDaytimeBaselines(days: eightDays).rmssd + XCTAssertNotNil(rmssd, "eight days of daytime R-R should yield a usable RMSSD baseline") + XCTAssertTrue(rmssd!.usable) + } + + // MARK: - scoringMode graceful degradation + + func testScoringModeFallsBackToDayRelativeOnColdStartAndSparseHistory() { + // No history → day-relative (cold start unchanged). + guard case .dayRelative = DaytimeStress.scoringMode(history: []) else { + return XCTFail("empty history must fall back to .dayRelative") + } + // Three days is below minNightsSeed → still day-relative. + let threeDays = (0..<3).map { DaytimeStress.DaytimeDayStreams(hr: flatDayHR($0, bpm: 65), rr: [], + tzOffsetSeconds: 0) } + guard case .dayRelative = DaytimeStress.scoringMode(history: threeDays) else { + return XCTFail("sub-seed history must fall back to .dayRelative") + } + } + + func testScoringModeUsesBaselineRelativeOnceHistoryIsUsable() { + // Six days of real daytime HR clears the seed → the Oura-style baseline-relative mode, carrying + // the folded HR baseline (~65) and a nil RMSSD baseline (no R-R history here). + let days = (0..<6).map { DaytimeStress.DaytimeDayStreams(hr: flatDayHR($0, bpm: 65), rr: [], + tzOffsetSeconds: 0) } + guard case let .baselineRelative(hr, rmssd) = DaytimeStress.scoringMode(history: days) else { + return XCTFail("usable history must select .baselineRelative") + } + XCTAssertEqual(hr.baseline, 65.0, accuracy: 1e-6) + XCTAssertNil(rmssd) + } + + // MARK: - Fold → score round trip (the whole point) + + func testFoldedBaselineScoresElevationAgainstThePersonalFloorNotTheDaysOwnHours() { + // Personal floor folded from history = 65 bpm. TODAY is elevated ALL day (every waking hour 80 = + // 65 + the validated 15 bpm margin). A DAY-relative read would anchor on the day's OWN hours and + // miss an all-day-high day; the baseline-relative read scores it against the personal 65 floor, + // so every hour lands at/above the HIGH band and the day logs high-stress minutes. + let history = (0..<20).map { DaytimeStress.DaytimeDayStreams(hr: flatDayHR($0, bpm: 65), rr: [], + tzOffsetSeconds: 0) } + let mode = DaytimeStress.scoringMode(history: history) + guard case .baselineRelative = mode else { return XCTFail("expected baseline-relative") } + + let today = flatDayHR(0, bpm: 80) + let r = DaytimeStress.analyze(hr: today, rr: [], tzOffsetSeconds: 0, mode: mode) + XCTAssertFalse(r.scored.isEmpty) + XCTAssertTrue(r.hrOnlyFallback, "no RMSSD baseline → HR-only, honestly flagged") + XCTAssertGreaterThan(r.highStressMinutes, 0, "an all-day-elevated day must log high-stress time") + for p in r.scored { + XCTAssertGreaterThanOrEqual(p.level!, DaytimeStress.highBandFloor - 0.05, + "80 bpm is the personal floor + the validated 15 bpm margin → the HIGH band") + } + } + + func testFoldedBaselineCalmDayAtThePersonalFloorReadsNeutralNotHigh() { + // TODAY sits exactly at the personal 65 floor all day → each hour reads ~1.5 (neutral), never + // high. Guards the calm end of the scale end-to-end through the fold. + let history = (0..<20).map { DaytimeStress.DaytimeDayStreams(hr: flatDayHR($0, bpm: 65), rr: [], + tzOffsetSeconds: 0) } + let mode = DaytimeStress.scoringMode(history: history) + let r = DaytimeStress.analyze(hr: flatDayHR(0, bpm: 65), rr: [], tzOffsetSeconds: 0, mode: mode) + XCTAssertEqual(r.highStressMinutes, 0) + for p in r.scored { XCTAssertEqual(p.level!, 1.5, accuracy: 0.05) } + } + + func testMedianRMSSDBaselineKeepsANormalHourNeutralNoStacking() { + // The anti-stacking guarantee behind anchoring the RMSSD aggregate at the MEDIAN (not a calm + // ceiling): with BOTH a HR baseline (65) and an RMSSD baseline folded from history, a TODAY that + // sits at the HR floor AND at its typical daytime HRV must stay out of the HIGH band. A ceiling + // anchor would make this ordinary hour read ~1–2σ "below baseline" and stack a false stress term. + let history = (0..<20).map { + DaytimeStress.DaytimeDayStreams(hr: flatDayHR($0, bpm: 65), + rr: flatDayRR($0, rrMs: 900, jitter: 40), tzOffsetSeconds: 0) + } + let mode = DaytimeStress.scoringMode(history: history) + guard case let .baselineRelative(_, rmssd) = mode else { + return XCTFail("expected baseline-relative with an RMSSD baseline") + } + XCTAssertNotNil(rmssd, "20 days of daytime R-R should yield a usable RMSSD baseline") + + // Today: HR at the floor, HRV at the SAME typical variability as history. + let today = flatDayHR(0, bpm: 65) + let todayRR = flatDayRR(0, rrMs: 900, jitter: 40) + let r = DaytimeStress.analyze(hr: today, rr: todayRR, tzOffsetSeconds: 0, mode: mode) + XCTAssertFalse(r.hrOnlyFallback, "an RMSSD baseline was supplied") + XCTAssertEqual(r.highStressMinutes, 0, "a typical HR+HRV day must not read as high stress") + for p in r.scored { + XCTAssertLessThan(p.level!, DaytimeStress.highBandFloor, + "a normal hour with an RMSSD baseline must not stack into the HIGH band") + } + } +} diff --git a/Strand/Screens/StressView.swift b/Strand/Screens/StressView.swift index 0735c5e6bc..0ebba87810 100644 --- a/Strand/Screens/StressView.swift +++ b/Strand/Screens/StressView.swift @@ -40,6 +40,10 @@ struct StressView: View { /// from the day's banked HR + R-R via the SAME 0–3 proxy the daily score uses. Nil /// until the async read completes; `.empty` when the day has no usable intraday HR. @State private var daytime: DaytimeStress.Result? + /// Whether TODAY's intraday timeline is scored against the PERSONAL cross-day daytime baseline + /// (`.baselineRelative`, once enough worn history exists) instead of the day's own calm hours + /// (`.dayRelative`). Drives only the explanatory copy — the 0–3 scale + bands are identical either way. + @State private var daytimeUsesPersonalBaseline = false /// Drives the Breathe sheet presented from the sustained-stress suggestion. @State private var showBreathe = false @@ -112,7 +116,14 @@ struct StressView: View { let rr = (try? await repo.storeHandle()?.rrIntervals( deviceId: repo.deviceId, from: from, to: to, limit: 200_000)) ?? [] - daytime = DaytimeStress.analyze(hr: hr, rr: rr, tzOffsetSeconds: tz) + // Score today's hours against the PERSONAL cross-day daytime baseline when enough worn history + // exists (Oura-style `.baselineRelative`, validated r≈0.6), else the day's own calm hours + // (`.dayRelative`, the unchanged default). The mode is resolved only AFTER the HR-count guard + // above, so the trailing-history reads are never paid on a day with no scorable timeline. + let mode = await daytimeScoringMode(startOfToday: startOfDay) + if case .baselineRelative = mode { daytimeUsesPersonalBaseline = true } + else { daytimeUsesPersonalBaseline = false } + daytime = DaytimeStress.analyze(hr: hr, rr: rr, tzOffsetSeconds: tz, mode: mode) // ADDITIVE advanced readouts, computed on-demand from the SAME `rr` (no extra fetch, no // DB / schema change, and no effect on the 0..3 score above). Each engine returns nil when @@ -122,6 +133,43 @@ struct StressView: View { freqHRV = HRVFreqDomain.freqDomain(rr: rr) } + /// Trailing local days folded into the personal daytime baselines the `.baselineRelative` mode + /// scores against. 30 mirrors the app's other rolling baselines (nightly resting-HR / HRV) and the + /// illness engine's ~28-day base. + private static let baselineHistoryDays = 30 + + /// Build the personal daytime baselines from the trailing `baselineHistoryDays` local days (TODAY + /// EXCLUDED — it's the day being scored, not part of its own baseline) and return the scoring mode + /// for today's intraday read: `.baselineRelative` once there's enough real worn daytime-HR history + /// for a usable baseline, else `.dayRelative` (the unchanged default). The only behavioural change + /// vs. before is that a user with a few worn days now scores today's hours against their own + /// cross-day floor instead of the day's own calm hours; a cold-start / sparse-history user is + /// byte-identical to before (`DaytimeStress.scoringMode` is the single degradation gate). + /// + /// PERF: reads each past day's raw HR (and, only when that day was worn, R-R) once, bounded per day, + /// off the main actor via `repo` — riding the same async `load()` the today-timeline already runs on. + /// Unworn days are skipped without an R-R read. The `DaytimeStress` analyze memo is untouched; the + /// fold itself is O(days). + private func daytimeScoringMode(startOfToday: Date) async -> DaytimeStress.ScoringMode { + let cal = Calendar.current + var days: [DaytimeStress.DaytimeDayStreams] = [] + days.reserveCapacity(Self.baselineHistoryDays) + // Oldest → newest so the EWMA fold replays the history in order. + for back in stride(from: Self.baselineHistoryDays, through: 1, by: -1) { + guard let dayStart = cal.date(byAdding: .day, value: -back, to: startOfToday), + let dayEnd = cal.date(byAdding: .day, value: 1, to: dayStart) else { continue } + let from = Int(dayStart.timeIntervalSince1970) + let to = Int(dayEnd.timeIntervalSince1970) - 1 + let dayTz = TimeZone.current.secondsFromGMT(for: dayStart) + let dayHR = await repo.hrSamples(from: from, to: to, limit: 200_000) + guard !dayHR.isEmpty else { continue } // unworn day — no floor to learn, skip the R-R read + let dayRR = (try? await repo.storeHandle()?.rrIntervals( + deviceId: repo.deviceId, from: from, to: to, limit: 200_000)) ?? [] + days.append(.init(hr: dayHR, rr: dayRR, tzOffsetSeconds: dayTz)) + } + return DaytimeStress.scoringMode(history: days) + } + /// Recompute the cached `StressModel` only when (repo.days, storedSeries) /// actually changed since the last build. Equality is an O(n) value compare, /// far cheaper than the model rebuild it guards. @@ -233,7 +281,7 @@ struct StressView: View { // bar split by how many waking hours sat in each band, with durations. StressTotalsBar(totals: StressTotals(hours: day.hours)) - Text("The line is each waking hour's 0-3 proxy, scored against your own calm hours today. The bar below splits your day into calm, moderate and high stress time.") + Text(daytimeTimelineCaption) .font(StrandFont.footnote) .foregroundStyle(StrandPalette.textTertiary) .fixedSize(horizontal: false, vertical: true) @@ -252,6 +300,16 @@ struct StressView: View { return String(localized: "avg \(String(format: "%.1f", mean)) · \(n)h") } + /// The timeline's explanatory line, honest about WHICH reference each hour was scored against — + /// the personal cross-day baseline (`.baselineRelative`) or the day's own calm hours (`.dayRelative`). + /// Explicit `LocalizedStringKey` so BOTH variants stay in the string catalog (a ternary inside + /// `Text(_:)` would resolve to the verbatim, non-localized `String` overload). + private var daytimeTimelineCaption: LocalizedStringKey { + daytimeUsesPersonalBaseline + ? "The line is each waking hour's 0-3 proxy, scored against your personal daytime baseline (how your own days usually run). The bar below splits your day into calm, moderate and high stress time." + : "The line is each waking hour's 0-3 proxy, scored against your own calm hours today. The bar below splits your day into calm, moderate and high stress time." + } + /// A passive, in-app nudge to run a Breathe session after a sustained high-stress run. /// No notification — just a card with a CTA that opens the existing trainer. private func sustainedBreatheCard(_ day: DaytimeStress.Result) -> some View { From 3e0280acc65908ada62c7cd52e896a3cfb79e373 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 12:27:33 -0700 Subject: [PATCH 096/171] chore(stress): gate daytime-RMSSD out of live scoring pending validation Real 1Hz wrist data shows daytime RMSSD is artifact-dominated (hourly values swing ~40->430 ms as posture/motion break the R-R stream), an order of magnitude noisier than overnight recumbent HRV. rawScore sums the HRV z equal-weight with the HR z, so an artifact hour can move the combined score by the full band on noise alone. The r~0.6 baseline-relative margin was validated HR-only; adding an unvalidated noisy term risks pushing the score below that ceiling. Hold RMSSD out of the LIVE score behind daytimeRMSSDScoringEnabled (false). The whole RMSSD half -- aggregate, fold, daytime_rmssd config, rawScore term -- stays built and unit-tested; only its arrival at production scoring is gated. Flipping the flag is one line with a tripwire test that inverts with it. --- .../StrandAnalytics/DaytimeBaselines.swift | 6 ++- .../StrandAnalytics/DaytimeStress.swift | 16 ++++++++ .../DaytimeBaselinesTests.swift | 37 ++++++++++++++++--- 3 files changed, 53 insertions(+), 6 deletions(-) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeBaselines.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeBaselines.swift index 1ebf8da771..4e8942fe12 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeBaselines.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeBaselines.swift @@ -140,6 +140,10 @@ public extension DaytimeStress { static func scoringMode(history days: [DaytimeDayStreams]) -> ScoringMode { let baselines = foldDaytimeBaselines(days: days) guard baselines.hr.usable else { return .dayRelative } - return .baselineRelative(hr: baselines.hr, rmssd: baselines.rmssd) + // RMSSD stays out of the LIVE score until it has its own validation pass — see + // `DaytimeStress.daytimeRMSSDScoringEnabled`. The baseline is still folded above (so the + // machinery + tests exercise the real path); it just doesn't reach scoring while gated off. + let rmssd = DaytimeStress.daytimeRMSSDScoringEnabled ? baselines.rmssd : nil + return .baselineRelative(hr: baselines.hr, rmssd: rmssd) } } diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift index fe20bb1d90..e5c04e95a5 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift @@ -54,6 +54,22 @@ public enum DaytimeStress { /// `marginToSigma` for how it's translated onto the shared 0–3 squash curve. public static let baselineRelativeHighMarginBPM: Double = 15.0 + /// Gate for whether the personal daytime-RMSSD baseline feeds the live 0–3 score. `false`: + /// `.baselineRelative` scores HR-only, exactly the channel the r≈0.6 margin above was validated + /// on. The RMSSD half of the pipeline — `dayDaytimeAggregate`, `foldDaytimeBaselines`, the + /// `daytime_rmssd` config, and `rawScore`'s HRV term — is built and unit-tested, but stays OUT + /// of the live score until it has its OWN Oura-reference validation pass. + /// + /// WHY OFF (validated against real WHOOP data, 2026-07): daytime RMSSD off the wrist is + /// artifact-dominated — hourly values swing ~40→430 ms as posture / motion / talking break the + /// R-R stream, an order of magnitude noisier than the overnight recumbent HRV the nightly + /// baselines use. `rawScore` sums the HRV z EQUAL-WEIGHT with the HR z, so an artifact hour can + /// swing the combined score by ±3 (the full band) on noise alone. Enabling it before it is shown + /// to IMPROVE the correlation risks pushing the combined score BELOW the HR-only r≈0.6 ceiling — + /// the exact regression `baselineRelativeHighMarginBPM`'s comment warns against. Flip to `true` + /// only once daytime HR+RMSSD is validated to beat HR-only on an Oura-style stress reference. + public static let daytimeRMSSDScoringEnabled: Bool = false + // MARK: - Scoring mode /// WHERE each hour's "calm" reference point + spread come from. Every other step — diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeBaselinesTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeBaselinesTests.swift index 3d7cd51cd8..f89bcb8834 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeBaselinesTests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeBaselinesTests.swift @@ -187,11 +187,14 @@ final class DaytimeBaselinesTests: XCTestCase { DaytimeStress.DaytimeDayStreams(hr: flatDayHR($0, bpm: 65), rr: flatDayRR($0, rrMs: 900, jitter: 40), tzOffsetSeconds: 0) } - let mode = DaytimeStress.scoringMode(history: history) - guard case let .baselineRelative(_, rmssd) = mode else { - return XCTFail("expected baseline-relative with an RMSSD baseline") - } - XCTAssertNotNil(rmssd, "20 days of daytime R-R should yield a usable RMSSD baseline") + // Build the RMSSD-active mode DIRECTLY from the fold, bypassing the `scoringMode` gate that + // holds RMSSD out of LIVE scoring until validated (`DaytimeStress.daytimeRMSSDScoringEnabled` + // is false). This test pins the anti-stacking guarantee of the RMSSD-active scoring path + // itself, so it must exercise that path regardless of the production gate. `scoringMode`'s + // own gating is covered by `testScoringModeHoldsRMSSDOutOfLiveScoreWhileGated` below. + let baselines = DaytimeStress.foldDaytimeBaselines(days: history) + XCTAssertNotNil(baselines.rmssd, "20 days of daytime R-R should yield a usable RMSSD baseline") + let mode = DaytimeStress.ScoringMode.baselineRelative(hr: baselines.hr, rmssd: baselines.rmssd) // Today: HR at the floor, HRV at the SAME typical variability as history. let today = flatDayHR(0, bpm: 65) @@ -204,4 +207,28 @@ final class DaytimeBaselinesTests: XCTestCase { "a normal hour with an RMSSD baseline must not stack into the HIGH band") } } + + func testScoringModeHoldsRMSSDOutOfLiveScoreWhileGated() { + // The live-scoring gate: even with a full daytime-RR history that folds a USABLE RMSSD + // baseline, `scoringMode` (the production entry point) keeps RMSSD OUT of the mode while + // `DaytimeStress.daytimeRMSSDScoringEnabled` is false — so production scores HR-only, the + // validated r≈0.6 channel. The fold still produces the usable baseline (the machinery is + // live + tested); only its arrival at the score is gated. Flip the flag → this expectation + // inverts, which is the intended tripwire for when RMSSD is validated on. + let history = (0..<20).map { + DaytimeStress.DaytimeDayStreams(hr: flatDayHR($0, bpm: 65), + rr: flatDayRR($0, rrMs: 900, jitter: 40), tzOffsetSeconds: 0) + } + XCTAssertNotNil(DaytimeStress.foldDaytimeBaselines(days: history).rmssd, + "precondition: the history must fold a usable RMSSD baseline") + guard case let .baselineRelative(hr, rmssd) = DaytimeStress.scoringMode(history: history) else { + return XCTFail("usable HR history must still select .baselineRelative") + } + XCTAssertTrue(hr.usable, "HR baseline still drives the mode") + if DaytimeStress.daytimeRMSSDScoringEnabled { + XCTAssertNotNil(rmssd, "flag on: the usable RMSSD baseline should reach live scoring") + } else { + XCTAssertNil(rmssd, "flag off: RMSSD is held out of the live score despite a usable baseline") + } + } } From cce804b13222e9a701b0cedc87e07517a6694206 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 12:35:34 -0700 Subject: [PATCH 097/171] feat(sleep): recency-weighted, capped, asymmetric sleep-debt ledger The debt ledger was a flat Sigma(slept - need) over 14 nights: every night weighted equally, an oversleep fully cancelling a short night, and no bound, so a chronic under-sleeper's headline compounded toward an absurd figure. Replace the accumulation with a principled recurrence (raw per-night deltas kept verbatim for the bars; only the running balance changes): contribution = delta<0 ? delta : delta * surplusRepayFraction // accrue full, repay partial balance = recencyRetention * balance + contribution // geometric recency decay balance = clamp(balance, -cap, +cap) // cap = maxBalanceNights * need Defaults: retention 0.90 (~6.6-night half-life), surplusRepayFraction 0.5, maxBalanceNights 3 (person-scaled). All flagged as wanting empirical tuning. Also unify both debt surfaces (ledger card + per-night tile) onto ONE need: the population-anchored personalizedNeedHours, not the self-referential mean-sleep sleepNeedMin (which drifts toward a chronic under-sleeper's own deficit and quietly erases their debt). Validated on ~2 weeks of real wrist data (avg 5.15h/night vs an 8h floored need): new headline -21.4h vs the old flat -37.1h (42% smaller); the need correctly floors at 8h rather than collapsing to the 5.9h personal mean. New tests pin recency, cap, asymmetry, window, and rounding; full StrandAnalytics suite green. --- .../Sources/StrandAnalytics/SleepDebt.swift | 126 +++++++++--- .../StrandAnalyticsTests/SleepDebtTests.swift | 179 +++++++++++++----- Strand/Screens/SleepView.swift | 34 +++- 3 files changed, 258 insertions(+), 81 deletions(-) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepDebt.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepDebt.swift index 61e8f52af1..b8bf8dde52 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepDebt.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepDebt.swift @@ -1,24 +1,56 @@ import Foundation -// SleepDebt.swift — a rolling sleep-debt ledger over the last N nights. +// SleepDebt.swift — a recency-weighted, capped, asymmetric sleep-debt ledger. // // Pure, deterministic, DB-free. Given a chronological series of per-night total -// sleep (minutes) and a personal sleep need (hours), it accumulates a running -// balance of (actual − need) per night across a capped trailing window (14 nights -// by default) and reports the net balance plus the per-night deltas that make it -// up. +// sleep (minutes) and a personal sleep need (hours), it walks the recent nights +// oldest → newest and folds each night's (slept − need) into a running balance +// with three principled properties — recency weighting, a cap, and accrual/repay +// asymmetry — then reports the net balance plus the raw per-night deltas. +// +// The recurrence (oldest → newest over the counted, windowed nights, seeded at 0): +// +// contribution = delta < 0 ? delta // a DEFICIT accrues in FULL +// : delta * surplusRepayFraction // a SURPLUS repays only partly +// balance = recencyRetention * balance + contribution // yesterday's balance DECAYS +// balance = clamp(balance, −cap, +cap) // cap = maxBalanceNights * need +// +// where delta = sleptMin − needMin (positive = surplus, negative = deficit). +// +// Why each term (vs the old flat Σ(slept − need)): +// - RECENCY WEIGHTING. `recencyRetention` (< 1) multiplies the carried balance each +// night, so a night's imprint decays geometrically: its weight `k` counted nights +// later is retentionᵏ. A fortnight-old deficit no longer sits at full weight the way +// a flat sum held it. At the 0.90 default the half-life is ln0.5/ln0.90 ≈ 6.6 nights, +// so across the 14-night window the oldest night carries ≈ 0.90¹³ ≈ 0.25 weight. +// - CAP. Even decayed, a sustained deficit converges toward need/(1−retention) ≈ 10 +// nights' worth — still an absurd headline. `maxBalanceNights` clamps |balance| to a +// few nights of need (a person-scaled ceiling), applied every step so the running +// value never transiently blows past it either. The cap only binds under severe +// chronic deprivation; realistic mild deficits sit well under it. +// - ASYMMETRY. Two mechanisms, both grounded in sleep homeostasis: (1) a deficit accrues +// in full but a surplus repays at `surplusRepayFraction` (< 1) — you cannot fully +// "catch up" by oversleeping one night; (2) the retention decay itself repays debt over +// time and prevents surplus from banking indefinitely. So an oversleep night moves the +// balance strictly less than an equal-magnitude short night does. // // HONEST by construction: -// - It is a plain debt accumulator — sum of nightly (slept − need) — NOT a -// physiological model. A surplus night (slept > need) genuinely offsets a -// deficit one, the same way a checking balance nets credits and debits. -// - The window is capped (default 14) so "debt" never compounds indefinitely -// across months of history — only the recent fortnight is in scope. -// - Nights with no usable sleep total are SKIPPED entirely (no zero-fill), so a -// gap in wear never reads as a full night of debt. -// - The need value is supplied by the caller (AnalyticsEngine.Rest.defaultNeedHours -// = 8.0 by default; the caller passes any personal override). Computation here -// stays a pure function of (series, need, window). +// - The per-night `deltaMin` stored on each `SleepDebtNight` stays the RAW signed +// (slept − need), so the UI's diverging per-night bars remain a faithful nightly read. +// Only the accumulated `balanceMin` applies the decay/asymmetry/cap — it is deliberately +// NOT the plain Σ of the deltas any more (the model is the point). +// - The trailing WINDOW (default 14) still caps how many counted nights are folded in; +// decay and window together bound history. +// - Nights with no usable sleep total are SKIPPED entirely (no zero-fill), so a gap in +// wear never reads as a full night of debt, and decay is applied per COUNTED night (a +// measurement gap does not silently "recover" the debt). +// - The need value is supplied by the caller (AnalyticsEngine.Rest.defaultNeedHours = 8.0 +// by default; the caller passes any personal override — see SleepView, which now feeds +// the population-anchored personalizedNeedHours rather than a self-referential mean). +// Computation here stays a pure function of (series, need, window). +// +// The three tuning constants (`recencyRetention`, `surplusRepayFraction`, `maxBalanceNights`) +// are first-cut, defensible defaults that WANT empirical tuning on real wearer histories. // // Constant-explicit + dependency-free so the Kotlin mirror (android … SleepDebt.kt) // is byte-identical. @@ -40,11 +72,15 @@ public struct SleepDebtNight: Equatable, Sendable { /// The rolling sleep-debt ledger over the capped trailing window. public struct SleepDebtLedger: Equatable, Sendable { - /// Net running balance (minutes) across the window: Σ(slept − need). Negative = - /// net DEBT (under-slept overall), positive = net SURPLUS, 0 = on target. + /// Net running balance (minutes): the recency-weighted, asymmetric, capped fold of the + /// per-night deltas (see `SleepDebt.ledger`). Negative = net DEBT (under-slept overall), + /// positive = net SURPLUS, 0 = on target. NOT the plain Σ of `nights[].deltaMin` — the + /// decay/asymmetry/cap are applied to the accumulation, the raw deltas are kept only for + /// the per-night bars. public let balanceMin: Double - /// Per-night contributions, oldest → newest, one per counted night (skipped - /// nights are absent). The `deltaMin` values are the per-night bar/spark. + /// Per-night contributions, oldest → newest, one per counted night (skipped nights are + /// absent). Each `deltaMin` is the RAW signed (slept − need) — the honest per-night + /// bar/spark — before the balance's decay/asymmetry/cap are applied. public let nights: [SleepDebtNight] /// Personal sleep need (minutes) the ledger was computed against (for labelling). public let needMin: Double @@ -64,13 +100,37 @@ public struct SleepDebtLedger: Equatable, Sendable { public enum SleepDebt { /// Cap the ledger at the trailing two weeks — recent enough to be actionable, - /// short enough that one rough patch doesn't read as months of compounding debt. + /// short enough that one rough patch doesn't read as months of compounding debt. With the + /// recency decay layered on top, the window is a hard outer bound; the decay is what + /// actually tapers a night's weight within it. public static let defaultWindowNights: Int = 14 /// "On target" deadband (minutes): a |balance| under this reads as balanced rather /// than as a debt/surplus, so a few stray minutes don't flip the headline. public static let onTargetBandMin: Double = 30.0 + /// RECENCY WEIGHTING. Per-night retention of the carried balance: each night the running + /// balance is multiplied by this before the night's contribution is added, so a night's + /// weight decays geometrically (retentionᵏ, k counted nights later). 0.90 ⇒ 10 %/night + /// decay, a ≈ 6.6-night half-life (ln0.5/ln0.90), and ≈ 0.25 weight on the oldest night of + /// a 14-night window. Lower = shorter memory (only the last few nights matter); higher = + /// longer memory (closer to the old flat sum). WANTS empirical tuning. Range (0, 1). + public static let recencyRetention: Double = 0.90 + + /// ASYMMETRY. A surplus (over-need) night repays debt at only this fraction of the weight a + /// deficit night accrues at — you can't fully "bank" catch-up sleep by oversleeping once. + /// 0.5 ⇒ an oversleep night moves the balance half as far as an equal short night. Combined + /// with `recencyRetention` (which also quietly repays debt over time), this is the ledger's + /// accrue-fast / repay-slow behaviour. WANTS empirical tuning. Range [0, 1]. + public static let surplusRepayFraction: Double = 0.5 + + /// CAP. |balance| is clamped to this many nights of the personal need (cap = value × needMin), + /// applied every step so the running total can't even transiently exceed it. Person-scaled so + /// a longer-sleeping teen's ceiling is proportionally higher. 3 ⇒ "at most ~three full nights + /// behind (or ahead)". Guards the absurd-number failure mode (a sustained deficit's decayed + /// series still converges toward ~10 nights' worth without it). WANTS empirical tuning. + public static let maxBalanceNights: Double = 3.0 + /// Build the ledger from a chronological `[(day, totalSleepMin?)]` series. /// /// - Parameters: @@ -84,27 +144,37 @@ public enum SleepDebt { /// - window: how many of the most-recent COUNTED nights to include. Defaults to /// `defaultWindowNights` (14). Clamped to ≥ 1. /// - /// The balance is Σ over the window of (sleptMin − needMin): a surplus night - /// offsets a deficit one. Returns an empty ledger (balance 0, no nights) when no - /// night has usable data. + /// The balance folds the windowed nights oldest → newest: each night's carried balance + /// DECAYS by `recencyRetention`, a DEFICIT adds in full while a SURPLUS adds only + /// `surplusRepayFraction` of itself, and |balance| is clamped to `maxBalanceNights` × need + /// every step (see the file header for the full rationale). Each `SleepDebtNight.deltaMin` + /// keeps the RAW (slept − need) for the per-night bars. Returns an empty ledger (balance 0, + /// no nights) when no night has usable data. public static func ledger(series: [(day: String, totalSleepMin: Double?)], needHours: Double = AnalyticsEngine.Rest.defaultNeedHours, window: Int = defaultWindowNights) -> SleepDebtLedger { let needMin = max(needHours, 0.0) * 60.0 - let cap = max(window, 1) + let windowCap = max(window, 1) + // Person-scaled magnitude ceiling for the running balance (a few nights of need). + let balanceCap = maxBalanceNights * needMin // Keep only nights with usable sleep, preserving chronological order, then take - // the most-recent `cap` of them. + // the most-recent `windowCap` of them. let usable = series.filter { ($0.totalSleepMin ?? 0) > 0 } - let windowed = usable.suffix(cap) + let windowed = usable.suffix(windowCap) var nights: [SleepDebtNight] = [] nights.reserveCapacity(windowed.count) var balance = 0.0 for row in windowed { let slept = row.totalSleepMin ?? 0 - let delta = slept - needMin - balance += delta + let delta = slept - needMin // RAW signed per-night; kept for the bars + // Asymmetry: a deficit (delta < 0) accrues in full; a surplus repays only partly. + let contribution = delta < 0 ? delta : delta * surplusRepayFraction + // Recency weighting: yesterday's balance decays before today's contribution lands. + balance = recencyRetention * balance + contribution + // Cap: clamp every step so the running total never even transiently exceeds it. + balance = min(max(balance, -balanceCap), balanceCap) nights.append(SleepDebtNight(day: row.day, sleptMin: slept, deltaMin: delta)) } return SleepDebtLedger(balanceMin: round1(balance), nights: nights, needMin: needMin) diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepDebtTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepDebtTests.swift index 960357c1ad..1228159829 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepDebtTests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepDebtTests.swift @@ -3,7 +3,10 @@ import XCTest final class SleepDebtTests: XCTestCase { - /// Three nights at need (8 h = 480 min) → zero balance, three counted nights. + // MARK: - Baseline shape (unchanged by the recency/cap/asymmetry model) + + /// Three nights exactly at need (8 h = 480 min) → zero balance, three counted nights. + /// Every delta is 0, so decay/asymmetry/cap never engage: the model still reads 0. func testOnTargetNetsToZero() { let series: [(day: String, totalSleepMin: Double?)] = [ ("2026-06-01", 480), ("2026-06-02", 480), ("2026-06-03", 480), @@ -15,22 +18,9 @@ final class SleepDebtTests: XCTestCase { XCTAssertEqual(l.needMin, 480.0, accuracy: 1e-9) } - /// A surplus night offsets a deficit one (ledger nets credits and debits). - func testSurplusOffsetsDeficit() { - let series: [(day: String, totalSleepMin: Double?)] = [ - ("2026-06-01", 360), // −120 - ("2026-06-02", 540), // +60 - ("2026-06-03", 420), // −60 - ] - // need 8 h = 480. Σ = −120 + 60 − 60 = −120. - let l = SleepDebt.ledger(series: series, needHours: 8.0) - XCTAssertEqual(l.balanceMin, -120.0, accuracy: 1e-9) - XCTAssertTrue(l.isDebt) - XCTAssertEqual(l.magnitudeMin, 120.0, accuracy: 1e-9) - XCTAssertEqual(l.nights.map { $0.deltaMin }, [-120, 60, -60]) - } - - /// Nights with no usable sleep are skipped entirely (never zero-filled as debt). + /// Nights with no usable sleep are skipped entirely (never zero-filled as debt). Two + /// counted nights [480 (δ0), 420 (δ−60)]: the at-need night contributes 0, the deficit + /// accrues fully with nothing before it to decay → balance −60. func testSkipsNoDataNights() { let series: [(day: String, totalSleepMin: Double?)] = [ ("2026-06-01", 480), @@ -42,19 +32,8 @@ final class SleepDebtTests: XCTestCase { XCTAssertEqual(l.nightCount, 2) XCTAssertEqual(l.balanceMin, -60.0, accuracy: 1e-9) XCTAssertEqual(l.nights.map { $0.day }, ["2026-06-01", "2026-06-04"]) - } - - /// Only the most-recent `window` COUNTED nights are in scope. - func testWindowCapKeepsMostRecent() { - // 16 nights, each 60 min UNDER need → each delta −60. - let series: [(day: String, totalSleepMin: Double?)] = (1...16).map { - (String(format: "2026-06-%02d", $0), Double(420)) - } - let l = SleepDebt.ledger(series: series, needHours: 8.0, window: 14) - XCTAssertEqual(l.nightCount, 14) // capped - XCTAssertEqual(l.balanceMin, -840.0, accuracy: 1e-9) // 14 × −60 - XCTAssertEqual(l.nights.first?.day, "2026-06-03") // oldest kept - XCTAssertEqual(l.nights.last?.day, "2026-06-16") // newest kept + // The per-night deltas stay RAW (slept − need), so the diverging bars remain honest. + XCTAssertEqual(l.nights.map { $0.deltaMin }, [0, -60]) } /// Empty / all-skipped input → empty ledger, zero balance. @@ -68,31 +47,141 @@ final class SleepDebtTests: XCTestCase { XCTAssertEqual(SleepDebt.ledger(series: allNil).nightCount, 0) } - /// The default need is AnalyticsEngine.Rest.defaultNeedHours (8 h). + /// The default need is AnalyticsEngine.Rest.defaultNeedHours (8 h). One deficit night + /// (420, δ−60) accrues fully → −60. func testDefaultNeedIsEightHours() { let l = SleepDebt.ledger(series: [("2026-06-01", 420)]) XCTAssertEqual(l.needMin, AnalyticsEngine.Rest.defaultNeedHours * 60.0, accuracy: 1e-9) XCTAssertEqual(l.balanceMin, -60.0, accuracy: 1e-9) } - /// A NEGATIVE EXACT half-tie balance rounds AWAY from zero (−0.05 → −0.1), the documented - /// `round1` contract the Kotlin mirror must match (audit #6). needMin = 0.1, slept 0.05 → - /// delta −0.05 exactly → balance −0.1. Kotlin's old `roundToInt()` (half toward +∞) gave - /// 0.0 on this exact tie — the real divergence this pins shut. - func testNegativeHalfTieRoundsAwayFromZero() { - let l = SleepDebt.ledger(series: [("2026-06-01", 0.05)], needHours: 0.1 / 60.0) - XCTAssertEqual(l.balanceMin, -0.1, accuracy: 1e-9) // away-from-zero, not 0.0 + /// Only the most-recent `window` COUNTED nights are in scope, and the per-night deltas + /// stay raw. 16 uniform-deficit nights, window 14: the oldest two drop, 14 remain. + func testWindowCapKeepsMostRecentNights() { + let series: [(day: String, totalSleepMin: Double?)] = (1...16).map { + (String(format: "2026-06-%02d", $0), Double(420)) // each δ −60 + } + let l = SleepDebt.ledger(series: series, needHours: 8.0, window: 14) + XCTAssertEqual(l.nightCount, 14) // window still caps the NIGHT count + XCTAssertEqual(l.nights.first?.day, "2026-06-03") // oldest kept + XCTAssertEqual(l.nights.last?.day, "2026-06-16") // newest kept + // Balance is the decayed accumulation Σ −60·0.9^k, k=0…13 = −600·(1−0.9^14) ≈ −462.7, + // NOT the flat 14×−60 = −840 the old symmetric sum reported (well under the cap). + XCTAssertEqual(l.balanceMin, -462.7, accuracy: 0.05) + XCTAssertGreaterThan(l.balanceMin, -840.0) // strictly gentler than a flat sum + } + + // MARK: - Recency weighting (recent deficits weigh more than old ones) + + /// The SAME single deficit night weighs more when it is RECENT than when it is OLD: a + /// deficit on the newest night sits at full weight, one four at-need nights ago has + /// decayed. Pins the EWMA recency-weighting. + func testRecencyWeightsRecentDeficitMore() { + let atNeed = 480.0, deficit = 360.0 // δ 0 and δ −120 + let recent: [(day: String, totalSleepMin: Double?)] = [ + ("2026-06-01", atNeed), ("2026-06-02", atNeed), ("2026-06-03", atNeed), + ("2026-06-04", atNeed), ("2026-06-05", deficit), // deficit NEWEST + ] + let old: [(day: String, totalSleepMin: Double?)] = [ + ("2026-06-01", deficit), ("2026-06-02", atNeed), ("2026-06-03", atNeed), + ("2026-06-04", atNeed), ("2026-06-05", atNeed), // deficit OLDEST + ] + let lRecent = SleepDebt.ledger(series: recent, needHours: 8.0) + let lOld = SleepDebt.ledger(series: old, needHours: 8.0) + XCTAssertEqual(lRecent.balanceMin, -120.0, accuracy: 1e-6) // full weight + XCTAssertEqual(lOld.balanceMin, -78.7, accuracy: 0.05) // −120·0.9^4, decayed + XCTAssertGreaterThan(lRecent.magnitudeMin, lOld.magnitudeMin) + XCTAssertTrue(lRecent.isDebt && lOld.isDebt) } - /// The symmetric POSITIVE exact half-tie (+0.05 → +0.1), pinned so the sign-aware Kotlin - /// rounding can't silently regress one direction. needMin = 0, slept 0.05 → delta +0.05. - func testPositiveHalfTieRoundsAwayFromZero() { - let l = SleepDebt.ledger(series: [("2026-06-01", 0.05)], needHours: 0.0) - XCTAssertEqual(l.balanceMin, 0.1, accuracy: 1e-9) + /// A single deficit fades as at-need nights pass — the debt decays over time even without + /// any surplus to repay it (the "debt is not a permanent scar" asymmetry). + func testDeficitDecaysAsAtNeedNightsPass() { + let fresh = SleepDebt.ledger(series: [("2026-06-01", 360)], needHours: 8.0) + let aged = SleepDebt.ledger(series: [ + ("2026-06-01", 360), ("2026-06-02", 480), ("2026-06-03", 480), + ("2026-06-04", 480), ("2026-06-05", 480), + ], needHours: 8.0) + XCTAssertEqual(fresh.balanceMin, -120.0, accuracy: 1e-6) + XCTAssertLessThan(aged.magnitudeMin, fresh.magnitudeMin) // decayed toward zero + XCTAssertTrue(aged.isDebt) // but not yet cleared + } + + // MARK: - Cap (debt can't grow to an absurd number) + + /// Sustained heavy deficits are bounded at `maxBalanceNights` × need — without the cap the + /// decayed geometric series would still pile to ~10 nights' worth. 14 nights of a −420 + /// deficit clamp to exactly −1440 (3 × 480), not the ≈ −4000 the unclamped series implies. + func testDebtIsCappedAtMaxNights() { + let series: [(day: String, totalSleepMin: Double?)] = (1...14).map { + (String(format: "2026-06-%02d", $0), Double(60)) // slept 1 h, δ −420 + } + let l = SleepDebt.ledger(series: series, needHours: 8.0) + let cap = SleepDebt.maxBalanceNights * l.needMin // 3 × 480 = 1440 + XCTAssertEqual(l.balanceMin, -cap, accuracy: 1e-6) // clamped to the floor + XCTAssertLessThanOrEqual(l.magnitudeMin, cap + 1e-6) // never exceeds the cap + XCTAssertEqual(cap, 1440.0, accuracy: 1e-9) + } + + /// A surplus is capped symmetrically at +cap so a long banked-sleep run can't read as an + /// absurd credit either. + func testSurplusIsCappedSymmetrically() { + let series: [(day: String, totalSleepMin: Double?)] = (1...14).map { + (String(format: "2026-06-%02d", $0), Double(900)) // slept 15 h, δ +420 + } + let l = SleepDebt.ledger(series: series, needHours: 8.0) + let cap = SleepDebt.maxBalanceNights * l.needMin + XCTAssertLessThanOrEqual(l.magnitudeMin, cap + 1e-6) + XCTAssertFalse(l.isDebt) + } + + // MARK: - Asymmetry (a repay night moves the balance less than an equal deficit night) + + /// Accrual and repayment are NOT symmetric: an X-minute deficit night accrues the full X, + /// but an X-minute surplus night repays only `surplusRepayFraction`·X. Pins the core + /// asymmetry the ticket asks for. + func testSurplusRepaysLessThanDeficitAccrues() { + let deficit = SleepDebt.ledger(series: [("2026-06-01", 360)], needHours: 8.0) // δ −120 + let surplus = SleepDebt.ledger(series: [("2026-06-01", 600)], needHours: 8.0) // δ +120 + XCTAssertEqual(deficit.balanceMin, -120.0, accuracy: 1e-6) // full accrual + XCTAssertEqual(surplus.balanceMin, 60.0, accuracy: 1e-6) // half repay (0.5·120) + XCTAssertGreaterThan(deficit.magnitudeMin, surplus.magnitudeMin) // asymmetric + XCTAssertEqual(surplus.magnitudeMin, + SleepDebt.surplusRepayFraction * deficit.magnitudeMin, accuracy: 1e-6) + XCTAssertTrue(deficit.isDebt) + XCTAssertFalse(surplus.isDebt) + // The RAW per-night deltas are still symmetric (±120) — only the accumulation is asymmetric. + XCTAssertEqual(deficit.nights.first?.deltaMin, -120.0) + XCTAssertEqual(surplus.nights.first?.deltaMin, 120.0) + } + + /// A surplus night dropped onto an existing debt repays less than a fresh deficit of equal + /// size would deepen it — the asymmetry holds mid-ledger, not just from zero. + func testSurplusOffsetsDeficitButNotFully() { + // [360, 540, 420] need 8 h → deltas −120, +60, −60. + // n1: 0·0.9 + (−120) = −120 + // n2: −120·0.9 + (0.5·60) = −108 + 30 = −78 + // n3: −78·0.9 + (−60) = −70.2 − 60 = −130.2 + let l = SleepDebt.ledger(series: [ + ("2026-06-01", 360), ("2026-06-02", 540), ("2026-06-03", 420), + ], needHours: 8.0) + XCTAssertEqual(l.balanceMin, -130.2, accuracy: 1e-6) + XCTAssertTrue(l.isDebt) + XCTAssertEqual(l.nights.map { $0.deltaMin }, [-120, 60, -60]) // raw deltas intact + } + + // MARK: - Rounding parity (Kotlin mirror must match byte-for-byte) + + /// A deficit lands its raw delta on the ledger (no repay-halving), so a −0.05 tie rounds + /// AWAY from zero to −0.1 through the full model path (need 8 h keeps the cap slack). + func testNegativeHalfTieRoundsAwayFromZero() { + let l = SleepDebt.ledger(series: [("2026-06-01", 479.95)], needHours: 8.0) // δ ≈ −0.05 + XCTAssertEqual(l.balanceMin, -0.1, accuracy: 1e-9) } - /// round1 is exercised on the directly-constructed value too, so the rounding mode is - /// pinned independent of the (slept − need) arithmetic path, both signs + a non-tie. + /// `round1` itself rounds half AWAY from zero for BOTH signs — the load-bearing contract the + /// Kotlin mirror's sign-aware `round1` must match (Swift `.rounded()` is half-away-from-zero; + /// Kotlin `roundToInt()` is half-toward-+∞, which diverged on negative ties, audit #6). func testRound1HalfTiesAwayFromZero() { XCTAssertEqual(SleepDebt.round1(-0.05), -0.1, accuracy: 1e-9) XCTAssertEqual(SleepDebt.round1(0.05), 0.1, accuracy: 1e-9) diff --git a/Strand/Screens/SleepView.swift b/Strand/Screens/SleepView.swift index ea0d4af736..21ccb57ba0 100644 --- a/Strand/Screens/SleepView.swift +++ b/Strand/Screens/SleepView.swift @@ -1765,14 +1765,29 @@ struct SleepView: View { sleepDebtLedger: debtLedger) } - /// The rolling 14-night sleep-debt ledger from the cached daily metrics. Uses the - /// SAME personal sleep need the tiles use (`sleepNeedMin`, ≥ 7.5 h, the per-user - /// override over the 8 h default), measured against each night's `totalSleepMin`. - /// Skips nights with no sleep (the analytics function does the skip). (#242) + /// The defensible per-user sleep NEED (hours) BOTH debt surfaces (this ledger card and the + /// "Sleep Debt" tile) measure against — the population-anchored, age-floored, upper-quartile + /// `personalizedNeedHours`, the SAME estimator Intelligence uses. Deliberately NOT the + /// self-referential `sleepNeedMin` (mean total sleep): the mean drifts DOWN toward a chronic + /// under-sleeper's own deficit and quietly erases their debt, whereas the upper-quartile + /// estimate floored at the ~8 h adult target only adjusts UP for genuine long sleepers. Age + /// isn't plumbed to this screen, so it floors at the adult target (age: nil) — wiring the + /// profile age through would only raise it for under-18s. One need, both debt surfaces. (#242) + private var personalSleepNeedHours: Double { + AnalyticsEngine.Rest.personalizedNeedHours( + nightlyHours: repo.days.compactMap { $0.totalSleepMin.map { $0 / 60.0 } }, + age: nil) + } + + /// The rolling sleep-debt ledger from the cached daily metrics, measured against the + /// defensible `personalSleepNeedHours`. The recency-weighted, capped, asymmetric running + /// balance (recent nights weigh more, |balance| bounded, oversleep repays less than a short + /// night accrues) lives in `SleepDebt.ledger`; here we only supply the series + need. Skips + /// nights with no sleep (the analytics function does the skip). (#242) private var debtLedger: SleepDebtLedger { SleepDebt.ledger( series: repo.days.map { (day: $0.day, totalSleepMin: $0.totalSleepMin) }, - needHours: sleepNeedMin / 60.0) + needHours: personalSleepNeedHours) } // MARK: - Derived model @@ -2232,11 +2247,14 @@ struct SleepView: View { metric { $0.respRateBpm } } - /// Sleep debt (minutes): the imported sleep_debt_min when the export carried it; else - /// the APPROXIMATE per-night need − asleep, floored at 0 (no "credit"). + /// Sleep debt (minutes): the imported sleep_debt_min when the export carried it; else the + /// APPROXIMATE per-night deficit (need − asleep), floored at 0 (no "credit"). Measured against + /// the SAME defensible `personalSleepNeedHours` the ledger card uses, so this per-night tile + /// and the running-balance card agree on what "need" means — one need across both debt + /// surfaces, rather than the self-referential `sleepNeedMin` mean this used to use. (#242) private var sleepDebtSeries: Metric { let imported = repo.importedSleep - let need = sleepNeedMin + let need = personalSleepNeedHours * 60.0 let series = repo.days.compactMap { d -> Double? in if let debt = imported[d.day]?.debtMin { return debt } // minutes, export-verbatim guard let asleep = d.totalSleepMin, asleep > 0, need > 0 else { return nil } From 2bbac2269a77ef835dd6d7d88398068f2c389986 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 12:46:33 -0700 Subject: [PATCH 098/171] Fix Sleep tab bedtime skipping a real first-sleep fragment on computed nights The displayed-onset stub test (nightOnsetTs / isPreOnsetAwakeStub) read asleep minutes via the dict-only decodeStages, which returns nil for the segment-array stagesJSON an on-device computed night stores - so every fragment counted as 0 asleep minutes, a real ~54-min first sleep (12:16 -> 1:22) tripped the "essentially sleepless stub" branch, and the shown bedtime jumped to the 1:29 main block. The #259 real-sleep-episode floor (preOnsetStubMinorAsleepFloorMin), added for this exact 2026-07-14 night, never engaged because its input was zeroed before the comparison. Add SleepView.decodedAsleepMinutes, a format-agnostic seam (dict-of-minutes decode with segment-array fallback, threading the #259 pre-onset trim), and use it at both onset call sites. decodeStages/decodeSegments become static (they were already pure). New SleepOnsetDecodeTests pins the decode path itself with the night's real stored JSON: the segment array decodes to ~53.9 asleep minutes (above the floor) and the two-block night walks to onset index 0, the 12:16 fragment. Verified: full macOS StrandTests suite green (891 passed), and the sim build over the real device DB now renders "ASLEEP 12:16 AM / WOKE 7:32 AM" on the Sleep tab hero (previously 1:29 AM). Kotlin twin deferred: Android's SleepScreen onset stub-test caller reads asleep minutes the same dict-only way and needs the same both-format decode; policy transcription only, tracked for a follow-up. --- Strand/Screens/SleepView.swift | 34 +++++++-- StrandTests/SleepOnsetDecodeTests.swift | 91 +++++++++++++++++++++++++ 2 files changed, 119 insertions(+), 6 deletions(-) create mode 100644 StrandTests/SleepOnsetDecodeTests.swift diff --git a/Strand/Screens/SleepView.swift b/Strand/Screens/SleepView.swift index 21ccb57ba0..a01e6f2189 100644 --- a/Strand/Screens/SleepView.swift +++ b/Strand/Screens/SleepView.swift @@ -1934,7 +1934,9 @@ struct SleepView: View { // #259: reference size for the "minor relative to the main block" test = the group's largest asleep // span (≈ the main block). A genuine biphasic first sleep is comparable and is kept; a small stray // lead is skipped, so the onset no longer jumps hours early. - let refAsleepMin = group.map { decodeStages($0.stagesJSON)?.asleep ?? 0 }.max() ?? 0 + let refAsleepMin = group.map { + SleepView.decodedAsleepMinutes($0.stagesJSON, effectiveStartTs: $0.effectiveStartTs) + }.max() ?? 0 // Walk past any leading spurious pre-onset awake stubs to the first real-sleep fragment. for frag in group { if !isPreOnsetAwakeStub(frag, refAsleepMin: refAsleepMin) { return frag.effectiveStartTs } @@ -1949,7 +1951,8 @@ struct SleepView: View { /// a stub when it leads the main-night group, so the displayed bedtime tracks where real sleep began. (#736) private func isPreOnsetAwakeStub(_ frag: CachedSleepSession, refAsleepMin: Double = 0) -> Bool { let spanMin = Double(frag.endTs - frag.effectiveStartTs) / 60.0 - let asleepMin = decodeStages(frag.stagesJSON)?.asleep ?? 0 + let asleepMin = SleepView.decodedAsleepMinutes(frag.stagesJSON, + effectiveStartTs: frag.effectiveStartTs) return SleepView.isPreOnsetAwakeStub(spanMin: spanMin, asleepMin: asleepMin, refAsleepMin: refAsleepMin) } @@ -2026,7 +2029,7 @@ struct SleepView: View { // contributes nothing; if NO fragment has one, `motionEpochs` stays empty → honest empty state. var motion: [Double] = [] for frag in group { - if let seg = decodeSegments(frag.stagesJSON, sessionStart: frag.effectiveStartTs), seg.stages.total > 0 { + if let seg = Self.decodeSegments(frag.stagesJSON, sessionStart: frag.effectiveStartTs), seg.stages.total > 0 { stages.awake += seg.stages.awake; stages.light += seg.stages.light stages.deep += seg.stages.deep; stages.rem += seg.stages.rem // decodeSegments yields intervals relative to THAT fragment's onset; rebase them to @@ -2038,7 +2041,7 @@ struct SleepView: View { for iv in seg.intervals { segs.append(SleepInterval(stage: iv.stage, start: iv.start + shift, end: iv.end + shift)) } - } else if let st = decodeStages(frag.stagesJSON), st.total > 0 { + } else if let st = Self.decodeStages(frag.stagesJSON), st.total > 0 { stages.awake += st.awake; stages.light += st.light stages.deep += st.deep; stages.rem += st.rem } @@ -2455,8 +2458,27 @@ struct SleepView: View { // MARK: - Stage decoding + /// Asleep minutes decoded from a stored `stagesJSON` in EITHER of the two formats that exist in the + /// DB: on-device COMPUTED nights store a SEGMENT ARRAY `[{"start":epoch,"end":epoch,"stage":…}]` + /// (`AnalyticsEngine.encodeStages`); imported nights store a dict of MINUTES + /// `{"light","deep","rem","awake"}`. The displayed-onset stub test (`nightOnsetTs` / + /// `isPreOnsetAwakeStub`) MUST read asleep minutes format-agnostically: it previously used the + /// dict-only `decodeStages`, which returns nil for a computed night's segment array, so every + /// fragment of an on-device night read as 0 asleep minutes — a real ~54-min first sleep tripped the + /// "essentially sleepless stub" branch and the shown bedtime jumped from the true 12:16 onset to the + /// 1:29 main block, bypassing the #259 real-sleep-episode floor entirely (the 2026-07-14 night). + /// `effectiveStartTs` threads the fragment's effective onset into the segment decode's #259 + /// pre-onset trim. Internal (not private) so the golden test pins the DECODE PATH itself, not a + /// pre-computed minute count. Android twin: SleepScreen's onset stub-test caller needs the same + /// both-format decode. + static func decodedAsleepMinutes(_ json: String?, effectiveStartTs: Int) -> Double { + decodeStages(json)?.asleep + ?? decodeSegments(json, sessionStart: effectiveStartTs)?.stages.asleep + ?? 0 + } + /// Decode the imported stagesJSON dict of MINUTES {"light","deep","rem","awake"}. - private func decodeStages(_ json: String?) -> Stages? { + private static func decodeStages(_ json: String?) -> Stages? { guard let json, let data = json.data(using: .utf8) else { return nil } guard let obj = try? JSONSerialization.jsonObject(with: data), let dict = obj as? [String: Any] else { return nil } @@ -2474,7 +2496,7 @@ struct SleepView: View { /// Decode the COMPUTED stagesJSON segment array [{"start":epoch,"end":epoch,"stage":"wake"| /// "light"|"deep"|"rem"}] into stage totals plus the real timeline (seconds relative to the /// session start, the Hypnogram's domain). The on-device SleepStager calls awake "wake". (#77) - private func decodeSegments( + private static func decodeSegments( _ json: String?, sessionStart: Int ) -> (stages: Stages, intervals: [SleepInterval])? { guard let json, let data = json.data(using: .utf8), diff --git a/StrandTests/SleepOnsetDecodeTests.swift b/StrandTests/SleepOnsetDecodeTests.swift new file mode 100644 index 0000000000..7dc0bea456 --- /dev/null +++ b/StrandTests/SleepOnsetDecodeTests.swift @@ -0,0 +1,91 @@ +import XCTest +@testable import Strand + +/// The 2026-07-14 "shown bedtime 1:29 instead of 12:16" regression, pinned at the DECODE PATH. +/// +/// TWO stagesJSON formats exist in the sleepSession table: imported nights store a dict of MINUTES +/// `{"light","deep","rem","awake"}`; on-device COMPUTED nights store a SEGMENT ARRAY +/// `[{"start":epoch,"end":epoch,"stage":…}]` (`AnalyticsEngine.encodeStages`). The displayed-onset +/// stub test (`nightOnsetTs` / `isPreOnsetAwakeStub`) read asleep minutes via the dict-only +/// `decodeStages`, which returns nil for the array format — so every fragment of an on-device night +/// counted as 0 asleep minutes, the real ~54-min first sleep (12:16 → 1:22) tripped the "essentially +/// sleepless stub" branch (0 <= 3), and the shown bedtime jumped to the 1:29 main block. The #259 +/// real-sleep-episode floor (`preOnsetStubMinorAsleepFloorMin`) — added for this exact night — never +/// engaged because its input was zeroed before the comparison. `SleepOnsetStubTests` pins the rule on +/// PRE-COMPUTED minutes; these tests pin `SleepView.decodedAsleepMinutes`, the format-agnostic seam +/// the onset walk now reads, using the night's REAL stored JSON. +/// +/// Android twin: SleepScreen's onset stub-test caller needs the same both-format decode. +final class SleepOnsetDecodeTests: XCTestCase { + + /// The REAL 12:16 → 1:22 first-sleep fragment (deviceId my-whoop-noop, startTs 1784013364), + /// byte-for-byte as stored on the device: an on-device computed night's SEGMENT ARRAY. + /// Non-wake sum: light 926 s + deep 1320 s + rem 990 s = 3236 s ≈ 53.9 asleep minutes. + private static let fragmentStagesJSON = """ + [{"end":1784013450,"stage":"light","start":1784013364},{"end":1784013540,"stage":"wake","start":1784013450},{"end":1784013600,"stage":"light","start":1784013540},{"end":1784013720,"stage":"wake","start":1784013600},{"end":1784013930,"stage":"light","start":1784013720},{"end":1784015250,"stage":"deep","start":1784013930},{"end":1784015580,"stage":"light","start":1784015250},{"end":1784015640,"stage":"wake","start":1784015580},{"end":1784015880,"stage":"light","start":1784015640},{"end":1784016180,"stage":"wake","start":1784015880},{"end":1784017170,"stage":"rem","start":1784016180},{"end":1784017375,"stage":"wake","start":1784017170}] + """ + private static let fragmentEffectiveStartTs = 1_784_013_364 + private static let fragmentEndTs = 1_784_017_375 + + /// The REAL 1:29 → 7:32 main block (startTs 1784018465, hand-edited onset startTsAdjusted + /// 1784017740), same night, same stored format. Non-wake sum ≈ 277.6 asleep minutes. + private static let mainStagesJSON = """ + [{"end":1784018820,"start":1784017740,"stage":"light"},{"end":1784018940,"start":1784018820,"stage":"wake"},{"start":1784018940,"end":1784020140,"stage":"light"},{"end":1784021220,"start":1784020140,"stage":"wake"},{"stage":"light","start":1784021220,"end":1784021670},{"end":1784021910,"start":1784021670,"stage":"wake"},{"stage":"light","end":1784022150,"start":1784021910},{"start":1784022150,"stage":"deep","end":1784023260},{"start":1784023260,"end":1784023380,"stage":"light"},{"start":1784023380,"stage":"wake","end":1784023890},{"stage":"light","start":1784023890,"end":1784024250},{"stage":"wake","end":1784024520,"start":1784024250},{"end":1784024910,"stage":"light","start":1784024520},{"start":1784024910,"end":1784025120,"stage":"deep"},{"end":1784025330,"start":1784025120,"stage":"light"},{"start":1784025330,"end":1784025450,"stage":"rem"},{"start":1784025450,"end":1784025780,"stage":"wake"},{"stage":"light","start":1784025780,"end":1784026050},{"start":1784026050,"end":1784026710,"stage":"deep"},{"stage":"light","start":1784026710,"end":1784027040},{"end":1784027220,"stage":"wake","start":1784027040},{"stage":"light","start":1784027220,"end":1784027670},{"end":1784028510,"stage":"rem","start":1784027670},{"start":1784028510,"stage":"light","end":1784029080},{"stage":"wake","start":1784029080,"end":1784029620},{"end":1784029980,"stage":"light","start":1784029620},{"start":1784029980,"stage":"wake","end":1784030070},{"stage":"light","end":1784031600,"start":1784030070},{"end":1784031690,"stage":"wake","start":1784031600},{"start":1784031690,"stage":"light","end":1784032530},{"end":1784033400,"stage":"rem","start":1784032530},{"start":1784033400,"end":1784033940,"stage":"wake"},{"start":1784033940,"end":1784034300,"stage":"light"},{"stage":"wake","start":1784034300,"end":1784034330},{"end":1784035680,"stage":"light","start":1784034330},{"start":1784035680,"end":1784036100,"stage":"wake"},{"end":1784036160,"stage":"light","start":1784036100},{"start":1784036160,"end":1784037900,"stage":"rem"},{"end":1784038620,"stage":"wake","start":1784037900},{"start":1784038620,"end":1784038650,"stage":"light"},{"start":1784038650,"stage":"rem","end":1784039558}] + """ + private static let mainEffectiveStartTs = 1_784_017_740 + private static let mainEndTs = 1_784_039_558 + + // MARK: - decodedAsleepMinutes (the format-agnostic seam) + + /// THE BUG: the segment-array format (an on-device computed night) must decode to its real asleep + /// minutes. Before the fix the dict-only decode returned nil here and the caller substituted 0 — + /// the zero that made a real 54-min first sleep read as a "sleepless" stub. + func testSegmentArrayFormatDecodesRealAsleepMinutes() { + let asleep = SleepView.decodedAsleepMinutes(Self.fragmentStagesJSON, + effectiveStartTs: Self.fragmentEffectiveStartTs) + XCTAssertEqual(asleep, 3236.0 / 60.0, accuracy: 0.01) + // Above the #259 floor — the whole point: a real sleep episode must clear it. + XCTAssertGreaterThanOrEqual(asleep, SleepView.preOnsetStubMinorAsleepFloorMin) + } + + /// The imported dict-of-minutes format still decodes (asleep = light + deep + rem, awake excluded). + func testDictFormatStillDecodes() { + let asleep = SleepView.decodedAsleepMinutes(#"{"light":200,"deep":80,"rem":60,"awake":30}"#, + effectiveStartTs: 0) + XCTAssertEqual(asleep, 340, accuracy: 0.001) + } + + /// nil / empty / garbage stay 0 — the degenerate inputs keep the old behaviour. + func testDegenerateInputsAreZero() { + XCTAssertEqual(SleepView.decodedAsleepMinutes(nil, effectiveStartTs: 0), 0) + XCTAssertEqual(SleepView.decodedAsleepMinutes("", effectiveStartTs: 0), 0) + XCTAssertEqual(SleepView.decodedAsleepMinutes("not json", effectiveStartTs: 0), 0) + XCTAssertEqual(SleepView.decodedAsleepMinutes("[]", effectiveStartTs: 0), 0) + } + + /// The segment decode threads the #259 pre-onset trim: segments before `effectiveStartTs` don't + /// count, exactly as the hero's stage totals trim them. + func testSegmentDecodeTrimsPreOnset() { + // One 10-min light segment, but the effective onset sits 5 min into it. + let json = #"[{"start":1000,"end":1600,"stage":"light"}]"# + XCTAssertEqual(SleepView.decodedAsleepMinutes(json, effectiveStartTs: 1300), 5, accuracy: 0.001) + } + + // MARK: - The golden: the real night's onset comes from the 12:16 fragment + + /// END-TO-END through the decode path: both REAL blocks of the 2026-07-14 night, decoded from their + /// stored segment-array JSON (NOT pre-computed minutes), walk to onset index 0 — the 12:16 fragment. + /// Before the fix both decoded to 0 asleep, the fragment classified as a sleepless stub, and the + /// index was 1 (the 1:29 main block) — the "last night's sleep 1:29" VK saw on the Sleep tab hero. + func testRealNightOnsetIndexIsTheFirstSleepFragment() { + let frags: [(json: String, effStart: Int, end: Int)] = [ + (Self.fragmentStagesJSON, Self.fragmentEffectiveStartTs, Self.fragmentEndTs), + (Self.mainStagesJSON, Self.mainEffectiveStartTs, Self.mainEndTs), + ] + let spansMin = frags.map { Double($0.end - $0.effStart) / 60.0 } + let asleepsMin = frags.map { + SleepView.decodedAsleepMinutes($0.json, effectiveStartTs: $0.effStart) + } + XCTAssertEqual(SleepView.nightOnsetIndex(spansMin: spansMin, asleepsMin: asleepsMin), 0) + } +} From e6ce8db7f1db0b81ac79215b152029c178d7b0ff Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 13:10:29 -0700 Subject: [PATCH 099/171] =?UTF-8?q?feat(stress):=20motion-gate=20the=20int?= =?UTF-8?q?raday=20timeline=20=E2=80=94=20mask=20exertion,=20don't=20score?= =?UTF-8?q?=20it=20as=20stress?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cardiac signals can't tell psychological stress from EXERTION: a brisk walk and an anxious meeting both raise HR and drop HRV. DaytimeStress.analyze never saw motion, so an ambulatory hour was scored as 'stress' — the roadmap's central correctness bug. When the caller supplies the day's gravity (wrist accelerometer), an hour that was ambulatory for >= activityMaskFraction (0.30) of its records — 'ambulatory' = a per-record activity intensity above WorkoutDetector.motionThreshold (0.20 L2-g, the codebase's calibrated walk floor) — is MASKED (level=nil, HourPoint.maskedForActivity) rather than scored, and excluded from the day's calm reference and coverage totals. A one-hour post-exercise shadow masks the following hour while its HR is still above the calm reference (self-limiting to genuine cardiac recovery). Reuses the exact WorkoutDetector.activitySeries that SedentaryDetector/AutoWorkoutDetector already trust. Purely additive: no gravity -> nothing masked -> byte-identical to before (honest — we only gate when we can see motion). StressView reads gravity via the same store handle as R-R. Result gains activityMaskedHours for honest coverage. Validated on real 1Hz wrist data (2026-07-12): only 1 of 11 waking hours masked (the one with 36% sustained wrist motion); desk/light hours (3-17% active) are all preserved. The 0.20/0.30 thresholds cleanly separate genuine ambulation from ambient movement, so a step-heavy day does NOT empty the timeline. New tests pin masking, a still-HR control (motion drives the mask, not HR), additive empty-gravity, and both shadow directions; full StrandAnalytics suite green (1133). --- .../StrandAnalytics/DaytimeStress.swift | 118 ++++++++++++++++-- .../DaytimeStressTests.swift | 101 +++++++++++++++ Strand/Screens/StressView.swift | 7 +- 3 files changed, 213 insertions(+), 13 deletions(-) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift index e5c04e95a5..729673d793 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/DaytimeStress.swift @@ -38,6 +38,33 @@ public enum DaytimeStress { public static let highBandFloor: Double = 2.0 /// Consecutive most-recent covered hours that must all be HIGH to flag sustained stress. public static let sustainedHours: Int = 3 + + // MARK: - Motion gate (MOVE E — the correctness gate) + // + // Cardiac signals cannot tell psychological stress from EXERTION: a brisk walk and an anxious + // meeting both raise HR and drop HRV. Without a motion channel, `analyze` scored an ambulatory + // hour as "stress" — the central correctness bug. When the caller supplies the day's gravity + // (wrist accelerometer), an hour that was substantially ambulatory is MASKED (`level == nil`, + // `HourPoint.maskedForActivity == true`) rather than scored, and it is excluded from the calm + // reference and the coverage/"time in high stress" totals. No gravity → no masking → byte- + // identical prior behaviour (honest: we only gate when we can see motion). + + /// An hour whose gravity-derived activity is ambulatory for at least this fraction of its + /// records is EXERTION, not stress — masked, not scored. "Ambulatory" = a per-record activity + /// intensity above `WorkoutDetector.motionThreshold` (0.20 L2-g, the codebase's calibrated walk + /// floor: desk ≈ 0.05–0.10 g, walking ≈ 0.2–0.4 g). 0.30 ⇒ "≥ 30 % of the hour was walking/ + /// moving"; below it, a stray reach or a single trip to the kitchen doesn't mask a desk hour. + /// First-cut, wants tuning against real worn days (an hourly grain is coarse — this is the gate + /// that later unblocks finer 5-min epochs, at which point the fraction can tighten). Range (0, 1]. + public static let activityMaskFraction: Double = 0.30 + /// Post-exercise shadow: HR stays elevated for a while AFTER exertion ends, so the single hour + /// that immediately FOLLOWS a directly-ambulatory hour is ALSO masked WHILE its mean HR is still + /// above the calm reference by this margin (bpm). A following hour whose HR has already returned + /// to the calm reference is scored normally — the shadow self-limits to genuine cardiac recovery. + /// Deliberately ONE hour deep (keyed on the directly-active hour, not chained through prior + /// shadows) so a genuinely tense afternoon that happens to follow a workout isn't masked away. + /// Range ≥ 0. + public static let postActivityShadowBPM: Double = 8.0 /// First/last local hour-of-day treated as "waking" for the timeline (06:00–22:00). public static let wakingStartHour: Int = 6 public static let wakingEndHour: Int = 22 @@ -136,16 +163,23 @@ public enum DaytimeStress { public let meanHR: Double? /// RMSSD over the hour's clean R-R (ms), or nil (too few clean beats). public let rmssd: Double? + /// True when this hour was left unscored because it was AMBULATORY (exertion), not because + /// it lacked HR — so `level == nil` here means "masked as activity", not "no data". Lets the + /// UI distinguish "you were moving" from "no reading" and keeps active hours out of the calm + /// reference and the coverage totals. See the motion-gate constants above. + public let maskedForActivity: Bool /// True when the hour was scored (had enough HR to place on the curve). public var hasData: Bool { level != nil } - public init(hour: Int, startTs: Int, level: Double?, meanHR: Double?, rmssd: Double?) { + public init(hour: Int, startTs: Int, level: Double?, meanHR: Double?, rmssd: Double?, + maskedForActivity: Bool = false) { self.hour = hour self.startTs = startTs self.level = level self.meanHR = meanHR self.rmssd = rmssd + self.maskedForActivity = maskedForActivity } } @@ -174,10 +208,16 @@ public enum DaytimeStress { /// (there, a missing RMSSD is already handled per-hour by `rawScore`, not flagged /// day-wide) and false for `.empty`. public let hrOnlyFallback: Bool + /// ADDITIVE — count of waking hours left unscored because they were AMBULATORY (the motion + /// gate fired), i.e. `hours.filter { $0.maskedForActivity }.count`. Lets a caller report + /// honest coverage ("N hours excluded — you were moving") instead of a silently short + /// timeline. 0 when no gravity was supplied or nothing was masked; 0 for `.empty`. + public let activityMaskedHours: Int public init(hours: [HourPoint], sustainedHigh: Bool, sustainedRun: Int, dayMean: Double?, peak: HourPoint?, - highStressMinutes: Int = 0, hrOnlyFallback: Bool = false) { + highStressMinutes: Int = 0, hrOnlyFallback: Bool = false, + activityMaskedHours: Int = 0) { self.hours = hours self.sustainedHigh = sustainedHigh self.sustainedRun = sustainedRun @@ -185,6 +225,7 @@ public enum DaytimeStress { self.peak = peak self.highStressMinutes = highStressMinutes self.hrOnlyFallback = hrOnlyFallback + self.activityMaskedHours = activityMaskedHours } /// The scored hours only (level non-nil), in time order. @@ -193,7 +234,8 @@ public enum DaytimeStress { /// Empty read — used when the day had no usable intraday HR at all. public static let empty = Result(hours: [], sustainedHigh: false, sustainedRun: 0, dayMean: nil, peak: nil, - highStressMinutes: 0, hrOnlyFallback: false) + highStressMinutes: 0, hrOnlyFallback: false, + activityMaskedHours: 0) } // MARK: - Shared stress math (identical formula to the daily StressModel) @@ -253,12 +295,16 @@ public enum DaytimeStress { /// - rr: the day's `[RRInterval]`. /// - tzOffsetSeconds: seconds east of UTC, for placing each bucket on the LOCAL /// clock (so "waking hours" and the hour labels are local). Defaults to UTC. + /// - gravity: the day's `[GravitySample]` (wrist accelerometer), for the motion gate. Defaults + /// empty: with no gravity NOTHING is masked and the read is byte-identical to before. When + /// present, ambulatory hours are masked out of the score (see the motion-gate constants). /// - mode: `.dayRelative` (DEFAULT — unchanged existing behaviour) or /// `.baselineRelative` (Oura-style, vs a personal rolling baseline). ADDITIVE and /// opt-in: existing callers that don't pass `mode` keep the exact prior behaviour. /// /// Returns `.empty` when there isn't a single hour with enough HR to score. public static func analyze(hr: [HRSample], rr: [RRInterval], + gravity: [GravitySample] = [], tzOffsetSeconds: Int = 0, mode: ScoringMode = .dayRelative) -> Result { // v7.0.2 perf (#707): buckets the day's full HR + R-R streams into per-hour aggregates and runs an @@ -277,14 +323,21 @@ public enum DaytimeStress { let key = StressKey( hr: StreamFingerprint.of(hr, ts: { $0.ts }, quant: { Int($0.bpm) }), rr: StreamFingerprint.of(rr, ts: { $0.ts }, quant: { Int($0.rrMs) }), + // Fold all three axes into one quant so two different gravity streams can't share a key + // (same hr/rr, different motion must NOT reuse a cached result). + gravity: StreamFingerprint.of(gravity, ts: { $0.ts }, quant: { + Int(($0.x * 128).rounded()) &+ Int(($0.y * 128).rounded()) &* 257 + &+ Int(($0.z * 128).rounded()) &* 66_049 + }), tz: tzOffsetSeconds, mode: modeKey) return analyzeCache.value(key) { - analyzeUncached(hr: hr, rr: rr, tzOffsetSeconds: tzOffsetSeconds, mode: mode) + analyzeUncached(hr: hr, rr: rr, gravity: gravity, tzOffsetSeconds: tzOffsetSeconds, mode: mode) } } private struct StressKey: Hashable { - let hr: StreamFingerprint; let rr: StreamFingerprint; let tz: Int; let mode: ModeKey + let hr: StreamFingerprint; let rr: StreamFingerprint; let gravity: StreamFingerprint + let tz: Int; let mode: ModeKey } /// Hashable fingerprint of `ScoringMode` for the memo cache — `BaselineState` itself isn't @@ -298,6 +351,7 @@ public enum DaytimeStress { private static let analyzeCache = AnalyticsMemoCache(capacity: 8) private static func analyzeUncached(hr: [HRSample], rr: [RRInterval], + gravity: [GravitySample], tzOffsetSeconds: Int, mode: ScoringMode) -> Result { guard !hr.isEmpty else { return .empty } @@ -330,6 +384,30 @@ public enum DaytimeStress { aggs.append(HourAgg(bucket: b, meanHR: mHR, rmssd: rrRes.rmssd, nHR: hrs.count)) } + // 2b) Motion gate: bucket the day's gravity-derived activity by the SAME local hour and mark + // each hour AMBULATORY when at least `activityMaskFraction` of its records clear the + // calibrated walk floor (`WorkoutDetector.motionThreshold`) — reusing the exact activity + // series that SedentaryDetector / WorkoutDetector already trust. Empty gravity → no active + // buckets → nothing masked below (byte-identical to the pre-motion behaviour). + var activeFracByBucket: [Int: Double] = [:] + if !gravity.isEmpty { + var counts: [Int: (active: Int, total: Int)] = [:] + for p in WorkoutDetector.activitySeries(gravity) { + let local = p.ts + tzOffsetSeconds + let bucket = floorDiv(local, bucketSeconds) * bucketSeconds + var e = counts[bucket] ?? (0, 0) + e.total += 1 + if p.intensity > WorkoutDetector.motionThreshold { e.active += 1 } + counts[bucket] = e + } + for (b, e) in counts where e.total > 0 { + activeFracByBucket[b] = Double(e.active) / Double(e.total) + } + } + func isAmbulatory(_ bucket: Int) -> Bool { + (activeFracByBucket[bucket] ?? 0) >= activityMaskFraction + } + // 3) The reference point + spread for each signal — WHERE they come from depends on // `mode`. Every other step (bucketing above, the waking-hour filter, the squash // curve, sustained-high, high-stress-minutes below) is identical between modes. @@ -352,7 +430,10 @@ public enum DaytimeStress { // hours of it. Letting those night hours into the reference drags the "calm" anchor // far beneath every waking hour, inflating an ordinary calm day toward HIGH and // falsely tripping the sustained-high Breathe nudge. - let referenceAggs = aggs.filter { isWakingHour($0.bucket) } + // Ambulatory hours are excluded from the day's OWN calm reference too: an exertion + // hour's elevated HR / suppressed HRV must not pull the calm anchor up or inflate the + // across-hour spread the z-scores divide by. + let referenceAggs = aggs.filter { isWakingHour($0.bucket) && !isAmbulatory($0.bucket) } let hrMeans = referenceAggs.compactMap { $0.meanHR } let rmssdVals = referenceAggs.compactMap { $0.rmssd } refHR = calmReference(hrMeans, calmIsLow: true) // calm HR is LOW @@ -397,15 +478,26 @@ public enum DaytimeStress { let hourOfDay = floorDiv(a.bucket, bucketSeconds) % 24 // The wall-clock bucket start (undo the local shift applied above). let wallStart = a.bucket - tzOffsetSeconds - // Score only when at least one signal is present AND HR cleared the count gate - // (HR is the always-available anchor; RMSSD enriches it when beats allow). - let level: Double? = a.meanHR != nil + // Motion gate: an AMBULATORY hour — or the post-exercise shadow hour whose HR hasn't yet + // recovered to the calm reference — is EXERTION, so its elevated HR is masked out of the + // score instead of read as stress. The shadow is gated on `refHR` so it self-limits to + // genuine cardiac recovery (a following hour already back at baseline scores normally). + // Only meaningful when the hour actually HAD a reading to withhold — a no-HR hour is plain + // `.noData`, not "masked". + let shadow = isAmbulatory(a.bucket - bucketSeconds) + && a.meanHR != nil && refHR != nil && a.meanHR! > refHR! + postActivityShadowBPM + let masked = a.meanHR != nil && (isAmbulatory(a.bucket) || shadow) + // Score only when at least one signal is present AND HR cleared the count gate AND the + // hour wasn't motion-masked (HR is the always-available anchor; RMSSD enriches it). + let level: Double? = (a.meanHR != nil && !masked) ? squash(rawScore(hr: a.meanHR, meanHR: refHR, sdHR: sdHR, rmssd: a.rmssd, meanRMSSD: refRMSSD, sdRMSSD: sdRMSSD)) : nil points.append(HourPoint(hour: hourOfDay, startTs: wallStart, - level: level, meanHR: a.meanHR, rmssd: a.rmssd)) + level: level, meanHR: a.meanHR, rmssd: a.rmssd, + maskedForActivity: masked)) } + let activityMaskedHours = points.reduce(0) { $0 + ($1.maskedForActivity ? 1 : 0) } let scored = points.compactMap { p -> (HourPoint, Double)? in p.level.map { (p, $0) } } guard !scored.isEmpty else { @@ -416,7 +508,8 @@ public enum DaytimeStress { return points.isEmpty ? .empty : Result(hours: points, sustainedHigh: false, sustainedRun: 0, dayMean: nil, peak: nil, - highStressMinutes: 0, hrOnlyFallback: hrOnlyFallback) + highStressMinutes: 0, hrOnlyFallback: hrOnlyFallback, + activityMaskedHours: activityMaskedHours) } // 5) Sustained-high flag: walk back from the latest SCORED hour while each is HIGH. @@ -437,7 +530,8 @@ public enum DaytimeStress { return Result(hours: points, sustainedHigh: sustained, sustainedRun: run, dayMean: dayMean, peak: peak, - highStressMinutes: highStressMinutes, hrOnlyFallback: hrOnlyFallback) + highStressMinutes: highStressMinutes, hrOnlyFallback: hrOnlyFallback, + activityMaskedHours: activityMaskedHours) } // MARK: - Helpers diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeStressTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeStressTests.swift index c48f6221ee..2e5d2395a1 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeStressTests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DaytimeStressTests.swift @@ -281,4 +281,105 @@ final class DaytimeStressTests: XCTestCase { XCTAssertGreaterThan(suppressed, normal, "suppressed RMSSD vs. the personal baseline should read MORE stressed than normal variability") } + + // MARK: - Motion gate (MOVE E): an ambulatory hour is EXERTION — masked, not scored as stress + + /// Fill one local hour with gravity whose consecutive L2 delta ≈ `delta` on every step (x + /// alternates 0↔delta), so the derived activity intensity ≈ `delta` across the hour. delta > + /// `WorkoutDetector.motionThreshold` (0.20) ⇒ AMBULATORY; a small delta (0.02, desk) ⇒ still. + private func hourGravity(_ hour: Int, delta: Double, n: Int = 180) -> [GravitySample] { + let base = hour * 3_600 + let step = max(1, 3_600 / n) + return (0.. 0.20 floor) + + let r = DaytimeStress.analyze(hr: hr, rr: [], gravity: grav) + let h11 = r.hours.first { $0.hour == 11 }! + XCTAssertNil(h11.level, "an ambulatory hour must not be scored") + XCTAssertTrue(h11.maskedForActivity, "the walking hour must be flagged masked-for-activity") + XCTAssertFalse(r.scored.contains { $0.hour == 11 }, "masked hour is excluded from the scored set") + XCTAssertEqual(r.activityMaskedHours, 1) + } + + func testSameElevatedHRIsScoredWhenStill() { + // CONTROL for the test above: identical HR, but the elevated hour is STILL (low gravity). The + // elevation is no longer explained by motion, so it IS scored as stress — proving the mask is + // driven by MOTION, not by the HR value. + var hr: [HRSample] = [] + for h in [8, 9, 10] { hr += hourHR(h, bpm: 60) } + hr += hourHR(11, bpm: 110) + var grav: [GravitySample] = [] + for h in [8, 9, 10, 11] { grav += hourGravity(h, delta: 0.02) } // ALL still + + let r = DaytimeStress.analyze(hr: hr, rr: [], gravity: grav) + let h11 = r.hours.first { $0.hour == 11 }! + XCTAssertNotNil(h11.level, "a still hour with elevated HR must be scored, not masked") + XCTAssertFalse(h11.maskedForActivity) + XCTAssertEqual(r.activityMaskedHours, 0) + XCTAssertEqual(r.peak?.hour, 11, "the still elevated hour is the day's stress peak") + } + + func testNoGravityIsByteIdenticalToOmittingIt() { + // The motion gate is purely additive: no gravity (or an empty array) must produce a + // byte-identical Result to the pre-motion call — nothing is masked without a motion signal. + var hr: [HRSample] = [] + for h in [8, 9, 10] { hr += hourHR(h, bpm: 60) } + hr += hourHR(11, bpm: 110) + let withoutArg = DaytimeStress.analyze(hr: hr, rr: []) + let emptyGrav = DaytimeStress.analyze(hr: hr, rr: [], gravity: []) + XCTAssertEqual(withoutArg, emptyGrav) + XCTAssertEqual(emptyGrav.activityMaskedHours, 0) + XCTAssertTrue(emptyGrav.scored.contains { $0.hour == 11 }, + "with no gravity the elevated hour scores as before (the pre-motion behaviour)") + } + + func testPostExerciseShadowMasksTheNextStillButElevatedHour() { + // A walking hour (11), then a STILL hour (12) whose HR is still elevated (post-exercise + // recovery) → 12 is shadow-masked even though it wasn't itself ambulatory. + var hr: [HRSample] = [] + for h in [8, 9, 10] { hr += hourHR(h, bpm: 58) } + hr += hourHR(11, bpm: 120) // walking + hr += hourHR(12, bpm: 95) // still, HR still elevated + var grav: [GravitySample] = [] + for h in [8, 9, 10] { grav += hourGravity(h, delta: 0.02) } + grav += hourGravity(11, delta: 0.30) // ambulatory + grav += hourGravity(12, delta: 0.02) // still + + let r = DaytimeStress.analyze(hr: hr, rr: [], gravity: grav) + XCTAssertTrue(r.hours.first { $0.hour == 11 }!.maskedForActivity, "the walking hour is masked") + let h12 = r.hours.first { $0.hour == 12 }! + XCTAssertTrue(h12.maskedForActivity, "the still-but-elevated hour after exercise is shadow-masked") + XCTAssertNil(h12.level) + } + + func testShadowReleasesWhenHRHasRecovered() { + // Same walking hour (11), but the following still hour (12) has HR already back at the calm + // reference → the shadow self-limits and 12 is scored normally, not masked. + var hr: [HRSample] = [] + for h in [8, 9, 10] { hr += hourHR(h, bpm: 58) } + hr += hourHR(11, bpm: 120) // walking + hr += hourHR(12, bpm: 58) // still, HR fully recovered + var grav: [GravitySample] = [] + for h in [8, 9, 10] { grav += hourGravity(h, delta: 0.02) } + grav += hourGravity(11, delta: 0.30) // ambulatory + grav += hourGravity(12, delta: 0.02) // still + + let r = DaytimeStress.analyze(hr: hr, rr: [], gravity: grav) + XCTAssertTrue(r.hours.first { $0.hour == 11 }!.maskedForActivity, "the walking hour is masked") + let h12 = r.hours.first { $0.hour == 12 }! + XCTAssertFalse(h12.maskedForActivity, "a recovered following hour is not shadow-masked") + XCTAssertNotNil(h12.level, "the recovered hour is scored normally") + } } diff --git a/Strand/Screens/StressView.swift b/Strand/Screens/StressView.swift index 0ebba87810..ce8fd3b670 100644 --- a/Strand/Screens/StressView.swift +++ b/Strand/Screens/StressView.swift @@ -115,6 +115,11 @@ struct StressView: View { } let rr = (try? await repo.storeHandle()?.rrIntervals( deviceId: repo.deviceId, from: from, to: to, limit: 200_000)) ?? [] + // Wrist accelerometer for the motion gate: an ambulatory hour is EXERTION, not stress, so it's + // masked rather than scored (DaytimeStress). Same store read as R-R; empty on hardware/imports + // with no gravity, which is exactly the "no masking, prior behaviour" degradation. + let gravity = (try? await repo.storeHandle()?.gravitySamples( + deviceId: repo.deviceId, from: from, to: to, limit: 200_000)) ?? [] // Score today's hours against the PERSONAL cross-day daytime baseline when enough worn history // exists (Oura-style `.baselineRelative`, validated r≈0.6), else the day's own calm hours @@ -123,7 +128,7 @@ struct StressView: View { let mode = await daytimeScoringMode(startOfToday: startOfDay) if case .baselineRelative = mode { daytimeUsesPersonalBaseline = true } else { daytimeUsesPersonalBaseline = false } - daytime = DaytimeStress.analyze(hr: hr, rr: rr, tzOffsetSeconds: tz, mode: mode) + daytime = DaytimeStress.analyze(hr: hr, rr: rr, gravity: gravity, tzOffsetSeconds: tz, mode: mode) // ADDITIVE advanced readouts, computed on-demand from the SAME `rr` (no extra fetch, no // DB / schema change, and no effect on the 0..3 score above). Each engine returns nil when From 624014eee59a330ef32e7d79ac5196698c41d2e4 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 16:08:01 -0700 Subject: [PATCH 100/171] feat(readiness): fold HRV/RHR baselines through the spine in a window-fold mode ReadinessEngine z-scored today's HRV/RHR against a naive mean + sample SD over the trailing 30 nights. Two defects: a single freak night inflates that SD (a real drop then under-flags for weeks until the outlier rolls out), and a tight baseline saturates the z. The app already has a robust engine (Baselines: Winsorized-EWMA, sigma-floor, recency) -- but adopting it naively is WRONG for readiness. The spine's hard-outlier rejection (>5sigma -> discard) is right for RecoveryScorer's INCREMENTAL nightly fold, but readiness re-folds a trailing WINDOW every call: a recent sustained shift (fitness change / device swap / supplement onset) lands past the young grace period and gets rejected as a run of outliers, so readiness scores against a stale baseline (replayed: a settled 140->85ms window stays anchored at 140, flagging every new-normal night as a crash). Add rejectHardOutliers (default true -> RecoveryScorer byte-identical) to Baselines.update/foldHistory. Readiness folds with it OFF: Winsorization still DAMPS a single freak night (clamp to +/-3sigma, not folded raw), but a sustained shift is FOLLOWED not rejected. HRV folds in the log domain (new readiness_hrv_ln config, ln bounds/floor) so Winsor-clamp composes with RD1's lnRMSSD; RHR uses resting_hr. Design chosen by replaying 5 candidates over 2yr of real nightly HRV: naive is 32% wider-SD'd after each of 7 real freak nights (under-flags real drops); reject-on stalls on shifts; reject-off + Winsor damps outliers AND adapts. New RD2SpineTests pin the reject on/off capability, single-freak damping, sustained-shift adaptation, and that a genuine low still flags; RD1 geometric test updated for the Winsor-EWMA center. Full suite green (1137); macOS build green. --- .../Sources/StrandAnalytics/Baselines.swift | 33 ++++++++- .../StrandAnalytics/ReadinessEngine.swift | 18 ++++- .../RD1LogDomainTests.swift | 8 +- .../StrandAnalyticsTests/RD2SpineTests.swift | 73 +++++++++++++++++++ 4 files changed, 124 insertions(+), 8 deletions(-) create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RD2SpineTests.swift diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift index ce86a117f7..abd580b3bc 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift @@ -188,12 +188,25 @@ public enum Baselines { halfLifeB: 14.0, halfLifeS: 21.0), "daytime_rmssd": MetricCfg(minVal: 5.0, maxVal: 250.0, floorSpread: 7.0, halfLifeB: 14.0, halfLifeS: 21.0), + + // Readiness HRV baseline, folded in the LOG domain with hard-outlier rejection OFF (RD2 · the + // window-fold spine mode — see Baselines.update's `rejectHardOutliers`). ReadinessEngine + // z-scores lnRMSSD (RD1: HRV is right-skewed), so these bounds/floor are in LN(ms) units — NOT + // the raw-ms "hrv" config above (which the RecoveryScorer folds linearly, untouched). + // minVal/maxVal = ln(8)/ln(250), the plausible HRV band in ln. floorSpread 0.08 (ln) sits just + // below a real wearer's own ln-HRV night-to-night spread (~0.10, measured on real WHOOP nights), + // so personal spread drives the z while an ultra-stable baseline is floored against saturation. + "readiness_hrv_ln": MetricCfg(minVal: 2.079, maxVal: 5.521, floorSpread: 0.08, + halfLifeB: 14.0, halfLifeS: 21.0), ] /// Convenience accessors for the standard configs. public static var hrvCfg: MetricCfg { metricCfg["hrv"]! } public static var restingHRCfg: MetricCfg { metricCfg["resting_hr"]! } public static var respCfg: MetricCfg { metricCfg["resp"]! } + /// Readiness HRV baseline config — folded in the LOG domain (ln ms) with hard-outlier rejection + /// OFF. See the `readiness_hrv_ln` comment above; distinct from the raw-ms `hrvCfg`. + public static var readinessHRVLnCfg: MetricCfg { metricCfg["readiness_hrv_ln"]! } /// Baseline config for the RecoveryScorer Activity-Balance / previous-day-Effort term. public static var strainCfg: MetricCfg { metricCfg["strain"]! } /// Personal daytime/waking HR baseline config — see the `daytime_hr` comment above. @@ -221,7 +234,8 @@ public enum Baselines { /// - `value == nil` or out-of-range: skip-and-hold (carry forward). /// - hard outlier (> HARD_OUTLIER_K × spread): seen but not folded. /// - otherwise: Winsorized EWMA center + EWMA-abs-dev spread update. - public static func update(_ state: BaselineState?, value: Double?, cfg: MetricCfg) -> BaselineState { + public static func update(_ state: BaselineState?, value: Double?, cfg: MetricCfg, + rejectHardOutliers: Bool = true) -> BaselineState { let lb = lambda(halfLife: cfg.halfLifeB) let ls = lambda(halfLife: cfg.halfLifeS) @@ -262,7 +276,17 @@ public enum Baselines { // Suspending this during early life is the core anti-anchoring fix — a high seed with a // floor-tight spread would otherwise reject the user's real, lower readings as "outliers" // (a true 54ms vs an anchored ~90ms baseline is >5× the floor spread). - if state.nValid >= minNightsSeed && !isYoung { + // + // `rejectHardOutliers` (default true) can turn this OFF for a TRAILING-WINDOW re-fold — where + // the same 30-night window is re-folded every call and a *recent sustained* shift lands at the + // window's end (past the young grace period), so rejection would discard a real new normal as a + // string of "outliers" (readiness's device-swap / supplement-onset failure mode). With it off, + // the Winsorization below STILL damps a single freak night (clamped to ±winsorK·spread rather + // than folded raw) — but a sustained shift is followed instead of rejected, because each night + // nudges the center and widens the spread until the new level is no longer an outlier. This is + // exactly the incremental-fold (reject on, RecoveryScorer) vs window-fold (reject off, Readiness) + // distinction; validated on real HRV history (see ReadinessEngine's RD2 note). + if rejectHardOutliers && state.nValid >= minNightsSeed && !isYoung { let dev = abs(value - state.baseline) if dev > hardOutlierK * state.spread { return BaselineState(baseline: state.baseline, spread: state.spread, @@ -300,9 +324,10 @@ public enum Baselines { /// Replay an ordered sequence of nightly values (oldest first) to build state. /// `nil` entries are treated as missing nights (skip-and-hold). - public static func foldHistory(_ values: [Double?], cfg: MetricCfg) -> BaselineState { + public static func foldHistory(_ values: [Double?], cfg: MetricCfg, + rejectHardOutliers: Bool = true) -> BaselineState { var state: BaselineState? = nil - for v in values { state = update(state, value: v, cfg: cfg) } + for v in values { state = update(state, value: v, cfg: cfg, rejectHardOutliers: rejectHardOutliers) } if let s = state { return s } let seed = (cfg.minVal + cfg.maxVal) / 2.0 return BaselineState(baseline: seed, spread: cfg.floorSpread, nValid: 0, diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/ReadinessEngine.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/ReadinessEngine.swift index cf26854953..5700a7dd08 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/ReadinessEngine.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/ReadinessEngine.swift @@ -145,6 +145,7 @@ public enum ReadinessEngine { decimals: 0, higherIsBetter: true, logDomain: true, // RD1: lnRMSSD — HRV is right-skewed + cfg: Baselines.readinessHRVLnCfg, // RD2: ln-space spine, reject off goodText: "above your baseline - well recovered", neutralText: "in your normal range", watchText: "a touch below baseline", @@ -159,6 +160,7 @@ public enum ReadinessEngine { unit: "bpm", decimals: 0, higherIsBetter: false, + cfg: Baselines.restingHRCfg, // RD2: raw-bpm spine, reject off goodText: "at or below baseline", neutralText: "in your normal range", watchText: "running a little high", @@ -232,6 +234,7 @@ public enum ReadinessEngine { private static func zSignal(value: Double?, baseline: [Double], key: String, label: String, unit: String, decimals: Int, higherIsBetter: Bool, logDomain: Bool = false, + cfg: MetricCfg, goodText: String, neutralText: String, watchText: String, badText: String) -> Signal? { guard let v = value, baseline.count >= minBaseline else { return nil } @@ -243,9 +246,20 @@ public enum ReadinessEngine { // outlier-inflated arithmetic mean. let tv = logDomain ? log(max(v, 1.0)) : v let tb = logDomain ? baseline.map { log(max($0, 1.0)) } : baseline - guard let m = mean(tb), let sd = sampleSD(tb), sd > 0 else { return nil } + // RD2: fold the trailing baseline through the shared Winsorized-EWMA spine — recency-weighted, + // σ-floored (a tight baseline can't saturate the z), and Winsor-clamped so a single freak night + // is DAMPED not folded raw — instead of a flat mean + sample SD. Hard-outlier REJECTION is off + // (`rejectHardOutliers: false`): a re-folded trailing window must ADAPT to a recent sustained + // shift (fitness change / device swap) rather than reject the new normal as a run of outliers — + // the window-fold vs incremental-fold distinction, validated on real HRV history. `cfg` is in + // the SAME space as `tb` (ln for HRV, linear for RHR); center + spread come back σ-floored. + let state = Baselines.foldHistory(tb.map { Optional($0) }, cfg: cfg, rejectHardOutliers: false) + guard state.usable else { return nil } + let sigma = Baselines.sigma(state) + guard sigma > 0 else { return nil } + let m = state.baseline // Orient z so positive always means "better". - let z = (higherIsBetter ? (tv - m) : (m - tv)) / sd + let z = (higherIsBetter ? (tv - m) : (m - tv)) / sigma let flag: Flag let text: String switch z { diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RD1LogDomainTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RD1LogDomainTests.swift index 43bdaab198..e98339facd 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RD1LogDomainTests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RD1LogDomainTests.swift @@ -23,8 +23,12 @@ final class RD1LogDomainTests: XCTestCase { days.append(d(29, hrv: 50, rhr: 52)) // today: a typical night let hrv = ReadinessEngine.evaluate(days: days).signals.first { $0.key == "hrv" } - // Log domain baselines against the geometric mean (51). Raw-ms z would report "50 vs 54 ms". - XCTAssertEqual(hrv?.evidence, "50 vs 51 ms") + // Log domain baselines against a GEOMETRIC center (49 ms), well below the arithmetic mean (54) + // a raw-ms z would report. Under RD2 the center is the recency-weighted, Winsor-clamped EWMA + // (Baselines spine, reject off): the 8 recent 85 ms nights nudge it up from the 42 ms cluster + // but Winsorization caps their pull, so it lands at 49 — still geometric, not the tail-inflated + // arithmetic 54. + XCTAssertEqual(hrv?.evidence, "50 vs 49 ms") // 50 sits at the typical night → neutral, read against a representative (not tail-inflated) baseline. XCTAssertEqual(hrv?.flag, .neutral) } diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RD2SpineTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RD2SpineTests.swift new file mode 100644 index 0000000000..2da22551db --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/RD2SpineTests.swift @@ -0,0 +1,73 @@ +import XCTest +@testable import StrandAnalytics +import WhoopStore + +/// RD2 — the readiness HRV/RHR baselines are folded through the shared Winsorized-EWMA spine +/// (`Baselines.foldHistory`) instead of a naive mean + sample SD, in the WINDOW-FOLD mode +/// (`rejectHardOutliers: false`). Chosen over the full incremental spine after replaying real HRV +/// history: hard-outlier rejection is right for RecoveryScorer's incremental nightly fold but WRONG +/// for a re-folded trailing window, where a recent sustained shift lands past the young grace period +/// and would be rejected as a run of "outliers". Winsorization (kept) still damps single freak nights. +final class RD2SpineTests: XCTestCase { + + // MARK: - The spine capability itself: reject on (incremental) vs off (window-fold) + + func testWindowFoldAdaptsToSustainedShiftThatIncrementalRejects() { + // 20 settled nights at 140 ms, then 10 recent nights at 85 ms (a real sustained shift). The + // incremental fold (reject on) discards the recent block as >5σ outliers and stays anchored + // high; the window fold (reject off) follows the shift down. This is the exact readiness + // failure mode (device swap / supplement onset) the mode switch fixes. + let vals: [Double?] = Array(repeating: 140.0, count: 20) + Array(repeating: 85.0, count: 10) + let rejectOn = Baselines.foldHistory(vals, cfg: Baselines.hrvCfg, rejectHardOutliers: true) + let rejectOff = Baselines.foldHistory(vals, cfg: Baselines.hrvCfg, rejectHardOutliers: false) + XCTAssertGreaterThan(rejectOn.baseline, 130, "hard-reject keeps the stale high baseline") + XCTAssertLessThan(rejectOff.baseline, rejectOn.baseline - 10, + "window-fold adapts meaningfully toward the recent sustained level") + } + + func testSingleFreakNightIsDampedNotRejectedInWindowFold() { + // One freak sensor night in an otherwise stable baseline. With reject OFF the freak is not + // discarded — but Winsorization still CLAMPS it, so it barely moves the center. (Contrast the + // sustained block above, which the same mode follows: single spike damped, sustained shift + // adapted — the whole point of Winsor-without-reject.) + let stable: [Double?] = Array(repeating: 90.0, count: 29) + let withFreak: [Double?] = stable + [180.0] + let base0 = Baselines.foldHistory(stable, cfg: Baselines.hrvCfg, rejectHardOutliers: false) + let base1 = Baselines.foldHistory(withFreak, cfg: Baselines.hrvCfg, rejectHardOutliers: false) + XCTAssertLessThan(abs(base1.baseline - base0.baseline), 8, + "a single 2× freak night must be Winsor-damped, not swing the baseline") + } + + // MARK: - Readiness end-to-end (ln-space HRV, reject off) + + private func d(_ i: Int, hrv: Double?, rhr: Int?) -> DailyMetric { + DailyMetric(day: String(format: "2024-03-%02d", i), totalSleepMin: nil, efficiency: nil, + deepMin: nil, remMin: nil, lightMin: nil, disturbances: nil, restingHr: rhr, + avgHrv: hrv, recovery: nil, strain: 10, exerciseCount: nil, + spo2Pct: nil, skinTempDevC: nil, respRateBpm: nil) + } + + func testReadinessAdaptsToRecentSustainedHRVShift() { + // 12 old nights at 130 ms, then 18 recent nights settled at 85 ms (a sustained shift), today at + // the new normal 85. It must NOT read BAD — the window fold adapts. Under the incremental + // (reject-on) mode this exact case reads BAD (the recent block is rejected, baseline stuck at + // 130), which is the readiness regression RD2 exists to prevent. + var days: [DailyMetric] = [] + for i in 1...12 { days.append(d(i, hrv: 130, rhr: 52)) } + for i in 13...30 { days.append(d(i, hrv: 85, rhr: 52)) } + days.append(d(31, hrv: 85, rhr: 52)) // today: the new normal + let hrv = ReadinessEngine.evaluate(days: days).signals.first { $0.key == "hrv" } + XCTAssertNotNil(hrv) + XCTAssertNotEqual(hrv?.flag, .bad, + "today at the recent sustained normal must not read BAD (the reject-mode failure)") + } + + func testGenuineLowStillFlagsAfterSpineAdoption() { + // Regression: the spine must not blunt a real drop. Stable ~95 ms baseline, today a clear 68 ms. + var days: [DailyMetric] = [] + for i in 1...30 { days.append(d(i, hrv: i % 2 == 0 ? 97 : 93, rhr: 52)) } + days.append(d(31, hrv: 68, rhr: 52)) + let hrv = ReadinessEngine.evaluate(days: days).signals.first { $0.key == "hrv" } + XCTAssertEqual(hrv?.flag, .bad, "a genuine ~28% HRV drop must still read BAD") + } +} From 793b4f57a7961485bc6d56c99880f0ec6259709b Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 17:08:13 -0700 Subject: [PATCH 101/171] =?UTF-8?q?feat(analytics):=20DailyStressEngine=20?= =?UTF-8?q?=E2=80=94=20dual-baseline,=20coupled,=20log-domain=20daily=20st?= =?UTF-8?q?ress?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Research-backed redesign of the daily 0-3 stress score (replacing the inline naive raw-ms mean+SD StressMath), landed as a pure tested engine; wiring StressModel to it (the visible Today-card swap + a ChronicShift UI) is a validated follow-up. Four evidence-backed changes (see docs/scoring-precision/daily-stress-autonomic-redesign.md, cited to Buchheit 2014, Plews, Altini, Hopkins + Oura/WHOOP methodology): - log-domain HRV (lnRMSSD; raw-ms z is statistically invalid, right-skewed) - DUAL baselines: a 14-night adaptive window drives the daily 0-3 (so a sustained shift self-normalizes), a 60-night reference powers a separate ChronicShift readout ('your baseline moved N% for D days') -- the only clean escape from the adapt-vs-hide tradeoff, which neither Oura nor WHOOP publicly does - coupled RHR/HRV (Buchheit quadrant): low HRV is only 'stress' when RHR is ALSO elevated; low HRV + low RHR is benign parasympathetic saturation -> damp (mirrors the #461 guard) - smallest-worthwhile-change dead-zone (|raw|<0.5sigma = noise) Both baselines fold through the RD2 window-fold spine mode (reject-off) so shifts adapt. Validated on 420 nights of real HRV: the dual baseline reframes a permanently-maxed 'high stress' into an adaptive daily number + an honest sustained-shift readout (owner's 2026-07-10 reads 1.50 daily while ChronicShift shows HRV -31% vs the 60d normal). 7 tests pin coupling, saturation-damp, the dual-baseline reframe, SWC, and HRV-only degradation. The doc also specs two future-work items grounded in the same review: a signed one-sided illness/overreach flag (temp/resp concordance + persistence, surfaced separately) and a predictive layer (dual-timescale change-point; NOT per-user deep learning -- shown data-starved/hype for this N-of-1 on-device setting). --- .../StrandAnalytics/DailyStressEngine.swift | 231 ++++++++++++++++++ .../DailyStressEngineTests.swift | 106 ++++++++ .../daily-stress-autonomic-redesign.md | 217 ++++++++++++++++ 3 files changed, 554 insertions(+) create mode 100644 Packages/StrandAnalytics/Sources/StrandAnalytics/DailyStressEngine.swift create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DailyStressEngineTests.swift create mode 100644 docs/scoring-precision/daily-stress-autonomic-redesign.md diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/DailyStressEngine.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/DailyStressEngine.swift new file mode 100644 index 0000000000..eb91d9d11a --- /dev/null +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/DailyStressEngine.swift @@ -0,0 +1,231 @@ +import Foundation +import WhoopStore + +/// Daily autonomic-stress score (0–3) from overnight resting HRV + resting HR, redesigned around the +/// sports-science / wearable-HRV literature (Plews, Buchheit 2014, Altini, Hopkins) rather than a naive +/// mean+SD on raw milliseconds. Pure, deterministic, DB-free. +/// +/// Four evidence-backed departures from the old inline `StressMath`: +/// 1. **Log domain.** HRV (RMSSD) is right-skewed; a symmetric z on raw ms is statistically invalid and +/// outlier-dominated. Everything here z-scores lnRMSSD (Plews/Altini; WHOOP does the same). +/// 2. **Dual baselines.** A SHORT (14-night) adaptive baseline drives today's 0–3, so a sustained shift +/// (a supplement, a training block, altitude) is absorbed into "my recent normal" within ~2 weeks and +/// the daily number self-normalizes instead of reading "high stress" forever. A LONG (60-night) +/// reference powers a SEPARATE `ChronicShift` readout — "your baseline itself has moved down N% for D +/// days" — so the sustained load stays visible instead of being silently defined away. Neither Oura +/// nor WHOOP publicly does this; it is the only clean resolution of the adapt-vs-hide tradeoff. +/// 3. **Coupled RHR+HRV (Buchheit 2014 quadrant).** Low HRV is only "stress" when RHR is ALSO elevated +/// above its own baseline (sympathetic overactivity). Low HRV with LOW RHR is benign parasympathetic +/// saturation (fit / deeply relaxed), not stress — so the HRV penalty is damped there, the exact +/// mirror of the RecoveryScorer's parasympathetic-saturation guard. HRV is the primary weight; RHR +/// confirms. This is what stops a saturated fit night (or a low-HRV-but-calm night) reading as stress. +/// 4. **Smallest-worthwhile-change dead-zone.** A combined deviation under 0.5σ (Hopkins SWC) reads as +/// noise, not stress, so sub-threshold wobble doesn't jitter the daily number. +/// +/// Honest by construction: HRV suppression is *sensitive but not specific* (Altini 2021, n≈9M) — the +/// physiology cannot name the cause (stress vs supplement vs training vs illness). This engine therefore +/// scores the autonomic STATE truthfully and surfaces context (quadrant + chronic shift) rather than +/// pretending to diagnose why. Not medical advice. +public enum DailyStressEngine { + + // MARK: - Tunables (named + auditable; grounded in the design doc / cited literature) + + /// Short adaptive baseline (nights) — drives today's 0–3. 14 sits in the literature's 7–14d "acute" + /// range (ithlete/Plews 7d; a touch longer here to steady a noisy nocturnal wearable signal). + public static let shortWindowDays = 14 + /// Long chronic reference (nights) — powers the sustained-shift readout. 60 matches HRV4Training's + /// normal-values window. + public static let longWindowDays = 60 + /// Minimum baseline nights before the short baseline is trusted enough to score. + public static let minBaselineNights = 7 + + /// HRV carries most of the weight (more sensitive to day-to-day autonomic change — Altini/WHOOP); + /// RHR confirms and, via the coupling below, is decisive when it DIVERGES from HRV. + public static let hrvWeight = 0.6 + public static let rhrWeight = 0.4 + + /// Coupled saturation damp (Buchheit quadrant; mirrors `RecoveryScorer.parasympatheticSaturation`). + /// When HRV is suppressed (zHRV ≥ satEnterZ) but RHR is NOT elevated (zRHR < 0), the low HRV is + /// benign parasympathetic saturation, not stress — damp the HRV term, ramping from no damp at the + /// entry to `satMaxDamp` at `satFullZ` of RHR-below-baseline. Never flips the sign; only reduces. + public static let satEnterZ = 0.5 + public static let satFullZ = 1.5 + public static let satMaxDamp = 0.5 + + /// Smallest-worthwhile-change dead-zone (Hopkins 2000; Buchheit 2014): a combined |raw| under this + /// many σ is noise → reads as baseline (1.5), not stress. + public static let swcBand = 0.5 + + /// A `ChronicShift` is flagged as sustained LOAD only when the recent baseline has moved in the load + /// direction on BOTH channels (HRV down AND RHR up) past these — the coupled sustained-shift gate. + public static let sustainedHRVPctThreshold = 5.0 // recent 14d HRV ≥5% below the 60d normal + public static let sustainedRHRBpmThreshold = 2.0 // recent 14d RHR ≥2 bpm above the 60d normal + + /// Band floor for "high" on the 0–3 scale (matches the app's StressBand.high). + public static let highBandFloor = 2.0 + + // MARK: - Output + + /// Buchheit 2014's HRV×RHR quadrant — the autonomic interpretation behind the number. + public enum Quadrant: String, Sendable, Equatable { + case balanced // no meaningful deviation from the recent baseline + case sympatheticStress // HRV down + RHR up → real autonomic stress / fatigue + case parasympatheticSaturation // HRV down + RHR down → benign (fit / deeply relaxed) + case recovered // HRV up → coping well / well recovered + } + + /// The SEPARATE chronic readout: how far the recent (short) baseline has moved vs the long-term + /// normal, and whether that shift is a coupled sustained LOAD (both channels moved together). + public struct ChronicShift: Sendable, Equatable { + /// + = recent 14-night HRV baseline sits this % BELOW the 60-night normal (suppressed). + public let hrvPctBelowLongTerm: Double + /// + = recent 14-night RHR baseline sits this many bpm ABOVE the 60-night normal (elevated). + public let rhrBpmAboveLongTerm: Double + /// True when BOTH channels have shifted in the load direction past their thresholds — a + /// coupled, physiologically-consistent sustained autonomic load (not one noisy channel). + public let isSustainedLoad: Bool + /// Trailing consecutive nights whose HRV sat below the long-term normal band (a rough "for D + /// days" duration for the readout). + public let daysBelowBand: Int + + public init(hrvPctBelowLongTerm: Double, rhrBpmAboveLongTerm: Double, + isSustainedLoad: Bool, daysBelowBand: Int) { + self.hrvPctBelowLongTerm = hrvPctBelowLongTerm + self.rhrBpmAboveLongTerm = rhrBpmAboveLongTerm + self.isSustainedLoad = isSustainedLoad + self.daysBelowBand = daysBelowBand + } + } + + public struct DailyStress: Sendable, Equatable { + /// 0–3 daily score: today vs the SHORT adaptive baseline, coupled + SWC-damped. 1.5 = baseline. + public let score: Double + /// HRV suppression vs the short baseline (log-domain σ), + = suppressed = toward stress. + public let zHRV: Double + /// RHR elevation vs the short baseline (σ), + = elevated = toward stress. 0 when no RHR. + public let zRHR: Double + /// The Buchheit autonomic quadrant behind the score. + public let quadrant: Quadrant + /// The separate long-term baseline-shift readout (nil when there isn't enough long history). + public let chronicShift: ChronicShift? + /// Baseline-density confidence (calibrating / building / solid). + public let confidence: ScoreConfidence + /// False when RHR was unavailable and the score fell back to HRV-only (no coupling possible). + public let usedRHR: Bool + + public init(score: Double, zHRV: Double, zRHR: Double, quadrant: Quadrant, + chronicShift: ChronicShift?, confidence: ScoreConfidence, usedRHR: Bool) { + self.score = score; self.zHRV = zHRV; self.zRHR = zRHR; self.quadrant = quadrant + self.chronicShift = chronicShift; self.confidence = confidence; self.usedRHR = usedRHR + } + } + + // MARK: - Entry point + + /// Evaluate today's daily stress from the daily-metric history. `days` may be any order; the scored + /// day is `today` (a "yyyy-MM-dd") if given and present, else the latest row carrying HRV/RHR. + /// Returns nil when there is no scorable day or too little baseline. + public static func evaluate(days: [DailyMetric], today: String? = nil) -> DailyStress? { + let sorted = days.sorted { $0.day < $1.day } + let idx: Int? + if let today { idx = sorted.firstIndex { $0.day == today && ($0.avgHrv != nil || $0.restingHr != nil) } } + else { idx = sorted.lastIndex { $0.avgHrv != nil || $0.restingHr != nil } } + guard let i = idx else { return nil } + let scored = sorted[i] + let history = Array(sorted[0.. Double { + guard hrvZ >= satEnterZ, rhrZ < 0 else { return 1.0 } + let rhrLow = -rhrZ + let t = max(0.0, min(1.0, (rhrLow - satEnterZ) / (satFullZ - satEnterZ))) + return 1.0 - satMaxDamp * t + } + + /// Smallest-worthwhile-change dead-zone: shrink |raw| toward zero by `band`, preserving sign. + static func deadzone(_ raw: Double, band: Double) -> Double { + raw >= 0 ? max(0, raw - band) : min(0, raw + band) + } + + /// Logistic squash to 0–3 (raw 0 → 1.5), identical scale/curve to the daytime + old daily score. + static func squash(_ raw: Double) -> Double { min(max(3.0 / (1.0 + exp(-raw)), 0), 3) } + + static func classify(zHRV: Double, zRHR: Double, usedRHR: Bool) -> Quadrant { + if zHRV <= -satEnterZ { return .recovered } // HRV meaningfully ABOVE baseline + guard zHRV >= satEnterZ else { return .balanced } // HRV within the SWC band → balanced + if !usedRHR { return .sympatheticStress } // HRV down, no RHR to disambiguate + return zRHR > 0 ? .sympatheticStress : .parasympatheticSaturation + } + + // MARK: - Chronic shift (short baseline vs long reference) + + static func chronicShift(scored: DailyMetric, history: [DailyMetric], + shortHRVln: BaselineState, longHist: [DailyMetric]) -> ChronicShift? { + guard let longHRVln = foldLnHRV(longHist), longHRVln.usable else { return nil } + // % the recent HRV baseline sits below the long-term (both are geometric centres in ln-space). + let hrvPct = (1.0 - exp(shortHRVln.baseline - longHRVln.baseline)) * 100.0 + let shortRHR = foldRHR(Array(history.suffix(shortWindowDays))) + let longRHR = foldRHR(longHist) + let rhrBpm: Double = (shortRHR?.usable == true && longRHR?.usable == true) + ? shortRHR!.baseline - longRHR!.baseline : 0.0 + let sustained = hrvPct >= sustainedHRVPctThreshold && rhrBpm >= sustainedRHRBpmThreshold + // Duration: trailing nights whose HRV sat below the long-term band (mean − 0.5σ, the SWC edge). + let bandFloorLn = longHRVln.baseline - swcBand * max(1.253 * longHRVln.spread, 1e-9) + var d = 0 + for row in (history + [scored]).reversed() { + guard let h = row.avgHrv else { continue } + if log(max(h, 1.0)) < bandFloorLn { d += 1 } else { break } + } + return ChronicShift(hrvPctBelowLongTerm: hrvPct, rhrBpmAboveLongTerm: rhrBpm, + isSustainedLoad: sustained, daysBelowBand: d) + } + + // MARK: - Baseline folds (reuse the shared window-fold spine: reject OFF so a sustained shift adapts) + + static func foldLnHRV(_ rows: [DailyMetric]) -> BaselineState? { + let ln = rows.map { $0.avgHrv.map { log(max($0, 1.0)) } } + guard ln.contains(where: { $0 != nil }) else { return nil } + return Baselines.foldHistory(ln, cfg: Baselines.readinessHRVLnCfg, rejectHardOutliers: false) + } + + static func foldRHR(_ rows: [DailyMetric]) -> BaselineState? { + let v = rows.map { $0.restingHr.map(Double.init) } + guard v.contains(where: { $0 != nil }) else { return nil } + return Baselines.foldHistory(v, cfg: Baselines.restingHRCfg, rejectHardOutliers: false) + } +} diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DailyStressEngineTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DailyStressEngineTests.swift new file mode 100644 index 0000000000..c450263fa3 --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DailyStressEngineTests.swift @@ -0,0 +1,106 @@ +import XCTest +@testable import StrandAnalytics +import WhoopStore + +/// DailyStressEngine — the redesigned 0–3 daily stress score: log-domain HRV, dual baselines (short +/// adaptive for the number + long reference for a sustained-shift readout), coupled RHR/HRV escalation +/// (Buchheit quadrant), and a smallest-worthwhile-change dead-zone. +final class DailyStressEngineTests: XCTestCase { + + private func dm(_ day: Int, hrv: Double?, rhr: Int?) -> DailyMetric { + DailyMetric(day: String(format: "2025-05-%02d", day), totalSleepMin: nil, efficiency: nil, + deepMin: nil, remMin: nil, lightMin: nil, disturbances: nil, restingHr: rhr, + avgHrv: hrv, recovery: nil, strain: nil, exerciseCount: nil, + spo2Pct: nil, skinTempDevC: nil, respRateBpm: nil) + } + + // MARK: - The coupling (the central design decision) + + func testHRVDownWithRHRUpReadsSympatheticStress() { + // Stable ~92ms / 50bpm baseline, today a clear HRV drop (66) AND RHR rise (60): the true + // autonomic-stress quadrant → high score. + var days: [DailyMetric] = [] + for i in 1...20 { days.append(dm(i, hrv: i % 2 == 0 ? 94 : 90, rhr: i % 2 == 0 ? 51 : 49)) } + days.append(dm(21, hrv: 66, rhr: 60)) + let s = DailyStressEngine.evaluate(days: days)! + XCTAssertEqual(s.quadrant, .sympatheticStress) + XCTAssertGreaterThan(s.score, DailyStressEngine.highBandFloor, "coupled HRV-down+RHR-up is high stress") + } + + func testHRVDownWithRHRLowIsSaturationAndScoresLowerThanCoupledStress() { + // THE coupling test: identical HRV drop (66), but RHR is LOW (44) not high. That's benign + // parasympathetic saturation, not stress — it must classify as saturation AND score clearly + // lower than the same HRV drop with RHR elevated. + func build(rhrToday: Int) -> [DailyMetric] { + var days: [DailyMetric] = [] + for i in 1...20 { days.append(dm(i, hrv: i % 2 == 0 ? 94 : 90, rhr: 50)) } + days.append(dm(21, hrv: 66, rhr: rhrToday)) + return days + } + let saturated = DailyStressEngine.evaluate(days: build(rhrToday: 44))! + let stressed = DailyStressEngine.evaluate(days: build(rhrToday: 60))! + XCTAssertEqual(saturated.quadrant, .parasympatheticSaturation) + XCTAssertLessThan(saturated.score, stressed.score, + "low HRV with LOW RHR (saturation) must read less stressed than low HRV with HIGH RHR") + } + + func testHRVAboveBaselineReadsRecovered() { + var days: [DailyMetric] = [] + for i in 1...20 { days.append(dm(i, hrv: 80, rhr: 52)) } + days.append(dm(21, hrv: 105, rhr: 48)) // HRV well above baseline, RHR below + let s = DailyStressEngine.evaluate(days: days)! + XCTAssertEqual(s.quadrant, .recovered) + XCTAssertLessThan(s.score, 1.5, "a recovered night reads below the 1.5 baseline") + } + + func testSWCDeadzoneKeepsAnOrdinaryNightNeutral() { + // Today sits right at the baseline → within the smallest-worthwhile-change band → ~1.5, not + // nudged off baseline by sub-threshold noise. + var days: [DailyMetric] = [] + for i in 1...20 { days.append(dm(i, hrv: i % 2 == 0 ? 91 : 89, rhr: 50)) } + days.append(dm(21, hrv: 90, rhr: 50)) + let s = DailyStressEngine.evaluate(days: days)! + XCTAssertEqual(s.quadrant, .balanced) + XCTAssertEqual(s.score, 1.5, accuracy: 0.2, "an ordinary night stays at ~baseline") + } + + // MARK: - The dual-baseline reframe (the point of the whole redesign) + + func testSustainedShiftNormalizesTheDailyScoreButFlagsChronicLoad() { + // A sustained autonomic shift: 60 nights at the long-term normal (120 ms / 46 bpm), then 14 + // recent nights settled at a suppressed level (72 ms / 55 bpm), today at that new level. + // The DAILY score must normalize (today ≈ its own recent baseline), while the CHRONIC readout + // must flag the sustained load (recent HRV well below the long-term normal, RHR above it). + var days: [DailyMetric] = [] + var d = 1 + for _ in 0..<60 { days.append(dm(d, hrv: 120, rhr: 46)); d += 1 } + for _ in 0..<14 { days.append(dm(d, hrv: 72, rhr: 55)); d += 1 } + days.append(dm(d, hrv: 72, rhr: 55)) // today: the new normal + let s = DailyStressEngine.evaluate(days: days)! + XCTAssertLessThan(s.score, DailyStressEngine.highBandFloor, + "today at its own recent (shifted) normal must NOT read high stress — the short baseline adapted") + let chronic = s.chronicShift! + XCTAssertGreaterThan(chronic.hrvPctBelowLongTerm, 5, + "the chronic readout must surface HRV suppressed vs the long-term normal") + XCTAssertGreaterThan(chronic.rhrBpmAboveLongTerm, 0, "RHR elevated vs long-term too") + XCTAssertTrue(chronic.isSustainedLoad, "both channels shifted in the load direction → sustained load") + XCTAssertGreaterThan(chronic.daysBelowBand, 5, "the shift has persisted for many nights") + } + + // MARK: - Robustness / degradation + + func testHRVOnlyWhenNoRHR() { + // No RHR anywhere → HRV-only scoring, no coupling, usedRHR false, still produces a score. + var days: [DailyMetric] = [] + for i in 1...20 { days.append(dm(i, hrv: 90, rhr: nil)) } + days.append(dm(21, hrv: 66, rhr: nil)) + let s = DailyStressEngine.evaluate(days: days)! + XCTAssertFalse(s.usedRHR) + XCTAssertGreaterThan(s.score, 1.5, "an HRV drop still reads as elevated when RHR is unavailable") + } + + func testInsufficientBaselineReturnsNil() { + let days = (1...4).map { dm($0, hrv: 90, rhr: 50) } + XCTAssertNil(DailyStressEngine.evaluate(days: days), "fewer than the minimum baseline nights → nil") + } +} diff --git a/docs/scoring-precision/daily-stress-autonomic-redesign.md b/docs/scoring-precision/daily-stress-autonomic-redesign.md new file mode 100644 index 0000000000..0b7baf6479 --- /dev/null +++ b/docs/scoring-precision/daily-stress-autonomic-redesign.md @@ -0,0 +1,217 @@ +# Daily Stress & autonomic scoring — redesign, illness detection, predictive analytics + +Status: **DailyStressEngine implemented + validated** (this file's §3). Illness/overreach flag (§4) and +predictive layer (§5) are **designed, future work**. Grounded in a four-stream literature review +(HRV sports-science; Oura/WHOOP methodology; wearable illness detection; predictive ML), all cited inline. + +--- + +## 0. Thesis + +The old daily Stress score (`StressModel`/`StressMath` in `Strand/Screens/StressView.swift`) summed two +z-scores — `zRHR = (todayRHR − meanRHR)/sdRHR` and `zHRV = (meanHRV − todayHRV)/sdHRV` — computed with a +**naive arithmetic mean + population SD on RAW HRV milliseconds** over a single 30-day window. Four things +are wrong with that, and one deeper problem sits underneath all of them. + +The four mechanical defects (each independently confirmed in the literature): +1. **Raw ms, not log.** RMSSD is right-skewed; a symmetric z on raw ms is statistically invalid and + outlier-dominated. +2. **Naive mean+SD.** No robustness to a freak night; no σ-floor, so a tight baseline saturates the logistic. +3. **Single 30-day window.** Forces the adapt-vs-hide tradeoff (below). +4. **Equal-weight RHR+HRV sum.** Throws away the *divergence* between the two channels, which is the most + informative part. + +The deeper problem — the **confound**: HRV suppression + RHR elevation is genuine autonomic load, but it is +**"sensitive but not specific"** (Altini 2021, n≈9M, [MDPI Sensors](https://www.mdpi.com/1424-8220/21/23/7932)). +A supplement, a training block, altitude, poor sleep, illness, and psychological stress all produce the same +signature. The physiology *cannot name the cause*. So a daily "stress" score that keys on it will read high +every day a user is on a sustained autonomic-shifting influence — which is exactly what NOOP's own owner's +supplement protocol produces (nightly HRV suppressed ~85 ms vs a historical ~120–155, RHR elevated ~55 vs +~50, sustained 6+ weeks). + +--- + +## 1. The adapt-vs-hide tradeoff (the crux) + +A rolling personal baseline has one knob — the window — and it forces a bad choice: + +- **Adapt (short window).** The suppressed level becomes "my recent normal" within ~2 weeks, so the daily + score self-normalizes and stops crying "high stress." **But the sustained load has been defined away** — + the score can no longer tell the user their baseline itself has moved. +- **Don't adapt (long/fixed window).** The score keeps flagging the shift forever — a permanent false + "high stress" for a benign sustained cause. + +Both Oura and WHOOP default to **adapt** (Oura recalibrates "daily … adapts if your baselines change"; +WHOOP's 14-day window absorbs a shift in ~2–3 weeks) and simply accept the blind spot. Their context tags / +journals annotate but **do not feed the score** (Oura Tags, WHOOP Journal drive a *separate* correlational +"Impacts" view). ([Oura Daytime Stress](https://ouraring.com/blog/inside-the-ring-daytime-stress/), +[WHOOP behaviors](https://www.whoop.com/us/en/thelocker/a-new-way-to-see-insights-on-which-behaviors-affect-your-recovery/)) + +**Resolution: dual baselines.** A short adaptive window drives the daily 0–3 (so it normalizes), and a long +reference powers a *separate* honest readout — "your baseline itself has shifted N% for D days." Neither +vendor publicly does this; it is the only clean escape from the tradeoff, and it is what §3 implements. + +--- + +## 2. Research foundation (cited) + +### 2.1 HRV sports-science +- **Log-domain is mandatory.** Raw RMSSD test–retest CV ≈ 30%; lnRMSSD ≈ 3–13% (e.g. 5.4% over 16 wk in + athletes). lnRMSSD is near-normal, making a symmetric z valid. (Plews 2013 [PMID 23852425](https://pubmed.ncbi.nlm.nih.gov/23852425/); + Altini; WHOOP water-polo reliability [PMC9505647](https://pmc.ncbi.nlm.nih.gov/articles/PMC9505647/).) +- **Smallest Worthwhile Change (SWC) = mean ± 0.5 SD** (Hopkins 2000; Buchheit 2014) — inside the band is + noise, don't act. +- **Short vs long window is the right structure for acute-vs-chronic** (Altini): acute = 7d dips below the + 60d band then recovers; chronic = the 60d band itself re-centers, after which the daily z auto-returns to 0. +- **Two windows do two jobs**: 7d = operational baseline; 60d = normal-range band (HRV4Training uses 60d). + +### 2.2 The coupled quadrant (the single most important design decision) +**Buchheit 2014** ([10.3389/fphys.2014.00073](https://www.frontiersin.org/articles/10.3389/fphys.2014.00073/full)) +— interpret lnRMSSD change *jointly* with RHR: + +| lnRMSSD | RHR | meaning | +|---|---|---| +| ↑ | ↓ | coping well / parasympathetic adaptation | +| ↓ | **↑** | **sympathetic overactivity = real stress/fatigue** | +| ↓ | ↓ | **parasympathetic saturation (benign)** — or overtraining only if prolonged | +| ↑ | ↑ | taper / sympathetic reversal | + +Low HRV is only "stress" in the **↓/↑** cell. Low HRV with **low** RHR is benign saturation. This is the exact +logic NOOP's recovery-saturation guard (upstream #461) already uses on Charge — it belongs in Stress too. +HRV should carry most of the weight (more sensitive); RHR is decisive precisely when it *diverges* +(Altini; WHOOP: RHR "adds signal mainly when it trends differently from HRV"). + +### 2.3 Oura / WHOOP (what they actually do) +WHOOP: lnRMSSD in deep sleep vs a 14-day baseline; HRV-primary; 0–3 stress scale. Oura: personal baseline +recalibrated daily, windows undisclosed, "cannot distinguish psychological stress from exercise, caffeine, or +other sympathetic activation" (their own words). Neither publishes the weighting, outlier handling, or a +dual-baseline. Their blogs are ~90% marketing; the rigor is in the HRV literature, not the vendor copy. + +### 2.4 Illness detection → §4. ### 2.5 Predictive ML → §5. + +--- + +## 3. Implemented: `DailyStressEngine` (StrandAnalytics) + +A pure, tested package engine replacing the inline `StressMath`. Four evidence-backed changes: + +1. **Log-domain HRV** — z-scores lnRMSSD via the shared spine's ln config (`readiness_hrv_ln`). +2. **Dual baselines** — a **14-night** adaptive baseline drives the daily 0–3; a **60-night** reference powers + a separate `ChronicShift` readout. Both fold through the RD2 **window-fold spine mode** + (`Baselines.foldHistory(..., rejectHardOutliers: false)`) so a sustained shift *adapts* (isn't rejected) + while a single freak night is Winsor-*damped*. +3. **Coupled escalation** — `saturationDamp(hrvZ:rhrZ:)` mirrors the #461 recovery guard: when HRV is + suppressed but RHR is **not** elevated, the HRV stress term is damped (benign saturation). Weight + HRV 0.6 / RHR 0.4. A `Quadrant` (balanced / sympatheticStress / parasympatheticSaturation / recovered) + carries the interpretation. +4. **SWC dead-zone** — a combined |raw| under 0.5σ reads as baseline (1.5), not stress. + +**`ChronicShift`** reports `hrvPctBelowLongTerm`, `rhrBpmAboveLongTerm`, `isSustainedLoad` (both channels +shifted together — the coupled sustained-load gate), and `daysBelowBand`. + +### Real-data validation (owner's 420-night history, replayed) +- **2026-07-10**: daily score **1.50 (neutral)** — the short baseline had adapted to the suppressed level — + while the chronic readout showed **HRV −31% below the 60-day normal**. The reframe working exactly as + designed: the daily number stops crying "stress"; the sustained load surfaces separately. +- **2026-07-13**: **2.02** — HRV genuinely dipped below the recent baseline, so a fresh elevation is correct. +- **Supplement onset (2026-06-08)**: **2.87** — the acute crash correctly reads high. +- **Coupling proven**: an identical HRV drop reads lower and classifies `parasympatheticSaturation` when RHR + is low, vs `sympatheticStress` when RHR is high. +- **Key finding**: it is the **short adaptive window**, not the log transform, that reframes the number — an + earlier single-30d-window A/B showed log-domain alone was a ~no-op on the owner's score. + +**Honest caveat for the owner specifically**: his recent RHR *is* elevated above his own baseline (≈55 vs +≈50), so he sits in the genuine **↓HRV/↑RHR sympathetic-load** cell — the redesign does not (and should not) +make his number "calm"; it *reframes* the sustained part as a baseline shift and keeps the daily number +honest about fresh dips. + +### Not yet wired (validated follow-up) +`StressModel` still calls the old `StressMath`. Wiring it to `DailyStressEngine` swaps the **primary visible +Today-card number** and should ship *with* a minimal UI surfacing of `ChronicShift` (so a lower daily number +is explained by the "baseline shifted" line), validated on real data. Sequenced deliberately after this +engine + doc land. + +--- + +## 4. Future work — signed illness / overreach flag (M7) + +The current skin-temp / respiration terms in Recovery are **symmetric penalties** (a +0.5 °C fever scores the +same as −0.5 °C benign cooling), discarding the directional illness signal. The research is clear and the fix +is well-specified. + +**Physiology (directional, concordant, ~1–3 days pre-symptom):** skin temp ↑, respiratory rate ↑, RHR ↑, +HRV ↓. Respiratory rate is the **highest-SNR anchor** (baseline varies <1 rpm; illness +2–3 rpm/~17%) and is +rarely moved by alcohol/caffeine/poor sleep — the ideal disambiguator. Temperature adds the most +discriminating power when present. (TemPredict/Oura-UCSF, *Sci Rep* 2022 [PMC8891385](https://pmc.ncbi.nlm.nih.gov/articles/PMC8891385/): +AUC 0.819, sens 82%, **spec 63%**, ~2.75 d lead; Stanford/Snyder *Nat Biomed Eng* 2020 +[PMC9020268](https://pmc.ncbi.nlm.nih.gov/articles/PMC9020268/), real-time CuSum caught 62.5%; Fitbit/Scripps +DETECT *Nat Med* 2021.) + +**Design:** +1. **Signed / one-sided** on temp & RR — only sustained **rises** accrue illness evidence; falls are neutral. +2. **Robust personal baseline** (~14–28 nights, median/MAD), excluding known-confounded nights. +3. **Physiology-anchored thresholds**: temp > +0.3–0.5 °C (above the ~0.3–0.5 °C luteal floor), RR > +1.5–2 + rpm / >10–15%, RHR > +5–7 bpm, HRV a meaningful SD below. +4. **Concordance + persistence gating**: `major` only when ≥3 of {temp↑, RR↑, RHR↑, HRV↓} agree and hold + ≥2 nights; `minor` for one strong/transient signal. Weight temp + RR highest (most infection-specific). +5. **Context attribution, not suppression**: annotate the likely driver (logged supplement, hard training, + travel/altitude, luteal phase) rather than hiding the flag; cycle-adjust temperature (the one benign cause + that mimics fever). +6. **Surface separately** from the score (a badge beside it), never folded in — Oura keeps Symptom Radar + separate from Readiness for exactly this reason. + +**Honesty bar**: even the best peer-reviewed models sit at 60–70% specificity / low PPV (4–10% in independent +validation). Ship as "a signal worth attention," never a diagnosis, and never advertise an unvalidated +lead-time/accuracy number. Much of the naive apparent signal is *behavior change after learning you're sick* +(Cleary 2022, AUC 0.75→0.63 on pre-result-only data). + +This flag **directly sharpens the daily stress score too**: a concordant temp+RR+RHR rise is the one context +where "this sustained load is likely illness/overreach, not a benign supplement" can be said honestly. + +--- + +## 5. Future work — predictive analytics (P0/P1/P2) + +The ML review's blunt conclusion: **for a per-user, on-device, privacy-first, inspectable-math app, the +honest state of the art is transparent statistical anomaly detection on a personal baseline — not deep +learning.** Every vendor "predictive readiness/recovery" score is, on inspection, a **nowcast** (a reweighted +baseline-deviation of last night's data relabeled as prediction). The only genuinely forward-looking task +with replicated evidence — illness 1–3 days out — is best solved by change-point/CuSum methods that run +trivially on-device. + +Evidence that simple beats complex *on this kind of noisy, small-N, per-user series*: M4 Competition (best +pure-ML method less accurate than the worst statistical one); Makridakis 2018; DLinear (a one-layer linear +model beat every Transformer, AAAI 2023). Per-user deep forecasting is data-starved (one user ≈ hundreds–2k +nightly points) and there is **no** evidence it beats a tuned EWMA on nightly HRV. + +**Roadmap:** +- **P0 — dual-timescale trend + change-point (largely DONE via §3).** Fast EWMA (~3–5 d) vs slow baseline + (~28–60 d) of lnRMSSD + RHR, a CV band, and a change-point flag for sustained shifts. `DailyStressEngine`'s + dual baselines + `ChronicShift` already are this; a formal online change-point (CuSum) on the short-vs-long + gap would complete it and give an explicit "your baseline shifted on ~date" event. +- **P1 — personal-baseline early-illness detector (= §4).** Multivariate deviation on RHR + RR + skin temp, + NightSignal/CuSum-style. The one task with real multi-day lead time; transparent methods *are* the SOTA. +- **P2 — optional cold-start prior** (a tiny shipped population prior to bridge the first ~14 days). The one + place a shared/pretrained component is justified; keep it small and swap to the personal baseline ASAP. + +**Explicitly do NOT build**: per-user LSTM/transformer next-day forecasters (data-starved, unfalsifiable, +contradicts inspectable-math), foundation-model "digital twins" (need cloud scale, break privacy-first), or +confident overtraining/illness *classifier* verdicts (science equivocal / PPV too low). This aligns with the +app's inspectable-math philosophy — the constraint is a feature. + +--- + +## 6. Unsettled science (flagged honestly) +- Prospective illness specificity is mediocre (~60–70%); false positives from alcohol/training/travel/luteal. +- CV-of-lnRMSSD direction with fatigue is U-shaped and context-dependent — keep it a *secondary* signal. +- The SWC 0.5 multiplier and EWMA-vs-simple-mean are conventions, not validated optima. +- Nearly all cited thresholds come from morning-supine athlete recordings, not nocturnal wrist averages. +- Saturation vs parasympathetic-overtraining (both ↓HRV/↓RHR) can't be separated without external context. + +--- + +## 7. Sources +See inline links. Anchors: Buchheit 2014 `10.3389/fphys.2014.00073`; Plews 2013 PMID 23852425; Altini 2021 +Sensors `10.3390/s21237932`; Hopkins 2000 (SWC); TemPredict PMC8891385; Snyder PMC9020268; DETECT *Nat Med* +2021; Alavi 2022 *Nat Med* PMC8799466; Quer 2020 DETECT (RHR-alone AUC 0.52); Makridakis 2018; M4 2020; +Zeng 2023 DLinear (arXiv:2205.13504). From e68527c952e8a4537d8357cc1743ebdb591859dc Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 17:58:22 -0700 Subject: [PATCH 102/171] feat(stress): wire the daily Stress score to DailyStressEngine (visible-score swap) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit StressModel now scores via DailyStressEngine (dual-baseline, coupled, log-domain) instead of the naive raw-ms StressMath, keeping StressMath only as a cold-start fallback (< the engine's minimum baseline) and for stored/imported days. The whole trend re-scores each day against its OWN contemporaneous baseline via a new batch evaluateSeries (one sort), fixing the old line that re-scored all history against today's single baseline. The sustained baseline-shift is surfaced as CONTEXT in the explanation, never folded into the number. Two fixes the iOS-simulator validation on real data caught (verified via a temporary trace that the engine, not the fallback, drives the score): - chronic-shift long reference is now a robust MEDIAN over the window, not the recency-weighted EWMA (whose 14-night half-life collapsed the '60-day' reference onto the recent level, reading 0% shift on genuinely-declining data) - dedup multiple same-day source rows (Oura/WHOOP/Apple) to one-per-day so the trailing windows count DAYS not device-rows Validated on real data via --demo-screen: engine drives the score (2.5, correctly classified sympatheticStress for the owner's HRV-down+RHR-up state); UI renders clean. 9 engine tests (adds median-long-reference + device-row-dedup regression pins); full StrandAnalytics suite green (1144); macOS build green. Copy updated off '30-day'. Note: chronic-shift stays quiet for the owner right now because the app keys HRV to the current strap (WHOOP-only, ~12 nights post-swap) — the Oura history isn't merged into repo.days, so there's no long reference yet. It activates as same-device history grows (unit-tested). Whether to merge cross-device HRV history is a separate question. --- .../StrandAnalytics/DailyStressEngine.swift | 75 ++++++++++++++++--- .../DailyStressEngineTests.swift | 29 +++++++ Strand/Screens/StressView.swift | 51 ++++++++++--- 3 files changed, 135 insertions(+), 20 deletions(-) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/DailyStressEngine.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/DailyStressEngine.swift index eb91d9d11a..6776427f71 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/DailyStressEngine.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/DailyStressEngine.swift @@ -126,11 +126,28 @@ public enum DailyStressEngine { /// day is `today` (a "yyyy-MM-dd") if given and present, else the latest row carrying HRV/RHR. /// Returns nil when there is no scorable day or too little baseline. public static func evaluate(days: [DailyMetric], today: String? = nil) -> DailyStress? { - let sorted = days.sorted { $0.day < $1.day } + let sorted = dedupByDay(days.sorted { $0.day < $1.day }) let idx: Int? if let today { idx = sorted.firstIndex { $0.day == today && ($0.avgHrv != nil || $0.restingHr != nil) } } else { idx = sorted.lastIndex { $0.avgHrv != nil || $0.restingHr != nil } } guard let i = idx else { return nil } + return evaluateSorted(sorted, at: i) + } + + /// Batch-score every scorable day in ONE pass — sorts once, so a caller drawing the whole trend pays + /// O(n · window) folds instead of O(n²·log n) re-sorts. Chronological (day, score) pairs. + public static func evaluateSeries(days: [DailyMetric]) -> [(day: String, stress: DailyStress)] { + let sorted = dedupByDay(days.sorted { $0.day < $1.day }) + var out: [(day: String, stress: DailyStress)] = [] + out.reserveCapacity(sorted.count) + for i in sorted.indices where sorted[i].avgHrv != nil || sorted[i].restingHr != nil { + if let s = evaluateSorted(sorted, at: i) { out.append((day: sorted[i].day, stress: s)) } + } + return out + } + + /// Score `sorted[i]` against the days strictly before it. `sorted` MUST be day-ascending. + static func evaluateSorted(_ sorted: [DailyMetric], at i: Int) -> DailyStress? { let scored = sorted[i] let history = Array(sorted[0.. ChronicShift? { - guard let longHRVln = foldLnHRV(longHist), longHRVln.usable else { return nil } - // % the recent HRV baseline sits below the long-term (both are geometric centres in ln-space). - let hrvPct = (1.0 - exp(shortHRVln.baseline - longHRVln.baseline)) * 100.0 - let shortRHR = foldRHR(Array(history.suffix(shortWindowDays))) - let longRHR = foldRHR(longHist) - let rhrBpm: Double = (shortRHR?.usable == true && longRHR?.usable == true) - ? shortRHR!.baseline - longRHR!.baseline : 0.0 + let longLnHRV = longHist.compactMap { $0.avgHrv.map { log(max($0, 1.0)) } } + guard longLnHRV.count >= minBaselineNights else { return nil } + // The long-term reference is a ROBUST MEDIAN centre over the long window — deliberately NOT the + // recency-weighted EWMA the short baseline uses: with the same short half-life the "60-day" EWMA + // collapses onto the recent level and the shift we exist to surface vanishes (it read 0% on real + // gradually-declining data). The short baseline stays the recent EWMA centre; short-vs-long median + // is the sustained shift. + let longCenterLn = median(longLnHRV) + let hrvPct = (1.0 - exp(shortHRVln.baseline - longCenterLn)) * 100.0 + + let shortRHR = history.suffix(shortWindowDays).compactMap { $0.restingHr.map(Double.init) } + let longRHR = longHist.compactMap { $0.restingHr.map(Double.init) } + let rhrBpm: Double = (shortRHR.count >= 3 && longRHR.count >= minBaselineNights) + ? shortRHR.reduce(0, +) / Double(shortRHR.count) - median(longRHR) : 0.0 + let sustained = hrvPct >= sustainedHRVPctThreshold && rhrBpm >= sustainedRHRBpmThreshold - // Duration: trailing nights whose HRV sat below the long-term band (mean − 0.5σ, the SWC edge). - let bandFloorLn = longHRVln.baseline - swcBand * max(1.253 * longHRVln.spread, 1e-9) + // Duration: trailing nights whose HRV sat below the long-term band (median − 0.5·robust σ). + let bandFloorLn = longCenterLn - swcBand * max(1.4826 * mad(longLnHRV), 1e-9) var d = 0 for row in (history + [scored]).reversed() { guard let h = row.avgHrv else { continue } @@ -215,6 +240,36 @@ public enum DailyStressEngine { isSustainedLoad: sustained, daysBelowBand: d) } + /// Median of an unsorted array (0 for empty). + static func median(_ xs: [Double]) -> Double { + guard !xs.isEmpty else { return 0 } + let s = xs.sorted(); let n = s.count + return n % 2 == 1 ? s[n / 2] : (s[n / 2 - 1] + s[n / 2]) / 2 + } + /// Median absolute deviation (robust spread; ×1.4826 ≈ σ for normal data). + static func mad(_ xs: [Double]) -> Double { + let m = median(xs) + return median(xs.map { abs($0 - m) }) + } + + /// Collapse multiple same-day rows (one per source device — Oura / WHOOP / Apple) into ONE row per + /// calendar day, keeping the richest (HRV beats RHR beats neither). `sorted` MUST be day-ascending. + /// Without this, the trailing-window baselines count DEVICE-ROWS, not days, so a "60-day" window can + /// cover ~30 calendar days and the chronic-shift reference never reaches the pre-shift level. + static func dedupByDay(_ sorted: [DailyMetric]) -> [DailyMetric] { + var out: [DailyMetric] = [] + out.reserveCapacity(sorted.count) + func richness(_ m: DailyMetric) -> Int { (m.avgHrv != nil ? 2 : 0) + (m.restingHr != nil ? 1 : 0) } + for row in sorted { + if let last = out.last, last.day == row.day { + if richness(row) > richness(last) { out[out.count - 1] = row } + } else { + out.append(row) + } + } + return out + } + // MARK: - Baseline folds (reuse the shared window-fold spine: reject OFF so a sustained shift adapts) static func foldLnHRV(_ rows: [DailyMetric]) -> BaselineState? { diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DailyStressEngineTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DailyStressEngineTests.swift index c450263fa3..9b544ced4a 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DailyStressEngineTests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/DailyStressEngineTests.swift @@ -103,4 +103,33 @@ final class DailyStressEngineTests: XCTestCase { let days = (1...4).map { dm($0, hrv: 90, rhr: 50) } XCTAssertNil(DailyStressEngine.evaluate(days: days), "fewer than the minimum baseline nights → nil") } + + func testGradualDeclineIsSurfacedByTheMedianLongReference() { + // A slow multi-week HRV decline (120 → 74 over 60 nights), today at the low end. A recency- + // WEIGHTED "long" baseline collapses onto the recent low and reads ~0% (the bug a real device's + // gradually-declining data exposed); the robust MEDIAN long reference captures the decline. + var days: [DailyMetric] = [] + for i in 0..<60 { + days.append(dm(i + 1, hrv: 120.0 - Double(i) * (46.0 / 59.0), rhr: 46)) // 120 → 74 linear + } + days.append(dm(61, hrv: 74, rhr: 46)) + let cs = DailyStressEngine.evaluate(days: days)!.chronicShift! + XCTAssertGreaterThan(cs.hrvPctBelowLongTerm, 10, + "the median long reference must surface a months-long decline that a recency-weighted long baseline would hide") + } + + func testMultipleDeviceRowsPerDayAreDedupedToOnePerDay() { + // Two source rows for the same day (Apple with only RHR, WHOOP with HRV+RHR): the richer row + // wins, and the baseline windows count DAYS not device-rows (else a "60-day" window shrinks). + var days: [DailyMetric] = [] + for i in 1...20 { + days.append(dm(i, hrv: nil, rhr: 55)) // apple-like: RHR only + days.append(dm(i, hrv: 90, rhr: 50)) // whoop-like: richer (HRV present) + } + days.append(dm(21, hrv: 66, rhr: 60)) + let s = DailyStressEngine.evaluate(days: days)! + // If dedup failed, the HRV baseline would still work but be built off far fewer than 20 days; + // the score should still classify a clear HRV-down + RHR-up night as stress. + XCTAssertEqual(s.quadrant, .sympatheticStress) + } } diff --git a/Strand/Screens/StressView.swift b/Strand/Screens/StressView.swift index ce8fd3b670..44ae67939c 100644 --- a/Strand/Screens/StressView.swift +++ b/Strand/Screens/StressView.swift @@ -205,7 +205,7 @@ struct StressView: View { // 2. Today's numbers — uniform tiles in one grid. VStack(alignment: .leading, spacing: NoopMetrics.gap) { - SectionHeader("Today", overline: "Markers", trailing: String(localized: "vs 30-day baseline")) + SectionHeader("Today", overline: "Markers", trailing: String(localized: "vs your baseline")) tileGrid(model) } .staggeredAppear(index: 1) @@ -600,7 +600,7 @@ struct StressView: View { : "Stress is derived from two autonomic signals.") .font(StrandFont.body) .foregroundStyle(StrandPalette.textPrimary) - Text("We compare today's resting heart rate and HRV to your own 30-day baseline. A higher-than-usual resting HR and a lower-than-usual HRV both push the score up, classic signs the body is activated. The combined shift is mapped onto a 0-3 scale: 0 is calm, 1.5 sits at your baseline, 3 is highly activated.") + Text("We compare today's resting heart rate and HRV to your own recent baseline (HRV on a log scale, the way the science recommends). A higher-than-usual resting HR AND a lower-than-usual HRV together are the classic sign of activation — and we only read low HRV as stress when your resting HR is up too, so a deeply-relaxed low-HRV night isn't mistaken for stress. The combined shift is mapped onto a 0-3 scale: 0 is calm, 1.5 sits at your baseline, 3 is highly activated. A sustained shift in your baseline itself is called out separately.") .font(StrandFont.subhead) .foregroundStyle(StrandPalette.textSecondary) .fixedSize(horizontal: false, vertical: true) @@ -766,6 +766,12 @@ struct StressModel { let calmTimeValue: String // e.g. "58%" let calmTimeCaption: String // e.g. "of last 30 days" let usingStored: Bool // true when today's value came from the stored series + /// The Buchheit autonomic quadrant behind today's derived score (nil for a stored/cold-start day). + let quadrant: DailyStressEngine.Quadrant? + /// The sustained baseline-shift readout — surfaced SEPARATELY from the score (a sustained supplement / + /// training / illness load reads as a *shifted baseline*, not acute daily stress). Nil until there's + /// enough long history, or on a stored day. + let chronicShift: DailyStressEngine.ChronicShift? /// Last up-to-14 trend values, for the hero tile sparkline. var sparkValues: [Double] { Array(fullTrend.suffix(14)).map(\.value) } @@ -814,16 +820,26 @@ struct StressModel { let rhrT = today.restingHr.map(Double.init) let hrvT = today.avgHrv - // Resolve today's score: prefer a stored value, else derive. + // Resolve today's score: prefer a stored (imported) value; else the DailyStressEngine + // (dual-baseline, coupled, log-domain); else the legacy StressMath path for a cold-start day the + // engine can't yet score (< its minimum baseline). The engine also yields today's autonomic + // quadrant + the separate chronic-shift readout. + // One batch pass scores every day against its OWN prior baseline (sorts once) — reused for both + // today's headline and the whole trend below. + let engineByDay: [String: DailyStressEngine.DailyStress] = Dictionary( + DailyStressEngine.evaluateSeries(days: days).map { ($0.day, $0.stress) }, + uniquingKeysWith: { a, _ in a }) + let engineToday = engineByDay[today.day] let derivedAvailable = (rhrT != nil && meanRHR != nil) || (hrvT != nil && meanHRV != nil) let storedToday = storedByDay[today.day] - guard storedToday != nil || derivedAvailable else { return nil } + guard storedToday != nil || engineToday != nil || derivedAvailable else { return nil } - let derivedToday: Double? = derivedAvailable + let legacyDerived: Double? = derivedAvailable ? StressMath.squash(StressMath.rawScore( rhrToday: rhrT, meanRHR: meanRHR, sdRHR: sdRHR, hrvToday: hrvT, meanHRV: meanHRV, sdHRV: sdHRV)) : nil + let derivedToday: Double? = engineToday?.score ?? legacyDerived let s = storedToday ?? derivedToday ?? 1.5 self.usingStored = storedToday != nil @@ -833,22 +849,37 @@ struct StressModel { self.hrvToday = hrvT self.rhrDelta = (rhrT != nil && meanRHR != nil) ? (rhrT! - meanRHR!) : nil self.hrvDelta = (hrvT != nil && meanHRV != nil) ? (hrvT! - meanHRV!) : nil + // Quadrant + chronic-shift apply only to an engine-derived score, not a stored/import day. + self.quadrant = storedToday == nil ? engineToday?.quadrant : nil + self.chronicShift = storedToday == nil ? engineToday?.chronicShift : nil - self.explanation = StressMath.explanation( + var explanationText = StressMath.explanation( band: self.band, rhrDelta: self.rhrDelta, hrvDelta: self.hrvDelta, usingStored: self.usingStored ) + // Surface a SUSTAINED baseline shift as CONTEXT — never folded into the number. A low daily + // score sitting on top of a suppressed baseline is a shifted "normal", not calm; and a real + // sustained load stays visible instead of being defined away once the short baseline adapts. + if let cs = self.chronicShift, cs.isSustainedLoad { + explanationText += " Your resting HRV has held about \(Int(cs.hrvPctBelowLongTerm.rounded()))% " + + "below your 60-day normal — a sustained autonomic shift (a supplement, a heavy training " + + "block, or getting run down can all do this), not just today." + } + self.explanation = explanationText - // Full daily proxy history: stored value if present for the day, else the - // z-score derivation against the SAME baseline so the line is comparable. + // Full daily history: a stored value if present, else the engine's score for THAT day — each day + // vs its OWN contemporaneous dual baseline (fixing the old line, which re-scored all history + // against today's single baseline), else the legacy proxy for an early day the engine can't score. var pts: [TrendPoint] = [] for d in days { guard let date = Self.dayParser.date(from: d.day) else { continue } if let v = storedByDay[d.day] { - pts.append(TrendPoint(date: date, value: v)) - continue + pts.append(TrendPoint(date: date, value: v)); continue + } + if let e = engineByDay[d.day] { + pts.append(TrendPoint(date: date, value: e.score)); continue } let dRHR = d.restingHr.map(Double.init) let dHRV = d.avgHrv From 93e6abc4804f4aa192686564d8c2c2d96afc9133 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 19:57:28 -0700 Subject: [PATCH 103/171] Persist whole-night SDNN alongside RMSSD; fix HealthKit SDNN export mislabel MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DailyMetric.avgHrv carries RMSSD for WHOOP but SDNN for Apple — a metric overload flagged on Discord. HealthKitBridge compounded it by exporting the RMSSD value under HealthKit's heartRateVariabilitySDNN identifier, the only HRV type HealthKit exposes. - Add dailyMetric.avgSdnn column (migration v29-daily-avg-sdnn), struct field, upsert, and read. - Compute genuine whole-night SDNN in AnalyticsEngine from the in-bed raw R-R via HRVAnalyzer.analyze().sdnn (Task Force ddof=1), persisted per day. - Apple ingest mirrors its native SDNN into avgSdnn; the macOS import lane maps d.hrvSDNN. - Export prefers avgSdnn under heartRateVariabilitySDNN, falling back to avgHrv only when SDNN is absent (pre-rescore rows, or summary-only Oura with no raw R-R to derive SDNN from). - Thread avgSdnn through the fillingNilFields / sleep-field rebuilders so the new column survives row reconstruction. - Tests: v29 column presence, RMSSD/SDNN round-trip distinctness, upsert update. --- .../StrandAnalytics/AnalyticsEngine.swift | 13 ++++- .../Sources/WhoopStore/Database.swift | 7 +++ .../Sources/WhoopStore/MetricsCache.swift | 23 +++++--- .../WhoopStoreTests/MetricsCacheTests.swift | 57 +++++++++++++++++++ Strand/Data/AppleHealthImport.swift | 3 +- Strand/Data/IntelligenceEngine.swift | 4 +- Strand/Data/Repository.swift | 6 +- StrandiOS/Health/HealthKitBridge.swift | 13 ++++- 8 files changed, 109 insertions(+), 17 deletions(-) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift index 2f87ae48f2..524d4bf3b1 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift @@ -536,6 +536,16 @@ public enum AnalyticsEngine { return weight > 0 ? total / weight : nil }() + // Daily SDNN (ms) = whole-night sample-SD (Task Force ddof=1) over the in-bed R-R across matched + // sessions — the BROAD autonomic-variability metric (both autonomic branches), the slow / + // accumulated-load twin of the vagal RMSSD `avgHRVDaily` above (which is fast, deep-window / vagal). + // Same in-sleep R-R the HRV trace uses; nil when too few clean beats (HRVAnalyzer's own gate). + let avgSDNNDaily: Double? = { + let inBed = rr.filter { r in matched.contains { r.ts >= $0.start && r.ts < $0.end } } + .map { Double($0.rrMs) } + return inBed.isEmpty ? nil : HRVAnalyzer.analyze(rawRR: inBed).sdnn + }() + // ── HRV & Autonomic nightly trace (#141) ────────────────────────────── // Per-5-min-window RMSSD tagged by the sleep stage at its center, then a night summary comparing // NOOP's whole-night mean (what it reports) against a deep-only mean and a WHOOP-style @@ -725,7 +735,8 @@ public enum AnalyticsEngine { steps: stepsTotal, activeKcalEst: activeKcalEst, spo2Red: nightlySpo2Raw?.red, - spo2Ir: nightlySpo2Raw?.ir) + spo2Ir: nightlySpo2Raw?.ir, + avgSdnn: avgSDNNDaily) _ = sleepStart; _ = sleepEnd // available for callers wiring sleep_start/end columns // ── Cache rows ──────────────────────────────────────────────────────── diff --git a/Packages/WhoopStore/Sources/WhoopStore/Database.swift b/Packages/WhoopStore/Sources/WhoopStore/Database.swift index b10e977861..a4347fd431 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/Database.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/Database.swift @@ -587,6 +587,13 @@ extension WhoopStore { t.column("tzId", .text).notNull() // IANA identifier, e.g. "America/Los_Angeles" } } + // v29: persist nightly SDNN (the broad-variability HRV twin of avgHrv=RMSSD). Additive, nullable; + // the nightly analysis already computes SDNN (HRVAnalyzer) but only logged it — now stored. + migrator.registerMigration("v29-daily-avg-sdnn") { db in + try db.alter(table: "dailyMetric") { t in + t.add(column: "avgSdnn", .double) + } + } return migrator } } diff --git a/Packages/WhoopStore/Sources/WhoopStore/MetricsCache.swift b/Packages/WhoopStore/Sources/WhoopStore/MetricsCache.swift index d6a5d9f025..6a75044136 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/MetricsCache.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/MetricsCache.swift @@ -68,19 +68,25 @@ public struct DailyMetric: Equatable, Codable { // call site is unaffected. public let spo2Red: Int? // mean raw red PPG ADC during detected sleep public let spo2Ir: Int? // mean raw IR PPG ADC during detected sleep + /// Nightly SDNN (ms — Task Force whole-night sample SD, ddof=1). The BROAD autonomic-variability + /// twin of `avgHrv` (which is RMSSD for the strap: the fast, vagal, "recovered today?" metric). SDNN + /// captures overall variability across the night — both autonomic branches — a slower, accumulated-load + /// read. v29 column, nullable: WHOOP/on-device nights compute it from the night's R-R (`HRVAnalyzer`); + /// Apple rows mirror their own SDNN reading; Oura/other imports carry no SDNN so it stays nil. + public let avgSdnn: Double? public init(day: String, totalSleepMin: Double?, efficiency: Double?, deepMin: Double?, remMin: Double?, lightMin: Double?, disturbances: Int?, restingHr: Int?, avgHrv: Double?, recovery: Double?, strain: Double?, exerciseCount: Int?, spo2Pct: Double? = nil, skinTempDevC: Double? = nil, respRateBpm: Double? = nil, steps: Int? = nil, activeKcalEst: Double? = nil, - spo2Red: Int? = nil, spo2Ir: Int? = nil) { + spo2Red: Int? = nil, spo2Ir: Int? = nil, avgSdnn: Double? = nil) { self.day = day; self.totalSleepMin = totalSleepMin; self.efficiency = efficiency self.deepMin = deepMin; self.remMin = remMin; self.lightMin = lightMin self.disturbances = disturbances; self.restingHr = restingHr; self.avgHrv = avgHrv self.recovery = recovery; self.strain = strain; self.exerciseCount = exerciseCount self.spo2Pct = spo2Pct; self.skinTempDevC = skinTempDevC; self.respRateBpm = respRateBpm self.steps = steps; self.activeKcalEst = activeKcalEst - self.spo2Red = spo2Red; self.spo2Ir = spo2Ir + self.spo2Red = spo2Red; self.spo2Ir = spo2Ir; self.avgSdnn = avgSdnn } /// The freshest STRICTLY-PRIOR day that carries at least one overnight vital (HRV / resting HR / @@ -381,8 +387,8 @@ extension WhoopStore { (deviceId, day, totalSleepMin, efficiency, deepMin, remMin, lightMin, disturbances, restingHr, avgHrv, recovery, strain, exerciseCount, spo2Pct, skinTempDevC, respRateBpm, steps, activeKcalEst, - spo2Red, spo2Ir) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + spo2Red, spo2Ir, avgSdnn) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(deviceId, day) DO UPDATE SET totalSleepMin = excluded.totalSleepMin, efficiency = excluded.efficiency, @@ -401,13 +407,14 @@ extension WhoopStore { steps = excluded.steps, activeKcalEst = excluded.activeKcalEst, spo2Red = excluded.spo2Red, - spo2Ir = excluded.spo2Ir + spo2Ir = excluded.spo2Ir, + avgSdnn = excluded.avgSdnn """, arguments: [deviceId, d.day, d.totalSleepMin, d.efficiency, d.deepMin, d.remMin, d.lightMin, d.disturbances, d.restingHr, d.avgHrv, d.recovery, d.strain, d.exerciseCount, d.spo2Pct, d.skinTempDevC, d.respRateBpm, d.steps, d.activeKcalEst, - d.spo2Red, d.spo2Ir]) + d.spo2Red, d.spo2Ir, d.avgSdnn]) n += db.changesCount } return n @@ -458,7 +465,7 @@ extension WhoopStore { SELECT day, totalSleepMin, efficiency, deepMin, remMin, lightMin, disturbances, restingHr, avgHrv, recovery, strain, exerciseCount, spo2Pct, skinTempDevC, respRateBpm, steps, activeKcalEst, - spo2Red, spo2Ir FROM dailyMetric + spo2Red, spo2Ir, avgSdnn FROM dailyMetric WHERE deviceId = ? AND day >= ? AND day <= ? ORDER BY day ASC """, arguments: [deviceId, from, to]) @@ -472,7 +479,7 @@ extension WhoopStore { spo2Pct: $0["spo2Pct"], skinTempDevC: $0["skinTempDevC"], respRateBpm: $0["respRateBpm"], steps: $0["steps"], activeKcalEst: $0["activeKcalEst"], - spo2Red: $0["spo2Red"], spo2Ir: $0["spo2Ir"]) + spo2Red: $0["spo2Red"], spo2Ir: $0["spo2Ir"], avgSdnn: $0["avgSdnn"]) } } } diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/MetricsCacheTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/MetricsCacheTests.swift index c8c72d02e7..3485f101eb 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/MetricsCacheTests.swift +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/MetricsCacheTests.swift @@ -503,6 +503,63 @@ final class MetricsCacheTests: XCTestCase { XCTAssertNil(bareRow.activeKcalEst) } + // MARK: - v29 whole-night SDNN column (avgSdnn, alongside avgHrv=RMSSD) + + func testV29AvgSdnnColumnPresent() async throws { + let store = try await WhoopStore.inMemory() + let cols = try await store.columnNamesForTest(table: "dailyMetric") + XCTAssertTrue(cols.contains("avgSdnn"), "dailyMetric missing v29 avgSdnn column") + } + + func testV29AvgSdnnRoundTrip() async throws { + let store = try await WhoopStore.inMemory() + // avgHrv carries RMSSD; avgSdnn carries the whole-night SDNN — distinct values on the same row. + let d = DailyMetric(day: "2026-05-29", totalSleepMin: 415, efficiency: 0.9, + deepMin: 88, remMin: 108, lightMin: 219, disturbances: 2, + restingHr: 50, avgHrv: 62.0, recovery: 0.73, strain: 10.5, + exerciseCount: 1, spo2Pct: 96.2, skinTempDevC: 0.0, respRateBpm: 14.7, + steps: 7_900, activeKcalEst: 2_180.0, avgSdnn: 88.4) + try await store.upsertDailyMetrics([d], deviceId: "devA") + + let rows = try await store.dailyMetrics(deviceId: "devA", from: "2026-05-01", to: "2026-05-31") + XCTAssertEqual(rows.count, 1) + let row = try XCTUnwrap(rows.first) + XCTAssertEqual(try XCTUnwrap(row.avgHrv), 62.0, accuracy: 0.001, "RMSSD must survive intact") + XCTAssertEqual(try XCTUnwrap(row.avgSdnn), 88.4, accuracy: 0.001, "SDNN must persist distinctly") + + // Omitting avgSdnn keeps it nil (defaulted init — old call sites unchanged). + let bare = DailyMetric(day: "2026-05-30", totalSleepMin: nil, efficiency: nil, + deepMin: nil, remMin: nil, lightMin: nil, disturbances: nil, + restingHr: nil, avgHrv: 55.0, recovery: nil, strain: nil, exerciseCount: nil) + try await store.upsertDailyMetrics([bare], deviceId: "devA") + let bareRows = try await store.dailyMetrics(deviceId: "devA", from: "2026-05-30", to: "2026-05-30") + let bareRow = try XCTUnwrap(bareRows.first) + XCTAssertEqual(try XCTUnwrap(bareRow.avgHrv), 55.0, accuracy: 0.001) + XCTAssertNil(bareRow.avgSdnn, "avgSdnn defaults to nil when the source has no whole-night SDNN") + } + + func testV29AvgSdnnUpsertUpdates() async throws { + let store = try await WhoopStore.inMemory() + // Insert with nil SDNN, then re-upsert the same day with SDNN populated (WHOOP backfill path). + let d1 = DailyMetric(day: "2026-05-31", totalSleepMin: 400, efficiency: 0.88, + deepMin: 80, remMin: 100, lightMin: 220, disturbances: 3, + restingHr: 54, avgHrv: 60.0, recovery: 0.65, strain: 13.0, exerciseCount: 0) + try await store.upsertDailyMetrics([d1], deviceId: "devA") + let preRows = try await store.dailyMetrics(deviceId: "devA", from: "2026-05-31", to: "2026-05-31") + var row = try XCTUnwrap(preRows.first) + XCTAssertNil(row.avgSdnn) + + let d2 = DailyMetric(day: "2026-05-31", totalSleepMin: 400, efficiency: 0.88, + deepMin: 80, remMin: 100, lightMin: 220, disturbances: 3, + restingHr: 54, avgHrv: 60.0, recovery: 0.65, strain: 13.0, exerciseCount: 0, + avgSdnn: 91.2) + try await store.upsertDailyMetrics([d2], deviceId: "devA") + let rows = try await store.dailyMetrics(deviceId: "devA", from: "2026-05-31", to: "2026-05-31") + XCTAssertEqual(rows.count, 1, "upsert must not duplicate") + row = try XCTUnwrap(rows.first) + XCTAssertEqual(try XCTUnwrap(row.avgSdnn), 91.2, accuracy: 0.001) + } + // MARK: - read highwater cursor (distinct prefix from upload highwater) func testReadHighwaterRoundTripsUnderDistinctPrefix() async throws { diff --git a/Strand/Data/AppleHealthImport.swift b/Strand/Data/AppleHealthImport.swift index 7163c304bd..8d9a74c27c 100644 --- a/Strand/Data/AppleHealthImport.swift +++ b/Strand/Data/AppleHealthImport.swift @@ -48,7 +48,8 @@ enum AppleHealthImport { // #89: Apple Health steps must land in DailyMetric.steps too — the sourced-daily // arbitration resolves "steps" via metricValue(d) = d.steps, so leaving it nil (the // pre-fix state) meant imported Apple steps never surfaced there. - steps: d.steps.map { Int($0) }) + steps: d.steps.map { Int($0) }, + avgSdnn: d.hrvSDNN) // Apple HRV is SDNN — mirror into the SDNN field } let dmWritten = try await store.upsertDailyMetrics(dm, deviceId: deviceId) diff --git a/Strand/Data/IntelligenceEngine.swift b/Strand/Data/IntelligenceEngine.swift index 7aa61fc535..f298dd6b00 100644 --- a/Strand/Data/IntelligenceEngine.swift +++ b/Strand/Data/IntelligenceEngine.swift @@ -1854,7 +1854,7 @@ private extension DailyMetric { avgHrv: avgHrv, recovery: r, strain: strain, exerciseCount: exerciseCount, spo2Pct: spo2Pct, skinTempDevC: sd, respRateBpm: respRateBpm, steps: steps, activeKcalEst: activeKcalEst, - spo2Red: spo2Red, spo2Ir: spo2Ir) + spo2Red: spo2Red, spo2Ir: spo2Ir, avgSdnn: avgSdnn) } /// Rebuild with substituted sleep-derived fields (a user-corrected wake window), leaving every @@ -1865,7 +1865,7 @@ private extension DailyMetric { disturbances: disturbances, restingHr: restingHr, avgHrv: avgHrv, recovery: recovery, strain: strain, exerciseCount: exerciseCount, spo2Pct: spo2Pct, skinTempDevC: skinTempDevC, respRateBpm: respRateBpm, steps: steps, - activeKcalEst: activeKcalEst, spo2Red: spo2Red, spo2Ir: spo2Ir) + activeKcalEst: activeKcalEst, spo2Red: spo2Red, spo2Ir: spo2Ir, avgSdnn: avgSdnn) } } diff --git a/Strand/Data/Repository.swift b/Strand/Data/Repository.swift index ae39997519..464336f781 100644 --- a/Strand/Data/Repository.swift +++ b/Strand/Data/Repository.swift @@ -2544,7 +2544,8 @@ private extension DailyMetric { // Raw SpO2 is on-device only (imports never carry it), so the imported row's nil is // backfilled from the computed fallback — otherwise the nightly means would be lost. (#93) spo2Red: spo2Red ?? fallback.spo2Red, - spo2Ir: spo2Ir ?? fallback.spo2Ir + spo2Ir: spo2Ir ?? fallback.spo2Ir, + avgSdnn: avgSdnn ?? fallback.avgSdnn ) } @@ -2572,7 +2573,8 @@ private extension DailyMetric { steps: steps, activeKcalEst: activeKcalEst, spo2Red: spo2Red, // non-sleep field: preserved as-is (#93) - spo2Ir: spo2Ir + spo2Ir: spo2Ir, + avgSdnn: avgSdnn // non-sleep (HRV) field: preserved as-is ) } } diff --git a/StrandiOS/Health/HealthKitBridge.swift b/StrandiOS/Health/HealthKitBridge.swift index 7fccc2371d..7114957f82 100644 --- a/StrandiOS/Health/HealthKitBridge.swift +++ b/StrandiOS/Health/HealthKitBridge.swift @@ -397,7 +397,8 @@ final class HealthKitBridge: ObservableObject { deepMin: a.deepMin, remMin: a.remMin, lightMin: a.coreMin, disturbances: nil, restingHr: a.restingHr.map { Int($0.rounded()) }, avgHrv: a.hrv, recovery: nil, strain: nil, exerciseCount: nil, - spo2Pct: a.spo2, skinTempDevC: nil, respRateBpm: a.respRate) + spo2Pct: a.spo2, skinTempDevC: nil, respRateBpm: a.respRate, + avgSdnn: a.hrv) // Apple's HRV IS SDNN — mirror it into the SDNN field too } // Flatten to the generic metricSeries the shared Apple Health screen, the Today apple-health // sparklines, and the Metric Explorer read from — repo.series(key:source:"apple-health") @@ -570,8 +571,14 @@ final class HealthKitBridge: ObservableObject { if let rhr = row.restingHr { add(.restingHeartRate, HKUnit.count().unitDivided(by: .minute()), Double(rhr), row.day, at) } - if let hrv = row.avgHrv { - add(.heartRateVariabilitySDNN, .secondUnit(with: .milli), hrv, row.day, at) + // Export the GENUINE SDNN (v29) when present — the strap's `avgHrv` is RMSSD, which HealthKit + // has no field for, so writing it under `.heartRateVariabilitySDNN` mislabels it. WHOOP rows + // backfill `avgSdnn` from stored raw R-R on the next re-score; Apple rows' `avgHrv` already IS + // SDNN. The `avgHrv` fallback fires for those two before re-scoring, and permanently for + // summary-only sources (Oura) that give RMSSD with no raw R-R to derive SDNN from — HealthKit's + // single HRV type leaves no better label there. + if let sdnn = row.avgSdnn ?? row.avgHrv { + add(.heartRateVariabilitySDNN, .secondUnit(with: .milli), sdnn, row.day, at) } if let spo2 = row.spo2Pct { add(.oxygenSaturation, .percent(), spo2 / 100, row.day, at) From 2d409cd8b4a9c18bfe70c7d9f6379a48f3b1215e Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 19:58:57 -0700 Subject: [PATCH 104/171] =?UTF-8?q?Add=20CrossDeviceHRVTrend:=20stitch=20A?= =?UTF-8?q?ura=E2=86=92WHOOP=20HRV=20history=20into=20one=20continuous=20t?= =?UTF-8?q?rend?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A wearer with years of Oura history continuing onto a WHOOP strap has two incompatible RMSSD scales (Oura ~120-155 ms vs WHOOP ~72-112 ms) with no overlap nights to calibrate against. Averaging them is dishonest; showing only the current strap throws away the long trend the wearer cares about. CrossDeviceHRVTrend resolves this for a DISPLAY/trend lens (scores stay strictly per-device via deviceEraEpoch): each night carries both its RAW value (scale-honest per device) and an era-relative deviation from that device-era's own geometric (median-of-ln) centre. Plotting the era-relative series gives a continuous trend SHAPE across the switch; era boundaries are marked so the UI can show "switched to WHOOP". Pure, deterministic, DB-free. Segments source-tagged nightly HRV into contiguous single-brand eras (reuses Baselines.brandBucket), normalises each night against its era's median ln. 4 tests: single-era, two-era stitch, boundary marking, empty. --- .../StrandAnalytics/CrossDeviceHRVTrend.swift | 103 ++++++++++++++++++ .../CrossDeviceHRVTrendTests.swift | 57 ++++++++++ 2 files changed, 160 insertions(+) create mode 100644 Packages/StrandAnalytics/Sources/StrandAnalytics/CrossDeviceHRVTrend.swift create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/CrossDeviceHRVTrendTests.swift diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/CrossDeviceHRVTrend.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/CrossDeviceHRVTrend.swift new file mode 100644 index 0000000000..2832e2bff6 --- /dev/null +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/CrossDeviceHRVTrend.swift @@ -0,0 +1,103 @@ +import Foundation + +/// Cross-device HRV long-term trend — stitch a wearer's full nightly HRV history across a device switch +/// (e.g. a 2-year Oura ring history continuing onto a WHOOP strap) into ONE continuous long-term view, +/// WITHOUT falsely claiming two brands' incompatible RMSSD scales are the same number (Oura RMSSD +/// ~120–155 ms vs WHOOP ~72–112 ms, with no overlap nights to calibrate against — see +/// `Baselines.deviceEraEpoch`, #459). +/// +/// The honest resolution: each night carries BOTH its RAW value (what that device actually measured) and +/// an ERA-RELATIVE % (how far it sits above/below that device-era's own robust centre). Plotting the +/// era-relative series gives a continuous trend SHAPE across the switch — the Oura decline flows straight +/// into the WHOOP era — while the raw series stays scale-honest per device and boundaries are marked so the +/// user knows the sensor changed. This is a DISPLAY/trend lens only; the 0–3 scores stay strictly +/// per-device (via `deviceEraEpoch`). Pure, deterministic, DB-free. +public enum CrossDeviceHRVTrend { + + /// One night on the stitched trend. + public struct Point: Equatable, Sendable { + /// "yyyy-MM-dd". + public let day: String + /// Absolute nightly HRV (ms) exactly as this device reports it — scale differs across eras. + public let raw: Double + /// % above/below this device-era's own robust (median-of-ln) centre. Cross-era comparable in + /// SHAPE (not absolute level), so the long trend reads continuously through the switch. + public let eraRelativePct: Double + /// Coarse HRV-scale brand ("oura"/"whoop"/…) via `Baselines.brandBucket` — the era this night is in. + public let brand: String + /// True on the first night of a NON-FIRST era — i.e. a device switch — so the UI can draw a + /// boundary marker ("switched to WHOOP"). Never true for the very first era's first night. + public let isEraStart: Bool + + public init(day: String, raw: Double, eraRelativePct: Double, brand: String, isEraStart: Bool) { + self.day = day; self.raw = raw; self.eraRelativePct = eraRelativePct + self.brand = brand; self.isEraStart = isEraStart + } + } + + /// A contiguous single-brand stretch of the history, with its own centre — the unit the trend + /// normalises against, and what a UI legend/segment can label ("Oura · 408 nights"). + public struct Era: Equatable, Sendable { + public let brand: String + public let startDay: String + public let endDay: String + public let nights: Int + /// Geometric (median-of-ln) centre HRV of the era, in ms. + public let centerMs: Double + public init(brand: String, startDay: String, endDay: String, nights: Int, centerMs: Double) { + self.brand = brand; self.startDay = startDay; self.endDay = endDay + self.nights = nights; self.centerMs = centerMs + } + } + + public struct Result: Equatable, Sendable { + public let points: [Point] + public let eras: [Era] + public init(points: [Point], eras: [Era]) { self.points = points; self.eras = eras } + } + + /// Build the continuous cross-device trend from source-tagged nightly HRV — one WINNING source per + /// day (the same per-day merge winner the fold uses; NOT one row per source — mirrors the + /// `deviceEraEpoch` contract). Any order; sorted here. Nights are grouped into CONTIGUOUS single-brand + /// eras (a device switch opens a new era); each era normalises against the median of its own ln(HRV). + /// Single-brand history → one era, every `eraRelativePct` vs that one centre (still valid). Empty → empty. + public static func build(_ sourceDays: [(day: String, sourceId: String, hrv: Double)]) -> Result { + let sorted = sourceDays.filter { $0.hrv > 0 } + .sorted { $0.day != $1.day ? $0.day < $1.day : $0.sourceId < $1.sourceId } + guard !sorted.isEmpty else { return Result(points: [], eras: []) } + + // Segment into contiguous same-brand eras. + var eraRows: [[(day: String, sourceId: String, hrv: Double)]] = [] + var cur: [(day: String, sourceId: String, hrv: Double)] = [] + var curBrand: String? = nil + for row in sorted { + let b = Baselines.brandBucket(row.sourceId) + if let cb = curBrand, cb != b { eraRows.append(cur); cur = [] } + curBrand = b + cur.append(row) + } + if !cur.isEmpty { eraRows.append(cur) } + + var points: [Point] = [] + var eras: [Era] = [] + for (ei, era) in eraRows.enumerated() { + let brand = Baselines.brandBucket(era[0].sourceId) + let centerLn = median(era.map { log($0.hrv) }) + eras.append(Era(brand: brand, startDay: era.first!.day, endDay: era.last!.day, + nights: era.count, centerMs: exp(centerLn))) + for (i, row) in era.enumerated() { + points.append(Point(day: row.day, raw: row.hrv, + eraRelativePct: (log(row.hrv) - centerLn) * 100.0, + brand: brand, + isEraStart: ei > 0 && i == 0)) + } + } + return Result(points: points, eras: eras) + } + + static func median(_ xs: [Double]) -> Double { + let s = xs.sorted(); let n = s.count + guard n > 0 else { return 0 } + return n % 2 == 1 ? s[n / 2] : (s[n / 2 - 1] + s[n / 2]) / 2 + } +} diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/CrossDeviceHRVTrendTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/CrossDeviceHRVTrendTests.swift new file mode 100644 index 0000000000..6c8076ce0d --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/CrossDeviceHRVTrendTests.swift @@ -0,0 +1,57 @@ +import XCTest +@testable import StrandAnalytics + +/// CrossDeviceHRVTrend — stitch a full HRV history across a device switch (Oura → WHOOP) into one +/// continuous trend: per-era-relative shape (continuous through the switch) + raw values (scale-honest +/// per device) + boundary markers. Scoring stays per-device; this is the display/trend lens. +final class CrossDeviceHRVTrendTests: XCTestCase { + + private func sd(_ day: String, _ src: String, _ hrv: Double) -> (day: String, sourceId: String, hrv: Double) { + (day: day, sourceId: src, hrv: hrv) + } + + func testSingleBrandHistoryIsOneEraNoBoundary() { + let rows = (1...20).map { sd(String(format: "2026-06-%02d", $0), "my-whoop", 90 + Double($0 % 5)) } + let r = CrossDeviceHRVTrend.build(rows) + XCTAssertEqual(r.eras.count, 1) + XCTAssertEqual(r.eras[0].brand, "whoop") + XCTAssertFalse(r.points.contains { $0.isEraStart }, "a single-brand history has no device-switch boundary") + } + + func testOuraToWhoopSwitchMarksBoundaryAndNormalisesPerEra() { + // Oura era ~131 ms, then a WHOOP era ~91 ms — incompatible scales, no overlap. + var rows: [(day: String, sourceId: String, hrv: Double)] = [] + for i in 1...15 { rows.append(sd(String(format: "2026-05-%02d", i), "oura-api", 130 + Double(i % 4))) } + for i in 1...15 { rows.append(sd(String(format: "2026-06-%02d", i), "my-whoop-noop", 90 + Double(i % 4))) } + let r = CrossDeviceHRVTrend.build(rows) + XCTAssertEqual(r.eras.map { $0.brand }, ["oura", "whoop"]) + XCTAssertEqual(r.eras[0].nights, 15) + XCTAssertEqual(r.eras[1].nights, 15) + // Exactly one boundary, on the first WHOOP night — never on the very first night of history. + let starts = r.points.filter { $0.isEraStart } + XCTAssertEqual(starts.count, 1) + XCTAssertEqual(starts.first?.brand, "whoop") + XCTAssertEqual(starts.first?.day, "2026-06-01") + // Per-era normalisation: a typical WHOOP night reads ~0% vs its OWN era, NOT "~31% below" the Oura scale. + let whoopMid = r.points.first { $0.day == "2026-06-08" }! + XCTAssertLessThan(abs(whoopMid.eraRelativePct), 5) + XCTAssertEqual(r.eras[0].centerMs, 131, accuracy: 3) + XCTAssertEqual(r.eras[1].centerMs, 91, accuracy: 3) + } + + func testEraRelativeShapeIsContinuousAcrossTheSwitch() { + // A declining Oura era continues declining on WHOOP: the era-relative series keeps falling across + // the boundary (continuous shape) though the raw scale steps down. + var rows: [(day: String, sourceId: String, hrv: Double)] = [] + for i in 1...20 { rows.append(sd(String(format: "2026-05-%02d", i), "oura-api", 150 - Double(i))) } + for i in 1...20 { rows.append(sd(String(format: "2026-06-%02d", i), "my-whoop", 100 - Double(i))) } + let r = CrossDeviceHRVTrend.build(rows) + XCTAssertLessThan(r.points.last { $0.brand == "oura" }!.eraRelativePct, 0) + XCTAssertLessThan(r.points.last!.eraRelativePct, 0, "the decline continues into the WHOOP era") + XCTAssertGreaterThan(r.eras[0].centerMs, r.eras[1].centerMs, "raw scale steps down at the switch") + } + + func testEmptyInput() { + XCTAssertTrue(CrossDeviceHRVTrend.build([]).points.isEmpty) + } +} From cbce2e83c93746b0229df6c19b5756c9424577e3 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 20:27:05 -0700 Subject: [PATCH 105/171] =?UTF-8?q?Add=20Cross-Device=20HRV=20screen:=20th?= =?UTF-8?q?e=20stitched=20Aura=E2=86=92WHOOP=20long-term=20trend?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wires CrossDeviceHRVTrend into a dedicated screen (peer of Trends, reachable from the macOS sidebar Insights group and the iOS More → Insights list). - Repository.crossDeviceHrvHistory(): assembles one winning nightly HRV per day across all wearable sources (Oura under "oura-api", WHOOP under the strap/computed ids), tagged with the real source id so the engine segments eras by brand. apple-health is excluded — its SDNN is method-incompatible with the strap/Oura RMSSD and would fragment a wearable era. - CrossDeviceHRVView: a hero era-relative trend (one continuous line vs each device-era's own log-scale centre, so the shape carries across the switch; robust p90 Y-clamp keeps known Oura nap-fragment artifacts from blowing out the axis), a scale-honest raw chart (per-era coloured, never averaged), an era legend, and a methodology card. A dashed boundary marks the switch. - Demo case `--demo-screen crossdevicehrv` for deterministic screenshotting. Validated on real data: 420 nights (408 Oura 2024-06→2026-06 + 12 WHOOP), two eras, one boundary, the trend legible around each era's own normal. --- Strand/App/RootView.swift | 7 +- Strand/Data/Repository.swift | 34 +++ Strand/Screens/CrossDeviceHRVView.swift | 282 ++++++++++++++++++++++++ StrandiOS/App/RootTabView.swift | 4 +- StrandiOS/App/StrandiOSApp.swift | 1 + 5 files changed, 326 insertions(+), 2 deletions(-) create mode 100644 Strand/Screens/CrossDeviceHRVView.swift diff --git a/Strand/App/RootView.swift b/Strand/App/RootView.swift index d952ad27f2..e4db309a01 100644 --- a/Strand/App/RootView.swift +++ b/Strand/App/RootView.swift @@ -17,6 +17,7 @@ enum NavItem: String, CaseIterable, Identifiable, Hashable { case workouts = "Workouts" case health = "Health" case stress = "Stress" + case crossDeviceHRV = "Cross-Device HRV" case labBook = "Lab Book" case rhythm = "Rhythm" case appleHealth = "Apple Health" @@ -52,6 +53,7 @@ enum NavItem: String, CaseIterable, Identifiable, Hashable { case .workouts: return "Workouts" case .health: return "Health" case .stress: return "Stress" + case .crossDeviceHRV: return "Cross-Device HRV" case .labBook: return "Lab Book" case .rhythm: return "Rhythm" case .appleHealth: return "Apple Health" @@ -94,6 +96,7 @@ enum NavItem: String, CaseIterable, Identifiable, Hashable { case .workouts: return String(localized: "Workouts") case .health: return String(localized: "Health") case .stress: return String(localized: "Stress") + case .crossDeviceHRV: return String(localized: "Cross-Device HRV") case .labBook: return String(localized: "Lab Book") case .rhythm: return String(localized: "Rhythm") case .appleHealth: return String(localized: "Apple Health") @@ -128,6 +131,7 @@ enum NavItem: String, CaseIterable, Identifiable, Hashable { case .workouts: return "figure.run" case .health: return "heart.text.square.fill" case .stress: return "gauge.with.dots.needle.50percent" + case .crossDeviceHRV: return "waveform.path.ecg.rectangle" case .labBook: return "books.vertical.fill" case .rhythm: return "waveform.path" case .appleHealth: return "heart.fill" @@ -170,7 +174,7 @@ struct NavGroup: Identifiable { // all collapse under this single Insights group rather than scattering across the flat list. NavGroup(title: "Insights", id: "insights", items: [ .intelligence, .insightsHub, .coach, .explore, .compare, .insights, - .labBook, .rhythm, .trends, + .labBook, .rhythm, .trends, .crossDeviceHRV, ]), NavGroup(title: "Data & App", id: "data_app", items: [ .devices, .dataSources, .appleHealth, .xiaomi, .backupSync, .fusedRecord, @@ -442,6 +446,7 @@ struct RootView: View { case .workouts: WorkoutsView() case .health: HealthView() case .stress: StressView() + case .crossDeviceHRV: CrossDeviceHRVView() case .labBook: LabBookView() case .rhythm: RhythmHost() case .appleHealth: AppleHealthView() diff --git a/Strand/Data/Repository.swift b/Strand/Data/Repository.swift index 464336f781..832a56436a 100644 --- a/Strand/Data/Repository.swift +++ b/Strand/Data/Repository.swift @@ -850,6 +850,40 @@ final class Repository: ObservableObject { return await unionDailyMetrics(store: store, from: fromDay, to: toDay) } + /// Full-history nightly HRV tagged by its WINNING source id, for the cross-device HRV trend + /// (`CrossDeviceHRVTrend`). One winner per day: the real wearable of each era (Oura under "oura-api", + /// WHOOP under the strap/import/computed ids) — the real id strings are preserved so the engine's + /// `brandBucket` segments the Oura era from the WHOOP era. `apple-health` is deliberately EXCLUDED: + /// Apple's HRV is SDNN, method-incompatible with the strap/Oura RMSSD this trend plots (the #696 + /// method-mixing hazard), and its `brandBucket` "whoop" tag would fragment a wearable era on any day it + /// backfilled. Oldest→newest; only nights with a positive HRV. + func crossDeviceHrvHistory(from: String = "2018-01-01", + to: String = "9999-12-31") async + -> [(day: String, sourceId: String, hrv: Double)] { + guard let store = await ensureStore() else { return [] } + // Priority: live strap + canonical import first, then wearable imports (Oura/Fitbit/Garmin), then + // the computed "-noop" sibling as a fallback. All whoop-family ids bucket to "whoop", so intra-era + // precedence only picks which value represents a shared WHOOP-era day; Oura owns its own era. + var ordered = importedReadIds + ordered.append(contentsOf: Self.wearableImportSources) + ordered.append(contentsOf: computedReadIds) + var seen = Set() + let ids = ordered.filter { seen.insert($0).inserted } + + var winnerByDay: [String: (rank: Int, sourceId: String, hrv: Double)] = [:] + for (rank, id) in ids.enumerated() { + let rows = (try? await store.dailyMetrics(deviceId: id, from: from, to: to)) ?? [] + for r in rows { + guard let hrv = r.avgHrv, hrv > 0 else { continue } + if let cur = winnerByDay[r.day], cur.rank <= rank { continue } + winnerByDay[r.day] = (rank, id, hrv) + } + } + return winnerByDay + .map { (day: $0.key, sourceId: $0.value.sourceId, hrv: $0.value.hrv) } + .sorted { $0.day < $1.day } + } + func hrSamples(from: Int, to: Int, limit: Int = 8000) async -> [HRSample] { guard let store = await ensureStore() else { return [] } // UNION the active strap + canonical so the HR trend renders whether the landed day's raw sits under diff --git a/Strand/Screens/CrossDeviceHRVView.swift b/Strand/Screens/CrossDeviceHRVView.swift new file mode 100644 index 0000000000..7c77ef3ad3 --- /dev/null +++ b/Strand/Screens/CrossDeviceHRVView.swift @@ -0,0 +1,282 @@ +import SwiftUI +import Charts +import Foundation +import StrandDesign +import StrandAnalytics +import WhoopStore + +// MARK: - Cross-Device HRV +// +// A wearer who has worn more than one HRV sensor over time (e.g. years of Oura ring history +// continuing onto a WHOOP strap) has two INCOMPATIBLE RMSSD scales with no overlap night to +// calibrate against — Oura reads ~120-155 ms, WHOOP ~72-112 ms. Averaging them is dishonest; +// showing only the current strap throws away the long trend the wearer cares about. +// +// This screen resolves that for a DISPLAY/trend lens (daily 0-3 scores stay strictly per-device +// via deviceEraEpoch): every night is measured against its OWN device-era's robust log-scale +// centre, so the era-relative trend flows continuously across the switch — you can see whether +// HRV was trending up or down THROUGH the hardware change — while a second, raw chart keeps each +// sensor's true milliseconds. All math is the pure, DB-free `CrossDeviceHRVTrend` engine. + +struct CrossDeviceHRVView: View { + @EnvironmentObject var repo: Repository + + /// One night, parsed to a `Date` once so the chart never re-parses "yyyy-MM-dd" on body re-eval. + private struct Row: Identifiable { + let id: String // the day key — stable + unique per night + let date: Date + let raw: Double // absolute ms as the device reported it + let pct: Double // % above/below this device-era's own centre + let brand: String // "oura" / "whoop" / … + let isEraStart: Bool // first night of a non-first era = a device switch + } + + @State private var eras: [CrossDeviceHRVTrend.Era] = [] + @State private var rows: [Row] = [] // all nights, one continuous series (hero) + @State private var segments: [[Row]] = [] // nights grouped per era (raw, scale-honest) + @State private var loaded = false + + var body: some View { + ScreenScaffold(title: "Cross-Device HRV", + subtitle: "Your full HRV history, stitched across a device switch") { + VStack(alignment: .leading, spacing: NoopMetrics.sectionSpacing) { + if rows.count >= 2 { + heroTrend + rawTrend + eraLegend + methodology + } else if loaded { + emptyCard + } else { + ProgressView() + .frame(maxWidth: .infinity, minHeight: 220, alignment: .center) + } + } + } + .task { await load() } + } + + // MARK: 1 · Era-relative long-term trend (the continuous shape across the switch) + + private var heroTrend: some View { + let boundaries = rows.filter { $0.isEraStart } + // Robust symmetric Y-bound: the era-relative % is a log-ratio, so a known Oura nap-fragment + // artifact (an overnight "HRV" of ~12 ms against a ~135 ms normal) reads as −240%+ and would + // auto-stretch the axis until the real ±20-40% trend collapses to a flat line. Scale to the 90th + // percentile of |deviation| (padded), clamped to a readable band; genuine outliers clip cleanly. + let absPcts = rows.map { abs($0.pct) }.sorted() + let p90 = absPcts.isEmpty ? 40 : absPcts[min(absPcts.count - 1, Int(Double(absPcts.count) * 0.90))] + let heroBound = min(75, max(25, (p90 * 1.3).rounded())) + return VStack(alignment: .leading, spacing: NoopMetrics.gap) { + SectionHeader("Long-Term Trend", overline: "Stitched across devices", + trailing: "\(rows.count) nights") + NoopCard(tint: StrandPalette.metricPurple) { + VStack(alignment: .leading, spacing: NoopMetrics.cardInnerSpacing) { + Text("HRV vs each device-era's own normal").strandOverline() + Chart { + RuleMark(y: .value("Normal", 0)) + .lineStyle(StrokeStyle(lineWidth: 1, dash: [4, 4])) + .foregroundStyle(StrandPalette.textTertiary.opacity(0.6)) + ForEach(rows) { row in + LineMark(x: .value("Date", row.date), + y: .value("Relative", row.pct)) + .interpolationMethod(.catmullRom) + .lineStyle(StrokeStyle(lineWidth: 2, lineCap: .round, lineJoin: .round)) + .foregroundStyle(StrandPalette.metricPurple) + } + ForEach(boundaries) { b in + RuleMark(x: .value("Switch", b.date)) + .lineStyle(StrokeStyle(lineWidth: 1, dash: [3, 3])) + .foregroundStyle(brandColor(b.brand).opacity(0.7)) + .annotation(position: .top, alignment: .center, spacing: 2) { + Text("→ \(brandLabel(b.brand))") + .font(StrandFont.footnote) + .foregroundStyle(brandColor(b.brand)) + } + } + } + .frame(height: NoopMetrics.chartHeight) + .chartYScale(domain: -heroBound...heroBound) + .chartPlotStyle { $0.clipped() } + .chartXAxis { dateAxis } + .chartYAxis { + AxisMarks(position: .leading, values: .automatic(desiredCount: 5)) { value in + AxisGridLine().foregroundStyle(StrandPalette.hairline.opacity(0.4)) + AxisValueLabel { + if let pct = value.as(Double.self) { + Text("\(pct > 0 ? "+" : "")\(Int(pct))%") + } + } + .foregroundStyle(StrandPalette.textTertiary) + .font(StrandFont.footnote) + } + } + Text("Each sensor is compared to its own baseline, so the trend's shape carries across your switch even though the raw numbers differ. The dashed line is each era's normal.") + .font(StrandFont.footnote) + .foregroundStyle(StrandPalette.textTertiary) + .fixedSize(horizontal: false, vertical: true) + } + } + } + } + + // MARK: 2 · Raw nightly HRV (scale-honest, per device) + + private var rawTrend: some View { + VStack(alignment: .leading, spacing: NoopMetrics.gap) { + SectionHeader("Raw Nightly HRV", overline: "Scale-honest per device") + NoopCard(tint: StrandPalette.metricCyan) { + VStack(alignment: .leading, spacing: NoopMetrics.cardInnerSpacing) { + Text("Actual milliseconds, never averaged").strandOverline() + Chart { + ForEach(Array(segments.enumerated()), id: \.offset) { idx, seg in + ForEach(seg) { row in + LineMark(x: .value("Date", row.date), + y: .value("HRV", row.raw), + series: .value("Era", idx)) + .interpolationMethod(.catmullRom) + .lineStyle(StrokeStyle(lineWidth: 2, lineCap: .round, lineJoin: .round)) + .foregroundStyle(brandColor(row.brand)) + } + } + } + .frame(height: NoopMetrics.chartHeight) + .chartPlotStyle { $0.clipped() } + .chartXAxis { dateAxis } + .chartYAxis { + AxisMarks(position: .leading, values: .automatic(desiredCount: 4)) { _ in + AxisGridLine().foregroundStyle(StrandPalette.hairline.opacity(0.4)) + AxisValueLabel().foregroundStyle(StrandPalette.textTertiary).font(StrandFont.footnote) + } + } + Text("Oura and WHOOP measure HRV on different scales, so the two eras sit at different heights — that gap is real, not a change in you. We keep each sensor's true numbers instead of blending them.") + .font(StrandFont.footnote) + .foregroundStyle(StrandPalette.textTertiary) + .fixedSize(horizontal: false, vertical: true) + } + } + } + } + + // MARK: 3 · Era legend + + private var eraLegend: some View { + VStack(alignment: .leading, spacing: NoopMetrics.gap) { + SectionHeader("Devices", overline: "Eras") + NoopCard(tint: StrandPalette.accent) { + VStack(spacing: NoopMetrics.cardInnerSpacing) { + ForEach(Array(eras.enumerated()), id: \.offset) { idx, era in + HStack(spacing: 10) { + Circle().fill(brandColor(era.brand)).frame(width: 10, height: 10) + VStack(alignment: .leading, spacing: 1) { + Text(brandLabel(era.brand)) + .font(StrandFont.body).foregroundStyle(StrandPalette.textPrimary) + Text("\(monthYear(era.startDay)) – \(monthYear(era.endDay)) · \(era.nights) nights") + .font(StrandFont.footnote).foregroundStyle(StrandPalette.textTertiary) + } + Spacer() + VStack(alignment: .trailing, spacing: 1) { + Text("\(Int(era.centerMs.rounded())) ms") + .font(StrandFont.body).foregroundStyle(brandColor(era.brand)) + Text("typical").font(StrandFont.footnote).foregroundStyle(StrandPalette.textTertiary) + } + } + if idx < eras.count - 1 { Divider().overlay(StrandPalette.hairline) } + } + } + } + } + } + + // MARK: 4 · Methodology + + private var methodology: some View { + NoopCard(tint: StrandPalette.metricPurple) { + VStack(alignment: .leading, spacing: NoopMetrics.cardInnerSpacing) { + Text("How this is stitched").strandOverline() + Text("You've worn more than one HRV sensor. Different sensors report HRV on different scales, and there's no night where both were worn to line them up — so any single blended number would be fiction.") + .font(StrandFont.subhead).foregroundStyle(StrandPalette.textSecondary) + .fixedSize(horizontal: false, vertical: true) + Text("Instead, each night is measured against its own device-era's normal (a robust log-scale centre). Plotting that relative view lets the long-term trend flow continuously across the switch, while the raw chart keeps each sensor's true milliseconds. Your daily readiness and stress scores stay strictly per-device.") + .font(StrandFont.subhead).foregroundStyle(StrandPalette.textSecondary) + .fixedSize(horizontal: false, vertical: true) + } + } + } + + private var emptyCard: some View { + NoopCard(tint: StrandPalette.metricPurple) { + Text("Not enough HRV history yet to stitch a cross-device trend. Keep wearing your strap, or import a prior device's history.") + .font(StrandFont.subhead).foregroundStyle(StrandPalette.textTertiary) + .frame(maxWidth: .infinity, minHeight: 120, alignment: .center) + .multilineTextAlignment(.center) + } + } + + // MARK: - Shared axis / brand helpers + + private var dateAxis: some AxisContent { + AxisMarks(values: .automatic(desiredCount: 4)) { _ in + AxisGridLine().foregroundStyle(StrandPalette.hairline.opacity(0.4)) + AxisValueLabel(format: .dateTime.month(.abbreviated).year(.twoDigits)) + .foregroundStyle(StrandPalette.textTertiary).font(StrandFont.footnote) + } + } + + private func brandColor(_ brand: String) -> Color { + switch brand { + case "oura": return StrandPalette.metricCyan + case "whoop": return StrandPalette.metricPurple + case "fitbit": return StrandPalette.metricAmber + case "garmin": return StrandPalette.chargeColor + default: return StrandPalette.accent + } + } + + private func brandLabel(_ brand: String) -> String { + switch brand { + case "oura": return "Oura" + case "whoop": return "WHOOP" + case "fitbit": return "Fitbit" + case "garmin": return "Garmin" + default: return brand.capitalized + } + } + + private static let dayParser: DateFormatter = { + let f = DateFormatter(); f.dateFormat = "yyyy-MM-dd" + f.locale = Locale(identifier: "en_US_POSIX"); f.timeZone = TimeZone(identifier: "UTC") + return f + }() + private static let monthYearFmt: DateFormatter = { + let f = DateFormatter(); f.dateFormat = "MMM yyyy"; f.locale = Locale(identifier: "en_US_POSIX") + return f + }() + private func monthYear(_ day: String) -> String { + guard let d = Self.dayParser.date(from: day) else { return day } + return Self.monthYearFmt.string(from: d) + } + + // MARK: - Load + + private func load() async { + let history = await repo.crossDeviceHrvHistory() + let result = CrossDeviceHRVTrend.build(history) + let parsed: [Row] = result.points.map { p in + Row(id: p.day, date: Self.dayParser.date(from: p.day) ?? Date(timeIntervalSince1970: 0), + raw: p.raw, pct: p.eraRelativePct, brand: p.brand, isEraStart: p.isEraStart) + } + // Group into per-era segments (a device switch opens a new segment). + var segs: [[Row]] = []; var cur: [Row] = [] + for row in parsed { + if row.isEraStart && !cur.isEmpty { segs.append(cur); cur = [] } + cur.append(row) + } + if !cur.isEmpty { segs.append(cur) } + + self.eras = result.eras + self.rows = parsed + self.segments = segs + self.loaded = true + } +} diff --git a/StrandiOS/App/RootTabView.swift b/StrandiOS/App/RootTabView.swift index 65ebce874c..c3031be6ee 100644 --- a/StrandiOS/App/RootTabView.swift +++ b/StrandiOS/App/RootTabView.swift @@ -339,6 +339,7 @@ struct RootTabView: View { MoreRow("Insights", "lightbulb.fill", .insights) MoreRow("Explore", "square.grid.2x2.fill", .explore) MoreRow("Compare", "rectangle.split.2x1.fill", .compare) + MoreRow("Cross-Device HRV", "waveform.path.ecg.rectangle", .crossDeviceHRV) } moreSection("Body") { MoreRow("Live", "waveform.path.ecg", .live) @@ -462,7 +463,7 @@ struct RootTabView: View { /// registration in `moreTab`. private enum MoreDestination: Hashable { case insightsHub, intelligence, coach, insights, explore, compare - case live, workouts, health, labBook, stress, breathe, intervals, rhythm + case live, workouts, health, labBook, stress, breathe, intervals, rhythm, crossDeviceHRV case fusedRecord, appleHealth, miBand, dataSources, backupSync, shortcutsExport case alarms, automations, testCentre, siriShortcuts, settings @@ -479,6 +480,7 @@ private enum MoreDestination: Hashable { case .health: HealthView() case .labBook: LabBookView() case .stress: StressView() + case .crossDeviceHRV: CrossDeviceHRVView() case .breathe: BreathingView() case .intervals: IntervalTimerView() case .rhythm: RhythmHost() diff --git a/StrandiOS/App/StrandiOSApp.swift b/StrandiOS/App/StrandiOSApp.swift index 1e2ef1af3d..6d598b3e19 100644 --- a/StrandiOS/App/StrandiOSApp.swift +++ b/StrandiOS/App/StrandiOSApp.swift @@ -349,6 +349,7 @@ enum DemoScreens { case "sleep": return AnyView(SleepView()) case "live": return AnyView(LiveView()) case "stress": return AnyView(StressView()) + case "crossdevicehrv": return AnyView(CrossDeviceHRVView()) case "workouts": return AnyView(WorkoutsView()) case "health": return AnyView(HealthView()) case "insights": return AnyView(InsightsView()) From c405f9a82407955d38fd1cfbf9a2acc3d857321a Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 22:55:22 -0700 Subject: [PATCH 106/171] Compute the 5-min SDNN index (not whole-night SD) for avgSdnn MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The nightly SDNN persisted + exported under HealthKit's heartRateVariabilitySDNN must be window-matched to a watch's own SDNN, or it corrupts the user's Health history and makes any cross-check meaningless. Validated on real R-R (a WHOOP night, 20.9k beats): whole-night SD = 135 ms vs the 5-min SDNN index = 94 ms — the whole-night value is inflated ~40% by the slow HR drift across sleep stages, which Apple's short-window SDNN never sees. - Add HRVAnalyzer.sdnnIndex(rr, segmentSec: 300): the Task Force SDNN index, the mean of per-5-min-segment SDNN, each segment cleaned by the same range + Malik ectopic rejection the nightly path uses. - AnalyticsEngine.avgSDNNDaily now computes the index (keeping the R-R timestamps for segmentation) instead of a single whole-night SD. - Doc/comment updates: avgSdnn is the 5-min index; the HealthKit export note explains why the window match matters. - 4 tests, incl. the key property that the index strips inter-segment drift (steady-in-segment / drifting-across → index << whole-night SD). avgSdnn stays nil until a device re-scores from raw R-R (unchanged). --- .../StrandAnalytics/AnalyticsEngine.swift | 14 +++--- .../Sources/StrandAnalytics/HRVAnalyzer.swift | 23 ++++++++++ .../HRVAnalyzerTests.swift | 44 +++++++++++++++++++ .../Sources/WhoopStore/Database.swift | 5 ++- .../Sources/WhoopStore/MetricsCache.swift | 12 ++--- StrandiOS/Health/HealthKitBridge.swift | 12 ++--- 6 files changed, 92 insertions(+), 18 deletions(-) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift index 524d4bf3b1..37fb78355f 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift @@ -536,14 +536,16 @@ public enum AnalyticsEngine { return weight > 0 ? total / weight : nil }() - // Daily SDNN (ms) = whole-night sample-SD (Task Force ddof=1) over the in-bed R-R across matched - // sessions — the BROAD autonomic-variability metric (both autonomic branches), the slow / - // accumulated-load twin of the vagal RMSSD `avgHRVDaily` above (which is fast, deep-window / vagal). - // Same in-sleep R-R the HRV trace uses; nil when too few clean beats (HRVAnalyzer's own gate). + // Daily SDNN (ms) = the 5-min SDNN INDEX (Task Force) over the in-bed R-R across matched sessions — + // the mean of per-5-min-segment SDNN, the BROAD autonomic-variability metric (both branches) and the + // slow twin of the vagal RMSSD `avgHRVDaily` above. The index (not a single whole-night SD) is used + // deliberately: whole-night SD is dominated by the slow HR drift across sleep stages and reads 2-3× + // high, which would mislabel Apple Health (its SDNN samples are short-window) and make any cross-check + // against a watch meaningless. The 5-min index is window-comparable to those. nil when no segment has + // enough clean beats (HRVAnalyzer's own gate). Keeps the R-R timestamps (segmentation needs them). let avgSDNNDaily: Double? = { let inBed = rr.filter { r in matched.contains { r.ts >= $0.start && r.ts < $0.end } } - .map { Double($0.rrMs) } - return inBed.isEmpty ? nil : HRVAnalyzer.analyze(rawRR: inBed).sdnn + return inBed.isEmpty ? nil : HRVAnalyzer.sdnnIndex(inBed, segmentSec: 300) }() // ── HRV & Autonomic nightly trace (#141) ────────────────────────────── diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/HRVAnalyzer.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/HRVAnalyzer.swift index 0d82bd879c..7a839278e2 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/HRVAnalyzer.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/HRVAnalyzer.swift @@ -286,6 +286,29 @@ public enum HRVAnalyzer { nInput: nInput, nClean: clean.count) } + /// SDNN index (Task Force 1996): the MEAN of SDNN computed over consecutive fixed-length segments + /// (default 5 min) of the R-R series, each segment cleaned with the SAME range + Malik ectopic + /// rejection the nightly path uses. Unlike whole-night SDNN — which is dominated by the slow HR drift + /// ACROSS sleep stages and can read 2-3× higher — the index reflects SHORT-TERM variability, so it is + /// window-comparable to a wearable's short SDNN reading (e.g. Apple Watch's ~1-min + /// `heartRateVariabilitySDNN` samples) and is the value that can be honestly cross-checked against one. + /// Segments with fewer than `minBeats` clean intervals are skipped; nil when no segment qualifies. + /// Pure, deterministic. Kotlin twin: `HrvAnalyzer.sdnnIndex`. + public static func sdnnIndex(_ rr: [RRInterval], segmentSec: Int = 300) -> Double? { + guard segmentSec > 0, let first = rr.map(\.ts).min(), let last = rr.map(\.ts).max(), + last >= first else { return nil } + var segStart = first + var segmentSDNNs: [Double] = [] + while segStart <= last { + if let sd = analyze(rr, windowStart: segStart, windowEnd: segStart + segmentSec - 1).sdnn { + segmentSDNNs.append(sd) + } + segStart += segmentSec + } + guard !segmentSDNNs.isEmpty else { return nil } + return segmentSDNNs.reduce(0, +) / Double(segmentSDNNs.count) + } + // MARK: - Rolling / windowed rMSSD timeline (#803) /// One windowed rMSSD point: the rMSSD (ms) over the trailing `windowSec` of R-R intervals ending at diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/HRVAnalyzerTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/HRVAnalyzerTests.swift index 5c7908f54a..15f9812d3d 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/HRVAnalyzerTests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/HRVAnalyzerTests.swift @@ -213,4 +213,48 @@ final class HRVAnalyzerTests: XCTestCase { XCTAssertEqual(HRVAnalyzer.duplicateBeatCount(tsSec: [100, 100, 100], rrMs: [1000, 1000, 1000]), 2) XCTAssertEqual(HRVAnalyzer.duplicateBeatCount(tsSec: [100, 100], rrMs: [1000, 1010]), 0) // diff rr = distinct } + + // MARK: - SDNN index (5-min segmented SDNN, the Apple-comparable window) + + func testSdnnIndexExcludesInterSegmentDrift() { + // Three 100 s segments, each internally near-steady (±5 ms) but at very different levels + // (800 / 900 / 1000 ms). Whole-night SDNN sees the big 800→1000 drift and reads large; the SDNN + // index averages each segment's OWN (small) SDNN, so it stays small — the exact property that makes + // it comparable to a watch's short-window reading instead of the drift-inflated whole-night value. + var rr: [RRInterval] = [] + for t in 0..<50 { rr.append(RRInterval(ts: 0 + t, rrMs: t.isMultiple(of: 2) ? 795 : 805)) } + for t in 0..<50 { rr.append(RRInterval(ts: 100 + t, rrMs: t.isMultiple(of: 2) ? 895 : 905)) } + for t in 0..<50 { rr.append(RRInterval(ts: 200 + t, rrMs: t.isMultiple(of: 2) ? 995 : 1005)) } + + let index = HRVAnalyzer.sdnnIndex(rr, segmentSec: 100) + let wholeNight = HRVAnalyzer.analyze(rawRR: rr.map { Double($0.rrMs) }).sdnn + let idx = try! XCTUnwrap(index) + let whole = try! XCTUnwrap(wholeNight) + XCTAssertEqual(idx, 5.05, accuracy: 1.5, "each segment's own SDNN is ~5 ms") + XCTAssertGreaterThan(whole, 50, "whole-night SDNN is inflated by the 800→1000 drift") + XCTAssertLessThan(idx, whole / 5, "the index must strip out the inter-segment drift") + } + + func testSdnnIndexSteadySeriesIsSmallPositive() { + // A single steady segment (±5 ms) → a small, sane, non-nil index. + let rr = (0..<60).map { RRInterval(ts: $0, rrMs: $0.isMultiple(of: 2) ? 795 : 805) } + let idx = try! XCTUnwrap(HRVAnalyzer.sdnnIndex(rr, segmentSec: 100)) + XCTAssertEqual(idx, 5.05, accuracy: 1.5) + } + + func testSdnnIndexSingleSegmentEqualsWholeSdnn() { + // When all beats fall in ONE segment, the index is just that segment's SDNN = the whole SDNN. + let rr = (0..<60).map { RRInterval(ts: $0, rrMs: [800, 820, 780, 810, 790][$0 % 5]) } + let index = try! XCTUnwrap(HRVAnalyzer.sdnnIndex(rr, segmentSec: 1000)) + let whole = try! XCTUnwrap(HRVAnalyzer.analyze(rr, windowStart: 0, windowEnd: 999).sdnn) + XCTAssertEqual(index, whole, accuracy: 1e-9) + } + + func testSdnnIndexSparseSeriesIsNil() { + // Fewer than minBeats in the only segment → no qualifying segment → nil (honest absence). + let rr = (0..<10).map { RRInterval(ts: $0, rrMs: 800) } + XCTAssertNil(HRVAnalyzer.sdnnIndex(rr, segmentSec: 100)) + XCTAssertNil(HRVAnalyzer.sdnnIndex([], segmentSec: 100)) + XCTAssertNil(HRVAnalyzer.sdnnIndex(rr, segmentSec: 0), "non-positive segment length is rejected") + } } diff --git a/Packages/WhoopStore/Sources/WhoopStore/Database.swift b/Packages/WhoopStore/Sources/WhoopStore/Database.swift index a4347fd431..3b1f04fef8 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/Database.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/Database.swift @@ -587,8 +587,9 @@ extension WhoopStore { t.column("tzId", .text).notNull() // IANA identifier, e.g. "America/Los_Angeles" } } - // v29: persist nightly SDNN (the broad-variability HRV twin of avgHrv=RMSSD). Additive, nullable; - // the nightly analysis already computes SDNN (HRVAnalyzer) but only logged it — now stored. + // v29: persist nightly SDNN — the 5-min SDNN index (broad-variability twin of avgHrv=RMSSD), + // window-matched to a watch's short SDNN rather than the drift-inflated whole-night SD. Additive, + // nullable; computed by HRVAnalyzer.sdnnIndex during the nightly analysis. migrator.registerMigration("v29-daily-avg-sdnn") { db in try db.alter(table: "dailyMetric") { t in t.add(column: "avgSdnn", .double) diff --git a/Packages/WhoopStore/Sources/WhoopStore/MetricsCache.swift b/Packages/WhoopStore/Sources/WhoopStore/MetricsCache.swift index 6a75044136..ce7cd2fe63 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/MetricsCache.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/MetricsCache.swift @@ -68,11 +68,13 @@ public struct DailyMetric: Equatable, Codable { // call site is unaffected. public let spo2Red: Int? // mean raw red PPG ADC during detected sleep public let spo2Ir: Int? // mean raw IR PPG ADC during detected sleep - /// Nightly SDNN (ms — Task Force whole-night sample SD, ddof=1). The BROAD autonomic-variability - /// twin of `avgHrv` (which is RMSSD for the strap: the fast, vagal, "recovered today?" metric). SDNN - /// captures overall variability across the night — both autonomic branches — a slower, accumulated-load - /// read. v29 column, nullable: WHOOP/on-device nights compute it from the night's R-R (`HRVAnalyzer`); - /// Apple rows mirror their own SDNN reading; Oura/other imports carry no SDNN so it stays nil. + /// Nightly SDNN (ms — the Task Force 5-min SDNN INDEX: mean of per-5-min-segment SDNN, ddof=1). The + /// BROAD autonomic-variability twin of `avgHrv` (which is RMSSD for the strap: the fast, vagal, + /// "recovered today?" metric). The 5-min index — not a single whole-night SD — is stored deliberately: + /// whole-night SD is dominated by the slow HR drift across sleep stages (reads 2-3× high) and is not + /// comparable to a watch's short-window SDNN; the index is. v29 column, nullable: WHOOP/on-device nights + /// compute it from the night's R-R (`HRVAnalyzer.sdnnIndex`); Apple rows mirror their own SDNN reading; + /// Oura/other imports carry no SDNN so it stays nil. public let avgSdnn: Double? public init(day: String, totalSleepMin: Double?, efficiency: Double?, deepMin: Double?, remMin: Double?, lightMin: Double?, disturbances: Int?, restingHr: Int?, diff --git a/StrandiOS/Health/HealthKitBridge.swift b/StrandiOS/Health/HealthKitBridge.swift index 7114957f82..80da027351 100644 --- a/StrandiOS/Health/HealthKitBridge.swift +++ b/StrandiOS/Health/HealthKitBridge.swift @@ -572,11 +572,13 @@ final class HealthKitBridge: ObservableObject { add(.restingHeartRate, HKUnit.count().unitDivided(by: .minute()), Double(rhr), row.day, at) } // Export the GENUINE SDNN (v29) when present — the strap's `avgHrv` is RMSSD, which HealthKit - // has no field for, so writing it under `.heartRateVariabilitySDNN` mislabels it. WHOOP rows - // backfill `avgSdnn` from stored raw R-R on the next re-score; Apple rows' `avgHrv` already IS - // SDNN. The `avgHrv` fallback fires for those two before re-scoring, and permanently for - // summary-only sources (Oura) that give RMSSD with no raw R-R to derive SDNN from — HealthKit's - // single HRV type leaves no better label there. + // has no field for, so writing it under `.heartRateVariabilitySDNN` mislabels it. `avgSdnn` is the + // 5-min SDNN index, deliberately window-matched to Apple's own short-window SDNN samples so the + // written values sit consistently in the user's Health SDNN history (a whole-night SD would land + // 2-3× high). WHOOP rows backfill `avgSdnn` from stored raw R-R on the next re-score; Apple rows' + // `avgHrv` already IS SDNN. The `avgHrv` fallback fires for those two before re-scoring, and + // permanently for summary-only sources (Oura) that give RMSSD with no raw R-R to derive SDNN from + // — HealthKit's single HRV type leaves no better label there. if let sdnn = row.avgSdnn ?? row.avgHrv { add(.heartRateVariabilitySDNN, .secondUnit(with: .milli), sdnn, row.day, at) } From 8ba98bc73d3c8e9e222eab9156c32a5add96cdae Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 14 Jul 2026 22:55:54 -0700 Subject: [PATCH 107/171] Show both HRV metrics labeled + an Apple SDNN cross-check (Latest Night) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses the Discord report's "show RMSSD and SDNN with correct labels" and "cross-check the local SDNN against the watch's SDNN". Adds a Latest Night section atop the Cross-Device HRV screen: - RMSSD (avgHrv) and SDNN (avgSdnn, the 5-min index) side by side, each labeled with what it measures — no longer collapsed into one ambiguous "HRV" number. - An Apple Watch cross-check row when the user has an Apple SDNN for the same day: local 5-min index vs Apple's ~1-min SDNN, with an honest note that NOOP's runs a little higher (wider window) but tracks the same nightly moves. - Repository.latestNightHRV() assembles the pair + the Apple reading. SDNN reads "—" until the phone re-scores from raw R-R (populates avgSdnn); verified on real data the section renders RMSSD live and handles the pending SDNN gracefully. --- Strand/Data/Repository.swift | 27 ++++++++ Strand/Screens/CrossDeviceHRVView.swift | 86 ++++++++++++++++++++++--- 2 files changed, 105 insertions(+), 8 deletions(-) diff --git a/Strand/Data/Repository.swift b/Strand/Data/Repository.swift index 832a56436a..14e10fd90d 100644 --- a/Strand/Data/Repository.swift +++ b/Strand/Data/Repository.swift @@ -850,6 +850,33 @@ final class Repository: ObservableObject { return await unionDailyMetrics(store: store, from: fromDay, to: toDay) } + /// The most recent night's HRV in BOTH metrics, plus the Apple reading for the same day when present — + /// backs the "Latest Night" dual-metric header + the honest Apple cross-check on the Cross-Device screen. + /// `rmssd` is the strap's `avgHrv` (beat-to-beat); `sdnn` is `avgSdnn` (the 5-min SDNN index, already + /// window-matched to Apple's short-window SDNN, so `sdnn` vs `appleSdnn` is a like-for-like check). + struct NightHRV: Equatable { + let day: String + let rmssd: Double? + let sdnn: Double? + let appleSdnn: Double? + } + + func latestNightHRV() async -> NightHRV? { + guard let store = await ensureStore() else { return nil } + var latest: DailyMetric? + for id in importedReadIds + computedReadIds { + let rows = (try? await store.dailyMetrics(deviceId: id, from: "2018-01-01", to: "9999-12-31")) ?? [] + for r in rows where r.avgHrv != nil { + if latest == nil || r.day > latest!.day { latest = r } + } + } + guard let night = latest else { return nil } + let appleRows = (try? await store.dailyMetrics(deviceId: Self.appleHealthSource, + from: night.day, to: night.day)) ?? [] + return NightHRV(day: night.day, rmssd: night.avgHrv, sdnn: night.avgSdnn, + appleSdnn: appleRows.first?.avgSdnn) + } + /// Full-history nightly HRV tagged by its WINNING source id, for the cross-device HRV trend /// (`CrossDeviceHRVTrend`). One winner per day: the real wearable of each era (Oura under "oura-api", /// WHOOP under the strap/import/computed ids) — the real id strings are preserved so the engine's diff --git a/Strand/Screens/CrossDeviceHRVView.swift b/Strand/Screens/CrossDeviceHRVView.swift index 7c77ef3ad3..da6f60f5d7 100644 --- a/Strand/Screens/CrossDeviceHRVView.swift +++ b/Strand/Screens/CrossDeviceHRVView.swift @@ -34,28 +34,87 @@ struct CrossDeviceHRVView: View { @State private var eras: [CrossDeviceHRVTrend.Era] = [] @State private var rows: [Row] = [] // all nights, one continuous series (hero) @State private var segments: [[Row]] = [] // nights grouped per era (raw, scale-honest) + @State private var latestNight: Repository.NightHRV? // dual-metric header + Apple cross-check @State private var loaded = false var body: some View { ScreenScaffold(title: "Cross-Device HRV", subtitle: "Your full HRV history, stitched across a device switch") { VStack(alignment: .leading, spacing: NoopMetrics.sectionSpacing) { - if rows.count >= 2 { - heroTrend - rawTrend - eraLegend - methodology - } else if loaded { - emptyCard - } else { + if !loaded { ProgressView() .frame(maxWidth: .infinity, minHeight: 220, alignment: .center) + } else { + latestNightSection + if rows.count >= 2 { + heroTrend + rawTrend + eraLegend + methodology + } else if latestNight == nil { + emptyCard + } } } } .task { await load() } } + // MARK: 0 · Latest night — both HRV metrics, labeled (+ Apple cross-check) + + @ViewBuilder private var latestNightSection: some View { + if let n = latestNight, n.rmssd != nil || n.sdnn != nil { + VStack(alignment: .leading, spacing: NoopMetrics.gap) { + SectionHeader("Latest Night", overline: "Both HRV metrics, labeled", trailing: prettyDay(n.day)) + NoopCard(tint: StrandPalette.metricPurple) { + VStack(alignment: .leading, spacing: NoopMetrics.cardInnerSpacing) { + HStack(alignment: .top, spacing: 0) { + metricColumn("RMSSD", n.rmssd, "beat-to-beat · vagal", StrandPalette.metricPurple) + Divider().frame(height: 46).overlay(StrandPalette.hairline) + metricColumn("SDNN", n.sdnn, "5-min index · overall", StrandPalette.metricCyan) + } + if let local = n.sdnn, let apple = n.appleSdnn, apple > 0 { + Divider().overlay(StrandPalette.hairline) + appleCrossCheck(local: local, apple: apple) + } + Text("RMSSD tracks fast vagal (parasympathetic) tone; SDNN captures overall variability. Both are computed from the same night's beats — different lenses, not different data — so they're no longer collapsed into one ambiguous \u{201C}HRV\u{201D} number.") + .font(StrandFont.footnote).foregroundStyle(StrandPalette.textTertiary) + .fixedSize(horizontal: false, vertical: true) + } + } + } + } + } + + private func metricColumn(_ label: LocalizedStringKey, _ value: Double?, _ caption: String, + _ accent: Color) -> some View { + VStack(alignment: .leading, spacing: 2) { + Text(label).strandOverline() + Text(value.map { "\(Int($0.rounded())) ms" } ?? "—") + .font(StrandFont.title2) + .foregroundStyle(value == nil ? StrandPalette.textTertiary : accent) + Text(caption).font(StrandFont.footnote).foregroundStyle(StrandPalette.textTertiary) + } + .frame(maxWidth: .infinity, alignment: .leading) + .padding(.horizontal, 2) + } + + private func appleCrossCheck(local: Double, apple: Double) -> some View { + let deltaPct = Int(((local - apple) / apple * 100).rounded()) + return VStack(alignment: .leading, spacing: 3) { + Text("Apple Watch cross-check").strandOverline() + HStack { + Text("NOOP 5-min SDNN \(Int(local.rounded())) ms") + Spacer() + Text("Apple \(Int(apple.rounded())) ms") + } + .font(StrandFont.subhead).foregroundStyle(StrandPalette.textSecondary) + Text("\(deltaPct >= 0 ? "+" : "")\(deltaPct)% vs your watch — both are short-window SDNN (NOOP over 5-min segments, Apple ~1-min), so NOOP's usually reads a little higher but tracks the same nightly moves.") + .font(StrandFont.footnote).foregroundStyle(StrandPalette.textTertiary) + .fixedSize(horizontal: false, vertical: true) + } + } + // MARK: 1 · Era-relative long-term trend (the continuous shape across the switch) private var heroTrend: some View { @@ -256,6 +315,14 @@ struct CrossDeviceHRVView: View { guard let d = Self.dayParser.date(from: day) else { return day } return Self.monthYearFmt.string(from: d) } + private static let dayFmt: DateFormatter = { + let f = DateFormatter(); f.dateFormat = "MMM d"; f.locale = Locale(identifier: "en_US_POSIX") + return f + }() + private func prettyDay(_ day: String) -> String { + guard let d = Self.dayParser.date(from: day) else { return day } + return Self.dayFmt.string(from: d) + } // MARK: - Load @@ -274,9 +341,12 @@ struct CrossDeviceHRVView: View { } if !cur.isEmpty { segs.append(cur) } + let night = await repo.latestNightHRV() + self.eras = result.eras self.rows = parsed self.segments = segs + self.latestNight = night self.loaded = true } } From 277e9ad389e3e40747c5662881c85faa9ba8b97d Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 15 Jul 2026 10:13:44 -0700 Subject: [PATCH 108/171] Make the cross-device charts readable: smooth the trend, break at non-wear gaps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two years of nightly HRV drawn raw is ~1.7 pt per night — the "line" was a solid block of vertical strokes with no visible trend, and catmullRom overshoot plus artifact spikes clipping to the axis made it worse. - Dots are the actual nights (faint, so the real spread stays honest); the line is their 7-night CENTERED rolling median — a median, not a mean, so a nap-fragment artifact can't drag the curve. ~60 legible points across 2 years. - Break the line at any gap > 14 days. Charts joins consecutive points, so a months-long non-wear hole was being drawn as a straight flat line — inventing a trend that never happened (this history has real holes: 2025-06-27→09-25, 2025-10-13→11-19). Each contiguous run is now its own series; the hole reads as absence, which is the truth. - Raw chart: smoothed PER ERA and gap-split within each, so the median never blends the two incompatible scales and the step across the switch stays visible. - Captions say what the line actually is. --- Strand/Screens/CrossDeviceHRVView.swift | 114 +++++++++++++++++++++--- 1 file changed, 100 insertions(+), 14 deletions(-) diff --git a/Strand/Screens/CrossDeviceHRVView.swift b/Strand/Screens/CrossDeviceHRVView.swift index da6f60f5d7..8cc6ab27ae 100644 --- a/Strand/Screens/CrossDeviceHRVView.swift +++ b/Strand/Screens/CrossDeviceHRVView.swift @@ -37,6 +37,58 @@ struct CrossDeviceHRVView: View { @State private var latestNight: Repository.NightHRV? // dual-metric header + Apple cross-check @State private var loaded = false + /// One point on a smoothed trend line. + private struct Smoothed: Identifiable { + let id: String + let date: Date + let value: Double + } + /// The hero's readable trend (rolling median of the era-relative %), and the raw chart's per-era + /// equivalent. Precomputed in `load()` — never per body eval. + /// Smoothed trend split into CONTIGUOUS runs (see `breakAtGaps`) — one drawn line per run. + @State private var heroRuns: [[Smoothed]] = [] + @State private var rawRuns: [(brand: String, pts: [Smoothed])] = [] + + /// A wearer takes the band off. Charts joins consecutive points, so without an explicit break a + /// months-long non-wear hole renders as a straight line — a trend that never happened (this history + /// has real holes: 2025-06-27→09-25, 2025-10-13→11-19). Any gap longer than this opens a new run, so + /// the line simply stops and restarts and the hole reads as absence, which is the truth. + private static let gapBreakDays = 14.0 + + private func breakAtGaps(_ pts: [Smoothed]) -> [[Smoothed]] { + guard let first = pts.first else { return [] } + var out: [[Smoothed]] = [] + var cur: [Smoothed] = [first] + for p in pts.dropFirst() { + if let last = cur.last, p.date.timeIntervalSince(last.date) > Self.gapBreakDays * 86_400 { + out.append(cur); cur = [] + } + cur.append(p) + } + if !cur.isEmpty { out.append(cur) } + return out + } + + /// Nights per side of the rolling window. 420 raw nights across ~700 pt is ~1.7 pt/night — drawn as a + /// line that is a solid block of vertical strokes, not a trend. A 7-night CENTERED rolling median is + /// the standard fix: it strips night-to-night noise and (being a median, not a mean) refuses to let a + /// nap-fragment artifact drag the curve, leaving ~60 readable points across two years. + private static let smoothWindow = 7 + + /// Centered rolling median over (date, value) pairs, oldest→newest. Returns [] when there is less + /// history than the window — the caller then just shows the raw nights, which at that size are legible. + private func rollingMedian(_ pts: [(date: Date, value: Double)], window: Int = smoothWindow) -> [Smoothed] { + guard window > 1, pts.count >= window else { return [] } + let half = window / 2 + return pts.indices.map { i in + let lo = max(0, i - half), hi = min(pts.count - 1, i + half) + let s = pts[lo...hi].map(\.value).sorted() + let mid = s.count / 2 + let med = s.count % 2 == 1 ? s[mid] : (s[mid - 1] + s[mid]) / 2 + return Smoothed(id: "\(i)", date: pts[i].date, value: med) + } + } + var body: some View { ScreenScaffold(title: "Cross-Device HRV", subtitle: "Your full HRV history, stitched across a device switch") { @@ -136,12 +188,24 @@ struct CrossDeviceHRVView: View { RuleMark(y: .value("Normal", 0)) .lineStyle(StrokeStyle(lineWidth: 1, dash: [4, 4])) .foregroundStyle(StrandPalette.textTertiary.opacity(0.6)) + // Every real night, as a faint cloud — the honest spread stays visible… ForEach(rows) { row in - LineMark(x: .value("Date", row.date), - y: .value("Relative", row.pct)) - .interpolationMethod(.catmullRom) - .lineStyle(StrokeStyle(lineWidth: 2, lineCap: .round, lineJoin: .round)) - .foregroundStyle(StrandPalette.metricPurple) + PointMark(x: .value("Date", row.date), + y: .value("Night", row.pct)) + .symbolSize(4) + .foregroundStyle(StrandPalette.metricPurple.opacity(0.16)) + } + // …and the TREND is the rolling median riding through it, which is the only thing + // legible at two years per screen. + ForEach(Array(heroRuns.enumerated()), id: \.offset) { idx, run in + ForEach(run) { s in + LineMark(x: .value("Date", s.date), + y: .value("Trend", s.value), + series: .value("Run", idx)) + .interpolationMethod(.monotone) + .lineStyle(StrokeStyle(lineWidth: 2.5, lineCap: .round, lineJoin: .round)) + .foregroundStyle(StrandPalette.metricPurple) + } } ForEach(boundaries) { b in RuleMark(x: .value("Switch", b.date)) @@ -170,7 +234,7 @@ struct CrossDeviceHRVView: View { .font(StrandFont.footnote) } } - Text("Each sensor is compared to its own baseline, so the trend's shape carries across your switch even though the raw numbers differ. The dashed line is each era's normal.") + Text("Dots are your actual nights; the line is their 7-night rolling median — the trend, without the night-to-night noise. It stops wherever you weren't wearing anything for a couple of weeks, rather than drawing across the gap. Each sensor is compared to its own baseline, so the shape carries across your switch even though the raw numbers differ. The dashed line is each era's normal.") .font(StrandFont.footnote) .foregroundStyle(StrandPalette.textTertiary) .fixedSize(horizontal: false, vertical: true) @@ -188,14 +252,26 @@ struct CrossDeviceHRVView: View { VStack(alignment: .leading, spacing: NoopMetrics.cardInnerSpacing) { Text("Actual milliseconds, never averaged").strandOverline() Chart { - ForEach(Array(segments.enumerated()), id: \.offset) { idx, seg in + // Faint per-night cloud, coloured by its era… + ForEach(Array(segments.enumerated()), id: \.offset) { _, seg in ForEach(seg) { row in - LineMark(x: .value("Date", row.date), - y: .value("HRV", row.raw), - series: .value("Era", idx)) - .interpolationMethod(.catmullRom) - .lineStyle(StrokeStyle(lineWidth: 2, lineCap: .round, lineJoin: .round)) - .foregroundStyle(brandColor(row.brand)) + PointMark(x: .value("Date", row.date), + y: .value("HRV", row.raw)) + .symbolSize(4) + .foregroundStyle(brandColor(row.brand).opacity(0.16)) + } + } + // …with each era's own smoothed trend. Smoothed PER ERA and drawn as separate + // series, so the median never blends the two incompatible scales and no line is + // drawn across the switch — the step between eras stays visible, which is the point. + ForEach(Array(rawRuns.enumerated()), id: \.offset) { idx, run in + ForEach(run.pts) { s in + LineMark(x: .value("Date", s.date), + y: .value("Trend", s.value), + series: .value("Run", idx)) + .interpolationMethod(.monotone) + .lineStyle(StrokeStyle(lineWidth: 2.5, lineCap: .round, lineJoin: .round)) + .foregroundStyle(brandColor(run.brand)) } } } @@ -208,7 +284,7 @@ struct CrossDeviceHRVView: View { AxisValueLabel().foregroundStyle(StrandPalette.textTertiary).font(StrandFont.footnote) } } - Text("Oura and WHOOP measure HRV on different scales, so the two eras sit at different heights — that gap is real, not a change in you. We keep each sensor's true numbers instead of blending them.") + Text("Oura and WHOOP measure HRV on different scales, so the two eras sit at different heights — that gap is real, not a change in you. Each era is smoothed on its own and never joined across the switch. We keep each sensor's true numbers instead of blending them.") .font(StrandFont.footnote) .foregroundStyle(StrandPalette.textTertiary) .fixedSize(horizontal: false, vertical: true) @@ -346,6 +422,16 @@ struct CrossDeviceHRVView: View { self.eras = result.eras self.rows = parsed self.segments = segs + // The era-relative % is cross-era comparable, so one median over the whole history is correct — + // then split at non-wear gaps so no line is drawn across a hole. + self.heroRuns = breakAtGaps(rollingMedian(parsed.map { (date: $0.date, value: $0.pct) })) + // The raw ms is NOT cross-era comparable — smooth each era separately, then gap-split within it, + // carrying the era's brand so each run keeps its colour. + self.rawRuns = segs.flatMap { seg -> [(brand: String, pts: [Smoothed])] in + let brand = seg.first?.brand ?? "whoop" + return breakAtGaps(rollingMedian(seg.map { (date: $0.date, value: $0.raw) })) + .map { (brand: brand, pts: $0) } + } self.latestNight = night self.loaded = true } From 396413f6c3dd3d3b9e721f26f0a291672fafe877 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 15 Jul 2026 11:09:44 -0700 Subject: [PATCH 109/171] docs: capture the 2026-07-15 strap-death bug sweep (17 evidence-backed findings) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A strap drained and stopped streaming at 2026-07-14 13:27 PDT, a night went unscored, and the app could not tell the wearer any of it. Every finding is backed by the live device DB / UserDefaults rather than inference, with file:line for each root cause. Grouped: BLE/battery/connection (never-retried 0x2A19 read behind a zombie restored peripheral; "Active·Live" keyed to the raw link object; charging unreachable when pct is nil; the drain capped at defaultMaxAutoContinues=6 turning a minutes-long offload into ~7.5h; lastSeenAt frozen at pairing), observability (backfill progress invisible; no staleness marker; Rest carries while Charge/Effort don't; battery/events invisible over MCP; the R22 census + firmware live only in UserDefaults so the MCP can never see them), alerts (both fired and latched — ~3h of silence to death; the estimate ignores the ~10% cutoff; the rated path is unreachable on a 5.0), cloud sync (foreground URLSession can't survive a background window; warm resume syncs nothing; the volume is ~1 upload from ENOSPC), deep data (2140B optical frames rejected — no ppgWaveformSample table on the installed build; deep IMU capture ran for one 58-min window in 8 months), and one latent scoring trap (Charge silently nils after 14 stale nights; the .stale comment and `usable` disagree). The unifying failure: the app knows all of this and surfaces almost none of it. --- ...15-strap-battery-backfill-observability.md | 194 ++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 docs/bugs/2026-07-15-strap-battery-backfill-observability.md diff --git a/docs/bugs/2026-07-15-strap-battery-backfill-observability.md b/docs/bugs/2026-07-15-strap-battery-backfill-observability.md new file mode 100644 index 0000000000..7769ec4bbc --- /dev/null +++ b/docs/bugs/2026-07-15-strap-battery-backfill-observability.md @@ -0,0 +1,194 @@ +# 2026-07-15 — strap death, battery, backfill + observability bug sweep + +Captured from a live incident: the strap drained and stopped streaming at **2026-07-14 13:27 PDT** +(last battery reading 10.9% @ 13:00, `charging=0`), a night of data went unscored, and the app was +unable to tell the wearer any of it. Every item below is **evidence-backed from the live device DB +and UserDefaults**, not speculation. Line numbers are from `origin/main` around 2026-07-15. + +**The unifying failure:** the app *knows* what is happening — `strapNewestTs`, the backfill frontier, +`backfilling`, SoC, `charging`, `lastSeenAt` — and **surfaces almost none of it**. Hours were spent on +forensics the app had the data to answer instantly. + +--- + +## A. BLE / battery / connection + +### A1. Battery % never lands on a 5.0/MG *(fix identified, ~1 line)* +The only 5/MG battery writer is standard `0x2A19` (`BLEManager.swift:3824-3830`, gated +`!= .whoop4`). Its one-shot `readValue` fires in `didDiscoverCharacteristicsFor` +(`:3467-3471`) — **pre-bond, on an unencrypted link**, which an MG rejects ("Authentication is +insufficient", see `:3476-3479`, `:3562-3567`). The failure hits the error branch (`:3806-3809`) +→ `log(...)` → `return`. **Never retried.** + +`:3586` calls `enableLiveNotifications(reason: "post-bond 5/MG")` with a comment claiming it recovers +"battery that failed pre-bond" — **it does not**; it only re-subscribes notify (`:2629-2654`), never +re-issues the read. HR recovers (notify pushes 1 Hz) so the UI says "Live"; battery never does. + +Proof it never landed: `LiveState.clearBiometrics()` (`:467-476`) clears `heartRate`/`rr`/ +`batterySamples` but **not** `batteryPct` — so `batteryPct == nil` proves `0x2A19` never delivered once. + +**Confirmed case (i):** the device log shows `Services discovered: 180D, 180A, 180F` — **180F IS +exposed**, so the characteristic exists and the read is simply never retried post-bond. + +**Fix:** re-issue the read for 5/MG at the end of `enableLiveNotifications` (idempotent; reuses the +3 existing call sites). **Fix 1b (more robust):** `FrameRouter.swift:202-205` already receives 5.0 +BATTERY_LEVEL events and reads `battery_charging` but **discards `battery_pct`** ("#77: flag only"). +The decode exists (`Interpreter.swift:742-753`, u16 @21, `/10`). The #77 stub-100 hazard is +**4.0-only**, so gating on family gives 5/MG a self-healing source refreshing ~every 8 min. + +### A2. "Active · Live" lies +`state.connected = true` is set in `didConnect` (`:3080`) **before** `discoverPrimaryServices` +(`:3113`) — true the instant the raw link exists, before any characteristic or byte. `DevicesView.swift:101,351` +shows "Active · Live" on `device.status == .active && live.connected`. Combined with iOS +state-restoration of a **dead** peripheral, the UI claimed "live" for a strap silent 21 hours. +**Fix:** gate the live indication on actual data flow / completed bond, not the raw link object. + +### A3. Charging can never display +`LiquidTodayView.swift:1529-1543`: the `bolt.fill` charging indicator is nested **inside** +`if let pct = live.batteryPct`. With pct nil the charging state is structurally unreachable — the +wearer was on the charger and the app could not say so. Also `:1546-1550`: `bolt.slash` is a +**no-data placeholder that reads as "battery dead"** (the wearer read it exactly that way). +**Fix:** render charging independent of pct; split "not connected" from "connected, no reading yet"; +accessibility label (`:1558`) should say "no reading yet". + +### A4. The app spins forever on a zombie restored peripheral +After iOS state-restoration returns a dead peripheral, the app marks it connected, the handshake never +completes, and it retries the **same dead handle** indefinitely — never cancelling, never re-scanning. +Observed: 40+ minutes of `Backfill: deferred — connect handshake not done yet` every 1–3 min, +persisting after the strap came off the charger. Only a **manual force-quit** recovered it. +**Fix:** bounded stalled-handshake timeout → cancel + re-scan. + +### A5. Backfill drain throttled by an arbitrary cap *(worst practical impact)* +`BLEManager.swift:320`: `static let defaultMaxAutoContinues = 6`. `shouldAutoContinue` (`:349`) +chains sessions back-to-back but every call site (`:1881`, `:1895`, `:1907`) caps at 6 → **7 sessions +per connection**, then falls back to `startBackfillTimer()` (`:2487`) → `backfillIntervalSeconds = 900` +(**15 min** between ~60s sessions). + +**Measured on device:** ~29 min of strap data recovered per session; ~18 h backlog ⇒ ~37 sessions ⇒ +7 free, then 30 × 15 min = **~7.5 hours** for data the strap streams in minutes. + +**The cap is redundant:** `shouldAutoContinue` already has `guard lastTrimAdvanced` (the real anti-spin +guard) plus future-clock checks. **Fix:** scale/bypass the cap while genuinely behind +(`strapNewestTs - frontier > behindGapSeconds` && `lastTrimAdvanced`), keeping a large runaway ceiling. +**Constraint:** each ~60s session blocks the WHOOP4 realtime keep-alive (`guard !backfilling`, see `:1509` +/ #160) — an aggressive drain must not starve live HR on a 4.0. + +**Workaround today:** `consecutiveCount` is **per connection** — force-quit + reopen resets it, buying +another ~7 sessions (~3.4 h of data) per cycle. + +### A6. Charging flag never lands +Newest live row: `ts 2026-07-15 10:58:06, soc=31.6, charging=0` **while actively charging**. Distinct +from A3 (there pct was nil; here pct exists and charging still reads 0). Unknown whether the 5/MG +emits the bit on this path at all. If it genuinely never reports it, the UI must **infer** charging +(rising SoC) or say nothing — never imply "not charging". + +### A7. `lastSeenAt` frozen at pairing → "last seen 18 days ago" +``` +my-whoop | added 2026-06-26 12:06:20 | lastSeen 2026-06-26 12:06:20 ← IDENTICAL +oura-api | added 2026-07-10 | lastSeen 2026-07-13 ← updates fine +``` +The strap's registry `lastSeenAt` is written once at pairing and **never refreshed**. Only the WHOOP +source is affected. + +--- + +## B. Observability (the wearer's #1 ask) + +### B1. Backfill progress is invisible +No way to see that a recovery is running, how far along, or how much remains. Direct quote: +> "All of this stuff, like backfill progress, should be something you can see on the app. It is kind of +> misleading me right now." + +The data exists (`strapNewestTs`, frontier/trim cursor, `backfilling`). Surface e.g. +*"Recovering strap history — July 14 14:31, ~15h behind"*. Pure, testable display-state function. + +### B2. Battery shown with no staleness marker +A 21-hour-old reading rendered identically to a live one. *"Last seen 21h ago at 11%"* would have +explained the entire incident instantly. + +### B3. Rest carries but Charge/Effort don't → a half-broken dashboard +On a dead-strap day the app shows a plausible Rest (carried from the last scored night) beside two +blanks, with **no signal that last night has no data**. Wearer repeatedly asked "are these stale?". +Verified: `2026-07-15` row is entirely NULL; the displayed Rest 72.4 / Charge 46.7 / Effort 25.2 are +**July 14's**, correctly computed. **Fix:** either carry consistently or say "no data for last night". + +### B4. Battery is invisible over MCP *(tool built, not deployed)* +The `battery` table (soc/mv/charging) and the `event` table (**11,457 rows**, `BATTERY_LEVEL` carrying +`battery_pct`) were never exposed. `battery_series`, `device_events`, `imu_coverage` are built + tested +in noop-cloud (branch `tz-upload`), **pending deploy**. + +### B5. Burst-capture census + firmware are not in the DB at all +`PuffinRawProbe.finishSession()` writes the R22 census JSON only to a file + +`UserDefaults(noop.probe.rawCensus)`; firmware lives in `UserDefaults(noop.lastFirmware)`, and neither +`device` nor `pairedDevice` has a firmware column. `ingest.ts` drops `settings.json` entirely. +⇒ **The MCP can never see R22 capture health or strap firmware** without a producer-side migration +(e.g. `probeRun(deviceId, ts, version, fw, capturedFrames, censusJSON)`). + +--- + +## C. Alerts + +### C1. Battery alerts fire once, then latch — no escalation +Device state: `batteryLowAlerted = true`, `batteryRuntimeAlerted = true` — **both fired, and the strap +still died.** `BatteryAlertPolicy` fires low once per discharge cycle, re-arming only above +`lowRearmAbove = 25`. The runtime alert fires at `runtimeAlertHours = 24` — at the measured 1.65 %/h +that is **39.6% SoC, ~18 h early** — then latches. Net: **~3 hours of silence from 15% to death.** +Both toggles default ON (`BehaviorStore.swift:100-101`), so that is not the explanation. +*Being fixed:* critical 12% alert + bedtime night-guard, each with independent gates. + +### C2. Runtime estimate ignores the strap's cutoff +`BatteryEstimator.swift:99`: `remaining = max(0, current) / rate` — time to **0%**. The strap appears +to stop near ~10%, so every estimate overstates runway. + +### C3. Rated-path predictive alert is unreachable on a 5.0 +`ratedHours = 288` (`BLEManager.swift:1005`) puts the 24 h line at **8.33% SoC — below the cutoff**. +A 5.0 falling back to the rated path therefore gets **no predictive alert, ever**. + +--- + +## D. Cloud sync + +### D1. Upload can never complete from the background *(fix built)* +`CloudSyncClient.swift:34` uses `URLSession.shared` (foreground). Zero `beginBackgroundTask`, zero +background `URLSessionConfiguration`. A push grants ~30 s; deflate-zipping ~400 MB (→ ~81.5 MB) blows +past it and in-flight tasks die on suspension. Device proof: `cloudsync.lastStatus` frozen at +`"Uploaded 81.5 MB · 12:21 PM"` yesterday — and it is written on **every** completion, success or +failure ⇒ nothing has *completed* since. + +### D2. Warm resume triggers no sync +`autoSyncIfDue`'s only iOS call site is `RootTabView.swift:133` inside `.task {}`, which does not +re-run on resume. `scenePhase == .active` (`StrandiOSApp.swift:205-223`) does not call CloudSync. + +### D3. Server volume ~1 upload from ENOSPC +1008 MiB total, ~461 used, **~547 free**; an ingest needs ~400 MB staged **alongside** the ~460 MB +mirror. `MAX_INGEST_BYTES = 768MB` exceeds the entire volume. Also `/ingest` logs nothing on receipt, +success, or the 400 path — it cannot distinguish "nothing arrived" from "arrived and was rejected". + +--- + +## E. Deep data / MG + +### E1. 2140 B optical frames are rejected — no home on this build +Live log: `Backfill: rejected frame[N] 2140B` (repeating). The live DB has **no `ppgWaveformSample` +table** (migrations top out at v28; upstream's `v27-ppg-waveform` / #415 is newer than the installed +build). ⇒ Every 2140 B optical buffer — **the SpO2/BP substrate** — is dropped on the floor. +**Action:** rebuild + install current fork main (which merged v27) to start storing them. + +### E2. 1244 B (IMU) frames also logged as rejected — unexplained +IMU decode demonstrably works (`imuActivity` = 2,945 rows), yet 1244 B frames appear in the reject log +during this offload. Needs investigation; do not guess. + +### E3. Deep IMU capture is barely running +`imuActivity`: **2,945 rows — all inside ONE 58-minute window across 8 months.** Whatever gates the +deep-buffer capture is almost never on. + +--- + +## F. Scoring (latent) + +### F1. Charge silently vanishes after 14 stale nights +`Baselines.swift:40` documents `case stale // usable but no update for > STALE_DAYS nights`, but +`:69` defines `usable = (status == .provisional || status == .trusted)` — **`.stale` excluded** — and +`staleDays = 14`. `RecoveryScorer.swift:417` then does `if !hrvBaselineUsable { return nil }`. +⇒ 14 nights without a valid HRV night ⇒ **Charge silently returns nil**, no explanation surfaced. +The comment and the code disagree; one of them is wrong. From 7f801540c2648b9ff205ec6e33ac0e1266ec14cb Mon Sep 17 00:00:00 2001 From: vishk23 Date: Wed, 15 Jul 2026 11:24:19 -0700 Subject: [PATCH 110/171] README: add fork preamble describing my upstream contributions Co-Authored-By: Claude Fable 5 --- README.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/README.md b/README.md index 1b3cb25510..6d3398222f 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,27 @@ +> ### About this fork +> +> This is **Vishnu Kchittibhooma's** working fork of [NOOP](https://github.com/ryanbr/noop), +> the offline WHOOP companion app by [@ryanbr](https://github.com/ryanbr) and contributors. +> The base app is their work; this fork is where my contributions to it are developed. +> +> **What I've contributed upstream:** 13 pull requests merged into `ryanbr/noop` +> ([see the merged PRs](https://github.com/ryanbr/noop/pulls?q=is%3Apr+author%3Avishk23+is%3Amerged)), +> concentrated in the sensor-protocol and analytics layers: +> +> - **WHOOP 5.0/MG raw sensor decoding** — 6-axis IMU offload buffer + activity features (#455), +> high-rate R22 deep buffers (#454), and persisting the v26 raw PPG waveform (#415) +> - **Sleep scoring** — motion-corroborated wake detection so elevated HR on a motionless +> wrist no longer scores WAKE (#465, #402) +> - **Analytics** — Recovery Index + Activity Balance terms in Charge (#417), an opt-in +> coarse workout-type classifier (#414) +> - **Import correctness** — Oura readiness/efficiency unit fixes across Swift + Kotlin +> +> This fork's `main` tracks upstream and carries my additional work in progress; see +> [branches](https://github.com/vishk23/noop/branches) for active experiments. +> The upstream README follows unchanged. + +--- +

NOOP

From 7942f667d545d8191cdc5a54b61e9738dd8d9f86 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 15 Jul 2026 11:53:59 -0700 Subject: [PATCH 111/171] =?UTF-8?q?Offload:=20stop=20archiving=20v20/v21?= =?UTF-8?q?=20as=20"undecodable"=20=E2=80=94=20they=20decode,=20and=20it?= =?UTF-8?q?=20costs=20~4x=20throughput?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reject archive exists to capture layouts we CANNOT decode, so they can be mapped later. v20 (2140 B) and v21 (1244 B) are already mapped: decodeWhoop5HistoricalV2021 parses them into optical/IMU channels. They only LOOK undecodable to the unmapped-layout predicate because it asks for heart_rate or gravity_x, which these layouts legitimately don't carry — precisely why v26 is already exempted one line above. The cost is not cosmetic. The strap banks one v18 + one v21 + one v20 per SECOND of history (3508 B, of which only v18's 124 B is stored), so a real backlog sends ~34 of every 51 records down this path. Each is hex-dumped to the strap log (a 2140 B frame = 4280 chars, which persistTail then serialises into a multi-MB plist) and appended to the 5 MB raw archive. Once that archive reaches its cap, EVERY chunk pays a read-all → hex-decode-every-line → rewrite-5 MB-and-fsync cycle, synchronously, with the trim ack waiting behind it. Measured on a real 16 h backlog (fw 50.40.1.0): throughput collapsed from ~3.7x real-time (archive freshly rotated, cheap append path) to ~1.0x within two minutes of the cap being reached — turning a ~3 h drain into ~12 h. It also drove the false "Synced, but 34 record(s) couldn't be decoded" banner. Fixtures are real header bytes from that capture. Kotlin twin needed: rejectedHistoricalRecords mirrors the Android implementation. --- .../WhoopProtocol/HistoricalStreams.swift | 15 +++++++++++++ .../RejectedHistoryTests.swift | 21 +++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift index 9da630c93f..8994a3eed7 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift @@ -74,6 +74,21 @@ public func rejectedHistoricalRecords(_ rawFrames: [[UInt8]], family: DeviceFami // different type byte, so they never pass this gate — they are excluded by construction. guard f.count > typeIndex, Int(f[typeIndex]) == 47 else { return false } if family == .whoop5, f.count > versionIndex, Int(f[versionIndex]) == 26 { return false } // v26 PPG: has its own durable stream (ppgWaveform), not this reject archive + // v20 (2140 B) / v21 (1244 B) are KNOWN-AND-DECODED layouts, not undecodable ones: + // `decodeWhoop5HistoricalV2021` parses them into optical/IMU channels. They only LOOK undecodable + // to the predicate below because that test asks for `heart_rate`/`gravity_x`, which these layouts + // legitimately do not carry — the same reason v26 is exempted one line above. + // + // Archiving them is actively harmful, not merely redundant. The strap banks one v18 + one v21 + + // one v20 per SECOND of history (3508 B, of which only v18's 124 B is stored), so a real backlog + // sends ~34 of every 51 records down this path. Each one is hex-dumped to the strap log and + // appended to the 5 MB raw archive; once that archive hits its cap EVERY chunk pays a + // read-all → hex-decode-every-line → rewrite-5 MB-and-fsync cycle, synchronously, and the trim + // ack waits behind it. Measured on a real 16 h backlog: offload throughput collapsed from ~3.7x + // real-time (archive freshly rotated, cheap append path) to ~1.0x within two minutes of the cap + // being reached — turning a ~3 h drain into ~12 h. The archive exists to capture layouts we + // CANNOT decode so they can be mapped later; these two are already mapped. + if family == .whoop5, f.count > versionIndex, Int(f[versionIndex]) == 20 || Int(f[versionIndex]) == 21 { return false } let p = parseFrame(f, family: family) // Envelope/CRC reject: parse failed outright or the CRC32 trailer mismatched. if !p.ok || p.crcOK == false { return true } diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/RejectedHistoryTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/RejectedHistoryTests.swift index 4101bd928a..f5441a3b5e 100644 --- a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/RejectedHistoryTests.swift +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/RejectedHistoryTests.swift @@ -77,6 +77,27 @@ final class RejectedHistoryTests: XCTestCase { XCTAssertTrue(rejectedHistoricalRecords([v26], family: .whoop5).isEmpty) } + func testWhoop5V20AndV21DeepBuffersExcluded() { + // Header bytes taken verbatim from a real 5/MG offload (fw 50.40.1.0): the strap banks one v18 + + // one v21 (1244 B) + one v20 (2140 B) per SECOND of history, so on a real backlog these are ~34 + // of every 51 records. Both DECODE fine (`decodeWhoop5HistoricalV2021` → optical/IMU channels); + // they only LOOK undecodable to the unmapped-layout predicate because they carry no heart_rate + // and no gravity_x — exactly why v26 is exempted too. Archiving them hex-dumps ~3.4 KB/s of + // history into the 5 MB raw archive, whose at-cap rewrite then blocks the trim ack. + let v21 = bytes("aa01d40401005c702f1580c585af000ea4566a1e45046400640003005bfb53fb66fb5dfb") + let v20 = bytes("aa0154080100b5b32f1481c585af000ea4566a1e4504001900001901160d042c1a032000") + XCTAssertEqual(v21[8], 47) // type-47 HISTORICAL_DATA… + XCTAssertEqual(v21[9], 21) // …version 21 — a known, decoded layout + XCTAssertEqual(v20[8], 47) + XCTAssertEqual(v20[9], 20) // …version 20 — likewise + XCTAssertTrue(rejectedHistoricalRecords([v21], family: .whoop5).isEmpty, + "v21 deep buffer decodes — it must not be archived as undecodable") + XCTAssertTrue(rejectedHistoricalRecords([v20], family: .whoop5).isEmpty, + "v20 deep buffer decodes — it must not be archived as undecodable") + // The whole batch shape a real offload delivers: nothing here belongs in the reject archive. + XCTAssertTrue(rejectedHistoricalRecords([v20, v21, v20, v21], family: .whoop5).isEmpty) + } + func testNonHistoricalFrameExcluded() { // A REALTIME_DATA (type-40) frame is live, not offload — never a history-loss candidate. let realtime = frameFromPayload([0x01, 0x02, 0x03], type: 40, seq: 0, cmd: 0) From 2b968d27a74c4a7ef80f7e79e6db85f1010490ec Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 15 Jul 2026 13:14:14 -0700 Subject: [PATCH 112/171] Backfill: stop capping a healthy recovery at 6 continues, and stop paying 60 s to notice a session ended Two duty-cycle bugs. Neither is in the per-chunk path, which is why staring at decode/SQLite/ack found nothing: offload is airtime-limited at ~25.4 KB/s and the app-side work is single-digit milliseconds per chunk. The losses are in the gaps BETWEEN sessions. 1. defaultMaxAutoContinues 6 -> 120. The old rationale ("6 x ~60s ~= 6 min of back-to-back draining -- enough to chew through a multi-night backlog") is arithmetically wrong. At ~7x real-time, 6 minutes of draining recovers ~42 minutes of history. Recovering one dead-strap night (~19 h) needs ~2.7 h of draining, so the cap fires mid-recovery and drops to backfillIntervalSeconds (900 s) -- a 6.7% duty cycle that turns a ~3 h drain into 12 h+. This is A5 in docs/bugs/2026-07-15-strap-battery-backfill-observability.md, measured on a real 18 h backlog. Guards 2 (strap still ahead of our frontier) and 3 (trim cursor advanced) are the real runaway protection and stop on evidence; the count is a last-resort backstop and 120 still bounds it. The cap resets on disconnect, so it never was a battery ceiling across a day -- only a ceiling on one continuous recovery, which is exactly when we want to keep going. 2. backfillIdleTimeoutSeconds 60 -> 15. Every session pays this in full as dead air on exit. 60 s was set without measuring the inter-frame tail. A 3,018-buffer arrival trace from a real 19 h drain: p50 64 ms, p90 106 ms, p99 273 ms, max 2,520 ms, with the tail at chunk boundaries. 15 s is ~6x the measured max and still clears the type-43 flood. Not dropping to the ~5 s the tail alone would justify -- that tail is one drain on one firmware, and the risk is asymmetric. Also removes the inline IMU decode from the offload delivery path (live frames keep it; the raw hex on the same line is the decoder's input, so the summary is recomputable offline). That is hygiene, not throughput -- measured 7.2x -> 7.7x. The comment now records why the 1244-B/2140-B gap asymmetry is airtime and not a stall, since it reads like a stall and has now fooled one reader. Kotlin twins needed: BackfillContinuation cap, idle timeout, PuffinDeepBufferLog. --- Strand/BLE/BLEManager.swift | 44 ++++++++++++++--- Strand/BLE/PuffinDeepBufferLog.swift | 57 ++++++++++++++++++++-- StrandTests/PuffinDeepBufferLogTests.swift | 23 +++++++++ 3 files changed, 112 insertions(+), 12 deletions(-) diff --git a/Strand/BLE/BLEManager.swift b/Strand/BLE/BLEManager.swift index d6d86505f5..9dd6c00860 100644 --- a/Strand/BLE/BLEManager.swift +++ b/Strand/BLE/BLEManager.swift @@ -314,10 +314,26 @@ struct Whoop5EmptyOffloadTracker { /// its idle timeout — the reported ~15-min sync). The stale/PAST-epoch case 2b actually exists for (#451) /// reads BEHIND the frontier, never future-dated, so it is untouched. struct BackfillContinuation { - /// Hard cap on consecutive auto-continues per connection (resets on disconnect). 6 × ~60s ≈ 6 min of - /// back-to-back draining — enough to chew through a multi-night backlog far faster than the 15-min - /// floor, without letting a misbehaving strap monopolise Bluetooth. - static let defaultMaxAutoContinues = 6 + /// Hard cap on consecutive auto-continues per connection (resets on disconnect), as a LAST-RESORT + /// backstop only. Guards 2 and 3 are the real protection and they stop on *evidence*: 2 stops the + /// moment the strap is no longer ahead of our frontier, 3 stops the moment a session fails to move + /// the trim cursor. A strap that is both genuinely behind AND genuinely handing over data is not + /// pathological — it is working, and cutting it off is the bug. + /// + /// The old value of 6 came with the rationale "6 × ~60s ≈ 6 min of back-to-back draining — enough to + /// chew through a multi-night backlog". That premise is arithmetically wrong. Offload runs at ~7x + /// real-time (airtime-limited at ~25.4 KB/s against the 3508 B the strap banks per second-of-history), + /// so six minutes of draining recovers ~42 minutes of history — not a multi-night backlog. Recovering + /// a single dead-strap night (~19 h) needs ~2.7 h of draining, i.e. ~50+ sessions. At 6 the cap fires + /// mid-recovery and drops to `backfillIntervalSeconds` (900 s) — a 6.7% duty cycle, turning a ~3 h + /// drain into ~12 h+. That is the "worst practical impact" recorded in + /// docs/bugs/2026-07-15-strap-battery-backfill-observability.md (A5), measured on a real 18 h backlog. + /// + /// 120 covers a full night's recovery with ~2x margin while still bounding a truly runaway loop that + /// somehow satisfies guards 2 and 3 forever. The cap resets on disconnect, so it was never a battery + /// ceiling across a day — only a ceiling on one continuous recovery, which is exactly when we want to + /// keep going. + static let defaultMaxAutoContinues = 120 /// How far ahead the strap must be (seconds) before "more backlog remains" is real, not clock noise. /// Matches StuckStrapDetector.behindGapSeconds (5 min) so the two agree on "behind". static let defaultBehindGapSeconds = 300 @@ -1643,10 +1659,22 @@ public final class BLEManager: NSObject, ObservableObject { /// Re-arm the idle watchdog. Called on every offload frame during backfill so the timer resets /// as long as the strap keeps sending HISTORY; if the strap goes silent the timer fires and we /// exit the session (the durable strap_trim cursor means the next session resumes where we left - /// off). Timeout is generous (60 s, not 20 s): the unstoppable ~2/s type-43 raw flood eats BLE - /// airtime, so genuine offload frames can arrive in bursts with multi-second lulls between chunks - /// — a short watchdog cut sessions short mid-drain. Longer = more records drained per session. - static let backfillIdleTimeoutSeconds = 60 + /// off). Timeout must clear the real inter-frame tail: the unstoppable ~2/s type-43 raw flood eats + /// BLE airtime, so genuine offload frames arrive with lulls between chunks — a short watchdog cut + /// sessions short mid-drain. + /// + /// 60 s was set without a measurement of that tail; it is ~24x the real worst case, and every session + /// pays it in full as dead air on the way out. A 3,018-buffer arrival trace from a real 19 h drain + /// puts inter-frame gaps at p50 64 ms / p90 106 ms / p99 273 ms / max 2,520 ms — the tail sits at + /// chunk boundaries (~51 records, where decode + insert + cursor + ack happen), not mid-chunk. 15 s is + /// ~6x that measured max, still clears the flood, and cuts the per-session exit cost by 4x. + /// + /// Deliberately NOT the ~5 s the tail alone would justify: that margin protects against a tail we have + /// sampled over ONE drain on ONE firmware, and the downside is asymmetric. Firing early is safe but + /// wasteful — the in-memory chunk is dropped un-acked, so the strap keeps it and the next session + /// refetches — whereas firing late only costs idle seconds. Buy the cheap 4x; leave the risky 3x until + /// the tail is instrumented across firmwares. + static let backfillIdleTimeoutSeconds = 15 private func armBackfillTimeout() { backfillTimeout?.cancel() let item = DispatchWorkItem { [weak self] in diff --git a/Strand/BLE/PuffinDeepBufferLog.swift b/Strand/BLE/PuffinDeepBufferLog.swift index 96732134ab..ba1597489e 100644 --- a/Strand/BLE/PuffinDeepBufferLog.swift +++ b/Strand/BLE/PuffinDeepBufferLog.swift @@ -31,7 +31,17 @@ final class PuffinDeepBufferLog { /// The 2140-B buffers are ~4.3 KB of hex each and arrive in bursts, so a bigger cap than the /// EVENT log: 60 MB live keeps roughly a few hours of accumulated high-rate bursts, ample to /// reverse the layout, and rotation bounds total disk at ~120 MB. - private static let softCapBytes = 60 * 1024 * 1024 + /// + /// Overridable via the `PuffinDeepBufferSoftCapMB` default because the 60 MB figure is sized for + /// "a few hours of bursts", which is the STEADY-STATE case. Recovering a dead-strap backlog is the + /// other case: the strap banks one 1244-B + one 2140-B buffer per SECOND of history, so a 16 h + /// backlog is ~190 MB raw / ~380 MB of hex — it would rotate away most of itself before the drain + /// ends, and the ack-trim frees those buffers from the strap permanently as it goes. A one-shot + /// raise is the only way to keep a full backlog intact. 0 or unset keeps the 60 MB default. + private static var softCapBytes: Int { + let mb = UserDefaults.standard.integer(forKey: "PuffinDeepBufferSoftCapMB") + return (mb > 0 ? mb : 60) * 1024 * 1024 + } /// WHOOP 5/MG inner-record type byte for the R22 deep packets (type 47 / 0x2F), at offset 8 (the /// same position `BLEManager.isOffloadFrame` / `PuffinEventLog` index). The 1 Hz historical rollup @@ -41,6 +51,23 @@ final class PuffinDeepBufferLog { /// Skip the ~124-B ≈1 Hz record; keep the 1244-/2140-B high-rate buffers. private static let minBufferBytes = 1000 + /// Lowercase hex, table-driven. NOT cosmetic: this runs on the MainActor inside the offload ack path, + /// and the obvious `frame.map { String(format: "%02x", $0) }.joined()` costs one Foundation formatter + /// call + one String allocation PER BYTE — ~3 ms for a 2140-B buffer. The strap banks one 1244-B and + /// one 2140-B buffer per second of history, so draining a backlog at 5x real-time formats ~170 + /// buffers/sec ≈ 400 ms of every second, and the strap sits idle waiting for the ack behind it. A + /// nibble lookup into a byte array is ~80x cheaper and allocates once. + private static let hexDigits: [UInt8] = Array("0123456789abcdef".utf8) + nonisolated static func hexString(_ bytes: [UInt8]) -> String { + var out = [UInt8]() + out.reserveCapacity(bytes.count * 2) + for b in bytes { + out.append(hexDigits[Int(b >> 4)]) + out.append(hexDigits[Int(b & 0x0F)]) + } + return String(decoding: out, as: UTF8.self) + } + /// Pure predicate: is `frame` a WHOOP 5/MG high-rate deep buffer? A reassembled frame's inner-record /// type byte sits at offset 8, so this needs `count > 8` before indexing. Extracted so the offset-8 /// magic number is unit-testable without a strap. @@ -54,7 +81,10 @@ final class PuffinDeepBufferLog { private var disabled = false private var isEnabled: Bool { - UserDefaults.standard.bool(forKey: PuffinFrameRecorder.enabledKey) + // LOCAL personal-capture build: force-ON + 600 MB cap to catch the 2026-07-14 dead-strap backlog + // whole (~380 MB of hex) before the ack-trim frees it from the strap. Never commit this. + UserDefaults.standard.set(600, forKey: "PuffinDeepBufferSoftCapMB") + return true } /// `/OpenWhoop/puffin-deepbuffers.jsonl` — OUTSIDE `puffin-captures/`, whose soft-cap @@ -96,14 +126,33 @@ final class PuffinDeepBufferLog { guard !disabled, Self.isDeepBuffer(frame), isEnabled else { return } let tsMs = Int(Date().timeIntervalSince1970 * 1000) let strapTs = Self.strapTs(frame).map { String($0) } ?? "null" - let hex = frame.map { String(format: "%02x", $0) }.joined() + let hex = Self.hexString(frame) // #423/#455: run the raw-IMU decoder on the 1244-B buffer so every captured IMU frame carries // its decoded activity summary (cadence/energy/jerk/gyro) inline beside the raw hex. This is the // first CALLER of `Whoop5RawImu.decode` outside its own tests — it exercises the decoder on real // device captures and makes each JSONL line self-checking (raw ↔ decode) with NO stored table, // migration, or downstream gate. Instrumentation only, per the derived-signal rule; the 2140-B // optical buffer stays raw-only (its layout isn't decoded yet). - let imu = Self.decodedImuField(frame) + // LIVE frames only — a hygiene measure, NOT a throughput fix. Keeping the honest numbers here so + // nobody re-derives the wrong conclusion from this file a third time: + // + // A real 19 h drain (3,018 buffers / 208 s) shows a mean gap of 89.1 ms after a 1244-B buffer vs + // 49.0 ms after a 2140-B one. That asymmetry is NOT this decode — it is pure airtime, and reading + // it as a stall is an off-by-one. A gap is the time for the NEXT frame(s) to arrive, not the + // previous frame's processing, and the ~124-B v18 record sits between the two logged buffers + // (it's below `minBufferBytes`, so it is invisible here). So the gap after the 1244-B spans + // 124+2140 = 2264 B and the gap after the 2140-B spans only 1244 B. One free parameter — the link + // rate — fixed by the total (3508 B / 138.1 ms = 25.4 KB/s) then predicts the SPLIT with zero + // further freedom: 89.13 ms and 48.97 ms, against 89.1 and 49.0 observed. The two gaps + // independently agree on the byte rate to 0.09%. The link is a metronome; there is no stall. + // Removing this decode measured 7.2x -> 7.7x, consistent with its true ~7 ms cost, not 40 ms. + // + // It is still right to skip during offload: it is per-frame work on the MainActor in the delivery + // path, it buys nothing there, and the raw `hex` on this very line is the decoder's own input — so + // the summary is recomputable offline from the archive at any time. Live frames arrive ~1/s, where + // the inline summary is free and makes each line self-checking. Capture captures; analysis is + // downstream. The real duty-cycle costs are the idle watchdog and the auto-continue cap. + let imu = isOffload ? "" : Self.decodedImuField(frame) let line = "{\"ts_ms\":\(tsMs),\"strap_ts\":\(strapTs),\"size\":\(frame.count)," + "\"offload\":\(isOffload),\"char\":\"\(char.uuidString.lowercased())\",\"hex\":\"\(hex)\"\(imu)}\n" do { diff --git a/StrandTests/PuffinDeepBufferLogTests.swift b/StrandTests/PuffinDeepBufferLogTests.swift index 9d5e0cb8b5..895e1d78e1 100644 --- a/StrandTests/PuffinDeepBufferLogTests.swift +++ b/StrandTests/PuffinDeepBufferLogTests.swift @@ -18,6 +18,29 @@ final class PuffinDeepBufferLogTests: XCTestCase { XCTAssertTrue(PuffinDeepBufferLog.isDeepBuffer(g)) } + func testHexStringMatchesFormatterForEveryByteValue() { + // Byte-for-byte parity with the `String(format: "%02x")` encoding it replaces, across the whole + // domain — the JSONL is a research archive, so a wrong nibble is silent corruption of the only + // copy of buffers the strap has already freed. + let all = (0...255).map { UInt8($0) } + XCTAssertEqual(PuffinDeepBufferLog.hexString(all), + all.map { String(format: "%02x", $0) }.joined()) + XCTAssertEqual(PuffinDeepBufferLog.hexString([]), "") + XCTAssertEqual(PuffinDeepBufferLog.hexString([0x00, 0x0F, 0xF0, 0xFF]), "000ff0ff") + } + + func testHexStringIsFastEnoughForTheAckPath() { + // Guards the regression that matters: this runs on the MainActor inside the offload ack path, so + // a per-byte Foundation formatter (~3 ms per 2140-B buffer) throttles the whole drain. Encoding + // one buffer must stay far under a millisecond. Threshold is deliberately loose — it's catching + // an 80x cliff, not micro-benchmarking. + let frame = (0..<2140).map { UInt8($0 & 0xFF) } + let start = Date() + for _ in 0..<100 { _ = PuffinDeepBufferLog.hexString(frame) } + let perBuffer = Date().timeIntervalSince(start) / 100 + XCTAssertLessThan(perBuffer, 0.001, "hex encode regressed to \(perBuffer * 1000) ms/buffer") + } + func testRejectsSmallType2FRecord() { // The ~124-B type-0x2F frame is the 1 Hz rollup NOOP already decodes — below the size gate, // it must NOT be logged as a high-rate buffer. From 2189f2559febf60ff6acd00caed3bcdde8d9ee53 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 15 Jul 2026 13:25:04 -0700 Subject: [PATCH 113/171] Revert force-enabled deep-buffer capture that should never have been committed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 2b968d27 swept a local personal-capture hack into the tree: `isEnabled` was made to `return true` unconditionally (and to write a 600 MB soft cap as a side effect of a getter). Its own comment said "Never commit this." It did. The effect was not cosmetic — it force-enabled the high-rate capture for every user, defeating the `PuffinFrameRecorder.enabledKey` opt-in, in an otherwise upstream-safe file. The log writes ~600 MB/day of hex for a strap worn continuously, so shipping this would fill a user's disk for a research capture they never asked for. `isEnabled` goes back to reading the Settings toggle. The soft-cap override stays: it's a real change and is default-60 MB, so it is inert unless deliberately set. --- Strand/BLE/PuffinDeepBufferLog.swift | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/Strand/BLE/PuffinDeepBufferLog.swift b/Strand/BLE/PuffinDeepBufferLog.swift index ba1597489e..124fa081a6 100644 --- a/Strand/BLE/PuffinDeepBufferLog.swift +++ b/Strand/BLE/PuffinDeepBufferLog.swift @@ -81,10 +81,7 @@ final class PuffinDeepBufferLog { private var disabled = false private var isEnabled: Bool { - // LOCAL personal-capture build: force-ON + 600 MB cap to catch the 2026-07-14 dead-strap backlog - // whole (~380 MB of hex) before the ack-trim frees it from the strap. Never commit this. - UserDefaults.standard.set(600, forKey: "PuffinDeepBufferSoftCapMB") - return true + UserDefaults.standard.bool(forKey: PuffinFrameRecorder.enabledKey) } /// `/OpenWhoop/puffin-deepbuffers.jsonl` — OUTSIDE `puffin-captures/`, whose soft-cap From 86d414ad48d84effb2d38d470c3cd81efaff40de Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 15 Jul 2026 14:14:17 -0700 Subject: [PATCH 114/171] =?UTF-8?q?Revert=20"Offload:=20stop=20archiving?= =?UTF-8?q?=20v20/v21=20as=20undecodable"=20=E2=80=94=20it=20silently=20de?= =?UTF-8?q?stroys=20them?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The exemption was justified by "they decode, for the same reason v26 is exempted one line above." That is backwards, and the difference is load-bearing. v26 is safe to exempt because it HAS a durable home: extractHistoricalStreams appends it to Streams.ppgWaveform (HistoricalStreams.swift:216) and WhoopStore persists it to ppgWaveformSample. The reject archive is redundant for it. v20/v21 have no home. decodeWhoop5HistoricalV2021 writes optical_ch0..2 / channel_b*_* into an in-memory ParsedFrame with ZERO production consumers on either platform — extractHistoricalStreams never reads those keys, and the frame is discarded. So with the exemption: not archived, not persisted, trim acked, and the strap frees the buffers permanently. PuffinDeepBufferLog does not catch them either — it is default-OFF and gated on PuffinFrameRecorder.enabledKey, a different key from the deep-data experiment. Two things the original missed. RawHistoryArchive.perVersionFloor (:27-32) names "WHOOP 5 v20/v21" as the rare-layout case its eviction floor was built for (#344) — the exemption deleted precisely what a prior fix exists to preserve. And BLEManager.swift:925-930 replays the archive and retro-decodes it on version bump: for v20, whose micro-decode is unsolved, that archive is the ONLY path by which the data is ever recovered once a decoder lands. The stated throughput rationale was also wrong: the "~3.7x -> ~1.0x collapse" was the strap being CAUGHT UP, not a bug — history cannot drain faster than the 3508 B/s at which it is banked. The real problems are real but belong elsewhere: the false "N records couldn't be decoded" banner should stop counting known layouts (BLEManager.swift:1801-1803), and the archive's at-cap read-all + hex-decode + 5 MB atomic rewrite per chunk should be fixed IN the archive. Neither justifies dropping the bytes. --- .../WhoopProtocol/HistoricalStreams.swift | 15 ------------- .../RejectedHistoryTests.swift | 21 ------------------- 2 files changed, 36 deletions(-) diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift index 8994a3eed7..9da630c93f 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift @@ -74,21 +74,6 @@ public func rejectedHistoricalRecords(_ rawFrames: [[UInt8]], family: DeviceFami // different type byte, so they never pass this gate — they are excluded by construction. guard f.count > typeIndex, Int(f[typeIndex]) == 47 else { return false } if family == .whoop5, f.count > versionIndex, Int(f[versionIndex]) == 26 { return false } // v26 PPG: has its own durable stream (ppgWaveform), not this reject archive - // v20 (2140 B) / v21 (1244 B) are KNOWN-AND-DECODED layouts, not undecodable ones: - // `decodeWhoop5HistoricalV2021` parses them into optical/IMU channels. They only LOOK undecodable - // to the predicate below because that test asks for `heart_rate`/`gravity_x`, which these layouts - // legitimately do not carry — the same reason v26 is exempted one line above. - // - // Archiving them is actively harmful, not merely redundant. The strap banks one v18 + one v21 + - // one v20 per SECOND of history (3508 B, of which only v18's 124 B is stored), so a real backlog - // sends ~34 of every 51 records down this path. Each one is hex-dumped to the strap log and - // appended to the 5 MB raw archive; once that archive hits its cap EVERY chunk pays a - // read-all → hex-decode-every-line → rewrite-5 MB-and-fsync cycle, synchronously, and the trim - // ack waits behind it. Measured on a real 16 h backlog: offload throughput collapsed from ~3.7x - // real-time (archive freshly rotated, cheap append path) to ~1.0x within two minutes of the cap - // being reached — turning a ~3 h drain into ~12 h. The archive exists to capture layouts we - // CANNOT decode so they can be mapped later; these two are already mapped. - if family == .whoop5, f.count > versionIndex, Int(f[versionIndex]) == 20 || Int(f[versionIndex]) == 21 { return false } let p = parseFrame(f, family: family) // Envelope/CRC reject: parse failed outright or the CRC32 trailer mismatched. if !p.ok || p.crcOK == false { return true } diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/RejectedHistoryTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/RejectedHistoryTests.swift index f5441a3b5e..4101bd928a 100644 --- a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/RejectedHistoryTests.swift +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/RejectedHistoryTests.swift @@ -77,27 +77,6 @@ final class RejectedHistoryTests: XCTestCase { XCTAssertTrue(rejectedHistoricalRecords([v26], family: .whoop5).isEmpty) } - func testWhoop5V20AndV21DeepBuffersExcluded() { - // Header bytes taken verbatim from a real 5/MG offload (fw 50.40.1.0): the strap banks one v18 + - // one v21 (1244 B) + one v20 (2140 B) per SECOND of history, so on a real backlog these are ~34 - // of every 51 records. Both DECODE fine (`decodeWhoop5HistoricalV2021` → optical/IMU channels); - // they only LOOK undecodable to the unmapped-layout predicate because they carry no heart_rate - // and no gravity_x — exactly why v26 is exempted too. Archiving them hex-dumps ~3.4 KB/s of - // history into the 5 MB raw archive, whose at-cap rewrite then blocks the trim ack. - let v21 = bytes("aa01d40401005c702f1580c585af000ea4566a1e45046400640003005bfb53fb66fb5dfb") - let v20 = bytes("aa0154080100b5b32f1481c585af000ea4566a1e4504001900001901160d042c1a032000") - XCTAssertEqual(v21[8], 47) // type-47 HISTORICAL_DATA… - XCTAssertEqual(v21[9], 21) // …version 21 — a known, decoded layout - XCTAssertEqual(v20[8], 47) - XCTAssertEqual(v20[9], 20) // …version 20 — likewise - XCTAssertTrue(rejectedHistoricalRecords([v21], family: .whoop5).isEmpty, - "v21 deep buffer decodes — it must not be archived as undecodable") - XCTAssertTrue(rejectedHistoricalRecords([v20], family: .whoop5).isEmpty, - "v20 deep buffer decodes — it must not be archived as undecodable") - // The whole batch shape a real offload delivers: nothing here belongs in the reject archive. - XCTAssertTrue(rejectedHistoricalRecords([v20, v21, v20, v21], family: .whoop5).isEmpty) - } - func testNonHistoricalFrameExcluded() { // A REALTIME_DATA (type-40) frame is live, not offload — never a history-loss candidate. let realtime = frameFromPayload([0x01, 0x02, 0x03], type: 40, seq: 0, cmd: 0) From d2ec5dc3c1434a21a25199224c4bc90a9b696cf9 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 15 Jul 2026 14:16:29 -0700 Subject: [PATCH 115/171] Backfill: cut the idle watchdog 60s -> 15s; keep the auto-continue cap at 6 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The watchdog is paid in full as dead air on every session exit, and 60s was set without measuring the inter-frame tail. A 3,018-buffer arrival trace from a real drain: p50 64 ms, p90 106 ms, p99 273 ms, max 2,520 ms, with the tail at chunk boundaries. 15s is ~6x the measured max and still clears the type-43 flood. Not dropping to the ~5s the tail alone would justify: that is one drain on one firmware, and the risk is asymmetric — firing early is safe but wasteful (the in-memory chunk is dropped un-acked, so the strap keeps it and the next session refetches), while firing late only costs idle seconds. Kotlin twin included. The cap stays at 6, and the reasoning is recorded so it is not "fixed" again: the evidence that it severed real recoveries came from a strap an experimental probe had switched into deep-data (R22) banking — 3508 B/s of history vs the default 124 B/s. A default strap banks 28x less, one session covers ~28x more history, and an 18 h backlog drains in ~2 sessions. Raising it would also widen the blast radius of the next guard-2 bug (#928/#1012 both burned the whole cap in EMPTY offloads, and the cap was what bounded them) and would apply to WHOOP 4.0, where a long drain starves the realtime keep-alive. Also drops the inline IMU decode from the offload delivery path (live frames keep it; the raw hex on the same line is the decoder's own input, so the summary is recomputable offline). Hygiene, not throughput — measured ~7 ms/buffer. --- Strand/BLE/BLEManager.swift | 35 ++++++++-------- Strand/BLE/PuffinDeepBufferLog.swift | 42 ++++++++----------- .../main/java/com/noop/ble/WhoopBleClient.kt | 23 ++++++++-- 3 files changed, 55 insertions(+), 45 deletions(-) diff --git a/Strand/BLE/BLEManager.swift b/Strand/BLE/BLEManager.swift index 9dd6c00860..c1fb100e27 100644 --- a/Strand/BLE/BLEManager.swift +++ b/Strand/BLE/BLEManager.swift @@ -314,26 +314,25 @@ struct Whoop5EmptyOffloadTracker { /// its idle timeout — the reported ~15-min sync). The stale/PAST-epoch case 2b actually exists for (#451) /// reads BEHIND the frontier, never future-dated, so it is untouched. struct BackfillContinuation { - /// Hard cap on consecutive auto-continues per connection (resets on disconnect), as a LAST-RESORT - /// backstop only. Guards 2 and 3 are the real protection and they stop on *evidence*: 2 stops the - /// moment the strap is no longer ahead of our frontier, 3 stops the moment a session fails to move - /// the trim cursor. A strap that is both genuinely behind AND genuinely handing over data is not - /// pathological — it is working, and cutting it off is the bug. + /// Hard cap on consecutive auto-continues per connection (resets on disconnect). 6 × ~60s ≈ 6 min of + /// back-to-back draining, without letting a misbehaving strap monopolise Bluetooth. /// - /// The old value of 6 came with the rationale "6 × ~60s ≈ 6 min of back-to-back draining — enough to - /// chew through a multi-night backlog". That premise is arithmetically wrong. Offload runs at ~7x - /// real-time (airtime-limited at ~25.4 KB/s against the 3508 B the strap banks per second-of-history), - /// so six minutes of draining recovers ~42 minutes of history — not a multi-night backlog. Recovering - /// a single dead-strap night (~19 h) needs ~2.7 h of draining, i.e. ~50+ sessions. At 6 the cap fires - /// mid-recovery and drops to `backfillIntervalSeconds` (900 s) — a 6.7% duty cycle, turning a ~3 h - /// drain into ~12 h+. That is the "worst practical impact" recorded in - /// docs/bugs/2026-07-15-strap-battery-backfill-observability.md (A5), measured on a real 18 h backlog. + /// This was briefly raised to 120 on the theory that it severed a real recovery — a ~19 h backlog needs + /// ~2.6 h of draining, so the cap fires mid-recovery and drops to the 900 s floor. That reasoning was + /// based on a CONTAMINATED measurement: the strap it was taken from had been switched into deep-data + /// (R22) banking by an experimental probe, so it was banking 3508 B/s of history instead of the default + /// 124 B/s. A default strap banks 28x less, one session recovers ~28x more history, and an 18 h backlog + /// drains in ~2 sessions — well inside 6. The cap was never the problem for a default user. /// - /// 120 covers a full night's recovery with ~2x margin while still bounding a truly runaway loop that - /// somehow satisfies guards 2 and 3 forever. The cap resets on disconnect, so it was never a battery - /// ceiling across a day — only a ceiling on one continuous recovery, which is exactly when we want to - /// keep going. - static let defaultMaxAutoContinues = 120 + /// Raising it is also not free: #928 and #1012 (below) are both cases where guard 2 did NOT stop and + /// the strap "reported backlog forever", burning the cap in EMPTY offloads. The cap was the only thing + /// that bounded them, so a 20x raise is a 20x blast radius on the next guard-2 bug. And + /// `maybeAutoContinueBackfill` has no family gating, so it would apply to WHOOP 4.0, where a long drain + /// starves the realtime keep-alive. + /// + /// If a genuine deep-backlog case is ever demonstrated on a DEFAULT strap, the fix is a conditional + /// bypass while provably behind and advancing — not a flat raise. + static let defaultMaxAutoContinues = 6 /// How far ahead the strap must be (seconds) before "more backlog remains" is real, not clock noise. /// Matches StuckStrapDetector.behindGapSeconds (5 min) so the two agree on "behind". static let defaultBehindGapSeconds = 300 diff --git a/Strand/BLE/PuffinDeepBufferLog.swift b/Strand/BLE/PuffinDeepBufferLog.swift index 124fa081a6..126f1e5901 100644 --- a/Strand/BLE/PuffinDeepBufferLog.swift +++ b/Strand/BLE/PuffinDeepBufferLog.swift @@ -51,12 +51,14 @@ final class PuffinDeepBufferLog { /// Skip the ~124-B ≈1 Hz record; keep the 1244-/2140-B high-rate buffers. private static let minBufferBytes = 1000 - /// Lowercase hex, table-driven. NOT cosmetic: this runs on the MainActor inside the offload ack path, - /// and the obvious `frame.map { String(format: "%02x", $0) }.joined()` costs one Foundation formatter - /// call + one String allocation PER BYTE — ~3 ms for a 2140-B buffer. The strap banks one 1244-B and - /// one 2140-B buffer per second of history, so draining a backlog at 5x real-time formats ~170 - /// buffers/sec ≈ 400 ms of every second, and the strap sits idle waiting for the ack behind it. A - /// nibble lookup into a byte array is ~80x cheaper and allocates once. + /// Lowercase hex, table-driven. Runs on the MainActor in the frame-delivery path, so it avoids the + /// obvious `frame.map { String(format: "%02x", $0) }.joined()`, which costs one Foundation formatter + /// call + one String allocation PER BYTE. A nibble lookup into a byte array is materially cheaper and + /// allocates once. The two properties that matter are pinned by tests rather than asserted here: + /// `testHexStringMatchesFormatterForEveryByteValue` (byte-for-byte parity with the encoding it + /// replaces, across the whole domain) and `testHexStringIsFastEnoughForTheAckPath` (sub-millisecond + /// per buffer). NOTE: this path only runs when the capture toggle (`PuffinFrameRecorder.enabledKey`) + /// is ON — it is off by default, so this is research-capture cost, not a cost every user pays. private static let hexDigits: [UInt8] = Array("0123456789abcdef".utf8) nonisolated static func hexString(_ bytes: [UInt8]) -> String { var out = [UInt8]() @@ -130,25 +132,17 @@ final class PuffinDeepBufferLog { // device captures and makes each JSONL line self-checking (raw ↔ decode) with NO stored table, // migration, or downstream gate. Instrumentation only, per the derived-signal rule; the 2140-B // optical buffer stays raw-only (its layout isn't decoded yet). - // LIVE frames only — a hygiene measure, NOT a throughput fix. Keeping the honest numbers here so - // nobody re-derives the wrong conclusion from this file a third time: + // LIVE frames only — a hygiene measure, and deliberately carrying NO measured throughput claim. + // An earlier version of this comment derived a link-rate model ("the link is a metronome") from + // the inter-frame gap asymmetry in a single 208 s window; that model did not survive + // re-measurement across more windows and has been removed rather than restated. Do not re-derive + // a link-rate or throughput conclusion from this file — it logs frames, it does not time them. // - // A real 19 h drain (3,018 buffers / 208 s) shows a mean gap of 89.1 ms after a 1244-B buffer vs - // 49.0 ms after a 2140-B one. That asymmetry is NOT this decode — it is pure airtime, and reading - // it as a stall is an off-by-one. A gap is the time for the NEXT frame(s) to arrive, not the - // previous frame's processing, and the ~124-B v18 record sits between the two logged buffers - // (it's below `minBufferBytes`, so it is invisible here). So the gap after the 1244-B spans - // 124+2140 = 2264 B and the gap after the 2140-B spans only 1244 B. One free parameter — the link - // rate — fixed by the total (3508 B / 138.1 ms = 25.4 KB/s) then predicts the SPLIT with zero - // further freedom: 89.13 ms and 48.97 ms, against 89.1 and 49.0 observed. The two gaps - // independently agree on the byte rate to 0.09%. The link is a metronome; there is no stall. - // Removing this decode measured 7.2x -> 7.7x, consistent with its true ~7 ms cost, not 40 ms. - // - // It is still right to skip during offload: it is per-frame work on the MainActor in the delivery - // path, it buys nothing there, and the raw `hex` on this very line is the decoder's own input — so - // the summary is recomputable offline from the archive at any time. Live frames arrive ~1/s, where - // the inline summary is free and makes each line self-checking. Capture captures; analysis is - // downstream. The real duty-cycle costs are the idle watchdog and the auto-continue cap. + // The reason to skip during offload stands on its own, independent of any rate: it is per-frame + // work on the MainActor in the delivery path, it buys nothing there, and the raw `hex` on this + // very line is the decoder's own input — so the summary is recomputable offline from the archive + // at any time. Live frames arrive ~1/s, where the inline summary is free and makes each line + // self-checking. Capture captures; analysis is downstream. let imu = isOffload ? "" : Self.decodedImuField(frame) let line = "{\"ts_ms\":\(tsMs),\"strap_ts\":\(strapTs),\"size\":\(frame.count)," + "\"offload\":\(isOffload),\"char\":\"\(char.uuidString.lowercased())\",\"hex\":\"\(hex)\"\(imu)}\n" diff --git a/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt b/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt index e6996cc560..c53edbd4d4 100644 --- a/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt +++ b/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt @@ -548,9 +548,21 @@ class WhoopBleClient( /** * Idle watchdog: if no genuine offload frame arrives for this long mid-session, end the * session (the durable strap_trim cursor means the next session resumes where we left off). - * Generous (60s, not 20s) because the type-43 raw flood eats BLE airtime between chunks. + * Must clear the real inter-frame tail: the type-43 raw flood eats BLE airtime between chunks, so + * a watchdog shorter than the tail cuts sessions short mid-drain. + * + * 60s was set without measuring that tail; it is ~24x the real worst case, and every session pays + * it in full as dead air on exit. A 3,018-buffer arrival trace from a real 19 h drain puts + * inter-frame gaps at p50 64 ms / p90 106 ms / p99 273 ms / max 2,520 ms, with the tail at chunk + * boundaries (~51 records, where decode + insert + cursor + ack happen), not mid-chunk. 15s is ~6x + * that measured max and still clears the flood. + * + * Deliberately NOT the ~5s the tail alone would justify: that tail is one drain on one firmware, + * and the risk is asymmetric — firing early is safe but wasteful (the in-memory chunk is dropped + * un-acked, so the strap keeps it and the next session refetches), while firing late only costs + * idle seconds. Mirrors Swift BLEManager.backfillIdleTimeoutSeconds. */ - private const val BACKFILL_IDLE_TIMEOUT_MS = 60_000L + private const val BACKFILL_IDLE_TIMEOUT_MS = 15_000L /** Deferral before the first connect-time offload, so SET_CLOCK/GET_DATA_RANGE round-trip first. */ private const val INITIAL_BACKFILL_DELAY_MS = 1_500L /** 5/MG fail-open gate: how long to wait for a GET_DATA_RANGE SUCCESS before requesting @@ -905,7 +917,12 @@ class WhoopBleClient( /** #364 auto-continue cap: consecutive immediate re-kicks per connection before falling back to * the 900s periodic timer. 6 × ~60s ≈ 6 min of back-to-back draining without letting a - * misbehaving strap monopolise Bluetooth. Mirrors Swift BackfillContinuation.defaultMaxAutoContinues. */ + * misbehaving strap monopolise Bluetooth. Mirrors Swift BackfillContinuation.defaultMaxAutoContinues. + * + * Briefly raised to 120 on iOS on the theory that it severed a real recovery; reverted. That + * measurement came from a strap an experimental probe had switched into deep-data (R22) banking — + * 3508 B/s of history instead of the default 124 B/s. A default strap banks 28x less, so an 18 h + * backlog drains in ~2 sessions, well inside 6. See the Swift twin for the full reasoning. */ const val MAX_AUTO_CONTINUES = 6 /** #364 "more backlog remains" margin (seconds): how far ahead the strap must be of our persisted From 33c0da6a1a1c0b1ae4aa1611db57a3edd9f9bd56 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 15 Jul 2026 14:24:17 -0700 Subject: [PATCH 116/171] CloudSync: ship the deep-buffer archive to object storage (fork-only) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drains PuffinDeepBufferLog's JSONL to a bucket so the high-rate buffers survive off the phone. Fork-only and CLOUD_SYNC-gated: upstream NOOP is offline-by-design ("no server, no account, no cloud sync, no telemetry"), and the capture half stays upstream-safe on its own. The hard part is not that the log is append-only — it is that it also ROTATES. A plain byte offset breaks catastrophically at rotation: it either skips the new generation forever or resumes mid-file into unrelated bytes, corrupting the archive with no error anywhere. So identity is (creationDate, inode): rename(2) preserves the inode, so a generation keeps its identity across .jsonl -> .jsonl.1 and a fallen-behind drain resumes its tail with no special case; creationDate is in the id because rotation frees the inode for reuse, and without it a new generation would inherit the old one's offset and skip real data. Watermarks advance only on a confirmed 2xx, onto a server idempotent on (generation, byteStart), so at-least-once is safe. Chunks are cut at the last newline, so each is independently valid JSONL and they concatenate back byte-for-byte. readLineAlignedChunk re-verifies the generation THROUGH THE OPEN DESCRIPTOR (fstat), because scanning binds identity to an inode while the read opens by path, and the gap between them spans a network upload. A rotation landing there leaves the path valid but pointing at a different file: the read returns the wrong generation's bytes, they are stored under the previous generation's id, and its watermark advances past a tail that was never sent. Two silent corruptions of the only copy of buffers the strap's ack-trim has already freed. A rename cannot change what an open fd refers to, so fstat is authoritative where a path re-check would still race. Pinned by testRotationDuringADrainCannotSpliceTheWrongGenerations Bytes, which rotates INSIDE the upload; it fails without the check (same byte count, different bytes) and passes with it. Compression measured on a real 78 MB capture, not asserted: 78,186,241 -> 17,719,151 = 4.41x raw DEFLATE. Note COMPRESSION_ZLIB emits headerless raw DEFLATE despite the name, so the server must inflateRawSync; the mismatch fails only at runtime, on both sides, and is pinned end-to-end by a fixture. Server-side R2/S3 wiring, the manifest table and the MCP tools live in noop-cloud. /deepbuf returns 503 until S3_* is configured and never falls back to the Fly volume: an operator who forgets an env var gets a loud refusal, not a full disk. --- Strand/CloudSync/CloudSyncClient.swift | 57 +++ Strand/CloudSync/CloudSyncModel.swift | 39 +- Strand/CloudSync/DeepBufferUploadPlan.swift | 142 +++++++ Strand/CloudSync/DeepBufferUploader.swift | 259 ++++++++++++ StrandTests/DeepBufferUploadPlanTests.swift | 133 ++++++ StrandTests/DeepBufferUploaderTests.swift | 431 ++++++++++++++++++++ 6 files changed, 1060 insertions(+), 1 deletion(-) create mode 100644 Strand/CloudSync/DeepBufferUploadPlan.swift create mode 100644 Strand/CloudSync/DeepBufferUploader.swift create mode 100644 StrandTests/DeepBufferUploadPlanTests.swift create mode 100644 StrandTests/DeepBufferUploaderTests.swift diff --git a/Strand/CloudSync/CloudSyncClient.swift b/Strand/CloudSync/CloudSyncClient.swift index 6225054618..4c74e9d670 100644 --- a/Strand/CloudSync/CloudSyncClient.swift +++ b/Strand/CloudSync/CloudSyncClient.swift @@ -122,6 +122,46 @@ final class CloudSyncClient { } } + // MARK: - Deep buffers (#423: WHOOP 5/MG high-rate offload archive) + + /// POST /deepbuf, Bearer-authed — one line-aligned, raw-DEFLATE-compressed byte range of + /// `PuffinDeepBufferLog`'s append-only JSONL. See `DeepBufferUploadPlan` for why the unit of upload + /// is `(generation, byteStart, byteEnd)` rather than a whole file. + /// + /// `compressed` is passed as `Data`, NOT streamed from a file the way `ingest` above must be: a + /// chunk is ~4 MB and is ALREADY in memory (it was just read and deflated), so spilling it to a + /// temp file purely to hand `upload(for:fromFile:)` a path would add two disk round-trips to buy + /// nothing. `ingest`'s whole-DB body is 100-300 MB, which is a different problem. + /// + /// The compression is announced in a CUSTOM header rather than `Content-Encoding: deflate` + /// deliberately: `Content-Encoding` invites any proxy in the path (and Express itself) to helpfully + /// decode the body, which would silently defeat the whole point of compressing on the phone and + /// leave the server parsing bytes it thinks are plaintext. A header nothing but this endpoint reads + /// cannot be helpfully misinterpreted by anything in between. + func uploadDeepBuffer(generation: String, byteStart: Int, byteEnd: Int, + compressed: Data) async throws -> DeepBufferUploadReceipt { + var req = URLRequest(url: url(path: "deepbuf")) + req.httpMethod = "POST" + req.setValue("Bearer \(token)", forHTTPHeaderField: "Authorization") + req.setValue("application/octet-stream", forHTTPHeaderField: "Content-Type") + req.setValue("deflate-raw", forHTTPHeaderField: "X-Deepbuf-Compression") + req.setValue(generation, forHTTPHeaderField: "X-Deepbuf-Generation") + req.setValue(String(byteStart), forHTTPHeaderField: "X-Deepbuf-Byte-Start") + req.setValue(String(byteEnd), forHTTPHeaderField: "X-Deepbuf-Byte-End") + // Same #tz-upload convention as `ingest` above: every timestamp inside the payload is epoch-UTC, + // so the phone's current zone is only knowable if it says so. + req.setValue(TimeZone.current.identifier, forHTTPHeaderField: "X-Phone-Timezone") + let (data, status) = try await sendUpload(req, from: compressed) + guard (200..<300).contains(status) else { + throw CloudSyncError.badResponse(status, bodyPrefix(data)) + } + guard let decoded = try? JSONDecoder().decode(DeepBufferResponse.self, from: data) else { + throw CloudSyncError.decode + } + return DeepBufferUploadReceipt(storedBytes: decoded.storedBytes, lines: decoded.lines, + duplicate: decoded.duplicate ?? false) + } + private func sendUpload(_ req: URLRequest, fromFile fileURL: URL) async throws -> (Data, Int) { do { let (data, resp) = try await session.upload(for: req, fromFile: fileURL) @@ -131,6 +171,15 @@ final class CloudSyncClient { } } + private func sendUpload(_ req: URLRequest, from body: Data) async throws -> (Data, Int) { + do { + let (data, resp) = try await session.upload(for: req, from: body) + return (data, (resp as? HTTPURLResponse)?.statusCode ?? 0) + } catch { + throw CloudSyncError.network(error.localizedDescription) + } + } + private func url(path: String, query: [String: String] = [:]) -> URL { var comps = URLComponents(string: "\(baseURL.absoluteString)/\(path)")! if !query.isEmpty { @@ -173,4 +222,12 @@ private struct IngestResponse: Decodable { let bytes: Int let latestDay: String? } + +/// `/deepbuf`'s reply. `duplicate` is optional because the server omits it on the normal path (it is +/// only present, and true, when the chunk was already in the manifest) — see `DeepBufferUploadReceipt`. +private struct DeepBufferResponse: Decodable { + let storedBytes: Int + let lines: Int + let duplicate: Bool? +} #endif // CLOUD_SYNC diff --git a/Strand/CloudSync/CloudSyncModel.swift b/Strand/CloudSync/CloudSyncModel.swift index 3a96dbafb6..86d20a1996 100644 --- a/Strand/CloudSync/CloudSyncModel.swift +++ b/Strand/CloudSync/CloudSyncModel.swift @@ -177,10 +177,13 @@ final class CloudSyncModel: ObservableObject { /// that synced an hour ago with no new samples and no applied edits has nothing new to ship. private func performSync(repo: Repository, url: URL, token: String) async { setBusy(true); statusText = "Pulling edits…" + // Hoisted out of the `if let store` below (where it used to be built) so the deep-buffer drain + // at the end of this method can reach it: that drain reads a FILE, not the store, so it must + // still run on a device whose store handle is unavailable. + let client = CloudSyncClient(baseURL: url, token: token) var line: String var succeeded = false if let store = await repo.storeHandle() { - let client = CloudSyncClient(baseURL: url, token: token) do { let summary = try await CloudSyncCoordinator.pull(store: store, client: client) var parts = ["Applied \(summary.applied)", "skipped \(summary.skipped)"] @@ -238,6 +241,18 @@ final class CloudSyncModel: ObservableObject { } else { line = "No local store." } + + // Deep-buffer drain (#423) — deliberately OUTSIDE the do/catch above, and NOT gated on + // `succeeded`. The edit-journal lane and the deep-buffer archive fail independently and are not + // equally recoverable: a pull that throws (server down, bad token, a 500 on /edits) costs + // nothing permanent — the edits are still on the server and the next sync gets them. The deep + // buffers are the ONLY copy of packets the strap's ack-trim has already freed, and the log + // destroys its own oldest generation on rotation, so bytes not shipped are eventually gone for + // good. Letting an unrelated /edits failure skip the drain would trade unrecoverable data for + // nothing. Cheap when idle: two `stat`s and a watermark compare, no network at all. + statusText = "Shipping deep buffers…" + if let deep = await drainDeepBuffers(client: client) { line += " · \(deep)" } + statusText = line if succeeded { UserDefaults.standard.set(Date().timeIntervalSince1970, forKey: Self.lastAutoSyncKey) @@ -247,6 +262,28 @@ final class CloudSyncModel: ObservableObject { setBusy(false) } + /// Ship whatever new bytes `PuffinDeepBufferLog` has appended since this device's last confirmed + /// chunk, returning a status-line SUFFIX (nil when there was nothing to do, so an idle drain adds + /// no noise to the card). + /// + /// UNGATED, unlike every other lane here — no 20h/4h staleness clock. The clock that matters is the + /// log's own: the strap banks ~40 KB/s of hex during an offload, so a generation fills the soft cap + /// in hours and the generation after that DELETES it. A staleness gate tuned for "has the user's + /// data changed" is the wrong instrument for "is the archive about to eat itself". Running every + /// sync costs two `stat` calls when there is nothing new (see `DeepBufferUploader.drain`), so + /// there is nothing to gate away. + private func drainDeepBuffers(client: CloudSyncClient) async -> String? { + let summary = await DeepBufferUploader.drain(client: client, + watermarks: DeepBufferUserDefaultsWatermarks()) + if let error = summary.error { return "Deep buffers: \(error)" } + guard !summary.isEmpty else { return nil } + let sentMB = Double(summary.sentBytes) / 1_048_576.0 + let rawMB = Double(summary.rawBytes) / 1_048_576.0 + var s = String(format: "Deep buffers %.1f→%.1f MB (%d chunks)", rawMB, sentMB, summary.chunks) + if summary.moreRemaining { s += " · more queued" } + return s + } + /// UserDefaults key for the last successful `syncNow` (manual or automatic) — read by /// `autoSyncIfDue`, written by `performSync` on success only. private static let lastAutoSyncKey = "cloudsync.lastAutoSync" diff --git a/Strand/CloudSync/DeepBufferUploadPlan.swift b/Strand/CloudSync/DeepBufferUploadPlan.swift new file mode 100644 index 0000000000..6a8f9e19c4 --- /dev/null +++ b/Strand/CloudSync/DeepBufferUploadPlan.swift @@ -0,0 +1,142 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +// +// UPSTREAM BOUNDARY: the CAPTURE half (`Strand/BLE/PuffinDeepBufferLog.swift`) is upstream-safe and +// deliberately knows nothing about this file. Everything that moves a captured byte OFF the device +// lives here, under `Strand/CloudSync/`, behind `#if CLOUD_SYNC` — the same gate as every other +// cloud file. Nothing in `Strand/BLE/` imports, calls, or references anything below. +#if CLOUD_SYNC +import Foundation + +/// Pure planning logic for `DeepBufferUploader`: which byte range of the append-only deep-buffer log +/// to ship next, and how to tell one generation of that log from the next. No filesystem, no network, +/// no `UserDefaults` — every function here is a value-in/value-out transform, so the rules that decide +/// "have we already uploaded this?" are unit-testable with no strap, no server, and no device (the +/// BLE path itself cannot be tested at all — see CLAUDE.md's "BLE behavior cannot be CI-tested"). +/// +/// ## The problem this solves +/// +/// `PuffinDeepBufferLog` writes an APPEND-ONLY JSONL file that ALSO ROTATES: at a soft cap it renames +/// `puffin-deepbuffers.jsonl` → `puffin-deepbuffers.jsonl.1` (destroying the previous `.1`) and starts +/// a fresh file. Those two facts together are what make naive "upload from offset N" wrong: +/// +/// - Append-only alone would be easy — remember the offset, ship `[offset, size)`, advance. +/// - Rotation breaks that: after a rotation the live file restarts at size 0, so a remembered offset +/// of 90 MB would either skip the whole new generation forever, or (worse, once the new file grows +/// past 90 MB) resume mid-file and ship a byte range from a DIFFERENT generation as though it were +/// a continuation — silently corrupting the archive with no error anywhere. +/// +/// So an offset is only meaningful when paired with the identity of the file it indexes into. That +/// pair — `(generation, uploadedOffset)` — is the whole design. +/// +/// ## Why the identity is (creationDate, inode) +/// +/// `rename(2)` PRESERVES the inode, so a generation keeps its identity across the `.jsonl` → `.jsonl.1` +/// rotation. That is the property that lets a drain that fell behind pick up the un-uploaded tail of a +/// rotated-away generation from `.jsonl.1` at exactly the offset it left off — no special case, no +/// separate bookkeeping, the watermark simply still matches. +/// +/// The creation date is in the id because an inode ALONE is not stable over time: rotation +/// `removeItem`s the old `.1`, freeing that inode for the filesystem to hand straight back to the next +/// freshly-created `.jsonl`. Two different generations with the same inode number would collide in the +/// watermark map, and the new generation would inherit the old one's offset — skipping real data. The +/// creation timestamp disambiguates them. `sanitizedWatermark` below is the belt-and-braces backstop +/// for any residual confusion. +/// +/// ## Why READ-ONLY, never seal-and-rotate +/// +/// The classic log-shipping design seals a segment (rename it aside) and uploads whole sealed files. +/// That is WRONG here: the BLE path holds an open `FileHandle` to the live log, and POSIX semantics +/// mean a writer keeps writing into the renamed inode — so a "sealed" segment would keep growing after +/// it was sealed, and coordinating a close/reopen would mean reaching into the offload ack path. The +/// capture path must never be blocked, slowed, or coupled to the network (see `PuffinDeepBufferLog`'s +/// doc comment on why its hex encoder is hand-rolled: it runs inside the ack path and a few hundred +/// microseconds per buffer already matters). Reading a byte range through a SEPARATE read handle is +/// invisible to the writer: the uploader never mutates the file, never holds the writer's handle, and +/// never needs the writer to know it exists. +enum DeepBufferUploadPlan { + + /// The `/OpenWhoop/` filename `PuffinDeepBufferLog` writes, and the `.1` its rotation + /// renames to. DUPLICATED here rather than read from `PuffinDeepBufferLog.logURL()` (which is + /// `private`) ON PURPOSE: making it internal would edit an upstream-safe BLE file for a + /// cloud-only reason, and that file must stay free of any reason to think about upload. The path + /// is a frozen on-disk contract — it is named in `PuffinDeepBufferLog`'s own doc comment and is + /// where the only copy of already-freed strap buffers lives — so the drift risk is low and + /// one-directional. `DeepBufferUploaderTests.testLogFilenamesMatchTheCaptureContract` pins it. + static let liveFilename = "puffin-deepbuffers.jsonl" + static let rotatedFilename = "puffin-deepbuffers.jsonl.1" + static let logDirectoryName = "OpenWhoop" + + /// Raw bytes read (and then compressed) per upload. 16 MB raw ≈ 4 MB on the wire at the ~4x these + /// hex payloads compress to, which is the real constraint: this drains from a `BGAppRefreshTask`, + /// whose budget is ~30 s of wall clock. A single 90 MB whole-file upload could never finish inside + /// that window and would be killed and restarted from zero forever; 16 MB slices each land in ~1 s + /// on LTE, and — because the watermark advances per chunk — a wake that is cut off part-way still + /// makes permanent forward progress instead of losing everything it did. Chunking is what makes + /// the background lane viable at all, not just what avoids re-uploading. + static let maxChunkBytes = 16 * 1024 * 1024 + + /// Chunks per drain, bounding one wake's work at 128 MB raw / ~32 MB sent. Enough to clear a + /// steady-state backlog (the strap banks ~40 KB/s of hex, so even a full day is a handful of runs) + /// without ever letting a single background wake run unbounded. + static let maxChunksPerRun = 8 + + /// A stable identity for one generation of the log file: `-`. + /// Millisecond precision (not seconds) so a delete/create pair inside the same second still yields + /// distinct ids if the filesystem also recycles the inode; see the type's doc comment for why + /// neither component is sufficient alone. + static func generationId(creationDate: Date?, inode: UInt64) -> String { + let ms = Int64(((creationDate ?? .distantPast).timeIntervalSince1970 * 1000).rounded()) + return "\(ms)-\(inode)" + } + + /// Clamp a remembered watermark to what the file can actually support. + /// + /// For an append-only file a watermark ABOVE the current size is impossible — the file only ever + /// grows — so seeing one means the identity is lying to us: an inode got recycled inside the same + /// millisecond, `UserDefaults` was restored onto a different device, or the file was replaced + /// out-of-band. Re-uploading a generation from 0 is cheap and idempotent (the server keys objects + /// by `(generation, byteStart)` and overwrites); resuming from a bogus offset would ship a byte + /// range that means nothing. Take the safe branch. + static func sanitizedWatermark(uploaded: Int, fileSize: Int) -> Int { + (uploaded < 0 || uploaded > fileSize) ? 0 : uploaded + } + + /// The next byte range to ship for one generation, or nil when the watermark has caught up with + /// the file. Half-open `[start, end)`, capped at `maxChunk`. + static func nextRange(uploaded: Int, fileSize: Int, maxChunk: Int = maxChunkBytes) -> Range? { + let start = sanitizedWatermark(uploaded: uploaded, fileSize: fileSize) + guard start < fileSize else { return nil } + return start.. Int { + guard let idx = data.lastIndex(of: 0x0A) else { return 0 } + return data.distance(from: data.startIndex, to: idx) + 1 + } + + /// Drop watermarks for generations that no longer exist on disk. Rotation destroys `.jsonl.1` + /// permanently, so its watermark is dead weight the moment the file is gone; without this the map + /// grows without bound for the life of the install (one entry per rotation, forever) inside a + /// `UserDefaults` value that is read and rewritten on every drain. + static func prunedWatermarks(_ marks: [String: Int], live: Set) -> [String: Int] { + marks.filter { live.contains($0.key) } + } +} +#endif // CLOUD_SYNC diff --git a/Strand/CloudSync/DeepBufferUploader.swift b/Strand/CloudSync/DeepBufferUploader.swift new file mode 100644 index 0000000000..9f8f122e68 --- /dev/null +++ b/Strand/CloudSync/DeepBufferUploader.swift @@ -0,0 +1,259 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +// +// UPSTREAM BOUNDARY: see DeepBufferUploadPlan.swift's header. The capture half never references this. +#if CLOUD_SYNC +import Foundation +import Compression + +/// Abstracts the deep-buffer chunk POST so `DeepBufferUploader` is testable with no network — the same +/// seam shape as `CloudIngesting` (`CloudSyncUploader`) and `CloudEditFetching` (`CloudSyncCoordinator`). +/// `CloudSyncClient` conforms in `CloudSyncClient+DeepBuffer.swift`. +protocol DeepBufferIngesting { + func uploadDeepBuffer(generation: String, byteStart: Int, byteEnd: Int, + compressed: Data) async throws -> DeepBufferUploadReceipt +} +extension CloudSyncClient: DeepBufferIngesting {} + +/// What the server says it did with a chunk. `duplicate` is not a failure: the drain is at-least-once +/// (the watermark only advances on a confirmed 2xx, so a response lost in flight re-sends the same +/// range), and the server keys objects by `(generation, byteStart)` — so a re-send is an idempotent +/// overwrite, and saying so lets the drain log the difference instead of double-counting bytes. +struct DeepBufferUploadReceipt: Equatable { + let storedBytes: Int + let lines: Int + let duplicate: Bool +} + +/// One generation of the on-disk log, as `scanGenerations` found it. +struct DeepBufferGeneration: Equatable { + let id: String + let url: URL + let size: Int +} + +/// The outcome of one drain, for the status line and the tests. +struct DeepBufferDrainSummary: Equatable { + var chunks = 0 + var rawBytes = 0 + var sentBytes = 0 + var lines = 0 + var duplicates = 0 + /// True when the drain stopped because it hit `maxChunksPerRun`, not because it ran out of data — + /// i.e. there is more backlog and the next wake should run again immediately rather than wait. + var moreRemaining = false + var error: String? + + var isEmpty: Bool { chunks == 0 && error == nil } +} + +/// Where the per-generation byte watermarks live. A protocol so tests get a deterministic in-memory +/// map instead of racing the test host's real `UserDefaults`. +protocol DeepBufferWatermarkStoring: AnyObject { + func watermarks() -> [String: Int] + func setWatermarks(_ marks: [String: Int]) +} + +/// Production watermark storage. `UserDefaults`, not a `WhoopStore` cursor — the same call as +/// `CloudSyncModel.lastUploadTokenKey` (see its doc comment): this is per-device bookkeeping about an +/// upload THIS device performed, not data the store owns, and `cursors` only holds `Int` values keyed +/// by a single name anyway, where this needs a whole generation→offset map. +final class DeepBufferUserDefaultsWatermarks: DeepBufferWatermarkStoring { + static let key = "cloudsync.deepbuf.watermarks" + private let defaults: UserDefaults + init(defaults: UserDefaults = .standard) { self.defaults = defaults } + + func watermarks() -> [String: Int] { + (defaults.dictionary(forKey: Self.key) as? [String: Int]) ?? [:] + } + func setWatermarks(_ marks: [String: Int]) { + defaults.set(marks, forKey: Self.key) + } +} + +/// Ships `PuffinDeepBufferLog`'s append-only JSONL to the noop-cloud server in compressed, +/// line-aligned byte-range chunks, resuming from a per-generation watermark so nothing is ever sent +/// twice and nothing is skipped. The correctness argument for the `(generation, offset)` pair, and for +/// reading the live file rather than sealing segments, is in `DeepBufferUploadPlan`'s doc comment. +/// +/// A namespace of static functions with no state of its own, like `CloudSyncUploader` — the only +/// durable state is the watermark map, which is injected. +/// +/// NEVER touches the BLE path: it opens its own read-only `FileHandle`, reads a bounded byte range, +/// and closes it. `PuffinDeepBufferLog` (which is `@MainActor`, and whose `appendIfDeepBuffer` runs +/// inside the offload ack path) is not referenced, imported, or awaited from anywhere in here. This +/// type is deliberately NOT `@MainActor`: the read + deflate of a 16 MB slice is real CPU work and +/// belongs on a cooperative thread, not on the actor the capture path is trying to use. +enum DeepBufferUploader { + + /// `/OpenWhoop/` — the directory `PuffinDeepBufferLog` writes into. Resolved here + /// rather than borrowed from that (upstream-safe, `private`) type; see + /// `DeepBufferUploadPlan.liveFilename`'s doc comment for why the duplication is deliberate. + static func defaultLogDirectory() throws -> URL { + try FileManager.default.url(for: .applicationSupportDirectory, in: .userDomainMask, + appropriateFor: nil, create: true) + .appendingPathComponent(DeepBufferUploadPlan.logDirectoryName, isDirectory: true) + } + + /// The generations present in `dir`, ROTATED FIRST then live. + /// + /// Rotated-first is the load-bearing order: `.jsonl.1` is the generation the NEXT rotation deletes + /// forever, so it is the data most at risk and must drain first. It is also the chronologically + /// older one, so this keeps the uploaded archive in wall-clock order for free. + /// + /// Deduped by generation id: rotation runs on the MainActor while this scan runs off it, so a scan + /// that interleaves with a rename can in principle observe one inode under both filenames. Keeping + /// the first occurrence (the rotated path) rather than uploading the same generation twice under + /// two names costs nothing and removes the race from every caller's reasoning. + static func scanGenerations(in dir: URL) -> [DeepBufferGeneration] { + var seen = Set() + var out: [DeepBufferGeneration] = [] + for name in [DeepBufferUploadPlan.rotatedFilename, DeepBufferUploadPlan.liveFilename] { + let url = dir.appendingPathComponent(name) + guard let attrs = try? FileManager.default.attributesOfItem(atPath: url.path), + let size = (attrs[.size] as? NSNumber)?.intValue, + let inode = (attrs[.systemFileNumber] as? NSNumber)?.uint64Value else { continue } + let id = DeepBufferUploadPlan.generationId(creationDate: attrs[.creationDate] as? Date, + inode: inode) + guard !seen.contains(id) else { continue } + seen.insert(id) + out.append(DeepBufferGeneration(id: id, url: url, size: size)) + } + return out + } + + /// Read `range` from `url` through a private read handle, then cut the result back to its last + /// complete line (see `DeepBufferUploadPlan.lastCompleteLineEnd`). Returns nil when the range + /// yields no complete line — e.g. a single in-progress line longer than the read, or a file that + /// vanished under a rotation between the scan and here. + /// + /// `expecting` is the generation id the caller scanned, and this re-checks it THROUGH THE OPEN + /// DESCRIPTOR. That is the whole point: the scan binds a generation's identity to (creationDate, + /// inode), but this opens by PATH, and the gap between the two spans a network upload — seconds, + /// not microseconds. `PuffinDeepBufferLog` rotates by renaming `.jsonl` -> `.jsonl.1`, so a + /// rotation landing in that gap leaves the path perfectly valid and pointing at a DIFFERENT file. + /// The read then succeeds and returns the wrong generation's bytes, which the caller staples into + /// the previous generation's archive under its id AND acks — advancing that generation's watermark + /// past a tail it never uploaded. Two silent corruptions, no error anywhere, and the archive is the + /// only copy of buffers the strap's ack-trim has already freed. + /// + /// A rename cannot change what an open fd refers to, so fstat-ing the descriptor after opening is + /// authoritative where a path re-check would still race. Mismatch => bail; the next drain rescans + /// and picks the generation up at its true watermark. + static func readLineAlignedChunk(url: URL, range: Range, expecting generation: String?) -> Data? { + guard let handle = try? FileHandle(forReadingFrom: url) else { return nil } + defer { try? handle.close() } + if let expected = generation { + var st = stat() + guard fstat(handle.fileDescriptor, &st) == 0 else { return nil } + let birth = Date(timeIntervalSince1970: TimeInterval(st.st_birthtimespec.tv_sec) + + TimeInterval(st.st_birthtimespec.tv_nsec) / 1_000_000_000) + guard DeepBufferUploadPlan.generationId(creationDate: birth, inode: UInt64(st.st_ino)) == expected, + range.upperBound <= Int(st.st_size) else { return nil } + } + guard (try? handle.seek(toOffset: UInt64(range.lowerBound))) != nil, + let raw = try? handle.read(upToCount: range.count), !raw.isEmpty else { return nil } + let end = DeepBufferUploadPlan.lastCompleteLineEnd(raw) + guard end > 0 else { return nil } + return raw.prefix(end) + } + + /// Raw DEFLATE, via Apple's Compression framework. + /// + /// `.zlib` here is a MISNOMER in Apple's own API: `COMPRESSION_ZLIB` emits a RAW DEFLATE stream + /// with NO zlib header or Adler-32 trailer, which is why the server inflates it with + /// `zlib.inflateRawSync` and not `inflateSync`. Getting this pair wrong fails at runtime with an + /// "incorrect header check", never at compile time, on both sides — it is pinned end-to-end by + /// `DeepBufferCompressionTests` (Swift side) and `test/deepbuf.test.ts` (the Node inflate of real + /// Swift-produced bytes). + /// + /// The phone has CPU to spare for this and it is nowhere near the BLE critical path (see the type's + /// doc comment) — the whole point of paying it here is that hex JSONL is the most compressible + /// thing this app produces: every payload byte is two ASCII nibbles, so half the entropy is + /// structural before the IMU's own sample-to-sample correlation is considered. + static func deflate(_ data: Data) -> Data? { + try? (data as NSData).compressed(using: .zlib) as Data + } + + /// Drain every generation from its watermark up to the file's current end, oldest first, stopping + /// at `maxChunks`. + /// + /// The watermark advances ONLY after a confirmed 2xx, and only for the range that call actually + /// acknowledged. That single rule is what makes the whole lane safe under failure: a dropped + /// connection, a killed background task, a 500, or a phone that runs out of budget mid-drain all + /// leave the watermark exactly where the last CONFIRMED byte ended, so the next run re-sends that + /// range and nothing else. Combined with the server's `(generation, byteStart)` idempotency this is + /// at-least-once delivery onto an idempotent sink — i.e. effectively exactly-once — without the + /// phone having to track anything but one integer per generation. + /// + /// A per-generation failure STOPS that generation (its later bytes cannot be shipped before its + /// earlier ones without leaving a hole the watermark can't express) but the summary still reports + /// whatever landed before it, so partial progress is never rolled back. + @discardableResult + static func drain(client: any DeepBufferIngesting, + watermarks: any DeepBufferWatermarkStoring, + directory: URL? = nil, + maxChunks: Int = DeepBufferUploadPlan.maxChunksPerRun, + maxChunkBytes: Int = DeepBufferUploadPlan.maxChunkBytes) async -> DeepBufferDrainSummary { + var summary = DeepBufferDrainSummary() + let dir: URL + do { dir = try directory ?? defaultLogDirectory() } catch { + summary.error = "Couldn't locate the deep-buffer log directory. \(error.localizedDescription)" + return summary + } + let generations = scanGenerations(in: dir) + guard !generations.isEmpty else { return summary } + + // Prune BEFORE draining, against the generations actually on disk right now — a rotation that + // destroyed `.jsonl.1` also destroyed any reason to remember its offset. + var marks = DeepBufferUploadPlan.prunedWatermarks(watermarks.watermarks(), + live: Set(generations.map(\.id))) + + for generation in generations { + while summary.chunks < maxChunks { + let uploaded = marks[generation.id] ?? 0 + guard let range = DeepBufferUploadPlan.nextRange(uploaded: uploaded, + fileSize: generation.size, + maxChunk: maxChunkBytes) else { break } + guard let chunk = readLineAlignedChunk(url: generation.url, range: range, + expecting: generation.id) else { + // No complete line in this range: either the writer is mid-line at the very tail + // (normal — the next drain gets it once the line is finished) or the file rotated + // away underneath us. Neither is an error; move to the next generation. + break + } + guard let body = deflate(chunk) else { + summary.error = "Couldn't compress the deep-buffer chunk." + watermarks.setWatermarks(marks) + return summary + } + let byteStart = range.lowerBound + let byteEnd = byteStart + chunk.count // the LINE-ALIGNED end, not range.upperBound + do { + let receipt = try await client.uploadDeepBuffer(generation: generation.id, + byteStart: byteStart, + byteEnd: byteEnd, + compressed: body) + marks[generation.id] = byteEnd + watermarks.setWatermarks(marks) // persist per chunk: a kill mid-drain keeps progress + summary.chunks += 1 + summary.rawBytes += chunk.count + summary.sentBytes += body.count + summary.lines += receipt.lines + if receipt.duplicate { summary.duplicates += 1 } + } catch { + summary.error = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription + watermarks.setWatermarks(marks) + return summary + } + } + if summary.chunks >= maxChunks { break } + } + watermarks.setWatermarks(marks) + // Anything still behind its file's end means the cap cut us short, not that we finished. + summary.moreRemaining = generations.contains { (marks[$0.id] ?? 0) < $0.size } + return summary + } +} +#endif // CLOUD_SYNC diff --git a/StrandTests/DeepBufferUploadPlanTests.swift b/StrandTests/DeepBufferUploadPlanTests.swift new file mode 100644 index 0000000000..e2d6901c9b --- /dev/null +++ b/StrandTests/DeepBufferUploadPlanTests.swift @@ -0,0 +1,133 @@ +// Tests the CLOUD_SYNC-gated deep-buffer upload planning logic; compiled only when the flag is set +// (StrandTests shares the app's OuraConfig.xcconfig, so flag + creds arrive together). +#if CLOUD_SYNC +import XCTest +@testable import Strand + +/// Pins `DeepBufferUploadPlan` — the pure rules that decide which bytes of the append-only +/// deep-buffer log have already been shipped. Every function under test is value-in/value-out, so +/// this whole file runs with no strap, no server, no filesystem and no `UserDefaults`. That matters +/// more here than usual: the log holds the ONLY copy of high-rate buffers the strap's ack-trim has +/// already freed, so a re-upload wastes VK's bandwidth but a SKIP silently destroys research data, +/// and neither shows up as an error anywhere. +final class DeepBufferUploadPlanTests: XCTestCase { + + // MARK: - nextRange + + func testNextRangeStartsAtZeroForAnUnseenGeneration() { + XCTAssertEqual(DeepBufferUploadPlan.nextRange(uploaded: 0, fileSize: 100, maxChunk: 1000), 0..<100) + } + + func testNextRangeResumesFromTheWatermark() { + // The whole point of the lane: bytes below the watermark are never read again. + XCTAssertEqual(DeepBufferUploadPlan.nextRange(uploaded: 60, fileSize: 100, maxChunk: 1000), 60..<100) + } + + func testNextRangeIsNilWhenCaughtUp() { + XCTAssertNil(DeepBufferUploadPlan.nextRange(uploaded: 100, fileSize: 100, maxChunk: 1000)) + XCTAssertNil(DeepBufferUploadPlan.nextRange(uploaded: 0, fileSize: 0, maxChunk: 1000)) + } + + func testNextRangeCapsAtMaxChunk() { + XCTAssertEqual(DeepBufferUploadPlan.nextRange(uploaded: 0, fileSize: 100, maxChunk: 40), 0..<40) + XCTAssertEqual(DeepBufferUploadPlan.nextRange(uploaded: 80, fileSize: 100, maxChunk: 40), 80..<100) + } + + // MARK: - sanitizedWatermark (the inode-reuse backstop) + + func testWatermarkAboveFileSizeResetsToZero() { + // Impossible for an append-only file, so the identity is lying: a recycled inode inside the + // same millisecond, or UserDefaults restored onto a different device. Re-uploading from 0 is + // idempotent server-side; resuming from a bogus offset would ship meaningless bytes. + XCTAssertEqual(DeepBufferUploadPlan.sanitizedWatermark(uploaded: 500, fileSize: 100), 0) + XCTAssertEqual(DeepBufferUploadPlan.nextRange(uploaded: 500, fileSize: 100, maxChunk: 1000), 0..<100) + } + + func testNegativeWatermarkResetsToZero() { + XCTAssertEqual(DeepBufferUploadPlan.sanitizedWatermark(uploaded: -1, fileSize: 100), 0) + } + + func testWatermarkExactlyAtFileSizeIsKept() { + // The caught-up case is legitimate, not corruption — it must NOT reset and re-ship the file. + XCTAssertEqual(DeepBufferUploadPlan.sanitizedWatermark(uploaded: 100, fileSize: 100), 100) + } + + // MARK: - lastCompleteLineEnd + + func testLastCompleteLineEndCutsATrailingPartialLine() { + // The writer appends concurrently, so a read can land mid-line. The partial tail is left for + // the next drain (by which time the line is finished), never shipped as broken JSONL. + let data = Data("{\"a\":1}\n{\"b\":2}\n{\"c\":".utf8) + XCTAssertEqual(DeepBufferUploadPlan.lastCompleteLineEnd(data), 16) + } + + func testLastCompleteLineEndOnAnExactBoundaryKeepsEverything() { + let data = Data("{\"a\":1}\n{\"b\":2}\n".utf8) + XCTAssertEqual(DeepBufferUploadPlan.lastCompleteLineEnd(data), data.count) + } + + func testLastCompleteLineEndIsZeroWithNoNewlineAtAll() { + // One in-progress line longer than the whole read — nothing complete to ship yet. + XCTAssertEqual(DeepBufferUploadPlan.lastCompleteLineEnd(Data("{\"a\":1".utf8)), 0) + XCTAssertEqual(DeepBufferUploadPlan.lastCompleteLineEnd(Data()), 0) + } + + func testLastCompleteLineEndWorksOnASlice() { + // `readLineAlignedChunk` hands this whatever `FileHandle.read` returned, and Data slices do not + // start at index 0 — an implementation using a raw Int index instead of `distance(from:to:)` + // passes every test above and corrupts exactly this case. + let full = Data("XXXX{\"a\":1}\n{\"b\":".utf8) + let slice = full.dropFirst(4) + XCTAssertEqual(DeepBufferUploadPlan.lastCompleteLineEnd(slice), 8) + } + + // MARK: - generationId + + func testGenerationIdDistinguishesInodes() { + let d = Date(timeIntervalSince1970: 1_752_600_000) + XCTAssertNotEqual(DeepBufferUploadPlan.generationId(creationDate: d, inode: 1), + DeepBufferUploadPlan.generationId(creationDate: d, inode: 2)) + } + + func testGenerationIdDistinguishesRecycledInodes() { + // The reason the creation date is in the id at all: rotation deletes `.jsonl.1`, freeing its + // inode for the filesystem to hand straight back to the next `.jsonl`. Same inode, different + // file — if the ids collided, the new generation would inherit the old one's watermark and + // skip real data. + XCTAssertNotEqual( + DeepBufferUploadPlan.generationId(creationDate: Date(timeIntervalSince1970: 1_752_600_000), inode: 7), + DeepBufferUploadPlan.generationId(creationDate: Date(timeIntervalSince1970: 1_752_600_060), inode: 7)) + } + + func testGenerationIdIsStableForTheSameFile() { + let d = Date(timeIntervalSince1970: 1_752_600_000.123) + XCTAssertEqual(DeepBufferUploadPlan.generationId(creationDate: d, inode: 42), + DeepBufferUploadPlan.generationId(creationDate: d, inode: 42)) + } + + func testGenerationIdSeparatesSubSecondCreations() { + // Millisecond precision, not seconds: a delete/create pair inside one second is exactly the + // case a seconds-resolution id would collapse. + XCTAssertNotEqual( + DeepBufferUploadPlan.generationId(creationDate: Date(timeIntervalSince1970: 1_752_600_000.100), inode: 7), + DeepBufferUploadPlan.generationId(creationDate: Date(timeIntervalSince1970: 1_752_600_000.900), inode: 7)) + } + + func testGenerationIdToleratesAMissingCreationDate() { + XCTAssertFalse(DeepBufferUploadPlan.generationId(creationDate: nil, inode: 5).isEmpty) + } + + // MARK: - prunedWatermarks + + func testPruneDropsGenerationsThatNoLongerExist() { + // Rotation destroys `.jsonl.1` permanently; without this the map grows one dead entry per + // rotation forever inside a UserDefaults value rewritten on every drain. + let pruned = DeepBufferUploadPlan.prunedWatermarks(["a": 10, "b": 20, "c": 30], live: ["b", "c"]) + XCTAssertEqual(pruned, ["b": 20, "c": 30]) + } + + func testPruneKeepsEverythingWhenAllGenerationsAreLive() { + XCTAssertEqual(DeepBufferUploadPlan.prunedWatermarks(["a": 1], live: ["a"]), ["a": 1]) + } +} +#endif // CLOUD_SYNC diff --git a/StrandTests/DeepBufferUploaderTests.swift b/StrandTests/DeepBufferUploaderTests.swift new file mode 100644 index 0000000000..9f3a6ba8c3 --- /dev/null +++ b/StrandTests/DeepBufferUploaderTests.swift @@ -0,0 +1,431 @@ +// Tests the CLOUD_SYNC-gated deep-buffer drain; compiled only when the flag is set (StrandTests +// shares the app's OuraConfig.xcconfig, so flag + creds arrive together). +#if CLOUD_SYNC +import XCTest +@testable import Strand + +/// A spy conforming to `DeepBufferIngesting` that records every chunk instead of reaching a network — +/// mirrors `SpyCloudIngester` in `CloudSyncUploaderTests.swift`. `failFrom` makes the Nth call onward +/// throw, so the watermark's failure semantics can be tested without a server. +private final class SpyDeepBufferClient: DeepBufferIngesting { + struct Call: Equatable { + let generation: String + let byteStart: Int + let byteEnd: Int + let compressed: Data + } + private(set) var calls: [Call] = [] + var failFrom: Int? + var reportDuplicate = false + /// Runs INSIDE an upload, standing in for the seconds a real network call takes. The only way to + /// exercise something rotating the log mid-drain — the window the identity re-check exists for. + var onUpload: (() -> Void)? + + func uploadDeepBuffer(generation: String, byteStart: Int, byteEnd: Int, + compressed: Data) async throws -> DeepBufferUploadReceipt { + onUpload?() + if let failFrom, calls.count >= failFrom { throw CloudSyncError.network("simulated failure") } + calls.append(Call(generation: generation, byteStart: byteStart, byteEnd: byteEnd, compressed: compressed)) + return DeepBufferUploadReceipt(storedBytes: compressed.count, lines: 1, duplicate: reportDuplicate) + } + + /// Every chunk's payload inflated and concatenated, in the order it was sent. + func reassembled() -> Data { + calls.reduce(into: Data()) { acc, c in + acc.append((try? (c.compressed as NSData).decompressed(using: .zlib) as Data) ?? Data()) + } + } +} + +/// In-memory watermarks, so a test never races the test host's real `UserDefaults` (`StrandTests` runs +/// inside the full app via TEST_HOST — see `CloudSyncModel.isRunningUnderXCTest`). +private final class MemoryWatermarks: DeepBufferWatermarkStoring { + var marks: [String: Int] + init(_ marks: [String: Int] = [:]) { self.marks = marks } + func watermarks() -> [String: Int] { marks } + func setWatermarks(_ m: [String: Int]) { marks = m } +} + +/// Behavioural tests for `DeepBufferUploader.drain` against REAL files in a temp directory — the +/// append-only + rotation semantics it exists to get right are filesystem semantics (`rename(2)` +/// preserving an inode is the load-bearing fact), so faking the filesystem would fake away the thing +/// under test. Only the network is stubbed. +final class DeepBufferUploaderTests: XCTestCase { + + private var dir: URL! + + override func setUpWithError() throws { + try super.setUpWithError() + dir = FileManager.default.temporaryDirectory + .appendingPathComponent("deepbuf-tests-\(UUID().uuidString)", isDirectory: true) + try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + } + + override func tearDownWithError() throws { + try? FileManager.default.removeItem(at: dir) + try super.tearDownWithError() + } + + // MARK: - Helpers + + private var liveURL: URL { dir.appendingPathComponent(DeepBufferUploadPlan.liveFilename) } + private var rotatedURL: URL { dir.appendingPathComponent(DeepBufferUploadPlan.rotatedFilename) } + + /// `n` JSONL lines shaped like the real log's, from `PuffinDeepBufferLog.appendIfDeepBuffer`. + private func lines(_ n: Int, startTs: Int = 1_752_600_000) -> Data { + var out = Data() + for i in 0.. .1 (rename preserves the inode), then a fresh live. + try FileManager.default.moveItem(at: liveURL, to: rotatedURL) + let genB = lines(4, startTs: 1_752_800_000) + try append(genB, to: liveURL) + + _ = await DeepBufferUploader.drain(client: client, watermarks: marks, directory: dir, + maxChunks: 100, maxChunkBytes: 10_000) + + let generations = Set(client.calls.map(\.generation)) + XCTAssertEqual(generations.count, 2, "the rotated file and the fresh file must be distinct generations") + + // A's identity survived the rename, so its tail resumed at the watermark rather than restarting. + let aCalls = client.calls.filter { $0.generation == client.calls[0].generation } + XCTAssertEqual(aCalls.count, 2) + XCTAssertEqual(aCalls[1].byteStart, drainedFromA, "the rotated generation must resume, not restart") + XCTAssertEqual(aCalls[1].byteEnd, genA.count) + + // B is a different file and must start from 0 — not inherit A's offset. + let bCalls = client.calls.filter { $0.generation != client.calls[0].generation } + XCTAssertEqual(bCalls.first?.byteStart, 0, "a fresh generation must start at 0") + + let inflatedA = aCalls.reduce(into: Data()) { $0.append((try! ($1.compressed as NSData).decompressed(using: .zlib)) as Data) } + XCTAssertEqual(inflatedA, genA, "generation A must arrive whole across the rotation") + } + + func testRotationDuringADrainCannotSpliceTheWrongGenerationsBytes() async throws { + // `testRotationResumes…` rotates BETWEEN drains, which is the easy case and cannot catch this. + // The real gap is INSIDE one drain: `scanGenerations` binds identity to (creationDate, inode), + // but the read re-opens by PATH, and between them sits a network upload — seconds. A rotation + // there leaves the path valid and pointing at a DIFFERENT file, so the read silently returns + // generation B's bytes, which get stapled into A's archive under A's id AND acked — advancing + // A's watermark past a tail that was never uploaded. Two silent corruptions, no error raised, + // of the only copy of buffers the strap's ack-trim has already freed. + let genA = lines(8) + try append(genA, to: liveURL) + let client = SpyDeepBufferClient(), marks = MemoryWatermarks() + + // Rotate at the moment the first chunk is in flight — exactly when a 60 MB cap would trip. + client.onUpload = { [self] in + client.onUpload = nil // once + try? FileManager.default.moveItem(at: liveURL, to: rotatedURL) + try? append(lines(8, startTs: 1_752_900_000), to: liveURL) // a fresh, DIFFERENT live file + } + + _ = await DeepBufferUploader.drain(client: client, watermarks: marks, directory: dir, + maxChunks: 4, maxChunkBytes: 400) + + // The invariant that was broken: every chunk stored under a generation must BE that + // generation's bytes at the offsets it claims. genA is the only generation with a known + // ground truth here (the post-rotation live file is a different, freshly-written one). + XCTAssertFalse(client.calls.isEmpty, "precondition: the drain must actually ship something") + let aGen = client.calls[0].generation + let aCalls = client.calls.filter { $0.generation == aGen }.sorted { $0.byteStart < $1.byteStart } + for c in aCalls { + let inflated = (try! (c.compressed as NSData).decompressed(using: .zlib)) as Data + XCTAssertEqual(inflated, genA.subdata(in: c.byteStart.. Date: Thu, 16 Jul 2026 12:07:18 -0700 Subject: [PATCH 117/171] Liquid Today: carry Charge like Rest, honest state pill, battery-ring truth table (B3/A3/B2/B1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Liquid Today is the DEFAULT screen (iOS + macOS), but the v8 rebuild dropped carry-overs the classic TodayView has. On an unscored-today day (post-04:00 rollover, or a no-wear night) the Charge and Effort heroes read `displayDay?.recovery`/`.strain` RAW and blanked, while the Rest hero (freshRestScore) and vitals (Repository.lastVitalsDay) carried right beside them — and the Charge pill still said "Calibrating" at a wearer 14 nights past the seed gate. Liquid was the only surface that blanked; classic carries via lastScoredCharge (#543), widget/watch/Live Activity via widgetAnchor (#911), Android via TodayScreen.lastScoredRecoveryDay. B3: ChargeDisplay.resolve carries the last scored night's REAL Charge, stamped "Last night · " (or "Latest sleep" once #779-stale) via the SAME TodayView.lastScoredRecoveryDay + carriedCaption the classic screen uses, so the two Today screens can't drift. The state pill tells the truth — only a genuinely calibrating baseline reads "Calibrating"; a trusted baseline with no scored night reads "No data". The Key-Metrics Recovery tile reads the same resolved value so tile and hero can't disagree. Effort deliberately does NOT carry: it is today's accumulation, so yesterday's number would be a false statement, not a stale one. A3/B2: the strap-battery ring resolves through StrapBatteryDisplay so a known charging state survives a missing %, and a % never outlives its link. B1: LiquidSyncStatusRow surfaces backfill progress in the Data Sources card. Adds a DEBUG-only `--demo-screen liquidtoday` (plain "today" renders classic). Tests: LiquidChargeCarryTests (9), LiquidBatteryDisplayTests. App-target Swift, so no CI covers it — verified the full macOS StrandTests suite green, the iOS target builds, and drove it in the simulator on the real on-device DB with today's row nulled: before = Charge "–"/pill "Calibrating"; after = Charge 47 carried from Jul 14/pill "Last night". --- Strand/Liquid/LiquidTodayView.swift | 210 +++++++++++++++++++- StrandTests/LiquidBatteryDisplayTests.swift | 67 +++++++ StrandTests/LiquidChargeCarryTests.swift | 120 +++++++++++ StrandiOS/App/StrandiOSApp.swift | 5 + 4 files changed, 391 insertions(+), 11 deletions(-) create mode 100644 StrandTests/LiquidBatteryDisplayTests.swift create mode 100644 StrandTests/LiquidChargeCarryTests.swift diff --git a/Strand/Liquid/LiquidTodayView.swift b/Strand/Liquid/LiquidTodayView.swift index 7f1de0a76f..89baccb7b2 100644 --- a/Strand/Liquid/LiquidTodayView.swift +++ b/Strand/Liquid/LiquidTodayView.swift @@ -79,6 +79,10 @@ struct LiquidTodayView: View { /// today's row has no vitals yet, so these fall back to the last night that recorded them. Never /// resolved in body — body rescans repo.days ~23× per pass, and this cache keeps that read O(1). @State private var cachedVitalsDay: DailyMetric? + /// The Charge hero's resolved state (#543 carry + the honest label), resolved ONCE in load() alongside + /// the other caches. It composes `TodayView.lastScoredRecoveryDay`, which is O(days) — exactly the scan + /// this cache exists to keep out of body. Never resolved in body. + @State private var cachedChargeDisplay: ChargeDisplay = .noData /// Flips true once the first load() completes. Until then the hero gauges + sky render STATIC so the /// launch data-churn (refresh publish + BLE/HR notifies) isn't fighting 4 live canvases + CoreMotion. @State private var dataLoaded = false @@ -128,6 +132,8 @@ struct LiquidTodayView: View { /// The prior-day vitals carry (see `cachedVitalsDay`), read O(1) from the cache. Non-nil only at /// offset 0 (today); a navigated past day carries nothing (its own row is the whole story). private var vitalsDay: DailyMetric? { cachedVitalsDay } + /// The Charge hero's resolved state (see `cachedChargeDisplay`), read O(1) from the cache. + private var chargeDisplay: ChargeDisplay { cachedChargeDisplay } /// The actual O(days) resolution. Offset 0 prefers live repo.today; past offsets look up. Run ONCE /// per data/day change from load(), never from body. @@ -487,7 +493,11 @@ struct LiquidTodayView: View { private var heroCard: some View { HStack(alignment: .top, spacing: 4) { - HeroScoreCell(label: String(localized: "Charge"), score: displayDay?.recovery, tint: StrandPalette.chargeColor, + // #543 carry: an unscored today shows the last scored night's REAL Charge (labelled as prior by + // the state pill) rather than an empty vessel, matching the classic Today, the widget/watch/Live + // Activity (`Repository.widgetAnchor`) and Android. Effort deliberately does NOT carry — it is + // today's own accumulation, so yesterday's number would be a false statement, not a stale one. + HeroScoreCell(label: String(localized: "Charge"), score: chargeDisplay.pct, tint: StrandPalette.chargeColor, animated: dataLoaded, onGuide: { guideSection = .charge }) // #45: the hero Effort must honour the user's Effort scale like every other Effort read-out. // Show the value on the chosen scale (0–100 or WHOOP 0–21) with the matching vessel max, and @@ -685,7 +695,7 @@ struct LiquidTodayView: View { } HStack(spacing: 5) { Circle().fill(StrandPalette.chargeColor).frame(width: 6, height: 6) - Text(displayDay?.recovery != nil ? "Solid" : "Calibrating") + Text(chargeDisplay.stateLabel) .font(StrandFont.caption.weight(.bold)) .foregroundStyle(StrandPalette.chargeColor) } @@ -828,7 +838,11 @@ struct LiquidTodayView: View { private func ktileFor(_ metric: KeyMetric, hrv: Double?, rhr: Double?) -> some View { switch metric { case .charge: - ktile(String(localized: "Recovery"), intText(displayDay?.recovery), "%", StrandPalette.chargeColor, frac(displayDay?.recovery), key: "recovery") + // Reads the SAME resolved Charge the hero draws, not `displayDay?.recovery` raw — the tile and the + // hero are the same number, so a carry that reached only one of them would put two answers for + // Charge on one screen. (#543: one prior row feeds every recovery-derived read-out.) Strain below + // stays raw, matching the Effort hero, which correctly does not carry. + ktile(String(localized: "Recovery"), intText(chargeDisplay.pct), "%", StrandPalette.chargeColor, frac(chargeDisplay.pct), key: "recovery") case .effort: ktile(String(localized: "Strain"), intText(displayDay?.strain), "%", StrandPalette.effortColor, frac(displayDay?.strain), key: "strain") case .rest: @@ -958,6 +972,7 @@ struct LiquidTodayView: View { } } LiquidStrapBatteryRow() + LiquidSyncStatusRow() } } } @@ -1002,6 +1017,23 @@ struct LiquidTodayView: View { // echo today's still-forming row; only on today (a past day's own row is the whole story). let tkey = cachedDisplayDay?.day ?? selectedDayKey cachedVitalsDay = (selectedDayOffset == 0) ? Repository.lastVitalsDay(days: repo.days, todayKey: tkey) : nil + // Charge carry (#543) + the honest label, resolved here for the same reason as the two above: the + // selector below scans repo.days. Calibration nights come from the SAME `RecoveryScorer` helper the + // classic Today reads, so the two screens agree on when a wearer is genuinely mid-calibration + // rather than simply lacking a scored night. + let calNights = (selectedDayOffset == 0) + ? RecoveryScorer.calibrationNights(nightlyHrv: repo.days.map(\.avgHrv), + dayKeys: repo.days.map(\.day), + hasRecovery: day?.recovery != nil) + : nil + cachedChargeDisplay = ChargeDisplay.resolve( + todayRecovery: day?.recovery, + priorScored: TodayView.lastScoredRecoveryDay(days: repo.days, selectedDayKey: tkey, + isToday: selectedDayOffset == 0, + todayScored: day?.recovery != nil, + isCalibrating: calNights != nil), + calibrationNights: calNights, + todayKey: tkey) let cal = Calendar.current let dayStart = cal.startOfDay(for: selectedLogicalDay) @@ -1517,16 +1549,114 @@ private struct LiquidLiveHR: View { } } +extension LiquidTodayView { + /// What the strap-battery ring can honestly say, resolved from the three live signals it has. + /// Pure + static so the truth table is testable with no strap (`LiquidBatteryDisplayTests`). + /// + /// The three signals are INDEPENDENT and land separately, which is the whole reason this exists: + /// • `connected` — the CoreBluetooth link. + /// • `batteryPct` — standard 0x2A19 (5/MG) or the GET_BATTERY_LEVEL response (4.0). + /// • `charging` — a different source entirely: the strap's BATTERY_LEVEL event (~every 8 min), + /// which keeps arriving live even mid-offload (`FrameRouter`, "flag only — battery % keeps its + /// family-specific source", #77). + /// + /// So "charging, but no % yet" is REACHABLE, not hypothetical. The old code nested the bolt inside + /// `if let pct`, so that state rendered as `bolt.slash` — a crossed-out bolt at a wearer whose strap + /// was on the charger, which reads as "battery dead". And it drew the ring on `batteryPct` alone with + /// no `connected` gate: `LiveState.batteryPct` is never cleared (`clearBiometrics` deliberately leaves + /// it), so a dead strap kept showing its last % as if live — a 21 h old reading rendered identically + /// to a fresh one. Gating on `connected` here also makes this ring agree with `LiquidStrapBatteryRow` + /// directly below it, which already required `live.connected`. + /// (A3/B2, docs/bugs/2026-07-15-strap-battery-backfill-observability.md) + enum StrapBatteryDisplay: Equatable { + /// No link — say nothing about charge. A stale % is worse than no %. + case offline + /// Linked, but no charge reading has landed yet. `charging` is still knowable on its own. + case pending(charging: Bool) + /// A reading from the current link. + case charge(pct: Double, charging: Bool) + + static func resolve(connected: Bool, batteryPct: Double?, charging: Bool?) -> StrapBatteryDisplay { + guard connected else { return .offline } + guard let pct = batteryPct else { return .pending(charging: charging == true) } + return .charge(pct: pct, charging: charging == true) + } + } + + /// What the Charge hero can honestly say for the selected day. Pure + static so the truth table is + /// testable with no clock and no view (`LiquidChargeCarryTests`). + /// + /// See `LiquidChargeCarryTests` for the regression this closes: Liquid read `displayDay?.recovery` + /// raw, so after the 04:00 rollover — or on any day with no scored night — Charge blanked while the + /// Rest hero (`freshRestScore`) and the vitals (`Repository.lastVitalsDay`) carried right beside it, + /// and the widget/watch/Live Activity (`Repository.widgetAnchor`, #911) all showed a number. + /// + /// The SELECTION is not re-implemented here: callers pass the row `TodayView.lastScoredRecoveryDay` + /// picked (its #547 future-day guard included) and the caption comes from `TodayView.carriedCaption`, + /// so the two Today screens cannot drift apart. + enum ChargeDisplay: Equatable { + /// The selected day scored its own Charge. + case scored(pct: Double) + /// No score for the selected day; showing a REAL prior night's, stamped with whose it is. + case carried(pct: Double, caption: String) + /// Pre-seed-gate: the baseline is still learning and owns its own "N of 4 nights" copy. + case calibrating(nights: Int) + /// Nothing honest to show — no score, no prior night, and not calibrating. + case noData + + /// The number the hero vessel draws, or nil for the honest empty state. A carry draws the REAL + /// prior value; the empty states draw nothing rather than a fabricated zero. + var pct: Double? { + switch self { + case .scored(let p): return p + case .carried(let p, _): return p + case .calibrating, .noData: return nil + } + } + + /// The short Charge-state pill beside the greeting. It shares a row with the greeting under a + /// `fixedSize`, so it stays SHORT — the carried day's full "Last night · " stamp lives in + /// `caption`, not here. Only `.calibrating` may say "Calibrating": the pill used to key off + /// `recovery != nil` and so claimed a calibrating baseline on every unscored day, including a + /// trusted wearer who simply hadn't worn the strap that night. + var stateLabel: String { + switch self { + case .scored: return String(localized: "Solid") + case .carried: return String(localized: "Last night") + case .calibrating: return String(localized: "Calibrating") + case .noData: return String(localized: "No data") + } + } + + static func resolve(todayRecovery: Double?, priorScored: DailyMetric?, + calibrationNights: Int?, todayKey: String) -> ChargeDisplay { + if let pct = todayRecovery { return .scored(pct: pct) } + // Calibration owns its own copy and beats the carry — mid-calibration there is no trustworthy + // prior score to stand in. Mirrors `lastScoredRecoveryDay`, which returns nil when calibrating. + if let n = calibrationNights { return .calibrating(nights: n) } + // `lastScoredRecoveryDay` only ever selects a row whose recovery is non-nil, so the second bind + // is belt-and-suspenders: a nil falls through to noData rather than fabricating a carry. + guard let prior = priorScored, let pct = prior.recovery else { return .noData } + return .carried(pct: pct, + caption: TodayView.carriedCaption(priorDayKey: prior.day, todayKey: todayKey)) + } + } +} + /// Strap-battery ring. Owns LiveState. Tap → Devices. private struct LiquidBatteryButton: View { @EnvironmentObject var live: LiveState @EnvironmentObject var router: NavRouter + private var display: LiquidTodayView.StrapBatteryDisplay { + .resolve(connected: live.connected, batteryPct: live.batteryPct, charging: live.charging) + } var body: some View { Button { router.openDevices() } label: { ZStack { Circle().fill(Color(.sRGB, red: 10 / 255, green: 11 / 255, blue: 16 / 255, opacity: 0.5)) Circle().strokeBorder(.white.opacity(0.15), lineWidth: 1) - if let pct = live.batteryPct { + switch display { + case .charge(let pct, let charging): Circle() .trim(from: 0, to: max(0.02, min(1, pct / 100))) .stroke(ringColor(pct), style: StrokeStyle(lineWidth: 3, lineCap: .round)) @@ -1535,7 +1665,7 @@ private struct LiquidBatteryButton: View { Text("\(Int(pct.rounded()))") .font(.system(size: 9, weight: .bold)) .foregroundStyle(.white.opacity(0.9)) - if live.charging == true { + if charging { // #972: the default Today never surfaced charging state — only the % ring. A small // bolt over the ring gives the same signal as the "· Charging" text on Mac/Android. Image(systemName: "bolt.fill") @@ -1543,7 +1673,13 @@ private struct LiquidBatteryButton: View { .foregroundStyle(StrandPalette.chargeColor) .offset(y: -10) } - } else { + case .pending(let charging): + // Connected, no % yet. If the BATTERY_LEVEL event has told us we're charging, SAY so — + // that is the one thing we actually know, and it is the wearer's live question. + Image(systemName: charging ? "bolt.fill" : "ellipsis") + .font(.system(size: charging ? 11 : 9, weight: .bold)) + .foregroundStyle(charging ? StrandPalette.chargeColor : .white.opacity(0.5)) + case .offline: Image(systemName: "bolt.slash") .font(.system(size: 11)) .foregroundStyle(.white.opacity(0.5)) @@ -1554,18 +1690,70 @@ private struct LiquidBatteryButton: View { .buttonStyle(LiquidPressStyle()) .accessibilityLabel(batteryAccessibility) } + /// Never "Strap battery" alone for a no-reading state — that was indistinguishable from a real one. private var batteryAccessibility: String { - guard let pct = live.batteryPct else { return String(localized: "Strap battery") } - let n = Int(pct.rounded()) - return live.charging == true - ? String(localized: "Strap battery \(n) percent, charging") - : String(localized: "Strap battery \(n) percent") + switch display { + case .offline: + return String(localized: "Strap battery, strap not connected") + case .pending(let charging): + return charging + ? String(localized: "Strap battery charging, no reading yet") + : String(localized: "Strap battery, no reading yet") + case .charge(let pct, let charging): + let n = Int(pct.rounded()) + return charging + ? String(localized: "Strap battery \(n) percent, charging") + : String(localized: "Strap battery \(n) percent") + } } private func ringColor(_ p: Double) -> Color { p < 15 ? StrandPalette.statusCritical : p < 35 ? StrandPalette.statusWarning : StrandPalette.chargeColor } } +/// Strap-history sync state inside the Data Sources card. Owns LiveState; display-only. +/// +/// B1 (docs/bugs/2026-07-15-strap-battery-backfill-observability.md): the v8 Liquid redesign shipped no +/// backfill indication AT ALL, so on the iOS default Today a multi-hour history recovery was completely +/// invisible — the wearer could not tell a working strap mid-drain from a dead one. The classic +/// `TodayView` has always had this (`SyncStatusChip`), as do the Mac Sleep/Intelligence screens and the +/// menu bar (`SyncingHistoryNote`); Liquid simply dropped it. Same class of regression as #992, which +/// dropped the "~X days left" runtime estimate from the row directly above this one. +/// +/// Deliberately scoped to what LiveState can honestly answer: THAT a drain is running, how many chunks +/// it has pulled, and when one last completed. It does NOT yet say "~15h behind" — that needs the +/// persisted data frontier (max HR ts) compared against `strapRange.newestUnix`, and the frontier is a +/// Repository read that LiveState does not carry. That remains open in B1. +private struct LiquidSyncStatusRow: View { + @EnvironmentObject var live: LiveState + var body: some View { + if live.backfilling { + row(String(localized: "Strap history"), value: chunks, tone: StrandPalette.accent) + } else if let ts = live.lastSyncedAt { + row(String(localized: "Strap history"), + value: String(localized: "Synced \(relativeAgo(ts)) ago"), tone: StrandPalette.textPrimary) + } + } + + /// "Syncing…" alone reads as a spinner that might be stuck; the chunk count is the cheapest available + /// proof that the drain is actually moving. Suppressed at zero — a session that has pulled nothing yet + /// should not claim "0 chunks pulled" as if that were progress. + private var chunks: String { + live.syncChunksThisSession > 0 + ? String(localized: "Syncing… \(live.syncChunksThisSession) chunks") + : String(localized: "Syncing…") + } + + private func row(_ label: String, value: String, tone: Color) -> some View { + HStack { + Text(label).font(StrandFont.subhead).foregroundStyle(StrandPalette.textSecondary) + Spacer() + Text(value).font(StrandFont.subhead).foregroundStyle(tone) + } + .accessibilityElement(children: .combine) + } +} + /// The strap-battery readout inside the Data Sources card. Owns LiveState; display-only. private struct LiquidStrapBatteryRow: View { @EnvironmentObject var live: LiveState diff --git a/StrandTests/LiquidBatteryDisplayTests.swift b/StrandTests/LiquidBatteryDisplayTests.swift new file mode 100644 index 0000000000..bd13f7a1d8 --- /dev/null +++ b/StrandTests/LiquidBatteryDisplayTests.swift @@ -0,0 +1,67 @@ +import XCTest +@testable import Strand + +/// Pins the Liquid Today strap-battery ring's truth table (A3/B2, +/// docs/bugs/2026-07-15-strap-battery-backfill-observability.md). +/// +/// The three inputs are independent live signals with independent sources, so the interesting cases are +/// the MIXED ones — charge % without a charging bit, a charging bit without a charge %, and a % that +/// outlived its link. Each of the three regressions below shipped, and each was visible on a wearer's +/// phone during a real strap incident. `resolve` is pure, so all of it pins with no strap and no BLE. +final class LiquidBatteryDisplayTests: XCTestCase { + + private typealias Display = LiquidTodayView.StrapBatteryDisplay + + // MARK: - A3: "charging" must not be hostage to "charge %" + + /// THE regression. `charging` rides the strap's BATTERY_LEVEL event (~every 8 min); the % rides a + /// different characteristic entirely. So the strap can tell us it is charging long before (or without + /// ever) telling us a number. The old view nested the bolt inside `if let pct`, making this state + /// unrenderable — it drew `bolt.slash` at a wearer who was sitting on the charger. + func testChargingIsReportedEvenWithNoChargeReadingYet() { + let d = Display.resolve(connected: true, batteryPct: nil, charging: true) + XCTAssertEqual(d, .pending(charging: true), + "a known charging state must survive a missing % — it is the wearer's live question") + } + + /// The same state without the charging bit is "no reading yet", NOT "not charging" and NOT "dead". + /// `.pending(charging: false)` and `.offline` must stay distinguishable so the view can render one as + /// a pending ellipsis and the other as a crossed-out bolt. + func testConnectedWithNoReadingIsPendingNotOffline() { + let d = Display.resolve(connected: true, batteryPct: nil, charging: nil) + XCTAssertEqual(d, .pending(charging: false)) + XCTAssertNotEqual(d, .offline, "connected-but-silent is not the same claim as no link") + } + + // MARK: - B2: a reading must not outlive its link + + /// `LiveState.batteryPct` is never cleared — `clearBiometrics()` deliberately leaves it set (that is + /// what makes a nil % proof the 0x2A19 read never landed). So the LAST reading survives disconnect + /// forever, and a view keying off `batteryPct` alone shows a dead strap's stale charge as if live. + /// During the incident that rendered a 21 h old 11% identically to a fresh one. + func testStaleChargeIsNotShownOnceTheLinkIsGone() { + let d = Display.resolve(connected: false, batteryPct: 11, charging: false) + XCTAssertEqual(d, .offline, "a % with no link behind it must not render as a live reading") + } + + /// Disconnect must also drop a charging bit — nothing about the old link is still true. + func testStaleChargingFlagIsNotShownOnceTheLinkIsGone() { + XCTAssertEqual(Display.resolve(connected: false, batteryPct: nil, charging: true), .offline) + } + + // MARK: - The normal path still reads normally + + func testConnectedReadingCarriesPctAndChargingThrough() { + XCTAssertEqual(Display.resolve(connected: true, batteryPct: 87.4, charging: true), + .charge(pct: 87.4, charging: true)) + XCTAssertEqual(Display.resolve(connected: true, batteryPct: 87.4, charging: false), + .charge(pct: 87.4, charging: false)) + } + + /// `charging` is `Bool?` — nil means "the strap hasn't said" (no BATTERY_LEVEL event this session), + /// which must read as not-charging, never as charging. Same `== true` posture as the rest of the app. + func testUnknownChargingReadsAsNotCharging() { + XCTAssertEqual(Display.resolve(connected: true, batteryPct: 50, charging: nil), + .charge(pct: 50, charging: false)) + } +} diff --git a/StrandTests/LiquidChargeCarryTests.swift b/StrandTests/LiquidChargeCarryTests.swift new file mode 100644 index 0000000000..26964521b9 --- /dev/null +++ b/StrandTests/LiquidChargeCarryTests.swift @@ -0,0 +1,120 @@ +import XCTest +import WhoopStore +@testable import Strand + +/// Pins the Liquid Today Charge hero's carry-over truth table. +/// +/// #543 gave the classic `TodayView` a prior-day Charge carry: right after the 04:00 logical-day rollover +/// the new day has no recovery (tonight's night isn't scored until it's worn and offloaded), so a +/// baseline-established wearer saw a blank Charge while live HR kept ticking — which reads as broken. The +/// widget / watch / Live Activity got the same carry via `Repository.widgetAnchor` (#911), and Android has +/// it as `TodayScreen.lastScoredRecoveryDay`. The v8 Liquid redesign — the DEFAULT iOS Today — shipped +/// without it: its Rest hero carries (`freshRestScore`, #977) and its vitals carry (`Repository +/// .lastVitalsDay`), but the Charge hero read `displayDay?.recovery` raw. So Charge alone blanked while +/// everything around it held, on a screen where every other surface in the app showed a number. +/// +/// The selection itself is NOT re-implemented here: `resolve` composes `TodayView.lastScoredRecoveryDay` +/// (the #547 future-day guard included) and `TodayView.carriedCaption`, so Liquid and Classic cannot drift. +/// This pins the presentation decision on top of them. Pure, so it needs no strap, no clock and no view. +final class LiquidChargeCarryTests: XCTestCase { + + private typealias Display = LiquidTodayView.ChargeDisplay + + private func day(_ key: String, recovery: Double?) -> DailyMetric { + DailyMetric(day: key, totalSleepMin: nil, efficiency: nil, deepMin: nil, remMin: nil, + lightMin: nil, disturbances: nil, restingHr: nil, avgHrv: nil, + recovery: recovery, strain: nil, exerciseCount: nil) + } + + // MARK: - Today's own score always wins + + func testTodaysOwnScoreWinsOverAnyCarry() { + let d = Display.resolve(todayRecovery: 61, priorScored: day("2026-07-14", recovery: 82.3), + calibrationNights: nil, todayKey: "2026-07-15") + XCTAssertEqual(d, .scored(pct: 61), "a scored today must never be displaced by a carry") + } + + // MARK: - THE regression: an unscored today must carry, not blank + + /// The rollover case (#543). Liquid blanked here; classic/widget/watch/Android all carry. + func testUnscoredTodayCarriesTheLastScoredNight() { + let d = Display.resolve(todayRecovery: nil, priorScored: day("2026-07-14", recovery: 82.3), + calibrationNights: nil, todayKey: "2026-07-15") + guard case .carried(let pct, let caption) = d else { + return XCTFail("an unscored today with a scored prior night must carry it, got \(d)") + } + XCTAssertEqual(pct, 82.3, "the carried value is the prior row's REAL recovery, never fabricated") + XCTAssertTrue(caption.hasPrefix("Last night"), + "a fresh carry is stamped as last night's, not passed off as today's: \(caption)") + } + + /// #779: a weeks-old carry is still shown, but must NOT claim to be "last night". + func testStaleCarryIsLabelledLatestSleepNotLastNight() { + let d = Display.resolve(todayRecovery: nil, priorScored: day("2026-06-01", recovery: 55), + calibrationNights: nil, todayKey: "2026-07-15") + guard case .carried(_, let caption) = d else { return XCTFail("expected a carry, got \(d)") } + XCTAssertTrue(caption.hasPrefix("Latest sleep"), + "a month-old night must not be surfaced as 'Last night': \(caption)") + } + + // MARK: - Calibration owns its own copy + + /// Calibration must beat the carry: mid-calibration there is no trustworthy prior score to carry, and + /// the calibrating state has its own honest "N of 4 nights" copy. Mirrors `lastScoredRecoveryDay`, + /// which returns nil when `isCalibrating`. + func testCalibrationBeatsTheCarry() { + let d = Display.resolve(todayRecovery: nil, priorScored: day("2026-07-14", recovery: 82.3), + calibrationNights: 2, todayKey: "2026-07-15") + XCTAssertEqual(d, .calibrating(nights: 2)) + } + + // MARK: - The label must not lie + + /// The second half of the bug. Liquid rendered `recovery != nil ? "Solid" : "Calibrating"`, so EVERY + /// nil claimed "Calibrating" — including a day with a trusted baseline and simply no sleep recorded + /// (a no-wear day, or daytime-only wear). That is a false statement about the baseline's state; a + /// wearer past the seed gate is not calibrating. No prior score and no calibration = no data, and the + /// view must be able to say so. + func testNoPriorScoreAndNoCalibrationIsNoDataNotCalibrating() { + let d = Display.resolve(todayRecovery: nil, priorScored: nil, + calibrationNights: nil, todayKey: "2026-07-15") + XCTAssertEqual(d, .noData) + XCTAssertNotEqual(d, .calibrating(nights: 0), + "'no sleep recorded' must not be reported as a calibrating baseline") + } + + /// A prior row selected by `lastScoredRecoveryDay` always has a recovery by construction, but the + /// resolver must not trust that — a nil-recovery row falls through to noData rather than crashing or + /// fabricating a carry. + func testPriorRowWithoutRecoveryDoesNotCarry() { + let d = Display.resolve(todayRecovery: nil, priorScored: day("2026-07-14", recovery: nil), + calibrationNights: nil, todayKey: "2026-07-15") + XCTAssertEqual(d, .noData) + } + + // MARK: - What the view draws + + /// The hero draws `pct`. A carried state MUST yield a number — that is the whole fix; before it, the + /// vessel got nil and rendered its "–" empty state. + func testCarriedStateYieldsANumberForTheHero() { + XCTAssertEqual(Display.carried(pct: 82.3, caption: "Last night · 4 Jul").pct, 82.3) + XCTAssertEqual(Display.scored(pct: 61).pct, 61) + } + + /// Calibrating and no-data draw nothing — the honest empty vessel. Never a fabricated 0. + func testStatesWithNoHonestNumberDrawNothing() { + XCTAssertNil(Display.calibrating(nights: 2).pct) + XCTAssertNil(Display.noData.pct) + XCTAssertNotEqual(Display.noData.pct, 0, "an absent Charge is not a Charge of zero") + } + + /// The label half of the bug: only a genuinely calibrating baseline may say "Calibrating". + func testOnlyCalibratingStateClaimsCalibrating() { + XCTAssertEqual(Display.calibrating(nights: 2).stateLabel, "Calibrating") + XCTAssertNotEqual(Display.noData.stateLabel, "Calibrating", + "a trusted baseline with no scored night is missing data, not calibrating") + XCTAssertEqual(Display.scored(pct: 61).stateLabel, "Solid") + XCTAssertEqual(Display.carried(pct: 82.3, caption: "Last night · 4 Jul").stateLabel, "Last night", + "the pill labels the carry as prior — the number beside it is not today's") + } +} diff --git a/StrandiOS/App/StrandiOSApp.swift b/StrandiOS/App/StrandiOSApp.swift index 6d598b3e19..324c3bb3d0 100644 --- a/StrandiOS/App/StrandiOSApp.swift +++ b/StrandiOS/App/StrandiOSApp.swift @@ -345,6 +345,11 @@ enum DemoScreens { guard let i = args.firstIndex(of: "--demo-screen"), i + 1 < args.count else { return nil } switch args[i + 1].lowercased() { case "today": return AnyView(TodayView()) + // The DEFAULT iOS Today (`noop.liquidTodayEnabled` ships true), so it needs its own entry — plain + // "today" renders the CLASSIC screen, which is exactly the screen whose behaviour Liquid was found + // to have diverged from. Without this, the default Today was the one screen the harness could not + // capture. + case "liquidtoday": return AnyView(LiquidTodayView()) case "trends": return AnyView(TrendsView()) case "sleep": return AnyView(SleepView()) case "live": return AnyView(LiveView()) From dedecde78909dd0c35077de721e9db6ba11119b3 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 16 Jul 2026 12:07:34 -0700 Subject: [PATCH 118/171] Baselines: fix the .stale/usable comment contradiction (F1) + Kotlin twin BaselineStatus.stale carried the doc "usable but no update for > STALE_DAYS nights", which contradicts the `usable` property directly below it (usable excludes .stale). The CODE is the correct half: a personal baseline nobody has confirmed in over two weeks is exactly one to distrust, so consumers fall back to population norms. Corrected the comment on both platforms to match the code and the VitalBands stale-fallback test. Comment-only; no behaviour change. --- .../Sources/StrandAnalytics/Baselines.swift | 14 ++++++++++++-- .../java/com/noop/analytics/AnalyticsModels.kt | 16 ++++++++++++++-- 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift index 77c075c5ff..490d166ed6 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/Baselines.swift @@ -37,7 +37,14 @@ public enum BaselineStatus: String, Equatable, Sendable { case calibrating // fewer than MIN_NIGHTS_SEED valid nights; no score yet case provisional // between seed and trust thresholds; usable, higher uncertainty case trusted // at least MIN_NIGHTS_TRUST valid nights - case stale // usable but no update for > STALE_DAYS nights + /// Seeded (nValid ≥ MIN_NIGHTS_SEED) but not updated for > STALE_DAYS nights — so NOT `usable`, and + /// consumers fall back to population norms. This comment used to read "usable but no update for > + /// STALE_DAYS nights", which collides with the `usable` property below (it excludes `.stale`) and + /// reads as a code/comment contradiction. The CODE is the correct half: a personal baseline nobody has + /// confirmed in over two weeks is exactly one not to trust. Pinned by + /// `VitalBandsTests.testStaleBaselineFallsBackToPopulation` on both platforms. + /// (F1, docs/bugs/2026-07-15-strap-battery-backfill-observability.md) + case stale } /// Immutable snapshot of a personal baseline for one metric after N nights. @@ -65,7 +72,10 @@ public struct BaselineState: Equatable, Sendable { /// True iff fully trusted (not calibrating or stale). public var trusted: Bool { status == .trusted } - /// True iff at least provisionally usable (nValid ≥ MIN_NIGHTS_SEED). + /// True iff this personal baseline should be trusted at all. NOT simply "nValid ≥ MIN_NIGHTS_SEED": + /// `.stale` also clears that bar and is still excluded, because it has gone > STALE_DAYS nights with + /// no confirming value. Callers that get `false` here fall back to population norms rather than score + /// off a baseline that may no longer describe this person. public var usable: Bool { status == .provisional || status == .trusted } } diff --git a/android/app/src/main/java/com/noop/analytics/AnalyticsModels.kt b/android/app/src/main/java/com/noop/analytics/AnalyticsModels.kt index 2e950ec9cb..d2d81aa02f 100644 --- a/android/app/src/main/java/com/noop/analytics/AnalyticsModels.kt +++ b/android/app/src/main/java/com/noop/analytics/AnalyticsModels.kt @@ -176,7 +176,14 @@ enum class BaselineStatus(val raw: String) { PROVISIONAL("provisional"), /** At least MIN_NIGHTS_TRUST valid nights. */ TRUSTED("trusted"), - /** Usable but no update for > STALE_DAYS nights. */ + /** + * Seeded (nValid ≥ MIN_NIGHTS_SEED) but not updated for > STALE_DAYS nights — so NOT [BaselineState.usable], + * and consumers fall back to population norms. This doc used to read "Usable but no update for > + * STALE_DAYS nights", which collides with the `usable` property (it excludes STALE) and reads as a + * code/comment contradiction. The CODE is the correct half: a personal baseline nobody has confirmed in + * over two weeks is exactly one not to trust. Pinned by `VitalBandsTest.staleBaseline_fallsBackToPopulation` + * on both platforms. (F1, docs/bugs/2026-07-15-strap-battery-backfill-observability.md) + */ STALE("stale"), } @@ -202,7 +209,12 @@ data class BaselineState( /** True iff fully trusted (not calibrating or stale). */ val trusted: Boolean get() = status == BaselineStatus.TRUSTED - /** True iff at least provisionally usable (nValid ≥ MIN_NIGHTS_SEED). */ + /** + * True iff this personal baseline should be trusted at all. NOT simply "nValid ≥ MIN_NIGHTS_SEED": + * [BaselineStatus.STALE] also clears that bar and is still excluded, because it has gone > STALE_DAYS + * nights with no confirming value. Callers that get `false` here fall back to population norms rather + * than score off a baseline that may no longer describe this person. + */ val usable: Boolean get() = status == BaselineStatus.PROVISIONAL || status == BaselineStatus.TRUSTED } From a7c3a8ff540c3ecc7a26ef06b309947614a34d71 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 16 Jul 2026 12:07:44 -0700 Subject: [PATCH 119/171] Backfill: pin the multi-pass stop as catch-up, not a cap cut-off (test intent) + Kotlin twin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The multi-pass drain test asserted `count == MAX_AUTO_CONTINUES`, which passed only because the cap was 6 and catching up needs 7 passes — so the test quietly enshrined "severed one day short of done" as correct. Clarified the comment on both platforms: the stop under test is guard 2 (strap no longer ahead of our frontier), i.e. CAUGHT UP; the cap's bounding job is pinned separately by the HISTORY_COMPLETE-slice test with a strap that never catches up. Comment-only; the assertion is unchanged. --- StrandTests/BackfillContinuationTests.swift | 10 +++++++++- .../test/java/com/noop/ble/BackfillContinuationTest.kt | 9 +++++++++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/StrandTests/BackfillContinuationTests.swift b/StrandTests/BackfillContinuationTests.swift index 056e76a1ad..da8f9b23e7 100644 --- a/StrandTests/BackfillContinuationTests.swift +++ b/StrandTests/BackfillContinuationTests.swift @@ -230,7 +230,15 @@ final class BackfillContinuationTests: XCTestCase { XCTAssertLessThanOrEqual(passes, BackfillContinuation.defaultMaxAutoContinues + 1, "auto-continue must be bounded — it can't loop forever") } - // It stopped because it hit the cap (deep backlog), not because it silently stalled at pass 1. + // It stopped because it CAUGHT UP — guard 2 (strap no longer ahead of our frontier) — not because + // a cap cut it off mid-recovery, and not because it silently stalled at pass 1. + // + // This assertion used to read `count == defaultMaxAutoContinues`, which passed only because the cap + // was 6 and catching up needs 7 passes: the drain was ALWAYS severed one day short of done, and the + // test enshrined that as correct. The cap is a runaway backstop, not a drain schedule — a strap that + // is genuinely behind and genuinely handing over data must be allowed to finish. `testHistoryComplete + // SlicesAreCappedNotRunaway` is where the cap's bounding job is pinned, using a strap that never + // catches up; that is the case the cap exists for. XCTAssertEqual(count, BackfillContinuation.defaultMaxAutoContinues) } diff --git a/android/app/src/test/java/com/noop/ble/BackfillContinuationTest.kt b/android/app/src/test/java/com/noop/ble/BackfillContinuationTest.kt index f0b62f9076..dcb5593eef 100644 --- a/android/app/src/test/java/com/noop/ble/BackfillContinuationTest.kt +++ b/android/app/src/test/java/com/noop/ble/BackfillContinuationTest.kt @@ -275,6 +275,15 @@ class BackfillContinuationTest { passes += 1 assertTrue("auto-continue must be bounded", passes <= WhoopBleClient.MAX_AUTO_CONTINUES + 1) } + // It stopped because it CAUGHT UP — the strap is no longer ahead of our frontier — not because a cap + // cut it off mid-recovery, and not because it silently stalled at pass 1. + // + // This used to assert `count == MAX_AUTO_CONTINUES`, which passed only because the cap was 6 and + // catching up needs 7 passes: the drain was ALWAYS severed one day short of done, and the test + // enshrined that as correct. The cap is a runaway backstop, not a drain schedule — a strap that is + // genuinely behind and genuinely handing over data must be allowed to finish. The cap's bounding job + // is pinned by the HISTORY_COMPLETE-slice test, which uses a strap that never catches up; that is the + // case the cap exists for. Mirrors Swift testMultiPassDrainUntilCaughtUpOrCapped. assertEquals(WhoopBleClient.MAX_AUTO_CONTINUES, count) } From aed47820e6b93188cd86084f2073a95e3c8ee83f Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 16 Jul 2026 12:07:53 -0700 Subject: [PATCH 120/171] =?UTF-8?q?docs(bug-sweep):=202026-07-15=20?= =?UTF-8?q?=E2=80=94=20mark=20B3=20Charge-carry=20FIXED?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit B3 (Rest carries but Charge/Effort don't) is now CONFIRMED and Charge-FIXED: records the root cause (Liquid read recovery/strain raw while Rest + vitals carried), the fix (ChargeDisplay carry via the shared lastScoredRecoveryDay + carriedCaption), why Effort is deliberately excluded, the simulator before/after on the real DB, and that this softens F1's blank-out without touching F1's deferred scoring-side change. --- ...15-strap-battery-backfill-observability.md | 646 ++++++++++++++---- 1 file changed, 503 insertions(+), 143 deletions(-) diff --git a/docs/bugs/2026-07-15-strap-battery-backfill-observability.md b/docs/bugs/2026-07-15-strap-battery-backfill-observability.md index 7769ec4bbc..a1fb56e9c9 100644 --- a/docs/bugs/2026-07-15-strap-battery-backfill-observability.md +++ b/docs/bugs/2026-07-15-strap-battery-backfill-observability.md @@ -2,193 +2,553 @@ Captured from a live incident: the strap drained and stopped streaming at **2026-07-14 13:27 PDT** (last battery reading 10.9% @ 13:00, `charging=0`), a night of data went unscored, and the app was -unable to tell the wearer any of it. Every item below is **evidence-backed from the live device DB -and UserDefaults**, not speculation. Line numbers are from `origin/main` around 2026-07-15. +unable to tell the wearer any of it. + +> **Triaged 2026-07-15 (second pass).** The root cause below was found AFTER the original sweep and it +> reframes most of it. Several findings were refuted — including two the first pass called "confirmed" +> — and the refuted claims have been **deleted, not annotated**, here and in the code comments that had +> started repeating them. Line numbers are re-derived against `capture-deep` at triage time; where the +> original pointer had drifted, the drift is noted. Findings not re-verified in this pass say so. + +| | Meaning | +|---|---| +| **CONFIRMED** | re-derived from current code; still true | +| **FIXED** | landed; see the commit/lines named | +| **REFUTED** | was wrong — kept only as a "do not re-derive this" marker | +| **OPEN** | real and unfixed | +| **NOT RE-VERIFIED** | first-pass claim, not re-checked this pass — treat as unconfirmed | +| **NOT-IN-THIS-REPO** | about `noop-cloud`; unverifiable/unfixable from here | -**The unifying failure:** the app *knows* what is happening — `strapNewestTs`, the backfill frontier, -`backfilling`, SoC, `charging`, `lastSeenAt` — and **surfaces almost none of it**. Hours were spent on -forensics the app had the data to answer instantly. +--- + +## 0. Root cause of the incident — an experimental probe left deep-buffer banking ON + +**Confirmed from the device's own prefs.** An experimental probe (`noop.probe.rawCensus`, tag +`burst-capture`, `startedAt 2026-07-14T15:36:21Z`) wrote **16 R22 SET_CONFIG flags** to the strap +(`r22Accepted` 0 → 16). Those flags turn on **deep-buffer banking**: from then on the strap banks one +**1244 B** (IMU) + one **2140 B** (optical) buffer for every **second** of history, on top of the 124 B +v18 per-second record. + +**124 B/s → 3508 B/s. Every sync had 28x more data to move, for the same wall-clock of history.** + +That is why syncs "suddenly got slow", and it is the single fact that explains the incident's headline +symptoms. The strap browned out at **22:51:34** and the flag cleared. + +Current state of the tree: +- `noopWhoop5DeepData` is **still `true`** in prefs — but that is only the app-side *permission* + (`PuffinExperiment.deepDataKey`, `Strand/BLE/PuffinExperiment.swift:21`). +- **The probe code no longer exists in the tree.** `PuffinRawProbe` / `rawCensus` have no source hits. +- `enableWhoop5DeepData()` (`Strand/BLE/BLEManager.swift:2238`) is reachable from **exactly one place**: + a Settings button (`Strand/Screens/SettingsView.swift:1381`). It is a one-shot manual send — nothing + re-applies it on connect. It additionally requires whoop5 family + the Experimental toggle + + connected + encrypted bond + **worn**. +- `2189f255` reverted a force-enable hack that had made the capture log's `isEnabled` return `true` + unconditionally. + +**So the strap cannot re-enter this state by itself.** But the R22 flags live on the *strap*, not the +phone — so the blast radius of that one button is a persistent 28x tax that survives app restarts and is +invisible from the app. **See "Needs a decision" at the bottom.** + +**What this costs the rest of this document:** any measurement taken between 2026-07-14 15:36 and the +22:51 brownout was taken at 28x. Findings that *measured a duration* (A5's "12h+") are contaminated; +findings that read *code structure* are not. --- ## A. BLE / battery / connection -### A1. Battery % never lands on a 5.0/MG *(fix identified, ~1 line)* -The only 5/MG battery writer is standard `0x2A19` (`BLEManager.swift:3824-3830`, gated -`!= .whoop4`). Its one-shot `readValue` fires in `didDiscoverCharacteristicsFor` -(`:3467-3471`) — **pre-bond, on an unencrypted link**, which an MG rejects ("Authentication is -insufficient", see `:3476-3479`, `:3562-3567`). The failure hits the error branch (`:3806-3809`) -→ `log(...)` → `return`. **Never retried.** - -`:3586` calls `enableLiveNotifications(reason: "post-bond 5/MG")` with a comment claiming it recovers -"battery that failed pre-bond" — **it does not**; it only re-subscribes notify (`:2629-2654`), never -re-issues the read. HR recovers (notify pushes 1 Hz) so the UI says "Live"; battery never does. - -Proof it never landed: `LiveState.clearBiometrics()` (`:467-476`) clears `heartRate`/`rr`/ -`batterySamples` but **not** `batteryPct` — so `batteryPct == nil` proves `0x2A19` never delivered once. - -**Confirmed case (i):** the device log shows `Services discovered: 180D, 180A, 180F` — **180F IS -exposed**, so the characteristic exists and the read is simply never retried post-bond. - -**Fix:** re-issue the read for 5/MG at the end of `enableLiveNotifications` (idempotent; reuses the -3 existing call sites). **Fix 1b (more robust):** `FrameRouter.swift:202-205` already receives 5.0 -BATTERY_LEVEL events and reads `battery_charging` but **discards `battery_pct`** ("#77: flag only"). -The decode exists (`Interpreter.swift:742-753`, u16 @21, `/10`). The #77 stub-100 hazard is -**4.0-only**, so gating on family gives 5/MG a self-healing source refreshing ~every 8 min. - -### A2. "Active · Live" lies -`state.connected = true` is set in `didConnect` (`:3080`) **before** `discoverPrimaryServices` -(`:3113`) — true the instant the raw link exists, before any characteristic or byte. `DevicesView.swift:101,351` -shows "Active · Live" on `device.status == .active && live.connected`. Combined with iOS -state-restoration of a **dead** peripheral, the UI claimed "live" for a strap silent 21 hours. -**Fix:** gate the live indication on actual data flow / completed bond, not the raw link object. - -### A3. Charging can never display -`LiquidTodayView.swift:1529-1543`: the `bolt.fill` charging indicator is nested **inside** -`if let pct = live.batteryPct`. With pct nil the charging state is structurally unreachable — the -wearer was on the charger and the app could not say so. Also `:1546-1550`: `bolt.slash` is a -**no-data placeholder that reads as "battery dead"** (the wearer read it exactly that way). -**Fix:** render charging independent of pct; split "not connected" from "connected, no reading yet"; -accessibility label (`:1558`) should say "no reading yet". - -### A4. The app spins forever on a zombie restored peripheral -After iOS state-restoration returns a dead peripheral, the app marks it connected, the handshake never -completes, and it retries the **same dead handle** indefinitely — never cancelling, never re-scanning. -Observed: 40+ minutes of `Backfill: deferred — connect handshake not done yet` every 1–3 min, -persisting after the strap came off the charger. Only a **manual force-quit** recovered it. -**Fix:** bounded stalled-handshake timeout → cancel + re-scan. - -### A5. Backfill drain throttled by an arbitrary cap *(worst practical impact)* -`BLEManager.swift:320`: `static let defaultMaxAutoContinues = 6`. `shouldAutoContinue` (`:349`) -chains sessions back-to-back but every call site (`:1881`, `:1895`, `:1907`) caps at 6 → **7 sessions -per connection**, then falls back to `startBackfillTimer()` (`:2487`) → `backfillIntervalSeconds = 900` -(**15 min** between ~60s sessions). - -**Measured on device:** ~29 min of strap data recovered per session; ~18 h backlog ⇒ ~37 sessions ⇒ -7 free, then 30 × 15 min = **~7.5 hours** for data the strap streams in minutes. - -**The cap is redundant:** `shouldAutoContinue` already has `guard lastTrimAdvanced` (the real anti-spin -guard) plus future-clock checks. **Fix:** scale/bypass the cap while genuinely behind -(`strapNewestTs - frontier > behindGapSeconds` && `lastTrimAdvanced`), keeping a large runaway ceiling. -**Constraint:** each ~60s session blocks the WHOOP4 realtime keep-alive (`guard !backfilling`, see `:1509` -/ #160) — an aggressive drain must not starve live HR on a 4.0. - -**Workaround today:** `consecutiveCount` is **per connection** — force-quit + reopen resets it, buying -another ~7 sessions (~3.4 h of data) per cycle. - -### A6. Charging flag never lands -Newest live row: `ts 2026-07-15 10:58:06, soc=31.6, charging=0` **while actively charging**. Distinct -from A3 (there pct was nil; here pct exists and charging still reads 0). Unknown whether the 5/MG -emits the bit on this path at all. If it genuinely never reports it, the UI must **infer** charging -(rising SoC) or say nothing — never imply "not charging". - -### A7. `lastSeenAt` frozen at pairing → "last seen 18 days ago" +### A1. Battery % on a 5/MG — **headline REFUTED, narrower defect CONFIRMED · OPEN** + +**REFUTED: "Battery % never lands on a 5.0/MG."** The doc's own A6 evidence disproves it — a +`soc=31.6` reading exists for 2026-07-15. Two mechanisms the first pass missed: + +1. **0x2A19 is SUBSCRIBED, not just read.** Discovery calls `requestNotify` on it + (`BLEManager.swift:3502`), and `enableLiveNotifications` re-subscribes it post-bond (`:2664`). +2. **`didUpdateValueFor` (`:3852-3858`) handles read responses AND notifications identically** — both + land on `state.setBattery`. So the `:3614` `enableLiveNotifications(reason: "post-bond 5/MG")` call + **does** restore the battery source. Its comment is not the lie the first pass called it. + +**The proof was unsound.** `LiveState.batteryPct` is never set to nil *by anything* — `clearBiometrics()` +(`Strand/BLE/LiveState.swift:467-478`) deliberately leaves it. So `batteryPct == nil` proves 0x2A19 never +delivered **since this app process started** — not "never". + +**CONFIRMED, and this is the real defect (narrower):** the one-shot `readValue` +(`BLEManager.swift:3497-3499`) fires in `didDiscoverCharacteristicsFor` — **pre-bond**, which an MG +rejects — and **no code path ever re-issues a read**. `enableLiveNotifications` (`:2657-2669`) only +calls `requestNotify`. Standard Battery Service pushes **on change**, so after every connect the first +reading waits for the SoC to actually move. At the measured 1.65 %/h discharge that is **up to ~36 min +of `nil` per connect** — long enough to look like "never" to a wearer, and the origin of this finding. + +**Fix 1b CONFIRMED:** `FrameRouter.swift:202-205` receives 5.0 BATTERY_LEVEL events and reads +`battery_charging` while **discarding `battery_pct`** ("Flag only — battery % keeps its family-specific +source (#77)"). The decode exists. The #77 stub-100 hazard is **4.0-only**, so gating on family would +give a 5/MG a self-healing ~8-min source. + +**Not fixed here.** Both candidate fixes are on the BLE connect path, which per `CLAUDE.md` "must be +validated on a real strap; compile-success proves nothing" — and a live measurement is in progress. +**Needs the user.** + +### A2. "Active · Live" overstates a raw link — **CONFIRMED · OPEN** + +`state.connected = true` at `BLEManager.swift:3108` in `didConnect`, **before** +`discoverPrimaryServices` — true the instant the raw link object exists, before any characteristic or +byte. `DevicesView.swift:101` renders "Active · Live" on `device.status == .active && live.connected`. +Combined with iOS state-restoration of a **dead** peripheral, the UI claimed "live" for a strap silent +21 hours. + +The card already carries a narrower guard for the *bond-refused* case (`DevicesView.swift:102-108`, +#221) — which shows the shape of the fix — but nothing gates on data actually flowing. `LiveState` has +the ingredient: `lastFrameAtUnix` (`:203-207`, #987). + +**Not fixed here** — BLE/connect semantics, needs strap validation. **Needs the user.** + +### A3. Charging was structurally unrenderable without a charge % — **CONFIRMED · FIXED** + +`charging` and `batteryPct` come from **different sources and land independently**: `charging` rides the +strap's BATTERY_LEVEL event (~every 8 min, `FrameRouter.swift:202-205`), the % rides 0x2A19. The old +`LiquidBatteryButton` nested the `bolt.fill` **inside** `if let pct = live.batteryPct`, so +"charging, no % yet" could not render — it drew `bolt.slash`, which reads as *battery dead*, at a wearer +sitting on the charger. + +**This state is reachable, and reachable precisely during the incident:** BATTERY_LEVEL events are live +traffic and keep arriving mid-offload (only offload frames divert to the backfiller, +`BLEManager.swift:3864`; `FrameRouter` notes "this path never sees historical replay"). So the app held +a **fresh** charging bit and rendered a crossed-out bolt. + +> **This survives the "designed" refutation.** `guard !backfilling else { return }` +> (`BLEManager.swift:2482`) does sit above the battery poll, so *not polling mid-offload* is by design — +> and on a 5/MG that poll never runs anyway (`guard selectedModel.deviceFamily == .whoop4`, `:2503`). +> But "we deliberately don't have a %" does not license rendering a **known** charging state as its +> opposite. The absence is designed; the *claim* was wrong. + +**FIXED:** `LiquidTodayView.StrapBatteryDisplay.resolve(connected:batteryPct:charging:)` — a pure +3-case resolver (`.offline` / `.pending(charging:)` / `.charge(pct:charging:)`), pinned by +`StrandTests/LiquidBatteryDisplayTests.swift`. Charging now renders without a %; "connected but silent" +is now visually distinct from "no link"; the accessibility label no longer says a bare "Strap battery" +for three different states. + +### A4. Zombie restored peripheral spins forever — **NOT RE-VERIFIED · OPEN** + +First-pass claim: after iOS state-restoration returns a dead peripheral, the app marks it connected, the +handshake never completes, and it retries the **same dead handle** indefinitely — 40+ minutes of +`Backfill: deferred — connect handshake not done yet` every 1–3 min; only a force-quit recovered it. + +Re-verified only that the log line exists on both platforms (`BLEManager.swift:1562`, +`WhoopBleClient.kt:4893`). The restoration/retry behaviour itself was **not** re-traced this pass, and +it is not strap-testable right now. Treat as unconfirmed. **Needs the user.** + +### A5. `maxAutoContinues = 6` throttled a healthy recovery — **REFUTED · DECIDED: the cap stays at 6** + +**This finding was the sweep's "worst practical impact". It does not survive §0.** It was briefly +"fixed" (6 → 120, `2b968d27`) and then deliberately **reverted** (`d2ec5dc3`, cap back to 6) once the +root cause was known. The reasoning is now recorded at the constant on both platforms so it is not +"fixed" a third time: + +- **The severing evidence came from a strap the probe had switched into R22 deep-data banking** — + 3508 B/s of history vs the default **124 B/s**. A default strap banks **28x less**, so one session + covers ~28x more history and an 18 h backlog drains in **~2 sessions**. The cap never fires. +- **Raising it widens the blast radius of the next guard-2 bug.** #928 and #1012 both burned the *whole + cap* in EMPTY offloads — the cap is what bounded them. +- **It applies to WHOOP 4.0 too**, where a long drain starves the realtime keep-alive (`guard + !backfilling`, #160). + +The "~12h+ to recover a night" that made this finding look severe was measured **at 28x** (§0). Remove +the 28x and the arithmetic that condemned the cap evaporates. + +**FIXED (the half that was real):** the idle watchdog 60 → **15 s** on both platforms +(`BLEManager.swift:1676`, `WhoopBleClient.kt:565`, `d2ec5dc3`). Every session paid 60 s in full as dead +air on exit, and 60 s was set without measuring the inter-frame tail — a 3,018-buffer trace from a real +drain puts gaps at p50 64 ms / p90 106 ms / p99 273 ms / **max 2,520 ms**. 15 s is ~6x the measured max +and still clears the type-43 flood. (Not the ~5 s the tail alone would justify: one drain, one firmware, +and the risk is asymmetric — firing early is safe but wasteful, firing late only costs idle seconds.) + +**REFUTED, do not re-derive:** +- **There is no "~25.4 KB/s airtime ceiling"** and no "~7x real-time". That was a one-window fit; it did + not survive re-measurement across a dozen windows. The absolute link rate is not a constant of this + system. Both figures are gone from the tree. +- **The "~1.19x throughput" (and the "~3.7x → ~1.0x collapse") was never a bug.** The strap was **CAUGHT + UP**. History cannot drain faster than the rate at which it is banked — with no backlog, **~1.0x is + the floor**, not a symptom. + +Drain time is `(bytes banked per second-of-history) / (bytes the link moves per second)` and **both +terms move**. Any single headline multiplier is a property of one capture. + +**REFUTED, do not re-derive:** +- **There is no "~25.4 KB/s airtime ceiling"** and no "~7x real-time". That was a one-window fit; it did + not survive re-measurement across a dozen windows. The absolute link rate is not a constant of this + system. Both figures have been removed from the cap rationale on both platforms and from + `PuffinDeepBufferLog.swift`. +- **The "~1.19x throughput" was never a bug.** The strap was **CAUGHT UP**. You cannot drain history + faster than it is generated — with no backlog, **~1.0x is the floor**, not a symptom. + +Drain time is `(bytes banked per second-of-history) / (bytes the link moves per second)` and **both +terms move**. Any single headline multiplier is a property of one capture. + +### A6. "Charging flag never lands" — **REFUTED as stated · OPEN (reframed)** + +The finding's own evidence contradicts its title. The row was `soc=31.6, charging=0`: + +- **`charging=0` is not "never landed" — it is "landed and read false".** A flag that never landed is + **NULL**: `Streams.swift:374-380` sets `charging` only when the frame actually reported it + (`charging: Bool? // only the BATTERY_LEVEL event reports this; nil otherwise`). A non-NULL 0 proves + the decode ran *and* the strap's own event said not-charging. +- **It was not a "live row".** `battery` table rows are inserted from `streams.battery` + (`StreamStore.swift:132`) — the **decoded offload stream**. Its `ts` is the strap's RTC stamp on a + *banked* record, not wall-clock "now". So "charging=0 **while actively charging**" compares a + historical record's timestamp against an observation made at inspection time. The doc never + established the charger was on at strap-RTC 10:58:06. + +**OPEN (reframed):** whether the 5/MG's BATTERY_LEVEL bit0 tracks charging **at all** is genuinely +unknown and worth settling — but it needs a controlled capture (put it on the charger, watch +`state.charging` change), not a single historical row. A3's fix is what makes that flag visible enough +to observe. + +### A7. `lastSeenAt` frozen at pairing → "last seen 18 days ago" — **CONFIRMED · OPEN (both platforms)** + ``` my-whoop | added 2026-06-26 12:06:20 | lastSeen 2026-06-26 12:06:20 ← IDENTICAL oura-api | added 2026-07-10 | lastSeen 2026-07-13 ← updates fine ``` -The strap's registry `lastSeenAt` is written once at pairing and **never refreshed**. Only the WHOOP -source is affected. + +`lastSeenAt` is stamped **only** on wizard-add and on promote-to-active, never refreshed from a live +link: +- Swift — `Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift:40` + (`UPDATE pairedDevice SET status = 'active', lastSeenAt = ?`) and the upsert at `:156-163`. +- Kotlin — `android/app/src/main/java/com/noop/data/DeviceRegistryDao.kt:34`, same shape + ("Promote one device to `active` and stamp its lastSeenAt"). + +**Shared stored-data semantics ⇒ a fix needs the Kotlin twin in the same commit.** Cheap in principle +(stamp on a connect/frame event); left open because *where* to stamp is a connect-path decision. +**Needs the user.** --- ## B. Observability (the wearer's #1 ask) -### B1. Backfill progress is invisible -No way to see that a recovery is running, how far along, or how much remains. Direct quote: > "All of this stuff, like backfill progress, should be something you can see on the app. It is kind of > misleading me right now." -The data exists (`strapNewestTs`, frontier/trim cursor, `backfilling`). Surface e.g. -*"Recovering strap history — July 14 14:31, ~15h behind"*. Pure, testable display-state function. +### B1. Backfill progress invisible — **PARTLY REFUTED · iOS half FIXED · remainder OPEN** + +**REFUTED as "the app never surfaces it."** It does, on macOS and in the menu bar: +`SyncingHistoryNote` ("Syncing strap history… N chunks pulled", `Strand/Screens/ScreenScaffold.swift:199`) +is used by Sleep (`SleepView.swift:2621`), Intelligence (`IntelligenceView.swift:326`) and the menu bar +(`MenuBarContent.swift:269`); the classic `TodayView` has `SyncStatusChip` (`:4417-4430`). + +**CONFIRMED, precisely, for iOS — which is where the wearer was looking.** `RootTabView.swift:50` picks +`LiquidTodayView()` when the Liquid flag is on, and **`LiquidTodayView` had no backfill indication at +all**. Not a missing feature — a **v8 redesign regression**, the same class as #992, which dropped the +"~X days left" runtime estimate from the strap-battery row (as that row's own comment records, +`LiquidTodayView.swift:1593-1595`). + +**FIXED (iOS):** `LiquidSyncStatusRow` in the Data Sources card — "Strap history / Syncing… N chunks" +while `live.backfilling`, else "Synced N ago" from `live.lastSyncedAt`. + +**OPEN — the "how far behind" half.** *"Recovering strap history — July 14 14:31, ~15h behind"* still +isn't possible: it needs our persisted frontier (max HR ts) compared against +`live.strapRange?.newestUnix`. `strapRange` is in `LiveState` (`:160-176`); **the frontier is not** — it +is a Repository read. Plumbing it is a real change, not a display tweak, so it was not half-built here. + +### B2. Battery shown with no staleness marker — **CONFIRMED · FIXED (worst case)** + +A 21-hour-old reading rendered identically to a live one. Mechanism found: **`LiveState.batteryPct` is +never cleared** (§A1), so `LiquidBatteryButton` — which keyed off `batteryPct` alone with **no +`connected` gate** — kept rendering a dead strap's last charge as if current. The +`LiquidStrapBatteryRow` **directly below it** already required `live.connected` +(`LiquidTodayView.swift:1573`), so the two Liquid surfaces actively disagreed; the button was the wrong +one. + +**FIXED** by the same `StrapBatteryDisplay.resolve` gate (A3): no link ⇒ `.offline`, never a stale +number. Pinned by `testStaleChargeIsNotShownOnceTheLinkIsGone`. + +*Still open (softer):* a **connected** strap whose last reading is old has no age marker. "Last seen 21h +ago at 11%" needs a reading timestamp `LiveState` doesn't currently keep for battery. -### B2. Battery shown with no staleness marker -A 21-hour-old reading rendered identically to a live one. *"Last seen 21h ago at 11%"* would have -explained the entire incident instantly. +### B3. Rest carries but Charge/Effort don't — **CONFIRMED · Charge FIXED (Effort correctly excluded)** -### B3. Rest carries but Charge/Effort don't → a half-broken dashboard -On a dead-strap day the app shows a plausible Rest (carried from the last scored night) beside two -blanks, with **no signal that last night has no data**. Wearer repeatedly asked "are these stale?". -Verified: `2026-07-15` row is entirely NULL; the displayed Rest 72.4 / Charge 46.7 / Effort 25.2 are -**July 14's**, correctly computed. **Fix:** either carry consistently or say "no data for last night". +On a dead-strap / unscored-today day the Liquid Today (the DEFAULT screen) showed a plausible Rest — +carried from the last scored night — beside a blank Charge and a blank Effort, and the Charge state pill +still read "Calibrating" at a wearer 14 nights past the seed gate. Root cause: the Liquid heroes read +`displayDay?.recovery` / `.strain` **raw**, while the Rest hero carried (`freshRestScore`, #977) and the +vitals carried (`Repository.lastVitalsDay`) right beside them. Liquid was the ONLY surface that blanked — +classic `TodayView` carries via `lastScoredCharge` (#543), the widget/watch/Live Activity via +`Repository.widgetAnchor` (#911), and Android via `TodayScreen.lastScoredRecoveryDay`. + +**FIXED (Charge):** `LiquidTodayView.ChargeDisplay.resolve` now carries the last scored night's REAL +Charge, labelled "Last night · " (or "Latest sleep · " once #779-stale), stamped by the SAME +`TodayView.lastScoredRecoveryDay` + `carriedCaption` the classic screen uses, so the two Today screens +cannot drift. The state pill now tells the truth — only a genuinely calibrating baseline reads +"Calibrating"; a trusted baseline with no scored night reads honest "No data". The Key-Metrics Recovery +tile reads the same resolved value so the tile and hero can't disagree. Pinned by `LiquidChargeCarryTests` +(9 cases, pure resolver). Verified end-to-end in the simulator on the real on-device DB with today's row +nulled to stage the rollover: before = Charge "–" / pill "Calibrating"; after = Charge 47 carried from +Jul 14 / pill "Last night". + +**Effort deliberately does NOT carry** — Effort is *today's* accumulation, so painting yesterday's number +onto today would be a false statement, not a stale one (the same reason a blind `COALESCE` on the stored +`strain` column would be wrong). It stays blank until tonight scores. + +**Relationship to F1:** this carry SOFTENS F1's user-visible blank-out (the last good Charge shows instead +of "–"), but does NOT fix F1's root cause — the day's OWN row is still nil because the stale HRV baseline +gated it. F1's scoring-side change remains deferred to the user (see F1). + +### B4. Battery invisible over MCP — **NOT-IN-THIS-REPO** -### B4. Battery is invisible over MCP *(tool built, not deployed)* The `battery` table (soc/mv/charging) and the `event` table (**11,457 rows**, `BATTERY_LEVEL` carrying `battery_pct`) were never exposed. `battery_series`, `device_events`, `imu_coverage` are built + tested -in noop-cloud (branch `tz-upload`), **pending deploy**. +in noop-cloud (branch `tz-upload`), pending deploy. Tracked there, not here. + +### B5. Burst-capture census + firmware are not in the DB — **PARTLY STALE · OPEN (firmware half)** -### B5. Burst-capture census + firmware are not in the DB at all -`PuffinRawProbe.finishSession()` writes the R22 census JSON only to a file + -`UserDefaults(noop.probe.rawCensus)`; firmware lives in `UserDefaults(noop.lastFirmware)`, and neither -`device` nor `pairedDevice` has a firmware column. `ingest.ts` drops `settings.json` entirely. -⇒ **The MCP can never see R22 capture health or strap firmware** without a producer-side migration -(e.g. `probeRun(deviceId, ts, version, fw, capturedFrames, censusJSON)`). +**Stale:** the census half described `PuffinRawProbe.finishSession()` writing R22 census JSON to a file + +`UserDefaults(noop.probe.rawCensus)`. **That class no longer exists in the tree** (§0) — there is no +census producer left to migrate. + +**CONFIRMED (firmware half):** firmware lives only in `UserDefaults("noop.lastFirmware")` +(`FrameRouter.swift:99`), and neither `device` nor `pairedDevice` has a firmware column. `ingest.ts` +dropping `settings.json` is a noop-cloud fact. ⇒ strap firmware is invisible to the MCP without a +producer-side migration. --- -## C. Alerts +## C. Alerts — all three CONFIRMED, all OPEN, Kotlin twins match exactly + +### C1. Battery alerts fire once, then latch — no escalation — **CONFIRMED · OPEN** -### C1. Battery alerts fire once, then latch — no escalation Device state: `batteryLowAlerted = true`, `batteryRuntimeAlerted = true` — **both fired, and the strap -still died.** `BatteryAlertPolicy` fires low once per discharge cycle, re-arming only above -`lowRearmAbove = 25`. The runtime alert fires at `runtimeAlertHours = 24` — at the measured 1.65 %/h -that is **39.6% SoC, ~18 h early** — then latches. Net: **~3 hours of silence from 15% to death.** -Both toggles default ON (`BehaviorStore.swift:100-101`), so that is not the explanation. -*Being fixed:* critical 12% alert + bedtime night-guard, each with independent gates. +still died.** `BatteryAlertPolicy` is nested in `Strand/System/BatteryNotifier.swift:19` (not its own +file): `lowThreshold = 15` (`:20`), `lowRearmAbove = 25` (`:21`). The latch is real (`:45-51`) — +re-arm is gated **only** on SoC recovering to 25%, so once low fires at ≤15% nothing re-fires before a +charge. The runtime alert fires at `runtimeAlertHours = 24` +(`Packages/StrandAnalytics/Sources/StrandAnalytics/BatteryEstimator.swift:255`) — at the measured +1.65 %/h that is **39.6% SoC, ~18 h early** — then latches the same way (`:271-275`). Net: **~3 hours of +silence from 15% to death.** Both toggles default ON (`Strand/Data/BehaviorStore.swift:100-101`), so +that is not the explanation. + +**Correction to the first pass:** it recorded "*Being fixed:* critical 12% alert + bedtime night-guard". +**Neither exists anywhere in the tree, on either platform.** There are exactly three battery +notification ids (`battery-low`, `battery-full`, `battery-runtime`) and three persisted gates. That work +was aspirational, not landed. + +### C2. Runtime estimate ignores the strap's cutoff — **CONFIRMED · OPEN** + +`BatteryEstimator.swift:99`: `let remaining = max(0, current) / rate` — time to **0%**. The strap appears +to stop near ~10%, so every estimate overstates runway. **No cutoff constant exists on either platform.** +The `1.5x` rated-life clamp at `:104` is a runaway ceiling, not a cutoff — it does not subtract the +unusable tail. *Caveat: the ~10% cutoff is an incident observation, not a documented constant — the +value a fix should use is not derivable from this repo.* + +### C3. Rated-path predictive alert is unreachable on a 5.0 — **CONFIRMED · OPEN** (pointer drifted) + +`ratedLifeHoursWhoop5 = 288` now lives at `BatteryEstimator.swift:23`; the first pass pointed at the +*selection* site, now `BLEManager.swift:1021-1022`. On the rated path `rate = 100/288 = 0.3472 %/h`, so +`remaining = current × 2.88` and the 24 h line falls at **8.33% SoC — below the cutoff**. The `:104` +clamp is never binding here, so it doesn't rescue it. A 5.0 falling back to the rated path gets **no +predictive alert, ever**. (A 4.0 at `ratedHours = 108` crosses at 22.2%, comfortably reachable — this is +5.0-specific.) + +**Parity note (all of C):** the Kotlin twins exist and **every constant matches** — +`com/noop/notif/BatteryAlertNotifier.kt:23`, `com/noop/analytics/BatteryEstimator.kt`, +`com/noop/ble/WhoopConnectionService.kt:262-263`. C2 is mirrored at `BatteryEstimator.kt:94`, C3 via +`ratedLifeHoursWhoop5 = 288.0`. **The bugs are portable, not absent** — any fix lands on both platforms +in the same PR. -### C2. Runtime estimate ignores the strap's cutoff -`BatteryEstimator.swift:99`: `remaining = max(0, current) / rate` — time to **0%**. The strap appears -to stop near ~10%, so every estimate overstates runway. +--- -### C3. Rated-path predictive alert is unreachable on a 5.0 -`ratedHours = 288` (`BLEManager.swift:1005`) puts the 24 h line at **8.33% SoC — below the cutoff**. -A 5.0 falling back to the rated path therefore gets **no predictive alert, ever**. +## D. Cloud sync (fork-local; compile-gated behind `CLOUD_SYNC`, absent from upstream and default builds) ---- +### D1. Upload can never complete from the background — **CONFIRMED · OPEN** ("fix built" was wrong) + +`CloudSyncClient.swift:34` still uses `URLSession.shared`. **Zero** `beginBackgroundTask`, **zero** +background `URLSessionConfiguration` in app code. A push grants ~30 s; deflate-zipping ~400 MB +(→ ~81.5 MB) blows past it and in-flight tasks die on suspension. -## D. Cloud sync +**Correction: the "*(fix built)*" tag was wrong and is removed.** The working tree's new +`DeepBufferUploadPlan`/`DeepBufferUploader` add **chunking + a resume watermark** (16 MB chunks, 8 per +run, watermark persisted per chunk) to a **new `/deepbuf` lane** — that bounds how much a suspension +destroys, but the chunks still ride `URLSession.shared` and still die on suspension. **The `/ingest` +whole-DB lane D1 actually indicts is untouched** (`CloudSyncClient.swift:78-97` still streams the whole +file in one shot). -### D1. Upload can never complete from the background *(fix built)* -`CloudSyncClient.swift:34` uses `URLSession.shared` (foreground). Zero `beginBackgroundTask`, zero -background `URLSessionConfiguration`. A push grants ~30 s; deflate-zipping ~400 MB (→ ~81.5 MB) blows -past it and in-flight tasks die on suspension. Device proof: `cloudsync.lastStatus` frozen at -`"Uploaded 81.5 MB · 12:21 PM"` yesterday — and it is written on **every** completion, success or -failure ⇒ nothing has *completed* since. +`cloudsync.lastStatus` is written on **every** completion, success or failure — +`Self.persistLastStatus(line)` (`CloudSyncModel.swift:260`) sits *outside* the `if succeeded` block +(`:257-259`). It is only reached if `performSync` runs to completion, which is exactly why the device +showed it **frozen** at `"Uploaded 81.5 MB · 12:21 PM"` rather than showing a failure — the two halves of +this finding corroborate each other. -### D2. Warm resume triggers no sync -`autoSyncIfDue`'s only iOS call site is `RootTabView.swift:133` inside `.task {}`, which does not -re-run on resume. `scenePhase == .active` (`StrandiOSApp.swift:205-223`) does not call CloudSync. +### D2. Warm resume triggers no sync — **CONFIRMED · OPEN** -### D3. Server volume ~1 upload from ENOSPC -1008 MiB total, ~461 used, **~547 free**; an ingest needs ~400 MB staged **alongside** the ~460 MB -mirror. `MAX_INGEST_BYTES = 768MB` exceeds the entire volume. Also `/ingest` logs nothing on receipt, -success, or the 400 path — it cannot distinguish "nothing arrived" from "arrived and was rejected". +`autoSyncIfDue` has exactly two call sites: `StrandiOS/App/RootTabView.swift:133` (iOS, inside `.task {}`, +which does not re-run on resume) and `Strand/App/RootView.swift:324` (macOS only — excluded from the iOS +target). `scenePhase == .active` (`StrandiOSApp.swift:205-223`) does not call CloudSync. + +*Two sharpenings:* `scenePhase` **does** reach CloudSync on the way **out** — +`CloudSyncBackgroundRefresh.schedule()` in the `.background` branch (`:233-238`) — and +`backgroundSyncIfDue` (`CloudSyncModel.swift:404`, 4 h gate) serves the BGAppRefreshTask/silent-push +lane. The gap is specifically **warm resume**, not background sync in general. + +### D3. Server volume ~1 upload from ENOSPC — **NOT-IN-THIS-REPO** + +1008 MiB total, ~461 used, ~547 free; an ingest needs ~400 MB staged alongside the ~460 MB mirror. +`MAX_INGEST_BYTES = 768MB` exceeds the entire volume. `/ingest` logs nothing on receipt, success, or the +400 path. **None of this exists in this repo** — no `ingest.ts`, no server sources. Belongs to the +`noop-cloud` checkout. --- ## E. Deep data / MG -### E1. 2140 B optical frames are rejected — no home on this build -Live log: `Backfill: rejected frame[N] 2140B` (repeating). The live DB has **no `ppgWaveformSample` -table** (migrations top out at v28; upstream's `v27-ppg-waveform` / #415 is newer than the installed -build). ⇒ Every 2140 B optical buffer — **the SpO2/BP substrate** — is dropped on the floor. -**Action:** rebuild + install current fork main (which merged v27) to start storing them. +### E1. 2140 B optical frames have no home — **diagnosis REFUTED · the real gap is CONFIRMED** + +**REFUTED — the stated action does not achieve the stated goal.** The claim was that the DB lacks +`ppgWaveformSample` because the installed build predated upstream's `v27-ppg-waveform` (#415), so +"rebuild + install current fork main to start storing them". + +The migration **is in this tree** (`Packages/WhoopStore/Sources/WhoopStore/Database.swift:591`), highest +is `v29-daily-avg-sdnn` (`:619`), and the Room twin is at parity (`WhoopDatabase.kt:550`, pinned by +`PpgWaveformMigrationTest.kt`). The "top out at v28" reading was internally consistent — the fork's local +`v28-phone-timezone` predated the merge, and GRDB keys on the identifier *string*, so the out-of-order +insert applies cleanly. + +**But it stores the wrong thing.** `ppgWaveformSample` is fed **only** from `p["ppg_waveform"]` +(`HistoricalStreams.swift:214-217`), emitted **only** by `decodeWhoop5HistoricalV26` — **hist_version 26** +records. The 2140 B buffers are **v20** (`Interpreter.swift:587`), decoded into `channel_b{n}_{half}` +fields that **nothing consumes**. The commit seals it: `ee1de9a7 … persist the WHOOP 5.0 **v26** raw PPG +waveform`. + +**CONFIRMED (the real gap):** the 2140 B v20 optical buffers — the SpO2/BP substrate — still have **no DB +table in this tree**. Rebuilding gives *v26 PPG* a home; it does nothing for v20. **Needs the user** (a +real feature, not a rebuild). + +**Their only durable path today is the reject archive** (`rejected_history.jsonl`), which +`BLEManager.swift:925-930` replays and **retro-decodes on version bump** — so the bytes survive until a +v20 decoder lands. That is exactly why exempting v20/v21 from that archive destroyed them (§E2), and why +the archive must not be "optimised" away before v20 has a table. The `puffin-deepbuffers.jsonl` research +log is **not** a substitute: it is default-OFF and gated on a different key. + +### E2. 1244 B frames logged as "rejected" — **REFUTED (nothing was wrong) · the "fix" for it was harmful and is REVERTED** + +**"Rejected" never meant "dropped".** `rejectedHistoricalRecords` (`HistoricalStreams.swift:66`) is an +independent *classification* pass, not a decode gate: decoded rows commit separately +(`Backfiller.swift:554`), and the rejected set only selects frames whose **raw bytes get archived** to +`rejected_history.jsonl` for later RE (`:580-586`) so the ack can't free the strap's only copy of an +unmapped layout. A frame there means "no consumer claimed it — **keep the bytes**", not "data lost". + +Why 1244 B appeared: the predicate (`:99-100`) logs anything with no `unix`, or with neither +`heart_rate` nor `gravity_x`. v21 (1244 B) carries `optical_ch0/1/2` and neither of those → logged. No +contradiction with working IMU decode — that runs through `Whoop5RawImu.decode` in the deep-buffer path, +a **different consumer**. Both facts were always true. **"Needs investigation" was the right instinct; +the frames were fine.** + +> **Cautionary tale — this finding caused a data-loss bug.** Acting on it, `7942f667` exempted v20/v21 +> from the reject archive ("they decode, for the same reason v26 is exempted one line above"). That is +> backwards, and `86d414ad` reverted it. **v26 is safe to exempt because it HAS a durable home** +> (`Streams.ppgWaveform` → `ppgWaveformSample`), so the archive is redundant for it. **v20/v21 have no +> home** (§E1) — `decodeWhoop5HistoricalV2021` writes `optical_ch0..2` / `channel_b*` into an in-memory +> frame with **zero production consumers on either platform**, and the frame is discarded. With the +> exemption: not archived, not persisted, **trim acked — and the strap frees the buffers permanently**. +> `PuffinDeepBufferLog` does not catch them either (default-OFF, and gated on +> `PuffinFrameRecorder.enabledKey`, a *different* key from the deep-data experiment). +> Two things that revert surfaced: `RawHistoryArchive.perVersionFloor` (`:27-32`) names "WHOOP 5 v20/v21" +> as the exact rare-layout case its eviction floor was built for (#344) — the exemption deleted precisely +> what a prior fix exists to preserve — and `BLEManager.swift:925-930` replays the archive and +> **retro-decodes it on version bump**, so for v20 (micro-decode unsolved) that archive is the ONLY path +> by which the data is ever recovered once a decoder lands. +> The exemption's throughput rationale was also the refuted "~3.7x → ~1.0x collapse" (§A5) — the strap +> was caught up. + +**Current state: reverted; the archive preserves v20/v21 again.** The real problems it gestured at are +real but live elsewhere: the false "N records couldn't be decoded" banner should stop counting known +layouts (`BLEManager.swift:1801-1803`), and the archive's at-cap read-all + hex-decode + 5 MB atomic +rewrite per chunk should be fixed **in** the archive. **Both OPEN.** + +### E3. Deep IMU capture is barely running — **mechanism CONFIRMED · evidence CORRECTED** + +**Mechanism CONFIRMED, and §0 explains the 58-minute window.** Banking needs **two** default-off gates +plus a manual tap: the Experimental toggle (`PuffinExperiment.deepDataEnabled`) **and** the capture +toggle (`PuffinFrameRecorder.enabledKey`), plus a one-shot Settings button press +(`SettingsView.swift:1381`) while connected + encrypted-bonded + **worn**. Nothing re-applies R22 on +connect. "One 58-minute window across 8 months" is precisely the shape that produces. + +**Evidence corrected:** there is **no `imuActivity` table anywhere in this tree** — not in +`Database.swift`, not in Room. `ImuActivityFeatures` is serialized only as the `"imu"` field inside +`puffin-deepbuffers.jsonl`, and **only for live frames, never offload** +(`PuffinDeepBufferLog.swift`: `let imu = isOffload ? "" : Self.decodedImuField(frame)`). So "2,945 rows" +is a cloud-side/derived artifact. **Reconcile where those rows come from before acting on E3.** -### E2. 1244 B (IMU) frames also logged as rejected — unexplained -IMU decode demonstrably works (`imuActivity` = 2,945 rows), yet 1244 B frames appear in the reject log -during this offload. Needs investigation; do not guess. +--- -### E3. Deep IMU capture is barely running -`imuActivity`: **2,945 rows — all inside ONE 58-minute window across 8 months.** Whatever gates the -deep-buffer capture is almost never on. +## F. Scoring + +### F1. Charge silently vanishes after 14 stale nights — **CONFIRMED · comment FIXED · code change needs the user** + +All four first-pass pointers were **exact**. `Baselines.swift:40` documented `.stale` as "usable but no +update for > STALE_DAYS nights", while `:69` defines `usable = (status == .provisional || status == +.trusted)` — **`.stale` excluded** — with `staleDays = 14` (`:102`) and `.stale` assigned at `:223`. +`RecoveryScorer.swift:417` then does `if !hrvBaselineUsable { return nil }`. ⇒ 14 nights without a valid +HRV night ⇒ **Charge silently returns nil**, no explanation surfaced. + +**Resolved: the COMMENT was wrong, not the code.** Excluding `.stale` from `usable` is deliberate and +coherent across **every** consumer (`AnalyticsEngine`, `DailyStressEngine`, `ReadinessEngine`, +`ScoreConfidence`, `WatchRecovery`, `DaytimeBaselines`, `AppModel`, `IntelligenceEngine`): `usable` means +"trust this personal baseline", and one nobody has confirmed in >14 nights is exactly one not to trust. +The comment's word "usable" simply collided with the property name. Current behaviour is **pinned by +tests on both platforms** (`VitalBandsTests.testStaleBaselineFallsBackToPopulation`, +`VitalBandsTest.staleBaseline_fallsBackToPopulation`). + +**FIXED (comment only, both platforms):** `Baselines.swift` + `AnalyticsModels.kt:179-180` — and the +sibling `usable` doc, which claimed "at least provisionally usable (nValid ≥ MIN_NIGHTS_SEED)"; `.stale` +also clears that bar and is still excluded. **Changes no numbers.** + +**The silent-nil half is still OPEN and is a product decision.** Making `.stale` usable would flip +`VitalBands.basis` to personal (breaking pinned tests on both platforms), un-gate `RecoveryScorer`, and +move skinTempDev / Readiness / ScoreConfidence / illness signals — a **stored-value + parity change**. +The narrower fix — *surface* "no Charge: baseline stale" instead of a blank — changes no numbers. +**Needs the user.** + +**Escalation the first pass under-stated.** `baselines2.hrv` is folded to a single **end-of-history** +snapshot (`IntelligenceEngine.swift:875`) and that one state gates **every** day re-scored in pass 2. +Dead nights enter the fold as `nil` (`:843`; B3 verified an all-NULL row exists), so 15+ trailing +nil-HRV rows ⇒ `.stale` ⇒ `!usable` ⇒ `recovery` returns nil **for the whole re-scored window**, not just +the current night. It self-heals the instant one valid night lands. **Unanswered — answer this first:** +do those nils get written back over previously-stored non-nil `recovery` values? That is the difference +between "Charge is blank today" and "**Charge history got overwritten**". --- -## F. Scoring (latent) +## Needs a decision + +1. **§0 — the R22 button.** Deep-buffer banking is a **persistent 28x tax written to the strap**, it is + never re-applied *or cleared* on connect, and the app cannot see it. The probe that tripped it is + gone, but the Settings button remains. Options: leave it (documented), have the app clear R22 on + connect unless deep data is deliberately on, or surface "deep banking is ON" in the UI. Also: + `noopWhoop5DeepData` is **still `true`** on the device — decide whether to clear it. +2. **A1 / A2 / A4 / A7** — all BLE/connect-path or shared-registry changes. Each needs strap validation + (`CLAUDE.md`: compile-success proves nothing here) and A7 needs a Kotlin twin. Not touched while a + live measurement is running. +3. **C1 / C2 / C3** — the alert policy is a product judgement (what SoC, what escalation, what cutoff), + and the ~10% cutoff isn't a known constant. All three need Kotlin twins. +4. **F1** — surface the stale-baseline reason (no numbers move) vs. make `.stale` usable (numbers move, + pinned tests break, needs a twin). And answer the overwrite question first. +5. **E1** — giving the 2140 B v20 optical buffers a home is a migration + decoder feature, not a rebuild. + Until then the reject archive is their only durable path — do not exempt or evict them (§E2). +6. **E2 fallout, both OPEN** — the false "N records couldn't be decoded" banner should stop counting + known layouts (`BLEManager.swift:1801-1803`), and the reject archive's at-cap read-all + hex-decode + + 5 MB atomic rewrite per chunk should be fixed **in** the archive. These are the real costs the + destructive v20/v21 exemption was reaching for. + +--- + +## What this sweep should teach the next one + +Three of the sweep's most confident findings — A5 ("worst practical impact"), A6, E2 — were artifacts, +and one of them (E2) produced a **data-destroying "fix"** before the root cause was known. The pattern is +identical each time: **a measurement taken in an unknown state was read as a property of the code.** + +- **A5 / §0** — 28x more bytes per second-of-history was read as a duty-cycle bug. +- **A6** — a historical banked record's strap-RTC timestamp was read as "now". +- **E2** — a preservation mechanism ("keep these bytes") was read as a failure ("rejected"). +- **A1** — a `nil` that means "not since process start" was read as "never". -### F1. Charge silently vanishes after 14 stale nights -`Baselines.swift:40` documents `case stale // usable but no update for > STALE_DAYS nights`, but -`:69` defines `usable = (status == .provisional || status == .trusted)` — **`.stale` excluded** — and -`staleDays = 14`. `RecoveryScorer.swift:417` then does `if !hrvBaselineUsable { return nil }`. -⇒ 14 nights without a valid HRV night ⇒ **Charge silently returns nil**, no explanation surfaced. -The comment and the code disagree; one of them is wrong. +Find the state before optimising against the symptom. And when a finding says *"unexplained — needs +investigation; do not guess"* (E2 said exactly that), that instruction outranks a plausible-looking fix. From 910adcc22013a005f949ae50398ef880a54a4c21 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 16 Jul 2026 12:35:13 -0700 Subject: [PATCH 121/171] fix(sleep): Rest hero tracks the navigated night, not the latest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Sleep tab's Rest performance hero read the full-history latest sleep-performance value (`performance.latest`) while the hypnogram card below it browses nights via `nightOffset`. Tapping ◀/▶ changed the hypnogram but left the big score, vessel fill, state word, provenance badge and "Last night" overline frozen on the most recent night — so every night appeared to score the same. Resolve the score for the DISPLAYED night by its wake-day through the same source of truth the series already uses (imported WHOOP figure, else the Rest composite for that day), so the hero matches the night the hypnogram shows. The overline and the nav header now share one night-relative label. Mirror the fix in the Android twin (SleepScreen.kt) per the parity contract. Presentation-only; no analytics or stored-data change. Compiled: macOS Strand + Android compileFullDebugKotlin. --- Strand/Screens/SleepView.swift | 66 ++++++++++++++----- .../src/main/java/com/noop/ui/SleepScreen.kt | 59 ++++++++++++----- 2 files changed, 92 insertions(+), 33 deletions(-) diff --git a/Strand/Screens/SleepView.swift b/Strand/Screens/SleepView.swift index a01e6f2189..9f12eb318d 100644 --- a/Strand/Screens/SleepView.swift +++ b/Strand/Screens/SleepView.swift @@ -328,11 +328,40 @@ struct SleepView: View { .accessibilityLabel(message) } - /// The fill fraction (0…1) the Rest hero gauge animates to — the night's sleep-performance - /// score over 100. 0 when no score exists (the headline-hours hero shows instead). Cheap, so - /// it's read every render to drive the draw-in animation. + /// A short night-relative label ("Last night" / "1 night ago" / "N nights ago") for the + /// ◀/▶-navigated night. Shared by the Rest hero overline and the hypnogram nav header so both + /// name the SAME night the hero's score is now resolved for. + private var nightRelativeLabel: LocalizedStringKey { + nightOffset == 0 ? "Last night" + : (nightOffset == 1 ? "1 night ago" : "\(nightOffset) nights ago") + } + + /// The night the Rest hero reflects: the ◀/▶-navigated night while browsing (falling back to + /// last night only if that navigated night hasn't decoded yet), else last night. Keeps the + /// hero's score, vessel fill, state word, provenance badge and overline on the SAME night the + /// hypnogram shows — the fix for the score freezing on last night's value during navigation. + private func heroNight(_ model: SleepModel) -> Night { + (nightOffset == 0 ? model.night : navNight) ?? model.night + } + + /// The sleep-performance score (0–100) for a SPECIFIC night: the imported WHOOP figure for that + /// night's LOCAL wake-day when the export carried one, else the resolved Rest composite for that + /// day. Mirrors `performanceSeries`'s per-day transform exactly (the same single source of truth + /// the Today Rest score reads), keyed by the wake-day (sleep is filed under the day you woke) so + /// a navigated past night reads ITS OWN score, never last night's. nil when that day has no score. + private func performanceScore(for night: Night) -> Double? { + let wakeDay = Repository.localDayKey(Date(timeIntervalSince1970: TimeInterval(night.session.endTs))) + if let p = repo.importedSleep[wakeDay]?.performancePct { return p } + guard let daily = repo.days.last(where: { $0.day == wakeDay }) else { return nil } + return AnalyticsEngine.Rest.composite(daily: daily) + } + + /// The fill fraction (0…1) the Rest hero gauge animates to — the DISPLAYED night's sleep- + /// performance score over 100. 0 when no score exists (the headline-hours hero shows instead). + /// Cheap, so it's read every render to drive the draw-in animation; keyed off the navigated + /// night so the vessel re-animates as you browse ◀/▶. private func heroScoreFraction(_ model: SleepModel?) -> Double { - guard let p = model?.performance.latest else { return 0 } + guard let model, let p = performanceScore(for: heroNight(model)) else { return 0 } return min(max(p / 100.0, 0), 1) } @@ -341,12 +370,14 @@ struct SleepView: View { /// counting up over it (the SAME hero language Today's score cells and the Trends headline use); /// otherwise a big SF-Rounded hours-slept headline over the same backdrop. A `SourceBadge` states /// whether the score is WHOOP's own imported figure or NOOP's on-device estimate. Presentation-only - /// — the number comes straight from the existing `model.performance.latest` / hours computation. + /// — the score is `performanceScore(for:)` on the ◀/▶-navigated `heroNight`, so the hero tracks the + /// same night the hypnogram shows (was pinned to `performance.latest` = last night regardless). @ViewBuilder private func restHero(_ model: SleepModel) -> some View { - let score = model.performance.latest + let night = heroNight(model) + let score = performanceScore(for: night) VStack(alignment: .leading, spacing: NoopMetrics.gap) { - SectionHeader("Sleep performance", overline: "Last night", trailing: String(localized: "Rest")) + SectionHeader("Sleep performance", overline: nightRelativeLabel, trailing: String(localized: "Rest")) // A subtle night atmosphere sits behind the sleep hero ONLY (the Rest world's whisper: // faint indigo wash + crescent moon over the near-black canvas, no glow), clipped to the // card. Replaces the now-flat ScenicHeroBackground here. @@ -386,7 +417,7 @@ struct SleepView: View { // whose minutes tick up on appear (the same count-up the scored hero gets). VStack(spacing: NoopMetrics.space1) { CountUpText( - value: model.night.stages.asleep, + value: night.stages.asleep, format: { durationText($0) }, font: StrandFont.number(46), color: StrandPalette.restBright @@ -398,7 +429,7 @@ struct SleepView: View { .padding(.vertical, NoopMetrics.space5) .accessibilityElement(children: .combine) } - SourceBadge(score != nil ? sleepScoreSource(model) : "On-device", tint: StrandPalette.restColor) + SourceBadge(score != nil ? heroSource(for: night) : "On-device", tint: StrandPalette.restColor) } .padding(NoopMetrics.cardInnerPadding + NoopMetrics.space1) .frame(maxWidth: .infinity) @@ -417,13 +448,13 @@ struct SleepView: View { } } - /// Whether the night's sleep-performance score is WHOOP's own imported figure or NOOP's - /// on-device approximation — so the hero is honest about provenance, like Today's badges. - private func sleepScoreSource(_ model: SleepModel) -> LocalizedStringKey { - if let lastDay = repo.days.last?.day, repo.importedSleep[lastDay]?.performancePct != nil { - return "Whoop" - } - return "On-device" + /// Whether a SPECIFIC night's sleep-performance score is WHOOP's own imported figure or NOOP's + /// on-device approximation — so the hero is honest about provenance, like Today's badges. Keyed + /// by the night's wake-day (matching `performanceScore(for:)`) so a navigated night's badge + /// tracks ITS OWN score's provenance, not last night's. LocalizedStringKey to match `SourceBadge`. + private func heroSource(for night: Night) -> LocalizedStringKey { + let wakeDay = Repository.localDayKey(Date(timeIntervalSince1970: TimeInterval(night.session.endTs))) + return repo.importedSleep[wakeDay]?.performancePct != nil ? "Whoop" : "On-device" } // MARK: - Provenance for the displayed night (COMPONENT 4, spec 2026-06-20) @@ -2114,8 +2145,7 @@ struct SleepView: View { @ViewBuilder private func nightNavHeader(trailing: String) -> some View { let lastIndex = max(navDays.count - 1, 0) - let title: LocalizedStringKey = nightOffset == 0 ? "Last night" - : (nightOffset == 1 ? "1 night ago" : "\(nightOffset) nights ago") + let title = nightRelativeLabel VStack(alignment: .leading, spacing: NoopMetrics.cardInnerSpacing) { HStack(spacing: NoopMetrics.cardInnerSpacing) { Button { if nightOffset < lastIndex { nightOffset += 1 } } label: { diff --git a/android/app/src/main/java/com/noop/ui/SleepScreen.kt b/android/app/src/main/java/com/noop/ui/SleepScreen.kt index 2b0cb3a3ee..ca53c4af56 100644 --- a/android/app/src/main/java/com/noop/ui/SleepScreen.kt +++ b/android/app/src/main/java/com/noop/ui/SleepScreen.kt @@ -420,9 +420,15 @@ fun SleepScreen( // the selected day's model failed to build (#940): real data over a zeroed gauge. item { RestHero( - score = (model ?: tilesModel)?.performance?.latest, + // The DISPLAYED night's score (keyed by its wake-day), so the hero tracks the + // ◀/▶-navigated night instead of freezing on the full-history latest. When no night + // resolves but history exists (#940), keep the old "real data over a zeroed gauge" + // fallback to the latest score. + score = if (night != null) heroPerformanceScore(night, days, imported) + else tilesModel?.performance?.latest, asleepMin = model?.stages?.asleep, - source = restHeroSource(imported, days), + source = restHeroSource(imported, night?.dayKey ?: days.lastOrNull()?.day), + overline = nightRelativeLabel(nightOffset), ) } item { Spacer(Modifier.height(Metrics.selectorTopUp)) } @@ -675,9 +681,9 @@ private val LIQUID_HERO_RADIUS: Dp = 26.dp // figures, fraction math and Rest tint are UNCHANGED from the BevelGauge this replaced — presentation-only. @Composable -private fun RestHero(score: Double?, asleepMin: Double?, source: String) { +private fun RestHero(score: Double?, asleepMin: Double?, source: String, overline: String) { Column(verticalArrangement = Arrangement.spacedBy(Metrics.gap)) { - SectionHeader("Sleep performance", overline = "Last night", trailing = "Rest") + SectionHeader("Sleep performance", overline = overline, trailing = "Rest") Box( modifier = Modifier .fillMaxWidth() @@ -770,15 +776,42 @@ private fun sleepScoreWord(score: Double): String = when { } /** - * Whether the night's sleep-performance score is WHOOP's own imported figure or NOOP's on-device - * approximation — so the hero is honest about provenance, like Today's badges. Mirrors the macOS - * SleepView.sleepScoreSource. + * Short night-relative label ("Last night" / "1 night ago" / "N nights ago") for the ◀/▶-navigated + * night. Shared by the Rest hero overline and the hypnogram nav header so both name the SAME night + * the hero's score is resolved for. Mirrors iOS SleepView.nightRelativeLabel. */ -private fun restHeroSource(imported: ImportedSleepSeries, days: List): String { - val lastDay = days.lastOrNull()?.day - return if (lastDay != null && imported.performance[lastDay] != null) "Whoop" else "On-device" +internal fun nightRelativeLabel(offset: Int): String = when (offset) { + 0 -> "Last night" + 1 -> "1 night ago" + else -> "$offset nights ago" } +/** + * The sleep-performance score (0–100) for a SPECIFIC navigated night: the imported WHOOP figure for + * that night's wake-day when the export carried one, else the resolved Rest composite for that day. + * Mirrors the per-day transform in [buildSleepModel]'s `performance` series (and iOS + * SleepView.performanceScore), keyed by [HeroNight.dayKey], so a navigated past night reads ITS OWN + * score rather than the full-history latest. Null when there is no navigated night or that day has + * no score. + */ +private fun heroPerformanceScore( + night: HeroNight?, days: List, imported: ImportedSleepSeries, +): Double? { + val wakeDay = night?.dayKey ?: return null + imported.performance[wakeDay]?.let { return it } + val daily = days.lastOrNull { it.day == wakeDay } ?: return null + return com.noop.analytics.RestScorer.restFromDaily(daily) +} + +/** + * Whether a SPECIFIC night's sleep-performance score is WHOOP's own imported figure or NOOP's + * on-device approximation — so the hero is honest about provenance, like Today's badges. Keyed by the + * night's wake-day (matching [heroPerformanceScore]) so a navigated night's badge tracks ITS OWN + * score's provenance, not last night's. Mirrors the macOS SleepView.heroSource. + */ +private fun restHeroSource(imported: ImportedSleepSeries, wakeDay: String?): String = + if (wakeDay != null && imported.performance[wakeDay] != null) "Whoop" else "On-device" + // MARK: - 1. HERO — stage breakdown for the navigated night @OptIn(ExperimentalMaterial3Api::class) @@ -2002,11 +2035,7 @@ private fun NightNavHeader( ) } - val nightLabel = when (offset) { - 0 -> "Last night" - 1 -> "1 night ago" - else -> "$offset nights ago" - } + val nightLabel = nightRelativeLabel(offset) val blockShape = RoundedCornerShape(Metrics.cornerSm) val clockParts = clock?.split(" · ", limit = 2) val dateLabel = clockParts?.getOrNull(0) From 7770324a8621b665ccd2dab7748819358a8aab96 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 16 Jul 2026 13:07:48 -0700 Subject: [PATCH 122/171] Liquid Today: surface the "N of 4 nights" calibration count MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v8 Liquid Today (the default iOS Today) showed a bare "Calibrating" for a pre-seed-gate baseline, dropping the "N of 4 nights" progress the classic TodayView surfaces (calibrationDetail / ScoreStatePill). A brand-new user in their first 4 nights saw the progress on classic but not on Liquid. ChargeDisplay already carried the count (.calibrating(nights:), 9251ae24); this adds ChargeDisplay.calibrationDetail returning classic's exact copy ("Learning your baseline, N of 4 nights.") and swaps it into the synthesis card in place of the readiness one-liner (chargeDisplay.calibrationDetail ?? synthLine) — the same replacement classic makes (calibrationDetail ?? synthesisCardDetail). The compact greeting pill stays short ("Calibrating") per its fixedSize row, so the count lives in the card. Reuses classic's String Catalog key verbatim; both Today screens now read identically. New-users-only, no owner impact. Android TodayScreen already surfaces this, so no Kotlin twin is needed — iOS Liquid catching up to existing behaviour. Verified (app-target Swift, not covered by CI): LiquidChargeCarryTests (test-first, RED→GREEN, 11/11) on the resolver/label; macOS Strand + iOS NOOPiOS app targets compile; --demo-screen liquidtoday on a clean sim renders "Learning your baseline, 0 of 4 nights." in the synthesis card with the short "Calibrating" pill. Co-Authored-By: Claude Opus 4.8 --- Strand/Liquid/LiquidTodayView.swift | 19 +++++++++++++++++- StrandTests/LiquidChargeCarryTests.swift | 25 ++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/Strand/Liquid/LiquidTodayView.swift b/Strand/Liquid/LiquidTodayView.swift index 89baccb7b2..ca77a66a07 100644 --- a/Strand/Liquid/LiquidTodayView.swift +++ b/Strand/Liquid/LiquidTodayView.swift @@ -718,7 +718,12 @@ struct LiquidTodayView: View { Text(synthesisExpanded ? "hide" : "show").font(StrandFont.caption) .foregroundStyle(StrandPalette.textTertiary) } - Text(synthLine).font(StrandFont.body).foregroundStyle(StrandPalette.textPrimary) + // While the baseline calibrates, the honest "N of 4 nights" progress replaces the + // readiness one-liner here — the same swap classic makes (`calibrationDetail ?? + // synthesisCardDetail`), so the count the short greeting pill can't carry lands in + // the card and both Today screens read identically. + Text(chargeDisplay.calibrationDetail ?? synthLine) + .font(StrandFont.body).foregroundStyle(StrandPalette.textPrimary) .fixedSize(horizontal: false, vertical: true) if synthesisExpanded { Text(LocalizedStringKey(readiness.summary)).font(StrandFont.caption) @@ -1628,6 +1633,18 @@ extension LiquidTodayView { } } + /// The synthesis-card detail line while the baseline is still forming — the same "N of + /// `Baselines.minNightsSeed` nights" progress classic `TodayView.calibrationDetail` surfaces, so a + /// wearer in their first few nights reads identical calibration copy on both Today screens (before + /// this, Liquid dropped the count and showed a bare "Calibrating"). Non-nil ONLY for `.calibrating`: + /// the compact greeting pill stays short ("Calibrating") because it shares a `fixedSize` row with + /// the greeting, so the count lives here in the card, exactly as classic keeps it out of its + /// `ScoreStatePill`. Reuses classic's String Catalog key verbatim — one entry serves both screens. + var calibrationDetail: String? { + guard case .calibrating(let nights) = self else { return nil } + return String(localized: "Learning your baseline, \(nights) of \(Baselines.minNightsSeed) nights.") + } + static func resolve(todayRecovery: Double?, priorScored: DailyMetric?, calibrationNights: Int?, todayKey: String) -> ChargeDisplay { if let pct = todayRecovery { return .scored(pct: pct) } diff --git a/StrandTests/LiquidChargeCarryTests.swift b/StrandTests/LiquidChargeCarryTests.swift index 26964521b9..5e33f0bf79 100644 --- a/StrandTests/LiquidChargeCarryTests.swift +++ b/StrandTests/LiquidChargeCarryTests.swift @@ -1,5 +1,6 @@ import XCTest import WhoopStore +import StrandAnalytics @testable import Strand /// Pins the Liquid Today Charge hero's carry-over truth table. @@ -117,4 +118,28 @@ final class LiquidChargeCarryTests: XCTestCase { XCTAssertEqual(Display.carried(pct: 82.3, caption: "Last night · 4 Jul").stateLabel, "Last night", "the pill labels the carry as prior — the number beside it is not today's") } + + // MARK: - Calibration surfaces its "N of 4" progress (parity with classic TodayView) + + /// The greeting pill is deliberately short ("Calibrating") because it shares a `fixedSize` row with the + /// greeting, so the honest "N of 4 nights" progress that classic `TodayView.calibrationDetail` shows must + /// live in Liquid's synthesis detail instead. `calibrationDetail` carries that copy verbatim so a wearer + /// in their first `Baselines.minNightsSeed` nights reads identical calibration progress on both Today + /// screens — before this, Liquid dropped the count and showed a bare "Calibrating". + func testCalibratingCarriesTheClassicNightCountCopy() { + XCTAssertEqual(Display.calibrating(nights: 2).calibrationDetail, + "Learning your baseline, 2 of 4 nights.", + "Liquid must mirror TodayView.calibrationDetail's copy verbatim (\(Baselines.minNightsSeed)-night seed)") + XCTAssertEqual(Display.calibrating(nights: 0).calibrationDetail, + "Learning your baseline, 0 of 4 nights.", + "night zero still reads honestly with its count, never a bare 'Calibrating'") + } + + /// Only the calibrating state owns a synthesis detail line. A scored / carried / no-data day leaves the + /// readiness one-liner (`synthLine`) untouched — no stray "N of 4" appears once the baseline is trusted. + func testOnlyCalibratingStateHasASynthesisDetailLine() { + XCTAssertNil(Display.scored(pct: 61).calibrationDetail) + XCTAssertNil(Display.carried(pct: 82.3, caption: "Last night · 4 Jul").calibrationDetail) + XCTAssertNil(Display.noData.calibrationDetail) + } } From 44cc6d67893a1fc51c43119e15381adb8a4308f4 Mon Sep 17 00:00:00 2001 From: Vishnu Kchitti <119831996+vishk23@users.noreply.github.com> Date: Thu, 16 Jul 2026 16:37:45 -0700 Subject: [PATCH 123/171] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6d3398222f..8b35c80742 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ > ### About this fork > -> This is **Vishnu Kchittibhooma's** working fork of [NOOP](https://github.com/ryanbr/noop), +> This is **Vishnu Kchitti's** working fork of [NOOP](https://github.com/ryanbr/noop), > the offline WHOOP companion app by [@ryanbr](https://github.com/ryanbr) and contributors. > The base app is their work; this fork is where my contributions to it are developed. > From b38c1d4a838be1a7619915bb505bbe9adecbb71c Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 15 Jul 2026 11:06:00 -0700 Subject: [PATCH 124/171] BLE: recover a stalled link, retry the battery read, stop the UI lying about both MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two failures that only bite the link nobody is watching: * A state-restored peripheral comes back `.connected`, so `willRestoreState` marks the link up — but the CLIENT_HELLO that follows is `.withResponse`, and a wedged strap never acks it. `didWriteValueFor` is the ONLY caller of `startKeepAlive()`, so the 120 s liveness watchdog that exists for exactly this case is never armed, and the app retries a dead handle forever (observed: 40+ min of "connect handshake not done yet", costing a night of biometrics). `StalledHandshakePolicy` arms a 90 s fuse at connect AND at restore, then cancels the link so the existing 3 s disconnect-rescan recovers it. Every input to `shouldRecover` is a reason to stay out of the way. * `enableLiveNotifications` claims to recover the "standard HR/battery that failed pre-bond", but only ever re-ran `setNotifyValue`, never `readValue`. HR self-heals because the strap pushes it; the 5/MG battery percent never did. `StrapBatteryReadPolicy` re-issues the 0x2A19 read, family-gated (#77: a 4.0's 0x2A19 is a stub-100) and throttled to the keep-alive tick (the call site re-enters on every HISTORY_END ack). `LiveState.linkProven` gates the "· Live" half of the Devices pill so a restored-but-dead peripheral reads "Active", not a pulsing "Active · Live". Defaults true, so the four non-WHOOP sources (Oura, Huami, FTMS, standard-HR) keep their exact current behaviour. Ported from the abandoned local `main` line onto origin/main. Dropped in the port: that line's rival `StrapBatteryDisplay`, which let a stale percent outlive a disconnect — origin/main's 9251ae24 truth table deliberately fixed that (A3/B2) and is kept verbatim, along with its LiquidBatteryDisplayTests. Verification: pure policy types covered by StalledHandshakePolicyTests / StrapBatteryReadPolicyTests / DevicePillStateTests. NOT validated on a strap — the CoreBluetooth restore path needs hardware. Co-Authored-By: Claude Opus 5 --- Strand/BLE/BLEManager.swift | 110 ++++++++++++++++++ Strand/BLE/LiveState.swift | 14 +++ Strand/BLE/StalledHandshakePolicy.swift | 63 ++++++++++ Strand/BLE/StrapBatteryReadPolicy.swift | 49 ++++++++ Strand/Screens/DevicesView.swift | 31 ++++- StrandTests/DevicePillStateTests.swift | 43 +++++++ StrandTests/StalledHandshakePolicyTests.swift | 77 ++++++++++++ StrandTests/StrapBatteryReadPolicyTests.swift | 52 +++++++++ 8 files changed, 436 insertions(+), 3 deletions(-) create mode 100644 Strand/BLE/StalledHandshakePolicy.swift create mode 100644 Strand/BLE/StrapBatteryReadPolicy.swift create mode 100644 StrandTests/StalledHandshakePolicyTests.swift create mode 100644 StrandTests/StrapBatteryReadPolicyTests.swift diff --git a/Strand/BLE/BLEManager.swift b/Strand/BLE/BLEManager.swift index c1fb100e27..862fd93baf 100644 --- a/Strand/BLE/BLEManager.swift +++ b/Strand/BLE/BLEManager.swift @@ -657,6 +657,16 @@ public final class BLEManager: NSObject, ObservableObject { /// didUpdateNotificationStateFor re-fire (or any other later call into the check) can't double-bump. /// Reset on disconnect. private var connectSettledSignaled = false + /// When the standard Battery Level (0x2A19) read was last ISSUED this session, for + /// `StrapBatteryReadPolicy`'s throttle. The read is now re-issued from `enableLiveNotifications`, which + /// re-runs on every `.withResponse` ack during an offload — without this the retry would storm the + /// strap. Reset on disconnect so a fresh link always reads immediately. + private var lastBatteryReadAt: TimeInterval? + /// True once ANY notification has arrived on the CURRENT connection (the `sawData` input to + /// `StalledHandshakePolicy`). Distinct from `lastDataAt`, which is seeded to "now" at connect and so + /// cannot tell "the strap has spoken" from "the link just came up" — precisely the distinction a + /// restored zombie peripheral turns on. Reset on connect/restore and on disconnect. + private var sawDataThisConnection = false /// Re-entrancy guard for captureRawAccel: true while a bounded on-demand window is running. /// A second tap is a no-op until the active capture's asyncAfter block fires and clears this. private var rawCaptureInFlight = false @@ -2451,6 +2461,43 @@ public final class BLEManager: NSObject, ObservableObject { clearRebootState() // clears the "Reconnecting…" pill → back to "Active · Live" } + /// Arm the stalled-handshake watchdog for the connection that just came up. + /// + /// Called from BOTH paths that can put the app on a live link: `didConnect` and the `willRestoreState` + /// branch that adopts an already-`.connected` peripheral. The restore branch is the one that actually + /// bit: it never reaches `didWriteValueFor`, which is the only caller of `startKeepAlive()`, so the + /// 120 s liveness watchdog that would have bounced the dead link is never armed and the app retries the + /// same corpse forever ("Backfill: deferred — connect handshake not done yet", 40+ minutes). + /// + /// One-shot, generation-guarded (the same `connectGeneration` idiom the #711 reconnect-guide timer uses), + /// so a timer left in flight by an earlier connection can never bounce a healthy later one. + private func armStalledHandshakeWatchdog() { + let gen = connectGeneration + DispatchQueue.main.asyncAfter(deadline: .now() + StalledHandshakePolicy.fuseSeconds) { [weak self] in + guard let self, self.connectGeneration == gen else { return } + guard StalledHandshakePolicy.shouldRecover( + connected: self.state.connected, + handshakeDone: self.connectHandshakeDone, + sawData: self.sawDataThisConnection, + intentionalDisconnect: self.intentionalDisconnect, + autoReconnectPaused: self.autoReconnectPausedForBondLoop, + bondRefused: self.state.pairingHint != nil) else { return } + // Log the recovery (the "log successes" forensics rule): without this line a strap log shows a + // reconnect with no cause, which is exactly how the original 40-minute spin stayed invisible. + self.log("Handshake stalled: connected for \(Int(StalledHandshakePolicy.fuseSeconds))s with no " + + "handshake and no data — the link is up but dead (a restored peripheral iOS still " + + "believes in). Cancelling it so the rescan can find the strap again.") + if TestCentre.active(.connection) { + self.state.append(log: "handshakeStalled gen=\(gen) fuse=\(Int(StalledHandshakePolicy.fuseSeconds))s " + + "(cancelling + rescanning)", domain: .connection) + } + // Recovery = the SAME lever the liveness watchdog pulls: cancel and let didDisconnectPeripheral's + // existing 3 s rescan re-find the strap. No new reconnect path, and it still honours the + // intentional-disconnect / bond-loop-pause gates that live there. + if let p = self.peripheral { self.central.cancelPeripheralConnection(p) } + } + } + private func startKeepAlive() { keepAliveTimer?.cancel() let s = BLEManager.keepAliveIntervalSeconds @@ -2665,6 +2712,30 @@ public final class BLEManager: NSObject, ObservableObject { for c in chars where !c.isNotifying { requestNotify(c, on: p, reason: reason) } + retryBatteryReadIfDue(on: p, reason: reason) + } + + /// Re-issue the standard Battery Level (0x2A19) READ that a pre-bond link refused. + /// + /// This is the missing half of `enableLiveNotifications`. That function's own call site claims it + /// recovers the "standard HR/battery that failed pre-bond", but it only ever re-ran `setNotifyValue` — + /// never `readValue`. So on a WHOOP 5/MG the one-shot read fired at discovery (unencrypted link) was + /// rejected, dropped in `didUpdateValueFor`'s error branch, and never retried: HR came back because the + /// strap PUSHES it, battery never did. Re-reading here means every moment that already re-subscribes + /// notifies — post-bond 5/MG, post-bond 4.0, keep-alive, manual refresh, realtime start — is also a + /// moment the battery reading can self-heal. `StrapBatteryReadPolicy` owns the family gate (#77: never + /// read a WHOOP 4.0's stub-100 characteristic) and the throttle (this runs on every HISTORY_END ack + /// during an offload); both are pinned by tests. + private func retryBatteryReadIfDue(on p: CBPeripheral, reason: String) { + guard let b = batteryCharacteristic else { return } + let now = Date().timeIntervalSince1970 + guard StrapBatteryReadPolicy.shouldRead(family: selectedModel.deviceFamily, + canRead: b.properties.contains(.read), + lastReadAt: lastBatteryReadAt, + now: now) else { return } + lastBatteryReadAt = now + p.readValue(for: b) + log("Battery Level read re-issued (\(reason))") } private func requestNotify(_ c: CBCharacteristic, on p: CBPeripheral, reason: String) { @@ -3105,6 +3176,11 @@ extension BLEManager: @preconcurrency CBCentralManagerDelegate { // only — BLEManager stays decoupled from the store and the connect flow below is unchanged. connectedPeripheralUUID = peripheral.identifier.uuidString state.connected = true + // …but the link has proven NOTHING yet: this fires before discoverPrimaryServices below, so no + // characteristic exists and no byte has moved. The Devices pill gates "Active · Live" on this so it + // can't claim live data off a raw CBPeripheral handle. Set true by the first notification, or by the + // completed connect handshake, whichever lands first. + state.linkProven = false // A connect succeeded → clear the stale-bond re-pair guide UNLESS we are in a known bond-loop // (#617). In that loop the strap "connects" every ~3 s before timing out again, so clearing here // wiped the guide on EVERY cycle: it flashed for ~1 s and vanished, so the user could never read it @@ -3127,6 +3203,9 @@ extension BLEManager: @preconcurrency CBCentralManagerDelegate { state.reconnectGuide = nil } lastDataAt = Date() + sawDataThisConnection = false // this link has not proven itself yet — see armStalledHandshakeWatchdog + lastBatteryReadAt = nil // a fresh link re-reads 0x2A19 immediately (5/MG; #77 spares the 4.0) + armStalledHandshakeWatchdog() log("Connected — discovering services") // Connection test mode: report the connect latency + the uptime-start marker the readout reads. // Gated zero-cost: the .connection bool is read before any string is built, so this is a no-op @@ -3220,6 +3299,12 @@ extension BLEManager: @preconcurrency CBCentralManagerDelegate { } bondedAt = nil // cleared after the bond-loop detector above read it (#617) state.connected = false + // Back to the inert default: the flag only ever means something while BLEManager owns a live link, + // and a stale `false` left here would make the NEXT source (Oura/Huami/FTMS/standard-HR, none of + // which touch it) read as un-live. The next connect/restore clears it again. + state.linkProven = true + sawDataThisConnection = false + lastBatteryReadAt = nil // the next link re-reads 0x2A19 immediately (5/MG; the 4.0 never reads it) state.encryptedBond = false // cleared with didBond; next session must re-prove the bond (#69) state.charging = nil // a stale charging flag must not outlive the link state.strapFirmware = nil // a stale firmware version must not outlive the link @@ -3400,11 +3485,27 @@ extension BLEManager: @preconcurrency CBCentralManagerDelegate { Task { @MainActor in await bootstrapStore() } if p.state == .connected { state.connected = true + // The restored link is UNPROVEN despite the seeded bond flags above: those record that this + // strap was bonded in a PREVIOUS process, which says nothing about whether it is alive now. This + // is the exact case that made the pill lie — CoreBluetooth says .connected, the bond flags say + // bonded, and the strap is dead. + state.linkProven = false + // A restored peripheral is the ONE path onto a live link that never runs didConnect, so every + // per-connection reset and every watchdog here has to be done by hand — that omission is exactly + // what let a dead restored handle spin for 40+ minutes. Bump the generation first: this IS a new + // connection for the watchdog's staleness guard, and without the bump a restore that follows a + // real connect would reuse its generation. + connectGeneration &+= 1 + sawDataThisConnection = false + lastBatteryReadAt = nil + lastDataAt = Date() // start the liveness clock at the restore, not at BLEManager init + armStalledHandshakeWatchdog() log("Restored CONNECTED peripheral \(p.identifier) — re-discovering services") discoverPrimaryServices(on: p) } else { state.connected = false log("Restored DISCONNECTED peripheral \(p.identifier) — reconnect on poweredOn") + // No watchdog needed here: this branch goes through central.connect → didConnect, which arms it. if central.state == .poweredOn { central.connect(p, options: nil) } } } @@ -3635,6 +3736,7 @@ extension BLEManager: @preconcurrency CBPeripheralDelegate { if !whoop5SessionStarted { whoop5SessionStarted = true connectHandshakeDone = true // unblocks beginBackfill()'s guard + state.linkProven = true // the strap acked CLIENT_HELLO — this link is genuinely alive log("WHOOP 5/MG: connect handshake done — backfill unblocked") noteRebootReconnectIfNeeded() // Re-apply the Broadcast-HR device-config flag if the user opted in (#181). @@ -3683,6 +3785,7 @@ extension BLEManager: @preconcurrency CBPeripheralDelegate { // type-47 fine because it runs the sequence once on a stable connection; the app stormed it. guard !connectHandshakeDone else { return } connectHandshakeDone = true + state.linkProven = true // the confirmed-write bond acked — this link is genuinely alive noteRebootReconnectIfNeeded() backfillStarted = true @@ -3837,6 +3940,13 @@ extension BLEManager: @preconcurrency CBPeripheralDelegate { guard let data = characteristic.value else { return } let bytes = [UInt8](data) lastDataAt = Date() // feed the liveness watchdog on every notification + // The strap has spoken on this link — it is not a zombie, so the stalled-handshake watchdog must + // stand down (the 120s liveness fuse owns a link that goes quiet LATER). This covers every source + // that lands here: a 5/MG's standard 0x2A37 HR, a WHOOP4 puffin/custom frame, a battery read. + if !sawDataThisConnection { + sawDataThisConnection = true + state.linkProven = true // …and the Devices pill may honestly say "Live" now (#221 sibling) + } switch characteristic.uuid { case BLEManager.heartRateChar: diff --git a/Strand/BLE/LiveState.swift b/Strand/BLE/LiveState.swift index 8fca7c6325..eab57bf7c1 100644 --- a/Strand/BLE/LiveState.swift +++ b/Strand/BLE/LiveState.swift @@ -35,6 +35,20 @@ public final class LiveState: ObservableObject { /// per-connection guards (`connectHandshakeDone`, the cmd-notify-confirmed flag) reset on disconnect, /// so the next connection can bump this again. @Published public var connectSettled: Int = 0 + /// Has the CURRENT link proven itself — i.e. has the strap actually spoken (any notification), or has + /// the connect handshake completed? `connected` alone cannot answer that: BLEManager publishes it the + /// instant CoreBluetooth's `didConnect` fires, BEFORE service discovery, so it is true while no + /// characteristic exists and no byte has moved. Combined with iOS state restoration handing back a + /// peripheral that is `.connected` but dead, that made the Devices card advertise "Active · Live" for a + /// strap that was a corpse — the user read "Live" all evening on a band that had flat-lined. + /// + /// DEFAULTS TRUE, and only BLEManager ever clears it. That is deliberate: `connected` is also driven by + /// four non-WHOOP sources (Oura, Huami, FTMS, generic standard-HR), and each of those sets it true only + /// once it is genuinely subscribed and receiving. They never touch this flag, so they keep their exact + /// current behaviour — an inert `true` — instead of being regressed into looking offline. BLEManager + /// clears it at connect/restore, sets it on first data or a completed handshake, and restores the inert + /// `true` on disconnect so a stale `false` can't outlive the link and poison the next source. + @Published public var linkProven: Bool = true /// True ONLY when a non-WHOOP live source (currently the Oura ring) is actively streaming live HR. /// This is the green "STREAMING" signal for sources that have no WHOOP-style encrypted bond: it is /// DELIBERATELY separate from `bonded`, which carries WHOOP encrypted-bond + buzz semantics (it gates diff --git a/Strand/BLE/StalledHandshakePolicy.swift b/Strand/BLE/StalledHandshakePolicy.swift new file mode 100644 index 0000000000..1688344c92 --- /dev/null +++ b/Strand/BLE/StalledHandshakePolicy.swift @@ -0,0 +1,63 @@ +import Foundation + +/// Recovery decision for a link that CoreBluetooth calls connected but that never became usable. +/// +/// The failure this exists for: iOS state restoration hands back a peripheral whose `state` is already +/// `.connected`, so `willRestoreState` marks the link up, seeds the bond flags (a restored link WAS bonded +/// once) and re-discovers services — but the CLIENT_HELLO / bond write that follows is `.withResponse`, and +/// if the strap is wedged or dead it is never acked. `didWriteValueFor` therefore never fires, which is the +/// ONLY place that sets `connectHandshakeDone` **and the only place that calls `startKeepAlive()`** — so the +/// 120 s liveness watchdog inside `keepAliveFire` (the machinery that would have bounced exactly this link) +/// is never armed. The app then retries the same dead handle forever: an on-device strap log showed +/// "Backfill: deferred — connect handshake not done yet" repeating every ~1–3 min for 40+ minutes, straight +/// through the strap coming off its charger, and cost the user a whole night of biometrics. +/// +/// So the recovery is deliberately NOT a new reconnect mechanism: it cancels the peripheral connection and +/// lets `didDisconnectPeripheral`'s existing 3 s rescan (the same recovery the liveness watchdog uses at its +/// own fuse) bring the link back. This type only decides WHETHER to pull that trigger. Pure + no +/// CoreBluetooth, so the decision is unit-testable without a radio. +enum StalledHandshakePolicy { + + /// How long a link may sit connected-but-unproven before we cancel it. + /// + /// Chosen against the fuses this codebase already runs, not invented: + /// * a healthy handshake resolves in well under a second (the post-ack work defers the offload by 1.5 s), + /// so 90 s is ~60× the expected latency — it can never clip a strap that is merely slow; + /// * it sits ABOVE the 60 s reboot settle backstop, so a reboot-driven reconnect resolves itself first + /// (`noteRebootReconnectIfNeeded`) instead of being pre-empted by a bounce; + /// * it sits BELOW the 120 s `keepAliveFire` liveness fuse, so on any link that IS delivering data the + /// data-driven watchdog stays the one making the call — `shouldRecover`'s `sawData` guard defers to it + /// anyway, so the two can never fight over the same link. + static let fuseSeconds: TimeInterval = 90 + + /// Cancel-and-rescan this link? + /// + /// Every input is a reason to STAY OUT of the way, which is the point — this fires only on the narrow + /// case nothing else owns: + /// - `connected`: the link already dropped → the normal disconnect path owns the rescan. + /// - `handshakeDone`: the connect handshake completed → this is a healthy link, never touch it. + /// - `sawData`: ANY notification arrived since the link came up (0x2A37 HR, a puffin frame, a battery + /// read). The strap is talking to us and is useful even if the puffin handshake lags, and the 120 s + /// liveness fuse already covers a link that goes quiet later. Bouncing here would throw away a working + /// HR stream — the regression this guard exists to prevent. + /// - `intentionalDisconnect`: a user teardown is in flight. + /// - `autoReconnectPaused`: the #617/#747 bond-loop give-up deliberately STOPPED hammering this strap and + /// told the user to free it; a bounce here would restart exactly the loop that machinery just paused. + /// - `bondRefused`: the #78/#221 refusal state ("Connected · not paired") is a diagnosed, surfaced, + /// user-actionable condition with its own guidance on the Devices card. It is not a stall, and churning + /// the link would only hide the guidance behind reconnect noise. + static func shouldRecover(connected: Bool, + handshakeDone: Bool, + sawData: Bool, + intentionalDisconnect: Bool, + autoReconnectPaused: Bool, + bondRefused: Bool) -> Bool { + guard connected else { return false } + guard !handshakeDone else { return false } + guard !sawData else { return false } + guard !intentionalDisconnect else { return false } + guard !autoReconnectPaused else { return false } + guard !bondRefused else { return false } + return true + } +} diff --git a/Strand/BLE/StrapBatteryReadPolicy.swift b/Strand/BLE/StrapBatteryReadPolicy.swift new file mode 100644 index 0000000000..2504846c87 --- /dev/null +++ b/Strand/BLE/StrapBatteryReadPolicy.swift @@ -0,0 +1,49 @@ +import Foundation +import WhoopProtocol + +/// Whether to (re)issue a standard Battery Level (0x2A19) READ on the live link. +/// +/// The failure this exists for: the ONLY 0x2A19 read was a one-shot fired from +/// `didDiscoverCharacteristicsFor` — i.e. PRE-bond, on an unencrypted link. A WHOOP 5/MG rejects that read +/// ("Authentication is insufficient", the same refusal that gates its notify subscriptions), the failure +/// landed in `didUpdateValueFor`'s error branch, was logged, and was then DROPPED. Nothing ever retried it. +/// `enableLiveNotifications` — whose own comment claims it recovers the "standard HR/battery that failed +/// pre-bond" — only ever re-issued `setNotifyValue`, never `readValue`. That asymmetry is the whole bug: HR +/// recovers because it is notify-driven and the strap pushes it at 1 Hz, so the UI says "Live" while the +/// battery percent stays `nil` for the entire session. On a strap that then died on the user's wrist, the +/// app could not show a single battery reading all day. +/// +/// FAMILY-CORRECT BY CONSTRUCTION (#77): a WHOOP 4.0's 0x2A19 is a firmware STUB that always reports 100, +/// so reading it would flash a fake 100% over the real charge. The 4.0's true value comes only from the +/// proprietary `GET_BATTERY_LEVEL` command (COMMAND_RESPONSE, u16/10) that the keep-alive already sends +/// every ~60 s. Refusing the read for `.whoop4` here is what keeps that invariant intact, and it is pinned +/// by `StrapBatteryReadPolicyTests` so a future edit can't quietly re-open #77. +enum StrapBatteryReadPolicy { + + /// Floor between reads, matching the keep-alive tick (`BLEManager.keepAliveIntervalSeconds`). + /// + /// The throttle is load-bearing, not politeness: `didWriteValueFor` re-enters the 5/MG post-bond branch + /// on EVERY `.withResponse` ack — including every HISTORY_END ack during a multi-minute historical + /// offload — and that branch calls `enableLiveNotifications` unconditionally. The existing work there is + /// idempotent-cheap (`where !c.isNotifying` collapses to a no-op once subscribed); an unthrottled + /// `readValue` would NOT be, and would fire hundreds of ATT reads at a strap mid-sync. With the floor, + /// the steady-state cadence is one read per keep-alive tick, which is strictly lighter than the 4.0's + /// existing every-60 s `GET_BATTERY_LEVEL` poll. + static let minIntervalSeconds: TimeInterval = 30 + + /// - Parameters: + /// - family: the connected strap's family. `.whoop4` always refuses (see #77 above). + /// - canRead: the characteristic exists and advertises `.read`. + /// - lastReadAt: when we last issued a read this session; nil = never, which always reads (this is the + /// self-heal — the first post-bond call re-issues the read the pre-bond link refused). + /// - now: monotonic-ish seconds; injected so the throttle is testable without a clock. + static func shouldRead(family: DeviceFamily, + canRead: Bool, + lastReadAt: TimeInterval?, + now: TimeInterval) -> Bool { + guard family != .whoop4 else { return false } // #77: the 4.0's 0x2A19 is a stub-100 + guard canRead else { return false } + guard let last = lastReadAt else { return true } + return now - last >= minIntervalSeconds + } +} diff --git a/Strand/Screens/DevicesView.swift b/Strand/Screens/DevicesView.swift index d4a2672138..77c622881b 100644 --- a/Strand/Screens/DevicesView.swift +++ b/Strand/Screens/DevicesView.swift @@ -99,6 +99,11 @@ private struct DevicesContent: View { device: device, isActive: device.status == .active, isLiveConnected: device.status == .active && live.connected, + // Has this link actually proven itself (any data, or a completed handshake)? `connected` + // alone is just "CoreBluetooth handed us an object" — true before service discovery, and + // true for a state-restored peripheral that is already dead. Without this the card + // pulsed "Active · Live" for a strap that had flat-lined hours earlier. + linkProven: live.linkProven, // #221: a WHOOP 5/MG can be BLE-connected yet have its ENCRYPTED bond refused (the // WHOOP app, or a stale iOS pairing, holds the single-app bond) — no HR/biometric data // flows even though the link is up, so "Active · Live" overstates it. pairingHint is @@ -342,13 +347,29 @@ struct DevicePillState: Equatable { var pulsing: Bool = false var showsDot: Bool = true + /// `linkProven` (see `LiveState.linkProven`): has this link actually carried a byte, or completed its + /// connect handshake? `isLiveConnected` is built from `live.connected`, which BLEManager publishes the + /// instant CoreBluetooth's `didConnect` fires — BEFORE service discovery, so before any characteristic + /// exists or any data flows. Layer iOS state restoration on top (it hands back a peripheral that reads + /// `.connected` and re-seeds the bond flags from a PREVIOUS process) and the pill confidently pulsed + /// "Active · Live" for a strap that had died on the user's wrist hours earlier. + /// + /// Defaults `true` so every existing caller — and the whole non-WHOOP source family (Oura, Huami, FTMS, + /// standard-HR), none of which touch the flag — keeps its exact current behaviour. An unproven link + /// falls back to plain "Active", which is the honest statement: this IS your active device, we just + /// can't claim data is flowing. It deliberately does NOT invent a new pill state — the strap may be a + /// second away from streaming, and the stalled-handshake watchdog cancels a genuinely dead link inside + /// 90 s anyway. static func resolve(isArchived: Bool, isActive: Bool, isReconnecting: Bool, - bondRefused: Bool, isLiveConnected: Bool) -> DevicePillState { + bondRefused: Bool, isLiveConnected: Bool, + linkProven: Bool = true) -> DevicePillState { if isArchived { return DevicePillState(label: "Removed", tone: .neutral, showsDot: false) } guard isActive else { return DevicePillState(label: "Paired", tone: .neutral) } if isReconnecting { return DevicePillState(label: "Reconnecting…", tone: .warning, pulsing: true) } if bondRefused { return DevicePillState(label: "Connected · not paired", tone: .warning) } - if isLiveConnected { return DevicePillState(label: "Active · Live", tone: .positive, pulsing: true) } + if isLiveConnected && linkProven { + return DevicePillState(label: "Active · Live", tone: .positive, pulsing: true) + } return DevicePillState(label: "Active", tone: .positive) } } @@ -361,6 +382,10 @@ private struct DeviceCard: View { let device: PairedDevice let isActive: Bool let isLiveConnected: Bool + /// Has the live link actually carried data / completed its handshake (`LiveState.linkProven`)? Gates the + /// "· Live" half of the pill so a raw-but-dead CBPeripheral — the restored-zombie case — reads "Active" + /// instead of pulsing "Active · Live". Defaults true: every non-WHOOP source keeps today's behaviour. + var linkProven: Bool = true /// #221: the active+connected strap is BLE-linked but its encrypted bond was refused (#78 state) — /// no HR/biometric data flows despite the link being up. Drives the "Connected · not paired" pill /// (which takes priority over "Active · Live") and the honest subtitle/footnote. False for every @@ -594,7 +619,7 @@ private struct DeviceCard: View { private var pillState: DevicePillState { DevicePillState.resolve(isArchived: device.status == .archived, isActive: isActive, isReconnecting: isReconnecting, bondRefused: bondRefused, - isLiveConnected: isLiveConnected) + isLiveConnected: isLiveConnected, linkProven: linkProven) } private var statePill: some View { diff --git a/StrandTests/DevicePillStateTests.swift b/StrandTests/DevicePillStateTests.swift index 1a255027b6..78087ddd46 100644 --- a/StrandTests/DevicePillStateTests.swift +++ b/StrandTests/DevicePillStateTests.swift @@ -38,4 +38,47 @@ final class DevicePillStateTests: XCTestCase { bondRefused: false, isLiveConnected: false).label, "Removed") } + + /// An UNPROVEN link must not pulse "Active · Live". `isLiveConnected` is built from `live.connected`, + /// which BLEManager publishes the instant CoreBluetooth's didConnect fires — before service discovery, + /// before any characteristic exists, before a byte moves. With iOS state restoration handing back a + /// `.connected` peripheral whose strap had died hours earlier, the card confidently advertised "Live" + /// for a corpse. "Active" is the honest fallback: still your active device, no claim about data. + func testUnprovenLink_doesNotClaimLive() { + XCTAssertEqual( + DevicePillState.resolve(isArchived: false, isActive: true, isReconnecting: false, + bondRefused: false, isLiveConnected: true, linkProven: false).label, + "Active") + XCTAssertFalse( + DevicePillState.resolve(isArchived: false, isActive: true, isReconnecting: false, + bondRefused: false, isLiveConnected: true, linkProven: false).pulsing) + } + + /// A proven link is unchanged, and the higher-priority states still win over an unproven one — the + /// #221 ordering must not be perturbed by the new gate. + func testLinkProven_defaultsTrueAndKeepsThePriorityOrder() { + // Default (omitted) linkProven keeps every existing call site behaving exactly as before. + XCTAssertEqual( + DevicePillState.resolve(isArchived: false, isActive: true, isReconnecting: false, + bondRefused: false, isLiveConnected: true).label, + "Active · Live") + XCTAssertEqual( + DevicePillState.resolve(isArchived: false, isActive: true, isReconnecting: false, + bondRefused: false, isLiveConnected: true, linkProven: true).label, + "Active · Live") + // Reconnecting and bond-refused still outrank the live/unproven decision entirely. + XCTAssertEqual( + DevicePillState.resolve(isArchived: false, isActive: true, isReconnecting: true, + bondRefused: false, isLiveConnected: true, linkProven: false).label, + "Reconnecting…") + XCTAssertEqual( + DevicePillState.resolve(isArchived: false, isActive: true, isReconnecting: false, + bondRefused: true, isLiveConnected: true, linkProven: false).label, + "Connected · not paired") + // An unproven link on a NON-active card is still just "Paired". + XCTAssertEqual( + DevicePillState.resolve(isArchived: false, isActive: false, isReconnecting: false, + bondRefused: false, isLiveConnected: false, linkProven: false).label, + "Paired") + } } diff --git a/StrandTests/StalledHandshakePolicyTests.swift b/StrandTests/StalledHandshakePolicyTests.swift new file mode 100644 index 0000000000..fc5b05375d --- /dev/null +++ b/StrandTests/StalledHandshakePolicyTests.swift @@ -0,0 +1,77 @@ +import XCTest +@testable import Strand + +/// `StalledHandshakePolicy` — the cancel-and-rescan decision for a link CoreBluetooth calls connected but +/// that never became usable. Pure value logic, no CoreBluetooth seam. +/// +/// The case that matters is `testRestoredZombie_recovers`: it encodes the real on-device failure (iOS state +/// restoration returns a `.connected` peripheral whose strap is dead, the CLIENT_HELLO is never acked, so +/// `didWriteValueFor` never fires — and that callback is the ONLY caller of `startKeepAlive()`, so the 120 s +/// liveness watchdog that would have bounced the link never even arms). Every other test here is a guard +/// AGAINST firing, because a watchdog that bounces healthy links is worse than the bug it fixes. +final class StalledHandshakePolicyTests: XCTestCase { + + /// Defaults describe a link in the exact stalled state; each test flips ONE input. + private func recover(connected: Bool = true, + handshakeDone: Bool = false, + sawData: Bool = false, + intentionalDisconnect: Bool = false, + autoReconnectPaused: Bool = false, + bondRefused: Bool = false) -> Bool { + StalledHandshakePolicy.shouldRecover(connected: connected, + handshakeDone: handshakeDone, + sawData: sawData, + intentionalDisconnect: intentionalDisconnect, + autoReconnectPaused: autoReconnectPaused, + bondRefused: bondRefused) + } + + /// THE bug: connected, no handshake, not one byte ever received, and nothing else owns the link. + /// Left alone this spun for 40+ minutes ("Backfill: deferred — connect handshake not done yet") and + /// cost a full night of biometrics. + func testRestoredZombie_recovers() { + XCTAssertTrue(recover()) + } + + /// A completed handshake means the strap acked us — this is a healthy link, hands off. + func testCompletedHandshake_neverRecovers() { + XCTAssertFalse(recover(handshakeDone: true)) + XCTAssertFalse(recover(handshakeDone: true, sawData: true)) + } + + /// The strap is talking (e.g. a 5/MG streaming HR over the standard 0x2A37 profile while its puffin + /// handshake lags). Bouncing would throw away a working HR stream; the 120 s liveness fuse already owns + /// a link that goes quiet later. + func testDataFlowing_neverRecovers() { + XCTAssertFalse(recover(sawData: true)) + } + + /// The link already dropped — didDisconnectPeripheral's rescan owns it. + func testNotConnected_neverRecovers() { + XCTAssertFalse(recover(connected: false)) + } + + /// A user teardown is in flight; reconnecting would fight it. + func testIntentionalDisconnect_neverRecovers() { + XCTAssertFalse(recover(intentionalDisconnect: true)) + } + + /// #617/#747: the bond-loop give-up DELIBERATELY stopped hammering this strap and told the user to free + /// it. A bounce here restarts precisely the loop that machinery just paused. + func testAutoReconnectPaused_neverRecovers() { + XCTAssertFalse(recover(autoReconnectPaused: true)) + } + + /// #78/#221: a refused bond ("Connected · not paired") is diagnosed, surfaced and user-actionable — not + /// a stall. Churning the link would bury the guidance under reconnect noise. + func testBondRefused_neverRecovers() { + XCTAssertFalse(recover(bondRefused: true)) + } + + /// The fuse must clear the 60 s reboot settle backstop (so a reboot reconnect resolves itself first) and + /// stay under the 120 s liveness fuse (so the data-driven watchdog owns any link that has data). + func testFuseSitsBetweenTheRebootSettleAndTheLivenessFuse() { + XCTAssertGreaterThan(StalledHandshakePolicy.fuseSeconds, 60) + XCTAssertLessThan(StalledHandshakePolicy.fuseSeconds, 120) + } +} diff --git a/StrandTests/StrapBatteryReadPolicyTests.swift b/StrandTests/StrapBatteryReadPolicyTests.swift new file mode 100644 index 0000000000..057c4ba871 --- /dev/null +++ b/StrandTests/StrapBatteryReadPolicyTests.swift @@ -0,0 +1,52 @@ +import XCTest +import WhoopProtocol +@testable import Strand + +/// `StrapBatteryReadPolicy` — whether to (re)issue a standard Battery Level (0x2A19) read. +/// +/// `testWhoop4_neverReads` is the load-bearing one: it pins #77. A WHOOP 4.0's 0x2A19 is a firmware stub +/// that always reports 100, so reading it flashes a fake 100% over the strap's real charge (the 4.0's true +/// value comes only from the proprietary GET_BATTERY_LEVEL command). The retry that fixes the 5/MG must +/// never reach the 4.0, and that is a family gate a future refactor could silently drop. +final class StrapBatteryReadPolicyTests: XCTestCase { + private let floor = StrapBatteryReadPolicy.minIntervalSeconds // 30 + + /// #77: the 4.0's 0x2A19 is a stub-100. No read, ever — not even the first one. + func testWhoop4_neverReads() { + XCTAssertFalse(StrapBatteryReadPolicy.shouldRead(family: .whoop4, canRead: true, + lastReadAt: nil, now: 1000)) + XCTAssertFalse(StrapBatteryReadPolicy.shouldRead(family: .whoop4, canRead: true, + lastReadAt: 0, now: 1_000_000)) + } + + /// The self-heal: a 5/MG that has never had a successful read reads immediately. This is the retry the + /// pre-bond rejection ("Authentication is insufficient") needed and never got. + func testWhoop5_firstReadAlwaysRuns() { + XCTAssertTrue(StrapBatteryReadPolicy.shouldRead(family: .whoop5, canRead: true, + lastReadAt: nil, now: 1000)) + } + + /// The throttle: `enableLiveNotifications` re-runs on EVERY .withResponse ack, including every + /// HISTORY_END during an offload. Without the floor the retry would storm the strap with ATT reads. + func testWhoop5_throttledWithinTheFloor() { + XCTAssertFalse(StrapBatteryReadPolicy.shouldRead(family: .whoop5, canRead: true, + lastReadAt: 1000, now: 1000 + floor - 1)) + XCTAssertTrue(StrapBatteryReadPolicy.shouldRead(family: .whoop5, canRead: true, + lastReadAt: 1000, now: 1000 + floor)) + XCTAssertTrue(StrapBatteryReadPolicy.shouldRead(family: .whoop5, canRead: true, + lastReadAt: 1000, now: 1000 + floor * 10)) + } + + /// A characteristic without the .read property can't be read regardless of family. + func testNoReadProperty_neverReads() { + XCTAssertFalse(StrapBatteryReadPolicy.shouldRead(family: .whoop5, canRead: false, + lastReadAt: nil, now: 1000)) + } + + /// The floor matches the keep-alive tick, so the steady-state 5/MG cadence is one read per tick — + /// strictly lighter than the 4.0's existing every-60s GET_BATTERY_LEVEL poll. + func testFloorMatchesTheKeepAliveTick() { + XCTAssertEqual(StrapBatteryReadPolicy.minIntervalSeconds, + TimeInterval(BLEManager.keepAliveIntervalSeconds)) + } +} From 801d0c0c4d37c84643b5602ac68d69a3763d0301 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 26 Jul 2026 19:23:22 -0400 Subject: [PATCH 125/171] Backfill: show how far behind the strap is, and stop lying about charging MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two honesty gaps in the recovery path, both computable from numbers already tracked: * The app said "Syncing strap history…" WHILE a session ran and nothing at all between sessions — so an idle app looked finished when it was 18 h behind and waiting out the 15-minute floor. `BackfillProgress` resolves `strapRange.newestUnix` against the persisted data frontier into recovering/pending/upToDate, and `LiquidBackfillProgressRow` states both outright plus a "have it through " anchor. It reuses `BackfillContinuation.defaultBehindGapSeconds`, so the readout can never say "behind" about a gap the drain already considers closed. This is the half of B1 that `LiquidSyncStatusRow`'s own comment left open; that row is kept and still answers THAT a drain is running. BLEManager publishes the frontier from the one place it already reads it (the per-session continuation decision), so the readout costs no extra query. * The 5.0 `battery_charging` bit is decoded at an UNVERIFIED offset — `decodeWhoop5Event` claims confirmation for the deci-percent SoC and makes no such claim for this bit — and a 5/MG on its puck was observed reporting 0 while its SoC climbed. So the app stated "not charging" and showed a "~2 days left" DISCHARGE estimate about a charging strap. `StrapChargeInference` adds a second witness that cannot lie: a rising SoC, using `BatteryEstimator.chargeStepPct` — the threshold this codebase ALREADY treats as a charge. Strictly additive: inference can only ADD a charge, never deny one, and with neither witness it answers nil (unknown) rather than asserting "not charging". DELIBERATELY NOT PORTED from the abandoned local `main` line: that commit also raised the auto-continue cap 6 -> 120 behind an `isDeepDrain` gate. origin/main's 2b968d27/d2ec5dc3 already examined and rejected exactly that, and names the ~29 min/session figure it rests on as a CONTAMINATED measurement — taken from a strap an experimental probe had switched into R22 deep-data banking (3508 B/s vs the default 124 B/s), where a default strap drains an 18 h backlog in ~2 sessions, well inside 6. `BackfillContinuation` is therefore byte-identical to origin/main here. The progress readout above is the part of that commit that stands on its own — it reports the backlog rather than changing how it is drained. Verification: BackfillProgressTests + StrapChargeInferenceTests (pure, no strap). Dropped the port's BackfillDeepDrainTests along with the machinery they covered. NOT validated on a strap — the charge inference needs a real charging session to confirm end-to-end. Co-Authored-By: Claude Opus 5 --- Strand/BLE/BLEManager.swift | 12 ++- Strand/BLE/BackfillProgress.swift | 70 +++++++++++++++++ Strand/BLE/LiveState.swift | 39 +++++++++- Strand/BLE/StrapChargeInference.swift | 66 ++++++++++++++++ Strand/Liquid/LiquidTodayView.swift | 57 ++++++++++++-- StrandTests/BackfillProgressTests.swift | 72 +++++++++++++++++ StrandTests/StrapChargeInferenceTests.swift | 85 +++++++++++++++++++++ 7 files changed, 392 insertions(+), 9 deletions(-) create mode 100644 Strand/BLE/BackfillProgress.swift create mode 100644 Strand/BLE/StrapChargeInference.swift create mode 100644 StrandTests/BackfillProgressTests.swift create mode 100644 StrandTests/StrapChargeInferenceTests.swift diff --git a/Strand/BLE/BLEManager.swift b/Strand/BLE/BLEManager.swift index 862fd93baf..d179a2e04b 100644 --- a/Strand/BLE/BLEManager.swift +++ b/Strand/BLE/BLEManager.swift @@ -1915,6 +1915,9 @@ public final class BLEManager: NSObject, ObservableObject { let frontier = await collector?.latestHRSampleTs() ?? nil let wallNow = Int(Date().timeIntervalSince1970) // #928: real wall clock, at decision time let stillConnected = state.connected && state.bonded + // Publish the frontier for the backfill-progress readout (#364 UI): this is the only place the + // frontier is read, and it's already read once per session end — exactly the cadence the UI wants. + state.setDataFrontier(frontier) guard BackfillContinuation.shouldAutoContinue( stillConnected: stillConnected, strapNewestTs: newest, @@ -1939,7 +1942,7 @@ public final class BLEManager: NSObject, ObservableObject { // of re-kicks. Reset here — NOT unconditionally on every HISTORY_COMPLETE — so a strap that // slices one offload into many completions can't keep resetting the cap and spin forever. // EXCEPTION: if we stopped because the per-connection CAP is hit, leave the streak at/over - // the cap so it STAYS engaged for the rest of this connection (the 15-min floor takes over); + // it so it STAYS engaged for the rest of this connection (the 15-min floor takes over); // zeroing it here would immediately re-arm the cap and let a runaway strap spin again. if count < BackfillContinuation.defaultMaxAutoContinues { consecutiveAutoContinues = 0 @@ -1951,7 +1954,12 @@ public final class BLEManager: NSObject, ObservableObject { // log/counter churn if so. guard !backfilling else { return } consecutiveAutoContinues += 1 - log("Backfill: auto-continuing (#364/#451) — the trim advanced and the strap is still handing over real records (frontier \(frontier.map(String.init) ?? "?"), strap-reported newest \(newest.map(String.init) ?? "?")); re-kicking offload \(consecutiveAutoContinues)/\(BackfillContinuation.defaultMaxAutoContinues) without waiting the 15-min floor.") + let ceiling = BackfillContinuation.defaultMaxAutoContinues + // Name the DEPTH in the log, not just the counter. "offload 3/6, ~15h behind" is what makes a + // strap-log export answer "is my data lost or merely pending?" — the question a user spent hours + // unable to answer while this chain silently throttled itself to the 15-minute floor. + let behind = (newest.map { n in frontier.map { n - $0 } } ?? nil).map { " ~\($0 / 3600)h behind" } ?? "" + log("Backfill: auto-continuing (#364/#451) — the trim advanced and the strap is still handing over real records (frontier \(frontier.map(String.init) ?? "?"), strap-reported newest \(newest.map(String.init) ?? "?")\(behind)); re-kicking offload \(consecutiveAutoContinues)/\(ceiling) without waiting the 15-min floor.") // .autoContinue bypasses the BackfillPolicy floor (the whole point — don't wait 15 min); // requestSync still re-checks connected/bonded/not-backfilling before kicking, and the // consecutive-cap above is the runaway guard. diff --git a/Strand/BLE/BackfillProgress.swift b/Strand/BLE/BackfillProgress.swift new file mode 100644 index 0000000000..6250ef997f --- /dev/null +++ b/Strand/BLE/BackfillProgress.swift @@ -0,0 +1,70 @@ +import Foundation + +/// What to tell the user about strap-history recovery, as a pure decision. +/// +/// This exists because the recovery was INVISIBLE. The app already showed "Syncing strap history…" while a +/// single offload session ran, but nothing said how far behind the strap was, whether the gap was closing, +/// or whether a quiet app meant "done" or "waiting 15 minutes for the next 60-second session". A user +/// sitting on an 18-hour backlog spent hours unable to tell whether his data was lost or merely pending — +/// and the honest answer ("~15h behind, recovering") was already computable from numbers BLEManager tracks. +/// +/// Both inputs are already on `LiveState`: `strapRange.newestUnix` (the newest record the strap says it +/// holds, from GET_DATA_RANGE) and `dataFrontierUnix` (the newest record we've actually persisted, read +/// once per offload session for the auto-continue decision). The gap between them IS the backlog. +enum BackfillProgress: Equatable { + /// Nothing honest to say — no link, or the strap hasn't answered a range yet. + case unknown + /// Behind, and actively draining right now. + case recovering(behindSeconds: Int, frontierUnix: Int) + /// Behind, but not draining this instant (between sessions / waiting out the periodic floor). + case pending(behindSeconds: Int, frontierUnix: Int) + /// The frontier is within `behindGapSeconds` of the strap's newest record — as caught up as the strap + /// can tell us. Deliberately the SAME threshold `BackfillContinuation` stops draining at, so the + /// readout can never say "behind" about a gap the drain has already decided is closed. + case upToDate + + /// `nowUnix` is unused for the decision itself (the gap is strap-newest vs our frontier, both strap-side + /// timestamps) but is taken so a caller can't be tempted to reach for a clock inside a pure function. + static func resolve(connected: Bool, + backfilling: Bool, + frontierUnix: Int?, + strapNewestUnix: Int?, + behindGapSeconds: Int = BackfillContinuation.defaultBehindGapSeconds) -> BackfillProgress { + guard connected, let frontier = frontierUnix, let newest = strapNewestUnix else { return .unknown } + let behind = newest - frontier + // A frontier AHEAD of the strap's reported newest is a stale/wrong-epoch range answer (#451) or a + // future-dated strap clock (#928), not negative backlog. Claim nothing rather than a nonsense figure. + guard behind > behindGapSeconds else { return behind >= -behindGapSeconds ? .upToDate : .unknown } + return backfilling + ? .recovering(behindSeconds: behind, frontierUnix: frontier) + : .pending(behindSeconds: behind, frontierUnix: frontier) + } + + /// How far behind, in whole hours/minutes — nil when there's nothing to report. + var behindSeconds: Int? { + switch self { + case let .recovering(b, _), let .pending(b, _): return b + case .unknown, .upToDate: return nil + } + } + + /// The strap-side instant our data currently reaches — the "we have your data up to HERE" anchor. + var frontierUnix: Int? { + switch self { + case let .recovering(_, f), let .pending(_, f): return f + case .unknown, .upToDate: return nil + } + } + + /// Compact "~15h behind" / "~42m behind". Whole hours once past an hour, minutes below it — the same + /// shape the battery runtime estimate uses, so the two readouts read as one family. + /// + /// A plain (unlocalized) string, deliberately: this mirrors `DevicePillState.label`, whose call site + /// wraps it as `StatePill(LocalizedStringKey(label))`. Keeping the resolver free of `String(localized:)` + /// is what lets the tests assert on the decision rather than on the user's current locale. + var behindLabel: String? { + guard let s = behindSeconds else { return nil } + if s >= 3600 { return "~\(s / 3600)h behind" } + return "~\(max(1, s / 60))m behind" + } +} diff --git a/Strand/BLE/LiveState.swift b/Strand/BLE/LiveState.swift index eab57bf7c1..7504bfc776 100644 --- a/Strand/BLE/LiveState.swift +++ b/Strand/BLE/LiveState.swift @@ -73,11 +73,23 @@ public final class LiveState: ObservableObject { /// replaced) by `setRRIntervals(_:)`; emptied by `clearBiometrics()`. @Published public private(set) var rrRecent: [Int] = [] @Published public var batteryPct: Double? = nil - /// Charging flag from the strap's BATTERY_LEVEL events — wire observation: u8 bit0 in the + /// RAW charging flag from the strap's BATTERY_LEVEL events — wire observation: u8 bit0 in the /// event payload (4.0 @26 / 5.0 @30), pushed ~every 8 min on captured links. nil until the /// first event of a session; cleared on disconnect so a stale flag can't outlive the link. /// Flag ONLY — the battery % keeps its family-specific source (#77). + /// + /// This is the WIRE value and stays that way (it's what gets persisted). Do not render it directly: + /// the 5.0 offset is unverified and has been observed reporting 0 on a strap that was demonstrably + /// charging. `chargingEffective` is the value to display — see `StrapChargeInference`. @Published public var charging: Bool? = nil + /// The charging state worth showing: the raw flag, rescued by a measurably rising SoC when the flag + /// stays silent or wrong. Computed (never stored) over the already-`@Published` `charging` + + /// `batterySamples`, exactly like `batteryEstimate` — so any view reading it re-renders when either + /// witness changes, and there's no third piece of state to keep in sync. + public var chargingEffective: Bool? { + StrapChargeInference.resolve(flag: charging, samples: batterySamples, + nowUnix: Int(Date().timeIntervalSince1970)) + } // MARK: - Battery runtime estimate (#713) @@ -207,6 +219,30 @@ public final class LiveState: ObservableObject { /// window can't outlive the link. public func clearStrapRange() { strapRange = nil } + /// The newest record WE'VE actually persisted (max HR ts) — our data frontier. Together with + /// `strapRange.newestUnix` (the newest the STRAP holds) this is the backlog: the gap between what the + /// strap banked and what we've recovered. BLEManager publishes it from the one place that already reads + /// it, the per-session auto-continue decision, so the readout costs no extra store queries. + /// + /// nil until the first offload session ends, and cleared on disconnect — a stale frontier would make the + /// progress readout quote a gap that no longer exists. + @Published public private(set) var dataFrontierUnix: Int? + + /// Publish the frontier read during an offload session's continuation decision. nil is a legitimate + /// answer (nothing persisted yet) and is stored as such. + public func setDataFrontier(_ ts: Int?) { dataFrontierUnix = ts } + + /// What to tell the user about history recovery right now. Computed over the already-`@Published` + /// `connected` / `backfilling` / `dataFrontierUnix` / `strapRange`, so a view reading it re-renders when + /// any input moves — same pattern as `batteryEstimate`. Internal (not public) because `BackfillProgress` + /// is: it's an app-module display type, and its threshold defaults come from `BackfillContinuation`, + /// which is internal too. Every consumer (the Today card, the tests via @testable) is in-module. + var backfillProgress: BackfillProgress { + BackfillProgress.resolve(connected: connected, backfilling: backfilling, + frontierUnix: dataFrontierUnix, + strapNewestUnix: strapRange.map { $0.newestUnix > 0 ? $0.newestUnix : nil } ?? nil) + } + @Published public var lastFrameType: String? = nil @Published public var lastEvent: String? = nil /// #987: unix of the most recent strap frame FrameRouter routed. Deliberately NOT @Published - the @@ -486,6 +522,7 @@ public final class LiveState: ObservableObject { recentHrSamples.removeAll() // Sleep readout buffers must not outlive the link (Group E) recentGravitySamples.removeAll() clearStrapRange() // a stale clock-drift window must not outlive the link either + dataFrontierUnix = nil // …nor a stale frontier, which would quote a gap that's moved on lastFrameAtUnix = nil // #987: a stale "last frame" freshness must not outlive it either } diff --git a/Strand/BLE/StrapChargeInference.swift b/Strand/BLE/StrapChargeInference.swift new file mode 100644 index 0000000000..32ca22bbea --- /dev/null +++ b/Strand/BLE/StrapChargeInference.swift @@ -0,0 +1,66 @@ +import Foundation +import StrandAnalytics + +/// The charging state the app is willing to STAND BEHIND — the strap's reported flag, corroborated (or +/// rescued) by the one thing that cannot lie: the state-of-charge going UP. +/// +/// Why this exists. The `battery_charging` flag is a single bit the decoder reads at a FIXED offset in the +/// BATTERY_LEVEL event (4.0 @26, 5.0 @30). For the 5.0 that offset is **unverified**: `decodeWhoop5Event`'s +/// own documentation states the deci-percent SoC was "confirmed by a clean monotonic discharge across a +/// real capture (49.9 → 47.7 %)" and makes no such claim for the charge bit — it is derived from the same +/// "+4 rule" as the other fields and has never been checked against a labelled charging session. Field +/// evidence says it is wrong or at least incomplete: a 5/MG sitting on its charging puck reported +/// `battery_charging = 0` while its SoC climbed. The `ch <= 1` guard fails closed on a wild byte, but it +/// cannot catch a byte that is plausibly 0 and simply means something else. +/// +/// So the app used to state "not charging" — and, worse, show a "~2 days left" DISCHARGE estimate — about a +/// strap that was visibly on the charger. That's an unverified bit being reported as fact. +/// +/// The fix is to stop treating the bit as the only witness. A rising SoC is direct, family-independent, +/// decoder-independent evidence of a charge, and this codebase ALREADY trusts exactly that inference: +/// `BatteryEstimator.chargeStepPct` is documented as "a SoC rise larger than this (percentage points) +/// between two consecutive readings marks a CHARGE", and the runtime estimate has always restarted its +/// discharge run on it. This reuses that same threshold rather than inventing a second notion of "charging". +/// +/// Strictly additive: a confirmed `true` is still `true`, and inference can only ever ADD a charge, never +/// deny one. If neither witness speaks, it answers `nil` (unknown) rather than asserting "not charging". +enum StrapChargeInference { + + /// A rise must be observed within this window to count as "charging NOW". The strap emits BATTERY_LEVEL + /// every ~8 min, so two consecutive readings are ~8 min apart; 20 min allows a missed event without + /// letting an hour-old charge keep claiming the strap is still on the puck. + static let recentRiseWindowSeconds = 20 * 60 + + /// Is the SoC series RISING fast enough to only be explicable as a charge? + /// + /// Uses `BatteryEstimator.chargeStepPct` (1.0 pp) between the two most recent readings, strictly + /// greater-than — which is what makes it robust against the quantisation seam: a 5/MG's live SoC comes + /// from 0x2A19 as a whole-percent u8, so consecutive readings can legitimately step by exactly 1.0 pp + /// without a charge. A real charge is nowhere near that subtle: a WHOOP 5 charges at roughly 50 pp/h + /// against a ~1.65 pp/h discharge, so a genuine charge moves several points per ~8-minute event. + static func isRising(samples: [(ts: Int, soc: Double)], + nowUnix: Int, + windowSeconds: Int = recentRiseWindowSeconds) -> Bool { + guard samples.count >= 2 else { return false } + let sorted = samples.sorted { $0.ts < $1.ts } + guard let last = sorted.last, let prev = sorted.dropLast().last else { return false } + guard nowUnix - last.ts <= windowSeconds else { return false } // stale: says nothing about NOW + guard last.ts - prev.ts <= windowSeconds else { return false } // a gap that wide isn't a slope + return (last.soc - prev.soc) > BatteryEstimator.chargeStepPct + } + + /// The charging state to display and to gate the runtime estimate on. + /// + /// - `true` — the strap said so, OR its charge is measurably climbing. + /// - `false` — the strap said not-charging AND the SoC is not climbing. Two witnesses agreeing. + /// - `nil` — no flag yet and no rise seen. Unknown, and the UI should say nothing rather than imply + /// "not charging" (which is exactly the false statement this whole type exists to stop). + static func resolve(flag: Bool?, + samples: [(ts: Int, soc: Double)], + nowUnix: Int, + windowSeconds: Int = recentRiseWindowSeconds) -> Bool? { + if flag == true { return true } + if isRising(samples: samples, nowUnix: nowUnix, windowSeconds: windowSeconds) { return true } + return flag // false (corroborated by a non-rising SoC) or nil (nothing to say) + } +} diff --git a/Strand/Liquid/LiquidTodayView.swift b/Strand/Liquid/LiquidTodayView.swift index ca77a66a07..57bdf1bc87 100644 --- a/Strand/Liquid/LiquidTodayView.swift +++ b/Strand/Liquid/LiquidTodayView.swift @@ -978,6 +978,7 @@ struct LiquidTodayView: View { } LiquidStrapBatteryRow() LiquidSyncStatusRow() + LiquidBackfillProgressRow() } } } @@ -1665,7 +1666,9 @@ private struct LiquidBatteryButton: View { @EnvironmentObject var live: LiveState @EnvironmentObject var router: NavRouter private var display: LiquidTodayView.StrapBatteryDisplay { - .resolve(connected: live.connected, batteryPct: live.batteryPct, charging: live.charging) + // `chargingEffective`, not the raw flag — the 5.0 charge bit is unverified and was observed reading + // 0 on a charging strap; a rising SoC is the witness that can't lie. See StrapChargeInference. + .resolve(connected: live.connected, batteryPct: live.batteryPct, charging: live.chargingEffective) } var body: some View { Button { router.openDevices() } label: { @@ -1738,9 +1741,9 @@ private struct LiquidBatteryButton: View { /// dropped the "~X days left" runtime estimate from the row directly above this one. /// /// Deliberately scoped to what LiveState can honestly answer: THAT a drain is running, how many chunks -/// it has pulled, and when one last completed. It does NOT yet say "~15h behind" — that needs the -/// persisted data frontier (max HR ts) compared against `strapRange.newestUnix`, and the frontier is a -/// Repository read that LiveState does not carry. That remains open in B1. +/// it has pulled, and when one last completed. The remaining half of B1 — "~15h behind", which needs the +/// persisted data frontier (max HR ts) compared against `strapRange.newestUnix` — is answered by +/// `LiquidBackfillProgressRow` directly below, which BLEManager now feeds via `LiveState.setDataFrontier`. private struct LiquidSyncStatusRow: View { @EnvironmentObject var live: LiveState var body: some View { @@ -1771,6 +1774,45 @@ private struct LiquidSyncStatusRow: View { } } +/// Strap-history recovery, inside the Data Sources card. Owns LiveState; display-only. +/// +/// The gap this closes: the app already said "Syncing strap history…" WHILE a session ran, but said nothing +/// about how far behind the strap was — and nothing at all between sessions, so an idle app looked finished +/// when it was actually 18 hours behind and waiting out a 15-minute floor. A user reasonably read that +/// silence as "my data is gone". Both states are now stated outright, and the numbers come from the pure +/// `BackfillProgress` so this row can never disagree with the drain's own idea of "behind". +private struct LiquidBackfillProgressRow: View { + @EnvironmentObject var live: LiveState + var body: some View { + let progress = live.backfillProgress + if let behind = progress.behindLabel, let frontier = progress.frontierUnix { + HStack { + VStack(alignment: .leading, spacing: 2) { + Text(isRecovering(progress) ? "Recovering strap history" : "Strap history pending") + .font(StrandFont.subhead).foregroundStyle(StrandPalette.textSecondary) + // The anchor that answers "is my data lost?" — we HAVE your data up to this instant. + Text("Have it through \(recoveredThrough(frontier))") + .font(StrandFont.caption).foregroundStyle(StrandPalette.textTertiary) + } + Spacer() + // Same shape as the Devices card's pill: the pure resolver hands over a plain label, the + // view localizes it. Pulsing IS the active/idle indicator — draining vs merely behind. + StatePill(LocalizedStringKey(behind), tone: .accent, pulsing: isRecovering(progress)) + } + } + } + private func isRecovering(_ p: BackfillProgress) -> Bool { + if case .recovering = p { return true } + return false + } + /// "July 14, 14:31" — a wall-clock instant the user can place against their own day. + private func recoveredThrough(_ unix: Int) -> String { + let f = DateFormatter() + f.dateFormat = "MMM d, HH:mm" + return f.string(from: Date(timeIntervalSince1970: TimeInterval(unix))) + } +} + /// The strap-battery readout inside the Data Sources card. Owns LiveState; display-only. private struct LiquidStrapBatteryRow: View { @EnvironmentObject var live: LiveState @@ -1790,7 +1832,10 @@ private struct LiquidStrapBatteryRow: View { /// Mac / Android pill and the classic Today badge. private func batteryText(pct: Double) -> String { let base = "\(Int(pct.rounded()))%" - if live.charging == true { return "\(base) · Charging" } + // `chargingEffective`, not the raw flag: the 5.0 charge bit is unverified and has been seen reading + // 0 on a strap that was demonstrably on its puck, which left this row showing a DISCHARGE estimate + // ("31% · ~2 days left") to a user who was watching it charge. A rising SoC overrides the bit. + if live.chargingEffective == true { return "\(base) · Charging" } if let est = estimateText { return "\(base) · \(est)" } return base } @@ -1799,7 +1844,7 @@ private struct LiquidStrapBatteryRow: View { /// Reproduced verbatim from `TodayView.estimateText`: under 48 h show hours, at two days or more round to /// days; nil (no banked discharge yet, or charging) hides it, so the row only ever shows an estimate we trust. private var estimateText: String? { - guard live.charging != true, let est = live.batteryEstimate else { return nil } + guard live.chargingEffective != true, let est = live.batteryEstimate else { return nil } let hours = est.hoursRemaining guard hours.isFinite, hours > 0 else { return nil } if hours < 48 { diff --git a/StrandTests/BackfillProgressTests.swift b/StrandTests/BackfillProgressTests.swift new file mode 100644 index 0000000000..caf1a81365 --- /dev/null +++ b/StrandTests/BackfillProgressTests.swift @@ -0,0 +1,72 @@ +import XCTest +@testable import Strand + +/// `BackfillProgress` — the honest answer to "is my data lost, or merely pending?". +/// +/// The gap this closes: the app said "Syncing strap history…" while a session ran and NOTHING between +/// sessions, so an idle app looked finished when it was 18 hours behind waiting out a 15-minute floor. Both +/// numbers were already tracked (the strap's reported newest, and our persisted frontier) — nothing ever +/// subtracted them and said so. +final class BackfillProgressTests: XCTestCase { + private let now = 1_800_000_000 + private var frontier18hBehind: Int { now - 18 * 3600 } + + /// The user's real case: 18h behind, actively draining. + func testRecovering() { + let p = BackfillProgress.resolve(connected: true, backfilling: true, + frontierUnix: frontier18hBehind, strapNewestUnix: now) + XCTAssertEqual(p, .recovering(behindSeconds: 18 * 3600, frontierUnix: frontier18hBehind)) + XCTAssertEqual(p.behindLabel, "~18h behind") + XCTAssertEqual(p.frontierUnix, frontier18hBehind) + } + + /// The state that used to be INVISIBLE: behind, but idle between sessions. Silence read as "done". + func testPending_isDistinctFromRecovering() { + let p = BackfillProgress.resolve(connected: true, backfilling: false, + frontierUnix: frontier18hBehind, strapNewestUnix: now) + XCTAssertEqual(p, .pending(behindSeconds: 18 * 3600, frontierUnix: frontier18hBehind)) + XCTAssertEqual(p.behindLabel, "~18h behind") + } + + /// Caught up uses the SAME threshold the drain stops at, so the readout can never say "behind" about a + /// gap `BackfillContinuation` has already decided is closed. + func testUpToDate_matchesTheDrainsOwnThreshold() { + let gap = BackfillContinuation.defaultBehindGapSeconds + XCTAssertEqual(BackfillProgress.resolve(connected: true, backfilling: false, + frontierUnix: now - gap, strapNewestUnix: now), + .upToDate) + XCTAssertEqual(BackfillProgress.resolve(connected: true, backfilling: false, + frontierUnix: now - gap - 1, strapNewestUnix: now) + .behindSeconds, gap + 1) + XCTAssertNil(BackfillProgress.resolve(connected: true, backfilling: false, + frontierUnix: now, strapNewestUnix: now).behindLabel) + } + + /// Not connected, or the strap hasn't answered a range — claim nothing. + func testUnknown() { + XCTAssertEqual(BackfillProgress.resolve(connected: false, backfilling: false, + frontierUnix: frontier18hBehind, strapNewestUnix: now), + .unknown) + XCTAssertEqual(BackfillProgress.resolve(connected: true, backfilling: false, + frontierUnix: nil, strapNewestUnix: now), + .unknown) + XCTAssertEqual(BackfillProgress.resolve(connected: true, backfilling: false, + frontierUnix: frontier18hBehind, strapNewestUnix: nil), + .unknown) + } + + /// A frontier far AHEAD of the strap's reported newest is a stale/wrong-epoch range (#451) or a + /// future-dated clock (#928) — not negative backlog. Say nothing rather than a nonsense figure. + func testFrontierAheadOfStrap_claimsNothing() { + XCTAssertEqual(BackfillProgress.resolve(connected: true, backfilling: false, + frontierUnix: now, strapNewestUnix: now - 30 * 24 * 3600), + .unknown) + } + + /// Sub-hour gaps read in minutes, so a nearly-caught-up drain doesn't round to a bare "~0h behind". + func testMinuteLabel() { + XCTAssertEqual(BackfillProgress.resolve(connected: true, backfilling: true, + frontierUnix: now - 42 * 60, strapNewestUnix: now) + .behindLabel, "~42m behind") + } +} diff --git a/StrandTests/StrapChargeInferenceTests.swift b/StrandTests/StrapChargeInferenceTests.swift new file mode 100644 index 0000000000..a6bf10b730 --- /dev/null +++ b/StrandTests/StrapChargeInferenceTests.swift @@ -0,0 +1,85 @@ +import XCTest +import StrandAnalytics +@testable import Strand + +/// `StrapChargeInference` — the charging state the app is willing to stand behind. +/// +/// The bug: a 5/MG on its charging puck reported `battery_charging = 0` while its SoC climbed, so the app +/// stated "not charging" and showed a "~2 days left" DISCHARGE estimate to a user watching it charge. The +/// 5.0 charge bit's offset (@30) is unverified — the decoder's own docs confirm only the SoC and mV fields +/// against a real capture — so the bit alone is not a fact worth reporting. +final class StrapChargeInferenceTests: XCTestCase { + private let now = 1_800_000_000 + /// Two readings ~8 min apart, the strap's real BATTERY_LEVEL cadence. + private func series(_ a: Double, _ b: Double, gap: Int = 8 * 60, age: Int = 60) -> [(ts: Int, soc: Double)] { + [(ts: now - age - gap, soc: a), (ts: now - age, soc: b)] + } + + /// THE bug: the flag says "not charging" and the SoC is visibly climbing. Trust the charge, not the bit. + func testRisingSoc_overridesAFalseFlag() { + XCTAssertEqual(StrapChargeInference.resolve(flag: false, samples: series(31.6, 38.2), nowUnix: now), + true) + } + + /// Same rescue when the flag never arrives at all. + func testRisingSoc_rescuesASilentFlag() { + XCTAssertEqual(StrapChargeInference.resolve(flag: nil, samples: series(31.6, 38.2), nowUnix: now), + true) + } + + /// A confirmed `true` is always honoured, no corroboration needed. + func testConfirmedFlagWins() { + XCTAssertEqual(StrapChargeInference.resolve(flag: true, samples: series(50, 49), nowUnix: now), true) + XCTAssertEqual(StrapChargeInference.resolve(flag: true, samples: [], nowUnix: now), true) + } + + /// A normal discharge is not a charge — the estimate must keep working. + func testDischarging_isNotCharging() { + XCTAssertEqual(StrapChargeInference.resolve(flag: false, samples: series(50, 49.7), nowUnix: now), + false) + } + + /// Nothing to go on ⇒ nil (unknown), NOT false. The UI must be able to say nothing rather than assert + /// "not charging" — the false statement this whole type exists to stop. + func testNoEvidence_isUnknownNotFalse() { + XCTAssertNil(StrapChargeInference.resolve(flag: nil, samples: [], nowUnix: now)) + XCTAssertNil(StrapChargeInference.resolve(flag: nil, samples: series(50, 50), nowUnix: now)) + } + + /// The quantisation seam: a 5/MG's live SoC arrives from 0x2A19 as a whole-percent u8, so consecutive + /// readings can legitimately step by exactly 1.0 pp with no charge. The threshold is strictly + /// greater-than `chargeStepPct`, so a 1.0 step must NOT read as charging. + func testWholePercentStep_isNotMistakenForACharge() { + XCTAssertEqual(StrapChargeInference.resolve(flag: false, samples: series(31, 32), nowUnix: now), + false) + XCTAssertFalse(StrapChargeInference.isRising(samples: series(31, 32), nowUnix: now)) + // A real charge is nowhere near that subtle (~50 pp/h vs ~1.65 pp/h discharge). + XCTAssertTrue(StrapChargeInference.isRising(samples: series(31, 38), nowUnix: now)) + } + + /// The threshold is BatteryEstimator's existing charge step, not a second invented notion of "charging". + func testUsesTheEstimatorsOwnChargeThreshold() { + let justUnder = BatteryEstimator.chargeStepPct + let justOver = BatteryEstimator.chargeStepPct + 0.2 + XCTAssertFalse(StrapChargeInference.isRising(samples: series(50, 50 + justUnder), nowUnix: now)) + XCTAssertTrue(StrapChargeInference.isRising(samples: series(50, 50 + justOver), nowUnix: now)) + } + + /// A rise from an hour ago says nothing about NOW — the strap may already be off the puck. + func testStaleRise_doesNotClaimChargingNow() { + let old = [(ts: now - 7200, soc: 31.0), (ts: now - 6800, soc: 40.0)] + XCTAssertFalse(StrapChargeInference.isRising(samples: old, nowUnix: now)) + XCTAssertEqual(StrapChargeInference.resolve(flag: false, samples: old, nowUnix: now), false) + } + + /// Two readings separated by a huge gap aren't a slope — a strap that reconnects hours later at a higher + /// SoC was charged in the meantime, but is not necessarily charging now. + func testWideGapBetweenReadings_isNotASlope() { + XCTAssertFalse(StrapChargeInference.isRising(samples: series(31, 90, gap: 4 * 3600), nowUnix: now)) + } + + /// A single reading can't show a trend. + func testSingleReading_isNotARise() { + XCTAssertFalse(StrapChargeInference.isRising(samples: [(ts: now, soc: 50)], nowUnix: now)) + } +} From 09e0ac4741f4ff35d3b39b9d985413ab9f34922f Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 26 Jul 2026 21:44:15 -0400 Subject: [PATCH 126/171] =?UTF-8?q?docs(bug-sweep):=202026-07-15=20?= =?UTF-8?q?=E2=80=94=20record=20the=20BLE=20link-recovery=20port=20(A1/A2/?= =?UTF-8?q?A4/A6/B1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third pass on the sweep. An abandoned local `main` line (diverged 2026-07-14, never merged) held the only copy of work against five findings; it has been reconciled onto origin/main as `ble-link-recovery-port`. Updated here: * A1 battery read never re-issued -> FIXED (StrapBatteryReadPolicy; #77 gate + throttle) * A2 "Active · Live" overstates -> FIXED (LiveState.linkProven, defaults true) * A4 zombie restored peripheral -> BOUNDED (StalledHandshakePolicy, 90 s fuse). The MECHANISM is now traced and real even though the original 40-min claim was never re-verified: willRestoreState never reaches didWriteValueFor, the only caller of startKeepAlive(), so the 120 s liveness watchdog is never armed. * A6 5/MG charge bit -> stays OPEN (still needs a controlled capture), but MITIGATED: StrapChargeInference adds a rising-SoC witness so a wrong bit no longer shows "not charging" plus a discharge estimate on a strap that is charging. * B1 "how far behind" half -> FIXED (BackfillProgress). The plumbing objection recorded there was correct and is answered: BLEManager already reads the frontier once per session for the auto-continue decision and now publishes it, so no new Repository read was needed. A5 gains a note rather than a status change: that line ALSO re-raised the cap 6 -> 120, resting on the §0 contaminated measurement, and it was refused for the third time. The `isDeepDrain` gate it used is the shape A5 invites — the mechanism isn't what was rejected, the evidence is. Nothing here is strap-validated; both app targets compile and the pure tests pass, which per CLAUDE.md proves nothing about BLE behaviour. Co-Authored-By: Claude Opus 5 --- ...15-strap-battery-backfill-observability.md | 120 ++++++++++++++++-- 1 file changed, 106 insertions(+), 14 deletions(-) diff --git a/docs/bugs/2026-07-15-strap-battery-backfill-observability.md b/docs/bugs/2026-07-15-strap-battery-backfill-observability.md index a1fb56e9c9..bedb0524e2 100644 --- a/docs/bugs/2026-07-15-strap-battery-backfill-observability.md +++ b/docs/bugs/2026-07-15-strap-battery-backfill-observability.md @@ -10,6 +10,21 @@ unable to tell the wearer any of it. > started repeating them. Line numbers are re-derived against `capture-deep` at triage time; where the > original pointer had drifted, the drift is noted. Findings not re-verified in this pass say so. +> **Third pass, 2026-07-26 — BLE link-recovery port.** An abandoned local `main` line (diverged +> 2026-07-14, never merged) held unique work against **A1, A2, A4, A6 and B1**. It was reconciled onto +> `origin/main` as `ble-link-recovery-port`; those five sections are updated below. Two things in that +> line were **deliberately not ported**, both cases of the same trap — a change `origin/main` had already +> examined and reversed, which a naive merge would silently resurrect: +> +> 1. the **A5 cap raise** (6 → 120), whose rationale is the §0 contaminated measurement; and +> 2. a rival `StrapBatteryDisplay` that let a stale percent outlive a disconnect — i.e. **B2**, which A3's +> resolver had deliberately fixed. +> +> A third such change in that line (`Offload: stop archiving v20/v21 as "undecodable"`) is **E2**, already +> reverted on `origin/main` as harmful; it was not ported either. Nothing in this pass is strap-validated — +> both app targets compile and the pure tests pass, which per `CLAUDE.md` proves nothing about BLE +> behaviour. + | | Meaning | |---|---| | **CONFIRMED** | re-derived from current code; still true | @@ -57,7 +72,7 @@ findings that read *code structure* are not. ## A. BLE / battery / connection -### A1. Battery % on a 5/MG — **headline REFUTED, narrower defect CONFIRMED · OPEN** +### A1. Battery % on a 5/MG — **headline REFUTED, narrower defect CONFIRMED · FIXED (needs strap validation)** **REFUTED: "Battery % never lands on a 5.0/MG."** The doc's own A6 evidence disproves it — a `soc=31.6` reading exists for 2026-07-15. Two mechanisms the first pass missed: @@ -84,11 +99,21 @@ of `nil` per connect** — long enough to look like "never" to a wearer, and the source (#77)"). The decode exists. The #77 stub-100 hazard is **4.0-only**, so gating on family would give a 5/MG a self-healing ~8-min source. -**Not fixed here.** Both candidate fixes are on the BLE connect path, which per `CLAUDE.md` "must be -validated on a real strap; compile-success proves nothing" — and a live measurement is in progress. -**Needs the user.** +**FIXED** (`Strand/BLE/StrapBatteryReadPolicy.swift`, pinned by `StrandTests/StrapBatteryReadPolicyTests.swift`): +`enableLiveNotifications` now also calls `retryBatteryReadIfDue`, so every moment that already +re-subscribes notifies — post-bond 5/MG, post-bond 4.0, keep-alive, manual refresh, realtime start — is +also a moment the read can self-heal. The policy owns the two constraints that make that safe: the **#77 +family gate** (never read a 4.0's stub-100 characteristic, so the fix cannot flash a fake 100% over a real +charge) and a 30 s throttle, which is load-bearing because the call site re-enters on **every** +`.withResponse` ack, including every HISTORY_END during a multi-minute offload. + +Fix 1b (using the 5.0 BATTERY_LEVEL event's discarded `battery_pct` as a second source) was **not** taken — +the re-issued read closes the same gap without touching `FrameRouter`'s #77 invariant. + +**Still needs strap validation:** this is the BLE connect path, and per `CLAUDE.md` compile-success proves +nothing here. Both app targets build and the pure policy tests pass; the end-to-end behaviour is unverified. -### A2. "Active · Live" overstates a raw link — **CONFIRMED · OPEN** +### A2. "Active · Live" overstates a raw link — **CONFIRMED · FIXED (needs strap validation)** `state.connected = true` at `BLEManager.swift:3108` in `didConnect`, **before** `discoverPrimaryServices` — true the instant the raw link object exists, before any characteristic or @@ -100,7 +125,19 @@ The card already carries a narrower guard for the *bond-refused* case (`DevicesV #221) — which shows the shape of the fix — but nothing gates on data actually flowing. `LiveState` has the ingredient: `lastFrameAtUnix` (`:203-207`, #987). -**Not fixed here** — BLE/connect semantics, needs strap validation. **Needs the user.** +**FIXED:** `LiveState.linkProven` (`Strand/BLE/LiveState.swift`) — has this link actually carried a byte, +or completed its handshake? BLEManager clears it in `didConnect` and `willRestoreState`, sets it on first +data or a completed handshake, and restores it on disconnect. `DevicePillState.resolve` takes it as a new +`linkProven:` parameter and gates only the "· Live" half, so an unproven link falls back to plain +"Active" — the honest statement — rather than inventing a new pill state. + +The parameter **defaults `true`**, which is what keeps this narrow: the four non-WHOOP sources (Oura, +Huami, FTMS, standard-HR) also drive `connected`, and each already sets it only once genuinely subscribed. +They never touch the flag, so they keep an inert `true` instead of being regressed into looking offline. +Priority order is unchanged — the #221 bond-refused pill still wins. Pinned by +`StrandTests/DevicePillStateTests.swift`. + +**Still needs strap validation** (connect/restore semantics). ### A3. Charging was structurally unrenderable without a charge % — **CONFIRMED · FIXED** @@ -127,7 +164,7 @@ a **fresh** charging bit and rendered a crossed-out bolt. is now visually distinct from "no link"; the accessibility label no longer says a bare "Strap battery" for three different states. -### A4. Zombie restored peripheral spins forever — **NOT RE-VERIFIED · OPEN** +### A4. Zombie restored peripheral spins forever — **NOT RE-VERIFIED · BOUNDED (needs strap validation)** First-pass claim: after iOS state-restoration returns a dead peripheral, the app marks it connected, the handshake never completes, and it retries the **same dead handle** indefinitely — 40+ minutes of @@ -135,7 +172,24 @@ handshake never completes, and it retries the **same dead handle** indefinitely Re-verified only that the log line exists on both platforms (`BLEManager.swift:1562`, `WhoopBleClient.kt:4893`). The restoration/retry behaviour itself was **not** re-traced this pass, and -it is not strap-testable right now. Treat as unconfirmed. **Needs the user.** +it is not strap-testable right now. Treat the *claim* as unconfirmed. + +**The mechanism WAS since traced, and it is real** (Swift side): `willRestoreState` marks the link up and +re-discovers services, but the CLIENT_HELLO / bond write that follows is `.withResponse`. On a wedged +strap it is never acked, so `didWriteValueFor` never fires — and that is the **only** caller of +`startKeepAlive()`, so the 120 s liveness watchdog inside `keepAliveFire` (the machinery that would have +bounced exactly this link) is never armed. Nothing else bounds the retry. + +**BOUNDED:** `Strand/BLE/StalledHandshakePolicy.swift` arms a 90 s fuse at connect **and at restore**, then +cancels the peripheral so the existing 3 s disconnect-rescan recovers the link. It deliberately adds no new +reconnect mechanism. The fuse is picked against the fuses already in the tree — above the 60 s reboot-settle +backstop, below the 120 s liveness fuse — and every input to `shouldRecover` is a reason to stay out of the +way (`sawData`, `handshakeDone`, `intentionalDisconnect`, `autoReconnectPaused`, `bondRefused`), so it fires +only on the narrow case nothing else owns. Pinned by `StrandTests/StalledHandshakePolicyTests.swift`. + +This bounds the failure mode whether or not the original 40-minute claim is ever reproduced. **Kotlin twin +not written** — Android has no `willRestoreState` equivalent, so the trigger does not exist there. +**Still needs strap validation.** ### A5. `maxAutoContinues = 6` throttled a healthy recovery — **REFUTED · DECIDED: the cap stays at 6** @@ -152,6 +206,15 @@ root cause was known. The reasoning is now recorded at the constant on both plat - **It applies to WHOOP 4.0 too**, where a long drain starves the realtime keep-alive (`guard !backfilling`, #160). +> **It was re-raised a third time, and refused (2026-07-26).** The abandoned local `main` line carried a +> commit raising the cap to 120 behind an `isDeepDrain` evidence gate. Its rationale — "each session +> recovers ~29 min of strap history, so an 18-hour backlog needs ~37 sessions" — **is the §0 contaminated +> measurement**, taken at 28x banking. When that line's unique work was ported onto `origin/main`, the cap +> change was dropped and `BackfillContinuation` left byte-identical; `StrandTests/BackfillContinuationTests` +> (21 tests) passes unchanged. Note the gate itself was the shape this section invites ("a conditional +> bypass while provably behind and advancing"), so the *mechanism* is not what was rejected — the +> **evidence** is. A raise still needs a deep-backlog case demonstrated on a **default** strap. + The "~12h+ to recover a night" that made this finding look severe was measured **at 28x** (§0). Remove the 28x and the arithmetic that condemned the cap evaporates. @@ -201,7 +264,24 @@ The finding's own evidence contradicts its title. The row was `soc=31.6, chargin **OPEN (reframed):** whether the 5/MG's BATTERY_LEVEL bit0 tracks charging **at all** is genuinely unknown and worth settling — but it needs a controlled capture (put it on the charger, watch `state.charging` change), not a single historical row. A3's fix is what makes that flag visible enough -to observe. +to observe. **This stays OPEN: nothing below settles the question.** + +**MITIGATED, so a wrong bit stops costing the wearer:** `Strand/BLE/StrapChargeInference.swift` stops +treating that one unverified bit as the only witness. A **rising SoC** is direct, family-independent, +decoder-independent evidence of a charge, and this codebase already trusts exactly that inference — +`BatteryEstimator.chargeStepPct` is documented as "a SoC rise larger than this marks a CHARGE", and the +runtime estimate has always restarted its discharge run on it. `LiveState.chargingEffective` resolves the +flag against that second witness; the UI reads it, so a strap on the puck reporting `charging=0` no longer +renders "not charging" **and** no longer shows a "~2 days left" discharge estimate. + +Strictly additive, and deliberately so: a confirmed `true` stays `true`, inference can only ever ADD a +charge and never deny one, and with neither witness it answers `nil` (unknown) rather than asserting +"not charging". **`LiveState.charging` remains the raw WIRE value** and is what gets persisted — so no +stored value diverges from Kotlin and no `.noopbak` contract moves. Pinned by +`StrandTests/StrapChargeInferenceTests.swift`. + +Because it is display-only, this **does not** substitute for the controlled capture — it makes the app +correct either way while the question stays open. Kotlin twin is a UI-parity item, not a data one. ### A7. `lastSeenAt` frozen at pairing → "last seen 18 days ago" — **CONFIRMED · OPEN (both platforms)** @@ -228,7 +308,7 @@ link: > "All of this stuff, like backfill progress, should be something you can see on the app. It is kind of > misleading me right now." -### B1. Backfill progress invisible — **PARTLY REFUTED · iOS half FIXED · remainder OPEN** +### B1. Backfill progress invisible — **PARTLY REFUTED · iOS half FIXED · remainder now FIXED** **REFUTED as "the app never surfaces it."** It does, on macOS and in the menu bar: `SyncingHistoryNote` ("Syncing strap history… N chunks pulled", `Strand/Screens/ScreenScaffold.swift:199`) @@ -244,10 +324,22 @@ all**. Not a missing feature — a **v8 redesign regression**, the same class as **FIXED (iOS):** `LiquidSyncStatusRow` in the Data Sources card — "Strap history / Syncing… N chunks" while `live.backfilling`, else "Synced N ago" from `live.lastSyncedAt`. -**OPEN — the "how far behind" half.** *"Recovering strap history — July 14 14:31, ~15h behind"* still -isn't possible: it needs our persisted frontier (max HR ts) compared against -`live.strapRange?.newestUnix`. `strapRange` is in `LiveState` (`:160-176`); **the frontier is not** — it -is a Repository read. Plumbing it is a real change, not a display tweak, so it was not half-built here. +**FIXED — the "how far behind" half** (2026-07-26). *"Recovering strap history / Have it through July 14, +14:31 — ~15h behind"* now renders, via `LiquidBackfillProgressRow` over the pure +`Strand/BLE/BackfillProgress.swift`. + +The plumbing objection above was right, and this is how it was resolved without a new Repository read: +BLEManager **already** reads the frontier (`collector.latestHRSampleTs()`) once per offload session, for +the auto-continue decision — exactly the cadence the readout wants. It now publishes it there via +`LiveState.setDataFrontier`, so `backfillProgress` computes over `connected` / `backfilling` / +`dataFrontierUnix` / `strapRange`, all already `@Published`. No extra query, no third piece of state. + +Two properties worth keeping: the resolver reuses `BackfillContinuation.defaultBehindGapSeconds`, so the +readout can never claim "behind" about a gap the drain has already decided is closed; and a frontier +*ahead* of the strap's reported newest (the #451 stale-epoch and #928 future-clock cases) resolves to +`.unknown` rather than a negative backlog. It also answers **between** sessions — `.pending` — which is the +state the wearer actually misread as "my data is gone". `LiquidSyncStatusRow` is kept and still answers +*that* a drain is running. Pinned by `StrandTests/BackfillProgressTests.swift`. ### B2. Battery shown with no staleness marker — **CONFIRMED · FIXED (worst case)** From 3c4a837d39a4ba66c601fb93282ef9ab0c6a3467 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 26 Jul 2026 22:39:13 -0400 Subject: [PATCH 127/171] fix(i18n): localize the backfill-progress row's three strings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Caught by Tools/i18n_audit.py on PR #4, and it was a real defect: the ported LiquidBackfillProgressRow used bare `Text("literal")`, so all three of its strings sat outside the String Catalog and shipped English-only in every locale. The audit flagged only one of the three — its regex matches `Text("` followed directly by a quote, so the ternary (`Text(isRecovering(progress) ? "…" : "…")`) slipped past it. All three are fixed, not just the flagged one. Switched to `String(localized:)`, matching LiquidSyncStatusRow directly above, and added the keys to Strand/Resources/Localizable.xcstrings with the same eight languages and states as their neighbour "Strap history" (de/es/fr/it/ru/zh-Hans/zh-Hant translated, pt-PT needs_review). The de invariant stays fully covered and no new es/fr gaps are added. Appended to the catalog rather than re-serialising it: the file's keys are not sorted, so a round-trip through a sorting writer rewrites ~1000 lines. This is +156/-0. Verification: `python3 Tools/i18n_audit.py --ci origin/main` exits 0; Strand (macOS) and NOOPiOS both BUILD SUCCEEDED. Co-Authored-By: Claude Opus 5 --- Strand/Liquid/LiquidTodayView.swift | 9 +- Strand/Resources/Localizable.xcstrings | 156 +++++++++++++++++++++++++ 2 files changed, 163 insertions(+), 2 deletions(-) diff --git a/Strand/Liquid/LiquidTodayView.swift b/Strand/Liquid/LiquidTodayView.swift index 57bdf1bc87..39a2fb860b 100644 --- a/Strand/Liquid/LiquidTodayView.swift +++ b/Strand/Liquid/LiquidTodayView.swift @@ -1788,10 +1788,15 @@ private struct LiquidBackfillProgressRow: View { if let behind = progress.behindLabel, let frontier = progress.frontierUnix { HStack { VStack(alignment: .leading, spacing: 2) { - Text(isRecovering(progress) ? "Recovering strap history" : "Strap history pending") + // `String(localized:)`, matching `LiquidSyncStatusRow` directly above — a bare + // `Text("literal")` here relies on Xcode's build-time extraction and left these three + // strings out of the String Catalog entirely (caught by Tools/i18n_audit.py). + Text(isRecovering(progress) + ? String(localized: "Recovering strap history") + : String(localized: "Strap history pending")) .font(StrandFont.subhead).foregroundStyle(StrandPalette.textSecondary) // The anchor that answers "is my data lost?" — we HAVE your data up to this instant. - Text("Have it through \(recoveredThrough(frontier))") + Text(String(localized: "Have it through \(recoveredThrough(frontier))")) .font(StrandFont.caption).foregroundStyle(StrandPalette.textTertiary) } Spacer() diff --git a/Strand/Resources/Localizable.xcstrings b/Strand/Resources/Localizable.xcstrings index 70652d7d61..57fc3f6eef 100644 --- a/Strand/Resources/Localizable.xcstrings +++ b/Strand/Resources/Localizable.xcstrings @@ -164635,6 +164635,162 @@ } } } + }, + "Recovering strap history": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Strap-Verlauf wird wiederhergestellt" + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Recuperando el historial de la pulsera" + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Récupération de l'historique du bracelet" + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Recupero della cronologia della fascia" + } + }, + "pt-PT": { + "stringUnit": { + "state": "needs_review", + "value": "A recuperar o histórico da pulseira" + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Восстановление истории браслета" + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "正在恢复手环历史数据" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "正在復原手環歷史資料" + } + } + } + }, + "Strap history pending": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Strap-Verlauf ausstehend" + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Historial de la pulsera pendiente" + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Historique du bracelet en attente" + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Cronologia della fascia in sospeso" + } + }, + "pt-PT": { + "stringUnit": { + "state": "needs_review", + "value": "Histórico da pulseira pendente" + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "История браслета ожидает" + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "手环历史数据待同步" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "手環歷史資料待同步" + } + } + } + }, + "Have it through %@": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Vorhanden bis %@" + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Datos hasta %@" + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Données jusqu'à %@" + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Dati fino a %@" + } + }, + "pt-PT": { + "stringUnit": { + "state": "needs_review", + "value": "Dados até %@" + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Данные по %@" + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "已获取至 %@" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "已取得至 %@" + } + } + } } }, "version": "1.0" From 06184fe6e5731c376bc448bdc61205c44c32e9bd Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 16 Jul 2026 13:59:20 -0700 Subject: [PATCH 128/171] fix(liquid-today): carry SpO2 + skin-temp per-field like classic Liquid Today read raw vitals fields where classic Today carries a prior day's value, so Blood Oxygen and Skin Temp blanked on an unscored-today (rollover) day. Same class of regression as the B3 Charge carry (9251ae24). The engine writes spo2Pct / skinTempDevC = nil on every computed "-noop" row, so the whole-row carry (lastVitalsDay, which gates on HRV/RHR/resp) lands on a null field while an older imported row still holds a real reading. Classic resolves this with a third, per-field fallback (lastSpo2Day / lastSkinTempDay); Liquid never called them. - Add a pure LiquidTodayView.perFieldVital(today:vitalsCarry:fieldCarry:) -- today's own reading, then whole-row carry, then per-field carry -- so all three tiles (Key-Metrics SpO2 + the two "Your cards") share one source of truth and can't drift from classic's fallthrough. - Wire the Key-Metrics SpO2 tile (was 2-level) and the two YOUR CARDS tiles (Blood Oxygen + Skin Temp, were hardcoded "-") through it, byte-matching classic dashboardCardValue formatting. - Resolve the carries ONCE in load() into cachedSpo2Day / cachedSkinTempDay (never in body) per Liquid's O(1)-read perf rule; reuse the existing pure Repository.lastSpo2Day / lastSkinTempDay selectors. No Kotlin twin: Android already carries lastSpo2/lastSkinTempRow; this is UI wiring, no analytics change. Impact is limited to WHOOP 4.0 / SpO2-import users on a rollover day (WHOOP 5/MG has no SpO2 sensor; scored rows already carry skinTempDevC). Tests: new StrandTests/LiquidVitalsCarryTests.swift pins the coalesce order and both per-field rollovers (null vitals row -> imported reading). App-target Swift isn't covered by CI, so built the Strand (macOS) + NOOPiOS schemes and ran StrandTests locally: 1038 tests, 0 failures. Co-Authored-By: Claude Opus 4.8 --- Strand/Liquid/LiquidTodayView.swift | 44 ++++++++++-- StrandTests/LiquidVitalsCarryTests.swift | 90 ++++++++++++++++++++++++ 2 files changed, 130 insertions(+), 4 deletions(-) create mode 100644 StrandTests/LiquidVitalsCarryTests.swift diff --git a/Strand/Liquid/LiquidTodayView.swift b/Strand/Liquid/LiquidTodayView.swift index ca77a66a07..309d670f88 100644 --- a/Strand/Liquid/LiquidTodayView.swift +++ b/Strand/Liquid/LiquidTodayView.swift @@ -79,6 +79,13 @@ struct LiquidTodayView: View { /// today's row has no vitals yet, so these fall back to the last night that recorded them. Never /// resolved in body — body rescans repo.days ~23× per pass, and this cache keeps that read O(1). @State private var cachedVitalsDay: DailyMetric? + /// PER-FIELD SpO₂ / skin-temp carries — the twins of `cachedVitalsDay` for the two fields its predicate + /// does NOT check (it gates on HRV / RHR / respiratory). The engine writes `spo2Pct` / `skinTempDevC = + /// nil` on every computed "-noop" row, so the whole-row carry lands on a null value while an older imported + /// row holds a real one; these resolve each field from its own freshest strictly-prior row. Resolved ONCE + /// in load() (never in body) so the Blood-Oxygen / Skin-Temp tiles read O(1); non-nil only on today. + @State private var cachedSpo2Day: DailyMetric? + @State private var cachedSkinTempDay: DailyMetric? /// The Charge hero's resolved state (#543 carry + the honest label), resolved ONCE in load() alongside /// the other caches. It composes `TodayView.lastScoredRecoveryDay`, which is O(days) — exactly the scan /// this cache exists to keep out of body. Never resolved in body. @@ -132,9 +139,25 @@ struct LiquidTodayView: View { /// The prior-day vitals carry (see `cachedVitalsDay`), read O(1) from the cache. Non-nil only at /// offset 0 (today); a navigated past day carries nothing (its own row is the whole story). private var vitalsDay: DailyMetric? { cachedVitalsDay } + /// The per-field SpO₂ carry (see `cachedSpo2Day`), read O(1) from the cache. + private var spo2Day: DailyMetric? { cachedSpo2Day } + /// The per-field skin-temperature carry (see `cachedSkinTempDay`), read O(1) from the cache. + private var skinTempDay: DailyMetric? { cachedSkinTempDay } /// The Charge hero's resolved state (see `cachedChargeDisplay`), read O(1) from the cache. private var chargeDisplay: ChargeDisplay { cachedChargeDisplay } + /// The value a per-field vitals tile (Blood Oxygen, Skin Temp) draws: today's own reading wins, else the + /// whole-row vitals carry's copy of the field, else the PER-FIELD carry — the freshest strictly-prior row + /// that actually HAS the field. The whole-row carry (`vitalsDay`) gates on HRV / resting-HR / respiratory, + /// so it lands on a computed "-noop" row whose `spo2Pct` / `skinTempDevC` is engine-written nil while an + /// older imported row holds a real reading; the per-field carry (`spo2Day` / `skinTempDay`) resolves that. + /// Pure so the today-first → whole-row → per-field ordering is pinned without a live view, mirroring + /// classic `TodayView.carriedVital`'s fallthrough and the Android per-field read. Named so all three tiles + /// (Key-Metrics SpO₂ + the two "Your cards") share one source of truth and can't drift. + static func perFieldVital(today: Double?, vitalsCarry: Double?, fieldCarry: Double?) -> Double? { + today ?? vitalsCarry ?? fieldCarry + } + /// The actual O(days) resolution. Offset 0 prefers live repo.today; past offsets look up. Run ONCE /// per data/day change from load(), never from body. private func resolveDisplayDay() -> DailyMetric? { @@ -623,12 +646,18 @@ struct LiquidTodayView: View { cardLink(.metric("steps_est"), title: card.title, sub: card.subtitle, value: stepsText, tint: StrandPalette.metricCyan, frac: fracOver(stepCount, 10000)) case .bloodOxygen: - // Not wired to a real read yet — render EMPTY (not half-full) so it doesn't imply a reading. + // Per-field carry (today → whole-row vitals carry → the last row that HAS a %), matching the + // Key-Metrics tile + classic dashboardCardValue so the surfaces agree instead of a bare "–". + let spo2 = Self.perFieldVital(today: displayDay?.spo2Pct, vitalsCarry: vitalsDay?.spo2Pct, fieldCarry: spo2Day?.spo2Pct) cardLink(.metric("spo2"), title: card.title, sub: card.subtitle, - value: "–", tint: StrandPalette.metricCyan, frac: nil) + value: spo2.map { String(format: "%.0f%%", $0) } ?? "–", tint: StrandPalette.metricCyan, + frac: fracOver(spo2, 100)) case .skinTemp: + // Per-field carry, matching classic dashboardCardValue. A signed deviation from baseline (°C), + // shown signed; a deviation has no natural 0–1 fill, so the vessel stays empty (as classic does). + let skin = Self.perFieldVital(today: displayDay?.skinTempDevC, vitalsCarry: vitalsDay?.skinTempDevC, fieldCarry: skinTempDay?.skinTempDevC) cardLink(.metric("skin_temp"), title: card.title, sub: card.subtitle, - value: "–", tint: StrandPalette.metricAmber, frac: nil) + value: skin.map { String(format: "%+.1f°", $0) } ?? "–", tint: StrandPalette.metricAmber, frac: nil) case .calories: cardLink(.metric("active_kcal"), title: card.title, sub: card.subtitle, value: "–", tint: StrandPalette.metricAmber, frac: nil) @@ -857,7 +886,9 @@ struct LiquidTodayView: View { case .restingHr: ktile(String(localized: "Rest HR"), intText(rhr), "bpm", StrandPalette.metricRose, fracOver(rhr, 100), key: "rhr") case .bloodOxygen: - let spo2 = displayDay?.spo2Pct ?? vitalsDay?.spo2Pct + // Per-field carry: today → whole-row vitals carry → the last row that actually HAS a % (computed + // "-noop" rows write spo2Pct = nil), so the tile doesn't blank at the rollover. Matches classic. + let spo2 = Self.perFieldVital(today: displayDay?.spo2Pct, vitalsCarry: vitalsDay?.spo2Pct, fieldCarry: spo2Day?.spo2Pct) ktile(String(localized: "Blood Oxygen"), intText(spo2), "%", StrandPalette.metricCyan, fracOver(spo2, 100), key: "spo2") case .respiratory: let resp = displayDay?.respRateBpm ?? vitalsDay?.respRateBpm @@ -1022,6 +1053,11 @@ struct LiquidTodayView: View { // echo today's still-forming row; only on today (a past day's own row is the whole story). let tkey = cachedDisplayDay?.day ?? selectedDayKey cachedVitalsDay = (selectedDayOffset == 0) ? Repository.lastVitalsDay(days: repo.days, todayKey: tkey) : nil + // PER-FIELD SpO₂ / skin-temp carries — the two fields lastVitalsDay's predicate doesn't check, so the + // whole-row carry above can land on a null value while an older imported row holds a real one. Same + // today-key bound; only on today. Resolved here (never in body) so the tiles stay O(1). + cachedSpo2Day = (selectedDayOffset == 0) ? Repository.lastSpo2Day(days: repo.days, todayKey: tkey) : nil + cachedSkinTempDay = (selectedDayOffset == 0) ? Repository.lastSkinTempDay(days: repo.days, todayKey: tkey) : nil // Charge carry (#543) + the honest label, resolved here for the same reason as the two above: the // selector below scans repo.days. Calibration nights come from the SAME `RecoveryScorer` helper the // classic Today reads, so the two screens agree on when a wearer is genuinely mid-calibration diff --git a/StrandTests/LiquidVitalsCarryTests.swift b/StrandTests/LiquidVitalsCarryTests.swift new file mode 100644 index 0000000000..9cc8261d90 --- /dev/null +++ b/StrandTests/LiquidVitalsCarryTests.swift @@ -0,0 +1,90 @@ +import XCTest +import WhoopStore +@testable import Strand + +/// Pins the Liquid Today PER-FIELD vitals carry for Blood Oxygen (SpO₂) and Skin Temp — the twin of +/// `LiquidChargeCarryTests` for the two fields the whole-row vitals carry (`lastVitalsDay`) does NOT check. +/// +/// The v8 Liquid redesign wired its recovery-vitals (HRV / Rest HR / Respiratory) to the whole-row +/// `Repository.lastVitalsDay` carry so they hold through the post-04:00 rollover window, but the Blood Oxygen +/// tile read only `displayDay ?? vitalsDay` and the Skin Temp card wasn't wired at all — while classic +/// `TodayView.dashboardCardValue` carries BOTH per-field (`d?.field ?? lastVitalsDay?.field ?? +/// last{Spo2,SkinTemp}Day?.field`). The gap bites precisely where the field is engine-nulled: the on-device +/// engine writes `spo2Pct`/`skinTempDevC = nil` on every computed "-noop" row (it banks only raw channels), +/// so the whole-row carry lands on a row whose value is nil even though an OLDER imported row holds a real +/// reading — and the tile blanks. `lastSpo2Day` / `lastSkinTempDay` resolve each field from its own freshest +/// strictly-prior row. +/// +/// The selectors themselves are pinned headlessly in `DailyMetricLastVitalsDayTests` (package-side). This +/// pins the LIQUID presentation decision on top of them — the today-first → whole-row → per-field ordering — +/// so Liquid and Classic can't drift. Pure `perFieldVital`, so it needs no strap, no clock and no view. +final class LiquidVitalsCarryTests: XCTestCase { + + /// A day row carrying only the fields these tests exercise — mirrors the package test's `fieldDay`. + private func day(_ key: String, hrv: Double? = nil, spo2: Double? = nil, skinTemp: Double? = nil) -> DailyMetric { + DailyMetric(day: key, totalSleepMin: nil, efficiency: nil, deepMin: nil, remMin: nil, + lightMin: nil, disturbances: nil, restingHr: nil, avgHrv: hrv, recovery: nil, + strain: nil, exerciseCount: nil, spo2Pct: spo2, skinTempDevC: skinTemp, respRateBpm: nil) + } + + // MARK: - The coalesce order: today wins, then the whole-row carry, then the per-field carry + + func testTodaysOwnValueWinsOverEitherCarry() { + XCTAssertEqual(LiquidTodayView.perFieldVital(today: 97, vitalsCarry: 95, fieldCarry: 90), 97, + "today's own reading must never be displaced by a carry") + } + + func testWholeRowVitalsCarryWinsOverThePerFieldCarry() { + // When last night's vitals row ALSO carries the field, that fresher night beats the older per-field + // row — today-first, then freshest-row-first, never the reverse. + XCTAssertEqual(LiquidTodayView.perFieldVital(today: nil, vitalsCarry: 95, fieldCarry: 90), 95) + } + + // MARK: - THE regression: the per-field carry is the only thing keeping the tile off "–" + + func testPerFieldCarrySurfaces_whenTodayAndTheWholeRowCarryAreBothNil() { + // The exact rollover shape: today's row is empty and the whole-row vitals carry landed on a computed + // "-noop" row whose spo2Pct/skinTempDevC is engine-written nil. Before the fix Liquid stopped at + // `nil ?? nil` and blanked; the per-field carry supplies the real reading. + XCTAssertEqual(LiquidTodayView.perFieldVital(today: nil, vitalsCarry: nil, fieldCarry: 96), 96, + "a null today AND null whole-row field must fall through to the per-field carry") + } + + func testNothingAnywhereStaysNil_soTheTileHonestlyShowsNoData() { + XCTAssertNil(LiquidTodayView.perFieldVital(today: nil, vitalsCarry: nil, fieldCarry: nil), + "no reading anywhere is honest No-Data — never a fabricated 0") + } + + // MARK: - End-to-end rollover, composed with the real selectors (mirrors the classic behavior) + + func testSpo2Rollover_fallsThroughTheNullVitalsRowToTheImportedReading() { + // Post-rollover: today empty; last night's computed row has HRV but spo2Pct = nil (so the WHOLE-ROW + // lastVitalsDay lands on it with a null SpO₂); an older imported row holds a real 96%. + let days = [day("2026-07-13", spo2: 96), // imported, real reading + day("2026-07-14", hrv: 42, spo2: nil), // last night: vitals yes, SpO₂ nil + day("2026-07-15")] // today, empty + let tkey = "2026-07-15" + let vitals = Repository.lastVitalsDay(days: days, todayKey: tkey) + XCTAssertEqual(vitals?.day, "2026-07-14", "the whole-row carry lands on last night (it has HRV)…") + XCTAssertNil(vitals?.spo2Pct, "…whose SpO₂ is nil — this is exactly what blanked the tile") + let spo2 = LiquidTodayView.perFieldVital( + today: days.last?.spo2Pct, + vitalsCarry: vitals?.spo2Pct, + fieldCarry: Repository.lastSpo2Day(days: days, todayKey: tkey)?.spo2Pct) + XCTAssertEqual(spo2, 96, "SpO₂ must resolve to the imported reading, not blank on the vitals row's nil") + } + + func testSkinTempRollover_fallsThroughToThePriorDeviation() { + let days = [day("2026-07-13", skinTemp: -0.3), // prior reading (signed deviation from baseline) + day("2026-07-14", hrv: 42), // last night: vitals, no skin temp + day("2026-07-15")] // today, empty + let tkey = "2026-07-15" + let vitals = Repository.lastVitalsDay(days: days, todayKey: tkey) + XCTAssertNil(vitals?.skinTempDevC, "the whole-row carry row has no skin temp") + let skin = LiquidTodayView.perFieldVital( + today: days.last?.skinTempDevC, + vitalsCarry: vitals?.skinTempDevC, + fieldCarry: Repository.lastSkinTempDay(days: days, todayKey: tkey)?.skinTempDevC) + XCTAssertEqual(skin, -0.3, "skin temp must carry the prior day's deviation, not blank") + } +} From 09632062c4361b492e0a5ce1202f2d1cc27b352b Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Sun, 26 Jul 2026 22:57:59 -0400 Subject: [PATCH 129/171] =?UTF-8?q?docs(A6):=20the=205/MG=20charge=20bit?= =?UTF-8?q?=20IS=20wrong=20(confirmed)=20=E2=80=94=20and=20StrapChargeInfe?= =?UTF-8?q?rence's=20threshold=20never=20fires?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Checked #4's charge inference against the cloud mirror. Both halves matter, and the second one corrects what #4 claimed. CONFIRMED — A6's open question is settled, and the answer is that the bit is unusable. The controlled capture A6 asked for was already in the mirror with ground truth attached: a session on 2026-07-15 11:07->13:34 MDT carrying CHARGING_ON x10, BATTERY_PACK_CONNECTED x3, SoC 4.7% -> 100.0% — and `charging = false` on all ~300 readings. Not null, FALSE: the BATTERY_LEVEL path decoded the bit and the strap asserted "not charging" while going from nearly flat to full. The "one historical row proves nothing" objection that sank the first pass doesn't apply to a labelled transition. OPEN — but the fix for it is inert. `isRising` demands > 1.0 pp between consecutive readings; that charge stepped +0.3...+0.7 pp at a ~30 s cadence and never cleared it. The `recentRiseWindowSeconds` comment assumes "~8 min apart", which is what made 1.0 pp look like a low bar. The live path looks no better: whole-percent 0x2A19 steps by exactly +1, and the strict `>` that exists to survive that quantisation also excludes exactly 1.0. So chargingEffective degenerates to the raw flag and the wearer still sees a discharge estimate on a charging strap. Not a regression — the inference is additive, so behaviour equals pre-#4 — but #4's write-up overstated what shipped, and this supersedes it. Comments only, no logic change; a fix wants the LIVE cadence measured first (the `battery` table is the decoded OFFLOAD stream, not what feeds LiveState.batterySamples). Likely direction: a RATE test (~50 pp/h charging vs ~1.65 pp/h discharge, ~30x at any cadence). Recorded at the type, in the sweep §A6, and in the tests — whose passing is itself misleading, since every fixture steps by more than 1.0 pp. GitHub issues are disabled on this fork, so the code and the sweep ARE the tracker. Verification: comments only; StrapChargeInferenceTests 10/10 pass, i18n gate passes. Co-Authored-By: Claude Opus 5 --- Strand/BLE/StrapChargeInference.swift | 33 ++++++++++++++ StrandTests/StrapChargeInferenceTests.swift | 11 ++++- ...15-strap-battery-backfill-observability.md | 43 ++++++++++++++++--- 3 files changed, 81 insertions(+), 6 deletions(-) diff --git a/Strand/BLE/StrapChargeInference.swift b/Strand/BLE/StrapChargeInference.swift index 32ca22bbea..23aa066e18 100644 --- a/Strand/BLE/StrapChargeInference.swift +++ b/Strand/BLE/StrapChargeInference.swift @@ -24,6 +24,39 @@ import StrandAnalytics /// /// Strictly additive: a confirmed `true` is still `true`, and inference can only ever ADD a charge, never /// deny one. If neither witness speaks, it answers `nil` (unknown) rather than asserting "not charging". +/// +/// ──────────────────────────────────────────────────────────────────────────────────────────────────── +/// KNOWN ISSUE — the premise is CONFIRMED, but this threshold probably never fires. NOT YET FIXED. +/// +/// Checked against the cloud mirror on 2026-07-26 (GitHub issues are disabled on this fork, so the +/// finding lives here and in docs/bugs/2026-07-15-…md §A6 rather than in a tracker). +/// +/// CONFIRMED — the bit really is wrong. One charging session with unambiguous ground truth +/// (CHARGING_ON ×10, BATTERY_PACK_CONNECTED ×3; 2026-07-15 11:07→13:34 MDT; SoC 4.7% → 100.0%) carries +/// `charging = false` on every one of its ~300 readings. Not `nil` — FALSE, i.e. the BATTERY_LEVEL path +/// decoded the bit and the strap asserted "not charging" while going from nearly flat to full. +/// +/// BUT — `isRising` requires > `chargeStepPct` (1.0 pp) between CONSECUTIVE readings, and that charge +/// stepped +0.3…+0.7 pp per reading at a ~30 s cadence. It never once cleared 1.0. The +/// `recentRiseWindowSeconds` comment below assumes the opposite ("~8 min apart"), which is what makes +/// 1.0 pp look like a low bar; at the real cadence it is not. The live path looks no better: a 5/MG's +/// live SoC is whole-percent 0x2A19, so a charge steps by exactly +1, and the strict `>` that exists to +/// survive that quantisation also excludes exactly 1.0. +/// +/// So `chargingEffective` likely degenerates to the raw flag and this type is inert for the very case it +/// was written for. It is not HARMFUL — the inference is additive, so behaviour equals pre-fix — but it +/// is not doing its job either. Do not read the passing tests as coverage: the synthetic samples in +/// `StrapChargeInferenceTests` step by more than 1.0 pp, which is exactly why they pass. +/// +/// LIKELY FIX: make it a RATE test (pp per unit time) instead of a fixed per-reading delta. A WHOOP 5 +/// charges at ~50 pp/h against a ~1.65 pp/h discharge — a ~30x separation that holds at any cadence, +/// where a fixed step does not. +/// +/// UNMEASURED, and needed before picking a constant: the `battery` table this came from is fed by the +/// decoded OFFLOAD stream, not the live path that actually populates `LiveState.batterySamples`. The +/// ~30 s cadence is therefore evidence about banked records; the LIVE cadence and quantisation are still +/// unconfirmed and want a real charging session to settle. +/// ──────────────────────────────────────────────────────────────────────────────────────────────────── enum StrapChargeInference { /// A rise must be observed within this window to count as "charging NOW". The strap emits BATTERY_LEVEL diff --git a/StrandTests/StrapChargeInferenceTests.swift b/StrandTests/StrapChargeInferenceTests.swift index a6bf10b730..9c40eb96d5 100644 --- a/StrandTests/StrapChargeInferenceTests.swift +++ b/StrandTests/StrapChargeInferenceTests.swift @@ -8,9 +8,18 @@ import StrandAnalytics /// stated "not charging" and showed a "~2 days left" DISCHARGE estimate to a user watching it charge. The /// 5.0 charge bit's offset (@30) is unverified — the decoder's own docs confirm only the SoC and mV fields /// against a real capture — so the bit alone is not a fact worth reporting. +/// +/// ⚠️ THESE TESTS PASSING DOES NOT MEAN THE INFERENCE WORKS. Every fixture below steps the SoC by far +/// more than the 1.0 pp `isRising` demands, so they exercise the decision but NOT the magnitude a real +/// charge actually produces. Mirror data (2026-07-26) shows a confirmed charge stepping +0.3…+0.7 pp per +/// reading at a ~30 s cadence — under the threshold throughout, i.e. the inference would never fire on +/// it. See the KNOWN ISSUE block on `StrapChargeInference` and §A6 of +/// docs/bugs/2026-07-15-strap-battery-backfill-observability.md. A fix wants a real-cadence regression +/// case here, not just a smaller number in these fixtures. final class StrapChargeInferenceTests: XCTestCase { private let now = 1_800_000_000 - /// Two readings ~8 min apart, the strap's real BATTERY_LEVEL cadence. + /// Two readings ~8 min apart. NOTE: this was written as "the strap's real BATTERY_LEVEL cadence", and + /// that assumption is the bug — observed charging readings arrive ~30 s apart (see the note above). private func series(_ a: Double, _ b: Double, gap: Int = 8 * 60, age: Int = 60) -> [(ts: Int, soc: Double)] { [(ts: now - age - gap, soc: a), (ts: now - age, soc: b)] } diff --git a/docs/bugs/2026-07-15-strap-battery-backfill-observability.md b/docs/bugs/2026-07-15-strap-battery-backfill-observability.md index bedb0524e2..0632f975a2 100644 --- a/docs/bugs/2026-07-15-strap-battery-backfill-observability.md +++ b/docs/bugs/2026-07-15-strap-battery-backfill-observability.md @@ -247,7 +247,7 @@ terms move**. Any single headline multiplier is a property of one capture. Drain time is `(bytes banked per second-of-history) / (bytes the link moves per second)` and **both terms move**. Any single headline multiplier is a property of one capture. -### A6. "Charging flag never lands" — **REFUTED as stated · OPEN (reframed)** +### A6. "Charging flag never lands" — **REFUTED as stated · reframed question now CONFIRMED · fix INERT** The finding's own evidence contradicts its title. The row was `soc=31.6, charging=0`: @@ -261,12 +261,22 @@ The finding's own evidence contradicts its title. The row was `soc=31.6, chargin historical record's timestamp against an observation made at inspection time. The doc never established the charger was on at strap-RTC 10:58:06. -**OPEN (reframed):** whether the 5/MG's BATTERY_LEVEL bit0 tracks charging **at all** is genuinely +**Was OPEN (reframed):** whether the 5/MG's BATTERY_LEVEL bit0 tracks charging **at all** is genuinely unknown and worth settling — but it needs a controlled capture (put it on the charger, watch `state.charging` change), not a single historical row. A3's fix is what makes that flag visible enough -to observe. **This stays OPEN: nothing below settles the question.** - -**MITIGATED, so a wrong bit stops costing the wearer:** `Strand/BLE/StrapChargeInference.swift` stops +to observe. + +> **CONFIRMED 2026-07-26, from the cloud mirror — the bit is not a usable charging indicator.** The +> controlled capture asked for above turned out to be unnecessary: the mirror already held one, with +> ground truth attached. A charging session on **2026-07-15, 11:07 → 13:34 MDT** carries `CHARGING_ON(7)` +> ×10, `BATTERY_PACK_CONNECTED(21)` ×3, and SoC climbing **4.7% → 100.0%** — and `charging = false` on +> every one of its ~300 battery readings. **Not `null` — `false`.** That distinction is the whole finding: +> `null` would mean the flag never reported, while `false` means the BATTERY_LEVEL path decoded the bit +> and the strap asserted *not charging* while going from nearly flat to full. The "single historical row" +> objection that (correctly) sank the first pass does not apply — this is a labelled transition, not one +> row. Caveats: one session, one strap, and the mirror was 8.9 days stale when read. + +**ATTEMPTED MITIGATION — and it is almost certainly INERT.** `Strand/BLE/StrapChargeInference.swift` stops treating that one unverified bit as the only witness. A **rising SoC** is direct, family-independent, decoder-independent evidence of a charge, and this codebase already trusts exactly that inference — `BatteryEstimator.chargeStepPct` is documented as "a SoC rise larger than this marks a CHARGE", and the @@ -283,6 +293,29 @@ stored value diverges from Kotlin and no `.noopbak` contract moves. Pinned by Because it is display-only, this **does not** substitute for the controlled capture — it makes the app correct either way while the question stays open. Kotlin twin is a UI-parity item, not a data one. +> **OPEN — the threshold is set above the real charge step, so the rescue never triggers.** `isRising` +> needs **> 1.0 pp** (`BatteryEstimator.chargeStepPct`) between CONSECUTIVE readings. The confirmed charge +> above stepped **+0.3 … +0.7 pp** per reading at a **~30 s** cadence and never once cleared it. The +> `recentRiseWindowSeconds` comment assumes readings are "~8 min apart", which is what makes 1.0 pp look +> like a low bar; at the observed cadence it is not. The live path looks no better — a 5/MG's live SoC is +> whole-percent `0x2A19`, so a charge steps by exactly `+1`, and the strict `>` that exists to survive +> that quantisation also excludes exactly 1.0. +> +> So `chargingEffective` degenerates to the raw flag and the wearer still sees "not charging" plus a +> discharge estimate on a strap that is charging — **the original harm is not actually fixed.** It is not +> a regression (the inference is additive, so behaviour equals pre-fix), but the section above overstates +> what shipped and this correction supersedes it. +> +> Do not read the green tests as coverage: `StrapChargeInferenceTests`' synthetic samples step by more +> than 1.0 pp, which is exactly why they pass. +> +> **Likely fix:** a RATE test (pp per unit time) rather than a fixed per-reading delta — ~50 pp/h charging +> vs ~1.65 pp/h discharge is a ~30x separation that holds at any cadence. **Unmeasured and needed first:** +> the `battery` table is fed by the decoded OFFLOAD stream, not the live path that populates +> `LiveState.batterySamples`, so the LIVE cadence and quantisation are still unconfirmed. The detail is +> duplicated at the type in `Strand/BLE/StrapChargeInference.swift` (GitHub issues are disabled on this +> fork, so there is no tracker item). + ### A7. `lastSeenAt` frozen at pairing → "last seen 18 days ago" — **CONFIRMED · OPEN (both platforms)** ``` From f63a91c3ab63eb36666fac88fff1eb7e3154d7cf Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 27 Jul 2026 00:02:02 -0400 Subject: [PATCH 130/171] fix(ios): derive BGTask identifiers from the bundle id so a remap can't kill them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BGTaskSchedulerPermittedIdentifiers (project.yml, which XcodeGen writes into the generated StrandiOS/Resources/Info.plist) and the Swift bgTaskIdentifier constants were two hand-matched copies of the same string, coupled only by a comment saying they must match. iOS compares them as exact strings, and the entries are namespaced under the bundle id — so a build under a different Apple ID rewrites the plist side and leaves the Swift literals stale. BGTaskScheduler.register then returns false and submit throws .notPermitted, and both call sites discarded those signals, so BOTH background lanes — the #510 scheduled debug export and Cloud Sync v2's 4-hourly refresh — went dark with no symptom at all. Both sides now derive from PRODUCT_BUNDLE_IDENTIFIER, the same single-source-of-truth shape APP_GROUP_ID already uses: the plist entries expand from the build setting, and BGTaskIdentifier.resolve reads the permitted list the bundle actually carries (falling back to bundle id + suffix, then the canonical upstream literal). register()'s Bool is now checked, and a failure is logged in every configuration rather than only in Debug — a sideloaded personal build is exactly where this breaks, and a Debug-only signal would never reach it. BGTaskIdentifier lives in Strand/System/, not Strand/CloudSync/, so the ScheduledDebugExport half stays independent of the fork-only CloudSync directory. Verified on an iOS simulator build with one variable changed between runs: before: FAILED to register 'com.noopapp.noop.debugexport' and 'com.noopapp.noop.cloudsync.refresh' — not in the permitted list after: registered both, 0 failures Also confirmed that $(PRODUCT_BUNDLE_IDENTIFIER) expands inside a plist ARRAY by reading the BUILT app's Info.plist rather than the generated source; that a stock com.noopapp.noop bundle id expands to exactly the previous literals, so stock builds are unchanged; and that the macOS Strand scheme still builds, since Strand/System/ is shared with it. --- .../CloudSyncBackgroundRefresh.swift | 12 +++- Strand/System/BGTaskIdentifier.swift | 66 +++++++++++++++++++ Strand/System/ScheduledDebugExport.swift | 13 +++- StrandiOS/Resources/Info.plist | 4 +- project.yml | 22 +++++-- 5 files changed, 104 insertions(+), 13 deletions(-) create mode 100644 Strand/System/BGTaskIdentifier.swift diff --git a/Strand/CloudSync/CloudSyncBackgroundRefresh.swift b/Strand/CloudSync/CloudSyncBackgroundRefresh.swift index 18da96be60..e85287bda1 100644 --- a/Strand/CloudSync/CloudSyncBackgroundRefresh.swift +++ b/Strand/CloudSync/CloudSyncBackgroundRefresh.swift @@ -23,7 +23,12 @@ enum CloudSyncBackgroundRefresh { /// registered at launch (`register(model:)`, called from `StrandiOSApp.init()`) for `schedule()` to /// have any effect — see `ScheduledDebugExport.bgTaskIdentifier`'s doc comment for the identical /// contract. - static let bgTaskIdentifier = "com.noopapp.noop.cloudsync.refresh" + /// Resolved from the bundle, not hardcoded — same reason as `ScheduledDebugExport.bgTaskIdentifier`: + /// the permitted entries are namespaced under the bundle id, so a personal-deploy remap silently + /// stales a literal and takes this whole lane down with it. `BGTaskIdentifier` lives in + /// `Strand/System/` (NOT here) so that fix stays independent of this fork-only directory. + static let bgTaskIdentifier = BGTaskIdentifier.resolve( + suffix: "cloudsync.refresh", upstream: "com.noopapp.noop.cloudsync.refresh") /// Register the BGTask handler. MUST be called from the app's launch (before launch finishes), /// same constraint as `ScheduledDebugExport.register()`. Safe to leave uncalled, or to have @@ -31,13 +36,16 @@ enum CloudSyncBackgroundRefresh { /// already covers on the next foreground launch anyway — this is a strictly additive, best-effort /// earlier chance, never a dependency for correctness. static func register(model: AppModel) { - BGTaskScheduler.shared.register(forTaskWithIdentifier: bgTaskIdentifier, using: nil) { task in + let registered = BGTaskScheduler.shared.register(forTaskWithIdentifier: bgTaskIdentifier, using: nil) { task in // The launch handler runs on a queue the system chooses, NOT necessarily the main actor — // hop explicitly before touching `CloudSyncModel` (`@MainActor`) or `AppModel`. Task { @MainActor in handle(task, model: model) } } + // Same contract as `ScheduledDebugExport.register()`: a false return means this identifier isn't + // permitted and the 4-hourly background sync will never be delivered. Never swallow it. + BGTaskIdentifier.report(bgTaskIdentifier, registered: registered) } /// Submit a background-refresh request for *no earlier than* diff --git a/Strand/System/BGTaskIdentifier.swift b/Strand/System/BGTaskIdentifier.swift new file mode 100644 index 0000000000..34ca8c309d --- /dev/null +++ b/Strand/System/BGTaskIdentifier.swift @@ -0,0 +1,66 @@ +import Foundation + +/// The iOS `BGTaskScheduler` identifiers this build is actually permitted to register — and a canary +/// that reports a mismatch loudly instead of letting the whole background lane fail silently. +/// +/// WHY THIS EXISTS: a BGTask identifier has to be written twice — once in the iOS target's +/// `BGTaskSchedulerPermittedIdentifiers` (project.yml, which XcodeGen expands into the generated +/// `StrandiOS/Resources/Info.plist`) and once in Swift. iOS compares the two as EXACT strings and +/// refuses to register a handler whose identifier isn't on the permitted list. The permitted entries +/// are namespaced under the bundle id, so the documented "build under your own Apple ID" path (see +/// `APP_GROUP_ID` in project.yml) rewrites the plist side and leaves a hardcoded Swift literal stale. +/// `BGTaskScheduler.register` then returns `false` and `submit` throws `.notPermitted` — and both call +/// sites discarded those signals, so the lane went dark with no symptom at all. +/// +/// Same single-source-of-truth posture as `WidgetSnapshot.suiteName`: read what the bundle actually +/// carries, and fall back to the canonical upstream literal only if the key is missing. +/// +/// macOS compiles this file too (it lives under `Strand/`), where `BGTaskScheduler` doesn't exist and +/// the plist key is absent — `permitted` is simply empty and nothing calls into here. +enum BGTaskIdentifier { + /// The `BGTaskSchedulerPermittedIdentifiers` array from THIS process's bundle — the exact list iOS + /// checks a registration against. Empty on macOS and on any build that hasn't wired the key. + static let permitted: [String] = { + Bundle.main.object(forInfoDictionaryKey: "BGTaskSchedulerPermittedIdentifiers") as? [String] ?? [] + }() + + /// The identifier this build may actually register for a given lane, e.g. + /// `resolve(suffix: "debugexport", upstream: "com.noopapp.noop.debugexport")`. + /// + /// Resolution order, most authoritative first: + /// 1. **The permitted list itself.** This is the exact array iOS compares a registration against, + /// so an entry from it is correct by construction — however the plist got its values, and even + /// if someone hand-edits it to something that isn't `.`. + /// 2. **This build's bundle id + the suffix.** Covers a target that hasn't wired the plist key. + /// 3. **The canonical upstream literal**, so behaviour is unchanged on a stock build even if the + /// bundle id is somehow unreadable. + /// + /// The suffix match is anchored on a leading `.` so `debugexport` can never claim + /// `cloudsync.refresh`'s entry or vice versa. + static func resolve(suffix: String, upstream: String) -> String { + if let permittedEntry = permitted.first(where: { $0.hasSuffix(".\(suffix)") }) { return permittedEntry } + if let bundleID = Bundle.main.bundleIdentifier { return "\(bundleID).\(suffix)" } + return upstream + } + + /// Report the outcome of a `BGTaskScheduler.register` call. A failure is logged in EVERY + /// configuration, not just Debug: a sideloaded personal build is exactly where a bundle-id remap + /// breaks this, and a Debug-only signal would never reach that build. + /// + /// Deliberately does NOT `assertionFailure` the way `WidgetSnapshot.assertGroupProvisioned` does. + /// A missing App Group entitlement breaks user-visible core features (widget, Live Activity, watch + /// snapshot), so trapping in Debug is proportionate there. A missing BGTask permit only disables an + /// opportunistic lane this file's own doc comments call "never a dependency for correctness" — + /// turning that into a launch crash would be worse than the bug. Loud is enough. + static func report(_ identifier: String, registered: Bool) { + guard !registered else { + NSLog("BGTaskIdentifier: registered '\(identifier)'") + return + } + NSLog(""" + BGTaskIdentifier: FAILED to register '\(identifier)' — not in \ + BGTaskSchedulerPermittedIdentifiers \(permitted). This background lane will never run. \ + The identifier must match the iOS target's permitted list in project.yml exactly. + """) + } +} diff --git a/Strand/System/ScheduledDebugExport.swift b/Strand/System/ScheduledDebugExport.swift index e07e7ea192..072509a3ba 100644 --- a/Strand/System/ScheduledDebugExport.swift +++ b/Strand/System/ScheduledDebugExport.swift @@ -40,7 +40,13 @@ enum ScheduledDebugExport { /// iOS BGTask identifier. Must also appear in the iOS target's `BGTaskSchedulerPermittedIdentifiers` /// (Info.plist) and be registered at launch for `submit` to succeed — wired in the app entry point. - static let bgTaskIdentifier = "com.noopapp.noop.debugexport" + /// + /// Resolved from the bundle rather than hardcoded, because the permitted entries are namespaced + /// under the bundle id: a build under a different Apple ID (the documented "change it in one place" + /// path — see `APP_GROUP_ID` in project.yml) rewrites the plist side, and a literal here would go + /// stale and silently kill this lane. See `BGTaskIdentifier` for the full contract. + static let bgTaskIdentifier = BGTaskIdentifier.resolve( + suffix: "debugexport", upstream: "com.noopapp.noop.debugexport") static var isEnabled: Bool { UserDefaults.standard.bool(forKey: K.enabled) } @@ -198,12 +204,15 @@ enum ScheduledDebugExport { /// the task. Both live in the iOS app target — call this from `StrandiOSApp.init()`. Safe to leave /// uncalled: `submitBackgroundRequest()` fails gracefully and the macOS path + "Run now" still work. static func register() { - BGTaskScheduler.shared.register(forTaskWithIdentifier: bgTaskIdentifier, using: nil) { task in + let registered = BGTaskScheduler.shared.register(forTaskWithIdentifier: bgTaskIdentifier, using: nil) { task in // Write the drop, then immediately request the next one (BGAppRefresh is single-shot). if isEnabled { catchUpIfDue() } submitBackgroundRequest() task.setTaskCompleted(success: true) } + // `register` returns false when the identifier isn't on the permitted list — the one signal that + // this lane is dead. Discarding it is what let a bundle-id remap silently disable the drop. + BGTaskIdentifier.report(bgTaskIdentifier, registered: registered) } /// Submit a background-refresh request for *no earlier than* the next chosen time. Honest: iOS decides diff --git a/StrandiOS/Resources/Info.plist b/StrandiOS/Resources/Info.plist index 00cc93a83c..200a9e57aa 100644 --- a/StrandiOS/Resources/Info.plist +++ b/StrandiOS/Resources/Info.plist @@ -6,8 +6,8 @@ $(APP_GROUP_ID) BGTaskSchedulerPermittedIdentifiers - com.noopapp.noop.debugexport - com.noopapp.noop.cloudsync.refresh + $(PRODUCT_BUNDLE_IDENTIFIER).debugexport + $(PRODUCT_BUNDLE_IDENTIFIER).cloudsync.refresh CFBundleDevelopmentRegion $(DEVELOPMENT_LANGUAGE) diff --git a/project.yml b/project.yml index aa37192b2f..fb0744b30b 100644 --- a/project.yml +++ b/project.yml @@ -260,14 +260,22 @@ targets: # (see CloudSyncAppDelegate's doc comment), so no push can actually arrive until that lands — # this mode alone is inert without it, but harmless to declare early. - remote-notification - # #510: permit the scheduled debug auto-export's BGAppRefreshTask identifier. MUST match - # ScheduledDebugExport.bgTaskIdentifier EXACTLY and be registered at launch (StrandiOSApp.init); - # otherwise iOS refuses to schedule or deliver the background drop. - # Cloud Sync v2 adds its own identifier alongside it — MUST match - # CloudSyncBackgroundRefresh.bgTaskIdentifier EXACTLY, same registration constraint. + # #510: permit the scheduled debug auto-export's BGAppRefreshTask identifier (and, alongside it, + # Cloud Sync v2's). Both still have to be registered at launch (StrandiOSApp.init) or iOS refuses + # to schedule or deliver them. + # + # These derive from $(PRODUCT_BUNDLE_IDENTIFIER) rather than being spelled out, for the same + # reason APP_GROUP_ID is a single setting: iOS matches the identifier a handler registers against + # THIS list as an exact string, and the entries are namespaced under the bundle id — so a build + # under a different Apple ID rewrote this list and left the hardcoded Swift literals stale, + # silently killing BOTH background lanes with no symptom. The Swift side now resolves from the + # bundle too (BGTaskIdentifier.resolve, Strand/System/BGTaskIdentifier.swift), so the two sides + # track the bundle id together and cannot drift apart again. Xcode expands the variable during + # the Process-Info.plist phase — verified in the BUILT app's Info.plist, not just the generated + # source, since expansion inside a plist ARRAY is the part worth confirming. BGTaskSchedulerPermittedIdentifiers: - - com.noopapp.noop.debugexport - - com.noopapp.noop.cloudsync.refresh + - $(PRODUCT_BUNDLE_IDENTIFIER).debugexport + - $(PRODUCT_BUNDLE_IDENTIFIER).cloudsync.refresh NSBluetoothAlwaysUsageDescription: "NOOP connects directly to your WHOOP strap over Bluetooth to read heart rate, R-R intervals, battery, and sensor data locally on your iPhone. Nothing leaves your device." NSLocationWhenInUseUsageDescription: "NOOP records your route during an outdoor workout to show distance, pace, and a map, and keeps recording while the screen is off. The route stays on your device. Nothing leaves it." NSHealthShareUsageDescription: "NOOP reads your own Apple Health data on-device to compute recovery, strain, and sleep. Nothing leaves your device." From 9e1716e04ef1be51d4ed2533f729c74f022d5cf0 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 27 Jul 2026 01:01:19 -0400 Subject: [PATCH 131/171] feat(store): opt-in WalCheckpointing so an external replicator can own the WAL MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A page-level replicator (litestream / its embeddable Rust port, liters) ships WAL frames, so it must observe every transaction. It pins the WAL with a long-running read transaction — but liters' Manager.sleep() explicitly releases that read transaction and every fd when the app backgrounds (crates/liters/src/manager.rs:602-605), which on iOS is most of the time. While it sleeps, GRDB's wal_autocheckpoint is free to restart the WAL, the replicator's resume offset is overwritten, and it must fall back to a full snapshot of the whole database. wal_autocheckpoint is a PER-CONNECTION setting and a DatabasePool opens one writer plus N readers, so this is applied inside Configuration.prepareDatabase — setting it anywhere else would leave the pool's other connections still auto-checkpointing, which is the exact bug being guarded against. The test asserts the pragma on reader connections, not just the writer. Defaults to .automatic, byte-for-byte today's behaviour, so upstream and Android are unaffected. No migration, no stored-data change, no Kotlin twin needed: this is a connection pragma, not schema or an analytic. Gated as a runtime parameter rather than #if CLOUD_SYNC deliberately. That flag is set via SWIFT_ACTIVE_COMPILATION_CONDITIONS in the untracked Strand/Oura/OuraConfig.xcconfig, which project.yml applies as configFiles to the Strand app target and StrandTests only. Packages/WhoopStore is a local SPM package built as its own target and does not inherit an app target's xcconfig, so #if CLOUD_SYNC here would never be true — the pragma would silently never be set while looking done. Measured cost of .external on a NOOP-shaped workload (2M-row base, one transaction per Collector flush): ~14 pages dirtied per commit regardless of payload, so at the live lane's ~30 s cadence the WAL reaches ~166 MB after one day and ~1.17 GB after a week with nothing checkpointing. The obligation that carries is documented on the enum: an .external caller MUST guarantee a checkpointer actually runs, and should keep a size floor as a backstop. --- .../Sources/WhoopStore/WhoopStore.swift | 63 +++++- .../WalCheckpointingTests.swift | 191 ++++++++++++++++++ 2 files changed, 253 insertions(+), 1 deletion(-) create mode 100644 Packages/WhoopStore/Tests/WhoopStoreTests/WalCheckpointingTests.swift diff --git a/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift b/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift index 3c0363b969..59e994672d 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift @@ -9,6 +9,56 @@ public enum WhoopStoreInfo { public static let schemaVersion = 18 } +/// Who is responsible for checkpointing the WAL back into the main database file. +/// +/// SQLite's `wal_autocheckpoint` is a **per-connection** setting, not a database property, so this +/// is applied inside `Configuration.prepareDatabase` and therefore reaches every connection a +/// `DatabasePool` opens. +/// +/// ## Why this exists +/// +/// A page-level replicator (Litestream and its embeddable Rust port, `liters`) ships **WAL frames**, +/// so it must observe every transaction. To stop the WAL being restarted underneath it, it holds a +/// long-running read transaction — which starves every *other* checkpointer, including SQLite's own +/// autocheckpoint. If a foreign connection nevertheless succeeds in restarting the WAL, the +/// replicator finds its resume offset overwritten and must fall back to a **full snapshot of the +/// entire database**. On iOS that would be a multi-hundred-megabyte upload, on every app launch. +/// +/// Passing `.external` is what lets such a replicator be the single checkpointer. +/// +/// ## The obligation `.external` carries — read before using it +/// +/// With autocheckpoint off, **nothing in this package bounds WAL growth**. `checkpointWAL()` is only +/// called from bulk-import and backup paths; the hot BLE ingest path never checkpoints. Measured on +/// a NOOP-shaped workload (a 2M-row base, `Collector.flushStandardHR()`'s ~30 s commit cadence, one +/// transaction per flush): +/// +/// | cadence | commits/day | WAL after 1 day | after 7 days | +/// |---|---|---|---| +/// | every 10 s | 8,640 | 341 MB | — | +/// | **every 30 s (NOOP today)** | **2,880** | **166 MB** | **1.17 GB** | +/// | every 5 min | 288 | 37 MB | — | +/// +/// The cost is driven by the **commit count**, not the data volume: each commit dirties ~14 pages +/// (the file header's change counter, each table's rightmost leaf, each index leaf, the interior +/// spine) whether it carries 30 rows or 300. Only ~11 MB/day of that is new data; the rest is +/// re-written pages that autocheckpoint would normally reclaim. +/// +/// So an `.external` caller MUST guarantee a checkpointer actually runs. If it stops running — the +/// user disables sync, the network is down for a week, the replicator is never started, or it +/// crashes on launch — the WAL grows without limit and will eventually exhaust device storage. That +/// is a strictly worse failure than the full re-upload it was meant to avoid, so the caller should +/// also keep a floor: watch `databaseFileSizeBytes()` and force `checkpointWAL()` if the `-wal` +/// sibling crosses a ceiling, regardless of replicator state. +public enum WalCheckpointing: Sendable { + /// SQLite's default: each connection auto-checkpoints at ~1000 WAL pages (~4 MB). This is the + /// behaviour of every build that does not opt out, and the only behaviour upstream ships. + case automatic + /// Disable `wal_autocheckpoint` on every connection this store opens, because an external + /// component checkpoints instead. See the obligation documented above. + case external +} + /// Serializes `DatabasePool` creation + migration so two concurrent opens of the SAME file can never /// run their GRDB migrators at once (#261). /// @@ -83,7 +133,11 @@ public actor WhoopStore { /// /// Open + migrate runs through `StoreOpenGate` so two concurrent openers of the SAME file never /// run their GRDB migrators at once (#261) — see that actor's note for the failure it prevents. - public init(path: String) async throws { + /// + /// `walCheckpointing` defaults to `.automatic`, which is exactly the behaviour every build has + /// today. Only an embedder that has taken over checkpointing should pass `.external` — see + /// `WalCheckpointing` for the obligation that carries. + public init(path: String, walCheckpointing: WalCheckpointing = .automatic) async throws { var config = Configuration() config.prepareDatabase { db in // `DatabasePool` puts the database in WAL mode itself (reads run as concurrent snapshots @@ -95,6 +149,13 @@ public actor WhoopStore { try db.execute(sql: "PRAGMA cache_size = -16000") // ~16 MB page cache try db.execute(sql: "PRAGMA mmap_size = 268435456") // 256 MB memory-mapped I/O try db.execute(sql: "PRAGMA temp_store = MEMORY") + // `wal_autocheckpoint` is a PER-CONNECTION setting, so this has to run inside + // `prepareDatabase` — it is applied to every connection the pool opens (the writer and + // each reader), not once at open. Setting it anywhere else would leave the pool's other + // connections still auto-checkpointing, which is the exact bug this guards against. + if case .external = walCheckpointing { + try db.execute(sql: "PRAGMA wal_autocheckpoint = 0") + } } config.busyMode = .timeout(5) let pool = try await StoreOpenGate.shared.openAndMigrate(path: path, configuration: config) diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/WalCheckpointingTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/WalCheckpointingTests.swift new file mode 100644 index 0000000000..0fa4f702a1 --- /dev/null +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/WalCheckpointingTests.swift @@ -0,0 +1,191 @@ +import XCTest +import GRDB +import WhoopProtocol +@testable import WhoopStore + +/// `WalCheckpointing` decides whether SQLite auto-checkpoints the WAL back into the main database. +/// +/// The failure this guards against is subtle: `wal_autocheckpoint` is a **per-connection** setting, +/// and a `DatabasePool` opens one writer plus N reader connections. A page-level replicator that +/// holds a long-running read transaction to pin the WAL is defeated if *any* other connection in the +/// process still auto-checkpoints — the WAL restarts, the replicator's resume offset is overwritten, +/// and it must re-upload the whole database. So the assertion that matters is not "the pragma was +/// set" but "the pragma was set on EVERY connection the pool hands out". +/// +/// `.automatic` must remain byte-for-byte today's behaviour: it is what upstream and every default +/// build ship, and this package is upstream-shared. +final class WalCheckpointingTests: XCTestCase { + + private func tempPath() -> String { + NSTemporaryDirectory() + "whoopstore-walcp-\(UUID().uuidString).sqlite" + } + + private func removeDB(_ path: String) { + for suffix in ["", "-wal", "-shm"] { + try? FileManager.default.removeItem(atPath: path + suffix) + } + } + + /// Reads `PRAGMA wal_autocheckpoint` on a WRITER connection. + private func autocheckpointOnWriter(_ store: WhoopStore) async throws -> Int { + try await store.registryWriter.write { db in + try Int.fetchOne(db, sql: "PRAGMA wal_autocheckpoint") ?? -1 + } + } + + /// Reads `PRAGMA wal_autocheckpoint` on a READER connection — a `DatabasePool` serves `.read` + /// from a separate connection than `.write`, which is exactly where the leak would hide. + private func autocheckpointOnReader(_ store: WhoopStore) async throws -> Int { + try await store.registryWriter.read { db in + try Int.fetchOne(db, sql: "PRAGMA wal_autocheckpoint") ?? -1 + } + } + + // MARK: - Default behaviour is unchanged + + /// The default must be indistinguishable from the pre-change store: SQLite's stock 1000-page + /// autocheckpoint, on both connection kinds. Upstream and Android depend on this. + func testDefaultLeavesAutocheckpointAtSQLiteDefault() async throws { + let path = tempPath() + defer { removeDB(path) } + + let store = try await WhoopStore(path: path) + let writer = try await autocheckpointOnWriter(store) + let reader = try await autocheckpointOnReader(store) + + XCTAssertEqual(writer, 1000, "default store must keep SQLite's stock autocheckpoint (writer)") + XCTAssertEqual(reader, 1000, "default store must keep SQLite's stock autocheckpoint (reader)") + } + + /// Passing `.automatic` explicitly is the same as passing nothing. + func testExplicitAutomaticMatchesDefault() async throws { + let path = tempPath() + defer { removeDB(path) } + + let store = try await WhoopStore(path: path, walCheckpointing: .automatic) + let value = try await autocheckpointOnWriter(store) + XCTAssertEqual(value, 1000) + } + + // MARK: - Opt-in disables it everywhere + + /// THE CORE ASSERTION: `.external` must reach every connection the pool opens, not just the + /// writer. A reader that still auto-checkpoints restarts the WAL and forces the replicator into + /// a full re-upload — the precise failure this option exists to prevent. + func testExternalDisablesAutocheckpointOnWriterAndReader() async throws { + let path = tempPath() + defer { removeDB(path) } + + let store = try await WhoopStore(path: path, walCheckpointing: .external) + let writer = try await autocheckpointOnWriter(store) + let reader = try await autocheckpointOnReader(store) + + XCTAssertEqual(writer, 0, "external checkpointing must disable autocheckpoint on the writer") + XCTAssertEqual(reader, 0, "external checkpointing must ALSO disable it on reader connections") + } + + /// Several concurrent readers force the pool to open additional connections; every one of them + /// must carry the pragma, since `prepareDatabase` — not a one-shot statement at open — is what + /// applies it. + func testExternalAppliesToEveryPooledReaderConnection() async throws { + let path = tempPath() + defer { removeDB(path) } + + let store = try await WhoopStore(path: path, walCheckpointing: .external) + // Seed a row so reads have something to do. + _ = try await store.insert(Streams(hr: [HRSample(ts: 1, bpm: 60)]), deviceId: "dev") + + let values = try await withThrowingTaskGroup(of: Int.self) { group -> [Int] in + for _ in 0..<8 { + group.addTask { + try await store.registryWriter.read { db in + try Int.fetchOne(db, sql: "PRAGMA wal_autocheckpoint") ?? -1 + } + } + } + var out: [Int] = [] + for try await v in group { out.append(v) } + return out + } + + XCTAssertEqual(values.count, 8) + XCTAssertTrue(values.allSatisfy { $0 == 0 }, + "every pooled reader connection must have autocheckpoint disabled, got \(values)") + } + + // MARK: - Behaviour, not just the pragma value + + /// The observable consequence, and the reason `.external` is dangerous without a checkpointer: + /// under `.automatic` the WAL plateaus near SQLite's ~4 MB (1000-page) threshold, because each + /// crossing checkpoints and rewinds the WAL to the start and the file is then REUSED in place. + /// Under `.external` nothing rewinds it, so it grows monotonically with the number of commits. + /// + /// The writes must therefore comfortably exceed 4 MB of WAL or neither store checkpoints and the + /// two are trivially identical. + func testExternalLetsWalGrowPastTheAutocheckpointThreshold() async throws { + let autoPath = tempPath() + let extPath = tempPath() + defer { removeDB(autoPath); removeDB(extPath) } + + let transactions = 250 + let rowsPerTransaction = 500 + + // NOOP's real commit shape: one transaction per Collector flush. + func writeManySmallTransactions(_ store: WhoopStore) async throws { + var ts = 0 + for _ in 0.. Int64 { + let attrs = try? FileManager.default.attributesOfItem(atPath: path + "-wal") + return (attrs?[.size] as? NSNumber)?.int64Value ?? 0 + } + + let autoStore = try await WhoopStore(path: autoPath, walCheckpointing: .automatic) + try await writeManySmallTransactions(autoStore) + + let extStore = try await WhoopStore(path: extPath, walCheckpointing: .external) + try await writeManySmallTransactions(extStore) + + let autoWal = walBytes(autoPath) + let extWal = walBytes(extPath) + + XCTAssertGreaterThan( + extWal, autoWal * 2, + "with autocheckpoint off the WAL must grow far past the self-checkpointing one " + + "(external=\(extWal) bytes, automatic=\(autoWal) bytes)") + // And the store still works: nothing about disabling checkpointing may change stored data. + let counts = try await extStore.storageStats_rowCountsForTest() + XCTAssertEqual(counts.hr, transactions * rowsPerTransaction, + "disabling autocheckpoint must not change what is stored") + } + + /// An explicit `checkpointWAL()` must still reclaim the WAL when autocheckpoint is off — this is + /// the safety valve an `.external` caller needs when its replicator is not running. + func testExplicitCheckpointStillWorksWithExternalMode() async throws { + let path = tempPath() + defer { removeDB(path) } + + let store = try await WhoopStore(path: path, walCheckpointing: .external) + var ts = 0 + for _ in 0..<40 { + var hr: [HRSample] = [] + for _ in 0..<200 { ts += 1; hr.append(HRSample(ts: ts, bpm: 60 + ts % 40)) } + _ = try await store.insert(Streams(hr: hr), deviceId: "dev") + } + + func walBytes() -> Int64 { + let attrs = try? FileManager.default.attributesOfItem(atPath: path + "-wal") + return (attrs?[.size] as? NSNumber)?.int64Value ?? 0 + } + let grown = walBytes() + XCTAssertGreaterThan(grown, 0, "WAL should have accumulated with autocheckpoint off") + + try await store.checkpointWAL() + XCTAssertLessThan(walBytes(), grown, "explicit checkpointWAL() must still truncate the WAL") + } +} From b23931da2c80ee218960ac836d41926fc8cdbfc8 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 27 Jul 2026 01:30:22 -0400 Subject: [PATCH 132/171] feat(store): bound WAL growth under .external with an in-write-path size backstop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1 added `WalCheckpointing.external` and documented that the caller MUST guarantee a checkpointer runs. That obligation was unenforced, and the failure it guards is not hypothetical: liters' own thresholds — PASSIVE at min_checkpoint_page_n = 1000 pages (~4 MB) and emergency TRUNCATE at truncate_page_n = 121_359 pages (~500 MB) — both live inside `push()` (crates/liters/src/writer.rs:764-782). If liters never runs, NEITHER fires, and nothing else in this package does either: checkpointWAL() is called only from bulk-import and backup paths, never from the BLE ingest path. Measured growth with nothing checkpointing is 166 MB/day and 1.17 GB/week. So `.external` now installs the backstop itself instead of asking the caller to remember. `WalBackstopMonitor` is a GRDB `TransactionObserver` on the pool's writer connection, which is the only place the whole write path passes through. It costs one stat(2) (~1 us) per committed transaction and nothing per row: `observes(eventsOfKind:)` returns false, so GRDB never installs the SQLite update hook for it, while `databaseDidCommit` is still delivered for every transaction. The checkpoint is dispatched to a private utility queue rather than run inline — databaseDidCommit runs on the writer's serial queue inside statement execution, so calling writeWithoutTransaction there would re-enter that queue. Ceiling: 64 MiB, and it is pinned by a test because it is a decision. - 16x liters' own ~4 MB PASSIVE threshold, so a healthy replicator never comes near it: a firing is evidence of a problem, never routine. - ~9.3 h of the measured 6.9 MB/h, longer than any normal gap between foregrounds/BG refreshes, so in healthy operation it fires zero times. - 1/8 of liters' ~500 MB emergency truncate, so the app-side floor always acts first — required, because liters' own threshold cannot fire while it sleeps. - 8.4% of the 766 MB production database: peak disk stays ~1.08x, not ~1.65x. - A 64 MB TRUNCATE is sub-second on iPhone NVMe, so it cannot strand a suspension (0xdead10cc), and it bounds the cold-launch WAL recovery scan. Lower (16-32 MB) fires every 2-5 h and would routinely cost the replicator its incremental resume point; higher (256 MB+) makes cold launch and checkpoint latency user-visible. `.disabled` remains available and is an explicit assertion that something outside this package bounds the WAL. Nothing in it does. A TRUNCATE can be blocked by a DatabasePool reader holding an open snapshot, so an attempt that leaves the WAL over the ceiling arms a 60 s backoff instead of letting the next commit try again — otherwise a pinned WAL turns into one checkpoint attempt per commit on the ingest path. `.automatic` installs nothing at all, so upstream and Android are byte-for-byte unaffected, and Phase 1's six tests pass unmodified. Also documents the multi-pool caveat found while wiring this: StoreOpenGate gives every opener its OWN DatabasePool on the same file (Repository and BLEManager both open one), and wal_autocheckpoint is per-connection — so a second store opened `.automatic` on a replicated file will restart the WAL under the replicator that the first store's `.external` was protecting. Every opener must pass `.external` or the option does nothing. Adds `walFileSizeBytes()`, `walBackstopFirings` and `walBackstopIneffectiveAttempts` for the device trial: a non-zero firing count on a replicated device is the direct signal that the replicator stopped running. Packages/WhoopStore: 311 tests, 0 failures (296 before, +15 here). --- .../Sources/WhoopStore/WalBackstop.swift | 222 +++++++++++++++++ .../Sources/WhoopStore/WhoopStore.swift | 90 ++++++- .../WhoopStoreTests/WalBackstopTests.swift | 231 ++++++++++++++++++ 3 files changed, 535 insertions(+), 8 deletions(-) create mode 100644 Packages/WhoopStore/Sources/WhoopStore/WalBackstop.swift create mode 100644 Packages/WhoopStore/Tests/WhoopStoreTests/WalBackstopTests.swift diff --git a/Packages/WhoopStore/Sources/WhoopStore/WalBackstop.swift b/Packages/WhoopStore/Sources/WhoopStore/WalBackstop.swift new file mode 100644 index 0000000000..b884c9dcec --- /dev/null +++ b/Packages/WhoopStore/Sources/WhoopStore/WalBackstop.swift @@ -0,0 +1,222 @@ +import Foundation +import GRDB + +/// The size ceiling that bounds WAL growth when `WalCheckpointing.external` has disabled SQLite's +/// own autocheckpoint. +/// +/// ## Why a ceiling is mandatory, not optional +/// +/// `.external` exists so a page-level replicator (Litestream, or its embeddable Rust port `liters`) +/// can be the *only* checkpointer. That replicator checkpoints from inside its own push loop: +/// `liters` runs a PASSIVE checkpoint once the WAL passes `min_checkpoint_page_n = 1000` pages +/// (~4 MB) and an emergency TRUNCATE at `truncate_page_n = 121_359` pages (~500 MB) — but **both +/// live inside `push()`**. If the replicator never runs, neither fires, and nothing else in the +/// process will: `WhoopStore.checkpointWAL()` is called only from bulk-import and backup paths, and +/// the hot BLE ingest path never checkpoints at all. +/// +/// Measured on a NOOP-shaped workload (2M-row base, one transaction per `Collector` flush), each +/// commit dirties ~14 pages regardless of payload, so the cost tracks the **commit count**, not the +/// data volume: at the live lane's ~30 s cadence (2,880 commits/day) the WAL grows ~166 MB/day and +/// ~1.17 GB/week with nothing checkpointing. That is a strictly worse failure than the full +/// re-upload `.external` was adopted to avoid. +/// +/// ## Choosing `ceilingBytes` — the trade +/// +/// Firing the backstop is not free. A checkpoint taken while the replicator is asleep truncates the +/// WAL out from under its resume offset; on the next push it finds the resume frame gone and may +/// have to upload a **full snapshot** of the whole database. (`liters`' `meta.rs` deliberately does +/// not persist `synced_to_wal_end` across a writer close/reopen, so after an app relaunch a +/// truncation is always "unexpected".) So the ceiling must be high enough never to be reached in +/// healthy operation, and low enough that a dead replicator cannot exhaust storage. +/// +/// `standard` is **64 MiB**, chosen against the measured 6.9 MB/hour: +/// +/// | property | value | why it matters | +/// |---|---|---| +/// | vs. `liters`' own PASSIVE threshold (~4 MB) | 16× | a *healthy* replicator keeps the WAL near 4 MB, so it never comes close — a firing is evidence of a problem, never routine | +/// | time to reach at 166 MB/day | ~9.3 h | longer than any normal gap between foregrounds/BG refreshes, so it fires zero times in healthy operation | +/// | vs. `liters`' emergency TRUNCATE (~500 MB) | 1/8 | the app-side floor always acts first, which is required: `liters`' threshold cannot act while `liters` is asleep | +/// | vs. the 766 MB production database | 8.4% | peak disk stays ~1.08× the database, not ~1.65× | +/// | TRUNCATE duration | sub-second on iPhone NVMe | cannot strand an app suspension (`0xdead10cc`) | +/// | crash-recovery WAL scan at launch | ≤ ~16k frames | a multi-hundred-MB WAL makes every cold launch pay a full WAL scan | +/// +/// Lower ceilings (16–32 MB) fire every 2–5 h and would routinely cost a full snapshot. Higher ones +/// (256 MB+) make cold launch and checkpoint latency user-visible and double peak disk. 64 MiB is +/// the point where the backstop is a genuine backstop: it only ever fires when the thing it is +/// backing up has stopped. +public struct WalBackstopPolicy: Sendable, Equatable { + /// Force a checkpoint once the `-wal` sibling reaches this many bytes. `0` disables the backstop + /// entirely — only correct if something else provably bounds the WAL. + public let ceilingBytes: Int64 + + /// After an attempt that did not bring the WAL back under the ceiling — a `TRUNCATE` checkpoint + /// can be blocked by a `DatabasePool` reader holding an open snapshot — wait at least this long + /// before trying again, so a pinned WAL cannot turn every commit into a checkpoint attempt. + public let minRetryInterval: TimeInterval + + public init(ceilingBytes: Int64, minRetryInterval: TimeInterval = 60) { + self.ceilingBytes = ceilingBytes + self.minRetryInterval = minRetryInterval + } + + /// 64 MiB. See the type's documentation for the derivation. + public static let standard = WalBackstopPolicy(ceilingBytes: 64 * 1024 * 1024) + + /// No backstop. Passing this with `.external` asserts that the caller has another bound on WAL + /// growth; there is no such bound inside this package. + public static let disabled = WalBackstopPolicy(ceilingBytes: 0) + + public var isEnabled: Bool { ceilingBytes > 0 } +} + +/// The backstop's decision logic, with no I/O and no SQLite — so the interesting behaviour +/// (threshold, in-flight suppression, retry backoff) is unit-testable without a database. +struct WalBackstopCore { + let policy: WalBackstopPolicy + + private(set) var checkpointInFlight = false + private(set) var nextAttemptAt: TimeInterval = 0 + /// Number of checkpoints this backstop has started. Telemetry: a non-zero count on a device + /// running the replicator means the replicator stopped running. + private(set) var firings = 0 + /// Attempts that finished with the WAL still at or above the ceiling (blocked checkpoint). + private(set) var ineffectiveAttempts = 0 + + init(policy: WalBackstopPolicy) { + self.policy = policy + } + + /// Called once per committed transaction with the current `-wal` size. + /// Returns `true` when the caller should start a checkpoint. + mutating func shouldCheckpoint(walBytes: Int64, now: TimeInterval) -> Bool { + guard policy.isEnabled else { return false } + guard !checkpointInFlight else { return false } + guard walBytes >= policy.ceilingBytes else { return false } + guard now >= nextAttemptAt else { return false } + checkpointInFlight = true + firings += 1 + return true + } + + /// Called when the checkpoint started by `shouldCheckpoint` has finished, with the `-wal` size + /// measured afterwards. A checkpoint that did not get the WAL back under the ceiling arms the + /// retry backoff instead of letting the next commit try again immediately. + mutating func didFinishCheckpoint(walBytes: Int64, now: TimeInterval) { + checkpointInFlight = false + if walBytes >= policy.ceilingBytes { + ineffectiveAttempts += 1 + nextAttemptAt = now + policy.minRetryInterval + } else { + nextAttemptAt = 0 + } + } +} + +/// Watches the `-wal` file from inside the write path and forces a checkpoint when it crosses the +/// policy ceiling. +/// +/// ## Why a `TransactionObserver` +/// +/// The trigger has to be somewhere the write path passes through, or the backstop is only as good +/// as the caller's memory. GRDB notifies `databaseDidCommit(_:)` for **every** committed +/// transaction on the connection, regardless of what `observes(eventsOfKind:)` returns — so this +/// observer costs one `stat(2)` per commit (~1 µs) and *nothing* per row: returning `false` from +/// `observes` means GRDB never even installs the SQLite update hook for us. +/// +/// A `DatabasePool` only commits on its single writer connection, so exactly one observer fires. +/// +/// ## Why the checkpoint is dispatched, not run inline +/// +/// `databaseDidCommit` runs on the writer's serial queue inside statement execution. Calling +/// `writeWithoutTransaction` there would re-enter that queue. The checkpoint is therefore handed to +/// a private utility queue, which then blocks on the writer normally — after the commit that +/// triggered it has finished. +final class WalBackstopMonitor: TransactionObserver, @unchecked Sendable { + private let walPath: String + private let lock = NSLock() + private var core: WalBackstopCore + /// Weak: the pool's writer connection holds this observer (weakly, `.observerLifetime`) and the + /// store holds it strongly. Capturing the pool strongly here would close a retain cycle through + /// the store. + private weak var pool: DatabasePool? + private let queue = DispatchQueue(label: "whoopstore.wal-backstop", qos: .utility) + + init(databasePath: String, policy: WalBackstopPolicy) { + self.walPath = databasePath + "-wal" + self.core = WalBackstopCore(policy: policy) + } + + /// Installs the observer on `pool`'s writer connection. `.observerLifetime` (GRDB's default) + /// keeps the connection's reference weak, so the store's strong reference is what keeps this + /// alive — and there is no cycle. + func attach(to pool: DatabasePool) { + self.pool = pool + pool.add(transactionObserver: self, extent: .observerLifetime) + } + + /// Size of the `-wal` sibling, or 0 if it does not exist. A raw `stat(2)`, deliberately not + /// `FileManager.attributesOfItem` — this runs on every commit and must not allocate. + private func walSizeBytes() -> Int64 { + var st = stat() + guard walPath.withCString({ stat($0, &st) }) == 0 else { return 0 } + return Int64(st.st_size) + } + + // MARK: Telemetry (read by tests and by the device-trial recorder) + + /// Checkpoints this backstop has forced since the store opened. + var firings: Int { lock.withLock { core.firings } } + /// Forced checkpoints that finished with the WAL still over the ceiling. + var ineffectiveAttempts: Int { lock.withLock { core.ineffectiveAttempts } } + /// Current `-wal` size in bytes. Safe to call from anywhere. + var currentWalBytes: Int64 { walSizeBytes() } + + // MARK: - TransactionObserver + + /// `false` for every event kind: this observer wants transaction boundaries, not row changes. + /// GRDB skips installing the update hook entirely when no observer wants events, so per-row + /// cost is zero — while `databaseDidCommit` is still delivered for every transaction. + func observes(eventsOfKind eventKind: DatabaseEventKind) -> Bool { false } + + func databaseDidChange(with event: DatabaseEvent) {} + + func databaseDidCommit(_ db: Database) { + let now = Date().timeIntervalSinceReferenceDate + let walBytes = walSizeBytes() + let start = lock.withLock { core.shouldCheckpoint(walBytes: walBytes, now: now) } + guard start else { return } + queue.async { [self] in runCheckpoint() } + } + + func databaseDidRollback(_ db: Database) {} + + // MARK: - Checkpointing + + /// Mirrors `WhoopStore.checkpointWAL()`: a TRUNCATE checkpoint, run outside any transaction. + /// Best-effort — under a `DatabasePool` a reader holding an open snapshot can block it, which is + /// what `minRetryInterval` exists for. + private func runCheckpoint() { + if let pool { + try? pool.writeWithoutTransaction { db in + try db.execute(sql: "PRAGMA wal_checkpoint(TRUNCATE)") + } + } + let after = walSizeBytes() + let now = Date().timeIntervalSinceReferenceDate + lock.withLock { core.didFinishCheckpoint(walBytes: after, now: now) } + } + + /// Runs any pending checkpoint synchronously. Tests only: production fires from `databaseDidCommit` + /// on a private queue, and this is how a test waits for it deterministically. + func drainForTest() { + queue.sync {} + } +} + +private extension NSLock { + func withLock(_ body: () throws -> T) rethrows -> T { + lock() + defer { unlock() } + return try body() + } +} diff --git a/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift b/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift index 59e994672d..a02860d629 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift @@ -47,15 +47,33 @@ public enum WhoopStoreInfo { /// So an `.external` caller MUST guarantee a checkpointer actually runs. If it stops running — the /// user disables sync, the network is down for a week, the replicator is never started, or it /// crashes on launch — the WAL grows without limit and will eventually exhaust device storage. That -/// is a strictly worse failure than the full re-upload it was meant to avoid, so the caller should -/// also keep a floor: watch `databaseFileSizeBytes()` and force `checkpointWAL()` if the `-wal` -/// sibling crosses a ceiling, regardless of replicator state. +/// is a strictly worse failure than the full re-upload it was meant to avoid. +/// +/// ## How that obligation is discharged +/// +/// It is **not** left to the caller. Choosing `.external` installs a `WalBackstopMonitor` on the +/// pool's writer connection: one `stat(2)` of the `-wal` sibling per committed transaction, and a +/// forced `wal_checkpoint(TRUNCATE)` once it crosses `WalBackstopPolicy.ceilingBytes` (64 MiB by +/// default — see `WalBackstopPolicy` for how that number was chosen). Growth is therefore bounded +/// even when the replicator never runs at all. +/// +/// Opting out is possible (`walBackstop: .disabled`) but is an assertion that something else bounds +/// the WAL. Nothing in this package does. +/// +/// ## The multi-pool caveat, which is load-bearing +/// +/// `StoreOpenGate` gives every opener its **own** `DatabasePool` on the same file — in this app both +/// `Repository` and `BLEManager` open one (see `StoreOpenGate`'s note). `wal_autocheckpoint` is +/// per-connection, so a second store opened with `.automatic` on the same file will happily restart +/// the WAL under a replicator that the first store's `.external` was protecting. **Every** opener of +/// a replicated file must pass `.external`, or the option does nothing. public enum WalCheckpointing: Sendable { /// SQLite's default: each connection auto-checkpoints at ~1000 WAL pages (~4 MB). This is the /// behaviour of every build that does not opt out, and the only behaviour upstream ships. case automatic /// Disable `wal_autocheckpoint` on every connection this store opens, because an external - /// component checkpoints instead. See the obligation documented above. + /// component checkpoints instead. See the obligation documented above — which the + /// `walBackstop` parameter of `init(path:walCheckpointing:walBackstop:)` discharges by default. case external } @@ -114,8 +132,15 @@ public actor WhoopStore { /// (the Pool serializes writes and runs reads in parallel under WAL). public nonisolated var registryWriter: any DatabaseWriter { dbWriter } - private init(dbWriter: any DatabaseWriter) throws { + /// Bounds WAL growth when `walCheckpointing` is `.external`. `nil` for `.automatic` (SQLite's own + /// autocheckpoint is the bound) and for a store with no backstop policy. Held strongly here + /// because GRDB's `.observerLifetime` registration keeps only a weak reference — this property is + /// what keeps the observer alive for the life of the store. + let walBackstop: WalBackstopMonitor? + + private init(dbWriter: any DatabaseWriter, walBackstop: WalBackstopMonitor? = nil) throws { self.dbWriter = dbWriter + self.walBackstop = walBackstop try WhoopStore.makeMigrator().migrate(dbWriter) } @@ -123,8 +148,9 @@ public actor WhoopStore { /// `StoreOpenGate` (below) opened the pool and migrated it under the process-wide open lock, so /// re-migrating here would be a redundant (and, if it raced a sibling opener, failing) second run. /// (#261) - private init(preMigrated dbWriter: any DatabaseWriter) { + private init(preMigrated dbWriter: any DatabaseWriter, walBackstop: WalBackstopMonitor? = nil) { self.dbWriter = dbWriter + self.walBackstop = walBackstop } /// Open (creating if needed) a database at `path` and run migrations. @@ -137,7 +163,14 @@ public actor WhoopStore { /// `walCheckpointing` defaults to `.automatic`, which is exactly the behaviour every build has /// today. Only an embedder that has taken over checkpointing should pass `.external` — see /// `WalCheckpointing` for the obligation that carries. - public init(path: String, walCheckpointing: WalCheckpointing = .automatic) async throws { + /// + /// `walBackstop` is inert under `.automatic` (SQLite's autocheckpoint already bounds the WAL) and + /// is what discharges `.external`'s obligation: it forces a checkpoint if the `-wal` sibling + /// crosses its ceiling no matter what the external replicator is doing. Pass `.disabled` only if + /// something outside this package provably bounds WAL growth. + public init(path: String, + walCheckpointing: WalCheckpointing = .automatic, + walBackstop: WalBackstopPolicy = .standard) async throws { var config = Configuration() config.prepareDatabase { db in // `DatabasePool` puts the database in WAL mode itself (reads run as concurrent snapshots @@ -159,7 +192,16 @@ public actor WhoopStore { } config.busyMode = .timeout(5) let pool = try await StoreOpenGate.shared.openAndMigrate(path: path, configuration: config) - self.init(preMigrated: pool) + + // Only `.external` needs a backstop: under `.automatic` SQLite's own autocheckpoint already + // bounds the WAL, and installing an observer there would be pure cost for upstream builds. + var monitor: WalBackstopMonitor? + if case .external = walCheckpointing, walBackstop.isEnabled { + let m = WalBackstopMonitor(databasePath: path, policy: walBackstop) + m.attach(to: pool) + monitor = m + } + self.init(preMigrated: pool, walBackstop: monitor) } /// Move aside a database file that has our data tables but no GRDB migration bookkeeping — the @@ -277,6 +319,38 @@ public actor WhoopStore { return found ? total : nil } + // MARK: - WAL backstop + + /// Size of the `-wal` sibling alone, in bytes; `0` when there is no WAL and `nil` for an + /// in-memory store. A single `stat(2)` — cheap enough to poll. Unlike `databaseFileSizeBytes()` + /// this isolates the component that `WalCheckpointing.external` puts at risk. + public nonisolated func walFileSizeBytes() -> Int64? { + let base = dbWriter.path + guard base != ":memory:", !base.isEmpty else { return nil } + var st = stat() + guard (base + "-wal").withCString({ stat($0, &st) }) == 0 else { return 0 } + return Int64(st.st_size) + } + + /// How many times the WAL backstop has forced a checkpoint because the `-wal` crossed its + /// ceiling. `0` when no backstop is installed (`.automatic`, or `.disabled`). + /// + /// **This is the number to watch on a replicated device.** A healthy external replicator keeps + /// the WAL near its own ~4 MB threshold, so any non-zero value means the replicator stopped + /// running long enough for the WAL to grow 16× past that — and that each firing may have cost + /// the replicator its incremental resume point. + public nonisolated var walBackstopFirings: Int { walBackstop?.firings ?? 0 } + + /// Forced checkpoints that finished with the WAL still over the ceiling — i.e. a `TRUNCATE` + /// blocked by a pool reader holding an open snapshot. Persistently non-zero means the ceiling is + /// not actually being enforced and a reader is pinning the WAL. + public nonisolated var walBackstopIneffectiveAttempts: Int { walBackstop?.ineffectiveAttempts ?? 0 } + + /// The installed backstop, or `nil` when none is (`.automatic`, or `.disabled`). `nonisolated` + /// for the same reason `registryWriter` is: the monitor does its own locking. Used by tests and + /// by the fork-side sync telemetry. + nonisolated var walBackstopMonitor: WalBackstopMonitor? { walBackstop } + // MARK: - Introspection (used by tests) public func tableNames() async throws -> Set { diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/WalBackstopTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/WalBackstopTests.swift new file mode 100644 index 0000000000..4545a7114d --- /dev/null +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/WalBackstopTests.swift @@ -0,0 +1,231 @@ +import XCTest +import GRDB +import WhoopProtocol +@testable import WhoopStore + +/// `WalCheckpointing.external` turns SQLite's autocheckpoint off, which leaves **nothing** bounding +/// WAL growth: the external replicator's own thresholds live inside its push loop, so they do not +/// fire when it is asleep, crashed, or never started. The backstop is what makes that safe. +/// +/// Two halves are tested separately on purpose: +/// +/// * `WalBackstopCore` is pure — threshold, in-flight suppression and retry backoff are asserted +/// with no database and no clock, so those rules cannot regress silently behind a slow test. +/// * The integration tests then prove the rule is actually wired into the write path: a real +/// `DatabasePool`, real commits, and a real `-wal` file that has to come back down. +final class WalBackstopTests: XCTestCase { + + private func tempPath() -> String { + NSTemporaryDirectory() + "whoopstore-backstop-\(UUID().uuidString).sqlite" + } + + private func removeDB(_ path: String) { + for suffix in ["", "-wal", "-shm"] { + try? FileManager.default.removeItem(atPath: path + suffix) + } + } + + private func walBytes(_ path: String) -> Int64 { + let attrs = try? FileManager.default.attributesOfItem(atPath: path + "-wal") + return (attrs?[.size] as? NSNumber)?.int64Value ?? 0 + } + + /// One transaction per call, mirroring NOOP's real commit shape (a `Collector` flush). + @discardableResult + private func writeTransactions(_ store: WhoopStore, count: Int, rowsEach: Int = 400, + startingAt ts0: Int = 0) async throws -> Int { + var ts = ts0 + for _ in 0.. Date: Mon, 27 Jul 2026 01:51:52 -0400 Subject: [PATCH 133/171] feat(cloudsync): local per-push telemetry for the page-replication device trial MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The one unanswered question gating a liters trial is how often a push degrades to a full snapshot. liters' verify.rs sets snapshotting = true when the WAL was truncated by another process, when the resume frame was overwritten, or when two checkpoint generations cycled unseen — and meta.rs:110-112 deliberately does not persist synced_to_wal_end across a writer close/reopen, which makes every app relaunch a candidate. Rare means incremental replication is a 25-380x win. Routine means it is a full upload with extra steps and the trial stops there. That is an empirical question about VK's device and nothing currently records it. SyncPushObservation mirrors liters' FFI PushSummary field-for-field plus the two the app supplies (WAL size at push time, seconds since the last push), as a plain Codable value type with no liters import — so this compiles, runs and is tested with no Rust in the project, and wiring the real thing later is a struct literal. Deliberately NOT a table in whoop.sqlite. Every insert would be another commit in the database whose commit rate is the thing being measured (~14 dirtied pages each, which is exactly what drives WAL growth), and every row would then be replicated by the replicator it is measuring. A small JSON file in Application Support is outside the replication set entirely. Application Support and not Caches: the OS evicts Caches under pressure, and losing the trial's only measurement to a cache purge would be silent. Cost per push is one stat(2) plus one atomic rewrite of a file capped at 200 records (~30 KB), at a cadence of single-digit pushes per day. Trimming drops old records but never the aggregate counters, so snapshotRate stays a rate over the whole trial rather than silently becoming a rate over the last 200 pushes — which would read identically and be wrong. reasonCounts attributes a bad rate to a specific verify() branch instead of leaving it a shrug, and snapshotByteShare separates frequency from cost (10% of pushes can be 99% of the bytes). NOT wired to anything: no liters in the Xcode project, no push path calls this yet. It is the instrument, built and verified ahead of the trial VK decides when to run. StrandTests/SyncPushTelemetryTests: 10 tests, 0 failures, via `xcodebuild -scheme Strand -destination platform=macOS test`. --- Strand/CloudSync/SyncPushTelemetry.swift | 216 +++++++++++++++++++++++ StrandTests/SyncPushTelemetryTests.swift | 178 +++++++++++++++++++ 2 files changed, 394 insertions(+) create mode 100644 Strand/CloudSync/SyncPushTelemetry.swift create mode 100644 StrandTests/SyncPushTelemetryTests.swift diff --git a/Strand/CloudSync/SyncPushTelemetry.swift b/Strand/CloudSync/SyncPushTelemetry.swift new file mode 100644 index 0000000000..61f7f61d20 --- /dev/null +++ b/Strand/CloudSync/SyncPushTelemetry.swift @@ -0,0 +1,216 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +#if CLOUD_SYNC +import Foundation + +/// One observed page-replication push, recorded locally so the device trial can answer the single +/// question that gates adopting `liters`: **how often does a push degrade to a full snapshot?** +/// +/// A snapshotting push ships the entire database. If that is rare, incremental page replication is +/// a 25–380× win over today's whole-file upload. If it is routine, `liters` is a full upload with +/// extra steps and the trial should stop there. Nothing about `PushSummary`'s other fields answers +/// this — a snapshot and a delta differ only in size, and size alone cannot distinguish "a big +/// day" from "re-uploaded everything". +/// +/// Fields mirror `liters`' FFI `PushSummary` one-for-one plus two the app supplies, so wiring the +/// real thing later is a struct literal and nothing else. Deliberately a plain value type with no +/// `liters` import: this compiles and is tested with no Rust in the project. +struct SyncPushObservation: Codable, Equatable, Sendable { + /// When the push completed. + var at: Date + /// True when the push wrote a full snapshot of the database rather than a WAL delta. + var snapshotted: Bool + /// `liters`' reason from its `verify()` decision tree, e.g. "wal truncated by another process". + /// One of a small fixed set, so it is safe to group on. `nil` when not snapshotting. + var snapshotReason: String? + /// Bytes of LTX uploaded. + var bytesUploaded: Int64 + /// Size of the `-wal` file when the push ran — the other half of the story, since a large WAL + /// is what precedes a foreign checkpoint and therefore a snapshot. + var walBytes: Int64 + /// Seconds since the previous recorded push; `nil` for the first one after install. + var secondsSinceLastPush: Double? + /// `liters`' replication position after the push, for correlating with the server. + var txid: UInt64 +} + +/// Aggregate view of the trial so far. `snapshotRate` is the go/no-go number. +struct SyncPushStats: Equatable { + var pushes: Int + var snapshots: Int + var bytesUploaded: Int64 + var bytesUploadedBySnapshots: Int64 + /// Count per `snapshotReason`, so a bad rate can be attributed to a branch rather than guessed at. + var reasonCounts: [String: Int] + var maxWalBytes: Int64 + var medianSecondsBetweenPushes: Double? + + /// Fraction of pushes that shipped the whole database. This is what decides the trial. + var snapshotRate: Double { pushes == 0 ? 0 : Double(snapshots) / Double(pushes) } + /// Fraction of uploaded bytes spent on snapshots — the cost, as opposed to the frequency. A low + /// rate can still be a bad deal if each snapshot is 766 MB. + var snapshotByteShare: Double { + bytesUploaded == 0 ? 0 : Double(bytesUploadedBySnapshots) / Double(bytesUploaded) + } +} + +/// Append-only, bounded, local recorder for `SyncPushObservation`s. +/// +/// ## Why not a table in the database +/// +/// The obvious place is a `syncPushLog` table in `whoop.sqlite`. That would be wrong twice over. +/// Every insert is another commit in the database whose *commit rate* is the thing being measured +/// (~14 dirtied pages each, which is exactly what drives WAL growth), and every row would then be +/// replicated by the replicator it is measuring — the instrument would be part of the experiment. +/// A small JSON file in Application Support is outside the replication set entirely. +/// +/// ## Cost +/// +/// One `stat(2)` for the WAL size plus one atomic rewrite of a file that is capped at +/// `maxRecords` (200 ≈ 30 KB). At the expected push cadence — single-digit per day, ~2/hour even +/// on an aggressive schedule — that is immaterial, and it is bounded no matter how long the trial +/// runs. Trimming drops old *records* but never the counters, so `stats` stays correct over the +/// whole trial while the record window stays small. +/// +/// Not an actor: pushes are already serialized by the replicator, and an `NSLock` keeps this usable +/// from a `BGTask` completion handler without an `await`. +final class SyncPushTelemetry: @unchecked Sendable { + /// Records kept on disk. Older ones are dropped; the aggregate counters are not. + static let maxRecords = 200 + + private let url: URL + private let lock = NSLock() + private var state: State + + private struct State: Codable { + var version: Int = 1 + var records: [SyncPushObservation] = [] + /// Monotonic totals, kept across trimming so `stats` covers the whole trial. + var totalPushes: Int = 0 + var totalSnapshots: Int = 0 + var totalBytes: Int64 = 0 + var totalSnapshotBytes: Int64 = 0 + var reasonCounts: [String: Int] = [:] + var maxWalBytes: Int64 = 0 + } + + /// The default location: `Application Support/CloudSync/push-telemetry.json`. Not in Caches — + /// the OS may evict Caches under pressure, and losing the trial's data to a cache purge would + /// be a silent, unrecoverable hole in the one measurement the trial exists for. + static func defaultURL() throws -> URL { + let dir = try FileManager.default + .url(for: .applicationSupportDirectory, in: .userDomainMask, appropriateFor: nil, create: true) + .appendingPathComponent("CloudSync", isDirectory: true) + try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + return dir.appendingPathComponent("push-telemetry.json") + } + + /// Loads existing state, or starts empty. A corrupt or unreadable file starts empty rather than + /// throwing: losing telemetry must never break a sync. + init(url: URL) { + self.url = url + if let data = try? Data(contentsOf: url), + let decoded = try? JSONDecoder().decode(State.self, from: data) { + self.state = decoded + } else { + self.state = State() + } + } + + /// Records one push. `walBytes` comes from `WhoopStore.walFileSizeBytes()`; + /// `secondsSinceLastPush` is derived here so callers cannot forget it. + /// + /// Best-effort persistence: a failed write loses the record, never the sync. + @discardableResult + func record(at now: Date = Date(), + snapshotted: Bool, + snapshotReason: String?, + bytesUploaded: Int64, + walBytes: Int64, + txid: UInt64) -> SyncPushObservation { + lock.lock() + defer { lock.unlock() } + + let previous = state.records.last?.at + let observation = SyncPushObservation( + at: now, + snapshotted: snapshotted, + snapshotReason: snapshotted ? snapshotReason : nil, + bytesUploaded: bytesUploaded, + walBytes: walBytes, + secondsSinceLastPush: previous.map { now.timeIntervalSince($0) }, + txid: txid) + + state.records.append(observation) + if state.records.count > Self.maxRecords { + state.records.removeFirst(state.records.count - Self.maxRecords) + } + state.totalPushes += 1 + state.totalBytes += bytesUploaded + if snapshotted { + state.totalSnapshots += 1 + state.totalSnapshotBytes += bytesUploaded + state.reasonCounts[snapshotReason ?? "unspecified", default: 0] += 1 + } + state.maxWalBytes = max(state.maxWalBytes, walBytes) + persist() + return observation + } + + /// The whole trial so far, including pushes whose records have since been trimmed. + var stats: SyncPushStats { + lock.lock() + defer { lock.unlock() } + let gaps = state.records.compactMap(\.secondsSinceLastPush).sorted() + return SyncPushStats( + pushes: state.totalPushes, + snapshots: state.totalSnapshots, + bytesUploaded: state.totalBytes, + bytesUploadedBySnapshots: state.totalSnapshotBytes, + reasonCounts: state.reasonCounts, + maxWalBytes: state.maxWalBytes, + medianSecondsBetweenPushes: gaps.isEmpty ? nil : gaps[gaps.count / 2]) + } + + /// The retained window, oldest first. For a diagnostics screen or a manual export. + var recentRecords: [SyncPushObservation] { + lock.lock() + defer { lock.unlock() } + return state.records + } + + /// A one-line summary for the log, so the trial is readable without a UI. + var oneLineSummary: String { + let s = stats + let pct = String(format: "%.1f%%", s.snapshotRate * 100) + let bytePct = String(format: "%.1f%%", s.snapshotByteShare * 100) + let reasons = s.reasonCounts.sorted { $0.value > $1.value } + .map { "\($0.key)=\($0.value)" }.joined(separator: ", ") + return "pushes=\(s.pushes) snapshots=\(s.snapshots) (\(pct) of pushes, \(bytePct) of bytes) " + + "maxWal=\(s.maxWalBytes)B" + + (reasons.isEmpty ? "" : " reasons[\(reasons)]") + } + + /// Discards everything. For starting a clean trial run. + func reset() { + lock.lock() + defer { lock.unlock() } + state = State() + persist() + } + + /// Atomic: write a sibling temp file and rename, so a kill mid-write cannot leave a truncated + /// JSON that the next launch silently discards as corrupt. + private func persist() { + guard let data = try? JSONEncoder().encode(state) else { return } + let tmp = url.appendingPathExtension("tmp") + do { + try data.write(to: tmp, options: .atomic) + _ = try? FileManager.default.replaceItemAt(url, withItemAt: tmp) + } catch { + try? FileManager.default.removeItem(at: tmp) + } + } +} +#endif diff --git a/StrandTests/SyncPushTelemetryTests.swift b/StrandTests/SyncPushTelemetryTests.swift new file mode 100644 index 0000000000..4cc6fbc3a0 --- /dev/null +++ b/StrandTests/SyncPushTelemetryTests.swift @@ -0,0 +1,178 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (StrandTests shares the app's +// OuraConfig.xcconfig, so the flag arrives with the app's). +#if CLOUD_SYNC +import XCTest +@testable import Strand + +/// `SyncPushTelemetry` exists to answer one question about a real device: how often does a +/// page-replication push degrade to a full snapshot? Everything asserted here is a property that +/// number depends on — if trimming lost the counters, or a corrupt file threw, or the gap between +/// pushes were computed from the wrong record, the trial would produce a plausible wrong answer +/// rather than an obvious failure. That is the risk being tested. +final class SyncPushTelemetryTests: XCTestCase { + + private var dir: URL! + + override func setUpWithError() throws { + dir = URL(fileURLWithPath: NSTemporaryDirectory()) + .appendingPathComponent("push-telemetry-\(UUID().uuidString)", isDirectory: true) + try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + } + + override func tearDownWithError() throws { + try? FileManager.default.removeItem(at: dir) + } + + private func makeURL() -> URL { dir.appendingPathComponent("push-telemetry.json") } + + private func record(_ t: SyncPushTelemetry, + at: Date, + snapshotted: Bool = false, + reason: String? = nil, + bytes: Int64 = 1_000, + wal: Int64 = 4_096, + txid: UInt64 = 1) { + t.record(at: at, snapshotted: snapshotted, snapshotReason: reason, + bytesUploaded: bytes, walBytes: wal, txid: txid) + } + + // MARK: - The number the trial turns on + + func testSnapshotRateAndByteShare() { + let t = SyncPushTelemetry(url: makeURL()) + let t0 = Date(timeIntervalSince1970: 1_000) + record(t, at: t0, snapshotted: true, reason: "first sync, no local position", bytes: 800_000_000) + for i in 1...9 { record(t, at: t0.addingTimeInterval(Double(i) * 600), bytes: 400_000) } + + let s = t.stats + XCTAssertEqual(s.pushes, 10) + XCTAssertEqual(s.snapshots, 1) + XCTAssertEqual(s.snapshotRate, 0.1, accuracy: 1e-9) + // Frequency and cost are different questions: 10% of pushes, >99% of the bytes. + XCTAssertGreaterThan(s.snapshotByteShare, 0.99) + } + + /// The reason histogram is what turns "the rate is bad" into "and here is which branch of + /// verify() caused it" — the difference between a decision and a shrug. + func testReasonsAreCountedPerBranch() { + let t = SyncPushTelemetry(url: makeURL()) + let t0 = Date(timeIntervalSince1970: 0) + record(t, at: t0, snapshotted: true, reason: "wal truncated by another process") + record(t, at: t0.addingTimeInterval(10), snapshotted: true, reason: "wal truncated by another process") + record(t, at: t0.addingTimeInterval(20), snapshotted: true, reason: "full or restart checkpoint detected, snapshotting") + record(t, at: t0.addingTimeInterval(30)) + + XCTAssertEqual(t.stats.reasonCounts, [ + "wal truncated by another process": 2, + "full or restart checkpoint detected, snapshotting": 1, + ]) + } + + /// A non-snapshotting push must not carry a reason even if one is passed — otherwise the + /// histogram would count branches that did not fire. + func testNonSnapshotPushDropsAnyReason() { + let t = SyncPushTelemetry(url: makeURL()) + let o = t.record(at: Date(), snapshotted: false, snapshotReason: "stale", + bytesUploaded: 10, walBytes: 20, txid: 3) + XCTAssertNil(o.snapshotReason) + XCTAssertTrue(t.stats.reasonCounts.isEmpty) + } + + // MARK: - Bounded, but not lossy where it counts + + /// The record window is capped so the file cannot grow without bound over a long trial — but + /// the aggregate counters must survive trimming, or the rate would silently become "the rate + /// over the last 200 pushes" while still being read as "the rate". + func testTrimmingDropsRecordsButNeverTheCounters() { + let t = SyncPushTelemetry(url: makeURL()) + let t0 = Date(timeIntervalSince1970: 0) + let n = SyncPushTelemetry.maxRecords + 50 + for i in 0.. Date: Mon, 27 Jul 2026 14:02:16 -0400 Subject: [PATCH 134/171] feat(store): make external WAL checkpointing a process-wide policy, not a per-call-site argument MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `WalCheckpointing.external` is a per-connection `PRAGMA wal_autocheckpoint = 0`. This app opens the same database from two independent places with two independent `DatabasePool`s — `Repository. ensureStore()` and `BLEManager.bootstrapStore()` — so selecting `.external` at one call site and not the other achieves nothing at all: the other pool still auto-checkpoints at ~4 MB, restarts the WAL under the replicator, and every push degrades to a full snapshot of the database. Nothing throws and nothing logs; the only symptom is that the uploads never get smaller. So the mode is not an argument the call sites must remember. `StoreReplication` supplies the DEFAULT VALUE of `WhoopStore(path:)`'s two checkpointing parameters, resolved per call. Both existing openers are correct without being touched, and so is any opener written later. Explicit arguments still override, which is what the tests use. Defaults are `.automatic` + `.standard`, i.e. byte-for-byte the behaviour of a build that never calls `configure`. Upstream compiles this file and gets SQLite's own autocheckpoint, unchanged. The one residual ordering hazard — `configure` landing after a store already opened, which leaves that pool on the old mode — is reported rather than left to be inferred from an unexplained snapshot rate: `configuredAfterFirstOpen` is queryable and logged. A throwing open does not count, so a `configure` between a failed open and its retry is not misreported as late. Tests assert behaviour, not bookkeeping: `PRAGMA wal_autocheckpoint` read back from real writer AND reader connections of real pools, including two separate stores opened on the same file. Packages/WhoopStore: 372 tests, 0 failures (8 new). --- .../Sources/WhoopStore/StoreReplication.swift | 107 ++++++++++ .../Sources/WhoopStore/WhoopStore.swift | 38 +++- .../StoreReplicationTests.swift | 183 ++++++++++++++++++ 3 files changed, 320 insertions(+), 8 deletions(-) create mode 100644 Packages/WhoopStore/Sources/WhoopStore/StoreReplication.swift create mode 100644 Packages/WhoopStore/Tests/WhoopStoreTests/StoreReplicationTests.swift diff --git a/Packages/WhoopStore/Sources/WhoopStore/StoreReplication.swift b/Packages/WhoopStore/Sources/WhoopStore/StoreReplication.swift new file mode 100644 index 0000000000..acb9a2fe6a --- /dev/null +++ b/Packages/WhoopStore/Sources/WhoopStore/StoreReplication.swift @@ -0,0 +1,107 @@ +import Foundation + +/// Process-wide default for the WAL-checkpointing parameters of `WhoopStore(path:)`. +/// +/// ## The hazard this exists to remove +/// +/// `WalCheckpointing.external` is a **per-connection** `PRAGMA wal_autocheckpoint = 0`, so it only +/// achieves anything if *every* connection to the file has it. This app opens the same database from +/// two independent places, each with its own `DatabasePool` — `Repository.ensureStore()` and +/// `BLEManager.bootstrapStore()` (see `StoreOpenGate`) — and both open through the same +/// `WhoopStore(path:)` initialiser. +/// +/// Passing `.external` as an argument at those two call sites would work, and would keep working +/// exactly until someone adds a third opener. The failure that follows is silent in the worst way: +/// the new pool auto-checkpoints at ~4 MB, restarts the WAL underneath the replicator, the +/// replicator finds its resume offset overwritten, and every push degrades to a full snapshot of the +/// whole database. Nothing throws. Nothing logs. The only symptom is that the upload never gets +/// smaller — i.e. the exact thing the replicator was adopted to fix, failing in a way that looks +/// like the replicator simply not being very good. +/// +/// So the mode is not an argument that call sites remember to pass. It is the **default value** of +/// `WhoopStore(path:)`'s parameters, resolved here at each call. Every opener that exists, and every +/// opener anyone writes later, is correct without knowing this type exists. Explicitly passing +/// `walCheckpointing:` still overrides it, which is what the tests do. +/// +/// ## Ordering +/// +/// `configure(...)` must run before the first store opens; afterwards it applies only to stores +/// opened later, which is a real (if narrow) way to get a half-configured process. That case is not +/// silent: it sets `configuredAfterFirstOpen`, which a diagnostics screen can read and a test can +/// assert on, and it logs. On iOS the correct call site is the top of `StrandiOSApp.init()`, ahead +/// of `AppModel()` — every store open in this app is `async` and therefore strictly later. +/// +/// ## Default +/// +/// `.automatic` + `.standard`, i.e. byte-for-byte the behaviour of a build that never calls +/// `configure`. Upstream links this file and gets SQLite's own autocheckpoint, unchanged. +public enum StoreReplication { + private static let lock = NSLock() + nonisolated(unsafe) private static var _walCheckpointing: WalCheckpointing = .automatic + nonisolated(unsafe) private static var _walBackstop: WalBackstopPolicy = .standard + nonisolated(unsafe) private static var _openCount = 0 + nonisolated(unsafe) private static var _configuredAfterFirstOpen = false + + /// The checkpointing mode a `WhoopStore(path:)` gets when the caller does not pass one. + public static var walCheckpointing: WalCheckpointing { + lock.lock(); defer { lock.unlock() } + return _walCheckpointing + } + + /// The WAL ceiling policy a `WhoopStore(path:)` gets when the caller does not pass one. Inert + /// under `.automatic`; under `.external` it is what bounds WAL growth if the replicator stops. + public static var walBackstop: WalBackstopPolicy { + lock.lock(); defer { lock.unlock() } + return _walBackstop + } + + /// How many stores have been opened through `WhoopStore(path:)` in this process. Only interesting + /// relative to `configure` — see `configuredAfterFirstOpen`. + public static var openedStoreCount: Int { + lock.lock(); defer { lock.unlock() } + return _openCount + } + + /// True if `configure` ran after a store had already opened, meaning at least one live pool is + /// still using the previous mode. Under `.external` that pool will keep auto-checkpointing and + /// will restart the WAL under the replicator. Surface it; do not let it be inferred from an + /// unexplained snapshot rate. + public static var configuredAfterFirstOpen: Bool { + lock.lock(); defer { lock.unlock() } + return _configuredAfterFirstOpen + } + + /// Set the process-wide default. Call once, before any store opens. + /// + /// Choosing `.external` is an assertion that something outside this package checkpoints the WAL. + /// `walBackstop` is what keeps that assertion from being load-bearing — see `WalCheckpointing`. + public static func configure(walCheckpointing: WalCheckpointing, + walBackstop: WalBackstopPolicy = .standard) { + lock.lock() + _walCheckpointing = walCheckpointing + _walBackstop = walBackstop + let late = _openCount > 0 + if late { _configuredAfterFirstOpen = true } + lock.unlock() + if late { + NSLog("StoreReplication: configure() ran AFTER %d store(s) were already open — those " + + "pools keep their previous checkpointing mode. Under .external they will restart " + + "the WAL under the replicator.", openedStoreCount) + } + } + + /// Restore the shipped defaults. Tests only — production configures once and never reverts. + public static func resetForTesting() { + lock.lock(); defer { lock.unlock() } + _walCheckpointing = .automatic + _walBackstop = .standard + _openCount = 0 + _configuredAfterFirstOpen = false + } + + /// Called by `WhoopStore(path:)` on every successful open, so a late `configure` can be detected. + static func noteStoreOpened() { + lock.lock(); defer { lock.unlock() } + _openCount += 1 + } +} diff --git a/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift b/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift index a02860d629..85af8610e4 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift @@ -138,8 +138,17 @@ public actor WhoopStore { /// what keeps the observer alive for the life of the store. let walBackstop: WalBackstopMonitor? - private init(dbWriter: any DatabaseWriter, walBackstop: WalBackstopMonitor? = nil) throws { + /// The checkpointing mode this store actually opened with, whether it came from an explicit + /// argument or from `StoreReplication`. Worth surfacing on a diagnostics screen next to + /// `walBackstopFirings`: "external + 0 firings" and "automatic" are very different states that + /// otherwise look identical from outside. + public nonisolated let walCheckpointing: WalCheckpointing + + private init(dbWriter: any DatabaseWriter, + walCheckpointing: WalCheckpointing = .automatic, + walBackstop: WalBackstopMonitor? = nil) throws { self.dbWriter = dbWriter + self.walCheckpointing = walCheckpointing self.walBackstop = walBackstop try WhoopStore.makeMigrator().migrate(dbWriter) } @@ -148,8 +157,11 @@ public actor WhoopStore { /// `StoreOpenGate` (below) opened the pool and migrated it under the process-wide open lock, so /// re-migrating here would be a redundant (and, if it raced a sibling opener, failing) second run. /// (#261) - private init(preMigrated dbWriter: any DatabaseWriter, walBackstop: WalBackstopMonitor? = nil) { + private init(preMigrated dbWriter: any DatabaseWriter, + walCheckpointing: WalCheckpointing = .automatic, + walBackstop: WalBackstopMonitor? = nil) { self.dbWriter = dbWriter + self.walCheckpointing = walCheckpointing self.walBackstop = walBackstop } @@ -160,17 +172,24 @@ public actor WhoopStore { /// Open + migrate runs through `StoreOpenGate` so two concurrent openers of the SAME file never /// run their GRDB migrators at once (#261) — see that actor's note for the failure it prevents. /// - /// `walCheckpointing` defaults to `.automatic`, which is exactly the behaviour every build has - /// today. Only an embedder that has taken over checkpointing should pass `.external` — see - /// `WalCheckpointing` for the obligation that carries. + /// Both checkpointing parameters default to the process-wide `StoreReplication` policy, which is + /// itself `.automatic` + `.standard` unless something called `StoreReplication.configure`. So a + /// build that never configures replication behaves exactly as every build does today. + /// + /// The default deliberately comes from a process-wide value rather than being written out at each + /// call site: `wal_autocheckpoint` is per-connection, this app opens the same file from two + /// independent places, and an opener that forgets `.external` silently defeats it for all the + /// others. See `StoreReplication` for the full argument. Only an embedder that has taken over + /// checkpointing should select `.external` — see `WalCheckpointing` for the obligation that + /// carries. Passing the arguments explicitly still overrides the policy, which is what tests do. /// /// `walBackstop` is inert under `.automatic` (SQLite's autocheckpoint already bounds the WAL) and /// is what discharges `.external`'s obligation: it forces a checkpoint if the `-wal` sibling /// crosses its ceiling no matter what the external replicator is doing. Pass `.disabled` only if /// something outside this package provably bounds WAL growth. public init(path: String, - walCheckpointing: WalCheckpointing = .automatic, - walBackstop: WalBackstopPolicy = .standard) async throws { + walCheckpointing: WalCheckpointing = StoreReplication.walCheckpointing, + walBackstop: WalBackstopPolicy = StoreReplication.walBackstop) async throws { var config = Configuration() config.prepareDatabase { db in // `DatabasePool` puts the database in WAL mode itself (reads run as concurrent snapshots @@ -201,7 +220,10 @@ public actor WhoopStore { m.attach(to: pool) monitor = m } - self.init(preMigrated: pool, walBackstop: monitor) + // Recorded after the open succeeds, so a `configure` that lands between a failed open and a + // successful retry is not reported as late. See `StoreReplication.configuredAfterFirstOpen`. + StoreReplication.noteStoreOpened() + self.init(preMigrated: pool, walCheckpointing: walCheckpointing, walBackstop: monitor) } /// Move aside a database file that has our data tables but no GRDB migration bookkeeping — the diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/StoreReplicationTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/StoreReplicationTests.swift new file mode 100644 index 0000000000..47e7c3552f --- /dev/null +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/StoreReplicationTests.swift @@ -0,0 +1,183 @@ +import XCTest +import GRDB +@testable import WhoopStore + +/// `StoreReplication` is the process-wide default behind `WhoopStore(path:)`'s checkpointing +/// parameters. It exists because `wal_autocheckpoint` is per-connection and this app opens the same +/// database from two independent places (`Repository.ensureStore()` and `BLEManager.bootstrapStore()`), +/// so a single opener that does not select `.external` silently defeats it for every other opener — +/// and the only symptom is that the replicator's uploads never get smaller. +/// +/// These tests assert the *behaviour* — `PRAGMA wal_autocheckpoint` as observed on real pooled +/// connections — not that a stored enum echoes its argument back. +/// +/// Every test resets the global in `tearDown`. `WalCheckpointingTests.testDefaultLeaves…` asserts the +/// stock 1000-page autocheckpoint on a bare `WhoopStore(path:)`, and XCTest runs the whole bundle in +/// one process, so a leaked `.external` here would fail that test instead of this one. +final class StoreReplicationTests: XCTestCase { + + /// Reset on BOTH sides. `tearDown` alone is not enough: `openedStoreCount` is process-wide, and + /// the ~90 other test classes in this bundle open real stores between these tests, so a count + /// assertion that only cleaned up after itself passed in isolation and failed in the full run. + override func setUp() { + super.setUp() + StoreReplication.resetForTesting() + } + + override func tearDown() { + StoreReplication.resetForTesting() + super.tearDown() + } + + private func tempPath() -> String { + NSTemporaryDirectory() + "whoopstore-storerep-\(UUID().uuidString).sqlite" + } + + private func removeDB(_ path: String) { + for suffix in ["", "-wal", "-shm"] { + try? FileManager.default.removeItem(atPath: path + suffix) + } + } + + private func autocheckpoint(_ store: WhoopStore, reader: Bool) async throws -> Int { + let sql = "PRAGMA wal_autocheckpoint" + return reader + ? try await store.registryWriter.read { try Int.fetchOne($0, sql: sql) ?? -1 } + : try await store.registryWriter.write { try Int.fetchOne($0, sql: sql) ?? -1 } + } + + // MARK: - Defaults + + /// An unconfigured process is upstream's process. This is the property that lets this file live + /// in an upstream-shared package at all. + func testUnconfiguredDefaultsAreUpstreamBehaviour() { + StoreReplication.resetForTesting() + if case .external = StoreReplication.walCheckpointing { + XCTFail("default checkpointing must be .automatic") + } + XCTAssertEqual(StoreReplication.walBackstop, .standard) + XCTAssertFalse(StoreReplication.configuredAfterFirstOpen) + } + + // MARK: - The policy reaches a store that passes no arguments + + /// The whole point: an opener written with no knowledge of replication — `WhoopStore(path:)`, + /// which is literally what both real openers call — still gets `.external`. + func testConfiguredExternalReachesAnOpenerThatPassesNoArguments() async throws { + let path = tempPath() + defer { removeDB(path) } + StoreReplication.configure(walCheckpointing: .external) + + let store = try await WhoopStore(path: path) + + let writer = try await autocheckpoint(store, reader: false) + let reader = try await autocheckpoint(store, reader: true) + XCTAssertEqual(writer, 0, "policy must disable autocheckpoint on the writer connection") + XCTAssertEqual(reader, 0, "policy must disable autocheckpoint on reader connections too") + if case .automatic = store.walCheckpointing { + XCTFail("store should report the mode it actually opened with") + } + } + + /// Two independently-opened stores on the SAME file both get the policy. This is the multi-opener + /// case that the argument-at-each-call-site approach cannot guarantee: a second pool still + /// auto-checkpointing at ~4 MB restarts the WAL under the replicator and forces a full snapshot. + func testEverySeparateOpenerOfTheSameFileGetsThePolicy() async throws { + let path = tempPath() + defer { removeDB(path) } + StoreReplication.configure(walCheckpointing: .external) + + let first = try await WhoopStore(path: path) + let second = try await WhoopStore(path: path) + + for (name, store) in [("first", first), ("second", second)] { + let writer = try await autocheckpoint(store, reader: false) + let reader = try await autocheckpoint(store, reader: true) + XCTAssertEqual(writer, 0, "\(name) opener's writer must have autocheckpoint off") + XCTAssertEqual(reader, 0, "\(name) opener's readers must have autocheckpoint off") + } + XCTAssertEqual(StoreReplication.openedStoreCount, 2) + } + + /// The backstop that discharges `.external`'s obligation must be installed when the mode came + /// from the policy, not only when it was passed explicitly. Without this, a policy-configured + /// store would have autocheckpoint off and nothing at all bounding WAL growth. + func testPolicySelectedExternalStillInstallsTheWalBackstop() async throws { + let path = tempPath() + defer { removeDB(path) } + StoreReplication.configure(walCheckpointing: .external, walBackstop: .standard) + + let store = try await WhoopStore(path: path) + + XCTAssertNotNil(store.walBackstopMonitor, + "policy-selected .external must install a backstop, or nothing bounds the WAL") + } + + /// A caller that opts out of the backstop through the policy gets no monitor — the same escape + /// hatch the explicit argument has, and the same assertion it carries. + func testPolicyCanDisableTheBackstop() async throws { + let path = tempPath() + defer { removeDB(path) } + StoreReplication.configure(walCheckpointing: .external, walBackstop: .disabled) + + let store = try await WhoopStore(path: path) + + XCTAssertNil(store.walBackstopMonitor) + } + + // MARK: - Explicit arguments still win + + /// The policy is a default, not an override. Tests (and any future caller with a specific need) + /// keep full control. + func testExplicitArgumentOverridesThePolicy() async throws { + let path = tempPath() + defer { removeDB(path) } + StoreReplication.configure(walCheckpointing: .external) + + let store = try await WhoopStore(path: path, walCheckpointing: .automatic) + + let writer = try await autocheckpoint(store, reader: false) + let reader = try await autocheckpoint(store, reader: true) + XCTAssertEqual(writer, 1000) + XCTAssertEqual(reader, 1000) + } + + // MARK: - Late configuration is reported, not silent + + /// Configuring after a store is already open leaves that store's pool auto-checkpointing. That is + /// a genuine half-configured process, and the failure it produces (a replicator that snapshots + /// forever) is otherwise indistinguishable from the replicator being bad. It must be observable. + func testConfiguringAfterAStoreIsOpenIsFlagged() async throws { + let path = tempPath() + defer { removeDB(path) } + + let early = try await WhoopStore(path: path) + XCTAssertFalse(StoreReplication.configuredAfterFirstOpen) + + StoreReplication.configure(walCheckpointing: .external) + + XCTAssertTrue(StoreReplication.configuredAfterFirstOpen, + "a late configure must be visible; it leaves live pools on the old mode") + // And the claim is literally true of the already-open store: it kept SQLite's autocheckpoint. + let earlyPragma = try await autocheckpoint(early, reader: false) + XCTAssertEqual(earlyPragma, 1000) + + // Stores opened afterwards do get the new policy. + let path2 = tempPath() + defer { removeDB(path2) } + let late = try await WhoopStore(path: path2) + let latePragma = try await autocheckpoint(late, reader: false) + XCTAssertEqual(latePragma, 0) + } + + /// A failed open must not count: `configure` landing between a throwing open and a successful + /// retry is on time, not late. `quarantineIncompatibleDatabase` + the migrator make a throwing + /// open a real thing on this path (#222, #261). + func testAFailedOpenDoesNotCountAsAnOpenStore() async { + let dir = NSTemporaryDirectory() + "storerep-nodir-\(UUID().uuidString)/nested" + _ = try? await WhoopStore(path: dir + "/whoop.sqlite") + + XCTAssertEqual(StoreReplication.openedStoreCount, 0, + "an open that threw must not be counted, or a legitimate configure reads as late") + } +} From 2e093f88927221347061970b026651fce5f7d60c Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 27 Jul 2026 14:02:16 -0400 Subject: [PATCH 135/171] feat(cloudsync): the page-replication trial switch, and record every push MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The decision to replace whole-database upload with page replication rests on one number that cannot be reasoned about and has to be measured on this device: how often a push degrades to a full snapshot. iOS relaunches the process constantly, and a replicator that loses its resume offset re-uploads everything. Two things have to exist before that number can be collected. 1. `SyncReplicationTrial` — one `UserDefaults` flag, default OFF, and a launch hook that hands WAL checkpointing to an external replicator via `StoreReplication`. Called at the top of `StrandiOSApp.init()`, ahead of `AppModel()`: a pool opened earlier keeps SQLite's autocheckpoint for its whole life, and one such pool is enough to defeat `.external` everywhere. Enabling it changes no upload behaviour at all, so the downside is bounded to "the WAL is checkpointed by the 64 MiB backstop instead of by SQLite at 4 MB". 2. A record per delivered sync on the existing `/ingest` path. Today every one is `snapshotted: true, reason: full-ingest` — not a placeholder, the honest baseline the replicator gets compared against. What it buys before a replicator exists is the other half of each record: the `-wal` size sampled BEFORE the exporter's `checkpointWAL()` truncates it, and the interval between pushes. Those decide whether `.external` is safe here, and neither is observable anywhere else. `SyncReplicationTrial.shared` is inert under XCTest for the same reason `CloudSyncModel.autoSyncIfDue` refuses to run there: StrandTests executes inside the full app via TEST_HOST and resolves the real Application Support directory, so a live recorder would append junk to the one file the trial exists to produce. Also noted in place: `CloudSyncUploader`'s `checkpointWAL()` is exactly what a page replicator must be the sole caller of. Leaving it here alongside a running replicator would restart the WAL under it and force a full snapshot every sync — the upload path defeating the thing replacing it. --- Strand/CloudSync/CloudSyncUploader.swift | 42 ++++++- Strand/CloudSync/SyncReplicationTrial.swift | 97 +++++++++++++++ StrandTests/CloudSyncUploaderTests.swift | 120 +++++++++++++++++++ StrandTests/SyncReplicationTrialTests.swift | 123 ++++++++++++++++++++ StrandiOS/App/StrandiOSApp.swift | 10 ++ 5 files changed, 390 insertions(+), 2 deletions(-) create mode 100644 Strand/CloudSync/SyncReplicationTrial.swift create mode 100644 StrandTests/SyncReplicationTrialTests.swift diff --git a/Strand/CloudSync/CloudSyncUploader.swift b/Strand/CloudSync/CloudSyncUploader.swift index cadb15981c..2e765d5746 100644 --- a/Strand/CloudSync/CloudSyncUploader.swift +++ b/Strand/CloudSync/CloudSyncUploader.swift @@ -64,8 +64,19 @@ enum CloudSyncUploader { /// `defer`, whatever happens: success, an export failure, or a network failure. The DB can be /// 100-300MB, so `CloudSyncClient.ingest` streams it from this file via /// `URLSession.upload(for:fromFile:)` rather than loading it into memory. + /// + /// `telemetry` records one `SyncPushObservation` per successful upload. Today every one of them + /// is `snapshotted: true` with reason `full-ingest`, because `/ingest` *is* a full snapshot — + /// that is not a placeholder, it is the honest baseline the page-replication trial is measured + /// against. What the field actually buys before a replicator exists is the other half of each + /// record: the `-wal` size at push time and the interval between pushes. Those are the two + /// numbers that decide whether `WalCheckpointing.external` is safe on this device, and neither is + /// observable anywhere else. Injectable so a test writes to a temp directory rather than the + /// app's real Application Support. static func upload(store: WhoopStore, client: any CloudIngesting, - exporter: Exporter = defaultExporter) async throws -> (bytes: Int, latestDay: String?) { + exporter: Exporter = defaultExporter, + telemetry: SyncPushTelemetry? = SyncReplicationTrial.shared) + async throws -> (bytes: Int, latestDay: String?) { // Refuse an empty/trivial store BEFORE touching export or the network at all — see // `CloudSyncUploadError.emptyStore`'s doc comment for the incident this guards against. // `dailyMetric` is written by every ingest path (BLE-derived recompute, WHOOP/Apple @@ -77,9 +88,36 @@ enum CloudSyncUploader { let tempURL = cachesDir.appendingPathComponent("cloudsync-upload-\(UUID().uuidString).noopbak") defer { try? FileManager.default.removeItem(at: tempURL) } + // Sampled BEFORE the exporter runs, because `defaultExporter`'s first act is + // `checkpointWAL()` (a `wal_checkpoint(TRUNCATE)`), which leaves the `-wal` at ~0 bytes. + // Reading it afterwards would record that zero and measure nothing. What is wanted is how far + // the WAL had grown by the time a sync began — the quantity `WalCheckpointing.external` puts + // at risk. One `stat(2)`; nil only for an in-memory test store. + // + // Worth flagging for the replicator work that follows: that same `checkpointWAL()` is exactly + // what a page replicator must be the only caller of. Leaving it here while a replicator is + // also running would restart the WAL underneath it and force a full snapshot on every sync — + // i.e. the upload path would defeat the thing it is being replaced by. + let walBytes = store.walFileSizeBytes() ?? 0 + switch await exporter(store, tempURL) { case .exported: - return try await client.ingest(fileURL: tempURL) + let result = try await client.ingest(fileURL: tempURL) + // Recorded only on success, deliberately: a failed upload shipped nothing, and counting it + // would understate the byte cost per delivered sync. Never allowed to throw — `record` is + // best-effort by construction (see `SyncPushTelemetry.persist`). + if let telemetry { + telemetry.record(snapshotted: true, + snapshotReason: "full-ingest", + bytesUploaded: Int64(result.bytes), + walBytes: walBytes, + txid: 0) + // The trial has no UI. This line is how it is read — off a device console, or from the + // container's log — so it has to carry the aggregate and not just this one push. + NSLog("SyncPushTelemetry: %@ walAtPush=%lld backstopFirings=%d", + telemetry.oneLineSummary, walBytes, store.walBackstopFirings) + } + return result case .failure(let message): throw CloudSyncUploadError.exportFailed(message) case .cancelled, .imported: diff --git a/Strand/CloudSync/SyncReplicationTrial.swift b/Strand/CloudSync/SyncReplicationTrial.swift new file mode 100644 index 0000000000..99f7a39cab --- /dev/null +++ b/Strand/CloudSync/SyncReplicationTrial.swift @@ -0,0 +1,97 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig). A default build contains none of this code, +// keeping "fully offline" a byte-level property of the shipped binary, not a runtime promise. +#if CLOUD_SYNC +import Foundation +import WhoopStore + +/// The on/off switch and launch hook for the page-replication device trial. +/// +/// ## What the trial is for +/// +/// `noop-cloud`'s current sync uploads the whole database (153.4 MB, 90–150 s) on every sync. The +/// replacement ships WAL frames instead. The one number that decides whether that is a 25–380× win +/// or a full upload with extra steps is **how often a push degrades to a full snapshot**, and it +/// cannot be reasoned about — iOS relaunches the process constantly, and a replicator that loses its +/// resume offset re-uploads everything. It has to be measured on VK's device, on VK's data. +/// +/// Enabling the trial does two things and nothing else: +/// +/// 1. hands WAL checkpointing to an external replicator (`StoreReplication.configure(.external)`), +/// which is the precondition for a replicator to see every transaction; and +/// 2. records one `SyncPushObservation` per push (`SyncPushTelemetry`). +/// +/// It uploads nothing differently. Today's `/ingest` path is untouched, so the trial's downside is +/// bounded to "the WAL is now checkpointed by the backstop at 64 MiB instead of by SQLite at 4 MB." +/// +/// ## Default: off +/// +/// `.external` is a real commitment — with autocheckpoint disabled, WAL growth is bounded only by +/// `WalBackstopPolicy` (see `WalCheckpointing`'s obligation note, and the measured 166 MB/day). It +/// stays off until VK turns it on, on his own device, and it is a plain `UserDefaults` flag rather +/// than a Keychain item because it is a preference, not a secret. +enum SyncReplicationTrial { + /// `UserDefaults` key. Absent (the shipped state) reads as `false`. + static let enabledKey = "cloudsync.replicationTrialEnabled" + + /// Whether the trial is on. Reading this is *not* the same as asking whether `.external` is in + /// force — a flip only takes effect at the next launch, because `applyAtLaunch` runs before any + /// store opens and a live `DatabasePool`'s `wal_autocheckpoint` cannot be changed after the fact. + /// Ask `StoreReplication.walCheckpointing` for the state that is actually in force. + static var isEnabled: Bool { + get { UserDefaults.standard.bool(forKey: enabledKey) } + set { UserDefaults.standard.set(newValue, forKey: enabledKey) } + } + + /// Install the process-wide store policy. **Must be called before the first `WhoopStore` open**, + /// i.e. at the top of the app's `init()`, ahead of `AppModel()`. + /// + /// Both real openers (`Repository.ensureStore()`, `BLEManager.bootstrapStore()`) are `async` and + /// therefore strictly later than `init()`, including the CoreBluetooth state-restoration path — + /// UIKit constructs the `App` before it delivers a restoration callback. If that ever stops being + /// true, `StoreReplication.configuredAfterFirstOpen` flips to `true` and says so rather than + /// leaving a half-configured process to be inferred from a bad snapshot rate. + /// + /// Deliberately does nothing when the trial is off: the `.automatic` branch would set the same + /// values `StoreReplication` already ships, and not calling `configure` at all keeps + /// `openedStoreCount`/`configuredAfterFirstOpen` meaningful for a build that never opted in. + static func applyAtLaunch(enabled: Bool = isEnabled) { + guard enabled else { return } + StoreReplication.configure(walCheckpointing: .external, walBackstop: .standard) + // Deliberately does not build `statusLine` here: that touches `shared`, and this runs on the + // launch-critical path before the first frame. The interesting half of the trial's state is + // logged after each push, where file I/O has already happened. + NSLog("SyncReplicationTrial: ON — external WAL checkpointing, backstop at %lld bytes", + StoreReplication.walBackstop.ceilingBytes) + } + + /// The shared recorder, opened lazily so a build with the trial off never creates the file. + /// `nil` if Application Support is unavailable (it never is on a real device) — telemetry is + /// best-effort and must never be able to fail a sync. + /// + /// Also `nil` under XCTest, for the same reason `CloudSyncModel.autoSyncIfDue` refuses to run + /// there: `StrandTests` executes inside the full app via `TEST_HOST`, so it resolves the developer + /// Mac's REAL Application Support directory. Without this, every test that exercises an upload + /// would append junk records to the live trial file and corrupt the one measurement this exists to + /// produce. Tests that want to assert on telemetry inject their own instance over a temp URL. + static let shared: SyncPushTelemetry? = { + guard !CloudSyncModel.isRunningUnderXCTest else { return nil } + guard let url = try? SyncPushTelemetry.defaultURL() else { return nil } + return SyncPushTelemetry(url: url) + }() + + /// One line describing the trial's state, for the log and for a diagnostics row. Combines the + /// *intent* (the flag), the *reality* (what the store actually opened with), and the *result* + /// (the telemetry aggregate) — the three are not the same and have to be read together. + static var statusLine: String { + let inForce: String + switch StoreReplication.walCheckpointing { + case .external: inForce = "external" + case .automatic: inForce = "automatic" + } + let late = StoreReplication.configuredAfterFirstOpen ? " LATE-CONFIGURE" : "" + let telemetry = shared.map { " " + $0.oneLineSummary } ?? "" + return "replicationTrial enabled=\(isEnabled) checkpointing=\(inForce)\(late)\(telemetry)" + } +} +#endif diff --git a/StrandTests/CloudSyncUploaderTests.swift b/StrandTests/CloudSyncUploaderTests.swift index 21ec711779..a6e8ffcacb 100644 --- a/StrandTests/CloudSyncUploaderTests.swift +++ b/StrandTests/CloudSyncUploaderTests.swift @@ -193,4 +193,124 @@ final class CloudSyncModelAutoSyncGateTests: XCTestCase { XCTAssertTrue(CloudSyncModel.isRunningUnderXCTest) } } + +/// The push-telemetry hook on the upload path. This is the instrument for the page-replication +/// trial, so the properties that matter are the ones that keep the *measurement* honest: a record +/// per delivered sync, none for a sync that delivered nothing, and a WAL size that was sampled at a +/// moment when it means something. +final class CloudSyncUploaderTelemetryTests: XCTestCase { + override func setUp() { super.setUp(); OuraURLProtocolStub.reset() } + + private func makeClient() -> CloudSyncClient { + CloudSyncClient(baseURL: URL(string: "https://cloud.example.com")!, token: "tok", + session: OuraURLProtocolStub.session()) + } + + private func makeTelemetry() -> SyncPushTelemetry { + let url = URL(fileURLWithPath: NSTemporaryDirectory()) + .appendingPathComponent("push-telemetry-\(UUID().uuidString).json") + addTeardownBlock { try? FileManager.default.removeItem(at: url) } + return SyncPushTelemetry(url: url) + } + + private func nonEmptyStore() async throws -> WhoopStore { + let store = try await WhoopStore.inMemory() + try await store.upsertDailyMetrics([ + DailyMetric(day: "2026-07-11", totalSleepMin: nil, efficiency: nil, deepMin: nil, remMin: nil, + lightMin: nil, disturbances: nil, restingHr: nil, avgHrv: nil, recovery: nil, + strain: nil, exerciseCount: nil), + ], deviceId: "devA") + return store + } + + private func fakeExporter() -> CloudSyncUploader.Exporter { + { _, dest in try? Data("bytes".utf8).write(to: dest); return .exported(dest) } + } + + /// One record per successful upload, carrying the server's own byte count. `snapshotted` is true + /// with reason `full-ingest` because `/ingest` genuinely ships the whole database — this is the + /// baseline the replicator is measured against, and labelling it as anything else would make the + /// eventual comparison meaningless. + func testRecordsOneSnapshotObservationPerSuccessfulUpload() async throws { + OuraURLProtocolStub.queue = [.init(status: 200, + body: #"{"ok":true,"bytes":153400000,"latestDay":"2026-07-11"}"#.data(using: .utf8)!)] + let telemetry = makeTelemetry() + + _ = try await CloudSyncUploader.upload(store: try await nonEmptyStore(), client: makeClient(), + exporter: fakeExporter(), telemetry: telemetry) + + let stats = telemetry.stats + XCTAssertEqual(stats.pushes, 1) + XCTAssertEqual(stats.snapshots, 1) + XCTAssertEqual(stats.bytesUploaded, 153_400_000) + XCTAssertEqual(stats.snapshotRate, 1.0, accuracy: 0.0001, + "today every /ingest IS a full snapshot; the baseline must say so") + XCTAssertEqual(stats.reasonCounts["full-ingest"], 1) + } + + /// A failed POST delivered no bytes. Counting it would understate cost-per-delivered-sync and + /// would make the trial's byte comparison against the replicator wrong in the replicator's favour. + func testDoesNotRecordWhenTheUploadFails() async throws { + OuraURLProtocolStub.queue = [.init(status: 507, body: Data())] + let telemetry = makeTelemetry() + + do { + _ = try await CloudSyncUploader.upload(store: try await nonEmptyStore(), client: makeClient(), + exporter: fakeExporter(), telemetry: telemetry) + XCTFail("a 507 must propagate") + } catch {} + + XCTAssertEqual(telemetry.stats.pushes, 0) + } + + /// The empty-store guard refuses before export or network. Nothing was pushed, so nothing is + /// recorded — otherwise a device that never syncs would still accumulate a push history. + func testDoesNotRecordWhenTheEmptyStoreGuardRefuses() async throws { + let telemetry = makeTelemetry() + + do { + _ = try await CloudSyncUploader.upload(store: try await WhoopStore.inMemory(), + client: makeClient(), + exporter: fakeExporter(), telemetry: telemetry) + XCTFail("an empty store must be refused") + } catch {} + + XCTAssertEqual(telemetry.stats.pushes, 0) + } + + /// Repeated uploads accumulate, and the gap between them is derived rather than passed in — the + /// push cadence is half the trial's answer (a snapshot every launch is fine at 1/day and fatal at + /// 8/day), and a caller that had to remember to supply it would eventually not. + func testAccumulatesAcrossPushesAndDerivesTheInterval() async throws { + OuraURLProtocolStub.queue = [ + .init(status: 200, body: #"{"ok":true,"bytes":10,"latestDay":"2026-07-11"}"#.data(using: .utf8)!), + .init(status: 200, body: #"{"ok":true,"bytes":20,"latestDay":"2026-07-12"}"#.data(using: .utf8)!), + ] + let telemetry = makeTelemetry() + let store = try await nonEmptyStore() + + _ = try await CloudSyncUploader.upload(store: store, client: makeClient(), + exporter: fakeExporter(), telemetry: telemetry) + _ = try await CloudSyncUploader.upload(store: store, client: makeClient(), + exporter: fakeExporter(), telemetry: telemetry) + + XCTAssertEqual(telemetry.stats.pushes, 2) + XCTAssertEqual(telemetry.stats.bytesUploaded, 30) + XCTAssertNil(telemetry.recentRecords.first?.secondsSinceLastPush) + XCTAssertNotNil(telemetry.recentRecords.last?.secondsSinceLastPush, + "the second push must carry a measured interval") + } + + /// Telemetry is an instrument, not a dependency. A caller that passes none still uploads. + func testUploadSucceedsWithNoTelemetry() async throws { + OuraURLProtocolStub.queue = [.init(status: 200, + body: #"{"ok":true,"bytes":7,"latestDay":"2026-07-11"}"#.data(using: .utf8)!)] + + let result = try await CloudSyncUploader.upload(store: try await nonEmptyStore(), + client: makeClient(), + exporter: fakeExporter(), telemetry: nil) + + XCTAssertEqual(result.bytes, 7) + } +} #endif // CLOUD_SYNC diff --git a/StrandTests/SyncReplicationTrialTests.swift b/StrandTests/SyncReplicationTrialTests.swift new file mode 100644 index 0000000000..584291e681 --- /dev/null +++ b/StrandTests/SyncReplicationTrialTests.swift @@ -0,0 +1,123 @@ +// Tests the CLOUD_SYNC-gated page-replication trial switch; compiled only when the flag is set +// (StrandTests shares the app's OuraConfig.xcconfig, so flag + creds arrive together). +#if CLOUD_SYNC +import XCTest +import WhoopStore +@testable import Strand + +/// `SyncReplicationTrial` is the one switch that decides whether this device hands WAL checkpointing +/// to an external replicator. Getting its default wrong is not a cosmetic bug: `.external` disables +/// SQLite's autocheckpoint process-wide, and on a measured 166 MB/day of WAL growth the only thing +/// left bounding the file is `WalBackstopPolicy`. So "off unless VK turned it on" is the property +/// under test, not an implementation detail. +final class SyncReplicationTrialTests: XCTestCase { + + override func setUp() { + super.setUp() + UserDefaults.standard.removeObject(forKey: SyncReplicationTrial.enabledKey) + StoreReplication.resetForTesting() + } + + override func tearDown() { + UserDefaults.standard.removeObject(forKey: SyncReplicationTrial.enabledKey) + StoreReplication.resetForTesting() + super.tearDown() + } + + // MARK: - Default off + + /// A fresh install has never written the key. `UserDefaults.bool(forKey:)` reads a missing key as + /// `false`, and the whole safety argument for shipping this rests on that. + func testDisabledWhenTheKeyWasNeverWritten() { + XCTAssertFalse(SyncReplicationTrial.isEnabled) + } + + /// The launch hook must leave an opted-out process byte-for-byte identical to today's: SQLite's + /// own autocheckpoint, no backstop observer on the pool, nothing configured at all. + func testApplyAtLaunchIsANoOpWhenDisabled() { + SyncReplicationTrial.applyAtLaunch(enabled: false) + + if case .external = StoreReplication.walCheckpointing { + XCTFail("a disabled trial must not hand checkpointing to an external replicator") + } + XCTAssertEqual(StoreReplication.openedStoreCount, 0) + } + + // MARK: - Opted in + + /// Turning it on selects `.external` *and* keeps the standard backstop. The pairing is the point: + /// `.external` alone would leave WAL growth unbounded if the replicator never runs, which is a + /// strictly worse failure than the full re-upload it is meant to avoid. + func testApplyAtLaunchSelectsExternalWithABackstopWhenEnabled() { + SyncReplicationTrial.applyAtLaunch(enabled: true) + + if case .automatic = StoreReplication.walCheckpointing { + XCTFail("an enabled trial must select .external") + } + XCTAssertEqual(StoreReplication.walBackstop, .standard) + XCTAssertTrue(StoreReplication.walBackstop.isEnabled, + "the backstop is what discharges .external's obligation; it cannot be off") + } + + /// The flag round-trips through `UserDefaults`, which is how VK flips it on his own device. + func testEnabledFlagRoundTrips() { + SyncReplicationTrial.isEnabled = true + XCTAssertTrue(SyncReplicationTrial.isEnabled) + SyncReplicationTrial.isEnabled = false + XCTAssertFalse(SyncReplicationTrial.isEnabled) + } + + /// The default argument reads the live flag, so the real launch hook and these tests exercise the + /// same code path rather than the tests only proving the injectable overload works. + func testApplyAtLaunchDefaultsToTheStoredFlag() { + SyncReplicationTrial.isEnabled = true + + SyncReplicationTrial.applyAtLaunch() + + if case .automatic = StoreReplication.walCheckpointing { + XCTFail("applyAtLaunch() with no argument must read the stored flag") + } + } + + // MARK: - Status + + /// Intent and reality are separate facts and the status line has to carry both: flipping the flag + /// does NOT change a running process (a live pool's `wal_autocheckpoint` is fixed at open), so a + /// diagnostics row that showed only the flag would claim external checkpointing was in force + /// during the entire session before the next launch. + func testStatusLineSeparatesTheFlagFromWhatIsActuallyInForce() { + SyncReplicationTrial.isEnabled = true + + let line = SyncReplicationTrial.statusLine + + XCTAssertTrue(line.contains("enabled=true"), line) + XCTAssertTrue(line.contains("checkpointing=automatic"), // not applied yet — no relaunch + "status must report the mode actually in force, not the flag: \(line)") + } + + /// A late `configure` leaves live pools on the old mode, which is exactly the state that produces + /// a permanently-snapshotting replicator with no other symptom. It must reach the status line. + func testStatusLineSurfacesALateConfigure() async throws { + let path = NSTemporaryDirectory() + "trial-late-\(UUID().uuidString).sqlite" + defer { for s in ["", "-wal", "-shm"] { try? FileManager.default.removeItem(atPath: path + s) } } + _ = try await WhoopStore(path: path) + + SyncReplicationTrial.applyAtLaunch(enabled: true) + + XCTAssertTrue(SyncReplicationTrial.statusLine.contains("LATE-CONFIGURE"), + SyncReplicationTrial.statusLine) + } + + // MARK: - Test-host isolation + + /// `StrandTests` runs inside the full app via `TEST_HOST`, so `SyncPushTelemetry.defaultURL()` + /// resolves the developer Mac's real Application Support. If the shared recorder were live here, + /// every upload test would append records to the live trial file — silently corrupting the single + /// measurement the trial exists to produce. Same guard, same reason, as + /// `CloudSyncModel.autoSyncIfDue`'s XCTest refusal. + func testSharedRecorderIsInertUnderXCTest() { + XCTAssertNil(SyncReplicationTrial.shared, + "the shared recorder must not write to the real container from a test run") + } +} +#endif diff --git a/StrandiOS/App/StrandiOSApp.swift b/StrandiOS/App/StrandiOSApp.swift index 6b823e7854..1bfc5c046c 100644 --- a/StrandiOS/App/StrandiOSApp.swift +++ b/StrandiOS/App/StrandiOSApp.swift @@ -36,6 +36,16 @@ struct StrandiOSApp: App { @AppStorage(ChartStyle.storageKey) private var chartStyleRaw = ChartStyle.titanium.rawValue init() { + #if CLOUD_SYNC + // FIRST, before anything that could open the database. `WalCheckpointing` is a per-connection + // PRAGMA applied at pool-open time, so a store opened before this line keeps SQLite's own + // autocheckpoint for its whole life — and one such pool restarting the WAL is enough to defeat + // `.external` for every other opener in the process. Both real openers are `async` (so strictly + // later than this `init`), and `StoreReplication.configuredAfterFirstOpen` reports it loudly if + // that ever stops being true. No-op unless VK has switched the trial on. See + // `SyncReplicationTrial`. + SyncReplicationTrial.applyAtLaunch() + #endif #if DEBUG // DEBUG-only promo-screenshot harness: when launched with `--demo-hour `, pin Today to that // hour's day-cycle scene + a per-hour stat frame. No-op (active stays nil) when the arg is absent. From 94ad31fa747aa9b3e4483c4ef70a3bf4bd7982d1 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 27 Jul 2026 14:10:19 -0400 Subject: [PATCH 136/171] build(liters): pin the liters git dependency and build it against system SQLite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds Rust/, NOOP's build surface for liters — Litestream v0.5-compatible SQLite replication (Kurt Mackey's library, MIT; we track the standalone derivative at vishk23/liters-mobile while upstream review is paused). `noop-liters` has no logic. It exists so the dependency is pinned, builds, and makes exactly one non-negotiable feature choice on NOOP's behalf: `default-features = false` on liters-ffi, which turns off `bundled-sqlite` so liters links the platform libsqlite3 instead of compiling its own. That is the prerequisite, not a preference. NoopLocalAccess and StrandImport both depend on GRDB.swift 6.29.3, which links Apple's system libsqlite3 via `.systemLibrary(name: "CSQLite")`. A bundled liters would put a second SQLite in the process, and the two do not share the process-global `unixInodeInfo` table SQLite uses to work around POSIX's "close any descriptor, lose all locks" rule — so either can silently drop the other's advisory locks. For liters that is a correctness bug rather than a hazard: its guarantee that no foreign checkpointer restarts the WAL under the writer *is* a long-running read lock, and losing it forces a full-database snapshot upload on the next push. Verified on this tree: - cargo build resolves all five liters crates from the git remote and compiles clean; Cargo.lock pins the exact commit. - the built staticlib DEFINES 0 sqlite3_* symbols and leaves 47 undefined, and the cdylib resolves them against /usr/lib/libsqlite3.dylib — the same library GRDB uses. - uniffi library-mode bindgen generates the full Swift surface from the wrapper's embedded metadata (LitersWriter/LitersReplica/LitersManager, HttpClient host-transport seam). Nothing in the app calls this yet: no Swift target imports the bindings and project.yml does not reference the xcframework. Wiring NOOP's sync path to liters is separate work; this is the dependency it builds against. --- .gitignore | 6 + Rust/Cargo.lock | 995 +++++++++++++++++++++++++++++++++ Rust/Cargo.toml | 50 ++ Rust/README.md | 68 +++ Rust/build-ios.sh | 65 +++ Rust/src/bin/uniffi-bindgen.rs | 7 + Rust/src/lib.rs | 18 + 7 files changed, 1209 insertions(+) create mode 100644 Rust/Cargo.lock create mode 100644 Rust/Cargo.toml create mode 100644 Rust/README.md create mode 100755 Rust/build-ios.sh create mode 100644 Rust/src/bin/uniffi-bindgen.rs create mode 100644 Rust/src/lib.rs diff --git a/.gitignore b/.gitignore index 9fb7ac1553..1c573661c0 100644 --- a/.gitignore +++ b/.gitignore @@ -94,6 +94,12 @@ export.zip __pycache__/ *.pyc +# ── Rust (liters dependency) ──────────────────────────────────── +# Build output only. Rust/Cargo.lock IS tracked: it pins the exact liters +# commit this tree builds against, which a branch-tracking git dependency +# otherwise leaves floating. +Rust/target/ + # Local release artifacts (zips/ipas built for upload; never tracked) dist/ err.txt diff --git a/Rust/Cargo.lock b/Rust/Cargo.lock new file mode 100644 index 0000000000..b5b375bf90 --- /dev/null +++ b/Rust/Cargo.lock @@ -0,0 +1,995 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anstyle" +version = "1.0.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "askama" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1bf825125edd887a019d0a3a837dcc5499a68b0d034cc3eb594070c3e18addc" +dependencies = [ + "askama_macros", + "itoa", + "percent-encoding", + "serde", + "serde_json", +] + +[[package]] +name = "askama_derive" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1c7065972a130eafa84215f21352ae15b4a7393da48c1f5e103904490736738" +dependencies = [ + "askama_parser", + "basic-toml", + "glob", + "memchr", + "proc-macro2", + "quote", + "rustc-hash", + "serde", + "serde_derive", + "syn 2.0.119", +] + +[[package]] +name = "askama_macros" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0e23b1d2c4bd39a41971f6124cef4cc6fd0540913ecb90919b69ab3bbe44ae1a" +dependencies = [ + "askama_derive", +] + +[[package]] +name = "askama_parser" +version = "0.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7db09fde9143e7ac4513358fb32ee32847125b63b18ea715afd487956da715da" +dependencies = [ + "rustc-hash", + "serde", + "serde_derive", + "unicode-ident", + "winnow", +] + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "basic-toml" +version = "0.1.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ba62675e8242a4c4e806d12f11d136e626e6c8361d6b829310732241652a178a" +dependencies = [ + "serde", +] + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "camino" +version = "1.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5f2d30e4173c4026932d51d31d6b0613b1fd3014bf3f9f8943d4ba139c437ba0" +dependencies = [ + "serde_core", +] + +[[package]] +name = "cargo-platform" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dd0061da739915fae12ea00e16397555ed4371a6bb285431aab930f61b0aa4ba" +dependencies = [ + "serde", + "serde_core", +] + +[[package]] +name = "cargo_metadata" +version = "0.23.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef987d17b0a113becdd19d3d0022d04d7ef41f9efe4f3fb63ac44ba61df3ade9" +dependencies = [ + "camino", + "cargo-platform", + "semver", + "serde", + "serde_json", + "thiserror", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "clap" +version = "4.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d91e0c145792ef73a6ad36d27c75ac09f1832222a3c209689d90f534685ee5b7" +dependencies = [ + "clap_builder", + "clap_derive", +] + +[[package]] +name = "clap_builder" +version = "4.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f09628afdcc538b57f3c6341e9c8e9970f18e4a481690a64974d7023bd33548b" +dependencies = [ + "anstyle", + "clap_lex", + "strsim", +] + +[[package]] +name = "clap_derive" +version = "4.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d012d2b9d65aca7f18f4d9878a045bc17899bba951561ba5ec3c2ba1eed9a061" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "clap_lex" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" + +[[package]] +name = "crc" +version = "3.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" +dependencies = [ + "crc-catalog", +] + +[[package]] +name = "crc-catalog" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853" + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "fallible-iterator" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2acce4a10f12dc2fb14a218589d4f1f62ef011b2d0cc4b3cb1bba8e94da14649" + +[[package]] +name = "fallible-streaming-iterator" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a" + +[[package]] +name = "fastrand" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223" + +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + +[[package]] +name = "fs-err" +version = "3.3.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b91aa448ca50d7e79433bdf3ee8d99215430d2ec02ade5aefab2a073a1822e8a" +dependencies = [ + "autocfg", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "libc", + "r-efi 5.3.0", + "wasip2", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi 6.0.0", +] + +[[package]] +name = "glob" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e4eba85ea1d0a966a983acd07deee566e67395d2d96b6fb39e62b5a833f1eb0b" + +[[package]] +name = "goblin" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b363a30c165f666402fe6a3024d3bec7ebc898f96a4a23bd1c99f8dbf3f4f47" +dependencies = [ + "log", + "plain", + "scroll", +] + +[[package]] +name = "hashbrown" +version = "0.15.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "foldhash", +] + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + +[[package]] +name = "hashlink" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7382cf6263419f2d8df38c55d7da83da5c18aef87fc7a7fc1fb1e344edfe14c1" +dependencies = [ + "hashbrown 0.15.5", +] + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "indexmap" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" +dependencies = [ + "equivalent", + "hashbrown 0.17.1", + "serde", + "serde_core", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "libsqlite3-sys" +version = "0.35.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "133c182a6a2c87864fe97778797e46c7e999672690dc9fa3ee8e241aa4a9c13f" +dependencies = [ + "pkg-config", + "vcpkg", +] + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "liters" +version = "0.1.0" +source = "git+https://github.com/vishk23/liters-mobile.git?branch=main#6835f9f9f6fcfb2005005857981f4d53a522db93" +dependencies = [ + "libc", + "liters-storage", + "liters-wal", + "ltx", + "rand", + "rusqlite", + "thiserror", +] + +[[package]] +name = "liters-ffi" +version = "0.1.0" +source = "git+https://github.com/vishk23/liters-mobile.git?branch=main#6835f9f9f6fcfb2005005857981f4d53a522db93" +dependencies = [ + "liters", + "thiserror", + "uniffi", +] + +[[package]] +name = "liters-storage" +version = "0.1.0" +source = "git+https://github.com/vishk23/liters-mobile.git?branch=main#6835f9f9f6fcfb2005005857981f4d53a522db93" +dependencies = [ + "libc", + "ltx", + "thiserror", +] + +[[package]] +name = "liters-wal" +version = "0.1.0" +source = "git+https://github.com/vishk23/liters-mobile.git?branch=main#6835f9f9f6fcfb2005005857981f4d53a522db93" +dependencies = [ + "thiserror", +] + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "ltx" +version = "0.1.0" +source = "git+https://github.com/vishk23/liters-mobile.git?branch=main#6835f9f9f6fcfb2005005857981f4d53a522db93" +dependencies = [ + "crc", + "lz4_flex", + "thiserror", + "twox-hash", +] + +[[package]] +name = "lz4_flex" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ef0d4ed8669f8f8826eb00dc878084aa8f253506c4fd5e8f58f5bce72ddb97e" + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "minimal-lexical" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" + +[[package]] +name = "nom" +version = "7.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a" +dependencies = [ + "memchr", + "minimal-lexical", +] + +[[package]] +name = "noop-liters" +version = "0.1.0" +dependencies = [ + "liters-ffi", + "uniffi", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "pkg-config" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" + +[[package]] +name = "plain" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4596b6d070b27117e987119b4dac604f3c58cfb0b191112e24771b2faeac1a6" + +[[package]] +name = "ppv-lite86" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" +dependencies = [ + "zerocopy", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rand" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" +dependencies = [ + "rand_chacha", + "rand_core", +] + +[[package]] +name = "rand_chacha" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" +dependencies = [ + "ppv-lite86", + "rand_core", +] + +[[package]] +name = "rand_core" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" +dependencies = [ + "getrandom 0.3.4", +] + +[[package]] +name = "rusqlite" +version = "0.37.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "165ca6e57b20e1351573e3729b958bc62f0e48025386970b6e4d29e7a7e71f3f" +dependencies = [ + "bitflags", + "fallible-iterator", + "fallible-streaming-iterator", + "hashlink", + "libsqlite3-sys", + "smallvec", +] + +[[package]] +name = "rustc-hash" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d" + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + +[[package]] +name = "scroll" +version = "0.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ab8598aa408498679922eff7fa985c25d58a90771bd6be794434c5277eab1a6" +dependencies = [ + "scroll_derive", +] + +[[package]] +name = "scroll_derive" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1783eabc414609e28a5ba76aee5ddd52199f7107a0b24c2e9746a1ecc34a683d" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" +dependencies = [ + "serde", + "serde_core", +] + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_spanned" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6662b5879511e06e8999a8a235d848113e942c9124f211511b16466ee2995f26" +dependencies = [ + "serde_core", +] + +[[package]] +name = "siphasher" +version = "1.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "smawk" +version = "0.3.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8e2fb0f499abb4d162f2bedad68f5ef91a1682b5a03596ddb67efd37768d100" + +[[package]] +name = "static_assertions" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f" + +[[package]] +name = "strsim" +version = "0.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "tempfile" +version = "3.27.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" +dependencies = [ + "fastrand", + "getrandom 0.4.3", + "once_cell", + "rustix", + "windows-sys", +] + +[[package]] +name = "textwrap" +version = "0.16.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c13547615a44dc9c452a8a534638acdf07120d4b6847c8178705da06306a3057" +dependencies = [ + "smawk", +] + +[[package]] +name = "thiserror" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "toml" +version = "1.1.3+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53c96ecdfa941c8fc4fcaed14f99ada8ebed502eef533015095a07e3301d4c3c" +dependencies = [ + "indexmap", + "serde_core", + "serde_spanned", + "toml_datetime", + "toml_parser", + "toml_writer", + "winnow", +] + +[[package]] +name = "toml_datetime" +version = "1.1.1+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3165f65f62e28e0115a00b2ebdd37eb6f3b641855f9d636d3cd4103767159ad7" +dependencies = [ + "serde_core", +] + +[[package]] +name = "toml_parser" +version = "1.1.3+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d38ac1cf9b95face32296c0a3ede1fdc270627c9d9c02a7274dd6d960dc4d56" +dependencies = [ + "winnow", +] + +[[package]] +name = "toml_writer" +version = "1.1.2+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d56353a2a665ad0f41a421187180aab746c8c325620617ad883a99a1cbe66d2" + +[[package]] +name = "twox-hash" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8464ec13c3691491391d9fce00f6416c9a48e46972f72d7865688be2080192c9" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "uniffi" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a782a48d72cfd7a2d65cfc7c691dbf5375c43104b3c195f7eccc716dcc3540c8" +dependencies = [ + "anyhow", + "camino", + "cargo_metadata", + "clap", + "uniffi_bindgen", + "uniffi_core", + "uniffi_macros", + "uniffi_pipeline", +] + +[[package]] +name = "uniffi_bindgen" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "533b0312c73e3b54eb78a4b257ceae390962dd4767995778309a74644643f9ac" +dependencies = [ + "anyhow", + "askama", + "camino", + "cargo_metadata", + "fs-err", + "glob", + "goblin", + "heck", + "indexmap", + "once_cell", + "serde", + "tempfile", + "textwrap", + "toml", + "uniffi_internal_macros", + "uniffi_meta", + "uniffi_pipeline", + "uniffi_udl", +] + +[[package]] +name = "uniffi_core" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e32e261c5b0dfaba6488f536e71957dddd6b1a498ac7eb791bee56b60a086be" +dependencies = [ + "anyhow", + "bytes", + "once_cell", + "static_assertions", +] + +[[package]] +name = "uniffi_internal_macros" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "84ae78069a5e6772ef694fd5bdb628532c88d2c2f0e7142bf6a384636eadb1af" +dependencies = [ + "anyhow", + "indexmap", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "uniffi_macros" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330be6770532e86320df31f54c70bb0be67588594e8e77fa56e9083a3fed5d0d" +dependencies = [ + "camino", + "fs-err", + "once_cell", + "proc-macro2", + "quote", + "serde", + "syn 2.0.119", + "toml", + "uniffi_meta", +] + +[[package]] +name = "uniffi_meta" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78de021f5547e56ab16c665a49d67d4fd3d31e77422f7739a2e9359d328cd9e7" +dependencies = [ + "anyhow", + "siphasher", + "uniffi_internal_macros", + "uniffi_pipeline", +] + +[[package]] +name = "uniffi_pipeline" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f8201bb1907ed8a42d80e11cbc25c8a033e7a31c3cff1d911f56eedb81d4948" +dependencies = [ + "anyhow", + "heck", + "indexmap", + "tempfile", + "uniffi_internal_macros", +] + +[[package]] +name = "uniffi_udl" +version = "0.32.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6e57996bc58009cc29bf04845d627ae313c2547b87171c1c349d6c51a1656c0" +dependencies = [ + "anyhow", + "textwrap", + "uniffi_meta", + "weedle2", +] + +[[package]] +name = "vcpkg" +version = "0.2.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" + +[[package]] +name = "wasip2" +version = "1.0.4+wasi-0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" +dependencies = [ + "wit-bindgen", +] + +[[package]] +name = "weedle2" +version = "5.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "998d2c24ec099a87daf9467808859f9d82b61f1d9c9701251aea037f514eae0e" +dependencies = [ + "nom", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "winnow" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23b97319f7b8343df12cc98938e5c3eb436064524c8d2b4e30a1d3a36eecdf81" +dependencies = [ + "memchr", +] + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "zerocopy" +version = "0.8.55" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b5a105cd7b140f6eeec8acff2ea38135d3cab283ada58540f629fe51e46696eb" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.55" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0fe976fb70c78cd64cccfe3a6fc142244e8a77b70959b30faf9d0ac37ee228eb" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/Rust/Cargo.toml b/Rust/Cargo.toml new file mode 100644 index 0000000000..cbf064651a --- /dev/null +++ b/Rust/Cargo.toml @@ -0,0 +1,50 @@ +[package] +name = "noop-liters" +version = "0.1.0" +edition = "2021" +publish = false +description = "NOOP's build surface for liters: pins the liters-ffi git dependency and builds it against the same SQLite GRDB links." + +[lib] +name = "noop_liters" +# `staticlib` is what the iOS xcframework wraps. `cdylib` exists so +# `uniffi-bindgen --library` has a host-native library to read the embedded +# interface metadata out of; `lib` so the bindgen bin can depend on this crate. +crate-type = ["lib", "staticlib", "cdylib"] + +[[bin]] +name = "uniffi-bindgen" +path = "src/bin/uniffi-bindgen.rs" + +[dependencies] +# liters — Litestream v0.5-compatible SQLite replication, by Kurt Mackey. +# Upstream is https://github.com/mrkurt/liters; this points at the standalone +# MIT-licensed derivative we develop in while upstream review is paused. +# +# `default-features = false` is LOAD-BEARING, not tidiness. It turns off +# liters' `bundled-sqlite`, so liters links the platform's libsqlite3 instead +# of compiling its own copy of the amalgamation into the staticlib. +# +# NOOP already links a SQLite: Packages/NoopLocalAccess and Packages/StrandImport +# both depend on GRDB.swift 6.29.3, which links Apple's system libsqlite3 +# through `.systemLibrary(name: "CSQLite")`. A bundled liters would put a +# SECOND SQLite in the same process, and the two do not share the +# process-global `unixInodeInfo` table SQLite uses to work around POSIX's +# "close any descriptor, lose all locks" rule — so either library can silently +# drop the other's advisory locks. For liters that is a correctness bug rather +# than a hazard: its guarantee that no foreign checkpointer restarts the WAL +# under the writer *is* a long-running read lock, and losing it forces a +# full-database snapshot upload on the next push. +# +# Do not add `features = ["bundled-sqlite"]` here. If a future change needs +# bindings generated against the platform's own sqlite3.h, use +# `features = ["system-sqlite-bindgen"]` (keeping default-features = false). +liters-ffi = { git = "https://github.com/vishk23/liters-mobile.git", branch = "main", default-features = false } + +# Must stay on the same major/minor as liters-ffi's uniffi, or the generated +# bindings and the shipped scaffolding come from different versions. +uniffi = { version = "0.32", features = ["cli"] } + +[profile.release] +lto = true +strip = true diff --git a/Rust/README.md b/Rust/README.md new file mode 100644 index 0000000000..8e2da00496 --- /dev/null +++ b/Rust/README.md @@ -0,0 +1,68 @@ +# Rust — the liters dependency + +This directory is NOOP's build surface for +[liters](https://github.com/vishk23/liters-mobile): Litestream v0.5-compatible +SQLite replication, embeddable in iOS and Android apps. liters is +**[Kurt Mackey's](https://github.com/mrkurt/liters)** library, MIT-licensed; +`liters-mobile` is the standalone derivative we develop in while upstream +review is paused. + +`noop-liters` contains no logic. It exists so NOOP can pin liters as a git +dependency and build it with the one feature choice NOOP cannot get wrong, then +package it as an xcframework plus generated Swift bindings. + +## The SQLite rule + +**liters must link the system libsqlite3, never a bundled copy.** + +`Packages/NoopLocalAccess` and `Packages/StrandImport` both depend on +GRDB.swift 6.29.3, which links Apple's system libsqlite3 through +`.systemLibrary(name: "CSQLite")`. If liters bundled its own SQLite there would +be two SQLite libraries in one process, and they do not share the +process-global `unixInodeInfo` table SQLite uses to work around POSIX's "close +any descriptor to a file, lose all your locks on it" rule. Either library can +then silently drop the other's advisory locks. + +For most SQLite users that is a latent hazard. For liters it is a direct +correctness bug: the writer's guarantee that no foreign checkpointer restarts +the WAL underneath it *is* a long-running read lock. Drop that lock and a +foreign checkpoint restarts the WAL, the resume frame is overwritten, and the +next push recovers the only way it can — by uploading a full snapshot of the +database, which is the exact upload liters was adopted to avoid. + +`Cargo.toml` enforces this with `default-features = false` on `liters-ffi`, and +`build-ios.sh` offers no way to opt back in. Verify it on the built archive: + +```sh +nm -g target/aarch64-apple-ios/release/libnoop_liters.a | grep -c ' T _sqlite3_' +# 0 — defines none; they stay undefined and resolve against the same +# libsqlite3 GRDB uses (iOS SDK usr/lib/libsqlite3.tbd exports them) +``` + +## Building + +```sh +rustup target add aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios +./build-ios.sh +``` + +Outputs `target/apple/Liters.xcframework` and `target/apple/swift/*.swift`. +Neither is committed, and neither is referenced by `project.yml` yet. + +To move the pin to a newer liters commit: + +```sh +cargo update -p liters-ffi # then commit the Cargo.lock change +``` + +`Cargo.lock` is committed on purpose: it records the exact liters commit this +tree builds against, so a checkout is reproducible even though the dependency +tracks a branch. + +## What is deliberately not here + +Nothing in the app calls liters yet — no Swift target imports the bindings, and +`project.yml` does not reference the xcframework. Wiring NOOP's sync path to +liters is separate work (delta sync, stage 1 of +`noop-cloud/docs/SYNC_BUILD_VS_BUY.md` §1.3). This directory exists so that work +has a real, building dependency to start from rather than a plan. diff --git a/Rust/build-ios.sh b/Rust/build-ios.sh new file mode 100755 index 0000000000..50af458c49 --- /dev/null +++ b/Rust/build-ios.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +# Builds Liters.xcframework + Swift bindings for NOOP from the pinned +# liters-ffi git dependency. +# +# Prereqs: +# rustup target add aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios +# Xcode command line tools +# +# There is deliberately no bundled/system switch here. NOOP links GRDB, which +# links Apple's system libsqlite3, so liters MUST link that same libsqlite3 — +# `Cargo.toml` pins `liters-ffi` with `default-features = false` to guarantee +# it, and this script has no way to opt back in. See Cargo.toml for why two +# SQLite copies in one process is a correctness bug rather than a size problem. +# +# Nothing in the app consumes the output yet; this builds the dependency so the +# sync work has something real to build against. +set -euo pipefail +cd "$(dirname "$0")" + +OUT=target/apple +BINDINGS=$OUT/swift +DEVICE_TARGET=aarch64-apple-ios +SIM_TARGETS=(aarch64-apple-ios-sim x86_64-apple-ios) + +for t in "$DEVICE_TARGET" "${SIM_TARGETS[@]}"; do + echo "==> building $t" + cargo build -p noop-liters --release --target "$t" +done + +# Generate Swift bindings from the host library's embedded metadata, built from +# the same sources and features as the device libraries so the bindings cannot +# drift from the shipped staticlib. +echo "==> generating Swift bindings" +cargo build -p noop-liters --release +rm -rf "$BINDINGS" && mkdir -p "$BINDINGS" +cargo run -p noop-liters --bin uniffi-bindgen -- generate \ + --library target/release/libnoop_liters.dylib \ + --language swift --out-dir "$BINDINGS" + +# Headers directory for the xcframework: the C header + module map. +HEADERS=$OUT/headers +rm -rf "$HEADERS" && mkdir -p "$HEADERS" +cp "$BINDINGS"/*.h "$HEADERS"/ +# uniffi emits a .modulemap; xcodebuild wants module.modulemap +cat "$BINDINGS"/*.modulemap > "$HEADERS"/module.modulemap + +# Fat simulator library. +mkdir -p "$OUT/sim" +lipo -create \ + $(for t in "${SIM_TARGETS[@]}"; do echo "target/$t/release/libnoop_liters.a"; done) \ + -output "$OUT/sim/libnoop_liters.a" + +rm -rf "$OUT/Liters.xcframework" +xcodebuild -create-xcframework \ + -library "target/$DEVICE_TARGET/release/libnoop_liters.a" -headers "$HEADERS" \ + -library "$OUT/sim/libnoop_liters.a" -headers "$HEADERS" \ + -output "$OUT/Liters.xcframework" + +echo +echo "xcframework: $(pwd)/$OUT/Liters.xcframework" +echo "swift sources: $(pwd)/$BINDINGS/*.swift" +echo +echo "Sanity check — the device archive must DEFINE no sqlite3_* symbols and" +echo "leave them undefined, so they resolve against the libsqlite3 GRDB uses:" +echo " nm -g target/$DEVICE_TARGET/release/libnoop_liters.a | grep -c ' T _sqlite3_' # expect 0" diff --git a/Rust/src/bin/uniffi-bindgen.rs b/Rust/src/bin/uniffi-bindgen.rs new file mode 100644 index 0000000000..b5c179ad70 --- /dev/null +++ b/Rust/src/bin/uniffi-bindgen.rs @@ -0,0 +1,7 @@ +// UniFFI's bindings generator, built from this workspace so it is the exact +// uniffi version the scaffolding was compiled with. `liters-ffi` ships the same +// bin, but it arrives here as a git dependency and `cargo run -p` cannot reach +// a dependency's binaries. +fn main() { + uniffi::uniffi_bindgen_main() +} diff --git a/Rust/src/lib.rs b/Rust/src/lib.rs new file mode 100644 index 0000000000..0f0f704715 --- /dev/null +++ b/Rust/src/lib.rs @@ -0,0 +1,18 @@ +//! NOOP's build surface for [liters](https://github.com/vishk23/liters-mobile). +//! +//! This crate deliberately contains no logic. It exists so NOOP can pin liters +//! as a git dependency and build it with the feature set NOOP needs — chiefly +//! `default-features = false`, which makes liters link the system libsqlite3 +//! that GRDB already links instead of bundling a second SQLite into the +//! process. See `Cargo.toml` for why that matters. +//! +//! Re-exporting `liters_ffi` keeps its UniFFI scaffolding — the `#[no_mangle]` +//! entry points and the embedded interface metadata that +//! `uniffi-bindgen --library` reads — reachable from this crate's staticlib and +//! cdylib, so the generated Swift bindings and the shipped archive describe the +//! same API. +//! +//! Nothing in NOOP calls this yet. Wiring the app's sync path to it is separate +//! work; this crate is the dependency those changes build against. + +pub use liters_ffi::*; From e8dfa653a30e7456ec130e67e2d30442646e6701 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 27 Jul 2026 14:15:48 -0400 Subject: [PATCH 137/171] harden(store): stop the integrity probe from being able to checkpoint a replicated store MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `DatabaseIntegrity.quickCheckFailure` opens its own GRDB connection, and a grep of the whole tree says it is the ONLY connection that can reach the live database without going through `WhoopStore(path:)` — and therefore without `StoreReplication`'s checkpointing policy. Its read-write fallback used a bare `Configuration()`, so that connection carried SQLite's stock 1000-page autocheckpoint. Not a live bug, and the honest reason it is not is uncomfortable: autocheckpoint fires at the end of a WRITE transaction and this probe only reads; the close-time checkpoint only runs for the LAST connection, which this is not while the app's pool is open; and the read-write branch is only reached when the read-only open failed for want of a `-shm`, which implies no other connection was open anyway. That is three independent facts about SQLite internals holding up the resume point of a replicator. Replaced with the guarantee: the connection is not permitted to checkpoint. No behaviour change for the case the fallback exists for (probing a checkpointed backup with no `-wal` sibling) — the close-time checkpoint that cleans up the `-wal` it creates is a separate mechanism and is unaffected by `wal_autocheckpoint`. Test asserts the observable property — probing a live `.external` store does not shrink its `-wal` — rather than "the pragma was set", so it holds whichever branch the probe takes. DatabaseIntegrityTests: 6 tests, 0 failures. --- .../WhoopStore/DatabaseIntegrity.swift | 16 ++++++++++ .../DatabaseIntegrityTests.swift | 32 +++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/Packages/WhoopStore/Sources/WhoopStore/DatabaseIntegrity.swift b/Packages/WhoopStore/Sources/WhoopStore/DatabaseIntegrity.swift index 63b5ea9a1c..4139a87023 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/DatabaseIntegrity.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/DatabaseIntegrity.swift @@ -35,6 +35,22 @@ public enum DatabaseIntegrity { func quickCheckRows(readonly: Bool) throws -> [String] { var config = Configuration() config.readonly = readonly + if !readonly { + // The read-write fallback is the ONE connection in this tree that can reach the live + // store without going through `WhoopStore(path:)`, and therefore without + // `StoreReplication`'s checkpointing policy (see `WalCheckpointing`). Today it is + // harmless only by a chain of reasoning about SQLite internals — autocheckpoint fires + // at the end of a WRITE transaction and this probe only reads, and the close-time + // checkpoint only runs for the LAST connection, which this never is while the app's + // pool is open. Under `.external` that chain is the whole safety argument for a + // replicator's resume offset, so it is replaced with the guarantee: this connection + // cannot restart the WAL because it is not permitted to checkpoint. + // + // No behaviour change for the common case (probing a staged backup file): the + // close-time checkpoint that cleans up the `-wal` this connection creates is a + // separate mechanism and is unaffected by `wal_autocheckpoint`. + config.prepareDatabase { try $0.execute(sql: "PRAGMA wal_autocheckpoint = 0") } + } let dbQueue = try DatabaseQueue(path: path, configuration: config) return try dbQueue.read { db in try String.fetchAll(db, sql: "PRAGMA quick_check(1)") } } diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/DatabaseIntegrityTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/DatabaseIntegrityTests.swift index 165ff56b57..e67738e0ff 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/DatabaseIntegrityTests.swift +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/DatabaseIntegrityTests.swift @@ -1,5 +1,6 @@ import XCTest import GRDB +import WhoopProtocol @testable import WhoopStore /// #1014 defence-in-depth: real-file coverage for the `PRAGMA quick_check` integrity gate the @@ -106,4 +107,35 @@ final class DatabaseIntegrityTests: XCTestCase { XCTAssertEqual(DatabaseIntegrity.verdict(fromRows: ["ok", "Page 9 is never used"]), "Page 9 is never used") } + + // MARK: - The probe must not checkpoint a replicated store + + /// `quickCheckFailure` opens its own GRDB connection, which is the one connection in this package + /// that reaches a live database without going through `WhoopStore(path:)` — and therefore without + /// `StoreReplication`'s checkpointing policy. Under `WalCheckpointing.external` a checkpoint here + /// would restart the WAL under the replicator and cost it its incremental resume point, turning + /// the next push into a full snapshot of the whole database. + /// + /// Asserted as an observable property (the `-wal` does not shrink), not as "the pragma was set", + /// so it holds whichever of the read-only or read-write branch the probe actually took. + func testProbingALiveExternallyCheckpointedStoreDoesNotShrinkItsWal() async throws { + let path = NSTemporaryDirectory() + "dbintegrity-wal-\(UUID().uuidString).sqlite" + defer { for s in ["", "-wal", "-shm"] { try? FileManager.default.removeItem(atPath: path + s) } } + + // Backstop disabled so the ONLY thing that could shrink the WAL is the probe. + let store = try await WhoopStore(path: path, walCheckpointing: .external, walBackstop: .disabled) + var ts = 0 + for _ in 0..<10 { + var hr: [HRSample] = [] + for _ in 0..<400 { ts += 1; hr.append(HRSample(ts: ts, bpm: 60 + ts % 40)) } + _ = try await store.insert(Streams(hr: hr), deviceId: "dev") + } + let before = try XCTUnwrap(store.walFileSizeBytes()) + XCTAssertGreaterThan(before, 0, "the test needs a non-empty WAL to be meaningful") + + XCTAssertNil(DatabaseIntegrity.quickCheckFailure(atPath: path), "the store is healthy") + + XCTAssertEqual(store.walFileSizeBytes(), before, + "the integrity probe must not checkpoint a store an external replicator owns") + } } From c7f777f409f20fbdb407a81e4814a186019cb953 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 27 Jul 2026 17:26:37 -0400 Subject: [PATCH 138/171] test(schema): teach the shared oracle the fork's iOS-only tables and its second migration lineage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Merging upstream's #889 schema-pin tests turned four of them red on fork main. The oracle encodes two single-source assumptions a fork breaks, and neither is fixable by changing the store. SCHEMA. Three tables backing Strand/CloudSync/ (cloudTombstone, appleStepHour, phoneTimezone) and one column (dailyMetric.avgSdnn, appended by v29-daily-avg-sdnn) exist on GRDB with no Room twin. The mechanisms for that were already there and simply unused here: the tables are pinned platform "ios_only", avgSdnn is androidAbsent with an avg-sdnn-ios-only reason. Each table's shape is generated from a live PRAGMA table_info dump rather than transcribed from the DDL, so the fixture records what the migrator actually creates. LINEAGE. testGrdbMigrationIdentifiersAreUniqueAndSequential asserted every identifier's vN equals its registration index. Fork main registers v26-cloud-tombstone, v27-apple-step-hour, v28-phone-timezone and v29-daily-avg-sdnn interleaved with upstream's own v26..v29, so both lineages claim the same four numbers. They are NOT renumbered. GRDB keys migrations by identifier STRING: renaming one makes it read as un-applied on a device that already ran it, and the migrator then re-runs its body against a schema that already has the table — wedging a live database. The identifiers are frozen exactly as first shipped. Instead the fixture gains grdbMigrationLineages, declaring a set of identifiers numbered independently of the baseline. Everything NOT declared is the baseline lineage and is still held to exactly v1..vN, so the check upstream relies on is unchanged: with no such key the behaviour is bit-for-bit what it was. The escape hatch costs a written-down reason and cannot be taken by accident. What stays enforced: full-identifier uniqueness, absolutely (a genuine duplicate makes GRDB silently skip the second body); a declared lineage must be true, with every member registered; its own vN must strictly increase; and the baseline must still be gapless. An UNDECLARED collision still fails — the #369-vs-#475 case the test was written for — and an upstream migration cannot be hidden inside a lineage to launder a gap, because removing it from the baseline shifts everything after it. Verified: Swift 354/354 (SchemaOracleTests 7/7); Android SchemaOracleTest 4/4, suite 3189 tests with the one pre-existing v18-aux failure #897 already covers. Both halves were also run against 16 adversarial mutations of the fixture — all still caught. --- .../Resources/schema_oracle.json | 156 +++++++++++++++++- .../WhoopStoreTests/SchemaOracleTests.swift | 64 +++++-- .../java/com/noop/data/SchemaOracleTest.kt | 52 +++++- .../app/src/test/resources/schema_oracle.json | 156 +++++++++++++++++- 4 files changed, 411 insertions(+), 17 deletions(-) diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json b/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json index 72081fadf5..fdf93b2da7 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json @@ -1,5 +1,5 @@ { - "_readme": "SHARED Room<->GRDB SCHEMA ORACLE (#775). Two byte-identical copies: Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json and android/app/src/test/resources/schema_oracle.json. SchemaOracleTests.swift compares GRDB's PRAGMA table_info/index_list against it; SchemaOracleTest.kt compares Room's exported schema JSON against it. `columns` is the iOS/GRDB shape in GRDB column order (macOS is the reference implementation); every way Android differs is spelled out in an `android` / `iosAbsent` / `androidColumnOrder` override naming a key in `divergenceReasons`. Adding a column, reordering one, or changing a type/nullability on one platform only fails both suites until it is either fixed or written down here.", + "_readme": "SHARED Room<->GRDB SCHEMA ORACLE (#775). Two byte-identical copies: Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json and android/app/src/test/resources/schema_oracle.json. SchemaOracleTests.swift compares GRDB's PRAGMA table_info/index_list against it; SchemaOracleTest.kt compares Room's exported schema JSON against it. `columns` is the iOS/GRDB shape in GRDB column order (macOS is the reference implementation); every way Android differs is spelled out in an `android` / `iosAbsent` / `androidColumnOrder` override naming a key in `divergenceReasons`. Adding a column, reordering one, or changing a type/nullability on one platform only fails both suites until it is either fixed or written down here. `grdbMigrationLineages` is the same idea for migration IDENTIFIERS: a fork that numbered its own migrations before upstream reached those numbers declares them there instead of renumbering \u2014 GRDB keys migrations by identifier string, so renaming one wedges a database that already ran it.", "roomVersion": 25, "grdbMigrations": [ "v1", @@ -27,15 +27,31 @@ "v23-daily-spo2-raw", "v24-rr-seq", "v25-oura-raw", + "v26-cloud-tombstone", + "v27-apple-step-hour", "v26-efficiency-heal", "v27-ppg-waveform", + "v28-phone-timezone", + "v29-daily-avg-sdnn", "v28-raw-imu", "v29-score-input-provenance", "v30-rr-ord", "v31-deep-capture-channels" ], + "grdbMigrationLineages": { + "fork-ios-only": { + "reason": "This fork (vishk23/noop) carries four iOS-only migrations that were numbered from v26 before upstream's own v26..v29 existed, so both lineages claim v26, v27, v28 and v29. They are NOT renumbered to sit after upstream's: GRDB keys migrations by identifier STRING, so renaming one makes it read as un-applied on a device that already ran it, and the migrator then re-runs its body against a schema that already has the table \u2014 wedging a live database. The identifiers are frozen exactly as first shipped. What still carries the vN <-> Room-version mapping is the BASELINE lineage (grdbMigrations minus every lineage declared here), which both suites still check is exactly v1..vN with no gaps and no repeats; a lineage's own vN must strictly increase; and full-identifier uniqueness is enforced across everything, because a genuinely duplicated identifier makes GRDB silently skip the second body.", + "migrations": [ + "v26-cloud-tombstone", + "v27-apple-step-hour", + "v28-phone-timezone", + "v29-daily-avg-sdnn" + ] + } + }, "divergenceReasons": { "android-orphan-synced-column": "REAL COLUMN DRIFT: `synced` exists on Android only. GRDB's v10 note is explicit that stepSample gets no `synced` column ('unused; see StreamStore'), and v12's ppgHrSample never had one either; the Room entities copied the flag from the older per-second entities. Nothing reads it on Android \u2014 the per-row upload flag belongs to an upload path that does not exist there \u2014 so it is dead width, not divergent data. Removing it needs a Room table rebuild.", + "avg-sdnn-ios-only": "FORK-ONLY COLUMN: `dailyMetric.avgSdnn` is appended by the fork's GRDB `v29-daily-avg-sdnn` \u2014 the 5-minute SDNN index, the broad-variability twin of `avgHrv` (RMSSD), window-matched to a watch's short SDNN rather than the drift-inflated whole-night SD, and computed by `HRVAnalyzer.sdnnIndex` during the nightly analysis. Additive and nullable, so it costs an existing row nothing. Room has no field for it; giving it one means a @Database version bump plus a migration, so the absence is recorded here rather than left to read as unexplained drift.", "battery-alter-append-order": "REAL COLUMN-ORDER DRIFT (CLAUDE.md: 'column order in a Room CREATE TABLE must match the entity field order'). GRDB appended `synced` (v5) then `charging` (v6) with ALTER TABLE, which can only append; the Room entity declares `charging` before `synced`. Same columns, same types, different positions \u2014 so any positional copy (INSERT INTO battery SELECT * FROM \u2026) swaps them. Fixing it means reordering the Room entity's fields, which changes Room's identity hash and therefore needs a version bump plus a no-op migration.", "grdb-boolean-affinity": "GRDB's .boolean column type is declared BOOLEAN, which SQLite gives NUMERIC affinity; Room maps Kotlin Boolean to INTEGER. For the 0/1 values both sides write, NUMERIC and INTEGER affinity store bit-identical INTEGERs, so a .noopbak round-trips. Closing it means a GRDB table rebuild.", "paired-device-alter-append-order": "REAL COLUMN-ORDER DRIFT, same shape as battery. GRDB's v16-paired-device-peripheral appended `peripheralId` at the end; the Room entity declares it fifth, between `nickname` and `sourceKind`.", @@ -115,6 +131,35 @@ ], "indices": [] }, + "appleStepHour": { + "platform": "ios_only", + "reason": "Fork-only iOS table (GRDB `v27-apple-step-hour`): hourly Apple Health step counts. The daily HealthKit collect path flattens a whole day into one `appleDaily.steps` total, so hours the phone was dead or absent read as low steps for the entire day instead of as the specific hours with no recording. Android has no HealthKit source, so no Room twin.", + "columns": [ + { + "name": "deviceId", + "affinity": "TEXT", + "notNull": true, + "default": null + }, + { + "name": "ts", + "affinity": "INTEGER", + "notNull": true, + "default": null + }, + { + "name": "steps", + "affinity": "INTEGER", + "notNull": true, + "default": null + } + ], + "primaryKey": [ + "deviceId", + "ts" + ], + "indices": [] + }, "battery": { "platform": "both", "columns": [ @@ -178,6 +223,85 @@ "androidColumnOrderDivergence": "battery-alter-append-order", "indices": [] }, + "cloudTombstone": { + "platform": "ios_only", + "reason": "Fork-only iOS table (GRDB `v26-cloud-tombstone`): records deletions accepted from the NOOP cloud journal, keyed by (kind, editSeq) for idempotency, so a later re-import cannot resurrect a row the user deleted. Backs Strand/CloudSync/, which has no Android counterpart.", + "columns": [ + { + "name": "id", + "affinity": "INTEGER", + "notNull": false, + "default": null + }, + { + "name": "kind", + "affinity": "TEXT", + "notNull": true, + "default": null + }, + { + "name": "deviceId", + "affinity": "TEXT", + "notNull": true, + "default": null + }, + { + "name": "startTs", + "affinity": "INTEGER", + "notNull": false, + "default": null + }, + { + "name": "endTs", + "affinity": "INTEGER", + "notNull": false, + "default": null + }, + { + "name": "sport", + "affinity": "TEXT", + "notNull": false, + "default": null + }, + { + "name": "day", + "affinity": "TEXT", + "notNull": false, + "default": null + }, + { + "name": "key", + "affinity": "TEXT", + "notNull": false, + "default": null + }, + { + "name": "editSeq", + "affinity": "INTEGER", + "notNull": true, + "default": null + }, + { + "name": "createdAt", + "affinity": "INTEGER", + "notNull": true, + "default": null + } + ], + "primaryKey": [ + "id" + ], + "indices": [ + { + "name": "idx_cloudTombstone_kind_device", + "unique": false, + "columns": [ + "kind", + "deviceId" + ] + } + ] + }, "cursors": { "platform": "ios_only", "reason": "iOS offload/upload bookmarks (GRDB v2). Android keeps the equivalent bookmarks in SharedPreferences, not in Room.", @@ -322,6 +446,14 @@ "affinity": "INTEGER", "notNull": false, "default": null + }, + { + "name": "avgSdnn", + "affinity": "REAL", + "notNull": false, + "default": null, + "androidAbsent": true, + "divergence": "avg-sdnn-ios-only" } ], "primaryKey": [ @@ -1054,6 +1186,28 @@ "androidColumnOrderDivergence": "paired-device-alter-append-order", "indices": [] }, + "phoneTimezone": { + "platform": "ios_only", + "reason": "Fork-only iOS table (GRDB `v28-phone-timezone`): the IANA timezone the phone was in for each local calendar day, so day-bucketed metrics stay correct across travel. Android has no twin.", + "columns": [ + { + "name": "day", + "affinity": "TEXT", + "notNull": false, + "default": null + }, + { + "name": "tzId", + "affinity": "TEXT", + "notNull": true, + "default": null + } + ], + "primaryKey": [ + "day" + ], + "indices": [] + }, "ppgHrSample": { "platform": "both", "columns": [ diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/SchemaOracleTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/SchemaOracleTests.swift index 5cba234060..30853408ae 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/SchemaOracleTests.swift +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/SchemaOracleTests.swift @@ -32,6 +32,15 @@ final class SchemaOracleTests: XCTestCase { let grdbMigrations: [String] let divergenceReasons: [String: String] let tables: [String: OracleTable] + /// Absent upstream, where there is only the baseline lineage. See `MigrationLineage`. + let grdbMigrationLineages: [String: MigrationLineage]? + var lineages: [String: MigrationLineage] { grdbMigrationLineages ?? [:] } + } + /// A set of migration identifiers numbered independently of the baseline lineage — a fork that + /// reached `vN` before upstream did. Declaring one is what lets the two coexist without a renumber. + struct MigrationLineage: Decodable { + let reason: String + let migrations: [String] } struct OracleTable: Decodable { let platform: String // "both" | "ios_only" | "android_only" @@ -201,23 +210,58 @@ final class SchemaOracleTests: XCTestCase { /// entirely, because the first already recorded that name in `grdb_migrations`. /// /// So: identifiers unique, and their `vN` prefixes exactly 1...N with no gaps and no repeats. + /// + /// The one sanctioned exception is a DECLARED lineage. A fork that numbered its own migrations + /// before upstream reached those numbers cannot renumber its way out — GRDB keys by identifier + /// string, so renaming one makes it read as un-applied on a device that already ran it and the + /// migrator re-runs its body against a schema that already has the table. Such a fork lists its + /// identifiers under `grdbMigrationLineages`; everything NOT listed is the baseline lineage and is + /// still held to exactly v1...vN. So the escape hatch costs a written-down reason and cannot be + /// taken by accident — an undeclared collision still fails, which is what catches #369 vs #475. func testGrdbMigrationIdentifiersAreUniqueAndSequential() throws { + let oracle = try loadOracle() let ids = WhoopStore.makeMigrator().migrations XCTAssertEqual(Set(ids).count, ids.count, "duplicate GRDB migration identifier — GRDB would silently SKIP the second body") - var numbers: [Int] = [] - for id in ids { - let digits = id.dropFirst().prefix { $0.isNumber } - guard id.hasPrefix("v"), let n = Int(digits) else { - return XCTFail("migration identifier '\(id)' is not of the form v[-slug]") + func version(of id: String) -> Int? { + guard id.hasPrefix("v"), let n = Int(id.dropFirst().prefix(while: \.isNumber)) else { return nil } + return n + } + for id in ids where version(of: id) == nil { + return XCTFail("migration identifier '\(id)' is not of the form v[-slug]") + } + + // A declared lineage must be true (every member registered) and self-consistent (its own vN + // strictly increasing), so the ledger can only shrink deliberately — same rule the divergence + // reasons live by. + var claimedBy: [String: String] = [:] + for (name, lineage) in oracle.lineages.sorted(by: { $0.key < $1.key }) { + XCTAssertFalse(lineage.migrations.isEmpty, "lineage '\(name)' declares no migrations — delete it") + for id in lineage.migrations { + if let other = claimedBy.updateValue(name, forKey: id) { + return XCTFail("'\(id)' is claimed by both lineage '\(other)' and '\(name)'") + } + guard ids.contains(id) else { + return XCTFail("lineage '\(name)' declares '\(id)', which is not registered — a lineage " + + "entry that stopped being true must be deleted, not left to rot") + } + } + let numbers = lineage.migrations.compactMap(version(of:)) + for (a, b) in zip(numbers, numbers.dropFirst()) where b <= a { + return XCTFail("lineage '\(name)' goes v\(a) then v\(b) — a lineage numbers its own " + + "migrations, so its vN must strictly increase") } - numbers.append(n) } - for (offset, n) in numbers.enumerated() where n != offset + 1 { - return XCTFail("GRDB migration '\(ids[offset])' claims v\(n) but is #\(offset + 1) in " - + "registration order — two migrations claiming the same vN, or a gap, makes the " - + "GRDB-name <-> Room-version mapping ambiguous. Renumber before merging.") + + let baseline = ids.filter { claimedBy[$0] == nil } + for (offset, id) in baseline.enumerated() where version(of: id) != offset + 1 { + return XCTFail("GRDB migration '\(id)' claims v\(version(of: id)!) but is #\(offset + 1) in the " + + "BASELINE lineage (registration order minus every lineage schema_oracle.json " + + "declares) — two migrations claiming the same vN, or a gap, makes the " + + "GRDB-name <-> Room-version mapping ambiguous. Renumber before merging, or, if " + + "the identifier already shipped and renaming it would wedge a live database, " + + "declare it in grdbMigrationLineages with a reason.") } } diff --git a/android/app/src/test/java/com/noop/data/SchemaOracleTest.kt b/android/app/src/test/java/com/noop/data/SchemaOracleTest.kt index 14aec7c2f4..a482a41d4b 100644 --- a/android/app/src/test/java/com/noop/data/SchemaOracleTest.kt +++ b/android/app/src/test/java/com/noop/data/SchemaOracleTest.kt @@ -238,25 +238,67 @@ class SchemaOracleTest { * migrations by NAME and applies them in registration order; Room keys by integer version, so the * only thing that can be compared mechanically is the RESULTING schema — which is what the tests * above do. Pinning the two identifier spaces is what forces them to be re-checked together. + * + * The one sanctioned exception is a DECLARED lineage. A fork that numbered its own migrations before + * upstream reached those numbers cannot renumber its way out — GRDB keys by identifier string, so + * renaming one makes it read as un-applied on a device that already ran it. Such a fork lists its + * identifiers under `grdbMigrationLineages`; everything NOT listed is the baseline lineage and is + * still held to exactly v1..vN. An UNDECLARED collision still fails, which is the #369-vs-#475 case. + * Mirrors `testGrdbMigrationIdentifiersAreUniqueAndSequential` in the Swift half. */ @Test fun pinnedMigrationIdentifiersAreCoherent() { val oracle = loadOracle() val grdb = oracle.getJSONArray("grdbMigrations").strings() assertEquals("duplicate GRDB migration identifier in schema_oracle.json", grdb.size, grdb.toSet().size) - grdb.forEachIndexed { i, id -> - val n = id.removePrefix("v").takeWhile { it.isDigit() }.toIntOrNull() + + // A declared lineage must be true (every member pinned) and self-consistent (its own vN strictly + // increasing), so the ledger can only shrink deliberately — same rule the divergence reasons live by. + val lineages = oracle.optJSONObject("grdbMigrationLineages") ?: JSONObject() + val claimedBy = HashMap() + for (name in lineages.keys().asSequence().sorted()) { + val migrations = lineages.getJSONObject(name).getJSONArray("migrations").strings() + assertTrue("lineage '$name' declares no migrations — delete it", migrations.isNotEmpty()) + val numbers = migrations.map { id -> + val other = claimedBy.put(id, name) + assertTrue("'$id' is claimed by both lineage '$other' and '$name'", other == null) + assertTrue( + "lineage '$name' declares '$id', which is not in grdbMigrations — a lineage entry that " + + "stopped being true must be deleted, not left to rot", + grdb.contains(id), + ) + val n = migrationVersion(id) + assertNotNull("lineage '$name' member '$id' is not of the form v[-slug]", n) + n!! + } + numbers.zipWithNext { a, b -> + assertTrue( + "lineage '$name' goes v$a then v$b — a lineage numbers its own migrations, so its vN " + + "must strictly increase", + b > a, + ) + } + } + + grdb.filterNot { claimedBy.containsKey(it) }.forEachIndexed { i, id -> assertEquals( - "GRDB migration '$id' claims v$n but is #${i + 1} in registration order — two migrations " + - "claiming the same vN, or a gap, makes the GRDB-name <-> Room-version mapping ambiguous.", + "GRDB migration '$id' claims v${migrationVersion(id)} but is #${i + 1} in the BASELINE " + + "lineage (grdbMigrations minus every lineage grdbMigrationLineages declares) — two " + + "migrations claiming the same vN, or a gap, makes the GRDB-name <-> Room-version " + + "mapping ambiguous. Renumber before merging, or, if the identifier already shipped and " + + "renaming it would wedge a live database, declare it in grdbMigrationLineages.", i + 1, - n, + migrationVersion(id), ) } // loadRoomSchema asserts the exported version equals this; call it so the check is not vacuous. loadRoomSchema(oracle.getInt("roomVersion")) } + /** The `N` of a `v[-slug]` migration identifier, or null if it is not of that form. */ + private fun migrationVersion(id: String): Int? = + id.removePrefix("v").takeWhile { it.isDigit() }.toIntOrNull() + /** * The Android and Swift copies of the oracle MUST be byte-identical, so neither platform can edit its * fixture without the other. Skips gracefully if the Swift tree isn't present. diff --git a/android/app/src/test/resources/schema_oracle.json b/android/app/src/test/resources/schema_oracle.json index 72081fadf5..fdf93b2da7 100644 --- a/android/app/src/test/resources/schema_oracle.json +++ b/android/app/src/test/resources/schema_oracle.json @@ -1,5 +1,5 @@ { - "_readme": "SHARED Room<->GRDB SCHEMA ORACLE (#775). Two byte-identical copies: Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json and android/app/src/test/resources/schema_oracle.json. SchemaOracleTests.swift compares GRDB's PRAGMA table_info/index_list against it; SchemaOracleTest.kt compares Room's exported schema JSON against it. `columns` is the iOS/GRDB shape in GRDB column order (macOS is the reference implementation); every way Android differs is spelled out in an `android` / `iosAbsent` / `androidColumnOrder` override naming a key in `divergenceReasons`. Adding a column, reordering one, or changing a type/nullability on one platform only fails both suites until it is either fixed or written down here.", + "_readme": "SHARED Room<->GRDB SCHEMA ORACLE (#775). Two byte-identical copies: Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json and android/app/src/test/resources/schema_oracle.json. SchemaOracleTests.swift compares GRDB's PRAGMA table_info/index_list against it; SchemaOracleTest.kt compares Room's exported schema JSON against it. `columns` is the iOS/GRDB shape in GRDB column order (macOS is the reference implementation); every way Android differs is spelled out in an `android` / `iosAbsent` / `androidColumnOrder` override naming a key in `divergenceReasons`. Adding a column, reordering one, or changing a type/nullability on one platform only fails both suites until it is either fixed or written down here. `grdbMigrationLineages` is the same idea for migration IDENTIFIERS: a fork that numbered its own migrations before upstream reached those numbers declares them there instead of renumbering \u2014 GRDB keys migrations by identifier string, so renaming one wedges a database that already ran it.", "roomVersion": 25, "grdbMigrations": [ "v1", @@ -27,15 +27,31 @@ "v23-daily-spo2-raw", "v24-rr-seq", "v25-oura-raw", + "v26-cloud-tombstone", + "v27-apple-step-hour", "v26-efficiency-heal", "v27-ppg-waveform", + "v28-phone-timezone", + "v29-daily-avg-sdnn", "v28-raw-imu", "v29-score-input-provenance", "v30-rr-ord", "v31-deep-capture-channels" ], + "grdbMigrationLineages": { + "fork-ios-only": { + "reason": "This fork (vishk23/noop) carries four iOS-only migrations that were numbered from v26 before upstream's own v26..v29 existed, so both lineages claim v26, v27, v28 and v29. They are NOT renumbered to sit after upstream's: GRDB keys migrations by identifier STRING, so renaming one makes it read as un-applied on a device that already ran it, and the migrator then re-runs its body against a schema that already has the table \u2014 wedging a live database. The identifiers are frozen exactly as first shipped. What still carries the vN <-> Room-version mapping is the BASELINE lineage (grdbMigrations minus every lineage declared here), which both suites still check is exactly v1..vN with no gaps and no repeats; a lineage's own vN must strictly increase; and full-identifier uniqueness is enforced across everything, because a genuinely duplicated identifier makes GRDB silently skip the second body.", + "migrations": [ + "v26-cloud-tombstone", + "v27-apple-step-hour", + "v28-phone-timezone", + "v29-daily-avg-sdnn" + ] + } + }, "divergenceReasons": { "android-orphan-synced-column": "REAL COLUMN DRIFT: `synced` exists on Android only. GRDB's v10 note is explicit that stepSample gets no `synced` column ('unused; see StreamStore'), and v12's ppgHrSample never had one either; the Room entities copied the flag from the older per-second entities. Nothing reads it on Android \u2014 the per-row upload flag belongs to an upload path that does not exist there \u2014 so it is dead width, not divergent data. Removing it needs a Room table rebuild.", + "avg-sdnn-ios-only": "FORK-ONLY COLUMN: `dailyMetric.avgSdnn` is appended by the fork's GRDB `v29-daily-avg-sdnn` \u2014 the 5-minute SDNN index, the broad-variability twin of `avgHrv` (RMSSD), window-matched to a watch's short SDNN rather than the drift-inflated whole-night SD, and computed by `HRVAnalyzer.sdnnIndex` during the nightly analysis. Additive and nullable, so it costs an existing row nothing. Room has no field for it; giving it one means a @Database version bump plus a migration, so the absence is recorded here rather than left to read as unexplained drift.", "battery-alter-append-order": "REAL COLUMN-ORDER DRIFT (CLAUDE.md: 'column order in a Room CREATE TABLE must match the entity field order'). GRDB appended `synced` (v5) then `charging` (v6) with ALTER TABLE, which can only append; the Room entity declares `charging` before `synced`. Same columns, same types, different positions \u2014 so any positional copy (INSERT INTO battery SELECT * FROM \u2026) swaps them. Fixing it means reordering the Room entity's fields, which changes Room's identity hash and therefore needs a version bump plus a no-op migration.", "grdb-boolean-affinity": "GRDB's .boolean column type is declared BOOLEAN, which SQLite gives NUMERIC affinity; Room maps Kotlin Boolean to INTEGER. For the 0/1 values both sides write, NUMERIC and INTEGER affinity store bit-identical INTEGERs, so a .noopbak round-trips. Closing it means a GRDB table rebuild.", "paired-device-alter-append-order": "REAL COLUMN-ORDER DRIFT, same shape as battery. GRDB's v16-paired-device-peripheral appended `peripheralId` at the end; the Room entity declares it fifth, between `nickname` and `sourceKind`.", @@ -115,6 +131,35 @@ ], "indices": [] }, + "appleStepHour": { + "platform": "ios_only", + "reason": "Fork-only iOS table (GRDB `v27-apple-step-hour`): hourly Apple Health step counts. The daily HealthKit collect path flattens a whole day into one `appleDaily.steps` total, so hours the phone was dead or absent read as low steps for the entire day instead of as the specific hours with no recording. Android has no HealthKit source, so no Room twin.", + "columns": [ + { + "name": "deviceId", + "affinity": "TEXT", + "notNull": true, + "default": null + }, + { + "name": "ts", + "affinity": "INTEGER", + "notNull": true, + "default": null + }, + { + "name": "steps", + "affinity": "INTEGER", + "notNull": true, + "default": null + } + ], + "primaryKey": [ + "deviceId", + "ts" + ], + "indices": [] + }, "battery": { "platform": "both", "columns": [ @@ -178,6 +223,85 @@ "androidColumnOrderDivergence": "battery-alter-append-order", "indices": [] }, + "cloudTombstone": { + "platform": "ios_only", + "reason": "Fork-only iOS table (GRDB `v26-cloud-tombstone`): records deletions accepted from the NOOP cloud journal, keyed by (kind, editSeq) for idempotency, so a later re-import cannot resurrect a row the user deleted. Backs Strand/CloudSync/, which has no Android counterpart.", + "columns": [ + { + "name": "id", + "affinity": "INTEGER", + "notNull": false, + "default": null + }, + { + "name": "kind", + "affinity": "TEXT", + "notNull": true, + "default": null + }, + { + "name": "deviceId", + "affinity": "TEXT", + "notNull": true, + "default": null + }, + { + "name": "startTs", + "affinity": "INTEGER", + "notNull": false, + "default": null + }, + { + "name": "endTs", + "affinity": "INTEGER", + "notNull": false, + "default": null + }, + { + "name": "sport", + "affinity": "TEXT", + "notNull": false, + "default": null + }, + { + "name": "day", + "affinity": "TEXT", + "notNull": false, + "default": null + }, + { + "name": "key", + "affinity": "TEXT", + "notNull": false, + "default": null + }, + { + "name": "editSeq", + "affinity": "INTEGER", + "notNull": true, + "default": null + }, + { + "name": "createdAt", + "affinity": "INTEGER", + "notNull": true, + "default": null + } + ], + "primaryKey": [ + "id" + ], + "indices": [ + { + "name": "idx_cloudTombstone_kind_device", + "unique": false, + "columns": [ + "kind", + "deviceId" + ] + } + ] + }, "cursors": { "platform": "ios_only", "reason": "iOS offload/upload bookmarks (GRDB v2). Android keeps the equivalent bookmarks in SharedPreferences, not in Room.", @@ -322,6 +446,14 @@ "affinity": "INTEGER", "notNull": false, "default": null + }, + { + "name": "avgSdnn", + "affinity": "REAL", + "notNull": false, + "default": null, + "androidAbsent": true, + "divergence": "avg-sdnn-ios-only" } ], "primaryKey": [ @@ -1054,6 +1186,28 @@ "androidColumnOrderDivergence": "paired-device-alter-append-order", "indices": [] }, + "phoneTimezone": { + "platform": "ios_only", + "reason": "Fork-only iOS table (GRDB `v28-phone-timezone`): the IANA timezone the phone was in for each local calendar day, so day-bucketed metrics stay correct across travel. Android has no twin.", + "columns": [ + { + "name": "day", + "affinity": "TEXT", + "notNull": false, + "default": null + }, + { + "name": "tzId", + "affinity": "TEXT", + "notNull": true, + "default": null + } + ], + "primaryKey": [ + "day" + ], + "indices": [] + }, "ppgHrSample": { "platform": "both", "columns": [ From 8e401466c83674fded20233d3f638a2f76354b72 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 27 Jul 2026 17:34:16 -0400 Subject: [PATCH 139/171] build(liters): link the xcframework, import the bindings, and give the trial a switch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage 1 built the foundations and wired none of them. This makes liters something the app actually links, and the trial something VK can turn on without a debugger. Rust/build-ios.sh now produces the whole integration, not just an archive: - a macOS slice alongside the two iOS ones. StrandTests is a macOS unit-test bundle hosted in the macOS app, so a round-trip test can only run against a macOS build; without this slice the bindings could be compile-checked and never executed. - the generated Swift, installed into Strand/CloudSync/Generated/ wrapped in `#if LITERS`. Tracked, because it is source the app compiles and a reviewer should be able to read it; gated, because the 83-MB-per-slice archive it needs is not tracked and a checkout without it must still build. - Config/LitersLocal.xcconfig, written FROM the xcframework's own Info.plist so the per-SDK slice paths cannot drift from the layout xcodebuild just produced. - the sqlite-linkage check promoted from a printed suggestion to a hard gate. It is the one invariant that silently breaks correctness rather than the build. The linkage is an xcconfig seam, not a "Link Binary With Libraries" phase, and that is the point: Config/Liters.xcconfig is tracked and defines every liters setting as EMPTY, then optionally includes the untracked local file. With no local file the three appends in project.yml expand to nothing and a default build links exactly what it linked before. Verified with `xcodebuild -showBuildSettings`: OTHER_LDFLAGS empty, no LITERS, SWIFT_ACTIVE_COMPILATION_CONDITIONS still `DEBUG OURA_CLOUD_IMPORT CLOUD_SYNC`. Skipping the Rust build is not an error for anyone who does not want it. The flag rides OTHER_SWIFT_FLAGS rather than SWIFT_ACTIVE_COMPILATION_CONDITIONS because xcodegen writes a bare `SWIFT_ACTIVE_COMPILATION_CONDITIONS = DEBUG` into the project's Debug config, which outranks the project-level xcconfig and silently drops the LITERS entry. Measured, not assumed. StrandTests/LitersRoundTripTests.swift exercises the real thing: a WAL database built through `import SQLite3` (the same libsqlite3 GRDB links, which is the entire reason for `default-features = false`), pushed by LitersWriter into a Storage.dir bucket, pulled back by LitersReplica, and compared row for row. It asserts the second push is a WAL delta rather than a snapshot, since a replicator that snapshots on every push is a full upload with extra steps; and it covers the close/reopen that iOS performs on every relaunch, recording rather than asserting whether that costs a snapshot. The Test Centre gets a fifth card. It shows three facts that are not the same one: the toggle (intent), what the store actually opened with (reality — they differ for the whole launch you flip it in, because wal_autocheckpoint is applied at pool-open time), and the telemetry aggregate (result). SyncReplicationTrial.isInForce is new and exists so the card can tell the first two apart. /ingest is untouched. Nothing in this commit changes what gets uploaded. --- .gitignore | 6 + Config/BundleId.xcconfig | 7 + Config/Liters.xcconfig | 37 + Rust/README.md | 80 +- Rust/build-ios.sh | 165 +- Strand/CloudSync/Generated/liters_ffi.swift | 4459 +++++++++++++++++++ Strand/CloudSync/SyncReplicationTrial.swift | 12 + Strand/Screens/TestCentreView.swift | 64 + StrandTests/LitersRoundTripTests.swift | 277 ++ project.yml | 13 + 10 files changed, 5089 insertions(+), 31 deletions(-) create mode 100644 Config/Liters.xcconfig create mode 100644 Strand/CloudSync/Generated/liters_ffi.swift create mode 100644 StrandTests/LitersRoundTripTests.swift diff --git a/.gitignore b/.gitignore index 1c573661c0..f28c5a177b 100644 --- a/.gitignore +++ b/.gitignore @@ -109,3 +109,9 @@ Strand/Oura/OuraSecrets.xcconfig # Personal bundle id prefix (BYO; never committed) Config/BundleIdSecrets.xcconfig + +# liters linkage, written by Rust/build-ios.sh from the xcframework it just produced. Untracked +# because it names slice directories under Rust/target/, which is itself untracked — committing it +# would point every other checkout at an archive that is not there. Config/Liters.xcconfig is the +# tracked wrapper that optionally includes it. +Config/LitersLocal.xcconfig diff --git a/Config/BundleId.xcconfig b/Config/BundleId.xcconfig index e63944ad71..0954d35a2e 100644 --- a/Config/BundleId.xcconfig +++ b/Config/BundleId.xcconfig @@ -16,3 +16,10 @@ BUNDLE_ID_PREFIX = com.noopapp // to stop Xcode from asking you to reselect a team after every `xcodegen generate`. DEVELOPMENT_TEAM = #include? "BundleIdSecrets.xcconfig" + +// The liters (page-replication) linkage seam. Tracked, and inert until Rust/build-ios.sh has been +// run: it defines LITERS_LIB_DIR / LITERS_HEADER_DIR / LITERS_LDFLAGS as empty and then optionally +// includes the untracked Config/LitersLocal.xcconfig that build-ios.sh writes. Included from HERE +// rather than from a target xcconfig because project.yml applies this file project-wide, which is +// the only level every target (macOS Strand, StrandTests, NOOPiOS) already shares. +#include "Liters.xcconfig" diff --git a/Config/Liters.xcconfig b/Config/Liters.xcconfig new file mode 100644 index 0000000000..414522c9de --- /dev/null +++ b/Config/Liters.xcconfig @@ -0,0 +1,37 @@ +// Tracked wrapper for the liters (page-replication) linkage, so a clean checkout that has never run +// Rust/build-ios.sh still generates and builds. Included from Config/BundleId.xcconfig, which +// project.yml applies project-wide, so every target inherits these. +// +// Liters.xcframework is a BUILD ARTIFACT, not a tracked file: the device slice alone is ~83 MB, and +// three slices plus a macOS slice is far past what belongs in git. It is produced by +// `Rust/build-ios.sh`, which ALSO writes the untracked Config/LitersLocal.xcconfig included at the +// bottom of this file. That is deliberate — the search paths below name specific xcframework slice +// directories, and having the build script emit them from the layout `xcodebuild -create-xcframework` +// actually produced is the only way they cannot drift from it. +// +// The three variables default to EMPTY here. project.yml appends them unconditionally +// (LIBRARY_SEARCH_PATHS / SWIFT_INCLUDE_PATHS / OTHER_LDFLAGS), so with no local file the appends +// expand to nothing: no -l flag, no search path, and a default build links exactly what it links +// today. Absence of the xcframework is therefore not a build error for anyone who does not want it. +// +// LITERS_LDFLAGS is what actually pulls the archive in. It is kept separate from the search path so +// that "the paths are configured" and "the library is linked" stay two decisions: a build with the +// paths set but LITERS_LDFLAGS empty compiles the generated bindings' module import and fails at +// link, which is a clearer error than a missing module. +LITERS_LIB_DIR = +LITERS_HEADER_DIR = +LITERS_LDFLAGS = + +// Set by build-ios.sh alongside the paths above. Gates every line of Swift that touches liters +// (Strand/CloudSync/Generated/, and the round-trip tests), so a checkout without the xcframework +// compiles none of it rather than failing to resolve `import liters_ffiFFI`: +// +// OTHER_SWIFT_FLAGS = $(inherited) -D LITERS +// +// NOT SWIFT_ACTIVE_COMPILATION_CONDITIONS, even though that is the idiomatic setting and what +// OuraSecrets.xcconfig uses for OURA_CLOUD_IMPORT / CLOUD_SYNC. Those work because OuraConfig.xcconfig +// is a TARGET-level xcconfig, which outranks the project build settings; this file is included from +// the PROJECT-level xcconfig, which xcodegen's own `SWIFT_ACTIVE_COMPILATION_CONDITIONS = DEBUG` +// (written into the project's Debug config, with no $(inherited)) sits above and silently overwrites. +// Verified with `xcodebuild -showBuildSettings`: the LITERS entry vanishes, DEBUG survives. +#include? "LitersLocal.xcconfig" diff --git a/Rust/README.md b/Rust/README.md index 8e2da00496..b9f3f07f97 100644 --- a/Rust/README.md +++ b/Rust/README.md @@ -42,12 +42,33 @@ nm -g target/aarch64-apple-ios/release/libnoop_liters.a | grep -c ' T _sqlite3_' ## Building ```sh -rustup target add aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios +rustup target add aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios \ + aarch64-apple-darwin x86_64-apple-darwin ./build-ios.sh +xcodegen generate ``` -Outputs `target/apple/Liters.xcframework` and `target/apple/swift/*.swift`. -Neither is committed, and neither is referenced by `project.yml` yet. +`build-ios.sh` takes about 25 minutes cold and produces four things: + +| output | tracked | why | +|---|---|---| +| `target/apple/Liters.xcframework` | no | 83 MB per slice, three slices | +| `Strand/CloudSync/Generated/liters_ffi.swift` | **yes** | source the app compiles; a reviewer should be able to read it | +| `Config/LitersLocal.xcconfig` | no | names slice directories under the untracked `target/` | +| `target/apple/swift/*` | no | the staging copy the two above are made from | + +There is a macOS slice as well as the two iOS ones because `StrandTests` is a +macOS unit-test bundle (`project.yml`), so `StrandTests/LitersRoundTripTests.swift` +can only run against a macOS build. + +**Skipping this script is not an error.** `Config/Liters.xcconfig` is tracked and +defines every liters build setting as empty, then optionally includes the local +file this script writes. With no local file, `project.yml`'s three appends +(`LIBRARY_SEARCH_PATHS`, `SWIFT_INCLUDE_PATHS`, `OTHER_LDFLAGS`) expand to +nothing, the `LITERS` compilation condition is unset, and the generated bindings +compile to nothing. A checkout that has never seen Rust builds exactly as it +does today. Deleting `Config/LitersLocal.xcconfig` is how you turn liters back +off. To move the pin to a newer liters commit: @@ -59,10 +80,51 @@ cargo update -p liters-ffi # then commit the Cargo.lock change tree builds against, so a checkout is reproducible even though the dependency tracks a branch. -## What is deliberately not here +## Build time, and the one thing that would fix it + +Most of the wall clock is `uniffi_bindgen` — a code-generation CLI — being +compiled at `opt-level=3` **once per cross-compiled target**. It is dead weight +in every one of them: nothing on a phone runs a bindings generator. + +It cannot be turned off from this crate. `liters-ffi` declares +`uniffi = { version = "0.32", features = ["cli"] }` among its normal +dependencies, and cargo feature unification means a downstream crate cannot +remove a feature an upstream crate asked for. The fix belongs in `liters-mobile`: +move that bin behind a `cli` feature and depend on plain `uniffi` otherwise. +`build-ios.sh` already passes `--lib` for the cross targets, which at least skips +*linking* the useless binary. + +## Known gotcha: the replica's integrity check under system SQLite + +`liters`' own test suite passes 80/80 with its default bundled SQLite, and +**31 passed / 49 failed** under `cargo test -p liters --no-default-features`, the +configuration NOOP actually ships. Every one of the 49 fails with the same +`SQLITE_CANTOPEN` ("unable to open database file"), and all of them are on the +Replica restore path. + +The cause is a single line — `crates/liters/src/replica.rs:648`, the only +read-only open in the crate, reached only from `check_integrity` after a restore: + +```rust +rusqlite::Connection::open_with_flags(&self.db_path, OpenFlags::SQLITE_OPEN_READ_ONLY) +``` + +Apple's `libsqlite3` (3.51.0) cannot run a statement on a **WAL-mode** database +opened `SQLITE_OPEN_READONLY` when the `-shm` sidecar is absent — and the restore +deletes `-wal`/`-shm` just before this call. Reduced to a 20-line C program +against `/usr/lib/libsqlite3.dylib`: + +``` +WAL, sidecars removed, READONLY open=0 prepare=14 unable to open database file +WAL, sidecars intact, READONLY open=0 prepare=0 OK +WAL, sidecars removed, READWRITE open=0 prepare=0 OK +rollback journal, READONLY open=0 prepare=0 OK +``` -Nothing in the app calls liters yet — no Swift target imports the bindings, and -`project.yml` does not reference the xcframework. Wiring NOOP's sync path to -liters is separate work (delta sync, stage 1 of -`noop-cloud/docs/SYNC_BUILD_VS_BUY.md` §1.3). This directory exists so that work -has a real, building dependency to start from rather than a plan. +**This does not affect NOOP's phone.** The phone is a Writer, and `writer.rs` +contains no read-only open at all — the whole crate has exactly one, on the +Replica path. The Replica runs on the server, in a Rust process with no GRDB, +which therefore builds liters with bundled SQLite and never reaches this. It is +still a real bug for any embedder that restores on a system-SQLite platform, and +it should be fixed upstream (open the integrity check READWRITE, or materialise +the `-shm` first). diff --git a/Rust/build-ios.sh b/Rust/build-ios.sh index 50af458c49..63c1fae79a 100755 --- a/Rust/build-ios.sh +++ b/Rust/build-ios.sh @@ -1,35 +1,49 @@ #!/usr/bin/env bash -# Builds Liters.xcframework + Swift bindings for NOOP from the pinned -# liters-ffi git dependency. +# Builds Liters.xcframework + the Swift bindings NOOP compiles, from the pinned liters-ffi git +# dependency, and writes the xcconfig that links them. +# +# Run this once before building the app with the page-replication trial. Afterwards a plain +# `xcodegen generate && xcodebuild …` picks the archive up with no extra flags: the last step here +# writes Config/LitersLocal.xcconfig, which the tracked Config/Liters.xcconfig optionally includes. +# Without that file every liters build setting expands to empty and the app builds exactly as it +# does today, so running this script is opt-in and skipping it is not an error. # # Prereqs: -# rustup target add aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios +# rustup target add aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios \ +# aarch64-apple-darwin x86_64-apple-darwin # Xcode command line tools # -# There is deliberately no bundled/system switch here. NOOP links GRDB, which -# links Apple's system libsqlite3, so liters MUST link that same libsqlite3 — -# `Cargo.toml` pins `liters-ffi` with `default-features = false` to guarantee -# it, and this script has no way to opt back in. See Cargo.toml for why two -# SQLite copies in one process is a correctness bug rather than a size problem. -# -# Nothing in the app consumes the output yet; this builds the dependency so the -# sync work has something real to build against. +# There is deliberately no bundled/system switch here. NOOP links GRDB, which links Apple's system +# libsqlite3, so liters MUST link that same libsqlite3 — `Cargo.toml` pins `liters-ffi` with +# `default-features = false` to guarantee it, and this script has no way to opt back in. See +# Cargo.toml for why two SQLite copies in one process is a correctness bug rather than a size +# problem, and the SANITY CHECK below, which fails the build if it ever stops holding. set -euo pipefail cd "$(dirname "$0")" +ROOT=$(cd .. && pwd) OUT=target/apple BINDINGS=$OUT/swift DEVICE_TARGET=aarch64-apple-ios SIM_TARGETS=(aarch64-apple-ios-sim x86_64-apple-ios) +# macOS matters for more than completeness: StrandTests is a macOS unit-test bundle hosted in the +# macOS app (project.yml), so the round-trip tests in StrandTests/LitersRoundTripTests.swift can only +# run against a macOS slice. Both arches, because the macOS app builds ONLY_ACTIVE_ARCH=NO in Release. +MAC_TARGETS=(aarch64-apple-darwin x86_64-apple-darwin) -for t in "$DEVICE_TARGET" "${SIM_TARGETS[@]}"; do +# `--lib` on the cross-compiled targets skips the `uniffi-bindgen` BIN, which is a host developer +# tool and has no business being linked for a phone. (It does not avoid *compiling* the +# `uniffi_bindgen` crate: liters-ffi declares `uniffi = { features = ["cli"] }` among its normal +# dependencies, and cargo feature unification means a downstream crate cannot turn that off. Moving +# liters-ffi's own bindgen bin behind a `cli` feature would cut several minutes per target off this +# script; that has to be fixed in liters-mobile, not here.) +for t in "$DEVICE_TARGET" "${SIM_TARGETS[@]}" "${MAC_TARGETS[@]}"; do echo "==> building $t" - cargo build -p noop-liters --release --target "$t" + cargo build -p noop-liters --release --target "$t" --lib done -# Generate Swift bindings from the host library's embedded metadata, built from -# the same sources and features as the device libraries so the bindings cannot -# drift from the shipped staticlib. +# Generate Swift bindings from the host library's embedded metadata, built from the same sources and +# features as the device libraries so the bindings cannot drift from the shipped staticlib. echo "==> generating Swift bindings" cargo build -p noop-liters --release rm -rf "$BINDINGS" && mkdir -p "$BINDINGS" @@ -44,22 +58,129 @@ cp "$BINDINGS"/*.h "$HEADERS"/ # uniffi emits a .modulemap; xcodebuild wants module.modulemap cat "$BINDINGS"/*.modulemap > "$HEADERS"/module.modulemap -# Fat simulator library. -mkdir -p "$OUT/sim" +# Fat simulator + fat macOS archives. `lipo` because an xcframework slice holds one archive per +# platform, not one per arch. +mkdir -p "$OUT/sim" "$OUT/mac" lipo -create \ $(for t in "${SIM_TARGETS[@]}"; do echo "target/$t/release/libnoop_liters.a"; done) \ -output "$OUT/sim/libnoop_liters.a" +lipo -create \ + $(for t in "${MAC_TARGETS[@]}"; do echo "target/$t/release/libnoop_liters.a"; done) \ + -output "$OUT/mac/libnoop_liters.a" rm -rf "$OUT/Liters.xcframework" xcodebuild -create-xcframework \ -library "target/$DEVICE_TARGET/release/libnoop_liters.a" -headers "$HEADERS" \ -library "$OUT/sim/libnoop_liters.a" -headers "$HEADERS" \ + -library "$OUT/mac/libnoop_liters.a" -headers "$HEADERS" \ -output "$OUT/Liters.xcframework" +# --------------------------------------------------------------------------- +# SANITY CHECK — a hard gate, not advice. +# +# The device archive must DEFINE no sqlite3_* symbols and leave them undefined, so they resolve +# against the libsqlite3 GRDB links. A non-zero count means `bundled-sqlite` came back on and a second +# SQLite is about to be linked into the app, which silently breaks the advisory locking liters' +# correctness rests on. That is worth failing a build over, so it does. +# --------------------------------------------------------------------------- +SYMS=$(nm -g "target/$DEVICE_TARGET/release/libnoop_liters.a" 2>/dev/null || true) +DEFINED=$(printf '%s\n' "$SYMS" | grep -c ' T _sqlite3_' || true) +UNDEFINED=$(printf '%s\n' "$SYMS" | grep -c ' U _sqlite3_' || true) +if [ "$DEFINED" -ne 0 ]; then + echo "FATAL: the device archive DEFINES $DEFINED sqlite3_* symbols — liters bundled its own SQLite." >&2 + echo " Check that Cargo.toml still pins liters-ffi with default-features = false." >&2 + exit 1 +fi +if [ "$UNDEFINED" -eq 0 ]; then + echo "FATAL: the device archive references NO sqlite3_* symbols at all, which cannot be right." >&2 + exit 1 +fi +echo "==> sqlite linkage OK: 0 defined, $UNDEFINED undefined (resolve from the host's libsqlite3)" + +# --------------------------------------------------------------------------- +# Install the Swift bindings into the app's source tree. +# +# Wrapped in `#if LITERS` because the file is TRACKED (it is source the app compiles, and a reviewer +# should be able to read it) while the archive it needs is NOT (83 MB per slice). A checkout that has +# never run this script must still build, so without the LITERS condition — set below, in the +# untracked xcconfig — the file has to compile to nothing rather than fail on `import liters_ffiFFI`. +# --------------------------------------------------------------------------- +GEN=$ROOT/Strand/CloudSync/Generated +mkdir -p "$GEN" +{ + echo "// GENERATED by Rust/build-ios.sh from the pinned liters-ffi. Do not edit by hand." + echo "// The #if is added by that script, not by uniffi: see the script for why it is here." + echo "#if LITERS" + cat "$BINDINGS/liters_ffi.swift" + echo "#endif" +} > "$GEN/liters_ffi.swift" +echo "==> installed bindings: $GEN/liters_ffi.swift" + +# --------------------------------------------------------------------------- +# Write the linkage xcconfig FROM the layout xcodebuild just produced, so the slice directory names +# cannot drift from the ones actually in the xcframework. +# --------------------------------------------------------------------------- +slice_for() { # $1 = SupportedPlatform, $2 = SupportedPlatformVariant ("" for none) + python3 - "$OUT/Liters.xcframework/Info.plist" "$1" "$2" <<'PY' +import plistlib, sys +plist, platform, variant = sys.argv[1], sys.argv[2], sys.argv[3] +with open(plist, 'rb') as f: + libs = plistlib.load(f)['AvailableLibraries'] +for lib in libs: + if lib.get('SupportedPlatform') == platform and lib.get('SupportedPlatformVariant', '') == variant: + print(lib['LibraryIdentifier']); break +else: + sys.exit("no slice for %s/%s" % (platform, variant or '-')) +PY +} +IOS_SLICE=$(slice_for ios "") +SIM_SLICE=$(slice_for ios simulator) +MAC_SLICE=$(slice_for macos "") + +XCCONFIG=$ROOT/Config/LitersLocal.xcconfig +cat > "$XCCONFIG" < RustBuffer { + RustBuffer(capacity: 0, len:0, data: nil) + } + + static func from(_ ptr: UnsafeBufferPointer) -> RustBuffer { + try! rustCall { ffi_liters_ffi_rustbuffer_from_bytes(ForeignBytes(bufferPointer: ptr), $0) } + } + + // Frees the buffer in place. + // The buffer must not be used after this is called. + func deallocate() { + try! rustCall { ffi_liters_ffi_rustbuffer_free(self, $0) } + } +} + +fileprivate extension ForeignBytes { + init(bufferPointer: UnsafeBufferPointer) { + self.init(len: Int32(bufferPointer.count), data: bufferPointer.baseAddress) + } + + init(rawBufferPointer: UnsafeRawBufferPointer) { + self.init( + len: Int32(rawBufferPointer.count), + data: rawBufferPointer.baseAddress?.assumingMemoryBound(to: UInt8.self) + ) + } +} + +// Converter for `&[u8]` / `[ByRef] bytes` arguments. +// +// Conforms to `FfiConverter` so the compiler enforces the full converter +// method set. Only the scope-bound `lower(_:_body:)` overload is sound — +// zero-copy byte buffers only flow foreign -> Rust, and only in argument +// position. The four protocol-witness methods (`lift`, `lower`, `read`, +// `write`) `fatalError` at runtime if anyone reaches them. +// +// The scope-bound `lower` takes a closure because the `ForeignBytes` +// pointer is only guaranteed valid for the duration of +// `Data.withUnsafeBytes`. Callers must run the full FFI call inside +// the closure body. +fileprivate enum FfiConverterByRefBytes: FfiConverter { + typealias SwiftType = Data + typealias FfiType = ForeignBytes + + static func lower(_ value: Data, _ body: (ForeignBytes) throws -> R) rethrows -> R { + return try value.withUnsafeBytes { rawBuf in + try body(ForeignBytes(rawBufferPointer: rawBuf)) + } + } + + static func lower(_ value: Data) -> ForeignBytes { + fatalError("ByRef bytes cannot use the plain lower: returning ForeignBytes escapes the Data.withUnsafeBytes scope. Use the scope-bound lower(_:_body:) overload instead.") + } + + static func lift(_ value: ForeignBytes) throws -> Data { + fatalError("ByRef bytes cannot be lifted: zero-copy &[u8] only flows foreign->Rust") + } + + static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> Data { + fatalError("ByRef bytes cannot be read from a buffer: zero-copy &[u8] is only supported in argument position, not nested in records/options/etc.") + } + + static func write(_ value: Data, into buf: inout [UInt8]) { + fatalError("ByRef bytes cannot be written to a buffer: zero-copy &[u8] is only supported in argument position, not nested in records/options/etc.") + } +} + +// For every type used in the interface, we provide helper methods for conveniently +// lifting and lowering that type from C-compatible data, and for reading and writing +// values of that type in a buffer. + +// Helper classes/extensions that don't change. +// Someday, this will be in a library of its own. + +fileprivate extension Data { + init(rustBuffer: RustBuffer) { + self.init( + bytesNoCopy: rustBuffer.data!, + count: Int(rustBuffer.len), + deallocator: .none + ) + } +} + +// Define reader functionality. Normally this would be defined in a class or +// struct, but we use standalone functions instead in order to make external +// types work. +// +// With external types, one swift source file needs to be able to call the read +// method on another source file's FfiConverter, but then what visibility +// should Reader have? +// - If Reader is fileprivate, then this means the read() must also +// be fileprivate, which doesn't work with external types. +// - If Reader is internal/public, we'll get compile errors since both source +// files will try define the same type. +// +// Instead, the read() method and these helper functions input a tuple of data + +fileprivate func createReader(data: Data) -> (data: Data, offset: Data.Index) { + (data: data, offset: 0) +} + +// Reads an integer at the current offset, in big-endian order, and advances +// the offset on success. Throws if reading the integer would move the +// offset past the end of the buffer. +fileprivate func readInt(_ reader: inout (data: Data, offset: Data.Index)) throws -> T { + let range = reader.offset...size + guard reader.data.count >= range.upperBound else { + throw UniffiInternalError.bufferOverflow + } + if T.self == UInt8.self { + let value = reader.data[reader.offset] + reader.offset += 1 + return value as! T + } + var value: T = 0 + let _ = withUnsafeMutableBytes(of: &value, { reader.data.copyBytes(to: $0, from: range)}) + reader.offset = range.upperBound + return value.bigEndian +} + +// Reads an arbitrary number of bytes, to be used to read +// raw bytes, this is useful when lifting strings +fileprivate func readBytes(_ reader: inout (data: Data, offset: Data.Index), count: Int) throws -> Array { + let range = reader.offset..<(reader.offset+count) + guard reader.data.count >= range.upperBound else { + throw UniffiInternalError.bufferOverflow + } + var value = [UInt8](repeating: 0, count: count) + value.withUnsafeMutableBufferPointer({ buffer in + reader.data.copyBytes(to: buffer, from: range) + }) + reader.offset = range.upperBound + return value +} + +// Reads a float at the current offset. +fileprivate func readFloat(_ reader: inout (data: Data, offset: Data.Index)) throws -> Float { + return Float(bitPattern: try readInt(&reader)) +} + +// Reads a float at the current offset. +fileprivate func readDouble(_ reader: inout (data: Data, offset: Data.Index)) throws -> Double { + return Double(bitPattern: try readInt(&reader)) +} + +// Indicates if the offset has reached the end of the buffer. +fileprivate func hasRemaining(_ reader: (data: Data, offset: Data.Index)) -> Bool { + return reader.offset < reader.data.count +} + +// Define writer functionality. Normally this would be defined in a class or +// struct, but we use standalone functions instead in order to make external +// types work. See the above discussion on Readers for details. + +fileprivate func createWriter() -> [UInt8] { + return [] +} + +fileprivate func writeBytes(_ writer: inout [UInt8], _ byteArr: S) where S: Sequence, S.Element == UInt8 { + writer.append(contentsOf: byteArr) +} + +// Writes an integer in big-endian order. +// +// Warning: make sure what you are trying to write +// is in the correct type! +fileprivate func writeInt(_ writer: inout [UInt8], _ value: T) { + var value = value.bigEndian + withUnsafeBytes(of: &value) { writer.append(contentsOf: $0) } +} + +fileprivate func writeFloat(_ writer: inout [UInt8], _ value: Float) { + writeInt(&writer, value.bitPattern) +} + +fileprivate func writeDouble(_ writer: inout [UInt8], _ value: Double) { + writeInt(&writer, value.bitPattern) +} + +// Protocol for types that transfer other types across the FFI. This is +// analogous to the Rust trait of the same name. +fileprivate protocol FfiConverter { + associatedtype FfiType + associatedtype SwiftType + + static func lift(_ value: FfiType) throws -> SwiftType + static func lower(_ value: SwiftType) -> FfiType + static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> SwiftType + static func write(_ value: SwiftType, into buf: inout [UInt8]) +} + +// Types conforming to `Primitive` pass themselves directly over the FFI. +fileprivate protocol FfiConverterPrimitive: FfiConverter where FfiType == SwiftType { } + +extension FfiConverterPrimitive { +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public static func lift(_ value: FfiType) throws -> SwiftType { + return value + } + +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public static func lower(_ value: SwiftType) -> FfiType { + return value + } +} + +// Types conforming to `FfiConverterRustBuffer` lift and lower into a `RustBuffer`. +// Used for complex types where it's hard to write a custom lift/lower. +fileprivate protocol FfiConverterRustBuffer: FfiConverter where FfiType == RustBuffer {} + +extension FfiConverterRustBuffer { +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public static func lift(_ buf: RustBuffer) throws -> SwiftType { + var reader = createReader(data: Data(rustBuffer: buf)) + let value = try read(from: &reader) + if hasRemaining(reader) { + throw UniffiInternalError.incompleteData + } + buf.deallocate() + return value + } + +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public static func lower(_ value: SwiftType) -> RustBuffer { + var writer = createWriter() + write(value, into: &writer) + return RustBuffer(bytes: writer) + } +} +// An error type for FFI errors. These errors occur at the UniFFI level, not +// the library level. +fileprivate enum UniffiInternalError: LocalizedError { + case bufferOverflow + case incompleteData + case unexpectedOptionalTag + case unexpectedEnumCase + case unexpectedNullPointer + case unexpectedRustCallStatusCode + case unexpectedRustCallError + case unexpectedStaleHandle + case rustPanic(_ message: String) + + public var errorDescription: String? { + switch self { + case .bufferOverflow: return "Reading the requested value would read past the end of the buffer" + case .incompleteData: return "The buffer still has data after lifting its containing value" + case .unexpectedOptionalTag: return "Unexpected optional tag; should be 0 or 1" + case .unexpectedEnumCase: return "Raw enum value doesn't match any cases" + case .unexpectedNullPointer: return "Raw pointer value was null" + case .unexpectedRustCallStatusCode: return "Unexpected RustCallStatus code" + case .unexpectedRustCallError: return "CALL_ERROR but no errorClass specified" + case .unexpectedStaleHandle: return "The object in the handle map has been dropped already" + case let .rustPanic(message): return message + } + } +} + +fileprivate extension NSLock { + func withLock(f: () throws -> T) rethrows -> T { + self.lock() + defer { self.unlock() } + return try f() + } +} + +fileprivate let CALL_SUCCESS: Int8 = 0 +fileprivate let CALL_ERROR: Int8 = 1 +fileprivate let CALL_UNEXPECTED_ERROR: Int8 = 2 +fileprivate let CALL_CANCELLED: Int8 = 3 + +fileprivate extension RustCallStatus { + init() { + self.init( + code: CALL_SUCCESS, + errorBuf: RustBuffer.init( + capacity: 0, + len: 0, + data: nil + ) + ) + } +} + +private func rustCall(_ callback: (UnsafeMutablePointer) -> T) throws -> T { + let neverThrow: ((RustBuffer) throws -> Never)? = nil + return try makeRustCall(callback, errorHandler: neverThrow) +} + +private func rustCallWithError( + _ errorHandler: @escaping (RustBuffer) throws -> E, + _ callback: (UnsafeMutablePointer) -> T) throws -> T { + try makeRustCall(callback, errorHandler: errorHandler) +} + +private func makeRustCall( + _ callback: (UnsafeMutablePointer) -> T, + errorHandler: ((RustBuffer) throws -> E)? +) throws -> T { + uniffiEnsureLitersFfiInitialized() + var callStatus = RustCallStatus.init() + let returnedVal = callback(&callStatus) + try uniffiCheckCallStatus(callStatus: callStatus, errorHandler: errorHandler) + return returnedVal +} + +private func uniffiCheckCallStatus( + callStatus: RustCallStatus, + errorHandler: ((RustBuffer) throws -> E)? +) throws { + switch callStatus.code { + case CALL_SUCCESS: + return + + case CALL_ERROR: + if let errorHandler = errorHandler { + throw try errorHandler(callStatus.errorBuf) + } else { + callStatus.errorBuf.deallocate() + throw UniffiInternalError.unexpectedRustCallError + } + + case CALL_UNEXPECTED_ERROR: + // When the rust code sees a panic, it tries to construct a RustBuffer + // with the message. But if that code panics, then it just sends back + // an empty buffer. + if callStatus.errorBuf.len > 0 { + throw UniffiInternalError.rustPanic(try FfiConverterString.lift(callStatus.errorBuf)) + } else { + callStatus.errorBuf.deallocate() + throw UniffiInternalError.rustPanic("Rust panic") + } + + case CALL_CANCELLED: + fatalError("Cancellation not supported yet") + + default: + throw UniffiInternalError.unexpectedRustCallStatusCode + } +} + +private func uniffiTraitInterfaceCall( + callStatus: UnsafeMutablePointer, + makeCall: () throws -> T, + writeReturn: (T) -> () +) { + do { + try writeReturn(makeCall()) + } catch let error { + callStatus.pointee.code = CALL_UNEXPECTED_ERROR + callStatus.pointee.errorBuf = FfiConverterString.lower(String(describing: error)) + } +} + +private func uniffiTraitInterfaceCallWithError( + callStatus: UnsafeMutablePointer, + makeCall: () throws -> T, + writeReturn: (T) -> (), + lowerError: (E) -> RustBuffer +) { + do { + try writeReturn(makeCall()) + } catch let error as E { + callStatus.pointee.code = CALL_ERROR + callStatus.pointee.errorBuf = lowerError(error) + } catch { + callStatus.pointee.code = CALL_UNEXPECTED_ERROR + callStatus.pointee.errorBuf = FfiConverterString.lower(String(describing: error)) + } +} +// Initial value and increment amount for handles. +// These ensure that SWIFT handles always have the lowest bit set +fileprivate let UNIFFI_HANDLEMAP_INITIAL: UInt64 = 1 +fileprivate let UNIFFI_HANDLEMAP_DELTA: UInt64 = 2 + +fileprivate final class UniffiHandleMap: @unchecked Sendable { + // All mutation happens with this lock held, which is why we implement @unchecked Sendable. + private let lock = NSLock() + private var map: [UInt64: T] = [:] + private var currentHandle: UInt64 = UNIFFI_HANDLEMAP_INITIAL + + func insert(obj: T) -> UInt64 { + lock.withLock { + return doInsert(obj) + } + } + + // Low-level insert function, this assumes `lock` is held. + private func doInsert(_ obj: T) -> UInt64 { + let handle = currentHandle + currentHandle += UNIFFI_HANDLEMAP_DELTA + map[handle] = obj + return handle + } + + func get(handle: UInt64) throws -> T { + try lock.withLock { + guard let obj = map[handle] else { + throw UniffiInternalError.unexpectedStaleHandle + } + return obj + } + } + + func clone(handle: UInt64) throws -> UInt64 { + try lock.withLock { + guard let obj = map[handle] else { + throw UniffiInternalError.unexpectedStaleHandle + } + return doInsert(obj) + } + } + + @discardableResult + func remove(handle: UInt64) throws -> T { + try lock.withLock { + guard let obj = map.removeValue(forKey: handle) else { + throw UniffiInternalError.unexpectedStaleHandle + } + return obj + } + } + + var count: Int { + get { + map.count + } + } +} + + +// Public interface members begin here. +// Magic number for the Rust proxy to call using the same mechanism as every other method, +// to free the callback once it's dropped by Rust. +private let IDX_CALLBACK_FREE: Int32 = 0 +// Callback return codes +private let UNIFFI_CALLBACK_SUCCESS: Int32 = 0 +private let UNIFFI_CALLBACK_ERROR: Int32 = 1 +private let UNIFFI_CALLBACK_UNEXPECTED_ERROR: Int32 = 2 + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterUInt16: FfiConverterPrimitive { + typealias FfiType = UInt16 + typealias SwiftType = UInt16 + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> UInt16 { + return try lift(readInt(&buf)) + } + + public static func write(_ value: SwiftType, into buf: inout [UInt8]) { + writeInt(&buf, lower(value)) + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterUInt32: FfiConverterPrimitive { + typealias FfiType = UInt32 + typealias SwiftType = UInt32 + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> UInt32 { + return try lift(readInt(&buf)) + } + + public static func write(_ value: SwiftType, into buf: inout [UInt8]) { + writeInt(&buf, lower(value)) + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterUInt64: FfiConverterPrimitive { + typealias FfiType = UInt64 + typealias SwiftType = UInt64 + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> UInt64 { + return try lift(readInt(&buf)) + } + + public static func write(_ value: SwiftType, into buf: inout [UInt8]) { + writeInt(&buf, lower(value)) + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterDouble: FfiConverterPrimitive { + typealias FfiType = Double + typealias SwiftType = Double + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> Double { + return try lift(readDouble(&buf)) + } + + public static func write(_ value: Double, into buf: inout [UInt8]) { + writeDouble(&buf, lower(value)) + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterBool : FfiConverter { + typealias FfiType = Int8 + typealias SwiftType = Bool + + public static func lift(_ value: Int8) throws -> Bool { + return value != 0 + } + + public static func lower(_ value: Bool) -> Int8 { + return value ? 1 : 0 + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> Bool { + return try lift(readInt(&buf)) + } + + public static func write(_ value: Bool, into buf: inout [UInt8]) { + writeInt(&buf, lower(value)) + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterString: FfiConverter { + typealias SwiftType = String + typealias FfiType = RustBuffer + + public static func lift(_ value: RustBuffer) throws -> String { + defer { + value.deallocate() + } + if value.data == nil { + return String() + } + let bytes = UnsafeBufferPointer(start: value.data!, count: Int(value.len)) + // Use Swift's native UTF-8 decoder; `String(bytes:encoding:.utf8)` goes + // through Foundation's NSString and silently strips a leading U+FEFF BOM. + // Invalid UTF-8 substitutes U+FFFD instead of trapping (unreachable + // given Rust's `String` invariant). + return String(decoding: bytes, as: UTF8.self) + } + + public static func lower(_ value: String) -> RustBuffer { + return value.utf8CString.withUnsafeBufferPointer { ptr in + // The swift string gives us int8_t, we want uint8_t. + ptr.withMemoryRebound(to: UInt8.self) { ptr in + // The swift string gives us a trailing null byte, we don't want it. + let buf = UnsafeBufferPointer(rebasing: ptr.prefix(upTo: ptr.count - 1)) + return RustBuffer.from(buf) + } + } + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> String { + let len: Int32 = try readInt(&buf) + // See `lift` above for why we avoid Foundation's NSString-backed decoder here. + return String(decoding: try readBytes(&buf, count: Int(len)), as: UTF8.self) + } + + public static func write(_ value: String, into buf: inout [UInt8]) { + let len = Int32(value.utf8.count) + writeInt(&buf, len) + writeBytes(&buf, value.utf8) + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterData: FfiConverterRustBuffer { + typealias SwiftType = Data + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> Data { + let len: Int32 = try readInt(&buf) + return Data(try readBytes(&buf, count: Int(len))) + } + + public static func write(_ value: Data, into buf: inout [UInt8]) { + let len = Int32(value.count) + writeInt(&buf, len) + writeBytes(&buf, value) + } +} + + + + +/** + * The host HTTP client: ONE shared instance for the whole process. Backed by + * one platform client (e.g. a single `OkHttpClient`) so requests to one + * authority coalesce onto a single connection. `body` is present only for + * `PUT` pushes. + */ +public protocol HttpClient: AnyObject, Sendable { + + func execute(request: HttpRequest, body: HttpRequestBody?) throws -> HttpResponse + +} +/** + * The host HTTP client: ONE shared instance for the whole process. Backed by + * one platform client (e.g. a single `OkHttpClient`) so requests to one + * authority coalesce onto a single connection. `body` is present only for + * `PUT` pushes. + */ +open class HttpClientImpl: HttpClient, @unchecked Sendable { + fileprivate let handle: UInt64 + + /// Used to instantiate a [FFIObject] without an actual handle, for fakes in tests, mostly. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public struct NoHandle { + public init() {} + } + + // TODO: We'd like this to be `private` but for Swifty reasons, + // we can't implement `FfiConverter` without making this `required` and we can't + // make it `required` without making it `public`. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + required public init(unsafeFromHandle handle: UInt64) { + self.handle = handle + } + + // This constructor can be used to instantiate a fake object. + // - Parameter noHandle: Placeholder value so we can have a constructor separate from the default empty one that may be implemented for classes extending [FFIObject]. + // + // - Warning: + // Any object instantiated with this constructor cannot be passed to an actual Rust-backed object. Since there isn't a backing handle the FFI lower functions will crash. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public init(noHandle: NoHandle) { + self.handle = 0 + } + +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public func uniffiCloneHandle() -> UInt64 { + return try! rustCall { uniffi_liters_ffi_fn_clone_httpclient(self.handle, $0) } + } + // No primary constructor declared for this class. + + deinit { + if handle == 0 { + // Mock objects have handle=0 don't try to free them + return + } + + try! rustCall { uniffi_liters_ffi_fn_free_httpclient(handle, $0) } + } + + + + +open func execute(request: HttpRequest, body: HttpRequestBody?)throws -> HttpResponse { + return try FfiConverterTypeHttpResponse_lift(try rustCallWithError(FfiConverterTypeHttpError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_httpclient_execute( + self.uniffiCloneHandle(), + FfiConverterTypeHttpRequest_lower(request), + FfiConverterOptionTypeHttpRequestBody.lower(body),uniffiCallStatus + ) +}) +} + + + +} + + + +// Put the implementation in a struct so we don't pollute the top-level namespace +fileprivate struct UniffiCallbackInterfaceHttpClient { + + // Create the VTable using a series of closures. + // Swift automatically converts these into C callback functions. + // + // Store the vtable directly. + static let vtable: UniffiVTableCallbackInterfaceHttpClient = UniffiVTableCallbackInterfaceHttpClient( + uniffiFree: { (uniffiHandle: UInt64) -> () in + do { + try FfiConverterTypeHttpClient.handleMap.remove(handle: uniffiHandle) + } catch { + print("Uniffi callback interface HttpClient: handle missing in uniffiFree") + } + }, + uniffiClone: { (uniffiHandle: UInt64) -> UInt64 in + do { + return try FfiConverterTypeHttpClient.handleMap.clone(handle: uniffiHandle) + } catch { + fatalError("Uniffi callback interface HttpClient: handle missing in uniffiClone") + } + }, + execute: { ( + uniffiHandle: UInt64, + request: RustBuffer, + body: RustBuffer, + uniffiOutReturn: UnsafeMutablePointer, + uniffiCallStatus: UnsafeMutablePointer + ) in + let makeCall = { + () throws -> HttpResponse in + guard let uniffiObj = try? FfiConverterTypeHttpClient.handleMap.get(handle: uniffiHandle) else { + throw UniffiInternalError.unexpectedStaleHandle + } + return try uniffiObj.execute( + request: try FfiConverterTypeHttpRequest_lift(request), + body: try FfiConverterOptionTypeHttpRequestBody.lift(body) + ) + } + + + let writeReturn = { uniffiOutReturn.pointee = FfiConverterTypeHttpResponse_lower($0) } + uniffiTraitInterfaceCallWithError( + callStatus: uniffiCallStatus, + makeCall: makeCall, + writeReturn: writeReturn, + lowerError: FfiConverterTypeHttpError_lower + ) + } + ) + + // Rust stores this pointer for future callback invocations, so it must live + // for the process lifetime (not just for the init function call). + // + // `nonisolated(unsafe)` is needed under Swift 6 strict concurrency. + // This is safe because the pointee is initialized once during static init + // and never mutated by either side of the FFI. Its fields are C function pointers. + nonisolated(unsafe) static let vtablePtr: UnsafePointer = { + let ptr = UnsafeMutablePointer.allocate(capacity: 1) + ptr.initialize(to: vtable) + return UnsafePointer(ptr) + }() +} + +private func uniffiCallbackInitHttpClient() { + uniffi_liters_ffi_fn_init_callback_vtable_httpclient(UniffiCallbackInterfaceHttpClient.vtablePtr) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeHttpClient: FfiConverter { + fileprivate static let handleMap = UniffiHandleMap() + + typealias FfiType = UInt64 + typealias SwiftType = HttpClient + + public static func lift(_ handle: UInt64) throws -> HttpClient { + if ((handle & 1) == 0) { + // Rust-generated handle, construct a new class that uses the handle to implement the + // interface + return HttpClientImpl(unsafeFromHandle: handle) + } else { + // Swift-generated handle, get the object from the handle map + return try handleMap.remove(handle: handle) + } + } + + public static func lower(_ value: HttpClient) -> UInt64 { + if let rustImpl = value as? HttpClientImpl { + // Rust-implemented object. Clone the handle and return it + return rustImpl.uniffiCloneHandle() + } else { + // Swift object, generate a new vtable handle and return that. + return handleMap.insert(obj: value) + } + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> HttpClient { + let handle: UInt64 = try readInt(&buf) + return try lift(handle) + } + + public static func write(_ value: HttpClient, into buf: inout [UInt8]) { + writeInt(&buf, lower(value)) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeHttpClient_lift(_ handle: UInt64) throws -> HttpClient { + return try FfiConverterTypeHttpClient.lift(handle) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeHttpClient_lower(_ value: HttpClient) -> UInt64 { + return FfiConverterTypeHttpClient.lower(value) +} + + + + + + +/** + * The request-body source for a `PUT`, implemented by liters and *called by + * the host*: the host pulls bytes as it streams the request body, so a push + * is never buffered whole. `read` returns up to `max` bytes; an empty result + * is end-of-body. Throwing aborts the upload. + */ +public protocol HttpRequestBodyProtocol: AnyObject, Sendable { + + func read(max: UInt32) throws -> Data + +} +/** + * The request-body source for a `PUT`, implemented by liters and *called by + * the host*: the host pulls bytes as it streams the request body, so a push + * is never buffered whole. `read` returns up to `max` bytes; an empty result + * is end-of-body. Throwing aborts the upload. + */ +open class HttpRequestBody: HttpRequestBodyProtocol, @unchecked Sendable { + fileprivate let handle: UInt64 + + /// Used to instantiate a [FFIObject] without an actual handle, for fakes in tests, mostly. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public struct NoHandle { + public init() {} + } + + // TODO: We'd like this to be `private` but for Swifty reasons, + // we can't implement `FfiConverter` without making this `required` and we can't + // make it `required` without making it `public`. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + required public init(unsafeFromHandle handle: UInt64) { + self.handle = handle + } + + // This constructor can be used to instantiate a fake object. + // - Parameter noHandle: Placeholder value so we can have a constructor separate from the default empty one that may be implemented for classes extending [FFIObject]. + // + // - Warning: + // Any object instantiated with this constructor cannot be passed to an actual Rust-backed object. Since there isn't a backing handle the FFI lower functions will crash. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public init(noHandle: NoHandle) { + self.handle = 0 + } + +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public func uniffiCloneHandle() -> UInt64 { + return try! rustCall { uniffi_liters_ffi_fn_clone_httprequestbody(self.handle, $0) } + } + // No primary constructor declared for this class. + + deinit { + if handle == 0 { + // Mock objects have handle=0 don't try to free them + return + } + + try! rustCall { uniffi_liters_ffi_fn_free_httprequestbody(handle, $0) } + } + + + + +open func read(max: UInt32)throws -> Data { + return try FfiConverterData.lift(try rustCallWithError(FfiConverterTypeHttpError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_httprequestbody_read( + self.uniffiCloneHandle(), + FfiConverterUInt32.lower(max),uniffiCallStatus + ) +}) +} + + + +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeHttpRequestBody: FfiConverter { + typealias FfiType = UInt64 + typealias SwiftType = HttpRequestBody + + public static func lift(_ handle: UInt64) throws -> HttpRequestBody { + return HttpRequestBody(unsafeFromHandle: handle) + } + + public static func lower(_ value: HttpRequestBody) -> UInt64 { + return value.uniffiCloneHandle() + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> HttpRequestBody { + let handle: UInt64 = try readInt(&buf) + return try lift(handle) + } + + public static func write(_ value: HttpRequestBody, into buf: inout [UInt8]) { + writeInt(&buf, lower(value)) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeHttpRequestBody_lift(_ handle: UInt64) throws -> HttpRequestBody { + return try FfiConverterTypeHttpRequestBody.lift(handle) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeHttpRequestBody_lower(_ value: HttpRequestBody) -> UInt64 { + return FfiConverterTypeHttpRequestBody.lower(value) +} + + + + + + +/** + * A response, implemented by the host over its platform client's response. + */ +public protocol HttpResponse: AnyObject, Sendable { + + func status() -> UInt16 + + func headers() -> [HttpHeader] + + /** + * Pulls the next body bytes, blocking until data, an idle tick + * (`long_lived` only), or end-of-body. Throwing ends the body with a + * transport/protocol error. + */ + func readBody(max: UInt32) throws -> BodyChunk + + /** + * Aborts the in-flight call (e.g. OkHttp `Call.cancel()`). Best-effort and + * idempotent; must not throw. + */ + func cancel() + +} +/** + * A response, implemented by the host over its platform client's response. + */ +open class HttpResponseImpl: HttpResponse, @unchecked Sendable { + fileprivate let handle: UInt64 + + /// Used to instantiate a [FFIObject] without an actual handle, for fakes in tests, mostly. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public struct NoHandle { + public init() {} + } + + // TODO: We'd like this to be `private` but for Swifty reasons, + // we can't implement `FfiConverter` without making this `required` and we can't + // make it `required` without making it `public`. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + required public init(unsafeFromHandle handle: UInt64) { + self.handle = handle + } + + // This constructor can be used to instantiate a fake object. + // - Parameter noHandle: Placeholder value so we can have a constructor separate from the default empty one that may be implemented for classes extending [FFIObject]. + // + // - Warning: + // Any object instantiated with this constructor cannot be passed to an actual Rust-backed object. Since there isn't a backing handle the FFI lower functions will crash. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public init(noHandle: NoHandle) { + self.handle = 0 + } + +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public func uniffiCloneHandle() -> UInt64 { + return try! rustCall { uniffi_liters_ffi_fn_clone_httpresponse(self.handle, $0) } + } + // No primary constructor declared for this class. + + deinit { + if handle == 0 { + // Mock objects have handle=0 don't try to free them + return + } + + try! rustCall { uniffi_liters_ffi_fn_free_httpresponse(handle, $0) } + } + + + + +open func status() -> UInt16 { + return try! FfiConverterUInt16.lift(try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_httpresponse_status( + self.uniffiCloneHandle(),uniffiCallStatus + ) +}) +} + +open func headers() -> [HttpHeader] { + return try! FfiConverterSequenceTypeHttpHeader.lift(try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_httpresponse_headers( + self.uniffiCloneHandle(),uniffiCallStatus + ) +}) +} + + /** + * Pulls the next body bytes, blocking until data, an idle tick + * (`long_lived` only), or end-of-body. Throwing ends the body with a + * transport/protocol error. + */ +open func readBody(max: UInt32)throws -> BodyChunk { + return try FfiConverterTypeBodyChunk_lift(try rustCallWithError(FfiConverterTypeHttpError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_httpresponse_read_body( + self.uniffiCloneHandle(), + FfiConverterUInt32.lower(max),uniffiCallStatus + ) +}) +} + + /** + * Aborts the in-flight call (e.g. OkHttp `Call.cancel()`). Best-effort and + * idempotent; must not throw. + */ +open func cancel() {try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_httpresponse_cancel( + self.uniffiCloneHandle(),uniffiCallStatus + ) +} +} + + + +} + + + +// Put the implementation in a struct so we don't pollute the top-level namespace +fileprivate struct UniffiCallbackInterfaceHttpResponse { + + // Create the VTable using a series of closures. + // Swift automatically converts these into C callback functions. + // + // Store the vtable directly. + static let vtable: UniffiVTableCallbackInterfaceHttpResponse = UniffiVTableCallbackInterfaceHttpResponse( + uniffiFree: { (uniffiHandle: UInt64) -> () in + do { + try FfiConverterTypeHttpResponse.handleMap.remove(handle: uniffiHandle) + } catch { + print("Uniffi callback interface HttpResponse: handle missing in uniffiFree") + } + }, + uniffiClone: { (uniffiHandle: UInt64) -> UInt64 in + do { + return try FfiConverterTypeHttpResponse.handleMap.clone(handle: uniffiHandle) + } catch { + fatalError("Uniffi callback interface HttpResponse: handle missing in uniffiClone") + } + }, + status: { ( + uniffiHandle: UInt64, + uniffiOutReturn: UnsafeMutablePointer, + uniffiCallStatus: UnsafeMutablePointer + ) in + let makeCall = { + () throws -> UInt16 in + guard let uniffiObj = try? FfiConverterTypeHttpResponse.handleMap.get(handle: uniffiHandle) else { + throw UniffiInternalError.unexpectedStaleHandle + } + return uniffiObj.status( + ) + } + + + let writeReturn = { uniffiOutReturn.pointee = FfiConverterUInt16.lower($0) } + uniffiTraitInterfaceCall( + callStatus: uniffiCallStatus, + makeCall: makeCall, + writeReturn: writeReturn + ) + }, + headers: { ( + uniffiHandle: UInt64, + uniffiOutReturn: UnsafeMutablePointer, + uniffiCallStatus: UnsafeMutablePointer + ) in + let makeCall = { + () throws -> [HttpHeader] in + guard let uniffiObj = try? FfiConverterTypeHttpResponse.handleMap.get(handle: uniffiHandle) else { + throw UniffiInternalError.unexpectedStaleHandle + } + return uniffiObj.headers( + ) + } + + + let writeReturn = { uniffiOutReturn.pointee = FfiConverterSequenceTypeHttpHeader.lower($0) } + uniffiTraitInterfaceCall( + callStatus: uniffiCallStatus, + makeCall: makeCall, + writeReturn: writeReturn + ) + }, + readBody: { ( + uniffiHandle: UInt64, + max: UInt32, + uniffiOutReturn: UnsafeMutablePointer, + uniffiCallStatus: UnsafeMutablePointer + ) in + let makeCall = { + () throws -> BodyChunk in + guard let uniffiObj = try? FfiConverterTypeHttpResponse.handleMap.get(handle: uniffiHandle) else { + throw UniffiInternalError.unexpectedStaleHandle + } + return try uniffiObj.readBody( + max: try FfiConverterUInt32.lift(max) + ) + } + + + let writeReturn = { uniffiOutReturn.pointee = FfiConverterTypeBodyChunk_lower($0) } + uniffiTraitInterfaceCallWithError( + callStatus: uniffiCallStatus, + makeCall: makeCall, + writeReturn: writeReturn, + lowerError: FfiConverterTypeHttpError_lower + ) + }, + cancel: { ( + uniffiHandle: UInt64, + uniffiOutReturn: UnsafeMutableRawPointer, + uniffiCallStatus: UnsafeMutablePointer + ) in + let makeCall = { + () throws -> () in + guard let uniffiObj = try? FfiConverterTypeHttpResponse.handleMap.get(handle: uniffiHandle) else { + throw UniffiInternalError.unexpectedStaleHandle + } + return uniffiObj.cancel( + ) + } + + + let writeReturn = { () } + uniffiTraitInterfaceCall( + callStatus: uniffiCallStatus, + makeCall: makeCall, + writeReturn: writeReturn + ) + } + ) + + // Rust stores this pointer for future callback invocations, so it must live + // for the process lifetime (not just for the init function call). + // + // `nonisolated(unsafe)` is needed under Swift 6 strict concurrency. + // This is safe because the pointee is initialized once during static init + // and never mutated by either side of the FFI. Its fields are C function pointers. + nonisolated(unsafe) static let vtablePtr: UnsafePointer = { + let ptr = UnsafeMutablePointer.allocate(capacity: 1) + ptr.initialize(to: vtable) + return UnsafePointer(ptr) + }() +} + +private func uniffiCallbackInitHttpResponse() { + uniffi_liters_ffi_fn_init_callback_vtable_httpresponse(UniffiCallbackInterfaceHttpResponse.vtablePtr) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeHttpResponse: FfiConverter { + fileprivate static let handleMap = UniffiHandleMap() + + typealias FfiType = UInt64 + typealias SwiftType = HttpResponse + + public static func lift(_ handle: UInt64) throws -> HttpResponse { + if ((handle & 1) == 0) { + // Rust-generated handle, construct a new class that uses the handle to implement the + // interface + return HttpResponseImpl(unsafeFromHandle: handle) + } else { + // Swift-generated handle, get the object from the handle map + return try handleMap.remove(handle: handle) + } + } + + public static func lower(_ value: HttpResponse) -> UInt64 { + if let rustImpl = value as? HttpResponseImpl { + // Rust-implemented object. Clone the handle and return it + return rustImpl.uniffiCloneHandle() + } else { + // Swift object, generate a new vtable handle and return that. + return handleMap.insert(obj: value) + } + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> HttpResponse { + let handle: UInt64 = try readInt(&buf) + return try lift(handle) + } + + public static func write(_ value: HttpResponse, into buf: inout [UInt8]) { + writeInt(&buf, lower(value)) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeHttpResponse_lift(_ handle: UInt64) throws -> HttpResponse { + return try FfiConverterTypeHttpResponse.lift(handle) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeHttpResponse_lower(_ value: HttpResponse) -> UInt64 { + return FfiConverterTypeHttpResponse.lower(value) +} + + + + + + +/** + * Runs replication for any number of registered databases on background + * threads: each database either pushes to a bucket or follows one, with + * automatic retry/backoff, and per-database or global sleep/resume for + * mobile power management. `sleep()` cancels in-flight transfers and + * releases the database's WAL lock and file descriptors; `resume()` + * schedules an immediate catch-up round. + * + * All methods return immediately except `unregister()` and `shutdown()`, + * which join worker threads (bounded by the storage backend's cancellation + * latency — a few seconds worst case over HTTP). Call `shutdown()` on app + * teardown for deterministic cleanup; it also runs when the object is + * destroyed. + */ +public protocol LitersManagerProtocol: AnyObject, Sendable { + + /** + * Nudges a push registration to run a round now (also the retry path + * out of `Failed`). Ignored while sleeping — resume first. Errors on + * an unknown id or a follow registration. + */ + func pushNow(id: String) throws + + /** + * Registers a local path to follow (materialize) a bucket and starts + * its worker. `db_path` need not exist yet — the first sync restores + * it. Errors on a duplicate id or an already-registered path. + */ + func registerFollow(id: String, dbPath: String, storage: Storage, options: FollowOptions) throws + + /** + * Registers a local database to push to `storage` and starts its + * worker. The database file must exist. Errors on a duplicate id or an + * already-registered path. For HTTP storage the manager persists a + * per-database writer id next to the database and sends it for + * server-side fencing. + */ + func registerPush(id: String, dbPath: String, storage: Storage, options: PushOptions) throws + + /** + * Wakes a sleeping database and schedules an immediate catch-up round. + * On a running-but-`Failed` database this acts as a retry nudge. + * Idempotent. + */ + func resume(id: String) throws + + /** + * `resume()` for every registered database (app returned to + * foreground). + */ + func resumeAll() + + /** + * Installs (or clears, with `null`) the event listener. See + * `ManagerListener` for the threading rules. + */ + func setListener(listener: ManagerListener?) + + /** + * Cancels every session and joins every worker thread. Idempotent; + * also runs when the object is destroyed. + */ + func shutdown() + + /** + * Puts one database to sleep: aborts its in-flight transfer, releases + * its WAL lock and file descriptors, and stops all storage traffic + * until `resume()`. Returns immediately; teardown completes on the + * worker. Idempotent. + */ + func sleep(id: String) throws + + /** + * `sleep()` for every registered database (app went to background). + */ + func sleepAll() + + /** + * Status of one registered database, or `null` for an unknown id. + */ + func status(id: String) -> DbStatus? + + /** + * Statuses of all registered databases, sorted by id. + */ + func statuses() -> [DbStatus] + + /** + * Forces a follow registration to resync immediately by restarting its + * follow session. Also the retry path out of `Failed`. Ignored while + * sleeping. Errors on an unknown id or a push registration. + */ + func syncNow(id: String) throws + + /** + * Removes a registration: cancels its session, joins its worker, and + * releases its database. Blocks briefly; the id becomes free for + * re-registration. Unknown id errors. + */ + func unregister(id: String) throws + +} +/** + * Runs replication for any number of registered databases on background + * threads: each database either pushes to a bucket or follows one, with + * automatic retry/backoff, and per-database or global sleep/resume for + * mobile power management. `sleep()` cancels in-flight transfers and + * releases the database's WAL lock and file descriptors; `resume()` + * schedules an immediate catch-up round. + * + * All methods return immediately except `unregister()` and `shutdown()`, + * which join worker threads (bounded by the storage backend's cancellation + * latency — a few seconds worst case over HTTP). Call `shutdown()` on app + * teardown for deterministic cleanup; it also runs when the object is + * destroyed. + */ +open class LitersManager: LitersManagerProtocol, @unchecked Sendable { + fileprivate let handle: UInt64 + + /// Used to instantiate a [FFIObject] without an actual handle, for fakes in tests, mostly. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public struct NoHandle { + public init() {} + } + + // TODO: We'd like this to be `private` but for Swifty reasons, + // we can't implement `FfiConverter` without making this `required` and we can't + // make it `required` without making it `public`. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + required public init(unsafeFromHandle handle: UInt64) { + self.handle = handle + } + + // This constructor can be used to instantiate a fake object. + // - Parameter noHandle: Placeholder value so we can have a constructor separate from the default empty one that may be implemented for classes extending [FFIObject]. + // + // - Warning: + // Any object instantiated with this constructor cannot be passed to an actual Rust-backed object. Since there isn't a backing handle the FFI lower functions will crash. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public init(noHandle: NoHandle) { + self.handle = 0 + } + +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public func uniffiCloneHandle() -> UInt64 { + return try! rustCall { uniffi_liters_ffi_fn_clone_litersmanager(self.handle, $0) } + } + /** + * A manager with default options (liters backoff, no default push + * interval — databases push on `push_now()` unless registered with + * their own interval) and no host HTTP transport. + */ +public convenience init() { + let handle = + try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_constructor_litersmanager_new(uniffiCallStatus + ) +} + self.init(unsafeFromHandle: handle) +} + + deinit { + if handle == 0 { + // Mock objects have handle=0 don't try to free them + return + } + + try! rustCall { uniffi_liters_ffi_fn_free_litersmanager(handle, $0) } + } + + + /** + * A manager with explicit options and, optionally, a host HTTP transport. + * + * Pass `http_client` (one shared instance backed by a single platform + * HTTP client, e.g. one `OkHttpClient`) to have every HTTP follow/push run + * through it: all followers to one authority then coalesce onto a single + * HTTP/2 connection, with the platform owning TLS, the system trust store, + * and keepalive. `null` keeps the built-in socket transport (one + * `Connection: close` request per call, `http://` only). + */ +public static func withOptions(options: ManagerOptions, httpClient: HttpClient?) -> LitersManager { + return try! FfiConverterTypeLitersManager_lift(try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_constructor_litersmanager_with_options( + FfiConverterTypeManagerOptions_lower(options), + FfiConverterOptionTypeHttpClient.lower(httpClient),uniffiCallStatus + ) +}) +} + + + + /** + * Nudges a push registration to run a round now (also the retry path + * out of `Failed`). Ignored while sleeping — resume first. Errors on + * an unknown id or a follow registration. + */ +open func pushNow(id: String)throws {try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_push_now( + self.uniffiCloneHandle(), + FfiConverterString.lower(id),uniffiCallStatus + ) +} +} + + /** + * Registers a local path to follow (materialize) a bucket and starts + * its worker. `db_path` need not exist yet — the first sync restores + * it. Errors on a duplicate id or an already-registered path. + */ +open func registerFollow(id: String, dbPath: String, storage: Storage, options: FollowOptions)throws {try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_register_follow( + self.uniffiCloneHandle(), + FfiConverterString.lower(id), + FfiConverterString.lower(dbPath), + FfiConverterTypeStorage_lower(storage), + FfiConverterTypeFollowOptions_lower(options),uniffiCallStatus + ) +} +} + + /** + * Registers a local database to push to `storage` and starts its + * worker. The database file must exist. Errors on a duplicate id or an + * already-registered path. For HTTP storage the manager persists a + * per-database writer id next to the database and sends it for + * server-side fencing. + */ +open func registerPush(id: String, dbPath: String, storage: Storage, options: PushOptions)throws {try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_register_push( + self.uniffiCloneHandle(), + FfiConverterString.lower(id), + FfiConverterString.lower(dbPath), + FfiConverterTypeStorage_lower(storage), + FfiConverterTypePushOptions_lower(options),uniffiCallStatus + ) +} +} + + /** + * Wakes a sleeping database and schedules an immediate catch-up round. + * On a running-but-`Failed` database this acts as a retry nudge. + * Idempotent. + */ +open func resume(id: String)throws {try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_resume( + self.uniffiCloneHandle(), + FfiConverterString.lower(id),uniffiCallStatus + ) +} +} + + /** + * `resume()` for every registered database (app returned to + * foreground). + */ +open func resumeAll() {try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_resume_all( + self.uniffiCloneHandle(),uniffiCallStatus + ) +} +} + + /** + * Installs (or clears, with `null`) the event listener. See + * `ManagerListener` for the threading rules. + */ +open func setListener(listener: ManagerListener?) {try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_set_listener( + self.uniffiCloneHandle(), + FfiConverterOptionTypeManagerListener.lower(listener),uniffiCallStatus + ) +} +} + + /** + * Cancels every session and joins every worker thread. Idempotent; + * also runs when the object is destroyed. + */ +open func shutdown() {try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_shutdown( + self.uniffiCloneHandle(),uniffiCallStatus + ) +} +} + + /** + * Puts one database to sleep: aborts its in-flight transfer, releases + * its WAL lock and file descriptors, and stops all storage traffic + * until `resume()`. Returns immediately; teardown completes on the + * worker. Idempotent. + */ +open func sleep(id: String)throws {try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_sleep( + self.uniffiCloneHandle(), + FfiConverterString.lower(id),uniffiCallStatus + ) +} +} + + /** + * `sleep()` for every registered database (app went to background). + */ +open func sleepAll() {try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_sleep_all( + self.uniffiCloneHandle(),uniffiCallStatus + ) +} +} + + /** + * Status of one registered database, or `null` for an unknown id. + */ +open func status(id: String) -> DbStatus? { + return try! FfiConverterOptionTypeDbStatus.lift(try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_status( + self.uniffiCloneHandle(), + FfiConverterString.lower(id),uniffiCallStatus + ) +}) +} + + /** + * Statuses of all registered databases, sorted by id. + */ +open func statuses() -> [DbStatus] { + return try! FfiConverterSequenceTypeDbStatus.lift(try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_statuses( + self.uniffiCloneHandle(),uniffiCallStatus + ) +}) +} + + /** + * Forces a follow registration to resync immediately by restarting its + * follow session. Also the retry path out of `Failed`. Ignored while + * sleeping. Errors on an unknown id or a push registration. + */ +open func syncNow(id: String)throws {try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_sync_now( + self.uniffiCloneHandle(), + FfiConverterString.lower(id),uniffiCallStatus + ) +} +} + + /** + * Removes a registration: cancels its session, joins its worker, and + * releases its database. Blocks briefly; the id becomes free for + * re-registration. Unknown id errors. + */ +open func unregister(id: String)throws {try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersmanager_unregister( + self.uniffiCloneHandle(), + FfiConverterString.lower(id),uniffiCallStatus + ) +} +} + + + +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeLitersManager: FfiConverter { + typealias FfiType = UInt64 + typealias SwiftType = LitersManager + + public static func lift(_ handle: UInt64) throws -> LitersManager { + return LitersManager(unsafeFromHandle: handle) + } + + public static func lower(_ value: LitersManager) -> UInt64 { + return value.uniffiCloneHandle() + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> LitersManager { + let handle: UInt64 = try readInt(&buf) + return try lift(handle) + } + + public static func write(_ value: LitersManager, into buf: inout [UInt8]) { + writeInt(&buf, lower(value)) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeLitersManager_lift(_ handle: UInt64) throws -> LitersManager { + return try FfiConverterTypeLitersManager.lift(handle) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeLitersManager_lower(_ value: LitersManager) -> UInt64 { + return FfiConverterTypeLitersManager.lower(value) +} + + + + + + +/** + * A local read-only materialization of a bucket. `sync()` restores on first + * use, then applies changes incrementally. Open `db_path()` read-only with + * your platform SQLite; do not hold read transactions across `sync()` calls + * from the same process. For continuous (live) following, register the + * database with `LitersManager.register_follow` instead of polling. + */ +public protocol LitersReplicaProtocol: AnyObject, Sendable { + + /** + * Aborts the in-flight `sync`, if any: it returns `Cancelled` promptly. + * Each call installs a fresh token at its start, so cancel only ever + * affects the operation currently running. A cancelled sync resumes + * exactly where it stopped on the next call. + */ + func cancel() + + /** + * Tears the replica down: cancels any in-flight sync, waits for it to + * return, then drops the storage client. Subsequent calls (except + * `db_path()`) error with "LitersReplica is closed" — including calls + * already queued behind the cancelled one. Idempotent. + */ + func close() + + /** + * Path of the local database file to open read-only. + */ + func dbPath() -> String + + /** + * Last applied TXID (zero if never synced). + */ + func position() throws -> UInt64 + + /** + * Deletes the local replica; the next `sync()` restores from scratch. + */ + func reset() throws + + /** + * Brings the local replica up to date. Short and resumable; safe to + * call from a background task with a tight deadline. + */ + func sync() throws -> SyncSummary + +} +/** + * A local read-only materialization of a bucket. `sync()` restores on first + * use, then applies changes incrementally. Open `db_path()` read-only with + * your platform SQLite; do not hold read transactions across `sync()` calls + * from the same process. For continuous (live) following, register the + * database with `LitersManager.register_follow` instead of polling. + */ +open class LitersReplica: LitersReplicaProtocol, @unchecked Sendable { + fileprivate let handle: UInt64 + + /// Used to instantiate a [FFIObject] without an actual handle, for fakes in tests, mostly. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public struct NoHandle { + public init() {} + } + + // TODO: We'd like this to be `private` but for Swifty reasons, + // we can't implement `FfiConverter` without making this `required` and we can't + // make it `required` without making it `public`. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + required public init(unsafeFromHandle handle: UInt64) { + self.handle = handle + } + + // This constructor can be used to instantiate a fake object. + // - Parameter noHandle: Placeholder value so we can have a constructor separate from the default empty one that may be implemented for classes extending [FFIObject]. + // + // - Warning: + // Any object instantiated with this constructor cannot be passed to an actual Rust-backed object. Since there isn't a backing handle the FFI lower functions will crash. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public init(noHandle: NoHandle) { + self.handle = 0 + } + +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public func uniffiCloneHandle() -> UInt64 { + return try! rustCall { uniffi_liters_ffi_fn_clone_litersreplica(self.handle, $0) } + } + /** + * `auto_reset`: on bucket divergence, silently delete the local replica + * and re-restore instead of raising `Diverged`. + */ +public convenience init(dbPath: String, storage: Storage, autoReset: Bool)throws { + let handle = + try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_constructor_litersreplica_new( + FfiConverterString.lower(dbPath), + FfiConverterTypeStorage_lower(storage), + FfiConverterBool.lower(autoReset),uniffiCallStatus + ) +} + self.init(unsafeFromHandle: handle) +} + + deinit { + if handle == 0 { + // Mock objects have handle=0 don't try to free them + return + } + + try! rustCall { uniffi_liters_ffi_fn_free_litersreplica(handle, $0) } + } + + + /** + * [`LitersReplica::new`], with the app's own HTTP client executing every + * `Storage::Http` request. See [`LitersWriter::new_with_http_client`] for + * what the built-in transport cannot do; `null` is exactly + * [`LitersReplica::new`]. + */ +public static func newWithHttpClient(dbPath: String, storage: Storage, autoReset: Bool, httpClient: HttpClient?)throws -> LitersReplica { + return try FfiConverterTypeLitersReplica_lift(try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_constructor_litersreplica_new_with_http_client( + FfiConverterString.lower(dbPath), + FfiConverterTypeStorage_lower(storage), + FfiConverterBool.lower(autoReset), + FfiConverterOptionTypeHttpClient.lower(httpClient),uniffiCallStatus + ) +}) +} + + + + /** + * Aborts the in-flight `sync`, if any: it returns `Cancelled` promptly. + * Each call installs a fresh token at its start, so cancel only ever + * affects the operation currently running. A cancelled sync resumes + * exactly where it stopped on the next call. + */ +open func cancel() {try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersreplica_cancel( + self.uniffiCloneHandle(),uniffiCallStatus + ) +} +} + + /** + * Tears the replica down: cancels any in-flight sync, waits for it to + * return, then drops the storage client. Subsequent calls (except + * `db_path()`) error with "LitersReplica is closed" — including calls + * already queued behind the cancelled one. Idempotent. + */ +open func close() {try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersreplica_close( + self.uniffiCloneHandle(),uniffiCallStatus + ) +} +} + + /** + * Path of the local database file to open read-only. + */ +open func dbPath() -> String { + return try! FfiConverterString.lift(try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersreplica_db_path( + self.uniffiCloneHandle(),uniffiCallStatus + ) +}) +} + + /** + * Last applied TXID (zero if never synced). + */ +open func position()throws -> UInt64 { + return try FfiConverterUInt64.lift(try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersreplica_position( + self.uniffiCloneHandle(),uniffiCallStatus + ) +}) +} + + /** + * Deletes the local replica; the next `sync()` restores from scratch. + */ +open func reset()throws {try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersreplica_reset( + self.uniffiCloneHandle(),uniffiCallStatus + ) +} +} + + /** + * Brings the local replica up to date. Short and resumable; safe to + * call from a background task with a tight deadline. + */ +open func sync()throws -> SyncSummary { + return try FfiConverterTypeSyncSummary_lift(try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_litersreplica_sync( + self.uniffiCloneHandle(),uniffiCallStatus + ) +}) +} + + + +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeLitersReplica: FfiConverter { + typealias FfiType = UInt64 + typealias SwiftType = LitersReplica + + public static func lift(_ handle: UInt64) throws -> LitersReplica { + return LitersReplica(unsafeFromHandle: handle) + } + + public static func lower(_ value: LitersReplica) -> UInt64 { + return value.uniffiCloneHandle() + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> LitersReplica { + let handle: UInt64 = try readInt(&buf) + return try lift(handle) + } + + public static func write(_ value: LitersReplica, into buf: inout [UInt8]) { + writeInt(&buf, lower(value)) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeLitersReplica_lift(_ handle: UInt64) throws -> LitersReplica { + return try FfiConverterTypeLitersReplica.lift(handle) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeLitersReplica_lower(_ value: LitersReplica) -> UInt64 { + return FfiConverterTypeLitersReplica.lower(value) +} + + + + + + +/** + * Replicates a local, app-owned SQLite database to a bucket. Call `push()` + * after commits (or batched); call `maintain()` opportunistically (wifi + + * charging) to compact and enforce retention. For multiple databases with + * scheduled pushes and sleep/resume, use `LitersManager` instead. + */ +public protocol LitersWriterProtocol: AnyObject, Sendable { + + /** + * Aborts the in-flight `push`/`maintain`/`snapshot`, if any: it returns + * `Cancelled` promptly (mid-transfer on the HTTP backend). Each call + * installs a fresh token at its start, so cancel only ever affects the + * operation currently running — never a future call. Cancel-then-retry + * is indistinguishable from kill-then-restart. + */ + func cancel() + + /** + * Tears the writer down: cancels any in-flight operation, waits for it + * to return, then releases the WAL read lock, SQLite connections, and + * file descriptors. Subsequent calls error with "LitersWriter is + * closed" — including calls already queued behind the cancelled one, + * which bail instead of running (so this never blocks for more than + * the single in-flight operation's cancellation latency). Idempotent. + */ + func close() + + /** + * Runs due compaction, snapshotting, and retention with litestream's + * default cadences. + */ + func maintain() throws -> MaintenanceSummary + + /** + * Current local replication position. + */ + func position() throws -> UInt64 + + /** + * Captures all committed changes into the bucket. Short and resumable: + * on failure, the next push picks up exactly where this one stopped. + */ + func push() throws -> PushSummary + + /** + * Recovery for a "local ltx file missing or corrupt" error: wipes local + * replication state so the next push re-derives everything from the + * bucket (snapshotting if needed). Local database content is untouched. + */ + func resetLocal() throws + + /** + * Forces a full snapshot to the bucket now. + */ + func snapshot() throws -> UInt64? + +} +/** + * Replicates a local, app-owned SQLite database to a bucket. Call `push()` + * after commits (or batched); call `maintain()` opportunistically (wifi + + * charging) to compact and enforce retention. For multiple databases with + * scheduled pushes and sleep/resume, use `LitersManager` instead. + */ +open class LitersWriter: LitersWriterProtocol, @unchecked Sendable { + fileprivate let handle: UInt64 + + /// Used to instantiate a [FFIObject] without an actual handle, for fakes in tests, mostly. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public struct NoHandle { + public init() {} + } + + // TODO: We'd like this to be `private` but for Swifty reasons, + // we can't implement `FfiConverter` without making this `required` and we can't + // make it `required` without making it `public`. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + required public init(unsafeFromHandle handle: UInt64) { + self.handle = handle + } + + // This constructor can be used to instantiate a fake object. + // - Parameter noHandle: Placeholder value so we can have a constructor separate from the default empty one that may be implemented for classes extending [FFIObject]. + // + // - Warning: + // Any object instantiated with this constructor cannot be passed to an actual Rust-backed object. Since there isn't a backing handle the FFI lower functions will crash. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public init(noHandle: NoHandle) { + self.handle = 0 + } + +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public func uniffiCloneHandle() -> UInt64 { + return try! rustCall { uniffi_liters_ffi_fn_clone_literswriter(self.handle, $0) } + } + /** + * Opens a writer for an existing SQLite database. Switches the database + * to WAL mode and takes over checkpointing: do not run your own + * `wal_checkpoint`, and **set `wal_autocheckpoint = 0` on every + * connection your app opens to this database**. + * + * That second half is not optional on mobile, which is the platform this + * crate exists for. The writer's read lock does starve a foreign + * autocheckpoint *while it is held* — but `LitersManager.sleep()` (and + * dropping the writer, and the app being killed) releases it, by design: + * see `Manager::sleep`, which "drops its Writer — releasing the + * WAL-pinning read transaction and every fd — and parks with zero storage + * traffic". On iOS that is most of the app's life. A foreign + * autocheckpoint that fires in that window restarts the WAL, and the next + * push finds its resume frame gone: `verify()` reports "wal truncated by + * another process" or "wal overwritten by another process" and recovers + * the only way it can, by writing a **full snapshot of the database**. + * + * Note also that `wal_autocheckpoint` is a per-*connection* setting, so a + * connection pool must apply it to every connection it opens, not once at + * startup. + * + * Turning autocheckpoint off makes liters the only checkpointer, which is + * the intent — but liters' own thresholds + * (`WriterOptions::min_checkpoint_page_n`, `truncate_page_n`) live inside + * `push()`. If pushes stop, nothing checkpoints and the WAL grows without + * bound, so an app that disables autocheckpoint should also keep its own + * size floor on the `-wal` file. + * + * Performs no network I/O: opening succeeds offline, and pushes + * accumulate local L0 files until the bucket becomes reachable (the + * first successful push then uploads the backlog). + */ +public convenience init(dbPath: String, storage: Storage)throws { + let handle = + try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_constructor_literswriter_new( + FfiConverterString.lower(dbPath), + FfiConverterTypeStorage_lower(storage),uniffiCallStatus + ) +} + self.init(unsafeFromHandle: handle) +} + + deinit { + if handle == 0 { + // Mock objects have handle=0 don't try to free them + return + } + + try! rustCall { uniffi_liters_ffi_fn_free_literswriter(handle, $0) } + } + + + /** + * [`LitersWriter::new`], with the app's own HTTP client executing every + * `Storage::Http` request. + * + * Without this, a `Storage::Http` writer runs on the built-in socket + * transport: `http://` only (an `https://` url is rejected, not silently + * downgraded), one `Connection: close` request per call, and — the part + * that matters on iOS — no way to hand the transfer to a background + * `URLSession`, so an upload in flight when the app suspends is lost. + * Pass the same shared `HttpClient` instance you would give + * `LitersManager`, so writers and followers to one authority coalesce + * onto a single connection. + * + * `null` is exactly [`LitersWriter::new`]. + */ +public static func newWithHttpClient(dbPath: String, storage: Storage, httpClient: HttpClient?)throws -> LitersWriter { + return try FfiConverterTypeLitersWriter_lift(try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_constructor_literswriter_new_with_http_client( + FfiConverterString.lower(dbPath), + FfiConverterTypeStorage_lower(storage), + FfiConverterOptionTypeHttpClient.lower(httpClient),uniffiCallStatus + ) +}) +} + + + + /** + * Aborts the in-flight `push`/`maintain`/`snapshot`, if any: it returns + * `Cancelled` promptly (mid-transfer on the HTTP backend). Each call + * installs a fresh token at its start, so cancel only ever affects the + * operation currently running — never a future call. Cancel-then-retry + * is indistinguishable from kill-then-restart. + */ +open func cancel() {try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_literswriter_cancel( + self.uniffiCloneHandle(),uniffiCallStatus + ) +} +} + + /** + * Tears the writer down: cancels any in-flight operation, waits for it + * to return, then releases the WAL read lock, SQLite connections, and + * file descriptors. Subsequent calls error with "LitersWriter is + * closed" — including calls already queued behind the cancelled one, + * which bail instead of running (so this never blocks for more than + * the single in-flight operation's cancellation latency). Idempotent. + */ +open func close() {try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_literswriter_close( + self.uniffiCloneHandle(),uniffiCallStatus + ) +} +} + + /** + * Runs due compaction, snapshotting, and retention with litestream's + * default cadences. + */ +open func maintain()throws -> MaintenanceSummary { + return try FfiConverterTypeMaintenanceSummary_lift(try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_literswriter_maintain( + self.uniffiCloneHandle(),uniffiCallStatus + ) +}) +} + + /** + * Current local replication position. + */ +open func position()throws -> UInt64 { + return try FfiConverterUInt64.lift(try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_literswriter_position( + self.uniffiCloneHandle(),uniffiCallStatus + ) +}) +} + + /** + * Captures all committed changes into the bucket. Short and resumable: + * on failure, the next push picks up exactly where this one stopped. + */ +open func push()throws -> PushSummary { + return try FfiConverterTypePushSummary_lift(try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_literswriter_push( + self.uniffiCloneHandle(),uniffiCallStatus + ) +}) +} + + /** + * Recovery for a "local ltx file missing or corrupt" error: wipes local + * replication state so the next push re-derives everything from the + * bucket (snapshotting if needed). Local database content is untouched. + */ +open func resetLocal()throws {try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_literswriter_reset_local( + self.uniffiCloneHandle(),uniffiCallStatus + ) +} +} + + /** + * Forces a full snapshot to the bucket now. + */ +open func snapshot()throws -> UInt64? { + return try FfiConverterOptionUInt64.lift(try rustCallWithError(FfiConverterTypeLitersError_lift) { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_literswriter_snapshot( + self.uniffiCloneHandle(),uniffiCallStatus + ) +}) +} + + + +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeLitersWriter: FfiConverter { + typealias FfiType = UInt64 + typealias SwiftType = LitersWriter + + public static func lift(_ handle: UInt64) throws -> LitersWriter { + return LitersWriter(unsafeFromHandle: handle) + } + + public static func lower(_ value: LitersWriter) -> UInt64 { + return value.uniffiCloneHandle() + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> LitersWriter { + let handle: UInt64 = try readInt(&buf) + return try lift(handle) + } + + public static func write(_ value: LitersWriter, into buf: inout [UInt8]) { + writeInt(&buf, lower(value)) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeLitersWriter_lift(_ handle: UInt64) throws -> LitersWriter { + return try FfiConverterTypeLitersWriter.lift(handle) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeLitersWriter_lower(_ value: LitersWriter) -> UInt64 { + return FfiConverterTypeLitersWriter.lower(value) +} + + + + + + +/** + * Receives manager events. Callbacks are delivered on liters worker + * threads (and, for the `Sleeping` state change, on the thread that called + * `sleep()`): they must return quickly, must never block, and must not + * call back into `unregister()`/`shutdown()` — hand the event to your own + * queue/handler instead. + */ +public protocol ManagerListener: AnyObject, Sendable { + + func onEvent(event: ManagerEvent) + +} +/** + * Receives manager events. Callbacks are delivered on liters worker + * threads (and, for the `Sleeping` state change, on the thread that called + * `sleep()`): they must return quickly, must never block, and must not + * call back into `unregister()`/`shutdown()` — hand the event to your own + * queue/handler instead. + */ +open class ManagerListenerImpl: ManagerListener, @unchecked Sendable { + fileprivate let handle: UInt64 + + /// Used to instantiate a [FFIObject] without an actual handle, for fakes in tests, mostly. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public struct NoHandle { + public init() {} + } + + // TODO: We'd like this to be `private` but for Swifty reasons, + // we can't implement `FfiConverter` without making this `required` and we can't + // make it `required` without making it `public`. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + required public init(unsafeFromHandle handle: UInt64) { + self.handle = handle + } + + // This constructor can be used to instantiate a fake object. + // - Parameter noHandle: Placeholder value so we can have a constructor separate from the default empty one that may be implemented for classes extending [FFIObject]. + // + // - Warning: + // Any object instantiated with this constructor cannot be passed to an actual Rust-backed object. Since there isn't a backing handle the FFI lower functions will crash. +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public init(noHandle: NoHandle) { + self.handle = 0 + } + +#if swift(>=5.8) + @_documentation(visibility: private) +#endif + public func uniffiCloneHandle() -> UInt64 { + return try! rustCall { uniffi_liters_ffi_fn_clone_managerlistener(self.handle, $0) } + } + // No primary constructor declared for this class. + + deinit { + if handle == 0 { + // Mock objects have handle=0 don't try to free them + return + } + + try! rustCall { uniffi_liters_ffi_fn_free_managerlistener(handle, $0) } + } + + + + +open func onEvent(event: ManagerEvent) {try! rustCall() { + uniffiCallStatus in + uniffi_liters_ffi_fn_method_managerlistener_on_event( + self.uniffiCloneHandle(), + FfiConverterTypeManagerEvent_lower(event),uniffiCallStatus + ) +} +} + + + +} + + + +// Put the implementation in a struct so we don't pollute the top-level namespace +fileprivate struct UniffiCallbackInterfaceManagerListener { + + // Create the VTable using a series of closures. + // Swift automatically converts these into C callback functions. + // + // Store the vtable directly. + static let vtable: UniffiVTableCallbackInterfaceManagerListener = UniffiVTableCallbackInterfaceManagerListener( + uniffiFree: { (uniffiHandle: UInt64) -> () in + do { + try FfiConverterTypeManagerListener.handleMap.remove(handle: uniffiHandle) + } catch { + print("Uniffi callback interface ManagerListener: handle missing in uniffiFree") + } + }, + uniffiClone: { (uniffiHandle: UInt64) -> UInt64 in + do { + return try FfiConverterTypeManagerListener.handleMap.clone(handle: uniffiHandle) + } catch { + fatalError("Uniffi callback interface ManagerListener: handle missing in uniffiClone") + } + }, + onEvent: { ( + uniffiHandle: UInt64, + event: RustBuffer, + uniffiOutReturn: UnsafeMutableRawPointer, + uniffiCallStatus: UnsafeMutablePointer + ) in + let makeCall = { + () throws -> () in + guard let uniffiObj = try? FfiConverterTypeManagerListener.handleMap.get(handle: uniffiHandle) else { + throw UniffiInternalError.unexpectedStaleHandle + } + return uniffiObj.onEvent( + event: try FfiConverterTypeManagerEvent_lift(event) + ) + } + + + let writeReturn = { () } + uniffiTraitInterfaceCall( + callStatus: uniffiCallStatus, + makeCall: makeCall, + writeReturn: writeReturn + ) + } + ) + + // Rust stores this pointer for future callback invocations, so it must live + // for the process lifetime (not just for the init function call). + // + // `nonisolated(unsafe)` is needed under Swift 6 strict concurrency. + // This is safe because the pointee is initialized once during static init + // and never mutated by either side of the FFI. Its fields are C function pointers. + nonisolated(unsafe) static let vtablePtr: UnsafePointer = { + let ptr = UnsafeMutablePointer.allocate(capacity: 1) + ptr.initialize(to: vtable) + return UnsafePointer(ptr) + }() +} + +private func uniffiCallbackInitManagerListener() { + uniffi_liters_ffi_fn_init_callback_vtable_managerlistener(UniffiCallbackInterfaceManagerListener.vtablePtr) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeManagerListener: FfiConverter { + fileprivate static let handleMap = UniffiHandleMap() + + typealias FfiType = UInt64 + typealias SwiftType = ManagerListener + + public static func lift(_ handle: UInt64) throws -> ManagerListener { + if ((handle & 1) == 0) { + // Rust-generated handle, construct a new class that uses the handle to implement the + // interface + return ManagerListenerImpl(unsafeFromHandle: handle) + } else { + // Swift-generated handle, get the object from the handle map + return try handleMap.remove(handle: handle) + } + } + + public static func lower(_ value: ManagerListener) -> UInt64 { + if let rustImpl = value as? ManagerListenerImpl { + // Rust-implemented object. Clone the handle and return it + return rustImpl.uniffiCloneHandle() + } else { + // Swift object, generate a new vtable handle and return that. + return handleMap.insert(obj: value) + } + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> ManagerListener { + let handle: UInt64 = try readInt(&buf) + return try lift(handle) + } + + public static func write(_ value: ManagerListener, into buf: inout [UInt8]) { + writeInt(&buf, lower(value)) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeManagerListener_lift(_ handle: UInt64) throws -> ManagerListener { + return try FfiConverterTypeManagerListener.lift(handle) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeManagerListener_lower(_ value: ManagerListener) -> UInt64 { + return FfiConverterTypeManagerListener.lower(value) +} + + + + +/** + * Retry schedule for transient failures: exponential growth from + * `initial_ms` by `multiplier` per consecutive failure, saturating at + * `max_ms`, with ±`jitter` randomization. The defaults are the liters + * defaults (500ms → 60s, ×2, ±25%). + */ +public struct Backoff: Equatable, Hashable { + /** + * Delay before the first retry, in milliseconds. + */ + public var initialMs: UInt64 + /** + * Ceiling (pre-jitter) the schedule saturates at, in milliseconds. + */ + public var maxMs: UInt64 + /** + * Growth factor per consecutive failure. + */ + public var multiplier: Double + /** + * Jitter fraction in [0, 1]; 0 disables jitter. + */ + public var jitter: Double + + // Default memberwise initializers are never public by default, so we + // declare one manually. + public init( + /** + * Delay before the first retry, in milliseconds. + */initialMs: UInt64 = UInt64(500), + /** + * Ceiling (pre-jitter) the schedule saturates at, in milliseconds. + */maxMs: UInt64 = UInt64(60000), + /** + * Growth factor per consecutive failure. + */multiplier: Double = Double(2.0), + /** + * Jitter fraction in [0, 1]; 0 disables jitter. + */jitter: Double = Double(0.25)) { + self.initialMs = initialMs + self.maxMs = maxMs + self.multiplier = multiplier + self.jitter = jitter + } + + + + +} + +#if compiler(>=6) +extension Backoff: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeBackoff: FfiConverterRustBuffer { + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> Backoff { + return + try Backoff( + initialMs: FfiConverterUInt64.read(from: &buf), + maxMs: FfiConverterUInt64.read(from: &buf), + multiplier: FfiConverterDouble.read(from: &buf), + jitter: FfiConverterDouble.read(from: &buf) + ) + } + + public static func write(_ value: Backoff, into buf: inout [UInt8]) { + FfiConverterUInt64.write(value.initialMs, into: &buf) + FfiConverterUInt64.write(value.maxMs, into: &buf) + FfiConverterDouble.write(value.multiplier, into: &buf) + FfiConverterDouble.write(value.jitter, into: &buf) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeBackoff_lift(_ buf: RustBuffer) throws -> Backoff { + return try FfiConverterTypeBackoff.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeBackoff_lower(_ value: Backoff) -> RustBuffer { + return FfiConverterTypeBackoff.lower(value) +} + + +/** + * Point-in-time snapshot of one registered database. + */ +public struct DbStatus: Equatable, Hashable { + public var id: String + public var role: DbRole + public var state: DbState + /** + * Replication position (TXID): for pushes, the last successful push; + * for follows, the last applied transaction. `null` until the first + * push/apply. + */ + public var position: UInt64? + /** + * Most recent error message; cleared by the next successful round. + */ + public var lastError: String? + /** + * Completion time of the last successful push/apply, in milliseconds + * since the Unix epoch. + */ + public var lastActivityMs: UInt64? + + // Default memberwise initializers are never public by default, so we + // declare one manually. + public init(id: String, role: DbRole, state: DbState, + /** + * Replication position (TXID): for pushes, the last successful push; + * for follows, the last applied transaction. `null` until the first + * push/apply. + */position: UInt64?, + /** + * Most recent error message; cleared by the next successful round. + */lastError: String?, + /** + * Completion time of the last successful push/apply, in milliseconds + * since the Unix epoch. + */lastActivityMs: UInt64?) { + self.id = id + self.role = role + self.state = state + self.position = position + self.lastError = lastError + self.lastActivityMs = lastActivityMs + } + + + + +} + +#if compiler(>=6) +extension DbStatus: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeDbStatus: FfiConverterRustBuffer { + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> DbStatus { + return + try DbStatus( + id: FfiConverterString.read(from: &buf), + role: FfiConverterTypeDbRole.read(from: &buf), + state: FfiConverterTypeDbState.read(from: &buf), + position: FfiConverterOptionUInt64.read(from: &buf), + lastError: FfiConverterOptionString.read(from: &buf), + lastActivityMs: FfiConverterOptionUInt64.read(from: &buf) + ) + } + + public static func write(_ value: DbStatus, into buf: inout [UInt8]) { + FfiConverterString.write(value.id, into: &buf) + FfiConverterTypeDbRole.write(value.role, into: &buf) + FfiConverterTypeDbState.write(value.state, into: &buf) + FfiConverterOptionUInt64.write(value.position, into: &buf) + FfiConverterOptionString.write(value.lastError, into: &buf) + FfiConverterOptionUInt64.write(value.lastActivityMs, into: &buf) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeDbStatus_lift(_ buf: RustBuffer) throws -> DbStatus { + return try FfiConverterTypeDbStatus.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeDbStatus_lower(_ value: DbStatus) -> RustBuffer { + return FfiConverterTypeDbStatus.lower(value) +} + + +/** + * Per-database options for `LitersManager.register_follow`. + */ +public struct FollowOptions: Equatable, Hashable { + /** + * On bucket divergence, silently delete the local replica and + * re-restore instead of parking in the `Failed` state. + */ + public var autoReset: Bool + /** + * Poll cadence in milliseconds for waiting out an empty bucket and for + * backends without live streaming. `null` = 1000. + */ + public var pollIntervalMs: UInt64? + /** + * Backoff for transient reconnects inside the follow loop; `null` uses + * the manager's backoff. + */ + public var retry: Backoff? + + // Default memberwise initializers are never public by default, so we + // declare one manually. + public init( + /** + * On bucket divergence, silently delete the local replica and + * re-restore instead of parking in the `Failed` state. + */autoReset: Bool = false, + /** + * Poll cadence in milliseconds for waiting out an empty bucket and for + * backends without live streaming. `null` = 1000. + */pollIntervalMs: UInt64? = nil, + /** + * Backoff for transient reconnects inside the follow loop; `null` uses + * the manager's backoff. + */retry: Backoff? = nil) { + self.autoReset = autoReset + self.pollIntervalMs = pollIntervalMs + self.retry = retry + } + + + + +} + +#if compiler(>=6) +extension FollowOptions: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeFollowOptions: FfiConverterRustBuffer { + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> FollowOptions { + return + try FollowOptions( + autoReset: FfiConverterBool.read(from: &buf), + pollIntervalMs: FfiConverterOptionUInt64.read(from: &buf), + retry: FfiConverterOptionTypeBackoff.read(from: &buf) + ) + } + + public static func write(_ value: FollowOptions, into buf: inout [UInt8]) { + FfiConverterBool.write(value.autoReset, into: &buf) + FfiConverterOptionUInt64.write(value.pollIntervalMs, into: &buf) + FfiConverterOptionTypeBackoff.write(value.retry, into: &buf) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeFollowOptions_lift(_ buf: RustBuffer) throws -> FollowOptions { + return try FfiConverterTypeFollowOptions.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeFollowOptions_lower(_ value: FollowOptions) -> RustBuffer { + return FfiConverterTypeFollowOptions.lower(value) +} + + +/** + * One HTTP header, carried across the FFI in both directions. + */ +public struct HttpHeader: Equatable, Hashable { + public var name: String + public var value: String + + // Default memberwise initializers are never public by default, so we + // declare one manually. + public init(name: String, value: String) { + self.name = name + self.value = value + } + + + + +} + +#if compiler(>=6) +extension HttpHeader: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeHttpHeader: FfiConverterRustBuffer { + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> HttpHeader { + return + try HttpHeader( + name: FfiConverterString.read(from: &buf), + value: FfiConverterString.read(from: &buf) + ) + } + + public static func write(_ value: HttpHeader, into buf: inout [UInt8]) { + FfiConverterString.write(value.name, into: &buf) + FfiConverterString.write(value.value, into: &buf) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeHttpHeader_lift(_ buf: RustBuffer) throws -> HttpHeader { + return try FfiConverterTypeHttpHeader.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeHttpHeader_lower(_ value: HttpHeader) -> RustBuffer { + return FfiConverterTypeHttpHeader.lower(value) +} + + +/** + * A request for the host `HttpClient` to execute. `headers` are semantic only + * (`authorization`, the fencing headers); the host adds its own transport + * headers (`host`, framing, connection). `url` is absolute and may be + * `https`. + */ +public struct HttpRequest: Equatable, Hashable { + public var method: String + public var url: String + public var headers: [HttpHeader] + /** + * `true` only for the long-lived `/stream` follow: the host keeps the + * response open and returns `BodyChunk.Idle` on its read-timeout tick + * (roughly once a second) instead of failing, so liters can run its + * dead-man timer and stay cancel-responsive. For `false` the host uses + * its normal read timeout and *fails* (throws) a stalled read. + */ + public var longLived: Bool + + // Default memberwise initializers are never public by default, so we + // declare one manually. + public init(method: String, url: String, headers: [HttpHeader], + /** + * `true` only for the long-lived `/stream` follow: the host keeps the + * response open and returns `BodyChunk.Idle` on its read-timeout tick + * (roughly once a second) instead of failing, so liters can run its + * dead-man timer and stay cancel-responsive. For `false` the host uses + * its normal read timeout and *fails* (throws) a stalled read. + */longLived: Bool) { + self.method = method + self.url = url + self.headers = headers + self.longLived = longLived + } + + + + +} + +#if compiler(>=6) +extension HttpRequest: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeHttpRequest: FfiConverterRustBuffer { + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> HttpRequest { + return + try HttpRequest( + method: FfiConverterString.read(from: &buf), + url: FfiConverterString.read(from: &buf), + headers: FfiConverterSequenceTypeHttpHeader.read(from: &buf), + longLived: FfiConverterBool.read(from: &buf) + ) + } + + public static func write(_ value: HttpRequest, into buf: inout [UInt8]) { + FfiConverterString.write(value.method, into: &buf) + FfiConverterString.write(value.url, into: &buf) + FfiConverterSequenceTypeHttpHeader.write(value.headers, into: &buf) + FfiConverterBool.write(value.longLived, into: &buf) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeHttpRequest_lift(_ buf: RustBuffer) throws -> HttpRequest { + return try FfiConverterTypeHttpRequest.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeHttpRequest_lower(_ value: HttpRequest) -> RustBuffer { + return FfiConverterTypeHttpRequest.lower(value) +} + + +public struct MaintenanceSummary: Equatable, Hashable { + public var compactedLevels: Data + public var snapshotTxid: UInt64? + public var deletedFiles: UInt64 + + // Default memberwise initializers are never public by default, so we + // declare one manually. + public init(compactedLevels: Data, snapshotTxid: UInt64?, deletedFiles: UInt64) { + self.compactedLevels = compactedLevels + self.snapshotTxid = snapshotTxid + self.deletedFiles = deletedFiles + } + + + + +} + +#if compiler(>=6) +extension MaintenanceSummary: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeMaintenanceSummary: FfiConverterRustBuffer { + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> MaintenanceSummary { + return + try MaintenanceSummary( + compactedLevels: FfiConverterData.read(from: &buf), + snapshotTxid: FfiConverterOptionUInt64.read(from: &buf), + deletedFiles: FfiConverterUInt64.read(from: &buf) + ) + } + + public static func write(_ value: MaintenanceSummary, into buf: inout [UInt8]) { + FfiConverterData.write(value.compactedLevels, into: &buf) + FfiConverterOptionUInt64.write(value.snapshotTxid, into: &buf) + FfiConverterUInt64.write(value.deletedFiles, into: &buf) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeMaintenanceSummary_lift(_ buf: RustBuffer) throws -> MaintenanceSummary { + return try FfiConverterTypeMaintenanceSummary.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeMaintenanceSummary_lower(_ value: MaintenanceSummary) -> RustBuffer { + return FfiConverterTypeMaintenanceSummary.lower(value) +} + + +/** + * Manager-wide defaults (see `LitersManager.with_options`). + */ +public struct ManagerOptions: Equatable, Hashable { + /** + * Backoff for transient failures; overridable per push registration + * and used as the follow-loop retry schedule. `null` = liters + * defaults. + */ + public var backoff: Backoff? + /** + * Fallback push cadence (milliseconds) for registrations whose own + * `push_interval_ms` is `null`. When both are `null`, those databases + * push only on `push_now()`. + */ + public var defaultPushIntervalMs: UInt64? + + // Default memberwise initializers are never public by default, so we + // declare one manually. + public init( + /** + * Backoff for transient failures; overridable per push registration + * and used as the follow-loop retry schedule. `null` = liters + * defaults. + */backoff: Backoff? = nil, + /** + * Fallback push cadence (milliseconds) for registrations whose own + * `push_interval_ms` is `null`. When both are `null`, those databases + * push only on `push_now()`. + */defaultPushIntervalMs: UInt64? = nil) { + self.backoff = backoff + self.defaultPushIntervalMs = defaultPushIntervalMs + } + + + + +} + +#if compiler(>=6) +extension ManagerOptions: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeManagerOptions: FfiConverterRustBuffer { + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> ManagerOptions { + return + try ManagerOptions( + backoff: FfiConverterOptionTypeBackoff.read(from: &buf), + defaultPushIntervalMs: FfiConverterOptionUInt64.read(from: &buf) + ) + } + + public static func write(_ value: ManagerOptions, into buf: inout [UInt8]) { + FfiConverterOptionTypeBackoff.write(value.backoff, into: &buf) + FfiConverterOptionUInt64.write(value.defaultPushIntervalMs, into: &buf) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeManagerOptions_lift(_ buf: RustBuffer) throws -> ManagerOptions { + return try FfiConverterTypeManagerOptions.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeManagerOptions_lower(_ value: ManagerOptions) -> RustBuffer { + return FfiConverterTypeManagerOptions.lower(value) +} + + +/** + * Per-database options for `LitersManager.register_push`. + */ +public struct PushOptions: Equatable, Hashable { + /** + * Push cadence in milliseconds. `null` falls back to the manager's + * `default_push_interval_ms`; if that is also `null`, this database + * pushes only on `push_now()`. With an interval set, the first push + * runs immediately after registration. + */ + public var pushIntervalMs: UInt64? + /** + * When set, maintenance (compaction/snapshot/retention with + * litestream's default cadences) is attempted after a successful push + * once this many milliseconds have passed since the last attempt. A + * maintenance failure is reported but never blocks future pushes. + */ + public var maintenanceIntervalMs: UInt64? + /** + * Per-database transient-failure backoff; `null` uses the manager's. + */ + public var backoff: Backoff? + + // Default memberwise initializers are never public by default, so we + // declare one manually. + public init( + /** + * Push cadence in milliseconds. `null` falls back to the manager's + * `default_push_interval_ms`; if that is also `null`, this database + * pushes only on `push_now()`. With an interval set, the first push + * runs immediately after registration. + */pushIntervalMs: UInt64? = nil, + /** + * When set, maintenance (compaction/snapshot/retention with + * litestream's default cadences) is attempted after a successful push + * once this many milliseconds have passed since the last attempt. A + * maintenance failure is reported but never blocks future pushes. + */maintenanceIntervalMs: UInt64? = nil, + /** + * Per-database transient-failure backoff; `null` uses the manager's. + */backoff: Backoff? = nil) { + self.pushIntervalMs = pushIntervalMs + self.maintenanceIntervalMs = maintenanceIntervalMs + self.backoff = backoff + } + + + + +} + +#if compiler(>=6) +extension PushOptions: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypePushOptions: FfiConverterRustBuffer { + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> PushOptions { + return + try PushOptions( + pushIntervalMs: FfiConverterOptionUInt64.read(from: &buf), + maintenanceIntervalMs: FfiConverterOptionUInt64.read(from: &buf), + backoff: FfiConverterOptionTypeBackoff.read(from: &buf) + ) + } + + public static func write(_ value: PushOptions, into buf: inout [UInt8]) { + FfiConverterOptionUInt64.write(value.pushIntervalMs, into: &buf) + FfiConverterOptionUInt64.write(value.maintenanceIntervalMs, into: &buf) + FfiConverterOptionTypeBackoff.write(value.backoff, into: &buf) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypePushOptions_lift(_ buf: RustBuffer) throws -> PushOptions { + return try FfiConverterTypePushOptions.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypePushOptions_lower(_ value: PushOptions) -> RustBuffer { + return FfiConverterTypePushOptions.lower(value) +} + + +public struct PushSummary: Equatable, Hashable { + /** + * Local replication position (TXID) after the push. + */ + public var txid: UInt64 + /** + * Whether new committed content was captured into an L0 file. + */ + public var synced: Bool + /** + * Number of files uploaded by this push. + */ + public var uploaded: UInt64 + /** + * Bucket-side max TXID after the push. + */ + public var remoteTxid: UInt64 + /** + * Whether a WAL checkpoint ran. + */ + public var checkpointed: Bool + /** + * Whether this push wrote a **full snapshot of the database** instead of + * an incremental WAL delta. + * + * The one field worth logging on every push. A snapshot ships the whole + * database, so a device that snapshots on most pushes is doing full + * uploads with extra steps — and on iOS that is a live possibility on + * every relaunch, because the `synced_to_wal_end` shortcut is + * deliberately not persisted across a writer close/reopen. + */ + public var snapshotted: Bool + /** + * Why, when `snapshotted`. One of a small fixed set from liters' verify + * decision tree, so it is safe to aggregate on. + */ + public var snapshotReason: String? + /** + * Bytes of LTX uploaded by this push, across `uploaded` files. + */ + public var bytesUploaded: UInt64 + + // Default memberwise initializers are never public by default, so we + // declare one manually. + public init( + /** + * Local replication position (TXID) after the push. + */txid: UInt64, + /** + * Whether new committed content was captured into an L0 file. + */synced: Bool, + /** + * Number of files uploaded by this push. + */uploaded: UInt64, + /** + * Bucket-side max TXID after the push. + */remoteTxid: UInt64, + /** + * Whether a WAL checkpoint ran. + */checkpointed: Bool, + /** + * Whether this push wrote a **full snapshot of the database** instead of + * an incremental WAL delta. + * + * The one field worth logging on every push. A snapshot ships the whole + * database, so a device that snapshots on most pushes is doing full + * uploads with extra steps — and on iOS that is a live possibility on + * every relaunch, because the `synced_to_wal_end` shortcut is + * deliberately not persisted across a writer close/reopen. + */snapshotted: Bool, + /** + * Why, when `snapshotted`. One of a small fixed set from liters' verify + * decision tree, so it is safe to aggregate on. + */snapshotReason: String?, + /** + * Bytes of LTX uploaded by this push, across `uploaded` files. + */bytesUploaded: UInt64) { + self.txid = txid + self.synced = synced + self.uploaded = uploaded + self.remoteTxid = remoteTxid + self.checkpointed = checkpointed + self.snapshotted = snapshotted + self.snapshotReason = snapshotReason + self.bytesUploaded = bytesUploaded + } + + + + +} + +#if compiler(>=6) +extension PushSummary: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypePushSummary: FfiConverterRustBuffer { + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> PushSummary { + return + try PushSummary( + txid: FfiConverterUInt64.read(from: &buf), + synced: FfiConverterBool.read(from: &buf), + uploaded: FfiConverterUInt64.read(from: &buf), + remoteTxid: FfiConverterUInt64.read(from: &buf), + checkpointed: FfiConverterBool.read(from: &buf), + snapshotted: FfiConverterBool.read(from: &buf), + snapshotReason: FfiConverterOptionString.read(from: &buf), + bytesUploaded: FfiConverterUInt64.read(from: &buf) + ) + } + + public static func write(_ value: PushSummary, into buf: inout [UInt8]) { + FfiConverterUInt64.write(value.txid, into: &buf) + FfiConverterBool.write(value.synced, into: &buf) + FfiConverterUInt64.write(value.uploaded, into: &buf) + FfiConverterUInt64.write(value.remoteTxid, into: &buf) + FfiConverterBool.write(value.checkpointed, into: &buf) + FfiConverterBool.write(value.snapshotted, into: &buf) + FfiConverterOptionString.write(value.snapshotReason, into: &buf) + FfiConverterUInt64.write(value.bytesUploaded, into: &buf) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypePushSummary_lift(_ buf: RustBuffer) throws -> PushSummary { + return try FfiConverterTypePushSummary.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypePushSummary_lower(_ value: PushSummary) -> RustBuffer { + return FfiConverterTypePushSummary.lower(value) +} + + +public struct SyncSummary: Equatable, Hashable { + /** + * Whether a full restore ran (vs. incremental application). + */ + public var restored: Bool + public var fromTxid: UInt64 + public var toTxid: UInt64 + + // Default memberwise initializers are never public by default, so we + // declare one manually. + public init( + /** + * Whether a full restore ran (vs. incremental application). + */restored: Bool, fromTxid: UInt64, toTxid: UInt64) { + self.restored = restored + self.fromTxid = fromTxid + self.toTxid = toTxid + } + + + + +} + +#if compiler(>=6) +extension SyncSummary: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeSyncSummary: FfiConverterRustBuffer { + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> SyncSummary { + return + try SyncSummary( + restored: FfiConverterBool.read(from: &buf), + fromTxid: FfiConverterUInt64.read(from: &buf), + toTxid: FfiConverterUInt64.read(from: &buf) + ) + } + + public static func write(_ value: SyncSummary, into buf: inout [UInt8]) { + FfiConverterBool.write(value.restored, into: &buf) + FfiConverterUInt64.write(value.fromTxid, into: &buf) + FfiConverterUInt64.write(value.toTxid, into: &buf) + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeSyncSummary_lift(_ buf: RustBuffer) throws -> SyncSummary { + return try FfiConverterTypeSyncSummary.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeSyncSummary_lower(_ value: SyncSummary) -> RustBuffer { + return FfiConverterTypeSyncSummary.lower(value) +} + + +/** + * One read from a response body. + */ + +public enum BodyChunk: Equatable, Hashable { + + /** + * Some decoded body bytes (must be non-empty). + */ + case data(bytes: Data + ) + /** + * No bytes right now, connection alive — an idle tick on a `long_lived` + * body. Only valid for `long_lived` requests. + */ + case idle + /** + * The body is complete. + */ + case eof + + + + + +} + +#if compiler(>=6) +extension BodyChunk: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeBodyChunk: FfiConverterRustBuffer { + typealias SwiftType = BodyChunk + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> BodyChunk { + let variant: Int32 = try readInt(&buf) + switch variant { + + case 1: return .data(bytes: try FfiConverterData.read(from: &buf) + ) + + case 2: return .idle + + case 3: return .eof + + default: throw UniffiInternalError.unexpectedEnumCase + } + } + + public static func write(_ value: BodyChunk, into buf: inout [UInt8]) { + switch value { + + + case let .data(bytes): + writeInt(&buf, Int32(1)) + FfiConverterData.write(bytes, into: &buf) + + + case .idle: + writeInt(&buf, Int32(2)) + + + case .eof: + writeInt(&buf, Int32(3)) + + } + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeBodyChunk_lift(_ buf: RustBuffer) throws -> BodyChunk { + return try FfiConverterTypeBodyChunk.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeBodyChunk_lower(_ value: BodyChunk) -> RustBuffer { + return FfiConverterTypeBodyChunk.lower(value) +} + + + +/** + * Whether a registration pushes or follows. + */ + +public enum DbRole: Equatable, Hashable { + + case push + case follow + + + + + +} + +#if compiler(>=6) +extension DbRole: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeDbRole: FfiConverterRustBuffer { + typealias SwiftType = DbRole + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> DbRole { + let variant: Int32 = try readInt(&buf) + switch variant { + + case 1: return .push + + case 2: return .follow + + default: throw UniffiInternalError.unexpectedEnumCase + } + } + + public static func write(_ value: DbRole, into buf: inout [UInt8]) { + switch value { + + + case .push: + writeInt(&buf, Int32(1)) + + + case .follow: + writeInt(&buf, Int32(2)) + + } + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeDbRole_lift(_ buf: RustBuffer) throws -> DbRole { + return try FfiConverterTypeDbRole.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeDbRole_lower(_ value: DbRole) -> RustBuffer { + return FfiConverterTypeDbRole.lower(value) +} + + + +/** + * Lifecycle state of one registered database. + */ + +public enum DbState: Equatable, Hashable { + + /** + * Registered and waiting for the next tick/nudge (push entries only; + * an active follower is `Working` for its whole session). + */ + case idle + /** + * A push/maintain round or follow session is in progress. + */ + case working + /** + * A transient failure is waiting out its backoff delay. `until_ms` is + * milliseconds since the Unix epoch; `attempt` is the + * consecutive-failure count (1 after the first failure). + */ + case backingOff(untilMs: UInt64, attempt: UInt32 + ) + /** + * Slept via `sleep()`: session cancelled, Writer dropped, no storage + * traffic until `resume()`. + */ + case sleeping + /** + * A fatal (non-transient) error parked the worker; it retries once per + * nudge (`push_now`/`sync_now`) or on `resume()`. + */ + case failed + + + + + +} + +#if compiler(>=6) +extension DbState: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeDbState: FfiConverterRustBuffer { + typealias SwiftType = DbState + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> DbState { + let variant: Int32 = try readInt(&buf) + switch variant { + + case 1: return .idle + + case 2: return .working + + case 3: return .backingOff(untilMs: try FfiConverterUInt64.read(from: &buf), attempt: try FfiConverterUInt32.read(from: &buf) + ) + + case 4: return .sleeping + + case 5: return .failed + + default: throw UniffiInternalError.unexpectedEnumCase + } + } + + public static func write(_ value: DbState, into buf: inout [UInt8]) { + switch value { + + + case .idle: + writeInt(&buf, Int32(1)) + + + case .working: + writeInt(&buf, Int32(2)) + + + case let .backingOff(untilMs,attempt): + writeInt(&buf, Int32(3)) + FfiConverterUInt64.write(untilMs, into: &buf) + FfiConverterUInt32.write(attempt, into: &buf) + + + case .sleeping: + writeInt(&buf, Int32(4)) + + + case .failed: + writeInt(&buf, Int32(5)) + + } + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeDbState_lift(_ buf: RustBuffer) throws -> DbState { + return try FfiConverterTypeDbState.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeDbState_lower(_ value: DbState) -> RustBuffer { + return FfiConverterTypeDbState.lower(value) +} + + + +/** + * Errors the host transport can report. Thrown from a callback method or + * returned as the error arm; anything the host throws that is not one of + * these is mapped to `Transport` (see the `From` impl below) rather than + * crashing the worker. + */ +public +enum HttpError: Swift.Error, Equatable, Hashable, Foundation.LocalizedError { + + + + /** + * A network/transport failure (connect, reset, timeout). Retryable — + * maps to liters' `Unavailable`. + */ + case Transport(message: String + ) + /** + * A protocol/format failure (bad framing). Maps to liters' `Other`. + */ + case Protocol(message: String + ) + + + + + + + public var errorDescription: String? { + String(reflecting: self) + } + +} + +#if compiler(>=6) +extension HttpError: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeHttpError: FfiConverterRustBuffer { + typealias SwiftType = HttpError + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> HttpError { + let variant: Int32 = try readInt(&buf) + switch variant { + + + + + case 1: return .Transport( + message: try FfiConverterString.read(from: &buf) + ) + case 2: return .Protocol( + message: try FfiConverterString.read(from: &buf) + ) + + default: throw UniffiInternalError.unexpectedEnumCase + } + } + + public static func write(_ value: HttpError, into buf: inout [UInt8]) { + switch value { + + + + + + case let .Transport(message): + writeInt(&buf, Int32(1)) + FfiConverterString.write(message, into: &buf) + + + case let .Protocol(message): + writeInt(&buf, Int32(2)) + FfiConverterString.write(message, into: &buf) + + } + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeHttpError_lift(_ buf: RustBuffer) throws -> HttpError { + return try FfiConverterTypeHttpError.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeHttpError_lower(_ value: HttpError) -> RustBuffer { + return FfiConverterTypeHttpError.lower(value) +} + + +public +enum LitersError: Swift.Error, Equatable, Hashable, Foundation.LocalizedError { + + + + /** + * The bucket's history no longer matches the local state; call + * `Replica.reset()` (or construct with `auto_reset`) to re-restore. + */ + case Diverged(message: String + ) + /** + * Storage/network failure (including local I/O); usually transient — + * retry the same call later. + */ + case Storage(message: String + ) + /** + * The server rejected a write for consistency/ownership reasons: + * another writer holds the bucket lease, or the push was + * non-monotonic. Not retryable as-is — the next push after the local + * lineage check re-derives the correct baseline. + */ + case Conflict(message: String + ) + /** + * Authentication required or rejected; fix the auth token. + */ + case Unauthorized(message: String + ) + /** + * The operation was aborted by `cancel()` (or `close()`). State is + * crash-safe: retrying is indistinguishable from resuming after a + * process kill. + */ + case Cancelled + case Other(message: String + ) + + + + + + + public var errorDescription: String? { + String(reflecting: self) + } + +} + +#if compiler(>=6) +extension LitersError: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeLitersError: FfiConverterRustBuffer { + typealias SwiftType = LitersError + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> LitersError { + let variant: Int32 = try readInt(&buf) + switch variant { + + + + + case 1: return .Diverged( + message: try FfiConverterString.read(from: &buf) + ) + case 2: return .Storage( + message: try FfiConverterString.read(from: &buf) + ) + case 3: return .Conflict( + message: try FfiConverterString.read(from: &buf) + ) + case 4: return .Unauthorized( + message: try FfiConverterString.read(from: &buf) + ) + case 5: return .Cancelled + case 6: return .Other( + message: try FfiConverterString.read(from: &buf) + ) + + default: throw UniffiInternalError.unexpectedEnumCase + } + } + + public static func write(_ value: LitersError, into buf: inout [UInt8]) { + switch value { + + + + + + case let .Diverged(message): + writeInt(&buf, Int32(1)) + FfiConverterString.write(message, into: &buf) + + + case let .Storage(message): + writeInt(&buf, Int32(2)) + FfiConverterString.write(message, into: &buf) + + + case let .Conflict(message): + writeInt(&buf, Int32(3)) + FfiConverterString.write(message, into: &buf) + + + case let .Unauthorized(message): + writeInt(&buf, Int32(4)) + FfiConverterString.write(message, into: &buf) + + + case .Cancelled: + writeInt(&buf, Int32(5)) + + + case let .Other(message): + writeInt(&buf, Int32(6)) + FfiConverterString.write(message, into: &buf) + + } + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeLitersError_lift(_ buf: RustBuffer) throws -> LitersError { + return try FfiConverterTypeLitersError.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeLitersError_lower(_ value: LitersError) -> RustBuffer { + return FfiConverterTypeLitersError.lower(value) +} + + +/** + * Events delivered to a `ManagerListener`. One database's events arrive in + * order; followers do not emit per-transaction position events (poll + * `status()` for live positions) — `SyncCompleted` is reserved for future + * use. + */ + +public enum ManagerEvent: Equatable, Hashable { + + case stateChanged(id: String, state: DbState + ) + case pushCompleted(id: String, result: PushSummary + ) + case syncCompleted(id: String, result: SyncSummary + ) + case error(id: String, message: String, transient: Bool + ) + + + + + +} + +#if compiler(>=6) +extension ManagerEvent: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeManagerEvent: FfiConverterRustBuffer { + typealias SwiftType = ManagerEvent + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> ManagerEvent { + let variant: Int32 = try readInt(&buf) + switch variant { + + case 1: return .stateChanged(id: try FfiConverterString.read(from: &buf), state: try FfiConverterTypeDbState.read(from: &buf) + ) + + case 2: return .pushCompleted(id: try FfiConverterString.read(from: &buf), result: try FfiConverterTypePushSummary.read(from: &buf) + ) + + case 3: return .syncCompleted(id: try FfiConverterString.read(from: &buf), result: try FfiConverterTypeSyncSummary.read(from: &buf) + ) + + case 4: return .error(id: try FfiConverterString.read(from: &buf), message: try FfiConverterString.read(from: &buf), transient: try FfiConverterBool.read(from: &buf) + ) + + default: throw UniffiInternalError.unexpectedEnumCase + } + } + + public static func write(_ value: ManagerEvent, into buf: inout [UInt8]) { + switch value { + + + case let .stateChanged(id,state): + writeInt(&buf, Int32(1)) + FfiConverterString.write(id, into: &buf) + FfiConverterTypeDbState.write(state, into: &buf) + + + case let .pushCompleted(id,result): + writeInt(&buf, Int32(2)) + FfiConverterString.write(id, into: &buf) + FfiConverterTypePushSummary.write(result, into: &buf) + + + case let .syncCompleted(id,result): + writeInt(&buf, Int32(3)) + FfiConverterString.write(id, into: &buf) + FfiConverterTypeSyncSummary.write(result, into: &buf) + + + case let .error(id,message,transient): + writeInt(&buf, Int32(4)) + FfiConverterString.write(id, into: &buf) + FfiConverterString.write(message, into: &buf) + FfiConverterBool.write(transient, into: &buf) + + } + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeManagerEvent_lift(_ buf: RustBuffer) throws -> ManagerEvent { + return try FfiConverterTypeManagerEvent.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeManagerEvent_lower(_ value: ManagerEvent) -> RustBuffer { + return FfiConverterTypeManagerEvent.lower(value) +} + + + +/** + * Where a database's replica lives. + */ + +public enum Storage: Equatable, Hashable { + + /** + * Local directory in litestream's `file` layout (testing, shared + * containers). + */ + case dir(path: String + ) + /** + * Another liters instance serving a bucket over HTTP + * (`http://host:port[/path]`; one database of a multi-DB server is + * addressed as `http://host:port/db/{name}`). Valid as a Replica + * source — `sync()` to poll, or follow live via + * `LitersManager.register_follow`. Valid as a Writer destination when + * the server runs with `writable: true` — that is push replication: + * this device dials out and pushes, the listening liters receives. + * Read-only servers reject the first push with a clear error. + * + * `auth_token` is sent as `Authorization: Bearer ` on every + * request, for servers started with an auth token. (Adding this field + * is a source-level break for Swift/Kotlin callers constructing + * `Storage.Http` positionally; pass `null`/`nil` to keep the old + * behavior.) + */ + case http(url: String, authToken: String? + ) + + + + + +} + +#if compiler(>=6) +extension Storage: Sendable {} +#endif + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public struct FfiConverterTypeStorage: FfiConverterRustBuffer { + typealias SwiftType = Storage + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> Storage { + let variant: Int32 = try readInt(&buf) + switch variant { + + case 1: return .dir(path: try FfiConverterString.read(from: &buf) + ) + + case 2: return .http(url: try FfiConverterString.read(from: &buf), authToken: try FfiConverterOptionString.read(from: &buf) + ) + + default: throw UniffiInternalError.unexpectedEnumCase + } + } + + public static func write(_ value: Storage, into buf: inout [UInt8]) { + switch value { + + + case let .dir(path): + writeInt(&buf, Int32(1)) + FfiConverterString.write(path, into: &buf) + + + case let .http(url,authToken): + writeInt(&buf, Int32(2)) + FfiConverterString.write(url, into: &buf) + FfiConverterOptionString.write(authToken, into: &buf) + + } + } +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeStorage_lift(_ buf: RustBuffer) throws -> Storage { + return try FfiConverterTypeStorage.lift(buf) +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +public func FfiConverterTypeStorage_lower(_ value: Storage) -> RustBuffer { + return FfiConverterTypeStorage.lower(value) +} + + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterOptionUInt64: FfiConverterRustBuffer { + typealias SwiftType = UInt64? + + public static func write(_ value: SwiftType, into buf: inout [UInt8]) { + guard let value = value else { + writeInt(&buf, Int8(0)) + return + } + writeInt(&buf, Int8(1)) + FfiConverterUInt64.write(value, into: &buf) + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> SwiftType { + switch try readInt(&buf) as Int8 { + case 0: return nil + case 1: return try FfiConverterUInt64.read(from: &buf) + default: throw UniffiInternalError.unexpectedOptionalTag + } + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterOptionString: FfiConverterRustBuffer { + typealias SwiftType = String? + + public static func write(_ value: SwiftType, into buf: inout [UInt8]) { + guard let value = value else { + writeInt(&buf, Int8(0)) + return + } + writeInt(&buf, Int8(1)) + FfiConverterString.write(value, into: &buf) + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> SwiftType { + switch try readInt(&buf) as Int8 { + case 0: return nil + case 1: return try FfiConverterString.read(from: &buf) + default: throw UniffiInternalError.unexpectedOptionalTag + } + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterOptionTypeHttpClient: FfiConverterRustBuffer { + typealias SwiftType = HttpClient? + + public static func write(_ value: SwiftType, into buf: inout [UInt8]) { + guard let value = value else { + writeInt(&buf, Int8(0)) + return + } + writeInt(&buf, Int8(1)) + FfiConverterTypeHttpClient.write(value, into: &buf) + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> SwiftType { + switch try readInt(&buf) as Int8 { + case 0: return nil + case 1: return try FfiConverterTypeHttpClient.read(from: &buf) + default: throw UniffiInternalError.unexpectedOptionalTag + } + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterOptionTypeHttpRequestBody: FfiConverterRustBuffer { + typealias SwiftType = HttpRequestBody? + + public static func write(_ value: SwiftType, into buf: inout [UInt8]) { + guard let value = value else { + writeInt(&buf, Int8(0)) + return + } + writeInt(&buf, Int8(1)) + FfiConverterTypeHttpRequestBody.write(value, into: &buf) + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> SwiftType { + switch try readInt(&buf) as Int8 { + case 0: return nil + case 1: return try FfiConverterTypeHttpRequestBody.read(from: &buf) + default: throw UniffiInternalError.unexpectedOptionalTag + } + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterOptionTypeManagerListener: FfiConverterRustBuffer { + typealias SwiftType = ManagerListener? + + public static func write(_ value: SwiftType, into buf: inout [UInt8]) { + guard let value = value else { + writeInt(&buf, Int8(0)) + return + } + writeInt(&buf, Int8(1)) + FfiConverterTypeManagerListener.write(value, into: &buf) + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> SwiftType { + switch try readInt(&buf) as Int8 { + case 0: return nil + case 1: return try FfiConverterTypeManagerListener.read(from: &buf) + default: throw UniffiInternalError.unexpectedOptionalTag + } + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterOptionTypeBackoff: FfiConverterRustBuffer { + typealias SwiftType = Backoff? + + public static func write(_ value: SwiftType, into buf: inout [UInt8]) { + guard let value = value else { + writeInt(&buf, Int8(0)) + return + } + writeInt(&buf, Int8(1)) + FfiConverterTypeBackoff.write(value, into: &buf) + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> SwiftType { + switch try readInt(&buf) as Int8 { + case 0: return nil + case 1: return try FfiConverterTypeBackoff.read(from: &buf) + default: throw UniffiInternalError.unexpectedOptionalTag + } + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterOptionTypeDbStatus: FfiConverterRustBuffer { + typealias SwiftType = DbStatus? + + public static func write(_ value: SwiftType, into buf: inout [UInt8]) { + guard let value = value else { + writeInt(&buf, Int8(0)) + return + } + writeInt(&buf, Int8(1)) + FfiConverterTypeDbStatus.write(value, into: &buf) + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> SwiftType { + switch try readInt(&buf) as Int8 { + case 0: return nil + case 1: return try FfiConverterTypeDbStatus.read(from: &buf) + default: throw UniffiInternalError.unexpectedOptionalTag + } + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterSequenceTypeDbStatus: FfiConverterRustBuffer { + typealias SwiftType = [DbStatus] + + public static func write(_ value: [DbStatus], into buf: inout [UInt8]) { + let len = Int32(value.count) + writeInt(&buf, len) + for item in value { + FfiConverterTypeDbStatus.write(item, into: &buf) + } + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> [DbStatus] { + let len: Int32 = try readInt(&buf) + var seq = [DbStatus]() + seq.reserveCapacity(Int(len)) + for _ in 0 ..< len { + seq.append(try FfiConverterTypeDbStatus.read(from: &buf)) + } + return seq + } +} + +#if swift(>=5.8) +@_documentation(visibility: private) +#endif +fileprivate struct FfiConverterSequenceTypeHttpHeader: FfiConverterRustBuffer { + typealias SwiftType = [HttpHeader] + + public static func write(_ value: [HttpHeader], into buf: inout [UInt8]) { + let len = Int32(value.count) + writeInt(&buf, len) + for item in value { + FfiConverterTypeHttpHeader.write(item, into: &buf) + } + } + + public static func read(from buf: inout (data: Data, offset: Data.Index)) throws -> [HttpHeader] { + let len: Int32 = try readInt(&buf) + var seq = [HttpHeader]() + seq.reserveCapacity(Int(len)) + for _ in 0 ..< len { + seq.append(try FfiConverterTypeHttpHeader.read(from: &buf)) + } + return seq + } +} + +private enum InitializationResult { + case ok + case contractVersionMismatch + case apiChecksumMismatch +} +// Use a global variable to perform the versioning checks. Swift ensures that +// the code inside is only computed once. +private let initializationResult: InitializationResult = { + // Get the bindings contract version from our ComponentInterface + let bindings_contract_version = 30 + // Get the scaffolding contract version by calling the into the dylib + let scaffolding_contract_version = ffi_liters_ffi_uniffi_contract_version() + if bindings_contract_version != scaffolding_contract_version { + return InitializationResult.contractVersionMismatch + } + if (uniffi_liters_ffi_checksum_method_httpclient_execute() != 20111) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_httprequestbody_read() != 60181) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_httpresponse_status() != 55118) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_httpresponse_headers() != 31518) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_httpresponse_read_body() != 54488) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_httpresponse_cancel() != 48762) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_push_now() != 60078) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_register_follow() != 44870) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_register_push() != 15559) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_resume() != 64190) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_resume_all() != 38642) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_set_listener() != 45114) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_shutdown() != 37060) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_sleep() != 32012) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_sleep_all() != 64691) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_status() != 22869) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_statuses() != 53853) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_sync_now() != 2233) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersmanager_unregister() != 3105) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersreplica_cancel() != 41779) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersreplica_close() != 45861) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersreplica_db_path() != 55063) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersreplica_position() != 41654) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersreplica_reset() != 44487) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_litersreplica_sync() != 37122) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_literswriter_cancel() != 9936) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_literswriter_close() != 22696) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_literswriter_maintain() != 48767) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_literswriter_position() != 6071) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_literswriter_push() != 60273) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_literswriter_reset_local() != 52534) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_literswriter_snapshot() != 32323) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_method_managerlistener_on_event() != 29970) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_constructor_litersmanager_new() != 6165) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_constructor_litersmanager_with_options() != 34269) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_constructor_litersreplica_new() != 23861) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_constructor_litersreplica_new_with_http_client() != 6821) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_constructor_literswriter_new() != 51245) { + return InitializationResult.apiChecksumMismatch + } + if (uniffi_liters_ffi_checksum_constructor_literswriter_new_with_http_client() != 15490) { + return InitializationResult.apiChecksumMismatch + } + + uniffiCallbackInitHttpClient() + uniffiCallbackInitHttpResponse() + uniffiCallbackInitManagerListener() + return InitializationResult.ok +}() + +// Make the ensure init function public so that other modules which have external type references to +// our types can call it. +public func uniffiEnsureLitersFfiInitialized() { + switch initializationResult { + case .ok: + break + case .contractVersionMismatch: + fatalError("UniFFI contract version mismatch: try cleaning and rebuilding your project") + case .apiChecksumMismatch: + fatalError("UniFFI API checksum mismatch: try cleaning and rebuilding your project") + } +} + +// swiftlint:enable all#endif diff --git a/Strand/CloudSync/SyncReplicationTrial.swift b/Strand/CloudSync/SyncReplicationTrial.swift index 99f7a39cab..af33014b86 100644 --- a/Strand/CloudSync/SyncReplicationTrial.swift +++ b/Strand/CloudSync/SyncReplicationTrial.swift @@ -43,6 +43,18 @@ enum SyncReplicationTrial { set { UserDefaults.standard.set(newValue, forKey: enabledKey) } } + /// Whether the trial is actually *running in this process*, as opposed to merely switched on. + /// + /// The two differ for exactly one launch after every flip, in both directions, and the difference is + /// the thing a user is most likely to get wrong: `applyAtLaunch` runs once in `init()` and + /// `wal_autocheckpoint` is a per-connection PRAGMA applied at pool-open time, so flipping the toggle + /// cannot retroactively change a pool that is already open. The Test Centre card shows this next to + /// the toggle so "on" and "measuring" are never conflated. + static var isInForce: Bool { + if case .external = StoreReplication.walCheckpointing { return true } + return false + } + /// Install the process-wide store policy. **Must be called before the first `WhoopStore` open**, /// i.e. at the top of the app's `init()`, ahead of `AppModel()`. /// diff --git a/Strand/Screens/TestCentreView.swift b/Strand/Screens/TestCentreView.swift index 4d14170561..13915e1671 100644 --- a/Strand/Screens/TestCentreView.swift +++ b/Strand/Screens/TestCentreView.swift @@ -41,6 +41,13 @@ struct TestCentreView: View { @AppStorage(PuffinExperiment.ppgHrSubLagInterpKey) private var ppgHrSubLagInterpEnabled = false @AppStorage(PuffinExperiment.hrvReadinessKey) private var hrvReadinessEnabled = false + #if CLOUD_SYNC + // Section 5: the page-replication trial. Bound to the SAME UserDefaults key SyncReplicationTrial + // reads at launch, so this toggle IS the switch rather than a mirror of it. Default false, which is + // what a missing key reads as, so a fresh install and an explicit "off" are the same state. + @AppStorage(SyncReplicationTrial.enabledKey) private var replicationTrialEnabled = false + #endif + /// True when the connected strap is a 5/MG, so the 5/MG experimental block shows. Mirrors the /// SettingsView gate (#22): a confident 4.0 owner never sees controls that cannot touch their strap. private var is5MG: Bool { selectedWhoopModelRaw == WhoopModel.whoop5mg.rawValue } @@ -60,6 +67,9 @@ struct TestCentreView: View { diagnosticToolsCard.staggeredAppear(index: 1) exportCard.staggeredAppear(index: 2) experimentalAlgorithmsCard.staggeredAppear(index: 3) + #if CLOUD_SYNC + replicationTrialCard.staggeredAppear(index: 4) + #endif } } .id(refreshToken) @@ -316,6 +326,60 @@ struct TestCentreView: View { } } + // MARK: - Section 5: Page-replication trial (CLOUD_SYNC only, opt-in, off by default) + + #if CLOUD_SYNC + /// The one UI for `SyncReplicationTrial`. Until this existed the switch was a bare `UserDefaults` + /// key with no way to flip it short of attaching a debugger, which meant the measurement it exists + /// to produce could not be started on the device that has the data. + /// + /// Three things are shown, deliberately, because they are three different facts: + /// + /// - the **intent** (this toggle, i.e. the persisted flag); + /// - the **reality** (`SyncReplicationTrial.statusLine`, which reports what the store actually + /// opened with) — these differ for the whole of the launch you flip the toggle in, because + /// `wal_autocheckpoint` is applied at pool-open time and cannot be changed on a live pool; and + /// - the **result so far** (the telemetry aggregate, carried inside `statusLine`). + /// + /// Reading only the first would let a user conclude the trial is running when it is not. + @ViewBuilder private var replicationTrialCard: some View { + NoopCard { + VStack(alignment: .leading, spacing: NoopMetrics.space3) { + Text("PAGE REPLICATION TRIAL") + .font(StrandFont.overline).tracking(StrandFont.overlineTracking) + .foregroundStyle(StrandPalette.textSecondary) + Text("Measures whether shipping only the changed pages of your database could replace the whole-database Cloud Sync upload. Opt-in, off by default.") + .font(StrandFont.caption).foregroundStyle(StrandPalette.textTertiary) + .fixedSize(horizontal: false, vertical: true) + + Toggle(isOn: $replicationTrialEnabled) { + Text("Hand WAL checkpointing to the replicator") + .font(StrandFont.subhead).foregroundStyle(StrandPalette.textPrimary) + } + .toggleStyle(.switch).tint(StrandPalette.accent) + .accessibilityLabel("Page replication trial") + + Text("When on, NOOP stops letting SQLite checkpoint the write-ahead log on its own and records one measurement per Cloud Sync upload. It does NOT change what gets uploaded: today's full-database upload is untouched, so the worst case is a larger write-ahead log between syncs. Takes effect the next time NOOP starts.") + .font(StrandFont.caption).foregroundStyle(StrandPalette.textTertiary) + .fixedSize(horizontal: false, vertical: true) + + Divider().overlay(StrandPalette.hairline) + + // The state actually in force, which is NOT the toggle above until the next launch. + Text(SyncReplicationTrial.statusLine) + .font(StrandFont.caption).foregroundStyle(StrandPalette.textSecondary) + .fixedSize(horizontal: false, vertical: true) + .textSelection(.enabled) + + if replicationTrialEnabled, !SyncReplicationTrial.isInForce { + Text("Restart NOOP to start measuring.") + .font(StrandFont.caption).foregroundStyle(StrandPalette.statusWarning) + } + } + } + } + #endif + // MARK: - Shared actions (same calls as the SettingsView controls these re-host) /// Re-anchor every baseline that feeds Charge from now, via the single cross-platform source of diff --git a/StrandTests/LitersRoundTripTests.swift b/StrandTests/LitersRoundTripTests.swift new file mode 100644 index 0000000000..29faa9defe --- /dev/null +++ b/StrandTests/LitersRoundTripTests.swift @@ -0,0 +1,277 @@ +// The liters round trip, run against real SQLite files on disk. +// +// Gated on BOTH flags. CLOUD_SYNC because this is fork-local sync work; LITERS because the symbols +// come from Liters.xcframework, which `Rust/build-ios.sh` produces and which is deliberately not +// tracked (see Config/Liters.xcconfig). A checkout that has never built the Rust side compiles none +// of this rather than failing to link. +#if CLOUD_SYNC && LITERS +import XCTest +import SQLite3 +@testable import Strand + +/// End-to-end proof that the Rust archive in `Liters.xcframework` actually replicates a database, +/// rather than merely linking. +/// +/// ## Why these tests open SQLite through `import SQLite3` and not GRDB +/// +/// The single load-bearing decision in `Rust/Cargo.toml` is `default-features = false`, which makes +/// liters link the *platform* `libsqlite3` instead of compiling its own copy of the amalgamation. +/// The reason is that two SQLite libraries in one process do not share the process-global +/// `unixInodeInfo` table SQLite uses to work around POSIX's "close any descriptor, lose all locks" +/// rule, so either can silently drop the other's advisory locks — and liters' correctness rests on a +/// long-running read lock. +/// +/// `import SQLite3` resolves to exactly the `libsqlite3.dylib` GRDB links through +/// `.systemLibrary(name: "CSQLite")`. So these tests are not a stand-in for the app's arrangement; +/// they *are* it: one process, one SQLite, opened from Swift on one side and from Rust on the other, +/// against the same file. A test written through GRDB would prove the same thing less directly and +/// drag a package dependency into a target that does not otherwise need one. +/// +/// ## Why a real file and not an in-memory database +/// +/// liters replicates WAL frames. An in-memory database has no WAL, no `-wal` sidecar, and no file +/// locks, so it would exercise none of the machinery under test and would pass whether or not the +/// linkage were correct. +final class LitersRoundTripTests: XCTestCase { + + private var dir: URL! + + override func setUpWithError() throws { + try super.setUpWithError() + dir = URL(fileURLWithPath: NSTemporaryDirectory()) + .appendingPathComponent("liters-rt-\(UUID().uuidString)", isDirectory: true) + try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + } + + override func tearDownWithError() throws { + if let dir { try? FileManager.default.removeItem(at: dir) } + try super.tearDownWithError() + } + + private var originPath: String { dir.appendingPathComponent("origin.sqlite").path } + private var bucketPath: String { dir.appendingPathComponent("bucket", isDirectory: true).path } + private var replicaPath: String { dir.appendingPathComponent("replica.sqlite").path } + + // MARK: - The round trip + + /// Writer pushes to a `Storage.dir` bucket, replica syncs out of it, and the rows arrive. + /// + /// This is the whole premise in one test: if the Swift bindings, the UniFFI scaffolding, the Rust + /// archive, and the platform `libsqlite3` are all wired correctly, a row written through Swift's + /// SQLite shows up in a second file that only Rust ever wrote. Any break in that chain fails here + /// rather than three integration layers later. + func testWriterPushesAndReplicaReceivesTheRows() throws { + try makeOriginDatabase(rows: 1...50) + + let writer = try LitersWriter(dbPath: originPath, storage: .dir(path: bucketPath)) + defer { writer.close() } + let push = try writer.push() + + XCTAssertTrue(push.synced, "the first push must capture the database's committed content") + XCTAssertGreaterThan(push.uploaded, 0, "the first push must upload at least one LTX file") + XCTAssertGreaterThan(push.bytesUploaded, 0) + XCTAssertGreaterThan(push.txid, 0) + + let replica = try LitersReplica(dbPath: replicaPath, storage: .dir(path: bucketPath), autoReset: true) + defer { replica.close() } + let sync = try replica.sync() + + XCTAssertGreaterThan(sync.toTxid, 0) + XCTAssertEqual(try readIds(at: replicaPath), Array(1...50), + "the replica must contain exactly the rows the origin committed") + } + + /// The second push must ship a WAL delta, not another copy of the database. + /// + /// `snapshotted` is the number the whole trial exists to measure. A replicator that snapshots on + /// every push is a full upload with extra steps, which is precisely what the current `/ingest` + /// path already does. Asserting it is `false` for a plain "write more rows, push again" is the + /// minimum bar; anything less and there is no reason to adopt this at all. + func testASecondPushIsIncrementalAndNotASnapshot() throws { + try makeOriginDatabase(rows: 1...10) + + let writer = try LitersWriter(dbPath: originPath, storage: .dir(path: bucketPath)) + defer { writer.close() } + let first = try writer.push() + XCTAssertTrue(first.synced) + + try appendRows(11...20, to: originPath) + let second = try writer.push() + + XCTAssertFalse(second.snapshotted, + "an ordinary follow-up push must ship a WAL delta. reason: \(second.snapshotReason ?? "nil")") + XCTAssertGreaterThan(second.txid, first.txid) + + let replica = try LitersReplica(dbPath: replicaPath, storage: .dir(path: bucketPath), autoReset: true) + defer { replica.close() } + _ = try replica.sync() + XCTAssertEqual(try readIds(at: replicaPath), Array(1...20)) + } + + /// An incremental sync onto an ALREADY-restored replica applies the delta rather than restoring. + /// + /// The server half of this design keeps `replica.db` current with repeated `sync()` calls, so + /// "second sync does not re-download the world" is a property of the deployment, not a nicety. + func testSecondSyncAppliesIncrementallyWithoutRestoring() throws { + try makeOriginDatabase(rows: 1...10) + let writer = try LitersWriter(dbPath: originPath, storage: .dir(path: bucketPath)) + defer { writer.close() } + _ = try writer.push() + + let replica = try LitersReplica(dbPath: replicaPath, storage: .dir(path: bucketPath), autoReset: true) + defer { replica.close() } + let firstSync = try replica.sync() + XCTAssertTrue(firstSync.restored, "the first sync onto an empty path is a restore by definition") + + try appendRows(11...30, to: originPath) + _ = try writer.push() + + let secondSync = try replica.sync() + XCTAssertFalse(secondSync.restored, "a replica that is already current must apply a delta, not restore") + XCTAssertGreaterThan(secondSync.toTxid, secondSync.fromTxid) + XCTAssertEqual(try readIds(at: replicaPath), Array(1...30)) + } + + // MARK: - The iOS-shaped failure mode + + /// Close the writer, reopen it, push. This is what iOS does to the app on every relaunch, and the + /// design document names it as the one thing that must be measured before committing: + /// + /// > the issue-#927 "expected truncation" shortcut is deliberately not trusted across an app + /// > restart — which on iOS is every single time. + /// + /// The test does not assert `snapshotted == false`, because a snapshot here would be *correct* + /// behaviour, not a bug — liters is refusing to trust a stale shortcut. It asserts the weaker, + /// genuinely required property: a reopened writer resumes against the same bucket and the replica + /// still converges. Whether the reopen costs a snapshot in practice is what + /// `SyncPushTelemetry.snapshotRate` measures on VK's device; recording the observed value here + /// documents which way it fell on this platform. + func testWriterReopenResumesAgainstTheSameBucket() throws { + try makeOriginDatabase(rows: 1...10) + + let first = try LitersWriter(dbPath: originPath, storage: .dir(path: bucketPath)) + let firstPush = try first.push() + first.close() + + try appendRows(11...25, to: originPath) + + let second = try LitersWriter(dbPath: originPath, storage: .dir(path: bucketPath)) + defer { second.close() } + let secondPush = try second.push() + + XCTAssertGreaterThan(secondPush.txid, firstPush.txid, + "a reopened writer must resume from the bucket, not start over at txid 0") + + let replica = try LitersReplica(dbPath: replicaPath, storage: .dir(path: bucketPath), autoReset: true) + defer { replica.close() } + _ = try replica.sync() + XCTAssertEqual(try readIds(at: replicaPath), Array(1...25), + "no commit may be lost across a writer close/reopen") + + // Not an assertion: the measurement. Printed so a CI log records which way the reopen fell. + print("[liters] reopen push snapshotted=\(secondPush.snapshotted) reason=\(secondPush.snapshotReason ?? "nil")") + } + + // MARK: - Errors cross the FFI as errors + + /// A failure has to arrive in Swift as a thrown `LitersError`, not a crash and not a silent + /// success. UniFFI's error mapping is generated code, so this checks the generated code works for + /// the one path the app will actually hit at 3am. + func testOpeningAMissingDatabaseThrows() { + let missing = dir.appendingPathComponent("nope/absent.sqlite").path + XCTAssertThrowsError(try LitersWriter(dbPath: missing, storage: .dir(path: bucketPath))) + } + + // MARK: - SQLite helpers (the platform libsqlite3, the same one GRDB links) + + /// Creates a WAL-mode database and inserts `rows`, through Apple's system SQLite. + private func makeOriginDatabase(rows: ClosedRange) throws { + var db: OpaquePointer? + let rc = sqlite3_open_v2(originPath, &db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, nil) + defer { sqlite3_close(db) } + guard rc == SQLITE_OK, db != nil else { + throw SQLiteFailure(code: rc, message: "open \(originPath)") + } + + // WAL is not incidental. liters replicates WAL frames, so a rollback-journal database would + // have nothing for it to ship and the round trip would prove nothing. + let mode = try queryString(db, "PRAGMA journal_mode = WAL;") + guard mode == "wal" else { throw SQLiteFailure(code: -1, message: "journal_mode is \(mode)") } + try exec(db, "CREATE TABLE sample (id INTEGER PRIMARY KEY, payload TEXT NOT NULL);") + try insert(db, rows) + } + + /// Runs a statement that returns one text column. `PRAGMA journal_mode = WAL` is such a + /// statement — it reports the mode actually in force, which is the only way to know the switch + /// took (it silently stays in rollback mode inside a transaction, for instance). + private func queryString(_ db: OpaquePointer?, _ sql: String) throws -> String { + var stmt: OpaquePointer? + let rc = sqlite3_prepare_v2(db, sql, -1, &stmt, nil) + guard rc == SQLITE_OK else { throw SQLiteFailure(code: rc, message: sql) } + defer { sqlite3_finalize(stmt) } + guard sqlite3_step(stmt) == SQLITE_ROW, let text = sqlite3_column_text(stmt, 0) else { + throw SQLiteFailure(code: -1, message: "no row from \(sql)") + } + return String(cString: text) + } + + /// Writes through a SECOND connection while the liters writer holds its read lock open. That is + /// the app's real arrangement — GRDB writing while the replicator watches — and it is the case + /// the read lock is designed to allow: it blocks foreign *checkpoints*, not foreign writes. + private func appendRows(_ rows: ClosedRange, to path: String) throws { + var db: OpaquePointer? + let rc = sqlite3_open_v2(path, &db, SQLITE_OPEN_READWRITE, nil) + defer { sqlite3_close(db) } + guard rc == SQLITE_OK, db != nil else { throw SQLiteFailure(code: rc, message: "reopen \(path)") } + try insert(db, rows) + } + + private func insert(_ db: OpaquePointer?, _ rows: ClosedRange) throws { + try exec(db, "BEGIN;") + for i in rows { + // Padded so each row is big enough that a batch spans more than one 4 KB page, which is + // what makes "the second push shipped a delta" a statement about pages and not about an + // empty WAL. + try exec(db, "INSERT INTO sample (id, payload) VALUES (\(i), '\(String(repeating: "x", count: 512))');") + } + try exec(db, "COMMIT;") + } + + /// Reads back every id, ascending. The assertion target for "did the bytes arrive". + private func readIds(at path: String) throws -> [Int] { + var db: OpaquePointer? + // READWRITE, not READONLY, and that is not arbitrary: Apple's libsqlite3 cannot run a + // statement on a WAL-mode database opened SQLITE_OPEN_READONLY when the -shm sidecar is + // absent, which it is right after a liters restore. Opening read-only here would fail with + // SQLITE_CANTOPEN for reasons that have nothing to do with whether the bytes arrived. + let rc = sqlite3_open_v2(path, &db, SQLITE_OPEN_READWRITE, nil) + defer { sqlite3_close(db) } + guard rc == SQLITE_OK, db != nil else { throw SQLiteFailure(code: rc, message: "open replica \(path)") } + + var stmt: OpaquePointer? + let prepared = sqlite3_prepare_v2(db, "SELECT id FROM sample ORDER BY id;", -1, &stmt, nil) + guard prepared == SQLITE_OK else { throw SQLiteFailure(code: prepared, message: "prepare select") } + defer { sqlite3_finalize(stmt) } + + var ids: [Int] = [] + while sqlite3_step(stmt) == SQLITE_ROW { ids.append(Int(sqlite3_column_int64(stmt, 0))) } + return ids + } + + private func exec(_ db: OpaquePointer?, _ sql: String) throws { + var err: UnsafeMutablePointer? + let rc = sqlite3_exec(db, sql, nil, nil, &err) + guard rc == SQLITE_OK else { + let message = err.map { String(cString: $0) } ?? sql + sqlite3_free(err) + throw SQLiteFailure(code: rc, message: message) + } + } + + private struct SQLiteFailure: Error, CustomStringConvertible { + let code: Int32 + let message: String + var description: String { "sqlite rc \(code): \(message)" } + } +} +#endif diff --git a/project.yml b/project.yml index 58f3752ff0..7c08978318 100644 --- a/project.yml +++ b/project.yml @@ -144,6 +144,13 @@ targets: # distributed (ad-hoc) build. The app uses no JIT / unsigned-exec memory, so it's safe. ENABLE_HARDENED_RUNTIME: YES COMBINE_HIDPI_IMAGES: YES + # liters (page-replication trial). All three variables are defined EMPTY in the tracked + # Config/Liters.xcconfig and only filled in by the untracked Config/LitersLocal.xcconfig that + # Rust/build-ios.sh writes, so on a checkout that has never built the Rust side these three + # lines append nothing and this target links exactly what it links today. See Config/Liters.xcconfig. + LIBRARY_SEARCH_PATHS: $(inherited) $(LITERS_LIB_DIR) + SWIFT_INCLUDE_PATHS: $(inherited) $(LITERS_HEADER_DIR) + OTHER_LDFLAGS: $(inherited) $(LITERS_LDFLAGS) dependencies: - package: WhoopProtocol - package: WhoopStore @@ -330,6 +337,12 @@ targets: GENERATE_INFOPLIST_FILE: NO TARGETED_DEVICE_FAMILY: "1,2" SWIFT_EMIT_LOC_STRINGS: YES + # liters (page-replication trial) — same inert-by-default seam as the macOS target above. + # The slice these resolve to is chosen per-SDK inside Config/LitersLocal.xcconfig, so the + # device and simulator builds pick up their own archive without a second setting here. + LIBRARY_SEARCH_PATHS: $(inherited) $(LITERS_LIB_DIR) + SWIFT_INCLUDE_PATHS: $(inherited) $(LITERS_HEADER_DIR) + OTHER_LDFLAGS: $(inherited) $(LITERS_LDFLAGS) dependencies: - package: WhoopProtocol - package: WhoopStore From eb700b2bd865227d7368da7ba3e61a81e46ebe9e Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 27 Jul 2026 17:40:15 -0400 Subject: [PATCH 140/171] docs(liters): say what the pin contains, and what it does not Two liters repos exist and only one is pinned. `vishk23/liters-mobile` is what Cargo.toml points at; `vishk23/liters` is the working clone, and it is the only place `fix(replica): bound and cancel the replica-file lock instead of blocking forever` lives. That commit has not been merged to liters-mobile main, so the F_SETLKW fix is NOT in the archive this directory builds, contrary to what a reader would reasonably assume from the repo description. Also records the SQLITE_CANTOPEN gotcha found while verifying the linkage: liters passes 80/80 of its own tests with bundled SQLite and 31/80 with the system SQLite NOOP actually links, every failure the same error, all on the Replica restore path, all from the crate's single read-only open (replica.rs:648). Reduced to a four-case C program against Apple's libsqlite3: a WAL database whose -shm is absent cannot run a statement when opened SQLITE_OPEN_READONLY. Both are Replica-side and so cannot affect the phone, which is a Writer and never opens read-only. Written down because "the linkage is verified at the symbol level" is not the same claim as "the behaviour is verified", and the difference was invisible until someone ran the suite in the shipping configuration. --- Rust/README.md | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/Rust/README.md b/Rust/README.md index b9f3f07f97..7c3bad9821 100644 --- a/Rust/README.md +++ b/Rust/README.md @@ -78,7 +78,32 @@ cargo update -p liters-ffi # then commit the Cargo.lock change `Cargo.lock` is committed on purpose: it records the exact liters commit this tree builds against, so a checkout is reproducible even though the dependency -tracks a branch. +tracks a branch. As of this commit the pin is +`liters-mobile@6835f9f9`, and the four commits between it and `liters-mobile` +`main` are documentation and test-harness changes only. + +### What the pin does NOT contain + +There are **two** liters repos, and only one of them is pinned here: + +| repo | role | +|---|---| +| `vishk23/liters-mobile` | public, MIT, **what `Cargo.toml` pins** | +| `vishk23/liters` | the working clone; carries `fix/replica-lock-deadlock` | + +`fix(replica): bound and cancel the replica-file lock instead of blocking +forever` — the F_SETLKW fix that replaces a kernel-blocking `fcntl(F_SETLKW)` +with a bounded `F_SETLK` poll so `CancelToken` can interrupt it — exists only in +`vishk23/liters`. It has **not** been merged to `liters-mobile` `main`, so it is +**not** in the archive this directory builds. Check before assuming otherwise: + +```sh +git -C log --all --oneline --grep="bound and cancel" # currently empty +``` + +Like the integrity-check gotcha above, it is a Replica-side fix and so does not +affect the phone, which is a Writer. Both should be landed on `liters-mobile` +before anything runs a Replica against a system-SQLite build. ## Build time, and the one thing that would fix it From d3cc8cba67d84338ca1efea8daff4f618fbcc752 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 27 Jul 2026 17:59:18 -0400 Subject: [PATCH 141/171] fix(liters): close the #if the generated bindings are wrapped in MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit uniffi's Swift output ends `// swiftlint:enable all` with no trailing newline, so `cat file; echo "#endif"` appended the directive to that comment line, where the comment ate it. The wrapper's `#if LITERS` was therefore never closed and every build with liters enabled died at EOF with "expected #else or #endif at end of conditional compilation block" — an error that points at the last line of a 4,459-line generated file and says nothing about the comment that caused it. `printf '\n#endif\n'` instead. Caught by the iOS simulator build, which is the first thing that actually compiled the file. --- Rust/build-ios.sh | 7 ++++++- Strand/CloudSync/Generated/liters_ffi.swift | 3 ++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/Rust/build-ios.sh b/Rust/build-ios.sh index 63c1fae79a..eea4022576 100755 --- a/Rust/build-ios.sh +++ b/Rust/build-ios.sh @@ -112,7 +112,12 @@ mkdir -p "$GEN" echo "// The #if is added by that script, not by uniffi: see the script for why it is here." echo "#if LITERS" cat "$BINDINGS/liters_ffi.swift" - echo "#endif" + # The leading newline is load-bearing. uniffi's output ends `// swiftlint:enable all` with NO + # trailing newline, so a plain `echo "#endif"` lands on the same line as that comment and is + # swallowed by it — producing a file whose `#if LITERS` is never closed. The compiler's complaint + # ("expected #else or #endif at end of conditional compilation block", pointing at EOF) does not + # mention the comment, so this is worth a comment rather than a rediscovery. + printf '\n#endif\n' } > "$GEN/liters_ffi.swift" echo "==> installed bindings: $GEN/liters_ffi.swift" diff --git a/Strand/CloudSync/Generated/liters_ffi.swift b/Strand/CloudSync/Generated/liters_ffi.swift index 25bc8cff3b..5fe6329841 100644 --- a/Strand/CloudSync/Generated/liters_ffi.swift +++ b/Strand/CloudSync/Generated/liters_ffi.swift @@ -4456,4 +4456,5 @@ public func uniffiEnsureLitersFfiInitialized() { } } -// swiftlint:enable all#endif +// swiftlint:enable all +#endif From 0a445be170c63ca53dc3eaaf11dcca1832ed6b94 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 27 Jul 2026 18:23:17 -0400 Subject: [PATCH 142/171] test(liters): split the round trip at the line system SQLite actually draws MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Running the round trip for real answered a question the symbol-level check could not. 5 tests, all green: testWriterPushesCommittedContent passed testASecondPushIsIncrementalAndNotASnapshot passed testWriterReopenResumesAgainstTheSameBucket passed testReplicaRestoreIsBrokenUnderSystemSQLite passed testOpeningAMissingDatabaseThrows passed The first version of this file asserted a writer->replica->rows round trip in one test and 4 of 5 failed, every one of them on `replica.sync()` and every one with `sqlite: unable to open database file`. Not one `writer.push()` failed. That is not a flaky suite, it is the boundary: the Writer half works in the linkage NOOP ships and the Replica half does not, for the reason documented in Rust/README.md (replica.rs:648 opens READ_ONLY, and Apple's libsqlite3 cannot run a statement on a WAL database with no -shm). The phone is a Writer. The server, which is the Replica, has no GRDB and so builds liters with bundled SQLite. So the tests are split along that line rather than papered over. The writer tests assert real behaviour. The replica test asserts the *bug*, deliberately, and says in its doc comment that it must be inverted into the real round trip the moment it starts failing — which is the moment liters is fixed. A skipped test would have recorded nothing; this one is a tripwire. The reopen test also produced the measurement SYNC_BUILD_VS_BUY.md §"The one thing that must be tested" asks for, and it is not the comfortable answer: [liters] reopen push snapshotted=true reason=wal truncated by another process A writer that closes, has a foreign connection write and close, then reopens, pays a FULL SNAPSHOT. That is the unmitigated case — this test does not install the `.external` policy that sets wal_autocheckpoint=0 — so it does not predict the device rate. It does show the mechanism is real and reachable, which is exactly why StoreReplication's `.external` policy is load-bearing rather than tidy. project.yml: StrandTests needed SWIFT_INCLUDE_PATHS and nothing else. Without it `@testable import Strand` fails to resolve `liters_ffiFFI` transitively even though the app compiled; with the link flags it would pull a second copy of the Rust staticlib into the test bundle, and UniFFI's handle map is static storage. --- Strand/CloudSync/Generated/liters_ffi.swift | 3 +- StrandTests/LitersRoundTripTests.swift | 101 ++++++++++---------- project.yml | 13 +++ 3 files changed, 67 insertions(+), 50 deletions(-) diff --git a/Strand/CloudSync/Generated/liters_ffi.swift b/Strand/CloudSync/Generated/liters_ffi.swift index 5fe6329841..25bc8cff3b 100644 --- a/Strand/CloudSync/Generated/liters_ffi.swift +++ b/Strand/CloudSync/Generated/liters_ffi.swift @@ -4456,5 +4456,4 @@ public func uniffiEnsureLitersFfiInitialized() { } } -// swiftlint:enable all -#endif +// swiftlint:enable all#endif diff --git a/StrandTests/LitersRoundTripTests.swift b/StrandTests/LitersRoundTripTests.swift index 29faa9defe..5bf8d47d71 100644 --- a/StrandTests/LitersRoundTripTests.swift +++ b/StrandTests/LitersRoundTripTests.swift @@ -52,16 +52,20 @@ final class LitersRoundTripTests: XCTestCase { private var bucketPath: String { dir.appendingPathComponent("bucket", isDirectory: true).path } private var replicaPath: String { dir.appendingPathComponent("replica.sqlite").path } - // MARK: - The round trip + // MARK: - The writer, which is the half the phone is - /// Writer pushes to a `Storage.dir` bucket, replica syncs out of it, and the rows arrive. + /// A push captures the database's committed content and uploads it. /// /// This is the whole premise in one test: if the Swift bindings, the UniFFI scaffolding, the Rust - /// archive, and the platform `libsqlite3` are all wired correctly, a row written through Swift's - /// SQLite shows up in a second file that only Rust ever wrote. Any break in that chain fails here - /// rather than three integration layers later. - func testWriterPushesAndReplicaReceivesTheRows() throws { + /// archive, and the platform `libsqlite3` are all wired correctly, a database written through + /// Swift's SQLite becomes LTX files in a bucket that only Rust ever wrote. Any break in that + /// chain fails here rather than three integration layers later. + func testWriterPushesCommittedContent() throws { try makeOriginDatabase(rows: 1...50) + // Establishes the baseline the replica half will be compared against once liters' restore + // works under system SQLite (see testReplicaRestoreIsBrokenUnderSystemSQLite), and proves the + // fixture really is 50 committed rows rather than an empty file liters would happily push. + XCTAssertEqual(try readIds(at: originPath), Array(1...50)) let writer = try LitersWriter(dbPath: originPath, storage: .dir(path: bucketPath)) defer { writer.close() } @@ -71,14 +75,7 @@ final class LitersRoundTripTests: XCTestCase { XCTAssertGreaterThan(push.uploaded, 0, "the first push must upload at least one LTX file") XCTAssertGreaterThan(push.bytesUploaded, 0) XCTAssertGreaterThan(push.txid, 0) - - let replica = try LitersReplica(dbPath: replicaPath, storage: .dir(path: bucketPath), autoReset: true) - defer { replica.close() } - let sync = try replica.sync() - - XCTAssertGreaterThan(sync.toTxid, 0) - XCTAssertEqual(try readIds(at: replicaPath), Array(1...50), - "the replica must contain exactly the rows the origin committed") + XCTAssertGreaterThan(push.remoteTxid, 0, "the bucket must know about the push that just happened") } /// The second push must ship a WAL delta, not another copy of the database. @@ -101,35 +98,6 @@ final class LitersRoundTripTests: XCTestCase { XCTAssertFalse(second.snapshotted, "an ordinary follow-up push must ship a WAL delta. reason: \(second.snapshotReason ?? "nil")") XCTAssertGreaterThan(second.txid, first.txid) - - let replica = try LitersReplica(dbPath: replicaPath, storage: .dir(path: bucketPath), autoReset: true) - defer { replica.close() } - _ = try replica.sync() - XCTAssertEqual(try readIds(at: replicaPath), Array(1...20)) - } - - /// An incremental sync onto an ALREADY-restored replica applies the delta rather than restoring. - /// - /// The server half of this design keeps `replica.db` current with repeated `sync()` calls, so - /// "second sync does not re-download the world" is a property of the deployment, not a nicety. - func testSecondSyncAppliesIncrementallyWithoutRestoring() throws { - try makeOriginDatabase(rows: 1...10) - let writer = try LitersWriter(dbPath: originPath, storage: .dir(path: bucketPath)) - defer { writer.close() } - _ = try writer.push() - - let replica = try LitersReplica(dbPath: replicaPath, storage: .dir(path: bucketPath), autoReset: true) - defer { replica.close() } - let firstSync = try replica.sync() - XCTAssertTrue(firstSync.restored, "the first sync onto an empty path is a restore by definition") - - try appendRows(11...30, to: originPath) - _ = try writer.push() - - let secondSync = try replica.sync() - XCTAssertFalse(secondSync.restored, "a replica that is already current must apply a delta, not restore") - XCTAssertGreaterThan(secondSync.toTxid, secondSync.fromTxid) - XCTAssertEqual(try readIds(at: replicaPath), Array(1...30)) } // MARK: - The iOS-shaped failure mode @@ -161,15 +129,52 @@ final class LitersRoundTripTests: XCTestCase { XCTAssertGreaterThan(secondPush.txid, firstPush.txid, "a reopened writer must resume from the bucket, not start over at txid 0") + XCTAssertTrue(secondPush.synced, "the commits made while closed must be captured on reopen") + + // Not an assertion: the measurement. Printed so a CI log records which way the reopen fell. + print("[liters] reopen push snapshotted=\(secondPush.snapshotted) reason=\(secondPush.snapshotReason ?? "nil")") + } + + // MARK: - The replica, which is the half the SERVER is, and which system SQLite breaks + + /// **This test asserts a bug, on purpose.** When it starts failing, liters has been fixed and the + /// assertion below should be inverted into the real round trip (restore, then compare + /// `readIds(at: replicaPath)` against the origin's rows). + /// + /// `Replica.sync()` cannot restore when liters is linked against the platform `libsqlite3` — + /// which is the configuration NOOP ships, and the whole point of `default-features = false`. + /// The cause is `crates/liters/src/replica.rs:648`, the only read-only open in the crate, reached + /// only from `check_integrity` after a restore: + /// + /// ```rust + /// rusqlite::Connection::open_with_flags(&self.db_path, OpenFlags::SQLITE_OPEN_READ_ONLY) + /// ``` + /// + /// Apple's libsqlite3 (3.51.0) cannot run a statement on a WAL-mode database opened + /// `SQLITE_OPEN_READONLY` when the `-shm` sidecar is absent — and the restore deletes `-wal`/ + /// `-shm` immediately before this call. Reduced to a four-case C program against + /// `/usr/lib/libsqlite3.dylib`; opening the same file READWRITE, or leaving the sidecars in + /// place, succeeds. liters' own suite passes 80/80 with its bundled SQLite and 31/80 without, + /// and every one of those 49 failures is on this path. + /// + /// **It does not block the phone.** NOOP is a Writer; `writer.rs` contains no read-only open at + /// all, and the tests above prove the writer half works in exactly this linkage. The Replica runs + /// on the server, in a Rust process with no GRDB, which therefore builds liters with bundled + /// SQLite and never reaches this. The test exists so that "we knew, and here is the boundary" + /// is a property of the codebase rather than of someone's memory. + func testReplicaRestoreIsBrokenUnderSystemSQLite() throws { + try makeOriginDatabase(rows: 1...10) + let writer = try LitersWriter(dbPath: originPath, storage: .dir(path: bucketPath)) + defer { writer.close() } + _ = try writer.push() let replica = try LitersReplica(dbPath: replicaPath, storage: .dir(path: bucketPath), autoReset: true) defer { replica.close() } - _ = try replica.sync() - XCTAssertEqual(try readIds(at: replicaPath), Array(1...25), - "no commit may be lost across a writer close/reopen") - // Not an assertion: the measurement. Printed so a CI log records which way the reopen fell. - print("[liters] reopen push snapshotted=\(secondPush.snapshotted) reason=\(secondPush.snapshotReason ?? "nil")") + XCTAssertThrowsError(try replica.sync(), "if this no longer throws, liters is fixed — invert this test") { error in + XCTAssertTrue("\(error)".contains("unable to open database file"), + "expected the known SQLITE_CANTOPEN from check_integrity, got: \(error)") + } } // MARK: - Errors cross the FFI as errors diff --git a/project.yml b/project.yml index 7c08978318..c70f810260 100644 --- a/project.yml +++ b/project.yml @@ -184,6 +184,19 @@ targets: # NOTE: must track the app target's PRODUCT_NAME ("NOOP Staging" since the staging rename). TEST_HOST: "$(BUILT_PRODUCTS_DIR)/NOOP Staging.app/Contents/MacOS/NOOP Staging" BUNDLE_LOADER: "$(TEST_HOST)" + # liters: SWIFT_INCLUDE_PATHS ONLY, deliberately — no LIBRARY_SEARCH_PATHS and no + # OTHER_LDFLAGS, unlike the two app targets. + # + # The include path is required. `Strand.swiftmodule` records `liters_ffiFFI` as a dependency + # (the generated bindings import it), and `@testable import Strand` makes StrandTests resolve + # that transitively — without this it fails with "Unable to resolve module dependency: + # 'liters_ffiFFI'" even though the app itself compiled fine. + # + # The link flags are required to be ABSENT. This bundle already resolves liters' symbols + # through BUNDLE_LOADER, so adding -lnoop_liters would pull a SECOND copy of the Rust + # staticlib into the test bundle. UniFFI keeps its object handle map in static storage; two + # copies means two maps, and a handle minted by one is meaningless to the other. + SWIFT_INCLUDE_PATHS: $(inherited) $(LITERS_HEADER_DIR) # --------------------------------------------------------------------------- # iOS app target (community build-from-source; never distributed via the App From cc9d1536c6dc7fd501812f54fbd1eb1ece4dcd9c Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 27 Jul 2026 18:26:06 -0400 Subject: [PATCH 143/171] fix(liters): regenerate the bindings from the corrected script The previous commit swept up a stale copy of the generated file. Editing Rust/build-ios.sh while an instance of it was still running meant bash kept reading the OLD script text from its byte offset, so that run re-installed the bindings with the pre-fix `echo "#endif"` and re-wrote the xcconfig with the SWIFT_ACTIVE_COMPILATION_CONDITIONS line that was already known not to survive xcodegen's Debug defaults. Both are outputs, so both silently reverted. Re-running the committed script from a clean shell produces the right thing and exercises it end to end: 3 xcframework slices, `sqlite linkage OK: 0 defined, 22 undefined`, 139/139 balanced conditionals, and OTHER_SWIFT_FLAGS in the xcconfig. --- Strand/CloudSync/Generated/liters_ffi.swift | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Strand/CloudSync/Generated/liters_ffi.swift b/Strand/CloudSync/Generated/liters_ffi.swift index 25bc8cff3b..5fe6329841 100644 --- a/Strand/CloudSync/Generated/liters_ffi.swift +++ b/Strand/CloudSync/Generated/liters_ffi.swift @@ -4456,4 +4456,5 @@ public func uniffiEnsureLitersFfiInitialized() { } } -// swiftlint:enable all#endif +// swiftlint:enable all +#endif From 4140d73193ced63a27c8147df5ffa80770a96d82 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 28 Jul 2026 00:02:50 -0400 Subject: [PATCH 144/171] feat(cloudsync): push pages through liters when the trial is on, /ingest when it is not MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nothing had ever pushed an LTX frame from the phone. The bindings round-tripped, the trial flag existed, and every sync still uploaded the whole database. This wires LitersWriter into the real push path behind the existing flag. Three conditions gate it, all cheap: the trial flag (UserDefaults, default false), `.external` actually being in force rather than merely requested, and the client having a liters destination (it has none when LITERS is not compiled in, and no test double has one). Any failure — unreachable sink, full volume, lease conflict, rotated token — falls through to /ingest, which is untouched and remains both the fallback and the recovery path. The branch sits BEFORE the exporter, deliberately. `defaultExporter`'s first act is `wal_checkpoint(TRUNCATE)`, which is precisely what a page replicator must be the sole caller of; exporting and then pushing would restart the WAL underneath the writer and force a full snapshot on every sync. LitersURLSessionClient exists because liters' built-in socket transport rejects https:// outright, so without it the path cannot reach noop-cloud at all. It streams a PUT body from liters' pull-based HttpRequestBody through bound streams, so a snapshot push does not buffer. It also rewrites content-length from the buffered body: URLSession does not reliably surface the header, and liters fails the whole push with "list level 0: missing content-length" without it — found by running it, not by reading. LitersReplicator holds ONE writer for the process lifetime, which is the load- bearing decision here. Measured against a real noop-cloud sink on the real 782 MB / 3.18M-row database: a writer opened and dropped around each push snapshotted every time (~312 MB each), because closing the last connection checkpoints and truncates the WAL and `wal_autocheckpoint = 0` does not stop that. One writer held across pushes snapshots once and then runs incremental — 143-176 KB per 10-minute delta, ~2.0 MB per 4-hour delta, against 164.7 MB for the /ingest upload of the same database. maintain() is deliberately not called: one run took 177 s and uploaded a fresh 312 MB snapshot, which is neither a BGTask-sized operation nor compatible with the point of the exercise. The cost is bucket growth (~12 MB/day), swept server-side. --- Strand/CloudSync/CloudSyncClient.swift | 9 +- Strand/CloudSync/CloudSyncUploader.swift | 110 +++++++++++- Strand/CloudSync/LitersHTTPClient.swift | 205 +++++++++++++++++++++++ Strand/CloudSync/LitersReplicator.swift | 116 +++++++++++++ 4 files changed, 437 insertions(+), 3 deletions(-) create mode 100644 Strand/CloudSync/LitersHTTPClient.swift create mode 100644 Strand/CloudSync/LitersReplicator.swift diff --git a/Strand/CloudSync/CloudSyncClient.swift b/Strand/CloudSync/CloudSyncClient.swift index 4c74e9d670..5fab409cee 100644 --- a/Strand/CloudSync/CloudSyncClient.swift +++ b/Strand/CloudSync/CloudSyncClient.swift @@ -27,8 +27,13 @@ enum CloudSyncError: LocalizedError, Equatable { /// The noop-cloud edit-journal client. Injected `session` makes it URLProtocol-testable. /// Networking lives here in the app target by design (mirrors Strand/Oura/OuraAPIClient.swift). final class CloudSyncClient { - private let baseURL: URL - private let token: String + /// Internal rather than private so the liters push path can address the *same* server with the + /// *same* credential this client already holds. Deliberately not a second settings read: a + /// server URL that has been through `CloudSyncModel.saveSettings`' validation and a token that + /// came out of the Keychain are exactly what `/ingest` is using, and the two paths must never + /// be able to disagree about where "the server" is. + let baseURL: URL + let token: String private let session: URLSession init(baseURL: URL, token: String, session: URLSession = .shared) { diff --git a/Strand/CloudSync/CloudSyncUploader.swift b/Strand/CloudSync/CloudSyncUploader.swift index 2e765d5746..58d663a13e 100644 --- a/Strand/CloudSync/CloudSyncUploader.swift +++ b/Strand/CloudSync/CloudSyncUploader.swift @@ -9,8 +9,30 @@ import WhoopStore /// mirrors `CloudSyncCoordinator`'s `CloudEditFetching` seam. `CloudSyncClient` conforms below. protocol CloudIngesting { func ingest(fileURL: URL) async throws -> (bytes: Int, latestDay: String?) + + /// Where a page-replication push would go, and with what credential — the same server and the + /// same token `ingest` uses. `nil` means "this ingester has no liters destination", which is + /// the correct answer for every test double and makes the liters branch inert for them without + /// each one having to opt out. + var litersDestination: (endpoint: String, token: String)? { get } +} + +extension CloudIngesting { + /// Default: no liters destination. Only `CloudSyncClient` overrides this. + var litersDestination: (endpoint: String, token: String)? { nil } +} + +extension CloudSyncClient: CloudIngesting { + var litersDestination: (endpoint: String, token: String)? { + #if LITERS + return (LitersReplicator.endpoint(base: baseURL), token) + #else + // The xcframework is not in this build, so there is nothing to push with. Reporting `nil` + // rather than an endpoint keeps the trial flag from selecting a path that cannot exist. + return nil + #endif + } } -extension CloudSyncClient: CloudIngesting {} /// The export half's user-facing failure. The network half of an upload throws `CloudSyncError` (same /// typed error every other CloudSync network call throws); this covers only "never got as far as @@ -100,6 +122,30 @@ enum CloudSyncUploader { // i.e. the upload path would defeat the thing it is being replaced by. let walBytes = store.walFileSizeBytes() ?? 0 + // --- page replication, when the trial is on ------------------------------------------- + // + // Placed HERE, before the exporter, and that placement is the whole design. `exporter`'s + // first act is `checkpointWAL()` — a `wal_checkpoint(TRUNCATE)` — which is precisely what a + // page replicator must be the sole caller of (see the note above `walBytes`). Running the + // export and then pushing would restart the WAL underneath the writer and force + // `snapshotted: true` on every push: the old path would defeat the new one while both ran. + // + // Three conditions, all required, all cheap to check: + // * the trial flag is on (`UserDefaults`, default false — a shipped build is unaffected); + // * `.external` is actually in force, not merely requested. `isEnabled` and `isInForce` + // differ for exactly one launch after every flip, and pushing while SQLite still owns + // autocheckpoint is the configuration that snapshots on every sync; + // * the client has a liters destination (it does not when LITERS is not compiled in, and + // no test double has one). + // + // A failure here is NOT fatal and NOT retried in place: it falls through to `/ingest` + // below, which is untouched and remains both the fallback and the recovery path. The one + // thing that must not happen is a sync that ships nothing and reports success. + if let result = await litersPushIfEnabled(client: client, walBytes: walBytes, + telemetry: telemetry) { + return result + } + switch await exporter(store, tempURL) { case .exported: let result = try await client.ingest(fileURL: tempURL) @@ -126,5 +172,67 @@ enum CloudSyncUploader { throw CloudSyncUploadError.exportFailed("The export step returned an unexpected result.") } } + + /// One page-replication push, or `nil` to mean "not this time — use `/ingest`". + /// + /// Returns `nil` for every reason a push does not happen (trial off, `.external` not in force, + /// no destination, liters not compiled in) *and* for a push that failed, because the caller's + /// only correct response to all of them is the same: fall through to the whole-database upload. + /// The distinction is preserved in the log, not in the control flow. + /// + /// `latestDay` is `nil` on this path by construction: `/liters` is a byte pipe into + /// `mirror.sqlite` and answers with liters' protocol, not with `/ingest`'s + /// `{ok,bytes,latestDay}`. `CloudSyncModel` uses only `bytes`. + private static func litersPushIfEnabled( + client: any CloudIngesting, walBytes: Int64, telemetry: SyncPushTelemetry? + ) async -> (bytes: Int, latestDay: String?)? { + #if LITERS + guard SyncReplicationTrial.isEnabled else { return nil } + // `isEnabled` is the intent; `isInForce` is the reality. Pushing while SQLite still owns + // wal_autocheckpoint means a foreign checkpoint can restart the WAL between pushes, and + // every push then ships the whole database — measurably, not theoretically. + guard SyncReplicationTrial.isInForce else { + NSLog("liters: trial is on but WAL checkpointing is still .automatic " + + "(restart pending) — using /ingest for this sync") + return nil + } + guard let dest = client.litersDestination else { return nil } + + do { + // The one real on-disk database, resolved exactly the way `DataBackup`/`FolderBackup` + // resolve it. Throwing is folded into the same catch: a store path we cannot even name + // is a reason to use `/ingest`, not a reason to fail the sync. + let dbPath = try StorePaths.defaultDatabasePath() + // Synchronous and blocking by liters' contract; kept off the cooperative pool's + // forward progress by running it on a detached background task. + let summary = try await Task.detached(priority: .utility) { + try LitersReplicator.shared.push(databasePath: dbPath, + endpoint: dest.endpoint, + token: dest.token) + }.value + + telemetry?.record(snapshotted: summary.snapshotted, + snapshotReason: summary.snapshotReason, + bytesUploaded: Int64(summary.bytesUploaded), + walBytes: walBytes, + txid: summary.txid) + NSLog("liters: pushed txid=%llu bytes=%llu snapshotted=%d reason=%@ walAtPush=%lld", + summary.txid, summary.bytesUploaded, summary.snapshotted ? 1 : 0, + summary.snapshotReason ?? "-", walBytes) + return (Int(summary.bytesUploaded), nil) + } catch { + // Deliberately swallowed. Every liters failure mode — an unreachable sink (503 from + // the proxy when LITERS_SINK_ENABLED is unset), a full volume (507), a lease conflict, + // a rotated token — is survivable by uploading the whole database instead, and a sync + // that ships data on the old path beats a sync that reports a new-path error. + NSLog("liters: push failed (%@) — falling back to /ingest for this sync", + String(describing: error)) + return nil + } + #else + _ = (client, walBytes, telemetry) + return nil + #endif + } } #endif // CLOUD_SYNC diff --git a/Strand/CloudSync/LitersHTTPClient.swift b/Strand/CloudSync/LitersHTTPClient.swift new file mode 100644 index 0000000000..3143673ffb --- /dev/null +++ b/Strand/CloudSync/LitersHTTPClient.swift @@ -0,0 +1,205 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig — see OuraConfig.xcconfig) AND LITERS is set (by the generated +// Config/LitersLocal.xcconfig, i.e. only after Rust/build-ios.sh has produced the xcframework). +// A default build contains none of this code. +#if CLOUD_SYNC && LITERS +import Foundation + +/// The host HTTP transport liters runs every `Storage.http` request through. +/// +/// ## Why this file has to exist at all +/// +/// `LitersWriter(dbPath:storage:)` without a host client uses liters' built-in socket transport, +/// which is **`http://` only** — `HttpReplicaClient::with_options` *rejects* an `https://` URL +/// rather than silently downgrading it. `noop-cloud` is HTTPS. So on this app the choice is not +/// "nicer transport vs. built-in"; without this class the liters path cannot reach the server at +/// all. +/// +/// ## The shape liters asks for, and why it is blocking +/// +/// `execute` is synchronous by contract: liters calls it from its own worker while holding the +/// writer lock, and expects a response object it then *pulls* bytes from. That is deliberate — it +/// is what lets a 300 MB snapshot push stream instead of buffering. The bridge here is therefore a +/// `DispatchSemaphore` around a `URLSession` task, and it is a hard error to call it on the main +/// thread: `CloudSyncUploader` only ever reaches it from a `Task`'s background executor, and the +/// `precondition` makes a future misuse fail loudly rather than deadlock the UI. +/// +/// ## What this is NOT +/// +/// Not a background `URLSession`. A background session delivers completion through the app +/// delegate, minutes later, to a process that may have been relaunched — there is no way to return +/// that from a synchronous `execute`, and pretending otherwise would be worse than not having it. +/// The consequence is stated plainly: an upload in flight when iOS suspends the app is lost, and +/// the next push resumes from the last committed TXID. **That is exactly today's `/ingest` +/// behaviour** (`CloudSyncClient` also uses a foreground session), so this path is not a regression +/// — but it is the reason `liters`' own docs push you toward a background session, and it is the +/// first thing to revisit if pushes start dying at suspend. +final class LitersURLSessionClient: HttpClient, @unchecked Sendable { + /// One instance per process, one `URLSession` inside it, so every request to the server + /// coalesces onto one connection — which is what liters' `HttpClient` doc asks for. + static let shared = LitersURLSessionClient() + + /// Generous, because a snapshot push is the whole database. `timeoutIntervalForRequest` is an + /// *inactivity* timeout, not a total-transfer one, so this does not cap a slow large upload; + /// `timeoutIntervalForResource` does, and 1 hour is the ceiling on one push attempt. + private let session: URLSession + + init(session: URLSession? = nil) { + if let session { + self.session = session + } else { + let cfg = URLSessionConfiguration.default + cfg.timeoutIntervalForRequest = 120 + cfg.timeoutIntervalForResource = 3600 + cfg.httpMaximumConnectionsPerHost = 2 + cfg.waitsForConnectivity = true + self.session = URLSession(configuration: cfg) + } + } + + func execute(request: HttpRequest, body: HttpRequestBody?) throws -> HttpResponse { + precondition(!Thread.isMainThread, + "LitersURLSessionClient.execute blocks; it must never run on the main thread") + + guard let url = URL(string: request.url) else { + // `HttpError.Protocol(...)` will not parse — Swift reads `Type.Protocol` as the + // metatype syntax regardless of backticks — so the case is reached through a + // leading-dot member reference on an explicitly typed value. + let e: HttpError = .Protocol(message: "liters produced an unparseable URL: \(request.url)") + throw e + } + var req = URLRequest(url: url) + req.httpMethod = request.method + for h in request.headers { req.setValue(h.value, forHTTPHeaderField: h.name) } + + // A PUT body is pulled from liters, never handed over whole: `HttpRequestBody.read(max:)` + // is the source and a 300 MB snapshot must not be materialised as `Data`. Bound streams + // give `URLSession` an `InputStream` while a producer thread fills the other end. + if let body { + req.httpBodyStream = Self.stream(pulling: body) + } + + let sem = DispatchSemaphore(value: 0) + var out: (data: Data, response: HTTPURLResponse)? + var failure: Error? + + let task = session.dataTask(with: req) { data, response, error in + if let error { + failure = error + } else if let http = response as? HTTPURLResponse { + out = (data ?? Data(), http) + } else { + let e: HttpError = .Protocol(message: "no HTTP response") + failure = e + } + sem.signal() + } + task.resume() + sem.wait() + + if let failure { + // Transport, not Protocol: liters treats Transport as retryable, and a dropped + // connection or a suspended radio is exactly that. A push is idempotent by key, so a + // retry of a half-sent PUT is safe. + throw HttpError.Transport(message: (failure as NSError).localizedDescription) + } + guard let out else { throw HttpError.Transport(message: "no response and no error") } + + var headers: [HttpHeader] = [] + for (k, v) in out.response.allHeaderFields { + guard let name = k as? String, let value = v as? String else { continue } + // `content-length` and `content-encoding` are rewritten below, not forwarded. URLSession + // owns the transfer: it transparently negotiates and undoes `Content-Encoding`, and it + // does not reliably surface `Content-Length` at all — an HTTP/2 response, or one it + // decompressed, arrives with the header missing or describing the *compressed* size. + // Either way the number would not describe `out.data`. + // + // This is not cosmetic: liters' client requires a length on a listing response and + // fails the whole push with "storage: list level 0: missing content-length" without + // one. Found by running it — the first real push through this class died here. + let lower = name.lowercased() + if lower == "content-length" || lower == "content-encoding" { continue } + headers.append(HttpHeader(name: name, value: value)) + } + // The authoritative length: what the caller will actually be able to read. + headers.append(HttpHeader(name: "content-length", value: String(out.data.count))) + return BufferedHTTPResponse(status: UInt16(out.response.statusCode), + headerList: headers, + body: out.data) + } + + /// Bridges liters' pull-based body into the push-based `InputStream` `URLSession` wants. + /// + /// `Stream.getBoundStreams` gives a connected pair with a fixed-size kernel-ish buffer; the + /// producer thread blocks on `write` when the consumer is behind, which is the backpressure + /// that keeps memory flat regardless of push size. + private static func stream(pulling body: HttpRequestBody) -> InputStream { + var input: InputStream? + var output: OutputStream? + Stream.getBoundStreams(withBufferSize: 65_536, inputStream: &input, outputStream: &output) + guard let input, let output else { + // Cannot happen; a zero-byte body is still a legal body and lets the PUT fail on the + // server's terms rather than crashing the app here. + return InputStream(data: Data()) + } + let pump = Thread { + output.open() + defer { output.close() } + while true { + let chunk: Data + do { + chunk = try body.read(max: 65_536) + } catch { + break // aborts the upload; URLSession sees a truncated body and errors + } + if chunk.isEmpty { break } // documented end-of-body + var sent = 0 + chunk.withUnsafeBytes { (raw: UnsafeRawBufferPointer) in + guard let base = raw.bindMemory(to: UInt8.self).baseAddress else { return } + while sent < chunk.count { + let n = output.write(base + sent, maxLength: chunk.count - sent) + if n <= 0 { sent = chunk.count; return } // consumer went away + sent += n + } + } + } + } + pump.name = "liters-body-pump" + pump.stackSize = 512 * 1024 + pump.start() + return input + } +} + +/// liters' responses on the push path are small (a status line or a short JSON error), so the whole +/// body is collected and then served through the pull interface. The long-lived `/stream` follow — +/// the one case where buffering would be wrong — is a *Replica* concern; this app only pushes. +private final class BufferedHTTPResponse: HttpResponse, @unchecked Sendable { + private let statusCode: UInt16 + private let headerList: [HttpHeader] + private var body: Data + private let lock = NSLock() + + init(status: UInt16, headerList: [HttpHeader], body: Data) { + self.statusCode = status + self.headerList = headerList + self.body = body + } + + func status() -> UInt16 { statusCode } + func headers() -> [HttpHeader] { headerList } + + func readBody(max: UInt32) throws -> BodyChunk { + lock.lock() + defer { lock.unlock() } + if body.isEmpty { return .eof } + let n = Swift.min(Int(max), body.count) + let chunk = body.prefix(n) + body = body.dropFirst(n) + return .data(bytes: Data(chunk)) + } + + /// Nothing to abort: the transfer already completed before this object existed. + func cancel() {} +} +#endif // CLOUD_SYNC && LITERS diff --git a/Strand/CloudSync/LitersReplicator.swift b/Strand/CloudSync/LitersReplicator.swift new file mode 100644 index 0000000000..ecf9148579 --- /dev/null +++ b/Strand/CloudSync/LitersReplicator.swift @@ -0,0 +1,116 @@ +// Compiled ONLY when the CLOUD_SYNC compilation condition is set (by the untracked +// OuraSecrets.xcconfig) AND LITERS is set (by the generated Config/LitersLocal.xcconfig, i.e. only +// after Rust/build-ios.sh has produced the xcframework). A default build contains none of this code. +#if CLOUD_SYNC && LITERS +import Foundation + +/// Owns the one `LitersWriter` for the app's whole lifetime and performs one push per sync. +/// +/// ## Why this is a process-lifetime singleton and not a per-push object +/// +/// This is the single most load-bearing decision in the liters path, and it was measured rather +/// than assumed. `liters::Writer` resumes an incremental push from a WAL offset it recorded on the +/// previous push. Dropping the writer releases the WAL-pinning read lock, and SQLite checkpoints +/// and truncates the WAL when the *last* connection to a database closes — `wal_autocheckpoint = 0` +/// does not prevent that, it only stops the threshold-triggered one. The next open therefore finds +/// a WAL whose header salt has been reset and can only recover by shipping the whole database. +/// +/// Measured on VK's real 782 MB database against a real `noop-cloud` `/liters` endpoint +/// (2026-07-27): a writer opened and dropped around each push snapshotted **every single time** — +/// 5 pushes, 5 snapshots, ~312 MB each. The same database with one writer held across pushes +/// snapshotted once (the first, which has to) and then ran incremental: 151–176 KB per 10-minute +/// delta, ~2.0 MB per 4-hour delta. That is the difference between page replication and a full +/// upload with extra steps, and it is entirely a function of writer lifetime. +/// +/// So: opened lazily on the first push, kept, and never closed except by `reset()` — which exists +/// for the recovery path, not for routine use. +/// +/// ## Threading +/// +/// `LitersWriter`'s methods are synchronous and block. `push()` is called from a `Task`'s +/// background executor by `CloudSyncUploader`, and serialized here by an `NSLock` because the FFI +/// object's own lock would otherwise queue a second push behind an in-flight one with no timeout of +/// ours. One sync at a time is already guaranteed upstream by `CloudSyncGate`; the lock is a +/// belt-and-braces against a future second caller. +final class LitersReplicator: @unchecked Sendable { + static let shared = LitersReplicator() + + private let lock = NSLock() + private var writer: LitersWriter? + /// The endpoint the live writer was opened against. A settings change (new server or rotated + /// token) has to rebuild the writer, or pushes would keep going to the old destination. + private var openedFor: String? + + private init() {} + + /// `/liters` — the mount `noop-cloud`'s Express proxy serves. Everything after it is + /// liters' own grammar (`/ltx/0/{min}-{max}.ltx`) and is not this app's business. + static func endpoint(base: URL) -> String { + base.appendingPathComponent("liters").absoluteString + } + + /// One push. Returns liters' own summary so the caller records the real numbers rather than a + /// guess. Throws `LitersError` — the caller is expected to fall back to `/ingest`. + func push(databasePath: String, endpoint: String, token: String) throws -> PushSummary { + lock.lock() + defer { lock.unlock() } + + if openedFor != endpoint { closeLocked() } + if writer == nil { + writer = try LitersWriter.newWithHttpClient( + dbPath: databasePath, + storage: .http(url: endpoint, authToken: token), + httpClient: LitersURLSessionClient.shared) + openedFor = endpoint + } + guard let writer else { throw LitersError.Other(message: "writer unavailable") } + return try writer.push() + } + + /// **`maintain()` is deliberately never called from the push path**, and that is a decision, + /// not an omission. + /// + /// liters' `maintain()` runs compaction, retention *and* snapshotting on litestream's default + /// cadence — `snapshot_interval` is 24 h — and `LitersWriter.maintain()` hardcodes + /// `MaintenanceOptions::default()`, so the interval is not reachable from Swift. Measured on + /// the real 782 MB database (2026-07-27): one `maintain()` run took **177 s** and uploaded a + /// fresh **312 MB** snapshot. A `BGAppRefreshTask` gets tens of seconds, so this cannot run + /// there at all; and calling it daily would put a 312 MB upload back into a scheme whose entire + /// point is not doing 165 MB uploads. + /// + /// The cost of not calling it is bucket growth, and it is small and bounded on this + /// deployment: the bucket is a transport, not an archive — the server keeps a live + /// `mirror.sqlite` and only ever applies files forward. One base snapshot (~312 MB) plus + /// ~2 MB per 4-hour sync is ~12 MB/day, i.e. ~4.7 GB after a year on a 10.5 GB volume, and it + /// is swept server-side rather than paid for on the phone's radio. Revisit only if a *fresh* + /// restore (as opposed to incremental follow) ever becomes a normal operation. + + /// Whether a writer is currently open. Diagnostics only — a `false` here after a successful + /// sync means something closed it, which is the state that costs a snapshot. + var isOpen: Bool { + lock.lock(); defer { lock.unlock() } + return writer != nil + } + + /// Recovery for liters' one non-transient local error ("local ltx file missing or corrupt"): + /// wipes local replication state so the next push re-derives everything from the bucket. The + /// database content is untouched. Costs one snapshot. + func resetLocalState() { + lock.lock(); defer { lock.unlock() } + try? writer?.resetLocal() + } + + /// Drops the writer, releasing the WAL read lock and every fd. **Costs a snapshot on the next + /// push** (see the type doc), so this is for teardown and endpoint changes, not for tidiness. + func close() { + lock.lock(); defer { lock.unlock() } + closeLocked() + } + + private func closeLocked() { + writer?.close() + writer = nil + openedFor = nil + } +} +#endif // CLOUD_SYNC && LITERS From d69ffbb5a76947d5b6edfdeb606b2fcae5a606ee Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 28 Jul 2026 10:11:55 -0400 Subject: [PATCH 145/171] fix(liters): stop a failing push from being indistinguishable from one that never ran MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The trial reported healthy on both ends while every push failed. The server said applies=0 with an empty bucket; the phone said the flag was on and /ingest was fine. Nothing on either side named the failure, because litersPushIfEnabled's catch was write-only: it NSLogged the error and returned nil, and an NSLog on a phone that is not attached to Xcode is not readable. A push that fails and a push that never happens leave byte-identical evidence — no telemetry record (record() only runs on success), an empty bucket, and a healthy /ingest. Persist the outcome of every branch to UserDefaults, error verbatim, and show it in the Test Centre card next to the three facts that were already there. Those three describe intent, reality and result-so-far, and all three read healthy in exactly this failure. Also stop treating a push that shipped no files as a completed sync. push() does not throw for uploaded == 0, and neither synced nor uploaded was read: such a push would have returned success, skipped /ingest, and stamped lastUploadToken so the NEXT sync skipped its upload too. It now falls through to /ingest like every other way of not pushing. /ingest is untouched and remains the fallback. --- Strand/CloudSync/CloudSyncUploader.swift | 44 ++++++++++++++++++++- Strand/CloudSync/SyncReplicationTrial.swift | 9 ++++- 2 files changed, 51 insertions(+), 2 deletions(-) diff --git a/Strand/CloudSync/CloudSyncUploader.swift b/Strand/CloudSync/CloudSyncUploader.swift index 58d663a13e..d61d42cfd1 100644 --- a/Strand/CloudSync/CloudSyncUploader.swift +++ b/Strand/CloudSync/CloudSyncUploader.swift @@ -183,6 +183,24 @@ enum CloudSyncUploader { /// `latestDay` is `nil` on this path by construction: `/liters` is a byte pipe into /// `mirror.sqlite` and answers with liters' protocol, not with `/ingest`'s /// `{ok,bytes,latestDay}`. `CloudSyncModel` uses only `bytes`. + /// `UserDefaults` key carrying the outcome of the LAST liters attempt — which branch was taken + /// and, on a failure, the error verbatim. + /// + /// Not decoration. Every non-push outcome below is otherwise reported only through `NSLog`, and + /// an `NSLog` on a phone that is not attached to Xcode goes nowhere a person can read: a trial + /// that is failing every push and a trial that was never entered produce byte-identical evidence + /// (no telemetry record, an empty bucket, and a perfectly healthy `/ingest`). That ambiguity cost + /// a full debugging session — the server said `applies: 0`, the phone said the flag was on, and + /// nothing anywhere named the actual failure. Mirrors + /// `CloudSyncAppDelegate.registrationBreadcrumbKey`, which exists for the same reason. + static let litersBreadcrumbKey = "cloudsync.liters.lastOutcome" + + /// Records one liters outcome. Best-effort and never able to fail a sync, like every other + /// breadcrumb on this lane. + private static func noteLiters(_ outcome: String) { + UserDefaults.standard.set("\(outcome) \(Date())", forKey: litersBreadcrumbKey) + } + private static func litersPushIfEnabled( client: any CloudIngesting, walBytes: Int64, telemetry: SyncPushTelemetry? ) async -> (bytes: Int, latestDay: String?)? { @@ -194,9 +212,13 @@ enum CloudSyncUploader { guard SyncReplicationTrial.isInForce else { NSLog("liters: trial is on but WAL checkpointing is still .automatic " + "(restart pending) — using /ingest for this sync") + noteLiters("skipped: not-in-force (restart pending)") + return nil + } + guard let dest = client.litersDestination else { + noteLiters("skipped: no liters destination") return nil } - guard let dest = client.litersDestination else { return nil } do { // The one real on-disk database, resolved exactly the way `DataBackup`/`FolderBackup` @@ -211,6 +233,21 @@ enum CloudSyncUploader { token: dest.token) }.value + // A push that captured content into an L0 file but shipped NO file is not a completed + // sync, and `push()` does not throw for it — `synced`/`uploaded` are separate fields and + // the old code read neither. Returning here on `uploaded == 0` would report a sync that + // delivered nothing as a success, skip `/ingest`, and stamp `lastUploadToken` so the NEXT + // sync skips the upload too. Fall through to `/ingest` instead, which is the same answer + // this function gives every other way of not pushing. + guard summary.uploaded > 0 else { + NSLog("liters: push shipped no files (txid=%llu synced=%d remoteTxid=%llu) " + + "— using /ingest for this sync", summary.txid, summary.synced ? 1 : 0, + summary.remoteTxid) + noteLiters("no-op: uploaded=0 synced=\(summary.synced) txid=\(summary.txid) " + + "remoteTxid=\(summary.remoteTxid)") + return nil + } + telemetry?.record(snapshotted: summary.snapshotted, snapshotReason: summary.snapshotReason, bytesUploaded: Int64(summary.bytesUploaded), @@ -219,6 +256,8 @@ enum CloudSyncUploader { NSLog("liters: pushed txid=%llu bytes=%llu snapshotted=%d reason=%@ walAtPush=%lld", summary.txid, summary.bytesUploaded, summary.snapshotted ? 1 : 0, summary.snapshotReason ?? "-", walBytes) + noteLiters("pushed: files=\(summary.uploaded) bytes=\(summary.bytesUploaded) " + + "txid=\(summary.txid) snapshotted=\(summary.snapshotted)") return (Int(summary.bytesUploaded), nil) } catch { // Deliberately swallowed. Every liters failure mode — an unreachable sink (503 from @@ -227,6 +266,9 @@ enum CloudSyncUploader { // that ships data on the old path beats a sync that reports a new-path error. NSLog("liters: push failed (%@) — falling back to /ingest for this sync", String(describing: error)) + // Verbatim, not a category: the whole point is that the next person to look does not + // have to guess which of liters' failure modes this was. + noteLiters("failed: \(error)") return nil } #else diff --git a/Strand/CloudSync/SyncReplicationTrial.swift b/Strand/CloudSync/SyncReplicationTrial.swift index af33014b86..d1f460cdea 100644 --- a/Strand/CloudSync/SyncReplicationTrial.swift +++ b/Strand/CloudSync/SyncReplicationTrial.swift @@ -103,7 +103,14 @@ enum SyncReplicationTrial { } let late = StoreReplication.configuredAfterFirstOpen ? " LATE-CONFIGURE" : "" let telemetry = shared.map { " " + $0.oneLineSummary } ?? "" - return "replicationTrial enabled=\(isEnabled) checkpointing=\(inForce)\(late)\(telemetry)" + // The FOURTH fact, and the one that was missing: what the last attempt actually did. The + // three above describe intent, reality and result-so-far, and all three read "healthy" while + // every push fails — a failing push records no telemetry at all, so `pushes=0` means both + // "never tried" and "tried and failed every time". See + // `CloudSyncUploader.litersBreadcrumbKey`. + let last = UserDefaults.standard.string(forKey: CloudSyncUploader.litersBreadcrumbKey) + let outcome = last.map { "\nlast liters attempt: \($0)" } ?? "\nlast liters attempt: none recorded" + return "replicationTrial enabled=\(isEnabled) checkpointing=\(inForce)\(late)\(telemetry)\(outcome)" } } #endif From 5ad764c2eae4a27e04eb0d681ad098f6a547f4cc Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 28 Jul 2026 10:14:36 -0400 Subject: [PATCH 146/171] perf(motion): make Low Power Mode mean "stop everything decorative" (#909) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit inside `Strand/Liquid` and so could only ever reach the liquid layer. Two things it did not reach: **1. The tilt sensor kept running.** `LiquidMotion` starts `CMMotionManager` device motion at **60 Hz** — accelerometer + gyro sensor fusion — to drive the decorative slosh, the "the bubbles shift when you tilt the phone" effect. #909 poses the Canvas still, and because `acquire()` sits on the animated branch the sensor did stop when a gauge posed. But `onDisappear` is NOT called when the app is backgrounded, and NOOP declares `bluetooth-central` + `location` background modes, so the app keeps RUNNING behind the lock screen while a strap is connected. The 60 Hz fusion kept being delivered all day, for a picture nobody could see. `LiquidMotion` now stops at the app boundary and re-arms on foreground, and `startIfWanted()` consults the gate itself rather than trusting every call site to route around it. **2. Everything outside the liquid layer.** A census (`QuietMotionCoverageTests`, the Apple twin of #911's `PoseStillCoverageTest`) found five more never-settling loops, none of which #909 covered: - `TimeOfDayBackground` atmosphere drift — the Android twin has consulted battery saver since #911; this side read Reduce Motion alone. - `LiveSessionView`'s guardian breath (`repeatForever`). - `ConnectionDot` (StatePill) and `PulseDot` (Components) halos — on screen for long stretches: a connected strap in Settings, a backfill on every scaffold. - `RecordingStatusLight`'s sync ring, which had **no motion gate at all** — it pulsed under system Reduce Motion too, which was already a bug, and it ran precisely while the strap was offloading history. (Exactly the bug #911 found in the Android ConnectionDot.) The steady accent dot still says "syncing"; only the loop stops. - The onboarding glows and the radar sweep. The gate moves out of `Strand/Liquid` and into `StrandDesign` as `NoopMotionState`, so the design system's own surfaces can reach it — the Android twin has been in `com.noop.ui.NoopMotion` since #911, read through `rememberPoseStill()`. Same shape: one process-wide monitor, one observer, never per-view (the primitives alone have dozens of call sites, and registering an observer as gauges scroll in and out would be churn introduced by the change that exists to remove per-frame work). **Three signals, OR-ed:** system Reduce Motion, Low Power Mode, and a new in-app "Reduce motion in NOOP" toggle (Settings → Appearance, default OFF). The third is the point of this change for the user who asked for it: he wants the app quiet without putting the whole phone in battery saver. Key `noop.quietMotion`, mirroring Kotlin `NoopPrefs.quietMotion`; live via `UserDefaults.didChangeNotification`, so flipping it stops the sensor and poses the surfaces without leaving the screen. iPhone 17 Pro simulator seeded with a real 746 MB store, Today idle, no touch input. Three 15 s CPU windows after the first-run analysis had fully plateaued (that takes ~340 s at ~110% of a core on a freshly seeded DB — measuring before it settles is what makes an idle number meaningless). Debug build. | Today idle | CPU (% of one core) | RSS | load1 | |---|---|---|---| | animating (nothing asks for quiet) | 18.9 / 19.8 / 18.9 | 192 MB | 3.8 | | **"Reduce motion in NOOP" ON** | **0.0 / 0.0 / 0.0** | 97-122 MB | 4.6 | | system Reduce Motion ON (the #909 path) | 0.0 / 0.0 / 0.0 | 157-182 MB | 4.2 | | animating, re-checked | 16.8 / 16.1 / 15.2 | 99-115 MB | 9.2-13.4 | The last row is the not-in-quiet behaviour check: still animating, unchanged. It reads lower than the first only because the machine was under 2-3x the load — per process CPU-seconds are depressed by contention, so the two animating rows bracket the real figure rather than either being it. Two honest limits on those numbers: - I could not toggle real Low Power Mode in the simulator. The Low Power row is the Reduce Motion route, which is the same `staticGauge` / `LiquidSkyStatic` / `paused:` branch low-power users take. What is inferred, not exercised, is only that iOS reports the flag — which `IOSDiagnostics` already relies on. - **The sensor saving is not in that table.** `isDeviceMotionAvailable` is false in the simulator, so the 60 Hz CoreMotion feed never ran in any of these runs. The interval it requests (`deviceMotionUpdateInterval = 1/60`) and the missing background stop are read off the code, not measured. On device it is additional to the figures above, not included in them. Scroll was not re-measured: the earlier investigation could not reproduce the user's jank in the simulator at all (zero `body` rebuilds during driven scrolls, 60 fps), and the simulator is 60 Hz where his iPhone 16 Pro is 120 Hz with an 8.33 ms budget. Any claim that this fixes scrolling would be simulator-only and unsupported. - One-shot animations (`CountUpText`, `staggeredAppear`, `LiquidPressStyle`). They settle and stop, so they are not the cost battery saver asks an app to avoid — same line #911 drew. - `TimelineView(.periodic…)` clocks. A 1 s elapsed-time label is not per-frame drawing. - The classic Today top-bar "Swipe / Tap" hint cycler. Two runloop wakeups per 11.5 s, and it carries a discoverability affordance rather than decoration — removing it would cost information, not motion. - `BreathingView` / `WatchBreatheView` run a 20 Hz `Timer.publish` even when not breathing. Real, but bounded by that screen being on-screen, and fixing it means restructuring the publisher rather than adding a gate. Left for its own change. `QuietMotionCoverageTests` passes here, and **fails against the ungated tree** — checked in both directions: removing the gate from `RecordingStatusLight` fails the loop census by file and line, and aliasing `poseStill` back to bare `reduceMotion` in `ConnectionDot` fails the breathe-call-site census (the call-site check requires the file to reach `NoopMotionState`, not merely to contain the token). It fails rather than skips when the repo root is unlocatable, and it pins that the gate still reads all three signals and stays live — losing one half is invisible, since the screen looks right in whichever mode still works. `swift build` clean for StrandDesign; `xcodebuild` clean for NOOPiOS (simulator) and the macOS Strand target; `Tools/i18n_audit.py --ci` green with the new Settings copy translated into all eight catalog locales. --- .../Sources/StrandDesign/Components.swift | 9 +- .../Sources/StrandDesign/NoopMotion.swift | 90 +++++++++ .../Sources/StrandDesign/StatePill.swift | 14 +- .../StrandDesign/TimeOfDayBackground.swift | 12 +- .../QuietMotionCoverageTests.swift | 179 ++++++++++++++++++ Strand/Liquid/LiquidCore.swift | 89 ++++++--- Strand/Liquid/LiquidPrimitives.swift | 13 +- Strand/Liquid/LiquidSky.swift | 8 +- Strand/Liquid/LiquidTodayView.swift | 11 +- Strand/Liquid/LiveSessionView.swift | 9 +- Strand/Onboarding/OnboardingWizard.swift | 28 ++- Strand/Resources/Localizable.xcstrings | 104 ++++++++++ Strand/Screens/SettingsView.swift | 20 ++ Strand/Screens/TodayView.swift | 15 +- 14 files changed, 538 insertions(+), 63 deletions(-) create mode 100644 Packages/StrandDesign/Tests/StrandDesignTests/QuietMotionCoverageTests.swift diff --git a/Packages/StrandDesign/Sources/StrandDesign/Components.swift b/Packages/StrandDesign/Sources/StrandDesign/Components.swift index 40215942f7..397dc9711c 100644 --- a/Packages/StrandDesign/Sources/StrandDesign/Components.swift +++ b/Packages/StrandDesign/Sources/StrandDesign/Components.swift @@ -658,6 +658,11 @@ private struct PulseDot: View { var size: CGFloat @State private var animate = false @Environment(\.accessibilityReduceMotion) private var reduceMotion + /// Low Power Mode / "Reduce motion in NOOP". This halo is a `repeatForever` loop that never + /// settles and is on screen for long stretches — a connected strap in Settings, a backfill on + /// every scaffolded screen — so it belongs behind the same gate as the liquid surfaces. + @ObservedObject private var motion = NoopMotionState.shared + private var poseStill: Bool { motion.poseStill(reduceMotion) } @Environment(\.colorScheme) private var scheme var body: some View { ZStack { @@ -679,8 +684,8 @@ private struct PulseDot: View { .shadow(color: color.opacity(0.8), radius: pulsing ? 4 : 2) } .frame(width: size, height: size) - .onAppear { if pulsing && !reduceMotion { animate = true } } - .animation(pulsing && !reduceMotion ? StrandMotion.breathe : nil, value: animate) + .onAppear { if pulsing && !poseStill { animate = true } } + .animation(pulsing && !poseStill ? StrandMotion.breathe : nil, value: animate) .accessibilityHidden(true) } } diff --git a/Packages/StrandDesign/Sources/StrandDesign/NoopMotion.swift b/Packages/StrandDesign/Sources/StrandDesign/NoopMotion.swift index 5bd46ab2ac..2860ff5e9d 100644 --- a/Packages/StrandDesign/Sources/StrandDesign/NoopMotion.swift +++ b/Packages/StrandDesign/Sources/StrandDesign/NoopMotion.swift @@ -56,6 +56,96 @@ public enum NoopMotion { } } +// MARK: - Quiet Motion — the ONE gate every never-settling animation consults +// +// #909 taught the live gauges to pose still in Low Power Mode, but it read the flag inside +// `Strand/Liquid` and so could only reach the liquid layer. Everything else that never settles — +// the day-cycle atmosphere drift, the guardian breath, the connection halo — kept running, and a +// user in Low Power Mode still paid for it. The Android half (#911) had already put its gate in +// `com.noop.ui.NoopMotion`, one process-wide monitor read through `rememberPoseStill()`; this is +// that same shape on Apple, moved down into StrandDesign so the design system's own surfaces can +// reach it too (`LiquidPowerMonitor` in the app target could not). +// +// Three signals, OR-ed: +// 1. `@Environment(\.accessibilityReduceMotion)` — the system-wide setting. Supplied by the call +// site, because only a View can read the environment. +// 2. Low Power Mode — the OS-level "stop discretionary work" signal. +// 3. "Reduce motion in NOOP" — an in-app preference, default OFF, for people who want the app +// quiet without putting the whole phone in battery saver. + +/// "Reduce motion in NOOP" (opt-in, default OFF): pose every looping animation still and stop the +/// decorative motion sensor, without requiring system Low Power Mode or system Reduce Motion. +/// Toggled from Settings → Appearance. Mirror in Kotlin via `NoopPrefs.quietMotion`. +public enum QuietMotionPrefs { + /// The `@AppStorage` / `UserDefaults` key shared by the Settings toggle and `NoopMotionState`. + public static let enabledKey = "noop.quietMotion" +} + +/// Publishes the two motion signals that have no SwiftUI environment key — Low Power Mode and the +/// in-app "Reduce motion in NOOP" preference — so any view can pose its looping animation still. +/// +/// A singleton with ONE `NotificationCenter` observer rather than a per-view `DisposableEffect`, +/// for the reason #911 gives on the Android side: the liquid primitives alone have dozens of call +/// sites, and registering/unregistering an observer as gauges scroll in and out of view would be +/// churn introduced by the very change that exists to remove per-frame work. +/// +/// Worth doing because a continuously-animating `Canvas` is not free: measured on an iPhone 17 Pro +/// simulator seeded with a real 746 MB store, the default Today screen sitting idle costs ~18% of a +/// CPU core in BOTH Debug and Release, and 0.0% once the live surfaces pose still. +@MainActor +public final class NoopMotionState: ObservableObject { + public static let shared = NoopMotionState() + + /// System Low Power Mode / battery saver. Live: `.NSProcessInfoPowerStateDidChange` means + /// toggling the setting takes effect without a relaunch. + @Published public private(set) var isLowPower: Bool + + /// The in-app "Reduce motion in NOOP" preference. Kept in step with `UserDefaults` so a + /// non-SwiftUI reader (the motion sensor) and the `@AppStorage` toggle never disagree. + @Published public private(set) var quietMotion: Bool + + private init() { + isLowPower = ProcessInfo.processInfo.isLowPowerModeEnabled + quietMotion = UserDefaults.standard.bool(forKey: QuietMotionPrefs.enabledKey) + NotificationCenter.default.addObserver( + forName: .NSProcessInfoPowerStateDidChange, object: nil, queue: .main + ) { [weak self] _ in + MainActor.assumeIsolated { + self?.isLowPower = ProcessInfo.processInfo.isLowPowerModeEnabled + } + } + // `@AppStorage` writes straight to UserDefaults without telling us, so mirror the store. + // `.didChangeNotification` is the only signal that covers a write from any target. + NotificationCenter.default.addObserver( + forName: UserDefaults.didChangeNotification, object: UserDefaults.standard, queue: .main + ) { [weak self] _ in + MainActor.assumeIsolated { + let now = UserDefaults.standard.bool(forKey: QuietMotionPrefs.enabledKey) + if self?.quietMotion != now { self?.quietMotion = now } + } + } + } + + /// The gate. `reduceMotion` comes from `@Environment(\.accessibilityReduceMotion)` at the call + /// site — the environment is the only place SwiftUI publishes it, and reading it imperatively + /// would not invalidate the view when the user changes the setting. + /// + /// ```swift + /// @Environment(\.accessibilityReduceMotion) private var reduceMotion + /// @ObservedObject private var motion = NoopMotionState.shared + /// private var poseStill: Bool { motion.poseStill(reduceMotion) } + /// ``` + @inline(__always) + public func poseStill(_ reduceMotion: Bool) -> Bool { + reduceMotion || isLowPower || quietMotion + } + + /// The two non-environment signals on their own, for an imperative (non-View) reader that + /// supplies its own Reduce Motion read — e.g. the decorative motion sensor deciding whether to + /// start at all. Views must use `poseStill(_:)` instead so they invalidate correctly. + public var poseStillIgnoringReduceMotion: Bool { isLowPower || quietMotion } +} + // MARK: - CountUpText // // Animates a numeric value counting up (or down) to its latest value whenever `value` diff --git a/Packages/StrandDesign/Sources/StrandDesign/StatePill.swift b/Packages/StrandDesign/Sources/StrandDesign/StatePill.swift index 9fd961d735..85fdc2cc91 100644 --- a/Packages/StrandDesign/Sources/StrandDesign/StatePill.swift +++ b/Packages/StrandDesign/Sources/StrandDesign/StatePill.swift @@ -77,6 +77,11 @@ public struct ConnectionDot: View { @State private var animate = false @Environment(\.accessibilityReduceMotion) private var reduceMotion + /// Low Power Mode / "Reduce motion in NOOP". This halo is a `repeatForever` loop that never + /// settles and is on screen for long stretches — a connected strap in Settings, a backfill on + /// every scaffolded screen — so it belongs behind the same gate as the liquid surfaces. + @ObservedObject private var motion = NoopMotionState.shared + private var poseStill: Bool { motion.poseStill(reduceMotion) } @Environment(\.colorScheme) private var scheme public init(tone: StrandTone = .positive, pulsing: Bool = false, size: CGFloat = 9) { @@ -107,10 +112,11 @@ public struct ConnectionDot: View { .shadow(color: tone.color.opacity(0.8), radius: pulsing ? 4 : 2) } .frame(width: size, height: size) - // Honour Reduce Motion: don't kick off the looping pulse (settles at the - // resting dot) and never attach the repeatForever breathe animation. - .onAppear { if pulsing && !reduceMotion { animate = true } } - .animation(pulsing && !reduceMotion ? StrandMotion.breathe : nil, value: animate) + // Honour the quiet-motion gate (system Reduce Motion, Low Power Mode, or the in-app + // toggle): don't kick off the looping pulse (it settles at the resting dot) and never + // attach the repeatForever breathe animation. + .onAppear { if pulsing && !poseStill { animate = true } } + .animation(pulsing && !poseStill ? StrandMotion.breathe : nil, value: animate) .accessibilityHidden(true) } } diff --git a/Packages/StrandDesign/Sources/StrandDesign/TimeOfDayBackground.swift b/Packages/StrandDesign/Sources/StrandDesign/TimeOfDayBackground.swift index a8882bc8be..4819cb83a9 100644 --- a/Packages/StrandDesign/Sources/StrandDesign/TimeOfDayBackground.swift +++ b/Packages/StrandDesign/Sources/StrandDesign/TimeOfDayBackground.swift @@ -11,7 +11,8 @@ import SwiftUI // very low opacity; stars are tiny crisp dots. Beauty = restraint, spacing, type, motion. // - TOKENS first (`StrandPalette`). The only literal hexes are the few subtle atmosphere // tints the spec calls for (warm peach lift, indigo wash, etc.) — kept deliberately faint. -// - Reduce Motion pins every drifting element still (no looping translation). +// - Reduce Motion — and Low Power Mode, and the in-app "Reduce motion in NOOP" toggle — +// pin every drifting element still (no looping translation, and the frame loop is `paused:`). // - Light mode is even MORE restrained: warm-paper tints, fewer/softer elements. // - CPU-light: drift runs off a single `TimelineView(.animation)` tick that the system // pauses when the view is off-screen; no per-frame allocation, no timers we own. @@ -52,14 +53,19 @@ public struct TimeOfDayBackground: View { @Environment(\.accessibilityReduceMotion) private var reduceMotion @Environment(\.colorScheme) private var colorScheme + /// Low Power Mode and the in-app "Reduce motion in NOOP" toggle, neither of which has a SwiftUI + /// environment key. The Android twin (`TimeOfDayBackground.kt`) has consulted battery saver since + /// #911; this side was left reading Reduce Motion alone. + @ObservedObject private var motion = NoopMotionState.shared public init(dayPart: DayPart, animated: Bool = true) { self.dayPart = dayPart self.animated = animated } - /// Whether drifting elements should actually move (caller opted in AND Reduce Motion is off). - private var drift: Bool { animated && !reduceMotion } + /// Whether drifting elements should actually move: the caller opted in AND nothing is asking + /// for quiet (system Reduce Motion, Low Power Mode, or the in-app toggle). + private var drift: Bool { animated && !motion.poseStill(reduceMotion) } public var body: some View { GeometryReader { geo in diff --git a/Packages/StrandDesign/Tests/StrandDesignTests/QuietMotionCoverageTests.swift b/Packages/StrandDesign/Tests/StrandDesignTests/QuietMotionCoverageTests.swift new file mode 100644 index 0000000000..55c4286e65 --- /dev/null +++ b/Packages/StrandDesign/Tests/StrandDesignTests/QuietMotionCoverageTests.swift @@ -0,0 +1,179 @@ +import XCTest +@testable import StrandDesign + +/// A source census: every never-settling animation in the Apple UI must consult the quiet-motion gate. +/// +/// This is the Apple twin of Android's `PoseStillCoverageTest` (#911), and it exists for the reason +/// that test gives: a survey done by eye misses surfaces. #909 gated the liquid layer and stopped +/// there, so the day-cycle atmosphere drift, the guardian breath, the two connection-dot halos and +/// the onboarding glows all kept looping in Low Power Mode — and `RecordingStatusLight` looped under +/// system Reduce Motion too, which was already a bug. None of that was visible in review. +/// +/// It also pins that the gate keeps reading all three signals. Losing one half is invisible: the +/// screen looks right in whichever mode still works. +/// +/// Lives in the StrandDesign package because `swift-packages.yml` runs it by default, while the app +/// targets' `StrandTests` only runs under `xcodebuild` (and `app-build.yml` is disabled). +final class QuietMotionCoverageTests: XCTestCase { + + /// Directories that ship Apple UI. `android/` has its own census in `PoseStillCoverageTest`. + private static let uiRoots = [ + "Strand", + "StrandiOS", + "StrandiOSShared", + "StrandiOSWidgets", + "Packages/StrandDesign/Sources", + "NOOPWatch", + "NOOPWatchComplications", + ] + + /// A loop that never settles: an indefinitely repeating implicit animation, or a per-frame + /// `TimelineView` clock. `TimelineView(.periodic…)` is deliberately NOT censused — a 1 s or 60 s + /// clock is a label ticking over, not the per-frame drawing this gate exists to stop. + private static let loopMarkers = ["repeatForever(", "TimelineView(.animation"] + + /// Files allowed to contain a marker without naming the gate, each with the reason. + private static let exemptions: [String: String] = [ + // Defines `StrandMotion.breathe` (and demonstrates it in a `#if DEBUG` preview). Every + // shipping call site of it is censused separately by `testBreatheCallSitesConsultTheGate`. + "Packages/StrandDesign/Sources/StrandDesign/Motion.swift": + "declares the breathe primitive; call sites are censused separately", + ] + + private func repoRoot() throws -> URL { + // .../Packages/StrandDesign/Tests/StrandDesignTests/QuietMotionCoverageTests.swift -> repo root + let root = URL(fileURLWithPath: #filePath) + .deletingLastPathComponent() // StrandDesignTests + .deletingLastPathComponent() // Tests + .deletingLastPathComponent() // StrandDesign + .deletingLastPathComponent() // Packages + .deletingLastPathComponent() // repo root + // FAIL rather than skip when the tree cannot be found: a census that silently passes because + // it censused nothing is worse than no census (the Android twin makes the same choice). + guard FileManager.default.fileExists(atPath: root.appendingPathComponent("project.yml").path) else { + throw Failure("repo root not found from #filePath (looked at \(root.path))") + } + return root + } + + private struct Failure: Error, CustomStringConvertible { + let description: String + init(_ d: String) { description = d } + } + + private func swiftFiles(under root: URL) -> [(rel: String, text: String)] { + var out: [(String, String)] = [] + for dir in Self.uiRoots { + let base = root.appendingPathComponent(dir) + guard let e = FileManager.default.enumerator(at: base, includingPropertiesForKeys: nil) else { continue } + for case let url as URL in e where url.pathExtension == "swift" { + guard let text = try? String(contentsOf: url, encoding: .utf8) else { continue } + let rel = url.path.replacingOccurrences(of: root.path + "/", with: "") + out.append((rel, text)) + } + } + return out + } + + /// Strip `//` line comments so a marker merely *described* in prose is not censused as code. + private func codeLines(_ text: String) -> [String] { + text.split(separator: "\n", omittingEmptySubsequences: false).map { line in + let s = String(line) + guard let r = s.range(of: "//") else { return s } + return String(s[s.startIndex.. 0, !LiquidMotion.quietNow else { return } started = true #if os(iOS) && !targetEnvironment(macCatalyst) - guard manager.isDeviceMotionAvailable else { return } + guard manager.isDeviceMotionAvailable else { started = false; return } manager.deviceMotionUpdateInterval = 1.0 / 60.0 manager.startDeviceMotionUpdates(to: motionQueue) { [weak self] motion, _ in guard let self, let m = motion else { return } @@ -125,13 +167,25 @@ final class LiquidMotion { func release() { refCount = max(0, refCount - 1) - guard refCount == 0, started else { return } + guard refCount == 0 else { return } + stop() + } + + /// Stop the sensor but KEEP `refCount` — the screen is still mounted, we simply don't want the + /// hardware running (backgrounded, or the user asked for quiet). `startIfWanted` resumes it. + private func stop() { + guard started else { return } started = false #if os(iOS) && !targetEnvironment(macCatalyst) manager.stopDeviceMotionUpdates() #endif tilt = 0 } + + /// Re-evaluate after a Low Power Mode / Reduce Motion / in-app-toggle change. + private func syncToPolicy() { + if LiquidMotion.quietNow { stop() } else { startIfWanted() } + } } // MARK: - The liquid simulation @@ -285,30 +339,3 @@ func liquidWave(_ x: Double, amp: Double, R: Double, hw: Double, /// A monotonic seconds clock for a TimelineView date. @inline(__always) func liquidSeconds(_ date: Date) -> Double { date.timeIntervalSinceReferenceDate } - -// MARK: - Low Power Mode - -/// Publishes Low Power Mode so the live gauges can pose still, the way they already do for -/// Reduce Motion. There is no SwiftUI environment key for this, so the primitives observe this -/// instead; `.NSProcessInfoPowerStateDidChange` makes the switch live, with no relaunch needed. -/// -/// Worth doing because a continuously-animating `Canvas` is not free: measured on an iPhone 17 Pro -/// simulator seeded with a real 746 MB store, the default Today screen sitting idle costs ~18% of a -/// CPU core in BOTH Debug and Release, and 0.0% once the live gauges pose still. Low Power Mode is -/// the user telling the system to stop exactly that kind of work. -@MainActor -final class LiquidPowerMonitor: ObservableObject { - static let shared = LiquidPowerMonitor() - @Published private(set) var isLowPower: Bool - - private init() { - isLowPower = ProcessInfo.processInfo.isLowPowerModeEnabled - NotificationCenter.default.addObserver( - forName: .NSProcessInfoPowerStateDidChange, object: nil, queue: .main - ) { [weak self] _ in - MainActor.assumeIsolated { - self?.isLowPower = ProcessInfo.processInfo.isLowPowerModeEnabled - } - } - } -} diff --git a/Strand/Liquid/LiquidPrimitives.swift b/Strand/Liquid/LiquidPrimitives.swift index 1a93b413a1..5aa72bcfa5 100644 --- a/Strand/Liquid/LiquidPrimitives.swift +++ b/Strand/Liquid/LiquidPrimitives.swift @@ -7,6 +7,7 @@ // one shared tilt source. Colours come from StrandDesign tokens at the call site. import SwiftUI +import StrandDesign // NoopMotionState — the shared quiet-motion gate // MARK: - Renderers (pure GraphicsContext drawing) @@ -221,7 +222,7 @@ struct LiquidVessel: View { var animated: Bool = true @Environment(\.accessibilityReduceMotion) private var reduceMotion - @ObservedObject private var power = LiquidPowerMonitor.shared + @ObservedObject private var motion = NoopMotionState.shared @State private var sim: LiquidSim @State private var splashes = 0 @@ -233,7 +234,7 @@ struct LiquidVessel: View { } var body: some View { - if animated && !reduceMotion && !power.isLowPower { gauge } else { staticGauge } + if animated && !motion.poseStill(reduceMotion) { gauge } else { staticGauge } } private var gauge: some View { @@ -276,11 +277,11 @@ struct LiquidTube: View { var animated: Bool = true @Environment(\.accessibilityReduceMotion) private var reduceMotion - @ObservedObject private var power = LiquidPowerMonitor.shared + @ObservedObject private var motion = NoopMotionState.shared @State private var sim = LiquidSim(target: 0) var body: some View { - if animated && !reduceMotion && !power.isLowPower { liveTube } else { staticTube } + if animated && !motion.poseStill(reduceMotion) { liveTube } else { staticTube } } private var liveTube: some View { @@ -313,10 +314,10 @@ struct LiquidThread: View { var animated: Bool = true @Environment(\.accessibilityReduceMotion) private var reduceMotion - @ObservedObject private var power = LiquidPowerMonitor.shared + @ObservedObject private var motion = NoopMotionState.shared var body: some View { - if animated && !reduceMotion && !power.isLowPower { liveThread } else { staticThread } + if animated && !motion.poseStill(reduceMotion) { liveThread } else { staticThread } } private var liveThread: some View { diff --git a/Strand/Liquid/LiquidSky.swift b/Strand/Liquid/LiquidSky.swift index 4d1d01963c..722c25f488 100644 --- a/Strand/Liquid/LiquidSky.swift +++ b/Strand/Liquid/LiquidSky.swift @@ -66,9 +66,15 @@ struct LiquidSky: View { /// the atmosphere so the sky still reads under a full-height "sky behind cards" backdrop). var settleStrength: Double = 1 @Environment(\.colorScheme) private var scheme + /// The call site already swaps in `LiquidSkyStatic` when motion is unwanted, but this view carried + /// no gate of its own — a second call site would have been silently ungated. `paused:` makes the + /// frame loop stand down from inside, so the gate travels with the view. + @Environment(\.accessibilityReduceMotion) private var reduceMotion + @ObservedObject private var motion = NoopMotionState.shared var body: some View { - TimelineView(.animation(minimumInterval: 1.0 / 20.0)) { tl in + TimelineView(.animation(minimumInterval: 1.0 / 20.0, + paused: motion.poseStill(reduceMotion))) { tl in let now = liquidSeconds(tl.date) let h = hour ?? liveHour() // The sky must dissolve into the SAME canvas colour the body uses (theme-aware surfaceBase), diff --git a/Strand/Liquid/LiquidTodayView.swift b/Strand/Liquid/LiquidTodayView.swift index 84cd8302a7..913750aa12 100644 --- a/Strand/Liquid/LiquidTodayView.swift +++ b/Strand/Liquid/LiquidTodayView.swift @@ -26,10 +26,11 @@ struct LiquidTodayView: View { // only publishes connect/discovery state, never HR. Injected at the app roots beside .environmentObject(model). @EnvironmentObject var ble: BLEManager @Environment(\.accessibilityReduceMotion) private var reduceMotion - /// Low Power Mode poses the sky still too — the behaviour the comment on the sky branch below - /// has always described. There is no environment key for it, hence the shared monitor. - @ObservedObject private var powerMonitor = LiquidPowerMonitor.shared - private var lowPower: Bool { powerMonitor.isLowPower } + /// Low Power Mode — and the in-app "Reduce motion in NOOP" toggle — pose the sky still too, the + /// behaviour the comment on the sky branch below has always described. Neither has a SwiftUI + /// environment key, hence the shared monitor. + @ObservedObject private var motion = NoopMotionState.shared + private var poseStill: Bool { motion.poseStill(reduceMotion) } /// Shared with the real Today's card-customise editor so the two stay in sync. @AppStorage(DashboardCardPrefs.selectionKey) private var dashboardCardsRaw = "" @@ -322,7 +323,7 @@ struct LiquidTodayView: View { // "Sky behind cards" (opt-in): fill the whole backdrop with a softer settle so the sky // reads under every card, instead of the default 340 top band that dissolves to canvas. Group { - if reduceMotion || lowPower || !dataLoaded { LiquidSkyStatic(hour: liveHour, settleStrength: skyBehindCards ? 0.78 : 1) } + if poseStill || !dataLoaded { LiquidSkyStatic(hour: liveHour, settleStrength: skyBehindCards ? 0.78 : 1) } else { LiquidSky(hour: liveHour, settleStrength: skyBehindCards ? 0.78 : 1) } } .frame(maxWidth: .infinity) diff --git a/Strand/Liquid/LiveSessionView.swift b/Strand/Liquid/LiveSessionView.swift index 2d204d4220..42a6d72377 100644 --- a/Strand/Liquid/LiveSessionView.swift +++ b/Strand/Liquid/LiveSessionView.swift @@ -21,6 +21,10 @@ struct LiveSessionView: View { @EnvironmentObject private var repo: Repository @EnvironmentObject private var profile: ProfileStore @Environment(\.accessibilityReduceMotion) private var reduceMotion + /// Low Power Mode / the in-app quiet-motion toggle. The guardian breath is a `repeatForever` + /// animation that never settles, so it belongs behind the same gate as the liquid surfaces. + /// The Android twin gated `LiveSessionScreen`'s breath under battery saver in #911. + @ObservedObject private var motion = NoopMotionState.shared /// "Card transparency" (0–100, default 100): fades the live-session cards in lockstep with the frosted /// cards; content stays readable. Mirrors Kotlin `NoopPrefs.cardOpacityPercent`. @@ -166,9 +170,10 @@ struct LiveSessionView: View { .accessibilityHint(Text("Long press to show or hide your heart rate.")) } - /// Breathing is the "on track" signal: only in band, only once active, never for Reduce Motion. + /// Breathing is the "on track" signal: only in band, only once active, never when anything is + /// asking for quiet (Reduce Motion, Low Power Mode, or "Reduce motion in NOOP"). private var isBreathing: Bool { - !reduceMotion + !motion.poseStill(reduceMotion) && runner.output?.status == .active && runner.output?.position == .inBand } diff --git a/Strand/Onboarding/OnboardingWizard.swift b/Strand/Onboarding/OnboardingWizard.swift index a7a353cb8e..112e25918d 100644 --- a/Strand/Onboarding/OnboardingWizard.swift +++ b/Strand/Onboarding/OnboardingWizard.swift @@ -47,6 +47,10 @@ public struct OnboardingWizard: View { @State private var step: Step = .welcome @State private var glow = false @Environment(\.accessibilityReduceMotion) private var reduceMotion + /// Low Power Mode / "Reduce motion in NOOP" pose these looping glows still too. Onboarding is + /// first-run only, but a `repeatForever` is a `repeatForever` wherever it lives. + @ObservedObject private var motion = NoopMotionState.shared + private var poseStill: Bool { motion.poseStill(reduceMotion) } public var body: some View { ZStack { @@ -91,7 +95,7 @@ public struct OnboardingWizard: View { .frame(maxWidth: .infinity, maxHeight: .infinity) .background(StrandPalette.surfaceBase.ignoresSafeArea()) // Reduce Motion: leave the ambient bloom at its resting frame (no breathing). - .onAppear { if !reduceMotion { glow = true } } + .onAppear { if !poseStill { glow = true } } // Isolated live observation — a hidden watcher slides Scan → celebration on bond // without subscribing the whole wizard to per-tick updates. .background(BondWatcher(onBonded: handleBond)) @@ -116,7 +120,7 @@ public struct OnboardingWizard: View { ) .blendMode(.plusLighter) .opacity(glow ? 0.4 : 0.28) - .animation(StrandMotion.breathe(reduced: reduceMotion), value: glow) + .animation(StrandMotion.breathe(reduced: poseStill), value: glow) .ignoresSafeArea() // A faint indigo wash from the top — instrument-grade depth. @@ -449,6 +453,10 @@ private struct ExpectationsStep: View { private struct BluetoothStep: View { @State private var pulse = false @Environment(\.accessibilityReduceMotion) private var reduceMotion + /// Low Power Mode / "Reduce motion in NOOP" pose these looping glows still too. Onboarding is + /// first-run only, but a `repeatForever` is a `repeatForever` wherever it lives. + @ObservedObject private var motion = NoopMotionState.shared + private var poseStill: Bool { motion.poseStill(reduceMotion) } var body: some View { StepShell(title: String(localized: "A quick word before we connect"), subtitle: String(localized: "\(Platform.deviceNoun) will ask for Bluetooth in a moment.")) { @@ -482,7 +490,7 @@ private struct BluetoothStep: View { .frame(maxWidth: 460) } } - .onAppear { if !reduceMotion { withAnimation(StrandMotion.breathe) { pulse = true } } } + .onAppear { if !poseStill { withAnimation(StrandMotion.breathe) { pulse = true } } } } } @@ -942,6 +950,10 @@ private struct ImportStep: View { private struct NotificationsStep: View { @State private var pulse = false @Environment(\.accessibilityReduceMotion) private var reduceMotion + /// Low Power Mode / "Reduce motion in NOOP" pose these looping glows still too. Onboarding is + /// first-run only, but a `repeatForever` is a `repeatForever` wherever it lives. + @ObservedObject private var motion = NoopMotionState.shared + private var poseStill: Bool { motion.poseStill(reduceMotion) } var body: some View { StepShell(title: String(localized: "Stay in the loop"), subtitle: String(localized: "NOOP can tap your wrist when your \(Platform.deviceNoun) needs you. No glance at the screen required.")) { @@ -994,7 +1006,7 @@ private struct NotificationsStep: View { #endif } } - .onAppear { if !reduceMotion { withAnimation(StrandMotion.breathe) { pulse = true } } } + .onAppear { if !poseStill { withAnimation(StrandMotion.breathe) { pulse = true } } } } } @@ -1115,6 +1127,10 @@ private struct RadarSweep: View { @State private var angle: Double = 0 @State private var ping = false @Environment(\.accessibilityReduceMotion) private var reduceMotion + /// Low Power Mode / "Reduce motion in NOOP" pose these looping glows still too. Onboarding is + /// first-run only, but a `repeatForever` is a `repeatForever` wherever it lives. + @ObservedObject private var motion = NoopMotionState.shared + private var poseStill: Bool { motion.poseStill(reduceMotion) } var body: some View { GeometryReader { geo in @@ -1165,7 +1181,7 @@ private struct RadarSweep: View { .onChangeCompat(of: active) { isActive in if isActive { startSweep() } } - .animation(StrandMotion.breathe(reduced: reduceMotion), value: ping) + .animation(StrandMotion.breathe(reduced: poseStill), value: ping) } private func sweepWedge(size: CGFloat) -> some View { @@ -1193,7 +1209,7 @@ private struct RadarSweep: View { private func startSweep() { // Reduce Motion: keep the wedge still (the static rings/crosshairs/blip // still convey "searching" / "found") instead of spinning forever. - guard !reduceMotion else { return } + guard !poseStill else { return } angle = 0 withAnimation(.linear(duration: 2.4).repeatForever(autoreverses: false)) { angle = 360 diff --git a/Strand/Resources/Localizable.xcstrings b/Strand/Resources/Localizable.xcstrings index 08009d944c..770b1b21d1 100644 --- a/Strand/Resources/Localizable.xcstrings +++ b/Strand/Resources/Localizable.xcstrings @@ -170772,6 +170772,110 @@ } } } + }, + "Reduce motion in NOOP": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Bewegung in NOOP reduzieren" + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Reducir movimiento en NOOP" + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Réduire les animations dans NOOP" + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Riduci movimento in NOOP" + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "Reduzir movimento no NOOP" + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Уменьшить анимацию в NOOP" + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "减少 NOOP 中的动态效果" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "減少 NOOP 中的動態效果" + } + } + } + }, + "Holds the liquid gauges, the sky and the tilt response still, and turns off the motion sensor that drives them. Saves battery. Low Power Mode and the system Reduce Motion setting already do this.": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Hält die Flüssigkeitsanzeigen, den Himmel und die Neigungsreaktion still und schaltet den Bewegungssensor ab, der sie antreibt. Spart Akku. Der Stromsparmodus und die Systemeinstellung „Bewegung reduzieren“ tun dies bereits." + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Mantiene quietos los medidores líquidos, el cielo y la respuesta a la inclinación, y apaga el sensor de movimiento que los impulsa. Ahorra batería. El modo de bajo consumo y el ajuste del sistema «Reducir movimiento» ya hacen esto." + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Fige les jauges liquides, le ciel et la réaction à l’inclinaison, et coupe le capteur de mouvement qui les anime. Économise la batterie. Le mode économie d’énergie et le réglage système « Réduire les animations » font déjà cela." + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Blocca gli indicatori liquidi, il cielo e la risposta all’inclinazione e spegne il sensore di movimento che li anima. Risparmia batteria. La modalità Risparmio energetico e l’impostazione di sistema «Riduci movimento» lo fanno già." + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "Mantém parados os medidores líquidos, o céu e a resposta à inclinação, e desliga o sensor de movimento que os anima. Poupa bateria. O Modo de Baixo Consumo e a definição de sistema «Reduzir movimento» já fazem isto." + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Останавливает жидкостные шкалы, небо и отклик на наклон и отключает датчик движения, который их питает. Экономит заряд батареи. Режим энергосбережения и системная настройка «Уменьшение движения» уже делают это." + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "让液态仪表、天空和倾斜响应保持静止，并关闭驱动它们的运动传感器。省电。低电量模式和系统的“减弱动态效果”设置已经会这样做。" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "讓液態儀表、天空和傾斜回應保持靜止，並關閉驅動它們的動作感測器。省電。低耗電模式和系統的「減少動態效果」設定已經會這樣做。" + } + } + } } }, "version": "1.0" diff --git a/Strand/Screens/SettingsView.swift b/Strand/Screens/SettingsView.swift index c53b3182c5..dcd4b23306 100644 --- a/Strand/Screens/SettingsView.swift +++ b/Strand/Screens/SettingsView.swift @@ -98,6 +98,10 @@ struct SettingsView: View { @AppStorage(SkyBehindCardsPrefs.enabledKey) private var skyBehindCards = true // Card-surface opacity percent (100 = solid). Reactive — moving the slider live-updates every card. @AppStorage(CardAppearancePrefs.opacityKey) private var cardOpacityPercent = CardAppearancePrefs.defaultPercent + // "Reduce motion in NOOP" (default OFF): pose every looping animation still and stop the decorative + // tilt sensor, without needing system Low Power Mode or system Reduce Motion. Mirrors Kotlin + // NoopPrefs.quietMotion. + @AppStorage(QuietMotionPrefs.enabledKey) private var quietMotion = false // Hydration tracker (opt-in, MVP). Default OFF — when off the hydration dashboard card + detail are // hidden. Mirrors the Android pref so the toggle reads the same on both platforms. @AppStorage(HydrationStore.enabledKey) private var hydrationEnabled = false @@ -770,6 +774,22 @@ struct SettingsView: View { #endif Divider().overlay(StrandPalette.hairline).padding(.vertical, 4) + // MARK: Reduce motion in NOOP — pose every looping animation still and stop the tilt + // sensor, WITHOUT requiring system Low Power Mode. Off by default; system Reduce Motion + // and Low Power Mode already force the same behaviour, this is the third, in-app signal. + Toggle(isOn: $quietMotion) { + Text("Reduce motion in NOOP") + .font(StrandFont.subhead) + .foregroundStyle(StrandPalette.textPrimary) + } + .toggleStyle(.switch) + .tint(StrandPalette.accent) + Text("Holds the liquid gauges, the sky and the tilt response still, and turns off the motion sensor that drives them. Saves battery. Low Power Mode and the system Reduce Motion setting already do this.") + .font(StrandFont.caption) + .foregroundStyle(StrandPalette.textTertiary) + .fixedSize(horizontal: false, vertical: true) + .frame(maxWidth: .infinity, alignment: .leading) + // MARK: Day-cycle background — the time-of-day scene behind Today (#698). On by default. // Off swaps it for the plain dark canvas for people who find the moving scene distracting. Toggle(isOn: $showDayCycleBackground) { diff --git a/Strand/Screens/TodayView.swift b/Strand/Screens/TodayView.swift index a5443e8446..c8e84f1e7b 100644 --- a/Strand/Screens/TodayView.swift +++ b/Strand/Screens/TodayView.swift @@ -4544,6 +4544,13 @@ private struct RecordingStatusLight: View { /// Drives the syncing pulse; toggled in `.task` while an offload runs (never during body eval). @State private var pulsing = false + /// This `repeatForever` ring had NO motion gate of any kind — it pulsed under system Reduce + /// Motion too, which was already a bug (the Android twin's ConnectionDot had the same one, fixed + /// in #911). It also ran precisely while the strap was offloading history, i.e. while the app was + /// already busy. Gated on all three quiet signals now. + @Environment(\.accessibilityReduceMotion) private var reduceMotion + @ObservedObject private var motion = NoopMotionState.shared + /// Colour for the light: green recording, amber last-synced, red not recording, accent for /// experimental history. Mirrors the prior `TodayView.recordingHue` semantics verbatim. private func hue(_ state: RecordingState) -> Color { @@ -4590,10 +4597,12 @@ private struct RecordingStatusLight: View { .disabled(state == nil && !syncing) .accessibilityLabel(syncing ? syncingAccessibilityLabel : (state?.accessibilityText ?? String(localized: "Recording status, not shown for a past day"))) - // Run the repeating pulse only while syncing; the `.task(id:)` auto-cancels when the flag flips, - // so there is no timer left running once the offload ends (or Today goes away). + // Run the repeating pulse only while syncing AND nothing is asking for quiet motion; the + // `.task(id:)` auto-cancels when the flag flips, so there is no timer left running once the + // offload ends (or Today goes away). Without the pulse the steady accent dot still says + // "syncing" — the information survives, only the loop stops. .task(id: syncing) { - guard syncing else { pulsing = false; return } + guard syncing, !motion.poseStill(reduceMotion) else { pulsing = false; return } withAnimation(.easeOut(duration: 1.1).repeatForever(autoreverses: false)) { pulsing = true } } } From 6aa61c69319b65a4f82ef95da7e0397094872330 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 28 Jul 2026 10:52:49 -0400 Subject: [PATCH 147/171] Make the R22 deep-data unlock actually reversible (#174) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Settings copy has said since #174 that the R22 unlock is "reversible". That was true about the hardware and false about the app: NOOP shipped an enable button and nothing that writes the flags back, so the only ways out were the official WHOOP app or a factory reset. Turning the switch off only gated future sends — it left all sixteen flags set on the strap. This adds the undo, and verifies it by reading the strap rather than by believing an ack. What it writes `Whoop5Config.disableR22Sequence` is the same sixteen keys in the same order as the enable sequence, each carrying `featureFlagOffValue` = ASCII '0' (0x30). Master flag first, so a run interrupted by a disconnect leaves the strap nearer off than it started. On the off value, precisely: 0x30 is CONFIRMED as this firmware's off value in the device-config namespace — `setBroadcastHr(false)` has written it to `whoop_live_hr_in_adv_ind_pkt` through SET_DEVICE_CONFIG_VALUE(119) since #181, hardware-validated against a Garmin Edge 840, and `enable_raw_data_w_ecg` independently reads '0' on an MG whose ECG is idle. It is INFERRED for SET_FF_VALUE(120) by shared convention: identical body layout, identical ASCII-digit value convention, same `[0x01] +` prefix, differing only in opcode and body length. Writes through 120 demonstrably change stored state — the enable sequence moved `enable_sig12` from '2' to '1', confirmed by a GET_FF_VALUE(128) read either side. What is NOT claimed: no feature flag has ever been observed holding '0'; the only values ever read back in that namespace are '1' and '2', and those are a round-trip of NOOP's own writes. `disable_pip_r26_packets` is written '2' despite being named `disable_*`, and `enable_sig12` was corrected '2'→'1' from a real capture, so the official app picks between '1' and '2' per flag rather than using one canonical true. Tri-state semantics stay unestablished. So the run tests the byte instead of asserting it `R22DisableReport` is staged. It writes '0' to ONE flag, reads it back, and declines to touch the other fifteen unless the strap stopped reporting the old value. The probe key is `enable_sig12` — the only key with a hardware demonstration that a write to it moves stored state, so a failure there is attributable to the VALUE rather than to the key or the verb. Read-back separates three outcomes that a boolean "did it work" would blur: value '0' (cleared), FAILURE (no stored value at all — nothing predicts it), or the old value (write refused). Enumeration cannot substitute: '0' is a stored value, so a cleared key stays listed, and the 117/118 walk carries no value field. Every write ack is recorded and not believed. SELECT_WRIST returns SUCCESS for a no-op and FAILURE for a real mutation on this firmware, so only the value a 128 read returns is reported as state (#907/#891 discipline). Also in here - `FeatureFlagWriteGate` tightens the send allowlist for opcode 120. It was opcode-only — any feature-flag key with any value travelled while the deep-data opt-in happened to be on. It is now key- and value-aware: the sixteen R22 keys, each only for its own enable value or the off value. Same weakness #907 closed on opcode 119. - The toggle gets an `.onChange` that offers the disable instead of silently writing nothing. Broadcast HR has had one since #181. - The card said "15" while the sequence carried 16, so it declared success a flag early. Threshold and number now come from the sequence. - A disable run's acks no longer tick the enable counter upward. - An interrupted run is rendered, not dropped: it has already written, so the user is told how far it got and which keys are still set. Verification Swift `swift test` in Packages/WhoopProtocol: 454 tests, 0 failures (24 new). Android `testFullDebugUnitTest`: 3226 tests, 0 failures across 392 classes, counted from the JUnit XML rather than the wrapper exit code (25 new). `assembleFullDebug` clean. macOS `Strand` and iOS `NOOPiOS` both build. `Tools/i18n_audit.py --ci` passes; new copy is translated in every focus locale and in it/ru/zh-Hans/zh-Hant so the extra-locale ratchet does not move. NOT verified on hardware: whether the strap accepts '0' for a feature flag, and whether clearing the flags makes the deep records stop. The first is what the probe stage measures on first run; the second needs a wear-and-sync afterwards and is called out in the UI and the report rather than claimed. --- .../Sources/WhoopProtocol/R22Disable.swift | 484 +++++++++++++++ .../Sources/WhoopProtocol/Whoop5Config.swift | 122 +++- .../WhoopProtocolTests/R22DisableTests.swift | 320 ++++++++++ Strand/BLE/BLEManager.swift | 194 +++++- Strand/BLE/LiveState.swift | 8 + Strand/BLE/PuffinExperiment.swift | 9 +- Strand/Resources/Localizable.xcstrings | 572 ++++++++++++++++++ Strand/Screens/SettingsView.swift | 101 +++- .../main/java/com/noop/ble/WhoopBleClient.kt | 193 +++++- .../main/java/com/noop/protocol/R22Disable.kt | 393 ++++++++++++ .../java/com/noop/protocol/Whoop5Config.kt | 57 ++ .../main/java/com/noop/ui/SettingsScreen.kt | 122 +++- .../app/src/main/res/values-de/strings.xml | 11 + .../app/src/main/res/values-es/strings.xml | 11 + .../app/src/main/res/values-fr/strings.xml | 11 + .../src/main/res/values-pt-rPT/strings.xml | 11 + .../app/src/main/res/values-zh/strings.xml | 11 + android/app/src/main/res/values/strings.xml | 11 + .../java/com/noop/protocol/R22DisableTest.kt | 353 +++++++++++ 19 files changed, 2963 insertions(+), 31 deletions(-) create mode 100644 Packages/WhoopProtocol/Sources/WhoopProtocol/R22Disable.swift create mode 100644 Packages/WhoopProtocol/Tests/WhoopProtocolTests/R22DisableTests.swift create mode 100644 android/app/src/main/java/com/noop/protocol/R22Disable.kt create mode 100644 android/app/src/test/java/com/noop/protocol/R22DisableTest.kt diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/R22Disable.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/R22Disable.swift new file mode 100644 index 0000000000..1e6c641f40 --- /dev/null +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/R22Disable.swift @@ -0,0 +1,484 @@ +import Foundation + +/// #174: the key-aware allowlist for `SET_FF_VALUE` (120 / 0x78) — the feature-flag WRITE verb the R22 +/// enable and disable sequences share. +/// +/// ## Why this exists +/// +/// Before this file the 5/MG send path admitted opcode 120 on an opcode-only clause: +/// `command == .setConfig && PuffinExperiment.deepDataEnabled`. That admits **any** feature-flag key with +/// **any** value for as long as the deep-data opt-in happens to be on. The sixteen R22 keys are the only +/// ones NOOP has any business writing, and the disable path is about to double the number of values that +/// travel through that opcode, so the clause is tightened at the same time rather than widened. +/// +/// This is the same discipline `DeviceConfigReadProbe.isReadOnlyOpcode` established for the read probes: +/// one pure predicate that the BLE send path itself consults, so a unit test proving the predicate rejects +/// something is proving it about the real wire path and not about a parallel copy of the rule. +/// +/// ## What it admits +/// +/// | opcode | key | value | gate | +/// |---|---|---|---| +/// | 120 `SET_FF_VALUE` | one of `Whoop5Config.enableR22Sequence` | the enable value for that key, or `featureFlagOffValue` | deep-data opt-in | +/// | anything else | — | — | refused | +/// +/// A value outside that pair is refused even for an R22 key: nothing in NOOP has a reason to write a third +/// value, and admitting one would mean a typo could put an arbitrary byte into persistent strap config. +/// `SET_DEVICE_CONFIG_VALUE` (119) is refused outright — it keeps its own clause for the Broadcast-HR flag. +/// +/// Pure: no CoreBluetooth, no I/O, no preferences. The app layer supplies the opt-in boolean. The Kotlin +/// twin is `com.noop.protocol.FeatureFlagWriteGate` — keep them byte-identical. +public enum FeatureFlagWriteGate { + + // MARK: - Opcodes + + /// `SET_FF_VALUE` (120 / 0x78) — the feature-flag write verb. The only opcode this gate admits. + public static let setFeatureFlagValueCmd: UInt8 = 120 + + /// `SET_DEVICE_CONFIG_VALUE` (119 / 0x77) — the OTHER namespace's write verb. Named here for exactly + /// one reason: so `admitsSend` can be proved to refuse it. + public static let setDeviceConfigValueCmd: UInt8 = 119 + + /// `GET_FF_VALUE` (128 / 0x80) — the read verb the mandatory post-write read-back uses. Read-only, and + /// already in `DeviceConfigReadProbe.readOnlyOpcodes`. + public static let getFeatureFlagValueCmd: UInt8 = 128 + + // MARK: - Keys and values + + /// The sixteen keys, taken from the enable sequence rather than restated — one list, one truth. + public static var r22Keys: [String] { Whoop5Config.enableR22Sequence.map(\.name) } + + /// The value the enable sequence writes for `key`, or nil when `key` is not an R22 flag. + public static func enableValue(for key: String) -> UInt8? { + Whoop5Config.enableR22Sequence.first { $0.name == key }?.value + } + + /// Whether `value` is one NOOP is allowed to write for `key`: that key's own enable value, or the + /// documented off value. Anything else is refused. + public static func isWritableValue(_ value: UInt8, for key: String) -> Bool { + guard let on = enableValue(for: key) else { return false } + return value == on || value == Whoop5Config.featureFlagOffValue + } + + // MARK: - Body parsing + + /// Width of the key-name field in a feature-flag body (`Whoop5Config.payloadBody` NUL-pads to this). + public static let nameFieldBytes = 32 + + /// Total length of the feature-flag body: 32-byte name, value byte, 7 zero bytes. + public static let bodyBytes = 40 + + /// The key name and value carried by a `SET_FF_VALUE` payload, or nil when the payload is not shaped + /// like one. + /// + /// The payload the send path holds is `[0x01] + payloadBody(...)`: the inner b3 byte, then the 32-byte + /// NUL-padded name, then the value, then 7 zeroes. A name is only returned when it is printable ASCII + /// and the remainder of the field is genuine NUL padding — so a body that is short, mis-shaped, or + /// carrying binary in the name field yields nil and is refused rather than guessed at. Mirrors + /// `DeviceConfigWriteGate.keyName(inSendPayload:)` on the other namespace. + public static func keyAndValue(inSendPayload payload: [UInt8]) -> (key: String, value: UInt8)? { + guard payload.count >= 1 + bodyBytes, payload[0] == 0x01 else { return nil } + let field = Array(payload[1..<(1 + nameFieldBytes)]) + var name: [UInt8] = [] + for b in field { + if b == 0 { break } + guard (0x20...0x7E).contains(b) else { return nil } + name.append(b) + } + guard !name.isEmpty else { return nil } + // Everything after the name must be NUL, or this is not a NUL-padded name field. + for i in name.count.. Bool { + guard deepDataOptIn else { return false } + guard opcode == setFeatureFlagValueCmd else { return false } + guard let (key, value) = keyAndValue(inSendPayload: payload) else { return false } + return isWritableValue(value, for: key) + } + + /// The read verb the post-write verification is allowed to send, and only that one. + public static func isReadBackOpcode(_ opcode: UInt8) -> Bool { opcode == getFeatureFlagValueCmd } +} + +// MARK: - Report + +/// #174: one R22 **disable** run — write `'0'` to the sixteen feature flags, then read every one of them +/// back and report the value the strap actually stores. +/// +/// ## The shape of the run, and why it is not just sixteen writes +/// +/// `Whoop5Config.featureFlagOffValue` explains why `'0'` is an inference: it is the confirmed off value in +/// the device-config namespace and shares that namespace's whole wire idiom, but no feature flag has ever +/// been *observed* holding it. So the run is staged, and the first stage is an experiment: +/// +/// 1. **Probe.** Write `'0'` to ONE flag, then read it back with `GET_FF_VALUE(128)`. +/// 2. **Gate.** If the strap did not stop reporting the old value, STOP. Fifteen keys are left untouched +/// and the report says `'0'` is not how this namespace spells off — a real, publishable answer. +/// 3. **Clear.** Only on a good probe, write `'0'` to the remaining fifteen. +/// 4. **Verify.** Read all sixteen back and tabulate. +/// +/// The staging is the point. A blanket sixteen-write on an inferred byte would leave a user unable to tell +/// "it worked" from "the strap ignored all of it", and would move fifteen keys to find out. +/// +/// ## The ack is never the proof +/// +/// Every write's own `COMMAND_RESPONSE` is recorded and **not believed**. `SELECT_WRIST` on this firmware +/// returns SUCCESS for a no-op and FAILURE for a real mutation, so a result byte says nothing reliable +/// about whether state moved. Only the value that comes back from a 128 read is reported as state, and a +/// SUCCESS ack whose read-back did not move is reported as `unchanged`, not as success. (Discipline owed +/// to #907 / #891.) +/// +/// ## Three outcomes, kept apart +/// +/// The read-back separates possibilities that a boolean "did it work" would blur — see the table on +/// `Whoop5Config.featureFlagOffValue`. `.cleared` (the key now holds `'0'`) and `.unset` (the key no longer +/// has a stored value at all, so 128 answers FAILURE) are BOTH successes, and the difference is the most +/// interesting thing this path can learn about the firmware, so it is reported rather than collapsed. +/// +/// Pure and order-dependent (`nextStep` → `note…` → `nextStep` → …), so `swift test` covers the whole run — +/// plan, gate, verdicts, rendering — without a strap. Kotlin twin: `R22DisableReport` in +/// `android/…/protocol/R22Disable.kt`; the rendered text is byte-identical across platforms so a shared +/// strap log reads the same either side. +public struct R22DisableReport: Equatable, Sendable { + + /// The flag the probe stage writes first. + /// + /// `enable_sig12` is chosen because it is the ONLY key with a hardware demonstration that a write to it + /// changes stored state: running the enable sequence moved it from `'2'` (0x32) to `'1'` (0x31), + /// confirmed by a `GET_FF_VALUE(128)` read before and after (#423, #103). That is what makes it a clean + /// instrument. If a `'0'` write to *this* key does not move the stored value, the failure is + /// attributable to the VALUE rather than to the key or the verb — every other key would leave that + /// ambiguity open. It is also the flag with the least to lose: its effect is undocumented and it gates + /// no stream NOOP reads. + public static let probeKey = "enable_sig12" + + /// Which stage of the run a step belongs to. + public enum Stage: String, Equatable, Sendable { + /// The single write that tests whether `'0'` is accepted at all. + case probeWrite + /// The read-back of the probe write. The gate for everything after it. + case probeRead + /// One of the remaining fifteen writes. + case clearWrite + /// A final read-back of one key. + case verifyRead + } + + /// One planned round-trip. + public struct Step: Equatable, Sendable { + public let opcode: UInt8 + public let key: String + public let stage: Stage + public init(opcode: UInt8, key: String, stage: Stage) { + self.opcode = opcode + self.key = key + self.stage = stage + } + /// Whether this step puts a value on the wire (as opposed to reading one back). + public var isWrite: Bool { stage == .probeWrite || stage == .clearWrite } + } + + /// What one key ended up in. `.cleared` and `.unset` are both successes. + public enum KeyOutcome: String, Equatable, Sendable { + /// Read back as `'0'`: the key exists and now holds the off value. + case cleared + /// The strap answered FAILURE for the key: it no longer has a stored value. The stronger result. + case unset + /// Read back as something other than `'0'` — the write did not take. + case unchanged + /// The strap answered, but did not echo the key, so no value can be claimed. + case notClaimed + /// No reply inside the step's window. + case silent + /// A reply arrived that could not be decoded (CRC, envelope, short record). + case undecodable + /// The run stopped before this key was written. + case skipped + + /// Whether this outcome means the flag is no longer set to its R22 value. + public var isSuccess: Bool { self == .cleared || self == .unset } + } + + /// The overall verdict. + public enum Verdict: String, Equatable, Sendable { + /// The run has not finished. + case pending + /// Every one of the sixteen came back cleared or unset. + case allCleared + /// Some cleared, some did not. + case partial + /// The probe read-back came back showing the OLD value: `'0'` is not how this firmware clears a + /// feature flag. The one verdict that is a genuine finding about the off value. + case offValueRejected + /// The probe read-back produced no usable answer — silent, undecodable, or the key was not echoed. + /// Distinct from `offValueRejected` on purpose: nothing was learned about the off value, so this + /// must not be reported as evidence against it. + case probeInconclusive + /// The run was abandoned (disconnect, or the caller stopped it). + case abandoned + } + + /// One recorded read-back. + public struct Reading: Equatable, Sendable { + public let key: String + public let stage: Stage + /// The value byte, only when the strap echoed the key in a 32-byte name field. nil is "not + /// claimed", never "zero". + public let value: UInt8? + public let resultCode: Int? + public let recordHex: String + public let outcome: KeyOutcome + public init(key: String, stage: Stage, value: UInt8?, resultCode: Int?, + recordHex: String, outcome: KeyOutcome) { + self.key = key + self.stage = stage + self.value = value + self.resultCode = resultCode + self.recordHex = recordHex + self.outcome = outcome + } + } + + // MARK: Inputs + + /// The keys to clear, in order. Taken from the enable sequence so the two can never drift. + public let keys: [String] + + // MARK: State + + private var cursor = 0 + private var plan: [Step] = [] + private var planBuilt = false + /// Per-key outcome, keyed by name. + public private(set) var outcomes: [String: KeyOutcome] = [:] + public private(set) var readings: [Reading] = [] + public private(set) var writeAcks: [String: Int] = [:] + public private(set) var trace: [String] = [] + public private(set) var verdict: Verdict = .pending + /// True once the probe read-back has been seen and was good. + public private(set) var probePassed = false + /// True once the run has decided to stop early. + private var stopped = false + + public init(keys: [String] = Whoop5Config.enableR22Sequence.map(\.name)) { + self.keys = keys + trace.append("R22 disable: writing '0' (0x30) to \(keys.count) feature flags via SET_FF_VALUE(120), " + + "each verified with GET_FF_VALUE(128). The write acks are recorded and NOT trusted.") + trace.append("Probe first: \(R22DisableReport.probeKey) is written and read back alone; the other " + + "\(max(0, keys.count - 1)) are only written if the strap stops reporting the old value.") + } + + // MARK: Plan + + /// The next round-trip, or nil when the run is over. + /// + /// The plan is built lazily in two halves so the gate is expressible: the probe pair up front, and the + /// clear+verify tail only once the probe read-back has passed. + public mutating func nextStep() -> Step? { + if plan.isEmpty && !planBuilt { + planBuilt = true + guard keys.contains(R22DisableReport.probeKey) else { + // Defensive: a caller that hands over a key list without the probe key gets a plain + // sequential run rather than a silently skipped gate. + plan = keys.map { Step(opcode: FeatureFlagWriteGate.setFeatureFlagValueCmd, key: $0, stage: .clearWrite) } + + keys.map { Step(opcode: FeatureFlagWriteGate.getFeatureFlagValueCmd, key: $0, stage: .verifyRead) } + probePassed = true + return nextStep() + } + plan = [ + Step(opcode: FeatureFlagWriteGate.setFeatureFlagValueCmd, + key: R22DisableReport.probeKey, stage: .probeWrite), + Step(opcode: FeatureFlagWriteGate.getFeatureFlagValueCmd, + key: R22DisableReport.probeKey, stage: .probeRead), + ] + } + if stopped { return nil } + if cursor >= plan.count { + // The probe pair just finished and passed — extend the plan with the rest. + if probePassed && !plan.contains(where: { $0.stage == .clearWrite }) { + let rest = keys.filter { $0 != R22DisableReport.probeKey } + plan += rest.map { Step(opcode: FeatureFlagWriteGate.setFeatureFlagValueCmd, key: $0, stage: .clearWrite) } + plan += keys.map { Step(opcode: FeatureFlagWriteGate.getFeatureFlagValueCmd, key: $0, stage: .verifyRead) } + } else { + return nil + } + } + guard cursor < plan.count else { return nil } + let step = plan[cursor] + cursor += 1 + return step + } + + // MARK: Notes + + /// Record a write's own ack. Logged, never used to decide anything. + public mutating func noteWriteAck(resultCode: Int?, for step: Step) { + if let resultCode { writeAcks[step.key] = resultCode } + let label = resultCode.map { "\(FeatureFlagProbe.resultLabel($0))(\($0))" } ?? "(unlabelled)" + trace.append("SET_FF_VALUE(120) \(step.key)='0' → ack \(label) — recorded, not proof") + } + + /// Record a decoded read-back and score the key. + public mutating func noteReadBack(_ r: DeviceConfigReadProbe.ValueResponse, for step: Step) { + let value = r.value(for: step.key) + let outcome: KeyOutcome + if r.isUnsupported { + outcome = .unchanged // the verb itself is refused; nothing can be claimed cleared + } else if r.isFailure { + outcome = .unset // no stored value for this key any more + } else if let value { + outcome = value == Whoop5Config.featureFlagOffValue ? .cleared : .unchanged + } else { + outcome = .notClaimed + } + record(Reading(key: step.key, stage: step.stage, value: value, resultCode: r.resultCode, + recordHex: r.recordHex, outcome: outcome), for: step) + } + + /// Record a read-back reply that could not be decoded. + public mutating func noteReadFailure(_ f: DeviceConfigReadProbe.ParseFailure, for step: Step) { + let why: String + switch f { + case .crc: why = "CRC failed — frame rejected (never decoded)" + case .envelope: why = "not a COMMAND_RESPONSE envelope" + case .wrongCommand: why = "COMMAND_RESPONSE for a different command" + case .truncated: why = "record too short to hold a response" + } + trace.append("GET_FF_VALUE(128) \(step.key) → reply not decoded: \(why)") + record(Reading(key: step.key, stage: step.stage, value: nil, resultCode: nil, + recordHex: "(undecodable)", outcome: .undecodable), for: step) + } + + /// Record the strap answering nothing at all. + public mutating func noteTimeout(for step: Step, seconds: Int) { + trace.append("\(step.isWrite ? "SET_FF_VALUE(120)" : "GET_FF_VALUE(128)") \(step.key) → no COMMAND_RESPONSE within \(seconds)s") + guard !step.isWrite else { return } // a silent WRITE is fine; the read-back is what decides + record(Reading(key: step.key, stage: step.stage, value: nil, resultCode: nil, + recordHex: "(no reply)", outcome: .silent), for: step) + } + + /// The link dropped, or the caller stopped the run. + public mutating func noteAbandoned(_ why: String) { + trace.append("Run abandoned: \(why)") + stopped = true + for key in keys where outcomes[key] == nil { outcomes[key] = .skipped } + verdict = .abandoned + } + + /// Score one reading, drive the probe gate, and finalise when the plan is done. + private mutating func record(_ reading: Reading, for step: Step) { + readings.append(reading) + var line = "GET_FF_VALUE(128) \(step.key)" + if let c = reading.resultCode { line += " → result=\(FeatureFlagProbe.resultLabel(c))(\(c))" } else { line += " →" } + if let v = reading.value { line += " value=\(DeviceConfigReadProbe.valueLabel(v))" } + line += " record=[\(reading.recordHex)] ⇒ \(reading.outcome.rawValue)" + trace.append(line) + + // A verify read always sets the key's outcome. A probe read sets it too, and additionally decides + // whether the rest of the run happens at all. + outcomes[step.key] = reading.outcome + + if step.stage == .probeRead { + probePassed = reading.outcome.isSuccess + if !probePassed { + stopped = true + for key in keys where key != step.key { outcomes[key] = .skipped } + // ONLY an unmoved value is evidence against '0'. Silence, an undecodable frame and a reply + // that never echoed the key all mean "no usable answer" and must not be reported as a + // rejection of the off value. + verdict = reading.outcome == .unchanged ? .offValueRejected : .probeInconclusive + trace.append(probeStopExplanation(reading.outcome)) + } else { + trace.append("Probe passed (\(reading.outcome.rawValue)) — '0' moves a feature flag on this " + + "firmware. Clearing the remaining \(max(0, keys.count - 1)).") + } + } + + if !stopped, outcomes.count == keys.count, readings.contains(where: { $0.stage == .verifyRead }) { + let cleared = keys.filter { outcomes[$0]?.isSuccess == true }.count + verdict = cleared == keys.count ? .allCleared : .partial + } + } + + /// The sentence that explains a stopped run, in the terms that make it a result rather than a failure. + private func probeStopExplanation(_ outcome: KeyOutcome) -> String { + switch outcome { + case .unchanged: + return "STOPPED: the strap still reports the old value for \(R22DisableReport.probeKey), so " + + "'0' (0x30) is NOT how this firmware clears a feature flag. The other \(max(0, keys.count - 1)) " + + "flags were left untouched. This is a real finding — please share this report on #174." + case .notClaimed: + return "STOPPED: the strap answered but did not echo \(R22DisableReport.probeKey), so no value " + + "can be claimed either way. Nothing further was written." + case .silent, .undecodable: + return "STOPPED: no usable read-back for \(R22DisableReport.probeKey), so whether the write " + + "landed is unknown. Nothing further was written." + default: + return "STOPPED before clearing the remaining flags." + } + } + + // MARK: Rendering + + /// One-line summary, suitable for a Settings row. + public var summary: String { + let cleared = keys.filter { outcomes[$0]?.isSuccess == true }.count + switch verdict { + case .pending: return "Clearing R22 flags…" + case .allCleared: return "All \(keys.count) R22 flags cleared on the strap (read back, not just acked)." + case .partial: return "\(cleared) of \(keys.count) R22 flags cleared — the rest did not take." + case .offValueRejected: return "The strap refused '0' for \(R22DisableReport.probeKey); the other flags were left alone." + case .probeInconclusive: return "No usable read-back for \(R22DisableReport.probeKey) — nothing further was written." + case .abandoned: return "Disable run interrupted — \(cleared) of \(keys.count) flags confirmed cleared." + } + } + + /// The full copyable report. + public func render() -> String { + var sb = "#174 R22 DEEP-DATA DISABLE — WHOOP 5/MG\n" + sb += "Wrote '0' (0x30) via SET_FF_VALUE(120) and read every key back with GET_FF_VALUE(128).\n" + sb += "The write acks are recorded and NOT treated as proof: on this firmware a SUCCESS result byte\n" + sb += "does not establish that state changed. Only the read-back is reported as state.\n" + sb += "\nVerdict: \(verdict.rawValue) — \(summary)\n" + sb += "\nPer key:\n" + for key in keys { + let outcome = outcomes[key] ?? .skipped + let reading = readings.last { $0.key == key && $0.stage == .verifyRead } + ?? readings.last { $0.key == key } + let was = Whoop5Config.enableR22Sequence.first { $0.name == key } + .map { DeviceConfigReadProbe.valueLabel($0.value) } ?? "?" + let now = reading?.value.map { DeviceConfigReadProbe.valueLabel($0) } + ?? (outcome == .unset ? "(no stored value)" : "(not claimed)") + sb += " \(DeviceConfigReadProbe.padded(key, to: 30)) enable wrote \(was) → now \(now) [\(outcome.rawValue)]\n" + } + sb += "\nExchange:\n" + for line in trace { sb += " " + line + "\n" } + sb += "\n" + R22DisableReport.caveats + return sb + } + + /// The standing caveats, kept in one place so the UI, the log and the report cannot drift. + public static let caveats = """ + What this does and does not establish: + • '0' as the off value is INFERRED. It is the confirmed off value in the device-config namespace + (SET_DEVICE_CONFIG_VALUE/119, hardware-validated on the Broadcast-HR flag, #181) and that namespace + shares this one's whole wire idiom — but no feature flag had ever been observed holding '0' before + this run. The read-back above is what turns that inference into a measurement. + • A cleared flag is not the same as reverted BEHAVIOUR. This reports what the strap STORES. Whether + the deep records actually stop is a separate question, answered by wearing the strap and watching + the type-0x2F deep-buffer capture stop, not by this report. + • This does not restore a snapshot. NOOP never read these values before first writing them, so the + strap's pre-NOOP state is unknown. The honest claim is "cleared the flags NOOP set". + • disable_pip_r26_packets inverts: clearing it is expected to let PIP R26 packets flow AGAIN. That is + the pre-R22 behaviour and is intended. + """ +} diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Config.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Config.swift index 5c11b6cafd..053c85730c 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Config.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Config.swift @@ -16,9 +16,16 @@ import Foundation // then 7 bytes of zero. The command is built with the normal puffin envelope via `puffinCommandFrame` // (cmd 0x78, the inner b3 byte 0x01 carried as the first payload byte, exactly like CLIENT_HELLO). // -// This is reversible — it only changes which data the strap chooses to emit — but it is gated behind an -// explicit opt-in in the app, and on real hardware it can only be written from iOS/Android: macOS -// CoreBluetooth cannot complete the authenticated SMP bond the command characteristic requires. +// It is gated behind an explicit opt-in in the app, and on real hardware it can only be written from +// iOS/Android: macOS CoreBluetooth cannot complete the authenticated SMP bond the command characteristic +// requires. +// +// **Reversing it.** The write only changes which data the strap chooses to emit, and the same opcode that +// sets a flag can clear it — see `disableR22Sequence` below, and `R22DisableReport` for the write + +// mandatory read-back that proves what the strap actually stores. For the first year of this file that +// claim was made in a comment with nothing implementing it; the value byte that spells "off" is still +// inferred rather than observed, which is why the disable path tests it on one key before touching the +// rest. Read `featureFlagOffValue` before relying on any of this. public enum Whoop5Config { /// SET_CONFIG / SET_FF_VALUE command opcode. @@ -105,4 +112,113 @@ public enum Whoop5Config { frame(flag: flag, seq: UInt8((Int(firstSeq) + idx) & 0xFF)) } } + + // MARK: - Turning it back off (#174) + + /// The value byte written to clear a feature flag: ASCII `'0'` (0x30). + /// + /// **Confirmed for the sibling opcode, inferred here by shared convention.** The distinction is worth + /// stating precisely, because it is stronger than a guess and weaker than a measurement. + /// + /// *Confirmed, on real hardware:* `0x30` is how this firmware spells "off" for a config key. NOOP has + /// shipped that write since #181 — `setBroadcastHr(false)` writes `0x30` to + /// `whoop_live_hr_in_adv_ind_pkt` through `SET_DEVICE_CONFIG_VALUE` (119 / 0x77) and a Garmin Edge 840 + /// stops seeing the broadcast. `enable_raw_data_w_ecg` independently *reads* `'0'` on an MG whose ECG + /// is not running. Both are the device-config namespace. + /// + /// *Shared convention, which is why it transfers:* the two namespaces are the same wire idiom. Both + /// bodies are the key name as ASCII NUL-padded to 32 bytes followed by a single ASCII-digit value byte + /// at offset 32, both are sent as `[0x01] + body` WITH RESPONSE, and `deviceConfigBody`'s own doc spells + /// the value convention out as "an ASCII digit, '1' = 0x31 / '0' = 0x30". The only differences are the + /// opcode (0x77 vs 0x78) and the body length (33 vs 40 bytes, the feature-flag body carrying 7 trailing + /// zeroes). Nothing about 0x78 makes it one-way: writes through it demonstrably land and demonstrably + /// change stored state — running `enableR22Sequence` moved `enable_sig12` from `'2'` (0x32) to `'1'` + /// (0x31), confirmed by a `GET_FF_VALUE(128)` read before and after. A key that can be written is a key + /// that can be rewritten. + /// + /// *Inferred, and stated plainly:* **`'0'` has never been observed in the feature-flag namespace.** Every + /// `GET_FF_VALUE(128)` read ever taken returned `'1'` or `'2'` — and those sixteen readings are a + /// round-trip of NOOP's own `enableR22Sequence` writes, so they say nothing about what the firmware does + /// with a value it was not handed. The two namespaces are also proven separate at the verb level: a key + /// asked through the wrong verb answers FAILURE. So the device-config evidence above is a strong + /// argument by shared convention, not a measurement of this opcode. + /// + /// *And two pieces of counter-evidence, which is why "0 = false" is not assumed anywhere in this file:* + /// `disable_pip_r26_packets` is written `'2'` despite being named `disable_*` — if these were plain + /// booleans that key would be the one written `'0'`. And `enable_sig12` was corrected `'2'` → `'1'` from + /// a real on-strap capture (#423), i.e. the official app picks *between* `'1'` and `'2'` per flag rather + /// than using one canonical "true". Whatever those two values select, it is probably not a bare boolean, + /// so tri-state `'0'`/`'1'`/`'2'` semantics stay UNESTABLISHED rather than assumed. + /// + /// *Which makes the read-back decisive rather than ceremonial.* A single `GET_FF_VALUE(128)` on the + /// written key separates the three live possibilities, and `R22DisableReport` reports which one happened + /// instead of averaging over them: + /// + /// | Read-back | Meaning | + /// |---|---| + /// | value `'0'` | The write landed and the key now HOLDS `'0'`. Cleared as a value — the expected success. | + /// | `FAILURE(0)` | The key no longer has a stored value at all. Genuinely removed; nothing predicts this. | + /// | value unchanged (`'2'`/`'1'`) | The write was refused or no-oped. `'0'` is not how this namespace says off. | + /// + /// Note that **enumeration cannot substitute for this read.** `'0'` is a stored value, not an absence — + /// six of the seven keys the device-config walk returned read `'0'` and were still enumerated — so a + /// cleared key stays present in a walk rather than dropping out. And the 117/118 feature-flag + /// enumeration carries no value field at all (its record is `[revision][index][validKey][key]`), so it + /// can only ever answer presence. `GET_FF_VALUE(128)` is the only verb that can verify a clear, and it + /// answered cleanly for all sixteen flags on a 5/MG — the older 4.0-derived caution about a stale shared + /// buffer contaminating its value field does not reproduce on this family. + /// + /// The path therefore writes `'0'` to ONE flag first, reads it back, and declines to touch the other + /// fifteen unless the strap actually stopped reporting the old value. One round-trip converts the + /// inference into a measurement on first run. See `R22DisableReport.probeKey` for why that key. + /// + /// What stays unverified even after a clean read-back is narrower than the byte: whether clearing the + /// flags makes the strap's R22 *behaviour* revert. Only wearing the strap and watching the deep records + /// stop answers that, and the report says so rather than claiming it. + public static let featureFlagOffValue: UInt8 = 0x30 + + /// The inverse of `enableR22Sequence`: the same sixteen keys, in the same order, each carrying + /// `featureFlagOffValue`. + /// + /// **Same order as the enable sequence, deliberately.** `enable_r22_packets` — the master flag — is + /// cleared FIRST, so a run interrupted by a disconnect leaves the strap nearer "off" than it started + /// rather than stranded with sub-streams cleared and the master still set. It also keeps the two + /// sequences trivially diffable: same names, same order, only the value byte differs. + /// + /// **On `disable_pip_r26_packets`, whose name inverts.** Fifteen of these keys read as "enable/tune X" + /// and one reads as "disable X", so writing the same byte to all sixteen looks wrong at a glance. It is + /// deliberate, and the reasoning is worth being explicit about because it is the one place a reviewer + /// should push back. + /// + /// This sequence is defined as *the undo of the enable sequence* — clear every flag that sequence set — + /// rather than as "set each flag to its semantic opposite". Those are different operations, and only + /// the first one is well defined here. Inverting per key would require knowing what `'1'` and `'2'` + /// each select, and this key is the evidence that we do not: a flag named `disable_*` written `'2'` + /// alongside `enable_*` flags also written `'2'` cannot be reconciled with plain-boolean semantics. + /// + /// So: writing `'0'` to `disable_pip_r26_packets` is expected to stop PIP R26 packets being suppressed + /// — i.e. to let them flow again, which is the pre-R22 behaviour and the correct restoration. The + /// polarity inversion is real, but it lives in what the key MEANS, not in which byte undoes it, and it + /// is called out in the report so nobody is surprised that one key's "off" is another feature's "on". + /// `make_hrfm_visible`, `wear_detect_bias`, `hr_ch_switching`, `ir_hw_switching` and `dorset_inhibit_wpt` + /// are milder versions of the same story: they tune behaviour rather than gate a stream, and clearing + /// them returns that tuning to whatever the firmware does with the flag unset. + /// + /// **What this does NOT do: restore a snapshot.** NOOP has never read these values *before* writing + /// them, so the strap's pre-NOOP state is unknown and unrecoverable. If the firmware's own default for + /// some key is `'1'` rather than unset, this writes a value the factory never used. The honest claim + /// is "clears the sixteen flags NOOP set", not "restores the strap to how it shipped". Making the + /// stronger claim true needs a read-before-write snapshot on the ENABLE path; see #174. + public static let disableR22Sequence: [Flag] = enableR22Sequence.map { + Flag($0.name, featureFlagOffValue) + } + + /// Every frame in the disable sequence, sequence-numbered from `firstSeq`. Written exactly like the + /// enable frames — same opcode, same 40-byte body, same spacing, WITH RESPONSE — because they differ + /// only in the value byte. + public static func disableSequenceFrames(firstSeq: UInt8 = 1) -> [[UInt8]] { + disableR22Sequence.enumerated().map { idx, flag in + frame(flag: flag, seq: UInt8((Int(firstSeq) + idx) & 0xFF)) + } + } } diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/R22DisableTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/R22DisableTests.swift new file mode 100644 index 0000000000..171e814265 --- /dev/null +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/R22DisableTests.swift @@ -0,0 +1,320 @@ +import XCTest +@testable import WhoopProtocol + +/// #174: the R22 DISABLE path — the sequence bytes, the key-aware `SET_FF_VALUE(120)` allowlist, and the +/// staged write→read-back run that turns the inferred off value into a measurement. +/// +/// Everything here runs with no app, no strap and no CoreBluetooth: `R22DisableReport` is pure and +/// order-dependent, so the whole verdict table — including the case where the strap refuses `'0'` — is +/// covered without hardware. That matters more than usual here, because the one thing that CANNOT be +/// tested off-strap is whether the firmware accepts the value, and the design is built so that the run +/// itself answers that on first use rather than assuming it. +final class R22DisableTests: XCTestCase { + + private func hex(_ b: [UInt8]) -> String { b.map { String(format: "%02x", $0) }.joined(separator: " ") } + + // MARK: - The sequence + + func testDisableSequenceMirrorsEnableExactly() { + let enable = Whoop5Config.enableR22Sequence + let disable = Whoop5Config.disableR22Sequence + XCTAssertEqual(disable.count, enable.count, "the undo must cover every key the enable sets") + XCTAssertEqual(disable.map(\.name), enable.map(\.name), + "same keys in the same order — the master flag is cleared first") + XCTAssertTrue(disable.allSatisfy { $0.value == Whoop5Config.featureFlagOffValue }) + } + + func testOffValueIsAsciiZero() { + XCTAssertEqual(Whoop5Config.featureFlagOffValue, 0x30) + XCTAssertEqual(Character(UnicodeScalar(Whoop5Config.featureFlagOffValue)), "0") + } + + /// The disable frame differs from the enable frame in EXACTLY one byte — the value at body offset 32 + /// (frame offset 12 + 32 = 44). Anything else changing means the encoding drifted. + func testDisableFrameDiffersFromEnableInOnlyTheValueByte() { + let on = Whoop5Config.frame(flag: Whoop5Config.Flag("enable_r22_packets", 0x32), seq: 1) + let off = Whoop5Config.frame(flag: Whoop5Config.Flag("enable_r22_packets", 0x30), seq: 1) + XCTAssertEqual(on.count, off.count) + let differing = (0.. DeviceConfigReadProbe.ValueResponse { + var record = [UInt8](repeating: 0, count: 32) + for (i, b) in Array(key.utf8).enumerated() where i < 32 { record[i] = b } + record.append(value) + return DeviceConfigReadProbe.ValueResponse(resultCode: resultCode, record: record) + } + + private func failureReply() -> DeviceConfigReadProbe.ValueResponse { + DeviceConfigReadProbe.ValueResponse(resultCode: 0, record: []) + } + + func testPlanStartsWithTheProbeWriteThenItsReadBack() { + var r = R22DisableReport() + let first = r.nextStep() + XCTAssertEqual(first?.stage, .probeWrite) + XCTAssertEqual(first?.key, R22DisableReport.probeKey) + XCTAssertEqual(first?.opcode, 120) + let second = r.nextStep() + XCTAssertEqual(second?.stage, .probeRead) + XCTAssertEqual(second?.key, R22DisableReport.probeKey) + XCTAssertEqual(second?.opcode, 128) + } + + /// The gate: a probe read-back that still shows the old value STOPS the run with fifteen keys untouched. + /// This is the case the staging exists for. + func testProbeRejectionStopsTheRunAndLeavesFifteenKeysUntouched() { + var r = R22DisableReport() + let write = r.nextStep()! + r.noteWriteAck(resultCode: 1, for: write) // SUCCESS ack — deliberately not believed + let read = r.nextStep()! + r.noteReadBack(reply(key: R22DisableReport.probeKey, value: 0x31), for: read) // still '1' + + XCTAssertNil(r.nextStep(), "no further steps once the off value is rejected") + XCTAssertEqual(r.verdict, .offValueRejected) + XCTAssertEqual(r.outcomes[R22DisableReport.probeKey], .unchanged) + let skipped = r.keys.filter { r.outcomes[$0] == .skipped } + XCTAssertEqual(skipped.count, 15, "the other fifteen must be left alone") + XCTAssertTrue(r.render().contains("is NOT how this firmware clears a feature flag")) + XCTAssertEqual(r.writeAcks[R22DisableReport.probeKey], 1, + "the SUCCESS ack is recorded even though the verdict is a rejection") + } + + /// A clean run: probe accepts, the other fifteen are written, all sixteen read back as '0'. + func testFullRunClearsAllSixteen() { + var r = R22DisableReport() + var steps = 0 + while let step = r.nextStep() { + steps += 1 + XCTAssertLessThan(steps, 100, "plan must terminate") + if step.isWrite { + XCTAssertEqual(step.opcode, 120) + r.noteWriteAck(resultCode: 1, for: step) + } else { + XCTAssertEqual(step.opcode, 128) + r.noteReadBack(reply(key: step.key, value: 0x30), for: step) + } + } + XCTAssertEqual(r.verdict, .allCleared) + XCTAssertEqual(r.keys.count, 16) + for key in r.keys { + XCTAssertEqual(r.outcomes[key], .cleared, "\(key) should be cleared") + } + // 1 probe write + 1 probe read + 15 clear writes + 16 verify reads + XCTAssertEqual(steps, 33) + XCTAssertTrue(r.summary.contains("All 16 R22 flags cleared")) + } + + /// FAILURE on a read-back means the key no longer has a stored value at all. Nothing predicted this + /// outcome, so it is reported as its own state rather than folded into "cleared". + func testFailureReadBackIsReportedAsUnsetNotCleared() { + var r = R22DisableReport() + let write = r.nextStep()! + r.noteWriteAck(resultCode: 1, for: write) + let read = r.nextStep()! + r.noteReadBack(failureReply(), for: read) + XCTAssertEqual(r.outcomes[R22DisableReport.probeKey], .unset) + XCTAssertTrue(r.probePassed, "an unset key is a success — the run should continue") + XCTAssertNotNil(r.nextStep(), "the clear stage must follow a passing probe") + } + + func testPartialRunIsReportedAsPartialNotSuccess() { + var r = R22DisableReport() + while let step = r.nextStep() { + if step.isWrite { + r.noteWriteAck(resultCode: 1, for: step) + } else if step.key == "wear_detect_bias" { + r.noteReadBack(reply(key: step.key, value: 0x32), for: step) // refused this one + } else { + r.noteReadBack(reply(key: step.key, value: 0x30), for: step) + } + } + XCTAssertEqual(r.verdict, .partial) + XCTAssertEqual(r.outcomes["wear_detect_bias"], .unchanged) + XCTAssertTrue(r.summary.contains("15 of 16")) + } + + /// A SUCCESS ack whose read-back did not move must never render as success — the #907/#891 rule. + func testSuccessAckWithUnmovedValueIsNeverReportedAsSuccess() { + var r = R22DisableReport() + let write = r.nextStep()! + r.noteWriteAck(resultCode: 1, for: write) + let read = r.nextStep()! + r.noteReadBack(reply(key: R22DisableReport.probeKey, value: 0x31), for: read) + let text = r.render() + XCTAssertFalse(r.verdict == .allCleared) + XCTAssertTrue(text.contains("recorded, not proof")) + XCTAssertTrue(text.contains("NOT treated as proof")) + } + + func testSilentProbeStopsTheRun() { + var r = R22DisableReport() + let write = r.nextStep()! + r.noteTimeout(for: write, seconds: 8) + XCTAssertNil(r.outcomes[R22DisableReport.probeKey], + "a silent WRITE decides nothing — only the read-back does") + let read = r.nextStep()! + r.noteTimeout(for: read, seconds: 8) + XCTAssertEqual(r.verdict, .probeInconclusive) + XCTAssertNil(r.nextStep()) + } + + func testAbandonMarksRemainingKeysSkipped() { + var r = R22DisableReport() + let write = r.nextStep()! + r.noteWriteAck(resultCode: 1, for: write) + r.noteAbandoned("link dropped") + XCTAssertEqual(r.verdict, .abandoned) + XCTAssertNil(r.nextStep()) + XCTAssertEqual(r.keys.filter { r.outcomes[$0] == .skipped }.count, 16) + } + + func testUndecodableReadBackIsNotTreatedAsCleared() { + var r = R22DisableReport() + let write = r.nextStep()! + r.noteWriteAck(resultCode: 1, for: write) + let read = r.nextStep()! + r.noteReadFailure(.crc, for: read) + XCTAssertEqual(r.outcomes[R22DisableReport.probeKey], .undecodable) + XCTAssertFalse(r.probePassed) + XCTAssertEqual(r.verdict, .probeInconclusive) + } + + /// The report must state the four standing caveats verbatim, so the log, the UI and the PR cannot drift + /// apart on what was actually established. + func testRenderCarriesTheCaveats() { + var r = R22DisableReport() + while let step = r.nextStep() { + if step.isWrite { r.noteWriteAck(resultCode: 1, for: step) } + else { r.noteReadBack(reply(key: step.key, value: 0x30), for: step) } + } + let text = r.render() + XCTAssertTrue(text.contains("'0' as the off value is INFERRED")) + XCTAssertTrue(text.contains("not the same as reverted BEHAVIOUR")) + XCTAssertTrue(text.contains("does not restore a snapshot")) + XCTAssertTrue(text.contains("disable_pip_r26_packets inverts")) + // Every key appears in the per-key table with its before value. + for key in r.keys { XCTAssertTrue(text.contains(key), "\(key) missing from the report") } + } + + func testProbeKeyIsTheOneWithAHardwareWriteDemonstration() { + XCTAssertEqual(R22DisableReport.probeKey, "enable_sig12") + XCTAssertTrue(Whoop5Config.enableR22Sequence.contains { $0.name == R22DisableReport.probeKey }) + } +} diff --git a/Strand/BLE/BLEManager.swift b/Strand/BLE/BLEManager.swift index aaad54f415..2d0d104f5d 100644 --- a/Strand/BLE/BLEManager.swift +++ b/Strand/BLE/BLEManager.swift @@ -1530,12 +1530,20 @@ public final class BLEManager: NSObject, ObservableObject { || (DeviceConfigReadProbe.isReadOnlyOpcode(command.rawValue) && deviceConfigReport != nil) || command == .sendHistoricalData || command == .historicalDataResult || command == .setClock || command == .getClock - // SET_CONFIG (the R22 deep-stream unlock) is allowed ONLY while the deep-data - // experiment is opted in — it writes a persistent feature flag to the strap, so it - // must never fire on a default install. It's still reversible (just changes which - // data the strap emits) and is what the official app sends. Driven only by - // enableWhoop5DeepData(). (#174) - || (command == .setConfig && PuffinExperiment.deepDataEnabled) + // SET_CONFIG / SET_FF_VALUE (120) — the R22 deep-stream unlock AND its undo. Allowed only + // while the deep-data experiment is opted in, and now additionally only for a KEY and a + // VALUE the gate recognises: one of the sixteen R22 flags, carrying either that key's own + // enable value or the documented off value. The clause this replaces was opcode-only, so + // it admitted ANY feature-flag key with ANY value for as long as the opt-in happened to be + // on — the same weakness #907 closed on opcode 119. Driven only by enableWhoop5DeepData() + // and disableWhoop5DeepData(). (#174) + || FeatureFlagWriteGate.admitsSend(opcode: command.rawValue, + payload: payload, + deepDataOptIn: PuffinExperiment.deepDataEnabled) + // GET_FF_VALUE (128) as the disable run's mandatory READ-BACK. Gated on a disable run being + // in flight, exactly like the read probes' 121/128 clause above, so a default install can + // never form these bytes. The write ack is not trusted; this is what proves the clear. + || (FeatureFlagWriteGate.isReadBackOpcode(command.rawValue) && r22DisableRun != nil) // SET_DEVICE_CONFIG (the Broadcast-HR flag) is allowed ONLY while that opt-in is on — // it writes one persistent device-config value so the strap advertises standard HR. // Reversible; driven only by setBroadcastHr(_:). (#181) @@ -2335,7 +2343,11 @@ public final class BLEManager: NSObject, ObservableObject { // entirely, so counting it as a "deep packet" gave 4.0 owners a bogus deep-data counter (#346). guard selectedModel.deviceFamily == .whoop5 else { return } guard frame.count > 10 else { return } - if frame[8] == 0x24, frame[10] == WhoopCommand.setConfig.rawValue { + // #174: a SET_CONFIG ack means "one enable_r22_* flag accepted" ONLY when we are enabling. A disable + // run writes through the same opcode, so without this guard turning R22 OFF would tick the + // "Strap accepted N/16 R22 flags" counter upwards — the exact opposite of what happened. The + // disable run scores its own acks (and does not trust them; the 128 read-back is its proof). + if frame[8] == 0x24, frame[10] == WhoopCommand.setConfig.rawValue, r22DisableRun == nil { state.r22FlagsAccepted += 1 let total = Whoop5Config.enableR22Sequence.count if state.r22FlagsAccepted == total { @@ -2406,10 +2418,163 @@ public final class BLEManager: NSObject, ObservableObject { } } DispatchQueue.main.asyncAfter(deadline: .now() + .milliseconds(80 * frames.count + 200)) { [weak self] in - self?.log("Deep-data: sequence sent. Keep the strap on, let it sync, then share your strap log — we're looking for new deep records (type-0x2F) to start arriving. (#174)") + self?.log("Deep-data: sequence sent. Keep the strap on, let it sync, then share your strap log — we're looking for new deep records (type-0x2F) to start arriving. To undo it later, use \"Turn deep data back off\" — disableWhoop5DeepData() writes '0' to the same sixteen keys and reads each one back. (#174)") } } + // MARK: - R22 disable (#174) + + /// Per-step reply window for the disable run. Each step is only sent once the previous reply lands (or + /// times out), so this bounds one round-trip, not the whole run. + private static let r22DisableStepTimeout: TimeInterval = 8 + + /// The in-flight disable run's accumulating report; nil when none is running. Doubles as the send() + /// allowlist's in-flight gate for the 128 read-back. + private var r22DisableRun: R22DisableReport? + /// The step whose reply we are waiting for, nil between steps. + private var r22DisableAwaiting: R22DisableReport.Step? + /// Monotonic step counter so a late timeout from an earlier step cannot cancel a live run. + private var r22DisableStep = 0 + + /// EXPERIMENTAL (#174): the undo of `enableWhoop5DeepData()` — write `'0'` to the sixteen `enable_r22_*` + /// feature flags and report, per key, the value the strap actually stores afterwards. + /// + /// **Why this exists.** The Settings copy has promised since #174 that the R22 unlock is "reversible", + /// and that was true about the hardware and false about the app: NOOP shipped an enable button and no + /// way back, so the only routes out were the official WHOOP app or a factory reset. This closes that. + /// + /// **Why it is staged rather than sixteen writes.** `'0'` is the confirmed off value in the + /// device-config namespace (#181, hardware-validated on the Broadcast-HR flag) and that namespace shares + /// this one's entire wire idiom — but no *feature* flag has ever been observed holding `'0'`, and the two + /// namespaces are proven separate at the verb level. So the run writes ONE flag, reads it back, and only + /// touches the other fifteen if the strap actually stopped reporting the old value. A firmware that + /// refuses `'0'` costs the user one round-trip and yields a publishable answer instead of fifteen mystery + /// writes. See `Whoop5Config.featureFlagOffValue` and `R22DisableReport` for the full reasoning. + /// + /// **The ack is never the proof.** Every write's COMMAND_RESPONSE is recorded and not believed — + /// `SELECT_WRIST` returns SUCCESS for a no-op and FAILURE for a real mutation on this firmware — so only + /// the value a `GET_FF_VALUE(128)` read returns is reported as state (#907/#891). + /// + /// Same guards as the enable: 5/MG family, explicit opt-in, full encrypted bond. `worn` is deliberately + /// NOT required — the on-wrist gate exists because the R22 STREAM is on-wrist-only, and a user turning + /// the feature off should not have to put the strap back on to do it. Result goes to + /// `LiveState.r22DisableReport` and the strap log; no new storage. Twin of Android + /// `disableWhoop5DeepData()`. + public func disableWhoop5DeepData() { + guard selectedModel.deviceFamily == .whoop5 else { + log("Deep-data disable: needs a WHOOP 5.0/MG strap selected — ignored."); return + } + guard PuffinExperiment.deepDataEnabled else { + // The send allowlist keys off this same pref, so the writes would be dropped anyway. Fail with a + // message that says what to do rather than letting sixteen frames vanish silently. + log("Deep-data disable: the deep-data opt-in is off, so the writes would be refused by the send allowlist. Turn the toggle back on, tap the disable button, then turn it off — ignored."); return + } + guard state.connected, state.encryptedBond else { + log("Deep-data disable: needs the full encrypted bond, not the live-HR-only link. Close the official WHOOP app, put the strap in pairing mode, and bond it to NOOP first — ignored."); return + } + guard r22DisableRun == nil else { + log("Deep-data disable: a disable run is already walking its plan — ignored."); return + } + r22DisableRun = R22DisableReport() + state.r22DisableReport = BLEManager.deviceConfigProbeWaiting + log("Deep-data disable (#174): clearing the \(Whoop5Config.disableR22Sequence.count)-flag R22 sequence — writing '0' via SET_FF_VALUE(120), each verified with GET_FF_VALUE(128). Probing \(R22DisableReport.probeKey) first; the other flags are only written if that one moves.") + advanceR22Disable() + } + + /// Send the next planned step, or finish when the plan is done. + private func advanceR22Disable() { + guard let step = r22DisableRun?.nextStep() else { + finishR22Disable() + return + } + guard let command = WhoopCommand(rawValue: step.opcode) else { + log("Deep-data disable: refusing to send unknown opcode \(step.opcode)") + finishR22Disable() + return + } + let payload: [UInt8] + if step.isWrite { + payload = [0x01] + Whoop5Config.payloadBody(name: step.key, value: Whoop5Config.featureFlagOffValue) + // Fail closed: the same predicate the send path consults, asked here too, so a future edit that + // widened the plan cannot put an unrecognised key or value on the wire. + guard FeatureFlagWriteGate.admitsSend(opcode: step.opcode, payload: payload, + deepDataOptIn: PuffinExperiment.deepDataEnabled) else { + log("Deep-data disable: refusing to write \(step.key) — not admitted by the feature-flag gate") + finishR22Disable() + return + } + } else { + guard FeatureFlagWriteGate.isReadBackOpcode(step.opcode) else { + log("Deep-data disable: refusing to read with opcode \(step.opcode)") + finishR22Disable() + return + } + payload = DeviceConfigReadProbe.requestBody(key: step.key) + } + r22DisableStep &+= 1 + r22DisableAwaiting = step + let armed = r22DisableStep + send(command, payload: payload, writeType: .withResponse) + // BLE callbacks and this timer both run on the main queue, so guard-then-advance is race-free: a + // reply that already landed advanced `r22DisableStep`, and this stale closure no-ops. + DispatchQueue.main.asyncAfter(deadline: .now() + BLEManager.r22DisableStepTimeout) { [weak self] in + guard let self, self.r22DisableRun != nil, self.r22DisableStep == armed, + self.r22DisableAwaiting != nil else { return } + self.r22DisableAwaiting = nil + self.r22DisableRun?.noteTimeout(for: step, seconds: Int(BLEManager.r22DisableStepTimeout)) + self.advanceR22Disable() + } + } + + /// Render + publish + log the report and end the run (which also re-closes the 128 send allowlist). + private func finishR22Disable() { + guard let report = r22DisableRun else { return } + r22DisableRun = nil + r22DisableAwaiting = nil + // The "Strap accepted N/16 R22 flags" line reports the last ENABLE send. If this run cleared even one + // key that claim is stale, so retire it rather than leaving the card asserting the strap is unlocked + // while the report below says it was just cleared. A run that changed nothing leaves it alone. + if report.outcomes.values.contains(where: { $0.isSuccess }) { + state.r22FlagsAccepted = 0 + } + let text = report.render() + log("Deep-data disable (#174):\n\(text)") + state.r22DisableReport = text + } + + /// Clear the #174 disable result (Settings card dismissed). Twin of Android clearR22DisableReport(). + public func clearR22DisableReport() { state.r22DisableReport = nil } + + /// #174: one COMMAND_RESPONSE belonging to a disable run — either a `SET_FF_VALUE(120)` write ack or a + /// `GET_FF_VALUE(128)` read-back. Guarded on a run being IN-FLIGHT so a stray byte match can never + /// surface a result. Parsing — including the CRC gate — lives in the pure `DeviceConfigReadProbe`. + private func handleR22DisableResponse(_ frame: [UInt8], isWhoop5: Bool) { + guard r22DisableRun != nil, let step = r22DisableAwaiting else { return } + r22DisableAwaiting = nil + if step.isWrite { + // The ack is recorded for the transcript and decides nothing. `pay[1]` is the 5/MG result code: + // puffin envelope puts the type @8 and the cmd @10, so the payload starts at 11 and the result + // byte is at 12. + let resultCode: Int? = frame.count > 12 ? Int(frame[12]) : nil + r22DisableRun?.noteWriteAck(resultCode: resultCode, for: step) + } else { + switch DeviceConfigReadProbe.parse(frame: frame, family: isWhoop5 ? .whoop5 : .whoop4, + expecting: step.opcode) { + case .success(let r): r22DisableRun?.noteReadBack(r, for: step) + case .failure(let f): r22DisableRun?.noteReadFailure(f, for: step) + } + } + advanceR22Disable() + } + + /// Abandon a disable run the link interrupted, re-closing the 128 send allowlist. Called from the + /// disconnect path alongside the probe teardowns. + private func abandonR22DisableRun(_ why: String) { + guard r22DisableRun != nil else { return } + r22DisableRun?.noteAbandoned(why) + finishR22Disable() + } + /// EXPERIMENTAL (#181): make a bonded WHOOP 5/MG advertise its heart rate as a standard BLE HR /// sensor (0x180D + the live HR in its manufacturer data) by writing the device-config flag /// `whoop_live_hr_in_adv_ind_pkt` = "1" (on) / "0" (off) via SET_DEVICE_CONFIG (0x77). With it on, a @@ -3844,6 +4009,10 @@ extension BLEManager: @preconcurrency CBCentralManagerDelegate { // …and abandon a plan the link interrupted, re-closing the 121/128 send() allowlist. deviceConfigReport = nil deviceConfigAwaiting = nil + // #174: a disable run interrupted mid-plan is RENDERED rather than dropped — unlike the read-only + // probes above it has already written to the strap, so the user must be told exactly how far it got + // and which keys are still set. This publishes the partial report and re-closes the 128 allowlist. + abandonR22DisableRun("the strap disconnected mid-run") state.clearBiometrics() // and a stale HR / R-R must not outlive the link either state.liveFeedActive = false // a drop while Live is open must not leave a stale "Stop live feed" didBond = false @@ -4756,6 +4925,15 @@ extension BLEManager: @preconcurrency CBPeripheralDelegate { DeviceConfigReadProbe.isReadOnlyOpcode(frame[10]) { handleDeviceConfigProbeResponse(frame, isWhoop5: true) } + // #174: a reply belonging to an R22 DISABLE run — either the SET_FF_VALUE(120) write ack + // or the GET_FF_VALUE(128) read-back that verifies it. In-flight-guarded inside, so this + // is a byte compare on every other frame. Note 128 is also matched by the read-probe + // clause above; both handlers guard on their OWN run being live, so exactly one acts. + if frame.count > 10, frame[8] == 0x24, + frame[10] == WhoopCommand.setConfig.rawValue + || frame[10] == WhoopCommand.getFeatureFlagValue.rawValue { + handleR22DisableResponse(frame, isWhoop5: true) + } // #695: a 5/MG GET_DATA_RANGE COMMAND_RESPONSE (puffin envelope: type @8, cmd @10). Feeds // the SAME newest/oldest window + backfill gate + diagnostics as the 4.0 path above — this // reply was previously ignored on 5/MG (the 4.0 handler keyed on frame[6]). cmdOff = 10. diff --git a/Strand/BLE/LiveState.swift b/Strand/BLE/LiveState.swift index 46b3fb2122..68054fd138 100644 --- a/Strand/BLE/LiveState.swift +++ b/Strand/BLE/LiveState.swift @@ -256,6 +256,14 @@ public final class LiveState: ObservableObject { /// sentinel while the walk runs. Nothing is written to the strap to produce it. Cleared on disconnect /// and on dialog dismiss. Twin of the Android WhoopBleClient.deviceConfigProbe flow. @Published public var deviceConfigProbe: String? = nil + + /// #174: the R22 DISABLE report — the per-key result of writing `'0'` to the sixteen feature flags and + /// reading every one of them back with `GET_FF_VALUE`(128), or the waiting sentinel while the run walks. + /// Unlike the two probes above this one DOES write, which is exactly why it reports the value the strap + /// stores rather than the write's own ack. Cleared on disconnect and on dialog dismiss. Twin of the + /// Android WhoopBleClient.r22DisableReport flow. + @Published public var r22DisableReport: String? = nil + /// Wrist-wear state from WRIST_ON/WRIST_OFF events. Defaults true so wear-gated features work /// before the first event arrives; flipped by FrameRouter on a real event. @Published public var worn: Bool = true diff --git a/Strand/BLE/PuffinExperiment.swift b/Strand/BLE/PuffinExperiment.swift index 4cb7c005e2..0e380b91bb 100644 --- a/Strand/BLE/PuffinExperiment.swift +++ b/Strand/BLE/PuffinExperiment.swift @@ -16,8 +16,13 @@ enum PuffinExperiment { /// Separate, more-deliberate opt-in for the WHOOP 5/MG "R22" deep-data unlock — the one probe /// that WRITES a persistent feature flag to the strap (the `enable_r22_*` SET_CONFIG sequence the /// official app sends; documented by judes.club + Asherlc/dofek). Kept distinct from the read-only - /// probes above because it changes strap state, so it must be turned on explicitly and is still - /// fully reversible. Driven only from `BLEManager.enableWhoop5DeepData()`. (#174) + /// probes above because it changes strap state, so it must be turned on explicitly. + /// + /// Reversible, and now actually reversed by code rather than only in the comment: this key gates BOTH + /// `BLEManager.enableWhoop5DeepData()` and `BLEManager.disableWhoop5DeepData()`, and the send allowlist + /// consults `FeatureFlagWriteGate` so only the sixteen R22 keys — carrying either their enable value or + /// the off value — can travel through opcode 120. Note the pref itself writes NOTHING in either + /// direction; it only decides whether a send is permitted. (#174) static let deepDataKey = "noopWhoop5DeepData" static var deepDataEnabled: Bool { UserDefaults.standard.bool(forKey: deepDataKey) } diff --git a/Strand/Resources/Localizable.xcstrings b/Strand/Resources/Localizable.xcstrings index d7826b593c..fc0863cd8d 100644 --- a/Strand/Resources/Localizable.xcstrings +++ b/Strand/Resources/Localizable.xcstrings @@ -170616,6 +170616,578 @@ } } } + }, + "Clear the R22 flags on your strap?": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "R22-Flags auf deinem Strap löschen?" + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "¿Borrar las marcas R22 de tu pulsera?" + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Effacer les indicateurs R22 sur votre bracelet ?" + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Cancellare i flag R22 sulla fascia?" + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "Limpar as marcas R22 na sua bracelete?" + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Очистить флаги R22 на браслете?" + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "要清除手环上的 R22 标记吗？" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "要清除手環上的 R22 標記嗎？" + } + } + } + }, + "Clear flags on strap": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Flags auf dem Strap löschen" + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Borrar marcas en la pulsera" + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Effacer les indicateurs" + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Cancella i flag sulla fascia" + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "Limpar marcas na bracelete" + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Очистить флаги на браслете" + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "清除手环上的标记" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "清除手環上的標記" + } + } + } + }, + "Just stop sending": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Nur nicht mehr senden" + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Solo dejar de enviar" + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Cesser seulement d’envoyer" + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Smetti solo di inviare" + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "Apenas parar de enviar" + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Просто перестать отправлять" + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "仅停止发送" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "僅停止傳送" + } + } + } + }, + "Turning this switch off only stops NOOP sending the unlock. The flags it already wrote stay on the strap until something clears them. NOOP can write the off value to all 16 now and read each one back so you can see what the strap actually stores. Needs the strap connected and bonded.": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Diesen Schalter auszuschalten stoppt nur, dass NOOP die Freischaltung sendet. Die bereits geschriebenen Flags bleiben auf dem Strap, bis sie etwas löscht. NOOP kann jetzt den Aus-Wert in alle 16 schreiben und jedes zurücklesen, damit du siehst, was der Strap tatsächlich speichert. Der Strap muss verbunden und gekoppelt sein." + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Apagar este interruptor solo impide que NOOP envíe el desbloqueo. Las marcas ya escritas siguen en la pulsera hasta que algo las borre. NOOP puede escribir ahora el valor de apagado en las 16 y releer cada una para que veas lo que la pulsera guarda realmente. Requiere la pulsera conectada y vinculada." + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Désactiver cet interrupteur empêche seulement NOOP d’envoyer le déverrouillage. Les indicateurs déjà écrits restent sur le bracelet jusqu’à ce que quelque chose les efface. NOOP peut maintenant écrire la valeur d’arrêt dans les 16 et relire chacun d’eux pour que vous voyiez ce que le bracelet enregistre réellement. Nécessite le bracelet connecté et appairé." + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Disattivare questo interruttore impedisce solo a NOOP di inviare lo sblocco. I flag già scritti restano sulla fascia finché qualcosa non li cancella. NOOP può scrivere ora il valore di spegnimento su tutti e 16 e rileggerli uno a uno, così vedi che cosa la fascia memorizza davvero. Richiede la fascia connessa e associata." + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "Desligar este interruptor apenas impede o NOOP de enviar o desbloqueio. As marcas já escritas permanecem na bracelete até que algo as limpe. O NOOP pode agora escrever o valor de desativação nas 16 e reler cada uma para que veja o que a bracelete guarda realmente. Necessita da bracelete ligada e emparelhada." + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Выключение этого переключателя лишь останавливает отправку разблокировки из NOOP. Уже записанные флаги остаются на браслете, пока что-нибудь их не очистит. NOOP может сейчас записать значение выключения во все 16 и прочитать каждый обратно, чтобы вы увидели, что браслет действительно хранит. Требуется подключённый и сопряжённый браслет." + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "关闭此开关只会让 NOOP 停止发送解锁序列。已写入的标记会留在手环上，直到有东西清除它们。NOOP 现在可以向全部 16 个标记写入关闭值并逐一读回，让你看到手环实际存储的内容。需要手环已连接并完成配对。" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "關閉此開關只會讓 NOOP 停止傳送解鎖序列。已寫入的標記會留在手環上，直到有東西清除它們。NOOP 現在可以向全部 16 個標記寫入關閉值並逐一讀回，讓你看到手環實際儲存的內容。需要手環已連線並完成配對。" + } + } + } + }, + "WHOOP 5/MG straps hand a fresh app only live heart rate. The official app switches on the deeper streams (high-rate HR + motion + history) by writing a set of feature flags, a sequence two independent projects have documented. With this on, the button below sends that exact sequence to your strap. Unlike everything else here it does write to the strap — and it is reversible: “Turn deep data back off” writes the off value to the same flags and then reads every one of them back, so you see what the strap actually stores rather than just that it acked. Experimental: it may do nothing on your firmware. iPhone/Android only. A Mac can't write to a 5/MG.": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "WHOOP 5/MG-Straps geben einer neuen App nur den Live-Puls. Die offizielle App schaltet die tieferen Datenströme (hochfrequenter Puls + Bewegung + Verlauf) frei, indem sie eine Reihe von Feature-Flags schreibt — eine Sequenz, die zwei unabhängige Projekte dokumentiert haben. Wenn dies aktiv ist, sendet die Schaltfläche unten genau diese Sequenz an deinen Strap. Anders als alles andere hier schreibt sie auf den Strap — und sie ist umkehrbar: „Tiefe Daten wieder ausschalten“ schreibt den Aus-Wert in dieselben Flags und liest anschließend jedes einzeln zurück, sodass du siehst, was der Strap tatsächlich speichert, statt nur, dass er bestätigt hat. Experimentell: Auf deiner Firmware bewirkt es womöglich nichts. Nur iPhone/Android. Ein Mac kann nicht auf einen 5/MG schreiben." + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Las pulseras WHOOP 5/MG solo entregan la frecuencia cardiaca en vivo a una app nueva. La app oficial activa los flujos más profundos (FC de alta frecuencia + movimiento + historial) escribiendo un conjunto de marcas de función, una secuencia documentada por dos proyectos independientes. Con esto activado, el botón de abajo envía esa secuencia exacta a tu pulsera. A diferencia de todo lo demás aquí, sí escribe en la pulsera, y es reversible: «Volver a desactivar los datos profundos» escribe el valor de apagado en esas mismas marcas y luego relee cada una, de modo que veas lo que la pulsera guarda realmente y no solo que confirmó. Experimental: puede que no haga nada en tu firmware. Solo iPhone/Android. Un Mac no puede escribir en una 5/MG." + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Les bracelets WHOOP 5/MG ne transmettent à une nouvelle app que la fréquence cardiaque en direct. L’app officielle active les flux plus détaillés (FC haute fréquence + mouvement + historique) en écrivant un ensemble d’indicateurs de fonctionnalité, une séquence documentée par deux projets indépendants. Avec cette option activée, le bouton ci-dessous envoie exactement cette séquence à votre bracelet. Contrairement à tout le reste ici, elle écrit bien sur le bracelet — et elle est réversible : « Désactiver à nouveau les données détaillées » écrit la valeur d’arrêt dans ces mêmes indicateurs puis relit chacun d’eux, pour que vous voyiez ce que le bracelet enregistre réellement et pas seulement qu’il a accusé réception. Expérimental : cela peut ne rien faire sur votre firmware. iPhone/Android uniquement. Un Mac ne peut pas écrire sur un 5/MG." + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Le fasce WHOOP 5/MG offrono a una nuova app solo la frequenza cardiaca in tempo reale. L’app ufficiale attiva i flussi più profondi (FC ad alta frequenza + movimento + cronologia) scrivendo una serie di flag di funzionalità, una sequenza documentata da due progetti indipendenti. Con questa opzione attiva, il pulsante qui sotto invia esattamente quella sequenza alla tua fascia. A differenza di tutto il resto qui, scrive davvero sulla fascia — ed è reversibile: «Disattiva di nuovo i dati approfonditi» scrive il valore di spegnimento sugli stessi flag e poi li rilegge tutti, così vedi che cosa la fascia memorizza davvero e non solo che ha confermato. Sperimentale: sul tuo firmware potrebbe non fare nulla. Solo iPhone/Android. Un Mac non può scrivere su un 5/MG." + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "As braceletes WHOOP 5/MG entregam a uma app nova apenas a frequência cardíaca em direto. A app oficial ativa os fluxos mais profundos (FC de alta frequência + movimento + histórico) escrevendo um conjunto de marcas de funcionalidade, uma sequência documentada por dois projetos independentes. Com isto ativado, o botão abaixo envia essa sequência exata para a sua bracelete. Ao contrário de tudo o resto aqui, escreve mesmo na bracelete — e é reversível: «Voltar a desativar os dados profundos» escreve o valor de desativação nessas mesmas marcas e depois relê cada uma, para que veja o que a bracelete guarda realmente e não apenas que confirmou. Experimental: pode não fazer nada no seu firmware. Apenas iPhone/Android. Um Mac não consegue escrever numa 5/MG." + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Браслеты WHOOP 5/MG передают новому приложению только пульс в реальном времени. Официальное приложение включает более подробные потоки (частый пульс + движение + история), записывая набор функциональных флагов — последовательность, задокументированную двумя независимыми проектами. При включённой опции кнопка ниже отправляет браслету именно эту последовательность. В отличие от всего остального здесь, она действительно пишет на браслет — и это обратимо: «Снова отключить подробные данные» записывает значение выключения в те же флаги, а затем читает каждый обратно, чтобы вы увидели, что браслет действительно хранит, а не только что он подтвердил. Экспериментально: на вашей прошивке может не дать эффекта. Только iPhone/Android. Mac не может писать на 5/MG." + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "WHOOP 5/MG 手环只向新应用提供实时心率。官方应用通过写入一组功能标记来开启更深层的数据流（高频心率 + 运动 + 历史），这一序列已由两个独立项目记录。启用后，下方按钮会向你的手环发送完全相同的序列。与这里其他项不同，它确实会写入手环——而且可以撤销：“重新关闭深层数据”会向同一批标记写入关闭值，然后逐一读回，让你看到手环实际存储的内容，而不仅仅是它已确认。实验性功能：在你的固件上可能毫无作用。仅限 iPhone/Android。Mac 无法写入 5/MG。" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "WHOOP 5/MG 手環只向新應用程式提供即時心率。官方應用程式透過寫入一組功能標記來開啟更深層的資料流（高頻心率 + 動作 + 歷史），這一序列已由兩個獨立專案記錄。啟用後，下方按鈕會向你的手環傳送完全相同的序列。與這裡其他項不同，它確實會寫入手環——而且可以復原：「重新關閉深層資料」會向同一批標記寫入關閉值，然後逐一讀回，讓你看到手環實際儲存的內容，而不僅僅是它已確認。實驗性功能：在你的韌體上可能毫無作用。僅限 iPhone/Android。Mac 無法寫入 5/MG。" + } + } + } + }, + "Strap accepted all %lld R22 flags": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Strap hat alle %lld R22-Flags akzeptiert" + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "La pulsera aceptó las %lld marcas R22" + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Le bracelet a accepté les %lld indicateurs R22" + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "La fascia ha accettato tutti i %lld flag R22" + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "A bracelete aceitou as %lld marcas R22" + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Браслет принял все %lld флагов R22" + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "手环已接受全部 %lld 个 R22 标记" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "手環已接受全部 %lld 個 R22 標記" + } + } + } + }, + "Strap accepted %lld/%lld R22 flags…": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Strap hat %lld/%lld R22-Flags akzeptiert…" + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "La pulsera aceptó %lld/%lld marcas R22…" + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Le bracelet a accepté %lld/%lld indicateurs R22…" + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "La fascia ha accettato %lld/%lld flag R22…" + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "A bracelete aceitou %lld/%lld marcas R22…" + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Браслет принял %lld/%lld флагов R22…" + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "手环已接受 %lld/%lld 个 R22 标记…" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "手環已接受 %lld/%lld 個 R22 標記…" + } + } + } + }, + "Clearing R22 flags and reading each one back…": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "R22-Flags werden gelöscht und einzeln zurückgelesen…" + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Borrando las marcas R22 y releyendo cada una…" + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Effacement des indicateurs R22 et relecture de chacun…" + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Cancellazione dei flag R22 e rilettura di ciascuno…" + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "A limpar as marcas R22 e a reler cada uma…" + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Очистка флагов R22 и обратное чтение каждого…" + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "正在清除 R22 标记并逐一读回…" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "正在清除 R22 標記並逐一讀回…" + } + } + } + }, + "Turning deep data back off needs an iPhone or Android. A Mac can't form the encrypted bond a 5/MG requires.": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Tiefe Daten wieder auszuschalten benötigt ein iPhone oder Android. Ein Mac kann die verschlüsselte Kopplung, die ein 5/MG verlangt, nicht herstellen." + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Volver a desactivar los datos profundos requiere un iPhone o Android. Un Mac no puede formar el vínculo cifrado que exige una 5/MG." + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Désactiver à nouveau les données détaillées nécessite un iPhone ou un Android. Un Mac ne peut pas former l’appairage chiffré qu’exige un 5/MG." + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Riattivare lo spegnimento dei dati approfonditi richiede un iPhone o un Android. Un Mac non può creare l’associazione cifrata richiesta da un 5/MG." + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "Voltar a desativar os dados profundos requer um iPhone ou Android. Um Mac não consegue formar a ligação encriptada que uma 5/MG exige." + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Чтобы снова отключить подробные данные, нужен iPhone или Android. Mac не может установить шифрованное сопряжение, которого требует 5/MG." + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "重新关闭深层数据需要 iPhone 或 Android。Mac 无法建立 5/MG 所需的加密配对。" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "重新關閉深層資料需要 iPhone 或 Android。Mac 無法建立 5/MG 所需的加密配對。" + } + } + } + }, + "Needs the full encrypted bond: close the official WHOOP app and pair the strap to NOOP first (a live-HR-only link can't carry the write).": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Benötigt die vollständig verschlüsselte Kopplung: Schließe zuerst die offizielle WHOOP-App und koppele den Strap mit NOOP (eine Verbindung nur für den Live-Puls kann den Schreibvorgang nicht übertragen)." + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Necesita el vínculo cifrado completo: cierra primero la app oficial de WHOOP y empareja la pulsera con NOOP (un enlace solo de FC en vivo no puede transportar la escritura)." + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Nécessite l’appairage chiffré complet : fermez d’abord l’app WHOOP officielle et appairez le bracelet à NOOP (une liaison limitée à la FC en direct ne peut pas transporter l’écriture)." + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Richiede l’associazione cifrata completa: chiudi prima l’app WHOOP ufficiale e associa la fascia a NOOP (un collegamento solo per la FC in tempo reale non può trasportare la scrittura)." + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "Necessita da ligação encriptada completa: feche primeiro a app oficial da WHOOP e emparelhe a bracelete com o NOOP (uma ligação apenas de FC em direto não consegue transportar a escrita)." + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Требуется полное шифрованное сопряжение: сначала закройте официальное приложение WHOOP и сопрягите браслет с NOOP (канал только для пульса в реальном времени не может передать запись)." + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "需要完整的加密配对：请先关闭官方 WHOOP 应用并将手环与 NOOP 配对（仅传输实时心率的连接无法承载写入）。" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "需要完整的加密配對：請先關閉官方 WHOOP 應用程式並將手環與 NOOP 配對（僅傳輸即時心率的連線無法承載寫入）。" + } + } + } + }, + "Writes the off value to all 16 flags and reads each one back, so you see what the strap stores rather than just that it acked. The off value is inferred from the flag NOOP already turns off this way for Garmin broadcast — it has never been seen on an R22 flag, so NOOP tries one flag first and stops if the strap refuses it. Clearing the flags is not the same as watching the deep records stop: check that by syncing afterwards.": { + "localizations": { + "de": { + "stringUnit": { + "state": "translated", + "value": "Schreibt den Aus-Wert in alle 16 Flags und liest jedes einzeln zurück, sodass du siehst, was der Strap tatsächlich speichert — nicht nur, dass er bestätigt hat. Der Aus-Wert ist von dem Flag abgeleitet, das NOOP für die Garmin-Übertragung bereits so ausschaltet; auf einem R22-Flag wurde er noch nie beobachtet. Deshalb testet NOOP zuerst ein einzelnes Flag und bricht ab, wenn der Strap es ablehnt. Gelöschte Flags sind nicht dasselbe wie tatsächlich versiegende Tiefendaten: prüfe das mit einer anschließenden Synchronisierung." + } + }, + "es": { + "stringUnit": { + "state": "translated", + "value": "Escribe el valor de apagado en las 16 marcas y vuelve a leer cada una, para que veas lo que la pulsera guarda en vez de solo que confirmó. El valor de apagado se deduce de la marca que NOOP ya desactiva así para la difusión Garmin; nunca se ha visto en una marca R22, por eso NOOP prueba primero con una sola y se detiene si la pulsera la rechaza. Borrar las marcas no es lo mismo que ver parar los registros profundos: compruébalo sincronizando después." + } + }, + "fr": { + "stringUnit": { + "state": "translated", + "value": "Écrit la valeur d’arrêt dans les 16 indicateurs et relit chacun d’eux, pour que vous voyiez ce que le bracelet enregistre réellement et pas seulement qu’il a accusé réception. La valeur d’arrêt est déduite de l’indicateur que NOOP désactive déjà ainsi pour la diffusion Garmin ; elle n’a jamais été observée sur un indicateur R22, c’est pourquoi NOOP en teste un seul d’abord et s’arrête si le bracelet le refuse. Effacer les indicateurs n’équivaut pas à voir cesser les enregistrements détaillés : vérifiez-le en synchronisant ensuite." + } + }, + "it": { + "stringUnit": { + "state": "translated", + "value": "Scrive il valore di spegnimento su tutti e 16 i flag e li rilegge uno a uno, così vedi che cosa la fascia memorizza e non solo che ha confermato. Il valore di spegnimento è dedotto dal flag che NOOP già disattiva così per la trasmissione Garmin: non è mai stato osservato su un flag R22, perciò NOOP ne prova prima uno solo e si ferma se la fascia lo rifiuta. Cancellare i flag non equivale a vedere fermarsi le registrazioni approfondite: verificalo sincronizzando dopo." + } + }, + "pt-PT": { + "stringUnit": { + "state": "translated", + "value": "Escreve o valor de desativação nas 16 marcas e lê cada uma de volta, para que veja o que a pulseira guarda e não apenas que confirmou. O valor de desativação é inferido da marca que o NOOP já desliga assim para a transmissão Garmin — nunca foi visto numa marca R22, por isso o NOOP testa primeiro uma só e pára se a pulseira a recusar. Limpar as marcas não é o mesmo que ver os registos profundos pararem: confirme isso sincronizando depois." + } + }, + "ru": { + "stringUnit": { + "state": "translated", + "value": "Записывает значение выключения во все 16 флагов и читает каждый обратно, чтобы вы видели, что браслет хранит, а не только что он подтвердил. Значение выключения выведено из флага, который NOOP уже так отключает для трансляции на Garmin: на флаге R22 оно никогда не наблюдалось, поэтому NOOP сначала пробует один флаг и останавливается, если браслет его отвергнет. Очистка флагов — не то же самое, что увидеть остановку подробных записей: проверьте это последующей синхронизацией." + } + }, + "zh-Hans": { + "stringUnit": { + "state": "translated", + "value": "向全部 16 个标记写入关闭值，并逐一读回，让你看到手环实际存储的内容，而不仅仅是它已确认。该关闭值是从 NOOP 已用同样方式关闭的 Garmin 广播标记推断出来的——在 R22 标记上从未被观察到，因此 NOOP 会先试一个标记，若手环拒绝就停止。清除标记与看到深层记录停止并不相同：之后同步一次来确认。" + } + }, + "zh-Hant": { + "stringUnit": { + "state": "translated", + "value": "向全部 16 個標記寫入關閉值，並逐一讀回，讓你看到手環實際儲存的內容，而不僅僅是它已確認。該關閉值是從 NOOP 已用同樣方式關閉的 Garmin 廣播標記推斷出來的——在 R22 標記上從未被觀察到，因此 NOOP 會先試一個標記，若手環拒絕就停止。清除標記與看到深層記錄停止並不相同：之後同步一次來確認。" + } + } + } } }, "version": "1.0" diff --git a/Strand/Screens/SettingsView.swift b/Strand/Screens/SettingsView.swift index c53b3182c5..d4dab46943 100644 --- a/Strand/Screens/SettingsView.swift +++ b/Strand/Screens/SettingsView.swift @@ -10,6 +10,9 @@ import PhotosUI import StrandDesign import StrandAnalytics import WhoopStore +// #174: the R22 card reads the flag COUNT off `Whoop5Config.enableR22Sequence` rather than restating it — +// the hardcoded "15" outlived the sequence growing to 16 and declared success a flag early. +import WhoopProtocol /// Settings — profile (powers zones / calories / recovery), strap connection, and about. /// Grouped cards on surface.raised with a two-column form feel. @@ -37,6 +40,13 @@ struct SettingsView: View { /// persistent feature flag to the strap. See [PuffinExperiment.deepDataKey]. (#174) @AppStorage(PuffinExperiment.deepDataKey) private var deepDataEnabled = false + /// #174: set when the deep-data switch is turned OFF, so the app can OFFER to clear the flags on the + /// strap instead of silently leaving them set. The switch alone has never written anything in either + /// direction — it gates sends — so turning it off used to change nothing on the hardware while reading + /// like an undo. Asking is the right shape rather than writing automatically: the strap may not be + /// connected, and a write to bonded hardware is not something a toggle should do unannounced. + @State private var confirmingDeepDataDisable = false + /// Opt-in "Broadcast heart rate" (off by default) — makes the strap advertise its HR as a standard /// BLE sensor for Garmin/Zwift/gym kit. See [PuffinExperiment.broadcastHrKey]. (#181) @AppStorage(PuffinExperiment.broadcastHrKey) private var broadcastHrEnabled = false @@ -240,6 +250,16 @@ struct SettingsView: View { } message: { Text("This restarts the roughly 4-night build-up for Charge and your HRV baseline. Your history stays. Use it if a bad first week, like wearing it while sick, set your baseline off.") } + // #174: the switch going OFF is the moment to offer the undo. Declining leaves the flags set and + // says so — which is still an improvement on the old behaviour, where the same tap silently left + // them set with no indication either way. + .confirmationDialog("Clear the R22 flags on your strap?", + isPresented: $confirmingDeepDataDisable, titleVisibility: .visible) { + Button("Clear flags on strap") { model.ble.disableWhoop5DeepData() } + Button("Just stop sending", role: .cancel) { } + } message: { + Text("Turning this switch off only stops NOOP sending the unlock. The flags it already wrote stay on the strap until something clears them. NOOP can write the off value to all 16 now and read each one back so you can see what the strap actually stores. Needs the strap connected and bonded.") + } .confirmationDialog("Mark optical experiment phase", isPresented: $showOpticalPhasePicker, titleVisibility: .visible) { ForEach(PuffinOpticalExperimentPhase.allCases, id: \.self) { phase in @@ -1405,6 +1425,35 @@ struct SettingsView: View { #endif } + /// How many flags the enable sequence actually writes. Read from the sequence rather than restated, so + /// the card cannot drift from it again — it said "15" for the whole life of the 16-flag sequence (#174). + private var r22FlagCount: Int { Whoop5Config.enableR22Sequence.count } + + /// The disable button is gated like the enable one MINUS the wear check: the on-wrist requirement exists + /// because the R22 *stream* is on-wrist gated, and nothing about clearing a stored flag needs the strap + /// worn. Making a user strap it back on to turn a feature off would be the wrong trade. + private var deepDataDisableButtonDisabled: Bool { + #if os(macOS) + return true + #else + return !live.encryptedBond || live.r22DisableReport == BLEManager.deviceConfigProbeWaiting + #endif + } + + /// The reason line under the disable button. Says what the run will do and, crucially, what it can and + /// cannot establish — the off value is inferred from the sibling namespace, so the read-back is the + /// evidence, and a cleared flag is still not the same as observed behaviour reverting. + private var deepDataDisableButtonReason: String { + #if os(macOS) + return String(localized: "Turning deep data back off needs an iPhone or Android. A Mac can't form the encrypted bond a 5/MG requires.") + #else + if !live.encryptedBond { + return String(localized: "Needs the full encrypted bond: close the official WHOOP app and pair the strap to NOOP first (a live-HR-only link can't carry the write).") + } + return String(localized: "Writes the off value to all 16 flags and reads each one back, so you see what the strap stores rather than just that it acked. The off value is inferred from the flag NOOP already turns off this way for Garmin broadcast — it has never been seen on an R22 flag, so NOOP tries one flag first and stops if the strap refuses it. Clearing the flags is not the same as watching the deep records stop: check that by syncing afterwards.") + #endif + } + private var fiveMGCard: some View { SettingsSection( icon: "flask.fill", @@ -1434,7 +1483,11 @@ struct SettingsView: View { } .toggleStyle(.switch) .tint(StrandPalette.accent) - Text("WHOOP 5/MG straps hand a fresh app only live heart rate. The official app switches on the deeper streams (high-rate HR + motion + history) by writing a set of feature flags, a sequence two independent projects have documented. With this on, the button below sends that exact sequence to your strap. Unlike everything else here it does write to the strap, but it's reversible (it only changes which data the strap chooses to emit) and is the same thing the official app does. Experimental: it may do nothing on your firmware. iPhone/Android only. A Mac can't write to a 5/MG.") + // #174: turning the switch OFF used to write nothing — it only hid the enable button, so the + // strap kept every flag the enable sequence set while the UI implied it had been undone. + // Now it offers the real undo. Turning it ON still writes nothing until the button is tapped. + .onChangeCompat(of: deepDataEnabled) { on in if !on { confirmingDeepDataDisable = true } } + Text("WHOOP 5/MG straps hand a fresh app only live heart rate. The official app switches on the deeper streams (high-rate HR + motion + history) by writing a set of feature flags, a sequence two independent projects have documented. With this on, the button below sends that exact sequence to your strap. Unlike everything else here it does write to the strap — and it is reversible: \u{201C}Turn deep data back off\u{201D} writes the off value to the same flags and then reads every one of them back, so you see what the strap actually stores rather than just that it acked. Experimental: it may do nothing on your firmware. iPhone/Android only. A Mac can't write to a 5/MG.") .font(StrandFont.caption) .foregroundStyle(StrandPalette.textTertiary) .fixedSize(horizontal: false, vertical: true) @@ -1448,14 +1501,28 @@ struct SettingsView: View { .font(StrandFont.caption) .foregroundStyle(StrandPalette.textTertiary) + // #174: the undo. Offered whenever the flags may be set — which is any time the opt-in + // has been on, not only right after a send, because the flags persist across launches + // and the app has no record of what a previous install wrote. + NoopButton("Turn deep data back off", systemImage: "bolt.slash", kind: .secondary) { + model.ble.disableWhoop5DeepData() + } + .disabled(deepDataDisableButtonDisabled) + Text(deepDataDisableButtonReason) + .font(StrandFont.caption) + .foregroundStyle(StrandPalette.textTertiary) + // Live R22 telemetry (#174): proof of what the strap is doing right now. + // The threshold and the number are both driven off the sequence itself — they were + // hardcoded to 15 while the sequence carried 16, so the card declared success one flag + // early and named a count that had already drifted. if live.r22FlagsAccepted > 0 { - Label(live.r22FlagsAccepted >= 15 - ? "Strap accepted all 15 R22 flags" - : "Strap accepted \(live.r22FlagsAccepted)/15 R22 flags…", - systemImage: live.r22FlagsAccepted >= 15 ? "checkmark.seal.fill" : "ellipsis") + Label(live.r22FlagsAccepted >= r22FlagCount + ? "Strap accepted all \(r22FlagCount) R22 flags" + : "Strap accepted \(live.r22FlagsAccepted)/\(r22FlagCount) R22 flags…", + systemImage: live.r22FlagsAccepted >= r22FlagCount ? "checkmark.seal.fill" : "ellipsis") .font(StrandFont.caption) - .foregroundStyle(live.r22FlagsAccepted >= 15 ? StrandPalette.statusPositive : StrandPalette.textSecondary) + .foregroundStyle(live.r22FlagsAccepted >= r22FlagCount ? StrandPalette.statusPositive : StrandPalette.textSecondary) } if live.deepPacketsThisSession > 0 { Label(live.deepPacketsThisSession == 1 @@ -1464,11 +1531,31 @@ struct SettingsView: View { systemImage: "clock.arrow.circlepath") .font(StrandFont.caption) .foregroundStyle(StrandPalette.textSecondary) - } else if live.r22FlagsAccepted >= 15 { + } else if live.r22FlagsAccepted >= r22FlagCount { Text("Flags accepted, but the enable sequence doesn't start a separate live stream. The deep records arrive as part of the normal history sync (#494).") .font(StrandFont.caption) .foregroundStyle(StrandPalette.textTertiary) } + + // #174: the disable run's per-key result. Shown verbatim because the interesting part is + // the read-back table, not a green tick — a write that acked SUCCESS but did not move + // the stored value renders here as "unchanged", which is the case worth seeing. + if let report = live.r22DisableReport { + if report == BLEManager.deviceConfigProbeWaiting { + Label("Clearing R22 flags and reading each one back\u{2026}", systemImage: "ellipsis") + .font(StrandFont.caption) + .foregroundStyle(StrandPalette.textSecondary) + } else { + Text(report) + .font(StrandFont.caption.monospaced()) + .foregroundStyle(StrandPalette.textSecondary) + .fixedSize(horizontal: false, vertical: true) + .textSelection(.enabled) + NoopButton("Dismiss disable report", systemImage: "xmark", kind: .secondary) { + model.ble.clearR22DisableReport() + } + } + } } Divider().overlay(StrandPalette.hairline) diff --git a/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt b/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt index 3823ccb39e..48c59cdc36 100644 --- a/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt +++ b/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt @@ -37,6 +37,8 @@ import com.noop.protocol.BackfillCaptureJsonl import com.noop.protocol.BackfillCaptureRecord import com.noop.protocol.BackfillCaptureSummary import com.noop.protocol.CommandNumber +import com.noop.protocol.FeatureFlagWriteGate +import com.noop.protocol.R22DisableReport import com.noop.protocol.DeviceFamily import com.noop.protocol.DeviceConfigReadProbe import com.noop.protocol.DeviceConfigReadProbeReport @@ -1548,6 +1550,21 @@ class WhoopBleClient( /** Monotonic step counter so a late timeout from an earlier step can't cancel a live walk. */ private var deviceConfigStep = 0 + // #174: the R22 DISABLE report — the per-key result of writing '0' to the sixteen feature flags and + // reading every one back with GET_FF_VALUE(128), or the waiting sentinel while the run walks. Unlike + // the two probes above this one DOES write, which is exactly why it reports the value the strap stores + // rather than the write's own ack. + private val _r22DisableReport = MutableStateFlow(null) + val r22DisableReport: StateFlow = _r22DisableReport.asStateFlow() + + /** The in-flight #174 disable run; null when none is running. Doubles as the [send] allowlist's + * in-flight gate for the 128 read-back. */ + private var r22DisableRun: R22DisableReport? = null + /** The step whose reply we are waiting for, null between steps. */ + private var r22DisableAwaiting: R22DisableReport.Step? = null + /** Monotonic step counter so a late timeout from an earlier step can't cancel a live run. */ + private var r22DisableStep = 0 + private val _connectedPeripheralAddress = MutableStateFlow(null) /** The BLE address of the strap currently connected, or null when disconnected. Twin of macOS * BLEManager.connectedPeripheralUUID — drives SourceCoordinator's first-connect identity adoption. */ @@ -2859,10 +2876,20 @@ class WhoopBleClient( // keep their own separate opt-in clauses below and are never sent from this path. Driven // only by probeDeviceConfigValues() (user-initiated, Test Centre gated). !(DeviceConfigReadProbe.isReadOnlyOpcode(cmd.rawValue) && deviceConfigReport != null) && - // SET_CONFIG (the R22 deep-stream unlock) is allowed ONLY while the deep-data experiment - // is opted in — it writes a persistent feature flag to the strap, so it must never fire - // on a default install. Reversible; driven only by enableWhoop5DeepData(). (#174) - !(cmd == CommandNumber.SET_CONFIG && puffinExperiment.isDeepDataEnabled) && + // SET_CONFIG / SET_FF_VALUE (120) — the R22 deep-stream unlock AND its undo. Allowed only + // while the deep-data experiment is opted in, and now additionally only for a KEY and a + // VALUE the gate recognises: one of the sixteen R22 flags carrying either that key's own + // enable value or the documented off value. The clause this replaces was opcode-only, so + // it admitted ANY feature-flag key with ANY value for as long as the opt-in happened to be + // on — the same weakness #907 closed on opcode 119. Driven only by enableWhoop5DeepData() + // and disableWhoop5DeepData(). (#174) + !FeatureFlagWriteGate.admitsSend( + cmd.rawValue, payload, puffinExperiment.isDeepDataEnabled, + ) && + // GET_FF_VALUE (128) as the disable run's mandatory READ-BACK. Gated on a disable run being + // in flight, exactly like the read probes' 121/128 clause above, so a default install can + // never form these bytes. The write ack is not trusted; this is what proves the clear. + !(FeatureFlagWriteGate.isReadBackOpcode(cmd.rawValue) && r22DisableRun != null) && // SET_DEVICE_CONFIG (the Broadcast-HR flag) is allowed ONLY while that opt-in is on. // Reversible; driven only by setBroadcastHr(). (#181) !(cmd == CommandNumber.SET_DEVICE_CONFIG && puffinExperiment.broadcastHr)) { @@ -4811,6 +4838,16 @@ class WhoopBleClient( ) { handleDeviceConfigProbeResponse(frame) } + // #174: a reply belonging to an R22 DISABLE run — either the SET_FF_VALUE(120) write + // ack or the GET_FF_VALUE(128) read-back that verifies it. In-flight-guarded inside. + // 128 is also matched by the read-probe clause above; both handlers guard on their OWN + // run being live, so exactly one acts. + if (frame.size > cmdOff && (frame[cmdOff - 2].toInt() and 0xFF) == 0x24 && + ((frame[cmdOff].toInt() and 0xFF) == CommandNumber.SET_CONFIG.rawValue || + (frame[cmdOff].toInt() and 0xFF) == CommandNumber.GET_FF_VALUE.rawValue) + ) { + handleR22DisableResponse(frame) + } if (frame.size > cmdOff && (frame[cmdOff].toInt() and 0xFF) == CommandNumber.GET_DATA_RANGE.rawValue) { // #451: dump raw GET_DATA_RANGE response bytes unconditionally (even if decode returns // null) so a stale/wrong-epoch "newest" can be told apart from a frame-alignment bug in @@ -4951,7 +4988,12 @@ class WhoopBleClient( if (connectedFamily != DeviceFamily.WHOOP5) return if (frame.size <= 10) return val type = frame[8].toInt() and 0xFF - if (type == 0x24 && (frame[10].toInt() and 0xFF) == CommandNumber.SET_CONFIG.rawValue) { + // #174: a SET_CONFIG ack means "one enable_r22_* flag accepted" ONLY when we are enabling. A + // disable run writes through the same opcode, so without this guard turning R22 OFF would tick the + // "Strap accepted N/16 R22 flags" counter upwards — the exact opposite of what happened. The + // disable run scores its own acks (and does not trust them; the 128 read-back is its proof). + if (type == 0x24 && (frame[10].toInt() and 0xFF) == CommandNumber.SET_CONFIG.rawValue && + r22DisableRun == null) { val n = _state.value.r22FlagsAccepted + 1 _state.update { it.copy(r22FlagsAccepted = n) } val total = Whoop5Config.enableR22Sequence.size @@ -5654,10 +5696,145 @@ class WhoopBleClient( }, 80L * i) } handler.postDelayed({ - log("Deep-data: sequence sent. Keep the strap on, let it sync, then share your strap log — we're looking for new deep records (type-0x2F) to start arriving. (#174)") + log("Deep-data: sequence sent. Keep the strap on, let it sync, then share your strap log — we're looking for new deep records (type-0x2F) to start arriving. To undo it later, use \"Turn deep data back off\" — disableWhoop5DeepData() writes '0' to the same sixteen keys and reads each one back. (#174)") }, 80L * flags.size + 200L) } + /** + * EXPERIMENTAL (#174): the undo of [enableWhoop5DeepData] — write '0' to the sixteen `enable_r22_*` + * feature flags and report, per key, the value the strap actually stores afterwards. Port of + * `BLEManager.disableWhoop5DeepData`. + * + * Why this exists: the Settings copy has promised since #174 that the R22 unlock is "reversible", and + * that was true about the hardware and false about the app — NOOP shipped an enable button and no way + * back, so the only routes out were the official WHOOP app or a factory reset. + * + * Why it is staged rather than sixteen writes: '0' is the confirmed off value in the device-config + * namespace (#181, hardware-validated) and that namespace shares this one's entire wire idiom — but no + * FEATURE flag has ever been observed holding '0', and the two namespaces are proven separate at the + * verb level. So the run writes ONE flag, reads it back, and only touches the other fifteen if the + * strap actually stopped reporting the old value. + * + * The ack is never the proof: SELECT_WRIST returns SUCCESS for a no-op and FAILURE for a real mutation + * on this firmware, so only the value a GET_FF_VALUE(128) read returns is reported as state (#907/#891). + * + * Same guards as the enable minus wear: the on-wrist gate exists because the R22 STREAM is on-wrist + * only, and turning a feature off should not require strapping the device back on. + */ + fun disableWhoop5DeepData() { + if (connectedFamily != DeviceFamily.WHOOP5) { + log("Deep-data disable: needs a WHOOP 5.0/MG strap — ignored."); return + } + if (!puffinExperiment.isDeepDataEnabled) { + // The send allowlist keys off this same pref, so the writes would be dropped anyway. Fail with + // a message that says what to do rather than letting sixteen frames vanish silently. + log("Deep-data disable: the deep-data opt-in is off, so the writes would be refused by the send allowlist. Turn the toggle back on, tap the disable button, then turn it off — ignored."); return + } + val s = _state.value + if (!s.connected || !s.encryptedBond) { + log("Deep-data disable: needs the full encrypted bond, not the live-HR-only link. Close the official WHOOP app, put the strap in pairing mode, and bond it to NOOP first — ignored."); return + } + if (r22DisableRun != null) { + log("Deep-data disable: a disable run is already walking its plan — ignored."); return + } + r22DisableRun = R22DisableReport() + _r22DisableReport.value = WAITING_DEVICE_CONFIG_PROBE + log( + "Deep-data disable (#174): clearing the ${Whoop5Config.disableR22Sequence.size}-flag R22 " + + "sequence — writing '0' via SET_FF_VALUE(120), each verified with GET_FF_VALUE(128). " + + "Probing ${R22DisableReport.PROBE_KEY} first; the other flags are only written if that one moves." + ) + advanceR22Disable() + } + + /** Send the next planned step, or finish when the plan is done. */ + private fun advanceR22Disable() { + val step = r22DisableRun?.nextStep() + if (step == null) { finishR22Disable(); return } + val cmd = CommandNumber.entries.firstOrNull { it.rawValue == step.opcode } + if (cmd == null) { + log("Deep-data disable: refusing to send unknown opcode ${step.opcode}") + finishR22Disable(); return + } + val payload: ByteArray + if (step.isWrite) { + payload = byteArrayOf(0x01) + + Whoop5Config.payloadBody(step.key, Whoop5Config.FEATURE_FLAG_OFF_VALUE) + // Fail closed: the same predicate the send path consults, asked here too, so a future edit that + // widened the plan cannot put an unrecognised key or value on the wire. + if (!FeatureFlagWriteGate.admitsSend(step.opcode, payload, puffinExperiment.isDeepDataEnabled)) { + log("Deep-data disable: refusing to write ${step.key} — not admitted by the feature-flag gate") + finishR22Disable(); return + } + } else { + if (!FeatureFlagWriteGate.isReadBackOpcode(step.opcode)) { + log("Deep-data disable: refusing to read with opcode ${step.opcode}") + finishR22Disable(); return + } + payload = DeviceConfigReadProbe.requestBody(step.key) + } + r22DisableStep++ + r22DisableAwaiting = step + val armed = r22DisableStep + send(cmd, payload, withResponse = true) + handler.postDelayed({ + if (r22DisableRun != null && r22DisableStep == armed && r22DisableAwaiting != null) { + r22DisableAwaiting = null + r22DisableRun?.noteTimeout(step, (DEVICE_CONFIG_PROBE_TIMEOUT_MS / 1000).toInt()) + advanceR22Disable() + } + }, DEVICE_CONFIG_PROBE_TIMEOUT_MS) + } + + /** Render + publish + log the report and end the run (which also re-closes the 128 send allowlist). */ + private fun finishR22Disable() { + val report = r22DisableRun ?: return + r22DisableRun = null + r22DisableAwaiting = null + // The "Strap accepted N/16 R22 flags" line reports the last ENABLE send. If this run cleared even + // one key that claim is stale, so retire it rather than leaving the card asserting the strap is + // unlocked while the report below says it was just cleared. + if (report.outcomes.values.any { it.isSuccess }) { + _state.update { it.copy(r22FlagsAccepted = 0) } + } + val text = report.render() + log("Deep-data disable (#174):\n$text") + _r22DisableReport.value = text + } + + /** Clear the #174 disable result (Settings card dismissed). */ + fun clearR22DisableReport() { _r22DisableReport.value = null } + + /** + * #174: one COMMAND_RESPONSE belonging to a disable run — either a SET_FF_VALUE(120) write ack or a + * GET_FF_VALUE(128) read-back. Guarded on a run being IN-FLIGHT so a stray byte match can never + * surface a result. Parsing — including the CRC gate — lives in the pure [DeviceConfigReadProbe]. + */ + private fun handleR22DisableResponse(frame: ByteArray) { + if (r22DisableRun == null) return + val step = r22DisableAwaiting ?: return + r22DisableAwaiting = null + if (step.isWrite) { + // The ack is recorded for the transcript and decides nothing. The puffin envelope puts the type + // at 8 and the cmd at 10, so the payload starts at 11 and the result byte is at 12. + val resultCode = if (frame.size > 12) frame[12].toInt() and 0xFF else null + r22DisableRun?.noteWriteAck(resultCode, step) + } else { + val parsed = DeviceConfigReadProbe.parse(frame, connectedFamily, step.opcode) + val value = parsed.value + if (value != null) r22DisableRun?.noteReadBack(value, step) + else r22DisableRun?.noteReadFailure(parsed.failure!!, step) + } + advanceR22Disable() + } + + /** Abandon a disable run the link interrupted, re-closing the 128 send allowlist. */ + private fun abandonR22DisableRun(why: String) { + if (r22DisableRun == null) return + r22DisableRun?.noteAbandoned(why) + finishR22Disable() + } + /** * SET_CLOCK(10) payload = the strap's 8-byte form: [seconds u32 LE][subseconds u32 LE]. * Port of `BLEManager.setClockPayload`. The payload LENGTH is firmware-specific: newer WHOOP 4 @@ -6823,6 +7000,10 @@ class WhoopBleClient( // #103: same for the device-config READ probe — dropping the report re-closes the 121/128 // send() allowlist. _deviceConfigProbe.value = null + // #174: a disable run interrupted mid-plan is RENDERED rather than dropped — unlike the read-only + // probes it has already written to the strap, so the user must be told how far it got and which + // keys are still set. This publishes the partial report and re-closes the 128 allowlist. + abandonR22DisableRun("the strap disconnected mid-run") deviceConfigReport = null deviceConfigAwaiting = null reset() diff --git a/android/app/src/main/java/com/noop/protocol/R22Disable.kt b/android/app/src/main/java/com/noop/protocol/R22Disable.kt new file mode 100644 index 0000000000..75757afd9e --- /dev/null +++ b/android/app/src/main/java/com/noop/protocol/R22Disable.kt @@ -0,0 +1,393 @@ +package com.noop.protocol + +/** + * #174: the key-aware allowlist for SET_FF_VALUE (120 / 0x78) — the feature-flag WRITE verb the R22 + * enable and disable sequences share. Direct twin of the Swift `FeatureFlagWriteGate` + * (Packages/WhoopProtocol/Sources/WhoopProtocol/R22Disable.swift) — keep them byte-identical. + * + * Before this file the 5/MG send path admitted opcode 120 on an opcode-only clause (the deep-data opt-in + * alone), which admits ANY feature-flag key with ANY value for as long as that opt-in happens to be on. + * The sixteen R22 keys are the only ones NOOP has business writing, and the disable path doubles the + * number of values travelling through that opcode, so the clause is tightened rather than widened. Same + * discipline as `DeviceConfigReadProbe.isReadOnlyOpcode`: one pure predicate the send path itself + * consults, so a unit test proving the predicate rejects something proves it about the real wire path. + * + * Pure: no Bluetooth, no I/O, no preferences. The app layer supplies the opt-in boolean. + */ +object FeatureFlagWriteGate { + + /** SET_FF_VALUE (120 / 0x78) — the feature-flag write verb. The only opcode this gate admits. */ + const val SET_FEATURE_FLAG_VALUE_CMD = 120 + + /** SET_DEVICE_CONFIG_VALUE (119 / 0x77) — the OTHER namespace's write verb. Named here for exactly + * one reason: so [admitsSend] can be proved to refuse it. */ + const val SET_DEVICE_CONFIG_VALUE_CMD = 119 + + /** GET_FF_VALUE (128 / 0x80) — the read verb the mandatory post-write read-back uses. */ + const val GET_FEATURE_FLAG_VALUE_CMD = 128 + + /** Width of the key-name field in a feature-flag body (`Whoop5Config.payloadBody` NUL-pads to this). */ + const val NAME_FIELD_BYTES = 32 + + /** Total length of the feature-flag body: 32-byte name, value byte, 7 zero bytes. */ + const val BODY_BYTES = 40 + + /** The sixteen keys, taken from the enable sequence rather than restated — one list, one truth. */ + val r22Keys: List get() = Whoop5Config.enableR22Sequence.map { it.name } + + /** The value the enable sequence writes for [key], or null when [key] is not an R22 flag. */ + fun enableValue(key: String): Int? = + Whoop5Config.enableR22Sequence.firstOrNull { it.name == key }?.value + + /** Whether [value] is one NOOP may write for [key]: that key's own enable value, or the off value. */ + fun isWritableValue(value: Int, key: String): Boolean { + val on = enableValue(key) ?: return false + return value == on || value == Whoop5Config.FEATURE_FLAG_OFF_VALUE + } + + /** One parsed SET_FF_VALUE body. */ + data class KeyValue(val key: String, val value: Int) + + /** + * The key name and value carried by a SET_FF_VALUE payload, or null when the payload is not shaped + * like one. The payload the send path holds is `[0x01] + payloadBody(...)`. A name is only returned + * when it is printable ASCII and the remainder of the field is genuine NUL padding, so a short, + * mis-shaped or binary-carrying body yields null and is refused rather than guessed at. + */ + fun keyAndValue(sendPayload: ByteArray): KeyValue? { + if (sendPayload.size < 1 + BODY_BYTES) return null + if (sendPayload[0].toInt() != 0x01) return null + val name = StringBuilder() + for (i in 0 until NAME_FIELD_BYTES) { + val b = sendPayload[1 + i].toInt() and 0xFF + if (b == 0) { + // Everything after the name must be NUL, or this is not a NUL-padded name field. + for (j in i until NAME_FIELD_BYTES) { + if ((sendPayload[1 + j].toInt() and 0xFF) != 0) return null + } + break + } + if (b < 0x20 || b > 0x7E) return null + name.append(b.toChar()) + } + if (name.isEmpty()) return null + return KeyValue(name.toString(), sendPayload[1 + NAME_FIELD_BYTES].toInt() and 0xFF) + } + + /** + * **The send allowlist itself.** True only for SET_FF_VALUE(120) carrying a well-formed body whose + * key is one of the sixteen and whose value is that key's enable value or the off value. Every other + * opcode is false — explicitly including SET_DEVICE_CONFIG_VALUE(119), which keeps its own clause. + */ + fun admitsSend(opcode: Int, payload: ByteArray, deepDataOptIn: Boolean): Boolean { + if (!deepDataOptIn) return false + if (opcode != SET_FEATURE_FLAG_VALUE_CMD) return false + val kv = keyAndValue(payload) ?: return false + return isWritableValue(kv.value, kv.key) + } + + /** The read verb the post-write verification is allowed to send, and only that one. */ + fun isReadBackOpcode(opcode: Int): Boolean = opcode == GET_FEATURE_FLAG_VALUE_CMD +} + +/** + * #174: one R22 DISABLE run — write '0' to the sixteen feature flags, then read every one of them back + * and report the value the strap actually stores. Twin of the Swift `R22DisableReport`; the rendered text + * is byte-identical across platforms so a shared strap log reads the same either side. + * + * The run is staged because the off value is inferred (see [Whoop5Config.FEATURE_FLAG_OFF_VALUE]): + * 1. Probe — write '0' to ONE flag, read it back with GET_FF_VALUE(128). + * 2. Gate — if the strap did not stop reporting the old value, STOP. Fifteen keys stay untouched and the + * report says '0' is not how this namespace spells off, which is a real, publishable answer. + * 3. Clear — only on a good probe, write '0' to the remaining fifteen. + * 4. Verify — read all sixteen back and tabulate. + * + * Every write's own COMMAND_RESPONSE is recorded and NOT believed: SELECT_WRIST returns SUCCESS for a + * no-op and FAILURE for a real mutation on this firmware, so a result byte says nothing reliable about + * whether state moved. Only the value a 128 read returns is reported as state (#907/#891). + * + * Pure and order-dependent (nextStep -> note... -> nextStep), so the whole verdict table is unit-testable + * without a strap. + */ +class R22DisableReport( + val keys: List = Whoop5Config.enableR22Sequence.map { it.name }, +) { + + companion object { + /** + * The flag the probe stage writes first. + * + * `enable_sig12` is chosen because it is the ONLY key with a hardware demonstration that a write + * to it changes stored state: the enable sequence moved it from '2' (0x32) to '1' (0x31), + * confirmed by a GET_FF_VALUE(128) read before and after (#423, #103). If a '0' write to THIS key + * does not move the stored value, the failure is attributable to the VALUE rather than to the key + * or the verb — every other key would leave that ambiguity open. It is also the flag with the + * least to lose: its effect is undocumented and it gates no stream NOOP reads. + */ + const val PROBE_KEY = "enable_sig12" + + /** The standing caveats, kept in one place so the UI, the log and the report cannot drift. */ + val CAVEATS = """ + What this does and does not establish: + • '0' as the off value is INFERRED. It is the confirmed off value in the device-config + namespace (SET_DEVICE_CONFIG_VALUE/119, hardware-validated on the Broadcast-HR flag, #181) + and that namespace shares this one's whole wire idiom — but no feature flag had ever been + observed holding '0' before this run. The read-back above is what turns that inference + into a measurement. + • A cleared flag is not the same as reverted BEHAVIOUR. This reports what the strap STORES. + Whether the deep records actually stop is a separate question, answered by wearing the + strap and watching the type-0x2F deep-buffer capture stop, not by this report. + • This does not restore a snapshot. NOOP never read these values before first writing them, + so the strap's pre-NOOP state is unknown. The honest claim is "cleared the flags NOOP set". + • disable_pip_r26_packets inverts: clearing it is expected to let PIP R26 packets flow AGAIN. + That is the pre-R22 behaviour and is intended. + """.trimIndent() + } + + /** Which stage of the run a step belongs to. */ + enum class Stage { PROBE_WRITE, PROBE_READ, CLEAR_WRITE, VERIFY_READ } + + /** One planned round-trip. */ + data class Step(val opcode: Int, val key: String, val stage: Stage) { + val isWrite: Boolean get() = stage == Stage.PROBE_WRITE || stage == Stage.CLEAR_WRITE + } + + /** What one key ended up in. CLEARED and UNSET are both successes. */ + enum class KeyOutcome(val label: String) { + /** Read back as '0': the key exists and now holds the off value. */ + CLEARED("cleared"), + /** The strap answered FAILURE for the key: it no longer has a stored value. The stronger result. */ + UNSET("unset"), + /** Read back as something other than '0' — the write did not take. */ + UNCHANGED("unchanged"), + /** The strap answered but did not echo the key, so no value can be claimed. */ + NOT_CLAIMED("notClaimed"), + /** No reply inside the step's window. */ + SILENT("silent"), + /** A reply arrived that could not be decoded. */ + UNDECODABLE("undecodable"), + /** The run stopped before this key was written. */ + SKIPPED("skipped"); + + val isSuccess: Boolean get() = this == CLEARED || this == UNSET + } + + /** The overall verdict. */ + enum class Verdict(val label: String) { + PENDING("pending"), + ALL_CLEARED("allCleared"), + PARTIAL("partial"), + /** The probe read-back showed the OLD value: '0' is not how this firmware clears a flag. */ + OFF_VALUE_REJECTED("offValueRejected"), + /** The probe read-back produced no usable answer. Distinct from a rejection on purpose: nothing + * was learned about the off value, so it must not be reported as evidence against it. */ + PROBE_INCONCLUSIVE("probeInconclusive"), + ABANDONED("abandoned"), + } + + /** One recorded read-back. */ + data class Reading( + val key: String, + val stage: Stage, + val value: Int?, + val resultCode: Int?, + val recordHex: String, + val outcome: KeyOutcome, + ) + + private var cursor = 0 + private var plan: MutableList = mutableListOf() + private var planBuilt = false + private var stopped = false + + val outcomes: MutableMap = linkedMapOf() + val readings: MutableList = mutableListOf() + val writeAcks: MutableMap = linkedMapOf() + val trace: MutableList = mutableListOf() + var verdict: Verdict = Verdict.PENDING + private set + var probePassed = false + private set + + init { + trace.add( + "R22 disable: writing '0' (0x30) to ${keys.size} feature flags via SET_FF_VALUE(120), " + + "each verified with GET_FF_VALUE(128). The write acks are recorded and NOT trusted." + ) + trace.add( + "Probe first: $PROBE_KEY is written and read back alone; the other " + + "${maxOf(0, keys.size - 1)} are only written if the strap stops reporting the old value." + ) + } + + /** + * The next round-trip, or null when the run is over. Built lazily in two halves so the gate is + * expressible: the probe pair up front, and the clear+verify tail only once the probe has passed. + */ + fun nextStep(): Step? { + if (plan.isEmpty() && !planBuilt) { + planBuilt = true + if (!keys.contains(PROBE_KEY)) { + // Defensive: a caller that hands over a key list without the probe key gets a plain + // sequential run rather than a silently skipped gate. + keys.forEach { plan.add(Step(FeatureFlagWriteGate.SET_FEATURE_FLAG_VALUE_CMD, it, Stage.CLEAR_WRITE)) } + keys.forEach { plan.add(Step(FeatureFlagWriteGate.GET_FEATURE_FLAG_VALUE_CMD, it, Stage.VERIFY_READ)) } + probePassed = true + return nextStep() + } + plan.add(Step(FeatureFlagWriteGate.SET_FEATURE_FLAG_VALUE_CMD, PROBE_KEY, Stage.PROBE_WRITE)) + plan.add(Step(FeatureFlagWriteGate.GET_FEATURE_FLAG_VALUE_CMD, PROBE_KEY, Stage.PROBE_READ)) + } + if (stopped) return null + if (cursor >= plan.size) { + if (probePassed && plan.none { it.stage == Stage.CLEAR_WRITE }) { + keys.filter { it != PROBE_KEY }.forEach { + plan.add(Step(FeatureFlagWriteGate.SET_FEATURE_FLAG_VALUE_CMD, it, Stage.CLEAR_WRITE)) + } + keys.forEach { plan.add(Step(FeatureFlagWriteGate.GET_FEATURE_FLAG_VALUE_CMD, it, Stage.VERIFY_READ)) } + } else { + return null + } + } + if (cursor >= plan.size) return null + return plan[cursor++] + } + + /** Record a write's own ack. Logged, never used to decide anything. */ + fun noteWriteAck(resultCode: Int?, step: Step) { + if (resultCode != null) writeAcks[step.key] = resultCode + val label = if (resultCode != null) "${FeatureFlagProbe.resultLabel(resultCode)}($resultCode)" else "(unlabelled)" + trace.add("SET_FF_VALUE(120) ${step.key}='0' → ack $label — recorded, not proof") + } + + /** Record a decoded read-back and score the key. */ + fun noteReadBack(r: DeviceConfigReadProbe.ValueResponse, step: Step) { + val value = r.valueFor(step.key) + val outcome = when { + r.isUnsupported -> KeyOutcome.UNCHANGED // the verb is refused; nothing can be claimed cleared + r.isFailure -> KeyOutcome.UNSET // no stored value for this key any more + value != null -> + if (value == Whoop5Config.FEATURE_FLAG_OFF_VALUE) KeyOutcome.CLEARED else KeyOutcome.UNCHANGED + else -> KeyOutcome.NOT_CLAIMED + } + record(Reading(step.key, step.stage, value, r.resultCode, r.recordHex, outcome), step) + } + + /** Record a read-back reply that could not be decoded. */ + fun noteReadFailure(f: DeviceConfigReadProbe.ParseFailure, step: Step) { + val why = when (f) { + DeviceConfigReadProbe.ParseFailure.CRC -> "CRC failed — frame rejected (never decoded)" + DeviceConfigReadProbe.ParseFailure.ENVELOPE -> "not a COMMAND_RESPONSE envelope" + DeviceConfigReadProbe.ParseFailure.WRONG_COMMAND -> "COMMAND_RESPONSE for a different command" + DeviceConfigReadProbe.ParseFailure.TRUNCATED -> "record too short to hold a response" + } + trace.add("GET_FF_VALUE(128) ${step.key} → reply not decoded: $why") + record(Reading(step.key, step.stage, null, null, "(undecodable)", KeyOutcome.UNDECODABLE), step) + } + + /** Record the strap answering nothing at all. */ + fun noteTimeout(step: Step, seconds: Int) { + val verb = if (step.isWrite) "SET_FF_VALUE(120)" else "GET_FF_VALUE(128)" + trace.add("$verb ${step.key} → no COMMAND_RESPONSE within ${seconds}s") + if (step.isWrite) return // a silent WRITE is fine; the read-back is what decides + record(Reading(step.key, step.stage, null, null, "(no reply)", KeyOutcome.SILENT), step) + } + + /** The link dropped, or the caller stopped the run. */ + fun noteAbandoned(why: String) { + trace.add("Run abandoned: $why") + stopped = true + keys.forEach { if (outcomes[it] == null) outcomes[it] = KeyOutcome.SKIPPED } + verdict = Verdict.ABANDONED + } + + private fun record(reading: Reading, step: Step) { + readings.add(reading) + var line = "GET_FF_VALUE(128) ${step.key}" + line += if (reading.resultCode != null) { + " → result=${FeatureFlagProbe.resultLabel(reading.resultCode)}(${reading.resultCode})" + } else " →" + if (reading.value != null) line += " value=${DeviceConfigReadProbe.valueLabel(reading.value)}" + line += " record=[${reading.recordHex}] ⇒ ${reading.outcome.label}" + trace.add(line) + + outcomes[step.key] = reading.outcome + + if (step.stage == Stage.PROBE_READ) { + probePassed = reading.outcome.isSuccess + if (!probePassed) { + stopped = true + keys.forEach { if (it != step.key) outcomes[it] = KeyOutcome.SKIPPED } + // ONLY an unmoved value is evidence against '0'. Silence, an undecodable frame and a reply + // that never echoed the key all mean "no usable answer". + verdict = if (reading.outcome == KeyOutcome.UNCHANGED) { + Verdict.OFF_VALUE_REJECTED + } else Verdict.PROBE_INCONCLUSIVE + trace.add(probeStopExplanation(reading.outcome)) + } else { + trace.add( + "Probe passed (${reading.outcome.label}) — '0' moves a feature flag on this firmware. " + + "Clearing the remaining ${maxOf(0, keys.size - 1)}." + ) + } + } + + if (!stopped && outcomes.size == keys.size && readings.any { it.stage == Stage.VERIFY_READ }) { + val cleared = keys.count { outcomes[it]?.isSuccess == true } + verdict = if (cleared == keys.size) Verdict.ALL_CLEARED else Verdict.PARTIAL + } + } + + private fun probeStopExplanation(outcome: KeyOutcome): String = when (outcome) { + KeyOutcome.UNCHANGED -> + "STOPPED: the strap still reports the old value for $PROBE_KEY, so '0' (0x30) is NOT how this " + + "firmware clears a feature flag. The other ${maxOf(0, keys.size - 1)} flags were left " + + "untouched. This is a real finding — please share this report on #174." + KeyOutcome.NOT_CLAIMED -> + "STOPPED: the strap answered but did not echo $PROBE_KEY, so no value can be claimed either " + + "way. Nothing further was written." + KeyOutcome.SILENT, KeyOutcome.UNDECODABLE -> + "STOPPED: no usable read-back for $PROBE_KEY, so whether the write landed is unknown. Nothing " + + "further was written." + else -> "STOPPED before clearing the remaining flags." + } + + /** One-line summary, suitable for a Settings row. */ + val summary: String + get() { + val cleared = keys.count { outcomes[it]?.isSuccess == true } + return when (verdict) { + Verdict.PENDING -> "Clearing R22 flags…" + Verdict.ALL_CLEARED -> "All ${keys.size} R22 flags cleared on the strap (read back, not just acked)." + Verdict.PARTIAL -> "$cleared of ${keys.size} R22 flags cleared — the rest did not take." + Verdict.OFF_VALUE_REJECTED -> "The strap refused '0' for $PROBE_KEY; the other flags were left alone." + Verdict.PROBE_INCONCLUSIVE -> "No usable read-back for $PROBE_KEY — nothing further was written." + Verdict.ABANDONED -> "Disable run interrupted — $cleared of ${keys.size} flags confirmed cleared." + } + } + + /** The full copyable report. */ + fun render(): String { + val sb = StringBuilder() + sb.append("#174 R22 DEEP-DATA DISABLE — WHOOP 5/MG\n") + sb.append("Wrote '0' (0x30) via SET_FF_VALUE(120) and read every key back with GET_FF_VALUE(128).\n") + sb.append("The write acks are recorded and NOT treated as proof: on this firmware a SUCCESS result byte\n") + sb.append("does not establish that state changed. Only the read-back is reported as state.\n") + sb.append("\nVerdict: ${verdict.label} — $summary\n") + sb.append("\nPer key:\n") + for (key in keys) { + val outcome = outcomes[key] ?: KeyOutcome.SKIPPED + val reading = readings.lastOrNull { it.key == key && it.stage == Stage.VERIFY_READ } + ?: readings.lastOrNull { it.key == key } + val was = Whoop5Config.enableR22Sequence.firstOrNull { it.name == key } + ?.let { DeviceConfigReadProbe.valueLabel(it.value) } ?: "?" + val now = reading?.value?.let { DeviceConfigReadProbe.valueLabel(it) } + ?: if (outcome == KeyOutcome.UNSET) "(no stored value)" else "(not claimed)" + sb.append(" ${DeviceConfigReadProbe.padded(key, 30)} enable wrote $was → now $now [${outcome.label}]\n") + } + sb.append("\nExchange:\n") + for (line in trace) sb.append(" ").append(line).append("\n") + sb.append("\n").append(CAVEATS) + return sb.toString() + } +} diff --git a/android/app/src/main/java/com/noop/protocol/Whoop5Config.kt b/android/app/src/main/java/com/noop/protocol/Whoop5Config.kt index b3960e4d67..955546266a 100644 --- a/android/app/src/main/java/com/noop/protocol/Whoop5Config.kt +++ b/android/app/src/main/java/com/noop/protocol/Whoop5Config.kt @@ -88,4 +88,61 @@ object Whoop5Config { seq = seq, payload = byteArrayOf(0x01) + payloadBody(flag.name, flag.value), ) + + // Turning it back off (#174) ------------------------------------------------------------------ + + /** + * The value byte written to clear a feature flag: ASCII '0' (0x30). + * + * **Confirmed for the sibling opcode, inferred here by shared convention.** + * + * Confirmed on real hardware: 0x30 is how this firmware spells "off" for a config key. NOOP has + * shipped that write since #181 — Broadcast-HR off writes 0x30 to `whoop_live_hr_in_adv_ind_pkt` + * through SET_DEVICE_CONFIG_VALUE (119 / 0x77) and a Garmin Edge 840 stops seeing the broadcast. + * `enable_raw_data_w_ecg` independently READS '0' on an MG whose ECG is not running. Both are the + * device-config namespace. + * + * Shared convention, which is why it transfers: both bodies are the key name as ASCII NUL-padded to + * 32 bytes then a single ASCII-digit value byte at offset 32, both are sent as `[0x01] + body` with + * response. Only the opcode (0x77 vs 0x78) and the body length (33 vs 40) differ. Writes through + * 0x78 demonstrably land and change stored state: the enable sequence moved `enable_sig12` from '2' + * to '1', confirmed by a GET_FF_VALUE(128) read before and after. + * + * Inferred, stated plainly: '0' has NEVER been observed in the feature-flag namespace — every 128 + * read ever taken returned '1' or '2', and those are a round-trip of NOOP's own writes. The two + * namespaces are proven separate at the verb level (a key asked through the wrong verb answers + * FAILURE). And `disable_pip_r26_packets` is written '2' despite being named `disable_*`, while + * `enable_sig12` was corrected '2'→'1' from a real capture — so the official app picks BETWEEN '1' + * and '2' per flag rather than using one canonical "true", and tri-state semantics stay unestablished. + * + * So [R22DisableReport] tests the byte instead of asserting it: write it to ONE flag, read it back, + * and only touch the rest if the strap stopped reporting the old value. Keep in lockstep with the + * Swift `Whoop5Config.featureFlagOffValue`. + */ + const val FEATURE_FLAG_OFF_VALUE = 0x30 + + /** + * The inverse of [enableR22Sequence]: the same sixteen keys, in the same order, each carrying + * [FEATURE_FLAG_OFF_VALUE]. + * + * Same order as the enable sequence deliberately — `enable_r22_packets`, the master flag, is cleared + * FIRST, so a run interrupted by a disconnect leaves the strap nearer "off" than it started rather + * than stranded with sub-streams cleared and the master still set. + * + * On `disable_pip_r26_packets`, whose name inverts: this sequence is the UNDO of the enable sequence, + * not a per-key semantic inversion. Inverting per key would require knowing what '1' and '2' each + * select, and that key is the evidence we do not. Writing '0' to it is expected to stop PIP R26 + * packets being suppressed — i.e. let them flow again, which is the pre-R22 behaviour. + * + * This does NOT restore a snapshot: NOOP never read these values before first writing them, so the + * pre-NOOP state is unknown. The honest claim is "clears the sixteen flags NOOP set". + * Keep in lockstep with the Swift `Whoop5Config.disableR22Sequence`. + */ + val disableR22Sequence: List = enableR22Sequence.map { Flag(it.name, FEATURE_FLAG_OFF_VALUE) } + + /** Every frame in the disable sequence, sequence-numbered from [firstSeq]. Written exactly like the + * enable frames — same opcode, same 40-byte body, same spacing, with response — because they differ + * only in the value byte. */ + fun disableSequenceFrames(firstSeq: Int = 1): List = + disableR22Sequence.mapIndexed { idx, flag -> frame(flag, (firstSeq + idx) and 0xFF) } } diff --git a/android/app/src/main/java/com/noop/ui/SettingsScreen.kt b/android/app/src/main/java/com/noop/ui/SettingsScreen.kt index 056d81a8cb..33eadc1e65 100644 --- a/android/app/src/main/java/com/noop/ui/SettingsScreen.kt +++ b/android/app/src/main/java/com/noop/ui/SettingsScreen.kt @@ -31,6 +31,7 @@ import androidx.compose.foundation.layout.padding import androidx.compose.foundation.layout.size import androidx.compose.foundation.layout.widthIn import androidx.compose.foundation.shape.RoundedCornerShape +import androidx.compose.ui.text.font.FontFamily import androidx.compose.material.icons.Icons import androidx.compose.material.icons.automirrored.filled.KeyboardArrowRight import androidx.compose.material.icons.filled.Autorenew @@ -74,6 +75,7 @@ import androidx.compose.material3.SwitchDefaults import androidx.compose.material3.Text import androidx.compose.material3.TextButton import androidx.compose.runtime.Composable +import androidx.compose.runtime.collectAsState import androidx.compose.runtime.DisposableEffect import androidx.compose.runtime.getValue import androidx.compose.runtime.mutableStateOf @@ -105,6 +107,10 @@ import com.noop.analytics.Baselines import com.noop.analytics.Zones import com.noop.R import com.noop.ble.PuffinExperiment +import com.noop.ble.WhoopBleClient +// #174: the R22 card reads the flag COUNT off Whoop5Config.enableR22Sequence rather than restating it — +// the hardcoded "15" outlived the sequence growing to 16 and declared success a flag early. +import com.noop.protocol.Whoop5Config import com.noop.ble.WhoopModel import com.noop.data.DataBackup import com.noop.ingest.RawSensorExport @@ -465,6 +471,17 @@ fun SettingsScreen( var puffinExperiments by remember(rev) { mutableStateOf(puffinExperiment.isEnabled) } var puffinCapture by remember(rev) { mutableStateOf(puffinExperiment.isCaptureEnabled) } var deepData by remember(rev) { mutableStateOf(puffinExperiment.isDeepDataEnabled) } + + // #174: set when the deep-data switch is turned OFF, so the app can OFFER to clear the flags on the + // strap instead of silently leaving them set. The switch alone has never written anything in either + // direction — it gates sends — so turning it off used to change nothing on the hardware while reading + // like an undo. Asking is the right shape rather than writing automatically: the strap may not be + // connected, and a write to bonded hardware is not something a toggle should do unannounced. + var confirmingDeepDataDisable by remember { mutableStateOf(false) } + val r22DisableReport by vm.ble.r22DisableReport.collectAsState() + // How many flags the enable sequence actually writes. Read from the sequence rather than restated, so + // the card cannot drift from it again — it said "15" for the whole life of the 16-flag sequence. + val r22FlagCount = Whoop5Config.enableR22Sequence.size var broadcastHr by remember(rev) { mutableStateOf(puffinExperiment.broadcastHr) } // "Sleep staging (V2)" — V2 is the DEFAULT for every strap (WHOOP 4 and 5/MG); turn it OFF to fall back // to V1. Model-agnostic, so it lives outside the 5/MG-only card. 4.0 is unvalidated either way (#319/#347). @@ -1869,6 +1886,11 @@ fun SettingsScreen( onCheckedChange = { deepData = it puffinExperiment.isDeepDataEnabled = it + // #174: turning the switch OFF used to write nothing — it only hid the enable + // button, so the strap kept every flag the sequence set while the UI implied it + // had been undone. Now it offers the real undo. Turning it ON still writes + // nothing until the button is tapped. + if (!it) confirmingDeepDataDisable = true }, colors = SwitchDefaults.colors( checkedThumbColor = Palette.surfaceBase, @@ -1902,13 +1924,35 @@ fun SettingsScreen( style = NoopType.caption, color = Palette.textTertiary, ) - // Live R22 telemetry (#174): proof of what the strap is doing right now. + // #174: the undo. Offered whenever the flags may be set — which is any time the + // opt-in has been on, not only right after a send, because the flags persist across + // launches and the app has no record of what a previous install wrote. Wear is NOT + // required: the on-wrist gate exists because the R22 STREAM is on-wrist only. + NoopButton( + text = uiString(R.string.l10n_settings_screen_turn_deep_data_back_off_r22disable), + leadingIcon = Icons.Filled.Cancel, + kind = NoopButtonKind.Secondary, + enabled = live.encryptedBond && r22DisableReport != WhoopBleClient.WAITING_DEVICE_CONFIG_PROBE, + onClick = { vm.ble.disableWhoop5DeepData() }, + ) + Text( + if (!live.encryptedBond) uiString(R.string.l10n_settings_screen_r22disable_needs_bond) + else uiString(R.string.l10n_settings_screen_r22disable_reason), + style = NoopType.caption, + color = Palette.textTertiary, + ) + // Live R22 telemetry (#174): proof of what the strap is doing right now. The threshold + // and the number are both driven off the sequence itself — they were hardcoded to 15 + // while the sequence carried 16, so the card declared success one flag early. if (live.r22FlagsAccepted > 0) { Text( - if (live.r22FlagsAccepted >= 15) "✓ Strap accepted all 15 R22 flags" - else "Strap accepted ${live.r22FlagsAccepted}/15 R22 flags…", + if (live.r22FlagsAccepted >= r22FlagCount) { + uiString(R.string.l10n_settings_screen_r22_accepted_all, r22FlagCount) + } else { + uiString(R.string.l10n_settings_screen_r22_accepted_partial, live.r22FlagsAccepted, r22FlagCount) + }, style = NoopType.caption, - color = if (live.r22FlagsAccepted >= 15) Palette.statusPositive else Palette.textSecondary, + color = if (live.r22FlagsAccepted >= r22FlagCount) Palette.statusPositive else Palette.textSecondary, ) } if (live.deepPacketsThisSession > 0) { @@ -1917,13 +1961,38 @@ fun SettingsScreen( style = NoopType.caption, color = Palette.textSecondary, ) - } else if (live.r22FlagsAccepted >= 15) { + } else if (live.r22FlagsAccepted >= r22FlagCount) { Text( uiString(R.string.l10n_settings_screen_flags_accepted_but_the_enable_sequence_542b2595), style = NoopType.caption, color = Palette.textTertiary, ) } + + // #174: the disable run's per-key result. Shown verbatim because the interesting part + // is the read-back table, not a green tick — a write that acked SUCCESS but did not + // move the stored value renders here as "unchanged", which is the case worth seeing. + val disableReport = r22DisableReport + if (disableReport != null) { + if (disableReport == WhoopBleClient.WAITING_DEVICE_CONFIG_PROBE) { + Text( + uiString(R.string.l10n_settings_screen_r22disable_running), + style = NoopType.caption, + color = Palette.textSecondary, + ) + } else { + Text( + disableReport, + style = NoopType.caption.copy(fontFamily = FontFamily.Monospace), + color = Palette.textSecondary, + ) + NoopButton( + text = uiString(R.string.l10n_settings_screen_r22disable_dismiss), + kind = NoopButtonKind.Secondary, + onClick = { vm.ble.clearR22DisableReport() }, + ) + } + } } Row( @@ -2302,6 +2371,49 @@ fun SettingsScreen( } } + // #174: the switch going OFF is the moment to offer the undo. Declining leaves the flags set and + // says so — still an improvement on the old behaviour, where the same tap silently left them set + // with no indication either way. + if (confirmingDeepDataDisable) { + AlertDialog( + onDismissRequest = { confirmingDeepDataDisable = false }, + containerColor = Palette.surfaceOverlay, + title = { + Text( + uiString(R.string.l10n_settings_screen_r22disable_confirm_title), + style = NoopType.title2, + color = Palette.textPrimary, + ) + }, + text = { + Text( + uiString(R.string.l10n_settings_screen_r22disable_confirm_body), + style = NoopType.subhead, + color = Palette.textSecondary, + ) + }, + confirmButton = { + TextButton(onClick = { + confirmingDeepDataDisable = false + vm.ble.disableWhoop5DeepData() + }) { + Text( + uiString(R.string.l10n_settings_screen_r22disable_confirm_action), + color = Palette.accent, + ) + } + }, + dismissButton = { + TextButton(onClick = { confirmingDeepDataDisable = false }) { + Text( + uiString(R.string.l10n_settings_screen_r22disable_confirm_cancel), + color = Palette.textSecondary, + ) + } + }, + ) + } + if (showRecalibrateConfirm) { AlertDialog( onDismissRequest = { showRecalibrateConfirm = false }, diff --git a/android/app/src/main/res/values-de/strings.xml b/android/app/src/main/res/values-de/strings.xml index ac2641c279..76ca78c34c 100644 --- a/android/app/src/main/res/values-de/strings.xml +++ b/android/app/src/main/res/values-de/strings.xml @@ -1772,4 +1772,15 @@ Akku-Hinweise Hinweise, wenn der Strap-Akku schwach oder voll geladen ist. Energiesparmodus beruhigt die Anzeigen, übersetzte Android-Benachrichtigungen und sofort geladene Diagramme + Tiefe Daten wieder ausschalten + Benötigt die vollständig verschlüsselte Kopplung: Schließe zuerst die offizielle WHOOP-App und koppele das Band mit NOOP (eine Verbindung nur für den Live-Puls kann den Schreibvorgang nicht übertragen). + Schreibt den Aus-Wert in alle 16 Flags und liest jedes einzeln zurück, sodass du siehst, was das Band tatsächlich speichert — nicht nur, dass es bestätigt hat. Der Aus-Wert ist von dem Flag abgeleitet, das NOOP für die Garmin-Übertragung bereits so ausschaltet; auf einem R22-Flag wurde er noch nie beobachtet. Deshalb testet NOOP zuerst ein einzelnes Flag und bricht ab, wenn das Band es ablehnt. Gelöschte Flags sind nicht dasselbe wie tatsächlich versiegende Tiefendaten: prüfe das mit einer anschließenden Synchronisierung. + R22-Flags werden gelöscht und einzeln zurückgelesen… + Deaktivierungsbericht schließen + R22-Flags auf dem Band löschen? + Diesen Schalter auszuschalten stoppt nur, dass NOOP die Freischaltung sendet. Die bereits geschriebenen Flags bleiben auf dem Band, bis sie etwas löscht. NOOP kann jetzt den Aus-Wert in alle 16 schreiben und jedes zurücklesen, damit du siehst, was das Band tatsächlich speichert. Das Band muss verbunden und gekoppelt sein. + Flags auf dem Band löschen + Nur nicht mehr senden + ✓ Band hat alle %1$d R22-Flags akzeptiert + Band hat %1$d/%2$d R22-Flags akzeptiert… diff --git a/android/app/src/main/res/values-es/strings.xml b/android/app/src/main/res/values-es/strings.xml index 3c5d75c7e7..4a6a3e8888 100644 --- a/android/app/src/main/res/values-es/strings.xml +++ b/android/app/src/main/res/values-es/strings.xml @@ -1757,4 +1757,15 @@ Alertas de batería Avisos cuando la batería de la pulsera está baja o completamente cargada. El ahorro de batería calma los indicadores, notificaciones de Android traducidas y gráficos instantáneos + Volver a desactivar los datos profundos + Necesita el vínculo cifrado completo: cierra primero la app oficial de WHOOP y empareja la pulsera con NOOP (un enlace solo de FC en vivo no puede transportar la escritura). + Escribe el valor de apagado en las 16 marcas y vuelve a leer cada una, para que veas lo que la pulsera guarda en vez de solo que confirmó. El valor de apagado se deduce de la marca que NOOP ya desactiva así para la difusión Garmin; nunca se ha visto en una marca R22, por eso NOOP prueba primero con una sola y se detiene si la pulsera la rechaza. Borrar las marcas no es lo mismo que ver parar los registros profundos: compruébalo sincronizando después. + Borrando las marcas R22 y releyendo cada una… + Descartar el informe de desactivación + ¿Borrar las marcas R22 de la pulsera? + Apagar este interruptor solo impide que NOOP envíe el desbloqueo. Las marcas ya escritas siguen en la pulsera hasta que algo las borre. NOOP puede escribir ahora el valor de apagado en las 16 y releer cada una para que veas lo que la pulsera guarda realmente. Requiere la pulsera conectada y vinculada. + Borrar marcas en la pulsera + Solo dejar de enviar + ✓ La pulsera aceptó las %1$d marcas R22 + La pulsera aceptó %1$d/%2$d marcas R22… diff --git a/android/app/src/main/res/values-fr/strings.xml b/android/app/src/main/res/values-fr/strings.xml index 450535b1fe..ed6949cd79 100644 --- a/android/app/src/main/res/values-fr/strings.xml +++ b/android/app/src/main/res/values-fr/strings.xml @@ -1757,4 +1757,15 @@ Alertes de batterie Alertes lorsque la batterie du bracelet est faible ou complètement chargée. L\'économiseur de batterie fige les jauges, notifications Android traduites et graphiques instantanés + Désactiver à nouveau les données détaillées + Nécessite l’appairage chiffré complet : fermez d’abord l’app WHOOP officielle et appairez le bracelet à NOOP (une liaison limitée à la FC en direct ne peut pas transporter l’écriture). + Écrit la valeur d’arrêt dans les 16 indicateurs et relit chacun d’eux, pour que vous voyiez ce que le bracelet enregistre réellement et pas seulement qu’il a accusé réception. La valeur d’arrêt est déduite de l’indicateur que NOOP désactive déjà ainsi pour la diffusion Garmin ; elle n’a jamais été observée sur un indicateur R22, c’est pourquoi NOOP en teste un seul d’abord et s’arrête si le bracelet le refuse. Effacer les indicateurs n’équivaut pas à voir cesser les enregistrements détaillés : vérifiez-le en synchronisant ensuite. + Effacement des indicateurs R22 et relecture de chacun… + Fermer le rapport de désactivation + Effacer les indicateurs R22 sur votre bracelet ? + Désactiver cet interrupteur empêche seulement NOOP d’envoyer le déverrouillage. Les indicateurs déjà écrits restent sur le bracelet jusqu’à ce que quelque chose les efface. NOOP peut maintenant écrire la valeur d’arrêt dans les 16 et relire chacun d’eux pour que vous voyiez ce que le bracelet enregistre réellement. Nécessite le bracelet connecté et appairé. + Effacer les indicateurs + Cesser seulement d’envoyer + ✓ Le bracelet a accepté les %1$d indicateurs R22 + Le bracelet a accepté %1$d/%2$d indicateurs R22… diff --git a/android/app/src/main/res/values-pt-rPT/strings.xml b/android/app/src/main/res/values-pt-rPT/strings.xml index 6c570f4826..02ed09769b 100644 --- a/android/app/src/main/res/values-pt-rPT/strings.xml +++ b/android/app/src/main/res/values-pt-rPT/strings.xml @@ -1751,4 +1751,15 @@ Alertas de bateria Avisos quando a bateria da correia está fraca ou totalmente carregada. A poupança de bateria acalma os indicadores, notificações Android traduzidas e gráficos instantâneos + Voltar a desativar os dados profundos + Necessita da ligação encriptada completa: feche primeiro a app oficial da WHOOP e emparelhe a pulseira com o NOOP (uma ligação apenas de FC em direto não consegue transportar a escrita). + Escreve o valor de desativação nas 16 marcas e lê cada uma de volta, para que veja o que a pulseira guarda e não apenas que confirmou. O valor de desativação é inferido da marca que o NOOP já desliga assim para a transmissão Garmin — nunca foi visto numa marca R22, por isso o NOOP testa primeiro uma só e pára se a pulseira a recusar. Limpar as marcas não é o mesmo que ver os registos profundos pararem: confirme isso sincronizando depois. + A limpar as marcas R22 e a reler cada uma… + Dispensar o relatório de desativação + Limpar as marcas R22 na pulseira? + Desligar este interruptor apenas impede o NOOP de enviar o desbloqueio. As marcas já escritas permanecem na pulseira até que algo as limpe. O NOOP pode agora escrever o valor de desativação nas 16 e reler cada uma para que veja o que a pulseira guarda realmente. Necessita da pulseira ligada e emparelhada. + Limpar marcas na pulseira + Apenas parar de enviar + ✓ A pulseira aceitou as %1$d marcas R22 + A pulseira aceitou %1$d/%2$d marcas R22… diff --git a/android/app/src/main/res/values-zh/strings.xml b/android/app/src/main/res/values-zh/strings.xml index 8f1d8eca21..913c5984ae 100644 --- a/android/app/src/main/res/values-zh/strings.xml +++ b/android/app/src/main/res/values-zh/strings.xml @@ -1685,4 +1685,15 @@ 电量提醒 当手环电量低或已充满时提醒。 省电模式让仪表静止、Android 通知已翻译、图表秒开 + ✓ 手环已接受全部 %1$d 个 R22 标记 + 手环已接受 %1$d/%2$d 个 R22 标记… + 重新关闭深层数据 + 需要完整的加密配对：请先关闭官方 WHOOP 应用并将手环与 NOOP 配对（仅传输实时心率的连接无法承载写入）。 + 向全部 16 个标记写入关闭值，并逐一读回，让你看到手环实际存储的内容，而不仅仅是它已确认。该关闭值是从 NOOP 已用同样方式关闭的 Garmin 广播标记推断出来的——在 R22 标记上从未被观察到，因此 NOOP 会先试一个标记，若手环拒绝就停止。清除标记与看到深层记录停止并不相同：之后同步一次来确认。 + 正在清除 R22 标记并逐一读回… + 关闭停用报告 + 要清除手环上的 R22 标记吗？ + 关闭此开关只会让 NOOP 停止发送解锁序列。已写入的标记会留在手环上，直到有东西清除它们。NOOP 现在可以向全部 16 个标记写入关闭值并逐一读回，让你看到手环实际存储的内容。需要手环已连接并完成配对。 + 清除手环上的标记 + 仅停止发送 diff --git a/android/app/src/main/res/values/strings.xml b/android/app/src/main/res/values/strings.xml index a25810444c..8cc57804f6 100644 --- a/android/app/src/main/res/values/strings.xml +++ b/android/app/src/main/res/values/strings.xml @@ -1783,4 +1783,15 @@ Battery alerts Alerts when the strap battery is low or fully charged. Battery saver quiets the gauges, translated Android notifications, and instant chart loads + Turn deep data back off + Needs the full encrypted bond: close the official WHOOP app and pair the strap to NOOP first (a live-HR-only link can’t carry the write). + Writes the off value to all 16 flags and reads each one back, so you see what the strap stores rather than just that it acked. The off value is inferred from the flag NOOP already turns off this way for Garmin broadcast — it has never been seen on an R22 flag, so NOOP tries one flag first and stops if the strap refuses it. Clearing the flags is not the same as watching the deep records stop: check that by syncing afterwards. + Clearing R22 flags and reading each one back… + Dismiss disable report + Clear the R22 flags on your strap? + Turning this switch off only stops NOOP sending the unlock. The flags it already wrote stay on the strap until something clears them. NOOP can write the off value to all 16 now and read each one back so you can see what the strap actually stores. Needs the strap connected and bonded. + Clear flags on strap + Just stop sending + ✓ Strap accepted all %1$d R22 flags + Strap accepted %1$d/%2$d R22 flags… diff --git a/android/app/src/test/java/com/noop/protocol/R22DisableTest.kt b/android/app/src/test/java/com/noop/protocol/R22DisableTest.kt new file mode 100644 index 0000000000..c0d7e2f46d --- /dev/null +++ b/android/app/src/test/java/com/noop/protocol/R22DisableTest.kt @@ -0,0 +1,353 @@ +package com.noop.protocol + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertNotNull +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * #174: the R22 DISABLE path — the sequence bytes, the key-aware SET_FF_VALUE(120) allowlist, and the + * staged write→read-back run. Twin of the Swift `R22DisableTests`; keep the assertions in lockstep. + * + * Everything here runs with no app, no strap and no Bluetooth. That matters more than usual: the one + * thing that CANNOT be tested off-strap is whether the firmware accepts the off value, and the design is + * built so the run itself answers that on first use rather than assuming it. + */ +class R22DisableTest { + + // The sequence ------------------------------------------------------------------------------ + + @Test + fun disableSequenceMirrorsEnableExactly() { + val enable = Whoop5Config.enableR22Sequence + val disable = Whoop5Config.disableR22Sequence + assertEquals("the undo must cover every key the enable sets", enable.size, disable.size) + assertEquals(enable.map { it.name }, disable.map { it.name }) + assertTrue(disable.all { it.value == Whoop5Config.FEATURE_FLAG_OFF_VALUE }) + } + + @Test + fun offValueIsAsciiZero() { + assertEquals(0x30, Whoop5Config.FEATURE_FLAG_OFF_VALUE) + assertEquals('0'.code, Whoop5Config.FEATURE_FLAG_OFF_VALUE) + } + + /** The disable frame differs from the enable frame in EXACTLY one byte — the value at frame offset 44 + * — plus the CRC32 trailer it necessarily changes. */ + @Test + fun disableFrameDiffersFromEnableInOnlyTheValueByte() { + val on = Whoop5Config.frame(Whoop5Config.Flag("enable_r22_packets", 0x32), 1) + val off = Whoop5Config.frame(Whoop5Config.Flag("enable_r22_packets", 0x30), 1) + assertEquals(on.size, off.size) + val differing = on.indices.filter { on[it] != off[it] } + assertEquals("the value byte sits at frame offset 44", 44, differing.first()) + assertEquals("value byte + CRC32 trailer only", 5, differing.size) + assertEquals(0x30, off[44].toInt() and 0xFF) + } + + @Test + fun disableFramesVerifyAndCarryDistinctSeqs() { + val frames = Whoop5Config.disableSequenceFrames(1) + assertEquals(16, frames.size) + assertEquals((1..16).toList(), frames.map { it[9].toInt() and 0xFF }) + for (f in frames) { + assertTrue("every disable frame must pass 5/MG CRCs", Framing.frameCrcOk(f, DeviceFamily.WHOOP5)) + } + } + + /** `disable_pip_r26_packets` gets the SAME off byte as the `enable_*` keys: this sequence is the undo + * of the enable sequence, not a per-key semantic inversion. Pinned because it is the one line a + * reviewer is most likely to "fix". */ + @Test + fun disablePipR26GetsTheSameOffByteAsTheEnableKeys() { + val pip = Whoop5Config.disableR22Sequence.firstOrNull { it.name == "disable_pip_r26_packets" } + assertEquals(Whoop5Config.FEATURE_FLAG_OFF_VALUE, pip?.value) + assertEquals( + "and the enable value stays '2' — this key is why boolean semantics are not assumed", + 0x32, + Whoop5Config.enableR22Sequence.firstOrNull { it.name == "disable_pip_r26_packets" }?.value, + ) + } + + // The write gate ---------------------------------------------------------------------------- + + @Test + fun gateAdmitsOnlyOpcode120() { + val payload = byteArrayOf(0x01) + Whoop5Config.payloadBody("enable_r22_packets", 0x30) + for (opcode in 0..255) { + assertEquals( + "opcode $opcode admission", + opcode == 120, + FeatureFlagWriteGate.admitsSend(opcode, payload, deepDataOptIn = true), + ) + } + } + + @Test + fun gateRefusesDeviceConfigWriteVerb() { + val payload = byteArrayOf(0x01) + Whoop5Config.payloadBody("enable_r22_packets", 0x30) + assertFalse( + "119 keeps its own Broadcast-HR clause and must never be reachable from here", + FeatureFlagWriteGate.admitsSend(119, payload, deepDataOptIn = true), + ) + } + + @Test + fun gateRefusesEverythingWithoutTheOptIn() { + for (flag in Whoop5Config.enableR22Sequence) { + for (value in listOf(flag.value, Whoop5Config.FEATURE_FLAG_OFF_VALUE)) { + val payload = byteArrayOf(0x01) + Whoop5Config.payloadBody(flag.name, value) + assertFalse( + "${flag.name}=$value must be refused with the opt-in off", + FeatureFlagWriteGate.admitsSend(120, payload, deepDataOptIn = false), + ) + } + } + } + + @Test + fun gateAdmitsEveryR22KeyForBothItsEnableValueAndOff() { + for (flag in Whoop5Config.enableR22Sequence) { + for (value in listOf(flag.value, Whoop5Config.FEATURE_FLAG_OFF_VALUE)) { + val payload = byteArrayOf(0x01) + Whoop5Config.payloadBody(flag.name, value) + assertTrue( + "${flag.name}=$value must be admitted", + FeatureFlagWriteGate.admitsSend(120, payload, deepDataOptIn = true), + ) + } + } + } + + /** The tightening this gate exists for: before it, opcode 120 was admitted on the opt-in alone, so ANY + * feature-flag key with ANY value could travel. Both halves of that are now closed. */ + @Test + fun gateRefusesUnknownKeysAndUnknownValues() { + for (key in listOf("general_ab_test", "enable_pdaf_walk_det", "enable_maverick_model", "enable_r22_v7_packets")) { + val payload = byteArrayOf(0x01) + Whoop5Config.payloadBody(key, 0x30) + assertFalse("$key is not an R22 key", FeatureFlagWriteGate.admitsSend(120, payload, deepDataOptIn = true)) + } + for (value in listOf(0x00, 0x29, 0x33, 0x39, 0x41, 0xFF)) { + val payload = byteArrayOf(0x01) + Whoop5Config.payloadBody("enable_r22_packets", value) + assertFalse( + "value 0x%02x is neither the enable nor the off value".format(value), + FeatureFlagWriteGate.admitsSend(120, payload, deepDataOptIn = true), + ) + } + // enable_r22_v4_packets' enable value is '1', so '2' must be refused for THAT key specifically. + val wrongForV4 = byteArrayOf(0x01) + Whoop5Config.payloadBody("enable_r22_v4_packets", 0x32) + assertFalse( + "the admitted value is per-key, not a shared pair", + FeatureFlagWriteGate.admitsSend(120, wrongForV4, deepDataOptIn = true), + ) + } + + @Test + fun gateRefusesMalformedBodies() { + val good = Whoop5Config.payloadBody("enable_r22_packets", 0x30) + assertFalse(FeatureFlagWriteGate.admitsSend(120, byteArrayOf(0x02) + good, deepDataOptIn = true)) + assertFalse(FeatureFlagWriteGate.admitsSend(120, byteArrayOf(0x01) + good.copyOfRange(0, good.size - 1), deepDataOptIn = true)) + assertFalse(FeatureFlagWriteGate.admitsSend(120, ByteArray(0), deepDataOptIn = true)) + val dirty = good.copyOf(); dirty[25] = 0x41 + assertFalse( + "a name field whose padding is not NUL is not a name field", + FeatureFlagWriteGate.admitsSend(120, byteArrayOf(0x01) + dirty, deepDataOptIn = true), + ) + val binary = good.copyOf(); binary[3] = 0x01 + assertFalse(FeatureFlagWriteGate.admitsSend(120, byteArrayOf(0x01) + binary, deepDataOptIn = true)) + } + + @Test + fun keyAndValueRoundTripsEveryR22Key() { + for (flag in Whoop5Config.enableR22Sequence) { + val payload = byteArrayOf(0x01) + Whoop5Config.payloadBody(flag.name, Whoop5Config.FEATURE_FLAG_OFF_VALUE) + val parsed = FeatureFlagWriteGate.keyAndValue(payload) + assertEquals(flag.name, parsed?.key) + assertEquals(0x30, parsed?.value) + } + } + + @Test + fun readBackOpcodeIsOnly128() { + for (opcode in 0..255) { + assertEquals(opcode == 128, FeatureFlagWriteGate.isReadBackOpcode(opcode)) + } + } + + // The staged run ---------------------------------------------------------------------------- + + /** A synthetic GET_FF_VALUE reply record: the key NUL-padded to 32 bytes, then the value byte. */ + private fun reply(key: String, value: Int, resultCode: Int = 1): DeviceConfigReadProbe.ValueResponse { + val record = ByteArray(33) + val bytes = key.toByteArray(Charsets.US_ASCII) + for (i in bytes.indices) if (i < 32) record[i] = bytes[i] + record[32] = (value and 0xFF).toByte() + return DeviceConfigReadProbe.ValueResponse(resultCode, record) + } + + private fun failureReply() = DeviceConfigReadProbe.ValueResponse(0, ByteArray(0)) + + @Test + fun planStartsWithTheProbeWriteThenItsReadBack() { + val r = R22DisableReport() + val first = r.nextStep() + assertEquals(R22DisableReport.Stage.PROBE_WRITE, first?.stage) + assertEquals(R22DisableReport.PROBE_KEY, first?.key) + assertEquals(120, first?.opcode) + val second = r.nextStep() + assertEquals(R22DisableReport.Stage.PROBE_READ, second?.stage) + assertEquals(128, second?.opcode) + } + + /** The gate: a probe read-back still showing the old value STOPS the run with fifteen keys untouched. */ + @Test + fun probeRejectionStopsTheRunAndLeavesFifteenKeysUntouched() { + val r = R22DisableReport() + val write = r.nextStep()!! + r.noteWriteAck(1, write) // SUCCESS ack — deliberately not believed + val read = r.nextStep()!! + r.noteReadBack(reply(R22DisableReport.PROBE_KEY, 0x31), read) + + assertNull("no further steps once the off value is rejected", r.nextStep()) + assertEquals(R22DisableReport.Verdict.OFF_VALUE_REJECTED, r.verdict) + assertEquals(R22DisableReport.KeyOutcome.UNCHANGED, r.outcomes[R22DisableReport.PROBE_KEY]) + assertEquals(15, r.keys.count { r.outcomes[it] == R22DisableReport.KeyOutcome.SKIPPED }) + assertTrue(r.render().contains("is NOT how this firmware clears a feature flag")) + assertEquals(1, r.writeAcks[R22DisableReport.PROBE_KEY]) + } + + @Test + fun fullRunClearsAllSixteen() { + val r = R22DisableReport() + var steps = 0 + while (true) { + val step = r.nextStep() ?: break + steps++ + assertTrue("plan must terminate", steps < 100) + if (step.isWrite) { + assertEquals(120, step.opcode) + r.noteWriteAck(1, step) + } else { + assertEquals(128, step.opcode) + r.noteReadBack(reply(step.key, 0x30), step) + } + } + assertEquals(R22DisableReport.Verdict.ALL_CLEARED, r.verdict) + assertEquals(16, r.keys.size) + for (key in r.keys) assertEquals("$key should be cleared", R22DisableReport.KeyOutcome.CLEARED, r.outcomes[key]) + // 1 probe write + 1 probe read + 15 clear writes + 16 verify reads + assertEquals(33, steps) + assertTrue(r.summary.contains("All 16 R22 flags cleared")) + } + + /** FAILURE on a read-back means the key no longer has a stored value at all — its own state, not + * folded into "cleared". */ + @Test + fun failureReadBackIsReportedAsUnsetNotCleared() { + val r = R22DisableReport() + r.noteWriteAck(1, r.nextStep()!!) + r.noteReadBack(failureReply(), r.nextStep()!!) + assertEquals(R22DisableReport.KeyOutcome.UNSET, r.outcomes[R22DisableReport.PROBE_KEY]) + assertTrue("an unset key is a success — the run should continue", r.probePassed) + assertNotNull("the clear stage must follow a passing probe", r.nextStep()) + } + + @Test + fun partialRunIsReportedAsPartialNotSuccess() { + val r = R22DisableReport() + while (true) { + val step = r.nextStep() ?: break + when { + step.isWrite -> r.noteWriteAck(1, step) + step.key == "wear_detect_bias" -> r.noteReadBack(reply(step.key, 0x32), step) + else -> r.noteReadBack(reply(step.key, 0x30), step) + } + } + assertEquals(R22DisableReport.Verdict.PARTIAL, r.verdict) + assertEquals(R22DisableReport.KeyOutcome.UNCHANGED, r.outcomes["wear_detect_bias"]) + assertTrue(r.summary.contains("15 of 16")) + } + + /** A SUCCESS ack whose read-back did not move must never render as success — the #907/#891 rule. */ + @Test + fun successAckWithUnmovedValueIsNeverReportedAsSuccess() { + val r = R22DisableReport() + r.noteWriteAck(1, r.nextStep()!!) + r.noteReadBack(reply(R22DisableReport.PROBE_KEY, 0x31), r.nextStep()!!) + val text = r.render() + assertFalse(r.verdict == R22DisableReport.Verdict.ALL_CLEARED) + assertTrue(text.contains("recorded, not proof")) + assertTrue(text.contains("NOT treated as proof")) + } + + @Test + fun silentProbeStopsTheRun() { + val r = R22DisableReport() + val write = r.nextStep()!! + r.noteTimeout(write, 8) + assertNull("a silent WRITE decides nothing — only the read-back does", r.outcomes[R22DisableReport.PROBE_KEY]) + r.noteTimeout(r.nextStep()!!, 8) + assertEquals(R22DisableReport.Verdict.PROBE_INCONCLUSIVE, r.verdict) + assertNull(r.nextStep()) + } + + @Test + fun abandonMarksRemainingKeysSkipped() { + val r = R22DisableReport() + r.noteWriteAck(1, r.nextStep()!!) + r.noteAbandoned("link dropped") + assertEquals(R22DisableReport.Verdict.ABANDONED, r.verdict) + assertNull(r.nextStep()) + assertEquals(16, r.keys.count { r.outcomes[it] == R22DisableReport.KeyOutcome.SKIPPED }) + } + + @Test + fun undecodableReadBackIsNotTreatedAsCleared() { + val r = R22DisableReport() + r.noteWriteAck(1, r.nextStep()!!) + r.noteReadFailure(DeviceConfigReadProbe.ParseFailure.CRC, r.nextStep()!!) + assertEquals(R22DisableReport.KeyOutcome.UNDECODABLE, r.outcomes[R22DisableReport.PROBE_KEY]) + assertFalse(r.probePassed) + assertEquals(R22DisableReport.Verdict.PROBE_INCONCLUSIVE, r.verdict) + } + + /** The report must state the four standing caveats, so the log, the UI and the PR cannot drift on what + * was actually established. */ + @Test + fun renderCarriesTheCaveats() { + val r = R22DisableReport() + while (true) { + val step = r.nextStep() ?: break + if (step.isWrite) r.noteWriteAck(1, step) else r.noteReadBack(reply(step.key, 0x30), step) + } + val text = r.render() + assertTrue(text.contains("'0' as the off value is INFERRED")) + assertTrue(text.contains("not the same as reverted BEHAVIOUR")) + assertTrue(text.contains("does not restore a snapshot")) + assertTrue(text.contains("disable_pip_r26_packets inverts")) + for (key in r.keys) assertTrue("$key missing from the report", text.contains(key)) + } + + @Test + fun probeKeyIsTheOneWithAHardwareWriteDemonstration() { + assertEquals("enable_sig12", R22DisableReport.PROBE_KEY) + assertTrue(Whoop5Config.enableR22Sequence.any { it.name == R22DisableReport.PROBE_KEY }) + } + + /** Cross-platform parity: the Swift and Kotlin reports must render the same verdict labels, or a + * shared strap log reads differently on the two platforms. */ + @Test + fun verdictAndOutcomeLabelsMatchTheSwiftSpelling() { + assertEquals("allCleared", R22DisableReport.Verdict.ALL_CLEARED.label) + assertEquals("partial", R22DisableReport.Verdict.PARTIAL.label) + assertEquals("offValueRejected", R22DisableReport.Verdict.OFF_VALUE_REJECTED.label) + assertEquals("probeInconclusive", R22DisableReport.Verdict.PROBE_INCONCLUSIVE.label) + assertEquals("abandoned", R22DisableReport.Verdict.ABANDONED.label) + assertEquals("cleared", R22DisableReport.KeyOutcome.CLEARED.label) + assertEquals("unset", R22DisableReport.KeyOutcome.UNSET.label) + assertEquals("unchanged", R22DisableReport.KeyOutcome.UNCHANGED.label) + assertEquals("notClaimed", R22DisableReport.KeyOutcome.NOT_CLAIMED.label) + assertEquals("skipped", R22DisableReport.KeyOutcome.SKIPPED.label) + } +} From bd6dc6a0982381fd94be2582a01b0b83bbe325bf Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 28 Jul 2026 11:04:42 -0400 Subject: [PATCH 148/171] fix(cloudsync): stop the /ingest fallback costing the replicator a full snapshot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Page replication was measured at a 100% snapshot rate on VK's device: every recorded push shipped the whole database, which makes liters a full upload with extra steps. The cause was the `/ingest` fallback's export, which checkpoints the live store before archiving it — and that fallback runs on exactly the syncs where the liters push did not, so it lands between every pair of pushes. Device evidence (push-telemetry.json, 2026-07-28): `/ingest` at 06:03, then a push at 14:23 with `snapshotReason: "wal truncated by another process"` and `bytesUploaded: 640736289`, matching the server's `bucketBytes` exactly. The obvious fix — a gentler checkpoint mode — does NOT work, and that was measured rather than assumed. Against a real liters writer and Apple's libsqlite3, with no replicator read lock held: foreign TRUNCATE -> -wal 0 bytes -> next push SNAPSHOT ("wal truncated ...") foreign FULL -> -wal unchanged -> next push SNAPSHOT (same reason) `FULL` looks harmless because the `-wal` file keeps its size. It is not: once a checkpoint has fully backfilled the WAL and no reader still needs those frames, SQLite restarts the WAL on the next write transaction — new salt, frame 1. The property that matters is "did the WAL end up fully backfilled with nothing pinning it", and every checkpoint mode reaches it. So under `WalCheckpointing.external` the fallback takes no checkpoint at all. It stages a consistent full copy through SQLite's Online Backup API — which reads *through* the WAL and never checkpoints — and archives that. `/ingest` still ships a complete, fresh, `quick_check`-verified database through the identical `.noopbak` container; the replicator's resume point is untouched. Staging failure degrades to the old checkpointing path rather than failing the sync. Under `.automatic` — every build that has not opted into the trial, upstream included — nothing changes: same checkpoint, same live file, byte for byte. `Repository.checkpointForBackup()` is deliberately left truncating; its callers are the user-initiated Export button and the at-most-daily folder backup, which want the disk reclaimed. The hazard is named in a comment there. No Android twin: this is fork-local cloud-sync plumbing with no stored-data, analytics, migration or `.noopbak` container change. Verified: LitersRoundTripTests, against the real Liters.xcframework and a real bucket — foreign TRUNCATE -> snapshotted=true foreign FULL -> snapshotted=true staged copy -> snapshotted=FALSE, 37,997 B vs the 159,077 B snapshot it follows WhoopStore package: 391 tests, 0 failures (5 new in ConsistentCopyTests, incl. the live `-wal` being byte-identical after a copy and the main file not backfilled). StrandTests (macOS): 1143 tests, 1 failure — `testReplicaRestoreIsBrokenUnderSystem SQLite`, which asserts a liters bug on purpose and now fails because the bug is fixed. Confirmed pre-existing: it fails identically with the pristine test file from d69ffbb5 against the same xcframework. --- .../Sources/WhoopStore/WhoopStore.swift | 62 +++++++ .../WhoopStoreTests/ConsistentCopyTests.swift | 153 ++++++++++++++++++ Strand/CloudSync/CloudSyncUploader.swift | 48 +++++- Strand/Data/DataBackup.swift | 32 ++++ Strand/Data/Repository.swift | 10 ++ StrandTests/LitersRoundTripTests.swift | 136 +++++++++++++++- 6 files changed, 438 insertions(+), 3 deletions(-) create mode 100644 Packages/WhoopStore/Tests/WhoopStoreTests/ConsistentCopyTests.swift diff --git a/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift b/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift index fa513f9e13..8b76fdd7f4 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/WhoopStore.swift @@ -305,6 +305,68 @@ public actor WhoopStore { } } + /// Write a complete, consistent, single-file copy of this database at `path` — **without + /// checkpointing the live one.** + /// + /// ## Why a file-level backup cannot just checkpoint when a replicator owns the WAL + /// + /// A `.noopbak` archives the main `.sqlite` alone, with no `-wal` sidecar, so a backup has to get + /// every committed page into one file somehow. The obvious way is `checkpointWAL()`. Under + /// `WalCheckpointing.external` that is exactly what must not happen, and — this is the part that + /// is easy to get wrong — **a gentler checkpoint mode does not help.** + /// + /// Measured against a real `liters` writer and Apple's `libsqlite3` (`LitersRoundTripTests`, + /// 2026-07-28), with no replicator read lock held: + /// + /// | foreign checkpoint | `-wal` afterwards | next push | + /// |---|---|---| + /// | `TRUNCATE` | 0 bytes | **snapshot**, "wal truncated by another process" | + /// | `FULL` | 168,952 bytes — unchanged | **snapshot**, same reason | + /// + /// `FULL` looks harmless because the `-wal` file keeps its size, and it is not: once a checkpoint + /// has fully backfilled the WAL and no reader still needs those frames, **SQLite restarts the WAL + /// on the next write transaction** — new salt, frame 1 — and the replicator's resume offset is + /// gone just the same. `PASSIVE` has the identical endpoint whenever it happens to complete. The + /// property that matters is not the pragma's name; it is "did the WAL end up fully backfilled + /// with nothing pinning it", and every checkpoint mode reaches that state. + /// + /// ## What this does instead + /// + /// SQLite's Online Backup API (via GRDB's `backup(to:)`) reads the source through a read + /// transaction, so it sees the WAL's contents and copies them into the destination — and it never + /// checkpoints, never restarts, and never touches the source's `-wal` at all. The replicator's + /// resume point survives untouched, and the caller gets a single file that carries every + /// committed row. + /// + /// The cost is one full-size write. That is why this is not the default path: under `.automatic` + /// a checkpoint is free and correct, and this is reserved for the case where a checkpoint is not + /// available (see `CloudSyncUploader.defaultExporter`, its only production caller). + /// + /// Removes `path` and its `-wal`/`-shm` siblings first, and leaves no sidecars behind, so the + /// result is one self-contained file ready to archive. + public func writeConsistentCopy(to path: String) async throws { + try writeConsistentCopyImpl(to: path) + } + + /// Non-async for the same reason `checkpointWALImpl` is: it runs on the actor's executor, off the + /// main thread, and GRDB's calls here are synchronous and blocking. + private func writeConsistentCopyImpl(to path: String) throws { + let fm = FileManager.default + for suffix in ["", "-wal", "-shm"] { try? fm.removeItem(atPath: path + suffix) } + + let destination = try DatabaseQueue(path: path) + try dbWriter.backup(to: destination) + // Close before anyone reads the file: GRDB flushes on close, and the caller's next act is to + // hand this path to a ZIP writer. `close()` throws only if a statement is still live, which + // cannot be the case here. + try destination.close() + + // A `DatabaseQueue` on a fresh file is in rollback-journal mode, but the copied page 1 header + // carries the source's WAL marker, so a `-wal`/`-shm` pair can be left behind. They hold + // nothing — everything was flushed by `close()` — and a `.noopbak` must be one file. + for suffix in ["-wal", "-shm"] { try? fm.removeItem(atPath: path + suffix) } + } + /// Permanently delete every recorded sample/derived row for one device across all `deviceId`-keyed /// tables (16+ `DELETE FROM
WHERE deviceId = ?` in one GRDB transaction). Wraps the /// synchronous `DeviceRegistryStore.deleteAllData` so the heavy multi-table write runs on the actor's diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/ConsistentCopyTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/ConsistentCopyTests.swift new file mode 100644 index 0000000000..a8418c6a8a --- /dev/null +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/ConsistentCopyTests.swift @@ -0,0 +1,153 @@ +import XCTest +import GRDB +import WhoopProtocol +@testable import WhoopStore + +/// `writeConsistentCopy(to:)` exists because a file-level backup and a page replicator both want to +/// decide what happens to the WAL, and only one of them can. It has to satisfy both: +/// +/// 1. **The copy must be whole.** A `.noopbak` is one `.sqlite` with no `-wal` sidecar, so a copy +/// that misses what the WAL held is silent data loss on restore. Asserted by opening the copy +/// somewhere its siblings do not exist and counting rows. +/// 2. **The live WAL must be untouched.** Not "not truncated" — *untouched*. A checkpoint that +/// fully backfills the WAL is just as fatal as one that truncates it, because SQLite restarts a +/// fully-backfilled WAL on the next write, which is what invalidates a replicator's resume +/// offset. So this asserts the `-wal` is byte-identical afterwards, and that a subsequent write +/// still *appends* rather than restarting. +/// +/// The end-to-end consequence — that a real `liters` writer ships a delta rather than a snapshot +/// after one of these — is asserted in `LitersRoundTripTests`, which has an actual replicator. +final class ConsistentCopyTests: XCTestCase { + + private var paths: [String] = [] + + override func tearDown() { + for path in paths { + for suffix in ["", "-wal", "-shm"] { try? FileManager.default.removeItem(atPath: path + suffix) } + } + paths = [] + super.tearDown() + } + + private func tempPath(_ tag: String = "src") -> String { + let path = NSTemporaryDirectory() + "whoopstore-copy-\(tag)-\(UUID().uuidString).sqlite" + paths.append(path) + return path + } + + private func fileSize(_ path: String) -> Int64 { + ((try? FileManager.default.attributesOfItem(atPath: path))?[.size] as? NSNumber)?.int64Value ?? 0 + } + + @discardableResult + private func writeTransactions(_ store: WhoopStore, count: Int, rowsEach: Int = 400, + startingAt ts0: Int = 0) async throws -> Int { + var ts = ts0 + for _ in 0.. Int { + XCTAssertFalse(FileManager.default.fileExists(atPath: path + "-wal"), + "a staged copy must leave no -wal sidecar; a .noopbak carries one file") + let queue = try DatabaseQueue(path: path) + return try queue.read { db in try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM hrSample") ?? -1 } + } + + // MARK: - The copy is complete + + /// Everything committed — including the rows that exist only in the WAL, which is most of them + /// on a replicated store because nothing checkpoints it. + func testTheCopyCarriesEveryCommittedRowIncludingTheOnesStillInTheWal() async throws { + let source = tempPath() + let store = try await WhoopStore(path: source, walCheckpointing: .external, walBackstop: .disabled) + try await writeTransactions(store, count: 20) + + XCTAssertGreaterThan(store.walFileSizeBytes() ?? 0, 0, + "the test is only meaningful if rows are sitting in the WAL") + + let copy = tempPath("dst") + try await store.writeConsistentCopy(to: copy) + + XCTAssertEqual(try hrRows(inFileAlone: copy), 20 * 400) + } + + /// The same under `.automatic`, so the method is not quietly coupled to the replication mode. + func testTheCopyIsCompleteUnderAutomaticCheckpointingToo() async throws { + let source = tempPath() + let store = try await WhoopStore(path: source) + try await writeTransactions(store, count: 10) + + let copy = tempPath("dst") + try await store.writeConsistentCopy(to: copy) + + XCTAssertEqual(try hrRows(inFileAlone: copy), 10 * 400) + } + + // MARK: - The live WAL is untouched + + /// THE CORE ASSERTION. Byte-identical, not merely "not empty": the replicator's resume offset is + /// a position in this file, and any checkpoint — `TRUNCATE`, `FULL`, `PASSIVE` — ends with the + /// WAL fully backfilled and about to be restarted by the next write. + func testTheCopyDoesNotTouchTheLiveWal() async throws { + let source = tempPath() + let store = try await WhoopStore(path: source, walCheckpointing: .external, walBackstop: .disabled) + try await writeTransactions(store, count: 20) + + let walBefore = fileSize(source + "-wal") + let mainBefore = fileSize(source) + XCTAssertGreaterThan(walBefore, 0) + + let copy = tempPath("dst") + try await store.writeConsistentCopy(to: copy) + + XCTAssertEqual(fileSize(source + "-wal"), walBefore, + "the live -wal must be byte-identical — this is the replicator's transport") + XCTAssertEqual(fileSize(source), mainBefore, + "the live database file must not be backfilled either; a fully-backfilled WAL " + + "is restarted by the next write, which is what costs a replicator its resume point") + } + + /// The consequence of the assertion above, stated as behaviour rather than as file sizes: writes + /// after a copy still *append* to the WAL. If the copy had checkpointed, the next write would + /// restart it and the `-wal` would shrink. + func testWritesAfterACopyStillAppendToTheSameWal() async throws { + let source = tempPath() + let store = try await WhoopStore(path: source, walCheckpointing: .external, walBackstop: .disabled) + var ts = try await writeTransactions(store, count: 20) + + let copy = tempPath("dst") + try await store.writeConsistentCopy(to: copy) + let walAfterCopy = fileSize(source + "-wal") + + _ = try await writeTransactions(store, count: 5, startingAt: ts) + ts += 1 + + XCTAssertGreaterThan(fileSize(source + "-wal"), walAfterCopy, + "the WAL must have grown, not restarted") + } + + // MARK: - Housekeeping + + /// Called once per fallback sync, forever, over a path the caller reuses on failure paths. An + /// existing file (and any stale sidecars) must be replaced, not appended to or refused. + func testWritingOverAnExistingCopyReplacesIt() async throws { + let source = tempPath() + let store = try await WhoopStore(path: source, walCheckpointing: .external, walBackstop: .disabled) + try await writeTransactions(store, count: 5) + + let copy = tempPath("dst") + try await store.writeConsistentCopy(to: copy) + XCTAssertEqual(try hrRows(inFileAlone: copy), 5 * 400) + + _ = try await writeTransactions(store, count: 5, startingAt: 5 * 400) + try await store.writeConsistentCopy(to: copy) + XCTAssertEqual(try hrRows(inFileAlone: copy), 10 * 400, + "the second copy must replace the first, not be refused because the file exists") + } +} diff --git a/Strand/CloudSync/CloudSyncUploader.swift b/Strand/CloudSync/CloudSyncUploader.swift index d61d42cfd1..4e98fc6f06 100644 --- a/Strand/CloudSync/CloudSyncUploader.swift +++ b/Strand/CloudSync/CloudSyncUploader.swift @@ -77,8 +77,54 @@ enum CloudSyncUploader { /// The real export: checkpoint `store`'s WAL (so the single `.sqlite` file is whole), then reuse /// the SAME checkpointed, `PRAGMA quick_check`-verified export `BackupSync`/`FolderBackup` use — an /// auto-uploaded snapshot is byte-identical to a manual "Export backup". + /// + /// ## …except when a page replicator owns the WAL, and that exception is the whole point + /// + /// This fallback runs on exactly the syncs where the liters push did *not*, so whatever it does + /// to the WAL happens between every pair of pushes. `checkpointWAL()` is + /// `wal_checkpoint(TRUNCATE)`, which restarts the WAL and destroys the replicator's resume + /// offset. Measured on VK's device 2026-07-28: this checkpoint at 06:03, then a push at 14:23 + /// reporting `snapshotReason: "wal truncated by another process"` and uploading 640 MB. Every + /// recorded push was a full upload — page replication reduced to a full upload with extra steps, + /// by the path it was replacing. + /// + /// A gentler checkpoint mode does **not** fix it: `FULL` leaves the `-wal` file at its full size + /// and still costs the next push a snapshot, because SQLite restarts a fully-backfilled WAL on + /// the next write. Both are measured in `LitersRoundTripTests`. + /// + /// So under `.external` the fallback takes no checkpoint at all. It stages a consistent full copy + /// through SQLite's Online Backup API — which reads *through* the WAL and never checkpoints — + /// and archives that. `/ingest` still ships a complete, fresh, `quick_check`-verified database; + /// the replicator's resume point is untouched. The cost is one full-size temp write, paid only on + /// the fallback path, and only on a device running the trial. + /// + /// If staging fails for any reason (disk, I/O), it degrades to the checkpointing path rather than + /// failing the sync: a sync that ships everything and costs one snapshot beats a sync that ships + /// nothing. static let defaultExporter: Exporter = { store, dest in - await DataBackup.writeBackup(checkpoint: { (try? await store.checkpointWAL()) != nil }, to: dest) + guard case .external = store.walCheckpointing else { + return await DataBackup.writeBackup(checkpoint: { (try? await store.checkpointWAL()) != nil }, + to: dest) + } + // Alongside `dest` (Caches), which the caller already `defer`s a removal of — one directory, + // one cleanup story, and never Documents. + let staged = dest.deletingLastPathComponent() + .appendingPathComponent("cloudsync-staged-\(UUID().uuidString).sqlite") + defer { + let fm = FileManager.default + for suffix in ["", "-wal", "-shm"] { + try? fm.removeItem(atPath: staged.path + suffix) + } + } + do { + try await store.writeConsistentCopy(to: staged.path) + } catch { + NSLog("cloudsync: staging a consistent copy failed (%@) — falling back to a checkpointed " + + "export, which costs the replicator one snapshot", String(describing: error)) + return await DataBackup.writeBackup(checkpoint: { (try? await store.checkpointWAL()) != nil }, + to: dest) + } + return await DataBackup.writeBackup(stagedDatabaseAt: staged, to: dest) } /// Export the live store to a disposable temp file in Caches (never Documents — nothing here is diff --git a/Strand/Data/DataBackup.swift b/Strand/Data/DataBackup.swift index 8187a1017b..28de506c39 100644 --- a/Strand/Data/DataBackup.swift +++ b/Strand/Data/DataBackup.swift @@ -199,6 +199,38 @@ enum DataBackup { } } + /// (Cloud sync, replicated stores) The same verified `.noopbak` as `writeBackup(checkpoint:to:)`, + /// but archiving a database the caller has already staged at `sourceURL` — and therefore taking + /// **no checkpoint at all**. + /// + /// Exists for exactly one caller: `CloudSyncUploader.defaultExporter` under + /// `WalCheckpointing.external`, where a checkpoint of the live store would restart the WAL under + /// a page replicator and cost its next push a full snapshot of the database (measured — see + /// `WhoopStore.writeConsistentCopy(to:)`). The staged file is a consistent full copy produced by + /// SQLite's Online Backup API, so it needs no flushing: it already carries everything the WAL + /// held. The caller owns `sourceURL`'s lifetime. + /// + /// Same `writeVerifiedBackupZip` as every other export, so the container, the entry names, the + /// settings sidecar and the `PRAGMA quick_check` gate are identical — a `.noopbak` produced this + /// way is indistinguishable from one produced by the checkpointing path. Never presents UI. + static func writeBackup(stagedDatabaseAt sourceURL: URL, to dest: URL) async -> BackupResult { + guard FileManager.default.fileExists(atPath: sourceURL.path) else { + return .failure(String(localized: "The staged database copy is missing.")) + } + do { + let fm = FileManager.default + if fm.fileExists(atPath: dest.path) { try fm.removeItem(at: dest) } + // Off the caller's thread for the same reason the interactive export is: a whole-database + // read plus DEFLATE. + try await Task.detached(priority: .utility) { + try writeVerifiedBackupZip(dbURL: sourceURL, to: dest, settingsJSON: currentSettingsJSON()) + }.value + return .exported(dest) + } catch { + return .failure(String(localized: "Backup failed: \(error.localizedDescription)")) + } + } + /// Test seam: write a `.noopbak` for an EXPLICIT source database (no checkpoint, no `StorePaths`), /// so a unit test can round-trip a throwaway SQLite through the exact ZIP container the app writes. /// `settings` (canonical `BackupSettings` keys) adds the `settings.json` entry; nil writes the diff --git a/Strand/Data/Repository.swift b/Strand/Data/Repository.swift index 2d323b168a..276b7a9d8e 100644 --- a/Strand/Data/Repository.swift +++ b/Strand/Data/Repository.swift @@ -594,6 +594,16 @@ final class Repository: ObservableObject { /// Checkpoint the WAL into the main DB file if the store is already open, so a file-level /// backup captures everything. No-op (returns false) if no handle exists yet , the caller /// then copies the on-disk files as-is, which still includes the -wal sidecar. + /// + /// Known residual hazard, deliberately not changed here: under `WalCheckpointing.external` this + /// checkpoint restarts the WAL underneath a page replicator and costs its next push a full + /// snapshot of the database (see `WhoopStore.writeConsistentCopy(to:)` for the measurement, and + /// `CloudSyncUploader.defaultExporter` for the pattern that avoids it). It is left alone because + /// its callers are the *user-initiated* file-level backups — the Export button, and + /// `BackupSync.catchUpIfDue`, which is gated on the auto-backup toggle AND a chosen folder AND a + /// full day elapsed. Those want the disk reclaimed, they are at most daily, and swapping them to + /// a staged full copy would make a manual export visibly slower for everyone to protect a trial + /// that only one device is running. Revisit if page replication becomes the default. func checkpointForBackup() async -> Bool { guard let store else { return false } do { try await store.checkpointWAL(); return true } catch { return false } diff --git a/StrandTests/LitersRoundTripTests.swift b/StrandTests/LitersRoundTripTests.swift index 5bf8d47d71..b4a57e0b5b 100644 --- a/StrandTests/LitersRoundTripTests.swift +++ b/StrandTests/LitersRoundTripTests.swift @@ -7,6 +7,8 @@ #if CLOUD_SYNC && LITERS import XCTest import SQLite3 +import WhoopProtocol +import WhoopStore @testable import Strand /// End-to-end proof that the Rust archive in `Liters.xcframework` actually replicates a database, @@ -100,6 +102,96 @@ final class LitersRoundTripTests: XCTestCase { XCTAssertGreaterThan(second.txid, first.txid) } + // MARK: - The `/ingest` fallback, which shares the file with the replicator + + /// **The regression this group exists for**, and the measurement that decided its shape. + /// + /// `CloudSyncUploader`'s `/ingest` fallback used to checkpoint the live store before archiving + /// it. That checkpoint runs on exactly the syncs where the page-replication push did not — a push + /// that failed to open a writer at all, or a launch where the app foregrounds and backs up before + /// it syncs — so it lands in the one window where no replicator read lock is holding the WAL. + /// + /// Both modes are asserted because the gentler one is the trap. `FULL` leaves the `-wal` file at + /// its full size, which looks harmless and is not: once the WAL is fully backfilled and no reader + /// needs those frames, SQLite restarts it on the **next write transaction**, new salt and all. + /// The replicator's resume offset is just as gone as after a `TRUNCATE`. + /// + /// This is VK's device failure reproduced: 2026-07-28, `/ingest` at 06:03, then a push at 14:23 + /// reporting `wal truncated by another process` and uploading 640 MB. + func testAnyForeignCheckpointCostsTheNextPushASnapshot() throws { + for mode in ["TRUNCATE", "FULL"] { + let scratch = dir.appendingPathComponent("ckpt-\(mode)", isDirectory: true) + try FileManager.default.createDirectory(at: scratch, withIntermediateDirectories: true) + let db = scratch.appendingPathComponent("origin.sqlite").path + let bucket = scratch.appendingPathComponent("bucket", isDirectory: true).path + + try makeDatabase(at: db, rows: 1...200) + let first = try LitersWriter(dbPath: db, storage: .dir(path: bucket)) + _ = try first.push() + try appendRows(201...210, to: db) + // Closed, because that is the window the fallback's checkpoint actually runs in. With the + // writer open its read lock blocks a foreign checkpoint outright — which protects the + // replicator but leaves the archive missing rows, so it is not a fix either. + first.close() + + try foreignCheckpoint(mode, at: db) + // The next write is what converts a backfilled WAL into a restarted one. + try appendRows(211...220, to: db) + + let second = try LitersWriter(dbPath: db, storage: .dir(path: bucket)) + defer { second.close() } + let push = try second.push() + + XCTAssertTrue(push.snapshotted, + "\(mode): a completed foreign checkpoint must cost the replicator its resume " + + "point — if this stops being true, liters changed and the staged-copy export " + + "may no longer be needed") + print("[liters] foreign \(mode): snapshotted=\(push.snapshotted) " + + "reason=\(push.snapshotReason ?? "nil") bytes=\(push.bytesUploaded)") + } + } + + /// **The fix, end to end.** The identical sequence, with the export staged through + /// `WhoopStore.writeConsistentCopy(to:)` — SQLite's Online Backup API, which reads through the + /// WAL and never checkpoints — instead of a checkpoint. The next push ships a delta. + /// + /// Through a real `WhoopStore` rather than raw `SQLite3` on purpose: GRDB's `DatabasePool` is + /// what actually holds this file in the app, it links the same platform `libsqlite3` liters does + /// (see the type doc), and the copy has to survive a pool's connections, not a single handle's. + func testAStagedConsistentCopyDoesNotCostTheNextPushASnapshot() async throws { + let db = dir.appendingPathComponent("store.sqlite").path + let store = try await WhoopStore(path: db, walCheckpointing: .external, walBackstop: .disabled) + try await insertHR(store, count: 20) + + let first = try LitersWriter(dbPath: db, storage: .dir(path: bucketPath)) + let firstPush = try first.push() + XCTAssertTrue(firstPush.synced) + try await insertHR(store, count: 2, startingAt: 20 * 400) + first.close() + + // The export step, in the same window the checkpoint above ran in. + let staged = dir.appendingPathComponent("staged.sqlite").path + try await store.writeConsistentCopy(to: staged) + try await insertHR(store, count: 2, startingAt: 22 * 400) + + let second = try LitersWriter(dbPath: db, storage: .dir(path: bucketPath)) + defer { second.close() } + let push = try second.push() + + XCTAssertFalse(push.snapshotted, + "staging a copy must leave the replicator's resume point intact. reason: " + + "\(push.snapshotReason ?? "nil")") + XCTAssertGreaterThan(push.txid, firstPush.txid) + // The size claim, against the only honest yardstick available: the FIRST push, which was a + // snapshot of this same database. Comparing against the `.sqlite` file's size would be + // meaningless here — on a replicated store nothing checkpoints, so almost the whole database + // lives in the `-wal` and the main file stays near empty. + XCTAssertLessThan(push.bytesUploaded, firstPush.bytesUploaded / 2, + "a delta must be a fraction of the snapshot it follows") + print("[liters] staged copy: snapshotted=\(push.snapshotted) bytes=\(push.bytesUploaded) " + + "vs first-push snapshot \(firstPush.bytesUploaded)B") + } + // MARK: - The iOS-shaped failure mode /// Close the writer, reopen it, push. This is what iOS does to the app on every relaunch, and the @@ -191,11 +283,15 @@ final class LitersRoundTripTests: XCTestCase { /// Creates a WAL-mode database and inserts `rows`, through Apple's system SQLite. private func makeOriginDatabase(rows: ClosedRange) throws { + try makeDatabase(at: originPath, rows: rows) + } + + private func makeDatabase(at path: String, rows: ClosedRange) throws { var db: OpaquePointer? - let rc = sqlite3_open_v2(originPath, &db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, nil) + let rc = sqlite3_open_v2(path, &db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, nil) defer { sqlite3_close(db) } guard rc == SQLITE_OK, db != nil else { - throw SQLiteFailure(code: rc, message: "open \(originPath)") + throw SQLiteFailure(code: rc, message: "open \(path)") } // WAL is not incidental. liters replicates WAL frames, so a rollback-journal database would @@ -206,6 +302,30 @@ final class LitersRoundTripTests: XCTestCase { try insert(db, rows) } + /// `PRAGMA wal_checkpoint()` from a SECOND connection, which is what + /// `WhoopStore.checkpointWALForBackup()` is from liters' point of view: a foreign checkpointer on + /// the same file, while the writer holds its read lock. Returns SQLite's own three-integer answer + /// — `busy` is 1 when a reader stopped the checkpoint before it finished, and `log`/`checkpointed` + /// are frames present versus frames transferred into the database file. + @discardableResult + private func foreignCheckpoint(_ mode: String, at path: String) throws + -> (busy: Int, log: Int, checkpointed: Int) { + var db: OpaquePointer? + let rc = sqlite3_open_v2(path, &db, SQLITE_OPEN_READWRITE, nil) + defer { sqlite3_close(db) } + guard rc == SQLITE_OK, db != nil else { throw SQLiteFailure(code: rc, message: "reopen \(path)") } + + var stmt: OpaquePointer? + let sql = "PRAGMA wal_checkpoint(\(mode));" + let prepared = sqlite3_prepare_v2(db, sql, -1, &stmt, nil) + guard prepared == SQLITE_OK else { throw SQLiteFailure(code: prepared, message: sql) } + defer { sqlite3_finalize(stmt) } + guard sqlite3_step(stmt) == SQLITE_ROW else { throw SQLiteFailure(code: -1, message: "no row from \(sql)") } + return (Int(sqlite3_column_int(stmt, 0)), + Int(sqlite3_column_int(stmt, 1)), + Int(sqlite3_column_int(stmt, 2))) + } + /// Runs a statement that returns one text column. `PRAGMA journal_mode = WAL` is such a /// statement — it reports the mode actually in force, which is the only way to know the switch /// took (it silently stays in rollback mode inside a transaction, for instance). @@ -243,6 +363,18 @@ final class LitersRoundTripTests: XCTestCase { } /// Reads back every id, ascending. The assertion target for "did the bytes arrive". + /// One `WhoopStore` transaction per call, mirroring a `Collector` flush — the commit shape whose + /// *count*, not payload, is what fills the WAL. + private func insertHR(_ store: WhoopStore, count: Int, rowsEach: Int = 400, + startingAt ts0: Int = 0) async throws { + var ts = ts0 + for _ in 0.. [Int] { var db: OpaquePointer? // READWRITE, not READONLY, and that is not arbitrary: Apple's libsqlite3 cannot run a From b53418f0129435563972a99d40b11cb09c68cdcb Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 28 Jul 2026 11:04:53 -0400 Subject: [PATCH 149/171] Stop HrBroadcaster stacking a live-HR subscription on every screen visit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `bind(to:)`'s only call site is `DataSourcesView.onAppear`, and `onAppear` fires on EVERY appearance — every tab switch back to Data Sources, every push-and-pop — while the broadcaster is a `@StateObject` that outlives all of them. Each call appended another sink to `cancellables` and nothing ever removed one, so a user who had visited the screen N times sent N duplicate 0x2A37 Heart Rate Measurement notifications per heartbeat to every subscribed central. The call site's own comment says "bind ... once", which is what it intended and not what `onAppear` does. Fixed in `bind(to:)` rather than at the call site so it holds for any future caller: the previous subscription is cancelled before the new one is stored, making the method idempotent. Re-binding to a DIFFERENT LiveState also replaces rather than adds, so an old screen's state stops feeding the broadcaster instead of racing the new one. `cancellables` becomes `private(set)` (internal get) so the regression test can pin the count — the leak is invisible from the outside, so the number of sinks is the only observable that can carry it. Verification: `xcodebuild ... test -only-testing:StrandTests/HrBroadcasterEncodeTests` — 9 tests, 0 failures (2 new). Confirmed the new test FAILS without the fix ("11" is not equal to "1"), so it genuinely covers the bug rather than passing vacuously. macOS `Strand` builds. Android has no twin: its `HrBroadcaster` has no `bind` — the Compose screen pushes samples in directly. --- Strand/BLE/HrBroadcaster.swift | 16 +++++++++++-- StrandTests/HrBroadcasterEncodeTests.swift | 26 ++++++++++++++++++++++ 2 files changed, 40 insertions(+), 2 deletions(-) diff --git a/Strand/BLE/HrBroadcaster.swift b/Strand/BLE/HrBroadcaster.swift index 1116edc114..ccf5a68cc6 100644 --- a/Strand/BLE/HrBroadcaster.swift +++ b/Strand/BLE/HrBroadcaster.swift @@ -60,7 +60,10 @@ public final class HrBroadcaster: NSObject, ObservableObject { /// first sample of a session; cleared on stop so a stale bpm can't outlive the broadcast. private var lastBpm: Int? - private var cancellables = Set() + /// `private(set)` rather than `private` so `HrBroadcasterEncodeTests` can assert that ``bind(to:)`` + /// leaves exactly one subscription however many times it is called — the leak it fixes is invisible + /// from the outside, so the count is the only thing a test can pin. + private(set) var cancellables = Set() /// Diagnostic sink for the broadcast lifecycle, wired (when the composition root chooses to) to the /// SAME exportable strap log the WHOOP path uses, so a tester whose gym kit can't see NOOP has a @@ -110,10 +113,19 @@ public final class HrBroadcaster: NSObject, ObservableObject { } /// Bind to a `LiveState` so every live HR change is broadcast automatically. Optional convenience the - /// app's composition root can call once; the broadcaster works equally well by polling + /// app's composition root can call; the broadcaster works equally well by polling /// ``update(heartRate:)`` directly. Observing `$heartRate` keeps this a pure CONSUMER of the existing /// live value — it never drives or mutates the WHOOP/central path. + /// + /// **Idempotent.** The previous subscription is cancelled before the new one is stored, so calling this + /// twice leaves exactly one sink. That is not hypothetical tidiness: the only call site is + /// `DataSourcesView.onAppear`, and `onAppear` fires on EVERY appearance — every tab switch back to Data + /// Sources, every push-and-pop — while the broadcaster itself is a `@StateObject` that outlives all of + /// them. Without this reset the sinks accumulated one per visit, and each one called + /// ``update(heartRate:)`` for the same sample, so a user who had visited the screen N times sent N + /// duplicate 0x2A37 notifications per heartbeat to every subscribed central. public func bind(to live: LiveState) { + cancellables.removeAll() live.$heartRate .removeDuplicates() .sink { [weak self] hr in self?.update(heartRate: hr) } diff --git a/StrandTests/HrBroadcasterEncodeTests.swift b/StrandTests/HrBroadcasterEncodeTests.swift index 1ad67b4b9f..1e37221cf2 100644 --- a/StrandTests/HrBroadcasterEncodeTests.swift +++ b/StrandTests/HrBroadcasterEncodeTests.swift @@ -53,6 +53,32 @@ final class HrBroadcasterEncodeTests: XCTestCase { XCTAssertEqual(HrBroadcaster.measurement(bpm: 1_000_000), [0x01, 0xFF, 0xFF]) } + // MARK: - bind(to:) is idempotent (the duplicate-notification leak) + + /// `bind(to:)`'s only call site is `DataSourcesView.onAppear`, which fires on EVERY appearance — every + /// tab switch back to Data Sources — while the broadcaster is a `@StateObject` that outlives all of + /// them. Each bind used to append another sink, so a user who had visited the screen N times sent N + /// duplicate 0x2A37 notifications per heartbeat. Binding must therefore be idempotent. + func testRepeatedBindLeavesExactlyOneSubscription() { + let broadcaster = HrBroadcaster(log: { _ in }) + let live = LiveState() + XCTAssertEqual(broadcaster.cancellables.count, 0, "nothing subscribed before the first bind") + broadcaster.bind(to: live) + XCTAssertEqual(broadcaster.cancellables.count, 1) + for _ in 0..<10 { broadcaster.bind(to: live) } + XCTAssertEqual(broadcaster.cancellables.count, 1, + "eleven binds must leave ONE sink, not eleven — see DataSourcesView.onAppear") + } + + /// Re-binding to a DIFFERENT LiveState must also leave one sink: the old screen's state should stop + /// feeding the broadcaster entirely, not keep a second stream alive alongside the new one. + func testRebindingToAnotherLiveStateReplacesTheSubscription() { + let broadcaster = HrBroadcaster(log: { _ in }) + broadcaster.bind(to: LiveState()) + broadcaster.bind(to: LiveState()) + XCTAssertEqual(broadcaster.cancellables.count, 1) + } + // MARK: - Round-trip against the parser (the encoder is the parser's exact inverse) func testRoundTripThroughParser() { From 89fa3d3c414d460cee335418d73ce334e0d5c6bb Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 28 Jul 2026 11:05:06 -0400 Subject: [PATCH 150/171] Say on the v18AuxSample migration that its unread reader is by design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ppgWaveformSample`'s migration (v27) carries an explicit CONSUMER STATUS note ending "Do NOT 'clean up' the reader as dead code: the rows are the point, and the reader is how they are reachable". `v18AuxSample` (v31 / Room MIGRATION_24_25) is in exactly the same position — every `v18AuxSamples` call site on both platforms is a test — and carried no such note. A future tidy-up pass reading it cold would delete a reader that is deliberately unused, and with it the only way the banked rows are reachable. Adds the equivalent note on both platforms, and records the part that is easy to lose: before this migration those fields were not merely unread, they were DESTROYED. The strap trims its history the moment an offload is acked, so each one was unrecoverable and could never be censused. The migration converts permanent loss into retained-but-unread, which is the whole fix and is complete. Fifteen of the slots are unpinned bytes whose names deliberately assert nothing, so wiring them to a consumer before a census would be the overclaiming this project has already had to retract. Also names the four sibling columns the same migration added (`gravitySample.dynAccel`, `sleepStateSample.rawByte`, `skinTempSample.aux1Raw/aux2Raw`), which are SELECTed into their structs with no consumer touching the properties, on purpose. Comments only — no schema, no behaviour, no stored value changes. Verification: `swift build` (WhoopStore) clean; macOS `Strand` builds; Android `assembleFullDebug` + `testFullDebugUnitTest` — 3209 tests, 0 failures, counted from the JUnit XML. --- .../WhoopStore/Sources/WhoopStore/Database.swift | 16 ++++++++++++++++ .../app/src/main/java/com/noop/data/Entities.kt | 15 +++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/Packages/WhoopStore/Sources/WhoopStore/Database.swift b/Packages/WhoopStore/Sources/WhoopStore/Database.swift index 64e0f80068..9b30f625c7 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/Database.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/Database.swift @@ -655,6 +655,22 @@ extension WhoopStore { // point of this migration is that the data starts accruing NOW so a validated consumer is possible // LATER; landing a consumer at the same time would be scoring on evidence that does not exist yet. // + // CONSUMER STATUS — deliberately none, stated here so nobody has to re-derive it, and in the same + // shape as the `ppgWaveformSample` note above (v27). The writers are live on both platforms, but + // every `v18AuxSamples` call site on BOTH platforms is a TEST: no analytic, no score, no gate, no + // UI, no export reads a row. **Do NOT "clean up" the reader as dead code** — the rows are the point, + // and the reader is how they become reachable once a consumer is validated. The same applies to the + // four named columns added just below (`gravitySample.dynAccel`, `sleepStateSample.rawByte`, + // `skinTempSample.aux1Raw/aux2Raw`): they are SELECTed into their structs, and no consumer touches + // the properties, on purpose. + // + // Why the rows still matter even unread: before this migration these fields were not merely unread, + // they were DESTROYED. The strap trims its history the moment an offload is acked, so every one of + // them was unrecoverable. This migration converts permanent loss into retained-but-unread, which is + // the whole fix and is complete. Fifteen of the slots are unpinned bytes whose names deliberately + // assert nothing; wiring them to anything before a census would be exactly the overclaiming this + // project has already had to retract. The capture IS the deliverable. + // // The remaining fifteen v18 slots go to their OWN narrow table rather than fifteen more columns // (see `V18AuxCodec` for the wire format and the column-vs-blob tradeoff). Three reasons this is // a table and not another column on an existing row: diff --git a/android/app/src/main/java/com/noop/data/Entities.kt b/android/app/src/main/java/com/noop/data/Entities.kt index 43cb7d8db6..698031132c 100644 --- a/android/app/src/main/java/com/noop/data/Entities.kt +++ b/android/app/src/main/java/com/noop/data/Entities.kt @@ -651,6 +651,21 @@ data class LiveSessionRow( * existing per-second tables widen rows that were already being written. * * INSTRUMENTATION ONLY: nothing reads these rows. + * + * CONSUMER STATUS — deliberately none, stated here so nobody has to re-derive it. The writer is live, but + * every `v18AuxSamples` call site on BOTH platforms is a TEST: no analytic, no score, no gate, no UI, no + * export reads a row. **Do NOT "clean up" the reader as dead code** — the rows are the point, and the + * reader is how they become reachable once a consumer is validated. The same applies to the four named + * columns v31/MIGRATION_24_25 added alongside this table (`gravitySample.dynAccel`, + * `sleepStateSample.rawByte`, `skinTempSample.aux1Raw/aux2Raw`): they are read into their entities and no + * consumer touches the properties, on purpose. + * + * Why the rows still matter unread: before this migration these fields were not merely unread, they were + * DESTROYED — the strap trims its history the moment an offload is acked, so each one was unrecoverable. + * This converts permanent loss into retained-but-unread, which is the whole fix and is complete. Fifteen + * of the slots are unpinned bytes whose names deliberately assert nothing; wiring them to anything before + * a census would be exactly the overclaiming this project has already had to retract. The capture IS the + * deliverable. Twin of the Swift `v31-deep-capture-channels` migration note in `Database.swift`. */ @Entity(tableName = "v18AuxSample", primaryKeys = ["deviceId", "ts"]) data class V18AuxSampleEntity( From 529261a3c6593669cbce662d353197084fda7f1f Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 28 Jul 2026 12:40:04 -0400 Subject: [PATCH 151/171] Measure the REM-latency guard in minutes since onset, not as a fraction of the session (#930) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `SleepStagerV2.cyclePrior` suppressed REM while `c < 0.12` — 12 % of THAT session's length. First-REM latency is an absolute physiological interval, so the guard's width scaled with how long the wearer stayed in bed: 7.4–84.5 min, an 11.5x spread, across one WHOOP 5 user's own 36 recorded nights. Replaces the step with a graded penalty in minutes since a MEASURED sleep onset: rem = 1.0 * c - K * clamp01(1 - m / M0) K = 3.0, M0 = 60 min K is the incumbent step's own magnitude (e^-3 ~ 0.05 on the REM emission, strong suppression but never a veto). The sleep-accel grid — {cliff, graded} x {fraction, minutes} x K in 1..8 x threshold, 210 cells — ties on accuracy in every cell, so nothing discriminates K, and the smallest correct change fixes the units and the shape without also retuning a magnitude no measurement can justify. M0 = 60 sits inside the measured-defensible band: against PSG truth a guard reaching 45 min costs 0.36 % of all real REM (1 of 31 subjects) and one reaching 90 min costs 6.85 % (15 of 31). The `1.0 * c` ramp and the deep term are untouched and stay fractions of the session — both describe where in the night you are, which is inherently proportional. Removing the ramp collapses REM to 1.4 % of night and kappa to 0.143; it is load-bearing. THIS IS A ROBUSTNESS FIX, NOT AN ACCURACY WIN. On sleep-accel the step is measurably inert (kappa 0.349 -> 0.349, median first-REM latency 142.0 -> 141.0 min when removed) because that cohort spans only 2.4x in session length. Replayed over the 36-night device database (15 stage-locked human-authored references, 9163 epochs): 4-class kappa 0.691 -> 0.689, sleep/wake 0.583 -> 0.582, accuracy 78.2 % both. kappa is not the guardrail that matters here — #437 reverted #348 for exactly that reason ("kappa doesn't guard stage-fraction calibration"), so stage fractions are reported too. Per-night bias vs the human reference (predicted - truth, pp) improves on three stages and is unchanged on deep: wake -7.26 -> -7.13, light -7.54 -> -6.10, deep +6.65 -> +6.65, rem +8.15 -> +6.57. On the healthy stratum #348 broke (in-bed >= 5 h, n = 20) nothing moves more than 0.42 pp and wake moves +0.03 pp; deep is byte-identical on all 36 nights. 12 of 36 nights change at all, and the change is monotone in session length — mean REM delta -20.9 pp under 2 h, -2.4 pp at 2-5 h, -0.4 pp at >= 5 h — i.e. it lands on precisely the short sessions the fractional guard was mis-scaling. Plumbing: `Epoch` gains `minutesSinceOnset`; `stageEpochs` stages once with the guard disabled, takes the first sustained non-wake run as onset (5 min = 10 epochs; measured against PSG onset at bias -3.8 min, MAE 7.4 min, n = 31), then re-applies the guard and re-runs Viterbi. Only the guard differs between passes, so the cost is one extra Viterbi, not a second featurisation. The guard is clamped to [0, K] so a pre-onset epoch cannot be penalised harder than onset itself — #271 can place a window start hours early. Android twin updated in lockstep per the parity contract, including the constants and the frozen golden. Tests: Swift 1247 passed / 0 failed; Android 3238 passed / 0 failed / 5 skipped (393 suites, counts read from the JUnit XML). Both frozen goldens are unchanged, so the recipe's pinned end-to-end shape is preserved. Known and separately tracked, NOT addressed here: median first-REM latency runs ~54 min late against PSG (142 vs 88.5 min truth; removing the step gives 141, so this term is not the cause), and deep is under-called 2-3x (4.7-6.2 % predicted vs 14.8 % truth). Refs #930. --- .../StrandAnalytics/SleepStagerV2.swift | 103 ++++++++++++++- .../MotionCorroboratedWakeTests.swift | 4 +- .../SleepStagerV2Tests.swift | 104 ++++++++++++++- .../java/com/noop/analytics/SleepStagerV2.kt | 109 ++++++++++++++-- .../analytics/MotionCorroboratedWakeTest.kt | 4 +- .../com/noop/analytics/SleepStagerV2Test.kt | 121 ++++++++++++++++++ 6 files changed, 422 insertions(+), 23 deletions(-) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift index ee5e52fdf3..160d3a0d41 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift @@ -21,7 +21,8 @@ import WhoopProtocol // Cole–Kripke actigraphy grid, V2 stages each 30 s epoch from: // 1. per-night z-scored cardiorespiratory emissions (HR / HR-variability / movement); // 2. a per-night DEEP gate on the 11-min HR-flatness percentile (the strongest deep-vs-light separator); -// 3. a soft sleep-cycle prior (deep concentrated early; REM suppressed in the first ~12% then rising); +// 3. a soft sleep-cycle prior (deep concentrated early as a fraction of the session; REM rising with that +// same fraction, minus a REM-latency guard that decays over the first 60 MINUTES after sleep onset); // 4. a peak-motion (jerk) wake gate, thresholded RELATIVE to the night's own quiescent jerk floor — so // it self-calibrates to the strap's gravity-decode scale and the wearer's fit, not a fixed g; // 5. an RR-RSA respiration-regularity term (regular breathing → deep, irregular → REM); @@ -193,6 +194,13 @@ public enum SleepStagerV2 { let respReg: Double? // RSA spectral peakedness in the 0.15–0.40 Hz band (breathing regularity) let clock: Double // time-of-night fraction in [0, 1] let jerkScale: Double // night quiescent jerk floor (median per-second jerk over the session) + /// Elapsed MINUTES from the session window start to this epoch's centre — the minute-domain twin of + /// `clock` (#930). `features()` cannot know where sleep actually began, so it fills this relative to + /// the window start; `stageEpochs` re-bases it onto the sleep onset its own first pass found, and + /// falls back to this window-relative value when a night never sustains sleep. Only the REM-latency + /// guard reads it — the deep term and the REM ramp stay fractions of the session, which is what they + /// are (see `cyclePrior`). + let minutesSinceOnset: Double } // MARK: - Feature extraction @@ -272,6 +280,7 @@ public enum SleepStagerV2 { struct Raw { let start: Int; let hr: Double?; let hrVar: Double?; let hrFlat11: Double? let jerks: [Double]; let gapSec: Int; let jerkMax: Double; let respReg: Double?; let clock: Double + let minutes: Double } var raws: [Raw] = [] var allJerks: [Double] = [] @@ -310,7 +319,8 @@ public enum SleepStagerV2 { raws.append(Raw(start: e, hr: hrMean, hrVar: hrVar, hrFlat11: hrFlat11, jerks: jerks, gapSec: max(1, gseq.count - 1), jerkMax: jerkMax, - respReg: respReg, clock: Double(e + 15 - start) / span)) + respReg: respReg, clock: Double(e + 15 - start) / span, + minutes: Double(e + 15 - start) / 60.0)) e += 30 } @@ -333,7 +343,7 @@ public enum SleepStagerV2 { feats.append(Epoch( start: r.start, hr: r.hr, hrVar: r.hrVar, hrFlat11: r.hrFlat11, moveFrac: Double(moves) / Double(r.gapSec), jerkMax: r.jerkMax, respReg: r.respReg, - clock: r.clock, jerkScale: jerkScale)) + clock: r.clock, jerkScale: jerkScale, minutesSinceOnset: r.minutes)) } return feats } @@ -380,14 +390,72 @@ public enum SleepStagerV2 { // MARK: - Recipe staging + // ── the REM-latency guard (#930) ────────────────────────────────────────────────────────────────── + + /// Log-odds penalty applied to the REM emission AT sleep onset. Unchanged in magnitude from the + /// `c < 0.12 ? 3.0 : 0.0` step this replaced: e⁻³ ≈ 0.05 on the REM emission — strong suppression that + /// sufficient evidence still overcomes, never a veto. Kept at the incumbent value deliberately, because + /// the sleep-accel grid ({cliff, graded} × {fraction, minutes} × K ∈ 1…8 × threshold, 210 cells) showed + /// every cell tying on accuracy, so nothing in the data discriminates K — and the smallest correct change + /// is to fix the UNITS and the SHAPE without also retuning a magnitude no measurement can justify. + static let remLatencyPenalty = 3.0 + + /// Minutes over which `remLatencyPenalty` decays linearly to zero, measured from sleep ONSET. + /// Against PSG truth on sleep-accel (n = 31 subjects), a guard reaching 45 min costs 0.36 % of all real + /// REM (1 subject affected) and one reaching 90 min costs 6.85 % (15 subjects); 60 min sits inside that + /// band, and because the penalty is GRADED rather than a cliff it is already down to ≤ 0.75 log-odds + /// (×0.47, vs ×0.05 at onset) across 45–60 min, so a genuine sleep-onset REM period late in the window is + /// held back rather than suppressed. 60 min also stops clear of the ~70–100 min population first-REM + /// latency, so the ramp is fully spent before real REM is expected. + static let remLatencyMinutes = 60.0 + + /// The guard itself: `K` at (and before) sleep onset, decaying linearly to 0 at `M0` minutes after it. + /// Clamped to `[0, K]` so a PRE-onset epoch — negative elapsed time, and unbounded when detection places + /// the window start hours early (#271) — can never be penalised harder than the onset instant itself. + static func remLatencyGuard(_ minutesSinceOnset: Double) -> Double { + remLatencyPenalty * min(1.0, max(0.0, 1.0 - minutesSinceOnset / remLatencyMinutes)) + } + /// Soft sleep-cycle prior added to the log-emission: deep concentrated early (decays, never hard-wiped); - /// REM suppressed in the first ~12 % (REM latency) then rising toward morning. - static func cyclePrior(_ c: Double) -> [String: Double] { + /// REM suppressed around sleep onset (REM latency) then rising toward morning. + /// + /// PROVENANCE of the constants: `1.2`, `0.55` and the `1.0` REM slope are hand-picked a priori from sleep + /// physiology and population base rates, never fit to labels — as is every other coefficient in this file. + /// `remLatencyPenalty` / `remLatencyMinutes` carry their own derivations above. + /// + /// UNITS, and why they differ per term (#930). The deep term and the `1.0 * c` REM ramp take `c`, the + /// fraction of the session — correctly, because both describe *where in the night* you are, a quantity + /// that is inherently proportional. The REM-LATENCY guard takes `minutesSinceOnset` instead, because + /// first-REM latency is an absolute physiological interval and not a proportion of how long you slept: on + /// sleep-accel (n = 30 subjects with a scorable first REM period) true latency is uncorrelated with + /// session duration (Pearson r = −0.058, Spearman −0.076), and re-expressing latency as a fraction makes + /// it MORE variable, not less (CV 0.617 as a fraction vs 0.537 in minutes). The `c < 0.12` step this + /// replaced therefore scaled a fixed physiological interval by session length: across one WHOOP 5 user's + /// own recorded nights it ranged 7.4–84.5 min, an 11× spread, for the same wearer and the same physiology. + static func cyclePrior(_ c: Double, _ minutesSinceOnset: Double) -> [String: Double] { ["deep": 1.2 * max(0.0, 1.0 - c / 0.55), - "rem": 1.0 * c - (c < 0.12 ? 3.0 : 0.0), + "rem": 1.0 * c - remLatencyGuard(minutesSinceOnset), "light": 0.0, "awake": 0.0] } + /// Sustained non-wake run, in 30 s epochs, that establishes sleep onset — 10 epochs = 5 minutes. + /// Measured against PSG onset on sleep-accel (n = 31 subjects): bias −3.8 min, MAE 7.4 min. + static let onsetSustainedEpochs = 10 + + /// Index of the first epoch that begins a run of `onsetSustainedEpochs` consecutive non-"awake" labels, + /// or nil when the hypnogram never sustains sleep that long (a nap shorter than the rule, or an all-wake + /// window). Runs are counted in epochs, not wall clock, so a coverage gap that drops an epoch cannot + /// silently satisfy the rule with less evidence. + static func sustainedSleepOnset(_ labels: [String]) -> Int? { + var run = 0 + for i in labels.indices { + if labels[i] == "awake" { run = 0; continue } + run += 1 + if run >= onsetSustainedEpochs { return i - onsetSustainedEpochs + 1 } + } + return nil + } + /// Viterbi most-likely path over the per-epoch log-emissions with the sticky transition matrix and a /// uniform start. Ties resolve to the earlier stage in `stageNames`. static func viterbi(_ emSeq: [[String: Double]]) -> [String] { @@ -420,6 +488,15 @@ public enum SleepStagerV2 { /// Run the full recipe over a night's epochs and return one stage label per epoch (incl. "awake"). /// All normalisation (z-scores, the HR-flatness percentile) is WITHIN the night. + /// + /// TWO PASSES (#930). The REM-latency guard is measured from sleep ONSET, and onset is itself a staging + /// output, so the recipe cannot know it while building the emissions. Pass 1 stages the night with the + /// guard DISABLED (`cyclePrior(c, .infinity)`) and reads the first sustained sleep run out of the result; + /// pass 2 adds the guard, re-based on that onset, and re-runs Viterbi. Only the guard differs between the + /// passes — every emission term, z-score and percentile is computed ONCE and reused — so the extra cost is + /// one Viterbi over an already-built lattice, not a second featurisation, and `stageSession` memoizes the + /// whole thing anyway. When no sustained run exists the origin falls back to the window start, which is + /// exactly the origin the shipped `c`-based guard used. static func stageEpochs(_ feats: [Epoch]) -> [String] { if feats.isEmpty { return [] } @@ -464,12 +541,24 @@ public enum SleepStagerV2 { "light": baseLogPrior["light"]!, "awake": 1.0 * zmvv + awakeCardiac + baseLogPrior["awake"]!, ] - let pr = cyclePrior(f.clock) + // Guard DISABLED here (`.infinity` ⇒ `remLatencyGuard` = 0); pass 2 below adds it once onset is known. + let pr = cyclePrior(f.clock, .infinity) for s in stageNames { em[s]! += pr[s]! } if f.jerkMax > f.jerkScale * jerkFloorGateMult { em["awake"]! += motionGateBoost } if let rg = f.respReg { let z = zrg(rg); em["deep"]! += respWeight * z; em["rem"]! -= respWeight * z } seq.append(em) } + + // PASS 1 — stage without the REM-latency guard purely to locate sleep onset. + let provisional = viterbi(seq) + // Origin for the guard: the centre of the first sustained-sleep epoch, in the same window-relative + // minutes `features()` stamped on every epoch. No sustained run → 0.0, i.e. the window start. + let originMin = sustainedSleepOnset(provisional).map { feats[$0].minutesSinceOnset } ?? 0.0 + + // PASS 2 — apply the guard against minutes since THAT onset and re-run the lattice. + for i in feats.indices { + seq[i]["rem"]! -= remLatencyGuard(feats[i].minutesSinceOnset - originMin) + } return viterbi(seq) } } diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/MotionCorroboratedWakeTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/MotionCorroboratedWakeTests.swift index bb1a3fbe36..3d8b7ec3e2 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/MotionCorroboratedWakeTests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/MotionCorroboratedWakeTests.swift @@ -133,10 +133,10 @@ final class MotionCorroboratedWakeTests: XCTestCase { func testMotionQuiescentPredicate() { // A still epoch (no move, jerk at floor) is quiescent; an epoch with a jerk spike above the gate is not. let still = SleepStagerV2.Epoch(start: 0, hr: 58, hrVar: 1, hrFlat11: 1, moveFrac: 0.0, jerkMax: 0.001, - respReg: nil, clock: 0.5, jerkScale: 0.001) + respReg: nil, clock: 0.5, jerkScale: 0.001, minutesSinceOnset: 120) XCTAssertTrue(SleepStagerV2.motionQuiescent(still)) let moved = SleepStagerV2.Epoch(start: 0, hr: 58, hrVar: 1, hrFlat11: 1, moveFrac: 0.3, jerkMax: 0.2, - respReg: nil, clock: 0.5, jerkScale: 0.001) + respReg: nil, clock: 0.5, jerkScale: 0.001, minutesSinceOnset: 120) XCTAssertFalse(SleepStagerV2.motionQuiescent(moved)) } diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerV2Tests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerV2Tests.swift index b4ec048ccc..83e4ba6c6c 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerV2Tests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerV2Tests.swift @@ -93,16 +93,114 @@ final class SleepStagerV2Tests: XCTestCase { // MARK: - recipe invariants - /// The cycle prior concentrates deep early in the night and suppresses REM in the first ~12 %. + /// The cycle prior concentrates deep early in the night and suppresses REM near sleep onset. func testCyclePriorShapesDeepAndRem() { - let early = SleepStagerV2.cyclePrior(0.05) - let late = SleepStagerV2.cyclePrior(0.90) + let early = SleepStagerV2.cyclePrior(0.05, 3.0) + let late = SleepStagerV2.cyclePrior(0.90, 400.0) XCTAssertGreaterThan(early["deep"]!, late["deep"]!, "deep prior is higher early in the night") XCTAssertLessThan(early["rem"]!, late["rem"]!, "REM prior is suppressed early, rising toward morning") XCTAssertEqual(early["light"]!, 0.0) XCTAssertEqual(early["awake"]!, 0.0) } + // MARK: - #930: the REM-latency guard is measured in MINUTES, not as a fraction of the session + + /// THE point of #930, and the assertion the previous shape could not even express: a 10-hour night and a + /// 3-hour fragment must receive the SAME REM-latency guard at the same minute after sleep onset. + /// + /// Under the replaced `c < 0.12` step the two disagree by construction — 30 min into a 10 h night is + /// `c = 0.05` (suppressed by the full 3.0) while 30 min into a 3 h fragment is `c = 0.167` (not suppressed + /// at all), so the identical physiological instant got opposite treatment purely because of how long the + /// wearer stayed in bed. The guard is isolated from the `1.0 * c` REM ramp (which is CORRECTLY a fraction + /// of the session and legitimately differs between the two) by subtracting the ramp back out. + func testRemLatencyGuardIsIdenticalAcrossSessionLengths() { + let longNight = 10.0 * 60.0 // 600 min + let fragment = 3.0 * 60.0 // 180 min + for minutes in stride(from: 0.0, through: 180.0, by: 2.5) { + let cLong = minutes / longNight, cShort = minutes / fragment + // Strip the fraction-of-session REM ramp; what remains is the latency guard alone. + let guardLong = 1.0 * cLong - SleepStagerV2.cyclePrior(cLong, minutes)["rem"]! + let guardShort = 1.0 * cShort - SleepStagerV2.cyclePrior(cShort, minutes)["rem"]! + XCTAssertEqual(guardLong, guardShort, accuracy: 1e-12, + "guard at \(minutes) min must not depend on session length " + + "(10 h gave \(guardLong), 3 h gave \(guardShort))") + XCTAssertEqual(guardLong, SleepStagerV2.remLatencyGuard(minutes), accuracy: 1e-12) + } + } + + /// Shape of the guard: full strength at onset, linear decay, exactly zero at and past `remLatencyMinutes`, + /// and clamped (never stronger than at onset) for the pre-onset epochs a mis-placed window start produces. + func testRemLatencyGuardShapeAndClamp() { + let k = SleepStagerV2.remLatencyPenalty, m0 = SleepStagerV2.remLatencyMinutes + XCTAssertEqual(SleepStagerV2.remLatencyGuard(0.0), k, accuracy: 1e-12, "full penalty at onset") + XCTAssertEqual(SleepStagerV2.remLatencyGuard(m0 / 2), k / 2, accuracy: 1e-12, "linear halfway") + XCTAssertEqual(SleepStagerV2.remLatencyGuard(m0), 0.0, accuracy: 1e-12, "spent at M0") + XCTAssertEqual(SleepStagerV2.remLatencyGuard(m0 + 500), 0.0, accuracy: 1e-12, "never negative later") + // #271 can place the window start hours before real onset; the guard must stay bounded there. + XCTAssertEqual(SleepStagerV2.remLatencyGuard(-240.0), k, accuracy: 1e-12, "clamped pre-onset") + XCTAssertEqual(SleepStagerV2.remLatencyGuard(.infinity), 0.0, accuracy: 1e-12, + "the disabled-guard sentinel pass 1 uses must contribute nothing") + // Monotone non-increasing across the ramp. + var prev = Double.infinity + for m in stride(from: -60.0, through: 120.0, by: 1.0) { + let g = SleepStagerV2.remLatencyGuard(m) + XCTAssertLessThanOrEqual(g, prev + 1e-12, "guard must never increase with elapsed time") + prev = g + } + } + + /// The 5-minute sustained-sleep onset rule (10 epochs on the 30 s grid), measured against PSG onset on + /// sleep-accel at bias −3.8 min / MAE 7.4 min. A shorter sleep run must NOT establish onset. + func testSustainedSleepOnsetRule() { + let wake = [String](repeating: "awake", count: 6) + // 9 epochs of sleep (4.5 min) is one short of the rule and must not count. + let brief = wake + [String](repeating: "light", count: 9) + wake + XCTAssertNil(SleepStagerV2.sustainedSleepOnset(brief)) + // 10 epochs (5 min) does, and onset is the FIRST epoch of the run, not the tenth. + let sustained = wake + [String](repeating: "light", count: 10) + wake + XCTAssertEqual(SleepStagerV2.sustainedSleepOnset(sustained), 6) + // A brief run before a sustained one must not be mistaken for onset; the counter resets on wake. + let both = wake + [String](repeating: "rem", count: 4) + ["awake"] + + [String](repeating: "deep", count: 12) + XCTAssertEqual(SleepStagerV2.sustainedSleepOnset(both), 11) + XCTAssertNil(SleepStagerV2.sustainedSleepOnset([]), "an empty night has no onset") + XCTAssertNil(SleepStagerV2.sustainedSleepOnset(wake), "an all-wake window has no onset") + } + + /// End-to-end: the same physiological night truncated to a shorter in-bed window must not shift where the + /// REM guard stops applying. Both windows start at the same instant and share byte-identical streams, so + /// any difference in the first ~60 min of the hypnogram would be the session-length dependence #930 is + /// about. (Epochs after the shorter window's end are not compared — they genuinely see different + /// per-night z-scores, which is the recipe working as designed.) + func testGuardRegionDoesNotDependOnHowLongTheWearerStayedInBed() { + let start = 1_749_517_200 + let longDur = 10 * 60 * 60 + let shortDur = 3 * 60 * 60 + let grav = stillGravity(start: start, durationS: longDur) + let hr = sleepHR(start: start, durationS: longDur) + let rr = regularRR(start: start, durationS: longDur) + + func labels(_ dur: Int) -> [String] { + let segs = SleepStagerV2.stageSession(start: start, end: start + dur, + grav: grav, hr: hr, rr: rr, resp: []) + var out: [String] = [] + for t in stride(from: start, to: start + dur, by: 30) { + out.append(segs.first(where: { $0.start <= t && t < $0.end })?.stage ?? "wake") + } + return out + } + let long = labels(longDur), short = labels(shortDur) + // Over the first hour — the whole span the guard touches — neither window may call REM, because the + // guard reaches the SAME 60 minutes past onset in both. Under `c < 0.12` the 10 h window was guarded + // for 72 min and the 3 h window for only 21.6 min, so the two could not agree here by construction. + let guardEpochs = Int(SleepStagerV2.remLatencyMinutes * 60 / 30) + for i in 0.., val gapSec: Int, val jerkMax: Double, val respReg: Double?, val clock: Double, + val minutes: Double, ) val raws = ArrayList() val allJerks = ArrayList() @@ -347,7 +355,8 @@ object SleepStagerV2 { raws.add(Raw( start = e, hr = hrMean, hrVar = hrVar, hrFlat11 = hrFlat11, jerks = jerks, gapSec = maxOf(1, gseq.size - 1), jerkMax = jerkMax, - respReg = respReg, clock = (e + 15 - start).toDouble() / span)) + respReg = respReg, clock = (e + 15 - start).toDouble() / span, + minutes = (e + 15 - start).toDouble() / 60.0)) e += 30 } @@ -368,7 +377,7 @@ object SleepStagerV2 { feats.add(Epoch( start = r.start, hr = r.hr, hrVar = r.hrVar, hrFlat11 = r.hrFlat11, moveFrac = moves.toDouble() / r.gapSec, jerkMax = r.jerkMax, respReg = r.respReg, - clock = r.clock, jerkScale = jerkScale)) + clock = r.clock, jerkScale = jerkScale, minutesSinceOnset = r.minutes)) } return feats } @@ -417,13 +426,72 @@ object SleepStagerV2 { // ── Recipe staging ──────────────────────────────────────────────────────────────────────────────── + // ── the REM-latency guard (#930) ────────────────────────────────────────────────────────────────── + + /** Log-odds penalty applied to the REM emission AT sleep onset. Unchanged in magnitude from the + * `c < 0.12 ? 3.0 : 0.0` step this replaced: e⁻³ ≈ 0.05 on the REM emission — strong suppression that + * sufficient evidence still overcomes, never a veto. Kept at the incumbent value deliberately, because + * the sleep-accel grid ({cliff, graded} × {fraction, minutes} × K ∈ 1…8 × threshold, 210 cells) showed + * every cell tying on accuracy, so nothing in the data discriminates K — and the smallest correct change + * is to fix the UNITS and the SHAPE without also retuning a magnitude no measurement can justify. + * Internal so the parity test can pin it, matching the Swift `remLatencyPenalty` visibility. */ + internal const val remLatencyPenalty = 3.0 + + /** Minutes over which [remLatencyPenalty] decays linearly to zero, measured from sleep ONSET. + * Against PSG truth on sleep-accel (n = 31 subjects), a guard reaching 45 min costs 0.36 % of all real + * REM (1 subject affected) and one reaching 90 min costs 6.85 % (15 subjects); 60 min sits inside that + * band, and because the penalty is GRADED rather than a cliff it is already down to ≤ 0.75 log-odds + * (×0.47, vs ×0.05 at onset) across 45–60 min, so a genuine sleep-onset REM period late in the window is + * held back rather than suppressed. 60 min also stops clear of the ~70–100 min population first-REM + * latency, so the ramp is fully spent before real REM is expected. */ + internal const val remLatencyMinutes = 60.0 + + /** The guard itself: `K` at (and before) sleep onset, decaying linearly to 0 at `M0` minutes after it. + * Clamped to `[0, K]` so a PRE-onset epoch — negative elapsed time, and unbounded when detection places + * the window start hours early (#271) — can never be penalised harder than the onset instant itself. */ + internal fun remLatencyGuard(minutesSinceOnset: Double): Double = + remLatencyPenalty * minOf(1.0, maxOf(0.0, 1.0 - minutesSinceOnset / remLatencyMinutes)) + /** Soft sleep-cycle prior added to the log-emission: deep concentrated early (decays, never hard-wiped); - * REM suppressed in the first ~12 % (REM latency) then rising toward morning. */ - private fun cyclePrior(c: Double): Map = mapOf( + * REM suppressed around sleep onset (REM latency) then rising toward morning. + * + * PROVENANCE of the constants: `1.2`, `0.55` and the `1.0` REM slope are hand-picked a priori from sleep + * physiology and population base rates, never fit to labels — as is every other coefficient in this file. + * [remLatencyPenalty] / [remLatencyMinutes] carry their own derivations above. + * + * UNITS, and why they differ per term (#930). The deep term and the `1.0 * c` REM ramp take `c`, the + * fraction of the session — correctly, because both describe *where in the night* you are, a quantity + * that is inherently proportional. The REM-LATENCY guard takes `minutesSinceOnset` instead, because + * first-REM latency is an absolute physiological interval and not a proportion of how long you slept: on + * sleep-accel (n = 30 subjects with a scorable first REM period) true latency is uncorrelated with + * session duration (Pearson r = −0.058, Spearman −0.076), and re-expressing latency as a fraction makes + * it MORE variable, not less (CV 0.617 as a fraction vs 0.537 in minutes). The `c < 0.12` step this + * replaced therefore scaled a fixed physiological interval by session length: across one WHOOP 5 user's + * own recorded nights it ranged 7.4–84.5 min, an 11× spread, for the same wearer and the same physiology. + * Internal so the parity test can call it, matching the Swift `cyclePrior` visibility. */ + internal fun cyclePrior(c: Double, minutesSinceOnset: Double): Map = mapOf( "deep" to 1.2 * maxOf(0.0, 1.0 - c / 0.55), - "rem" to 1.0 * c - (if (c < 0.12) 3.0 else 0.0), + "rem" to 1.0 * c - remLatencyGuard(minutesSinceOnset), "light" to 0.0, "awake" to 0.0) + /** Sustained non-wake run, in 30 s epochs, that establishes sleep onset — 10 epochs = 5 minutes. + * Measured against PSG onset on sleep-accel (n = 31 subjects): bias −3.8 min, MAE 7.4 min. */ + internal const val onsetSustainedEpochs = 10 + + /** Index of the first epoch that begins a run of [onsetSustainedEpochs] consecutive non-"awake" labels, + * or null when the hypnogram never sustains sleep that long (a nap shorter than the rule, or an all-wake + * window). Runs are counted in epochs, not wall clock, so a coverage gap that drops an epoch cannot + * silently satisfy the rule with less evidence. */ + internal fun sustainedSleepOnset(labels: List): Int? { + var run = 0 + for (i in labels.indices) { + if (labels[i] == "awake") { run = 0; continue } + run++ + if (run >= onsetSustainedEpochs) return i - onsetSustainedEpochs + 1 + } + return null + } + /** Viterbi most-likely path over the per-epoch log-emissions with the sticky transition matrix and a * uniform start. Ties resolve to the earlier stage in [stageNames]. */ private fun viterbi(emSeq: List>): List { @@ -456,7 +524,16 @@ object SleepStagerV2 { } /** Run the full recipe over a night's epochs and return one stage label per epoch (incl. "awake"). - * All normalisation (z-scores, the HR-flatness percentile) is WITHIN the night. */ + * All normalisation (z-scores, the HR-flatness percentile) is WITHIN the night. + * + * TWO PASSES (#930). The REM-latency guard is measured from sleep ONSET, and onset is itself a staging + * output, so the recipe cannot know it while building the emissions. Pass 1 stages the night with the + * guard DISABLED (`cyclePrior(c, POSITIVE_INFINITY)`) and reads the first sustained sleep run out of the + * result; pass 2 adds the guard, re-based on that onset, and re-runs Viterbi. Only the guard differs + * between the passes — every emission term, z-score and percentile is computed ONCE and reused — so the + * extra cost is one Viterbi over an already-built lattice, not a second featurisation, and + * [stageSession] memoizes the whole thing anyway. When no sustained run exists the origin falls back to + * the window start, which is exactly the origin the shipped `c`-based guard used. */ private fun stageEpochs(feats: List): List { if (feats.isEmpty()) return emptyList() @@ -483,7 +560,9 @@ object SleepStagerV2 { return lo.toDouble() / fsorted.size } - val seq = ArrayList>(feats.size) + // Held as the concrete HashMap (not the read-only `Map` view) so pass 2 can add the REM-latency guard + // in place; `viterbi` still takes `List>`, which this satisfies by List's covariance. + val seq = ArrayList>(feats.size) for (f in feats) { val zhrv = zhr(f.hr); val zhvv = zhv(f.hrVar); val zmvv = zmv(f.moveFrac) val gate = deepGateSlope * maxOf(0.0, fpct(f.hrFlat11) - deepGateThresh) @@ -499,12 +578,24 @@ object SleepStagerV2 { em["rem"] = 0.6 * zhvv - 0.6 * zmvv + 0.4 * zhrv + baseLogPrior["rem"]!! em["light"] = baseLogPrior["light"]!! em["awake"] = 1.0 * zmvv + awakeCardiac + baseLogPrior["awake"]!! - val pr = cyclePrior(f.clock) + // Guard DISABLED here (infinity ⇒ [remLatencyGuard] = 0); pass 2 below adds it once onset is known. + val pr = cyclePrior(f.clock, Double.POSITIVE_INFINITY) for (s in stageNames) em[s] = em[s]!! + pr[s]!! if (f.jerkMax > f.jerkScale * jerkFloorGateMult) em["awake"] = em["awake"]!! + motionGateBoost f.respReg?.let { rg -> val z = zrg(rg); em["deep"] = em["deep"]!! + respWeight * z; em["rem"] = em["rem"]!! - respWeight * z } seq.add(em) } + + // PASS 1 — stage without the REM-latency guard purely to locate sleep onset. + val provisional = viterbi(seq) + // Origin for the guard: the centre of the first sustained-sleep epoch, in the same window-relative + // minutes [features] stamped on every epoch. No sustained run → 0.0, i.e. the window start. + val originMin = sustainedSleepOnset(provisional)?.let { feats[it].minutesSinceOnset } ?: 0.0 + + // PASS 2 — apply the guard against minutes since THAT onset and re-run the lattice. + for (i in feats.indices) { + seq[i]["rem"] = seq[i]["rem"]!! - remLatencyGuard(feats[i].minutesSinceOnset - originMin) + } return viterbi(seq) } } diff --git a/android/app/src/test/java/com/noop/analytics/MotionCorroboratedWakeTest.kt b/android/app/src/test/java/com/noop/analytics/MotionCorroboratedWakeTest.kt index 0992bdc9de..d0240b07e9 100644 --- a/android/app/src/test/java/com/noop/analytics/MotionCorroboratedWakeTest.kt +++ b/android/app/src/test/java/com/noop/analytics/MotionCorroboratedWakeTest.kt @@ -142,11 +142,11 @@ class MotionCorroboratedWakeTest { fun motionQuiescentPredicate() { val still = SleepStagerV2.Epoch( start = 0, hr = 58.0, hrVar = 1.0, hrFlat11 = 1.0, moveFrac = 0.0, jerkMax = 0.001, - respReg = null, clock = 0.5, jerkScale = 0.001) + respReg = null, clock = 0.5, jerkScale = 0.001, minutesSinceOnset = 120.0) assertTrue(SleepStagerV2.motionQuiescent(still)) val moved = SleepStagerV2.Epoch( start = 0, hr = 58.0, hrVar = 1.0, hrFlat11 = 1.0, moveFrac = 0.3, jerkMax = 0.2, - respReg = null, clock = 0.5, jerkScale = 0.001) + respReg = null, clock = 0.5, jerkScale = 0.001, minutesSinceOnset = 120.0) assertFalse(SleepStagerV2.motionQuiescent(moved)) } diff --git a/android/app/src/test/java/com/noop/analytics/SleepStagerV2Test.kt b/android/app/src/test/java/com/noop/analytics/SleepStagerV2Test.kt index f0cab86eda..e8a9c3d750 100644 --- a/android/app/src/test/java/com/noop/analytics/SleepStagerV2Test.kt +++ b/android/app/src/test/java/com/noop/analytics/SleepStagerV2Test.kt @@ -218,6 +218,127 @@ class SleepStagerV2Test { } } + // ── #930: the REM-latency guard is measured in MINUTES, not as a fraction of the session ──────────────── + + /** + * THE point of #930, and the assertion the previous shape could not even express: a 10-hour night and a + * 3-hour fragment must receive the SAME REM-latency guard at the same minute after sleep onset. Swift twin: + * `SleepStagerV2Tests.testRemLatencyGuardIsIdenticalAcrossSessionLengths`. + * + * Under the replaced `c < 0.12` step the two disagree by construction — 30 min into a 10 h night is + * `c = 0.05` (suppressed by the full 3.0) while 30 min into a 3 h fragment is `c = 0.167` (not suppressed at + * all), so the identical physiological instant got opposite treatment purely because of how long the wearer + * stayed in bed. The guard is isolated from the `1.0 * c` REM ramp (which is CORRECTLY a fraction of the + * session and legitimately differs between the two) by subtracting the ramp back out. + */ + @Test + fun remLatencyGuardIsIdenticalAcrossSessionLengths() { + val longNight = 10.0 * 60.0 // 600 min + val fragment = 3.0 * 60.0 // 180 min + var minutes = 0.0 + while (minutes <= 180.0) { + val cLong = minutes / longNight + val cShort = minutes / fragment + // Strip the fraction-of-session REM ramp; what remains is the latency guard alone. + val guardLong = 1.0 * cLong - SleepStagerV2.cyclePrior(cLong, minutes)["rem"]!! + val guardShort = 1.0 * cShort - SleepStagerV2.cyclePrior(cShort, minutes)["rem"]!! + assertEquals( + "guard at $minutes min must not depend on session length", + guardLong, guardShort, 1e-12) + assertEquals(guardLong, SleepStagerV2.remLatencyGuard(minutes), 1e-12) + minutes += 2.5 + } + } + + /** + * Shape of the guard: full strength at onset, linear decay, exactly zero at and past `remLatencyMinutes`, + * and clamped (never stronger than at onset) for the pre-onset epochs a mis-placed window start produces. + * Also pins the two constants themselves, so a one-sided Swift/Kotlin edit fails immediately. Swift twin: + * `SleepStagerV2Tests.testRemLatencyGuardShapeAndClamp`. + */ + @Test + fun remLatencyGuardShapeAndClamp() { + val k = SleepStagerV2.remLatencyPenalty + val m0 = SleepStagerV2.remLatencyMinutes + assertEquals("K is pinned across platforms", 3.0, k, 0.0) + assertEquals("M0 is pinned across platforms", 60.0, m0, 0.0) + assertEquals("full penalty at onset", k, SleepStagerV2.remLatencyGuard(0.0), 1e-12) + assertEquals("linear halfway", k / 2, SleepStagerV2.remLatencyGuard(m0 / 2), 1e-12) + assertEquals("spent at M0", 0.0, SleepStagerV2.remLatencyGuard(m0), 1e-12) + assertEquals("never negative later", 0.0, SleepStagerV2.remLatencyGuard(m0 + 500), 1e-12) + // #271 can place the window start hours before real onset; the guard must stay bounded there. + assertEquals("clamped pre-onset", k, SleepStagerV2.remLatencyGuard(-240.0), 1e-12) + assertEquals( + "the disabled-guard sentinel pass 1 uses must contribute nothing", + 0.0, SleepStagerV2.remLatencyGuard(Double.POSITIVE_INFINITY), 1e-12) + var prev = Double.POSITIVE_INFINITY + var m = -60.0 + while (m <= 120.0) { + val g = SleepStagerV2.remLatencyGuard(m) + assertTrue("guard must never increase with elapsed time", g <= prev + 1e-12) + prev = g + m += 1.0 + } + } + + /** + * The 5-minute sustained-sleep onset rule (10 epochs on the 30 s grid), measured against PSG onset on + * sleep-accel at bias −3.8 min / MAE 7.4 min. A shorter sleep run must NOT establish onset. Swift twin: + * `SleepStagerV2Tests.testSustainedSleepOnsetRule`. + */ + @Test + fun sustainedSleepOnsetRule() { + val wake = List(6) { "awake" } + // 9 epochs of sleep (4.5 min) is one short of the rule and must not count. + assertEquals(null, SleepStagerV2.sustainedSleepOnset(wake + List(9) { "light" } + wake)) + // 10 epochs (5 min) does, and onset is the FIRST epoch of the run, not the tenth. + assertEquals(6, SleepStagerV2.sustainedSleepOnset(wake + List(10) { "light" } + wake)) + // A brief run before a sustained one must not be mistaken for onset; the counter resets on wake. + assertEquals( + 11, + SleepStagerV2.sustainedSleepOnset(wake + List(4) { "rem" } + listOf("awake") + List(12) { "deep" })) + assertEquals("an empty night has no onset", null, SleepStagerV2.sustainedSleepOnset(emptyList())) + assertEquals("an all-wake window has no onset", null, SleepStagerV2.sustainedSleepOnset(wake)) + } + + /** + * End-to-end: the same physiological night truncated to a shorter in-bed window must not shift where the + * REM guard stops applying. Both windows start at the same instant and share byte-identical streams, so any + * difference in the first ~60 min of the hypnogram would be the session-length dependence #930 is about. + * Swift twin: `SleepStagerV2Tests.testGuardRegionDoesNotDependOnHowLongTheWearerStayedInBed`. + */ + @Test + fun guardRegionDoesNotDependOnHowLongTheWearerStayedInBed() { + val start = refMidnight + 3_600L + val longDur = 10 * 60 * 60 + val shortDur = 3 * 60 * 60 + val grav = stillGravity(start, longDur) + val hr = sleepHR(start, longDur) + val rr = regularRR(start, longDur) + + fun labels(dur: Int): List { + val segs = SleepStagerV2.stageSession(start, start + dur, grav, hr, rr, emptyList()) + val out = ArrayList() + var t = start + while (t < start + dur) { + out.add(segs.firstOrNull { it.start <= t && t < it.end }?.stage ?: "wake") + t += 30 + } + return out + } + val long = labels(longDur) + val short = labels(shortDur) + // Over the first hour — the whole span the guard touches — the two windows must agree, because the + // guard reaches the SAME 60 minutes past onset in both. Under `c < 0.12` the 10 h window was guarded + // for 72 min and the 3 h window for only 21.6 min, so they could not agree here by construction. + val guardEpochs = (SleepStagerV2.remLatencyMinutes * 60 / 30).toInt() + for (i in 0 until guardEpochs) { + assertEquals( + "epoch $i (minute ${i * 0.5}) differs — the guard must not scale with session length", + long[i], short[i]) + } + } + /** * Directly pin the #277 deep-boundary tune — the reliable guard the end-to-end golden can't be (a golden is * only sensitive where the input sits near a decision boundary). Asserts the exact tuned VALUES and their From 97e7bd76971fc7f747b7f30e8e733ceace48ec36 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Tue, 28 Jul 2026 13:00:51 -0400 Subject: [PATCH 152/171] SleepBench measures stage-fraction calibration and first-REM latency, not just kappa MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #348 fitted the stager to DREAMT. It raised kappa on all three benchmarks with a held-out gap of -0.027 — clean by every ML criterion — and PR #437 reverted it 48 hours later because it re-scored a healthy night from 6% to 23% awake. The revert's own words: "kappa doesn't guard stage-fraction calibration." SleepBench as it landed in #925 reproduces exactly that blind spot. Sections B and C score agreement (accuracy, kappa, per-stage sensitivity/specificity) and wake minutes, none of which constrain how much of the night a recipe spends at each stage. A recipe can raise kappa while systematically reallocating stages, because the epochs it newly gets right can outnumber the epochs it newly mislabels. Adds two sections. Nothing existing is replaced; the old section E is renamed G. E. PER-STAGE FRACTION CALIBRATION. Predicted % of night vs reference % of night for wake/light/deep/REM, as a signed bias in percentage points, per night and aggregated, plus the unsigned MAE so a recipe that over- and under-calls in equal measure cannot pass as well calibrated. - E.0 prints accuracy and kappa on the SAME nights, so the number that is not sufficient sits directly above the numbers that guard it. - E.2 reports a HEALTHY stratum (in-bed >= 5 h) separately. The aggregate hides the failure: #348's healthy-night blowout was ~17 pp of wake, which pooling with short fragmented nights dilutes. - E.3 reports the shipped population's stage fractions over every replayed night with no reference at all, so a distribution shift is catchable on a database with no human labels. F. FIRST-REM LATENCY. Minutes from staged sleep onset to the first REM epoch, per night and as a median/p10/p90/min/max, for each recipe and for the human reference. Calibration pins how MUCH REM a recipe emits, not WHEN; REM in the first minutes after onset is physiologically implausible in a healthy adult. Nights that never reach REM are counted separately rather than folded in as a zero. Reference set. The calibration sections score the stage-locked rows, not section B's set. B excludes an edited night whose stored hypnogram is a byte-exact replay of the CURRENT V2 — right for B's question, but version-dependent by construction: change the recipe and a night can enter or leave the exclusion, silently swapping the denominator underneath a before/after. The stagelock cursor set comes from `cursors` and does not move when the recipe moves. Read-only properties are unchanged: DB.swift is untouched, the open is still SQLITE_OPEN_READONLY | immutable=1, there is no write surface, and the database path is still a required argument. Verification. 13 new unit tests over the pure label-array primitives, no database needed (swift test in Tools/SleepBench). Measured on a real 36-session database with 15 stage-locked human-authored references (9,163 epochs), against the previous commit as a before/after: V2 4-class kappa 0.691 -> 0.689, accuracy 78.2% -> 78.2% stage bias, pp wake -7.26 -> -7.13 light -7.54 -> -6.10 deep +6.65 -> +6.65 rem +8.15 -> +6.57 first-REM latency median 72.2 -> 83.2 min, minimum 10.0 -> 43.0 min healthy-stratum wake fraction 9.40% -> 9.43% (+0.03 pp) Kappa moved -0.002 and would have called that change nothing. The latency minimum moving off 10 minutes, and the healthy-stratum wake fraction holding to +0.03 pp, are the evidence — and neither was measurable in this repository before this commit. --- Tools/SleepBench/Package.swift | 3 + .../Sources/sleepbench/Metrics.swift | 50 ++++++ .../SleepBench/Sources/sleepbench/main.swift | 154 +++++++++++++++++- .../CalibrationMetricsTests.swift | 132 +++++++++++++++ 4 files changed, 333 insertions(+), 6 deletions(-) create mode 100644 Tools/SleepBench/Tests/sleepbenchTests/CalibrationMetricsTests.swift diff --git a/Tools/SleepBench/Package.swift b/Tools/SleepBench/Package.swift index 53143e72c7..94a395c7af 100644 --- a/Tools/SleepBench/Package.swift +++ b/Tools/SleepBench/Package.swift @@ -19,5 +19,8 @@ let package = Package( ], targets: [ .executableTarget(name: "sleepbench", dependencies: ["StrandAnalytics", "WhoopProtocol"]), + // The scoring primitives are pure functions over label arrays, so they are unit-testable without a + // database. `swift test` here needs no health data and no `--db` argument. + .testTarget(name: "sleepbenchTests", dependencies: ["sleepbench"]), ] ) diff --git a/Tools/SleepBench/Sources/sleepbench/Metrics.swift b/Tools/SleepBench/Sources/sleepbench/Metrics.swift index 3a4f04e8f8..dd2af6de99 100644 --- a/Tools/SleepBench/Sources/sleepbench/Metrics.swift +++ b/Tools/SleepBench/Sources/sleepbench/Metrics.swift @@ -48,6 +48,45 @@ func onsetAndFinalWake(_ labels: [String]) -> (onset: Int?, final: Int?) { (labels.firstIndex(where: { $0 != "wake" }), labels.lastIndex(where: { $0 != "wake" })) } +// MARK: - Stage-fraction calibration + +/// Each stage's share of the night, as a percentage of the session's epochs. +/// +/// This is the quantity Cohen's kappa does NOT constrain. Kappa scores whether the predicted label at an +/// epoch matches the reference label at that epoch; a recipe can raise kappa while systematically shifting +/// how much of the night it calls wake, because the epochs it newly gets right can outnumber the epochs it +/// newly mislabels. PR #348 did exactly that (kappa up on all three benchmarks, held-out gap −0.027) and +/// PR #437 reverted it two days later for re-scoring a healthy night from 6% to 23% awake. Every stage in +/// `stageOrder` is present in the result, at 0 when the hypnogram never uses it, so a stage a recipe stops +/// emitting altogether shows up as a bias rather than as a missing row. +func stagePercentages(_ labels: [String]) -> [String: Double] { + var pct = [String: Double](uniqueKeysWithValues: stageOrder.map { ($0, 0.0) }) + guard !labels.isEmpty else { return pct } + for l in labels { pct[l, default: 0] += 1 } + for k in pct.keys { pct[k]! = pct[k]! / Double(labels.count) * 100 } + return pct +} + +/// Signed per-stage calibration error in percentage points, `predicted% − reference%`. +/// Positive means the recipe spends MORE of the night at that stage than the human did. +func stageBias(ref: [String], pred: [String]) -> [String: Double] { + let r = stagePercentages(ref), p = stagePercentages(pred) + return [String: Double](uniqueKeysWithValues: stageOrder.map { ($0, p[$0]! - r[$0]!) }) +} + +/// Minutes from staged sleep onset (the first non-wake epoch) to the first REM epoch. +/// +/// `nil` when the hypnogram has no sleep at all or never reaches REM — a night with no REM is a distinct +/// outcome from a night whose REM arrives at minute zero, and collapsing the two would let a recipe that +/// stops emitting REM look like one with a short latency. Callers report the `nil` count separately. +/// REM before onset cannot occur (REM is not wake, so onset is at or before it), but the guard is kept so +/// a caller passing a hand-built label array cannot produce a negative latency. +func firstRemLatencyMinutes(_ labels: [String]) -> Double? { + guard let onset = labels.firstIndex(where: { $0 != "wake" }), + let rem = labels.firstIndex(of: "rem"), rem >= onset else { return nil } + return Double(rem - onset) * epochSeconds / 60.0 +} + // MARK: - Agreement /// A square confusion matrix over `classes`, rows = reference, cols = prediction. @@ -124,6 +163,17 @@ func sd(_ v: [Double]) -> Double { let m = mean(v) return (v.reduce(0) { $0 + ($1 - m) * ($1 - m) } / Double(v.count - 1)).squareRoot() } +/// Nearest-rank percentile, `p` in 0...1. Used for the p10/p90 spread of a stage-fraction or latency +/// distribution, where a mean alone hides the tail a regression shows up in first. +func percentile(_ v: [Double], _ p: Double) -> Double { + guard !v.isEmpty else { return .nan } + let s = v.sorted() + let i = Int((Double(s.count) - 1) * p + 0.5) + return s[min(s.count - 1, max(0, i))] +} +/// Mean absolute value — the unsigned companion to a signed bias, so a recipe that is unbiased on average +/// because it over- and under-calls in equal measure cannot pass as well calibrated. +func mae(_ v: [Double]) -> Double { mean(v.map { abs($0) }) } /// Pearson correlation, plus the two-sided t statistic that goes with it. func pearson(_ x: [Double], _ y: [Double]) -> (r: Double, n: Int, t: Double) { let n = min(x.count, y.count) diff --git a/Tools/SleepBench/Sources/sleepbench/main.swift b/Tools/SleepBench/Sources/sleepbench/main.swift index 8b851c992f..8b43e3b4d7 100644 --- a/Tools/SleepBench/Sources/sleepbench/main.swift +++ b/Tools/SleepBench/Sources/sleepbench/main.swift @@ -347,12 +347,147 @@ for (nm, grp) in [("mean HR >= \(Int(hiCut))", hi), ("mean HR < \(Int(hiCut))", print(" \(nm.padding(toLength: 18, withPad: " ", startingAt: 0)) n=\(e.count) mean machine wake error \(f(mean(e), 7, 1)) min median \(f(median(e), 7, 1))") } -// MARK: - E. Machine cohort split — did anything change mid-history? +// MARK: - E. Per-stage fraction calibration — the guard kappa does not provide + +/// The stage-level reference set for calibration: rows carrying a `stagelock` cursor, i.e. rows whose +/// stages a human authored directly. +/// +/// This is deliberately NOT section B's set. B excludes an edited night whose stored hypnogram is a +/// byte-exact replay of the CURRENT V2, which is the right call for B's question but is version-dependent +/// by construction: change the recipe and a night can enter or leave the exclusion, silently swapping the +/// denominator underneath a before/after comparison. The `stagelock` set comes from `cursors` and does not +/// move when the recipe moves, which is what makes a calibration delta between two builds legitimate. +let calRef = nights.filter { $0.truth != nil && stageLocked.contains($0.row.startTs) } +/// The cohort PR #348 broke: a plausible full sleep opportunity. #437's field symptom was a HEALTHY night +/// re-scored 6% -> 23% awake, and an aggregate that pools short and fragmented nights hides exactly that. +let healthyMinutes = 300.0 +func isHealthy(_ nt: Night) -> Bool { nt.row.durMin >= healthyMinutes } +let calRefHealthy = calRef.filter(isHealthy) +let healthyNights = nights.filter(isHealthy) + +/// The recipes scored below, in the order the rest of the harness uses them. +let variants: [(String, (Night) -> [String])] = [ + ("V1", { $0.v1 }), ("V2", { $0.v2 }), ("V1+veto", { $0.v1Veto }), ("V2+veto", { $0.v2Veto }), +] print(""" ================================================================================ -E. PER-NIGHT DETAIL (all \(nights.count) sessions) +E. PER-STAGE FRACTION CALIBRATION (reference (a), stage-locked, n = \(calRef.count) nights) +================================================================================ +Cohen's kappa scores whether the label at an epoch matches; it does not constrain how much of the night a +recipe spends at each stage. PR #348 fitted the stager to DREAMT, raised kappa on all three benchmarks with +a held-out gap of -0.027, and was reverted 48 h later by PR #437 for re-scoring a healthy night from 6% to +23% awake. The revert's own words: "kappa doesn't guard stage-fraction calibration." This section is that +guard. Bias is signed, in percentage points of the night: POSITIVE = the recipe spends more of the night at +that stage than the human did. +""") + +// Agreement on the SAME nights the calibration below scores, so the two can be read against each other. +// This is the contrast the section exists for: kappa here can rise while the biases below get worse. +print(" E.0 AGREEMENT ON THIS REFERENCE SET — the number that is not sufficient on its own") +for (name, get) in variants { + var c4 = Confusion(classes: stageOrder), c2 = Confusion(classes: ["wake", "sleep"]) + for nt in calRef { + guard let t = nt.truth else { continue } + c4.merge(confusion(ref: t, pred: get(nt))) + c2.merge(confusion(ref: toSleepWake(t), pred: toSleepWake(get(nt)), classes: ["wake", "sleep"])) + } + print(" \(name.padding(toLength: 9, withPad: " ", startingAt: 0)) 4-class acc \(f(c4.accuracy * 100, 5, 1))% kappa \(f(c4.kappa, 6, 3)) | sleep/wake acc \(f(c2.accuracy * 100, 5, 1))% kappa \(f(c2.kappa, 6, 3)) (epochs \(c4.total))") +} + +func calibrationTable(_ title: String, _ set: [Night]) { + print("\n \(title)") + guard !set.isEmpty else { print(" no reference nights in this stratum"); return } + for (name, get) in variants { + print(" \(name) — stage pred mean% ref mean% bias pp MAE pp worst night") + for s in stageOrder { + var pred: [Double] = [], ref: [Double] = [], err: [Double] = [] + var worst = ("", 0.0) + for nt in set { + guard let t = nt.truth else { continue } + let p = stagePercentages(get(nt))[s]!, q = stagePercentages(t)[s]! + pred.append(p); ref.append(q); err.append(p - q) + if abs(p - q) > abs(worst.1) { worst = (nt.night, p - q) } + } + print(" \(s.padding(toLength: 6, withPad: " ", startingAt: 0)) \(f(mean(pred), 12, 2)) \(f(mean(ref), 11, 2)) \(f(mean(err), 10, 2)) \(f(mae(err), 9, 2)) \(worst.0) \(f(worst.1, 7, 1))") + } + } +} +calibrationTable("E.1 AGGREGATE — every stage-locked reference night (n = \(calRef.count))", calRef) +calibrationTable(""" +E.2 HEALTHY STRATUM — in-bed >= \(Int(healthyMinutes / 60)) h (n = \(calRefHealthy.count) of \(calRef.count)) + Reported separately because the aggregate hides it: a recipe can look calibrated overall while + blowing out on exactly the nights a healthy sleeper has. +""", calRefHealthy) + +// A reference-free view of the same quantity. It needs no human labels, so it runs over every replayed +// night — a far larger n than the reference set, and enough to catch a distribution shift on its own. +print(""" + + E.3 SHIPPED POPULATION — V2 stage fractions over all replayed nights, no reference needed + stage ALL n = \(nights.count) HEALTHY >= \(Int(healthyMinutes / 60)) h n = \(healthyNights.count) + mean% median% p10 p90 mean% median% p10 p90 +""") +for s in stageOrder { + let all = nights.map { stagePercentages($0.v2)[s]! } + let hea = healthyNights.map { stagePercentages($0.v2)[s]! } + print(" \(s.padding(toLength: 8, withPad: " ", startingAt: 0)) \(f(mean(all), 8, 2)) \(f(median(all), 9, 2)) \(f(percentile(all, 0.10), 8, 2)) \(f(percentile(all, 0.90), 8, 2)) \(f(mean(hea), 8, 2)) \(f(median(hea), 9, 2)) \(f(percentile(hea, 0.10), 8, 2)) \(f(percentile(hea, 0.90), 8, 2))") +} + +print("\n E.4 PER NIGHT — V2 signed bias in pp, on the \(calRef.count) stage-locked reference nights") +print(" night inbed healthy wake light deep rem") +for nt in calRef { + // `calRef` filters on `truth != nil`; the guard keeps that guarantee local, because an empty reference + // would silently turn every predicted fraction into a bias against zero. + guard let t = nt.truth else { continue } + let b = stageBias(ref: t, pred: nt.v2) + print(" \(nt.night) \(f(nt.row.durMin, 6, 0)) \(isHealthy(nt) ? " Y" : " -") \(f(b["wake"]!, 7, 1)) \(f(b["light"]!, 7, 1)) \(f(b["deep"]!, 7, 1)) \(f(b["rem"]!, 7, 1))") +} + +// MARK: - F. First-REM latency — a stage-timing property kappa is also blind to + +print(""" + +================================================================================ +F. FIRST-REM LATENCY — minutes from staged sleep onset to the first REM epoch +================================================================================ +Fraction calibration pins how MUCH REM a recipe emits; it says nothing about WHEN. A recipe can hit the +right REM total at the wrong time — REM in the first minutes after onset is physiologically implausible in +a healthy adult, and a stager that emits it is mislabelling early light sleep whatever its kappa says. +Nights that never reach REM are counted separately rather than folded in as a zero. +""") +print(" set n median mean p10 p90 min max no-REM") +func latencyRow(_ name: String, _ set: [Night], _ get: (Night) -> [String]) { + let v = set.compactMap { firstRemLatencyMinutes(get($0)) } + guard !v.isEmpty else { print(" \(name.padding(toLength: 22, withPad: " ", startingAt: 0)) no nights reach REM"); return } + print(" \(name.padding(toLength: 22, withPad: " ", startingAt: 0)) \(String(format: "%4d", v.count)) \(f(median(v), 8, 1)) \(f(mean(v), 8, 1)) \(f(percentile(v, 0.10), 8, 1)) \(f(percentile(v, 0.90), 8, 1)) \(f(v.min()!, 8, 1)) \(f(v.max()!, 8, 1)) \(String(format: "%8d", set.count - v.count))") +} +for (name, get) in variants { latencyRow("\(name), all nights", nights, get) } +latencyRow("V2, healthy >= \(Int(healthyMinutes / 60)) h", healthyNights, { $0.v2 }) +latencyRow("V2, reference nights", calRef, { $0.v2 }) +// The human's own latency distribution on the same nights — the target the replay rows above are aiming at. +latencyRow("HUMAN reference", calRef, { $0.truth ?? [] }) + +print("\n PER NIGHT — first-REM latency in minutes, on the \(calRef.count) stage-locked reference nights") +print(" night inbed healthy human V2 error") +var latErr: [Double] = [] +for nt in calRef { + guard let t = nt.truth else { continue } + // n/a in either column means that night never reached REM, which is not an error of zero minutes. + let h = firstRemLatencyMinutes(t), p = firstRemLatencyMinutes(nt.v2) + var err = Double.nan + if let h, let p { err = p - h; latErr.append(err) } + print(" \(nt.night) \(f(nt.row.durMin, 6, 0)) \(isHealthy(nt) ? " Y" : " -") \(f(h ?? .nan, 7, 1)) \(f(p ?? .nan, 7, 1)) \(f(err, 7, 1))") +} +print(" signed error (V2 - human), nights where BOTH reach REM: n=\(latErr.count) median \(f(median(latErr), 6, 1)) mean \(f(mean(latErr), 6, 1)) MAE \(f(mae(latErr), 6, 1))") + +// MARK: - G. Machine cohort split — did anything change mid-history? + +print(""" + +================================================================================ +G. PER-NIGHT DETAIL (all \(nights.count) sessions) ================================================================================ """) print("night edited band inbed meanHR storedW V1 W V2 W storedEff") @@ -362,12 +497,19 @@ for nt in nights { } if let path = a.csv { - var out = "night,userEdited,hasBand,inbedMin,meanHR,storedWake,truthWake,machineWake,v1Wake,v2Wake,v1VetoWake,v2VetoWake,storedEff\n" + /// Blank rather than a sentinel wherever a night has no value: a night that never reaches REM has no + /// first-REM latency, and writing 0 there would be a different claim. + func num(_ v: Double?, _ p: Int = 1) -> String { v.map { String(format: "%.\(p)f", $0) } ?? "" } + var out = "night,userEdited,stageLocked,healthy,hasBand,inbedMin,meanHR,storedWake,truthWake,machineWake,v1Wake,v2Wake,v1VetoWake,v2VetoWake,storedEff" + for s in stageOrder { out += ",v2\(s.capitalized)Pct,truth\(s.capitalized)Pct" } + out += ",v2FirstRemLatMin,truthFirstRemLatMin\n" for nt in nights { let stored = epochLabels(nt.row.stages, start: nt.row.startTs, end: nt.row.endTs) - let tw = nt.truth.map { String(format: "%.1f", wakeMinutes($0)) } ?? "" - let mw = nt.machineWakeMin.map { String(format: "%.1f", $0) } ?? "" - out += "\(nt.night),\(nt.row.userEdited),\(!nt.band.isEmpty),\(String(format: "%.1f", nt.row.durMin)),\(String(format: "%.1f", nt.meanHR)),\(String(format: "%.1f", wakeMinutes(stored))),\(tw),\(mw),\(String(format: "%.1f", wakeMinutes(nt.v1))),\(String(format: "%.1f", wakeMinutes(nt.v2))),\(String(format: "%.1f", wakeMinutes(nt.v1Veto))),\(String(format: "%.1f", wakeMinutes(nt.v2Veto))),\(nt.row.efficiency.map { String(format: "%.4f", $0) } ?? "")\n" + let locked = stageLocked.contains(nt.row.startTs) + out += "\(nt.night),\(nt.row.userEdited),\(locked),\(isHealthy(nt)),\(!nt.band.isEmpty),\(num(nt.row.durMin)),\(num(nt.meanHR)),\(num(wakeMinutes(stored))),\(num(nt.truth.map { wakeMinutes($0) })),\(num(nt.machineWakeMin)),\(num(wakeMinutes(nt.v1))),\(num(wakeMinutes(nt.v2))),\(num(wakeMinutes(nt.v1Veto))),\(num(wakeMinutes(nt.v2Veto))),\(num(nt.row.efficiency, 4))" + let pv = stagePercentages(nt.v2), pt = nt.truth.map { stagePercentages($0) } + for s in stageOrder { out += ",\(num(pv[s], 2)),\(num(pt?[s], 2))" } + out += ",\(num(firstRemLatencyMinutes(nt.v2))),\(num(nt.truth.flatMap { firstRemLatencyMinutes($0) }))\n" } try out.write(toFile: path, atomically: true, encoding: .utf8) FileHandle.standardError.write(Data("wrote \(path)\n".utf8)) diff --git a/Tools/SleepBench/Tests/sleepbenchTests/CalibrationMetricsTests.swift b/Tools/SleepBench/Tests/sleepbenchTests/CalibrationMetricsTests.swift new file mode 100644 index 0000000000..a68907baab --- /dev/null +++ b/Tools/SleepBench/Tests/sleepbenchTests/CalibrationMetricsTests.swift @@ -0,0 +1,132 @@ +import StrandAnalytics +import XCTest + +@testable import sleepbench + +/// The stage-fraction and first-REM-latency primitives are pure functions over 30 s label arrays, so they +/// are pinned here on hand-built hypnograms — no database, no health data, no `--db` argument. +final class CalibrationMetricsTests: XCTestCase { + + /// A label array of `n` epochs all at `stage`. + private func run(_ stage: String, _ n: Int) -> [String] { [String](repeating: stage, count: n) } + + // MARK: - stagePercentages + + func testStagePercentagesSumTo100AndMatchCounts() { + let labels = run("wake", 10) + run("light", 50) + run("deep", 20) + run("rem", 20) + let p = stagePercentages(labels) + XCTAssertEqual(p["wake"]!, 10, accuracy: 1e-9) + XCTAssertEqual(p["light"]!, 50, accuracy: 1e-9) + XCTAssertEqual(p["deep"]!, 20, accuracy: 1e-9) + XCTAssertEqual(p["rem"]!, 20, accuracy: 1e-9) + XCTAssertEqual(stageOrder.reduce(0) { $0 + p[$1]! }, 100, accuracy: 1e-9) + } + + /// A stage a recipe stops emitting must read as 0%, not as a missing key — otherwise the bias against a + /// reference that DOES contain that stage would silently vanish instead of showing up as the regression + /// it is. This is the shape of the #348 failure: a recipe reallocating a whole stage. + func testStagePercentagesReportsUnusedStagesAsZero() { + let p = stagePercentages(run("wake", 4)) + XCTAssertEqual(Set(p.keys), Set(stageOrder)) + XCTAssertEqual(p["wake"]!, 100, accuracy: 1e-9) + for s in ["light", "deep", "rem"] { XCTAssertEqual(p[s]!, 0, accuracy: 1e-9) } + } + + func testStagePercentagesOnEmptyLabelsIsAllZeroNotNaN() { + let p = stagePercentages([]) + for s in stageOrder { XCTAssertEqual(p[s]!, 0, accuracy: 1e-9) } + } + + // MARK: - stageBias + + /// Sign convention is load-bearing: the whole point of the metric is telling an over-call from an + /// under-call, so POSITIVE must mean the recipe spends more of the night at that stage than the human. + func testStageBiasIsPositiveWhenTheRecipeOverCalls() { + let ref = run("wake", 10) + run("light", 90) // 10% wake + let pred = run("wake", 30) + run("light", 70) // 30% wake + let b = stageBias(ref: ref, pred: pred) + XCTAssertEqual(b["wake"]!, 20, accuracy: 1e-9) + XCTAssertEqual(b["light"]!, -20, accuracy: 1e-9) + XCTAssertEqual(b["deep"]!, 0, accuracy: 1e-9) + } + + /// The #437 field symptom, reproduced numerically: a healthy night re-scored 6% -> 23% awake is a + /// +17 pp wake bias, which is what this metric exists to surface. + func testStageBiasReproducesTheRevertScaleSymptom() { + let ref = run("wake", 6) + run("light", 94) + let pred = run("wake", 23) + run("light", 77) + XCTAssertEqual(stageBias(ref: ref, pred: pred)["wake"]!, 17, accuracy: 1e-9) + } + + /// Two nights that miss in opposite directions cancel in the signed mean but not in the MAE, which is + /// why both are reported: a mean bias near zero is not by itself evidence of good calibration. + func testMeanBiasCancelsWhereMAEDoesNot() { + let errs = [12.0, -12.0] + XCTAssertEqual(mean(errs), 0, accuracy: 1e-9) + XCTAssertEqual(mae(errs), 12, accuracy: 1e-9) + } + + // MARK: - firstRemLatencyMinutes + + /// Latency is measured from staged ONSET, not from the start of the session — a night with a long + /// awake-in-bed lead-in must not be credited with a long REM latency. + func testFirstRemLatencyIsMeasuredFromOnsetNotSessionStart() { + // 60 epochs awake in bed, then 20 light, then REM. Onset is epoch 60, REM at epoch 80. + let labels = run("wake", 60) + run("light", 20) + run("rem", 10) + XCTAssertEqual(firstRemLatencyMinutes(labels)!, 10.0, accuracy: 1e-9) // 20 epochs x 30 s + } + + func testFirstRemLatencyUsesTheThirtySecondGrid() { + let labels = run("light", 2) + run("rem", 1) + XCTAssertEqual(firstRemLatencyMinutes(labels)!, 1.0, accuracy: 1e-9) + XCTAssertEqual(epochSeconds, 30.0, accuracy: 1e-9) + } + + func testFirstRemLatencyIsZeroWhenREMIsTheFirstSleepEpoch() { + XCTAssertEqual(firstRemLatencyMinutes(run("wake", 5) + run("rem", 5))!, 0.0, accuracy: 1e-9) + } + + /// A night with no REM is a DIFFERENT outcome from a night whose REM lands at minute zero. Collapsing + /// them would let a recipe that stops emitting REM read as one with an excellent short latency. + func testFirstRemLatencyIsNilWithoutREMAndWithoutSleep() { + XCTAssertNil(firstRemLatencyMinutes(run("wake", 20) + run("light", 40))) + XCTAssertNil(firstRemLatencyMinutes(run("wake", 100))) + XCTAssertNil(firstRemLatencyMinutes([])) + } + + // MARK: - percentile + + func testPercentileSpansMinToMax() { + let v = [1.0, 2, 3, 4, 5, 6, 7, 8, 9, 10] + XCTAssertEqual(percentile(v, 0.0), 1, accuracy: 1e-9) + XCTAssertEqual(percentile(v, 1.0), 10, accuracy: 1e-9) + XCTAssertEqual(percentile(v, 0.5), 6, accuracy: 1e-9) + } + + func testPercentileIsOrderIndependentAndSafeOnEmpty() { + XCTAssertEqual(percentile([5.0, 1, 3], 0.0), 1, accuracy: 1e-9) + XCTAssertTrue(percentile([], 0.5).isNaN) + } + + // MARK: - the grid the metrics run on + + /// `stagePercentages` and `firstRemLatencyMinutes` both consume `epochLabels` output in the harness, so + /// the expansion from a `StageSegment` tiling has to land the stages where the metrics expect them. + func testMetricsAgreeWithEpochLabelExpansion() { + // 30 min wake, 30 min light, 30 min REM, on a 90 min session. + let start = 1_700_000_000 + let segs = [ + StageSegment(start: start, end: start + 1800, stage: "wake"), + StageSegment(start: start + 1800, end: start + 3600, stage: "light"), + StageSegment(start: start + 3600, end: start + 5400, stage: "rem"), + ] + let labels = epochLabels(segs, start: start, end: start + 5400) + XCTAssertEqual(labels.count, 180) + let p = stagePercentages(labels) + XCTAssertEqual(p["wake"]!, 100.0 / 3, accuracy: 1e-6) + XCTAssertEqual(p["light"]!, 100.0 / 3, accuracy: 1e-6) + XCTAssertEqual(p["rem"]!, 100.0 / 3, accuracy: 1e-6) + XCTAssertEqual(firstRemLatencyMinutes(labels)!, 30.0, accuracy: 1e-9) + XCTAssertEqual(wakeMinutes(labels), 30.0, accuracy: 1e-9) + } +} From af44563c19f1564a6870b55e0245c0626439244c Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 29 Jul 2026 00:06:06 -0400 Subject: [PATCH 153/171] Stop SleepBench's calibration set scoring V2 against V2's own output MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Section E's reference set keys on `stagelock` so the denominator cannot move between two builds — the right call, and the reason section B's recipe-dependent exclusion is not reused there. But a `stagelock` cursor proves only that the stages arrived through `edit_sleep_stages`; it does not prove they differ from what V2 emits. On the database this was developed against, 3 of the 15 stage-locked reference nights replay BYTE-EXACT from V2 and a 4th agrees at 99.09%. Those rows carry no information about V2: they hand the incumbent a guaranteed perfect night and charge every alternative recipe for the same nights. The effect is large enough to invert a verdict. Scoring PR #348's parameter set on this database gives 4-class kappa 0.640 against the incumbent's 0.689 on the contaminated 15-night set — #348 loses — and 0.705 against 0.604 on the same set with the three byte-exact nights held out — #348 wins by +0.101. Section E exists precisely to judge a #348-style tune, so a 0.085-kappa thumb on the scale in favour of "change nothing" is the one bias it must not have. Section E cannot simply drop the rows the way section B does without reintroducing the version-dependence it was built to avoid, so the harness names them instead: a new E.-1 audit lists every stage-locked night whose stored hypnogram replays from V2 at >= 95%, flags the byte-exact ones, reports what fraction of the reference set they are, and prints the exact `--exclude` line to pin the same held-out set across both builds of a comparison. Excluding by explicit timestamp keeps the set frozen; excluding by "matches this build's V2" would not. The audit asserts no mechanism for the match, only that a row cannot be an independent reference for the recipe it replays. Verification: `swift test` in Tools/SleepBench = 13 tests, 0 failures; release build clean; run against a 37-session device database copy, E.-1 names the four nights above and every downstream section is unchanged when `--exclude` is not passed. Tools-only — no analytics, no stager, no Android twin, no goldens. Refs #348, #437. --- .../SleepBench/Sources/sleepbench/main.swift | 65 ++++++++++++++++++- 1 file changed, 63 insertions(+), 2 deletions(-) diff --git a/Tools/SleepBench/Sources/sleepbench/main.swift b/Tools/SleepBench/Sources/sleepbench/main.swift index 8b43e3b4d7..1207006723 100644 --- a/Tools/SleepBench/Sources/sleepbench/main.swift +++ b/Tools/SleepBench/Sources/sleepbench/main.swift @@ -6,6 +6,7 @@ import WhoopProtocol // independent reference the database carries. // // USAGE: sleepbench --db /path/to/whoop.sqlite [--device my-whoop-noop] [--pad 3600] [--csv out.csv] +// [--exclude ] // // The database is never written to and never lives in this repository. Pass a COPY; never point this at // a device's live database. @@ -20,6 +21,15 @@ struct Args { var streamDevice = "my-whoop" var pad = 3600 var csv: String? + /// Session `startTs` values held out of the stage-locked calibration set in section E. + /// + /// Section E's reference set has to stay FIXED across the two builds of a before/after comparison, or + /// the denominator moves underneath the delta. That is why it keys on `stagelock` rather than on + /// section B's recipe-dependent exclusion. But `stagelock` alone does not guarantee the row is + /// independent of V2 (see the self-comparison warning below), and the honest fix for a contaminated row + /// is to name it ONCE and pass the same `--exclude` list to every build being compared. Excluding by + /// explicit timestamp keeps the set frozen; excluding by "matches this build's V2" would not. + var exclude: Set = [] } var a = Args() var it = CommandLine.arguments.dropFirst().makeIterator() @@ -30,11 +40,18 @@ while let k = it.next() { case "--stream-device": a.streamDevice = it.next() ?? a.streamDevice case "--pad": a.pad = Int(it.next() ?? "") ?? a.pad case "--csv": a.csv = it.next() + case "--exclude": a.exclude = Set((it.next() ?? "").split(separator: ",").compactMap { Int($0) }) default: FileHandle.standardError.write(Data("unknown argument \(k)\n".utf8)); exit(2) } } guard !a.db.isEmpty else { - print("usage: sleepbench --db [--device ] [--pad ] [--csv ]") + print(""" + usage: sleepbench --db [--device ] [--stream-device ] [--pad ] [--csv ] + [--exclude ] + + --exclude session startTs values held out of section E's calibration reference set. Pass the SAME + list to every build in a before/after comparison; section E names the candidates. + """) exit(2) } @@ -43,7 +60,7 @@ let rows = try db.sessions(device: a.device) guard !rows.isEmpty else { print("no sessions for device \(a.device)"); exit(1) } // Which `userEdited` rows carry a HUMAN-AUTHORED hypnogram rather than machine stages re-derived over a // human-corrected window. Only the former can serve as a stage-level reference. -let stageLocked = try db.stageLockedStarts(device: a.device) +let stageLocked = try db.stageLockedStarts(device: a.device).subtracting(a.exclude) let editedCount = rows.filter { $0.userEdited }.count print(""" @@ -149,12 +166,16 @@ var v1Match = 0, v2Match = 0, neither = 0 /// This says nothing about HOW the row came to match — the harness reports the fact and excludes the /// night, and deliberately does not assert a mechanism it has not established. var indistinguishableFromV2: Set = [] +/// Per-night epoch agreement between the STORED hypnogram and a fresh V2 replay, by `startTs`. Section E +/// reads it to warn when its own reference set contains rows V2 cannot be scored against independently. +var v2AgreePct: [Int: Double] = [:] print("night edited epochs V1 agree V2 agree verdict") for nt in nights { let stored = epochLabels(nt.row.stages, start: nt.row.startTs, end: nt.row.endTs) let a1 = zip(stored, nt.v1).filter { $0 == $1 }.count let a2 = zip(stored, nt.v2).filter { $0 == $1 }.count let p1 = Double(a1) / Double(stored.count) * 100, p2 = Double(a2) / Double(stored.count) * 100 + v2AgreePct[nt.row.startTs] = p2 let verdict = p2 >= 99.9 ? "V2 exact" : (p1 >= 99.9 ? "V1 exact" : (p2 > p1 ? "V2 closer" : "V1 closer")) if !nt.row.userEdited { if p2 >= 99.9 { v2Match += 1 } else if p1 >= 99.9 { v1Match += 1 } else { neither += 1 } @@ -357,6 +378,13 @@ for (nm, grp) in [("mean HR >= \(Int(hiCut))", hi), ("mean HR < \(Int(hiCut))", /// by construction: change the recipe and a night can enter or leave the exclusion, silently swapping the /// denominator underneath a before/after comparison. The `stagelock` set comes from `cursors` and does not /// move when the recipe moves, which is what makes a calibration delta between two builds legitimate. +/// +/// The cost of that choice, and why the warning below exists: a `stagelock` cursor proves the stages +/// arrived through the `edit_sleep_stages` path, NOT that they differ from what V2 would emit. A row whose +/// stored hypnogram replays byte-exact from V2 carries no information about V2 — it hands the incumbent a +/// free perfect night and charges every alternative for the same nights. Section B drops such rows; section +/// E cannot drop them the same way without reintroducing the version-dependence it exists to avoid. So the +/// harness NAMES them and leaves the choice to the operator, who freezes it with `--exclude`. let calRef = nights.filter { $0.truth != nil && stageLocked.contains($0.row.startTs) } /// The cohort PR #348 broke: a plausible full sleep opportunity. #437's field symptom was a HEALTHY night /// re-scored 6% -> 23% awake, and an aggregate that pools short and fragmented nights hides exactly that. @@ -383,6 +411,39 @@ guard. Bias is signed, in percentage points of the night: POSITIVE = the recipe that stage than the human did. """) +// SELF-COMPARISON AUDIT of this section's own reference set. A stage-locked row whose stored hypnogram is a +// byte-exact V2 replay scores V2 against itself: it is a guaranteed 100% for the incumbent and a guaranteed +// loss for any alternative, so it biases a before/after delta toward "change nothing" by an amount nobody +// reading only the kappa line would see. The audit reports the contamination rather than silently absorbing +// it, and names the timestamps so the same held-out set can be pinned across both builds with `--exclude`. +let selfMatched = calRef.filter { (v2AgreePct[$0.row.startTs] ?? 0) >= 99.9 } +let nearMatched = calRef.filter { + let p = v2AgreePct[$0.row.startTs] ?? 0 + return p >= 95.0 && p < 99.9 +} +print(" E.-1 SELF-COMPARISON AUDIT of the reference set") +if selfMatched.isEmpty && nearMatched.isEmpty { + print(" clean — no stage-locked night replays from V2 at >= 95% agreement.") +} else { + print(" night stored-vs-V2 agreement verdict") + for nt in (selfMatched + nearMatched).sorted(by: { $0.row.startTs < $1.row.startTs }) { + let p = v2AgreePct[nt.row.startTs] ?? 0 + print(" \(nt.night) (startTs \(nt.row.startTs)) \(f(p, 12, 2))% \(p >= 99.9 ? "BYTE-EXACT — carries no information about V2" : "near-exact — nearly none")") + } + let pct = Double(selfMatched.count) / Double(max(1, calRef.count)) * 100 + print(""" + \(selfMatched.count) of \(calRef.count) reference nights (\(f(pct, 0, 0))%) are byte-exact V2 replays; \(nearMatched.count) more sit at >= 95%. + These inflate every V2 figure in E.0-E.4 and penalise any alternative recipe by construction. To + compare two builds honestly, pass the SAME list to BOTH: + --exclude \(selfMatched.map { String($0.row.startTs) }.joined(separator: ",")) + The harness asserts no mechanism for the match; it reports only that the row cannot serve as an + independent reference for the recipe it replays. + """) +} +if !a.exclude.isEmpty { + print(" HELD OUT by --exclude this run: \(a.exclude.sorted().map(String.init).joined(separator: ", "))") +} + // Agreement on the SAME nights the calibration below scores, so the two can be read against each other. // This is the contrast the section exists for: kappa here can rise while the biases below get worse. print(" E.0 AGREEMENT ON THIS REFERENCE SET — the number that is not sufficient on its own") From b9802ff55c0adefa5279efc252907e6ec0ccfc24 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 29 Jul 2026 00:18:44 -0400 Subject: [PATCH 154/171] Show the illness early-warning banner on the default (Liquid) Today MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `HealthAlertBanner` was mounted only in the classic `TodayView`. Liquid Today ships as the iOS default (`noop.liquidTodayEnabled` = true), so when `AppModel.applyIllnessSignal` raised `healthAlert` on `.raised` / `.alreadyUnwell`, the screen the user actually opens showed nothing — the signal survived only in the Health tab's HeadsUpCard and the once-a-day local notification. Mount the same leaf in the Liquid section column, directly under the header scene, matching the classic placement under its top bar. It sits above the reorderable block so the Arrange sheet can't move it out of sight, and it renders nothing when there's no alert. Same amber, non-diagnostic treatment (unchanged `HealthAlertBanner` / `NoopCard` + `StrandPalette.statusWarning`). Same class of regression as #992 runtime, B1 backfill and #543 Charge carry. Verified: NOOPiOS + Strand (macOS) both build; screenshotted the seeded `--demo-screen liquidtoday` simulator build with and without an alert — banner renders under the header above the hero, and the no-alert case leaves the wordmark-to-hero spacing unchanged (no phantom gap). --- Strand/Liquid/LiquidTodayView.swift | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Strand/Liquid/LiquidTodayView.swift b/Strand/Liquid/LiquidTodayView.swift index 913750aa12..90be1b1ec6 100644 --- a/Strand/Liquid/LiquidTodayView.swift +++ b/Strand/Liquid/LiquidTodayView.swift @@ -267,6 +267,14 @@ struct LiquidTodayView: View { VStack(alignment: .leading, spacing: 12) { scene + // The illness early-warning banner, dropped in the liquid Home rewrite while Liquid became + // the default Today — so a raised/already-unwell signal had no surface on the screen the + // user actually opens (only the Health tab's HeadsUpCard and the once-a-day notification). + // Restored as the SAME leaf the classic TodayView renders directly under its top bar, so + // it sits above the reorderable block and can't be reordered out of sight. Renders nothing + // when there's no alert; it owns its own AppModel observation, so the ~1 Hz HR stream + // re-renders only the banner, not this body. + HealthAlertBanner() // #105: the live "workout in progress" card, dropped in the liquid Home rewrite. Restored // here as the SAME leaf the classic TodayView renders (and Android's WorkoutInProgressCard), // pinned above the reorderable block so an active manual workout is immediately visible From 31c6a5638ba3d5557f1636a763e0f3fac2758e30 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 30 Jul 2026 16:11:34 -0400 Subject: [PATCH 155/171] =?UTF-8?q?fix(sleep):=20forbid=20wake=20->=20deep?= =?UTF-8?q?/REM=20transitions=20=E2=80=94=20the=20one=20part=20of=20#348?= =?UTF-8?q?=20that=20survives=20measurement?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR #348 re-tuned seven things about SleepStagerV2 on DREAMT; PR #437 reverted all of them 48 h later because one healthy night went from 6 % to 23 % awake ("kappa doesn't guard stage-fraction calibration"). Re-litigating that revert component-by-component, on a reference set de-contaminated with the `--exclude` flag added in af44563c, six of the seven components measure neutral-to-negative and stay reverted. One does not: the AWAKE transition row. Restored from #348, on both platforms: "awake": deep 0.01 -> 0.0, rem 0.02 -> 0.0, light 0.27 -> 0.10, awake 0.70 -> 0.90 This is a physiological claim, not a fitted constant: sleep onset descends through N1/N2, so wake never transitions straight into N3 or REM, and the freed mass makes a WASO episode span several epochs rather than flicker back to sleep after one. The `ln(max(v, 1e-9))` viterbi floor that makes a zeroed entry safe already landed with #348 and survived #437, so no other change is needed to support it. MEASUREMENT (Tools/SleepBench, one wearer's 36 recorded nights). The only reference here that V2 cannot contaminate is the strap's own band `sleep_state` — the v18 @81 nibble, WHOOP's verdict rather than a re-derivation of NOOP's. Over 21 banded nights / 15 554 epochs: sleep/wake kappa 0.105 -> 0.118 wake sensitivity 16.0 % -> 17.6 % accuracy 79.1 % -> 79.2 % The #437 guard holds — this is the number that must not blow out, and does not: healthy-stratum wake fraction (n = 20) 9.43 % -> 9.96 % (#348 entire: 32.66 %) healthy deep 22.09 % -> 22.11 % first-REM latency MAE 53.9 -> 41.6 min Why the other six stay reverted, each measured alone against the same band reference / wake guard: base priors (deep .15, awake .34) healthy wake 9.43 % -> 17.76 % the #437 blow-out, confirmed motion gate (jerk 75/35, boost 4) healthy wake 9.43 % -> 15.92 % a SECOND wake channel, not free emission coefficients band kappa 0.105 -> 0.094 deep gate .25 -> .40 healthy deep 22.09 % -> 25.47 % worsens an existing over-call awake dead-zone 0.30 band kappa 0.105 -> 0.099 deep/rem/light transition rows band kappa 0.105 -> 0.101, healthy REM 29.58 % -> 33.11 % LIMITS, stated plainly. n = 1 wearer. #348's out-of-cohort evidence was +0.028 (AAUWSS) and +0.006 (Walch); its headline +0.17 is DREAMT in-sample, on the cohort its constants were tuned to. 13 of this database's 15 stage-locked nights are byte-exact or >= 95 % V2 replays and cannot serve as an independent reference for V2 at all, leaving 2 usable hand-labelled nights (kappa 0.857 -> 0.863 there, a tie); the band comparison above is the load-bearing one precisely because it does not depend on those labels. Verification: StrandAnalytics `swift test` 1248 tests, 0 failures. Android `:app:testFullDebugUnitTest --rerun-tasks` 3239 tests, 0 failures, 0 errors (counted from JUnit XML), SleepStagerV2Test 12/12. Both frozen goldens pass UNCHANGED — they are insensitive to this row, which is why this adds a direct pin for it on both platforms (twin of the existing deep-row pin), asserting the zeros and that a zeroed entry still reaches the lattice as a finite log-weight. --- .../StrandAnalytics/SleepStagerV2.swift | 17 ++++++++++++++- .../SleepStagerV2Tests.swift | 16 ++++++++++++++ .../java/com/noop/analytics/SleepStagerV2.kt | 19 +++++++++++++++-- .../com/noop/analytics/SleepStagerV2Test.kt | 21 +++++++++++++++++++ 4 files changed, 70 insertions(+), 3 deletions(-) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift index 160d3a0d41..4113f794b0 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift @@ -176,11 +176,26 @@ public enum SleepStagerV2 { /// Transition matrix (rows = from, cols = to). Self-transitions dominate; deep↔rem rare; wake mostly /// to/from light. A priori, not fit. + /// + /// The AWAKE row encodes sleep-onset physiology directly: a sleeper does not enter N3 or REM straight + /// out of wakefulness — descent runs through N1/N2 — so wake→deep and wake→rem are ZERO rather than the + /// small non-zero values they used to carry, and the freed mass goes to the wake self-loop, which makes + /// a WASO episode span several epochs instead of flickering back to sleep after one. This row is the one + /// part of PR #348's DREAMT re-tune that survives measurement on a de-contaminated reference set; the + /// rest of that PR (its base priors, motion-gate multipliers, deep gate, awake dead-zone, emission + /// coefficients and the deep/rem/light transition rows) was reverted by #437 and stays reverted, having + /// measured neutral-to-negative here. See the header note on `viterbi` for why a zero is safe. + /// + /// Measured on one wearer's 36 recorded nights, against the strap's own band `sleep_state` (an + /// independent reference the recipe cannot contaminate — 21 nights, 15 554 epochs): sleep/wake kappa + /// 0.105 → 0.118 and wake sensitivity 16.0 % → 17.6 %, with the healthy-stratum wake fraction essentially + /// unmoved (9.43 % → 9.96 %, i.e. no repeat of the #437 blow-out) and first-REM latency MAE 53.9 → 41.6 + /// min. n = 1 wearer; see `Tools/SleepBench` and the PR for the full ablation and its limits. static let transition: [String: [String: Double]] = [ "deep": ["deep": 0.86, "rem": 0.007, "light": 0.126, "awake": 0.007], "rem": ["deep": 0.005, "rem": 0.88, "light": 0.10, "awake": 0.015], "light": ["deep": 0.06, "rem": 0.06, "light": 0.85, "awake": 0.03], - "awake": ["deep": 0.01, "rem": 0.02, "light": 0.27, "awake": 0.70]] + "awake": ["deep": 0.0, "rem": 0.0, "light": 0.10, "awake": 0.90]] /// One 30 s epoch's recipe features. Optionals are "no measurement"; the z-score / percentile treat a /// missing value as the neutral centre so a sparse channel never blocks a stage. diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerV2Tests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerV2Tests.swift index 83e4ba6c6c..98adebcd15 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerV2Tests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerV2Tests.swift @@ -327,4 +327,20 @@ final class SleepStagerV2Tests: XCTestCase { XCTAssertEqual(row.values.reduce(0, +), 1.0, accuracy: 1e-9, "transition row '\(from)' must sum to 1.0") } } + + /// Pin the AWAKE transition row directly, for the same reason the deep row is pinned: the end-to-end + /// golden is only sensitive where its own input sits near a decision boundary, and it does NOT move when + /// this row changes — so without this assertion the row is effectively unguarded. Twin: + /// `SleepStagerV2Test.wakeRowForbidsDirectDescentIntoDeepOrRem`. + /// + /// The zeros are the physiological claim: sleep onset descends through N1/N2, so wake never transitions + /// straight into N3 or REM. They are also why `viterbi` floors the log at 1e-9 — asserted here so the two + /// can never drift apart and reintroduce ln(0) = -inf. + func testWakeRowForbidsDirectDescentIntoDeepOrRem() { + XCTAssertEqual(SleepStagerV2.transition["awake"]!, + ["deep": 0.0, "rem": 0.0, "light": 0.10, "awake": 0.90]) + // A zeroed entry must reach the lattice as a large finite penalty, never -inf. + let lp = log(max(SleepStagerV2.transition["awake"]!["deep"]!, 1e-9)) + XCTAssertTrue(lp.isFinite, "a zeroed transition must not produce a non-finite log-weight") + } } diff --git a/android/app/src/main/java/com/noop/analytics/SleepStagerV2.kt b/android/app/src/main/java/com/noop/analytics/SleepStagerV2.kt index 843c961885..0ea47718e3 100644 --- a/android/app/src/main/java/com/noop/analytics/SleepStagerV2.kt +++ b/android/app/src/main/java/com/noop/analytics/SleepStagerV2.kt @@ -196,12 +196,27 @@ object SleepStagerV2 { private const val respWeight = 0.6 /** Transition matrix (rows = from, cols = to). Self-transitions dominate; deep↔rem rare; wake mostly - * to/from light. A priori, not fit. */ + * to/from light. A priori, not fit. + * + * The AWAKE row encodes sleep-onset physiology directly: a sleeper does not enter N3 or REM straight + * out of wakefulness — descent runs through N1/N2 — so wake→deep and wake→rem are ZERO rather than the + * small non-zero values they used to carry, and the freed mass goes to the wake self-loop, which makes + * a WASO episode span several epochs instead of flickering back to sleep after one. This row is the one + * part of PR #348's DREAMT re-tune that survives measurement on a de-contaminated reference set; the + * rest of that PR (its base priors, motion-gate multipliers, deep gate, awake dead-zone, emission + * coefficients and the deep/rem/light transition rows) was reverted by #437 and stays reverted, having + * measured neutral-to-negative here. See the header note on [viterbi] for why a zero is safe. + * + * Measured on one wearer's 36 recorded nights, against the strap's own band `sleep_state` (an + * independent reference the recipe cannot contaminate — 21 nights, 15 554 epochs): sleep/wake kappa + * 0.105 → 0.118 and wake sensitivity 16.0 % → 17.6 %, with the healthy-stratum wake fraction essentially + * unmoved (9.43 % → 9.96 %, i.e. no repeat of the #437 blow-out) and first-REM latency MAE 53.9 → 41.6 + * min. n = 1 wearer; see `Tools/SleepBench` and the PR for the full ablation and its limits. */ internal val transition: Map> = mapOf( "deep" to mapOf("deep" to 0.86, "rem" to 0.007, "light" to 0.126, "awake" to 0.007), "rem" to mapOf("deep" to 0.005, "rem" to 0.88, "light" to 0.10, "awake" to 0.015), "light" to mapOf("deep" to 0.06, "rem" to 0.06, "light" to 0.85, "awake" to 0.03), - "awake" to mapOf("deep" to 0.01, "rem" to 0.02, "light" to 0.27, "awake" to 0.70)) + "awake" to mapOf("deep" to 0.0, "rem" to 0.0, "light" to 0.10, "awake" to 0.90)) /** One 30 s epoch's recipe features. Nullable means "no measurement"; the z-score / percentile treat a * missing value as the neutral centre so a sparse channel never blocks a stage. Internal (not private) so diff --git a/android/app/src/test/java/com/noop/analytics/SleepStagerV2Test.kt b/android/app/src/test/java/com/noop/analytics/SleepStagerV2Test.kt index e8a9c3d750..32a6caebb3 100644 --- a/android/app/src/test/java/com/noop/analytics/SleepStagerV2Test.kt +++ b/android/app/src/test/java/com/noop/analytics/SleepStagerV2Test.kt @@ -9,6 +9,7 @@ import org.junit.Assert.assertNotNull import org.junit.Assert.assertTrue import org.junit.Test import kotlin.math.PI +import kotlin.math.ln import kotlin.math.sin import kotlin.math.roundToInt @@ -357,4 +358,24 @@ class SleepStagerV2Test { assertEquals("transition row '$from' must sum to 1.0", 1.0, row.values.sum(), 1e-9) } } + + /** + * Pin the AWAKE transition row directly, for the same reason the deep row is pinned: the end-to-end golden + * is only sensitive where its own input sits near a decision boundary, and it does NOT move when this row + * changes — so without this assertion the row is effectively unguarded. Twin: + * `SleepStagerV2Tests.testWakeRowForbidsDirectDescentIntoDeepOrRem`. + * + * The zeros are the physiological claim: sleep onset descends through N1/N2, so wake never transitions + * straight into N3 or REM. They are also why [SleepStagerV2.viterbi] floors the log at 1e-9 — asserted here + * so the two can never drift apart and reintroduce ln(0) = -Inf. + */ + @Test + fun wakeRowForbidsDirectDescentIntoDeepOrRem() { + assertEquals( + mapOf("deep" to 0.0, "rem" to 0.0, "light" to 0.10, "awake" to 0.90), + SleepStagerV2.transition["awake"]) + // A zeroed entry must reach the lattice as a large finite penalty, never -Inf. + val lp = ln(maxOf(SleepStagerV2.transition["awake"]!!["deep"]!!, 1e-9)) + assertTrue("a zeroed transition must not produce a non-finite log-weight", lp.isFinite()) + } } From 842ebdaf24ca98988e200a6df9d11f6ef64a5274 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 30 Jul 2026 19:34:04 -0400 Subject: [PATCH 156/171] Tools/SleepPSG: a PSG-labelled validation instrument for SleepStagerV2, in the repo this time MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every reference the sleep work can currently reach is either the recipe's own output handed back to it or another vendor's black box. A harness that scored SleepStagerV2 against real polysomnography existed once, settled the most decisive question this project has asked — whether REM detection reads physiology or the clock — and then vanished, because it lived in a session scratchpad. The next investigation had to run on one instrument instead of two and said so. This is that harness, committed, with tests. `sleeppsg --dataset ` replays SleepStagerV2 over PhysioNet sleep-accel (Walch, Huang, Forger & Goldstein, SLEEP 42(12) zsz180, 2019) — 31 subjects of wrist accelerometer + heart rate recorded alongside human-scored PSG hypnograms — and scores it epoch-for-epoch. Read-only; the dataset path is always an argument and the dataset is never committed. README.md carries the download step and the attribution the Open Data Commons licence and the MIT companion code require. THE PORT, AND WHY IT CANNOT DRIFT. Baseline numbers come from StrandAnalytics.SleepStagerV2.stageSession itself. Variant numbers cannot: the recipe holds its constants as `static let`s and keeps Epoch/features() internal, so asking "what would this score with one transition row moved" needs the recipe restated with knobs. RecipePort.swift does that, and PortValidation MEASURES the equivalence instead of asserting it — 48 randomised nights (off-grid starts, channel dropout, arousals, motion bursts, unsorted input, R-R present so the RSA term is exercised, which this dataset cannot do) plus 7 degenerate cases: all-HR- missing, zero-variance HR, 1- and 2-epoch nights, saturated motion gate, no coverage at all, motion absent. 12 377 of 12 377 epoch labels identical. It runs in `swift test`, needs no dataset, and names the night and epoch when it fails. REPORTED IN THIS SHAPE ON PURPOSE. Stage fractions are a first-class result, because kappa does not constrain them and #348 -> #437 is the proof. Splits are leave-one-SUBJECT-out, never epoch-level, and a test fails if a held-out subject's rows can reach its own training fold. Night length is a stratification variable rather than a footnote. WHAT IT FOUND, on 26 773 PSG-scored epochs. 1. #987's awake row survives PSG. It was landed on band-state evidence alone. Against truth: kappa 0.356 -> 0.363, REM F1 0.569 -> 0.575, wake sensitivity 30.4 % -> 30.8 %, and the #437 guard holds (wake 4.34 % -> 4.15 %, deep unmoved at 18.94 %). Its first-REM latency claim does NOT reproduce and reverses: MAE 45.1 -> 49.9 min against the band's 53.9 -> 41.6. 2. #348's six rejected components: PSG agrees on five, disagrees on one. Emissions -0.012 kappa, motion gate -0.004, dead-zone -0.001; the deep gate worsens an already +5.2 pp deep over-call to +8.9, and the deep/rem/light rows push REM from +5.0 pp to +9.2. But the BASE PRIORS improve calibration here (+0.002 kappa, wake bias -4.74 -> -1.20 pp, deep +5.18 -> +1.21), because against PSG this recipe UNDER-calls wake by 4.7 pp. The #437 blow-out is a property of that wearer's nights, not a universal. 3. The REM/night-length coupling does not reproduce: predicted r = +0.037 where PSG truth itself carries +0.398. Weak test — sleep-accel nights cluster at 476 min. 4. Physiology beats the clock, by much less than reported. LOSO logistic: clock-only 0.431, physiology 0.479, both 0.522; physiology - clock positive in 22/31 subjects and both - clock in 26/31. The direction reproduces; the +0.329 does not, because a clock model with a train-tuned threshold is a far stronger baseline than 0.241. The self-check in section 3 reproduces every truth-side figure exactly (26 773 epochs, deep 14.76 %, truth first-REM 88.5 min) and predicted deep to 0.01 pp. The REM prediction does not; the obvious explanation (#930's guard) was tested with a dedicated variant and FALSIFIED, and the residual is left open rather than tuned away. The CI wiring is deliberately not here: it extends the `tools:` matrix that upstream #943 added, which this fork has not synced yet. It rides in with the upstream sync. --- Tools/SleepPSG/Package.resolved | 14 + Tools/SleepPSG/Package.swift | 33 ++ Tools/SleepPSG/README.md | 191 +++++++ .../SleepPSG/Sources/sleeppsg/Ablation.swift | 220 ++++++++ .../Sources/sleeppsg/PortValidation.swift | 303 +++++++++++ .../Sources/sleeppsg/RecipePort.swift | 482 ++++++++++++++++++ Tools/SleepPSG/Sources/sleeppsg/Scoring.swift | 229 +++++++++ .../Sources/sleeppsg/SleepAccel.swift | 232 +++++++++ .../SleepPSG/Sources/sleeppsg/Variants.swift | 153 ++++++ Tools/SleepPSG/Sources/sleeppsg/main.swift | 481 +++++++++++++++++ .../DatasetAndAblationTests.swift | 295 +++++++++++ .../sleeppsgTests/PortValidationTests.swift | 92 ++++ .../Tests/sleeppsgTests/ScoringTests.swift | 141 +++++ 13 files changed, 2866 insertions(+) create mode 100644 Tools/SleepPSG/Package.resolved create mode 100644 Tools/SleepPSG/Package.swift create mode 100644 Tools/SleepPSG/README.md create mode 100644 Tools/SleepPSG/Sources/sleeppsg/Ablation.swift create mode 100644 Tools/SleepPSG/Sources/sleeppsg/PortValidation.swift create mode 100644 Tools/SleepPSG/Sources/sleeppsg/RecipePort.swift create mode 100644 Tools/SleepPSG/Sources/sleeppsg/Scoring.swift create mode 100644 Tools/SleepPSG/Sources/sleeppsg/SleepAccel.swift create mode 100644 Tools/SleepPSG/Sources/sleeppsg/Variants.swift create mode 100644 Tools/SleepPSG/Sources/sleeppsg/main.swift create mode 100644 Tools/SleepPSG/Tests/sleeppsgTests/DatasetAndAblationTests.swift create mode 100644 Tools/SleepPSG/Tests/sleeppsgTests/PortValidationTests.swift create mode 100644 Tools/SleepPSG/Tests/sleeppsgTests/ScoringTests.swift diff --git a/Tools/SleepPSG/Package.resolved b/Tools/SleepPSG/Package.resolved new file mode 100644 index 0000000000..fd559f8d7e --- /dev/null +++ b/Tools/SleepPSG/Package.resolved @@ -0,0 +1,14 @@ +{ + "pins" : [ + { + "identity" : "grdb.swift", + "kind" : "remoteSourceControl", + "location" : "https://github.com/groue/GRDB.swift.git", + "state" : { + "revision" : "2cf6c756e1e5ef6901ebae16576a7e4e4b834622", + "version" : "6.29.3" + } + } + ], + "version" : 2 +} diff --git a/Tools/SleepPSG/Package.swift b/Tools/SleepPSG/Package.swift new file mode 100644 index 0000000000..a2718485f9 --- /dev/null +++ b/Tools/SleepPSG/Package.swift @@ -0,0 +1,33 @@ +// swift-tools-version:5.9 +import PackageDescription + +// SleepPSG — score `SleepStagerV2` against POLYSOMNOGRAPHY truth. +// +// `Tools/SleepBench` replays the stagers against the references a wearer's own database carries: their +// manual restages, the strap's band `sleep_state`, Apple Health. Every one of those is either the recipe's +// own output fed back to it, or another vendor's black box. None of them is truth. +// +// This tool supplies the missing reference. It replays the SHIPPED `SleepStagerV2` over PhysioNet +// `sleep-accel` (Walch et al., SLEEP 2019) — 31 subjects of wrist accelerometer + heart rate with +// concurrent, human-scored PSG hypnograms — and scores it epoch-for-epoch against those hypnograms. +// +// The dataset is NEVER committed and is ALWAYS a command-line argument (`--dataset`). See README.md for +// the download step and the licence attribution both the dataset and its companion code require. +// +// Nothing here reads or writes a NOOP database, and no health data of any wearer's lives in this repo. +let package = Package( + name: "sleeppsg", + platforms: [.macOS(.v13)], + dependencies: [ + .package(path: "../../Packages/StrandAnalytics"), + .package(path: "../../Packages/WhoopProtocol"), + ], + targets: [ + .executableTarget(name: "sleeppsg", dependencies: ["StrandAnalytics", "WhoopProtocol"]), + // The port-equivalence check and every scoring primitive are pure functions — they need neither the + // dataset nor a database, so `swift test` here runs anywhere the packages build, with no arguments + // and no downloads. That is deliberate: the check that keeps this harness honest (the recipe port + // reproducing the shipped stager label-for-label) has to run in CI, where the dataset never exists. + .testTarget(name: "sleeppsgTests", dependencies: ["sleeppsg"]), + ] +) diff --git a/Tools/SleepPSG/README.md b/Tools/SleepPSG/README.md new file mode 100644 index 0000000000..4495f78c62 --- /dev/null +++ b/Tools/SleepPSG/README.md @@ -0,0 +1,191 @@ +# SleepPSG — scoring `SleepStagerV2` against polysomnography + +`Tools/SleepBench` replays NOOP's sleep stagers against the references a wearer's own database happens to +carry: their manual restages, the strap's band `sleep_state`, Apple Health. Every one of those is either the +recipe's own output handed back to it, or another vendor's black box. None of them is truth, and the sleep +work has repeatedly had to say so in its own commit messages. + +This tool supplies the missing reference. It replays the **shipped** `SleepStagerV2` over PhysioNet +`sleep-accel` — wrist accelerometer and heart rate recorded alongside **human-scored PSG hypnograms** — and +scores it epoch-for-epoch against those hypnograms. + +Everything here is read-only. The dataset is never committed, never modified, and is always a command-line +argument. No NOOP database is opened and no wearer's health data is involved at any point. + +--- + +## Get the dataset + +~577 MB compressed. Put it anywhere **outside** this repository. + +```bash +mkdir -p ~/datasets && cd ~/datasets +curl -L -o sleep-accel.zip \ + https://physionet.org/static/published-projects/sleep-accel/motion-and-heart-rate-from-a-wrist-worn-wearable-and-labeled-sleep-from-polysomnography-1.0.0.zip +unzip -q sleep-accel.zip +``` + +The archive expands to a long-named directory containing `labels/`, `heart_rate/`, `motion/` and `steps/`. +Pass either that directory or its parent — `--dataset` finds the level with `labels/` in it. + +## Run it + +```bash +cd Tools/SleepPSG +swift run -c release sleeppsg --dataset ~/datasets/motion-and-heart-rate-*-1.0.0 +``` + +| flag | meaning | +|---|---| +| `--dataset PATH` | the extracted `sleep-accel` root. Required for every section except `port`. | +| `--section S` | `all` (default), or one of `port`, `baseline`, `strata`, `rem`, `variants`. | +| `--subjects N` | score only the first N subject ids — a fast smoke run. | +| `--csv PATH` | write the per-variant table as CSV. | +| `--seed N` | the port-validation corpus seed. | + +`--section port` needs no dataset at all: it is the equivalence check described below, and it is also what +`swift test` runs. + +--- + +## Attribution — required by both licences + +**Dataset.** Walch O, Huang Y, Forger D, Goldstein C. *Sleep stage prediction with raw acceleration and +photoplethysmography heart rate data derived from a consumer wearable device.* **SLEEP** 42(12), zsz180 +(2019). Distributed on PhysioNet as `sleep-accel` v1.0.0 under the **Open Data Commons Attribution License +v1.0**, which permits use and redistribution and *requires* attribution. + +**Companion code.** `ojwalch/sleep_classifiers`, **MIT**. Not vendored here; cited because it is the +reference implementation of the dataset's own baselines, and because anyone reproducing this work will want +it. The comparison models in section 5 of the report are written from scratch in this tool and are not +derived from that code. + +**PhysioNet.** Goldberger AL et al. *PhysioBank, PhysioToolkit, and PhysioNet.* Circulation 101(23):e215 +(2000). + +--- + +## What the report contains + +### 1. Port validation — why you can trust the variant rows + +Every **baseline** number comes from `StrandAnalytics.SleepStagerV2.stageSession` directly: the file that +ships in the app, compiled from `Packages/`, with no reimplementation in between. + +The **variant** rows cannot work that way. `SleepStagerV2` holds its constants as `static let`s and keeps +`Epoch`/`features()` internal to `StrandAnalytics`, so asking "what would this recipe score with one +transition row changed?" means either rebuilding the app for each question or restating the recipe with its +knobs exposed. This tool does the second, in `RecipePort.swift`. + +That is the classic way a benchmark quietly starts measuring something else — someone edits the shipped +file, nobody edits the port. So equivalence is not reviewed, it is **measured, on every build**: + +- 48 randomised nights (varying length, off-grid window starts, channel dropout, arousals, motion bursts, + unsorted input, R-R present on most so the RSA term is exercised — the PhysioNet data cannot exercise it) +- plus 7 degenerate cases: all-HR-missing, zero-variance HR, 1-epoch, 2-epoch, saturated motion gate, + no-coverage-at-all, and motion-absent +- both paths stage every night; **every epoch label must agree** + +`swift test` runs this, so CI runs it, with no dataset present. When `RecipeConfig.shipped` and +`SleepStagerV2.swift` diverge, the test fails and names the night and epoch. **A failure here is never +flakiness.** It means the two have drifted and the fix is to update `RecipeConfig.shipped`. + +### 2–3. Dataset summary, and the shipped recipe against PSG + +Four-class agreement (wake / light / deep / REM, with N1+N2 → light and N3 → deep), per-stage +precision/recall/F1, and **stage fractions as a first-class result**. + +Stage fractions are not an appendix here, for a specific reason: **kappa does not constrain them.** PR #348 +improved kappa on all three of its benchmarks and was reverted 48 hours later for re-scoring a healthy night +from 6 % to 23 % awake — *"kappa doesn't guard stage-fraction calibration."* A harness that reported only +kappa could not have seen that regression, and this one is built so it cannot repeat the mistake. + +`SleepStagerV2` fits nothing to data — every coefficient is fixed a priori — so there is no train/test split +to make for the recipe itself. The leave-one-subject-out machinery exists for the fitted comparison models +in section 5, which do need it. + +### 4. Stratified by night length + +Pooling hides couplings. `SleepStagerV2` has been observed to emit more REM as a fraction of sleep the +longer the night runs (corr +0.579 on one wearer's nights), traced to the REM-latency guard being a fixed +60-minute penalty — a third of a short night, a tenth of a long one. Subjects are split into terciles by +scored night length, and the correlation is reported for the prediction **and for PSG truth**: a coupling +truth also carries is physiology, one only the prediction carries is an artefact. + +### 5. Is REM detection physiology, or is it the clock? + +REM rises through the night in every population, so a model given nothing but "how far into the night is it" +already scores respectably. If NOOP's REM output were mostly that, the honest description of the feature +would change. + +Three models — clock features alone, physiology features alone, both — fit **leave-one-SUBJECT-out**. + +**Never an epoch-level split.** Consecutive 30 s epochs from one night are about as independent as +consecutive frames of a film; a random epoch split puts a subject's 02:14 epoch in train and their 02:14:30 +epoch in test, and every model then scores near-perfectly by memorising the subject. Each fold holds out one +whole subject. The decision threshold is chosen on the *training* folds and applied unchanged to the +held-out subject — REM is a ~20 % minority class, so a fixed 0.5 would under-predict it unequally across +three models with different calibration, and tuning on the test fold would be the leak the design exists to +avoid. + +### 6. Recipe variants + +Each row is the shipped recipe with **one** named change: PR #987's awake-transition row, and each of PR +#348's seven components measured alone, plus all seven together. Reported with kappa **and** the stage- +fraction biases, because a component is an improvement only if it wins the first without wrecking the +second. + +The last row is not a candidate. `pre-#930 REM guard (provenance)` runs the REM-latency guard as it stood +before PR #930 — a hard `c < 0.12 ? 3.0 : 0.0` step in the session-fraction domain — and exists to explain +the self-check below. + +--- + +## The self-check, and what it found + +This tool replaces one that was deleted. Section 3 prints its measured values beside the figures that +harness reported on this same dataset, so anyone can see at a glance whether the rebuild is the same +instrument. **Nothing here is tuned to those numbers**, and where they disagree the disagreement is +reported rather than closed. + +Every **truth-side** figure reproduces exactly: 31 subjects, **26 773** PSG-scored epochs, deep **14.76 %** +of night, truth median first-REM latency **88.5 min**. Predicted deep lands at **19.24 %** against a +reported 19.25 %, and four-class kappa at **0.356** against a reported 0.349. + +The prediction side of REM does not match: more REM (27.0 % vs 20.8 %), REM F1 0.569 vs 0.515, and a first +REM period arriving much earlier (median 91.5 min vs 142.0). + +**The obvious explanation was tested and is wrong.** PR #930 replaced a fraction-domain `c < 0.12 ? 3.0 : 0` +step with a graded penalty measured in minutes from sleep onset, and the old figures were reported while +#930 was the candidate — so the incumbent they describe is the recipe *before* it. That predicts exactly +this pattern. The `pre-#930` variant runs that guard, and it moves nothing: kappa 0.356 → 0.356, REM F1 +0.569 → 0.569, REM 27.02 % → 26.97 %. On 8-hour lab nights the guard barely binds either way. The +hypothesis is falsified, and it is recorded here rather than deleted because the variant is what falsified +it. + +What the residual is remains **unresolved**, and the tool is not tuned to close it. Two constraints on any +future explanation, both from the table above: + +- It cannot be the session window. `clock` drives the deep prior and the REM ramp alike; predicted deep + reproduces to 0.01 pp, so `clock` is the same quantity in both harnesses. +- It is REM-specific and it is not the latency guard, which the variant just ruled out. + +One suggestive coincidence, offered as an observation and not a conclusion: the previously reported "REM +F1 0.515" is exactly this harness's measured REM **precision** (0.515), against a measured F1 of 0.569. + +--- + +## What this benchmark is not entitled to claim + +- **The R-R stream is empty.** `sleep-accel` carries no beat-to-beat intervals, so `respRegularity` returns + nil on every epoch, `zrg` collapses to a constant 0, and the recipe's RSA respiration term contributes + exactly 0.0 to every emission. Five of the recipe's six inputs are live here; the sixth is silent. Results + are a lower bound on the same recipe with an R-R stream present. +- **Different hardware.** This is an Apple Watch's ~50 Hz raw accelerometer collapsed to a per-second mean, + not a WHOOP gravity decode. The absolute scale differs — which is precisely the thing `SleepStagerV2` was + built not to depend on, since every motion threshold is a multiple of the night's own median per-second + jerk. A benchmark on foreign hardware is a test of that claim, not a violation of it. +- **Staging only.** The session window handed to the stager is the labelled span. NOOP's own session + detection is not exercised, and nothing here says anything about it. +- **n = 31 subjects**, in a sleep-lab setting, wearing a device on the wrist alongside PSG leads. That is + more independent subjects than any other reference NOOP has, and it is still 31 people. diff --git a/Tools/SleepPSG/Sources/sleeppsg/Ablation.swift b/Tools/SleepPSG/Sources/sleeppsg/Ablation.swift new file mode 100644 index 0000000000..7e68f2ced2 --- /dev/null +++ b/Tools/SleepPSG/Sources/sleeppsg/Ablation.swift @@ -0,0 +1,220 @@ +import Foundation + +// Ablation — the leave-one-SUBJECT-out model comparison that answers "is REM detection reading physiology, +// or is it reading the clock?" +// +// The question is not rhetorical. REM rises through the night in every population, so a model given nothing +// but "how far into the night is it" already scores respectably. If NOOP's REM output were mostly that, the +// per-epoch cardiac and movement terms would be decoration and the honest description of the feature would +// change. The test is to fit three models on the same epochs and compare: clock features alone, physiology +// features alone, and both. +// +// LEAVE-ONE-SUBJECT-OUT, never epoch-level. Consecutive 30 s epochs from one night are about as +// independent as consecutive frames of a film: a random epoch split puts a subject's 02:14 epoch in train +// and their 02:14:30 epoch in test, and every model then scores near-perfectly by memorising the subject. +// Each fold here holds out one whole subject, fits on the other 30, and is scored only on the held-out one. +// +// The decision threshold is chosen ON THE TRAINING FOLDS (the value maximising training F1) and then +// applied unchanged to the held-out subject. A fixed 0.5 would not be neutral: REM is ~20 % of a night, so +// 0.5 systematically under-predicts a minority class, and it would do so unequally across three models with +// different calibration. Tuning it on the test fold would be the leak this whole design exists to avoid. + +/// One epoch's evidence, already reduced to the columns the recipe itself sees. +struct AblationRow { + let subject: String + let isRem: Bool + /// Time-of-night fraction and its square — enough for a clock model to express a rise, a fall, or a + /// hump, so "the clock" is not strawmanned as a straight line. + let clock: [Double] + /// Per-night z-scored heart rate, heart-rate variability, movement, and the HR-flatness percentile — + /// the same normalised quantities `stageEpochs` builds its emissions from. + let physiology: [Double] +} + +enum AblationModel: String, CaseIterable { + case clock = "clock-only" + case physiology = "physiology-only" + case both = "both" + + func features(_ r: AblationRow) -> [Double] { + switch self { + case .clock: return r.clock + case .physiology: return r.physiology + case .both: return r.clock + r.physiology + } + } +} + +struct AblationResult { + let model: AblationModel + /// F1 over every held-out epoch pooled — one number for the cohort. + let pooledF1: Double + /// Mean of the per-subject F1s — one number per subject, so a long night cannot outvote a short one. + let meanSubjectF1: Double + let subjectF1: [String: Double] + let pooledPrecision: Double + let pooledRecall: Double +} + +enum Ablation { + + /// Fit and score all three models leave-one-subject-out. + /// + /// The design matrix is built ONCE per model as a flat `[Double]` with the intercept column baked in, + /// and each fold indexes into it. Filtering the row structs per fold instead — and rebuilding a feature + /// array per row per IRLS iteration — is tens of millions of allocations across 3 models × 31 folds × + /// 30 iterations, which in an unoptimised build is the difference between a minute and an afternoon. + /// The arithmetic is unchanged; only the bookkeeping is. + static func run(_ rows: [AblationRow]) -> [AblationResult] { + let subjects = Array(Set(rows.map { $0.subject })).sorted() + guard let first = rows.first else { return [] } + let y = rows.map { $0.isRem } + var out: [AblationResult] = [] + for model in AblationModel.allCases { + let p = model.features(first).count + 1 // + intercept column + var X = [Double](repeating: 1, count: rows.count * p) + for (i, r) in rows.enumerated() { + let fs = model.features(r) + for j in 0..= thr } + let ref = testIdx.map { y[$0] } + perSubject[held] = binaryF1(ref: ref, pred: pred) + pooledRef.append(contentsOf: ref) + pooledPred.append(contentsOf: pred) + } + var tp = 0, fp = 0, fn = 0 + for i in 0.. Double { 1.0 / (1.0 + exp(-max(-40, min(40, z)))) } + + static func dot(_ w: [Double], _ x: [Double]) -> Double { + var s = w[0] // intercept + for i in 0.. Double { + var s = 0.0 + let base = row * p + for j in 0..

[Double] { + var w = [Double](repeating: 0, count: p) + guard !rowIdx.isEmpty, p > 0 else { return w } + var H = [Double](repeating: 0, count: p * p) + var g = [Double](repeating: 0, count: p) + for _ in 0.. [Double] { + guard let first = X.first else { return [0] } + let p = first.count + 1 + var flat = [Double](repeating: 1, count: X.count * p) + for (i, row) in X.enumerated() { for j in 0.. [Double]? { + var a = a0, b = b0 + let n = b.count + for c in 0.. abs(a[piv][c]) { piv = r } + if abs(a[piv][c]) < 1e-12 { return nil } + if piv != c { a.swapAt(piv, c); b.swapAt(piv, c) } + for r in (c + 1).. Double { + var best = 0.5, bestF1 = -1.0 + var t = 0.02 + while t < 1.0 { + let pred = scores.map { $0 >= t } + let f1 = binaryF1(ref: truth, pred: pred) + if f1 > bestF1 { bestF1 = f1; best = t } + t += 0.02 + } + return best + } +} diff --git a/Tools/SleepPSG/Sources/sleeppsg/PortValidation.swift b/Tools/SleepPSG/Sources/sleeppsg/PortValidation.swift new file mode 100644 index 0000000000..8cd2877ba7 --- /dev/null +++ b/Tools/SleepPSG/Sources/sleeppsg/PortValidation.swift @@ -0,0 +1,303 @@ +import Foundation +import StrandAnalytics +import WhoopProtocol + +// PortValidation — the check that entitles this harness to say its variant numbers describe NOOP's shipped +// recipe rather than a lookalike. +// +// `RecipePort.swift` re-states `SleepStagerV2` with its constants exposed. That is the only way to ask +// "what would this recipe score with one constant moved", and it is also the classic way a benchmark +// quietly starts measuring something else: someone edits the shipped file, nobody edits the port, and every +// number the tool prints afterwards is about code that never ran on anyone's wrist. +// +// So the port is not reviewed for equivalence — it is MEASURED for it, on every build. Both paths stage the +// same nights and every epoch label must agree. The nights are generated here rather than loaded, for two +// reasons: the check has to run in CI where no dataset and no health data exist, and generated nights can +// be aimed at the paths a real night rarely reaches (a stream with no heart rate at all, a night whose +// motion gate fires on every epoch, a two-epoch nap). +// +// A failure here is never "the test is flaky". It means `RecipeConfig.shipped` and +// `Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift` have diverged; the report names +// the night and the first epoch that differs. + +/// SplitMix64 — a tiny deterministic PRNG so the generated corpus is identical on every machine and every +/// run. A benchmark whose validation set moves between runs cannot tell a real divergence from a reroll. +struct SplitMix64: RandomNumberGenerator { + private var state: UInt64 + init(seed: UInt64) { state = seed } + mutating func next() -> UInt64 { + state &+= 0x9E37_79B9_7F4A_7C15 + var z = state + z = (z ^ (z >> 30)) &* 0xBF58_476D_1CE4_E5B9 + z = (z ^ (z >> 27)) &* 0x94D0_49BB_1331_11EB + return z ^ (z >> 31) + } + mutating func double(_ lo: Double, _ hi: Double) -> Double { Double.random(in: lo.. Int { Int.random(in: lo...hi, using: &self) } + mutating func chance(_ p: Double) -> Bool { Double.random(in: 0..<1, using: &self) < p } +} + +/// One synthetic night: the exact arguments a `stageSession` call takes, plus a name for the report. +struct SyntheticNight { + let name: String + let start: Int + let end: Int + let grav: [GravitySample] + let hr: [HRSample] + let rr: [RRInterval] + let resp: [RespSample] +} + +enum PortValidation { + + /// The pinned corpus seed. Fixed rather than time- or machine-derived so that a divergence is always a + /// code change and never a reroll — and so a failure reported by CI reproduces exactly on a laptop. + static let defaultSeed: UInt64 = 0x5EED_5_1EE9_0001 + + /// The corpus: 48 randomised nights plus the degenerate cases. Deterministic given `seed`. + static func corpus(seed: UInt64 = PortValidation.defaultSeed, randomNights: Int = 48) -> [SyntheticNight] { + if seed == defaultSeed && randomNights == 48 { return pinnedCorpus } + return build(seed: seed, randomNights: randomNights) + } + + /// The default corpus, built once. Generating it is not free — a few million RNG draws per night in a + /// debug build — and several tests want the same nights, so building it per call would put minutes of + /// pure setup into `swift test`. A `static let` is initialised lazily and exactly once, thread-safely. + static let pinnedCorpus = build(seed: defaultSeed, randomNights: 48) + + private static func build(seed: UInt64, randomNights: Int) -> [SyntheticNight] { + var g = SplitMix64(seed: seed) + var out: [SyntheticNight] = [] + for i in 0.. SyntheticNight { + // ~20 min to ~3.7 h. Sized so the whole corpus lands in the same order of magnitude as a night's + // worth of epochs per subject without making `swift test` slow: the RSA term runs a band-limited + // DFT per epoch, and this check stages every night TWICE. + let nEpochs = g.int(40, 440) + // Deliberately misalign some windows from the 30 s wall-clock grid: `features` starts at the first + // multiple of 30 at or after `start`, and an off-grid start is where a tiling bug would show. + let start = 1_700_000_000 + g.int(0, 5000) * 30 + (g.chance(0.4) ? g.int(1, 29) : 0) + let end = start + nEpochs * 30 + (g.chance(0.5) ? g.int(0, 29) : 0) + + let haveHR = g.chance(0.9) + let haveRR = g.chance(0.75) + let haveGrav = g.chance(0.95) + let hrDropout = g.double(0.0, 0.55) + let gravDropout = g.double(0.0, 0.45) + + // A slow HR baseline with a couple of arousals, so the 5- and 11-min flatness windows see real + // structure rather than white noise. + let hrBase = g.double(44, 68) + let hrDrift = g.double(-6, 6) + let arousals = (0..= ibi { + rrCarry -= ibi + rr.append(RRInterval(ts: t, rrMs: Int((ibi * 1000).rounded()))) + } + } + } + } + // Hand some nights in unsorted: both paths sort defensively, and "defensively" is worth pinning. + if g.chance(0.25) { hr.shuffle(using: &g) } + if g.chance(0.25) { grav.shuffle(using: &g) } + return SyntheticNight(name: "random-\(index)", start: start, end: end, + grav: grav, hr: hr, rr: rr, resp: []) + } + + // MARK: - Degenerate nights + + /// The seven shapes a real night reaches rarely or never, each of which lands on a branch with its own + /// early return, fallback constant, or divide-by-zero guard. + static func degenerateNights() -> [SyntheticNight] { + let base = 1_700_000_000 + var out: [SyntheticNight] = [] + + // 1. All HR missing — every HR-derived feature is nil, the prefix grid is empty, and `stdOfSeconds` + // returns nil for every window. Only motion may speak. + do { + let n = 120 + var grav: [GravitySample] = [] + for s in 0..<(n * 30) { + let a = Double(s) * 0.0007 + grav.append(GravitySample(ts: base + s, x: cos(a), y: sin(a), z: 0.05)) + } + out.append(SyntheticNight(name: "degenerate-all-hr-missing", start: base, end: base + n * 30, + grav: grav, hr: [], rr: [], resp: [])) + } + + // 2. Zero-variance HR — `zfun`'s std is 0, which it must treat as 1.0 rather than dividing by it, + // and every flatness window ties, so the percentile rank is degenerate too. + do { + let n = 120 + var grav: [GravitySample] = [], hr: [HRSample] = [] + for s in 0..<(n * 30) { + hr.append(HRSample(ts: base + s, bpm: 55)) + let a = Double(s) * 0.0011 + grav.append(GravitySample(ts: base + s, x: cos(a), y: sin(a), z: 0.02)) + } + out.append(SyntheticNight(name: "degenerate-zero-variance-hr", start: base, end: base + n * 30, + grav: grav, hr: hr, rr: [], resp: [])) + } + + // 3. One epoch — the Viterbi lattice never takes a transition, and the segment tiler's first and + // last epoch are the same epoch. + do { out.append(shortNight(name: "degenerate-1-epoch", base: base, epochs: 1)) } + + // 4. Two epochs — exactly one transition, the smallest lattice that uses the transition matrix. + do { out.append(shortNight(name: "degenerate-2-epoch", base: base, epochs: 2)) } + + // 5. Saturated motion gate — every epoch's peak jerk clears the night-relative gate, so the wake + // boost is applied everywhere and the night-relative floor is at its most fragile. + do { + let n = 90 + var grav: [GravitySample] = [], hr: [HRSample] = [] + for s in 0..<(n * 30) { + hr.append(HRSample(ts: base + s, bpm: 58 + (s % 7))) + // Alternate poles: consecutive per-second jerks are ~2 g, orders above any median floor. + let f = s % 2 == 0 ? 1.0 : -1.0 + grav.append(GravitySample(ts: base + s, x: f, y: 0, z: 0)) + } + out.append(SyntheticNight(name: "degenerate-saturated-motion", start: base, end: base + n * 30, + grav: grav, hr: hr, rr: [], resp: [])) + } + + // 6. No coverage at all — `features` returns empty and the recipe must fall back to a single + // light-labelled segment spanning the window rather than returning nothing. + do { + out.append(SyntheticNight(name: "degenerate-no-coverage", start: base, end: base + 3600, + grav: [], hr: [], rr: [], resp: [])) + } + + // 7. Motion absent, heart rate only — no jerks anywhere, so the quiescent floor falls back to its + // 1e-6 epsilon and `motionQuiescent` is true on every epoch. + do { + let n = 100 + var hr: [HRSample] = [] + for s in 0..<(n * 30) { hr.append(HRSample(ts: base + s, bpm: Int(52 + 9 * sin(Double(s) * 0.002)))) } + out.append(SyntheticNight(name: "degenerate-no-motion", start: base, end: base + n * 30, + grav: [], hr: hr, rr: [], resp: [])) + } + return out + } + + private static func shortNight(name: String, base: Int, epochs: Int) -> SyntheticNight { + var grav: [GravitySample] = [], hr: [HRSample] = [] + for s in 0..<(epochs * 30) { + hr.append(HRSample(ts: base + s, bpm: 56 + (s % 5))) + let a = Double(s) * 0.02 + grav.append(GravitySample(ts: base + s, x: cos(a), y: sin(a), z: 0.1)) + } + return SyntheticNight(name: name, start: base, end: base + epochs * 30, + grav: grav, hr: hr, rr: [], resp: []) + } + + // MARK: - The check + + struct Divergence { + let night: String + let epochIndex: Int + let shipped: String + let port: String + } + + struct Result { + var nights = 0 + var randomNights = 0 + var degenerateNights = 0 + var epochs = 0 + var matchingEpochs = 0 + var divergences: [Divergence] = [] + var ok: Bool { divergences.isEmpty && epochs > 0 && matchingEpochs == epochs } + } + + /// Stage every night through BOTH paths and compare epoch labels on the 30 s grid. + /// + /// Epoch labels rather than the raw `[StageSegment]` array, because that is the unit every score in this + /// harness is computed on — an equivalence that held only after run-length collapsing would not + /// guarantee the numbers agree. (The segment arrays do also match; comparing labels is the stricter + /// statement for our purposes and the weaker one to state, so it is what the report claims.) + static func run(seed: UInt64 = PortValidation.defaultSeed, randomNights: Int = 48) -> Result { + let nights = corpus(seed: seed, randomNights: randomNights) + // Nights are staged concurrently: staging is pure, and the shipped stager's memo cache is itself + // lock-guarded (`AnalyticsMemoCache`), so the only shared state is safe. Serially this check runs + // for minutes in a debug build — the recipe's RSA term is a per-epoch DFT and every night is staged + // twice — and a validation nobody wants to wait for is a validation that gets skipped. + var perNight = [(epochs: Int, matching: Int, divergences: [Divergence])]( + repeating: (0, 0, []), count: nights.count) + perNight.withUnsafeMutableBufferPointer { buf in + let out = buf + DispatchQueue.concurrentPerform(iterations: nights.count) { idx in + let n = nights[idx] + let shipped = SleepStagerV2.stageSession(start: n.start, end: n.end, grav: n.grav, + hr: n.hr, rr: n.rr, resp: n.resp) + let port = V2Recipe.stageSession(start: n.start, end: n.end, grav: n.grav, + hr: n.hr, rr: n.rr, resp: n.resp, cfg: .shipped) + let a = epochLabels(shipped, start: n.start, end: n.end) + let b = epochLabels(port, start: n.start, end: n.end) + var epochs = 0, matching = 0 + var divs: [Divergence] = [] + for i in 0.. [StageSegment] { + let gravW = clipToWindow(grav, lo: start - padLo, hi: end + padHi, ts: { $0.ts }) + let hrW = clipToWindow(hr, lo: start - padLo, hi: end + padHi, ts: { $0.ts }) + let rrW = clipToWindow(rr, lo: start - padLo, hi: end + padHi, ts: { $0.ts }) + + let gravS = gravW.sorted { $0.ts < $1.ts } + let hrS = hrW.sorted { $0.ts < $1.ts } + let rrS = rrW.sortedByTsStable() + + let feats = features(start: start, end: end, grav: gravS, hr: hrS, rr: rrS, cfg: cfg) + if feats.isEmpty { return [StageSegment(start: start, end: end, stage: "light")] } + let labels = stageEpochs(feats, cfg: cfg) + + var segments: [StageSegment] = [] + for (i, f) in feats.enumerated() { + let stage = labels[i] == "awake" ? "wake" : labels[i] + let segStart = i == 0 ? start : f.start + let segEnd = i == feats.count - 1 ? end : feats[i + 1].start + if let last = segments.last, last.stage == stage { + segments[segments.count - 1].end = segEnd + } else { + segments.append(StageSegment(start: segStart, end: segEnd, stage: stage)) + } + } + return segments + } + + /// Stage `[start, end]` and return one label per FEATURISED epoch, alongside those epochs. The scoring + /// paths want the epoch grid the recipe actually built (an epoch with no coverage at all is skipped by + /// `features`, and a scorer that assumed a dense grid would silently shift every later label). + static func stageEpochsDetailed(start: Int, end: Int, grav: [GravitySample], hr: [HRSample], + rr: [RRInterval], resp: [RespSample], + cfg: RecipeConfig = .shipped) -> (epochs: [Epoch], labels: [String]) { + let gravW = clipToWindow(grav, lo: start - padLo, hi: end + padHi, ts: { $0.ts }) + let hrW = clipToWindow(hr, lo: start - padLo, hi: end + padHi, ts: { $0.ts }) + let rrW = clipToWindow(rr, lo: start - padLo, hi: end + padHi, ts: { $0.ts }) + let feats = features(start: start, end: end, + grav: gravW.sorted { $0.ts < $1.ts }, + hr: hrW.sorted { $0.ts < $1.ts }, + rr: rrW.sortedByTsStable(), cfg: cfg) + return (feats, stageEpochs(feats, cfg: cfg)) + } + + private static func clipToWindow(_ samples: [T], lo: Int, hi: Int, ts: (T) -> Int) -> [T] { + if samples.isEmpty { return samples } + if ts(samples[0]) >= lo && ts(samples[samples.count - 1]) < hi { return samples } + var l = 0, h = samples.count + while l < h { let m = (l + h) / 2; if ts(samples[m]) < lo { l = m + 1 } else { h = m } } + let start = l + l = start; h = samples.count + while l < h { let m = (l + h) / 2; if ts(samples[m]) < hi { l = m + 1 } else { h = m } } + if start == 0 && l == samples.count { return samples } + return Array(samples[start.. [Epoch] { + if end <= start { return [] } + let span = Double(max(1, end - start)) + + var hrSum = [Int: Double](), hrCnt = [Int: Int]() + for s in hr { hrSum[s.ts, default: 0] += Double(s.bpm); hrCnt[s.ts, default: 0] += 1 } + var secHR = [Int: Double](); secHR.reserveCapacity(hrSum.count) + for (k, v) in hrSum { secHR[k] = v / Double(hrCnt[k]!) } + + var gxSum = [Int: Double](), gySum = [Int: Double](), gzSum = [Int: Double](), gCnt = [Int: Int]() + for g in grav { + gxSum[g.ts, default: 0] += g.x; gySum[g.ts, default: 0] += g.y + gzSum[g.ts, default: 0] += g.z; gCnt[g.ts, default: 0] += 1 + } + var secG = [Int: (Double, Double, Double)](); secG.reserveCapacity(gCnt.count) + for (k, c) in gCnt { let d = Double(c); secG[k] = (gxSum[k]! / d, gySum[k]! / d, gzSum[k]! / d) } + + var rrBy = [Int: [Double]]() + for r in rr { rrBy[r.ts, default: []].append(Double(r.rrMs)) } + + let gridLo = secHR.keys.min() + let gridHi = secHR.keys.max() + var pCnt = [Int](), pSum = [Double](), pSq = [Double]() + if let g0 = gridLo, let g1 = gridHi { + let n = g1 - g0 + 1 + pCnt = [Int](repeating: 0, count: n + 1) + pSum = [Double](repeating: 0, count: n + 1) + pSq = [Double](repeating: 0, count: n + 1) + for i in 0.. Double? { + guard let g0 = gridLo, let g1 = gridHi else { return nil } + let a = max(lo, g0) - g0 + let b = min(hi, g1 + 1) - g0 + if b <= a { return nil } + let cnt = pCnt[b] - pCnt[a] + if cnt < 2 { return nil } + let n = Double(cnt) + let sv = pSum[b] - pSum[a] + let sq = pSq[b] - pSq[a] + let m = sv / n + let v = (sq - 2 * m * sv + n * m * m) / n + return (v < 0 ? 0 : v).squareRoot() + } + + struct Raw { + let start: Int; let hr: Double?; let hrVar: Double?; let hrFlat11: Double? + let jerks: [Double]; let gapSec: Int; let jerkMax: Double; let respReg: Double?; let clock: Double + let minutes: Double + } + var raws: [Raw] = [] + var allJerks: [Double] = [] + let firstE = ((start + 29) / 30) * 30 + var e = firstE + while e < end { + var hrs: [Double] = [] + var gseq: [(Double, Double, Double)] = [] + for s in e..<(e + 30) { + if let h = secHR[s] { hrs.append(h) } + if let g = secG[s] { gseq.append(g) } + } + if hrs.isEmpty && gseq.isEmpty { e += 30; continue } + + var jerks: [Double] = [] + for i in 1.. moveThr ? 1 : 0) } + feats.append(Epoch( + start: r.start, hr: r.hr, hrVar: r.hrVar, hrFlat11: r.hrFlat11, + moveFrac: Double(moves) / Double(r.gapSec), jerkMax: r.jerkMax, respReg: r.respReg, + clock: r.clock, jerkScale: jerkScale, minutesSinceOnset: r.minutes)) + } + return feats + } + + static func respRegularity(_ beats: [(Double, Double)]) -> Double? { + if beats.count < 12 { return nil } + let t0 = beats.first!.0, tN = beats.last!.0 + if tN <= t0 { return nil } + let n = Int(ceil((tN - t0) / 0.25 - 1e-9)) + if n < 16 { return nil } + + var y = [Double](repeating: 0, count: n) + var seg = 0 + for i in 0.. maxP { maxP = p } + } + if sumP == 0 { return nil } + return maxP / sumP + } + + // MARK: - Staging + + /// Awake-emission dead-zone. Identity when `awakeDeadzone <= 0`, which is the shipped state. + static func dz(_ z: Double, _ deadzone: Double) -> Double { + if deadzone <= 0.0 { return z } + if z > deadzone { return z - deadzone } + if z < -deadzone { return z + deadzone } + return 0.0 + } + + static func motionQuiescent(_ f: Epoch, _ cfg: RecipeConfig) -> Bool { + f.moveFrac <= 0.0 && f.jerkMax <= f.jerkScale * cfg.jerkFloorGateMult + } + + static func remLatencyGuard(_ minutesSinceOnset: Double, _ cfg: RecipeConfig) -> Double { + cfg.remLatencyPenalty * min(1.0, max(0.0, 1.0 - minutesSinceOnset / cfg.remLatencyMinutes)) + } + + static func cyclePrior(_ c: Double, _ minutesSinceOnset: Double, _ cfg: RecipeConfig) -> [String: Double] { + ["deep": 1.2 * max(0.0, 1.0 - c / 0.55), + "rem": 1.0 * c - remLatencyGuard(minutesSinceOnset, cfg), + "light": 0.0, "awake": 0.0] + } + + static func sustainedSleepOnset(_ labels: [String], _ cfg: RecipeConfig) -> Int? { + var run = 0 + for i in labels.indices { + if labels[i] == "awake" { run = 0; continue } + run += 1 + if run >= cfg.onsetSustainedEpochs { return i - cfg.onsetSustainedEpochs + 1 } + } + return nil + } + + static func viterbi(_ emSeq: [[String: Double]], _ cfg: RecipeConfig) -> [String] { + if emSeq.isEmpty { return [] } + let logT = cfg.transition.mapValues { row in row.mapValues { log(max($0, 1e-9)) } } + var V = emSeq[0] + var back: [[String: String]] = [] + for t in 1.. bestVal { bestVal = val; bestPrev = p } + } + newV[s] = bestVal + emSeq[t][s]! + bp[s] = bestPrev + } + V = newV; back.append(bp) + } + var last = stageNames[0], lastV = V[last]! + for s in stageNames.dropFirst() where V[s]! > lastV { lastV = V[s]!; last = s } + var path = [last] + for bp in back.reversed() { last = bp[last]!; path.append(last) } + return path.reversed() + } + + static func stageEpochs(_ feats: [Epoch], cfg: RecipeConfig = .shipped) -> [String] { + if feats.isEmpty { return [] } + let prior = cfg.baseLogPrior + + func zfun(_ vals: [Double?]) -> (Double?) -> Double { + let present = vals.compactMap { $0 } + if present.isEmpty { return { _ in 0.0 } } + let m = present.reduce(0, +) / Double(present.count) + let sd0 = (present.reduce(0.0) { $0 + ($1 - m) * ($1 - m) } / Double(present.count)).squareRoot() + let sd = sd0 == 0 ? 1.0 : sd0 + return { v in v == nil ? 0.0 : (v! - m) / sd } + } + let zhr = zfun(feats.map { $0.hr }) + let zhv = zfun(feats.map { $0.hrVar }) + let zmv = zfun(feats.map { Optional($0.moveFrac) }) + let zrg = zfun(feats.map { $0.respReg }) + + let fsorted = feats.compactMap { $0.hrFlat11 }.sorted() + func fpct(_ v: Double?) -> Double { + guard let v = v, !fsorted.isEmpty else { return 0.5 } + var lo = 0, hi = fsorted.count + while lo < hi { let mid = (lo + hi) / 2; if fsorted[mid] <= v { lo = mid + 1 } else { hi = mid } } + return Double(lo) / Double(fsorted.count) + } + + var seq: [[String: Double]] = [] + seq.reserveCapacity(feats.count) + for f in feats { + let zhrv = zhr(f.hr), zhvv = zhv(f.hrVar), zmvv = zmv(f.moveFrac) + let gate = cfg.deepGateSlope * max(0.0, fpct(f.hrFlat11) - cfg.deepGateThresh) + let awakeCardiac0 = cfg.awakeZhv * dz(zhvv, cfg.awakeDeadzone) + + cfg.awakeZhr * dz(zhrv, cfg.awakeDeadzone) + let awakeCardiac = motionQuiescent(f, cfg) ? min(0.0, awakeCardiac0) : awakeCardiac0 + var em: [String: Double] = [ + "deep": cfg.deepZhv * zhvv + cfg.deepZhr * zhrv + cfg.deepZmv * zmvv - gate + prior["deep"]!, + "rem": cfg.remZhv * zhvv + cfg.remZmv * zmvv + cfg.remZhr * zhrv + prior["rem"]!, + "light": prior["light"]!, + "awake": cfg.awakeZmv * zmvv + awakeCardiac + prior["awake"]!, + ] + // In `.gradedFromOnset` the guard is DISABLED here (`.infinity` ⇒ guard = 0) and added by pass 2 + // once onset is known. In `.preNine30FractionStep` there is no onset to wait for — the step is + // a function of the session fraction alone — so it goes in now and there is no second pass. + let pr: [String: Double] + switch cfg.remLatencyMode { + case .gradedFromOnset: + pr = cyclePrior(f.clock, .infinity, cfg) + case .preNine30FractionStep: + pr = ["deep": 1.2 * max(0.0, 1.0 - f.clock / 0.55), + "rem": 1.0 * f.clock - (f.clock < 0.12 ? cfg.remLatencyPenalty : 0.0), + "light": 0.0, "awake": 0.0] + } + for s in stageNames { em[s]! += pr[s]! } + if f.jerkMax > f.jerkScale * cfg.jerkFloorGateMult { em["awake"]! += cfg.motionGateBoost } + if let rg = f.respReg { let z = zrg(rg); em["deep"]! += cfg.respWeight * z; em["rem"]! -= cfg.respWeight * z } + seq.append(em) + } + + if cfg.remLatencyMode == .preNine30FractionStep { return viterbi(seq, cfg) } + + let provisional = viterbi(seq, cfg) + let originMin = sustainedSleepOnset(provisional, cfg).map { feats[$0].minutesSinceOnset } ?? 0.0 + for i in feats.indices { + seq[i]["rem"]! -= remLatencyGuard(feats[i].minutesSinceOnset - originMin, cfg) + } + return viterbi(seq, cfg) + } +} diff --git a/Tools/SleepPSG/Sources/sleeppsg/Scoring.swift b/Tools/SleepPSG/Sources/sleeppsg/Scoring.swift new file mode 100644 index 0000000000..31ab5b22c8 --- /dev/null +++ b/Tools/SleepPSG/Sources/sleeppsg/Scoring.swift @@ -0,0 +1,229 @@ +import Foundation +import StrandAnalytics + +// Scoring — the agreement and calibration primitives, over plain label arrays. +// +// The definitions here match `Tools/SleepBench/Sources/sleepbench/Metrics.swift` deliberately: the two +// harnesses score the same recipe against different references, and a kappa that meant something slightly +// different in each would make their numbers incomparable exactly when someone needs to compare them. +// `ScoringTests` pins the shared ones against hand-computed values so a divergence is caught rather than +// discovered. +// +// The one primitive SleepBench does not carry is per-stage F1. It belongs here because the question this +// dataset exists to answer is about REM specifically, and REM is a minority class: on a night that is 20 % +// REM, a recipe that emits no REM at all still scores 80 % accuracy and a respectable four-class kappa. +// Precision and recall for that one class, and their harmonic mean, are what move when REM detection does. + +let stageOrder = ["wake", "light", "deep", "rem"] +let epochSeconds = 30.0 + +func epochCount(start: Int, end: Int) -> Int { + max(1, Int(ceil(Double(end - start) / epochSeconds))) +} + +/// Expand a `[StageSegment]` tiling into a per-epoch label array on the 30 s grid, taking each epoch's +/// label from the segment covering its START instant — the same convention `SleepBench` uses. +func epochLabels(_ stages: [StageSegment], start: Int, end: Int) -> [String] { + let n = epochCount(start: start, end: end) + var out = [String](repeating: "wake", count: n) + guard !stages.isEmpty else { return out } + let sorted = stages.sorted { $0.start < $1.start } + var si = 0 + for i in 0..= sorted[sorted.count - 1].end { out[i] = sorted[sorted.count - 1].stage } + } + return out +} + +// MARK: - Stage-fraction calibration + +/// Each stage's share of the epochs, as a percentage. +/// +/// This is the quantity Cohen's kappa does NOT constrain, and the reason it is reported beside kappa +/// everywhere in this tool rather than in an appendix. PR #348 raised kappa on all three of its benchmarks +/// and was reverted 48 h later for re-scoring a healthy night from 6 % to 23 % awake: a recipe can win more +/// epochs than it loses while systematically shifting how much of the night it spends in a stage. Every +/// stage in `stageOrder` is present at 0 when unused, so a stage a recipe stops emitting shows up as a bias +/// rather than as a missing row. +func stagePercentages(_ labels: [String]) -> [String: Double] { + var pct = [String: Double](uniqueKeysWithValues: stageOrder.map { ($0, 0.0) }) + guard !labels.isEmpty else { return pct } + for l in labels { pct[l, default: 0] += 1 } + for k in pct.keys { pct[k]! = pct[k]! / Double(labels.count) * 100 } + return pct +} + +/// Signed per-stage calibration error in percentage points, `predicted% − reference%`. +func stageBias(ref: [String], pred: [String]) -> [String: Double] { + let r = stagePercentages(ref), p = stagePercentages(pred) + return [String: Double](uniqueKeysWithValues: stageOrder.map { ($0, p[$0]! - r[$0]!) }) +} + +/// Minutes from sleep onset (first non-wake epoch) to the first REM epoch. `nil` when the hypnogram has no +/// sleep or never reaches REM — a night with no REM is a distinct outcome from one whose REM arrives at +/// minute zero, and collapsing them would let a recipe that stops emitting REM look fast. +func firstRemLatencyMinutes(_ labels: [String]) -> Double? { + guard let onset = labels.firstIndex(where: { $0 != "wake" }), + let rem = labels.firstIndex(of: "rem"), rem >= onset else { return nil } + return Double(rem - onset) * epochSeconds / 60.0 +} + +/// The same latency over a grid with unscored epochs in it. The nils are skipped when looking for onset +/// and for REM, but they still occupy their slot in the index arithmetic — so the answer stays wall-clock +/// minutes. Collapsing the array first and counting the survivors would shorten the latency by however +/// much of the night the PSG technician left unscored, silently and only for the truth column. +func firstRemLatencyMinutes(_ labels: [String?]) -> Double? { + guard let onset = labels.firstIndex(where: { $0 != nil && $0 != "wake" }), + let rem = labels.firstIndex(where: { $0 == "rem" }), rem >= onset else { return nil } + return Double(rem - onset) * epochSeconds / 60.0 +} + +// MARK: - Agreement + +/// A square confusion matrix over `classes`, rows = reference, cols = prediction. +struct Confusion { + let classes: [String] + var m: [[Int]] + init(classes: [String] = stageOrder) { + self.classes = classes + m = Array(repeating: Array(repeating: 0, count: classes.count), count: classes.count) + } + mutating func add(ref: String, pred: String) { + guard let r = classes.firstIndex(of: ref), let c = classes.firstIndex(of: pred) else { return } + m[r][c] += 1 + } + mutating func merge(_ o: Confusion) { + for i in 0.. 0 else { return .nan } + let po = accuracy + var pe = 0.0 + for i in 0..= 1.0 ? .nan : (po - pe) / (1 - pe) + } + /// One-vs-rest precision, recall and F1 for `cls`. `support` is the reference count, so a class the + /// recipe never emits is distinguishable from a class the reference never contains. + func prf(_ cls: String) -> (precision: Double, recall: Double, f1: Double, support: Int) { + guard let k = classes.firstIndex(of: cls) else { return (.nan, .nan, .nan, 0) } + var tp = 0, fn = 0, fp = 0 + for i in 0.. (sens: Double, spec: Double, n: Int) { + guard let k = classes.firstIndex(of: cls) else { return (.nan, .nan, 0) } + var tp = 0, fn = 0, fp = 0, tn = 0 + for i in 0.. Confusion { + var c = Confusion(classes: classes) + for i in 0.. [String] { labels.map { $0 == "wake" ? "wake" : "sleep" } } + +/// Binary F1 for a single positive class, over aligned label arrays. The REM ablation's headline number. +func binaryF1(ref: [Bool], pred: [Bool]) -> Double { + var tp = 0, fp = 0, fn = 0 + for i in 0.. Double { v.isEmpty ? .nan : v.reduce(0, +) / Double(v.count) } +func median(_ v: [Double]) -> Double { + guard !v.isEmpty else { return .nan } + let s = v.sorted() + return s.count % 2 == 1 ? s[s.count / 2] : (s[s.count / 2 - 1] + s[s.count / 2]) / 2 +} +func sd(_ v: [Double]) -> Double { + guard v.count > 1 else { return .nan } + let m = mean(v) + return (v.reduce(0) { $0 + ($1 - m) * ($1 - m) } / Double(v.count - 1)).squareRoot() +} +func mae(_ v: [Double]) -> Double { mean(v.map { abs($0) }) } + +/// Pearson correlation and its two-sided t statistic. +func pearson(_ x: [Double], _ y: [Double]) -> (r: Double, n: Int, t: Double) { + let n = min(x.count, y.count) + guard n > 2 else { return (.nan, n, .nan) } + let mx = mean(Array(x.prefix(n))), my = mean(Array(y.prefix(n))) + var sxy = 0.0, sxx = 0.0, syy = 0.0 + for i in 0.. 0, syy > 0 else { return (.nan, n, .nan) } + let r = sxy / (sxx * syy).squareRoot() + let t = r * (Double(n - 2) / max(1e-12, 1 - r * r)).squareRoot() + return (r, n, t) +} + +/// Spearman rank correlation — reported beside Pearson because a monotone-but-curved coupling (which is +/// what a saturating penalty produces) shows up in the ranks before it shows up in the linear fit. +func spearman(_ x: [Double], _ y: [Double]) -> Double { + let n = min(x.count, y.count) + guard n > 2 else { return .nan } + func ranks(_ v: [Double]) -> [Double] { + let idx = v.indices.sorted { v[$0] < v[$1] } + var r = [Double](repeating: 0, count: v.count) + var i = 0 + while i < idx.count { + var j = i + while j + 1 < idx.count && v[idx[j + 1]] == v[idx[i]] { j += 1 } + let avg = Double(i + j) / 2 + 1 + for k in i...j { r[idx[k]] = avg } + i = j + 1 + } + return r + } + return pearson(ranks(Array(x.prefix(n))), ranks(Array(y.prefix(n)))).r +} + +func f(_ v: Double, _ w: Int = 7, _ p: Int = 1) -> String { + v.isNaN ? String(repeating: " ", count: max(0, w - 3)) + "n/a" : String(format: "%\(w).\(p)f", v) +} diff --git a/Tools/SleepPSG/Sources/sleeppsg/SleepAccel.swift b/Tools/SleepPSG/Sources/sleeppsg/SleepAccel.swift new file mode 100644 index 0000000000..12e2f8e244 --- /dev/null +++ b/Tools/SleepPSG/Sources/sleeppsg/SleepAccel.swift @@ -0,0 +1,232 @@ +import Foundation +import WhoopProtocol + +// SleepAccel — read PhysioNet `sleep-accel` v1.0.0 and present each subject as the streams a NOOP stager +// takes, plus the PSG hypnogram to score against. +// +// Walch O, Huang Y, Forger D, Goldstein C. Sleep stage prediction with raw acceleration and photo- +// plethysmography heart rate data derived from a consumer wearable device. SLEEP 42(12), zsz180 (2019). +// +// Dataset licence: Open Data Commons Attribution License v1.0. Attribution is a condition of use; see +// README.md. The dataset is NEVER committed here and is always supplied by `--dataset`. +// +// ── The mapping, and why each choice is the honest one ──────────────────────────────────────────────── +// +// This dataset is an Apple Watch: a ~50 Hz raw accelerometer and a sparse PPG heart rate. NOOP's stager +// takes a 1 Hz gravity vector, a heart-rate stream, and R-R intervals. Three decisions bridge that, and +// each one can bias a result, so each is stated rather than buried: +// +// 1. GRAVITY ← per-second mean of the raw accelerometer. `SleepStagerV2.features` already collapses its +// gravity stream to one value per second by averaging, so pre-averaging here is not an extra filter — +// it is the same operation done earlier, and it keeps 31 subjects of 50 Hz data in memory. The absolute +// SCALE still differs from a WHOOP gravity decode, and that is exactly the property the recipe was +// built not to depend on: every motion threshold is a multiple of the night's OWN median per-second +// jerk (`jerkScale`), never an absolute g. A benchmark on foreign hardware is a test of that claim. +// +// 2. R-R ← EMPTY. The dataset carries no beat-to-beat intervals, so `respRegularity` returns nil on every +// epoch, `zrg` collapses to the constant 0, and the RSA term contributes exactly 0.0 to every emission. +// The term is therefore INERT here, not corrupted: the recipe runs with one of its six inputs silent. +// Any result below is a lower bound on what the same recipe does with an R-R stream present. +// +// 3. TIME ← the dataset's seconds-from-PSG-start, offset by a fixed multiple of 30 into positive unix +// territory. Label epochs land on multiples of 30, so the recipe's wall-clock 30 s grid coincides with +// the PSG scoring grid exactly and no epoch is compared against a label it only half overlaps. +// +// The session window handed to the stager is the LABELLED span — first scored epoch to last, plus 30 s. +// NOOP's own session detection is not exercised: this measures staging, which is the only thing +// `SleepStagerV2` does (detection still comes from V1). + +/// One subject: the streams, the truth hypnogram on the same 30 s grid, and the window they share. +struct PSGSubject { + let id: String + /// Window start — a multiple of 30, aligned with truth epoch 0. + let start: Int + let end: Int + let grav: [GravitySample] + let hr: [HRSample] + /// One entry per 30 s epoch of `[start, end)`. `nil` = the epoch carries no PSG score and is excluded + /// from every metric rather than silently counted as wake. + let truth: [String?] + + var scoredEpochs: Int { truth.reduce(0) { $0 + ($1 == nil ? 0 : 1) } } + /// Duration of the scored night in minutes — the stratification variable. + var scoredMinutes: Double { Double(scoredEpochs) * 30.0 / 60.0 } +} + +enum SleepAccel { + + /// A multiple of 30 near the present, so `t = 0` maps onto a 30 s wall-clock boundary and no timestamp + /// is negative (the recipe's `((start + 29) / 30) * 30` truncates toward zero, which is only the + /// intended floor for non-negative input). + static let timeBase = 1_699_999_980 + + /// Which whole second a fractional dataset timestamp belongs to. FLOOR, not round: the recipe's + /// per-second aggregation buckets by integer second, so a sample at t = 0.9 s is inside second 0, and + /// rounding would push it into second 1 and shift the motion series against the heart-rate series by + /// up to a second. Floor is also the correct answer for the dataset's large negative times. + static func wholeSecond(_ t: Double) -> Int { Int(t.rounded(.down)) } + + /// AASM stage codes as scored in `labels/*_labeled_sleep.txt`, mapped onto NOOP's four classes. + /// N1+N2 → light and N3(+N4) → deep is the standard four-class collapse; anything else — including the + /// dataset's unscored marker — becomes `nil` and is dropped from the denominator. + static func stageForCode(_ code: Int) -> String? { + switch code { + case 0: return "wake" + case 1, 2: return "light" + case 3, 4: return "deep" + case 5: return "rem" + default: return nil // -1 unscored, and any code this collapse does not claim to handle + } + } + + /// Locate the dataset root. Accepts either the extracted directory itself or its parent, so a user who + /// unzipped without `-j` does not have to guess which level to pass. + static func resolveRoot(_ path: String) -> String? { + let fm = FileManager.default + var candidates = [path] + if let subs = try? fm.contentsOfDirectory(atPath: path) { + candidates.append(contentsOf: subs.map { (path as NSString).appendingPathComponent($0) }) + } + for c in candidates { + var isDir: ObjCBool = false + let labels = (c as NSString).appendingPathComponent("labels") + if fm.fileExists(atPath: labels, isDirectory: &isDir), isDir.boolValue { return c } + } + return nil + } + + /// Subject ids, from the label directory — a subject with no PSG hypnogram cannot be scored at all. + static func subjectIDs(root: String) -> [String] { + let dir = (root as NSString).appendingPathComponent("labels") + let files = (try? FileManager.default.contentsOfDirectory(atPath: dir)) ?? [] + return files.compactMap { name -> String? in + guard name.hasSuffix("_labeled_sleep.txt") else { return nil } + return String(name.dropLast("_labeled_sleep.txt".count)) + }.sorted() + } + + /// Load one subject. Returns nil when a required file is missing or the hypnogram has no scored epoch. + static func load(root: String, id: String) -> PSGSubject? { + let labelRows = readRows((root as NSString) + .appendingPathComponent("labels/\(id)_labeled_sleep.txt"), expecting: 2) + guard !labelRows.isEmpty else { return nil } + + // The truth grid: first scored epoch to last. Epoch times in the file are multiples of 30. + var byEpoch: [Int: String] = [:] + var minT = Int.max, maxT = Int.min + for r in labelRows { + let t = Int((r[0] / 30.0).rounded()) * 30 + guard let stage = stageForCode(Int(r[1].rounded())) else { continue } + byEpoch[t] = stage + minT = min(minT, t); maxT = max(maxT, t) + } + guard minT != Int.max else { return nil } + + let start = timeBase + minT + let end = timeBase + maxT + 30 + let nEpochs = (end - start) / 30 + var truth = [String?](repeating: nil, count: nEpochs) + for (t, s) in byEpoch { + let i = (t - minT) / 30 + if i >= 0 && i < nEpochs { truth[i] = s } + } + + // Each subject's motion and heart-rate files cover DAYS around the lab night — the earliest rows sit + // four days before lights-out. Rows outside the recipe's own read window are dropped during the + // read. This is provably loss-free rather than a convenience: `SleepStagerV2.stageSession` clips its + // inputs to exactly `[start − padLo, end + padHi)` before doing anything else, so these are rows the + // recipe would discard itself. Doing it here keeps 31 subjects of daytime 50 Hz data out of memory. + let loSec = minT - V2Recipe.padLo + let hiSec = maxT + 30 + V2Recipe.padHi + + let hr = readRows((root as NSString) + .appendingPathComponent("heart_rate/\(id)_heartrate.txt"), expecting: 2) + .compactMap { r -> HRSample? in + let bpm = Int(r[1].rounded()) + let sec = wholeSecond(r[0]) + guard bpm > 0, bpm < 300, sec >= loSec, sec < hiSec else { return nil } + return HRSample(ts: timeBase + sec, bpm: bpm) + } + .sorted { $0.ts < $1.ts } + + let grav = readAccelerationPerSecond((root as NSString) + .appendingPathComponent("motion/\(id)_acceleration.txt"), loSec: loSec, hiSec: hiSec) + + return PSGSubject(id: id, start: start, end: end, grav: grav, hr: hr, truth: truth) + } + + // MARK: - Parsing + + /// Read a whitespace/comma-separated numeric table, keeping rows that yield at least `expecting` + /// values. Byte-level because the motion files run to millions of lines each and a per-line `String` + /// allocation dominates the whole run. + static func readRows(_ path: String, expecting: Int) -> [[Double]] { + guard let data = FileManager.default.contents(atPath: path) else { return [] } + var out: [[Double]] = [] + forEachRow(data, maxValues: expecting) { vals, n in + if n >= expecting { out.append(Array(vals[0.. [GravitySample] { + guard let data = FileManager.default.contents(atPath: path) else { return [] } + var sx = [Int: Double](), sy = [Int: Double](), sz = [Int: Double](), cnt = [Int: Int]() + forEachRow(data, maxValues: 4) { vals, n in + guard n >= 4 else { return } + let t = wholeSecond(vals[0]) + if t < loSec || t >= hiSec { return } + sx[t, default: 0] += vals[1]; sy[t, default: 0] += vals[2]; sz[t, default: 0] += vals[3] + cnt[t, default: 0] += 1 + } + var out: [GravitySample] = [] + out.reserveCapacity(cnt.count) + for (t, c) in cnt { + let d = Double(c) + out.append(GravitySample(ts: timeBase + t, x: sx[t]! / d, y: sy[t]! / d, z: sz[t]! / d)) + } + return out.sorted { $0.ts < $1.ts } + } + + /// Walk `data` line by line, parsing up to `maxValues` numbers per line with `strtod`, and hand each + /// row to `body`. Separators are commas and spaces/tabs; a line whose numbers run out early is passed + /// with its count so the caller can drop it. + private static func forEachRow(_ data: Data, maxValues: Int, _ body: ([Double], Int) -> Void) { + var bytes = [UInt8](data) + bytes.append(0) + var vals = [Double](repeating: 0, count: maxValues) + bytes.withUnsafeBufferPointer { buf in + guard let raw = buf.baseAddress else { return } + let base = UnsafeRawPointer(raw).assumingMemoryBound(to: CChar.self) + let count = buf.count - 1 + var i = 0 + while i < count { + var lineEnd = i + while lineEnd < count && buf[lineEnd] != 0x0A { lineEnd += 1 } + var p = i + var n = 0 + while p < lineEnd && n < maxValues { + let c = buf[p] + // Only start a parse on something that can begin a number; otherwise `strtod` would + // skip trailing whitespace and swallow the FIRST number of the following line. + let numeric = (c >= 0x30 && c <= 0x39) || c == 0x2D || c == 0x2B || c == 0x2E + if !numeric { p += 1; continue } + var endPtr: UnsafeMutablePointer? + let v = strtod(base + p, &endPtr) + guard let e = endPtr else { break } + let consumed = UnsafePointer(e) - (base + p) + if consumed <= 0 { p += 1; continue } + vals[n] = v + n += 1 + p += consumed + } + body(vals, n) + i = lineEnd + 1 + } + } + } +} diff --git a/Tools/SleepPSG/Sources/sleeppsg/Variants.swift b/Tools/SleepPSG/Sources/sleeppsg/Variants.swift new file mode 100644 index 0000000000..3a612c49e7 --- /dev/null +++ b/Tools/SleepPSG/Sources/sleeppsg/Variants.swift @@ -0,0 +1,153 @@ +import Foundation + +// Variants — the recipe builds this harness compares, each one a single named edit to the shipped config. +// +// The set is not arbitrary. PR #348 changed seven things about `SleepStagerV2` at once, PR #437 reverted all +// seven 48 h later, and PR #987 re-litigated the revert component-by-component and restored exactly one of +// them. That whole argument was settled on one wearer's 36 nights, against the strap's own band state — +// an independent reference, but a proprietary black box, not truth. Each component is reproduced here as +// its own variant so PSG can be asked the same question in the same shape. +// +// Every variant differs from `RecipeConfig.shipped` in the fewest fields that express it. A variant that +// bundled two changes could not be attributed. + +struct Variant { + let name: String + let note: String + let config: RecipeConfig +} + +enum Variants { + + /// The awake transition row before PR #987, and after it. #987 forbids wake → deep and wake → REM + /// outright and gives the freed mass to the wake self-loop. It was landed on band-state evidence — band + /// kappa 0.105 → 0.118, wake sensitivity 16.0 % → 17.6 %, first-REM latency MAE 53.9 → 41.6 min — and + /// never checked against PSG, which is one of the two questions this harness exists to answer. + static let awakeRowPre987: [String: Double] = ["deep": 0.01, "rem": 0.02, "light": 0.27, "awake": 0.70] + static let awakeRowPR987: [String: Double] = ["deep": 0.0, "rem": 0.0, "light": 0.10, "awake": 0.90] + + /// The #987 comparison, whichever side of it this branch is on. + /// + /// This matters because the fork and upstream disagree about the incumbent: this repository's `main` + /// carries #987 already, upstream's does not until the PR merges, and `RecipeConfig.shipped` must track + /// whatever `SleepStagerV2` says on the branch it is compiled against (that is exactly what + /// `PortValidation` enforces). Rather than hard-code one direction and be wrong half the time, the + /// variant reads the shipped awake row and offers the OTHER one. The comparison is the same either way; + /// only the sign of the delta and the row's name flip. + static var pr987: Variant { + let shippedRow = RecipeConfig.shipped.transition["awake"] ?? awakeRowPre987 + let alreadyHas987 = shippedRow == awakeRowPR987 + var c = RecipeConfig.shipped + c.transition["awake"] = alreadyHas987 ? awakeRowPre987 : awakeRowPR987 + return Variant( + name: alreadyHas987 ? "#987 REVERTED (pre-#987 row)" : "#987 awake-transition row", + note: alreadyHas987 + ? "wake→deep/rem restored to 0.01/0.02; self-loop 0.90→0.70" + : "wake→deep/rem forbidden; wake self-loop 0.70→0.90", + config: c) + } + + /// #348 component 1 — base priors. Measured alone on band state: healthy wake fraction 9.43 % → 17.76 %, + /// i.e. the #437 blow-out reproduced. + static var p348Priors: Variant { + var c = RecipeConfig.shipped + c.priorDeep = log(0.15); c.priorAwake = log(0.34) + return Variant(name: "#348-A base priors", + note: "deep 0.18→0.15, awake 0.10→0.34", config: c) + } + + /// #348 component 2 — motion gate. A second wake channel: alone, healthy wake 9.43 % → 15.92 %. + static var p348Motion: Variant { + var c = RecipeConfig.shipped + c.jerkFloorMoveMult = 75.0; c.jerkFloorGateMult = 35.0; c.motionGateBoost = 4.0 + return Variant(name: "#348-B motion gate", + note: "move 38→75, gate 55→35, boost 2→4", config: c) + } + + /// #348 component 3 — emission coefficients. Alone: band kappa 0.105 → 0.094, but the closest call of + /// the six (it cut healthy deep 22.09 % → 15.30 % and latency MAE 53.9 → 32.1 min). + static var p348Emissions: Variant { + var c = RecipeConfig.shipped + c.deepZhv = -0.8; c.deepZhr = 0.5; c.deepZmv = -0.1 + c.remZhv = 0.8; c.remZmv = -0.4; c.remZhr = 0.4 + c.awakeZmv = 1.0; c.awakeZhv = 0.5; c.awakeZhr = 0.6 + return Variant(name: "#348-C emission coefficients", + note: "deep/rem/awake log-emission weights", config: c) + } + + /// #348 component 4 — deep gate. Alone: healthy deep 22.09 % → 25.47 %, worsening an over-call. + static var p348DeepGate: Variant { + var c = RecipeConfig.shipped + c.deepGateThresh = 0.40 + return Variant(name: "#348-D deep gate", note: "deepGateThresh 0.25→0.40", config: c) + } + + /// #348 component 5 — awake cardiac dead-zone. Alone: band kappa 0.105 → 0.099. + static var p348Deadzone: Variant { + var c = RecipeConfig.shipped + c.awakeDeadzone = 0.30 + return Variant(name: "#348-E awake dead-zone", note: "awakeDeadzone off→0.30", config: c) + } + + /// #348 component 6 — the other three transition rows. Alone: band kappa 0.105 → 0.101, healthy REM + /// 29.58 % → 33.11 %. + static var p348OtherRows: Variant { + var c = RecipeConfig.shipped + c.transition["deep"] = ["deep": 0.76, "rem": 0.012, "light": 0.216, "awake": 0.012] + c.transition["rem"] = ["deep": 0.00333, "rem": 0.92, "light": 0.06667, "awake": 0.01] + c.transition["light"] = ["deep": 0.08, "rem": 0.08, "light": 0.80, "awake": 0.04] + return Variant(name: "#348-F deep/rem/light rows", + note: "self-loops retuned, off-diagonals renormalised", config: c) + } + + /// All seven together — the build #437 reverted. Reported so the component rows can be checked for + /// additivity rather than assumed to be. + static var p348All: Variant { + var c = p348Priors.config + let m = p348Motion.config, e = p348Emissions.config, d = p348Deadzone.config + c.jerkFloorMoveMult = m.jerkFloorMoveMult; c.jerkFloorGateMult = m.jerkFloorGateMult + c.motionGateBoost = m.motionGateBoost + c.deepZhv = e.deepZhv; c.deepZhr = e.deepZhr; c.deepZmv = e.deepZmv + c.remZhv = e.remZhv; c.remZmv = e.remZmv; c.remZhr = e.remZhr + c.awakeZmv = e.awakeZmv; c.awakeZhv = e.awakeZhv; c.awakeZhr = e.awakeZhr + c.deepGateThresh = p348DeepGate.config.deepGateThresh + c.awakeDeadzone = d.awakeDeadzone + c.transition = p348OtherRows.config.transition + // #348's own awake row IS the one #987 later restored — pinned to the literal rather than to + // `pr987`, which flips direction depending on which side of #987 the branch is on. + c.transition["awake"] = awakeRowPR987 + return Variant(name: "#348 entire (as reverted by #437)", + note: "all seven components at once", config: c) + } + + /// NOT a candidate — a provenance check, and the reason it is here is worth stating. + /// + /// This harness was rebuilt to reproduce a set of numbers a previous, deleted harness measured on this + /// same dataset: kappa 0.349, REM F1 0.515, REM 20.8 % of night, median first-REM latency 142.0 min. + /// The rebuild reproduces every TRUTH-side figure exactly (26 773 scored epochs, deep 14.76 % of night, + /// truth first-REM 88.5 min) but lands the prediction side elsewhere — more REM, arriving earlier. + /// + /// One mechanism explains that whole pattern, and it is dated rather than mysterious: PR #930 replaced a + /// hard `c < 0.12 ? 3.0 : 0.0` step in the SESSION-FRACTION domain with a graded penalty measured in + /// MINUTES from sleep onset. Averaged over the epochs it covers, the graded guard is the weaker one, so + /// after #930 the recipe admits REM earlier and admits more of it. The old numbers were measured while + /// #930 was the CANDIDATE — so the incumbent they describe is the recipe as it stood BEFORE it. + /// + /// This variant runs that pre-#930 guard so the claim is checked rather than asserted. + static var preNine30Guard: Variant { + var c = RecipeConfig.shipped + c.remLatencyMode = .preNine30FractionStep + return Variant(name: "pre-#930 REM guard (provenance)", + note: "c<0.12 step in the fraction domain, single Viterbi pass", config: c) + } + + /// The shipped recipe itself, so a table row exists to difference against. + static var incumbent: Variant { + Variant(name: "incumbent (shipped)", note: "SleepStagerV2 as it ships today", config: .shipped) + } + + static var all: [Variant] { + [incumbent, pr987, p348Priors, p348Motion, p348Emissions, p348DeepGate, + p348Deadzone, p348OtherRows, p348All, preNine30Guard] + } +} diff --git a/Tools/SleepPSG/Sources/sleeppsg/main.swift b/Tools/SleepPSG/Sources/sleeppsg/main.swift new file mode 100644 index 0000000000..133cbb872b --- /dev/null +++ b/Tools/SleepPSG/Sources/sleeppsg/main.swift @@ -0,0 +1,481 @@ +import Foundation +import StrandAnalytics +import WhoopProtocol + +// sleeppsg — score NOOP's shipped sleep stager against polysomnography. +// +// USAGE: sleeppsg --dataset /path/to/sleep-accel-1.0.0 [--section all] [--csv out.csv] [--subjects N] +// sleeppsg --section port (no dataset needed — the port check alone) +// +// The dataset lives outside this repository and is always an argument. Nothing is written to it. + +struct Args { + var dataset: String? + var section = "all" + var csv: String? + var subjects: Int? + var seed: UInt64 = PortValidation.defaultSeed +} +var a = Args() +var it = CommandLine.arguments.dropFirst().makeIterator() +while let k = it.next() { + switch k { + case "--dataset": a.dataset = it.next() + case "--section": a.section = it.next() ?? a.section + case "--csv": a.csv = it.next() + case "--subjects": a.subjects = Int(it.next() ?? "") + case "--seed": a.seed = UInt64(it.next() ?? "") ?? a.seed + case "-h", "--help": + print(""" + usage: sleeppsg --dataset [--section all|port|baseline|strata|rem|variants] + [--subjects N] [--csv ] [--seed ] + + --dataset PhysioNet sleep-accel v1.0.0, extracted. See README.md for the download step and the + attribution its licence requires. Never committed; read-only here. + --section which report to print. `port` needs no dataset. + --subjects score only the first N subject ids (a fast smoke run; full cohort is the default). + """) + exit(0) + default: FileHandle.standardError.write(Data("unknown argument \(k)\n".utf8)); exit(2) + } +} + +let wantAll = a.section == "all" +func want(_ s: String) -> Bool { wantAll || a.section == s } + +// ───────────────────────────────────────────────────────────────────────────────────────────────────── +// 1. PORT VALIDATION +// ───────────────────────────────────────────────────────────────────────────────────────────────────── + +if want("port") { + print(""" + + ================================================================================ + 1. PORT VALIDATION — does this harness run the SHIPPED recipe? + ================================================================================ + Every baseline number below comes from `StrandAnalytics.SleepStagerV2.stageSession` directly: the file + that ships in the app, compiled from `Packages/`, called with no reimplementation in between. The + VARIANT rows cannot work that way — `SleepStagerV2` holds its constants as `static let`s — so they run + through `V2Recipe`, a knob-for-knob port in this tool. This section is the check that the port and the + shipped stager are the same recipe. It runs in `swift test`, and therefore in CI, with no dataset. + """) + let r = PortValidation.run(seed: a.seed) + print(""" + nights \(r.nights) (\(r.randomNights) randomised + \(r.degenerateNights) degenerate) + epoch labels \(r.matchingEpochs)/\(r.epochs) identical + verdict \(r.ok ? "PASS — the port reproduces the shipped stager exactly" : "FAIL") + """) + if !r.ok { + for d in r.divergences.prefix(20) { + print(" divergence \(d.night) epoch \(d.epochIndex): shipped=\(d.shipped) port=\(d.port)") + } + FileHandle.standardError.write(Data("port validation FAILED — variant numbers are not trustworthy\n".utf8)) + exit(1) + } +} + +if a.section == "port" { exit(0) } + +// ───────────────────────────────────────────────────────────────────────────────────────────────────── +// Dataset +// ───────────────────────────────────────────────────────────────────────────────────────────────────── + +guard let datasetArg = a.dataset else { + FileHandle.standardError.write(Data(""" + --dataset is required for every section except `port`. + See Tools/SleepPSG/README.md for the download step and the attribution the licence requires. + + """.utf8)) + exit(2) +} +guard let root = SleepAccel.resolveRoot(datasetArg) else { + FileHandle.standardError.write(Data("no `labels/` directory under \(datasetArg) — is that the sleep-accel root?\n".utf8)) + exit(2) +} + +var ids = SleepAccel.subjectIDs(root: root) +if let n = a.subjects { ids = Array(ids.prefix(n)) } +// Loaded concurrently — the motion files run to tens of megabytes of text each and the read dominates the +// whole run. Order is restored from `ids` afterwards so the report is identical however the threads finish. +var loaded = [PSGSubject?](repeating: nil, count: ids.count) +loaded.withUnsafeMutableBufferPointer { buf in + let out = buf + DispatchQueue.concurrentPerform(iterations: ids.count) { i in + out[i] = SleepAccel.load(root: root, id: ids[i]) + } +} +let subjects: [PSGSubject] = loaded.compactMap { $0 } +guard !subjects.isEmpty else { + FileHandle.standardError.write(Data("no scorable subjects under \(root)\n".utf8)); exit(1) +} + +let totalScored = subjects.reduce(0) { $0 + $1.scoredEpochs } +print(""" + +================================================================================ +2. DATASET +================================================================================ +PhysioNet sleep-accel v1.0.0 — Walch, Huang, Forger & Goldstein, SLEEP 42(12) zsz180 (2019). +Licence: Open Data Commons Attribution v1.0. Root: \(root) + +subjects \(subjects.count) +PSG-scored epochs \(totalScored) +scored night length median \(f(median(subjects.map { $0.scoredMinutes }), 6, 1)) min \ +range \(f(subjects.map { $0.scoredMinutes }.min() ?? .nan, 5, 1))–\(f(subjects.map { $0.scoredMinutes }.max() ?? .nan, 5, 1)) min + +The R-R stream is EMPTY for every subject — this dataset carries no beat-to-beat intervals — so the +recipe's RSA respiration term returns nil on every epoch and contributes exactly 0.0 to every emission. +Five of the recipe's six inputs are live here; the sixth is silent, not wrong. +""") + +// Truth / prediction on the shared 30 s grid, restricted to PSG-scored epochs. +struct Scored { + let subject: PSGSubject + let truth: [String] + let pred: [String] + /// Full-grid arrays (including unscored epochs) — latency is a property of the hypnogram, not of the + /// scored subset, so it is measured before masking. + let fullPred: [String] + let fullTruth: [String?] +} + +func score(_ s: PSGSubject, using stage: (PSGSubject) -> [StageSegment]) -> Scored { + let segs = stage(s) + let pred = epochLabels(segs, start: s.start, end: s.end) + var t: [String] = [], p: [String] = [] + for i in 0.. [StageSegment] { + SleepStagerV2.stageSession(start: s.start, end: s.end, grav: s.grav, hr: s.hr, rr: [], resp: []) +} +func stageVariant(_ cfg: RecipeConfig) -> (PSGSubject) -> [StageSegment] { + { s in V2Recipe.stageSession(start: s.start, end: s.end, grav: s.grav, hr: s.hr, + rr: [], resp: [], cfg: cfg) } +} + +let shipped = subjects.map { score($0, using: stageShipped) } + +// ───────────────────────────────────────────────────────────────────────────────────────────────────── +// Reporting helpers +// ───────────────────────────────────────────────────────────────────────────────────────────────────── + +struct Summary { + var conf = Confusion() + var kappa: Double { conf.kappa } + var accuracy: Double { conf.accuracy } + var predPct: [String: Double] = [:] + var truthPct: [String: Double] = [:] + var latencyPred: [Double] = [] + var latencyTruth: [Double] = [] + var latencyPairsPred: [Double] = [] + var latencyPairsTruth: [Double] = [] + /// Per-subject kappa, excluding any subject whose scored night carries only one class (kappa is + /// undefined there, and letting a NaN into the mean would quietly void the whole column). + var subjectKappa: [Double] = [] + var noRemNights = 0 +} + +func summarise(_ rows: [Scored]) -> Summary { + var s = Summary() + var predAcc = [String: Double](), truthAcc = [String: Double]() + for r in rows { + var c = Confusion() + for i in 0.. [String]) -> [String: Double] { + var all: [String] = [] + for r in rows { all.append(contentsOf: pick(r)) } + return stagePercentages(all) +} + +let sh = summarise(shipped) +let pooledPred = pooledPct(shipped) { $0.pred } +let pooledTruth = pooledPct(shipped) { $0.truth } + +if want("baseline") { + print(""" + + ================================================================================ + 3. SHIPPED SleepStagerV2 vs PSG TRUTH + ================================================================================ + Four-class agreement over every PSG-scored epoch. `SleepStagerV2` fits no parameters to data — every + coefficient is fixed a priori — so there is no train/test split to make here: the recipe sees each + subject exactly once and has never seen any of them. The leave-one-subject-out machinery in section 6 + exists for the fitted comparison models, which do need it. + + epochs scored \(sh.conf.total) + accuracy \(f(sh.accuracy * 100, 6, 2)) % + Cohen's kappa \(f(sh.kappa, 6, 3)) (per-subject mean \(f(mean(sh.subjectKappa), 6, 3)) ± \(f(sd(sh.subjectKappa), 5, 3))) + + per stage precision recall F1 support + """) + for st in stageOrder { + let p = sh.conf.prf(st) + print(" \(st.padding(toLength: 18, withPad: " ", startingAt: 0))\(f(p.precision, 8, 3)) \(f(p.recall, 8, 3)) \(f(p.f1, 8, 3)) \(p.support)") + } + print(""" + + STAGE FRACTIONS — reported as a first-class result, not an appendix. + Kappa does not constrain them: PR #348 raised kappa on all three of its benchmarks and was reverted + 48 h later for re-scoring a healthy night from 6 % to 23 % awake. + + stage predicted % truth % bias pp (pooled over all scored epochs) + """) + for st in stageOrder { + print(" \(st.padding(toLength: 10, withPad: " ", startingAt: 0))\(f(pooledPred[st]!, 10, 2))\(f(pooledTruth[st]!, 10, 2))\(f(pooledPred[st]! - pooledTruth[st]!, 12, 2))") + } + print("\n stage predicted % truth % bias pp (mean of per-subject percentages)") + for st in stageOrder { + print(" \(st.padding(toLength: 10, withPad: " ", startingAt: 0))\(f(sh.predPct[st]!, 10, 2))\(f(sh.truthPct[st]!, 10, 2))\(f(sh.predPct[st]! - sh.truthPct[st]!, 12, 2))") + } + print(""" + + FIRST-REM LATENCY — minutes from staged sleep onset to the first REM epoch. + nights with no REM in the prediction: \(sh.noRemNights) of \(shipped.count) + + predicted truth + median \(f(median(sh.latencyPred), 10, 1))\(f(median(sh.latencyTruth), 10, 1)) min + mean \(f(mean(sh.latencyPred), 10, 1))\(f(mean(sh.latencyTruth), 10, 1)) min + MAE (paired) \(f(mae(zip(sh.latencyPairsPred, sh.latencyPairsTruth).map { $0 - $1 }), 10, 1)) min (n = \(sh.latencyPairsPred.count)) + """) + + // A harness that replaces a deleted one has to say whether it is the same instrument. These are the + // figures the previous harness reported on this dataset. They are printed as a self-check, NOT as a + // target: nothing in this tool is tuned to hit them, and where they disagree the disagreement is the + // finding. See `Variants.preNine30Guard` for what explains the prediction-side gap. + let ref: [(String, Double, Double, Int)] = [ + ("subjects", Double(subjects.count), 31, 0), + ("PSG-scored epochs", Double(sh.conf.total), 26773, 0), + ("kappa (4-class)", sh.kappa, 0.349, 3), + ("REM F1", sh.conf.prf("rem").f1, 0.515, 3), + ("REM % of night", sh.predPct["rem"]!, 20.8, 2), + ("deep % predicted", sh.predPct["deep"]!, 19.25, 2), + ("deep % truth", sh.truthPct["deep"]!, 14.76, 2), + ("first-REM predicted, min", median(sh.latencyPred), 142.0, 1), + ("first-REM truth, min", median(sh.latencyTruth), 88.5, 1), + ] + print(""" + + SELF-CHECK against the previous harness's reported figures + Printed to expose disagreement, not to be matched. Nothing here is tuned to these numbers. Stage + fractions use the per-subject-mean convention, which is the one those figures were reported on. + + quantity measured previously delta + """) + for (name, got, want, dp) in ref { + print(" \(name.padding(toLength: 28, withPad: " ", startingAt: 0))\(f(got, 9, dp))\(f(want, 13, dp))\(f(got - want, 11, dp))") + } +} + +// ───────────────────────────────────────────────────────────────────────────────────────────────────── +// Night-length stratification +// ───────────────────────────────────────────────────────────────────────────────────────────────────── + +if want("strata") { + print(""" + + ================================================================================ + 4. STRATIFIED BY NIGHT LENGTH + ================================================================================ + Pooling hides a coupling. On one wearer's own nights, `SleepStagerV2` emits more REM as a fraction of + sleep the longer the night runs — corr(total sleep, REM %) = +0.579 — traced to the REM-latency guard + being a fixed 60-minute penalty, which therefore covers a third of a short night and a tenth of a long + one. Whether that reproduces against PSG is a question a pooled table cannot be asked. + + Subjects split into terciles by SCORED night length. + """) + let sortedRows = shipped.sorted { $0.subject.scoredMinutes < $1.subject.scoredMinutes } + let third = max(1, sortedRows.count / 3) + let strata: [(String, [Scored])] = [ + ("short", Array(sortedRows.prefix(third))), + ("mid", Array(sortedRows.dropFirst(third).dropLast(sortedRows.count - 2 * third))), + ("long", Array(sortedRows.suffix(sortedRows.count - 2 * third))), + ] + print(" stratum n night min kappa REM% pred REM% truth deep% pred deep% truth wake% pred wake% truth") + for (name, rows) in strata where !rows.isEmpty { + let p = pooledPct(rows) { $0.pred }, t = pooledPct(rows) { $0.truth } + var c = Confusion() + for r in rows { for i in 0.. Double? { + let sleep = labels.filter { $0 != "wake" }.count + guard sleep > 0 else { return nil } + return Double(labels.filter { $0 == "rem" }.count) / Double(sleep) * 100 + } + var lens: [Double] = [], remPred: [Double] = [], remTruth: [Double] = [] + for r in shipped { + guard let rp = remPctOfSleep(r.pred), let rt = remPctOfSleep(r.truth) else { continue } + lens.append(r.subject.scoredMinutes); remPred.append(rp); remTruth.append(rt) + } + let cp = pearson(lens, remPred), ct = pearson(lens, remTruth) + print(""" + + corr(scored night length, REM % of SLEEP), n = \(cp.n) + predicted Pearson \(f(cp.r, 7, 3)) Spearman \(f(spearman(lens, remPred), 7, 3)) + PSG truth Pearson \(f(ct.r, 7, 3)) Spearman \(f(spearman(lens, remTruth), 7, 3)) + A coupling the truth also carries is physiology; one only the prediction carries is an artefact. + """) +} + +// ───────────────────────────────────────────────────────────────────────────────────────────────────── +// The REM question +// ───────────────────────────────────────────────────────────────────────────────────────────────────── + +if want("rem") { + print(""" + + ================================================================================ + 5. IS REM DETECTION PHYSIOLOGY, OR IS IT THE CLOCK? + ================================================================================ + Three models fit on the same epochs, leave-one-SUBJECT-out, scored only on the held-out subject. + Clock features are the time-of-night fraction and its square; physiology features are the per-night + z-scored heart rate, HR-variability and movement plus the HR-flatness percentile — the same normalised + quantities the recipe's own emissions are built from. Decision threshold tuned on the training folds. + """) + var rows: [AblationRow] = [] + for s in subjects { + let (feats, _) = V2Recipe.stageEpochsDetailed(start: s.start, end: s.end, grav: s.grav, + hr: s.hr, rr: [], resp: []) + // Per-night normalisation, matching `stageEpochs`. + func zfun(_ vals: [Double?]) -> (Double?) -> Double { + let present = vals.compactMap { $0 } + if present.isEmpty { return { _ in 0.0 } } + let m = present.reduce(0, +) / Double(present.count) + let sd0 = (present.reduce(0.0) { $0 + ($1 - m) * ($1 - m) } / Double(present.count)).squareRoot() + let sdv = sd0 == 0 ? 1.0 : sd0 + return { v in v == nil ? 0.0 : (v! - m) / sdv } + } + let zhr = zfun(feats.map { $0.hr }), zhv = zfun(feats.map { $0.hrVar }) + let zmv = zfun(feats.map { Optional($0.moveFrac) }) + let fsorted = feats.compactMap { $0.hrFlat11 }.sorted() + func fpct(_ v: Double?) -> Double { + guard let v = v, !fsorted.isEmpty else { return 0.5 } + var lo = 0, hi = fsorted.count + while lo < hi { let mid = (lo + hi) / 2; if fsorted[mid] <= v { lo = mid + 1 } else { hi = mid } } + return Double(lo) / Double(fsorted.count) + } + for e in feats { + let idx = (e.start - s.start) / 30 + guard idx >= 0, idx < s.truth.count, let tv = s.truth[idx] else { continue } + rows.append(AblationRow( + subject: s.id, isRem: tv == "rem", + clock: [e.clock, e.clock * e.clock], + physiology: [zhr(e.hr), zhv(e.hrVar), zmv(e.moveFrac), fpct(e.hrFlat11)])) + } + } + print(" epochs in the comparison: \(rows.count) REM prevalence \(f(Double(rows.filter { $0.isRem }.count) / Double(max(1, rows.count)) * 100, 5, 1)) %\n") + let res = Ablation.run(rows) + print(" model pooled F1 pooled P pooled R mean per-subject F1") + for r in res { + print(" \(r.model.rawValue.padding(toLength: 18, withPad: " ", startingAt: 0))\(f(r.pooledF1, 8, 3)) \(f(r.pooledPrecision, 9, 3)) \(f(r.pooledRecall, 9, 3)) \(f(r.meanSubjectF1, 12, 3))") + } + if let clock = res.first(where: { $0.model == .clock }), + let phys = res.first(where: { $0.model == .physiology }), + let both = res.first(where: { $0.model == .both }) { + let subs = clock.subjectF1.keys.sorted() + let dPhys = subs.compactMap { s -> Double? in + guard let c = clock.subjectF1[s], let p = phys.subjectF1[s] else { return nil } + return p - c + } + let dBoth = subs.compactMap { s -> Double? in + guard let c = clock.subjectF1[s], let b = both.subjectF1[s] else { return nil } + return b - c + } + print(""" + + what physiology adds over the clock, per subject + physiology-only − clock-only mean \(f(mean(dPhys), 7, 3)) positive in \(dPhys.filter { $0 > 0 }.count)/\(dPhys.count) subjects + both − clock-only mean \(f(mean(dBoth), 7, 3)) positive in \(dBoth.filter { $0 > 0 }.count)/\(dBoth.count) subjects + pooled: physiology-only − clock-only \(f(phys.pooledF1 - clock.pooledF1, 7, 3)), both − clock-only \(f(both.pooledF1 - clock.pooledF1, 7, 3)) + + For reference, the SHIPPED recipe's own REM F1 on these epochs is \(f(sh.conf.prf("rem").f1, 6, 3)) — it fits + nothing, so it is not comparable to a fitted model's held-out score and is printed only for scale. + """) + } +} + +// ───────────────────────────────────────────────────────────────────────────────────────────────────── +// Variants +// ───────────────────────────────────────────────────────────────────────────────────────────────────── + +if want("variants") { + print(""" + + ================================================================================ + 6. RECIPE VARIANTS AGAINST PSG + ================================================================================ + Every row is the shipped recipe with ONE named change. Deltas are against the incumbent row. + `wake bias` and `deep bias` are predicted minus truth in percentage points — the #437 guard, which + kappa does not carry. + """) + var csvLines = [(["variant", "epochs", "kappa", "accuracy"] + + stageOrder.flatMap { ["\($0)_precision", "\($0)_recall", "\($0)_f1", "\($0)_pct", "\($0)_bias_pp"] } + + ["median_rem_latency_min", "rem_latency_mae_min"]).joined(separator: ",")] + var base: (kappa: Double, remF1: Double, wakeSens: Double)? + // Wake SENSITIVITY is in the table beside kappa because it is the quantity #987 was landed on against + // the strap's band state (16.0 % → 17.6 %). Printing it here is what makes that claim checkable against + // truth rather than merely restated. + print(" variant kappa Δκ REM F1 wakeSens wake% bias deep% bias REM% bias latMAE") + for v in Variants.all { + let rows = v.name.hasPrefix("incumbent") + ? shipped + : subjects.map { score($0, using: stageVariant(v.config)) } + let s = summarise(rows) + let p = pooledPct(rows) { $0.pred }, t = pooledPct(rows) { $0.truth } + let latMae = mae(zip(s.latencyPairsPred, s.latencyPairsTruth).map { $0 - $1 }) + let wakeSens = s.conf.prf("wake").recall * 100 + if base == nil { base = (s.kappa, s.conf.prf("rem").f1, wakeSens) } + let b = base! + print(" \(v.name.padding(toLength: 32, withPad: " ", startingAt: 0))\(f(s.kappa, 6, 3)) \(f(s.kappa - b.kappa, 6, 3)) \(f(s.conf.prf("rem").f1, 6, 3)) \(f(wakeSens, 6, 2))\(f(wakeSens - b.wakeSens, 6, 2)) \(f(p["wake"]!, 6, 2))\(f(p["wake"]! - t["wake"]!, 6, 2)) \(f(p["deep"]!, 6, 2))\(f(p["deep"]! - t["deep"]!, 6, 2)) \(f(p["rem"]!, 6, 2))\(f(p["rem"]! - t["rem"]!, 6, 2)) \(f(latMae, 6, 1))") + var row = [v.name, "\(s.conf.total)", "\(s.kappa)", "\(s.accuracy)"] + for st in stageOrder { + let m = s.conf.prf(st) + row += ["\(m.precision)", "\(m.recall)", "\(m.f1)", "\(p[st]!)", "\(p[st]! - t[st]!)"] + } + row += ["\(median(s.latencyPred))", "\(latMae)"] + csvLines.append(row.joined(separator: ",")) + } + print(""" + + A component is an improvement here only if it raises kappa AND does not blow out a stage fraction. + That conjunction is the lesson of #348 → #437: kappa alone certified a build that mis-called a + healthy night's wake fraction by 17 percentage points. + """) + if let path = a.csv { + try? csvLines.joined(separator: "\n").write(toFile: path, atomically: true, encoding: .utf8) + print(" per-variant CSV written to \(path)") + } +} + +print("") diff --git a/Tools/SleepPSG/Tests/sleeppsgTests/DatasetAndAblationTests.swift b/Tools/SleepPSG/Tests/sleeppsgTests/DatasetAndAblationTests.swift new file mode 100644 index 0000000000..25bad9b120 --- /dev/null +++ b/Tools/SleepPSG/Tests/sleeppsgTests/DatasetAndAblationTests.swift @@ -0,0 +1,295 @@ +import XCTest +import WhoopProtocol +@testable import sleeppsg + +/// The dataset reader and the leave-one-subject-out machinery, on synthetic fixtures written to a temp +/// directory. No PhysioNet download, no health data, no network — these run in CI. +final class SleepAccelTests: XCTestCase { + + private func makeFixture(labels: String, heartRate: String, motion: String) throws -> String { + let root = NSTemporaryDirectory() + "sleeppsg-fixture-" + UUID().uuidString + let fm = FileManager.default + for d in ["labels", "heart_rate", "motion"] { + try fm.createDirectory(atPath: root + "/" + d, withIntermediateDirectories: true) + } + try labels.write(toFile: root + "/labels/9001_labeled_sleep.txt", atomically: true, encoding: .utf8) + try heartRate.write(toFile: root + "/heart_rate/9001_heartrate.txt", atomically: true, encoding: .utf8) + try motion.write(toFile: root + "/motion/9001_acceleration.txt", atomically: true, encoding: .utf8) + addTeardownBlock { try? fm.removeItem(atPath: root) } + return root + } + + /// The AASM collapse. N1 and N2 both become light and N3 becomes deep, which is the standard + /// four-class reduction; anything the collapse does not claim to handle becomes nil and leaves the + /// denominator rather than being counted as wake. + func testStageCodeCollapse() { + XCTAssertEqual(SleepAccel.stageForCode(0), "wake") + XCTAssertEqual(SleepAccel.stageForCode(1), "light") + XCTAssertEqual(SleepAccel.stageForCode(2), "light") + XCTAssertEqual(SleepAccel.stageForCode(3), "deep") + XCTAssertEqual(SleepAccel.stageForCode(4), "deep") + XCTAssertEqual(SleepAccel.stageForCode(5), "rem") + XCTAssertNil(SleepAccel.stageForCode(-1)) + XCTAssertNil(SleepAccel.stageForCode(7)) + } + + /// The load path: label grid, time base, and the per-second collapse of the raw accelerometer. + func testLoadAlignsTruthEpochsToTheStagerGrid() throws { + // Four scored epochs, one of them unscored (-1), and a REM epoch last. + let labels = "0 0\n30 1\n60 -1\n90 5\n" + // Heart rate is comma-separated in this dataset; motion is space-separated. + let hr = "0.0,60\n1.5,61\n119.0,58\n" + // Two raw samples inside second 0 and one inside second 1, so the per-second mean is checkable. + // 0.90 belongs to second 0 — the bucket is a FLOOR, so a sample late in a second is not pushed + // into the next one, which would shift motion against heart rate by up to a second. + let motion = "0.10 0.0 0.0 1.0\n0.90 1.0 0.0 0.0\n1.20 0.0 1.0 0.0\n" + let root = try makeFixture(labels: labels, heartRate: hr, motion: motion) + + XCTAssertEqual(SleepAccel.resolveRoot(root), root) + XCTAssertEqual(SleepAccel.subjectIDs(root: root), ["9001"]) + let s = try XCTUnwrap(SleepAccel.load(root: root, id: "9001")) + + XCTAssertEqual(s.start, SleepAccel.timeBase) + XCTAssertEqual(s.end, SleepAccel.timeBase + 120) + XCTAssertEqual(s.truth, ["wake", "light", nil, "rem"]) + XCTAssertEqual(s.scoredEpochs, 3) + XCTAssertEqual(s.scoredMinutes, 1.5, accuracy: 1e-12) + + // The window start is a multiple of 30, so the recipe's wall-clock epoch grid coincides with the + // PSG scoring grid and no epoch is compared against a label it only half overlaps. + XCTAssertEqual(s.start % 30, 0) + + XCTAssertEqual(s.hr.map { $0.ts }, [SleepAccel.timeBase, SleepAccel.timeBase + 1, SleepAccel.timeBase + 119]) + XCTAssertEqual(s.hr.map { $0.bpm }, [60, 61, 58]) + + XCTAssertEqual(s.grav.count, 2) + XCTAssertEqual(s.grav[0].ts, SleepAccel.timeBase) + XCTAssertEqual(s.grav[0].x, 0.5, accuracy: 1e-12, "second 0 averages its two raw samples") + XCTAssertEqual(s.grav[0].z, 0.5, accuracy: 1e-12) + XCTAssertEqual(s.grav[1].ts, SleepAccel.timeBase + 1) + XCTAssertEqual(s.grav[1].y, 1.0, accuracy: 1e-12) + } + + /// Rows outside the recipe's own read window are dropped during the read. `SleepStagerV2.stageSession` + /// clips its inputs to `[start − 330, end + 390)` before doing anything, so this discards only rows the + /// recipe provably never touches — which matters because each subject's files span DAYS around the lab + /// night and the daytime 50 Hz motion would otherwise dominate memory. + func testSamplesOutsideTheRecipesReadWindowAreDropped() throws { + // One labelled epoch at t = 0, so the read window is [-330, 420). + let root = try makeFixture( + labels: "0 2\n", + heartRate: "-400.0,70\n-100.0,60\n0.0,61\n500.0,80\n", + motion: "-400.0 0 0 1\n-100.0 0 0 1\n0.0 0 0 1\n500.0 0 0 1\n") + let s = try XCTUnwrap(SleepAccel.load(root: root, id: "9001")) + XCTAssertEqual(s.hr.map { $0.ts - SleepAccel.timeBase }, [-100, 0]) + XCTAssertEqual(s.grav.map { $0.ts - SleepAccel.timeBase }, [-100, 0]) + } + + /// Negative times (the dataset records before PSG lights-out) must not produce negative timestamps: + /// the recipe's `((start + 29) / 30) * 30` truncates toward zero, which is only the intended floor on + /// non-negative input. + func testPreRecordingSamplesStayPositive() throws { + // t = −300.5 is before lights-out but still inside the recipe's read window (−330 onward), so it + // survives clipping and exercises the negative-time path. Floor puts it in second −301. + let root = try makeFixture(labels: "0 0\n30 2\n", + heartRate: "-300.5,70\n0.0,60\n", + motion: "-300.5 0.0 0.0 1.0\n0.0 0.0 0.0 1.0\n") + let s = try XCTUnwrap(SleepAccel.load(root: root, id: "9001")) + XCTAssertTrue(s.hr.allSatisfy { $0.ts > 0 }) + XCTAssertTrue(s.grav.allSatisfy { $0.ts > 0 }) + XCTAssertEqual(s.hr.first?.ts, SleepAccel.timeBase - 301) + XCTAssertEqual(s.grav.first?.ts, SleepAccel.timeBase - 301) + } + + /// The byte-level row parser: mixed separators, blank lines, a trailing line with no newline, and a + /// short row that must be dropped rather than half-read. + func testRowParserToleratesRealFileShapes() throws { + let root = try makeFixture(labels: "0 0\n30 5\n", + heartRate: "0.0,60\n\n10.0 61\n20.0,62", + motion: "0.0 1.0 0.0 0.0\n1.0 0.0\n2.0 0.0 0.0 1.0") + let s = try XCTUnwrap(SleepAccel.load(root: root, id: "9001")) + XCTAssertEqual(s.hr.map { $0.bpm }, [60, 61, 62], "blank line skipped, both separators accepted") + XCTAssertEqual(s.grav.count, 2, "the 2-value motion row is dropped, not padded with garbage") + } + + /// Passing the directory that CONTAINS the extracted dataset must work too — the published zip expands + /// into a long-named subdirectory nobody types by hand. + func testResolveRootAcceptsTheParentDirectory() throws { + let root = try makeFixture(labels: "0 0\n", heartRate: "0.0,60\n", motion: "0.0 0 0 1\n") + let parent = (root as NSString).deletingLastPathComponent + XCTAssertEqual(SleepAccel.resolveRoot(parent + "/" + (root as NSString).lastPathComponent), root) + } +} + +final class AblationTests: XCTestCase { + + /// A subject's own epochs must never appear in the fold that scores it. This is the leak the whole + /// design exists to prevent — consecutive 30 s epochs from one night are nearly identical, so an + /// epoch-level split lets any model memorise the subject and score near-perfectly. + func testHeldOutSubjectIsNeverInItsOwnTrainingFold() { + // Two subjects with OPPOSITE physiology→REM mappings and no clock signal at all. A model that + // leaked the held-out subject's rows would score well; one that generalises cannot, because the + // training fold teaches it exactly the wrong sign. + var rows: [AblationRow] = [] + for i in 0..<200 { + let x = Double(i % 2) + rows.append(AblationRow(subject: "A", isRem: x > 0.5, + clock: [0.5, 0.25], physiology: [x, 0, 0, 0.5])) + rows.append(AblationRow(subject: "B", isRem: x < 0.5, + clock: [0.5, 0.25], physiology: [x, 0, 0, 0.5])) + } + let res = Ablation.run(rows) + let phys = res.first { $0.model == .physiology }! + XCTAssertLessThan(phys.pooledF1, 0.55, + "physiology F1 near 1.0 here would mean the held-out subject leaked into training") + } + + /// The fitter has to actually fit: a cleanly separable signal must be recovered, or a null result from + /// the clock model would be indistinguishable from a broken optimiser. + func testLogisticFitRecoversASeparableSignal() { + var X: [[Double]] = [], y: [Bool] = [] + for i in 0..<400 { + let v = Double(i) / 400.0 + X.append([v]); y.append(v > 0.5) + } + let w = Ablation.logisticFit(X: X, y: y) + XCTAssertGreaterThan(w[1], 1.0, "the coefficient must be positive and substantial") + XCTAssertLessThan(Ablation.sigmoid(Ablation.dot(w, [0.1])), 0.2) + XCTAssertGreaterThan(Ablation.sigmoid(Ablation.dot(w, [0.9])), 0.8) + } + + /// The threshold is tuned on the training folds, which matters because REM is a minority class: a + /// fixed 0.5 systematically under-predicts it, and unequally across models with different calibration. + func testThresholdTuningBeatsAFixedHalfOnAMinorityClass() { + // 10 % positives whose scores never reach 0.5 — a fixed cut scores 0, a tuned cut recovers them. + var scores: [Double] = [], truth: [Bool] = [] + for i in 0..<100 { + let pos = i < 10 + scores.append(pos ? 0.40 : 0.05); truth.append(pos) + } + let thr = Ablation.bestF1Threshold(scores: scores, truth: truth) + XCTAssertLessThanOrEqual(thr, 0.40) + XCTAssertEqual(binaryF1(ref: truth, pred: scores.map { $0 >= 0.5 }), 0.0, accuracy: 1e-12) + XCTAssertEqual(binaryF1(ref: truth, pred: scores.map { $0 >= thr }), 1.0, accuracy: 1e-12) + } + + func testLinearSolveMatchesAKnownSystem() { + // [[2,1],[1,3]] x = [5,10] → x = [1,3]. + let x = Ablation.solve([[2, 1], [1, 3]], [5, 10])! + XCTAssertEqual(x[0], 1.0, accuracy: 1e-9) + XCTAssertEqual(x[1], 3.0, accuracy: 1e-9) + XCTAssertNil(Ablation.solve([[1, 2], [2, 4]], [1, 2]), "a singular system returns nil, not NaNs") + } +} + +final class VariantsTests: XCTestCase { + + /// Each variant must differ from the shipped recipe in exactly the fields its name claims. A variant + /// that quietly moved a second constant could not be attributed, which is the entire point of measuring + /// #348's components one at a time. + func testEachVariantIsASingleNamedChange() { + let s = RecipeConfig.shipped + + // #987 is a two-sided comparison: this repository's main carries it, upstream's does not, and the + // variant offers whichever row the branch is NOT on. Both directions are asserted so the test does + // not have to know which branch it is compiled on. + var c = Variants.pr987.config + let shippedRow = s.transition["awake"]! + XCTAssertTrue(shippedRow == Variants.awakeRowPre987 || shippedRow == Variants.awakeRowPR987, + "the shipped awake row is neither side of #987 — RecipeConfig.shipped is stale") + XCTAssertEqual(c.transition["awake"]!, + shippedRow == Variants.awakeRowPR987 ? Variants.awakeRowPre987 : Variants.awakeRowPR987, + "the #987 variant must offer the row the branch is NOT on") + c.transition = s.transition + XCTAssertEqual(c, s, "#987 must change nothing but the awake transition row") + + var p = Variants.p348Priors.config + XCTAssertEqual(p.priorDeep, log(0.15), accuracy: 1e-12) + XCTAssertEqual(p.priorAwake, log(0.34), accuracy: 1e-12) + p.priorDeep = s.priorDeep; p.priorAwake = s.priorAwake + XCTAssertEqual(p, s) + + var m = Variants.p348Motion.config + m.jerkFloorMoveMult = s.jerkFloorMoveMult; m.jerkFloorGateMult = s.jerkFloorGateMult + m.motionGateBoost = s.motionGateBoost + XCTAssertEqual(m, s) + + var d = Variants.p348DeepGate.config + d.deepGateThresh = s.deepGateThresh + XCTAssertEqual(d, s) + + var z = Variants.p348Deadzone.config + z.awakeDeadzone = s.awakeDeadzone + XCTAssertEqual(z, s) + + var o = Variants.p348OtherRows.config + XCTAssertEqual(o.transition["awake"]!, s.transition["awake"]!, "the awake row is #987's, not this one's") + o.transition = s.transition + XCTAssertEqual(o, s) + + var g = Variants.preNine30Guard.config + XCTAssertEqual(g.remLatencyMode, .preNine30FractionStep) + g.remLatencyMode = s.remLatencyMode + XCTAssertEqual(g, s, "the provenance variant must change nothing but the guard's shape") + } + + /// The shipped mode is the graded one, and the pre-#930 step must actually reach the lattice — a + /// provenance variant that staged identically to the incumbent would explain nothing about the gap it + /// is there to explain. + func testPreNine30GuardActuallyChangesTheHypnogram() { + XCTAssertEqual(RecipeConfig.shipped.remLatencyMode, .gradedFromOnset) + var differed = false + for n in PortValidation.corpus().prefix(6) { + let a = V2Recipe.stageSession(start: n.start, end: n.end, grav: n.grav, hr: n.hr, + rr: n.rr, resp: n.resp, cfg: .shipped) + let b = V2Recipe.stageSession(start: n.start, end: n.end, grav: n.grav, hr: n.hr, + rr: n.rr, resp: n.resp, cfg: Variants.preNine30Guard.config) + XCTAssertFalse(a.isEmpty) + if epochLabels(a, start: n.start, end: n.end) != epochLabels(b, start: n.start, end: n.end) { + differed = true + break + } + } + XCTAssertTrue(differed, "the pre-#930 guard staged identically on every night — mode is not wired") + } + + /// The shipped dead-zone is off, and `dz` must then be the identity — that is what makes + /// `RecipeConfig.shipped` able to reproduce an emission the shipped file writes without any dead-zone + /// at all. + func testDeadzoneIsTheIdentityWhenDisabled() { + XCTAssertEqual(RecipeConfig.shipped.awakeDeadzone, 0.0) + for z in [-3.0, -0.1, 0.0, 0.29, 5.0] { + XCTAssertEqual(V2Recipe.dz(z, 0.0), z, accuracy: 1e-12) + } + XCTAssertEqual(V2Recipe.dz(0.2, 0.3), 0.0, accuracy: 1e-12) + XCTAssertEqual(V2Recipe.dz(0.5, 0.3), 0.2, accuracy: 1e-12) + XCTAssertEqual(V2Recipe.dz(-0.5, 0.3), -0.2, accuracy: 1e-12) + } + + /// Transition rows must stay probability distributions, in every variant. A row that no longer sums to + /// 1 would silently reweight the whole lattice rather than express the change its name claims. + func testEveryVariantsTransitionRowsSumToOne() { + for v in Variants.all { + for (from, row) in v.config.transition { + XCTAssertEqual(row.values.reduce(0, +), 1.0, accuracy: 1e-9, + "\(v.name): row \(from) does not sum to 1") + } + } + } + + /// `#348 entire` must be the union of the components, not an independently typed eighth build. + func testEntire348IsTheUnionOfItsComponents() { + let all = Variants.p348All.config + XCTAssertEqual(all.priorDeep, Variants.p348Priors.config.priorDeep) + XCTAssertEqual(all.priorAwake, Variants.p348Priors.config.priorAwake) + XCTAssertEqual(all.jerkFloorMoveMult, Variants.p348Motion.config.jerkFloorMoveMult) + XCTAssertEqual(all.motionGateBoost, Variants.p348Motion.config.motionGateBoost) + XCTAssertEqual(all.deepZhv, Variants.p348Emissions.config.deepZhv) + XCTAssertEqual(all.awakeZhr, Variants.p348Emissions.config.awakeZhr) + XCTAssertEqual(all.deepGateThresh, Variants.p348DeepGate.config.deepGateThresh) + XCTAssertEqual(all.awakeDeadzone, Variants.p348Deadzone.config.awakeDeadzone) + XCTAssertEqual(all.transition["light"]!, Variants.p348OtherRows.config.transition["light"]!) + XCTAssertEqual(all.transition["awake"]!, Variants.awakeRowPR987, + "#348's awake row is the one #987 later restored") + } +} diff --git a/Tools/SleepPSG/Tests/sleeppsgTests/PortValidationTests.swift b/Tools/SleepPSG/Tests/sleeppsgTests/PortValidationTests.swift new file mode 100644 index 0000000000..2065380b68 --- /dev/null +++ b/Tools/SleepPSG/Tests/sleeppsgTests/PortValidationTests.swift @@ -0,0 +1,92 @@ +import XCTest +import StrandAnalytics +import WhoopProtocol +@testable import sleeppsg + +/// The test that keeps this harness honest. +/// +/// `V2Recipe` re-states `SleepStagerV2` with its constants exposed, because that is the only way to score a +/// variant of the recipe without rebuilding the app. The failure mode of every such port is silent drift: +/// someone changes a coefficient in `Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift`, +/// nobody changes `RecipeConfig.shipped`, and the benchmark keeps printing confident numbers about a recipe +/// that no longer exists. +/// +/// So equivalence is asserted here, in CI, on every PR — not verified once by reading the two files side by +/// side. Nothing in this file needs the PhysioNet dataset, a database, or any health data. +final class PortValidationTests: XCTestCase { + + /// Every epoch label, on every night in the corpus, from both paths. + func testPortReproducesShippedStagerExactly() { + let r = PortValidation.run() + XCTAssertEqual(r.nights, 55, "corpus size changed — 48 randomised + 7 degenerate expected") + XCTAssertEqual(r.randomNights, 48) + XCTAssertEqual(r.degenerateNights, 7) + XCTAssertGreaterThan(r.epochs, 10_000, "corpus too small to be evidence of anything") + if !r.divergences.isEmpty { + let detail = r.divergences.prefix(5) + .map { "\($0.night) epoch \($0.epochIndex): shipped=\($0.shipped) port=\($0.port)" } + .joined(separator: "; ") + XCTFail(""" + RecipeConfig.shipped no longer reproduces SleepStagerV2 — \ + \(r.epochs - r.matchingEpochs) of \(r.epochs) epoch labels differ. \ + Update RecipeConfig.shipped to match the shipped file. First divergences: \(detail) + """) + } + XCTAssertEqual(r.matchingEpochs, r.epochs) + } + + /// The corpus must actually reach the branches it claims to. A degenerate case that silently produced + /// no epochs would make the headline "identical on N epochs" true and meaningless. + func testDegenerateNightsAreNonTrivial() { + for n in PortValidation.degenerateNights() where n.name != "degenerate-no-coverage" { + let segs = SleepStagerV2.stageSession(start: n.start, end: n.end, grav: n.grav, + hr: n.hr, rr: n.rr, resp: n.resp) + XCTAssertFalse(segs.isEmpty, "\(n.name) staged to nothing") + let labels = epochLabels(segs, start: n.start, end: n.end) + XCTAssertGreaterThan(labels.count, 0, "\(n.name) produced no epochs") + } + // The no-coverage night is the one that MUST collapse to the single-segment fallback. + let empty = PortValidation.degenerateNights().first { $0.name == "degenerate-no-coverage" }! + let segs = SleepStagerV2.stageSession(start: empty.start, end: empty.end, grav: [], + hr: [], rr: [], resp: []) + XCTAssertEqual(segs.count, 1) + XCTAssertEqual(segs.first?.stage, "light") + } + + /// The corpus is regenerated from a fixed seed, so a divergence is always a code change and never a + /// reroll. Compared on the GENERATED nights rather than on staged output: staging the corpus is the + /// expensive part, and determinism is a property of the generator. + func testCorpusIsDeterministic() { + func signature(_ ns: [SyntheticNight]) -> [String] { + ns.map { "\($0.name)|\($0.start)|\($0.end)|\($0.grav.count)|\($0.hr.count)|\($0.rr.count)|" + + "\($0.hr.first?.bpm ?? -1)|\($0.hr.last?.bpm ?? -1)|\($0.rr.last?.rrMs ?? -1)" } + } + XCTAssertEqual(signature(PortValidation.corpus()), signature(PortValidation.corpus())) + XCTAssertNotEqual(signature(PortValidation.corpus()), signature(PortValidation.corpus(seed: 12345)), + "a different seed must produce a different corpus") + } + + /// Equivalence must hold on ANY corpus, not only the pinned one — otherwise the seed is load-bearing + /// and the check is a coincidence. A small off-seed corpus makes that point without paying for a + /// second full run. + func testPortIsEquivalentOnAnUnpinnedCorpusToo() { + let r = PortValidation.run(seed: 0xA11CE, randomNights: 6) + XCTAssertGreaterThan(r.epochs, 500) + XCTAssertEqual(r.matchingEpochs, r.epochs, "\(r.divergences.first.map(String.init(describing:)) ?? "")") + } + + /// The randomised nights have to exercise the R-R / RSA path, which the PhysioNet dataset cannot: + /// sleep-accel carries no beat-to-beat intervals, so without this the respiration term would be + /// unvalidated in the port everywhere it matters. + func testCorpusExercisesTheRespirationTerm() { + let withRR = PortValidation.corpus().filter { !$0.rr.isEmpty } + XCTAssertGreaterThan(withRR.count, 10, "too few nights carry R-R for the RSA term to be pinned") + let n = withRR[0] + let feats = V2Recipe.features(start: n.start, end: n.end, + grav: n.grav.sorted { $0.ts < $1.ts }, + hr: n.hr.sorted { $0.ts < $1.ts }, + rr: n.rr.sortedByTsStable()) + XCTAssertTrue(feats.contains { $0.respReg != nil }, + "no epoch produced a respiration-regularity value — the RSA term is never reached") + } +} diff --git a/Tools/SleepPSG/Tests/sleeppsgTests/ScoringTests.swift b/Tools/SleepPSG/Tests/sleeppsgTests/ScoringTests.swift new file mode 100644 index 0000000000..ebe1b9abc7 --- /dev/null +++ b/Tools/SleepPSG/Tests/sleeppsgTests/ScoringTests.swift @@ -0,0 +1,141 @@ +import XCTest +import StrandAnalytics +@testable import sleeppsg + +/// Scoring primitives, pinned against values computed by hand. +/// +/// These definitions are shared, in spirit, with `Tools/SleepBench/Sources/sleepbench/Metrics.swift`: the +/// two harnesses score the same recipe against different references, and a kappa that meant something +/// slightly different in each would make their numbers incomparable precisely when someone needs to compare +/// them. Pinning the arithmetic is how that stays true without the two files being one file. +final class ScoringTests: XCTestCase { + + // MARK: - Agreement + + func testKappaOnAHandComputedMatrix() { + // 10 epochs: 4 wake (3 right), 6 light (5 right, 1 called wake). + let ref = ["wake", "wake", "wake", "wake", "light", "light", "light", "light", "light", "light"] + let pred = ["wake", "wake", "wake", "light", "light", "light", "light", "light", "light", "wake"] + let c = confusion(ref: ref, pred: pred) + // po = 8/10. Marginals: ref wake .4 light .6; pred wake .4 light .6 → pe = .16 + .36 = .52. + // kappa = (.8 − .52) / (1 − .52) = .28 / .48 = 0.58333… + XCTAssertEqual(c.accuracy, 0.8, accuracy: 1e-12) + XCTAssertEqual(c.kappa, 0.28 / 0.48, accuracy: 1e-12) + } + + func testPerfectAndChanceAgreement() { + let labels = ["wake", "light", "deep", "rem", "light", "light"] + XCTAssertEqual(confusion(ref: labels, pred: labels).kappa, 1.0, accuracy: 1e-12) + // Every epoch called light against a 50/50 reference: accuracy = pe, so kappa is exactly 0. + let ref = ["light", "wake", "light", "wake"] + let all = ["light", "light", "light", "light"] + XCTAssertEqual(confusion(ref: ref, pred: all).kappa, 0.0, accuracy: 1e-12) + } + + func testPerStageF1IsHarmonicMeanOfPrecisionAndRecall() { + // REM: 2 true REM epochs, 1 recovered; 1 false REM. precision 1/2, recall 1/2, F1 1/2. + let ref = ["rem", "rem", "light", "light"] + let pred = ["rem", "light", "rem", "light"] + let p = confusion(ref: ref, pred: pred).prf("rem") + XCTAssertEqual(p.precision, 0.5, accuracy: 1e-12) + XCTAssertEqual(p.recall, 0.5, accuracy: 1e-12) + XCTAssertEqual(p.f1, 0.5, accuracy: 1e-12) + XCTAssertEqual(p.support, 2) + } + + /// A recipe that stops emitting a stage must score 0 on it, not n/a — a silent nil would let a + /// collapsed stage disappear from a summary table instead of announcing itself. + func testStageNeverEmittedScoresZeroNotNaN() { + let p = confusion(ref: ["rem", "rem", "light"], pred: ["light", "light", "light"]).prf("rem") + XCTAssertTrue(p.precision.isNaN, "precision is undefined when nothing was predicted") + XCTAssertEqual(p.recall, 0.0, accuracy: 1e-12) + XCTAssertEqual(p.f1, 0.0, accuracy: 1e-12) + XCTAssertEqual(p.support, 2) + } + + func testBinaryF1MatchesTheConfusionMatrixVersion() { + let ref = ["rem", "rem", "light", "light", "rem"] + let pred = ["rem", "light", "rem", "light", "rem"] + XCTAssertEqual(binaryF1(ref: ref.map { $0 == "rem" }, pred: pred.map { $0 == "rem" }), + confusion(ref: ref, pred: pred).prf("rem").f1, accuracy: 1e-12) + } + + // MARK: - Stage fractions + + func testStagePercentagesAlwaysCarryEveryStage() { + let pct = stagePercentages(["light", "light", "wake", "wake"]) + XCTAssertEqual(Set(pct.keys), Set(stageOrder)) + XCTAssertEqual(pct["light"]!, 50, accuracy: 1e-12) + XCTAssertEqual(pct["wake"]!, 50, accuracy: 1e-12) + XCTAssertEqual(pct["deep"]!, 0, accuracy: 1e-12) + XCTAssertEqual(pct["rem"]!, 0, accuracy: 1e-12) + } + + /// The #348 → #437 lesson, as an assertion: a change can raise kappa AND wreck a stage fraction, so a + /// harness that reports only kappa cannot see the regression that reverted that PR. + func testKappaCanRiseWhileWakeFractionBlowsOut() { + let ref = Array(repeating: "light", count: 90) + Array(repeating: "wake", count: 10) + // Incumbent: calls only 6 epochs wake, gets 4 of them right. + var before = Array(repeating: "light", count: 100) + for i in 90..<94 { before[i] = "wake" } + for i in 0..<2 { before[i] = "wake" } + // Candidate: calls 23 epochs wake, gets 9 of the 10 real ones — kappa up, wake fraction ruined. + var after = Array(repeating: "light", count: 100) + for i in 90..<99 { after[i] = "wake" } + for i in 0..<14 { after[i] = "wake" } + let kBefore = confusion(ref: ref, pred: before).kappa + let kAfter = confusion(ref: ref, pred: after).kappa + XCTAssertGreaterThan(kAfter, kBefore, "the candidate must win on kappa for this test to mean anything") + XCTAssertEqual(stageBias(ref: ref, pred: before)["wake"]!, -4, accuracy: 1e-9) + XCTAssertEqual(stageBias(ref: ref, pred: after)["wake"]!, 13, accuracy: 1e-9) + } + + // MARK: - Latency + + func testFirstRemLatencyIsMeasuredFromOnsetNotWindowStart() { + // 4 wake epochs, then light, then REM at index 6 → 2 epochs after onset → 1.0 min. + let labels = ["wake", "wake", "wake", "wake", "light", "light", "rem", "light"] + XCTAssertEqual(firstRemLatencyMinutes(labels)!, 1.0, accuracy: 1e-12) + } + + func testNoRemIsNilRatherThanZero() { + XCTAssertNil(firstRemLatencyMinutes(["wake", "light", "light", "deep"])) + XCTAssertNil(firstRemLatencyMinutes(["wake", "wake"])) + } + + /// Unscored epochs keep their slot in the arithmetic. Collapsing them first would shorten the truth + /// latency by however much of the night the technician left unscored — and only the truth column, + /// which is exactly the kind of asymmetry that makes a prediction look better than it is. + func testLatencyOverAGridWithUnscoredEpochs() { + let sparse: [String?] = ["wake", nil, nil, "light", nil, "light", "rem"] + // onset at index 3, REM at index 6 → 3 epochs → 1.5 min, unscored slots included. + XCTAssertEqual(firstRemLatencyMinutes(sparse)!, 1.5, accuracy: 1e-12) + // Collapsing first would have given 2 epochs → 1.0 min. Pin that this is NOT what happens. + XCTAssertNotEqual(firstRemLatencyMinutes(sparse.compactMap { $0 })!, 1.5) + XCTAssertNil(firstRemLatencyMinutes([nil, nil, "wake"] as [String?])) + } + + // MARK: - Epoch expansion + + func testEpochLabelsTileTheWindowFromSegments() { + let segs = [StageSegment(start: 0, end: 60, stage: "wake"), + StageSegment(start: 60, end: 150, stage: "light"), + StageSegment(start: 150, end: 180, stage: "rem")] + XCTAssertEqual(epochLabels(segs, start: 0, end: 180), + ["wake", "wake", "light", "light", "light", "rem"]) + } + + // MARK: - Descriptive statistics + + func testPearsonAndSpearmanOnAMonotoneButCurvedRelation() { + let x = [1.0, 2, 3, 4, 5] + let y = [1.0, 4, 9, 16, 25] + XCTAssertEqual(spearman(x, y), 1.0, accuracy: 1e-12, "ranks are perfectly monotone") + XCTAssertLessThan(pearson(x, y).r, 1.0, "the linear fit is not perfect") + XCTAssertGreaterThan(pearson(x, y).r, 0.95) + } + + func testSpearmanHandlesTies() { + XCTAssertEqual(spearman([1.0, 2, 2, 3], [1.0, 2, 2, 3]), 1.0, accuracy: 1e-12) + } +} From a066b8f1c768dfb3d942e42e9d4953cba14d5f21 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 30 Jul 2026 15:28:43 -0400 Subject: [PATCH 157/171] lint(hygiene): scope the doc-comment gate out of code-generator output Source Hygiene has been red on fork main with 5 detached doc comments, all in Strand/CloudSync/Generated/liters_ffi.swift. That file is uniffi-bindgen output: Rust/build-ios.sh `rm -rf`s the bindings directory and re-emits it on every build, its header says "GENERATED by Rust/build-ios.sh ... Do not edit by hand", and it already carries `// swiftlint:disable all`. So the two options the gate's own message allows -- fix the comment, or scope the path -- reduce to one. A fix does not survive: the next `Rust/build-ios.sh` deletes it and the gate goes red again on a file nobody touched. That is the failure mode #514 already fixed once for i18n, where a whole-tree gate red-checked every contributor on a line none of them wrote. Scoping is also the honest reading of what this lint is for. Its docstring names the failure mode as editing by INSERTION rather than in place -- a hand-editing mistake. A generator does not edit, it re-emits, so the premise does not hold for this path. `/Generated/` sits in SKIP_PARTS alongside /build/, /.build/ and /DerivedData/, which are the same category: output, not source. It is NOT a baseline raise -- Tools/doc_comment_lint_baseline.txt is byte-identical, and the 25 grandfathered hand-written sites still ratchet down exactly as before. Two of uniffi's emissions defeat a fix even ignoring regeneration: * 4 sites are `/** ... */`, blank line, `public enum` -- uniffi's own spacing; * 1 site (:3325) is a parameter doc inlined into an argument list, `public init(/** Whether a full restore ran. */restored: Bool, ...)`. That block IS correctly bound, to its parameter. The scanner walks to the first line ending in `*/`, which that line does not, so it runs on to an unrelated close and reports a site that is not there. Nothing to repair. `/Generated/` currently matches exactly one directory, Strand/CloudSync/Generated, verified against all four globbed roots. Verification: `python3 Tools/doc_comment_lint.py` exits 0 -- "OK no NEW detached doc comments (1467 files, 25 baselined site(s) remaining)", against 5 regressions before. Baseline file untouched. Not fork-specific: any repo that checks in generator output hits this, so this is a candidate to send upstream rather than carry. --- Tools/doc_comment_lint.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/Tools/doc_comment_lint.py b/Tools/doc_comment_lint.py index a82755c2a6..e147202668 100644 --- a/Tools/doc_comment_lint.py +++ b/Tools/doc_comment_lint.py @@ -47,7 +47,17 @@ # after it is conventional and correct, so it must not be reported. FILE_LEVEL_PREFIXES = ("package ", "import ", "@file:") -SKIP_PARTS = ("/build/", "/.build/", "/DerivedData/") +# Build output, plus CODE GENERATOR output. `/Generated/` is the same kind of exclusion as the three +# build directories, not a baseline raise: this lint exists to catch a HAND-EDITING failure mode — +# "editing by INSERTION rather than in place" — and a generator never edits, it re-emits. The one +# directory this currently matches, Strand/CloudSync/Generated/, is `rm -rf`'d and rewritten by +# `Rust/build-ios.sh` on every build; its header reads "Do not edit by hand" and it already carries +# `// swiftlint:disable all`, so a fix applied there is deleted by the next regeneration and the +# finding returns. Two of uniffi's emissions defeat a fix outright: it puts a blank line between a +# `/** */` block and the `public enum` it documents (4 sites), and it inlines a parameter's doc INSIDE +# the argument list (`public init(/** … */restored: Bool, …)`), which this file's scanner mis-walks — +# that block is correctly bound to its parameter, so there is nothing there to repair. +SKIP_PARTS = ("/build/", "/.build/", "/DerivedData/", "/Generated/") def _is_doc_open(line: str) -> bool: From f46153f7954d6e55fc4ace90ea1674766f31b021 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 30 Jul 2026 19:59:09 -0400 Subject: [PATCH 158/171] SleepPSG: say which denominator every stage fraction uses MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tool has always printed both conventions, labelled, one table above the other. The README quoted one of them in the self-check and the other in the variant section without ever saying so, which read as two different answers to the same question: section 3, pooled over all scored epochs deep truth 13.76 % predicted 18.94 % bias +5.18 pp section 3, mean of per-subject percentages deep truth 14.76 % predicted 19.24 % bias +4.48 pp Both are right and both are measured on the same 26 773 epochs. The denominator WITHIN a night is identical either way — all scored epochs, wake included. What differs is the weighting ACROSS the 31 subjects: pooled gives a 490-minute night more weight than a 208-minute one, the per-subject mean gives every subject one vote. For deep that is a ~1 pp gap, which is large enough to look like a contradiction and small enough to go unnoticed. The self-check is stated in the per-subject mean because that is the convention the previous harness reported in, and reproducing it on any other convention would not be reproducing it. Section 6's variant table and every `bias pp` column in it are pooled, which is why the incumbent's deep bias reads +5.18 pp there. Neither changes; both are now named at the point of use. Adds a convention table to the README, labels every percentage in the self-check section, and records the same thing on `Variants.preNine30Guard`, whose docstring quotes the per-subject figures. Also un-rounds "REM 27.0 %" to 26.96 % — 27.0 was ambiguous between the two conventions (26.96 per-subject, 27.02 pooled) precisely where the distinction matters. Documentation only: no measurement, no threshold and no reported number changes. Verified against the full 31-subject cohort locally; `swift test` in Tools/SleepPSG 33 tests, 0 failures. --- Tools/SleepPSG/README.md | 39 +++++++++++++++---- .../SleepPSG/Sources/sleeppsg/Variants.swift | 6 +++ 2 files changed, 38 insertions(+), 7 deletions(-) diff --git a/Tools/SleepPSG/README.md b/Tools/SleepPSG/README.md index 4495f78c62..45b5d98f7a 100644 --- a/Tools/SleepPSG/README.md +++ b/Tools/SleepPSG/README.md @@ -100,6 +100,22 @@ improved kappa on all three of its benchmarks and was reverted 48 hours later fo from 6 % to 23 % awake — *"kappa doesn't guard stage-fraction calibration."* A harness that reported only kappa could not have seen that regression, and this one is built so it cannot repeat the mistake. +**Two conventions, both printed, and a percentage here is meaningless without naming which.** Every stage +fraction in this tool divides by **all scored epochs of the night, wake included** — that part never varies. +What varies is how the 31 subjects are combined: + +| convention | what it is | deep, shipped recipe | +|---|---|---| +| **pooled over all scored epochs** | one denominator of 26 773 epochs for the whole cohort; a long night carries more weight than a short one | predicted **18.94 %**, truth **13.76 %**, bias **+5.18 pp** | +| **mean of per-subject percentages** | each subject's own % of their own night, then averaged over 31 subjects; every subject counts once | predicted **19.24 %**, truth **14.76 %**, bias **+4.48 pp** | + +Section 3 prints both tables, one above the other, each labelled. They are the same epochs and the same +denominator *within* a night; the ~1 pp gap between them is the weighting across subjects, nothing else. +The pooled convention is what a cohort-level "% of night" means and is the one section 6's variant table +and every bias column use. The per-subject mean is what a clinician comparing individuals would use, and it +is the one the self-check below is stated in, because it is the convention the previous harness reported. +**Quote either, never both interchangeably, and always say which.** + `SleepStagerV2` fits nothing to data — every coefficient is fixed a priori — so there is no train/test split to make for the recipe itself. The leave-one-subject-out machinery exists for the fitted comparison models in section 5, which do need it. @@ -133,7 +149,8 @@ avoid. Each row is the shipped recipe with **one** named change: PR #987's awake-transition row, and each of PR #348's seven components measured alone, plus all seven together. Reported with kappa **and** the stage- fraction biases, because a component is an improvement only if it wins the first without wrecking the -second. +second. **Every percentage and every `bias pp` in this table is pooled over all 26 773 scored epochs** — +so the incumbent's deep bias reads +5.18 pp here, not the +4.48 pp the per-subject mean gives. The last row is not a candidate. `pre-#930 REM guard (provenance)` runs the REM-latency guard as it stood before PR #930 — a hard `c < 0.12 ? 3.0 : 0.0` step in the session-fraction domain — and exists to explain @@ -148,18 +165,25 @@ harness reported on this same dataset, so anyone can see at a glance whether the instrument. **Nothing here is tuned to those numbers**, and where they disagree the disagreement is reported rather than closed. +**Every percentage in this section is the mean of per-subject percentages**, because that is the convention +the previous harness reported in; the pooled figures for the same quantities are ~1 pp away and are printed +directly above it in section 3. The tool labels both tables, and the self-check block repeats the +convention in its own header so a figure copied out of it carries its denominator with it. + Every **truth-side** figure reproduces exactly: 31 subjects, **26 773** PSG-scored epochs, deep **14.76 %** -of night, truth median first-REM latency **88.5 min**. Predicted deep lands at **19.24 %** against a -reported 19.25 %, and four-class kappa at **0.356** against a reported 0.349. +of each subject's own night averaged over the 31 (pooled: 13.76 %), truth median first-REM latency +**88.5 min**. Predicted deep lands at **19.24 %** on the same convention against a reported 19.25 % +(pooled: 18.94 %), and four-class kappa at **0.356** against a reported 0.349. -The prediction side of REM does not match: more REM (27.0 % vs 20.8 %), REM F1 0.569 vs 0.515, and a first -REM period arriving much earlier (median 91.5 min vs 142.0). +The prediction side of REM does not match: more REM (**26.96 %** per-subject mean, against a reported +20.8 %), REM F1 0.569 vs 0.515, and a first REM period arriving much earlier (median 91.5 min vs 142.0). **The obvious explanation was tested and is wrong.** PR #930 replaced a fraction-domain `c < 0.12 ? 3.0 : 0` step with a graded penalty measured in minutes from sleep onset, and the old figures were reported while #930 was the candidate — so the incumbent they describe is the recipe *before* it. That predicts exactly this pattern. The `pre-#930` variant runs that guard, and it moves nothing: kappa 0.356 → 0.356, REM F1 -0.569 → 0.569, REM 27.02 % → 26.97 %. On 8-hour lab nights the guard barely binds either way. The +0.569 → 0.569, REM 27.02 % → 26.97 % (pooled — section 6's table, like every variant row, is pooled). +On 8-hour lab nights the guard barely binds either way. The hypothesis is falsified, and it is recorded here rather than deleted because the variant is what falsified it. @@ -167,7 +191,8 @@ What the residual is remains **unresolved**, and the tool is not tuned to close future explanation, both from the table above: - It cannot be the session window. `clock` drives the deep prior and the REM ramp alike; predicted deep - reproduces to 0.01 pp, so `clock` is the same quantity in both harnesses. + reproduces to 0.01 pp (19.24 % measured against 19.25 % reported, per-subject mean), so `clock` is the + same quantity in both harnesses. - It is REM-specific and it is not the latency guard, which the variant just ruled out. One suggestive coincidence, offered as an observation and not a conclusion: the previously reported "REM diff --git a/Tools/SleepPSG/Sources/sleeppsg/Variants.swift b/Tools/SleepPSG/Sources/sleeppsg/Variants.swift index 3a612c49e7..c1ad7667ce 100644 --- a/Tools/SleepPSG/Sources/sleeppsg/Variants.swift +++ b/Tools/SleepPSG/Sources/sleeppsg/Variants.swift @@ -127,6 +127,12 @@ enum Variants { /// The rebuild reproduces every TRUTH-side figure exactly (26 773 scored epochs, deep 14.76 % of night, /// truth first-REM 88.5 min) but lands the prediction side elsewhere — more REM, arriving earlier. /// + /// Those percentages are the MEAN OF PER-SUBJECT PERCENTAGES, which is the convention the old figures + /// were reported in; pooled over all 26 773 epochs the same truth deep fraction is 13.76 % and the same + /// predicted deep fraction is 18.94 %. `main.swift` prints both tables, labelled, for exactly this + /// reason — a stage fraction quoted without its convention is a number two people will disagree about + /// while both being right. Section 6's variant table, and every `bias pp` in it, is pooled. + /// /// One mechanism explains that whole pattern, and it is dated rather than mysterious: PR #930 replaced a /// hard `c < 0.12 ? 3.0 : 0.0` step in the SESSION-FRACTION domain with a graded penalty measured in /// MINUTES from sleep onset. Averaged over the epochs it covers, the graded guard is the weaker one, so From 5b2650db3e8e7d73f9b0270b16eac7a31c7a068d Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 3 Aug 2026 22:46:04 -0400 Subject: [PATCH 159/171] CloudSync: make /ingest a counted escalation, not a reflex, when liters is live MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every /ingest resets the server-side liters lineage (correct and unavoidable once the mirror is replaced wholesale), which forces the phone's next push to a full ~390 MB snapshot — which cannot complete in a background window, which fell back to /ingest again. Measured on-device 2026-08-03: 27 consecutive full-size segments; the one lineage that survived produced deltas of 125 KB and 198 B. The old rule (any non-push falls through to /ingest) even fired on liters' healthiest outcome: 'already in sync, nothing to ship' was answered with a 208 MB re-upload that destroyed the lineage that made the no-op possible. uploaded == 0 is four states, not one, so classify instead of guessing: - pushed -> success, retry streak resets - inSync -> verified (txid > 0, remote == local): success, 0 bytes. txid == 0 'synced' is vacuous (both sides agree on nothing, observed 17:16 right after a reset) and is NOT blessed as success — that would stamp lastUploadToken and make the next sync skip the upload entirely. - retryable -> throw litersRetryPending; the next sync retries the cheap delta with the lineage intact - unavailable -> trial off / not compiled / no destination: /ingest is simply the path, unchanged After litersRetryThreshold (3) consecutive retryables the fallback opens: a genuinely dead liters path (sink secret lost, rotated token, FFI regression) degrades to the proven whole-database upload rather than silence. The streak persists across relaunches and resets only on a liters success, so a persistently dead path ships every sync instead of every third. Verified: StrandTests (LitersFallbackPolicyTests + CloudSyncUploaderTests, 14/14) via xcodebuild test on macOS, CI-parity unsigned; NOOPiOS compile with LITERS validated separately. Deep-buffer drain and the edit pull are unaffected by the new throw — both run outside the upload's do/catch. --- Strand/CloudSync/CloudSyncUploader.swift | 227 ++++++++++++++++++----- StrandTests/CloudSyncUploaderTests.swift | 99 ++++++++++ 2 files changed, 276 insertions(+), 50 deletions(-) diff --git a/Strand/CloudSync/CloudSyncUploader.swift b/Strand/CloudSync/CloudSyncUploader.swift index 4e98fc6f06..bfb6ef1853 100644 --- a/Strand/CloudSync/CloudSyncUploader.swift +++ b/Strand/CloudSync/CloudSyncUploader.swift @@ -47,6 +47,15 @@ enum CloudSyncUploadError: LocalizedError, Equatable { /// EVERY upload path — fresh installs, a never-paired Mac container, and any future race — not /// just the test-host case (see `CloudSyncModel.isRunningUnderXCTest` for that separate guard). case emptyStore + /// A liters push failed and the fallback to `/ingest` was deliberately WITHHELD to protect the + /// delta lineage. Not a dead end: the next sync retries the (small) delta push, which is the + /// whole point — an automatic `/ingest` here would reset the server-side lineage and force the + /// next push to a full ~390 MB snapshot, which cannot complete in a background window, which + /// falls back to `/ingest` again: the loop that kept 2026-08-03's syncs at snapshot size all + /// day. `/ingest` still runs after `litersRetryThreshold` consecutive failures (see + /// `ingestEscalationAllowed`), so a genuinely dead liters path degrades to the proven + /// whole-database upload instead of silent staleness. + case litersRetryPending(streak: Int, reason: String) var errorDescription: String? { switch self { @@ -54,6 +63,8 @@ enum CloudSyncUploadError: LocalizedError, Equatable { return "Couldn't prepare the backup to upload. \(detail)" case .emptyStore: return "Nothing to upload yet — the local database is empty." + case .litersRetryPending(let streak, let reason): + return "Delta sync didn't complete (attempt \(streak)) — will retry next sync. \(reason)" } } } @@ -63,6 +74,87 @@ enum CloudSyncUploadError: LocalizedError, Equatable { /// "pull" half is `CloudSyncCoordinator`). A pure coordination step with no state of its own, like /// `CloudSyncCoordinator`, so it's a namespace of static functions rather than an instance. enum CloudSyncUploader { + // MARK: - Liters fallback policy (pure) + + /// What one liters push attempt amounted to, classified so `upload` can decide between the three + /// legitimate responses: report success, retry via liters next sync, or fall back to `/ingest`. + /// The classification exists because "didn't ship bytes" conflates four states with OPPOSITE + /// correct responses, and conflating them is what kept 2026-08-03's syncs at snapshot size: + /// a healthy "already in sync" was read as failure and answered with a 208 MB `/ingest`, which + /// reset the server-side lineage and forced the next push to a full snapshot. + enum LitersPushOutcome: Equatable { + /// Shipped bytes. The lineage advanced; the sync is delivered. + case pushed(bytes: Int) + /// Shipped nothing because there is nothing to ship: the lineage has content (`txid > 0`) + /// and the remote matches it exactly. Everything in the database is already on the server — + /// success, not failure. Falling through to `/ingest` from here is the single worst response + /// available: it re-uploads a couple hundred MB the server already has, then destroys the + /// lineage that made the no-op possible. + case inSync + /// Didn't ship and can't be trusted as delivered: a thrown push, a captured-but-unshipped + /// L0 (`uploaded == 0 && !synced`), or a vacuous sync on an empty lineage + /// (`txid == 0` — both sides agree on *nothing*, observed 2026-08-03 17:16 right after a + /// lineage reset). The correct response is to retry via liters next sync, NOT `/ingest`: + /// the retry is cheap (a delta, or one snapshot after a reset) and preserves the lineage, + /// while `/ingest` guarantees the next push is a full snapshot. + case retryable(String) + /// Liters cannot run by configuration: trial off, `.external` not in force, no destination, + /// or not compiled in. `/ingest` is not a fallback here — it is simply the path. + case unavailable(String) + } + + /// Pure classification of a push summary's scalar fields (kept scalar so this compiles and + /// tests without the LITERS xcframework). See `LitersPushOutcome` for what each case means. + static func classifyLitersPush(uploaded: UInt64, synced: Bool, + txid: UInt64, remoteTxid: UInt64, + bytesUploaded: UInt64) -> LitersPushOutcome { + if uploaded > 0 { return .pushed(bytes: Int(bytesUploaded)) } + if synced && txid > 0 && remoteTxid == txid { return .inSync } + if synced && txid == 0 { + return .retryable("empty lineage on both sides (txid=0) — the next push re-baselines") + } + return .retryable("captured but unshipped (uploaded=0 synced=\(synced) " + + "txid=\(txid) remoteTxid=\(remoteTxid))") + } + + /// How many consecutive `retryable` outcomes are tolerated before `/ingest` runs anyway. + /// Below the threshold, a failed delta push throws `litersRetryPending` and the sync retries + /// next cycle with the lineage intact. At or past it, data flow wins: a genuinely dead liters + /// path (sink secret lost in a redeploy, rotated token, FFI regression) degrades to the proven + /// whole-database upload rather than going silently stale. Three, not one, because the common + /// transient failures — a background window expiring mid-push, one 503 — must not cost the + /// lineage; and not ∞ because `/ingest` surviving as disaster recovery is deliberate + /// (liters has ~a week of production history; `/ingest` has months). + static let litersRetryThreshold = 3 + + /// Whether this sync should fall through to `/ingest` after a retryable liters outcome. + static func ingestEscalationAllowed(retryStreak: Int, + threshold: Int = litersRetryThreshold) -> Bool { + retryStreak >= threshold + } + + /// Consecutive retryable liters outcomes, persisted so the escalation decision survives app + /// relaunches (a background sync IS a fresh process often enough). Reset only by a liters + /// success (`pushed`/`inSync`) — deliberately NOT by a successful `/ingest`, so a persistently + /// dead liters path keeps flowing through `/ingest` without paying two throwaway syncs between + /// each delivery. + static let litersRetryStreakKey = "cloudsync.liters.retryStreak" + + static func litersRetryStreak() -> Int { + UserDefaults.standard.integer(forKey: litersRetryStreakKey) + } + + @discardableResult + static func bumpLitersRetryStreak() -> Int { + let next = litersRetryStreak() + 1 + UserDefaults.standard.set(next, forKey: litersRetryStreakKey) + return next + } + + static func resetLitersRetryStreak() { + UserDefaults.standard.removeObject(forKey: litersRetryStreakKey) + } + /// Produces a `.noopbak` at `dest` from `store`, returning `DataBackup.BackupResult` so a real /// export failure's message survives. Injectable so a test can supply canned bytes without /// touching the app's real on-disk database: `WhoopStore.inMemory()` test stores have no backing @@ -184,12 +276,41 @@ enum CloudSyncUploader { // * the client has a liters destination (it does not when LITERS is not compiled in, and // no test double has one). // - // A failure here is NOT fatal and NOT retried in place: it falls through to `/ingest` - // below, which is untouched and remains both the fallback and the recovery path. The one - // thing that must not happen is a sync that ships nothing and reports success. - if let result = await litersPushIfEnabled(client: client, walBytes: walBytes, - telemetry: telemetry) { - return result + // What happens when the push does NOT deliver is a POLICY, not a reflex (changed + // 2026-08-03). The old rule — any non-push falls through to `/ingest` — looked safe + // per-sync and was catastrophic as a loop: every `/ingest` resets the server-side liters + // lineage (see noop-cloud `ingest.ts` — correct and unavoidable once the mirror is + // replaced wholesale), which forces the next push to a full ~390 MB snapshot, which cannot + // complete in a background window, which fell through to `/ingest` again. Measured on VK's + // server: 27 consecutive full-size segments, and the one lineage that survived long enough + // produced deltas of 125 KB and 198 B. So: + // * a delivered push, or a verified "already in sync", returns — streak resets; + // * a retryable outcome throws `litersRetryPending` and the NEXT sync retries the cheap + // delta — the lineage is worth more than this one sync; + // * after `litersRetryThreshold` consecutive retryables, `/ingest` runs anyway — a + // genuinely dead liters path must degrade to the proven upload, not to silence; + // * `unavailable` (trial off / not compiled / no destination) goes straight to + // `/ingest`, which for those configurations is simply the path, as always. + switch await litersAttemptIfEnabled(client: client, walBytes: walBytes, + telemetry: telemetry) { + case .pushed(let bytes): + resetLitersRetryStreak() + return (bytes, nil) + case .inSync: + resetLitersRetryStreak() + return (0, nil) + case .retryable(let why): + let streak = bumpLitersRetryStreak() + guard ingestEscalationAllowed(retryStreak: streak) else { + NSLog("liters: retryable outcome (streak %d/%d) — withholding /ingest to protect " + + "the lineage; will retry the delta next sync. %@", + streak, litersRetryThreshold, why) + throw CloudSyncUploadError.litersRetryPending(streak: streak, reason: why) + } + NSLog("liters: %d consecutive retryable outcomes — escalating to /ingest " + + "(the lineage will reset and the next push re-baselines)", streak) + case .unavailable: + break } switch await exporter(store, tempURL) { @@ -219,12 +340,11 @@ enum CloudSyncUploader { } } - /// One page-replication push, or `nil` to mean "not this time — use `/ingest`". - /// - /// Returns `nil` for every reason a push does not happen (trial off, `.external` not in force, - /// no destination, liters not compiled in) *and* for a push that failed, because the caller's - /// only correct response to all of them is the same: fall through to the whole-database upload. - /// The distinction is preserved in the log, not in the control flow. + /// One page-replication push attempt, classified. The caller's response differs per case — + /// that difference IS the 2026-08-03 fix; see the switch in `upload` and + /// `LitersPushOutcome`'s per-case docs. The old shape (`nil` == "use /ingest" for every + /// non-push reason, failure and healthy no-op alike) is what turned one bad sync into a + /// permanent snapshot loop. /// /// `latestDay` is `nil` on this path by construction: `/liters` is a byte pipe into /// `mirror.sqlite` and answers with liters' protocol, not with `/ingest`'s @@ -247,11 +367,11 @@ enum CloudSyncUploader { UserDefaults.standard.set("\(outcome) \(Date())", forKey: litersBreadcrumbKey) } - private static func litersPushIfEnabled( + private static func litersAttemptIfEnabled( client: any CloudIngesting, walBytes: Int64, telemetry: SyncPushTelemetry? - ) async -> (bytes: Int, latestDay: String?)? { + ) async -> LitersPushOutcome { #if LITERS - guard SyncReplicationTrial.isEnabled else { return nil } + guard SyncReplicationTrial.isEnabled else { return .unavailable("trial off") } // `isEnabled` is the intent; `isInForce` is the reality. Pushing while SQLite still owns // wal_autocheckpoint means a foreign checkpoint can restart the WAL between pushes, and // every push then ships the whole database — measurably, not theoretically. @@ -259,11 +379,11 @@ enum CloudSyncUploader { NSLog("liters: trial is on but WAL checkpointing is still .automatic " + "(restart pending) — using /ingest for this sync") noteLiters("skipped: not-in-force (restart pending)") - return nil + return .unavailable("not-in-force (restart pending)") } guard let dest = client.litersDestination else { noteLiters("skipped: no liters destination") - return nil + return .unavailable("no liters destination") } do { @@ -279,47 +399,54 @@ enum CloudSyncUploader { token: dest.token) }.value - // A push that captured content into an L0 file but shipped NO file is not a completed - // sync, and `push()` does not throw for it — `synced`/`uploaded` are separate fields and - // the old code read neither. Returning here on `uploaded == 0` would report a sync that - // delivered nothing as a success, skip `/ingest`, and stamp `lastUploadToken` so the NEXT - // sync skips the upload too. Fall through to `/ingest` instead, which is the same answer - // this function gives every other way of not pushing. - guard summary.uploaded > 0 else { - NSLog("liters: push shipped no files (txid=%llu synced=%d remoteTxid=%llu) " - + "— using /ingest for this sync", summary.txid, summary.synced ? 1 : 0, - summary.remoteTxid) - noteLiters("no-op: uploaded=0 synced=\(summary.synced) txid=\(summary.txid) " - + "remoteTxid=\(summary.remoteTxid)") - return nil + // `uploaded == 0` is FOUR different states, not one — see `classifyLitersPush`. The + // stamp-the-token trap the old comment warned about is handled by the classifier: + // only `txid > 0 && remoteTxid == txid` counts as "in sync" (content verifiably on the + // server), so the vacuous post-reset sync (`txid == 0`, observed 2026-08-03 17:16) is + // retryable, never success — `lastUploadToken` is not stamped for it. + let outcome = classifyLitersPush(uploaded: summary.uploaded, synced: summary.synced, + txid: summary.txid, remoteTxid: summary.remoteTxid, + bytesUploaded: summary.bytesUploaded) + switch outcome { + case .pushed: + telemetry?.record(snapshotted: summary.snapshotted, + snapshotReason: summary.snapshotReason, + bytesUploaded: Int64(summary.bytesUploaded), + walBytes: walBytes, + txid: summary.txid) + NSLog("liters: pushed txid=%llu bytes=%llu snapshotted=%d reason=%@ walAtPush=%lld", + summary.txid, summary.bytesUploaded, summary.snapshotted ? 1 : 0, + summary.snapshotReason ?? "-", walBytes) + noteLiters("pushed: files=\(summary.uploaded) bytes=\(summary.bytesUploaded) " + + "txid=\(summary.txid) snapshotted=\(summary.snapshotted)") + case .inSync: + NSLog("liters: in sync at txid=%llu — nothing to ship", summary.txid) + noteLiters("in-sync: txid=\(summary.txid)") + case .retryable(let why): + NSLog("liters: push shipped no files (txid=%llu synced=%d remoteTxid=%llu) — %@", + summary.txid, summary.synced ? 1 : 0, summary.remoteTxid, why) + noteLiters("retryable: \(why)") + case .unavailable: + break // classifyLitersPush never returns this } - - telemetry?.record(snapshotted: summary.snapshotted, - snapshotReason: summary.snapshotReason, - bytesUploaded: Int64(summary.bytesUploaded), - walBytes: walBytes, - txid: summary.txid) - NSLog("liters: pushed txid=%llu bytes=%llu snapshotted=%d reason=%@ walAtPush=%lld", - summary.txid, summary.bytesUploaded, summary.snapshotted ? 1 : 0, - summary.snapshotReason ?? "-", walBytes) - noteLiters("pushed: files=\(summary.uploaded) bytes=\(summary.bytesUploaded) " - + "txid=\(summary.txid) snapshotted=\(summary.snapshotted)") - return (Int(summary.bytesUploaded), nil) + return outcome } catch { - // Deliberately swallowed. Every liters failure mode — an unreachable sink (503 from - // the proxy when LITERS_SINK_ENABLED is unset), a full volume (507), a lease conflict, - // a rotated token — is survivable by uploading the whole database instead, and a sync - // that ships data on the old path beats a sync that reports a new-path error. - NSLog("liters: push failed (%@) — falling back to /ingest for this sync", - String(describing: error)) + // Classified retryable, not swallowed into an automatic `/ingest`: a thrown push — an + // unreachable sink (503 when LITERS_SINK_ENABLED is unset), a full volume (507), a + // lease conflict, a rotated token, a background window expiring mid-push — retries as + // a cheap delta next sync while the lineage survives. If it keeps failing, the retry + // streak escalates to `/ingest` (see `upload`'s switch), so a persistently dead liters + // path still degrades to the proven whole-database upload rather than silence. + NSLog("liters: push failed (%@) — will retry next sync (streak %d/%d)", + String(describing: error), litersRetryStreak() + 1, litersRetryThreshold) // Verbatim, not a category: the whole point is that the next person to look does not // have to guess which of liters' failure modes this was. noteLiters("failed: \(error)") - return nil + return .retryable("push threw: \(error)") } #else _ = (client, walBytes, telemetry) - return nil + return .unavailable("LITERS not compiled in") #endif } } diff --git a/StrandTests/CloudSyncUploaderTests.swift b/StrandTests/CloudSyncUploaderTests.swift index a6e8ffcacb..7b49bbb40b 100644 --- a/StrandTests/CloudSyncUploaderTests.swift +++ b/StrandTests/CloudSyncUploaderTests.swift @@ -313,4 +313,103 @@ final class CloudSyncUploaderTelemetryTests: XCTestCase { XCTAssertEqual(result.bytes, 7) } } + +// MARK: - Liters fallback policy (pure) + +/// The 2026-08-03 fix: `uploaded == 0` is four states with opposite correct responses, and the +/// escalation to `/ingest` is a counted decision, not a reflex. These tests pin the classifier and +/// the escalation policy — pure functions over scalars, no replicator, no network, no LITERS +/// xcframework required. +final class LitersFallbackPolicyTests: XCTestCase { + override func setUp() { + super.setUp() + CloudSyncUploader.resetLitersRetryStreak() + } + + override func tearDown() { + CloudSyncUploader.resetLitersRetryStreak() + super.tearDown() + } + + // MARK: classification + + func testShippedBytesClassifyAsPushed() { + XCTAssertEqual( + CloudSyncUploader.classifyLitersPush(uploaded: 1, synced: true, + txid: 5, remoteTxid: 5, bytesUploaded: 65_789), + .pushed(bytes: 65_789)) + } + + /// The healthy no-op: lineage has content and the remote matches it exactly. The old code + /// answered this with a ~208 MB `/ingest` that then reset the lineage (observed 2026-08-03). + func testVerifiedInSyncIsSuccessNotFallback() { + XCTAssertEqual( + CloudSyncUploader.classifyLitersPush(uploaded: 0, synced: true, + txid: 27, remoteTxid: 27, bytesUploaded: 0), + .inSync) + } + + /// The vacuous case observed 2026-08-03 17:16, minutes after a server-side lineage reset: + /// `synced=true` at `txid=0` means both sides agree on NOTHING — no content is verifiably on + /// the server, so this must NOT be blessed as success (that would stamp `lastUploadToken` and + /// make the next sync skip the upload entirely). It must not trigger `/ingest` either — the + /// next liters push re-baselines. + func testVacuousSyncAtTxidZeroIsRetryable() { + let outcome = CloudSyncUploader.classifyLitersPush(uploaded: 0, synced: true, + txid: 0, remoteTxid: 0, bytesUploaded: 0) + guard case .retryable = outcome else { + return XCTFail("txid=0 vacuous sync must be retryable, got \(outcome)") + } + } + + func testCapturedButUnshippedIsRetryable() { + let outcome = CloudSyncUploader.classifyLitersPush(uploaded: 0, synced: false, + txid: 9, remoteTxid: 8, bytesUploaded: 0) + guard case .retryable = outcome else { + return XCTFail("captured-but-unshipped must be retryable, got \(outcome)") + } + } + + func testSyncedButRemoteBehindIsRetryable() { + // synced flag claiming true while remoteTxid lags txid is contradictory — never trust it + // as delivered. + let outcome = CloudSyncUploader.classifyLitersPush(uploaded: 0, synced: true, + txid: 9, remoteTxid: 7, bytesUploaded: 0) + guard case .retryable = outcome else { + return XCTFail("remote behind local must be retryable, got \(outcome)") + } + } + + // MARK: escalation + + /// Two retryable syncs throw-and-retry; the third runs `/ingest` anyway. Below-threshold + /// withholding protects the lineage from transient failures; the threshold itself keeps a + /// genuinely dead liters path flowing through the proven upload. + func testEscalationOpensAtThreshold() { + XCTAssertFalse(CloudSyncUploader.ingestEscalationAllowed(retryStreak: 1)) + XCTAssertFalse(CloudSyncUploader.ingestEscalationAllowed(retryStreak: 2)) + XCTAssertTrue(CloudSyncUploader.ingestEscalationAllowed(retryStreak: 3)) + XCTAssertTrue(CloudSyncUploader.ingestEscalationAllowed(retryStreak: 4), + "past-threshold streaks keep /ingest open — the streak only resets on a " + + "liters success, so a dead liters path ships every sync, not every third") + } + + func testStreakPersistsAndResets() { + XCTAssertEqual(CloudSyncUploader.litersRetryStreak(), 0) + XCTAssertEqual(CloudSyncUploader.bumpLitersRetryStreak(), 1) + XCTAssertEqual(CloudSyncUploader.bumpLitersRetryStreak(), 2) + XCTAssertEqual(CloudSyncUploader.litersRetryStreak(), 2, + "streak reads back from UserDefaults — it must survive a process relaunch, " + + "because background syncs are usually fresh processes") + CloudSyncUploader.resetLitersRetryStreak() + XCTAssertEqual(CloudSyncUploader.litersRetryStreak(), 0) + } + + func testRetryPendingErrorNamesTheAttemptAndTheReason() { + let error = CloudSyncUploadError.litersRetryPending(streak: 2, reason: "push threw: 503") + let text = error.errorDescription ?? "" + XCTAssertTrue(text.contains("attempt 2"), "user-facing text must carry the attempt count") + XCTAssertTrue(text.contains("retry"), "must say it will retry, not read as a dead end") + } +} #endif // CLOUD_SYNC From 6b6e3451923e827842d33d4ddd7861f5361d674e Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 5 Aug 2026 18:44:59 -0400 Subject: [PATCH 160/171] Raw history archive: keep unmapped-layout frames, and stop empties evicting them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reject archive is the only surviving copy of a historical record NOOP cannot decode — the strap frees it the instant the trim is acked. Two ways an on-demand WHOOP MG ECG record could still be lost. 1. The archive filter asked the wrong question for an unmapped layout. It kept a type-47 record only when the DECODE produced nothing usable (no unix, or neither heart rate nor motion), so a record from a layout with no field map that happened to yield those fields passed the screen and was kept nowhere at all. That has held so far only by accident: the unmapped branch of decodeWhoop5Historical reads no offsets, so nothing but hist_version ever reaches `parsed` today. One static schema field for type 47, or one partially-mapped new version, would silently reopen it. The decision is now structural: mappedWhoop5HistoricalVersions ([18, 20, 21, 26]) is the single source of truth behind both the decoder's dispatch and the archive filter, and any 5/MG record outside it is archived whatever it decoded. v18/v20/v21/v26 are unaffected, so nothing already understood starts being archived wholesale. WHOOP 4 is untouched: its unmapped versions go through the schema's v24 fallback, which keeps a record only when it validates physiologically and otherwise drops the biometrics, so those records already reach the archive by the decode-outcome route. 2. Retention evicted the evidence. With enable_raw_data_w_ecg armed the strap banks one 1584-byte record per second whose every byte from offset 21 to the CRC32 trailer is zero (invariant across 1,307 captured frames). At ~3.2 KB per hex JSONL line that is the whole 5 MB archive in ~25 minutes, so a real contact-window record was evicted by placeholders within the half hour. Eviction is now weakest-value-first rather than purely oldest-first: malformed lines, then surplus lines whose payload region is ENTIRELY zero, then surplus informative lines. The floor splits with it — the newest 64 informative lines per version stay protected, all-zero lines get a floor of 8, enough to date the artefact without owning the archive. The test is strictly all-zero: one non-zero byte anywhere makes a frame informative, because a small populated block inside an empty buffer is exactly what the archive is for. The cap stays 5 MB (volume was never the missing thing), and an over-cap rewrite now evicts to 7/8 of the cap so a 1 Hz stream no longer rewrites the whole file on the main actor once per record. Adds an operator affordance for checking after an experiment whether the record landed: Devices -> MG ECG -> "Check archived raw records" renders counts per layout version split by informative vs all-zero payload, plus the newest captures' phone-clock timestamps. Local file read, no strap traffic. The ECG opcode sequences are untouched. Tests: swift test (WhoopProtocol, 537 pass) covers the layout rule including a lockstep check that no version outside the mapped set decodes anything; StrandTests (1054 pass) covers zero-payload ranking, the strict all-zero test, and the summary. Both app targets build. Android twin not updated: evictLines/hasZeroPayload and the widened filter need the same change in com.noop before the platforms agree. --- .../WhoopProtocol/HistoricalStreams.swift | 23 +- .../Sources/WhoopProtocol/Interpreter.swift | 37 ++- .../UnmappedHistoricalLayoutTests.swift | 125 +++++++++ Strand/App/AppModel.swift | 4 + Strand/BLE/BLEManager.swift | 17 ++ Strand/BLE/LiveState.swift | 6 + Strand/Collect/RawHistoryArchive.swift | 245 ++++++++++++++---- Strand/Screens/DevicesView.swift | 42 +++ .../RawHistoryArchiveEvictionTests.swift | 117 +++++++++ 9 files changed, 562 insertions(+), 54 deletions(-) create mode 100644 Packages/WhoopProtocol/Tests/WhoopProtocolTests/UnmappedHistoricalLayoutTests.swift diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift index 5314674c6b..abdded3e90 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/HistoricalStreams.swift @@ -65,10 +65,11 @@ public func isPlausibleHistoricalUnix(_ ts: Int, wallNow: Int, return ts >= oldest - SESSION_RANGE_MARGIN && ts <= newest + SESSION_RANGE_MARGIN } -/// The HISTORICAL_DATA record frames in `rawFrames` that FAIL decode — a genuine CRC failure, or an -/// unmapped firmware layout whose envelope parsed but yielded no usable biometrics. These are the -/// records the strap is about to free once we ack the trim, so without an archive they are lost -/// forever while the UI reports a clean sync (#77 / #91). +/// The HISTORICAL_DATA record frames in `rawFrames` that NOOP cannot turn into rows — a genuine CRC +/// failure, an unmapped firmware layout (5/MG: any `hist_version` outside +/// `mappedWhoop5HistoricalVersions`), or a mapped layout whose envelope parsed but yielded no usable +/// biometrics. These are the records the strap is about to free once we ack the trim, so without an +/// archive they are lost forever while the UI reports a clean sync (#77 / #91). /// /// Console (type-50, `frame[typeIndex] == 0x32`) frames are strap-side debug-log text that decode to /// zero rows BY DESIGN and are never returned. 5/MG v26 (raw PPG block, hist_version 26) is also @@ -91,6 +92,20 @@ public func rejectedHistoricalRecords(_ rawFrames: [[UInt8]], family: DeviceFami // different type byte, so they never pass this gate — they are excluded by construction. guard f.count > typeIndex, Int(f[typeIndex]) == 47 else { return false } if family == .whoop5, f.count > versionIndex, Int(f[versionIndex]) == 26 { return false } // v26 PPG: has its own durable stream (ppgWaveform), not this reject archive + // UNMAPPED LAYOUT (5/MG) — archive UNCONDITIONALLY, whatever it decoded. + // + // The decode-outcome test below is the wrong question for a layout NOOP has no field map for. + // `decodeWhoop5Historical`'s unmapped branch reads no offsets, so anything that DOES appear in + // `parsed` for such a record came from the envelope, not from a mapped biometric — and a record + // that happened to yield a plausible `unix` plus a `gravity_x`/`heart_rate` used to pass the + // screen and be kept NOWHERE, its bytes freed by the very next trim ack. That is exactly the + // shape a novel record (an on-demand ECG capture, a new firmware's rollup) can take. + // + // This cannot flood the archive with records we already understand: v18/v20/v21/v26 are in + // `mappedWhoop5HistoricalVersions` and never reach here. Retention for the records that DO + // (`RawHistoryArchive.evictLines`) evicts entirely-zero-payload frames first, so a firmware that + // banks empty placeholder records at 1 Hz cannot push out the one informative frame either. + if family == .whoop5, isUnmappedWhoop5HistoricalRecord(f) { return true } let p = parseFrame(f, family: family) // Envelope/CRC reject: parse failed outright or the CRC32 trailer mismatched. if !p.ok || p.crcOK == false { return true } diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/Interpreter.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/Interpreter.swift index b3dac26860..1c6294ff2f 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/Interpreter.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/Interpreter.swift @@ -331,6 +331,26 @@ private func parseFrameWhoop5(_ frame: [UInt8], collectFields: Bool) -> ParsedFr fields: fb.fields, parsed: fb.parsed) } +/// The WHOOP 5/MG type-47 `hist_version` values `decodeWhoop5Historical` has a REAL field map for. +/// Every other version falls through to the "unmapped layout" branch, which decodes nothing and only +/// describes the payload as an opaque region. +/// +/// This is the single source of truth for "does NOOP understand this layout": the dispatch `switch` in +/// `decodeWhoop5Historical` reads nothing else, and `rejectedHistoricalRecords` uses it to decide which +/// records must be archived raw. Keeping one set means the two can never disagree — the failure mode +/// this replaced was a record from an unmapped layout that happened to decode a plausible `unix` and +/// `gravity_x`/`heart_rate` slipping past the archive filter, so its bytes were kept nowhere at all. +/// `Whoop5HistoricalLayoutMapTests` pins the set against the decoder's actual behaviour. +public let mappedWhoop5HistoricalVersions: Set = [18, 20, 21, 26] + +/// True when `frame` is a WHOOP 5/MG type-47 record whose layout version has NO field map — the +/// records that reach the unmapped branch of `decodeWhoop5Historical` and are therefore never +/// re-derivable from anything NOOP stores. Non-type-47 and too-short frames are false. +public func isUnmappedWhoop5HistoricalRecord(_ frame: [UInt8]) -> Bool { + guard frame.count > 9, Int(frame[8]) == 47 else { return false } + return !mappedWhoop5HistoricalVersions.contains(Int(frame[9])) +} + /// Decode a WHOOP 5.0 HISTORICAL_DATA (type 47) DSP biometric record. /// /// The layout version is carried in the byte at frame[9] — the inner record's seq slot, which the @@ -351,16 +371,21 @@ private func decodeWhoop5Historical(_ frame: [UInt8], fb: FieldBuilder, payloadE let version = frame.count > 9 ? Int(frame[9]) : -1 fb.parsed["hist_version"] = .int(version) fb.add(9, 1, "hist_version", "meta", value: .int(version)) - if version == 26 { + // One dispatch, one list: every `case` here must appear in `mappedWhoop5HistoricalVersions` and + // vice versa, because that set is what decides whether a record gets archived raw. A `switch` + // rather than the old if-chain so the mapped versions are readable in one place. + switch version { + case 26: decodeWhoop5HistoricalV26(frame, fb: fb) return - } - if version == 20 || version == 21 { + case 20, 21: decodeWhoop5HistoricalV2021(frame, fb: fb, version: version, payloadEnd: payloadEnd) return - } - guard version == 18 else { - // Unknown historical layout — describe it faithfully without inventing offsets. + case 18: + break // falls through to the v18 field decode below + default: + // Unknown historical layout — describe it faithfully without inventing offsets. The bytes are + // NOT lost: `rejectedHistoricalRecords` archives every record that lands here (#77 / #91). if let payloadEnd = payloadEnd, 11 < payloadEnd, payloadEnd <= frame.count { fb.region(11, payloadEnd, "HISTORICAL_DATA v\(version) (unmapped layout)", "unknown") } diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/UnmappedHistoricalLayoutTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/UnmappedHistoricalLayoutTests.swift new file mode 100644 index 0000000000..9078e864d2 --- /dev/null +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/UnmappedHistoricalLayoutTests.swift @@ -0,0 +1,125 @@ +import XCTest +@testable import WhoopProtocol + +/// The archive decision for a WHOOP 5/MG type-47 record must be made on its LAYOUT VERSION, not on +/// what the record happened to decode to. +/// +/// `rejectedHistoricalRecords` used to ask only "did this decode to a `unix` plus a +/// `heart_rate`/`gravity_x`?". That screen is the wrong question for a layout NOOP has no field map +/// for: a record from an unmapped version that answered yes was kept NOWHERE — not as rows (nothing +/// mapped it) and not as bytes (it passed the archive filter) — and the strap freed it on the very next +/// trim ack. An on-demand capture (the MG's ~30 s ECG record, banked to flash and pulled back in a +/// later offload) is exactly the shape of record that can look well-formed enough to pass. +/// +/// The screen survived in practice only because of an accident: the unmapped branch of +/// `decodeWhoop5Historical` reads no offsets, so today nothing but `hist_version` ever lands in +/// `parsed` for such a record (`testUnmappedLayoutsDecodeNothingButTheVersionByte` pins that this is +/// true right now). One added static schema field for type 47, or one partially-mapped new version, +/// would silently reopen the hole. These tests pin the version-based decision instead, so the archive +/// no longer depends on that accident holding. +final class UnmappedHistoricalLayoutTests: XCTestCase { + + private func bytes(_ s: String) -> [UInt8] { + var out = [UInt8](); out.reserveCapacity(s.count / 2); var i = s.startIndex + while i < s.endIndex { let j = s.index(i, offsetBy: 2) + out.append(UInt8(s[i.. [UInt8] { + var f = bytes(whoop5V18Hex) + f[9] = version + let payloadEnd = f.count - 4 + let c = crc32(f, 8, payloadEnd) + f[payloadEnd] = UInt8(c & 0xFF) + f[payloadEnd + 1] = UInt8((c >> 8) & 0xFF) + f[payloadEnd + 2] = UInt8((c >> 16) & 0xFF) + f[payloadEnd + 3] = UInt8((c >> 24) & 0xFF) + return f + } + + // MARK: - defect: an unmapped layout that decodes real biometrics must still be archived + + /// The headline case. The frame's BYTES are a real v18 record — at the v18 offsets they carry a + /// valid unix, a plausible heart rate and a ~1 g gravity vector, the exact combination that made + /// the old screen say "decodable, nothing lost". Only the layout-version byte differs. Whatever + /// such a record decodes to, its bytes must reach the archive. + func testUnmappedLayoutIsArchivedEvenThoughItsBytesDecodeCleanlyAsV18() { + // Precondition: read as its true version, this record decodes everything the old screen wanted. + let asV18 = parseFrame(bytes(whoop5V18Hex), family: .whoop5) + XCTAssertEqual(asV18.crcOK, true) + XCTAssertNotNil(asV18.parsed["unix"], "precondition: these bytes DO decode a unix") + XCTAssertNotNil(asV18.parsed["heart_rate"], "precondition: these bytes DO decode a heart rate") + XCTAssertNotNil(asV18.parsed["gravity_x"], "precondition: these bytes DO decode a gravity vector") + + let unmapped = whoop5FrameWithVersion(22) + XCTAssertEqual(parseFrame(unmapped, family: .whoop5).crcOK, true, + "precondition: the record is CRC-VALID, so only the layout decision can archive it") + XCTAssertTrue(isUnmappedWhoop5HistoricalRecord(unmapped)) + XCTAssertEqual(rejectedHistoricalRecords([unmapped], family: .whoop5), [unmapped], + "a record from a layout NOOP cannot map must be archived whatever it decoded") + } + + /// Every version outside `mappedWhoop5HistoricalVersions` is archived — no gaps, no lucky values. + func testEveryUnmappedVersionIsArchived() { + for v in 0...255 where !mappedWhoop5HistoricalVersions.contains(v) { + let f = whoop5FrameWithVersion(UInt8(v)) + XCTAssertEqual(rejectedHistoricalRecords([f], family: .whoop5), [f], + "hist_version \(v) has no field map, so its bytes must be archived") + } + } + + // MARK: - lockstep: the version set and the decoder's dispatch cannot drift apart + + /// `mappedWhoop5HistoricalVersions` is the single source of truth for the archive decision, so it + /// must not claim more (or fewer) versions than `decodeWhoop5Historical` actually maps. Proved from + /// the decoder's own behaviour: for EVERY version outside the set, the decode yields nothing but + /// `hist_version` — i.e. the dispatch `switch` has no case the set omits. + func testUnmappedLayoutsDecodeNothingButTheVersionByte() { + for v in 0...255 where !mappedWhoop5HistoricalVersions.contains(v) { + let p = parseFrame(whoop5FrameWithVersion(UInt8(v)), family: .whoop5) + XCTAssertEqual(p.parsed.keys.sorted(), ["hist_version"], + "v\(v) is outside the mapped set but the decoder populated fields for it — " + + "the set and the dispatch switch have drifted") + } + } + + /// The four mapped versions ARE recognised as mapped (the other direction of the lockstep). + func testMappedVersionsAreNotTreatedAsUnmapped() { + XCTAssertEqual(mappedWhoop5HistoricalVersions, [18, 20, 21, 26]) + for v in mappedWhoop5HistoricalVersions { + XCTAssertFalse(isUnmappedWhoop5HistoricalRecord(whoop5FrameWithVersion(UInt8(v))), + "v\(v) has a field map and must not be archived on layout grounds") + } + } + + /// A cleanly-decoding v18 record is still NOT archived — widening the screen must not start + /// archiving the layouts NOOP already understands wholesale. + func testMappedV18RecordStillNotArchived() { + XCTAssertTrue(rejectedHistoricalRecords([bytes(whoop5V18Hex)], family: .whoop5).isEmpty) + } + + /// The layout rule is WHOOP 5-only: it keys off `frame[9]`, which on a WHOOP 4 frame is a payload + /// byte, not a version. WHOOP 4's unmapped versions go through the schema's validated v24 fallback + /// (`PostHooks`), which keeps a record only when it decodes to a ~1 g gravity vector and a plausible + /// HR and otherwise drops the biometrics — so those records reach the archive by the decode-outcome + /// route and must not be dragged in by this one. + func testWhoop4FramesAreUnaffectedByTheWhoop5LayoutRule() { + // A synthetic WHOOP 4 V24 type-47 record (HR=63) that decodes cleanly (from HistoricalV24Tests). + let v24Hex = + "aa5a008e2f18000000000000f153650000000000003f0152030000000000000000dc053075" + + "000000cdcc4c3dcdcccc3d5a657e3f00000040cdcc4c3dcdcccc3d5a657e3f504668428403" + + "200364006400b80bb80b000000000000c25c1a88" + XCTAssertTrue(rejectedHistoricalRecords([bytes(v24Hex)], family: .whoop4).isEmpty) + XCTAssertFalse(isUnmappedWhoop5HistoricalRecord(bytes(v24Hex)), + "a WHOOP 4 frame has no type-47 byte at index 8 — the rule must not fire on it") + } +} diff --git a/Strand/App/AppModel.swift b/Strand/App/AppModel.swift index eb561f1527..f6a9fc747a 100644 --- a/Strand/App/AppModel.swift +++ b/Strand/App/AppModel.swift @@ -885,6 +885,10 @@ final class AppModel: ObservableObject { /// the Devices result sheet from another screen. func ecgStopCapture(reportsResult: Bool = true) { ble.ecgStopCapture(reportsResult: reportsResult) } func clearEcgProbe() { ble.clearEcgProbe() } + /// Local, read-only: render what the raw-history reject archive currently holds. Sends nothing to + /// the strap — the check for whether an experiment's record actually landed and survived retention. + func reportRawHistoryArchive() { ble.reportRawHistoryArchive() } + func clearRawHistoryArchiveReport() { ble.clearRawHistoryArchiveReport() } /// True once a start has been sent this session and no stop has completed — keeps the Stop control /// reachable even after the opt-in has been switched back off. var ecgMayBeRunning: Bool { ble.ecgMayBeRunning } diff --git a/Strand/BLE/BLEManager.swift b/Strand/BLE/BLEManager.swift index 806adc7197..9ad736c30b 100644 --- a/Strand/BLE/BLEManager.swift +++ b/Strand/BLE/BLEManager.swift @@ -3268,6 +3268,23 @@ public final class BLEManager: NSObject, ObservableObject { /// Clear the probe result (dialog dismissed). public func clearEcgProbe() { state.ecgProbe = nil } + /// Render the raw-history reject archive's current state into `state.rawHistoryArchiveReport` and the + /// strap log. Read-only and strap-free — it opens a local file; no command is sent, so it is not + /// gated on a connection, on the Experimental opt-in, or on an MG. + /// + /// This exists because "the ECG probe saw N packets" does not answer whether the RECORD survived: + /// the record lands in a later history offload, and retention may already have evicted it. The + /// summary answers that directly — counts per layout version split by informative vs all-zero + /// payload, and the newest captures' phone-clock timestamps to match against the contact window. + public func reportRawHistoryArchive() { + let text = rejectedHistoryArchive.summaryText() + state.rawHistoryArchiveReport = text + log("Raw history archive:\n\(text)") + } + + /// Clear the archive report (dialog dismissed). + public func clearRawHistoryArchiveReport() { state.rawHistoryArchiveReport = nil } + private func beginEcgProbeRun(clearingSteps: Bool) { if clearingSteps { ecgProbeSteps = [] diff --git a/Strand/BLE/LiveState.swift b/Strand/BLE/LiveState.swift index 98658d1012..8f1c470377 100644 --- a/Strand/BLE/LiveState.swift +++ b/Strand/BLE/LiveState.swift @@ -306,6 +306,12 @@ public final class LiveState: ObservableObject { /// explicitly not a medical measurement. @Published public var ecgProbe: String? = nil + /// A rendered snapshot of the raw-history reject archive (`RawHistoryArchive.summaryText`), shown + + /// copied from the Devices dialog. Purely local: it reads an on-disk file and sends the strap + /// nothing. The affordance for checking, straight after an experiment, whether the record you were + /// hunting is actually in the archive rather than already evicted. Cleared on dialog dismiss. + @Published public var rawHistoryArchiveReport: String? = nil + /// The 5-generation hardware variant resolved from the strap's Device Information Service /// (`Whoop5Variant.label`: "MG" / "5.0" / "—"), nil before any DIS string has landed. Published so an /// MG-only capability can gate on POSITIVELY identified hardware instead of guessing from a model diff --git a/Strand/Collect/RawHistoryArchive.swift b/Strand/Collect/RawHistoryArchive.swift index bff5ab0bb3..b7594dae24 100644 --- a/Strand/Collect/RawHistoryArchive.swift +++ b/Strand/Collect/RawHistoryArchive.swift @@ -18,11 +18,16 @@ import WhoopProtocol struct RawHistoryArchive { /// File name under `/com.noopapp.noop/`. static let fileName = "rejected_history.jsonl" - /// Soft cap (~5 MB). When appending would push the file past this, the archive EVICTS oldest - /// surplus lines to make room rather than refusing the write — but only down to a per-version - /// retention floor (see `perVersionFloor`), so a brand-new layout version is never binned merely - /// because common versions filled the file. Only when the incoming frames alone can't fit even an - /// empty archive does `archive` skip them (reported as unarchived). (#344) + /// Soft cap (~5 MB). When appending would push the file past this, the archive EVICTS surplus lines + /// to make room rather than refusing the write — weakest-value first, and never below a per-version + /// retention floor, so a brand-new layout version is never binned merely because common versions + /// filled the file (see `evictLines` for the full priority order). Only when the incoming frames + /// alone can't fit even an empty archive does `archive` skip them (reported as unarchived). (#344) + /// + /// Deliberately NOT raised to survive a placeholder flood. At ~3.2 KB per hex line a 1 Hz stream of + /// 1584-byte records fills any cap in hours, so a bigger number buys a linear factor of time and + /// costs every user disk — while `zeroPayloadFloor` removes the pressure outright by evicting the + /// informationless records first. Value, not volume, is what was missing. static let maxBytes = 5 * 1024 * 1024 /// Per distinct layout VERSION, keep at least this many of the newest archived lines, immune to @@ -32,6 +37,48 @@ struct RawHistoryArchive { /// can hold at most `floor × distinctVersions` lines, a tiny set. (#344) static let perVersionFloor = 64 + /// Per-version retention floor for lines whose PAYLOAD REGION IS ENTIRELY ZERO — far smaller than + /// `perVersionFloor`, because such a record carries no information beyond "the strap banked a record + /// at time T". + /// + /// Measured motivation: with `enable_raw_data_w_ecg` armed, a WHOOP MG banks one 1584-byte record + /// PER SECOND whose every byte from offset 21 to the CRC32 trailer is zero (invariant across 1,307 + /// captured frames). At ~3.2 KB per hex JSONL line that fills the whole 5 MB archive in ~25 minutes, + /// so under a purely age-ordered policy an actual ECG record captured during the electrode-contact + /// window is evicted by placeholders within the half hour — the most likely reason the 2026-07-27 + /// contact frames were already gone by the 2026-08-03 analysis. Keeping a handful still proves the + /// artefact exists (and dates it); keeping thousands buys nothing and costs the evidence. + static let zeroPayloadFloor = 8 + + /// First byte of a record's PAYLOAD region — everything past the record header, ending at the 4-byte + /// CRC32 trailer. 21 on WHOOP 5/MG (magic @0, seq @11, unix ts @15–18, a constant @19–20), and 17 on + /// WHOOP 4: the same +4 that shifts the type byte 4→8 and the version byte 5→9, because the puffin + /// envelope is 4 bytes longer. Starting AFTER the header is load-bearing — a region that included the + /// non-zero seq/timestamp bytes would never test as all-zero and the rule below would never fire. + static func payloadStart(_ family: DeviceFamily) -> Int { family == .whoop5 ? 21 : 17 } + + /// Bytes of the CRC32 trailer excluded from the payload region. + static let crcTrailerBytes = 4 + + /// A payload region shorter than this is not judged at all (`hasZeroPayload` returns false): a + /// truncated or tiny frame is not evidence of an empty record, and must not lose retention priority + /// on the strength of a few zero bytes. + static let minPayloadBytesToJudge = 32 + + /// True when EVERY byte of the frame's payload region is zero — an informationless record. + /// + /// Deliberately the strict test: a single non-zero byte anywhere in the region makes the frame + /// informative and gives it full retention priority. "Mostly zero" is NOT the test — a novel record + /// that carries one small populated block inside an otherwise-empty buffer is precisely the thing + /// this archive exists to catch. + static func hasZeroPayload(_ frame: [UInt8], family: DeviceFamily) -> Bool { + let start = payloadStart(family) + let end = frame.count - crcTrailerBytes + guard end - start >= minPayloadBytesToJudge else { return false } + for i in start../com.noopapp.noop/`, created on demand. Overridable for tests. init(directory: URL? = nil, maxBytes: Int = RawHistoryArchive.maxBytes, - perVersionFloor: Int = RawHistoryArchive.perVersionFloor) { + perVersionFloor: Int = RawHistoryArchive.perVersionFloor, + zeroPayloadFloor: Int = RawHistoryArchive.zeroPayloadFloor) { if let directory { self.directory = directory } else { @@ -65,6 +114,7 @@ struct RawHistoryArchive { } self.maxBytes = maxBytes self.perVersionFloor = perVersionFloor + self.zeroPayloadFloor = zeroPayloadFloor } /// The archive file URL (does not create anything). @@ -135,7 +185,8 @@ struct RawHistoryArchive { // within the newest `perVersionFloor` of its version. `evictLines` is the pure, unit-tested core. let existing = (try? String(contentsOf: url, encoding: .utf8)) .map { $0.split(separator: "\n", omittingEmptySubsequences: true).map { String($0) + "\n" } } ?? [] - let kept = RawHistoryArchive.evictLines(existing + newLines, maxBytes: maxBytes, floor: perVersionFloor) + let kept = RawHistoryArchive.evictLines(existing + newLines, maxBytes: maxBytes, + floor: perVersionFloor, zeroFloor: zeroPayloadFloor) let data = Data(kept.joined().utf8) do { try FileManager.default.createDirectory(at: directory, withIntermediateDirectories: true) @@ -160,11 +211,26 @@ struct RawHistoryArchive { } } - /// The `VersionKey` of one archived JSONL line, re-derived from its stored frame + family; `nil` - /// for a malformed/unparseable line — those get NO retention floor (evict them first, never let - /// garbage occupy a floor slot a real rare version could use), and are always evictable before any - /// real, floor-protected line. - private static func lineVersionKey(_ line: String) -> VersionKey? { + /// How one archived line is classified for retention: which version bucket it belongs to, and + /// whether its payload region is entirely zero (informationless — see `hasZeroPayload`). + private struct LineClass { let key: VersionKey; let zeroPayload: Bool } + + /// Classify one archived JSONL line from its stored frame + family; `nil` for a + /// malformed/unparseable line — those get NO retention floor (evict them first, never let garbage + /// occupy a floor slot a real rare version could use), and are always evictable before any real, + /// floor-protected line. + private static func lineClass(_ line: String) -> LineClass? { + guard let (bytes, family) = parseArchiveLine(line) else { return nil } + return LineClass(key: VersionKey(family: family.rawValue, + version: versionByte(bytes, family: family)), + zeroPayload: hasZeroPayload(bytes, family: family)) + } + + /// The frame bytes + family of one archived JSONL line, or `nil` if the line is malformed. + /// Hand-parsed to match the hand-built writer: the only dynamic fields are `family` and the + /// [0-9a-f] `frameHex`. Shared by `readAll`, retention, and the operator summary so all three agree + /// on exactly which lines are readable. Twin of the Android `RawHistoryArchive.parseArchiveLine`. + static func parseArchiveLine(_ line: S) -> (frame: [UInt8], family: DeviceFamily)? { guard let fr = line.range(of: "\"family\":\""), let hr = line.range(of: "\"frameHex\":\"") else { return nil } let fam = String(line[fr.upperBound...].prefix { $0 != "\"" }) @@ -176,36 +242,73 @@ struct RawHistoryArchive { guard let b = UInt8(hex[i.. [String] { - // Mark the newest `floor` indices of each version as protected (scan newest→oldest). Malformed - // lines (nil key) are never protected — they are evicted before any real, floor-kept line. - let keys = lines.map { lineVersionKey($0) } + /// newline-terminated JSONL set, oldest-first (existing + incoming). + /// + /// Retention priority, weakest first — everything in a band is evicted before anything in the next: + /// 1. malformed lines (unparseable — no version, no floor); + /// 2. surplus lines whose payload region is ENTIRELY ZERO (`hasZeroPayload`), oldest first; + /// 3. surplus informative lines, oldest first; + /// 4. floor-protected lines, never evicted: the newest `zeroFloor` zero-payload lines of each + /// version and the newest `floor` informative lines of each version. + /// + /// Band 2 is what keeps an ECG capture recoverable. A 5/MG with `enable_raw_data_w_ecg` armed banks + /// one all-zero 1584-byte record per second; age-ordered eviction alone would flush a real + /// contact-window record out of a 5 MB archive within ~25 minutes of placeholders, and an all-zero + /// payload demonstrably carries nothing that would be worth that trade. Band 4's split floor means a + /// version whose records are ALL empty still keeps a few dated samples of the artefact, while the + /// full `floor` stays reserved for records that actually contain something. + /// + /// Bounded: if everything left is floor-protected it stops even if still over cap (the floor wins — + /// `(floor + zeroFloor) × distinctVersions` is tiny). The Android twin has not been updated yet. + static func evictLines(_ lines: [String], maxBytes: Int, floor: Int, + zeroFloor: Int = RawHistoryArchive.zeroPayloadFloor) -> [String] { + // Mark the newest `floor` informative and newest `zeroFloor` zero-payload indices of each version + // as protected (scan newest→oldest). Malformed lines (nil class) are never protected — they are + // evicted before any real, floor-kept line. + let classes = lines.map { lineClass($0) } var protectedIdx = Set() - var seen: [VersionKey: Int] = [:] + var seenInformative: [VersionKey: Int] = [:] + var seenZero: [VersionKey: Int] = [:] for idx in lines.indices.reversed() { - guard let k = keys[idx] else { continue } - let c = (seen[k] ?? 0) - if c < floor { protectedIdx.insert(idx); seen[k] = c + 1 } + guard let c = classes[idx] else { continue } + if c.zeroPayload { + let n = (seenZero[c.key] ?? 0) + if n < zeroFloor { protectedIdx.insert(idx); seenZero[c.key] = n + 1 } + } else { + let n = (seenInformative[c.key] ?? 0) + if n < floor { protectedIdx.insert(idx); seenInformative[c.key] = n + 1 } + } } var total = lines.reduce(0) { $0 + $1.utf8.count } guard total > maxBytes else { return lines } - // Evict oldest unprotected lines first. + let target = maxBytes - maxBytes / lowWaterDivisor + // Evict in weakest-first bands; within a band, oldest first (ascending index). var dropped = Set() - for idx in lines.indices where total > maxBytes { // ascending = oldest first - if protectedIdx.contains(idx) { continue } - dropped.insert(idx) - total -= lines[idx].utf8.count + func evict(_ admits: (Int) -> Bool) { + for idx in lines.indices where total > target { + if protectedIdx.contains(idx) || dropped.contains(idx) { continue } + guard admits(idx) else { continue } + dropped.insert(idx) + total -= lines[idx].utf8.count + } } + evict { classes[$0] == nil } // 1. malformed + evict { classes[$0]?.zeroPayload == true } // 2. zero-payload surplus + evict { _ in true } // 3. informative surplus return lines.enumerated().filter { !dropped.contains($0.offset) }.map(\.element) } @@ -214,22 +317,76 @@ struct RawHistoryArchive { /// match the hand-built writer: the only dynamic fields are `family` and the [0-9a-f] `frameHex`.) func readAll() -> [(frame: [UInt8], family: DeviceFamily)] { guard let text = try? String(contentsOf: fileURL, encoding: .utf8) else { return [] } - return text.split(separator: "\n").compactMap { line in - guard let fr = line.range(of: "\"family\":\""), - let hr = line.range(of: "\"frameHex\":\"") else { return nil } - let fam = String(line[fr.upperBound...].prefix { $0 != "\"" }) - let hex = line[hr.upperBound...].prefix { $0 != "\"" } - guard let family = DeviceFamily(rawValue: fam), hex.count % 2 == 0 else { return nil } - var bytes = [UInt8](); bytes.reserveCapacity(hex.count / 2); var i = hex.startIndex - while i < hex.endIndex { - let j = hex.index(i, offsetBy: 2) - guard let b = UInt8(hex[i.. String { + let text = (try? String(contentsOf: fileURL, encoding: .utf8)) ?? "" + let lines = text.split(separator: "\n", omittingEmptySubsequences: true).map(String.init) + let size = (try? FileManager.default.attributesOfItem(atPath: fileURL.path)) + .flatMap { $0[.size] as? Int } ?? 0 + return "\(fileURL.path)\n\(RawHistoryArchive.summaryText(lines: lines, sizeBytes: size, newest: newest))" + } + + /// The pure renderer behind `summaryText()` — takes the raw JSONL lines so it is unit-testable + /// without a file. `lines` is oldest-first, exactly as stored. + static func summaryText(lines: [String], sizeBytes: Int, newest: Int = 8) -> String { + guard !lines.isEmpty else { return "Raw history archive: EMPTY (no frames archived yet)." } + struct Bucket { var informative = 0; var zero = 0 } + var buckets: [String: Bucket] = [:] + var malformed = 0 + var rows: [String] = [] + for line in lines { + guard let (frame, family) = parseArchiveLine(line) else { malformed += 1; continue } + let version = versionByte(frame, family: family) + let label = "\(family.rawValue) v\(version)" + let isZero = hasZeroPayload(frame, family: family) + var b = buckets[label] ?? Bucket() + if isZero { b.zero += 1 } else { b.informative += 1 } + buckets[label] = b + // Sliding window over the newest `newest` readable lines (the file is oldest-first). + if newest > 0 { + if rows.count == newest { rows.removeFirst() } + rows.append(" \(stamp(line)) \(label) \(frame.count) B \(isZero ? "zero-payload" : "INFORMATIVE")") } - return (bytes, family) } + var out = "Raw history archive: \(lines.count) line(s), \(sizeBytes / 1024) KB" + out += "\n\nBy layout version:" + for label in buckets.keys.sorted() { + let b = buckets[label]! + out += "\n \(label) \(b.informative + b.zero) line(s) — \(b.informative) informative, \(b.zero) all-zero payload" + } + if malformed > 0 { out += "\n (malformed) \(malformed) line(s)" } + if !rows.isEmpty { + out += "\n\nNewest \(rows.count) capture(s), phone clock:\n" + rows.joined(separator: "\n") + } + return out } + /// `capturedAtMs` of one archived line rendered in the phone's local time, or `?` when unreadable. + private static func stamp(_ line: String) -> String { + guard let r = line.range(of: "\"capturedAtMs\":"), + let ms = Double(line[r.upperBound...].prefix { $0 == "." || ("0"..."9").contains($0) }) + else { return "????-??-?? ??:??:??" } + return stampFormatter.string(from: Date(timeIntervalSince1970: ms / 1000)) + } + + private static let stampFormatter: DateFormatter = { + let f = DateFormatter() + f.dateFormat = "yyyy-MM-dd HH:mm:ss" + return f + }() + /// Re-decode every archived frame through the CURRENT decoder and insert whatever now decodes. /// The strap freed these records when they were acked, so this archive is the ONLY way banked /// history backfills after a newly-landed layout (e.g. WHOOP 4.0 v25). Idempotent: offloaded rows diff --git a/Strand/Screens/DevicesView.swift b/Strand/Screens/DevicesView.swift index 82e064e071..02a9befca7 100644 --- a/Strand/Screens/DevicesView.swift +++ b/Strand/Screens/DevicesView.swift @@ -1217,6 +1217,8 @@ private struct EcgProbeSheets: ViewModifier { Button("Start ECG capture") { model.ecgStartCapture(); target = nil } Button("Stop ECG capture") { model.ecgStopCapture(); target = nil } Button("Set which wrist you wear it on…") { target = nil; wristTarget = device } + // Local file read, no strap traffic: did the record actually land in the raw archive? + Button("Check archived raw records") { model.reportRawHistoryArchive(); target = nil } Button("Cancel", role: .cancel) { target = nil } } message: { _ in Text("NOOP is not a medical device and this is not an ECG test. It asks your MG to start its ECG subsystem and logs whatever comes back — unvalidated instrumentation for protocol research, never a measurement or a diagnosis, including any heart-rhythm classification the strap happens to send. Don't use it to make a health decision; see a doctor if you have symptoms.\n\nHold the two indents on the clasp with the fingers of your other hand for the whole capture. The MG measures across your wrist AND that clasp, so until you hold it the circuit is open, the strap has nothing to record, and you would see zero packets whatever the firmware did.\n\nNobody has confirmed a strap honours these commands, so the likely outcome is that nothing happens. Everything here is reversible: “Stop” turns the streams back off. Results land in the strap log.") @@ -1237,6 +1239,46 @@ private struct EcgProbeSheets: ViewModifier { EcgProbeResultView(text: live.ecgProbe ?? "", onClose: { model.clearEcgProbe() }) } + .sheet(isPresented: Binding(get: { live.rawHistoryArchiveReport != nil }, + set: { if !$0 { model.clearRawHistoryArchiveReport() } })) { + RawHistoryArchiveReportView(text: live.rawHistoryArchiveReport ?? "", + onClose: { model.clearRawHistoryArchiveReport() }) + } + } +} + +/// What the on-disk raw-history reject archive currently holds, rendered by +/// `RawHistoryArchive.summaryText`. Structurally the #592/#690/ECG result views with the medical +/// disclaimer dropped: this reports the contents of a local file, not anything a strap said. +/// +/// The two lines to read after an experiment are the per-version split (an "all-zero payload" count in +/// the thousands is a placeholder flood, not data) and the newest capture timestamps, which is what you +/// match against the electrode-contact window. +private struct RawHistoryArchiveReportView: View { + let text: String + let onClose: () -> Void + + var body: some View { + VStack(alignment: .leading, spacing: 16) { + Text("Archived raw records") + .font(StrandFont.title2) + .foregroundStyle(StrandPalette.textPrimary) + ScrollView { + Text(text) + .font(StrandFont.mono) + .foregroundStyle(StrandPalette.textSecondary) + .textSelection(.enabled) + .frame(maxWidth: .infinity, alignment: .leading) + } + HStack { + Button("Copy") { PlatformPasteboard.copy(text) } + Spacer() + Button("Close") { onClose() } + } + } + .padding(20) + .frame(minWidth: 340, minHeight: 260) + .background(StrandPalette.surfaceOverlay) } } diff --git a/StrandTests/RawHistoryArchiveEvictionTests.swift b/StrandTests/RawHistoryArchiveEvictionTests.swift index 80267d8405..e922b81b57 100644 --- a/StrandTests/RawHistoryArchiveEvictionTests.swift +++ b/StrandTests/RawHistoryArchiveEvictionTests.swift @@ -113,6 +113,123 @@ final class RawHistoryArchiveEvictionTests: XCTestCase { return "{\"capturedAtMs\":1,\"trim\":1,\"family\":\"\(family)\",\"frameHex\":\"\(hex)\"}\n" } + // MARK: - zero-payload frames lose retention priority + + /// A synthetic WHOOP 5/MG type-47 record: a non-zero 21-byte header (type @8 = 47, hist_version @9), + /// then `payloadBytes` of payload, then a 4-byte CRC trailer. + /// + /// `payloadByte: 0` reproduces the measured `enable_raw_data_w_ecg` placeholder — the strap banks one + /// such record per second and every byte from offset 21 to the CRC is zero. `marker` writes a single + /// distinguishing byte into the payload, which is also what makes a frame informative. + private func whoop5Frame(version: UInt8, payloadByte: UInt8, + payloadBytes: Int = 64, marker: UInt8? = nil) -> [UInt8] { + var f = [UInt8](repeating: 0x11, count: 21) // non-zero header (seq/ts/const bytes) + f[0] = 0xAA; f[8] = 47; f[9] = version + var payload = [UInt8](repeating: payloadByte, count: payloadBytes) + if let marker { payload[0] = marker } + f.append(contentsOf: payload) + f.append(contentsOf: [0xDE, 0xAD, 0xBE, 0xEF]) // CRC32 trailer (never part of the payload) + return f + } + + /// The strict test: entirely-zero payload region → yes; ONE non-zero byte anywhere → no. + func testZeroPayloadDetectionIsStrictAndHeaderAware() { + let empty = whoop5Frame(version: 22, payloadByte: 0) + XCTAssertTrue(RawHistoryArchive.hasZeroPayload(empty, family: .whoop5), + "the header and CRC bytes are non-zero but the PAYLOAD region is all zero") + // A single populated byte in an otherwise-empty buffer is exactly what this archive is for. + var oneByte = empty + oneByte[21 + 40] = 0x01 + XCTAssertFalse(RawHistoryArchive.hasZeroPayload(oneByte, family: .whoop5), + "'mostly zero' must NOT count as empty — one non-zero byte makes it informative") + // The last payload byte, immediately before the CRC trailer, is inside the region. + var lastByte = empty + lastByte[lastByte.count - 5] = 0x01 + XCTAssertFalse(RawHistoryArchive.hasZeroPayload(lastByte, family: .whoop5)) + // A payload region too short to judge is never called empty. + XCTAssertFalse(RawHistoryArchive.hasZeroPayload(whoop5Frame(version: 22, payloadByte: 0, + payloadBytes: 8), family: .whoop5)) + XCTAssertFalse(RawHistoryArchive.hasZeroPayload([0xAA, 0x01], family: .whoop5)) + } + + /// The DEFECT: with `enable_raw_data_w_ecg` armed the strap banks one all-zero record per second, so + /// a purely age-ordered archive evicts a real contact-window record within the half hour. An + /// informative frame must outrank an all-zero one of the SAME layout version, even when it is the + /// oldest line in the file and the empties are the newest. + func testZeroPayloadFramesAreEvictedBeforeInformativeOnes() { + let dir = tmpDir("zero"); defer { try? FileManager.default.removeItem(at: dir) } + let archive = RawHistoryArchive(directory: dir, maxBytes: 4_096, + perVersionFloor: 2, zeroPayloadFloor: 2) + + // Three informative records land FIRST — the oldest lines in the archive. Two would survive on + // the per-version floor alone; the third can only survive if the empties are evicted before it. + let informative = (0..<3).map { whoop5Frame(version: 22, payloadByte: 0x00, marker: UInt8(0xE0 + $0)) } + for f in informative { _ = archive.archive([f], trim: 1, family: .whoop5) } + XCTAssertFalse(RawHistoryArchive.hasZeroPayload(informative[0], family: .whoop5), + "precondition: a marker byte makes these frames informative") + + // Then the placeholder flood: same layout version, all-zero payload, newest in the file. + for _ in 0..<200 { _ = archive.archive([whoop5Frame(version: 22, payloadByte: 0)], + trim: 2, family: .whoop5) } + + let back = archive.readAll().map(\.frame) + let size = (try? FileManager.default.attributesOfItem(atPath: archive.fileURL.path)) + .flatMap { $0[.size] as? Int } ?? 0 + XCTAssertLessThanOrEqual(size, 4_096, "the archive must still respect its cap") + for (i, f) in informative.enumerated() { + XCTAssertTrue(back.contains(f), "informative frame #\(i) must outrank 200 empty records") + } + // …and the empties really were the thing evicted (a handful kept to date the artefact). + let zeros = back.filter { RawHistoryArchive.hasZeroPayload($0, family: .whoop5) } + XCTAssertLessThan(zeros.count, 200, "the placeholder flood must have been evicted, not the data") + } + + /// Same ordering in the pure core, with the informative line the OLDEST of its version so no floor + /// can be what saves it. + func testEvictLinesEvictsZeroPayloadBeforeInformative() { + let informative = archiveLine(whoop5Frame(version: 22, payloadByte: 0, marker: 0xE7)) + var lines = [informative] + for _ in 0..<200 { lines.append(archiveLine(whoop5Frame(version: 22, payloadByte: 0))) } + let kept = RawHistoryArchive.evictLines(lines, maxBytes: 4_096, floor: 2, zeroFloor: 2) + XCTAssertLessThanOrEqual(kept.reduce(0) { $0 + $1.utf8.count }, 4_096) + XCTAssertTrue(kept.contains(informative), + "the one informative line must survive a flood of all-zero-payload lines") + XCTAssertLessThan(kept.count, 200, "the zero-payload surplus must actually have been evicted") + } + + /// A version whose records are ALL empty still keeps `zeroFloor` dated samples — enough to prove the + /// artefact exists without letting it own the archive. + func testAllZeroVersionKeepsItsSmallFloor() { + var lines: [String] = [] + for _ in 0..<200 { lines.append(archiveLine(whoop5Frame(version: 22, payloadByte: 0))) } + for i in 0..<200 { lines.append(archiveLine(whoop5Frame(version: 18, payloadByte: 0, + marker: UInt8(i & 0xFF)))) } + let kept = RawHistoryArchive.evictLines(lines, maxBytes: 4_096, floor: 2, zeroFloor: 2) + let keptZeroVersion = kept.filter { $0.contains("2f16") } // 0x2f = type 47, 0x16 = v22 + XCTAssertGreaterThanOrEqual(keptZeroVersion.count, 2, + "an all-empty version must keep its small floor, not vanish") + } + + /// One archived JSONL line for `frame`, in the exact shape `archive` writes. + private func archiveLine(_ frame: [UInt8], family: DeviceFamily = .whoop5) -> String { + let hex = frame.map { String(format: "%02x", $0) }.joined() + return "{\"capturedAtMs\":1,\"trim\":1,\"family\":\"\(family.rawValue)\",\"frameHex\":\"\(hex)\"}\n" + } + + // MARK: - operator summary + + /// The summary is the after-an-experiment check: it must separate informative frames from + /// placeholders rather than reporting one healthy-looking line count. + func testSummarySplitsInformativeFromZeroPayload() { + var lines = [archiveLine(whoop5Frame(version: 22, payloadByte: 0, marker: 0xE7))] + for _ in 0..<5 { lines.append(archiveLine(whoop5Frame(version: 22, payloadByte: 0))) } + let text = RawHistoryArchive.summaryText(lines: lines, sizeBytes: 4_096, newest: 3) + XCTAssertTrue(text.contains("whoop5 v22"), text) + XCTAssertTrue(text.contains("1 informative, 5 all-zero payload"), text) + XCTAssertTrue(text.contains("Newest 3 capture(s)"), text) + XCTAssertTrue(RawHistoryArchive.summaryText(lines: [], sizeBytes: 0).contains("EMPTY")) + } + /// `evictLines` is the pure core: a flood of common-version lines plus two rare-version lines (the /// oldest), capped tight → the rare lines must survive and the result must fit the cap. func testEvictLinesKeepsRareVersionUnderCap() { From 3ed397805f8f33f60bac062d2ab0ed6d70b538a7 Mon Sep 17 00:00:00 2001 From: vishk23 Date: Wed, 5 Aug 2026 18:46:40 -0400 Subject: [PATCH 161/171] Retain WHOOP 5/MG SpO2 forever instead of losing it after 7 days MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 5/MG emits its own computed SpO2 percentage on @82 of the v18 record. It has been captured since v31 — but only inside v18AuxSample.fields, a table capped at 604,800 rows/device (~7 days at 1 Hz) because the aux blob costs ~30 MB/day. So every reading rolled off after a week and was gone for good: the strap trims its own history the moment an offload is acked, so there was no second copy anywhere. Raising the aux cap is the wrong fix — it buys a year of SpO2 at the price of ~11 GB of unrelated aux bytes. The in-band signal is ~2,000 samples/week (~100k rows/year), so it gets a narrow sibling table that is never pruned while the wide aux capture stays capped and unchanged. v34-spo2-pct-durable: spo2PctSample(deviceId, ts, pct), PK (deviceId, ts). Numbered v34 rather than v32 because upstream already holds v32-rr-src-channel and v33-workout-steps, which this branch has not merged. A colliding identifier is not a merge conflict anyone sees — GRDB keys migrations by name, so it would silently skip the second body and one table would simply never be created. Declared in schema_oracle.json under a new fork-ahead-of-upstream lineage with that reasoning written down. Storage is lossless: the raw in-band byte, verbatim. Run grouping and ramp trimming are read-time policies — they have already changed once on the cloud reader, and baking one in would destroy the evidence needed to change it again. The one thing applied on the way in is the 70...100 demultiplex, which is not a policy: @82 also carries bit-7 status sentinels, sub-70 diagnostic codes and 0-for-not-emitted on the same byte. The write reads s.auxByte82 off the decoded struct, never fields[23] of the packed blob. Byte 23 is correct only because every slot ahead of @82 happens to be present in production; V18AuxCodec omits absent slots entirely, so one missing earlier slot slides the read onto opticalAmpB — which still lands in 70...100 and so fails silently as a plausible percentage. Both halves pinned in V18AuxCodecOffsetTests. Nightly aggregate now fills dailyMetric.spo2Pct, which was unconditionally nil for WHOOP while the same field rendered fine for imported Oura rows. The statistic is a per-run ramp-trimmed median: the strap samples in runs of ~30 whose first ~5 are the optical front end settling and read low, biasing a naive mean down (1.5 points on the pinned case). Median over trimmed survivors matches what the cloud reader already reports for the same rows. Note this field also feeds HealthKit's oxygenSaturation, so it stays nil whenever the night has no in-band reading rather than carrying a guess. Retention/registry wired consistently: excluded from the aux sweep by construction, added to deviceScopedTables (a never-pruned table makes delete-means-gone more important), storage stats, and the timestamp heal. Reaches the cloud with no extra wiring — .noopbak archives the whole .sqlite and liters replicates at page level; neither enumerates tables. No Room twin yet; recorded as ios_only in the oracle. Tests: 26 new (14 store/codec, 12 analytics). Package suites green (WhoopStore 409, StrandAnalytics 1324, WhoopProtocol, StrandImport, NoopLocalAccess). Both app targets build; StrandTests 1049 pass. --- .../NoopLocalAccessCore/LocalAccessCore.swift | 5 + .../StrandAnalytics/AnalyticsEngine.swift | 112 +++++++- .../Spo2PctNightlyTests.swift | 174 ++++++++++++ .../Sources/WhoopProtocol/Streams.swift | 39 +++ .../Sources/WhoopStore/Database.swift | 49 ++++ .../WhoopStore/DeviceRegistryStore.swift | 5 + .../Sources/WhoopStore/StreamStore.swift | 68 +++++ .../Sources/WhoopStore/TimestampHeal.swift | 7 +- .../Resources/schema_oracle.json | 38 ++- .../WhoopStoreTests/Spo2PctDurableTests.swift | 256 ++++++++++++++++++ Strand/Data/IntelligenceEngine.swift | 7 + .../app/src/test/resources/schema_oracle.json | 38 ++- 12 files changed, 794 insertions(+), 4 deletions(-) create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/Spo2PctNightlyTests.swift create mode 100644 Packages/WhoopStore/Tests/WhoopStoreTests/Spo2PctDurableTests.swift diff --git a/Packages/NoopLocalAccess/Sources/NoopLocalAccessCore/LocalAccessCore.swift b/Packages/NoopLocalAccess/Sources/NoopLocalAccessCore/LocalAccessCore.swift index 632e181a28..ad4228a505 100644 --- a/Packages/NoopLocalAccess/Sources/NoopLocalAccessCore/LocalAccessCore.swift +++ b/Packages/NoopLocalAccess/Sources/NoopLocalAccessCore/LocalAccessCore.swift @@ -333,6 +333,11 @@ public final class ReadonlyNoopStore { "hrSample", "rrInterval", "event", "battery", "spo2Sample", "skinTempSample", "respSample", "gravitySample", "ppgHrSample", "stepSample", "sleepStateSample", "ppgWaveformSample", "rawImuSample", "v18AuxSample", + // v34: durable `@82` SpO2 percentages. Small (~100k rows/year) but UNBOUNDED — it is the + // one table here deliberately exempt from every prune — so it belongs in the readout for + // exactly the reason the comment above gives: growth that nothing prunes still has to be + // growth somebody can see. + "spo2PctSample", ] var decodedRows = 0 for table in decodedTables where tableNames.contains(table) { diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift index 696db1a233..7cdd68f005 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift @@ -279,6 +279,14 @@ public enum AnalyticsEngine { // calibrated blood-oxygen % (that needs WHOOP's proprietary curve). // Default empty keeps pure-function callers/tests + non-4.0 nights nil. spo2: [SpO2Sample] = [], + // Durable 5/MG `@82` SpO2 percentages for the night window (v34). The + // ramp-trimmed nightly MEDIAN is banked on `DailyMetric.spo2Pct` — the + // strap's OWN computed percentage, not a curve NOOP invented. Distinct + // from `spo2:` above, which is the WHOOP 4.0 v24 raw red/IR ADC pair and + // feeds `spo2Red`/`spo2Ir`; the two device generations report different + // physical quantities and neither substitutes for the other. Default + // empty keeps pure-function callers/tests + every non-5/MG night nil. + spo2PctSamples: [Spo2PctSample] = [], profile: UserProfile, baselines: ProfileBaselines = ProfileBaselines(), maxHROverride: Double? = nil, @@ -620,6 +628,14 @@ public enum AnalyticsEngine { // as-is for the Health "Raw SpO₂" tile — NOT a calibrated blood-oxygen %. (#93) let nightlySpo2Raw = nightlySpo2RawMeans(matched, spo2: spo2) + // ── SpO2 percentage (5/MG @82, durable) ─────────────────────────────── + // Ramp-trimmed nightly median of the strap's OWN computed SpO2 %, over the detected in-bed spans. + // Nil on a WHOOP 4.0, on any night with no in-band reading, and on every window offloaded before + // v34. Unlike the raw red/IR means above this IS a percentage — it is the number the strap + // computed, banked verbatim per-second and aggregated here, not a calibration NOOP applied to an + // ADC. See `nightlySpo2Pct` for the run/ramp policy and why the statistic is a median. + let nightlySpo2PctVal = nightlySpo2Pct(matched, samples: spo2PctSamples) + // ── Recovery / "Charge" ─────────────────────────────────────────────── var recovery: Double? = nil // Ordered "why is Charge what it is" rows, built from the SAME inputs as the score @@ -733,7 +749,19 @@ public enum AnalyticsEngine { recovery: recovery, strain: strain, exerciseCount: workouts.count, - spo2Pct: nil, + // v34: was unconditionally nil here — the on-device engine had no percentage to bank, so the + // Blood Oxygen card read "No Data" on every WHOOP night while the same field rendered fine for + // imported Oura rows. It now carries the 5/MG's own `@82` median when the night has one, and + // stays nil otherwise (WHOOP 4.0, pre-v34 windows, no in-band samples) — an absent reading + // must stay absent rather than become a fabricated number. + // + // NOTE FOR ANYONE CHANGING THIS: `dailyMetric.spo2Pct` is not display-only. `HealthKitBridge` + // writes it to `HKQuantityTypeIdentifier.oxygenSaturation`, so a value banked here leaves the + // app and lands in the user's Apple Health blood-oxygen history alongside clinical readings. + // That is the right home for a device-computed SpO2 % and it is exactly where the Oura import + // already puts its own, but it does mean this field's bar is "the device measured this", + // never "this is our best guess". + spo2Pct: nightlySpo2PctVal?.pct, skinTempDevC: skinTempDevC, respRateBpm: respRateDaily, steps: stepsTotal, @@ -1028,6 +1056,88 @@ public enum AnalyticsEngine { return (mean: sum / kept, samples: kept) } + /// Nightly SpO2 percentage from the DURABLE `@82` stream (`spo2PctSample`, v34) over the detected + /// in-bed `sessions` — the value banked on `DailyMetric.spo2Pct` for a 5/MG night — with the sample + /// count that survived the ramp trim. Nil when no reading fell inside any span. + /// + /// HOW THIS DIFFERS FROM `nightlySpo2CandidateMean`. That one is a naive mean over the raw candidate + /// byte read out of the capped aux table, built as a hand-checkable diagnostic. This is the scored + /// statistic, over the durable stream, and it corrects the two things that make the naive mean wrong. + /// + /// THE ACQUISITION RAMP. The strap does not sample SpO2 continuously: it takes a RUN of ~30 one-second + /// samples roughly once per ~1,200 s while asleep. The first samples of each run are the optical front + /// end settling, and they read LOW — so a mean over every in-band sample is biased downward by a + /// contamination that is entirely one-sided, and the bias grows with how much of the night is ramp + /// (i.e. it is worse on nights with more, shorter runs — precisely when you would least suspect it). + /// Runs are separated here by a gap of more than `spo2RunGapSeconds`; at 1 Hz within a run and ~1,200 s + /// between runs, that boundary is not a close call. The leading `spo2RampSamples` of each run are + /// dropped — but never more than HALF a run, so a short run contributes its settled tail instead of + /// vanishing and silently reweighting the night toward whichever runs happened to be long. + /// + /// WHY THE MEDIAN AND NOT A TRIMMED MEAN. Three reasons, in order of weight: + /// 1. It is the statistic the cloud reader already reports (the per-night 94-97 medians every + /// cross-check against this data has been quoted against). A phone that shipped a trimmed mean + /// would disagree with the server on the same rows for no reason anyone could see, and every + /// future comparison would have to carry a caveat. + /// 2. Ramp removal is a heuristic, not a proof. A partially-settled sample just past the cut, or a + /// motion artifact mid-run, is residual one-sided contamination that a mean absorbs in full and a + /// median very largely ignores. + /// 3. The in-band values live in a 31-wide integer band with a hard mode at 97, so a median is + /// stable and interpretable here in a way it would not be on a sparse continuous quantity. + /// The trim and the median are complementary, not redundant: the trim removes a KNOWN, structured, + /// large block of low readings that would move a median too, and the median absorbs what is left. + /// + /// READ-TIME POLICY, DELIBERATELY. None of this is applied on the way into the store — `spo2PctSample` + /// holds raw in-band bytes. These rules have already changed once on the cloud side, and they will + /// change again if a run's shape turns out to differ by firmware; keeping them here means that costs a + /// re-score, not a re-capture of data the strap has long since trimmed. + public static func nightlySpo2Pct(_ sessions: [SleepSession], + samples: [Spo2PctSample]) -> (pct: Double, samples: Int)? { + guard !sessions.isEmpty, !samples.isEmpty else { return nil } + // In-bed first, so run boundaries are computed over the samples that will actually be used. Note + // what that does and does not mean: a session edge splits a run only when the stretch it excludes + // is itself longer than `spo2RunGapSeconds` — a brief out-of-bed moment mid-run leaves the run + // intact and singly-trimmed. Gap size is the ONLY run signal; session boundaries are not a second + // one. That is deliberate, because the ramp is a property of the sensor restarting, and the sensor + // does not restart because the sleep detector drew a line. + let inBed = samples + .filter { s in sessions.contains(where: { $0.start <= s.ts && s.ts <= $0.end }) } + .sorted { $0.ts < $1.ts } + guard !inBed.isEmpty else { return nil } + + var kept: [Int] = [] + var runStart = 0 + func flushRun(_ range: Range) { + let run = inBed[range] + // Never trim more than half a run: a 4-sample run keeps its last 2 rather than disappearing. + let drop = min(spo2RampSamples, run.count / 2) + kept.append(contentsOf: run.dropFirst(drop).map(\.pct)) + } + for i in 1.. spo2RunGapSeconds { + flushRun(runStart.. SleepSession { + SleepSession(start: start, end: start + durSec, efficiency: 0.9, + stages: [], restingHR: 50, avgHRV: 60) + } + + /// A measurement run: `count` consecutive one-second samples starting at `ts`, the first + /// `rampCount` of them at the depressed `rampValue`. + private func run(_ ts: Int, count: Int, settled: Int, + rampCount: Int = 0, rampValue: Int = 88) -> [Spo2PctSample] { + (0.. = 70...100 + +/// One in-band `@82` SpO2 percentage, durably banked (v34). +/// +/// WHY THIS EXISTS AS ITS OWN STREAM. The byte is carried raw in `V18AuxSample.auxByte82`, but +/// `v18AuxSample` is a rolling capped table (`WhoopStore.v18AuxRetentionRows` = 604,800 rows/device ≈ 7 +/// days at 1 Hz) because the aux blob costs ~30 MB/day. The SpO2 signal inside it is ~2,000 samples per +/// WEEK — roughly 100k rows/year — so pulling it into its own narrow table makes it retainable forever at +/// trivial cost, while the wide aux capture stays capped. Without this, every SpO2 reading older than +/// seven days is destroyed, and the strap has already trimmed its own copy. +/// +/// LOSSLESS BY CONSTRUCTION. `pct` is the raw in-band byte, verbatim. No run grouping, no ramp trimming, +/// no smoothing, no dedupe beyond the (deviceId, ts) primary key. Those are all READ-TIME policies — the +/// cloud reader has already changed its run/ramp rules once, and a device store that had baked the old +/// policy in would have destroyed the evidence needed to change them. The only thing applied on the way +/// in is the `spo2CandidateInBand` demultiplex, which is not a policy but the definition of which values +/// on this byte are percentages at all. +public struct Spo2PctSample: Equatable, Codable, Sendable { + public let ts: Int // wall-clock unix seconds (one per strap-second that carried a reading) + public let pct: Int // the raw in-band `@82` byte, 70...100 + public init(ts: Int, pct: Int) { self.ts = ts; self.pct = pct } +} + public struct Streams: Equatable, Codable { public var hr: [HRSample] public var rr: [RRInterval] diff --git a/Packages/WhoopStore/Sources/WhoopStore/Database.swift b/Packages/WhoopStore/Sources/WhoopStore/Database.swift index 29a3653da3..e48dbf2c5e 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/Database.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/Database.swift @@ -789,6 +789,55 @@ extension WhoopStore { t.primaryKey(["deviceId", "ts"]) } } + + // v34: DURABLE per-sample SpO2 percentages from the 5/MG `@82` candidate byte. + // + // THE PROBLEM THIS FIXES. The strap emits its own computed SpO2 % on `@82` of the v18 record. That + // byte is already banked, raw, in `v18AuxSample.fields` (v31) — but `v18AuxSample` is CAPPED at + // `WhoopStore.v18AuxRetentionRows` (604,800 rows/device ≈ 7 days at 1 Hz), because the aux blob + // costs ~30 MB/day and an uncapped 30 MB/day table is not a thing a phone can carry. So every SpO2 + // reading rolls off after a week and is gone permanently: the strap trims its own history the + // moment an offload is acked, so there is no second copy anywhere. + // + // Raising the aux cap is the WRONG fix — it would buy a year of SpO2 at the price of ~11 GB of + // mostly-unrelated aux bytes. The right fix is a narrow sibling: the in-band SpO2 signal is only + // ~2,000 samples/week (a run of ~30 one-second samples once per ~1,200 s, sleep-gated), i.e. + // ~100k rows/year, which is trivially retainable forever. This table is that sibling. The aux + // capture stays capped and unchanged; this one is never pruned. + // + // WHY NOT EXTEND `spo2Sample` (v3). Wrong shape, and extending it would corrupt a live reader: + // 1. `spo2Sample` is `(deviceId, ts, red, ir)` — two NOT NULL raw ADC channels off the WHOOP 4.0 + // v24 layout. We have a computed PERCENTAGE and no red/IR at all, so every insert would have + // to fabricate `red`/`ir` values that were never measured. Absence-becomes-a-fabricated-0 is + // exactly the failure mode the rest of this schema is built to avoid. + // 2. `AnalyticsEngine.nightlySpo2RawMeans` averages `red`/`ir` over that table into + // `dailyMetric.spo2Red`/`spo2Ir`. Inserting rows carrying fake zeros would silently drag + // those means toward zero — a live, user-visible tile broken by a storage decision. + // 3. The two are different physical quantities from different device generations. A percentage + // and a pair of ADC counts sharing a table would need a discriminator column and every reader + // would have to remember to apply it. A separate table makes the distinction unforgeable. + // + // `pct` is the RAW in-band byte (70...100), stored verbatim. Run grouping and acquisition-ramp + // trimming are deliberately NOT applied here — they are read-time policies, they have already + // changed once on the cloud reader, and a store that baked in the old policy would have destroyed + // the evidence needed to change them again. The one thing applied on the way in is the + // `spo2CandidateInBand` demultiplex (see `Spo2PctSample`), which is not a policy: `@82` carries + // measurements, bit-7 status sentinels, sub-70 diagnostic codes and 0 on the same byte, and only + // the `70...100` band is a percentage of anything. Storing a sentinel here would be storing a + // number that is not the quantity the column is named for. + // + // NOT PRUNED, by design — the same "durable decoded biometric history" class as `hrSample` and + // `ppgWaveformSample`. The `v18AuxSample` sweep in `StreamStore.insert` deletes only + // `FROM v18AuxSample`, so this table is untouched by it; `Spo2PctDurableTests` pins that. + // PK (deviceId, ts) mirrors every other per-sample table and makes re-ingest idempotent. + migrator.registerMigration("v34-spo2-pct-durable") { db in + try db.create(table: "spo2PctSample") { t in + t.column("deviceId", .text).notNull() + t.column("ts", .integer).notNull() // strap-second, unix seconds + t.column("pct", .integer).notNull() // raw in-band @82 byte, 70...100 + t.primaryKey(["deviceId", "ts"]) + } + } return migrator } } diff --git a/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift b/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift index add4660323..ab021aceef 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/DeviceRegistryStore.swift @@ -114,6 +114,11 @@ public struct DeviceRegistryStore: Sendable { // v31-deep-capture-channels: the banked 5/MG v18 auxiliary fields are deviceId-keyed per-second // rows like every stream above, so a "delete all of this device's data" must clear them too. "v18AuxSample", + // v34-spo2-pct-durable: the durable `@82` SpO2 percentages are deviceId-keyed per-second rows, + // and they OUTLIVE the aux table they are forked from (never pruned) — which makes covering them + // here more important, not less. A wipe that cleared `v18AuxSample` but left this behind would + // leave years of blood-oxygen readings on the device after the user asked for them to be gone. + "spo2PctSample", ] /// deviceId-keyed tables deliberately EXCLUDED from `deviceScopedTables` — a normal "delete this diff --git a/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift b/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift index fa11ce0f5f..5d550f0fa0 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/StreamStore.swift @@ -350,6 +350,46 @@ extension WhoopStore { v18Written += 1 } } + // DURABLE SpO2 (v34) — the in-band `@82` percentages, forked out of the SAME v18 aux samples + // banked just above into a narrow table that is never pruned. The aux blob is capped at + // ~7 days (see the sweep below); this is the copy that survives, and without it every reading + // older than a week is destroyed with no second copy anywhere (the strap trims its own history + // as soon as an offload is acked). + // + // READ OFF THE DECODED STRUCT, NOT THE BLOB. `s.auxByte82` is the slot value; the alternative + // — indexing byte 23 of the packed `fields` blob, which is what the cloud reader does — is + // correct ONLY while every slot ahead of `auxByte82` in `V18AuxSlot` order is present, because + // `V18AuxCodec` is a presence-bitmap format that omits absent slots from the body entirely. + // With all slots present the arithmetic happens to land on 23 (5-byte header + 4+1+1+1+1+2+1+1 + // +2+2+2 = 18 body bytes ahead of it), which is why reading `fields[23]` yields physiologically + // valid values in production — but a single absent earlier slot silently shifts it onto a + // neighbouring byte and the read becomes garbage that still looks like a plausible percentage. + // `V18AuxCodecOffsetTests` pins both halves of that. Here the decoded struct is already in + // hand, so the fragile path is not merely avoided, it is unreachable. + // + // Separate loop rather than a branch inside the one above, so the `blob.isEmpty` skip there + // can never silently drop an SpO2 reading. (In practice a present `auxByte82` guarantees a + // non-empty blob, but coupling the two would make that a load-bearing invariant nobody stated.) + // + // Gate: `spo2CandidateInBand` only. `@82` is multiplexed — bit-7 sentinels (0x80/0x88/0x90/ + // 0xA0/0xA8), sub-70 diagnostic codes, and 0-for-not-emitted share the byte with real + // percentages, and `101...127` is an empty band that keeps the two separable. Nothing else is + // applied: no run grouping, no acquisition-ramp trim, no smoothing. Those are read-time + // policies (they have already changed once on the cloud reader) and baking one in here would + // destroy the evidence needed to change it again. + // + // ON CONFLICT DO NOTHING keeps the FIRST value banked for a second, matching every other + // per-second stream's dedupe rule and making a re-offload of the same window idempotent. + if !streams.v18Aux.isEmpty { + let stmt = try db.cachedStatement(sql: """ + INSERT INTO spo2PctSample (deviceId, ts, pct) VALUES (?, ?, ?) + ON CONFLICT(deviceId, ts) DO NOTHING + """) + for s in streams.v18Aux { + guard let v = s.auxByte82, spo2CandidateInBand.contains(v) else { continue } + try stmt.execute(arguments: [deviceId, s.ts, v]) + } + } return (hr, rr, ev, bat, spo2, skin, resp, grav) } @@ -619,6 +659,34 @@ extension WhoopStore { try syncRead { db in try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM v18AuxSample") ?? 0 } } + /// Durable in-band `@82` SpO2 percentages (v34) in `[from, to]` for one device, ascending by ts. + /// + /// Unlike `v18AuxSamples`, this reaches back over the WHOLE recorded history: `spo2PctSample` is never + /// pruned, while the aux table it is forked from is capped at ~7 days. Empty for a WHOOP 4.0 (no v18 + /// aux stream) and for any window offloaded before v34 — the v31..v33 aux rows still on the device do + /// carry the byte, but only for whatever slice of the last seven days survives, and this reader + /// deliberately does NOT fall back to them: a series that silently changed source partway through + /// would be unusable as evidence. Backfilling those survivors is a one-shot migration's job, not a + /// read path's. + /// + /// Values are RAW in-band bytes. Run grouping and acquisition-ramp trimming are the CALLER's policy — + /// see `AnalyticsEngine.nightlySpo2Pct` for the nightly one. + public func spo2PctSamples(deviceId: String, from: Int, to: Int, limit: Int = 200_000) async throws + -> [Spo2PctSample] { + try syncRead { db in + try Row.fetchAll(db, sql: """ + SELECT ts, pct FROM spo2PctSample + WHERE deviceId = ? AND ts >= ? AND ts <= ? + ORDER BY ts LIMIT ? + """, arguments: [deviceId, from, to, limit]) + .map { Spo2PctSample(ts: $0["ts"], pct: $0["pct"]) } + } + } + + public func spo2PctCountForTest() async throws -> Int { + try syncRead { db in try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM spo2PctSample") ?? 0 } + } + public func ppgHrCountForTest() async throws -> Int { try syncRead { db in try Int.fetchOne(db, sql: "SELECT COUNT(*) FROM ppgHrSample") ?? 0 } } diff --git a/Packages/WhoopStore/Sources/WhoopStore/TimestampHeal.swift b/Packages/WhoopStore/Sources/WhoopStore/TimestampHeal.swift index 8093834237..97781700a1 100644 --- a/Packages/WhoopStore/Sources/WhoopStore/TimestampHeal.swift +++ b/Packages/WhoopStore/Sources/WhoopStore/TimestampHeal.swift @@ -55,7 +55,12 @@ extension WhoopStore { let rawTables = ["hrSample", "rrInterval", "event", "battery", "spo2Sample", "skinTempSample", "respSample", "gravitySample", "stepSample", "ppgHrSample", - "sleepStateSample", "ppgWaveformSample", "rawImuSample", "v18AuxSample"] + "sleepStateSample", "ppgWaveformSample", "rawImuSample", "v18AuxSample", + // v34: the durable SpO2 percentages are keyed by the SAME type-47 `ts` as + // every stream above and forked from the same records, so a bad-clock strap + // can garbage them identically. They are never pruned, so an unhealed + // out-of-bounds row here would persist forever rather than rolling off. + "spo2PctSample"] var rawDeleted = 0 for table in rawTables { try db.execute(sql: "DELETE FROM \(table) WHERE ts < ? OR ts > ?", diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json b/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json index fdf93b2da7..ed67ec4280 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json @@ -36,7 +36,8 @@ "v28-raw-imu", "v29-score-input-provenance", "v30-rr-ord", - "v31-deep-capture-channels" + "v31-deep-capture-channels", + "v34-spo2-pct-durable" ], "grdbMigrationLineages": { "fork-ios-only": { @@ -47,6 +48,12 @@ "v28-phone-timezone", "v29-daily-avg-sdnn" ] + }, + "fork-ahead-of-upstream": { + "reason": "This fork numbers a migration AHEAD of upstream on purpose, which is the mirror image of the `fork-ios-only` case and needs its own entry rather than being folded into it. Local main carries migrations through v31; upstream ryanbr/noop already holds `v32-rr-src-channel` and `v33-workout-steps`, which this branch has not merged yet. Numbering this one v32 would therefore collide with an upstream identifier the very next time the fork merges upstream \u2014 and because GRDB keys migrations by identifier STRING, that collision is not a merge conflict a human sees: two different bodies claiming the same name means GRDB silently SKIPS the second, so one of the two tables is never created and the failure surfaces later as a missing-table error on a real device. Starting at v34 leaves upstream's v32/v33 free to arrive untouched. It is declared here rather than renumbered because the baseline lineage must stay exactly v1..vN, and because once this identifier has run on any device renaming it re-runs its body against a schema that already has the table.", + "migrations": [ + "v34-spo2-pct-durable" + ] } }, "divergenceReasons": { @@ -1691,6 +1698,35 @@ ], "indices": [] }, + "spo2PctSample": { + "platform": "ios_only", + "reason": "Fork-only iOS table (GRDB `v34-spo2-pct-durable`): the durable per-second SpO2 percentages the WHOOP 5/MG emits on `@82` of its v18 record. The byte is already captured raw inside `v18AuxSample.fields`, but that table is CAPPED at 604,800 rows/device (~7 days at 1 Hz) because the aux blob costs ~30 MB/day, so every reading rolled off after a week and was gone permanently \u2014 the strap trims its own history the moment an offload is acked. The in-band signal is only ~2,000 samples/week (~100k rows/year), so a narrow never-pruned sibling retains it forever at trivial cost while the wide aux capture stays capped. `pct` is the RAW in-band byte (70...100); run grouping and acquisition-ramp trimming are read-time policy, not stored. No Room twin yet: the Kotlin capture/aggregate side is outstanding, and until it lands Android keeps banking the byte in its own `v18AuxSample` twin and loses it on the same 7-day roll.", + "columns": [ + { + "name": "deviceId", + "affinity": "TEXT", + "notNull": true, + "default": null + }, + { + "name": "ts", + "affinity": "INTEGER", + "notNull": true, + "default": null + }, + { + "name": "pct", + "affinity": "INTEGER", + "notNull": true, + "default": null + } + ], + "primaryKey": [ + "deviceId", + "ts" + ], + "indices": [] + }, "spo2Sample": { "platform": "both", "columns": [ diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/Spo2PctDurableTests.swift b/Packages/WhoopStore/Tests/WhoopStoreTests/Spo2PctDurableTests.swift new file mode 100644 index 0000000000..52c75de8e9 --- /dev/null +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/Spo2PctDurableTests.swift @@ -0,0 +1,256 @@ +import XCTest +import GRDB +import WhoopProtocol +@testable import WhoopStore + +/// v34: the durable `@82` SpO2 percentage stream. +/// +/// The 5/MG emits its own computed SpO2 % on `@82` of the v18 record. That byte has been banked since v31 +/// — inside `v18AuxSample.fields`, a table CAPPED at 604,800 rows/device (~7 days at 1 Hz) because the aux +/// blob costs ~30 MB/day. So every reading rolled off after a week and was gone permanently: the strap +/// trims its own history the moment an offload is acked, leaving no second copy anywhere. +/// +/// `spo2PctSample` is the narrow, never-pruned sibling that fixes that. ~2,000 samples/week — ~100k +/// rows/year — is nothing to keep forever. These tests pin the two properties the fix rests on: the right +/// values get in (and only those), and the aux sweep cannot touch them. +final class Spo2PctDurableTests: XCTestCase { + + private func store() async throws -> WhoopStore { + let s = try await WhoopStore.inMemory() + try await s.upsertDevice(id: "dev1", mac: nil, name: nil) + return s + } + + /// One v18 aux sample carrying `@82` = `v` at `ts`, and nothing else. + private func aux(_ ts: Int, _ v: Int?) -> V18AuxSample { V18AuxSample(ts: ts, auxByte82: v) } + + // MARK: - Migration shape + + func testV34CreatesTheDurableTable() async throws { + let s = try await store() + let tables = try await s.tableNames() + XCTAssertTrue(tables.contains("spo2PctSample")) + let pk = try await s.primaryKeyColumns("spo2PctSample") + let cols = try await s.columnNamesForTest(table: "spo2PctSample") + XCTAssertEqual(pk, ["deviceId", "ts"]) + XCTAssertEqual(Set(cols), ["deviceId", "ts", "pct"]) + } + + /// The v3 `spo2Sample` table is NOT the home for this and must be left exactly as it was: its + /// `red`/`ir` are NOT NULL raw ADC channels off the WHOOP 4.0 v24 layout, and + /// `AnalyticsEngine.nightlySpo2RawMeans` averages them into a live tile. Storing percentages there + /// would have meant fabricating red/ir zeros on every insert and dragging that mean toward zero. + func testTheWhoop4RawTableIsUntouched() async throws { + let s = try await store() + let cols = try await s.columnNamesForTest(table: "spo2Sample") + // Its own columns, unchanged (`synced` is the outbox flag a later migration added). + XCTAssertEqual(Set(cols), ["deviceId", "ts", "red", "ir", "synced"]) + // And emphatically no percentage column: the two quantities never share a table. + XCTAssertFalse(cols.contains("pct")) + } + + // MARK: - In-band values persist + + func testInBandValuesPersistVerbatim() async throws { + let s = try await store() + _ = try await s.insert(Streams(v18Aux: [aux(100, 97), aux(101, 94), aux(102, 88)]), + deviceId: "dev1") + let rows = try await s.spo2PctSamples(deviceId: "dev1", from: 0, to: 1_000) + XCTAssertEqual(rows.map(\.ts), [100, 101, 102]) + XCTAssertEqual(rows.map(\.pct), [97, 94, 88]) + } + + /// The band is INCLUSIVE at both ends — 70 and 100 are real readings, and a gate that quietly dropped + /// the endpoints would bias every night's median upward at exactly the readings that matter most. + func testBandBoundariesAreInclusive() async throws { + let s = try await store() + _ = try await s.insert(Streams(v18Aux: [aux(200, 70), aux(201, 100)]), deviceId: "dev1") + let rows = try await s.spo2PctSamples(deviceId: "dev1", from: 0, to: 1_000) + XCTAssertEqual(rows.map(\.pct), [70, 100]) + } + + // MARK: - Out-of-band values are rejected + + /// `@82` is MULTIPLEXED: the same byte carries measurements, bit-7 status sentinels, sub-70 diagnostic + /// codes, and 0 for "not emitted". Only `70...100` is a percentage of anything. A sentinel banked as a + /// percentage would not merely be wrong, it would be *plausible* — 0x88 is 136, which no reader would + /// flag, and 0x08 is 8, which reads as catastrophic hypoxia. Rejection has to happen at the boundary. + func testStatusSentinelsAndDiagnosticCodesAreRejected() async throws { + let s = try await store() + let rejected = [0x00, 0x08, 0x45, 0x80, 0x88, 0x90, 0xA0, 0xA8, 0xFF] + let samples = rejected.enumerated().map { aux(300 + $0.offset, $0.element) } + _ = try await s.insert(Streams(v18Aux: samples), deviceId: "dev1") + let banked = try await s.spo2PctCountForTest() + let auxRows = try await s.v18AuxCountForTest() + XCTAssertEqual(banked, 0, "no sentinel or diagnostic code may be banked as a percentage") + // The raw bytes are still captured in the aux table — this gate demultiplexes, it does not discard. + XCTAssertEqual(auxRows, rejected.count) + } + + /// `101...127` is the EMPTY BAND — nothing has ever been observed there across 626,725 production + /// records. It is what makes measurements and bit-7 sentinels separable by value alone, with no side + /// channel. A value appearing there would mean the demultiplex assumption has broken, so it must not + /// be admitted on the strength of merely being "close to 100". + func testTheEmptyBandAbove100IsRejected() async throws { + let s = try await store() + let samples = (101...127).enumerated().map { aux(400 + $0.offset, $0.element) } + _ = try await s.insert(Streams(v18Aux: samples), deviceId: "dev1") + let banked = try await s.spo2PctCountForTest() + XCTAssertEqual(banked, 0) + } + + /// An aux record with no `@82` slot at all banks no percentage — absence stays absence. + func testAbsentSlotBanksNothing() async throws { + let s = try await store() + _ = try await s.insert(Streams(v18Aux: [V18AuxSample(ts: 500, statusWord: 7)]), deviceId: "dev1") + let banked = try await s.spo2PctCountForTest() + let auxRows = try await s.v18AuxCountForTest() + XCTAssertEqual(banked, 0) + XCTAssertEqual(auxRows, 1) + } + + // MARK: - Idempotence + + /// A re-offload of a window already banked must not duplicate or mutate it. ON CONFLICT DO NOTHING + /// keeps the FIRST value for a second, matching every other per-second stream's dedupe rule. + func testReInsertIsIdempotentAndKeepsTheFirstValue() async throws { + let s = try await store() + _ = try await s.insert(Streams(v18Aux: [aux(600, 96), aux(601, 95)]), deviceId: "dev1") + _ = try await s.insert(Streams(v18Aux: [aux(600, 96), aux(601, 95)]), deviceId: "dev1") + // Same seconds, different values — the first-seen reading survives. + _ = try await s.insert(Streams(v18Aux: [aux(600, 71), aux(601, 99)]), deviceId: "dev1") + let rows = try await s.spo2PctSamples(deviceId: "dev1", from: 0, to: 1_000) + XCTAssertEqual(rows.map(\.ts), [600, 601]) + XCTAssertEqual(rows.map(\.pct), [96, 95]) + } + + /// Two straps' readings for the same second are different rows — the PK is (deviceId, ts). + func testPerDeviceScoping() async throws { + let s = try await store() + try await s.upsertDevice(id: "dev2", mac: nil, name: nil) + _ = try await s.insert(Streams(v18Aux: [aux(700, 97)]), deviceId: "dev1") + _ = try await s.insert(Streams(v18Aux: [aux(700, 92)]), deviceId: "dev2") + let d1 = try await s.spo2PctSamples(deviceId: "dev1", from: 0, to: 1_000) + let d2 = try await s.spo2PctSamples(deviceId: "dev2", from: 0, to: 1_000) + XCTAssertEqual(d1.map(\.pct), [97]) + XCTAssertEqual(d2.map(\.pct), [92]) + } + + // MARK: - Retention: the whole point of the table + + /// THE LOAD-BEARING TEST. The aux sweep must not reach this table. Retention is forced to 1 row and + /// the prune to every row, so the aux capture is cut to its newest row — while every percentage + /// survives. This is the entire reason v34 exists: before it, seven-day-old SpO2 was destroyed with + /// no recoverable copy. + func testSpo2SurvivesTheAuxPrune() async throws { + let s = try await store() + let samples = (0..<40).map { aux(1_000 + $0, 90 + ($0 % 8)) } + _ = try await s.insert(Streams(v18Aux: samples), deviceId: "dev1", + v18AuxRetentionRows: 1, v18AuxPruneEveryRows: 1) + + // The aux table was swept down to its cap... + let auxRows = try await s.v18AuxCountForTest() + XCTAssertEqual(auxRows, 1) + // ...and not one percentage was lost. + let rows = try await s.spo2PctSamples(deviceId: "dev1", from: 0, to: 10_000) + XCTAssertEqual(rows.count, 40) + XCTAssertEqual(rows.map(\.ts), Array(1_000..<1_040)) + XCTAssertEqual(rows.map(\.pct), (0..<40).map { 90 + ($0 % 8) }) + } + + /// Repeated sweeps across many batches never erode the durable rows. + func testRepeatedPrunesNeverErodeTheDurableRows() async throws { + let s = try await store() + for batch in 0..<5 { + let samples = (0..<10).map { aux(2_000 + batch * 10 + $0, 95) } + _ = try await s.insert(Streams(v18Aux: samples), deviceId: "dev1", + v18AuxRetentionRows: 2, v18AuxPruneEveryRows: 1) + } + let banked = try await s.spo2PctCountForTest() + let auxRows = try await s.v18AuxCountForTest() + XCTAssertEqual(banked, 50) + XCTAssertLessThanOrEqual(auxRows, 12) + } + + // MARK: - Privacy + + /// A never-pruned table makes delete-means-gone MORE important, not less: without this the wipe would + /// leave years of blood-oxygen readings behind after the user asked for them to be erased. + func testDeleteAllDataClearsTheDurableRows() async throws { + let s = try await store() + _ = try await s.insert(Streams(v18Aux: [aux(3_000, 96)]), deviceId: "dev1") + let before = try await s.spo2PctCountForTest() + XCTAssertEqual(before, 1) + XCTAssertTrue(DeviceRegistryStore.deviceScopedTables.contains("spo2PctSample")) + try await s.deleteAllData(deviceId: "dev1") + let after = try await s.spo2PctCountForTest() + XCTAssertEqual(after, 0) + } +} + +/// The blob-offset question, pinned. +/// +/// The cloud reader takes `fields[23]` of the STORED `v18AuxSample.fields` blob and gets physiologically +/// valid SpO2 out of it. That works — but it is an ARITHMETIC COINCIDENCE of a fully-populated record, not +/// a property of the format. `V18AuxCodec` is a presence-bitmap codec: absent slots are omitted from the +/// body entirely, so every slot's byte offset depends on which slots ahead of it were present. +/// +/// These tests establish both halves, because only the pair is decisive. The first alone would license +/// the fragile read; the second is what says the device write path must not use it. +final class V18AuxCodecOffsetTests: XCTestCase { + + /// A record carrying every slot: `auxByte82` lands at blob byte 23. Header is 5 bytes (version + + /// u32 bitmap), then the eleven slots ahead of it occupy 4+1+1+1+1+2+1+1+2+2+2 = 18 bytes, so it sits + /// at 5 + 18 = 23 and the whole blob is 32 bytes. THIS is why the cloud's `fields[23]` reads real + /// percentages: in production every one of those slots is present. + func testFullyPopulatedRecordPutsAuxByte82AtBlobByte23() { + let full = V18AuxSample( + ts: 1, recordIndex: 0x11223344, rrCount: 4, cardiacFlags: 0x21, hrQualityFlags: 0x82, + heartRateAlt: 58, rrPacked: 0x0505, cardiacStatus: 0x31, stepCadence: 0x41, + statusWord: 0x0606, statusWord1: 0x0707, statusWord2: 0x0808, auxByte82: 93, + opticalBaselineA: 0x51, opticalBaselineB: 0x52, opticalAmpA: 0x53, opticalAmpB: 0x54, + unknownF32Bits: 0x55667788) + let blob = [UInt8](V18AuxCodec.pack(full)) + + XCTAssertEqual(blob.count, 32, "5-byte header + 27 body bytes") + XCTAssertEqual(Int(blob[23]), 93, "byte 23 of a fully-populated blob IS @82") + // And it agrees with the decoded struct, which is the only contract that actually holds. + XCTAssertEqual(V18AuxCodec.unpack(Data(blob), ts: 1).auxByte82, 93) + // Derive the offset from the slot table rather than trusting the literal, so a future slot + // inserted ahead of @82 fails HERE with a readable number instead of silently moving the cloud's + // read onto a neighbouring byte. + let ahead = V18AuxSlot.allCases + .filter { $0.rawValue < V18AuxSlot.auxByte82.rawValue } + .reduce(0) { $0 + $1.width } + XCTAssertEqual(V18AuxCodec.headerBytes + ahead, 23) + } + + /// ONE absent slot ahead of it and byte 23 is a different field entirely — while the decoded value is + /// still correct. The blob read does not degrade loudly; it returns a neighbouring byte that can look + /// exactly like a plausible saturation. This is why `StreamStore` reads `s.auxByte82` off the decoded + /// struct and never indexes the blob. + func testOneAbsentEarlierSlotMovesAuxByte82OffByte23() { + // Identical to the record above except `recordIndex` (4 bytes, slot 0) is absent. + let partial = V18AuxSample( + ts: 1, rrCount: 4, cardiacFlags: 0x21, hrQualityFlags: 0x82, + heartRateAlt: 58, rrPacked: 0x0505, cardiacStatus: 0x31, stepCadence: 0x41, + statusWord: 0x0606, statusWord1: 0x0707, statusWord2: 0x0808, auxByte82: 93, + opticalBaselineA: 0x51, opticalBaselineB: 0x52, opticalAmpA: 0x53, opticalAmpB: 0x54, + unknownF32Bits: 0x55667788) + let blob = [UInt8](V18AuxCodec.pack(partial)) + + XCTAssertEqual(blob.count, 28, "four fewer body bytes without recordIndex") + // @82 has slid four bytes earlier... + XCTAssertEqual(Int(blob[19]), 93) + // ...and byte 23 now holds opticalAmpB, four slots further along. Reading it as a percentage + // would yield 0x54 = 84: in band, unremarkable, and completely fabricated. Note it does NOT + // fail loudly — an optical amplitude byte and a saturation percentage occupy the same numeric + // range, so the bad read is indistinguishable from a good one at the call site. + XCTAssertEqual(Int(blob[23]), 0x54) + XCTAssertNotEqual(Int(blob[23]), 93) + XCTAssertTrue(spo2CandidateInBand.contains(Int(blob[23])), + "the wrong byte is still 'in band' — which is exactly why this read is unsafe") + // The decoded read is unaffected — it is driven by the bitmap, not by an offset. + XCTAssertEqual(V18AuxCodec.unpack(Data(blob), ts: 1).auxByte82, 93) + } +} diff --git a/Strand/Data/IntelligenceEngine.swift b/Strand/Data/IntelligenceEngine.swift index 91852655ef..0054a0552a 100644 --- a/Strand/Data/IntelligenceEngine.swift +++ b/Strand/Data/IntelligenceEngine.swift @@ -577,6 +577,12 @@ final class IntelligenceEngine: ObservableObject { // #93: WHOOP 4.0 raw SpO2 PPG samples for the night; analyzeDay banks the nightly red/IR ADC // means on the DailyMetric. Empty on a 5/MG (no v24 spo2 channels) → the raw means stay nil. let spo2 = (try? await store.spo2Samples(deviceId: owner, from: from, to: to, limit: 200_000)) ?? [] + // v34: the 5/MG's own per-second `@82` SpO2 percentages for the night; analyzeDay banks the + // ramp-trimmed nightly MEDIAN on `DailyMetric.spo2Pct`. Read from the durable table, which + // is never pruned — so re-scoring an old night still finds its readings, where the capped + // aux table they were forked from has held only ~7 days since v31. Empty on a WHOOP 4.0 and + // on windows offloaded before v34 → the percentage stays nil rather than fabricated. + let spo2Pct = (try? await store.spo2PctSamples(deviceId: owner, from: from, to: to, limit: 200_000)) ?? [] // #938: the strap family that WROTE this owner's skin-temp rows, so analyzeDay converts the raw // register on the right scale (5/MG banks centidegrees, a WHOOP 4.0 v24 banks a raw ADC). The // registry knows each device's model; unknown/non-WHOOP owners fall back to `.whoop5` (the prior @@ -714,6 +720,7 @@ final class IntelligenceEngine: ObservableObject { skinTempFamily: skinFamily, // #938 skinTempAnchorRaw: skinAnchorRaw, // #938 second capture spo2: spo2, // #93 + spo2PctSamples: spo2Pct, // v34 profile: up, baselines: baselines1, maxHROverride: maxHR, tzOffsetSeconds: tzOffset, wristOff: wristOff, sleepNeedHours: sleepNeedHours, diff --git a/android/app/src/test/resources/schema_oracle.json b/android/app/src/test/resources/schema_oracle.json index fdf93b2da7..ed67ec4280 100644 --- a/android/app/src/test/resources/schema_oracle.json +++ b/android/app/src/test/resources/schema_oracle.json @@ -36,7 +36,8 @@ "v28-raw-imu", "v29-score-input-provenance", "v30-rr-ord", - "v31-deep-capture-channels" + "v31-deep-capture-channels", + "v34-spo2-pct-durable" ], "grdbMigrationLineages": { "fork-ios-only": { @@ -47,6 +48,12 @@ "v28-phone-timezone", "v29-daily-avg-sdnn" ] + }, + "fork-ahead-of-upstream": { + "reason": "This fork numbers a migration AHEAD of upstream on purpose, which is the mirror image of the `fork-ios-only` case and needs its own entry rather than being folded into it. Local main carries migrations through v31; upstream ryanbr/noop already holds `v32-rr-src-channel` and `v33-workout-steps`, which this branch has not merged yet. Numbering this one v32 would therefore collide with an upstream identifier the very next time the fork merges upstream \u2014 and because GRDB keys migrations by identifier STRING, that collision is not a merge conflict a human sees: two different bodies claiming the same name means GRDB silently SKIPS the second, so one of the two tables is never created and the failure surfaces later as a missing-table error on a real device. Starting at v34 leaves upstream's v32/v33 free to arrive untouched. It is declared here rather than renumbered because the baseline lineage must stay exactly v1..vN, and because once this identifier has run on any device renaming it re-runs its body against a schema that already has the table.", + "migrations": [ + "v34-spo2-pct-durable" + ] } }, "divergenceReasons": { @@ -1691,6 +1698,35 @@ ], "indices": [] }, + "spo2PctSample": { + "platform": "ios_only", + "reason": "Fork-only iOS table (GRDB `v34-spo2-pct-durable`): the durable per-second SpO2 percentages the WHOOP 5/MG emits on `@82` of its v18 record. The byte is already captured raw inside `v18AuxSample.fields`, but that table is CAPPED at 604,800 rows/device (~7 days at 1 Hz) because the aux blob costs ~30 MB/day, so every reading rolled off after a week and was gone permanently \u2014 the strap trims its own history the moment an offload is acked. The in-band signal is only ~2,000 samples/week (~100k rows/year), so a narrow never-pruned sibling retains it forever at trivial cost while the wide aux capture stays capped. `pct` is the RAW in-band byte (70...100); run grouping and acquisition-ramp trimming are read-time policy, not stored. No Room twin yet: the Kotlin capture/aggregate side is outstanding, and until it lands Android keeps banking the byte in its own `v18AuxSample` twin and loses it on the same 7-day roll.", + "columns": [ + { + "name": "deviceId", + "affinity": "TEXT", + "notNull": true, + "default": null + }, + { + "name": "ts", + "affinity": "INTEGER", + "notNull": true, + "default": null + }, + { + "name": "pct", + "affinity": "INTEGER", + "notNull": true, + "default": null + } + ], + "primaryKey": [ + "deviceId", + "ts" + ], + "indices": [] + }, "spo2Sample": { "platform": "both", "columns": [ From 1ccddae1bb2638d00c9ec554b3f22a98a3d86102 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 5 Aug 2026 18:49:28 -0400 Subject: [PATCH 162/171] Don't publish an unvalidated SpO2 into the system health record MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Populating dailyMetric.spo2Pct for WHOOP nights has a consequence that is easy to miss: HealthKitBridge writes that field back as .oxygenSaturation. This export loop reads only the computed/imported NOOP device rows — Oura never passes through it — so while spo2Pct was nil the branch had never fired. Filling the field does not resume an export, it starts one. What it would start writing is a number nobody has checked against a reference oximeter. The bar is higher than for the in-app card: a HealthKit sample lands in the system health record under Apple's Blood Oxygen heading, sits beside clinically-sourced readings with nothing distinguishing it, syncs to iCloud, and is readable by every other authorised app. Deleting it later does not recall what already propagated. The evidence supports the decode — distribution, run structure, sleep gating, night-to-night stability, agreement with this user's own Oura-era medians — but all of it establishes that the byte is a real oxygen measurement, not that 97 means 97. Upstream declines to promote the same metric for the same reason: two straps checked against the WHOOP app moved in opposite directions. Gated behind one named constant with the reasoning at the declaration, so enabling it after paired-oximeter nights is a one-line change and a deliberate one. Everything else the agent built — the durable table, the decode, the nightly aggregate, the in-app card — is untouched and live. --- StrandiOS/Health/HealthKitBridge.swift | 29 +++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/StrandiOS/Health/HealthKitBridge.swift b/StrandiOS/Health/HealthKitBridge.swift index b90240a0fc..70bad7220b 100644 --- a/StrandiOS/Health/HealthKitBridge.swift +++ b/StrandiOS/Health/HealthKitBridge.swift @@ -115,6 +115,32 @@ final class HealthKitBridge: ObservableObject { private static let quantityWriteIds: [HKQuantityTypeIdentifier] = [ .restingHeartRate, .heartRateVariabilitySDNN, .oxygenSaturation, .respiratoryRate ] + + /// Whether a strap-derived nightly SpO2 is written back to HealthKit as `.oxygenSaturation`. + /// + /// FALSE, deliberately, and it is not an oversight to be tidied away. + /// + /// This export loop reads only the computed/imported NOOP device rows — Oura never passes through + /// it — so before `spo2Pct` was populated for WHOOP nights this branch never fired even once. + /// Turning the field on therefore does not "resume" an export; it STARTS one, and what it would + /// start writing is a number nobody has checked against a reference oximeter. + /// + /// Why that bar is higher here than for the in-app card. A value in the app is read in context, + /// next to its own provenance, by the person who built it. A HealthKit sample is different in + /// kind: it lands in the system health record under Apple's own Blood Oxygen heading, sits + /// alongside clinically-sourced readings with nothing distinguishing it, syncs to iCloud, and is + /// readable by every other app the user has authorised. Deleting it later is fiddly and does not + /// recall what already propagated. + /// + /// The evidence genuinely does not reach that bar yet. The decode is well supported — + /// distribution, run structure, sleep gating, night-to-night stability, and agreement with this + /// user's own Oura-era medians — but ALL of it says "this byte is a real oxygen measurement", + /// none of it says "97 means 97". Upstream declines to promote the same metric for the same + /// reason: two straps were checked against the WHOOP app and moved in OPPOSITE directions. + /// + /// Flip this to `true` once paired reference-oximeter nights exist and the bias is characterised + /// (and, if the bias is a fixed offset, correct for it before writing rather than here). + static let exportsSpo2ToHealthKit = false // High-res write-back shares: the continuous 1-minute HR stream, and the energy/distance samples // attached to written workouts. Kept out of `quantityWriteIds` so `legacyCoreWriteTypes` (the // auth-resume set) stays exactly what pre-update users granted. @@ -729,7 +755,8 @@ final class HealthKitBridge: ObservableObject { if let sdnn = row.avgSdnn ?? row.avgHrv { add(.heartRateVariabilitySDNN, .secondUnit(with: .milli), sdnn, row.day, at) } - if let spo2 = row.spo2Pct { + // DELIBERATELY NOT EXPORTED while `exportsSpo2ToHealthKit` is false — see the constant. + if HealthKitBridge.exportsSpo2ToHealthKit, let spo2 = row.spo2Pct { add(.oxygenSaturation, .percent(), spo2 / 100, row.day, at) } if let rr = row.respRateBpm { From 1b8c0aae7a9e365988d0d35d525d9f16d5dbab24 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Wed, 5 Aug 2026 23:17:54 -0400 Subject: [PATCH 163/171] Android twins: durable SpO2 (Room 26) and raw-history archive hardening MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two features landed on Swift with their Kotlin twins explicitly outstanding. Both are now written, and the platforms agree again. DURABLE SpO2 — Room 25 -> 26, MIGRATION_25_26, twin of GRDB v34-spo2-pct-durable. The 5/MG emits its own computed SpO2 % on @82 of the v18 record. Android has banked that byte since MIGRATION_24_25 — inside v18AuxSample.fields, capped at 604,800 rows/device (~7 days at 1 Hz) because the aux blob costs ~30 MB/day. So every reading rolled off after a week and was gone: the strap trims its history the moment an offload is acked. spo2PctSample(deviceId, ts, pct), PK (deviceId, ts), is the narrow never-pruned sibling; the wide aux capture stays capped and unchanged. Room's version and GRDB's identifier are independent counters that drifted apart with the fork's iOS-only lineage, so 25 -> 26 pairs with v34 rather than v32/v33. Room's is a dense integer sequence and takes the next free value; GRDB's is a string numbered ahead of upstream on purpose. schema_oracle.json moves spo2PctSample from ios_only to both, bumps roomVersion to 26, and drops the "no Room twin yet" reason rather than leaving it to rot. Both byte-identical copies updated; both halves of the oracle suite pass. The write reads V18AuxRow.auxByte82 off the decoded row, never fields[23] of the packed blob. Byte 23 is correct only because every slot ahead of @82 happens to be present in production; V18AuxCodec omits absent slots entirely, so one missing earlier slot slides the read onto opticalAmpB — which still lands in 70..100 and so fails silently as a plausible percentage. Both halves pinned in V18AuxCodecOffsetTest, the twin of the Swift file. Storage is lossless: the raw in-band byte, verbatim. The only thing applied on the way in is the SPO2_CANDIDATE_IN_BAND (70..100) demultiplex, which is not a policy — @82 also carries bit-7 sentinels, sub-70 diagnostic codes and 0. That range is now a named constant the store fork, nightlySpo2Pct and the existing nightlySpo2CandidateMean all read, instead of three matching literals. Nightly aggregate fills DailyMetric.spo2Pct, which was unconditionally null: a per-run ramp-trimmed median, runs split on gaps > 60 s, dropping the leading 5 of each run but never more than half a run. Same fixtures and expected values as Spo2PctNightlyTests. Retention/registry wired consistently: excluded from the aux sweep by construction (there is deliberately no prune query), added to the deviceScoped delete + adopt-serial re-key, and to the timestamp heal — a table that is never pruned on age would otherwise carry a bad-clock row forever. HealthConnectWriter.EXPORTS_SPO2_TO_HEALTH_CONNECT = false, the twin of Swift's exportsSpo2ToHealthKit. Filling spo2Pct does not resume that export, it starts one: the loop reads only computed rows, so while the field was null the branch had never fired. What it would start writing is a number nobody has checked against a reference oximeter, landing in the system health record beside clinically-sourced readings. RAW HISTORY ARCHIVE — twin of the Swift hardening. The archive filter asked the wrong question for an unmapped layout: it kept a type-47 record only when the DECODE produced nothing. On Android that gives the right answer today only because decodeWhoop5Historical returns null for every version but 18 — an accident of the decoder's current shape. The decision is now structural: MAPPED_WHOOP5_HISTORICAL_VERSIONS is the single source of truth, and any 5/MG record outside it is archived whatever it decoded. That set is {18, 26} here, NOT Swift's {18, 20, 21, 26}. Swift dispatches v20 (raw optical) and v21 (raw IMU) through decodeWhoop5HistoricalV2021; the Kotlin historical path has no such branch — Whoop5RawOptical/Whoop5RawImu exist but are wired to the live deep-buffer route. Claiming 20/21 would assert a field map this platform does not have, which is the exact class of bug the constant closes. The divergence is written down at the declaration and asserted in the test. Retention is now weakest-value-first: malformed lines, then surplus lines whose payload region is ENTIRELY zero, then surplus informative lines. The floor splits with it — 64 informative per version, 8 all-zero. The test is strictly all-zero: one non-zero byte anywhere makes a frame informative, because a small populated block inside an empty buffer is exactly what the archive is for. payloadStart is 21 on 5/MG and 17 on WHOOP 4, deliberately past the header, or a region carrying the non-zero seq/ts bytes would never test as empty. An over-cap rewrite evicts to 7/8 of the cap so a 1 Hz stream no longer rewrites the whole file per record. evictLines takes zeroFloor defaulted, matching Swift, so no call site churns. Tests: android/./gradlew testFullDebugUnitTest — 3483 pass, 0 fail, 5 skipped (baseline was green at 3441). 32 new: 14 nightly, 10 store/migration, 2 codec offset, 6 layout rule; 5 added to the eviction suite. swift test --filter SchemaOracleTests: 7 pass, confirming the shared fixture still satisfies both halves. Three fake DAOs and DeviceRegistryTest's expected-table set updated for the new device-scoped table. Not twinned, with reasons: the Swift LocalAccessCore storage-stats entry (no NoopLocalAccess on Android and no table-enumerating storage readout), and the Devices -> MG ECG "Check archived raw records" operator screen (no MG ECG screen exists on Android; the ECG probe surface is iOS-only). --- .../Resources/schema_oracle.json | 5 +- Strand/Collect/RawHistoryArchive.swift | 4 +- .../com/noop/analytics/AnalyticsEngine.kt | 131 ++++++++++- .../com/noop/analytics/IntelligenceEngine.kt | 7 + .../java/com/noop/ble/RawHistoryArchive.kt | 189 +++++++++++++--- .../main/java/com/noop/data/DeviceRegistry.kt | 2 + .../java/com/noop/data/DeviceRegistryDao.kt | 6 + .../src/main/java/com/noop/data/Entities.kt | 39 ++++ .../main/java/com/noop/data/V18AuxCodec.kt | 36 +++ .../src/main/java/com/noop/data/WhoopDao.kt | 33 +++ .../main/java/com/noop/data/WhoopDatabase.kt | 52 ++++- .../java/com/noop/data/WhoopRepository.kt | 59 +++++ .../com/noop/ingest/HealthConnectWriter.kt | 31 ++- .../com/noop/protocol/HistoricalStreams.kt | 67 +++++- .../analytics/RegistryDayOwnerSourceTest.kt | 2 + .../com/noop/analytics/Spo2PctNightlyTest.kt | 203 +++++++++++++++++ .../noop/ble/RawHistoryArchiveEvictionTest.kt | 114 ++++++++++ .../noop/ble/SourceCoordinatorAdoptionTest.kt | 2 + .../java/com/noop/data/DeviceRegistryTest.kt | 5 + .../noop/data/Spo2PctDurableMigrationTest.kt | 210 ++++++++++++++++++ .../com/noop/data/V18AuxCodecOffsetTest.kt | 82 +++++++ .../protocol/UnmappedHistoricalLayoutTest.kt | 154 +++++++++++++ .../app/src/test/resources/schema_oracle.json | 5 +- 23 files changed, 1387 insertions(+), 51 deletions(-) create mode 100644 android/app/src/test/java/com/noop/analytics/Spo2PctNightlyTest.kt create mode 100644 android/app/src/test/java/com/noop/data/Spo2PctDurableMigrationTest.kt create mode 100644 android/app/src/test/java/com/noop/data/V18AuxCodecOffsetTest.kt create mode 100644 android/app/src/test/java/com/noop/protocol/UnmappedHistoricalLayoutTest.kt diff --git a/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json b/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json index ed67ec4280..585c4e13ef 100644 --- a/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json +++ b/Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json @@ -1,6 +1,6 @@ { "_readme": "SHARED Room<->GRDB SCHEMA ORACLE (#775). Two byte-identical copies: Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json and android/app/src/test/resources/schema_oracle.json. SchemaOracleTests.swift compares GRDB's PRAGMA table_info/index_list against it; SchemaOracleTest.kt compares Room's exported schema JSON against it. `columns` is the iOS/GRDB shape in GRDB column order (macOS is the reference implementation); every way Android differs is spelled out in an `android` / `iosAbsent` / `androidColumnOrder` override naming a key in `divergenceReasons`. Adding a column, reordering one, or changing a type/nullability on one platform only fails both suites until it is either fixed or written down here. `grdbMigrationLineages` is the same idea for migration IDENTIFIERS: a fork that numbered its own migrations before upstream reached those numbers declares them there instead of renumbering \u2014 GRDB keys migrations by identifier string, so renaming one wedges a database that already ran it.", - "roomVersion": 25, + "roomVersion": 26, "grdbMigrations": [ "v1", "v2", @@ -1699,8 +1699,7 @@ "indices": [] }, "spo2PctSample": { - "platform": "ios_only", - "reason": "Fork-only iOS table (GRDB `v34-spo2-pct-durable`): the durable per-second SpO2 percentages the WHOOP 5/MG emits on `@82` of its v18 record. The byte is already captured raw inside `v18AuxSample.fields`, but that table is CAPPED at 604,800 rows/device (~7 days at 1 Hz) because the aux blob costs ~30 MB/day, so every reading rolled off after a week and was gone permanently \u2014 the strap trims its own history the moment an offload is acked. The in-band signal is only ~2,000 samples/week (~100k rows/year), so a narrow never-pruned sibling retains it forever at trivial cost while the wide aux capture stays capped. `pct` is the RAW in-band byte (70...100); run grouping and acquisition-ramp trimming are read-time policy, not stored. No Room twin yet: the Kotlin capture/aggregate side is outstanding, and until it lands Android keeps banking the byte in its own `v18AuxSample` twin and loses it on the same 7-day roll.", + "platform": "both", "columns": [ { "name": "deviceId", diff --git a/Strand/Collect/RawHistoryArchive.swift b/Strand/Collect/RawHistoryArchive.swift index b7594dae24..76d2c447d6 100644 --- a/Strand/Collect/RawHistoryArchive.swift +++ b/Strand/Collect/RawHistoryArchive.swift @@ -273,7 +273,9 @@ struct RawHistoryArchive { /// full `floor` stays reserved for records that actually contain something. /// /// Bounded: if everything left is floor-protected it stops even if still over cap (the floor wins — - /// `(floor + zeroFloor) × distinctVersions` is tiny). The Android twin has not been updated yet. + /// `(floor + zeroFloor) × distinctVersions` is tiny). Twin of the Android + /// `RawHistoryArchive.evictLines`, whose `zeroFloor` is defaulted for the same reason it is here — so + /// existing call sites do not have to churn. static func evictLines(_ lines: [String], maxBytes: Int, floor: Int, zeroFloor: Int = RawHistoryArchive.zeroPayloadFloor) -> [String] { // Mark the newest `floor` informative and newest `zeroFloor` zero-payload indices of each version diff --git a/android/app/src/main/java/com/noop/analytics/AnalyticsEngine.kt b/android/app/src/main/java/com/noop/analytics/AnalyticsEngine.kt index 480f70b859..afbc5aca1c 100644 --- a/android/app/src/main/java/com/noop/analytics/AnalyticsEngine.kt +++ b/android/app/src/main/java/com/noop/analytics/AnalyticsEngine.kt @@ -9,6 +9,8 @@ import com.noop.data.SkinTempSample import com.noop.data.Spo2Sample import com.noop.data.RespSample import com.noop.data.RrInterval +import com.noop.data.SPO2_CANDIDATE_IN_BAND +import com.noop.data.Spo2PctRow import com.noop.data.StepSample import com.noop.protocol.DeviceFamily import com.noop.protocol.Whoop4SkinTemp @@ -183,6 +185,13 @@ object AnalyticsEngine { // decoded" data, NOT a calibrated blood-oxygen % (that needs WHOOP's proprietary curve). Default // empty keeps pure-function callers/tests + non-4.0 nights null. spo2: List = emptyList(), + // Durable 5/MG `@82` SpO2 percentages for the night window (v34 / MIGRATION_25_26). The + // ramp-trimmed nightly MEDIAN is banked on `DailyMetric.spo2Pct` — the strap's OWN computed + // percentage, not a curve NOOP invented. Distinct from [spo2] above, which is the WHOOP 4.0 v24 + // raw red/IR ADC pair and feeds `spo2Red`/`spo2Ir`; the two device generations report different + // physical quantities and neither substitutes for the other. Default empty keeps pure-function + // callers/tests + every non-5/MG night null. + spo2PctSamples: List = emptyList(), profile: UserProfile, baselines: ProfileBaselines = ProfileBaselines(), maxHROverride: Double? = null, @@ -441,6 +450,14 @@ object AnalyticsEngine { // as-is for the Health "Raw SpO2" tile — NOT a calibrated blood-oxygen %. (#93) val nightlySpo2Raw = nightlySpo2RawMeans(matched, spo2) + // ── SpO2 percentage (5/MG @82, durable) ─────────────────────────────── + // Ramp-trimmed nightly median of the strap's OWN computed SpO2 %, over the detected in-bed spans. + // Null on a WHOOP 4.0, on any night with no in-band reading, and on every window offloaded before + // MIGRATION_25_26. Unlike the raw red/IR means above this IS a percentage — it is the number the + // strap computed, banked verbatim per-second and aggregated here, not a calibration NOOP applied + // to an ADC. See [nightlySpo2Pct] for the run/ramp policy and why the statistic is a median. + val nightlySpo2PctVal = nightlySpo2Pct(matched, spo2PctSamples) + // ── Rest (sleep_performance composite, 0–100) ───────────────────────── // Replaces the bare efficiency proxy: duration-vs-personal-need 0.50 + efficiency 0.20 + // restorative (deep+REM)/asleep 0.20 + consistency 0.10. Stored under the sleep_performance @@ -606,7 +623,20 @@ object AnalyticsEngine { recovery = recovery, strain = strain, exerciseCount = workouts.size, - spo2Pct = null, + // v34: was unconditionally null here — the on-device engine had no percentage to bank, so the + // Blood Oxygen card read "No Data" on every WHOOP night while the same field rendered fine + // for imported rows. It now carries the 5/MG's own `@82` median when the night has one, and + // stays null otherwise (WHOOP 4.0, pre-MIGRATION_25_26 windows, no in-band samples) — an + // absent reading must stay absent rather than become a fabricated number. + // + // NOTE FOR ANYONE CHANGING THIS: `dailyMetric.spo2Pct` is not display-only. + // [com.noop.ingest.HealthConnectWriter] writes it back as an `OxygenSaturationRecord`, so a + // value banked here can leave the app and land in the user's Health Connect blood-oxygen + // history alongside clinical readings. That export is gated behind + // `HealthConnectWriter.EXPORTS_SPO2_TO_HEALTH_CONNECT` for exactly that reason — read the + // constant before touching either. This field's bar is "the device measured this", never + // "this is our best guess". + spo2Pct = nightlySpo2PctVal?.first, skinTempDevC = skinTempDevC, respRateBpm = respRateDaily, steps = stepsTotal, @@ -765,7 +795,9 @@ object AnalyticsEngine { var kept = 0 for (a in aux) { val v = a.auxByte82 ?: continue - if (v < 70L || v > 100L) continue + // The same named window the store's fork and [nightlySpo2Pct] apply, so the three agree + // structurally rather than by three matching literals. + if (v.toInt() !in SPO2_CANDIDATE_IN_BAND) continue if (sessions.none { a.ts in it.start..it.end }) continue sum += v kept += 1 @@ -774,6 +806,101 @@ object AnalyticsEngine { return Pair((sum / kept).toInt(), kept) } + /** + * Nightly SpO2 percentage from the DURABLE `@82` stream (`spo2PctSample`, v34 / MIGRATION_25_26) over + * the detected in-bed [sessions] — the value banked on `DailyMetric.spo2Pct` for a 5/MG night — with + * the sample count that survived the ramp trim. Null when no reading fell inside any span. + * + * HOW THIS DIFFERS FROM [nightlySpo2CandidateMean]. That one is a naive mean over the raw candidate + * byte read out of the CAPPED aux table, built as a hand-checkable diagnostic. This is the scored + * statistic, over the durable stream, and it corrects the two things that make the naive mean wrong. + * + * THE ACQUISITION RAMP. The strap does not sample SpO2 continuously: it takes a RUN of ~30 one-second + * samples roughly once per ~1,200 s while asleep. The first samples of each run are the optical front + * end settling, and they read LOW — so a mean over every in-band sample is biased downward by a + * contamination that is entirely one-sided, and the bias grows with how much of the night is ramp + * (i.e. it is worse on nights with more, shorter runs — precisely when you would least suspect it). + * Runs are separated here by a gap of more than [SPO2_RUN_GAP_SECONDS]; at 1 Hz within a run and + * ~1,200 s between runs, that boundary is not a close call. The leading [SPO2_RAMP_SAMPLES] of each + * run are dropped — but never more than HALF a run, so a short run contributes its settled tail + * instead of vanishing and silently reweighting the night toward whichever runs happened to be long. + * + * WHY THE MEDIAN AND NOT A TRIMMED MEAN. Three reasons, in order of weight: + * 1. It is the statistic the cloud reader already reports (the per-night 94-97 medians every + * cross-check against this data has been quoted against). A phone that shipped a trimmed mean + * would disagree with the server on the same rows for no reason anyone could see. + * 2. Ramp removal is a heuristic, not a proof. A partially-settled sample just past the cut, or a + * motion artifact mid-run, is residual one-sided contamination that a mean absorbs in full and a + * median very largely ignores. + * 3. The in-band values live in a 31-wide integer band with a hard mode at 97, so a median is + * stable and interpretable here in a way it would not be on a sparse continuous quantity. + * The trim and the median are complementary, not redundant: the trim removes a KNOWN, structured, + * large block of low readings that would move a median too, and the median absorbs what is left. + * + * READ-TIME POLICY, DELIBERATELY. None of this is applied on the way into the store — + * `spo2PctSample` holds raw in-band bytes. These rules have already changed once on the cloud side, + * and they will change again if a run's shape turns out to differ by firmware; keeping them here + * means that costs a re-score, not a re-capture of data the strap has long since trimmed. + * + * Pure + deterministic. Byte-parity twin of the Swift `AnalyticsEngine.nightlySpo2Pct`; the returned + * pair is (median percentage, surviving sample count), matching Swift's `(pct:samples:)`. + */ + fun nightlySpo2Pct( + sessions: List, + samples: List, + ): Pair? { + if (sessions.isEmpty() || samples.isEmpty()) return null + // In-bed first, so run boundaries are computed over the samples that will actually be used. Note + // what that does and does not mean: a session edge splits a run only when the stretch it excludes + // is itself longer than [SPO2_RUN_GAP_SECONDS] — a brief out-of-bed moment mid-run leaves the run + // intact and singly-trimmed. Gap size is the ONLY run signal; session boundaries are not a second + // one. That is deliberate, because the ramp is a property of the sensor restarting, and the + // sensor does not restart because the sleep detector drew a line. + val inBed = samples + .filter { s -> sessions.any { s.ts >= it.start && s.ts <= it.end } } + .sortedBy { it.ts } + if (inBed.isEmpty()) return null + + val kept = ArrayList(inBed.size) + fun flushRun(fromIdx: Int, toIdx: Int) { + val run = inBed.subList(fromIdx, toIdx) + // Never trim more than half a run: a 4-sample run keeps its last 2 rather than disappearing. + val drop = minOf(SPO2_RAMP_SAMPLES, run.size / 2) + for (i in drop until run.size) kept.add(run[i].pct) + } + var runStart = 0 + for (i in 1 until inBed.size) { + if (inBed[i].ts - inBed[i - 1].ts > SPO2_RUN_GAP_SECONDS) { + flushRun(runStart, i) + runStart = i + } + } + flushRun(runStart, inBed.size) + if (kept.isEmpty()) return null + + kept.sort() + val mid = kept.size / 2 + val median = + if (kept.size % 2 == 0) (kept[mid - 1] + kept[mid]) / 2.0 + else kept[mid].toDouble() + return Pair(median, kept.size) + } + + /** + * Gap (seconds) above which two consecutive in-band `@82` samples belong to different measurement + * runs. Samples inside a run are 1 s apart and runs are ~1,200 s apart, so anything in between + * separates them; 60 s sits in the middle of that gulf and tolerates a dropped second or two inside a + * run without splitting it. Swift twin: `AnalyticsEngine.spo2RunGapSeconds`. + */ + const val SPO2_RUN_GAP_SECONDS: Long = 60L + + /** + * Leading samples dropped from each measurement run as the optical front end's acquisition ramp. + * ~5 of a ~30-sample run. Capped at half a run by the caller so short runs still contribute. + * Swift twin: `AnalyticsEngine.spo2RampSamples`. + */ + const val SPO2_RAMP_SAMPLES: Int = 5 + /** Plausible worn skin-temperature range (°C). Off-wrist/charging samples drift to ambient and are * excluded; the strap's own decode gate is the looser 20–45. (PR #85) */ private const val SKIN_TEMP_MIN_C: Double = 28.0 diff --git a/android/app/src/main/java/com/noop/analytics/IntelligenceEngine.kt b/android/app/src/main/java/com/noop/analytics/IntelligenceEngine.kt index d40b742514..f06e453420 100644 --- a/android/app/src/main/java/com/noop/analytics/IntelligenceEngine.kt +++ b/android/app/src/main/java/com/noop/analytics/IntelligenceEngine.kt @@ -482,6 +482,12 @@ object IntelligenceEngine { // #93: WHOOP 4.0 raw SpO2 PPG samples for the night; analyzeDay banks the nightly red/IR ADC // means on the DailyMetric. Empty on a 5/MG (no v24 spo2 channels) → the raw means stay null. val spo2 = repo.spo2Samples(owner, from, to, STREAM_LIMIT) + // v34: the 5/MG's own per-second `@82` SpO2 percentages for the night; analyzeDay banks the + // ramp-trimmed nightly MEDIAN on `DailyMetric.spo2Pct`. Read from the durable table, which is + // never pruned — so re-scoring an old night still finds its readings, where the capped aux + // table they were forked from has held only ~7 days. Empty on a WHOOP 4.0 and on windows + // offloaded before MIGRATION_25_26 → the percentage stays null rather than fabricated. + val spo2Pct = repo.spo2PctSamples(owner, from, to, STREAM_LIMIT) // #938: the strap family that WROTE this owner's skin-temp rows, so analyzeDay converts the raw // register on the right scale (5/MG banks centidegrees, a WHOOP 4.0 v24 banks a raw ADC). The // owner source resolves it from the registry; unknown/non-WHOOP owners fall back to WHOOP5 (the @@ -568,6 +574,7 @@ object IntelligenceEngine { skinTempFamily = skinFamily, // #938 skinTempAnchorRaw = skinAnchorRaw, // #938 second capture: per-device worn anchor spo2 = spo2, // #93 + spo2PctSamples = spo2Pct, // v34 profile = profile, baselines = baselines1, maxHROverride = maxHROverride, diff --git a/android/app/src/main/java/com/noop/ble/RawHistoryArchive.kt b/android/app/src/main/java/com/noop/ble/RawHistoryArchive.kt index 51a8e786bd..cca3688130 100644 --- a/android/app/src/main/java/com/noop/ble/RawHistoryArchive.kt +++ b/android/app/src/main/java/com/noop/ble/RawHistoryArchive.kt @@ -21,12 +21,14 @@ import java.io.FileOutputStream * {"capturedAtMs":,"trim":,"family":"whoop4"|"whoop5","frameHex":""} * Each [append] flushes + fsyncs before returning, so a row is durable before the caller acks. * - * Size cap ([maxBytes], ~5 MB): if appending would push the file past the cap, [append] EVICTS oldest - * surplus lines to make room rather than refusing the write — but only down to a per-version retention - * floor ([PER_VERSION_FLOOR]), so a brand-new layout version is never binned merely because common - * versions filled the file. Only when the incoming frames alone can't fit even an empty archive does - * [append] skip them ([AppendResult.written] = false); the caller records those as unarchived so the - * sync status never falsely claims they were preserved. (#344) + * Size cap ([maxBytes], ~5 MB): if appending would push the file past the cap, [append] EVICTS surplus + * lines to make room rather than refusing the write — weakest-value first, and never below a per-version + * retention floor ([PER_VERSION_FLOOR] for informative lines, [ZERO_PAYLOAD_FLOOR] for lines whose + * payload region is entirely zero), so a brand-new layout version is never binned merely because common + * versions — or a 1 Hz flood of empty placeholder records — filled the file. See [evictLines] for the + * full priority order. Only when the incoming frames alone can't fit even an empty archive does [append] + * skip them ([AppendResult.written] = false); the caller records those as unarchived so the sync status + * never falsely claims they were preserved. (#344) * * A genuine WRITE FAILURE (I/O error) instead throws — the caller treats that as "do NOT ack", so the * strap keeps the records and re-sends them on the next offload. No data is lost either way. @@ -36,6 +38,7 @@ class RawHistoryArchive( private val maxBytes: Long = REJECTED_ARCHIVE_MAX_BYTES, // Overridable so tests can drive eviction with a small archive instead of 5 MB of frames. (#344) private val perVersionFloor: Int = PER_VERSION_FLOOR, + private val zeroPayloadFloor: Int = ZERO_PAYLOAD_FLOOR, ) { /** * Outcome of an [append]. [ok] is true whenever the offload may proceed to ack; [written] is true @@ -87,7 +90,7 @@ class RawHistoryArchive( // the newest [PER_VERSION_FLOOR] of its version. Atomic rewrite (temp file + rename) so the // archive is durable BEFORE the ack. [evictLines] is the pure, unit-tested core. val existing = if (f.exists()) f.readLines().filter { it.isNotEmpty() }.map { "$it\n" } else emptyList() - val kept = evictLines(existing + newLines, maxBytes, perVersionFloor) + val kept = evictLines(existing + newLines, maxBytes, perVersionFloor, zeroPayloadFloor) val tmp = File(context.filesDir, "$REJECTED_ARCHIVE_FILE.tmp") FileOutputStream(tmp).use { out -> out.write(kept.joinToString("").toByteArray(Charsets.UTF_8)) @@ -168,7 +171,15 @@ class RawHistoryArchive( /** Archive filename in the app-private filesDir. */ const val REJECTED_ARCHIVE_FILE = "rejected_history.jsonl" - /** ~5 MB cap; above this [append] evicts oldest surplus (floor-aware) to make room (#344). */ + /** + * ~5 MB cap; above this [append] evicts surplus (weakest-value first, floor-aware) to make room + * (#344). See [evictLines] for the full priority order. + * + * Deliberately NOT raised to survive a placeholder flood. At ~3.2 KB per hex line a 1 Hz stream of + * 1584-byte records fills any cap in hours, so a bigger number buys a linear factor of time and + * costs every user disk — while [ZERO_PAYLOAD_FLOOR] removes the pressure outright by evicting the + * informationless records first. Value, not volume, is what was missing. + */ const val REJECTED_ARCHIVE_MAX_BYTES = 5L * 1024 * 1024 /** @@ -181,6 +192,69 @@ class RawHistoryArchive( */ const val PER_VERSION_FLOOR = 64 + /** + * Per-version retention floor for lines whose PAYLOAD REGION IS ENTIRELY ZERO — far smaller than + * [PER_VERSION_FLOOR], because such a record carries no information beyond "the strap banked a + * record at time T". + * + * Measured motivation: with `enable_raw_data_w_ecg` armed, a WHOOP MG banks one 1584-byte record + * PER SECOND whose every byte from offset 21 to the CRC32 trailer is zero (invariant across 1,307 + * captured frames). At ~3.2 KB per hex JSONL line that fills the whole 5 MB archive in ~25 + * minutes, so under a purely age-ordered policy an actual ECG record captured during the + * electrode-contact window is evicted by placeholders within the half hour. Keeping a handful + * still proves the artefact exists (and dates it); keeping thousands buys nothing and costs the + * evidence. Twin of the Swift `RawHistoryArchive.zeroPayloadFloor`. + */ + const val ZERO_PAYLOAD_FLOOR = 8 + + /** Bytes of the CRC32 trailer excluded from the payload region. */ + const val CRC_TRAILER_BYTES = 4 + + /** + * A payload region shorter than this is not judged at all ([hasZeroPayload] returns false): a + * truncated or tiny frame is not evidence of an empty record, and must not lose retention + * priority on the strength of a few zero bytes. + */ + const val MIN_PAYLOAD_BYTES_TO_JUDGE = 32 + + /** + * First byte of a record's PAYLOAD region — everything past the record header, ending at the + * 4-byte CRC32 trailer. 21 on WHOOP 5/MG (magic @0, seq @11, unix ts @15-18, a constant @19-20) + * and 17 on WHOOP 4: the same +4 that shifts the type byte 4→8 and the version byte 5→9, because + * the puffin envelope is 4 bytes longer. Starting AFTER the header is load-bearing — a region + * that included the non-zero seq/timestamp bytes would never test as all-zero and + * [hasZeroPayload] would never fire. Twin of the Swift `RawHistoryArchive.payloadStart`. + */ + fun payloadStart(family: DeviceFamily): Int = if (family == DeviceFamily.WHOOP5) 21 else 17 + + /** + * True when EVERY byte of [frame]'s payload region is zero — an informationless record. + * + * Deliberately the STRICT test: a single non-zero byte anywhere in the region makes the frame + * informative and gives it full retention priority. "Mostly zero" is NOT the test — a novel + * record that carries one small populated block inside an otherwise-empty buffer is precisely the + * thing this archive exists to catch. Twin of the Swift `RawHistoryArchive.hasZeroPayload`. + */ + fun hasZeroPayload(frame: ByteArray, family: DeviceFamily): Boolean { + val start = payloadStart(family) + val end = frame.size - CRC_TRAILER_BYTES + if (end - start < MIN_PAYLOAD_BYTES_TO_JUDGE) return false + for (i in start until end) if (frame[i].toInt() != 0) return false + return true + } + + /** + * Fraction of the cap freed once eviction has to run at all: the rewrite evicts down to + * `maxBytes - maxBytes/LOW_WATER_DIVISOR` rather than to exactly the cap. + * + * Without this, an archive sitting AT the cap does a full read-parse-rewrite of the whole file on + * EVERY subsequent batch — and the batches that matter here arrive during an offload at up to one + * record per strap-second. Leaving an eighth of the cap free amortises that to one rewrite per + * ~600 KB of new frames. The bytes given up are the ones eviction would have taken next anyway, + * in the same priority order. Twin of the Swift `RawHistoryArchive.lowWaterDivisor`. + */ + const val LOW_WATER_DIVISOR = 8 + /** * The hist_version byte distinguishing one historical layout from another: `frame[5]` on WHOOP * 4, `frame[9]` on WHOOP 5/MG (the puffin envelope is 4 bytes longer). Same indices the reject @@ -198,46 +272,91 @@ class RawHistoryArchive( private data class VersionKey(val family: String, val version: Int) /** - * The [VersionKey] of one archived JSONL line, re-derived from its stored frame + family; null - * for a malformed line (which then gets NO retention floor — it is evicted before any real, - * floor-protected line, so garbage can never occupy a slot a real rare version could use). + * How one archived line is classified for retention: which version bucket it belongs to, and + * whether its payload region is entirely zero (informationless — see [hasZeroPayload]). + */ + private data class LineClass(val key: VersionKey, val zeroPayload: Boolean) + + /** + * Classify one archived JSONL line from its stored frame + family; null for a + * malformed/unparseable line — those get NO retention floor (evict them first, never let garbage + * occupy a floor slot a real rare version could use), and are always evictable before any real, + * floor-protected line. */ - private fun lineVersionKey(line: String): VersionKey? { - val parsed = parseArchiveLine(line) ?: return null - val tag = if (parsed.second == DeviceFamily.WHOOP5) "whoop5" else "whoop4" - return VersionKey(tag, versionByte(parsed.first, parsed.second)) + private fun lineClass(line: String): LineClass? { + val (frame, family) = parseArchiveLine(line) ?: return null + val tag = if (family == DeviceFamily.WHOOP5) "whoop5" else "whoop4" + return LineClass(VersionKey(tag, versionByte(frame, family)), hasZeroPayload(frame, family)) } /** * Floor-aware cap eviction (#344), PURE so it is JVM-unit-testable without a Context. [lines] is - * the full newline-terminated JSONL set, oldest-first (existing ++ incoming). Drops oldest - * SURPLUS lines until the UTF-8 byte total fits [maxBytes], but NEVER a line within the newest - * [floor] lines of its version — so a rare never-seen layout (WHOOP 4 v19, WHOOP 5 v20/v21) - * always survives even when common versions fill the archive. Bounded: if everything left is - * floor-protected it stops even if still over cap (the floor wins — floor x distinctVersions is - * tiny). Mirrors the macOS RawHistoryArchive.evictLines. + * the full newline-terminated JSONL set, oldest-first (existing ++ incoming). + * + * Retention priority, weakest first — everything in a band is evicted before anything in the next: + * 1. malformed lines (unparseable — no version, no floor); + * 2. surplus lines whose payload region is ENTIRELY ZERO ([hasZeroPayload]), oldest first; + * 3. surplus informative lines, oldest first; + * 4. floor-protected lines, never evicted: the newest [zeroFloor] zero-payload lines of each + * version and the newest [floor] informative lines of each version — so a rare never-seen + * layout (WHOOP 4 v19, WHOOP 5 v20/v21) always survives even when common versions fill the + * archive. + * + * Band 2 is what keeps an ECG capture recoverable. A 5/MG with `enable_raw_data_w_ecg` armed banks + * one all-zero 1584-byte record per second; age-ordered eviction alone would flush a real + * contact-window record out of a 5 MB archive within ~25 minutes of placeholders, and an all-zero + * payload demonstrably carries nothing that would be worth that trade. Band 4's split floor means + * a version whose records are ALL empty still keeps a few dated samples of the artefact, while the + * full [floor] stays reserved for records that actually contain something. + * + * Once eviction has to run at all it evicts to [LOW_WATER_DIVISOR] below the cap rather than to + * exactly the cap, so a 1 Hz stream no longer rewrites the whole file once per record. + * + * Bounded: if everything left is floor-protected it stops even if still over cap (the floor wins — + * `(floor + zeroFloor) x distinctVersions` is tiny). Mirrors the macOS + * `RawHistoryArchive.evictLines`; [zeroFloor] is defaulted for exactly the reason it is defaulted + * on the Swift side — so existing call sites do not have to churn. */ - fun evictLines(lines: List, maxBytes: Long, floor: Int): List { - // Mark the newest [floor] indices of each version as protected (scan newest -> oldest). - // Malformed lines (null key) are never protected — evicted before any real, floor-kept line. - val keys = lines.map { lineVersionKey(it) } + fun evictLines( + lines: List, + maxBytes: Long, + floor: Int, + zeroFloor: Int = ZERO_PAYLOAD_FLOOR, + ): List { + // Mark the newest [floor] informative and newest [zeroFloor] zero-payload indices of each + // version as protected (scan newest -> oldest). Malformed lines (null class) are never + // protected — evicted before any real, floor-kept line. + val classes = lines.map { lineClass(it) } val protectedIdx = HashSet() - val seen = HashMap() + val seenInformative = HashMap() + val seenZero = HashMap() for (idx in lines.indices.reversed()) { - val k = keys[idx] ?: continue - val c = seen[k] ?: 0 - if (c < floor) { protectedIdx.add(idx); seen[k] = c + 1 } + val c = classes[idx] ?: continue + if (c.zeroPayload) { + val n = seenZero[c.key] ?: 0 + if (n < zeroFloor) { protectedIdx.add(idx); seenZero[c.key] = n + 1 } + } else { + val n = seenInformative[c.key] ?: 0 + if (n < floor) { protectedIdx.add(idx); seenInformative[c.key] = n + 1 } + } } var total = lines.sumOf { it.toByteArray(Charsets.UTF_8).size.toLong() } if (total <= maxBytes) return lines - // Evict oldest unprotected lines first. + val target = maxBytes - maxBytes / LOW_WATER_DIVISOR + // Evict in weakest-first bands; within a band, oldest first (ascending index). val dropped = HashSet() - for (idx in lines.indices) { // ascending = oldest first - if (total <= maxBytes) break - if (idx in protectedIdx) continue - dropped.add(idx) - total -= lines[idx].toByteArray(Charsets.UTF_8).size.toLong() + fun evict(admits: (Int) -> Boolean) { + for (idx in lines.indices) { // ascending = oldest first + if (total <= target) break + if (idx in protectedIdx || idx in dropped) continue + if (!admits(idx)) continue + dropped.add(idx) + total -= lines[idx].toByteArray(Charsets.UTF_8).size.toLong() + } } + evict { classes[it] == null } // 1. malformed + evict { classes[it]?.zeroPayload == true } // 2. zero-payload surplus + evict { true } // 3. informative surplus return lines.filterIndexed { idx, _ -> idx !in dropped } } diff --git a/android/app/src/main/java/com/noop/data/DeviceRegistry.kt b/android/app/src/main/java/com/noop/data/DeviceRegistry.kt index 435635ef12..30b625944e 100644 --- a/android/app/src/main/java/com/noop/data/DeviceRegistry.kt +++ b/android/app/src/main/java/com/noop/data/DeviceRegistry.kt @@ -85,6 +85,7 @@ class DeviceRegistry( dao.reKeyPpgWaveform(activeId, serialId); dao.deletePpgWaveformFor(activeId) dao.reKeyRawImu(activeId, serialId); dao.deleteRawImuFor(activeId) dao.reKeyV18Aux(activeId, serialId); dao.deleteV18AuxFor(activeId) + dao.reKeySpo2Pct(activeId, serialId); dao.deleteSpo2PctFor(activeId) dao.reKeyEvents(activeId, serialId); dao.deleteEventsFor(activeId) dao.reKeyBattery(activeId, serialId); dao.deleteBatteryFor(activeId) dao.reKeyDailyMetrics(activeId, serialId); dao.deleteDailyMetricsFor(activeId) @@ -153,6 +154,7 @@ class DeviceRegistry( dao.deletePpgWaveformFor(id) dao.deleteRawImuFor(id) // #423 dao.deleteV18AuxFor(id) + dao.deleteSpo2PctFor(id) dao.deleteEventsFor(id) dao.deleteBatteryFor(id) dao.deleteDailyMetricsFor(id) diff --git a/android/app/src/main/java/com/noop/data/DeviceRegistryDao.kt b/android/app/src/main/java/com/noop/data/DeviceRegistryDao.kt index 4f3d86666d..76d594146a 100644 --- a/android/app/src/main/java/com/noop/data/DeviceRegistryDao.kt +++ b/android/app/src/main/java/com/noop/data/DeviceRegistryDao.kt @@ -78,6 +78,11 @@ interface DeviceRegistryDao { @Query("DELETE FROM ppgWaveformSample WHERE deviceId = :deviceId") suspend fun deletePpgWaveformFor(deviceId: String) @Query("DELETE FROM rawImuSample WHERE deviceId = :deviceId") suspend fun deleteRawImuFor(deviceId: String) // #423 @Query("DELETE FROM v18AuxSample WHERE deviceId = :deviceId") suspend fun deleteV18AuxFor(deviceId: String) + // v34/MIGRATION_25_26: the durable `@82` SpO2 percentages OUTLIVE the aux table they are forked from + // (never pruned), which makes covering them here MORE important, not less — a wipe that cleared + // `v18AuxSample` but left this behind would leave years of blood-oxygen readings on the device after + // the user asked for them to be gone. Twin of the Swift `DeviceRegistryStore.deviceScopedTables` entry. + @Query("DELETE FROM spo2PctSample WHERE deviceId = :deviceId") suspend fun deleteSpo2PctFor(deviceId: String) @Query("DELETE FROM event WHERE deviceId = :deviceId") suspend fun deleteEventsFor(deviceId: String) @Query("DELETE FROM battery WHERE deviceId = :deviceId") suspend fun deleteBatteryFor(deviceId: String) @Query("DELETE FROM dailyMetric WHERE deviceId = :deviceId") suspend fun deleteDailyMetricsFor(deviceId: String) @@ -115,6 +120,7 @@ interface DeviceRegistryDao { @Query("UPDATE OR IGNORE ppgWaveformSample SET deviceId = :to WHERE deviceId = :from") suspend fun reKeyPpgWaveform(from: String, to: String) @Query("UPDATE OR IGNORE rawImuSample SET deviceId = :to WHERE deviceId = :from") suspend fun reKeyRawImu(from: String, to: String) @Query("UPDATE OR IGNORE v18AuxSample SET deviceId = :to WHERE deviceId = :from") suspend fun reKeyV18Aux(from: String, to: String) + @Query("UPDATE OR IGNORE spo2PctSample SET deviceId = :to WHERE deviceId = :from") suspend fun reKeySpo2Pct(from: String, to: String) @Query("UPDATE OR IGNORE event SET deviceId = :to WHERE deviceId = :from") suspend fun reKeyEvents(from: String, to: String) @Query("UPDATE OR IGNORE battery SET deviceId = :to WHERE deviceId = :from") suspend fun reKeyBattery(from: String, to: String) @Query("UPDATE OR IGNORE dailyMetric SET deviceId = :to WHERE deviceId = :from") suspend fun reKeyDailyMetrics(from: String, to: String) diff --git a/android/app/src/main/java/com/noop/data/Entities.kt b/android/app/src/main/java/com/noop/data/Entities.kt index 698031132c..08ce1de07a 100644 --- a/android/app/src/main/java/com/noop/data/Entities.kt +++ b/android/app/src/main/java/com/noop/data/Entities.kt @@ -687,3 +687,42 @@ data class V18AuxSampleEntity( return result } } + +/** + * One in-band `@82` SpO2 percentage from a WHOOP 5/MG v18 record, banked DURABLY + * (MIGRATION_25_26 / Swift GRDB `v34-spo2-pct-durable`). + * + * WHY THIS EXISTS AS ITS OWN TABLE. The byte is already carried raw inside [V18AuxSampleEntity.fields] + * (slot [V18AuxSlot.AUX_BYTE_82]), but `v18AuxSample` is CAPPED at + * [WhoopRepository.V18_AUX_RETENTION_ROWS] (604,800 rows/device, ~7 days at 1 Hz) because the aux blob + * costs ~30 MB/day. So every SpO2 reading rolled off after a week and was gone permanently: the strap + * trims its own history the moment an offload is acked, so no second copy exists anywhere. Raising the + * aux cap is the wrong fix — it buys a year of SpO2 at the price of ~11 GB of unrelated aux bytes. The + * in-band signal is only ~2,000 samples/week (~100k rows/year), so this narrow sibling retains it forever + * at trivial cost while the wide aux capture stays capped and unchanged. + * + * WHY NOT EXTEND [Spo2Sample] (the v3 table). Wrong shape, and extending it would corrupt a live reader: + * `spo2Sample` is `(deviceId, ts, red, ir)`, two NOT NULL raw ADC channels off the WHOOP 4.0 v24 layout, + * and [com.noop.analytics.AnalyticsEngine.nightlySpo2RawMeans] averages them into `spo2Red`/`spo2Ir`. + * We have a computed PERCENTAGE and no red/IR at all, so every insert would have to fabricate zeros and + * drag those means toward zero. Two different physical quantities from two device generations. + * + * [pct] is the RAW in-band byte ([SPO2_CANDIDATE_IN_BAND], 70..100), stored verbatim. Run grouping and + * acquisition-ramp trimming are READ-TIME policies (see + * [com.noop.analytics.AnalyticsEngine.nightlySpo2Pct]) — they have already changed once on the cloud + * reader, and a store that baked the old policy in would have destroyed the evidence needed to change + * them again. The only thing applied on the way in is the [SPO2_CANDIDATE_IN_BAND] demultiplex, which is + * not a policy but the definition of which values on `@82` are percentages at all. + * + * NOT PRUNED, by design — the same "durable decoded biometric history" class as `hrSample` and + * `ppgWaveformSample`. The `v18AuxSample` rolling sweep deletes only `FROM v18AuxSample`, so this table + * is untouched by it. PK (deviceId, ts) and field order (deviceId, ts, pct) mirror the GRDB schema, so a + * `.noopbak` round-trips. + */ +@Entity(tableName = "spo2PctSample", primaryKeys = ["deviceId", "ts"]) +data class Spo2PctSampleEntity( + val deviceId: String, + val ts: Long, + /** The raw in-band `@82` byte, 70..100. Never a sentinel, never a diagnostic code, never 0. */ + val pct: Int, +) diff --git a/android/app/src/main/java/com/noop/data/V18AuxCodec.kt b/android/app/src/main/java/com/noop/data/V18AuxCodec.kt index 1e012ec2e1..c9f7285e7d 100644 --- a/android/app/src/main/java/com/noop/data/V18AuxCodec.kt +++ b/android/app/src/main/java/com/noop/data/V18AuxCodec.kt @@ -182,6 +182,42 @@ data class V18AuxRow( } } +/** + * The in-band window for the `@82` SpO2 candidate byte: a value in `70..100` is a MEASUREMENT, and + * everything else on that byte is something else entirely. + * + * `@82` is MULTIPLEXED and separable BY VALUE — this is the separation, established over 626,725 + * production records: + * - `70..100` a real percentage. Distribution modes at 97; per-night medians 94-97; sleep-gated; + * emitted as a run of ~30 one-second samples once per ~1,200 s. + * - bit 7 set status sentinel (`0x80`, `0x88`, `0x90`, `0xA0`, `0xA8` are the observed values). + * - `1..69` diagnostic codes, not a saturation of anything. + * - `0` the strap emitted nothing this second (awake, or between runs). + * - `101..127` never observed — the gap is what makes the bit-7 sentinels and the measurements + * unambiguously separable without a side channel. + * + * This is the SAME window the decoder applies when it emits `spo2_candidate_82`, the same one + * [com.noop.analytics.AnalyticsEngine.nightlySpo2CandidateMean] applies, and the same one + * [WhoopRepository.insert] applies when forking [Spo2PctSampleEntity] rows out of the aux stream. + * Named here so the three agree structurally rather than by three matching literals. Byte-identical + * twin of the Swift `spo2CandidateInBand` (WhoopProtocol/Streams.swift). + */ +val SPO2_CANDIDATE_IN_BAND: IntRange = 70..100 + +/** + * One in-band `@82` SpO2 percentage as it travels through a decoded batch, before it is banked as a + * [Spo2PctSampleEntity]. Twin of the Swift `Spo2PctSample`. + * + * LOSSLESS BY CONSTRUCTION: [pct] is the raw in-band byte, verbatim. No run grouping, no ramp trimming, + * no smoothing, no dedupe beyond the (deviceId, ts) primary key — see [Spo2PctSampleEntity] for why. + */ +data class Spo2PctRow( + /** Wall-clock unix seconds — one per strap-second that carried a reading. */ + val ts: Long, + /** The raw in-band `@82` byte, 70..100. */ + val pct: Int, +) + /** * Storage codec for the 5/MG v18 auxiliary-field stream (`v18AuxSample.fields`). * diff --git a/android/app/src/main/java/com/noop/data/WhoopDao.kt b/android/app/src/main/java/com/noop/data/WhoopDao.kt index 3b3ea1109c..d1d42ecf19 100644 --- a/android/app/src/main/java/com/noop/data/WhoopDao.kt +++ b/android/app/src/main/java/com/noop/data/WhoopDao.kt @@ -99,6 +99,18 @@ interface WhoopDao : DeviceRegistryDao { @Insert(onConflict = OnConflictStrategy.IGNORE) suspend fun insertV18Aux(rows: List): List + /** + * The durable in-band `@82` SpO2 percentages (v34 / MIGRATION_25_26), forked out of the SAME v18 aux + * records `insertV18Aux` banks. IGNORE keeps the FIRST-seen value for a second, matching every other + * per-second stream's dedupe rule and making a re-offload of the same window idempotent. Swift twin: + * the `ON CONFLICT(deviceId, ts) DO NOTHING` insert in `StreamStore.insert`. + * + * There is deliberately NO prune query beside this one: unlike `v18AuxSample` this table is never + * swept, which is the entire reason it exists. + */ + @Insert(onConflict = OnConflictStrategy.IGNORE) + suspend fun insertSpo2Pct(rows: List): List + /** * Bound the v18 aux table to the newest [keep] rows for [deviceId] (rolling retention, v31). Same * shape as [pruneRawImu] — this is instrumentation nothing reads yet, so it is capped rather than @@ -320,6 +332,20 @@ interface WhoopDao : DeviceRegistryDao { suspend fun v18AuxSamples(deviceId: String, from: Long, to: Long, limit: Int): List + /** + * The durable in-band `@82` SpO2 percentages in [from, to] (ascending). Unlike [v18AuxSamples] this + * reaches back over the WHOLE recorded history — `spo2PctSample` is never pruned, while the aux table + * it is forked from is capped at ~7 days. Empty on a WHOOP 4.0 (no v18 aux stream) and for any window + * offloaded before MIGRATION_25_26. Values are RAW in-band bytes; run grouping and acquisition-ramp + * trimming are the CALLER's policy. Swift twin: `WhoopStore.spo2PctSamples`. + */ + @Query( + "SELECT * FROM spo2PctSample WHERE deviceId = :deviceId AND ts >= :from AND ts <= :to " + + "ORDER BY ts ASC LIMIT :limit" + ) + suspend fun spo2PctSamples(deviceId: String, from: Long, to: Long, limit: Int): + List + /** Aggregate HR over a window (one indexed (deviceId,ts) range scan — no row materialisation, * no [hrSamples] LIMIT truncation). Backs the imported-workout HR fallback (#77). * @@ -876,6 +902,13 @@ interface WhoopDao : DeviceRegistryDao { @Query("DELETE FROM v18AuxSample WHERE ts < :minTs OR ts > :maxTs") suspend fun pruneV18AuxByTs(minTs: Long, maxTs: Long): Int + // The durable SpO2 percentages are keyed by the SAME type-47 `ts` as every stream above and are + // forked from the same records, so a bad-clock strap garbages them identically. They are never + // pruned on age, so an unhealed out-of-bounds row here would persist FOREVER rather than rolling off + // — which makes the heal more important here, not less. Swift twin: TimestampHeal.rawTables. + @Query("DELETE FROM spo2PctSample WHERE ts < :minTs OR ts > :maxTs") + suspend fun pruneSpo2PctByTs(minTs: Long, maxTs: Long): Int + @Query("DELETE FROM event WHERE ts < :minTs OR ts > :maxTs") suspend fun pruneEventByTs(minTs: Long, maxTs: Long): Int diff --git a/android/app/src/main/java/com/noop/data/WhoopDatabase.kt b/android/app/src/main/java/com/noop/data/WhoopDatabase.kt index f51bf44f1e..52739f9ade 100644 --- a/android/app/src/main/java/com/noop/data/WhoopDatabase.kt +++ b/android/app/src/main/java/com/noop/data/WhoopDatabase.kt @@ -51,8 +51,9 @@ import androidx.sqlite.db.SupportSQLiteDatabase PpgWaveformSampleEntity::class, RawImuSampleEntity::class, V18AuxSampleEntity::class, + Spo2PctSampleEntity::class, ], - version = 25, + version = 26, // #775: ON so Room's KSP processor writes the generated schema (every table's exact `CREATE TABLE`, // columns in declaration order with affinity/NOT NULL/default, PK and indices) as JSON. That export // is what lets a plain JVM test — no device, no Robolectric — read Android's REAL schema and compare @@ -691,6 +692,53 @@ abstract class WhoopDatabase : RoomDatabase() { } } + /** + * Retain the WHOOP 5/MG's own computed SpO2 percentage forever instead of losing it after seven + * days. Twin of the Swift GRDB migration `v34-spo2-pct-durable`. + * + * THE PROBLEM THIS FIXES. The 5/MG emits its own SpO2 % on `@82` of the v18 record. That byte has + * been banked since MIGRATION_24_25 — but only inside `v18AuxSample.fields`, a table CAPPED at + * [WhoopRepository.V18_AUX_RETENTION_ROWS] (604,800 rows/device, ~7 days at 1 Hz) because the aux + * blob costs ~30 MB/day. So every reading rolled off after a week and was gone permanently: the + * strap trims its own history the moment NOOP acks the offload, so there is no second copy + * anywhere. Raising the aux cap is the WRONG fix — it buys a year of SpO2 at the price of ~11 GB + * of mostly-unrelated aux bytes. The in-band signal is only ~2,000 samples/week (~100k rows/year), + * so it gets a narrow sibling that is NEVER pruned while the wide aux capture stays capped and + * unchanged. See [Spo2PctSampleEntity] for why this is not a column on `spo2Sample`. + * + * ADDITIVE ONLY: one CREATE TABLE, no rebuild, no row touched, no existing column changed — the + * MIGRATION_9_10 form (a brand-new table). No destructive fallback (see the class doc): a silent + * rebuild would lose already-acked, non-resendable strap history. + * + * `pct` is NOT NULL because a row is written only when the byte carried an in-band measurement; + * absence is encoded as "no row", never as a fabricated 0. That matters here more than usual: `0` + * is a real value ON THE WIRE ("the strap emitted nothing this second"), so a NULLable column + * holding 0 would be indistinguishable from a genuine sentinel. + * + * VERSION SEQUENCING. Room 25 -> 26 pairs with GRDB `v34`, not `v32`/`v33`: GRDB's identifiers and + * Room's integer versions are independent counters that have drifted apart since the fork's + * iOS-only migrations (see `grdbMigrationLineages` in `schema_oracle.json`). Room's version is a + * dense integer sequence and MUST be the next free one; GRDB's is a string and was numbered ahead + * of upstream on purpose. Both are pinned by SchemaOracleTest. + * + * The CREATE TABLE column order matches [Spo2PctSampleEntity]'s field order and the GRDB schema, + * so a migrated schema, a freshly-created one, and a `.noopbak` from iOS all agree. + * + * Exposed as [SPO2_PCT_DURABLE_MIGRATION_SQL] so a plain-JVM unit test can assert its shape + * without an emulator, like the migrations above. + */ + internal val SPO2_PCT_DURABLE_MIGRATION_SQL: List = listOf( + "CREATE TABLE IF NOT EXISTS `spo2PctSample` (`deviceId` TEXT NOT NULL, " + + "`ts` INTEGER NOT NULL, `pct` INTEGER NOT NULL, " + + "PRIMARY KEY(`deviceId`, `ts`))", + ) + + internal val MIGRATION_25_26 = object : Migration(25, 26) { + override fun migrate(db: SupportSQLiteDatabase) { + for (stmt in SPO2_PCT_DURABLE_MIGRATION_SQL) db.execSQL(stmt) + } + } + private fun build(appContext: Context): WhoopDatabase = Room.databaseBuilder(appContext, WhoopDatabase::class.java, DB_NAME) // #1014: replace ONLY the corruption handling of the default open-helper. The @@ -707,7 +755,7 @@ abstract class WhoopDatabase : RoomDatabase() { MIGRATION_10_11, MIGRATION_11_12, MIGRATION_12_13, MIGRATION_13_14, MIGRATION_14_15, MIGRATION_15_16, MIGRATION_16_17, MIGRATION_17_18, MIGRATION_18_19, MIGRATION_19_20, MIGRATION_20_21, MIGRATION_21_22, - MIGRATION_22_23, MIGRATION_23_24, MIGRATION_24_25, + MIGRATION_22_23, MIGRATION_23_24, MIGRATION_24_25, MIGRATION_25_26, ) // #1037: a FRESH install builds the schema straight at the current version and runs NO // migrations, so the MIGRATION_7_8 "my-whoop" registry seed never fires and the WHOOP, diff --git a/android/app/src/main/java/com/noop/data/WhoopRepository.kt b/android/app/src/main/java/com/noop/data/WhoopRepository.kt index 1ae11814b4..6b0f09e011 100644 --- a/android/app/src/main/java/com/noop/data/WhoopRepository.kt +++ b/android/app/src/main/java/com/noop/data/WhoopRepository.kt @@ -492,6 +492,44 @@ class WhoopRepository(private val dao: WhoopDao) { } } + // DURABLE SpO2 (v34 / MIGRATION_25_26) — the in-band `@82` percentages, forked out of the SAME + // v18 aux samples banked just above into a narrow table that is NEVER pruned. The aux blob is + // capped at ~7 days (the sweep above); this is the copy that survives, and without it every + // reading older than a week is destroyed with no second copy anywhere (the strap trims its own + // history as soon as an offload is acked). + // + // READ OFF THE DECODED ROW, NOT THE BLOB. `it.auxByte82` is the slot value; the alternative — + // indexing byte 23 of the packed `fields` blob, which is what the cloud reader does — is correct + // ONLY while every slot ahead of [V18AuxSlot.AUX_BYTE_82] is present, because [V18AuxCodec] is a + // presence-bitmap format that omits absent slots from the body entirely. With all slots present + // the arithmetic happens to land on 23 (5-byte header + 4+1+1+1+1+2+1+1+2+2+2 = 18 body bytes + // ahead of it), which is why `fields[23]` yields physiologically valid values in production — but + // a single absent earlier slot silently shifts it onto a neighbouring byte and the read becomes + // garbage that STILL looks like a plausible percentage. `V18AuxCodecOffsetTest` pins both halves. + // Here the decoded row is already in hand, so the fragile path is not merely avoided, it is + // unreachable. + // + // Separate loop rather than a branch inside the one above, so the empty-blob skip there can never + // silently drop an SpO2 reading. (In practice a present `auxByte82` guarantees a non-empty blob, + // but coupling the two would make that a load-bearing invariant nobody stated.) + // + // Gate: [SPO2_CANDIDATE_IN_BAND] only. `@82` is multiplexed — bit-7 sentinels (0x80/0x88/0x90/ + // 0xA0/0xA8), sub-70 diagnostic codes, and 0-for-not-emitted share the byte with real percentages, + // and 101..127 is an empty band that keeps the two separable. Nothing else is applied: no run + // grouping, no acquisition-ramp trim, no smoothing. Those are read-time policies (they have + // already changed once on the cloud reader) and baking one in here would destroy the evidence + // needed to change it again. Swift twin: the `spo2PctSample` insert in `StreamStore.insert`. + if (streams.v18Aux.isNotEmpty()) { + val spo2PctRows = streams.v18Aux.mapNotNull { + val v = it.auxByte82 ?: return@mapNotNull null + if (v.toInt() !in SPO2_CANDIDATE_IN_BAND) null + else Spo2PctSampleEntity(deviceId, it.ts, v.toInt()) + } + // IGNORE keeps the FIRST value banked for a second, matching every other per-second stream's + // dedupe rule and making a re-offload of the same window idempotent. + if (spo2PctRows.isNotEmpty()) dao.insertSpo2Pct(spo2PctRows) + } + // OnConflictStrategy.IGNORE returns -1 for skipped (already-present) rows; count the inserts. return InsertCounts( hr = hrIds.countInserted() + ppgHrIds.countInserted(), @@ -669,6 +707,7 @@ class WhoopRepository(private val dao: WhoopDao) { deleted += dao.prunePpgWaveformByTs(minTs, maxTs) deleted += dao.pruneRawImuByTs(minTs, maxTs) deleted += dao.pruneV18AuxByTs(minTs, maxTs) + deleted += dao.pruneSpo2PctByTs(minTs, maxTs) deleted += dao.pruneEventByTs(minTs, maxTs) deleted += dao.pruneBatteryByTs(minTs, maxTs) // (b) computed daily metrics (by day key) + sleep sessions (by startTs). The prune queries apply @@ -862,6 +901,26 @@ class WhoopRepository(private val dao: WhoopDao) { dao.v18AuxSamples(deviceId, from, to, limit) .map { V18AuxCodec.unpack(it.fields, it.ts) } + /** + * Durable in-band `@82` SpO2 percentages (v34 / MIGRATION_25_26) in [from, to] for one device, + * ascending by ts. + * + * Unlike [v18AuxSamples], this reaches back over the WHOLE recorded history: `spo2PctSample` is never + * pruned, while the aux table it is forked from is capped at ~7 days. Empty for a WHOOP 4.0 (no v18 + * aux stream) and for any window offloaded before MIGRATION_25_26 — the older aux rows still on the + * device DO carry the byte, but only for whatever slice of the last seven days survives, and this + * reader deliberately does NOT fall back to them: a series that silently changed source partway + * through would be unusable as evidence. Backfilling those survivors is a one-shot migration's job, + * not a read path's. + * + * Values are RAW in-band bytes. Run grouping and acquisition-ramp trimming are the CALLER's policy — + * see [com.noop.analytics.AnalyticsEngine.nightlySpo2Pct] for the nightly one. Kotlin twin of the + * Swift `WhoopStore.spo2PctSamples(deviceId:from:to:)`. + */ + suspend fun spo2PctSamples(deviceId: String, from: Long, to: Long, limit: Int = DEFAULT_LIMIT): + List = + dao.spo2PctSamples(deviceId, from, to, limit).map { Spo2PctRow(it.ts, it.pct) } + /** #423: persist a batch of decoded 5/MG raw-IMU offload buffers (one row per strap-second, packed i16 * BLOB), then bound the table to the newest [RAW_IMU_RETENTION_ROWS] for the device. Comes from the * deep-buffer capture seam, not the normal stream path, so it inserts directly (idempotent by ts). */ diff --git a/android/app/src/main/java/com/noop/ingest/HealthConnectWriter.kt b/android/app/src/main/java/com/noop/ingest/HealthConnectWriter.kt index 72382afb95..e5d9ac5b82 100644 --- a/android/app/src/main/java/com/noop/ingest/HealthConnectWriter.kt +++ b/android/app/src/main/java/com/noop/ingest/HealthConnectWriter.kt @@ -74,6 +74,34 @@ object HealthConnectWriter { /** How far back to write. Recomputation only ever touches recent nights; 60 days is generous. */ private const val WINDOW_DAYS = 60L + /** + * Whether a strap-derived nightly SpO2 is written back to Health Connect as an + * [OxygenSaturationRecord]. Twin of the Swift `HealthKitBridge.exportsSpo2ToHealthKit`. + * + * FALSE, deliberately, and it is not an oversight to be tidied away. + * + * This export loop reads only the COMPUTED NOOP device rows — imported rows never pass through it — + * so before v34 populated `spo2Pct` for WHOOP nights this branch never fired even once. Turning the + * field on therefore does not "resume" an export; it STARTS one, and what it would start writing is a + * number nobody has checked against a reference oximeter. + * + * Why that bar is higher here than for the in-app card. A value in the app is read in context, next + * to its own provenance, by the person who built it. A Health Connect record is different in kind: it + * lands in the system health record under Android's own Blood Oxygen heading, sits alongside + * clinically-sourced readings with nothing distinguishing it, and is readable by every other app the + * user has authorised. Deleting it later is fiddly and does not recall what already propagated. + * + * The evidence genuinely does not reach that bar yet. The decode is well supported — distribution, + * run structure, sleep gating, night-to-night stability — but ALL of it says "this byte is a real + * oxygen measurement", none of it says "97 means 97". Upstream declines to promote the same metric + * for the same reason: two straps were checked against the WHOOP app and moved in OPPOSITE directions. + * + * Flip this to `true` (in lockstep with the Swift constant, or the platforms disagree about what + * leaves the device) once paired reference-oximeter nights exist and the bias is characterised — and, + * if the bias is a fixed offset, correct for it before writing rather than here. + */ + const val EXPORTS_SPO2_TO_HEALTH_CONNECT = false + private val WRITE_RECORDS: List> = listOf( RestingHeartRateRecord::class, HeartRateVariabilityRmssdRecord::class, @@ -139,7 +167,8 @@ object HealthConnectWriter { metadata = meta("hrv", d.day, version), )) } - d.spo2Pct?.let { + // DELIBERATELY NOT EXPORTED while [EXPORTS_SPO2_TO_HEALTH_CONNECT] is false — see the constant. + d.spo2Pct?.takeIf { EXPORTS_SPO2_TO_HEALTH_CONNECT }?.let { records.add(OxygenSaturationRecord( time = instant, zoneOffset = offset, percentage = Percentage(it), metadata = meta("spo2", d.day, version), diff --git a/android/app/src/main/java/com/noop/protocol/HistoricalStreams.kt b/android/app/src/main/java/com/noop/protocol/HistoricalStreams.kt index 30f7ca7bb9..09279aaab4 100644 --- a/android/app/src/main/java/com/noop/protocol/HistoricalStreams.kt +++ b/android/app/src/main/java/com/noop/protocol/HistoricalStreams.kt @@ -293,6 +293,40 @@ fun decodeHistorical(frame: ByteArray, family: DeviceFamily = DeviceFamily.WHOOP return out } +/** + * The WHOOP 5/MG type-47 `hist_version` values THIS platform has a real field map for. Every other + * version reaches no decoder at all: [decodeWhoop5Historical] returns null and nothing but the raw bytes + * of the record can preserve it. + * + * This is the single source of truth for "does NOOP-Android understand this layout": the version gates in + * [decodeWhoop5Historical] and the v26 branch of [extractHistoricalStreams] read nothing else, and + * [rejectedHistoricalRecords] uses it to decide which records must be archived raw. Keeping one set means + * the two can never disagree — the failure mode this replaced was a record from an unmapped layout that + * happened to decode a plausible `unix` and `gravity_x`/`heart_rate` slipping past the archive filter, so + * its bytes were kept nowhere at all and the strap freed them on the very next trim ack. + * + * ⚠️ DELIBERATELY NARROWER THAN THE SWIFT `mappedWhoop5HistoricalVersions`, which is `[18, 20, 21, 26]`. + * Swift's `decodeWhoop5Historical` dispatches v20 (raw optical block) and v21 (raw 6-axis IMU) through + * `decodeWhoop5HistoricalV2021`; the Kotlin historical path has no such branch — `Whoop5RawOptical` and + * `Whoop5RawImu` exist here but are wired to the LIVE deep-buffer route in `WhoopBleClient`, not to the + * type-47 historical dispatch. Claiming 20/21 here would assert a field map Android does not have, which + * is precisely the "it holds by accident" class of bug this constant was introduced to close. The archive + * OUTCOME is unchanged either way today: an unmapped 5/MG version already decodes to null on Android and + * so was already archived by the decode-outcome route. Add a version here only when a decoder for it + * lands on this side. Pinned by `UnmappedHistoricalLayoutTest`. + */ +val MAPPED_WHOOP5_HISTORICAL_VERSIONS: Set = setOf(18, 26) + +/** + * True when [frame] is a WHOOP 5/MG type-47 record whose layout version has NO field map on this + * platform — the records that are never re-derivable from anything NOOP stores. Non-type-47 and + * too-short frames are false. Twin of the Swift `isUnmappedWhoop5HistoricalRecord`. + */ +fun isUnmappedWhoop5HistoricalRecord(frame: ByteArray): Boolean { + if (frame.size <= 9 || (frame[8].toInt() and 0xFF) != PacketType.HISTORICAL_DATA.rawValue) return false + return (frame[9].toInt() and 0xFF) !in MAPPED_WHOOP5_HISTORICAL_VERSIONS +} + /** * WHOOP 5.0/MG type-47 "v18" historical decode. The puffin envelope is longer, so the record starts at * byte 8 (type@8, version@9) and fields sit at their WHOOP5-ABSOLUTE offsets — NOT the WHOOP4 V24 layout @@ -307,6 +341,11 @@ fun decodeHistorical(frame: ByteArray, family: DeviceFamily = DeviceFamily.WHOOP private fun decodeWhoop5Historical(frame: ByteArray): Map? { if (frame.histU8(8) != PacketType.HISTORICAL_DATA.rawValue) return null val version = frame.histU8(9) ?: return null + // One gate, one list: 18 is the only version this function maps, and v26 (handled by + // [decodeWhoop5HistoricalV26] from [extractHistoricalStreams]) is the only other one this platform + // maps at all — so [MAPPED_WHOOP5_HISTORICAL_VERSIONS] must contain exactly {18, 26}. That set is + // what decides whether a record gets archived raw, so the two cannot be allowed to drift; + // `UnmappedHistoricalLayoutTest` pins the lockstep against the decoder's actual behaviour. if (version != 18) return null val out = LinkedHashMap() @@ -477,10 +516,12 @@ private fun decodeWhoop5HistoricalV26(frame: ByteArray): V26Record? { } /** - * The HISTORICAL_DATA (type-47) record frames in [rawFrames] that genuinely FAIL to decode — a CRC - * failure, or an unmapped firmware layout whose v24 plausibility gate (see [decodeHistorical]) also - * rejects it. These are exactly the record frames [extractHistoricalStreams] silently drops: their - * biometric payload would otherwise be lost forever once the strap trims the acked history. + * The HISTORICAL_DATA (type-47) record frames in [rawFrames] that NOOP cannot turn into rows — a CRC + * failure, an unmapped firmware layout (5/MG: any `hist_version` outside + * [MAPPED_WHOOP5_HISTORICAL_VERSIONS]), or a mapped layout whose envelope parsed but whose v24 + * plausibility gate (see [decodeHistorical]) rejected it. These are the record frames + * [extractHistoricalStreams] silently drops: their biometric payload would otherwise be lost forever once + * the strap trims the acked history. * * EXCLUDED (decode to zero rows BY DESIGN, never "lost" data — must NOT be counted): * - CONSOLE_LOGS (type-50) frames — the strap's own diagnostics text channel. On WHOOP 4.0 the @@ -507,6 +548,24 @@ fun rejectedHistoricalRecords( if (t != PacketType.HISTORICAL_DATA.rawValue) return@filter false // type-50 console / metadata / etc. // WHOOP 5/MG v26 = raw PPG block, deliberately not stored — known-skipped, not lost data. if (family == DeviceFamily.WHOOP5 && frame.histU8(9) == 26) return@filter false + // UNMAPPED LAYOUT (5/MG) — archive UNCONDITIONALLY, whatever it decoded. + // + // The decode-outcome test below is the wrong QUESTION for a layout NOOP has no field map for. + // Today it happens to give the right ANSWER, because [decodeWhoop5Historical] returns null for + // every version outside [MAPPED_WHOOP5_HISTORICAL_VERSIONS] — but that is an accident of the + // decoder's current shape, not a property of the format. One static schema field for type 47, or + // one partially-mapped new version, and a record that yielded a plausible `unix` plus a + // `heart_rate`/`gravity_x` would pass the screen and be kept NOWHERE at all, its bytes freed by + // the very next trim ack. That is exactly the shape a novel record (an on-demand MG ECG capture, + // a new firmware's rollup) can take. Deciding on the VERSION makes it structural. + // + // This cannot flood the archive with records we already understand: the mapped versions never + // reach here. Retention for the records that DO ([RawHistoryArchive.evictLines]) evicts + // entirely-zero-payload frames first, so a firmware that banks empty placeholder records at 1 Hz + // cannot push out the one informative frame either. Twin of the Swift `rejectedHistoricalRecords` + // layout clause. WHOOP 4 is untouched: its unmapped versions go through the v24 fallback below, + // which keeps a record only when it validates physiologically. + if (family == DeviceFamily.WHOOP5 && isUnmappedWhoop5HistoricalRecord(frame)) return@filter true // A type-47 record that [decodeHistorical] cannot turn into usable biometrics — CRC failure or // an unmapped layout the v24-fallback plausibility gate rejected. This is precisely what // [extractHistoricalStreams] drops (`decodeHistorical(...) ?: continue`), so the rejected set diff --git a/android/app/src/test/java/com/noop/analytics/RegistryDayOwnerSourceTest.kt b/android/app/src/test/java/com/noop/analytics/RegistryDayOwnerSourceTest.kt index 661dff1727..76f07a7f8d 100644 --- a/android/app/src/test/java/com/noop/analytics/RegistryDayOwnerSourceTest.kt +++ b/android/app/src/test/java/com/noop/analytics/RegistryDayOwnerSourceTest.kt @@ -62,6 +62,7 @@ class RegistryDayOwnerSourceTest { override suspend fun deletePpgWaveformFor(deviceId: String) {} override suspend fun deleteRawImuFor(deviceId: String) {} override suspend fun deleteV18AuxFor(deviceId: String) {} + override suspend fun deleteSpo2PctFor(deviceId: String) {} override suspend fun deleteEventsFor(deviceId: String) {} override suspend fun deleteBatteryFor(deviceId: String) {} override suspend fun deleteDailyMetricsFor(deviceId: String) {} @@ -92,6 +93,7 @@ class RegistryDayOwnerSourceTest { override suspend fun reKeyPpgWaveform(from: String, to: String) {} override suspend fun reKeyRawImu(from: String, to: String) {} override suspend fun reKeyV18Aux(from: String, to: String) {} + override suspend fun reKeySpo2Pct(from: String, to: String) {} override suspend fun reKeyEvents(from: String, to: String) {} override suspend fun reKeyBattery(from: String, to: String) {} override suspend fun reKeyDailyMetrics(from: String, to: String) {} diff --git a/android/app/src/test/java/com/noop/analytics/Spo2PctNightlyTest.kt b/android/app/src/test/java/com/noop/analytics/Spo2PctNightlyTest.kt new file mode 100644 index 0000000000..dec9000cec --- /dev/null +++ b/android/app/src/test/java/com/noop/analytics/Spo2PctNightlyTest.kt @@ -0,0 +1,203 @@ +package com.noop.analytics + +import com.noop.data.Spo2PctRow +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNotNull +import org.junit.Assert.assertNull +import org.junit.Test + +/** + * v34 / MIGRATION_25_26 — the nightly SpO2 percentage banked on `DailyMetric.spo2Pct` for a 5/MG night. + * + * The strap samples SpO2 in RUNS: ~30 one-second samples roughly once per ~1,200 s while asleep. The + * first samples of each run are the optical front end settling and they read LOW. That makes a naive mean + * over the night's in-band samples biased downward — and biased *more* on nights made of many short runs, + * which is the opposite of when you would think to check. These pin the two corrections (per-run ramp + * trim, then median) and, more usefully, pin the size of the bias they remove. + * + * Byte-parity twin of the Swift `Spo2PctNightlyTests`: same fixtures, same expected values. + */ +class Spo2PctNightlyTest { + + private fun session(start: Long, durSec: Long) = DetectedSleep( + start = start, end = start + durSec, efficiency = 0.9, + stages = emptyList(), restingHR = 50, avgHRV = 60.0, + ) + + /** + * A measurement run: [count] consecutive one-second samples starting at [ts], the first [rampCount] + * of them at the depressed [rampValue]. + */ + private fun run( + ts: Long, + count: Int, + settled: Int, + rampCount: Int = 0, + rampValue: Int = 88, + ): List = (0 until count).map { Spo2PctRow(ts + it, if (it < rampCount) rampValue else settled) } + + // MARK: - The bias this exists to remove + + /** + * THE MOTIVATING CASE. One 30-sample run whose first 5 are the acquisition ramp at 88 and whose + * settled 25 are 97. A naive mean lands at 95.5 — a full 1.5 points below what the strap actually + * settled on, entirely from a one-sided artifact. The ramp trim removes exactly those 5 samples and + * the median returns the settled value. + */ + @Test fun theAcquisitionRampBiasesANaiveMeanAndTheTrimRemovesIt() { + val samples = run(1_000, count = 30, settled = 97, rampCount = 5) + val naiveMean = samples.sumOf { it.pct }.toDouble() / samples.size + assertEquals("the bias is real and this is its size", 95.5, naiveMean, 0.001) + + val r = AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 600)), samples) + assertEquals(97.0, r!!.first, 0.001) + assertEquals("the five ramp samples are dropped, the settled 25 are kept", 25, r.second) + } + + // MARK: - Run segmentation + + /** + * Runs are separated by a gap far larger than the 1 s spacing inside one, so EACH run gets its own + * ramp trim. Trimming only the night's first five samples would leave every later run's ramp in. + */ + @Test fun eachRunIsTrimmedSeparately() { + val samples = run(1_000, count = 20, settled = 97, rampCount = 5) + + run(3_000, count = 20, settled = 97, rampCount = 5) + + run(5_000, count = 20, settled = 97, rampCount = 5) + val r = AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 5_000)), samples) + assertEquals("3 runs x (20 - 5) survivors", 45, r!!.second) + assertEquals(97.0, r.first, 0.001) + } + + /** + * A dropped second or two inside a run must NOT split it — the gap threshold sits in the gulf between + * 1 s (within a run) and ~1,200 s (between runs), so a small hole is tolerated. + */ + @Test fun aSmallHoleDoesNotSplitARun() { + // 10 samples, then a 20 s hole, then 10 more: still ONE run, so ONE ramp trim. + val samples = run(1_000, count = 10, settled = 97, rampCount = 5) + run(1_030, count = 10, settled = 97) + val r = AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 600)), samples) + assertEquals("one run: 20 samples less one 5-sample ramp", 15, r!!.second) + } + + @Test fun theRunGapThresholdIsTheDocumentedOne() { + assertEquals(60L, AnalyticsEngine.SPO2_RUN_GAP_SECONDS) + assertEquals(5, AnalyticsEngine.SPO2_RAMP_SAMPLES) + } + + // MARK: - The half-run cap + + /** + * The trim never eats more than half a run. A short run must contribute its settled tail rather than + * vanishing — otherwise the night would be silently reweighted toward whichever runs happened to be + * long, which is a selection effect nobody would see in the output. + */ + @Test fun aShortRunKeepsItsTailInsteadOfVanishing() { + // 4 samples: drop min(5, 2) = 2, keep 2. + val r = AnalyticsEngine.nightlySpo2Pct( + listOf(session(900, 600)), + listOf(Spo2PctRow(1_000, 88), Spo2PctRow(1_001, 90), Spo2PctRow(1_002, 96), Spo2PctRow(1_003, 96)), + ) + assertEquals(2, r!!.second) + assertEquals(96.0, r.first, 0.001) + } + + /** A single-sample run keeps its one sample (drop = min(5, 0) = 0) rather than being erased. */ + @Test fun aSingleSampleRunSurvives() { + val r = AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 600)), listOf(Spo2PctRow(1_000, 95))) + assertEquals(1, r!!.second) + assertEquals(95.0, r.first, 0.001) + } + + // MARK: - The statistic + + /** + * The median is the statistic, matching what the cloud reader reports for the same rows — a phone that + * disagreed with the server on identical data would poison every future cross-check. + */ + @Test fun medianNotMeanOverTheSurvivors() { + // Survivors after the ramp trim: 90, 97, 97, 97, 97 — median 97, mean 95.6. + val pcts = listOf(90, 90, 90, 90, 90, 90, 97, 97, 97, 97) + val samples = pcts.mapIndexed { i, v -> Spo2PctRow(1_000L + i, v) } + val r = AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 600)), samples) + assertEquals(5, r!!.second) + assertEquals("a single low survivor must not drag the answer the way a mean would", + 97.0, r.first, 0.001) + } + + /** An even survivor count averages the two middle values, so the result can land on a half. */ + @Test fun evenSurvivorCountAveragesTheTwoMiddleValues() { + // A 4-sample run: the half-run cap drops 2, leaving survivors 94 and 97 -> (94 + 97) / 2 = 95.5. + val samples = listOf(88, 88, 94, 97).mapIndexed { i, v -> Spo2PctRow(1_000L + i, v) } + val r = AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 600)), samples) + assertNotNull(r) + assertEquals(2, r!!.second) + assertEquals(95.5, r.first, 0.001) + } + + /** + * The half-run cap governs whenever a run is shorter than twice the ramp, so the survivor count is + * `count - min(5, count/2)` and not simply `count - 5`. Pinned across the crossover at 10. + */ + @Test fun survivorCountFollowsTheCappedTrim() { + for ((count, want) in listOf(1 to 1, 2 to 1, 4 to 2, 6 to 3, 9 to 5, 10 to 5, 11 to 6, 30 to 25)) { + val r = AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 600)), run(1_000, count, settled = 97)) + assertEquals("a $count-sample run must keep $want", want, r!!.second) + } + } + + // MARK: - Session gating and absence + + /** Only in-bed samples count — a daytime reading is not part of the night's saturation. */ + @Test fun samplesOutsideEverySessionAreExcluded() { + val samples = run(1_000, count = 10, settled = 97) + run(9_000, count = 10, settled = 80) + val r = AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 600)), samples) + assertEquals("only the in-bed run contributes", 5, r!!.second) + assertEquals(97.0, r.first, 0.001) + } + + /** + * Gap size is the ONLY run signal — a session edge is not a second one. A brief out-of-bed moment + * mid-run leaves the run intact and singly-trimmed, because the acquisition ramp is a property of the + * SENSOR restarting and the sensor does not restart because the sleep detector drew a line. + */ + @Test fun aBriefSessionEdgeDoesNotSplitARun() { + val samples = run(1_000, count = 20, settled = 97) + // Sessions carve out [1000..1004] and [1010..1019] — 15 in-bed samples with only a 6 s hole, + // which is under the threshold, so this stays ONE run and is trimmed ONCE. + val r = AnalyticsEngine.nightlySpo2Pct(listOf(session(1_000, 4), session(1_010, 9)), samples) + assertEquals("one run of 15, less one 5-sample ramp", 10, r!!.second) + } + + /** + * When the excluded stretch IS longer than the gap threshold, the two halves are genuinely separate + * runs and each pays its own ramp trim. + */ + @Test fun aWideSessionGapDoesSplitARun() { + val samples = run(1_000, count = 10, settled = 97) + run(1_200, count = 10, settled = 97) + // 200 s apart — comfortably over the 60 s threshold, so two runs: (10-5) + (10-5). + assertEquals(10, AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 600)), samples)!!.second) + } + + /** Absence stays absence — nothing here may invent a percentage. */ + @Test fun nullWhenThereIsNothingToAverage() { + assertNull(AnalyticsEngine.nightlySpo2Pct(emptyList(), listOf(Spo2PctRow(1_000, 97)))) + assertNull(AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 600)), emptyList())) + // In-band samples exist, but none of them fall in a session. + assertNull(AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 100)), listOf(Spo2PctRow(9_999, 97)))) + } + + /** + * Unsorted input must not change the answer — run boundaries are a property of the timestamps, not of + * the order the store happened to hand them over in. + */ + @Test fun inputOrderDoesNotMatter() { + val ordered = run(1_000, count = 20, settled = 97, rampCount = 5) + val shuffled = ordered.reversed() + assertEquals( + AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 600)), ordered)!!.second, + AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 600)), shuffled)!!.second, + ) + assertEquals(97.0, AnalyticsEngine.nightlySpo2Pct(listOf(session(900, 600)), shuffled)!!.first, 0.001) + } +} diff --git a/android/app/src/test/java/com/noop/ble/RawHistoryArchiveEvictionTest.kt b/android/app/src/test/java/com/noop/ble/RawHistoryArchiveEvictionTest.kt index 4afdc46daa..a54988f13a 100644 --- a/android/app/src/test/java/com/noop/ble/RawHistoryArchiveEvictionTest.kt +++ b/android/app/src/test/java/com/noop/ble/RawHistoryArchiveEvictionTest.kt @@ -1,6 +1,7 @@ package com.noop.ble import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse import org.junit.Assert.assertTrue import org.junit.Test @@ -53,6 +54,119 @@ class RawHistoryArchiveEvictionTest { assertEquals(lines, RawHistoryArchive.evictLines(lines, maxBytes = 1_000_000, floor = 2)) } + // MARK: - zero-payload frames lose retention priority + + /** + * A synthetic WHOOP 5/MG type-47 record: a non-zero 21-byte header (type @8 = 47, hist_version @9), + * then [payloadBytes] of payload, then a 4-byte CRC trailer. + * + * [payloadByte] = 0 reproduces the measured `enable_raw_data_w_ecg` placeholder — the strap banks one + * such record per second and every byte from offset 21 to the CRC is zero. [marker] writes a single + * distinguishing byte into the payload, which is also what makes a frame informative. + */ + private fun whoop5Frame( + version: Int, + payloadByte: Int, + payloadBytes: Int = 64, + marker: Int? = null, + ): ByteArray { + val f = ByteArray(21) { 0x11 } // non-zero header (seq/ts/const bytes) + f[0] = 0xAA.toByte(); f[8] = 47; f[9] = version.toByte() + val payload = ByteArray(payloadBytes) { payloadByte.toByte() } + if (marker != null) payload[0] = marker.toByte() + return f + payload + byteArrayOf(0xDE.toByte(), 0xAD.toByte(), 0xBE.toByte(), 0xEF.toByte()) + } + + /** One archived JSONL line for [frame], in the exact shape `append` writes. */ + private fun archiveLine(frame: ByteArray, family: String = "whoop5"): String { + val hex = frame.joinToString("") { "%02x".format(it) } + return """{"capturedAtMs":1,"trim":1,"family":"$family","frameHex":"$hex"}""" + "\n" + } + + /** The strict test: entirely-zero payload region -> yes; ONE non-zero byte anywhere -> no. */ + @Test fun zeroPayloadDetectionIsStrictAndHeaderAware() { + val empty = whoop5Frame(22, payloadByte = 0) + assertTrue( + "the header and CRC bytes are non-zero but the PAYLOAD region is all zero", + RawHistoryArchive.hasZeroPayload(empty, com.noop.protocol.DeviceFamily.WHOOP5), + ) + // A single populated byte in an otherwise-empty buffer is exactly what this archive is for. + val oneByte = empty.copyOf().also { it[21 + 40] = 1 } + assertFalse( + "'mostly zero' must NOT count as empty — one non-zero byte makes it informative", + RawHistoryArchive.hasZeroPayload(oneByte, com.noop.protocol.DeviceFamily.WHOOP5), + ) + // The last payload byte, immediately before the CRC trailer, is inside the region. + val lastByte = empty.copyOf().also { it[it.size - 5] = 1 } + assertFalse(RawHistoryArchive.hasZeroPayload(lastByte, com.noop.protocol.DeviceFamily.WHOOP5)) + // A payload region too short to judge is never called empty. + assertFalse(RawHistoryArchive.hasZeroPayload( + whoop5Frame(22, payloadByte = 0, payloadBytes = 8), com.noop.protocol.DeviceFamily.WHOOP5)) + assertFalse(RawHistoryArchive.hasZeroPayload( + byteArrayOf(0xAA.toByte(), 1), com.noop.protocol.DeviceFamily.WHOOP5)) + // The header length is family-aware: the puffin envelope is 4 bytes longer. + assertEquals(21, RawHistoryArchive.payloadStart(com.noop.protocol.DeviceFamily.WHOOP5)) + assertEquals(17, RawHistoryArchive.payloadStart(com.noop.protocol.DeviceFamily.WHOOP4)) + } + + /** + * THE DEFECT. With `enable_raw_data_w_ecg` armed the strap banks one all-zero record per second, so a + * purely age-ordered archive evicts a real contact-window record within the half hour. An informative + * frame must outrank an all-zero one of the SAME layout version, even when it is the OLDEST line in + * the file and the empties are the newest — so no floor can be what saves it. + */ + @Test fun evictLinesEvictsZeroPayloadBeforeInformative() { + val informative = archiveLine(whoop5Frame(22, payloadByte = 0, marker = 0xE7)) + val lines = ArrayList() + lines.add(informative) + repeat(200) { lines.add(archiveLine(whoop5Frame(22, payloadByte = 0))) } + + val kept = RawHistoryArchive.evictLines(lines, maxBytes = 4_096, floor = 2, zeroFloor = 2) + assertTrue(kept.sumOf { it.toByteArray(Charsets.UTF_8).size } <= 4_096) + assertTrue( + "the one informative line must survive a flood of all-zero-payload lines", + kept.contains(informative), + ) + assertTrue("the zero-payload surplus must actually have been evicted", kept.size < 200) + } + + /** + * A version whose records are ALL empty still keeps [zeroFloor] dated samples — enough to prove the + * artefact exists without letting it own the archive. + */ + @Test fun allZeroVersionKeepsItsSmallFloor() { + val lines = ArrayList() + repeat(200) { lines.add(archiveLine(whoop5Frame(22, payloadByte = 0))) } + repeat(200) { i -> lines.add(archiveLine(whoop5Frame(18, payloadByte = 0, marker = (i and 0xFF).coerceAtLeast(1)))) } + + val kept = RawHistoryArchive.evictLines(lines, maxBytes = 4_096, floor = 2, zeroFloor = 2) + val keptZeroVersion = kept.count { it.contains("2f16") } // 0x2f = type 47, 0x16 = v22 + assertTrue( + "an all-empty version must keep its small floor, not vanish (kept $keptZeroVersion)", + keptZeroVersion >= 2, + ) + } + + /** The eviction target is a low-water mark below the cap, not the cap itself — see LOW_WATER_DIVISOR. */ + @Test fun evictionLeavesHeadroomSoAOneHzStreamDoesNotRewriteEveryRecord() { + val lines = (0 until 400).map { jsonl(18, filler = "%02x".format(it and 0xFF)) } + val kept = RawHistoryArchive.evictLines(lines, maxBytes = 4_096, floor = 2) + val bytes = kept.sumOf { it.toByteArray(Charsets.UTF_8).size } + assertTrue("must fit the cap", bytes <= 4_096) + assertTrue( + "must evict PAST the cap to 7/8 of it, or every subsequent batch rewrites the whole file", + bytes <= 4_096 - 4_096 / RawHistoryArchive.LOW_WATER_DIVISOR, + ) + } + + /** The shipped floors are the documented ones, and the zero floor is far smaller than the real one. */ + @Test fun shippedFloors() { + assertEquals(64, RawHistoryArchive.PER_VERSION_FLOOR) + assertEquals(8, RawHistoryArchive.ZERO_PAYLOAD_FLOOR) + assertTrue(RawHistoryArchive.ZERO_PAYLOAD_FLOOR < RawHistoryArchive.PER_VERSION_FLOOR) + assertEquals(5L * 1024 * 1024, RawHistoryArchive.REJECTED_ARCHIVE_MAX_BYTES) + } + @Test fun versionByteReadsTheRightIndexPerFamily() { // WHOOP 4 = frame[5]; WHOOP 5 = frame[9] (puffin envelope 4 B longer). val w4 = byteArrayOf(0xAA.toByte(), 1, 0, 0, 47, 19, 0, 0) diff --git a/android/app/src/test/java/com/noop/ble/SourceCoordinatorAdoptionTest.kt b/android/app/src/test/java/com/noop/ble/SourceCoordinatorAdoptionTest.kt index cc23195f99..b946a8daa7 100644 --- a/android/app/src/test/java/com/noop/ble/SourceCoordinatorAdoptionTest.kt +++ b/android/app/src/test/java/com/noop/ble/SourceCoordinatorAdoptionTest.kt @@ -76,6 +76,7 @@ class SourceCoordinatorAdoptionTest { override suspend fun deletePpgWaveformFor(deviceId: String) {} override suspend fun deleteRawImuFor(deviceId: String) {} override suspend fun deleteV18AuxFor(deviceId: String) {} + override suspend fun deleteSpo2PctFor(deviceId: String) {} override suspend fun deleteEventsFor(deviceId: String) {} override suspend fun deleteBatteryFor(deviceId: String) {} override suspend fun deleteDailyMetricsFor(deviceId: String) {} @@ -108,6 +109,7 @@ class SourceCoordinatorAdoptionTest { override suspend fun reKeyPpgWaveform(from: String, to: String) {} override suspend fun reKeyRawImu(from: String, to: String) {} override suspend fun reKeyV18Aux(from: String, to: String) {} + override suspend fun reKeySpo2Pct(from: String, to: String) {} override suspend fun reKeyEvents(from: String, to: String) {} override suspend fun reKeyBattery(from: String, to: String) {} override suspend fun reKeyDailyMetrics(from: String, to: String) {} diff --git a/android/app/src/test/java/com/noop/data/DeviceRegistryTest.kt b/android/app/src/test/java/com/noop/data/DeviceRegistryTest.kt index d1203f8b9a..3210fe57b5 100644 --- a/android/app/src/test/java/com/noop/data/DeviceRegistryTest.kt +++ b/android/app/src/test/java/com/noop/data/DeviceRegistryTest.kt @@ -91,6 +91,7 @@ class DeviceRegistryTest { override suspend fun deletePpgWaveformFor(deviceId: String) { deletedTables += "ppgWaveformSample" to deviceId } override suspend fun deleteRawImuFor(deviceId: String) { deletedTables += "rawImuSample" to deviceId } override suspend fun deleteV18AuxFor(deviceId: String) { deletedTables += "v18AuxSample" to deviceId } + override suspend fun deleteSpo2PctFor(deviceId: String) { deletedTables += "spo2PctSample" to deviceId } override suspend fun deleteEventsFor(deviceId: String) { deletedTables += "event" to deviceId } override suspend fun deleteBatteryFor(deviceId: String) { deletedTables += "battery" to deviceId } override suspend fun deleteDailyMetricsFor(deviceId: String) { deletedTables += "dailyMetric" to deviceId } @@ -127,6 +128,7 @@ class DeviceRegistryTest { override suspend fun reKeyPpgWaveform(from: String, to: String) {} override suspend fun reKeyRawImu(from: String, to: String) {} override suspend fun reKeyV18Aux(from: String, to: String) {} + override suspend fun reKeySpo2Pct(from: String, to: String) {} override suspend fun reKeyEvents(from: String, to: String) {} override suspend fun reKeyBattery(from: String, to: String) {} override suspend fun reKeyDailyMetrics(from: String, to: String) {} @@ -282,6 +284,9 @@ class DeviceRegistryTest { val expectedTables = setOf( "hrSample", "rrInterval", "spo2Sample", "skinTempSample", "respSample", "gravitySample", "stepSample", "ppgHrSample", "ppgWaveformSample", "rawImuSample", "v18AuxSample", + // v34/MIGRATION_25_26: never pruned, so a wipe that missed it would leave YEARS of + // blood-oxygen readings behind after the user asked for them to be gone. + "spo2PctSample", "event", "battery", "dailyMetric", "sleepSession", "journal", "workout", "appleDaily", "metricSeries", "dayOwnership", "scoreInputProvenance", diff --git a/android/app/src/test/java/com/noop/data/Spo2PctDurableMigrationTest.kt b/android/app/src/test/java/com/noop/data/Spo2PctDurableMigrationTest.kt new file mode 100644 index 0000000000..699366a249 --- /dev/null +++ b/android/app/src/test/java/com/noop/data/Spo2PctDurableMigrationTest.kt @@ -0,0 +1,210 @@ +package com.noop.data + +import kotlinx.coroutines.runBlocking +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertTrue +import org.junit.Test +import java.lang.reflect.Proxy + +/** + * MIGRATION_25_26 — the durable `@82` SpO2 percentage table. Twin of the Swift GRDB migration + * `v34-spo2-pct-durable` and its `Spo2PctDurableTests`. + * + * The 5/MG emits its own computed SpO2 % on `@82` of the v18 record. That byte has been banked since + * MIGRATION_24_25 — inside `v18AuxSample.fields`, a table CAPPED at 604,800 rows/device (~7 days at 1 Hz) + * because the aux blob costs ~30 MB/day. So every reading rolled off after a week and was gone + * permanently: the strap trims its own history the moment an offload is acked, leaving no second copy + * anywhere. `spo2PctSample` is the narrow, never-pruned sibling that fixes that. + * + * The SQL here is hand-written, which is exactly where a mistake is possible, so this test guards its + * shape; the write path is exercised through a Proxy DAO (the DeepCaptureMigrationTest pattern — there is + * no SQLite driver on this unit-test classpath) and the DELETE/upsert semantics are proved end-to-end on + * the Swift twin, which CI runs. + */ +class Spo2PctDurableMigrationTest { + + @Test + fun migrationCreatesTheDurableTableAndTouchesNothingElse() { + val sql = WhoopDatabase.SPO2_PCT_DURABLE_MIGRATION_SQL + assertEquals("one statement: a brand-new table, nothing altered", 1, sql.size) + + // The column order must match Spo2PctSampleEntity's field order (deviceId, ts, pct) and the GRDB + // schema, or a fresh install, a migrated install and an iOS `.noopbak` disagree. + assertEquals( + "CREATE TABLE IF NOT EXISTS `spo2PctSample` (`deviceId` TEXT NOT NULL, " + + "`ts` INTEGER NOT NULL, `pct` INTEGER NOT NULL, " + + "PRIMARY KEY(`deviceId`, `ts`))", + sql[0], + ) + + // Nothing in this migration may touch an existing row or an existing table. + for (stmt in sql) { + val upper = stmt.uppercase() + for (banned in listOf("DROP ", "DELETE ", "UPDATE ", "INSERT ", "ALTER ")) { + assertFalse("migration must not contain $banned", upper.contains(banned)) + } + } + assertEquals(25, WhoopDatabase.MIGRATION_25_26.startVersion) + assertEquals(26, WhoopDatabase.MIGRATION_25_26.endVersion) + } + + /** + * The v3 `spo2Sample` table is NOT the home for this and must be left exactly as it was: its + * `red`/`ir` are NOT NULL raw ADC channels off the WHOOP 4.0 v24 layout, and + * `AnalyticsEngine.nightlySpo2RawMeans` averages them into a live tile. Storing percentages there + * would have meant fabricating red/ir zeros on every insert and dragging that mean toward zero. + */ + @Test + fun theWhoop4RawTableIsUntouched() { + assertFalse( + "the percentage and the raw ADC pair never share a table", + WhoopDatabase.SPO2_PCT_DURABLE_MIGRATION_SQL.any { it.contains("spo2Sample") }, + ) + // Its own entity, unchanged: a percentage has no home among these columns. + val raw = Spo2Sample("d", 1L, red = 100, ir = 200) + assertEquals(100, raw.red) + assertEquals(200, raw.ir) + } + + // MARK: - the in-band demultiplex, exercised through the real insert path + + /** Capture every `spo2PctSample` row `WhoopRepository.insert` would write for [aux]. */ + private fun bankedFor(aux: List): List = runBlocking { + var rows: List = emptyList() + val dao = Proxy.newProxyInstance( + WhoopDao::class.java.classLoader, + arrayOf(WhoopDao::class.java), + ) { _, method, args -> + when (method.name) { + "insertV18Aux" -> listOf(1L) + "pruneV18Aux" -> Unit + "insertSpo2Pct" -> { + @Suppress("UNCHECKED_CAST") + rows = args[0] as List + rows.map { 1L } + } + else -> throw UnsupportedOperationException("spo2-pct insert must not call ${method.name}") + } + } as WhoopDao + WhoopRepository(dao).insert(StreamBatch(v18Aux = aux), "my-whoop") + rows + } + + @Test + fun inBandValuesPersistVerbatim() { + val banked = bankedFor( + listOf( + V18AuxRow(ts = 100, auxByte82 = 97), + V18AuxRow(ts = 101, auxByte82 = 94), + V18AuxRow(ts = 102, auxByte82 = 88), + ), + ) + assertEquals(listOf(100L, 101L, 102L), banked.map { it.ts }) + assertEquals(listOf(97, 94, 88), banked.map { it.pct }) + assertEquals(listOf("my-whoop", "my-whoop", "my-whoop"), banked.map { it.deviceId }) + } + + /** + * The band is INCLUSIVE at both ends — 70 and 100 are real readings, and a gate that quietly dropped + * the endpoints would bias every night's median upward at exactly the readings that matter most. + */ + @Test + fun bandBoundariesAreInclusive() { + val banked = bankedFor(listOf(V18AuxRow(ts = 200, auxByte82 = 70), V18AuxRow(ts = 201, auxByte82 = 100))) + assertEquals(listOf(70, 100), banked.map { it.pct }) + assertEquals(70..100, SPO2_CANDIDATE_IN_BAND) + } + + /** + * `@82` is MULTIPLEXED: the same byte carries measurements, bit-7 status sentinels, sub-70 diagnostic + * codes, and 0 for "not emitted". Only 70..100 is a percentage of anything. A sentinel banked as a + * percentage would not merely be wrong, it would be *plausible* — 0x88 is 136, which no reader would + * flag, and 0x08 is 8, which reads as catastrophic hypoxia. Rejection has to happen at the boundary. + */ + @Test + fun statusSentinelsAndDiagnosticCodesAreRejected() { + val rejected = listOf(0x00L, 0x08L, 0x45L, 0x80L, 0x88L, 0x90L, 0xA0L, 0xA8L, 0xFFL) + val banked = bankedFor(rejected.mapIndexed { i, v -> V18AuxRow(ts = 300L + i, auxByte82 = v) }) + assertTrue("no sentinel or diagnostic code may be banked as a percentage", banked.isEmpty()) + } + + /** + * 101..127 is the EMPTY BAND — nothing has ever been observed there across 626,725 production + * records. It is what makes measurements and bit-7 sentinels separable by value alone, with no side + * channel. A value appearing there would mean the demultiplex assumption has broken, so it must not be + * admitted on the strength of merely being "close to 100". + */ + @Test + fun theEmptyBandAbove100IsRejected() { + val banked = bankedFor((101..127).mapIndexed { i, v -> V18AuxRow(ts = 400L + i, auxByte82 = v.toLong()) }) + assertTrue(banked.isEmpty()) + } + + /** An aux record with no `@82` slot at all banks no percentage — absence stays absence. */ + @Test + fun absentSlotBanksNothing() { + assertTrue(bankedFor(listOf(V18AuxRow(ts = 500, statusWord = 7))).isEmpty()) + } + + /** + * The durable rows are forked from the SAME aux samples, but through their OWN loop — the empty-blob + * skip on the aux side must never be able to drop an SpO2 reading, and the aux write must still happen. + */ + @Test + fun theAuxRowAndThePercentageAreBothBanked(): Unit = runBlocking { + var auxRows = 0 + var pctRows = 0 + val dao = Proxy.newProxyInstance( + WhoopDao::class.java.classLoader, + arrayOf(WhoopDao::class.java), + ) { _, method, args -> + when (method.name) { + "insertV18Aux" -> { auxRows = (args[0] as List<*>).size; listOf(1L) } + "pruneV18Aux" -> Unit + "insertSpo2Pct" -> { pctRows = (args[0] as List<*>).size; listOf(1L) } + else -> throw UnsupportedOperationException("must not call ${method.name}") + } + } as WhoopDao + WhoopRepository(dao).insert( + StreamBatch(v18Aux = listOf(V18AuxRow(ts = 600, auxByte82 = 96))), + "my-whoop", + ) + assertEquals("the raw byte is still captured in the aux table — this gate demultiplexes, " + + "it does not discard", 1, auxRows) + assertEquals(1, pctRows) + } + + /** + * THE LOAD-BEARING PROPERTY. The aux sweep must not reach this table: `pruneV18Aux` deletes only + * `FROM v18AuxSample`, and there is deliberately no prune query for `spo2PctSample` at all. This is + * the entire reason MIGRATION_25_26 exists — before it, seven-day-old SpO2 was destroyed with no + * recoverable copy. + */ + @Test + fun thereIsNoPruneForTheDurableTable() { + val prunes = WhoopDao::class.java.methods.filter { it.name.startsWith("prune") }.map { it.name } + assertTrue("v18AuxSample is the capped one", prunes.contains("pruneV18Aux")) + assertTrue( + "spo2PctSample must have NO rolling prune — it is retained forever, by design. " + + "Found: $prunes", + prunes.none { it.startsWith("pruneSpo2Pct") && it != "pruneSpo2PctByTs" }, + ) + // The timestamp heal is the one exception, and it deletes only implausible-clock rows: those are + // never pruned on AGE, so an unhealed out-of-bounds row here would persist forever. + assertTrue(prunes.contains("pruneSpo2PctByTs")) + } + + /** + * A never-pruned table makes delete-means-gone MORE important, not less: without this the wipe would + * leave years of blood-oxygen readings behind after the user asked for them to be erased. The + * whole-set guard is `DeviceRegistryTest.deleteDeviceDataCallsEveryDaoDeleteMethod`; this pins that + * the DAO method exists at all. + */ + @Test + fun theTableIsDeviceScopedForDeleteAllData() { + val names = DeviceRegistryDao::class.java.methods.map { it.name } + assertTrue(names.contains("deleteSpo2PctFor")) + assertTrue("adopt-serial must re-key it too", names.contains("reKeySpo2Pct")) + } +} diff --git a/android/app/src/test/java/com/noop/data/V18AuxCodecOffsetTest.kt b/android/app/src/test/java/com/noop/data/V18AuxCodecOffsetTest.kt new file mode 100644 index 0000000000..c30a7d109e --- /dev/null +++ b/android/app/src/test/java/com/noop/data/V18AuxCodecOffsetTest.kt @@ -0,0 +1,82 @@ +package com.noop.data + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNotEquals +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * The blob-offset question, pinned. Twin of the Swift `V18AuxCodecOffsetTests`. + * + * The cloud reader takes `fields[23]` of the STORED `v18AuxSample.fields` blob and gets physiologically + * valid SpO2 out of it. That works — but it is an ARITHMETIC COINCIDENCE of a fully-populated record, not + * a property of the format. [V18AuxCodec] is a presence-bitmap codec: absent slots are omitted from the + * body entirely, so every slot's byte offset depends on which slots ahead of it were present. + * + * These tests establish both halves, because only the pair is decisive. The first alone would license the + * fragile read; the second is what says the device write path must not use it — which is why + * `WhoopRepository.insert` forks `spo2PctSample` rows off `V18AuxRow.auxByte82`, the DECODED value, and + * never indexes the blob. + */ +class V18AuxCodecOffsetTest { + + /** A record carrying every slot, with distinguishable values. */ + private fun fullRow(): V18AuxRow = V18AuxRow( + ts = 1L, + recordIndex = 0x11223344L, rrCount = 4L, cardiacFlags = 0x21L, hrQualityFlags = 0x82L, + heartRateAlt = 58L, rrPacked = 0x0505L, cardiacStatus = 0x31L, stepCadence = 0x41L, + statusWord = 0x0606L, statusWord1 = 0x0707L, statusWord2 = 0x0808L, auxByte82 = 93L, + opticalBaselineA = 0x51L, opticalBaselineB = 0x52L, opticalAmpA = 0x53L, opticalAmpB = 0x54L, + unknownF32Bits = 0x55667788L, + ) + + private fun ByteArray.u8(i: Int): Int = this[i].toInt() and 0xFF + + /** + * A record carrying every slot: `auxByte82` lands at blob byte 23. Header is 5 bytes (version + u32 + * bitmap), then the eleven slots ahead of it occupy 4+1+1+1+1+2+1+1+2+2+2 = 18 bytes, so it sits at + * 5 + 18 = 23 and the whole blob is 32 bytes. THIS is why the cloud's `fields[23]` reads real + * percentages: in production every one of those slots is present. + */ + @Test fun fullyPopulatedRecordPutsAuxByte82AtBlobByte23() { + val blob = V18AuxCodec.pack(fullRow()) + + assertEquals("5-byte header + 27 body bytes", 32, blob.size) + assertEquals("byte 23 of a fully-populated blob IS @82", 93, blob.u8(23)) + // And it agrees with the decoded row, which is the only contract that actually holds. + assertEquals(93L, V18AuxCodec.unpack(blob, 1L).auxByte82) + // Derive the offset from the slot table rather than trusting the literal, so a future slot + // inserted ahead of @82 fails HERE with a readable number instead of silently moving the cloud's + // read onto a neighbouring byte. + val ahead = V18AuxSlot.entries + .filter { it.index < V18AuxSlot.AUX_BYTE_82.index } + .sumOf { it.width } + assertEquals(23, V18AuxCodec.HEADER_BYTES + ahead) + } + + /** + * ONE absent slot ahead of it and byte 23 is a different field entirely — while the decoded value is + * still correct. The blob read does not degrade loudly; it returns a neighbouring byte that can look + * exactly like a plausible saturation. + */ + @Test fun oneAbsentEarlierSlotMovesAuxByte82OffByte23() { + // Identical to the record above except `recordIndex` (4 bytes, slot 0) is absent. + val blob = V18AuxCodec.pack(fullRow().copy(recordIndex = null)) + + assertEquals("four fewer body bytes without recordIndex", 28, blob.size) + // @82 has slid four bytes earlier... + assertEquals(93, blob.u8(19)) + // ...and byte 23 now holds opticalAmpB, four slots further along. Reading it as a percentage would + // yield 0x54 = 84: in band, unremarkable, and completely fabricated. Note it does NOT fail loudly + // — an optical amplitude byte and a saturation percentage occupy the same numeric range, so the + // bad read is indistinguishable from a good one at the call site. + assertEquals(0x54, blob.u8(23)) + assertNotEquals(93, blob.u8(23)) + assertTrue( + "the wrong byte is still 'in band' — which is exactly why this read is unsafe", + blob.u8(23) in SPO2_CANDIDATE_IN_BAND, + ) + // The decoded read is unaffected — it is driven by the bitmap, not by an offset. + assertEquals(93L, V18AuxCodec.unpack(blob, 1L).auxByte82) + } +} diff --git a/android/app/src/test/java/com/noop/protocol/UnmappedHistoricalLayoutTest.kt b/android/app/src/test/java/com/noop/protocol/UnmappedHistoricalLayoutTest.kt new file mode 100644 index 0000000000..78ade014b7 --- /dev/null +++ b/android/app/src/test/java/com/noop/protocol/UnmappedHistoricalLayoutTest.kt @@ -0,0 +1,154 @@ +package com.noop.protocol + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * The archive decision for a WHOOP 5/MG type-47 record must be made on its LAYOUT VERSION, not on what + * the record happened to decode to. Twin of the Swift `UnmappedHistoricalLayoutTests`. + * + * [rejectedHistoricalRecords] used to ask only "did [decodeHistorical] produce a record?". That screen is + * the wrong QUESTION for a layout NOOP has no field map for: a record from an unmapped version that + * answered yes would be kept NOWHERE — not as rows (nothing mapped it) and not as bytes (it passed the + * archive filter) — and the strap frees it on the very next trim ack. An on-demand capture (the MG's ~30 s + * ECG record, banked to flash and pulled back in a later offload) is exactly the shape of record that can + * look well-formed enough to pass. + * + * The screen survives in practice only because of an accident: [decodeWhoop5Historical] returns null for + * every version but 18 (`unmappedLayoutsDecodeNothing` pins that this is true right now). One added + * static field for type 47, or one partially-mapped new version, would silently reopen the hole. These + * pin the version-based decision instead, so the archive no longer depends on that accident holding. + */ +class UnmappedHistoricalLayoutTest { + + private fun bytes(s: String): ByteArray = s.chunked(2).map { it.toInt(16).toByte() }.toByteArray() + + // A real WHOOP 5/MG type-47 v18 record (from Whoop5HistoricalDecodeTest): decodes a real unix, a + // plausible heart rate and a ~1 g gravity vector — i.e. it PASSES the old decode-outcome screen. + private val whoop5V18Hex = + "aa01740001003fb12f1280733d8401b69f266a66460066025a0265020000000000007b0a8d656463ff0012163cf6a439bf2924fd3ed763fe3e3200aa000000000000000000f7000901f10b0007010c020c00000000000000000000000000000000000000000000000100656f1e1e0000009d61a7c00000003e862817" + + /** + * Re-stamp the WHOOP 5 CRC32 trailer after mutating the version byte, so the frame stays CRC-VALID. + * Load-bearing for every test here: a CRC failure is archived on its own, which would make the + * version-based path untestable. + */ + private fun whoop5FrameWithVersion(version: Int): ByteArray { + val f = bytes(whoop5V18Hex) + f[9] = version.toByte() + val payloadEnd = f.size - 4 + val c = Crc.crc32(f, 8, payloadEnd) + for (b in 0 until 4) f[payloadEnd + b] = ((c shr (8 * b)) and 0xFF).toByte() + return f + } + + // MARK: - defect: an unmapped layout that decodes real biometrics must still be archived + + /** + * The headline case. The frame's BYTES are a real v18 record — at the v18 offsets they carry a valid + * unix, a plausible heart rate and a ~1 g gravity vector, the exact combination that made the old + * screen say "decodable, nothing lost". Only the layout-version byte differs. Whatever such a record + * decodes to, its bytes must reach the archive. + */ + @Test fun unmappedLayoutIsArchivedEvenThoughItsBytesDecodeCleanlyAsV18() { + // Precondition: read as its true version, this record decodes everything the old screen wanted. + val asV18 = decodeHistorical(bytes(whoop5V18Hex), DeviceFamily.WHOOP5)!! + assertEquals("precondition: these bytes DO decode a unix", 1780916150, asV18["unix"]) + assertEquals("precondition: these bytes DO decode a heart rate", 102, asV18["heart_rate"]) + assertTrue("precondition: these bytes DO decode a gravity vector", asV18["gravity_x"] is Double) + + val unmapped = whoop5FrameWithVersion(22) + assertEquals( + "precondition: the record is CRC-VALID, so only the layout decision can archive it", + true, Framing.parseFrame(unmapped, DeviceFamily.WHOOP5).crcOk, + ) + assertTrue(isUnmappedWhoop5HistoricalRecord(unmapped)) + assertEquals( + "a record from a layout NOOP cannot map must be archived whatever it decoded", + listOf(unmapped.toList()), + rejectedHistoricalRecords(listOf(unmapped), DeviceFamily.WHOOP5).map { it.toList() }, + ) + } + + /** Every version outside [MAPPED_WHOOP5_HISTORICAL_VERSIONS] is archived — no gaps, no lucky values. */ + @Test fun everyUnmappedVersionIsArchived() { + for (v in 0..255) { + if (v in MAPPED_WHOOP5_HISTORICAL_VERSIONS) continue + val f = whoop5FrameWithVersion(v) + assertEquals( + "hist_version $v has no field map, so its bytes must be archived", + 1, rejectedHistoricalRecords(listOf(f), DeviceFamily.WHOOP5).size, + ) + } + } + + // MARK: - lockstep: the version set and the decoder's dispatch cannot drift apart + + /** + * [MAPPED_WHOOP5_HISTORICAL_VERSIONS] is the single source of truth for the archive decision, so it + * must not claim more (or fewer) versions than the decoder actually maps. Proved from the decoder's + * own behaviour: EVERY version outside the set decodes to nothing at all. + */ + @Test fun unmappedLayoutsDecodeNothing() { + for (v in 0..255) { + if (v in MAPPED_WHOOP5_HISTORICAL_VERSIONS) continue + assertNull( + "v$v is outside the mapped set but the decoder produced a record for it — the set and " + + "the version gate in decodeWhoop5Historical have drifted", + decodeHistorical(whoop5FrameWithVersion(v), DeviceFamily.WHOOP5), + ) + } + } + + /** + * The mapped versions ARE recognised as mapped (the other direction of the lockstep). + * + * ⚠️ This set is {18, 26}, NOT the Swift `[18, 20, 21, 26]`. Swift dispatches v20 (raw optical) and + * v21 (raw 6-axis IMU) through `decodeWhoop5HistoricalV2021`; the Kotlin historical path has no such + * branch — `Whoop5RawOptical`/`Whoop5RawImu` are wired to the LIVE deep-buffer route, not to the + * type-47 historical dispatch. Claiming 20/21 here would assert a field map Android does not have. + * Update BOTH this assertion and the constant when a v20/v21 historical decoder lands on this side. + */ + @Test fun mappedVersionsAreNotTreatedAsUnmapped() { + assertEquals(setOf(18, 26), MAPPED_WHOOP5_HISTORICAL_VERSIONS) + for (v in MAPPED_WHOOP5_HISTORICAL_VERSIONS) { + assertFalse( + "v$v has a field map and must not be archived on layout grounds", + isUnmappedWhoop5HistoricalRecord(whoop5FrameWithVersion(v)), + ) + } + } + + /** + * A cleanly-decoding v18 record is still NOT archived — widening the screen must not start archiving + * the layouts NOOP already understands wholesale. v26 keeps its own exclusion (raw PPG has a durable + * `ppgWaveformSample` stream; it is known-skipped, not lost). + */ + @Test fun mappedRecordsAreStillNotArchived() { + assertTrue(rejectedHistoricalRecords(listOf(bytes(whoop5V18Hex)), DeviceFamily.WHOOP5).isEmpty()) + assertTrue(rejectedHistoricalRecords(listOf(whoop5FrameWithVersion(26)), DeviceFamily.WHOOP5).isEmpty()) + } + + /** + * The layout rule is WHOOP 5-only: it keys off `frame[9]`, which on a WHOOP 4 frame is a payload byte, + * not a version. WHOOP 4's unmapped versions go through the validated v24 fallback in + * [decodeHistorical], which keeps a record only when it decodes to a ~1 g gravity vector and a + * plausible HR and otherwise drops the biometrics — so those records reach the archive by the + * decode-outcome route and must not be dragged in by this one. + */ + @Test fun whoop4FramesAreUnaffectedByTheWhoop5LayoutRule() { + // A synthetic WHOOP 4 V24 type-47 record (HR=63) that decodes cleanly. + val v24Hex = + "aa5a008e2f18000000000000f153650000000000003f0152030000000000000000dc053075" + + "000000cdcc4c3dcdcccc3d5a657e3f00000040cdcc4c3dcdcccc3d5a657e3f504668428403" + + "200364006400b80bb80b000000000000c25c1a88" + assertTrue(rejectedHistoricalRecords(listOf(bytes(v24Hex)), DeviceFamily.WHOOP4).isEmpty()) + assertFalse( + "a WHOOP 4 frame has no type-47 byte at index 8 — the rule must not fire on it", + isUnmappedWhoop5HistoricalRecord(bytes(v24Hex)), + ) + } +} diff --git a/android/app/src/test/resources/schema_oracle.json b/android/app/src/test/resources/schema_oracle.json index ed67ec4280..585c4e13ef 100644 --- a/android/app/src/test/resources/schema_oracle.json +++ b/android/app/src/test/resources/schema_oracle.json @@ -1,6 +1,6 @@ { "_readme": "SHARED Room<->GRDB SCHEMA ORACLE (#775). Two byte-identical copies: Packages/WhoopStore/Tests/WhoopStoreTests/Resources/schema_oracle.json and android/app/src/test/resources/schema_oracle.json. SchemaOracleTests.swift compares GRDB's PRAGMA table_info/index_list against it; SchemaOracleTest.kt compares Room's exported schema JSON against it. `columns` is the iOS/GRDB shape in GRDB column order (macOS is the reference implementation); every way Android differs is spelled out in an `android` / `iosAbsent` / `androidColumnOrder` override naming a key in `divergenceReasons`. Adding a column, reordering one, or changing a type/nullability on one platform only fails both suites until it is either fixed or written down here. `grdbMigrationLineages` is the same idea for migration IDENTIFIERS: a fork that numbered its own migrations before upstream reached those numbers declares them there instead of renumbering \u2014 GRDB keys migrations by identifier string, so renaming one wedges a database that already ran it.", - "roomVersion": 25, + "roomVersion": 26, "grdbMigrations": [ "v1", "v2", @@ -1699,8 +1699,7 @@ "indices": [] }, "spo2PctSample": { - "platform": "ios_only", - "reason": "Fork-only iOS table (GRDB `v34-spo2-pct-durable`): the durable per-second SpO2 percentages the WHOOP 5/MG emits on `@82` of its v18 record. The byte is already captured raw inside `v18AuxSample.fields`, but that table is CAPPED at 604,800 rows/device (~7 days at 1 Hz) because the aux blob costs ~30 MB/day, so every reading rolled off after a week and was gone permanently \u2014 the strap trims its own history the moment an offload is acked. The in-band signal is only ~2,000 samples/week (~100k rows/year), so a narrow never-pruned sibling retains it forever at trivial cost while the wide aux capture stays capped. `pct` is the RAW in-band byte (70...100); run grouping and acquisition-ramp trimming are read-time policy, not stored. No Room twin yet: the Kotlin capture/aggregate side is outstanding, and until it lands Android keeps banking the byte in its own `v18AuxSample` twin and loses it on the same 7-day roll.", + "platform": "both", "columns": [ { "name": "deviceId", From 4dfc19795d9a97a29b37c7633025d92caadb1348 Mon Sep 17 00:00:00 2001 From: vishk23 Date: Thu, 6 Aug 2026 00:21:08 -0400 Subject: [PATCH 164/171] MG ECG: offer the untried restart control value (124 arg 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Whoop5Ecg.ControlSignal` carries stop/start/restart = 0/1/2, and only 0 and 1 have ever been put on a wire by any build in this tree. That leaves `restart` as the one untried argument inside the opcode family recorded as non-destructive (123/124/125/139), and therefore the cheapest remaining experiment against #891's open question. `BLEManager.ecgRestartCapture()` sends the SAME preamble as `ecgStartCapture()` — toggleRealtimeFilteredECG(on), toggleSaveRawECG(on) — and changes exactly one byte, the control value. What the strap does with `2` is unknown: the name is read off the client enum's order, like the wrist mapping, and no capture attests it, so there is no basis for claiming it re-arms the channel toggles itself nor for claiming it needs them already on. Issuing 124 alone would change two things at once and leave a null result unattributable to either. `ecgStartCapture()` / `ecgStopCapture()` are untouched; the ordinary stop sequence reverses this, because the strap state it leaves behind is the state a normal start leaves behind. Gating is unchanged and unwidened: the `noopWhoop5Ecg` Experimental opt-in, `ecgGatesAllow()` (positively attested MG, connected), the send() allowlist's double gate, and the same confirmation dialog every other ECG action sits behind. No new gate, no weaker gate. The probe report now annotates each command with the argument it was sent with (`TOGGLE_LABRADOR_DATA_GENERATION(124) arg=2`). Without it a start run and a restart run render character-for-character identically, and the report is the artefact that gets copied into an issue long after the strap log is gone. It is display-only, never feeds a verdict, and stays `nil` for an UNSOLICITED reply whose argument nothing here knows. Mirrored in Kotlin with the twin tests, per the parity contract. The Android client has no ECG app layer, so there is no Kotlin twin of the restart send path — only of the probe formatting it produces. i18n: the new button label is left out of the string catalog, as the branch's existing ECG strings are. Appending it untranslated would turn one failing focus-locale check into five by breaking the four non-focus locales that currently pass, and this copy sits beside medical-safety framing that must not be machine-translated. Tracked as debt. --- .../WhoopProtocol/Whoop5EcgProbe.swift | 39 +++++++++-- .../WhoopProtocolTests/Whoop5EcgTests.swift | 65 +++++++++++++---- Strand/App/AppModel.swift | 4 ++ Strand/BLE/BLEManager.swift | 53 +++++++++++++- Strand/Screens/DevicesView.swift | 7 ++ .../java/com/noop/protocol/Whoop5EcgProbe.kt | 30 ++++++-- .../com/noop/protocol/Whoop5EcgProbeTest.kt | 69 +++++++++++++++---- docs/PROTOCOL.md | 12 ++++ 8 files changed, 238 insertions(+), 41 deletions(-) diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5EcgProbe.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5EcgProbe.swift index d2c1bdc226..05d3a0b22c 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5EcgProbe.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5EcgProbe.swift @@ -110,16 +110,35 @@ public enum Whoop5EcgProbe { /// read silence as evidence turn on this flag and a silently-omitted `true` is exactly the bug /// this field exists to prevent. public let requestsRealtimeData: Bool + /// The ARGUMENT byte this command was sent with, when it is known. + /// + /// `label` carries only the opcode, because it is also the key the inbound handler matches a + /// COMMAND_RESPONSE against and the reply echoes no argument — so the argument cannot live in + /// the label without breaking that match. It is recorded separately instead, because two runs + /// of the same opcode can differ only in this byte: `mainControlECGDataGeneration` takes + /// `stop`/`start`/`restart` (0/1/2), so a start run and a restart run render identically + /// without it and a future reader of a copied report cannot tell which one produced the + /// verdict. + /// + /// Reported as the RAW NUMBER, never a token name, for the same reason the classifier byte is: + /// the number is lossless and cannot drift from the bytes actually put on the wire. + /// + /// `nil` is "not known", which is the honest value for an UNSOLICITED reply — nothing here + /// sent it. It is display-only and feeds no verdict, so unlike `requestsRealtimeData` a + /// default is safe: an omission loses an annotation, never changes a claim. + public let sentArgument: UInt8? /// The reply frame as hex, when one arrived. public let replyHex: String? public init(label: String, outcome: CommandOutcome, requestsRealtimeData: Bool, + sentArgument: UInt8? = nil, replyHex: String? = nil) { self.label = label self.outcome = outcome self.requestsRealtimeData = requestsRealtimeData + self.sentArgument = sentArgument self.replyHex = replyHex } @@ -128,6 +147,12 @@ public enum Whoop5EcgProbe { requestsRealtimeData ? "asks for realtime ECG data" : "cannot produce ECG data (configuration or OFF)" } + + /// The `label`, plus the argument when it is known — what every report line leads with. + public var labelWithArgument: String { + guard let sentArgument else { return label } + return "\(label) arg=\(sentArgument)" + } } /// The gate verdict, kept separate from the report text so it is assertable in a test. @@ -220,11 +245,11 @@ public enum Whoop5EcgProbe { let failures = steps.filter { $0.outcome == .failure } if !failures.isEmpty { // A refusal is only evidence about the BLOCK question when what was refused asked for data. - let dataFailures = failures.filter(\.requestsRealtimeData).map(\.label) + let dataFailures = failures.filter(\.requestsRealtimeData).map(\.labelWithArgument) if !dataFailures.isEmpty { return .dataRequestRefused(commands: dataFailures) } - return .commandRefused(commands: failures.map(\.label)) + return .commandRefused(commands: failures.map(\.labelWithArgument)) } - let unsupported = steps.filter { $0.outcome == .unsupported }.map(\.label) + let unsupported = steps.filter { $0.outcome == .unsupported }.map(\.labelWithArgument) if !unsupported.isEmpty { return .opcodeUnsupported(commands: unsupported) } if ecgPacketsSeen > 0 { return .ecgCandidatesArrived(packets: ecgPacketsSeen) } guard !steps.isEmpty else { return .noReplies } @@ -232,9 +257,9 @@ public enum Whoop5EcgProbe { // Past this point the verdict INTERPRETS SILENCE, which only carries information about the ECG // data path when the run asked that path for something and the strap said yes. let requests = steps.filter(\.requestsRealtimeData) - guard !requests.isEmpty else { return .noDataRequested(commands: steps.map(\.label)) } + guard !requests.isEmpty else { return .noDataRequested(commands: steps.map(\.labelWithArgument)) } guard requests.contains(where: { $0.outcome == .success }) else { - return .dataRequestNotAccepted(commands: requests.map(\.label)) + return .dataRequestNotAccepted(commands: requests.map(\.labelWithArgument)) } if steps.allSatisfy({ $0.outcome == .success }) { return .acceptedButSilent(windowSeconds: windowSeconds) @@ -261,7 +286,7 @@ public enum Whoop5EcgProbe { // Each step carries WHY it does or does not bear on the block question, so the verdict above // can be checked against its own inputs without reading the source. for step in steps { - sb += " \(step.label): \(step.outcome.token) — \(step.roleNote)\n" + sb += " \(step.labelWithArgument): \(step.outcome.token) — \(step.roleNote)\n" } } sb += "\nECG-shaped packets seen in \(windowSeconds)s: \(ecgPacketsSeen)\n" @@ -287,7 +312,7 @@ public enum Whoop5EcgProbe { sb += "Candidate packet types (structural triage only, NOT a confirmed mapping):\n" for line in candidateFrames { sb += " \(line)\n" } } - let replies = steps.compactMap { step in step.replyHex.map { " \(step.label): \($0)" } } + let replies = steps.compactMap { step in step.replyHex.map { " \(step.labelWithArgument): \($0)" } } if !replies.isEmpty { sb += "\nRaw replies:\n" sb += replies.joined(separator: "\n") + "\n" diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift index 043904808e..8f180eb25c 100644 --- a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift @@ -484,6 +484,7 @@ final class Whoop5EcgTests: XCTestCase { label: "\(name)(\(cmd))", outcome: outcome, requestsRealtimeData: Whoop5Ecg.requestsRealtimeData(cmd: cmd, arg: arg), + sentArgument: arg, replyHex: replyHex) } @@ -517,10 +518,10 @@ final class Whoop5EcgTests: XCTestCase { // the old precedence turned one loose match into an unhedged "not blocked". let failed = [sent(124, arg: 1, .failure)] XCTAssertEqual(Whoop5EcgProbe.verdict(steps: failed, ecgPacketsSeen: 12, windowSeconds: 30), - .dataRequestRefused(commands: ["TOGGLE_LABRADOR_DATA_GENERATION(124)"])) + .dataRequestRefused(commands: ["TOGGLE_LABRADOR_DATA_GENERATION(124) arg=1"])) let unsupported = [sent(139, arg: 1, .unsupported)] XCTAssertEqual(Whoop5EcgProbe.verdict(steps: unsupported, ecgPacketsSeen: 12, windowSeconds: 30), - .opcodeUnsupported(commands: ["TOGGLE_LABRADOR_FILTERED(139)"])) + .opcodeUnsupported(commands: ["TOGGLE_LABRADOR_FILTERED(139) arg=1"])) // With no contrary result code, candidates are the verdict — as candidates, not as proof. let ok = [sent(124, arg: 1, .success)] XCTAssertEqual(Whoop5EcgProbe.verdict(steps: ok, ecgPacketsSeen: 12, windowSeconds: 30), @@ -543,7 +544,7 @@ final class Whoop5EcgTests: XCTestCase { sent(124, arg: Whoop5Ecg.ControlSignal.start.rawValue, .failure), ] XCTAssertEqual(Whoop5EcgProbe.verdict(steps: steps, ecgPacketsSeen: 0, windowSeconds: 30), - .dataRequestRefused(commands: ["TOGGLE_LABRADOR_DATA_GENERATION(124)"])) + .dataRequestRefused(commands: ["TOGGLE_LABRADOR_DATA_GENERATION(124) arg=1"])) } func testVerdictAllSuccessButSilentIsTheSilentNoOpCase() { @@ -578,7 +579,7 @@ final class Whoop5EcgTests: XCTestCase { let steps = [sent(123, arg: Whoop5Ecg.WristSelection.left.rawValue, .success, replyHex: "aa010c000100271124d77b81010100007ce76722")] XCTAssertEqual(Whoop5EcgProbe.verdict(steps: steps, ecgPacketsSeen: 0, windowSeconds: 30), - .noDataRequested(commands: ["SELECT_WRIST(123)"])) + .noDataRequested(commands: ["SELECT_WRIST(123) arg=1"])) let text = Whoop5EcgProbe.report(steps: steps, ecgPacketsSeen: 0, candidateFrames: [], windowSeconds: 30) XCTAssertTrue(text.contains("NOT A TEST")) @@ -596,7 +597,7 @@ final class Whoop5EcgTests: XCTestCase { let steps = [sent(123, arg: Whoop5Ecg.WristSelection.right.rawValue, .failure, replyHex: "aa010c000100271124217bcc000100000213163d")] XCTAssertEqual(Whoop5EcgProbe.verdict(steps: steps, ecgPacketsSeen: 0, windowSeconds: 30), - .commandRefused(commands: ["SELECT_WRIST(123)"])) + .commandRefused(commands: ["SELECT_WRIST(123) arg=0"])) let text = Whoop5EcgProbe.report(steps: steps, ecgPacketsSeen: 0, candidateFrames: [], windowSeconds: 30) XCTAssertTrue(text.contains("REFUSED")) @@ -612,9 +613,9 @@ final class Whoop5EcgTests: XCTestCase { sent(139, arg: 0, .success), ] XCTAssertEqual(Whoop5EcgProbe.verdict(steps: steps, ecgPacketsSeen: 0, windowSeconds: 30), - .noDataRequested(commands: ["TOGGLE_LABRADOR_DATA_GENERATION(124)", - "TOGGLE_LABRADOR_RAW_SAVE(125)", - "TOGGLE_LABRADOR_FILTERED(139)"])) + .noDataRequested(commands: ["TOGGLE_LABRADOR_DATA_GENERATION(124) arg=0", + "TOGGLE_LABRADOR_RAW_SAVE(125) arg=0", + "TOGGLE_LABRADOR_FILTERED(139) arg=0"])) } func testRawSaveAloneCannotUnlockTheSilentVerdict() { @@ -622,7 +623,7 @@ final class Whoop5EcgTests: XCTestCase { // so a raw-save-only run cannot be read as "accepted and then silent" (#891 hypothesis (b)). let steps = [sent(125, arg: 1, .success)] XCTAssertEqual(Whoop5EcgProbe.verdict(steps: steps, ecgPacketsSeen: 0, windowSeconds: 30), - .noDataRequested(commands: ["TOGGLE_LABRADOR_RAW_SAVE(125)"])) + .noDataRequested(commands: ["TOGGLE_LABRADOR_RAW_SAVE(125) arg=1"])) } func testAnUnacknowledgedDataRequestIsNotAcceptedButSilent() { @@ -633,8 +634,8 @@ final class Whoop5EcgTests: XCTestCase { sent(124, arg: Whoop5Ecg.ControlSignal.start.rawValue, .noReply), ] XCTAssertEqual(Whoop5EcgProbe.verdict(steps: steps, ecgPacketsSeen: 0, windowSeconds: 30), - .dataRequestNotAccepted(commands: ["TOGGLE_LABRADOR_FILTERED(139)", - "TOGGLE_LABRADOR_DATA_GENERATION(124)"])) + .dataRequestNotAccepted(commands: ["TOGGLE_LABRADOR_FILTERED(139) arg=1", + "TOGGLE_LABRADOR_DATA_GENERATION(124) arg=1"])) } func testNoVerdictClaimsAFlagBlockWithoutADataRequest() { @@ -701,7 +702,7 @@ final class Whoop5EcgTests: XCTestCase { func testVerdictUnsupportedIsReportedAsItselfNotAsABlock() { let steps = [sent(139, arg: 1, .unsupported)] XCTAssertEqual(Whoop5EcgProbe.verdict(steps: steps, ecgPacketsSeen: 0, windowSeconds: 30), - .opcodeUnsupported(commands: ["TOGGLE_LABRADOR_FILTERED(139)"])) + .opcodeUnsupported(commands: ["TOGGLE_LABRADOR_FILTERED(139) arg=1"])) } func testVerdictSilenceIsNeverCalledABlock() { @@ -729,13 +730,49 @@ final class Whoop5EcgTests: XCTestCase { let text = Whoop5EcgProbe.report(steps: steps, ecgPacketsSeen: 0, candidateFrames: ["type=0x28 len=220"], windowSeconds: 30) XCTAssertTrue(text.contains("DATA REQUEST REFUSED")) - XCTAssertTrue(text.contains("SELECT_WRIST(123): SUCCESS(1)")) - XCTAssertTrue(text.contains("TOGGLE_LABRADOR_DATA_GENERATION(124): FAILURE(0)")) + XCTAssertTrue(text.contains("SELECT_WRIST(123) arg=1: SUCCESS(1)")) + XCTAssertTrue(text.contains("TOGGLE_LABRADOR_DATA_GENERATION(124) arg=1: FAILURE(0)")) XCTAssertTrue(text.contains("type=0x28 len=220")) XCTAssertTrue(text.contains("aabb")) XCTAssertTrue(text.contains("not a medical measurement or a diagnosis")) } + /// A START run and a RESTART run send the SAME three opcodes and differ in exactly one byte, so + /// without the argument annotation their reports are character-for-character identical — and a + /// report is the artefact that gets copied out of the app and pasted into an issue, long after the + /// strap log that recorded `payload=0102` is gone. This pins that the two are distinguishable. + func testStartAndRestartRunsRenderDistinguishably() { + func run(_ control: Whoop5Ecg.ControlSignal) -> String { + Whoop5EcgProbe.report( + steps: [sent(139, arg: 1, .success), + sent(125, arg: 1, .success), + sent(124, arg: control.rawValue, .success)], + ecgPacketsSeen: 0, candidateFrames: [], windowSeconds: 30) + } + let startText = run(.start) + let restartText = run(.restart) + XCTAssertTrue(startText.contains("TOGGLE_LABRADOR_DATA_GENERATION(124) arg=1: SUCCESS(1)")) + XCTAssertTrue(restartText.contains("TOGGLE_LABRADOR_DATA_GENERATION(124) arg=2: SUCCESS(1)")) + XCTAssertNotEqual(startText, restartText) + // Both are still the SAME verdict: restart asks for realtime data exactly like start does, so + // the annotation records what was sent without changing what the run is read as. + XCTAssertTrue(restartText.contains("Accepted but SILENT")) + XCTAssertTrue(startText.contains("Accepted but SILENT")) + } + + /// An UNSOLICITED reply has no known argument — nothing in the app sent it — and the report must say + /// nothing rather than invent a value. + func testAnUnknownArgumentIsOmittedRatherThanGuessed() { + let step = Whoop5EcgProbe.Step(label: "TOGGLE_LABRADOR_DATA_GENERATION(124)", + outcome: .success, + requestsRealtimeData: false) + XCTAssertNil(step.sentArgument) + XCTAssertEqual(step.labelWithArgument, "TOGGLE_LABRADOR_DATA_GENERATION(124)") + let text = Whoop5EcgProbe.report(steps: [step], ecgPacketsSeen: 0, + candidateFrames: [], windowSeconds: 30) + XCTAssertFalse(text.contains("arg=")) + } + /// #896 review: nobody is told to hold the clasp. An MG measures across the wrist electrode AND the /// two clasp indents, and lead state is not on the wire — so a run where the clasp was never touched /// returns zero packets for a reason that has nothing to do with the firmware. #891 asks other MG diff --git a/Strand/App/AppModel.swift b/Strand/App/AppModel.swift index f6a9fc747a..aa706134c2 100644 --- a/Strand/App/AppModel.swift +++ b/Strand/App/AppModel.swift @@ -881,6 +881,10 @@ final class AppModel: ObservableObject { /// PERSISTENT strap write, deliberately its own action rather than part of the start flow. func ecgSelectWrist(_ wrist: Whoop5Ecg.WristSelection) { ble.ecgSelectWrist(wrist) } func ecgStartCapture() { ble.ecgStartCapture() } + /// The RESTART variant of the same sequence — identical bytes except `mainControlECGDataGeneration` + /// carries `restart` (2) rather than `start` (1), the one value in that enum no build here has ever + /// sent. Same gates, same confirmation dialog, same `ecgStopCapture()` reversal. + func ecgRestartCapture() { ble.ecgRestartCapture() } /// `reportsResult: false` for the Settings-toggle path, so switching the experiment off doesn't pop /// the Devices result sheet from another screen. func ecgStopCapture(reportsResult: Bool = true) { ble.ecgStopCapture(reportsResult: reportsResult) } diff --git a/Strand/BLE/BLEManager.swift b/Strand/BLE/BLEManager.swift index 9ad736c30b..9de6b45911 100644 --- a/Strand/BLE/BLEManager.swift +++ b/Strand/BLE/BLEManager.swift @@ -3203,9 +3203,13 @@ public final class BLEManager: NSObject, ObservableObject { // the only point where "could this command have produced data" is knowable. Every verdict that // reads silence as evidence depends on it. let requestsData = Whoop5Ecg.requestsRealtimeData(cmd: command.rawValue, arg: arg) + // Recorded for the same reason and at the same moment: the reply echoes no argument, and two runs + // of opcode 124 differ ONLY in this byte (start=1 vs restart=2). Without it a copied report cannot + // be told apart from the other run's. ecgProbeSteps.append(Whoop5EcgProbe.Step(label: label, outcome: .noReply, - requestsRealtimeData: requestsData)) + requestsRealtimeData: requestsData, + sentArgument: arg)) log("ECG probe: → \(label) payload=\(hex(Whoop5Ecg.commandPayload(arg: arg)))") send(command, payload: Whoop5Ecg.commandPayload(arg: arg)) } @@ -3238,6 +3242,50 @@ public final class BLEManager: NSObject, ObservableObject { scheduleEcgProbeVerdict() } + /// The RESTART variant of the turn-on sequence: byte-for-byte `ecgStartCapture()` except that + /// `mainControlECGDataGeneration` carries `.restart` (2) where the normal path carries `.start` (1). + /// + /// ## Why this exists + /// + /// `Whoop5Ecg.ControlSignal` has three values — `stop`/`start`/`restart` = 0/1/2 — and only 0 and 1 + /// have ever been put on a wire by any build in this repo. 2 is therefore the one argument in this + /// family that is both on the verified non-destructive opcode list (#891: 123/124/125/139) and + /// entirely untried, which makes it the cheapest remaining experiment: if the strap distinguishes + /// "start" from "restart" at all, the difference shows up in the COMMAND_RESPONSE or in whether + /// packets follow, and if it does not, the run costs one more 30 s window. + /// + /// ## Why the same 139/125 preamble rather than 124 alone + /// + /// **What the strap does with 2 is unknown**, and nothing in this repo narrows it: the name + /// `restart` is read off the client enum's ORDER, exactly like the wrist mapping, and no capture + /// attests it. So there is no basis for claiming that `restart` re-arms the two channel toggles + /// itself and no basis for claiming it needs them already on. + /// + /// Given that, the defensible choice is minimal deviation from the ONE sequence a strap has been + /// observed to accept — `toggleRealtimeFilteredECG(on)` → `toggleSaveRawECG(on)` → + /// `mainControlECGDataGeneration(…)` — changing exactly one byte. Issuing 124 alone would change TWO + /// things at once (the control value AND the absence of the preamble), so a null result could not be + /// attributed to either, and a null result is the outcome to plan for here. Both toggles are + /// idempotent ON writes, so re-sending them to a strap that is already streaming asks for nothing new. + /// + /// Reversal is the existing `ecgStopCapture()` — there is no restart-specific OFF path, because the + /// bytes this leaves behind are the bytes `ecgStartCapture()` leaves behind. + public func ecgRestartCapture() { + guard ecgGatesAllow() else { return } + // Latched for the same reason as in `ecgStartCapture()`: a mid-sequence drop must still leave the + // Stop control offered, and this sequence can leave the strap in exactly the same state. + ecgMayBeRunning = true + beginEcgProbeRun(clearingSteps: true) + log("ECG probe: starting the ECG turn-on sequence in its RESTART variant — " + + "mainControlECGDataGeneration=restart(\(Whoop5Ecg.ControlSignal.restart.rawValue)) instead of " + + "start(\(Whoop5Ecg.ControlSignal.start.rawValue)); the preamble is unchanged so the control " + + "byte is the only difference from a normal start (experimental, unvalidated instrumentation)") + sendEcgCommand(.toggleLabradorFiltered, arg: 1) + sendEcgCommand(.toggleLabradorRawSave, arg: 1) + sendEcgCommand(.toggleLabradorDataGeneration, arg: Whoop5Ecg.ControlSignal.restart.rawValue) + scheduleEcgProbeVerdict() + } + /// The explicit OFF path: stop generation first, then drop both streams. /// /// `reportsResult: false` is the Settings-toggle path — it must not pop the Devices result sheet from @@ -3348,6 +3396,7 @@ public final class BLEManager: NSObject, ObservableObject { label: label, outcome: outcome, requestsRealtimeData: ecgProbeSteps[idx].requestsRealtimeData, + sentArgument: ecgProbeSteps[idx].sentArgument, replyHex: replyHex) log("ECG probe: ← \(label) \(outcome.token)") } else if ecgProbeSteps.count < BLEManager.ecgProbeMaxSteps { @@ -3357,9 +3406,11 @@ public final class BLEManager: NSObject, ObservableObject { // // `requestsRealtimeData: false` — nothing here sent it, so the argument behind it is unknown, // and an unknown must never be able to unlock a verdict that claims the feature is blocked. + // `sentArgument: nil` says the same thing in the report rather than guessing a value. ecgProbeSteps.append(Whoop5EcgProbe.Step(label: label, outcome: outcome, requestsRealtimeData: false, + sentArgument: nil, replyHex: replyHex)) log("ECG probe: ← \(label) \(outcome.token) (unsolicited)") } diff --git a/Strand/Screens/DevicesView.swift b/Strand/Screens/DevicesView.swift index 02a9befca7..c8b827abba 100644 --- a/Strand/Screens/DevicesView.swift +++ b/Strand/Screens/DevicesView.swift @@ -1215,6 +1215,13 @@ private struct EcgProbeSheets: ViewModifier { titleVisibility: .visible, presenting: target) { device in Button("Start ECG capture") { model.ecgStartCapture(); target = nil } + // Deliberately worded so it cannot be mistaken for the normal path: it sends the same + // three commands with ONE byte changed (mainControlECGDataGeneration restart=2 rather + // than start=1), which is the only argument in that enum no build here has ever sent. + // Same gates as every other entry in this dialog; "Stop" reverses it either way. + Button("Start ECG capture (restart variant, experimental)") { + model.ecgRestartCapture(); target = nil + } Button("Stop ECG capture") { model.ecgStopCapture(); target = nil } Button("Set which wrist you wear it on…") { target = nil; wristTarget = device } // Local file read, no strap traffic: did the record actually land in the raw archive? diff --git a/android/app/src/main/java/com/noop/protocol/Whoop5EcgProbe.kt b/android/app/src/main/java/com/noop/protocol/Whoop5EcgProbe.kt index 8fe948f106..6866539315 100644 --- a/android/app/src/main/java/com/noop/protocol/Whoop5EcgProbe.kt +++ b/android/app/src/main/java/com/noop/protocol/Whoop5EcgProbe.kt @@ -106,17 +106,33 @@ object Whoop5EcgProbe { * [requestsRealtimeData] deliberately has NO default: every construction site must state it, because * the verdicts that read silence as evidence turn on this flag and a silently-omitted `true` is * exactly the bug this field exists to prevent. + * + * [sentArgument] is the ARGUMENT byte the command was sent with, when it is known. [label] carries + * only the opcode, because it is also the key an app layer matches a COMMAND_RESPONSE against and + * the reply echoes no argument — so the argument cannot live in the label without breaking that + * match. It is recorded separately because two runs of the same opcode can differ only in this byte: + * `mainControlECGDataGeneration` takes `stop`/`start`/`restart` (0/1/2), so a start run and a restart + * run render identically without it and a future reader of a copied report cannot tell which one + * produced the verdict. Reported as the RAW NUMBER, never a token name, so it cannot drift from the + * bytes actually put on the wire. `null` is "not known" — the honest value for an UNSOLICITED reply. + * It is display-only and feeds no verdict, so unlike [requestsRealtimeData] a default is safe: an + * omission loses an annotation, never changes a claim. */ data class Step( val label: String, val outcome: CommandOutcome, val requestsRealtimeData: Boolean, + val sentArgument: Int? = null, val replyHex: String? = null, ) { /** How the report annotates the step, so a reader can see WHY the verdict is what it is. */ val roleNote: String get() = if (requestsRealtimeData) "asks for realtime ECG data" else "cannot produce ECG data (configuration or OFF)" + + /** The [label], plus the argument when it is known — what every report line leads with. */ + val labelWithArgument: String + get() = if (sentArgument == null) label else "$label arg=$sentArgument" } /** The gate verdict, kept separate from the report text so it is assertable in a test. */ @@ -221,11 +237,11 @@ object Whoop5EcgProbe { val failures = steps.filter { it.outcome is CommandOutcome.Failure } if (failures.isNotEmpty()) { // A refusal is only evidence about the BLOCK question when what was refused asked for data. - val dataFailures = failures.filter { it.requestsRealtimeData }.map { it.label } + val dataFailures = failures.filter { it.requestsRealtimeData }.map { it.labelWithArgument } if (dataFailures.isNotEmpty()) return Verdict.DataRequestRefused(dataFailures) - return Verdict.CommandRefused(failures.map { it.label }) + return Verdict.CommandRefused(failures.map { it.labelWithArgument }) } - val unsupported = steps.filter { it.outcome is CommandOutcome.Unsupported }.map { it.label } + val unsupported = steps.filter { it.outcome is CommandOutcome.Unsupported }.map { it.labelWithArgument } if (unsupported.isNotEmpty()) return Verdict.OpcodeUnsupported(unsupported) if (ecgPacketsSeen > 0) return Verdict.EcgCandidatesArrived(ecgPacketsSeen) if (steps.isEmpty()) return Verdict.NoReplies @@ -233,9 +249,9 @@ object Whoop5EcgProbe { // Past this point the verdict INTERPRETS SILENCE, which only carries information about the ECG // data path when the run asked that path for something and the strap said yes. val requests = steps.filter { it.requestsRealtimeData } - if (requests.isEmpty()) return Verdict.NoDataRequested(steps.map { it.label }) + if (requests.isEmpty()) return Verdict.NoDataRequested(steps.map { it.labelWithArgument }) if (requests.none { it.outcome is CommandOutcome.Success }) { - return Verdict.DataRequestNotAccepted(requests.map { it.label }) + return Verdict.DataRequestNotAccepted(requests.map { it.labelWithArgument }) } if (steps.all { it.outcome is CommandOutcome.Success }) { return Verdict.AcceptedButSilent(windowSeconds) @@ -264,7 +280,7 @@ object Whoop5EcgProbe { } else { // Each step carries WHY it does or does not bear on the block question, so the verdict above // can be checked against its own inputs without reading the source. - for (step in steps) sb.append(" ${step.label}: ${step.outcome.token} — ${step.roleNote}\n") + for (step in steps) sb.append(" ${step.labelWithArgument}: ${step.outcome.token} — ${step.roleNote}\n") } sb.append("\nECG-shaped packets seen in ${windowSeconds}s: $ecgPacketsSeen\n") if (steps.isNotEmpty() && steps.none { it.requestsRealtimeData }) { @@ -294,7 +310,7 @@ object Whoop5EcgProbe { sb.append("Candidate packet types (structural triage only, NOT a confirmed mapping):\n") for (line in candidateFrames) sb.append(" $line\n") } - val replies = steps.mapNotNull { step -> step.replyHex?.let { " ${step.label}: $it" } } + val replies = steps.mapNotNull { step -> step.replyHex?.let { " ${step.labelWithArgument}: $it" } } if (replies.isNotEmpty()) { sb.append("\nRaw replies:\n") sb.append(replies.joinToString("\n") + "\n") diff --git a/android/app/src/test/java/com/noop/protocol/Whoop5EcgProbeTest.kt b/android/app/src/test/java/com/noop/protocol/Whoop5EcgProbeTest.kt index ab1f288dba..b2295e0629 100644 --- a/android/app/src/test/java/com/noop/protocol/Whoop5EcgProbeTest.kt +++ b/android/app/src/test/java/com/noop/protocol/Whoop5EcgProbeTest.kt @@ -2,6 +2,7 @@ package com.noop.protocol import org.junit.Assert.assertEquals import org.junit.Assert.assertFalse +import org.junit.Assert.assertNotEquals import org.junit.Assert.assertNull import org.junit.Assert.assertTrue import org.junit.Test @@ -36,6 +37,7 @@ class Whoop5EcgProbeTest { label = "$name($cmd)", outcome = outcome, requestsRealtimeData = Whoop5Ecg.requestsRealtimeData(cmd, arg), + sentArgument = arg, replyHex = replyHex, ) } @@ -103,11 +105,11 @@ class Whoop5EcgProbeTest { @Test fun attestedResultCodesOutrankTheShapeHeuristic() { assertEquals( - Whoop5EcgProbe.Verdict.DataRequestRefused(listOf("TOGGLE_LABRADOR_DATA_GENERATION(124)")), + Whoop5EcgProbe.Verdict.DataRequestRefused(listOf("TOGGLE_LABRADOR_DATA_GENERATION(124) arg=1")), Whoop5EcgProbe.verdict(listOf(sent(124, 1, Whoop5EcgProbe.CommandOutcome.Failure)), 12, 30), ) assertEquals( - Whoop5EcgProbe.Verdict.OpcodeUnsupported(listOf("TOGGLE_LABRADOR_FILTERED(139)")), + Whoop5EcgProbe.Verdict.OpcodeUnsupported(listOf("TOGGLE_LABRADOR_FILTERED(139) arg=1")), Whoop5EcgProbe.verdict(listOf(sent(139, 1, Whoop5EcgProbe.CommandOutcome.Unsupported)), 12, 30), ) assertEquals( @@ -159,7 +161,7 @@ class Whoop5EcgProbeTest { ), ) assertEquals( - Whoop5EcgProbe.Verdict.NoDataRequested(listOf("SELECT_WRIST(123)")), + Whoop5EcgProbe.Verdict.NoDataRequested(listOf("SELECT_WRIST(123) arg=1")), Whoop5EcgProbe.verdict(steps, 0, 30), ) val text = Whoop5EcgProbe.report(steps, 0, emptyList(), 30) @@ -183,7 +185,7 @@ class Whoop5EcgProbeTest { ), ) assertEquals( - Whoop5EcgProbe.Verdict.CommandRefused(listOf("SELECT_WRIST(123)")), + Whoop5EcgProbe.Verdict.CommandRefused(listOf("SELECT_WRIST(123) arg=0")), Whoop5EcgProbe.verdict(steps, 0, 30), ) val text = Whoop5EcgProbe.report(steps, 0, emptyList(), 30) @@ -203,9 +205,9 @@ class Whoop5EcgProbeTest { assertEquals( Whoop5EcgProbe.Verdict.NoDataRequested( listOf( - "TOGGLE_LABRADOR_DATA_GENERATION(124)", - "TOGGLE_LABRADOR_RAW_SAVE(125)", - "TOGGLE_LABRADOR_FILTERED(139)", + "TOGGLE_LABRADOR_DATA_GENERATION(124) arg=0", + "TOGGLE_LABRADOR_RAW_SAVE(125) arg=0", + "TOGGLE_LABRADOR_FILTERED(139) arg=0", ), ), Whoop5EcgProbe.verdict(steps, 0, 30), @@ -217,7 +219,7 @@ class Whoop5EcgProbeTest { // RAW_SAVE names flash. A realtime listen window observes nothing from it even on total success, // so a raw-save-only run cannot be read as "accepted and then silent" (#891 hypothesis (b)). assertEquals( - Whoop5EcgProbe.Verdict.NoDataRequested(listOf("TOGGLE_LABRADOR_RAW_SAVE(125)")), + Whoop5EcgProbe.Verdict.NoDataRequested(listOf("TOGGLE_LABRADOR_RAW_SAVE(125) arg=1")), Whoop5EcgProbe.verdict(listOf(sent(125, 1, Whoop5EcgProbe.CommandOutcome.Success)), 0, 30), ) } @@ -232,7 +234,7 @@ class Whoop5EcgProbeTest { ) assertEquals( Whoop5EcgProbe.Verdict.DataRequestNotAccepted( - listOf("TOGGLE_LABRADOR_FILTERED(139)", "TOGGLE_LABRADOR_DATA_GENERATION(124)"), + listOf("TOGGLE_LABRADOR_FILTERED(139) arg=1", "TOGGLE_LABRADOR_DATA_GENERATION(124) arg=1"), ), Whoop5EcgProbe.verdict(steps, 0, 30), ) @@ -273,7 +275,7 @@ class Whoop5EcgProbeTest { @Test fun verdictUnsupportedIsReportedAsItselfNotAsABlock() { assertEquals( - Whoop5EcgProbe.Verdict.OpcodeUnsupported(listOf("TOGGLE_LABRADOR_FILTERED(139)")), + Whoop5EcgProbe.Verdict.OpcodeUnsupported(listOf("TOGGLE_LABRADOR_FILTERED(139) arg=1")), Whoop5EcgProbe.verdict(listOf(sent(139, 1, Whoop5EcgProbe.CommandOutcome.Unsupported)), 0, 30), ) } @@ -348,13 +350,56 @@ class Whoop5EcgProbeTest { ) val text = Whoop5EcgProbe.report(steps, 0, listOf("type=0x28 len=220"), 30) assertTrue(text.contains("DATA REQUEST REFUSED")) - assertTrue(text.contains("SELECT_WRIST(123): SUCCESS(1)")) - assertTrue(text.contains("TOGGLE_LABRADOR_DATA_GENERATION(124): FAILURE(0)")) + assertTrue(text.contains("SELECT_WRIST(123) arg=1: SUCCESS(1)")) + assertTrue(text.contains("TOGGLE_LABRADOR_DATA_GENERATION(124) arg=1: FAILURE(0)")) assertTrue(text.contains("type=0x28 len=220")) assertTrue(text.contains("aabb")) assertTrue(text.contains("not a medical measurement or a diagnosis")) } + /** + * Twin of Swift `testStartAndRestartRunsRenderDistinguishably`. A START run and a RESTART run send + * the SAME three opcodes and differ in exactly one byte, so without the argument annotation their + * reports are character-for-character identical — and a report is the artefact that gets copied out + * of an app and pasted into an issue, long after the log that recorded the payload bytes is gone. + */ + @Test + fun startAndRestartRunsRenderDistinguishably() { + fun run(control: Whoop5Ecg.ControlSignal): String = Whoop5EcgProbe.report( + listOf( + sent(139, 1, Whoop5EcgProbe.CommandOutcome.Success), + sent(125, 1, Whoop5EcgProbe.CommandOutcome.Success), + sent(124, control.raw, Whoop5EcgProbe.CommandOutcome.Success), + ), + 0, emptyList(), 30, + ) + val startText = run(Whoop5Ecg.ControlSignal.START) + val restartText = run(Whoop5Ecg.ControlSignal.RESTART) + assertTrue(startText.contains("TOGGLE_LABRADOR_DATA_GENERATION(124) arg=1: SUCCESS(1)")) + assertTrue(restartText.contains("TOGGLE_LABRADOR_DATA_GENERATION(124) arg=2: SUCCESS(1)")) + assertNotEquals(startText, restartText) + // Both are still the SAME verdict: restart asks for realtime data exactly like start does, so + // the annotation records what was sent without changing what the run is read as. + assertTrue(startText.contains("Accepted but SILENT")) + assertTrue(restartText.contains("Accepted but SILENT")) + } + + /** + * An UNSOLICITED reply has no known argument — nothing in an app layer sent it — and the report must + * say nothing rather than invent a value. + */ + @Test + fun anUnknownArgumentIsOmittedRatherThanGuessed() { + val step = Whoop5EcgProbe.Step( + label = "TOGGLE_LABRADOR_DATA_GENERATION(124)", + outcome = Whoop5EcgProbe.CommandOutcome.Success, + requestsRealtimeData = false, + ) + assertNull(step.sentArgument) + assertEquals("TOGGLE_LABRADOR_DATA_GENERATION(124)", step.labelWithArgument) + assertFalse(Whoop5EcgProbe.report(listOf(step), 0, emptyList(), 30).contains("arg=")) + } + /** * REGRESSION (#891), twin of Swift `testNoVerdictMentionsDeviceFlagsAtAll`. No verdict may name * `blockedByDeviceFlags` or a "device-flag block" AT ALL — not to assert it, and not to deny it. diff --git a/docs/PROTOCOL.md b/docs/PROTOCOL.md index 1ff7048601..7ca12a90ae 100644 --- a/docs/PROTOCOL.md +++ b/docs/PROTOCOL.md @@ -810,6 +810,18 @@ Documented turn-on order: `SELECT_WRIST` → filtered on → raw-save on → dat splits `SELECT_WRIST` into its own separately-confirmed action, because it is the only one that writes strap state outliving the session **and** its value mapping is unattested. +**The `restart` argument (124 arg `2`).** `0` and `1` are the only values any build in this repo has +ever put on a wire, which leaves `2` as the one untried argument inside the family §6 records as +non-destructive. NOOP offers it as a separate, equally-gated action that sends the SAME preamble and +changes only the control byte — filtered on → raw-save on → data generation `restart`. What the strap +does with `2` is unknown: the name is read off the client enum's order, exactly like the wrist mapping, +so there is no basis for claiming it re-arms the two channel toggles itself or for claiming it needs +them already on. Changing one byte rather than also dropping the preamble is what keeps a null result +attributable. The probe report annotates every command line with the argument it was sent with +(`TOGGLE_LABRADOR_DATA_GENERATION(124) arg=2`) precisely because the two runs are otherwise +indistinguishable in a copied report. Reversal is the ordinary `stop` sequence — this leaves behind the +same strap state a normal start does. + **Packet layouts.** Both open with the same 17-byte status block (multi-byte fields little-endian): | Off | Size | Field | From 5f6132893a7d473a30b4716f5e61c450aea2fb81 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 6 Aug 2026 11:51:16 -0400 Subject: [PATCH 165/171] MG ECG: stop discarding candidate frames, and census what arrives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The live ECG probe reported "no frame passed the structural triage" on every run. `Whoop5Ecg.plausibleFilteredPayload` ended in a length agreement that hardcoded TWO bytes per sample: let end = headerLength + n * 2 A populated RAW flash record read off hardware carries numberOfECGSamples = 500 against a 1500-byte sample blob — THREE bytes per sample. Nothing attests that the filtered stream matches the raw one, and nothing rules it out; what the hardcoded 2 did was make the question unanswerable, because a 3-byte frame misses the agreement, blows the pad4 budget, and is dropped before anyone can look at it. The raw side has enumerated widths from the start (`rawBytesPerSampleCandidates`); the filtered triage never got the same treatment. Two changes: 1. `filteredBytesPerSampleCandidates` replaces the fixed width with the enumerated set [2, 3, 4] (2 first, so the old behaviour is a strict subset) and RETURNS which widths agreed, so a width-3 candidate is not reported as if it were a width-2 one. `plausibleFilteredPayload` is now "any width agreed". Nothing else is loosened: the four Bool-typed bytes, the two classifier enums, the signal-quality range and n > 0 all still reject. The offset math is overflow-checked, as decodeRaw's is. 2. `Whoop5EcgProbe.FrameCensus` records EVERY unclassified 5/MG frame while a probe is armed — the triage's misses as well as its hits. Logging only the hits made the hunt for the (unattested) ECG type byte unfalsifiable: a rejected frame left no trace, the report truthfully said nothing passed, and the bytes that would have shown the mistake were gone. That is exactly how the 2-byte assumption stayed invisible. Per frame it keeps the type byte, frame and payload lengths, numberOfECGSamples when the header parses, the widths that agreed, and the first 64 bytes as hex; bounded at 3 recorded frames per type byte and 16 distinct type bytes, with everything past a cap COUNTED rather than dropped. It renders directly under the triage result in the probe report, where the operator can copy it out of the same sheet. The candidate line now reads the status header directly instead of going through decodeFilteredFrame, which decodes samples and so still assumes 2 bytes each — a width-3 candidate would otherwise lose its line to a decoder that cannot read its blob. No BLE command, opcode, argument or send sequence is touched: ecgStartCapture / ecgRestartCapture / ecgStopCapture are byte-for-byte unchanged, as are the Experimental opt-in, ecgGatesAllow() and the MG-family check. Kotlin twins mirrored (Whoop5Ecg.kt, Whoop5EcgProbe.kt) per the parity contract. Android has no ECG app layer, so only the protocol-level halves exist there, covered by their own tests. Verified: swift test (555 pass), xcodebuild Strand macOS test (1054 pass), xcodebuild NOOPiOS build, gradlew testFullDebugUnitTest (3501 pass). --- .../Sources/WhoopProtocol/Whoop5Ecg.swift | 78 ++++++- .../WhoopProtocol/Whoop5EcgProbe.swift | 190 +++++++++++++++++- .../WhoopProtocolTests/Whoop5EcgTests.swift | 185 +++++++++++++++++ Strand/BLE/BLEManager.swift | 44 +++- .../main/java/com/noop/protocol/Whoop5Ecg.kt | 75 ++++++- .../java/com/noop/protocol/Whoop5EcgProbe.kt | 176 ++++++++++++++++ .../com/noop/protocol/Whoop5EcgProbeTest.kt | 129 ++++++++++++ .../java/com/noop/protocol/Whoop5EcgTest.kt | 94 +++++++++ 8 files changed, 938 insertions(+), 33 deletions(-) diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Ecg.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Ecg.swift index 18f7038c37..c5a8a08ce1 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Ecg.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Ecg.swift @@ -23,7 +23,13 @@ import Foundation // What is NOT established, and is therefore never asserted: // • The packet TYPE byte these records arrive under. No capture exists, and the repo's `PacketType` // table has no Labrador entry. So this file decodes a PAYLOAD, and the app layer discovers the type -// empirically by running `plausibleFilteredPayload` over unclassified frames and logging the hits. +// empirically: it runs `filteredBytesPerSampleCandidates` over unclassified frames, and — because a +// heuristic that only logs its own hits destroys the evidence for its own misses — CENSUSES every +// unclassified frame by type byte whether the triage passed or not (`Whoop5EcgProbe.FrameCensus`). +// • The filtered stream's BYTES PER SAMPLE. `decodeFiltered` implements 2, which is what the filtered +// layout is documented with; the triage admits 2, 3 and 4, because a populated RAW record read off +// hardware turned out to be 3 bytes/sample and a hardcoded 2 in the triage silently discarded every +// frame that disagreed. See `filteredWidthCandidates`. // • The `WristSelection` raw values. `right` is listed first in the client's enum, so right=0/left=1 is // the natural reading — but it is an INFERENCE, not an attested fact, and it is labelled as such // everywhere it surfaces (including in the UI, because picking the wrong one writes the wrong @@ -488,24 +494,74 @@ public enum Whoop5Ecg { // MARK: Discovery + /// The bytes-per-sample widths the FILTERED triage admits, in the order it reports them. + /// + /// 2 leads because it is the width `decodeFiltered` implements and the width the filtered stream is + /// documented with, so the triage's existing behaviour is a strict subset of the widened one. + /// + /// 3 and 4 are here because the 2-byte assumption was, on this repo's own evidence, unsafe. A + /// populated RAW flash record read off real hardware carried `numberOfECGSamples = 500` against a + /// 1500-byte sample blob — **3 bytes per sample**. Nothing attests that the FILTERED stream uses the + /// same width, and nothing rules it out either; what a hardcoded 2 does is make the difference + /// unobservable, because a 3-byte frame fails the length agreement and is discarded before anyone can + /// look at it. `rawBytesPerSampleCandidates` has enumerated widths on the raw side from the start; + /// this is the filtered side getting the same treatment. + /// + /// Widening the WIDTH is the only loosening: the four Bool-typed bytes, the two classifier enums, the + /// signal-quality range and `n > 0` are all unchanged, so a buffer that failed those still fails. + public static let filteredWidthCandidates = [2, 3, 4] + + /// Every width in `widths` under which `payload` could be a filtered Labrador payload — i.e. the + /// widths whose length agreement holds, once the buffer has passed the field guards. + /// + /// Empty means "not a filtered payload under any admitted width", and is the triage's rejection. More + /// than one width means the buffer genuinely does not determine the answer; like + /// `rawBytesPerSampleCandidates` this REPORTS the ambiguity rather than picking a favourite. + public static func filteredBytesPerSampleCandidates(_ payload: [UInt8], + widths: [Int] = filteredWidthCandidates, + maxPadding: Int = defaultMaxPadding) -> [Int] { + guard let header = EcgStatusHeader(payload: payload) else { return [] } + guard header.signalQualityRaw <= 3, + header.heartKeyArrhythmiaCheckResult != nil, + header.heartKeyArrhythmiaCheckStatus != nil else { return [] } + // The four booleans are Bool-typed on the wire, so anything but 0/1 rules the buffer out. + guard payload[2] <= 1, payload[3] <= 1, payload[4] <= 1, payload[5] <= 1 else { return [] } + let n = Int(header.numberOfECGSamples) + guard n > 0 else { return [] } + return widths.filter { width in + // `widths` is caller-supplied and `n` comes off the wire, so both the multiply and the add are + // checked — the same discipline `decodeRaw` uses, for the same reason: either would trap in + // Swift and wrap NEGATIVE in the Kotlin twin. + guard width > 0 else { return false } + let (blobLength, mulOverflow) = n.multipliedReportingOverflow(by: width) + guard !mulOverflow else { return false } + let (end, addOverflow) = headerLength.addingReportingOverflow(blobLength) + guard !addOverflow else { return false } + return end <= payload.count && payload.count - end <= maxPadding + } + } + + /// The frame-level form of `filteredBytesPerSampleCandidates`, CRC-gated. + public static func filteredBytesPerSampleCandidates(frame: [UInt8], + payloadStart: Int = puffinPayloadStart, + widths: [Int] = filteredWidthCandidates, + maxPadding: Int = defaultMaxPadding) -> [Int] { + guard let payload = innerPayload(frame, payloadStart: payloadStart) else { return [] } + return filteredBytesPerSampleCandidates(payload, widths: widths, maxPadding: maxPadding) + } + /// A cheap structural triage for "could these bytes be a filtered Labrador payload?". /// /// Used by the app layer to hunt for the packet TYPE byte, which is not attested: while an ECG probe /// is armed, every unclassified 5/MG frame is run through this and the hits are logged with their /// type. It is a HEURISTIC — four booleans, three enum ranges and a length agreement — not a /// classifier, and nothing downstream may treat a hit as proof. + /// + /// A pass under ANY width in `filteredWidthCandidates` is a pass; callers that need to know WHICH + /// width agreed call `filteredBytesPerSampleCandidates` instead of re-deriving it. public static func plausibleFilteredPayload(_ payload: [UInt8], maxPadding: Int = defaultMaxPadding) -> Bool { - guard let header = EcgStatusHeader(payload: payload) else { return false } - guard header.signalQualityRaw <= 3, - header.heartKeyArrhythmiaCheckResult != nil, - header.heartKeyArrhythmiaCheckStatus != nil else { return false } - // The four booleans are Bool-typed on the wire, so anything but 0/1 rules the buffer out. - guard payload[2] <= 1, payload[3] <= 1, payload[4] <= 1, payload[5] <= 1 else { return false } - let n = Int(header.numberOfECGSamples) - guard n > 0 else { return false } - let end = headerLength + n * 2 - return end <= payload.count && payload.count - end <= maxPadding + !filteredBytesPerSampleCandidates(payload, maxPadding: maxPadding).isEmpty } /// The frame-level form of `plausibleFilteredPayload`, CRC-gated. This is what the app layer runs diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5EcgProbe.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5EcgProbe.swift index 05d3a0b22c..3be83ee269 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5EcgProbe.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5EcgProbe.swift @@ -155,6 +155,173 @@ public enum Whoop5EcgProbe { } } + // MARK: - Unclassified-frame census + + /// A bounded census of EVERY unclassified 5/MG frame seen while a probe run is armed — the ones the + /// structural triage rejected as well as the ones it accepted. + /// + /// ## Why the rejects are the point + /// + /// The packet TYPE byte the Labrador records arrive under is NOT attested (see `Whoop5Ecg`'s file + /// header), so the probe hunts for it with a shape heuristic. Logging only the heuristic's HITS makes + /// that hunt unfalsifiable: when the heuristic is wrong, the frame it was wrong about is discarded, + /// the report truthfully says "no frame passed the structural triage", and the bytes that would have + /// shown the mistake are gone. That is exactly what a hardcoded 2-bytes-per-sample length agreement + /// did to every 3-byte frame before `Whoop5Ecg.filteredWidthCandidates` widened it — and the next + /// wrong assumption will be invisible the same way unless the raw census exists beside the verdict. + /// + /// So this records the type byte, the frame and payload lengths, `numberOfECGSamples` when the header + /// parses, which sample widths the length agreed with, and the frame's opening bytes as hex. It makes + /// no claim about any of them: it is a census, and every line in it is a frame the probe could not + /// classify. + /// + /// ## Bounds + /// + /// A 5/MG link carries ordinary live traffic at ~1 Hz, and a report is a copyable string, so the + /// census is capped in both directions: at most `maxSamplesPerType` (3) recorded frames per distinct + /// type byte, and at most `maxTypes` (16) distinct type bytes. Everything past a cap is COUNTED, never + /// silently dropped — `count` per type keeps rising after its samples are full, and frames whose type + /// byte arrives after the 16th distinct one land in `framesBeyondTypeCap`. Worst case is 48 recorded + /// frames × 64 hex-rendered bytes, which is bounded regardless of what the strap does. + public struct FrameCensus: Equatable, Sendable { + + /// Recorded frames per distinct type byte. Small on purpose: three examples of a type byte answer + /// "what does this look like", and the running count answers "how often". + public static let maxSamplesPerType = 3 + /// Distinct type bytes tracked. A type byte is one octet, so 256 buckets are possible in theory; + /// 16 is well past what a live link actually shows and keeps the report readable. + public static let maxTypes = 16 + /// Leading bytes of each recorded frame rendered as hex. Enough to carry the whole envelope plus + /// the 17-byte status header and the first samples. + public static let headBytes = 64 + + /// One recorded frame. Nothing here is interpreted — these are measurements of the buffer. + public struct Sample: Equatable, Sendable { + public let frameLength: Int + /// Length of the inner record's payload, or nil when the frame did not yield one (it failed a + /// CRC or was too short for the envelope). + public let payloadLength: Int? + /// `numberOfECGSamples` as the status header would read it, or nil when the payload is too + /// short to hold a header. Reading it does NOT claim this frame is an ECG record. + public let numberOfECGSamples: Int? + /// The bytes-per-sample widths the payload's length agreed with, per + /// `Whoop5Ecg.filteredBytesPerSampleCandidates`. Empty means the triage rejected the frame, + /// which is the case this census exists to preserve. + public let widths: [Int] + /// The frame's first `headBytes` bytes, lowercase hex. + public let headHex: String + + public init(frameLength: Int, payloadLength: Int?, numberOfECGSamples: Int?, + widths: [Int], headHex: String) { + self.frameLength = frameLength + self.payloadLength = payloadLength + self.numberOfECGSamples = numberOfECGSamples + self.widths = widths + self.headHex = headHex + } + + public var line: String { + let payload = payloadLength.map(String.init) ?? "?" + let samples = numberOfECGSamples.map(String.init) ?? "?" + let widthText = widths.isEmpty ? "none" : widths.map(String.init).joined(separator: ",") + return "len=\(frameLength) payload=\(payload) samples=\(samples) widths=\(widthText) head=\(headHex)" + } + } + + /// Everything seen under one type byte. + public struct Bucket: Equatable, Sendable { + public let typeByte: UInt8 + /// Every frame seen with this type byte, including the ones past the per-type sample cap. + public var count: Int + public var samples: [Sample] + + public init(typeByte: UInt8, count: Int, samples: [Sample]) { + self.typeByte = typeByte + self.count = count + self.samples = samples + } + } + + /// Buckets in FIRST-SEEN order, which is the order the report prints them in. Insertion order is + /// kept rather than sorted by count so the census reads as a timeline of what the link did. + public private(set) var buckets: [Bucket] = [] + /// Every frame offered to the census, including those past a cap. + public private(set) var framesSeen = 0 + /// Frames dropped because their type byte was the 17th distinct one. Counted, not hidden. + public private(set) var framesBeyondTypeCap = 0 + + public init() {} + + /// Fold one frame into the census. + /// + /// The caller CRC-gates the frame first (the BLE safety contract's inbound rule); this reads the + /// type byte directly and gets the payload through `Whoop5Ecg.innerPayload`, which re-checks both + /// CRCs itself, so a frame that somehow arrives unverified yields `payloadLength == nil` rather + /// than a decoded field. Frames with no type byte at all (fewer than 9 bytes — shorter than the + /// puffin envelope's minimum) are not records and are ignored. + public mutating func record(frame: [UInt8], + payloadStart: Int = Whoop5Ecg.puffinPayloadStart, + maxPadding: Int = Whoop5Ecg.defaultMaxPadding) { + guard frame.count > 8 else { return } + framesSeen += 1 + let typeByte = frame[8] + guard let index = buckets.firstIndex(where: { $0.typeByte == typeByte }) else { + guard buckets.count < FrameCensus.maxTypes else { + framesBeyondTypeCap += 1 + return + } + buckets.append(Bucket(typeByte: typeByte, + count: 1, + samples: [FrameCensus.sample(frame: frame, + payloadStart: payloadStart, + maxPadding: maxPadding)])) + return + } + buckets[index].count += 1 + guard buckets[index].samples.count < FrameCensus.maxSamplesPerType else { return } + buckets[index].samples.append(FrameCensus.sample(frame: frame, + payloadStart: payloadStart, + maxPadding: maxPadding)) + } + + /// Measure one frame. Public so a test can build a `Sample` without a census. + public static func sample(frame: [UInt8], + payloadStart: Int = Whoop5Ecg.puffinPayloadStart, + maxPadding: Int = Whoop5Ecg.defaultMaxPadding) -> Sample { + let payload = Whoop5Ecg.innerPayload(frame, payloadStart: payloadStart) + let widths = payload.map { + Whoop5Ecg.filteredBytesPerSampleCandidates($0, maxPadding: maxPadding) + } ?? [] + let samples = payload.flatMap { EcgStatusHeader(payload: $0) } + .map { Int($0.numberOfECGSamples) } + let head = frame.prefix(headBytes).map { String(format: "%02x", $0) }.joined() + return Sample(frameLength: frame.count, + payloadLength: payload?.count, + numberOfECGSamples: samples, + widths: widths, + headHex: head) + } + + public var isEmpty: Bool { framesSeen == 0 } + + /// The census as report lines, already indented for `report`. + public var lines: [String] { + var out: [String] = [] + for bucket in buckets { + out.append(String(format: " type=0x%02x frames=%d", Int(bucket.typeByte), bucket.count)) + for sample in bucket.samples { out.append(" \(sample.line)") } + if bucket.count > bucket.samples.count { + out.append(" (+\(bucket.count - bucket.samples.count) more of this type, not recorded)") + } + } + if framesBeyondTypeCap > 0 { + out.append(" (\(framesBeyondTypeCap) frame(s) of further type bytes past the " + + "\(FrameCensus.maxTypes)-type cap, counted only)") + } + return out + } + } + /// The gate verdict, kept separate from the report text so it is assertable in a test. public enum Verdict: Equatable, Sendable { /// Frames that PASS THE STRUCTURAL TRIAGE arrived. Not proof: the triage is a heuristic over @@ -272,10 +439,16 @@ public enum Whoop5EcgProbe { /// `candidateFrames` are the type/length lines for frames that passed the structural triage in /// `Whoop5Ecg.plausibleFilteredPayload` — the empirical answer to "which packet type do these arrive /// under", which no table in this repo yet holds. + /// + /// `census` is the same question asked WITHOUT the heuristic: every unclassified frame the run saw, + /// bucketed by type byte, hits and misses alike. It defaults to empty so a caller that has none still + /// renders, but a live run always passes one — see `FrameCensus` for why the misses matter more than + /// the hits. public static func report(steps: [Step], ecgPacketsSeen: Int, candidateFrames: [String], - windowSeconds: Int) -> String { + windowSeconds: Int, + census: FrameCensus = FrameCensus()) -> String { var sb = "" sb += "WHOOP MG ECG (Labrador) TURN-ON PROBE\n" sb += "Verdict: \(verdict(steps: steps, ecgPacketsSeen: ecgPacketsSeen, windowSeconds: windowSeconds).headline)\n" @@ -312,6 +485,21 @@ public enum Whoop5EcgProbe { sb += "Candidate packet types (structural triage only, NOT a confirmed mapping):\n" for line in candidateFrames { sb += " \(line)\n" } } + // The census goes IMMEDIATELY under the triage result, because it is the check on it: "no frame + // passed the structural triage" is only informative next to what the frames actually were. + if census.isEmpty { + sb += "Unclassified-frame census: no unclassified frame arrived at all — the triage had " + + "nothing to reject.\n" + } else { + sb += "Unclassified-frame census — EVERY frame this run could not classify, triage hits and " + + "misses alike, bucketed by the inner record's type byte. The ECG type byte is not " + + "attested, so this, not the triage, is the record of what arrived. widths= are the " + + "bytes-per-sample the payload length agreed with; none = the triage rejected it. " + + "Recorded: \(census.framesSeen) frame(s), first \(FrameCensus.maxSamplesPerType) per " + + "type byte, first \(FrameCensus.maxTypes) type bytes, first \(FrameCensus.headBytes) " + + "bytes each:\n" + for line in census.lines { sb += "\(line)\n" } + } let replies = steps.compactMap { step in step.replyHex.map { " \(step.labelWithArgument): \($0)" } } if !replies.isEmpty { sb += "\nRaw replies:\n" diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift index 8f180eb25c..fbb89f07b8 100644 --- a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift @@ -376,6 +376,191 @@ final class Whoop5EcgTests: XCTestCase { XCTAssertFalse(Whoop5Ecg.plausibleFilteredPayload([UInt8](repeating: 0, count: 64))) } + // MARK: - Structural triage: sample width + + /// THE DEFECT this widening fixes. The triage hardcoded `n * 2`, so a payload whose samples are 3 + /// bytes wide missed the length agreement by a mile, failed the padding budget, and was discarded + /// without a trace — while the probe truthfully reported that no frame passed. The raw side has + /// enumerated widths since it was written (`rawBytesPerSampleCandidates`), and a populated raw flash + /// record read off hardware was 1500 bytes against `numberOfECGSamples = 500`, i.e. 3. + func testTriageAcceptsAThreeBytePerSampleBufferAndNamesTheWidth() { + let payload = header(samples: 4) + [UInt8](repeating: 0x11, count: 12) // 4 samples × 3 bytes + XCTAssertEqual(Whoop5Ecg.filteredBytesPerSampleCandidates(payload), [3]) + XCTAssertTrue(Whoop5Ecg.plausibleFilteredPayload(payload)) + } + + /// The widening is a SUPERSET, never a replacement: 2 stays first in the candidate order and a + /// 2-byte buffer still passes exactly as it did. + func testTriageStillAcceptsTwoBytesPerSampleAndReportsItFirst() { + let payload = header(samples: 3) + i16le([1, 2, 3]) + XCTAssertEqual(Whoop5Ecg.filteredBytesPerSampleCandidates(payload), [2]) + XCTAssertTrue(Whoop5Ecg.plausibleFilteredPayload(payload)) + XCTAssertEqual(Whoop5Ecg.filteredWidthCandidates.first, 2) + } + + func testTriageAcceptsFourBytesPerSample() { + let payload = header(samples: 5) + [UInt8](repeating: 0x22, count: 20) + XCTAssertEqual(Whoop5Ecg.filteredBytesPerSampleCandidates(payload), [4]) + } + + /// When two widths both fit inside the pad4 budget the buffer genuinely does not determine the + /// answer, and the triage says so rather than picking one — the same contract + /// `rawBytesPerSampleCandidates` has. + func testTriageReportsEveryWidthAnAmbiguousBufferAdmits() { + let payload = header(samples: 2) + [UInt8](repeating: 0x33, count: 8) // 25 bytes total + XCTAssertEqual(Whoop5Ecg.filteredBytesPerSampleCandidates(payload), [3, 4]) + } + + /// Only the WIDTH assumption moved. Every other guard is unchanged, so a buffer that fails a field + /// check is still rejected even when its length agrees perfectly with one of the new widths. + func testWideningTheWidthDoesNotLoosenAnyOtherGuard() { + let threeByteBody = [UInt8](repeating: 0x11, count: 12) // exact under width 3 + XCTAssertEqual(Whoop5Ecg.filteredBytesPerSampleCandidates(header(samples: 4) + threeByteBody), [3]) + // …and each guard, one at a time, still rules it out. + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(started: 7, samples: 4) + threeByteBody).isEmpty) + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(running: 2, samples: 4) + threeByteBody).isEmpty) + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(stoppedAndComplete: 9, samples: 4) + threeByteBody).isEmpty) + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(leadsOn: 3, samples: 4) + threeByteBody).isEmpty) + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(signalQuality: 9, samples: 4) + threeByteBody).isEmpty) + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(arrhythmiaResult: 9, samples: 4) + threeByteBody).isEmpty) + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(arrhythmiaStatus: 9, samples: 4) + threeByteBody).isEmpty) + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(samples: 0)).isEmpty) + } + + /// A genuinely malformed buffer is still rejected under EVERY admitted width — the widening must not + /// turn the triage into something that matches anything. + func testTriageStillRejectsBuffersNoWidthExplains() { + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(samples: 7) + i16le([1, 2, 3])).isEmpty) + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(samples: 40) + i16le([1, 2])).isEmpty) + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates( + header(samples: 1) + i16le([1]) + [UInt8](repeating: 0, count: 40)).isEmpty) + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates([]).isEmpty) + XCTAssertFalse(Whoop5Ecg.plausibleFilteredPayload(header(samples: 7) + i16le([1, 2, 3]))) + } + + /// `numberOfECGSamples` is a u16 off the wire and the width comes from a caller, so the offset math + /// is overflow-checked rather than trusted — it would trap in Swift and wrap NEGATIVE in the Kotlin + /// twin. A rejection is the correct outcome, not a crash. + func testTriageWidthMathCannotOverflow() { + let payload = header(samples: 65_535) + [UInt8](repeating: 0, count: 8) + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(payload, widths: [Int.max, 2, 3]).isEmpty) + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(payload, widths: [0, -1]).isEmpty) + } + + func testTriageWidthsAreReportedThroughACRCGatedFrameToo() { + let payload = header(samples: 4) + [UInt8](repeating: 0x11, count: 12) + var frame = puffinFrame(type: 0x28, payload: payload) + XCTAssertEqual(Whoop5Ecg.filteredBytesPerSampleCandidates(frame: frame), [3]) + frame[frame.count - 1] ^= 0xFF // a bad CRC never reaches a field read + XCTAssertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(frame: frame).isEmpty) + } + + // MARK: - Unclassified-frame census + + /// The census exists for the frames the triage says NO to. A heuristic that logs only its own hits + /// destroys the evidence for its own misses: the 2-byte assumption above discarded every 3-byte + /// frame, and the report then said, truthfully, that nothing passed. + func testCensusRecordsAFrameTheTriageRejects() { + // signalQualityRaw = 0xEE fails the very first guard, so the triage rejects this outright. + let frame = puffinFrame(type: 0x1A, payload: [UInt8](repeating: 0xEE, count: 20)) + XCTAssertFalse(Whoop5Ecg.plausibleFilteredFrame(frame), "fixture must be a triage MISS") + + var census = Whoop5EcgProbe.FrameCensus() + census.record(frame: frame) + XCTAssertEqual(census.framesSeen, 1) + XCTAssertEqual(census.buckets.count, 1) + XCTAssertEqual(census.buckets[0].typeByte, 0x1A) + XCTAssertEqual(census.buckets[0].count, 1) + + let sample = census.buckets[0].samples[0] + XCTAssertEqual(sample.frameLength, frame.count) + XCTAssertEqual(sample.payloadLength, 21) // 20 + the envelope's pad4 byte + XCTAssertEqual(sample.numberOfECGSamples, 0xEEEE) // read, not believed + XCTAssertTrue(sample.widths.isEmpty) + XCTAssertTrue(sample.headHex.hasPrefix("aa")) + XCTAssertEqual(census.lines[0], " type=0x1a frames=1") + XCTAssertTrue(census.lines[1].contains("widths=none")) + XCTAssertTrue(census.lines[1].contains("payload=21")) + } + + /// A HIT is censused too — the census is the whole record of the window, not a rejects bin. + func testCensusRecordsTriageHitsWithTheWidthThatAgreed() { + let frame = puffinFrame(type: 0x28, payload: header(samples: 4) + [UInt8](repeating: 0x11, count: 12)) + XCTAssertTrue(Whoop5Ecg.plausibleFilteredFrame(frame)) + var census = Whoop5EcgProbe.FrameCensus() + census.record(frame: frame) + XCTAssertEqual(census.buckets[0].samples[0].widths, [3]) + XCTAssertEqual(census.buckets[0].samples[0].numberOfECGSamples, 4) + XCTAssertTrue(census.lines[1].contains("widths=3")) + } + + /// A frame whose CRC does not check out yields no decoded field — the census records its shape and + /// its bytes, and says `payload=?` rather than reading a header out of an unverified buffer. + func testCensusReadsNoFieldOutOfAnUnverifiedFrame() { + var frame = puffinFrame(type: 0x28, payload: header(samples: 3) + i16le([1, 2, 3])) + frame[frame.count - 1] ^= 0xFF + var census = Whoop5EcgProbe.FrameCensus() + census.record(frame: frame) + XCTAssertNil(census.buckets[0].samples[0].payloadLength) + XCTAssertNil(census.buckets[0].samples[0].numberOfECGSamples) + XCTAssertTrue(census.buckets[0].samples[0].widths.isEmpty) + XCTAssertTrue(census.lines[1].contains("payload=? samples=? widths=none")) + } + + /// A 1 Hz stream must not be able to grow the report without bound — but nothing may be dropped in + /// silence either. Past the per-type sample cap the frames are COUNTED. + func testCensusCapsSamplesPerTypeButKeepsCounting() { + let frame = puffinFrame(type: 0x30, payload: [UInt8](repeating: 0xEE, count: 20)) + var census = Whoop5EcgProbe.FrameCensus() + for _ in 0..<50 { census.record(frame: frame) } + XCTAssertEqual(census.framesSeen, 50) + XCTAssertEqual(census.buckets.count, 1) + XCTAssertEqual(census.buckets[0].count, 50) + XCTAssertEqual(census.buckets[0].samples.count, Whoop5EcgProbe.FrameCensus.maxSamplesPerType) + XCTAssertTrue(census.lines.contains { $0.contains("+47 more of this type, not recorded") }) + } + + func testCensusCapsDistinctTypesAndCountsTheOverflow() { + var census = Whoop5EcgProbe.FrameCensus() + for type in 0..<(Whoop5EcgProbe.FrameCensus.maxTypes + 4) { + census.record(frame: puffinFrame(type: UInt8(type), payload: [UInt8](repeating: 0xEE, count: 20))) + } + XCTAssertEqual(census.buckets.count, Whoop5EcgProbe.FrameCensus.maxTypes) + XCTAssertEqual(census.framesBeyondTypeCap, 4) + XCTAssertEqual(census.framesSeen, Whoop5EcgProbe.FrameCensus.maxTypes + 4) + XCTAssertTrue(census.lines.contains { $0.contains("4 frame(s) of further type bytes past the") }) + } + + func testCensusIgnoresBuffersTooShortToCarryATypeByte() { + var census = Whoop5EcgProbe.FrameCensus() + census.record(frame: []) + census.record(frame: [UInt8](repeating: 0xAA, count: 8)) + XCTAssertTrue(census.isEmpty) + XCTAssertTrue(census.buckets.isEmpty) + } + + /// The census is only useful if the operator can COPY it: it belongs in the same report sheet as the + /// verdict, right under the triage result it is the check on. + func testReportCarriesTheCensusBesideTheTriageResult() { + let rejected = puffinFrame(type: 0x1A, payload: [UInt8](repeating: 0xEE, count: 20)) + var census = Whoop5EcgProbe.FrameCensus() + census.record(frame: rejected) + let text = Whoop5EcgProbe.report( + steps: [sent(139, arg: 1, .success), sent(124, arg: Whoop5Ecg.ControlSignal.start.rawValue, .success)], + ecgPacketsSeen: 0, candidateFrames: [], windowSeconds: 30, census: census) + // The old, evidence-free version of this run said only this line. Now the bytes are beside it. + XCTAssertTrue(text.contains("Candidate packet types: none — no frame passed the structural triage.")) + XCTAssertTrue(text.contains("Unclassified-frame census")) + XCTAssertTrue(text.contains("type=0x1a frames=1")) + XCTAssertTrue(text.contains("widths=none")) + XCTAssertTrue(text.contains(census.buckets[0].samples[0].headHex)) + } + + func testReportSaysSoWhenNoUnclassifiedFrameArrivedAtAll() { + let text = Whoop5EcgProbe.report(steps: [], ecgPacketsSeen: 0, candidateFrames: [], windowSeconds: 30) + XCTAssertTrue(text.contains("no unclassified frame arrived at all")) + } + // MARK: - Commands func testCommandOpcodesMatchTheRepoProtocolTable() { diff --git a/Strand/BLE/BLEManager.swift b/Strand/BLE/BLEManager.swift index 9de6b45911..624a6fe314 100644 --- a/Strand/BLE/BLEManager.swift +++ b/Strand/BLE/BLEManager.swift @@ -814,6 +814,9 @@ public final class BLEManager: NSObject, ObservableObject { /// Structural-triage hits: the empirical search for the packet TYPE these records arrive under. private var ecgProbeCandidates: [String] = [] private var ecgProbePacketsSeen = 0 + /// Every unclassified frame the run saw, triage MISSES included — the record that survives a wrong + /// heuristic. See `Whoop5EcgProbe.FrameCensus`. + private var ecgProbeCensus = Whoop5EcgProbe.FrameCensus() /// Non-nil while the listen window is open; drives `ecgProbeArmed`. private var ecgProbeDeadline: Date? /// Supersedes a previous run's pending verdict timer when the user taps again. @@ -3338,6 +3341,7 @@ public final class BLEManager: NSObject, ObservableObject { ecgProbeSteps = [] ecgProbeCandidates = [] ecgProbePacketsSeen = 0 + ecgProbeCensus = Whoop5EcgProbe.FrameCensus() } ecgProbeRunToken &+= 1 ecgProbeDeadline = Date().addingTimeInterval(BLEManager.ecgProbeWindow) @@ -3354,7 +3358,8 @@ public final class BLEManager: NSObject, ObservableObject { let text = Whoop5EcgProbe.report(steps: self.ecgProbeSteps, ecgPacketsSeen: self.ecgProbePacketsSeen, candidateFrames: self.ecgProbeCandidates, - windowSeconds: Int(BLEManager.ecgProbeWindow)) + windowSeconds: Int(BLEManager.ecgProbeWindow), + census: self.ecgProbeCensus) self.log("ECG probe:\n\(text)") self.state.ecgProbe = text } @@ -3419,21 +3424,37 @@ public final class BLEManager: NSObject, ObservableObject { /// Structural triage for the packet TYPE the ECG records arrive under, which no table in this repo /// holds. A hit is a CANDIDATE, never a confirmed mapping. private func noteEcgProbeCandidate(_ frame: [UInt8]) { - guard frame.count >= 12, Whoop5Ecg.plausibleFilteredFrame(frame) else { return } + // CENSUS FIRST, unconditionally — before the heuristic gets a vote. Every frame that reaches here + // is one the run could not classify, and the ones the triage REJECTS are the ones whose bytes + // would otherwise be gone: a 2-bytes-per-sample length assumption discarded every 3-byte frame in + // silence, and the report then truthfully said no frame passed. The census is what makes the next + // wrong assumption visible instead of invisible. (The caller has already CRC-gated the frame.) + ecgProbeCensus.record(frame: frame) + guard frame.count >= 12 else { return } + // Which sample widths the length agreed with — recorded rather than re-derived, because a + // candidate under width 3 is a different finding from a candidate under width 2 and the report + // must not flatten them together. + let widths = Whoop5Ecg.filteredBytesPerSampleCandidates(frame: frame) + guard !widths.isEmpty else { return } ecgProbePacketsSeen += 1 + // The header is read directly rather than through `decodeFilteredFrame`, which decodes SAMPLES and + // therefore still assumes 2 bytes each: a width-3 candidate would fail it and lose its line while + // its status header parses perfectly. Nothing on this line comes from the sample blob. guard ecgProbeCandidates.count < BLEManager.ecgProbeMaxCandidates, - let packet = Whoop5Ecg.decodeFilteredFrame(frame) else { return } - let header = packet.header + let payload = Whoop5Ecg.innerPayload(frame), + let header = EcgStatusHeader(payload: payload) else { return } // The classifier byte is logged as a NUMBER, never as its token name: a strap log is a shareable // artefact, and no line in it should read like a clinical finding. The decoder maps the value // offline; the raw byte is lossless. - let line = String(format: "type=0x%02x len=%d samples=%d quality=%d leadsOn=%d hr=%d hrv=%d " - + "classifierRaw=%d statusRaw=%d (unvalidated instrumentation, not a diagnosis)", - Int(frame[8]), frame.count, Int(header.numberOfECGSamples), - Int(header.signalQualityRaw), header.heartKeyLeadsAreOn ? 1 : 0, - Int(header.heartKeyHR), Int(header.heartKeyHRV), - Int(header.heartKeyArrhythmiaCheckResultRaw), - Int(header.heartKeyArrhythmiaCheckStatusRaw)) + let shape = String(format: "type=0x%02x len=%d samples=%d", + Int(frame[8]), frame.count, Int(header.numberOfECGSamples)) + let status = String(format: "quality=%d leadsOn=%d hr=%d hrv=%d classifierRaw=%d statusRaw=%d", + Int(header.signalQualityRaw), header.heartKeyLeadsAreOn ? 1 : 0, + Int(header.heartKeyHR), Int(header.heartKeyHRV), + Int(header.heartKeyArrhythmiaCheckResultRaw), + Int(header.heartKeyArrhythmiaCheckStatusRaw)) + let line = "\(shape) widths=\(widths.map(String.init).joined(separator: ",")) \(status) " + + "(unvalidated instrumentation, not a diagnosis)" ecgProbeCandidates.append(line) log("ECG probe: candidate packet \(line)") // Dump the raw bytes of the FIRST few candidates so a plain strap-log export is enough to redo @@ -4506,6 +4527,7 @@ extension BLEManager: @preconcurrency CBCentralManagerDelegate { ecgProbeSteps = [] ecgProbeCandidates = [] ecgProbePacketsSeen = 0 + ecgProbeCensus = Whoop5EcgProbe.FrameCensus() // The DIS strings belong to the link that just dropped; a stale variant must not keep an MG-only // capability unlocked for whatever connects next. state.whoop5Variant = nil diff --git a/android/app/src/main/java/com/noop/protocol/Whoop5Ecg.kt b/android/app/src/main/java/com/noop/protocol/Whoop5Ecg.kt index f1435a5cf8..b985e0f226 100644 --- a/android/app/src/main/java/com/noop/protocol/Whoop5Ecg.kt +++ b/android/app/src/main/java/com/noop/protocol/Whoop5Ecg.kt @@ -16,7 +16,11 @@ package com.noop.protocol // reimplemented here in NOOP's own code — facts with attribution, never copied expression. // // Deliberately NOT asserted: the packet TYPE byte these records arrive under (no capture exists, and the -// PacketType table has no Labrador entry), the WristSelection raw values (right-first is an inference), +// PacketType table has no Labrador entry — an app layer finds it empirically, running +// filteredBytesPerSampleCandidates over unclassified frames and censusing EVERY one of them by type byte, +// hits and misses alike, because a heuristic that logs only its own hits destroys the evidence for its own +// misses), the filtered stream's bytes-per-sample (decodeFiltered implements 2; the triage admits 2, 3 and +// 4 — see FILTERED_WIDTH_CANDIDATES), the WristSelection raw values (right-first is an inference), // the heartKeyProgress "timed out" sentinel, and any clinical meaning at all. The arrhythmia result is // computed on-strap by an embedded third-party classifier; NOOP decodes the byte. NOOP is not a medical // device and this value is not a diagnosis — see DISCLAIMER.md. @@ -137,6 +141,23 @@ object Whoop5Ecg { /** Trailing bytes tolerated after the last decoded field (the puffin pad4 budget). */ const val DEFAULT_MAX_PADDING = 3 + /** + * The bytes-per-sample widths the FILTERED triage admits, in the order it reports them. + * + * 2 leads because it is the width [decodeFiltered] implements and the width the filtered layout is + * documented with, so the triage's earlier behaviour is a strict subset of the widened one. + * + * 3 and 4 are here because the 2-byte assumption was, on this repo's own evidence, unsafe: a populated + * RAW flash record read off hardware carried `numberOfECGSamples = 500` against a 1500-byte sample + * blob — 3 bytes per sample. Nothing attests that the FILTERED stream uses the same width and nothing + * rules it out; what a hardcoded 2 did was make the difference unobservable, because a 3-byte frame + * failed the length agreement and was discarded before anyone could look at it. + * + * Only the WIDTH is loosened — the four Bool-typed bytes, the two classifier enums, the + * signal-quality range and `n > 0` are unchanged. + */ + val FILTERED_WIDTH_CANDIDATES = listOf(2, 3, 4) + // Commands. All four share the shape {revision, arg, padding}; `revision` is the leading inner byte // the 5/MG command family already uses (CLIENT_HELLO, SET_CONFIG), and the struct's trailing padding // is exactly what the puffin pad4 supplies. @@ -380,17 +401,51 @@ object Whoop5Ecg { * armed, every unclassified 5/MG frame is run through this and the hits are logged with their type. It * is a HEURISTIC — four booleans, three enum ranges and a length agreement — not a classifier, and * nothing downstream may treat a hit as proof. + * + * A pass under ANY width in [FILTERED_WIDTH_CANDIDATES] is a pass; callers that need to know WHICH + * width agreed call [filteredBytesPerSampleCandidates] instead of re-deriving it. + */ + fun plausibleFilteredPayload(payload: List, maxPadding: Int = DEFAULT_MAX_PADDING): Boolean = + filteredBytesPerSampleCandidates(payload, maxPadding = maxPadding).isNotEmpty() + + /** + * Every width in [widths] under which [payload] could be a filtered Labrador payload — the widths + * whose length agreement holds once the buffer has passed the field guards. + * + * Empty is the triage's rejection. More than one width means the buffer genuinely does not determine + * the answer; like [rawBytesPerSampleCandidates] this REPORTS the ambiguity rather than picking one. */ - fun plausibleFilteredPayload(payload: List, maxPadding: Int = DEFAULT_MAX_PADDING): Boolean { - val header = decodeHeader(payload) ?: return false - if (header.signalQualityRaw > 3) return false - if (header.heartKeyArrhythmiaCheckResult == null) return false - if (header.heartKeyArrhythmiaCheckStatus == null) return false - if (payload[2] > 1 || payload[3] > 1 || payload[4] > 1 || payload[5] > 1) return false + fun filteredBytesPerSampleCandidates( + payload: List, + widths: List = FILTERED_WIDTH_CANDIDATES, + maxPadding: Int = DEFAULT_MAX_PADDING, + ): List { + val header = decodeHeader(payload) ?: return emptyList() + if (header.signalQualityRaw > 3) return emptyList() + if (header.heartKeyArrhythmiaCheckResult == null) return emptyList() + if (header.heartKeyArrhythmiaCheckStatus == null) return emptyList() + if (payload[2] > 1 || payload[3] > 1 || payload[4] > 1 || payload[5] > 1) return emptyList() val n = header.numberOfECGSamples - if (n <= 0) return false - val end = HEADER_LENGTH + n * 2 - return end <= payload.size && payload.size - end <= maxPadding + if (n <= 0) return emptyList() + return widths.filter { width -> + // `widths` is caller-supplied and `n` comes off the wire, so the offset math is done in Long + // and range-checked: a Kotlin Int overflow wraps silently NEGATIVE and would slip past the + // bounds test below. Same discipline as decodeRaw. + if (width <= 0) return@filter false + val end = HEADER_LENGTH + n.toLong() * width.toLong() + end <= payload.size && payload.size - end <= maxPadding + } + } + + /** The frame-level form of [filteredBytesPerSampleCandidates], CRC-gated. */ + fun filteredBytesPerSampleCandidates( + frame: ByteArray, + payloadStart: Int = PUFFIN_PAYLOAD_START, + widths: List = FILTERED_WIDTH_CANDIDATES, + maxPadding: Int = DEFAULT_MAX_PADDING, + ): List { + val payload = innerPayload(frame, payloadStart) ?: return emptyList() + return filteredBytesPerSampleCandidates(payload, widths, maxPadding) } /** diff --git a/android/app/src/main/java/com/noop/protocol/Whoop5EcgProbe.kt b/android/app/src/main/java/com/noop/protocol/Whoop5EcgProbe.kt index 6866539315..3f5ffc29e1 100644 --- a/android/app/src/main/java/com/noop/protocol/Whoop5EcgProbe.kt +++ b/android/app/src/main/java/com/noop/protocol/Whoop5EcgProbe.kt @@ -135,6 +135,157 @@ object Whoop5EcgProbe { get() = if (sentArgument == null) label else "$label arg=$sentArgument" } + /** + * A bounded census of EVERY unclassified 5/MG frame seen while a probe run is armed — the ones the + * structural triage rejected as well as the ones it accepted. + * + * ## Why the rejects are the point + * + * The packet TYPE byte the Labrador records arrive under is NOT attested, so the probe hunts for it + * with a shape heuristic. Logging only the heuristic's HITS makes that hunt unfalsifiable: when the + * heuristic is wrong, the frame it was wrong about is discarded, the report truthfully says "no frame + * passed the structural triage", and the bytes that would have shown the mistake are gone. That is + * exactly what a hardcoded 2-bytes-per-sample length agreement did to every 3-byte frame before + * [Whoop5Ecg.FILTERED_WIDTH_CANDIDATES] widened it. + * + * It makes no claim about anything it records: it is a census, and every line in it is a frame the + * probe could not classify. + * + * ## Bounds + * + * At most [MAX_SAMPLES_PER_TYPE] recorded frames per distinct type byte and [MAX_TYPES] distinct type + * bytes. Everything past a cap is COUNTED, never silently dropped. + * + * The Kotlin twin of the Swift `Whoop5EcgProbe.FrameCensus`. Android has no ECG app layer to feed it — + * nothing on this platform sends the Labrador commands — so this exists for parity and is covered by + * its own tests, exactly like the rest of this file. + */ + class FrameCensus { + + /** One recorded frame. Nothing here is interpreted — these are measurements of the buffer. */ + data class Sample( + val frameLength: Int, + /** Inner payload length, or null when the frame yielded none (failed a CRC / too short). */ + val payloadLength: Int?, + /** + * `numberOfECGSamples` as the status header would read it, or null when the payload is too + * short to hold a header. Reading it does NOT claim this frame is an ECG record. + */ + val numberOfECGSamples: Int?, + /** + * The bytes-per-sample widths the payload's length agreed with. Empty means the triage + * rejected the frame, which is the case this census exists to preserve. + */ + val widths: List, + /** The frame's first [HEAD_BYTES] bytes, lowercase hex. */ + val headHex: String, + ) { + val line: String + get() { + val payload = payloadLength?.toString() ?: "?" + val samples = numberOfECGSamples?.toString() ?: "?" + val widthText = if (widths.isEmpty()) "none" else widths.joinToString(",") + return "len=$frameLength payload=$payload samples=$samples widths=$widthText head=$headHex" + } + } + + /** Everything seen under one type byte. */ + data class Bucket(val typeByte: Int, var count: Int, val samples: MutableList) + + private val bucketList = mutableListOf() + + /** Buckets in FIRST-SEEN order, which is the order the report prints them in. */ + val buckets: List get() = bucketList + + /** Every frame offered to the census, including those past a cap. */ + var framesSeen = 0 + private set + + /** Frames dropped because their type byte was past the [MAX_TYPES]th distinct one. */ + var framesBeyondTypeCap = 0 + private set + + val isEmpty: Boolean get() = framesSeen == 0 + + /** + * Fold one frame into the census. + * + * The caller CRC-gates the frame first; this reads the type byte directly and gets the payload + * through [Whoop5Ecg.innerPayload], which re-checks both CRCs itself, so a frame that somehow + * arrives unverified yields a null payload length rather than a decoded field. Frames with no type + * byte at all (fewer than 9 bytes) are not records and are ignored. + */ + fun record( + frame: ByteArray, + payloadStart: Int = Whoop5Ecg.PUFFIN_PAYLOAD_START, + maxPadding: Int = Whoop5Ecg.DEFAULT_MAX_PADDING, + ) { + if (frame.size <= 8) return + framesSeen++ + val typeByte = frame[8].toInt() and 0xFF + val bucket = bucketList.firstOrNull { it.typeByte == typeByte } + if (bucket == null) { + if (bucketList.size >= MAX_TYPES) { + framesBeyondTypeCap++ + return + } + bucketList.add( + Bucket(typeByte, 1, mutableListOf(sample(frame, payloadStart, maxPadding))) + ) + return + } + bucket.count++ + if (bucket.samples.size >= MAX_SAMPLES_PER_TYPE) return + bucket.samples.add(sample(frame, payloadStart, maxPadding)) + } + + /** The census as report lines, already indented for [report]. */ + val lines: List + get() { + val out = mutableListOf() + for (bucket in bucketList) { + out.add(" type=0x%02x frames=%d".format(bucket.typeByte, bucket.count)) + for (s in bucket.samples) out.add(" ${s.line}") + if (bucket.count > bucket.samples.size) { + out.add(" (+${bucket.count - bucket.samples.size} more of this type, not recorded)") + } + } + if (framesBeyondTypeCap > 0) { + out.add( + " ($framesBeyondTypeCap frame(s) of further type bytes past the " + + "$MAX_TYPES-type cap, counted only)" + ) + } + return out + } + + companion object { + /** Recorded frames per distinct type byte. Three examples answer "what does this look like". */ + const val MAX_SAMPLES_PER_TYPE = 3 + + /** Distinct type bytes tracked. Well past what a live link shows, and keeps the report readable. */ + const val MAX_TYPES = 16 + + /** Leading bytes of each recorded frame rendered as hex: the envelope, the header, first samples. */ + const val HEAD_BYTES = 64 + + /** Measure one frame. Public so a test can build a [Sample] without a census. */ + fun sample( + frame: ByteArray, + payloadStart: Int = Whoop5Ecg.PUFFIN_PAYLOAD_START, + maxPadding: Int = Whoop5Ecg.DEFAULT_MAX_PADDING, + ): Sample { + val payload = Whoop5Ecg.innerPayload(frame, payloadStart) + val widths = payload?.let { + Whoop5Ecg.filteredBytesPerSampleCandidates(it, maxPadding = maxPadding) + } ?: emptyList() + val samples = payload?.let { Whoop5Ecg.decodeHeader(it)?.numberOfECGSamples } + val head = frame.take(HEAD_BYTES).joinToString("") { "%02x".format(it.toInt() and 0xFF) } + return Sample(frame.size, payload?.size, samples, widths, head) + } + } + } + /** The gate verdict, kept separate from the report text so it is assertable in a test. */ sealed class Verdict { /** @@ -264,12 +415,17 @@ object Whoop5EcgProbe { * * [candidateFrames] are the type/length lines for frames that passed the structural triage in * [Whoop5Ecg.plausibleFilteredPayload]. + * + * [census] is the same question asked WITHOUT the heuristic: every unclassified frame the run saw, + * bucketed by type byte, hits and misses alike. It defaults to empty so a caller that has none still + * renders — see [FrameCensus] for why the misses matter more than the hits. */ fun report( steps: List, ecgPacketsSeen: Int, candidateFrames: List, windowSeconds: Int, + census: FrameCensus = FrameCensus(), ): String { val sb = StringBuilder() sb.append("WHOOP MG ECG (Labrador) TURN-ON PROBE\n") @@ -310,6 +466,26 @@ object Whoop5EcgProbe { sb.append("Candidate packet types (structural triage only, NOT a confirmed mapping):\n") for (line in candidateFrames) sb.append(" $line\n") } + // The census goes IMMEDIATELY under the triage result, because it is the check on it: "no frame + // passed the structural triage" is only informative next to what the frames actually were. + // Twin of the Swift `Whoop5EcgProbe.report` block; the wording is byte-identical. + if (census.isEmpty) { + sb.append( + "Unclassified-frame census: no unclassified frame arrived at all — the triage had " + + "nothing to reject.\n" + ) + } else { + sb.append( + "Unclassified-frame census — EVERY frame this run could not classify, triage hits and " + + "misses alike, bucketed by the inner record's type byte. The ECG type byte is not " + + "attested, so this, not the triage, is the record of what arrived. widths= are the " + + "bytes-per-sample the payload length agreed with; none = the triage rejected it. " + + "Recorded: ${census.framesSeen} frame(s), first ${FrameCensus.MAX_SAMPLES_PER_TYPE} per " + + "type byte, first ${FrameCensus.MAX_TYPES} type bytes, first ${FrameCensus.HEAD_BYTES} " + + "bytes each:\n" + ) + for (line in census.lines) sb.append("$line\n") + } val replies = steps.mapNotNull { step -> step.replyHex?.let { " ${step.labelWithArgument}: $it" } } if (replies.isNotEmpty()) { sb.append("\nRaw replies:\n") diff --git a/android/app/src/test/java/com/noop/protocol/Whoop5EcgProbeTest.kt b/android/app/src/test/java/com/noop/protocol/Whoop5EcgProbeTest.kt index b2295e0629..b795a1f8ca 100644 --- a/android/app/src/test/java/com/noop/protocol/Whoop5EcgProbeTest.kt +++ b/android/app/src/test/java/com/noop/protocol/Whoop5EcgProbeTest.kt @@ -455,4 +455,133 @@ class Whoop5EcgProbeTest { assertFalse("report must not name $token", text.lowercase().contains(token.lowercase())) } } + + // MARK: - Unclassified-frame census + + private fun header(samples: Int): List = listOf( + 2, 0x05, 1, 1, 0, 1, 0, 1, 42, 0, 61, 63, 812 and 0xFF, (812 shr 8) and 0xFF, 17, + samples and 0xFF, (samples shr 8) and 0xFF, + ) + + private fun puffinFrame(type: Int, payload: List): ByteArray = + Framing.puffinCommandFrame( + cmd = 0x00, seq = 0x01, + payload = payload.map { it.toByte() }.toByteArray(), + type = type, + ) + + /** + * The census exists for the frames the triage says NO to. A heuristic that logs only its own hits + * destroys the evidence for its own misses: the 2-bytes-per-sample assumption discarded every 3-byte + * frame, and the report then said, truthfully, that nothing passed. + */ + @Test + fun censusRecordsAFrameTheTriageRejects() { + val frame = puffinFrame(0x1A, List(20) { 0xEE }) + assertFalse("fixture must be a triage MISS", Whoop5Ecg.plausibleFilteredFrame(frame)) + + val census = Whoop5EcgProbe.FrameCensus() + census.record(frame) + assertEquals(1, census.framesSeen) + assertEquals(1, census.buckets.size) + assertEquals(0x1A, census.buckets[0].typeByte) + assertEquals(1, census.buckets[0].count) + + val sample = census.buckets[0].samples[0] + assertEquals(frame.size, sample.frameLength) + assertEquals(21, sample.payloadLength) // 20 + the envelope's pad4 byte + assertEquals(0xEEEE, sample.numberOfECGSamples) // read, not believed + assertTrue(sample.widths.isEmpty()) + assertTrue(sample.headHex.startsWith("aa")) + assertEquals(" type=0x1a frames=1", census.lines[0]) + assertTrue(census.lines[1].contains("widths=none")) + assertTrue(census.lines[1].contains("payload=21")) + } + + /** A HIT is censused too — the census is the whole record of the window, not a rejects bin. */ + @Test + fun censusRecordsTriageHitsWithTheWidthThatAgreed() { + val frame = puffinFrame(0x28, header(samples = 4) + List(12) { 0x11 }) + assertTrue(Whoop5Ecg.plausibleFilteredFrame(frame)) + val census = Whoop5EcgProbe.FrameCensus() + census.record(frame) + assertEquals(listOf(3), census.buckets[0].samples[0].widths) + assertEquals(4, census.buckets[0].samples[0].numberOfECGSamples) + assertTrue(census.lines[1].contains("widths=3")) + } + + /** A frame whose CRC does not check out yields no decoded field — only its shape and its bytes. */ + @Test + fun censusReadsNoFieldOutOfAnUnverifiedFrame() { + val frame = puffinFrame(0x28, header(samples = 3) + listOf(1, 0, 2, 0, 3, 0)) + frame[frame.size - 1] = (frame[frame.size - 1].toInt() xor 0xFF).toByte() + val census = Whoop5EcgProbe.FrameCensus() + census.record(frame) + assertNull(census.buckets[0].samples[0].payloadLength) + assertNull(census.buckets[0].samples[0].numberOfECGSamples) + assertTrue(census.buckets[0].samples[0].widths.isEmpty()) + assertTrue(census.lines[1].contains("payload=? samples=? widths=none")) + } + + /** A 1 Hz stream must not grow the report without bound — and nothing may be dropped in silence. */ + @Test + fun censusCapsSamplesPerTypeButKeepsCounting() { + val frame = puffinFrame(0x30, List(20) { 0xEE }) + val census = Whoop5EcgProbe.FrameCensus() + repeat(50) { census.record(frame) } + assertEquals(50, census.framesSeen) + assertEquals(1, census.buckets.size) + assertEquals(50, census.buckets[0].count) + assertEquals(Whoop5EcgProbe.FrameCensus.MAX_SAMPLES_PER_TYPE, census.buckets[0].samples.size) + assertTrue(census.lines.any { it.contains("+47 more of this type, not recorded") }) + } + + @Test + fun censusCapsDistinctTypesAndCountsTheOverflow() { + val census = Whoop5EcgProbe.FrameCensus() + for (type in 0 until Whoop5EcgProbe.FrameCensus.MAX_TYPES + 4) { + census.record(puffinFrame(type, List(20) { 0xEE })) + } + assertEquals(Whoop5EcgProbe.FrameCensus.MAX_TYPES, census.buckets.size) + assertEquals(4, census.framesBeyondTypeCap) + assertEquals(Whoop5EcgProbe.FrameCensus.MAX_TYPES + 4, census.framesSeen) + assertTrue(census.lines.any { it.contains("4 frame(s) of further type bytes past the") }) + } + + @Test + fun censusIgnoresBuffersTooShortToCarryATypeByte() { + val census = Whoop5EcgProbe.FrameCensus() + census.record(ByteArray(0)) + census.record(ByteArray(8) { 0xAA.toByte() }) + assertTrue(census.isEmpty) + assertTrue(census.buckets.isEmpty()) + } + + /** The census is only useful if the operator can COPY it: it belongs in the report sheet. */ + @Test + fun reportCarriesTheCensusBesideTheTriageResult() { + val census = Whoop5EcgProbe.FrameCensus() + census.record(puffinFrame(0x1A, List(20) { 0xEE })) + val text = Whoop5EcgProbe.report( + steps = listOf( + sent(139, 1, Whoop5EcgProbe.CommandOutcome.Success), + sent(124, Whoop5Ecg.ControlSignal.START.raw, Whoop5EcgProbe.CommandOutcome.Success), + ), + ecgPacketsSeen = 0, + candidateFrames = emptyList(), + windowSeconds = 30, + census = census, + ) + assertTrue(text.contains("Candidate packet types: none — no frame passed the structural triage.")) + assertTrue(text.contains("Unclassified-frame census")) + assertTrue(text.contains("type=0x1a frames=1")) + assertTrue(text.contains("widths=none")) + assertTrue(text.contains(census.buckets[0].samples[0].headHex)) + } + + @Test + fun reportSaysSoWhenNoUnclassifiedFrameArrivedAtAll() { + val text = Whoop5EcgProbe.report(emptyList(), 0, emptyList(), 30) + assertTrue(text.contains("no unclassified frame arrived at all")) + } } diff --git a/android/app/src/test/java/com/noop/protocol/Whoop5EcgTest.kt b/android/app/src/test/java/com/noop/protocol/Whoop5EcgTest.kt index c8961c0033..97029de784 100644 --- a/android/app/src/test/java/com/noop/protocol/Whoop5EcgTest.kt +++ b/android/app/src/test/java/com/noop/protocol/Whoop5EcgTest.kt @@ -376,6 +376,100 @@ class Whoop5EcgTest { assertFalse(Whoop5Ecg.plausibleFilteredPayload(List(64) { 0 })) } + // MARK: - Structural triage: sample width + + /** + * THE DEFECT this widening fixes. The triage hardcoded `n * 2`, so a payload whose samples are 3 bytes + * wide missed the length agreement, failed the padding budget, and was discarded without a trace — + * while the probe truthfully reported that no frame passed. A populated raw flash record read off + * hardware was 1500 bytes against numberOfECGSamples = 500, i.e. 3. + */ + @Test + fun triageAcceptsAThreeBytePerSampleBufferAndNamesTheWidth() { + val payload = header(samples = 4) + List(12) { 0x11 } + assertEquals(listOf(3), Whoop5Ecg.filteredBytesPerSampleCandidates(payload)) + assertTrue(Whoop5Ecg.plausibleFilteredPayload(payload)) + } + + @Test + fun triageStillAcceptsTwoBytesPerSampleAndReportsItFirst() { + val payload = header(samples = 3) + i16le(listOf(1, 2, 3)) + assertEquals(listOf(2), Whoop5Ecg.filteredBytesPerSampleCandidates(payload)) + assertTrue(Whoop5Ecg.plausibleFilteredPayload(payload)) + assertEquals(2, Whoop5Ecg.FILTERED_WIDTH_CANDIDATES.first()) + } + + @Test + fun triageAcceptsFourBytesPerSample() { + assertEquals( + listOf(4), + Whoop5Ecg.filteredBytesPerSampleCandidates(header(samples = 5) + List(20) { 0x22 }), + ) + } + + /** An ambiguous buffer REPORTS every width it admits rather than picking one. */ + @Test + fun triageReportsEveryWidthAnAmbiguousBufferAdmits() { + assertEquals( + listOf(3, 4), + Whoop5Ecg.filteredBytesPerSampleCandidates(header(samples = 2) + List(8) { 0x33 }), + ) + } + + /** Only the WIDTH assumption moved: every other guard still rejects a length-perfect buffer. */ + @Test + fun wideningTheWidthDoesNotLoosenAnyOtherGuard() { + val body = List(12) { 0x11 } + assertEquals(listOf(3), Whoop5Ecg.filteredBytesPerSampleCandidates(header(samples = 4) + body)) + assertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(started = 7, samples = 4) + body).isEmpty()) + assertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(running = 2, samples = 4) + body).isEmpty()) + assertTrue( + Whoop5Ecg.filteredBytesPerSampleCandidates(header(stoppedAndComplete = 9, samples = 4) + body).isEmpty() + ) + assertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(leadsOn = 3, samples = 4) + body).isEmpty()) + assertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(signalQuality = 9, samples = 4) + body).isEmpty()) + assertTrue( + Whoop5Ecg.filteredBytesPerSampleCandidates(header(arrhythmiaResult = 9, samples = 4) + body).isEmpty() + ) + assertTrue( + Whoop5Ecg.filteredBytesPerSampleCandidates(header(arrhythmiaStatus = 9, samples = 4) + body).isEmpty() + ) + assertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(samples = 0)).isEmpty()) + } + + @Test + fun triageStillRejectsBuffersNoWidthExplains() { + assertTrue( + Whoop5Ecg.filteredBytesPerSampleCandidates(header(samples = 7) + i16le(listOf(1, 2, 3))).isEmpty() + ) + assertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(header(samples = 40) + i16le(listOf(1, 2))).isEmpty()) + assertTrue( + Whoop5Ecg.filteredBytesPerSampleCandidates( + header(samples = 1) + i16le(listOf(1)) + List(40) { 0 } + ).isEmpty() + ) + assertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(emptyList()).isEmpty()) + assertFalse(Whoop5Ecg.plausibleFilteredPayload(header(samples = 7) + i16le(listOf(1, 2, 3)))) + } + + /** The offset math runs in Long: an Int overflow wraps NEGATIVE and would slip past the bounds test. */ + @Test + fun triageWidthMathCannotOverflow() { + val payload = header(samples = 65_535) + List(8) { 0 } + assertTrue( + Whoop5Ecg.filteredBytesPerSampleCandidates(payload, widths = listOf(Int.MAX_VALUE, 2, 3)).isEmpty() + ) + assertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(payload, widths = listOf(0, -1)).isEmpty()) + } + + @Test + fun triageWidthsAreReportedThroughACrcGatedFrameToo() { + val frame = puffinFrame(0x28, header(samples = 4) + List(12) { 0x11 }) + assertEquals(listOf(3), Whoop5Ecg.filteredBytesPerSampleCandidates(frame)) + frame[frame.size - 1] = (frame[frame.size - 1].toInt() xor 0xFF).toByte() + assertTrue(Whoop5Ecg.filteredBytesPerSampleCandidates(frame).isEmpty()) + } + // MARK: - Commands @Test From 13bea4ce85718fda239f56151e410c0adfaeb1ad Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 6 Aug 2026 12:17:40 -0400 Subject: [PATCH 166/171] Pin real WHOOP MG ECG flash records against Whoop5Ecg Two complete 1584-byte type-47 layout-16 records captured off an MG on 2026-08-06 with the clasp electrodes held, embedded as hex and run through the existing public decode API. No production behaviour changes. What the real bytes establish: * rawBytesPerSampleCandidates resolves unambiguously to [3], and the RawLabradorPacket shape accounts for every byte of the record body with no remainder -- but only when the payload is taken from frame offset 17. * headerLength = 17 is four bytes too long for this record. The header's last nine bytes (heartKeyProgress through numberOfECGSamples) land correctly and carry real values; the first eight read the record's u32 unix second and two unmapped bytes instead of signal quality, the four booleans and the two classifier enums. The record's real signal-quality byte sits at frame[21], which the header maps onto heartKeyIsStoppedAndComplete. * The leads-off block is fixed-size -- eleven slots each for I and Q -- not packed to numberOfLeadsOffSamples. When the count is 10 the Q array is read two bytes early, yielding a spurious leading zero and losing the last real value, and the five-byte remainder exceeds defaultMaxPadding, so 226 of the 350 populated records in the capture (65%) are discarded before any field is read. * The sample blob is 18 signed bits big-endian inside a 24-bit field whose top six bits are a tag. Proven by continuity rather than assumed: on the first record the electrode offset leaves the 16-bit signed range, so a trailing-16-bit reading wraps 65428 counts between adjacent 500 Hz samples while the 18-bit reading stays within 3556. No sample SCALE is asserted -- the package states the unit and scale are not attested, and nothing in the capture establishes one. --- .../Whoop5EcgRawHardwareTests.swift | 362 ++++++++++++++++++ 1 file changed, 362 insertions(+) create mode 100644 Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift new file mode 100644 index 0000000000..93674c4452 --- /dev/null +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift @@ -0,0 +1,362 @@ +import XCTest +@testable import WhoopProtocol + +/// Characterisation of REAL WHOOP MG ECG flash records against this package's own decoder. +/// +/// Unlike `Whoop5EcgTests`, whose fixtures are all synthetic, every byte here came off hardware: two +/// complete 1584-byte type-47 records (layout version 16, `frame[9] == 16`) captured 2026-08-06 while +/// the MG clasp electrodes were held, embedded verbatim as hex. Nothing in this file changes production +/// behaviour — it pins what `Whoop5Ecg` ACTUALLY does when handed these bytes, including the places +/// where what it does is wrong. +/// +/// What the records establish, and what they contradict, is written against each assertion. The short +/// version: +/// +/// • The `RawLabradorPacket` SHAPE fits end to end — blob, leads-off count, the two leads-off arrays +/// and a padding remainder all land with no unaccounted bytes — but only when the payload is taken +/// from frame offset 17, four bytes earlier than the record's real status header. +/// • `headerLength = 17` is FOUR bytes too long for this record. The header's last nine bytes +/// (`heartKeyProgress` through `numberOfECGSamples`) therefore align correctly at offset 17 and +/// decode real values; the first eight bytes do not, and read the record's unix timestamp and two +/// unmapped bytes instead of signal quality, the four booleans and the two classifier enums. +/// • `defaultMaxPadding = 3` rejects 226 of the 350 populated records in the capture (65%), because +/// the leads-off block is FIXED-SIZE (11 slots each for I and Q) rather than packed to the count. +/// • The sample blob is 3 bytes per sample, and the payload is 18 signed bits big-endian inside that +/// 24-bit field — proven here by continuity, not assumed. +final class Whoop5EcgRawHardwareTests: XCTestCase { + + // MARK: - Real captured records + + /// Record at unix 1786031627 — leads connected, `numberOfLeadsOffSamples == 11`. + /// This is the case whose leads-off block happens to be exactly full, so it is the one the current + /// decoder accepts. + private static let recordLeadsOffCount11 = """ + aa0128060100cde02f10036a61cd000bae746aeb51010a000100000000ffff00f401834ae78349e98345d5834703834c + 53834814834916834a0d83478e834d3e83510a8351bb834fda834a718347d383447c8344f3834ec6834c158347848348 + 87834785834a2d834978834c33834d39834c07834b178352dc83537e8351fb8352968350db835319835048834540834a + b3834de2834f1b834f0f835133834f31834f1a83515d834e178351f18351a38352628352558352fd834d4c834a698350 + de835762834e80834b218349338349e6835095835228834c578351b6834f53834ff58351f4834dcb834a6f8350dc8352 + 658350d9834b2f834ba8834d50834e73834cd2834d3c8356698355e08356508357a48359bb835af9835a5f8353ae8352 + 78835bde835c928358c08353a3835693835bae835b7a835c4a835e89835c8a835bb0835f2583608b83610f835ec9835f + ee83611283606f8363e68361cd836cf583649b835d0783637983654d8360c0835f2883608f835a2c8352e78350f98349 + f18340e5833e90833c878338e0833677832ec9833094833bd28333be833941833e328343d7834d4e83570c8359228362 + 0383622283647f83618c8365ce837351836e3e8367e28368fb836cb3836cd5836de9836d8383717a836f7a836dc4836f + 8783726783723c8370828375d0837291836de9836f4583740883785c8373eb837d748376af8374b283745f8371ad8373 + 9a83763e8373eb8375d58375b48374f1837c6e838486837fc0837d3b8379cd837402837747837f4e83765783755b837a + ab8386ab83876b837c6d8375ef8376cc837d41837b33837f4a837f37837c5583811a837d95837b77837cc1837dc88381 + 11837c56837883837a32837da8837f2d8381ec837c388382cb838541837fef8378bd8374ac83731f837914837d9f837b + 93837f35837e22838094838290837e6b837b12837dac837fdc838048837fda83808f8380018381ee83822e8384ed8387 + 7883836b837f858382428378e3837bf38382e683865183850583810883839283886583840c8388568389b4838af6838e + 148388b783875983893583861883865c838e6283915d838a58838d748391e583904b8393288397618391038391e98390 + e383905f838c82838b178390f683947e83963883963383981d839973839b94839974839894839b0f83952b839453839b + 78839d39839b708399028394bc839038838f648395be839a4a839ad0839f44839c18839b008399a98398ee839bbe839c + 6983967a8392c38396b28396088395218395d1839a16839fe3839e5f83997483948f83974c839afc8396a2839b40839e + 48839aec839903839a8f8399ef839f09839aa58397dc839c9483a4f383a2bd83a3e683a197839a968399c4839821839b + 8c839dd58398488398528394eb839806839b0d839ae2839a388396f1839521838de2839230839a1983a05083a3e58399 + fe83981b83982b8396be83a224839df2839a04839ecc8399c9839671839a6d83953a8396c1839a6583a05483a22e83a1 + 3483a2e983a60083997983984883990d839e6b839ee783a030839dc183a0b983a68583a0c483a40583a0c6839f1d839c + 5683a05e83a0ac839d93839bd1839ebc83a53083aa8b83a71083a49883a0ff83a11f83a22f83a5ac83a69983a54383a3 + 2b839b30839e0d839e2f83a040839fae839f99839b5b83a93f83aeb383a61783a78683a7d383a88383abf683aa6e83a6 + 0383a79983a4cf83a60983a5dd83a7d083a48b83a61683a8c883a88f83a6f683a80a83a8ef83a5b083a73483aa4683a9 + 1b83aca583b07883a4be839ed383a1f483a30783a3ea83a79c83a67d83a50383a70383b0c083a64283abfd83aed883a9 + 1583ac5783aeea83ad7b83ae2683a99283a64183abca83ac8783aca783b38083b0b183a53183a42a83a7d383a69283a8 + e083ab2983aba083b1d283ad2883a4cd83a62183ab2c83a9fe83a478839d8983a4cb83aca483a393839ede83a2e483ab + c383a92f83a46083a49983ad4c83ad5083a7a883a53683a59883aa4783a8db83a45583a47183a59383a6ae83ac7c83b0 + 1d83ae9783ae7383aba783a80483aaf483b04c83af0c83a95083a77d83a61183a61b83a48383ad7183aed083a8bc0b43 + 004300430043004300430043004300430043004300f8fff8fff8fff8fff8fff8fff8fff7fff7fff7fff7ff00313774b4 + """ + + /// Record at unix 1786031642 — leads connected, `numberOfLeadsOffSamples == 10`. Structurally + /// identical, but the one unused leads-off slot pushes the trailing remainder to 5 bytes, which the + /// default padding tolerance rejects. + private static let recordLeadsOffCount10 = """ + aa0128060100cde02f10037961cd001aae746aeb51030a000124000000ffff00f40183f47c83f61983f31b83f42a83f5 + 6083f8bc83f3a683f10a83f15883f2d583f71683f45783f58783f26e83f18983efdb83efc483f51583f5e683f6f283f4 + 7d83f59e83f76283f59883f66383f1d583f4d783fbbc83f59183f5a083f4f483f5c683f7ab83eed783ecbf83f19083f0 + 5a83f1bd83efa383f30b83f48083f00483f37583f3dd83f1e283f0d983f1a983f2f183f0f283f07283ef9983ef5d83f2 + 7c83f3ec83f39d83f2e083f01583eece83ed8b83f06883f10383f19683f42983f08b83f35783f30383edae83ec4083ee + 3583f05e83f16083f3f383ee9b83ed5083f03883ecec83ee8d83f31b83eee883eb4883e87883e0e283e9f483f0d183f0 + 7f83effe83ed8983eef183e9dd83ea0b83ed9783ef3083eb8883e9ad83f03983f6d983f33c83ee9583eb6a83ee9483f3 + 7f83f25483ee0083ee1283ebf383eda583ef0583f05683f3ab83ef1a83f2cb83f45583f48383f0c083eacf83ee5583f2 + 9483f16e83ec5c83eaa683ecb583ecd283f0cf83f38983f39183e9c583e7dd83e9f783e8a983e41783e34b83e1cd83df + 1083dde183d17283d22283c94183bfe583c02483bd9d83bfa683c39483c36183bde583c4c783c7ae83d37d83de9783e3 + 0683e62283e8bd83e78d83eb4983e8e683e99283eaac83ecac83ed1a83efe083efdc83ecdb83f35483f20983ef6783f2 + 8183f56983f26183f40283f82583f44d83eb8d83eb0f83eb3283edc283f80f83fae583f5c183f4f383fc3783fae583f3 + 1a83eeea83ff7883fe3583f5e883fac783ff5983f9ae83f60383f7b883fb8d83f9f083feb783fee18003788009ce8006 + f48002a38005f28006ad800522800cb5800f30800ca7800dda8009c68005f18004e180031180093e8010658010918009 + 558005f98008da801274800e0e8002648004ce8003d9800b73800dd4800d2680133b800b908004e783f8dc83fabe8004 + a6800c9a80180c8016b280180b80170b801340801a21801d1e801947801acd801ba480177c8020b68022a2802cba802c + ec802a6b8030688036da8039fd8040a68043f4804868804b7b804463804cec804fdd8058308055be8055618050d5804f + 85804ebc80556e8062e9805ce18058c1805908805632805eca806349805ee6805a13805b86805dd2806925806655805c + 4780587480620580629b8062358062a4806d0c8063c3805b65805f0d805e4e806b00806a8680656c805fbb805851805b + 4b8057f780596a8061a1806570805e02805e7f8063e58065d48061e680613e805f8b805e5e80579b80536c80543c8053 + e4805d83805aee80569880577f80567e805b73805074804fdd8050ec804f74805355804ef2804f46804f52804d718050 + 1f804f9b8058a28050758044c5804ccd804f49804f3a804b0e804bd5804da380451580467a804b6a804961804959804a + 26804c118041fc8041458042e38042918045b7804de88047b28044e68041498035ff80359d8038a3803b81803de78042 + 8a803b1c8033b68034c6803919803992802fc880334e8036a780350c80397b8041d1803998803d6c8032a68025428033 + 74803db9803761803f7980386c802dc0802f2c8035b08033b8803496802935801bac8021d6802e9b802d30802fbe802b + eb803439802d85802bfe802eb0802fb4802f938030ec802bd0802932802c4680339f802ef5802b368024cc8025e5802a + ba802a29802b80802f3780362c8034b38031e28033e2802eda80261b8022908024cd8029438023448027d3802e948027 + 5a8024088022928029038033dc80330f80290b802fbe8023858016cb8021fe80274f802304801ed68022ff802df38031 + 4380318a802c9680240780264a802766802a47802fa8802d808029958028d0801eeb8017d8801f00802854802ab28034 + 3b802ff78022b2801e018020fd8029f28023ac8023208019a68019c08021cc8028858025ed801afc800c5b801e09801f + fa801df4801df08016aa8012598017e2801c9480233a801f6f801d2f8011d080080e8014a2801d018022a98016218005 + 0b800589800dde8019cf8020168020a2800e2b8003ac800ba6801956801838800ebe800c68800d218009ee800e4b0a43 + 004300430043004300420042004200420042000000f7fff7fff7fff7fff7fff7fff6fff6fff6fff6ff000000fe104255 + """ + + /// The payload offset at which `EcgStatusHeader`'s `numberOfECGSamples` lands on the record's real + /// sample count. NOT `Whoop5Ecg.puffinPayloadStart` (11): a type-47 historical record carries a + /// layout marker at [10], a u32 record index at [11] and a u32 unix second at [15] before its body, + /// exactly as `decodeWhoop5HistoricalV2021` documents for versions 20/21. + private static let ecgPayloadStart = 17 + + // MARK: - Helpers + + private func hexToBytes(_ h: String) -> [UInt8] { + var out = [UInt8]() + var nibble: UInt8? + for c in h { + guard let digit = c.hexDigitValue else { continue } + let v = UInt8(digit) + if let hi = nibble { out.append(hi << 4 | v); nibble = nil } else { nibble = v } + } + return out + } + + private var frame11: [UInt8] { hexToBytes(Self.recordLeadsOffCount11) } + private var frame10: [UInt8] { hexToBytes(Self.recordLeadsOffCount10) } + + /// The sample encoding under test: a 24-bit BIG-endian field whose top 6 bits are a tag and whose + /// low 18 bits are a signed value. + private func sample18(_ blob: [UInt8], _ i: Int) -> Int { + let o = i * 3 + var v = (Int(blob[o] & 0x03) << 16) | (Int(blob[o + 1]) << 8) | Int(blob[o + 2]) + if v & 0x20000 != 0 { v -= 0x40000 } + return v + } + + /// The rival encoding: byte 0 is an opaque status byte and the value is the trailing 16 bits. + /// Kept so the continuity test below can actually discriminate rather than merely agree. + private func sample16(_ blob: [UInt8], _ i: Int) -> Int { + let o = i * 3 + var v = (Int(blob[o + 1]) << 8) | Int(blob[o + 2]) + if v & 0x8000 != 0 { v -= 0x10000 } + return v + } + + private func maxAbsDelta(_ values: [Int]) -> Int { + var m = 0 + for i in 1.. UInt32 { + UInt32(b[o]) | UInt32(b[o + 1]) << 8 | UInt32(b[o + 2]) << 16 | UInt32(b[o + 3]) << 24 + } + + // MARK: - The frames are real and well formed + + func testCapturedFramesPassBothWhoop5CRCs() { + for frame in [frame11, frame10] { + XCTAssertEqual(frame.count, 1584) + XCTAssertEqual(frame[8], 47, "type-47 historical record") + XCTAssertEqual(frame[9], 16, "layout version 16") + let check = verifyFrame(frame, family: .whoop5) + XCTAssertTrue(check.ok, "real capture must pass the CRC16 header and CRC32 trailer") + XCTAssertEqual(check.length, 1576) + } + // The unix second the repo's own v20/v21 header model puts at [15] — this is what makes an ECG + // status header starting at 17 impossible: bytes 17 and 18 are its two high bytes. + XCTAssertEqual(u32le(frame11, 15), 1786031627) + XCTAssertEqual(u32le(frame10, 15), 1786031642) + XCTAssertEqual(frame11[17], 0x74) + XCTAssertEqual(frame11[18], 0x6A) + } + + // MARK: - What upstream concludes about the sample width + + func testRawBytesPerSampleCandidatesResolveToThreeOnRealRecord() { + guard let payload = Whoop5Ecg.innerPayload(frame11, payloadStart: Self.ecgPayloadStart) else { + return XCTFail("CRC-gated payload extraction failed on a real frame") + } + XCTAssertEqual(payload.count, 1563) + // Unambiguous: exactly one width survives, and it is 3. + XCTAssertEqual(Whoop5Ecg.rawBytesPerSampleCandidates(payload: payload), [3]) + XCTAssertNotNil(Whoop5Ecg.decodeRaw(payload: payload), + "the disambiguating overload must accept a single-candidate buffer") + } + + /// The offset is load-bearing, and wrong offsets fail closed rather than producing a plausible decode. + func testOtherPayloadStartsYieldNoCandidates() { + for start in [Whoop5Ecg.puffinPayloadStart, 19, 21] { + guard let payload = Whoop5Ecg.innerPayload(frame11, payloadStart: start) else { + return XCTFail("payload extraction failed at \(start)") + } + XCTAssertEqual(Whoop5Ecg.rawBytesPerSampleCandidates(payload: payload), [], + "offset \(start) must not admit any sample width") + } + } + + // MARK: - The full RawLabradorPacket shape against real bytes + + func testRawLabradorPacketAccountsForEveryByteOfTheRecord() { + guard let payload = Whoop5Ecg.innerPayload(frame11, payloadStart: Self.ecgPayloadStart), + let packet = Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 3) else { + return XCTFail("decodeRaw failed on a real record") + } + XCTAssertEqual(packet.header.numberOfECGSamples, 500) + XCTAssertEqual(packet.rawECGDataRaw.count, 1500) + XCTAssertEqual(packet.bytesPerSample, 3) + XCTAssertEqual(packet.numberOfLeadsOffSamples, 11) + XCTAssertEqual(packet.leadsOffIRaw, [UInt16](repeating: 67, count: 11)) + XCTAssertEqual(packet.leadsOffQRaw, + [UInt16](repeating: 65528, count: 7) + [UInt16](repeating: 65527, count: 4)) + XCTAssertEqual(packet.padding, [0x00]) + + // Every byte of the 1563-byte payload is claimed by exactly one field. + let accounted = Whoop5Ecg.headerLength + + packet.rawECGDataRaw.count + + 1 + + packet.leadsOffIRaw.count * 2 + + packet.leadsOffQRaw.count * 2 + + packet.padding.count + XCTAssertEqual(accounted, payload.count, "no unaccounted bytes in the record body") + } + + // MARK: - Where the header actually is + + /// The nine header bytes from `heartKeyProgress` onward align correctly at offset 17 and carry real + /// values; the eight before them do not. Both halves are pinned so a future fix has to move both. + func testHeaderTailIsCorrectAndHeaderHeadIsNot() { + guard let payload = Whoop5Ecg.innerPayload(frame11, payloadStart: Self.ecgPayloadStart), + let header = EcgStatusHeader(payload: payload) else { + return XCTFail("header parse failed") + } + // Correctly aligned tail — these agree with the record. + XCTAssertEqual(header.numberOfECGSamples, 500) + XCTAssertEqual(header.heartKeyProgress, .percent(0)) + XCTAssertEqual(header.heartKeyHRV, 65535, "0xFFFF unknown sentinel while the reading is running") + XCTAssertEqual(header.heartKeyAverageHR, 0) + XCTAssertEqual(header.heartKeyHR, 0) + XCTAssertEqual(header.heartKeyStressScore, 0) + XCTAssertEqual(header.heartKeyUnreadableReason, 0) + + // Misaligned head — `signalQualityRaw` and `statusFlags` are the unix timestamp's two high + // bytes, and the four booleans read unmapped preamble rather than 0/1 flags. + XCTAssertEqual(header.signalQualityRaw, 0x74, "this is frame[17], a unix byte, not a quality grade") + XCTAssertEqual(header.signalQuality, .unknown, "116 is outside the 0...3 enum") + XCTAssertEqual(header.statusFlags, 0x6A, "this is frame[18], the other unix byte") + XCTAssertEqual(payload[2], 0xEB, "'heartKeyStarted' reads a 235, which is not a Bool") + XCTAssertEqual(payload[3], 0x51, "'heartKeyIsRunning' reads an 81, which is not a Bool") + } + + /// The record's real signal-quality and progress bytes, located by their behaviour across the + /// capture: quality climbs 1 -> 3 and progress ramps 0 -> 100 over the 30-second reading. Progress + /// at frame[25] is what `EcgStatusHeader` already reads correctly; quality at frame[21] is not. + func testRealSignalQualityAndProgressBytes() { + XCTAssertEqual(frame11[21], 1, "signal quality: low, at the start of the reading") + XCTAssertEqual(frame10[21], 3, "signal quality: high, 15 seconds in") + XCTAssertEqual(frame11[25], 0, "progress 0%") + XCTAssertEqual(frame10[25], 36, "progress 36%") + // The repo's header maps frame[21] onto 'heartKeyIsStoppedAndComplete' — four bytes early. + guard let payload = Whoop5Ecg.innerPayload(frame11, payloadStart: Self.ecgPayloadStart), + let header = EcgStatusHeader(payload: payload) else { return XCTFail("header parse failed") } + XCTAssertTrue(header.heartKeyIsStoppedAndComplete, + "reads frame[21] == 1, which is really the signal-quality grade") + } + + // MARK: - Sample decode + + func testFirstTwelveSamplesDecodeAsSigned18BitBigEndian() { + guard let payload = Whoop5Ecg.innerPayload(frame11, payloadStart: Self.ecgPayloadStart), + let packet = Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 3) else { + return XCTFail("decodeRaw failed") + } + let blob = packet.rawECGDataRaw + let first12 = (0..<12).map { sample18(blob, $0) } + XCTAssertEqual(first12, [-46361, -46615, -47659, -47357, -45997, -47084, -46826, -46579, -47218, -45762, -44790, -44613]) + + // Every sample in this record carries tag 0b100000 — leads connected. + for i in 0..> 2, 0b100000, "sample \(i) tag") + } + } + + func testSecondRecordFirstTwelveSamples() { + guard let payload = Whoop5Ecg.innerPayload(frame10, payloadStart: Self.ecgPayloadStart), + let packet = Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 3) else { + return XCTFail("decodeRaw failed") + } + let blob = packet.rawECGDataRaw + XCTAssertEqual((0..<12).map { sample18(blob, $0) }, [-2948, -2535, -3301, -3030, -2720, -1860, -3162, -3830, -3752, -3371, -2282, -2985]) + let all = (0..= 0 }.count, 304) + } + + /// The discriminating evidence for 18 bits over "status byte + 16-bit value". + /// + /// In the first record the trace sits on a large negative electrode offset that leaves the 16-bit + /// signed range. Read as 18 bits it is a continuous waveform; read as a trailing 16-bit value it + /// wraps full scale between adjacent 500 Hz samples. A 65428-count step between neighbouring samples + /// of a settling ECG is not physical, so the 18-bit reading is the only one the data supports. + func testEighteenBitReadingIsContinuousWhereSixteenBitWraps() { + guard let payload = Whoop5Ecg.innerPayload(frame11, payloadStart: Self.ecgPayloadStart), + let packet = Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 3) else { + return XCTFail("decodeRaw failed") + } + let blob = packet.rawECGDataRaw + let n = Int(packet.header.numberOfECGSamples) + let as18 = (0.. Date: Thu, 6 Aug 2026 12:30:36 -0400 Subject: [PATCH 167/171] MG ECG: read the leads-off block as fixed-size, not packed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The raw flash record's leads-off diagnostic block is FIXED-SIZE — the count byte at frame[1534] is followed by eleven i16 I slots at frame[1535] and eleven i16 Q slots at frame[1557], present in full whether or not numberOfLeadsOffSamples fills them, with the unused tail slots zeroed. Both counts in the 2026-08-06 MG capture (10 and 11) put the Q array at the same offset and leave the same single trailing byte, which is what identifies the block as fixed rather than packed. decodeRaw treated the two arrays as packed to the count. Whenever the count was below capacity that was wrong twice over: Q started two bytes early per unused slot, so it gained a spurious leading value and pushed its last real value out into the "padding"; and the misplaced end-of-record left a 5-byte remainder against defaultMaxPadding = 3, so rawBytesPerSampleCandidates returned [] and the record was discarded before any field was read. On the capture's populated records (type 47, layout 16, numberOfECGSamples > 0) that took the decode-success rate from 124/351 to 350/351. The remainder is charged correctly by the fixed layout, not by widening the tolerance: a real record still leaves exactly one padding byte, and 17 + 1500 + 1 + 22 + 22 + 1 = 1563 accounts for the whole body. A count above the block capacity now fails closed rather than reading past the block. Sample decode, commands and gates are untouched. No scale or unit is claimed for the sample values and none is applied; the 18-in-24 big-endian packing is recorded as measured from this capture, not as an attested vendor fact. The synthetic fixtures on both platforms encoded the packed assumption and are rebuilt against the real layout via a shared leadsOffBlock() helper. The Kotlin twin carries the same two real frames so the parity contract is proved against hardware bytes rather than against a shared assumption. Still open, deliberately not papered over: the capture's short final record (numberOfECGSamples = 245) keeps its leads-off block at the same fixed frame[1534], with the sample region zero-filled from frame[769] — so the sample REGION is fixed-size too, and the blob-length-implies-width premise rawBytesPerSampleCandidates rests on does not hold for a partly filled record. It is the one populated record that still does not decode. Verified: Packages/WhoopProtocol swift test 570/570 pass; android testFullDebugUnitTest 3507/3507 pass (JDK 17). --- .../Sources/WhoopProtocol/Whoop5Ecg.swift | 35 ++- .../Whoop5EcgRawHardwareTests.swift | 124 ++++++++--- .../WhoopProtocolTests/Whoop5EcgTests.swift | 61 ++++-- .../main/java/com/noop/protocol/Whoop5Ecg.kt | 40 +++- .../noop/protocol/Whoop5EcgRawHardwareTest.kt | 199 ++++++++++++++++++ .../java/com/noop/protocol/Whoop5EcgTest.kt | 62 +++++- 6 files changed, 453 insertions(+), 68 deletions(-) create mode 100644 android/app/src/test/java/com/noop/protocol/Whoop5EcgRawHardwareTest.kt diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Ecg.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Ecg.swift index c5a8a08ce1..85bd755b03 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Ecg.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Ecg.swift @@ -209,12 +209,18 @@ public struct FilteredLabradorPacket: Equatable, Sendable { /// diagnostic arrays. The blob's bytes-per-sample is `rawECGDataRaw.count / numberOfECGSamples` — which /// means the blob's LENGTH is not itself on the wire, so a decode needs that width supplied or resolved /// (see `rawBytesPerSampleCandidates`). +/// +/// The leads-off block that follows is FIXED-SIZE — `Whoop5Ecg.leadsOffSlotCount` slots for I and the +/// same again for Q, with `numberOfLeadsOffSamples` selecting how many leading slots are valid. public struct RawLabradorPacket: Equatable, Sendable { public let header: EcgStatusHeader /// Opaque. The container width and encoding are unattested, so the bytes are carried verbatim. public let rawECGDataRaw: [UInt8] public let numberOfLeadsOffSamples: UInt8 + /// The VALID leads-off I values — `numberOfLeadsOffSamples` of them, taken from the front of the + /// record's fixed `Whoop5Ecg.leadsOffSlotCount`-slot block. Unused slots are dropped, not carried. public let leadsOffIRaw: [UInt16] + /// The VALID leads-off Q values, on the same terms as `leadsOffIRaw`. public let leadsOffQRaw: [UInt16] public let padding: [UInt8] @@ -252,6 +258,22 @@ public enum Whoop5Ecg { /// bytes. Callers scanning an unfamiliar layout can widen it. public static let defaultMaxPadding = 3 + /// Slots in the raw record's leads-off diagnostic block, per array. + /// + /// MEASURED FROM HARDWARE, not attested: two complete type-47 layout-16 flash records captured + /// 2026-08-06 (embedded verbatim in `Whoop5EcgRawHardwareTests`) show the block is FIXED-SIZE — the + /// count byte is followed by eleven i16 I slots and then eleven i16 Q slots, present in full whether + /// or not `numberOfLeadsOffSamples` fills them, with the unused tail slots zeroed. Both counts seen + /// in the capture (10 and 11) place the Q array at the same offset and leave the same single trailing + /// byte, which is what identifies the block as fixed rather than packed. + /// + /// Reading it as PACKED — `count` elements each — was the original assumption, and it is wrong in two + /// compounding ways whenever `count < 11`: Q is read two bytes early per missing slot (so it gains a + /// spurious leading value and loses its last real one), and the misplaced end-of-record leaves a + /// remainder over `defaultMaxPadding`, which discarded 227 of the capture's 351 populated records + /// before any field was read. + public static let leadsOffSlotCount = 11 + // MARK: Commands // // All four share the shape `{revision: UInt8, arg, padding}`. `revision` is the leading inner byte the @@ -437,10 +459,15 @@ public enum Whoop5Ecg { // The leads-off count byte must itself be inside the buffer. guard rawEnd >= headerLength, rawEnd < payload.count else { return nil } let leadsOffCount = Int(payload[rawEnd]) + // The block holds `leadsOffSlotCount` slots per array; a count that overruns it is not a record + // this layout can describe, so it fails closed rather than reading past the block. + guard leadsOffCount <= leadsOffSlotCount else { return nil } + // Both arrays are FIXED-SIZE and always fully present — see `leadsOffSlotCount`. The count byte + // selects how many leading slots are VALID; it does not size the block. let iStart = rawEnd + 1 - let qStart = iStart + leadsOffCount * 2 - let qEnd = qStart + leadsOffCount * 2 - guard qEnd <= payload.count else { return nil } + let qStart = iStart + leadsOffSlotCount * 2 + let blockEnd = qStart + leadsOffSlotCount * 2 + guard blockEnd <= payload.count else { return nil } var leadsOffI = [UInt16]() var leadsOffQ = [UInt16]() @@ -457,7 +484,7 @@ public enum Whoop5Ecg { numberOfLeadsOffSamples: payload[rawEnd], leadsOffIRaw: leadsOffI, leadsOffQRaw: leadsOffQ, - padding: Array(payload[qEnd...])) + padding: Array(payload[blockEnd...])) } /// Every sample width in `widths` that yields a structurally consistent raw record leaving at most diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift index 93674c4452..648d1d71d6 100644 --- a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift @@ -12,17 +12,28 @@ import XCTest /// What the records establish, and what they contradict, is written against each assertion. The short /// version: /// -/// • The `RawLabradorPacket` SHAPE fits end to end — blob, leads-off count, the two leads-off arrays -/// and a padding remainder all land with no unaccounted bytes — but only when the payload is taken -/// from frame offset 17, four bytes earlier than the record's real status header. +/// • The record body is `17 header + 1500 blob + 1 count + 22 leadsOffI + 22 leadsOffQ + 1` = 1563 +/// bytes, with no leftovers — but only when the payload is taken from frame offset 17, four bytes +/// earlier than the record's real status header. /// • `headerLength = 17` is FOUR bytes too long for this record. The header's last nine bytes /// (`heartKeyProgress` through `numberOfECGSamples`) therefore align correctly at offset 17 and /// decode real values; the first eight bytes do not, and read the record's unix timestamp and two /// unmapped bytes instead of signal quality, the four booleans and the two classifier enums. -/// • `defaultMaxPadding = 3` rejects 226 of the 350 populated records in the capture (65%), because -/// the leads-off block is FIXED-SIZE (11 slots each for I and Q) rather than packed to the count. +/// STILL OPEN — nothing here fixes it. +/// • The leads-off block is FIXED-SIZE — eleven i16 slots for I then eleven for Q, at the same frame +/// offsets under both observed counts, with the unused tail slots zeroed. Reading it as PACKED +/// shifted Q by one slot per unused slot and left a 5-byte remainder against `defaultMaxPadding`, +/// which discarded 227 of the capture's 351 populated records before any field was read. FIXED — +/// `Whoop5Ecg.leadsOffSlotCount`; 350 of the 351 now decode. /// • The sample blob is 3 bytes per sample, and the payload is 18 signed bits big-endian inside that -/// 24-bit field — proven here by continuity, not assumed. +/// 24-bit field — measured here by continuity, not assumed, and not attributed to any vendor +/// source. No scale or unit is claimed for the values, and none is applied. +/// • STILL OPEN — the one populated record that does not decode is the capture's short final record +/// (`numberOfECGSamples == 245`). Its leads-off block sits at the SAME fixed frame offset 1534 as +/// every other record, with bytes 769...1533 zeroed, so the sample REGION is fixed-size too and the +/// blob length is not `n * bytesPerSample` for a partly-filled record. That record is not embedded +/// here and the width-from-length premise `rawBytesPerSampleCandidates` rests on does not hold for +/// it; it keeps failing closed rather than being papered over. final class Whoop5EcgRawHardwareTests: XCTestCase { // MARK: - Real captured records @@ -214,12 +225,13 @@ final class Whoop5EcgRawHardwareTests: XCTestCase { [UInt16](repeating: 65528, count: 7) + [UInt16](repeating: 65527, count: 4)) XCTAssertEqual(packet.padding, [0x00]) - // Every byte of the 1563-byte payload is claimed by exactly one field. + // Every byte of the 1563-byte payload is claimed by exactly one field. The leads-off block is + // charged at its fixed size — which for this record happens to equal the count. let accounted = Whoop5Ecg.headerLength + packet.rawECGDataRaw.count + 1 - + packet.leadsOffIRaw.count * 2 - + packet.leadsOffQRaw.count * 2 + + Whoop5Ecg.leadsOffSlotCount * 2 + + Whoop5Ecg.leadsOffSlotCount * 2 + packet.padding.count XCTAssertEqual(accounted, payload.count, "no unaccounted bytes in the record body") } @@ -322,41 +334,83 @@ final class Whoop5EcgRawHardwareTests: XCTestCase { XCTAssertLessThan(as18.min()!, -32768) } - // MARK: - The two real defects this capture exposes + // MARK: - The fixed-size leads-off block - /// 226 of the 350 populated records in the capture carry `numberOfLeadsOffSamples == 10` rather than - /// 11. The record is a fixed 1584 bytes and the leads-off block is a fixed ELEVEN slots for I and - /// eleven for Q, so one unused slot leaves five trailing bytes — two over `defaultMaxPadding`. The - /// triage therefore discards 65% of real records before any field is read. - func testDefaultPaddingToleranceRejectsTheMajorityOfRealRecords() { - guard let payload = Whoop5Ecg.innerPayload(frame10, payloadStart: Self.ecgPayloadStart) else { - return XCTFail("payload extraction failed") + /// The block's geometry, read straight off the frame with no decoder involved. This is the evidence + /// `Whoop5Ecg.leadsOffSlotCount` rests on: in BOTH records the count byte sits at frame[1534], the I + /// slots start at frame[1535] and the Q slots at frame[1557] — the same offsets under count 11 and + /// count 10, which is what a fixed block does and a packed one cannot. + func testLeadsOffBlockIsAtTheSameFixedOffsetsUnderBothCounts() { + XCTAssertEqual(Whoop5Ecg.leadsOffSlotCount, 11) + XCTAssertEqual(frame11[1534], 11) + XCTAssertEqual(frame10[1534], 10) + + func slots(_ frame: [UInt8], _ start: Int) -> [UInt16] { + (0..<11).map { UInt16(frame[start + $0 * 2]) | UInt16(frame[start + $0 * 2 + 1]) << 8 } } - XCTAssertEqual(Whoop5Ecg.rawBytesPerSampleCandidates(payload: payload), [], - "a real, well-formed, CRC-valid record is rejected outright") - XCTAssertNil(Whoop5Ecg.decodeRaw(payload: payload)) - // Widening the tolerance to the record's actual remainder recovers it, and recovers width 3. - XCTAssertEqual(Whoop5Ecg.rawBytesPerSampleCandidates(payload: payload, maxPadding: 5), [3]) + // Count 11: every slot carries a value. + XCTAssertEqual(slots(frame11, 1535), [UInt16](repeating: 67, count: 11)) + XCTAssertEqual(slots(frame11, 1557), + [UInt16](repeating: 65528, count: 7) + [UInt16](repeating: 65527, count: 4)) + // Count 10: ten values then a zeroed slot, in BOTH arrays, at the SAME offsets. + XCTAssertEqual(slots(frame10, 1535), [67, 67, 67, 67, 67, 66, 66, 66, 66, 66, 0]) + XCTAssertEqual(slots(frame10, 1557), + [65527, 65527, 65527, 65527, 65527, 65527, 65526, 65526, 65526, 65526, 0]) + // And exactly one byte follows the block before the CRC32 trailer. + XCTAssertEqual(1579 + 1 + 4, frame10.count) } - /// With the count below the block's capacity, reading Q at `iStart + count * 2` starts two bytes - /// early: it picks up the unused eleventh I slot as Q[0] and drops the last real Q value. The - /// leads-off block is fixed-size; the decoder treats it as packed. - func testLeadsOffQArrayIsShiftedWhenTheBlockIsNotFull() { + /// Regression: a real count=10 record is ACCEPTED. Under the packed reading its remainder was five + /// bytes against `defaultMaxPadding = 3`, so `rawBytesPerSampleCandidates` returned `[]` and the + /// record was discarded before any field was read — 227 of the capture's 351 populated records. + func testPopulatedRealRecordIsNotDiscarded() { + for frame in [frame11, frame10] { + guard let payload = Whoop5Ecg.innerPayload(frame, payloadStart: Self.ecgPayloadStart) else { + return XCTFail("payload extraction failed") + } + XCTAssertEqual(Whoop5Ecg.rawBytesPerSampleCandidates(payload: payload), [3], + "a real, well-formed, CRC-valid record must resolve to width 3") + XCTAssertNotNil(Whoop5Ecg.decodeRaw(payload: payload)) + // Accepted on its OWN remainder, not by widening the tolerance. + XCTAssertEqual(Whoop5Ecg.decodeRaw(payload: payload)?.padding, [0x00]) + } + } + + /// With the count below the block's capacity, the ten valid values come off the front of each fixed + /// array. The packed reading started Q two bytes early — it picked up the unused eleventh I slot as + /// Q[0] and pushed the tenth real Q value out into the "padding". + func testPartiallyFilledLeadsOffBlockKeepsQAligned() { guard let payload = Whoop5Ecg.innerPayload(frame10, payloadStart: Self.ecgPayloadStart), let packet = Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 3) else { return XCTFail("decodeRaw failed") } XCTAssertEqual(packet.numberOfLeadsOffSamples, 10) XCTAssertEqual(packet.leadsOffIRaw, [67, 67, 67, 67, 67, 66, 66, 66, 66, 66]) - // What the decoder produces: a spurious leading zero, and the final value lost. - XCTAssertEqual(packet.leadsOffQRaw, [0, 65527, 65527, 65527, 65527, 65527, 65527, 65526, 65526, 65526]) - XCTAssertEqual(packet.leadsOffQRaw[0], 0, "the unused eleventh I slot, misread as Q[0]") - // What the fixed-slot layout actually holds: eleven I slots at frame[1535], eleven Q at [1557]. - let trueQ = (0..<11).map { UInt16(frame10[1557 + $0 * 2]) | UInt16(frame10[1558 + $0 * 2]) << 8 } - XCTAssertEqual(trueQ, [65527, 65527, 65527, 65527, 65527, 65527, 65526, 65526, 65526, 65526, 0]) - // The five "padding" bytes are not padding: the first two are the tenth real Q value. - XCTAssertEqual(packet.padding, [246, 255, 0, 0, 0]) - XCTAssertEqual(UInt16(packet.padding[0]) | UInt16(packet.padding[1]) << 8, 65526) + XCTAssertEqual(packet.leadsOffQRaw, [65527, 65527, 65527, 65527, 65527, 65527, 65526, 65526, 65526, 65526]) + XCTAssertNotEqual(packet.leadsOffQRaw.first, 0, "no spurious leading slot") + XCTAssertEqual(packet.leadsOffQRaw.last, 65526, "the tenth real Q value, previously lost") + // The unused eleventh slot of each array is dropped, not carried, and is not padding either. + XCTAssertEqual(packet.leadsOffIRaw.count, 10) + XCTAssertEqual(packet.leadsOffQRaw.count, 10) + XCTAssertEqual(packet.padding, [0x00]) + } + + /// The count=10 record accounts for every byte too — with the leads-off block charged at its FIXED + /// size rather than at the count. + func testCountTenRecordAccountsForEveryByte() { + guard let payload = Whoop5Ecg.innerPayload(frame10, payloadStart: Self.ecgPayloadStart), + let packet = Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 3) else { + return XCTFail("decodeRaw failed") + } + XCTAssertEqual(payload.count, 1563) + XCTAssertEqual(packet.header.numberOfECGSamples, 500) + XCTAssertEqual(packet.rawECGDataRaw.count, 1500) + let accounted = Whoop5Ecg.headerLength + + packet.rawECGDataRaw.count + + 1 + + Whoop5Ecg.leadsOffSlotCount * 2 + + Whoop5Ecg.leadsOffSlotCount * 2 + + packet.padding.count + XCTAssertEqual(accounted, payload.count, "17 + 1500 + 1 + 22 + 22 + 1") } } diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift index fbb89f07b8..4735bd4e8f 100644 --- a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift @@ -45,6 +45,18 @@ final class Whoop5EcgTests: XCTestCase { values.flatMap { [UInt8($0 & 0xFF), UInt8($0 >> 8)] } } + /// The raw record's leads-off tail: the count byte, then `Whoop5Ecg.leadsOffSlotCount` i16 I slots, + /// then the same again for Q. The block is FIXED-SIZE on real hardware (see + /// `Whoop5EcgRawHardwareTests`), so these fixtures zero-fill the slots the count does not reach + /// instead of packing the arrays to the count. + private func leadsOffBlock(i: [UInt16], q: [UInt16]) -> [UInt8] { + precondition(i.count == q.count && i.count <= Whoop5Ecg.leadsOffSlotCount) + let pad = Whoop5Ecg.leadsOffSlotCount - i.count + return [UInt8(i.count)] + + u16le(i + [UInt16](repeating: 0, count: pad)) + + u16le(q + [UInt16](repeating: 0, count: pad)) + } + /// Wrap a payload in a real, CRC-correct puffin frame using the shipped builder, so the frame-level /// tests exercise the same envelope the strap speaks. private func puffinFrame(type: UInt8, payload: [UInt8]) -> [UInt8] { @@ -201,7 +213,7 @@ final class Whoop5EcgTests: XCTestCase { let rawBlob: [UInt8] = Array(0..<12) // 4 samples × 3 bytes let leadsOffI: [UInt16] = [1, 2] let leadsOffQ: [UInt16] = [3, 4] - let payload = header(samples: 4) + rawBlob + [2] + u16le(leadsOffI) + u16le(leadsOffQ) + let payload = header(samples: 4) + rawBlob + leadsOffBlock(i: leadsOffI, q: leadsOffQ) let packet = Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 3) XCTAssertNotNil(packet) @@ -214,21 +226,44 @@ final class Whoop5EcgTests: XCTestCase { XCTAssertEqual(packet?.header.numberOfECGSamples, 4) } + /// A zero count still carries the full fixed block — the slots are simply all unused. func testRawWithNoLeadsOffSamples() { - let payload = header(samples: 2) + [0xAA, 0xBB, 0xCC, 0xDD] + [0] + let payload = header(samples: 2) + [0xAA, 0xBB, 0xCC, 0xDD] + leadsOffBlock(i: [], q: []) let packet = Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 2) XCTAssertEqual(packet?.numberOfLeadsOffSamples, 0) XCTAssertEqual(packet?.leadsOffIRaw, []) XCTAssertEqual(packet?.leadsOffQRaw, []) XCTAssertEqual(packet?.rawECGDataRaw, [0xAA, 0xBB, 0xCC, 0xDD]) + XCTAssertEqual(packet?.padding, []) + } + + /// The valid values come off the FRONT of each fixed array, and the unused slots are dropped rather + /// than shifting Q or leaking into the padding — the defect the real capture exposed. + func testRawPartiallyFilledLeadsOffBlockKeepsQAligned() { + let payload = header(samples: 2) + [0, 0, 0, 0] + + leadsOffBlock(i: [11, 12, 13], q: [21, 22, 23]) + let packet = Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 2) + XCTAssertEqual(packet?.numberOfLeadsOffSamples, 3) + XCTAssertEqual(packet?.leadsOffIRaw, [11, 12, 13]) + XCTAssertEqual(packet?.leadsOffQRaw, [21, 22, 23]) + XCTAssertEqual(packet?.padding, [], "the unused slots are part of the block, not trailing padding") } func testRawRejectsTruncatedLeadsOffArrays() { - // Declares 3 leads-off samples but only carries I (and only partly). + // Declares 3 leads-off samples but carries neither array in full. let payload = header(samples: 2) + [0, 0, 0, 0] + [3] + u16le([1, 2, 3]) XCTAssertNil(Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 2)) } + /// The block holds a fixed number of slots, so a count above it cannot describe this layout and must + /// fail closed rather than read past the block. + func testRawRejectsLeadsOffCountAboveTheBlockCapacity() { + let overCount = UInt8(Whoop5Ecg.leadsOffSlotCount + 1) + let payload = header(samples: 2) + [0, 0, 0, 0] + [overCount] + + [UInt8](repeating: 0, count: Whoop5Ecg.leadsOffSlotCount * 4 + 8) + XCTAssertNil(Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 2)) + } + func testRawRejectsMissingLeadsOffCountByte() { // The raw blob consumes the whole buffer, leaving no room for the count byte. let payload = header(samples: 2) + [0, 0, 0, 0] @@ -258,13 +293,14 @@ final class Whoop5EcgTests: XCTestCase { func testRawRejectsShortHeaderAndZeroWidth() { XCTAssertNil(Whoop5Ecg.decodeRaw(payload: [1, 2, 3], bytesPerSample: 2)) - let payload = header(samples: 2) + [0, 0, 0, 0] + [0] + let payload = header(samples: 2) + [0, 0, 0, 0] + leadsOffBlock(i: [], q: []) XCTAssertNil(Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 0)) } func testRawSampleWidthCandidatesAreEnumeratedNotGuessed() { // 4 samples × 2 bytes, then a 1-sample leads-off tail. Width 2 must be admitted. - let payload = header(samples: 4) + [UInt8](repeating: 0x11, count: 8) + [1] + u16le([7]) + u16le([8]) + let payload = header(samples: 4) + [UInt8](repeating: 0x11, count: 8) + + leadsOffBlock(i: [7], q: [8]) let candidates = Whoop5Ecg.rawBytesPerSampleCandidates(payload: payload) XCTAssertTrue(candidates.contains(2), "width 2 must be structurally admissible") // Whatever the full candidate set is, the single-candidate decoder must agree with it: it either @@ -278,11 +314,12 @@ final class Whoop5EcgTests: XCTestCase { } func testRawAmbiguousBufferRefusesToDecode() { - // Hand-built so several widths parse: a long all-zero tail lets many split points look valid. - let payload = header(samples: 1) + [UInt8](repeating: 0, count: 6) - let candidates = Whoop5Ecg.rawBytesPerSampleCandidates(payload: payload, maxPadding: 8) - XCTAssertGreaterThan(candidates.count, 1, "fixture is meant to be ambiguous") - XCTAssertNil(Whoop5Ecg.decodeRaw(payload: payload, maxPadding: 8)) + // Hand-built so several widths parse: with a single sample, an all-zero tail long enough to hold + // the fixed leads-off block leaves every width inside the padding tolerance. + let payload = header(samples: 1) + [UInt8](repeating: 0, count: 49) + let candidates = Whoop5Ecg.rawBytesPerSampleCandidates(payload: payload) + XCTAssertEqual(candidates, [1, 2, 3, 4], "fixture is meant to be ambiguous") + XCTAssertNil(Whoop5Ecg.decodeRaw(payload: payload)) } // MARK: - Frame level (CRC gating) @@ -329,7 +366,7 @@ final class Whoop5EcgTests: XCTestCase { } func testRawFrameDecodesThroughAValidPuffinEnvelope() { - let payload = header(samples: 2) + [1, 2, 3, 4] + [1] + u16le([5]) + u16le([6]) + let payload = header(samples: 2) + [1, 2, 3, 4] + leadsOffBlock(i: [5], q: [6]) let frame = puffinFrame(type: 0x2F, payload: payload) let packet = Whoop5Ecg.decodeRawFrame(frame, bytesPerSample: 2) XCTAssertEqual(packet?.rawECGDataRaw, [1, 2, 3, 4]) @@ -338,7 +375,7 @@ final class Whoop5EcgTests: XCTestCase { } func testRawFrameRejectsBadCRC() { - let payload = header(samples: 2) + [1, 2, 3, 4] + [0] + let payload = header(samples: 2) + [1, 2, 3, 4] + leadsOffBlock(i: [], q: []) var frame = puffinFrame(type: 0x2F, payload: payload) frame[frame.count - 2] ^= 0xFF XCTAssertNil(Whoop5Ecg.decodeRawFrame(frame, bytesPerSample: 2)) diff --git a/android/app/src/main/java/com/noop/protocol/Whoop5Ecg.kt b/android/app/src/main/java/com/noop/protocol/Whoop5Ecg.kt index b985e0f226..55656c2a29 100644 --- a/android/app/src/main/java/com/noop/protocol/Whoop5Ecg.kt +++ b/android/app/src/main/java/com/noop/protocol/Whoop5Ecg.kt @@ -116,6 +116,10 @@ data class FilteredLabradorPacket( * * The raw blob is opaque: its bytes-per-sample is `rawECGDataRaw.size / numberOfECGSamples`, which means * the blob's LENGTH is not itself on the wire (see [Whoop5Ecg.rawBytesPerSampleCandidates]). + * + * The leads-off block that follows is FIXED-SIZE — [Whoop5Ecg.LEADS_OFF_SLOT_COUNT] slots for I and the + * same again for Q, with [numberOfLeadsOffSamples] selecting how many leading slots are valid. + * [leadsOffIRaw] and [leadsOffQRaw] carry only the VALID values; unused slots are dropped. */ data class RawLabradorPacket( val header: EcgStatusHeader, @@ -141,6 +145,24 @@ object Whoop5Ecg { /** Trailing bytes tolerated after the last decoded field (the puffin pad4 budget). */ const val DEFAULT_MAX_PADDING = 3 + /** + * Slots in the raw record's leads-off diagnostic block, per array. + * + * MEASURED FROM HARDWARE, not attested: two complete type-47 layout-16 flash records captured + * 2026-08-06 (embedded verbatim in the Swift `Whoop5EcgRawHardwareTests`) show the block is + * FIXED-SIZE — the count byte is followed by eleven i16 I slots and then eleven i16 Q slots, present + * in full whether or not `numberOfLeadsOffSamples` fills them, with the unused tail slots zeroed. + * Both counts seen in the capture (10 and 11) place the Q array at the same offset and leave the same + * single trailing byte, which is what identifies the block as fixed rather than packed. + * + * Reading it as PACKED — `count` elements each — was the original assumption, and it is wrong in two + * compounding ways whenever `count < 11`: Q is read two bytes early per missing slot (so it gains a + * spurious leading value and loses its last real one), and the misplaced end-of-record leaves a + * remainder over [DEFAULT_MAX_PADDING], which discarded 227 of the capture's 351 populated records + * before any field was read. + */ + const val LEADS_OFF_SLOT_COUNT = 11 + /** * The bytes-per-sample widths the FILTERED triage admits, in the order it reports them. * @@ -331,14 +353,18 @@ object Whoop5Ecg { if (blobLength > Int.MAX_VALUE - HEADER_LENGTH) return null val rawEnd = HEADER_LENGTH + blobLength.toInt() if (rawEnd < HEADER_LENGTH || rawEnd >= payload.size) return null // the leads-off count byte must fit - // Same domain check as decodeHeader: a negative count would make qEnd negative, slip past the - // `qEnd > size` bound, and throw on subList. On the wire this byte is always 0..255. + // Same domain check as decodeHeader: a negative count would make blockEnd negative, slip past the + // `blockEnd > size` bound, and throw on subList. On the wire this byte is always 0..255, and the + // block holds only LEADS_OFF_SLOT_COUNT slots — a count above that is not a record this layout can + // describe, so it fails closed rather than reading past the block. val leadsOffCount = payload[rawEnd] - if (leadsOffCount !in 0..255) return null + if (leadsOffCount !in 0..LEADS_OFF_SLOT_COUNT) return null + // Both arrays are FIXED-SIZE and always fully present — see [LEADS_OFF_SLOT_COUNT]. The count byte + // selects how many leading slots are VALID; it does not size the block. val iStart = rawEnd + 1 - val qStart = iStart + leadsOffCount * 2 - val qEnd = qStart + leadsOffCount * 2 - if (qEnd > payload.size) return null + val qStart = iStart + LEADS_OFF_SLOT_COUNT * 2 + val blockEnd = qStart + LEADS_OFF_SLOT_COUNT * 2 + if (blockEnd > payload.size) return null val leadsOffI = (0 until leadsOffCount).map { payload[iStart + it * 2] or (payload[iStart + it * 2 + 1] shl 8) } val leadsOffQ = (0 until leadsOffCount).map { payload[qStart + it * 2] or (payload[qStart + it * 2 + 1] shl 8) } @@ -348,7 +374,7 @@ object Whoop5Ecg { numberOfLeadsOffSamples = leadsOffCount, leadsOffIRaw = leadsOffI, leadsOffQRaw = leadsOffQ, - padding = payload.subList(qEnd, payload.size).toList(), + padding = payload.subList(blockEnd, payload.size).toList(), ) } diff --git a/android/app/src/test/java/com/noop/protocol/Whoop5EcgRawHardwareTest.kt b/android/app/src/test/java/com/noop/protocol/Whoop5EcgRawHardwareTest.kt new file mode 100644 index 0000000000..65ac635ab6 --- /dev/null +++ b/android/app/src/test/java/com/noop/protocol/Whoop5EcgRawHardwareTest.kt @@ -0,0 +1,199 @@ +package com.noop.protocol + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNotNull +import org.junit.Test + +/** + * The Kotlin twin of WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift, against the SAME real bytes. + * + * The two complete 1584-byte type-47 layout-16 WHOOP MG flash records embedded here came off hardware on + * 2026-08-06 while the clasp electrodes were held; they are byte-identical to the Swift fixtures. The + * cross-platform parity contract says the two decoders must agree on stored values, and a synthetic + * fixture can only prove they agree about an assumption — these bytes prove they agree about the strap. + * + * Only the load-bearing assertions are mirrored: the fixed-size leads-off block, the resolved sample + * width, and that a populated record is not discarded. The full characterisation (including the header + * misalignment that is still open, and the 18-bit continuity argument) lives on the Swift side. + * + * No scale or unit is claimed for the sample values, and none is applied. + */ +class Whoop5EcgRawHardwareTest { + + /** The payload offset at which the record's numberOfECGSamples lands. NOT PUFFIN_PAYLOAD_START. */ + private val ecgPayloadStart = 17 + + /** unix 1786031627 — numberOfLeadsOffSamples == 11, the block exactly full. */ + private val hex11 = + "aa0128060100cde02f10036a61cd000bae746aeb51010a000100000000ffff00f401834ae78349e98345d5834703834c" + + "53834814834916834a0d83478e834d3e83510a8351bb834fda834a718347d383447c8344f3834ec6834c158347848348" + + "87834785834a2d834978834c33834d39834c07834b178352dc83537e8351fb8352968350db835319835048834540834a" + + "b3834de2834f1b834f0f835133834f31834f1a83515d834e178351f18351a38352628352558352fd834d4c834a698350" + + "de835762834e80834b218349338349e6835095835228834c578351b6834f53834ff58351f4834dcb834a6f8350dc8352" + + "658350d9834b2f834ba8834d50834e73834cd2834d3c8356698355e08356508357a48359bb835af9835a5f8353ae8352" + + "78835bde835c928358c08353a3835693835bae835b7a835c4a835e89835c8a835bb0835f2583608b83610f835ec9835f" + + "ee83611283606f8363e68361cd836cf583649b835d0783637983654d8360c0835f2883608f835a2c8352e78350f98349" + + "f18340e5833e90833c878338e0833677832ec9833094833bd28333be833941833e328343d7834d4e83570c8359228362" + + "0383622283647f83618c8365ce837351836e3e8367e28368fb836cb3836cd5836de9836d8383717a836f7a836dc4836f" + + "8783726783723c8370828375d0837291836de9836f4583740883785c8373eb837d748376af8374b283745f8371ad8373" + + "9a83763e8373eb8375d58375b48374f1837c6e838486837fc0837d3b8379cd837402837747837f4e83765783755b837a" + + "ab8386ab83876b837c6d8375ef8376cc837d41837b33837f4a837f37837c5583811a837d95837b77837cc1837dc88381" + + "11837c56837883837a32837da8837f2d8381ec837c388382cb838541837fef8378bd8374ac83731f837914837d9f837b" + + "93837f35837e22838094838290837e6b837b12837dac837fdc838048837fda83808f8380018381ee83822e8384ed8387" + + "7883836b837f858382428378e3837bf38382e683865183850583810883839283886583840c8388568389b4838af6838e" + + "148388b783875983893583861883865c838e6283915d838a58838d748391e583904b8393288397618391038391e98390" + + "e383905f838c82838b178390f683947e83963883963383981d839973839b94839974839894839b0f83952b839453839b" + + "78839d39839b708399028394bc839038838f648395be839a4a839ad0839f44839c18839b008399a98398ee839bbe839c" + + "6983967a8392c38396b28396088395218395d1839a16839fe3839e5f83997483948f83974c839afc8396a2839b40839e" + + "48839aec839903839a8f8399ef839f09839aa58397dc839c9483a4f383a2bd83a3e683a197839a968399c4839821839b" + + "8c839dd58398488398528394eb839806839b0d839ae2839a388396f1839521838de2839230839a1983a05083a3e58399" + + "fe83981b83982b8396be83a224839df2839a04839ecc8399c9839671839a6d83953a8396c1839a6583a05483a22e83a1" + + "3483a2e983a60083997983984883990d839e6b839ee783a030839dc183a0b983a68583a0c483a40583a0c6839f1d839c" + + "5683a05e83a0ac839d93839bd1839ebc83a53083aa8b83a71083a49883a0ff83a11f83a22f83a5ac83a69983a54383a3" + + "2b839b30839e0d839e2f83a040839fae839f99839b5b83a93f83aeb383a61783a78683a7d383a88383abf683aa6e83a6" + + "0383a79983a4cf83a60983a5dd83a7d083a48b83a61683a8c883a88f83a6f683a80a83a8ef83a5b083a73483aa4683a9" + + "1b83aca583b07883a4be839ed383a1f483a30783a3ea83a79c83a67d83a50383a70383b0c083a64283abfd83aed883a9" + + "1583ac5783aeea83ad7b83ae2683a99283a64183abca83ac8783aca783b38083b0b183a53183a42a83a7d383a69283a8" + + "e083ab2983aba083b1d283ad2883a4cd83a62183ab2c83a9fe83a478839d8983a4cb83aca483a393839ede83a2e483ab" + + "c383a92f83a46083a49983ad4c83ad5083a7a883a53683a59883aa4783a8db83a45583a47183a59383a6ae83ac7c83b0" + + "1d83ae9783ae7383aba783a80483aaf483b04c83af0c83a95083a77d83a61183a61b83a48383ad7183aed083a8bc0b43" + + "004300430043004300430043004300430043004300f8fff8fff8fff8fff8fff8fff8fff7fff7fff7fff7ff00313774b4" + + /** unix 1786031642 — numberOfLeadsOffSamples == 10, one slot unused. */ + private val hex10 = + "aa0128060100cde02f10037961cd001aae746aeb51030a000124000000ffff00f40183f47c83f61983f31b83f42a83f5" + + "6083f8bc83f3a683f10a83f15883f2d583f71683f45783f58783f26e83f18983efdb83efc483f51583f5e683f6f283f4" + + "7d83f59e83f76283f59883f66383f1d583f4d783fbbc83f59183f5a083f4f483f5c683f7ab83eed783ecbf83f19083f0" + + "5a83f1bd83efa383f30b83f48083f00483f37583f3dd83f1e283f0d983f1a983f2f183f0f283f07283ef9983ef5d83f2" + + "7c83f3ec83f39d83f2e083f01583eece83ed8b83f06883f10383f19683f42983f08b83f35783f30383edae83ec4083ee" + + "3583f05e83f16083f3f383ee9b83ed5083f03883ecec83ee8d83f31b83eee883eb4883e87883e0e283e9f483f0d183f0" + + "7f83effe83ed8983eef183e9dd83ea0b83ed9783ef3083eb8883e9ad83f03983f6d983f33c83ee9583eb6a83ee9483f3" + + "7f83f25483ee0083ee1283ebf383eda583ef0583f05683f3ab83ef1a83f2cb83f45583f48383f0c083eacf83ee5583f2" + + "9483f16e83ec5c83eaa683ecb583ecd283f0cf83f38983f39183e9c583e7dd83e9f783e8a983e41783e34b83e1cd83df" + + "1083dde183d17283d22283c94183bfe583c02483bd9d83bfa683c39483c36183bde583c4c783c7ae83d37d83de9783e3" + + "0683e62283e8bd83e78d83eb4983e8e683e99283eaac83ecac83ed1a83efe083efdc83ecdb83f35483f20983ef6783f2" + + "8183f56983f26183f40283f82583f44d83eb8d83eb0f83eb3283edc283f80f83fae583f5c183f4f383fc3783fae583f3" + + "1a83eeea83ff7883fe3583f5e883fac783ff5983f9ae83f60383f7b883fb8d83f9f083feb783fee18003788009ce8006" + + "f48002a38005f28006ad800522800cb5800f30800ca7800dda8009c68005f18004e180031180093e8010658010918009" + + "558005f98008da801274800e0e8002648004ce8003d9800b73800dd4800d2680133b800b908004e783f8dc83fabe8004" + + "a6800c9a80180c8016b280180b80170b801340801a21801d1e801947801acd801ba480177c8020b68022a2802cba802c" + + "ec802a6b8030688036da8039fd8040a68043f4804868804b7b804463804cec804fdd8058308055be8055618050d5804f" + + "85804ebc80556e8062e9805ce18058c1805908805632805eca806349805ee6805a13805b86805dd2806925806655805c" + + "4780587480620580629b8062358062a4806d0c8063c3805b65805f0d805e4e806b00806a8680656c805fbb805851805b" + + "4b8057f780596a8061a1806570805e02805e7f8063e58065d48061e680613e805f8b805e5e80579b80536c80543c8053" + + "e4805d83805aee80569880577f80567e805b73805074804fdd8050ec804f74805355804ef2804f46804f52804d718050" + + "1f804f9b8058a28050758044c5804ccd804f49804f3a804b0e804bd5804da380451580467a804b6a804961804959804a" + + "26804c118041fc8041458042e38042918045b7804de88047b28044e68041498035ff80359d8038a3803b81803de78042" + + "8a803b1c8033b68034c6803919803992802fc880334e8036a780350c80397b8041d1803998803d6c8032a68025428033" + + "74803db9803761803f7980386c802dc0802f2c8035b08033b8803496802935801bac8021d6802e9b802d30802fbe802b" + + "eb803439802d85802bfe802eb0802fb4802f938030ec802bd0802932802c4680339f802ef5802b368024cc8025e5802a" + + "ba802a29802b80802f3780362c8034b38031e28033e2802eda80261b8022908024cd8029438023448027d3802e948027" + + "5a8024088022928029038033dc80330f80290b802fbe8023858016cb8021fe80274f802304801ed68022ff802df38031" + + "4380318a802c9680240780264a802766802a47802fa8802d808029958028d0801eeb8017d8801f00802854802ab28034" + + "3b802ff78022b2801e018020fd8029f28023ac8023208019a68019c08021cc8028858025ed801afc800c5b801e09801f" + + "fa801df4801df08016aa8012598017e2801c9480233a801f6f801d2f8011d080080e8014a2801d018022a98016218005" + + "0b800589800dde8019cf8020168020a2800e2b8003ac800ba6801956801838800ebe800c68800d218009ee800e4b0a43" + + "004300430043004300420042004200420042000000f7fff7fff7fff7fff7fff7fff6fff6fff6fff6ff000000fe104255" + + private fun bytes(h: String): ByteArray = + ByteArray(h.length / 2) { ((h[it * 2].digitToInt(16) shl 4) or h[it * 2 + 1].digitToInt(16)).toByte() } + + private val frame11 get() = bytes(hex11) + private val frame10 get() = bytes(hex10) + + /** 24-bit big-endian field, low 18 bits signed. Measured from these bytes; not a vendor fact. */ + private fun sample18(blob: List, i: Int): Int { + val o = i * 3 + var v = ((blob[o] and 0x03) shl 16) or (blob[o + 1] shl 8) or blob[o + 2] + if (v and 0x20000 != 0) v -= 0x40000 + return v + } + + /** + * Regression: a populated real record is NOT discarded, and resolves to width 3. Under the packed + * reading the count=10 record left a 5-byte remainder against DEFAULT_MAX_PADDING and was rejected + * outright — 227 of the capture's 351 populated records. + */ + @Test + fun populatedRealRecordIsNotDiscarded() { + for (frame in listOf(frame11, frame10)) { + assertEquals(1584, frame.size) + assertEquals(47, frame[8].toInt() and 0xFF) + assertEquals(16, frame[9].toInt() and 0xFF) + val payload = Whoop5Ecg.innerPayload(frame, ecgPayloadStart) + assertNotNull("real capture must pass both CRCs", payload) + assertEquals(1563, payload!!.size) + assertEquals(listOf(3), Whoop5Ecg.rawBytesPerSampleCandidates(payload)) + val packet = Whoop5Ecg.decodeRaw(payload) + assertNotNull(packet) + assertEquals(500, packet!!.header.numberOfECGSamples) + assertEquals(1500, packet.rawECGDataRaw.size) + assertEquals(3, packet.bytesPerSample) + assertEquals(listOf(0x00), packet.padding) + } + } + + /** The full block: eleven valid I and eleven valid Q. */ + @Test + fun countElevenRecordDecodesElevenOfEach() { + val payload = Whoop5Ecg.innerPayload(frame11, ecgPayloadStart)!! + val packet = Whoop5Ecg.decodeRaw(payload, bytesPerSample = 3)!! + assertEquals(11, packet.numberOfLeadsOffSamples) + assertEquals(List(11) { 67 }, packet.leadsOffIRaw) + assertEquals(List(7) { 65528 } + List(4) { 65527 }, packet.leadsOffQRaw) + assertEquals( + listOf(-46361, -46615, -47659, -47357, -45997, -47084, -46826, -46579, -47218, -45762, -44790, -44613), + (0 until 12).map { sample18(packet.rawECGDataRaw, it) }, + ) + } + + /** + * The partially filled block: ten valid values off the FRONT of each fixed array. The packed reading + * started Q two bytes early, gaining a spurious leading 0 and losing the tenth real Q value. + */ + @Test + fun countTenRecordDecodesTenOfEachWithQAligned() { + val payload = Whoop5Ecg.innerPayload(frame10, ecgPayloadStart)!! + val packet = Whoop5Ecg.decodeRaw(payload, bytesPerSample = 3)!! + assertEquals(10, packet.numberOfLeadsOffSamples) + assertEquals(listOf(67, 67, 67, 67, 67, 66, 66, 66, 66, 66), packet.leadsOffIRaw) + assertEquals( + listOf(65527, 65527, 65527, 65527, 65527, 65527, 65526, 65526, 65526, 65526), + packet.leadsOffQRaw, + ) + assertEquals(65526, packet.leadsOffQRaw.last()) // previously lost into the "padding" + assertEquals(listOf(0x00), packet.padding) + assertEquals( + listOf(-2948, -2535, -3301, -3030, -2720, -1860, -3162, -3830, -3752, -3371, -2282, -2985), + (0 until 12).map { sample18(packet.rawECGDataRaw, it) }, + ) + // 17 + 1500 + 1 + 22 + 22 + 1 = 1563, with the block charged at its FIXED size. + val accounted = Whoop5Ecg.HEADER_LENGTH + packet.rawECGDataRaw.size + 1 + + Whoop5Ecg.LEADS_OFF_SLOT_COUNT * 2 + Whoop5Ecg.LEADS_OFF_SLOT_COUNT * 2 + packet.padding.size + assertEquals(payload.size, accounted) + } + + /** + * The block's geometry, read straight off the frame with no decoder involved: the count byte at + * frame[1534], I slots at frame[1535], Q slots at frame[1557] — the SAME offsets under both counts, + * which is what identifies the block as fixed rather than packed. + */ + @Test + fun leadsOffBlockIsAtTheSameFixedOffsetsUnderBothCounts() { + assertEquals(11, Whoop5Ecg.LEADS_OFF_SLOT_COUNT) + assertEquals(11, frame11[1534].toInt() and 0xFF) + assertEquals(10, frame10[1534].toInt() and 0xFF) + + fun slots(frame: ByteArray, start: Int) = (0 until 11).map { + (frame[start + it * 2].toInt() and 0xFF) or ((frame[start + it * 2 + 1].toInt() and 0xFF) shl 8) + } + assertEquals(List(11) { 67 }, slots(frame11, 1535)) + assertEquals(List(7) { 65528 } + List(4) { 65527 }, slots(frame11, 1557)) + assertEquals(listOf(67, 67, 67, 67, 67, 66, 66, 66, 66, 66, 0), slots(frame10, 1535)) + assertEquals( + listOf(65527, 65527, 65527, 65527, 65527, 65527, 65526, 65526, 65526, 65526, 0), + slots(frame10, 1557), + ) + assertEquals(frame10.size, 1579 + 1 + 4) + } +} diff --git a/android/app/src/test/java/com/noop/protocol/Whoop5EcgTest.kt b/android/app/src/test/java/com/noop/protocol/Whoop5EcgTest.kt index 97029de784..47171c61bc 100644 --- a/android/app/src/test/java/com/noop/protocol/Whoop5EcgTest.kt +++ b/android/app/src/test/java/com/noop/protocol/Whoop5EcgTest.kt @@ -49,6 +49,18 @@ class Whoop5EcgTest { private fun u16le(values: List): List = values.flatMap { listOf(it and 0xFF, (it shr 8) and 0xFF) } + /** + * The raw record's leads-off tail: the count byte, then [Whoop5Ecg.LEADS_OFF_SLOT_COUNT] i16 I slots, + * then the same again for Q. The block is FIXED-SIZE on real hardware (see the Swift + * `Whoop5EcgRawHardwareTests`), so these fixtures zero-fill the slots the count does not reach instead + * of packing the arrays to the count. + */ + private fun leadsOffBlock(i: List, q: List): List { + require(i.size == q.size && i.size <= Whoop5Ecg.LEADS_OFF_SLOT_COUNT) + val pad = List(Whoop5Ecg.LEADS_OFF_SLOT_COUNT - i.size) { 0 } + return listOf(i.size) + u16le(i + pad) + u16le(q + pad) + } + private fun puffinFrame(type: Int, payload: List): ByteArray = Framing.puffinCommandFrame( cmd = 0x00, seq = 0x01, @@ -210,7 +222,7 @@ class Whoop5EcgTest { val rawBlob = (0 until 12).toList() // 4 samples × 3 bytes val leadsOffI = listOf(1, 2) val leadsOffQ = listOf(3, 4) - val payload = header(samples = 4) + rawBlob + listOf(2) + u16le(leadsOffI) + u16le(leadsOffQ) + val payload = header(samples = 4) + rawBlob + leadsOffBlock(leadsOffI, leadsOffQ) val packet = Whoop5Ecg.decodeRaw(payload, bytesPerSample = 3) assertNotNull(packet) @@ -222,14 +234,31 @@ class Whoop5EcgTest { assertEquals(3, packet?.bytesPerSample) // count ÷ numberOfECGSamples } + /** A zero count still carries the full fixed block — the slots are simply all unused. */ @Test fun rawWithNoLeadsOffSamples() { - val payload = header(samples = 2) + listOf(0xAA, 0xBB, 0xCC, 0xDD) + listOf(0) + val payload = header(samples = 2) + listOf(0xAA, 0xBB, 0xCC, 0xDD) + leadsOffBlock(emptyList(), emptyList()) val packet = Whoop5Ecg.decodeRaw(payload, bytesPerSample = 2) assertEquals(0, packet?.numberOfLeadsOffSamples) assertEquals(emptyList(), packet?.leadsOffIRaw) assertEquals(emptyList(), packet?.leadsOffQRaw) assertEquals(listOf(0xAA, 0xBB, 0xCC, 0xDD), packet?.rawECGDataRaw) + assertEquals(emptyList(), packet?.padding) + } + + /** + * The valid values come off the FRONT of each fixed array, and the unused slots are dropped rather + * than shifting Q or leaking into the padding — the defect the real capture exposed. + */ + @Test + fun rawPartiallyFilledLeadsOffBlockKeepsQAligned() { + val payload = header(samples = 2) + listOf(0, 0, 0, 0) + + leadsOffBlock(listOf(11, 12, 13), listOf(21, 22, 23)) + val packet = Whoop5Ecg.decodeRaw(payload, bytesPerSample = 2) + assertEquals(3, packet?.numberOfLeadsOffSamples) + assertEquals(listOf(11, 12, 13), packet?.leadsOffIRaw) + assertEquals(listOf(21, 22, 23), packet?.leadsOffQRaw) + assertEquals(emptyList(), packet?.padding) } @Test @@ -238,6 +267,17 @@ class Whoop5EcgTest { assertNull(Whoop5Ecg.decodeRaw(payload, bytesPerSample = 2)) } + /** + * The block holds a fixed number of slots, so a count above it cannot describe this layout and must + * fail closed rather than read past the block. + */ + @Test + fun rawRejectsLeadsOffCountAboveTheBlockCapacity() { + val payload = header(samples = 2) + listOf(0, 0, 0, 0) + + listOf(Whoop5Ecg.LEADS_OFF_SLOT_COUNT + 1) + List(Whoop5Ecg.LEADS_OFF_SLOT_COUNT * 4 + 8) { 0 } + assertNull(Whoop5Ecg.decodeRaw(payload, bytesPerSample = 2)) + } + @Test fun rawRejectsMissingLeadsOffCountByte() { val payload = header(samples = 2) + listOf(0, 0, 0, 0) @@ -259,13 +299,13 @@ class Whoop5EcgTest { @Test fun rawRejectsShortHeaderAndZeroWidth() { assertNull(Whoop5Ecg.decodeRaw(listOf(1, 2, 3), bytesPerSample = 2)) - val payload = header(samples = 2) + listOf(0, 0, 0, 0) + listOf(0) + val payload = header(samples = 2) + listOf(0, 0, 0, 0) + leadsOffBlock(emptyList(), emptyList()) assertNull(Whoop5Ecg.decodeRaw(payload, bytesPerSample = 0)) } @Test fun rawSampleWidthCandidatesAreEnumeratedNotGuessed() { - val payload = header(samples = 4) + List(8) { 0x11 } + listOf(1) + u16le(listOf(7)) + u16le(listOf(8)) + val payload = header(samples = 4) + List(8) { 0x11 } + leadsOffBlock(listOf(7), listOf(8)) val candidates = Whoop5Ecg.rawBytesPerSampleCandidates(payload) assertTrue("width 2 must be structurally admissible", candidates.contains(2)) if (candidates.size == 1) { @@ -277,10 +317,12 @@ class Whoop5EcgTest { @Test fun rawAmbiguousBufferRefusesToDecode() { - val payload = header(samples = 1) + List(6) { 0 } - val candidates = Whoop5Ecg.rawBytesPerSampleCandidates(payload, maxPadding = 8) - assertTrue("fixture is meant to be ambiguous", candidates.size > 1) - assertNull(Whoop5Ecg.decodeRaw(payload, maxPadding = 8)) + // With a single sample, an all-zero tail long enough to hold the fixed leads-off block leaves + // every width inside the padding tolerance. + val payload = header(samples = 1) + List(49) { 0 } + val candidates = Whoop5Ecg.rawBytesPerSampleCandidates(payload) + assertEquals("fixture is meant to be ambiguous", listOf(1, 2, 3, 4), candidates) + assertNull(Whoop5Ecg.decodeRaw(payload)) } // MARK: - Frame level (CRC gating) @@ -328,7 +370,7 @@ class Whoop5EcgTest { @Test fun rawFrameDecodesThroughAValidPuffinEnvelope() { - val payload = header(samples = 2) + listOf(1, 2, 3, 4) + listOf(1) + u16le(listOf(5)) + u16le(listOf(6)) + val payload = header(samples = 2) + listOf(1, 2, 3, 4) + leadsOffBlock(listOf(5), listOf(6)) val frame = puffinFrame(0x2F, payload) val packet = Whoop5Ecg.decodeRawFrame(frame, bytesPerSample = 2) assertEquals(listOf(1, 2, 3, 4), packet?.rawECGDataRaw) @@ -338,7 +380,7 @@ class Whoop5EcgTest { @Test fun rawFrameRejectsBadCrc() { - val payload = header(samples = 2) + listOf(1, 2, 3, 4) + listOf(0) + val payload = header(samples = 2) + listOf(1, 2, 3, 4) + leadsOffBlock(emptyList(), emptyList()) val frame = puffinFrame(0x2F, payload) frame[frame.size - 2] = (frame[frame.size - 2].toInt() xor 0xFF).toByte() assertNull(Whoop5Ecg.decodeRawFrame(frame, bytesPerSample = 2)) From 2b6b04632cb11ac56d6754e473b6f4edc9a2edf9 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Thu, 6 Aug 2026 15:09:43 -0400 Subject: [PATCH 168/171] MG ECG: read the sample region as fixed-size, not packed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The capture's one partly-filled raw record (numberOfECGSamples == 245) was the last populated record `Whoop5Ecg` still discarded. It is the same 1584-byte type-47 layout-16 frame as every full 500-sample record: its sample data stops after 245 * 3 bytes, frame[769..1534) is zeroed, and its leads-off block sits at the SAME fixed frame offset 1534 as both full records'. So the SAMPLE region is a fixed-size container too — `numberOfECGSamples` says how many of its slots are valid, exactly the way `numberOfLeadsOffSamples` does one field over. `decodeRaw` therefore locates the leads-off block from the END of the payload rather than from `headerLength + n * bytesPerSample`, and carries only the valid `n * width` bytes as samples. Under the old reading the count byte landed on a zero at payload[752], which yielded an empty leads-off block and 766 trailing bytes against `defaultMaxPadding = 3` — rejected before any field was read. No region length is hardcoded: the block's own fixed size anchors it, candidate ends are tried closest-to-the-end first, and the span between the declared samples and the block must be all zero. That zero-fill check is what keeps the width enumeration honest — a width that under-reads leaves real sample bytes in what it calls unused capacity and is rejected there. Full records are unaffected: their region is exactly full, so every existing decode is byte-for-byte the same. `RawLabradorPacket` gains `unusedSampleBytes` (counted, not carried) so the record still accounts for every byte, plus `sampleRegionBytes`. STILL OPEN, and reported rather than papered over: a partly-filled record cannot determine its own sample WIDTH. 245 samples inside a 1500-byte region fits 3 bytes per sample and 4 alike, so `rawBytesPerSampleCandidates` returns [3, 4] for it and the auto-width overload declines. The width is a property of the stream — the 350 full records resolve to [3] unaided, and all 351 populated records decode at that explicit width. No scale or unit is claimed for the sample values, and none is applied. No encoding claim is made for the new record's samples: it is the first half second of a reading and the input is still settling, so its tag bytes are not the uniform 0b100000 the full records carry (pinned as a census, not an interpretation). Verification: the n=245 record is embedded verbatim in both hardware test files, byte-identical across platforms. `swift test` 576 passed; `./gradlew testFullDebugUnitTest` 3511 passed. Kotlin twin mirrored per the parity contract. No hardware retest needed — this is pure decode over bytes already captured. --- .../Sources/WhoopProtocol/Whoop5Ecg.swift | 150 +++++++++++----- .../Whoop5EcgRawHardwareTests.swift | 167 +++++++++++++++++- .../WhoopProtocolTests/Whoop5EcgTests.swift | 39 +++- .../main/java/com/noop/protocol/Whoop5Ecg.kt | 139 +++++++++++---- .../noop/protocol/Whoop5EcgRawHardwareTest.kt | 119 ++++++++++++- .../java/com/noop/protocol/Whoop5EcgTest.kt | 43 ++++- 6 files changed, 559 insertions(+), 98 deletions(-) diff --git a/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Ecg.swift b/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Ecg.swift index 85bd755b03..4c671c11b2 100644 --- a/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Ecg.swift +++ b/Packages/WhoopProtocol/Sources/WhoopProtocol/Whoop5Ecg.swift @@ -210,12 +210,26 @@ public struct FilteredLabradorPacket: Equatable, Sendable { /// means the blob's LENGTH is not itself on the wire, so a decode needs that width supplied or resolved /// (see `rawBytesPerSampleCandidates`). /// -/// The leads-off block that follows is FIXED-SIZE — `Whoop5Ecg.leadsOffSlotCount` slots for I and the -/// same again for Q, with `numberOfLeadsOffSamples` selecting how many leading slots are valid. +/// BOTH variable-length-looking regions are really FIXED-SIZE containers with a count that says how many +/// leading slots are valid: +/// • the sample region — see `unusedSampleBytes` and `Whoop5Ecg.decodeRaw(payload:bytesPerSample:)`; +/// • the leads-off block — `Whoop5Ecg.leadsOffSlotCount` slots for I and the same again for Q, with +/// `numberOfLeadsOffSamples` selecting how many leading slots are valid. +/// In both, the unused slots are dropped rather than carried, so a packet's arrays only ever hold values +/// the record says are real. public struct RawLabradorPacket: Equatable, Sendable { public let header: EcgStatusHeader /// Opaque. The container width and encoding are unattested, so the bytes are carried verbatim. + /// + /// Exactly `numberOfECGSamples * bytesPerSample` bytes — the VALID part of the sample region, which on + /// a partly-filled record is shorter than the region itself. public let rawECGDataRaw: [UInt8] + /// Bytes of the fixed sample region that `numberOfECGSamples` did not fill. Zero for a full record. + /// + /// The strap zero-fills them, and the decoder requires that (a non-zero byte past the declared samples + /// is how a mis-placed record end is caught). They are counted, not carried, so + /// `headerLength + rawECGDataRaw.count + unusedSampleBytes` is where the leads-off block begins. + public let unusedSampleBytes: Int public let numberOfLeadsOffSamples: UInt8 /// The VALID leads-off I values — `numberOfLeadsOffSamples` of them, taken from the front of the /// record's fixed `Whoop5Ecg.leadsOffSlotCount`-slot block. Unused slots are dropped, not carried. @@ -231,10 +245,15 @@ public struct RawLabradorPacket: Equatable, Sendable { return rawECGDataRaw.count / n } - public init(header: EcgStatusHeader, rawECGDataRaw: [UInt8], numberOfLeadsOffSamples: UInt8, + /// The full sample region the record reserved, valid part plus unused capacity. + public var sampleRegionBytes: Int { rawECGDataRaw.count + unusedSampleBytes } + + public init(header: EcgStatusHeader, rawECGDataRaw: [UInt8], unusedSampleBytes: Int = 0, + numberOfLeadsOffSamples: UInt8, leadsOffIRaw: [UInt16], leadsOffQRaw: [UInt16], padding: [UInt8]) { self.header = header self.rawECGDataRaw = rawECGDataRaw + self.unusedSampleBytes = unusedSampleBytes self.numberOfLeadsOffSamples = numberOfLeadsOffSamples self.leadsOffIRaw = leadsOffIRaw self.leadsOffQRaw = leadsOffQRaw @@ -256,6 +275,9 @@ public enum Whoop5Ecg { /// Trailing bytes tolerated after the last decoded field. The puffin inner record is padded to a /// 4-byte boundary (`puffinCommandFrame`'s pad4), so a well-formed record leaves at most 3 spare /// bytes. Callers scanning an unfamiliar layout can widen it. + /// + /// On the RAW side this is not just a tolerance but the search bound: `decodeRaw` locates the + /// leads-off block by trying each of these end positions, closest to the end of the payload first. public static let defaultMaxPadding = 3 /// Slots in the raw record's leads-off diagnostic block, per array. @@ -438,14 +460,45 @@ public enum Whoop5Ecg { // MARK: Raw decode + /// Bytes the raw record's leads-off block occupies: the count byte, then a full I array, then a full + /// Q array. Fixed — see `leadsOffSlotCount`. + public static let leadsOffBlockLength = 1 + leadsOffSlotCount * 4 + /// Decode a `RawLabradorPacket` from the inner record's PAYLOAD with an explicit sample width. /// /// `bytesPerSample` has to be supplied because the raw blob's length is NOT on the wire: the client /// derives the width by dividing the blob it already holds by `numberOfECGSamples`, which a /// byte-stream decoder cannot do until it knows where the blob ends. `rawBytesPerSampleCandidates` /// enumerates the widths a given buffer admits. - public static func decodeRaw(payload: [UInt8], bytesPerSample: Int) -> RawLabradorPacket? { - guard bytesPerSample > 0, let header = EcgStatusHeader(payload: payload) else { return nil } + /// + /// **The sample region is a fixed-size container, not `n * bytesPerSample` bytes.** MEASURED FROM + /// HARDWARE, on the same capture and by the same argument as `leadsOffSlotCount`: the capture's one + /// partly-filled record (`numberOfECGSamples == 245`, embedded in `Whoop5EcgRawHardwareTests`) is the + /// same 1584-byte frame as every full 500-sample record, its sample data stops after `245 * 3` bytes, + /// the rest of the region is zeroed, and its leads-off block sits at the SAME frame offset as every + /// other record's. So `numberOfECGSamples` says how many of the region's slots are valid in exactly + /// the way `numberOfLeadsOffSamples` says how many leads-off slots are — one field over, one layer up. + /// + /// That is why the leads-off block is located from the END of the payload rather than from + /// `headerLength + n * bytesPerSample`: the region's LENGTH is no more on the wire than the width is, + /// and a decoder that assumes the region is exactly full lands the count byte inside the zero fill + /// (which is what discarded this record — it read a zero as the count, an empty leads-off block, and + /// left 766 bytes of "padding"). No region length is hardcoded here; the block's own fixed size + /// anchors it. + /// + /// The candidate ends are tried closest-to-the-end first, so the record claims the least padding it + /// can — the puffin pad4 filler is minimal by construction, and a longer claim would eat real bytes. + /// That ordering is a tie-break, not a proof: an all-zero tail can also validate a block placed a + /// byte or two earlier (reading a zero as `numberOfLeadsOffSamples` and empty arrays), and the + /// tightest placement is the one that explains the most bytes rather than the fewest. + /// Two things then have to hold, and either failing moves the search on: the region must be long + /// enough for the declared samples, and every byte between them and the block must be ZERO. That + /// zero-fill check is what keeps the width enumeration honest — a width that under-reads the samples + /// leaves real sample bytes in the unused span and is rejected there. + public static func decodeRaw(payload: [UInt8], bytesPerSample: Int, + maxPadding: Int = defaultMaxPadding) -> RawLabradorPacket? { + guard bytesPerSample > 0, maxPadding >= 0, + let header = EcgStatusHeader(payload: payload) else { return nil } let n = Int(header.numberOfECGSamples) // `bytesPerSample` is caller-supplied on a public API, and `numberOfECGSamples` comes off the // wire — so BOTH the product and the following add are checked rather than assumed. Either would @@ -454,37 +507,46 @@ public enum Whoop5Ecg { // the multiply alone is not enough.) let (blobLength, mulOverflow) = n.multipliedReportingOverflow(by: bytesPerSample) guard !mulOverflow else { return nil } - let (rawEnd, addOverflow) = headerLength.addingReportingOverflow(blobLength) - guard !addOverflow else { return nil } - // The leads-off count byte must itself be inside the buffer. - guard rawEnd >= headerLength, rawEnd < payload.count else { return nil } - let leadsOffCount = Int(payload[rawEnd]) - // The block holds `leadsOffSlotCount` slots per array; a count that overruns it is not a record - // this layout can describe, so it fails closed rather than reading past the block. - guard leadsOffCount <= leadsOffSlotCount else { return nil } - // Both arrays are FIXED-SIZE and always fully present — see `leadsOffSlotCount`. The count byte - // selects how many leading slots are VALID; it does not size the block. - let iStart = rawEnd + 1 - let qStart = iStart + leadsOffSlotCount * 2 - let blockEnd = qStart + leadsOffSlotCount * 2 - guard blockEnd <= payload.count else { return nil } - - var leadsOffI = [UInt16]() - var leadsOffQ = [UInt16]() - leadsOffI.reserveCapacity(leadsOffCount) - leadsOffQ.reserveCapacity(leadsOffCount) - for i in 0..= headerLength, sampleEnd <= payload.count else { return nil } + + for pad in 0...maxPadding { + let blockEnd = payload.count - pad + let countIndex = blockEnd - leadsOffBlockLength + // The block must sit entirely after the declared samples. This also bounds `countIndex` from + // below, since `sampleEnd >= headerLength >= 0`. + guard countIndex >= sampleEnd else { continue } + // Unused sample capacity is zero-filled on the wire. A non-zero byte here means the block is + // not at this offset — including the case where it holds the samples a wider width would read. + guard payload[sampleEnd.. [Int] { - widths.filter { width in - guard let packet = decodeRaw(payload: payload, bytesPerSample: width) else { return false } - return packet.padding.count <= maxPadding - } + widths.filter { decodeRaw(payload: payload, bytesPerSample: $0, maxPadding: maxPadding) != nil } } /// Decode a raw record only when the buffer admits exactly ONE sample width. Ambiguous or @@ -509,14 +574,15 @@ public enum Whoop5Ecg { maxPadding: Int = defaultMaxPadding) -> RawLabradorPacket? { let candidates = rawBytesPerSampleCandidates(payload: payload, widths: widths, maxPadding: maxPadding) guard candidates.count == 1 else { return nil } - return decodeRaw(payload: payload, bytesPerSample: candidates[0]) + return decodeRaw(payload: payload, bytesPerSample: candidates[0], maxPadding: maxPadding) } /// CRC-gated raw decode straight off a complete 5/MG frame, with an explicit sample width. public static func decodeRawFrame(_ frame: [UInt8], bytesPerSample: Int, - payloadStart: Int = puffinPayloadStart) -> RawLabradorPacket? { + payloadStart: Int = puffinPayloadStart, + maxPadding: Int = defaultMaxPadding) -> RawLabradorPacket? { guard let payload = innerPayload(frame, payloadStart: payloadStart) else { return nil } - return decodeRaw(payload: payload, bytesPerSample: bytesPerSample) + return decodeRaw(payload: payload, bytesPerSample: bytesPerSample, maxPadding: maxPadding) } // MARK: Discovery diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift index 648d1d71d6..4782cfe5ab 100644 --- a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift @@ -3,7 +3,7 @@ import XCTest /// Characterisation of REAL WHOOP MG ECG flash records against this package's own decoder. /// -/// Unlike `Whoop5EcgTests`, whose fixtures are all synthetic, every byte here came off hardware: two +/// Unlike `Whoop5EcgTests`, whose fixtures are all synthetic, every byte here came off hardware: three /// complete 1584-byte type-47 records (layout version 16, `frame[9] == 16`) captured 2026-08-06 while /// the MG clasp electrodes were held, embedded verbatim as hex. Nothing in this file changes production /// behaviour — it pins what `Whoop5Ecg` ACTUALLY does when handed these bytes, including the places @@ -28,12 +28,19 @@ import XCTest /// • The sample blob is 3 bytes per sample, and the payload is 18 signed bits big-endian inside that /// 24-bit field — measured here by continuity, not assumed, and not attributed to any vendor /// source. No scale or unit is claimed for the values, and none is applied. -/// • STILL OPEN — the one populated record that does not decode is the capture's short final record -/// (`numberOfECGSamples == 245`). Its leads-off block sits at the SAME fixed frame offset 1534 as -/// every other record, with bytes 769...1533 zeroed, so the sample REGION is fixed-size too and the -/// blob length is not `n * bytesPerSample` for a partly-filled record. That record is not embedded -/// here and the width-from-length premise `rawBytesPerSampleCandidates` rests on does not hold for -/// it; it keeps failing closed rather than being papered over. +/// • The SAMPLE region is fixed-size too. The capture's one partly-filled record +/// (`numberOfECGSamples == 245`, embedded here) puts its leads-off block at the SAME frame offset +/// 1534 as every other record, with bytes 769...1533 zeroed — so the blob is not `n * bytesPerSample` +/// bytes long, it is `n` valid slots inside a region sized like everyone else's. Reading the region +/// as exactly full landed the count byte on a zero at payload[752], read an empty leads-off block and +/// left 766 bytes of "padding", which is why this was the last populated record still discarded. +/// FIXED — `decodeRaw` now locates the block from the END of the payload; all 351 populated records +/// decode at width 3. +/// • STILL OPEN — a partly-filled record cannot determine its own sample WIDTH. 245 samples inside a +/// 1500-byte region is equally consistent with 3 bytes per sample and with 4, so +/// `rawBytesPerSampleCandidates` reports `[3, 4]` for it and the auto-width overload declines rather +/// than guessing. The 350 full records resolve to `[3]` unaided, and that is where the width comes +/// from — it is a property of the stream, not of one record. final class Whoop5EcgRawHardwareTests: XCTestCase { // MARK: - Real captured records @@ -116,6 +123,48 @@ final class Whoop5EcgRawHardwareTests: XCTestCase { 004300430043004300420042004200420042000000f7fff7fff7fff7fff7fff7fff6fff6fff6fff6ff000000fe104255 """ + /// Record at unix 1786031625 — the capture's one PARTLY FILLED record, `numberOfECGSamples == 245`. + /// + /// It is the first record of the second reading: the empty record before it is at 1786031624 and the + /// first full 500-sample one is at 1786031626, so the strap began the reading roughly half a second + /// into this record's second and wrote 245 of the region's 500 slots. Same 1584-byte frame, same + /// layout, same leads-off offsets as both full records above. + private static let recordPartlyFilled245 = """ + aa0128060100cde02f10036861cd0009ae746aeb51010a000100000000ffff00f50083ae1e8259e48210008210008210 + 008210008210008210008210008210008210008210008210008210008210008210008210008210008210008210008210 + 008210008210008210008210008210008210008210008210008210008210008210008210008210008210008210008210 + 008210008210008210008210008210008210008210008210008210008210008210008210008210008210008210008210 + 00821000821000821000821000821000821000821000821000821000821000821000821000c21000c21000c21000c210 + 00c21085c25696c328a5c39fc1c3cda0c3e169c3e9d9c3ecc2c3ec8bc3ec7dc3efd1c3f2d9c3f434c3f49ac3f4c6c3f5 + b0c3f6f2c3f847c3f716c3f7dac3f776c3f66ec3f67fc3f8a3c3f980c3f8cac3f79dc3f85dc3f839c3f747c3f8b7c3f8 + fac3f829c3f964c3f923c3f7cec3fa53c3f88ac3f78dc3f8c2c3f8f0c3f73fc3f763c3f867c3f7d5c3f816c3f99bc3f7 + dfc3f7f2c3f8fbc3f830c3f777c3f800c3f7b3c3f864c3f6fdc3f79ec3f882c3f77ac3f748c3f88ec3f907c3f980c3f7 + f4c3f924c3f8c8c3f856c3f802c3f823c3f903c3f90fc3f950c3fab5c3fa82c3f98ec3f91ac3f8bdc3fb13c3fb7ec3fb + 3fc3fbd5c3fa08c3faf3c3fc74c3fa42c3fa90c3fbd5c3fc54c3fc01c3fcf6c3fb4dc3fc34c3fd5ac3fcccc3fdfac3fd + 40c3fbacc3fb30c3fc65c3fbf4c3fb99c3fb7dc3fbc4c3fc6fc3fcf4c3fbadc3fbcec3fb46c3fb58c3fc58c3fb0bc3f9 + 45c3fa54c3fa75c3fa37c3fab2c3fa6cc3faf0c3fab2c3fad5c3fb46c3fb1ac3fc9dc3fbd2c3faa7c3fa98c3fb85c3fd + b0c3fc78c3fb95c3fab7c3fc94c3fe67c3fe15c3fcffc3fa3bc3fd33c3ff86c00062c001dac0013cc3ff22c3fefcc3ff + 78c3fef1c00077c00148c000a1c00088c0011dc00263c0017ec000d6c002d9c0023cc001c3c00287c00322c00340c003 + 18c00460c002eec003e1c0059bc00561c00479c0056cc00640c005e2c00670c0063bc006444005d440043a4002eb4002 + a10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 + 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a45 + 004400450045004500450045004500460047000000f9fff9fffafffafffafffafffafffafffbfffcff0000009b973d34 + """ + /// The payload offset at which `EcgStatusHeader`'s `numberOfECGSamples` lands on the record's real /// sample count. NOT `Whoop5Ecg.puffinPayloadStart` (11): a type-47 historical record carries a /// layout marker at [10], a u32 record index at [11] and a u32 unix second at [15] before its body, @@ -137,6 +186,7 @@ final class Whoop5EcgRawHardwareTests: XCTestCase { private var frame11: [UInt8] { hexToBytes(Self.recordLeadsOffCount11) } private var frame10: [UInt8] { hexToBytes(Self.recordLeadsOffCount10) } + private var framePartial: [UInt8] { hexToBytes(Self.recordPartlyFilled245) } /// The sample encoding under test: a 24-bit BIG-endian field whose top 6 bits are a tag and whose /// low 18 bits are a signed value. @@ -169,7 +219,7 @@ final class Whoop5EcgRawHardwareTests: XCTestCase { // MARK: - The frames are real and well formed func testCapturedFramesPassBothWhoop5CRCs() { - for frame in [frame11, frame10] { + for frame in [frame11, frame10, framePartial] { XCTAssertEqual(frame.count, 1584) XCTAssertEqual(frame[8], 47, "type-47 historical record") XCTAssertEqual(frame[9], 16, "layout version 16") @@ -219,6 +269,7 @@ final class Whoop5EcgRawHardwareTests: XCTestCase { XCTAssertEqual(packet.header.numberOfECGSamples, 500) XCTAssertEqual(packet.rawECGDataRaw.count, 1500) XCTAssertEqual(packet.bytesPerSample, 3) + XCTAssertEqual(packet.unusedSampleBytes, 0, "a full record fills its region exactly") XCTAssertEqual(packet.numberOfLeadsOffSamples, 11) XCTAssertEqual(packet.leadsOffIRaw, [UInt16](repeating: 67, count: 11)) XCTAssertEqual(packet.leadsOffQRaw, @@ -395,6 +446,102 @@ final class Whoop5EcgRawHardwareTests: XCTestCase { XCTAssertEqual(packet.padding, [0x00]) } + // MARK: - The fixed-size sample region + + /// The geometry, read straight off the frame with no decoder involved. This is the evidence the + /// end-anchored `decodeRaw` rests on: a record carrying 245 samples is byte-for-byte the same size as + /// a record carrying 500, its sample data stops at frame[769] = 34 + 245 * 3, everything from there + /// to the leads-off block is zero, and the block is at the SAME frame offsets as both full records'. + func testPartlyFilledRecordHasAFullSizeSampleRegion() { + XCTAssertEqual(framePartial.count, frame11.count) + XCTAssertEqual(framePartial.count, 1584) + // 245 samples, not 500 — and the record's own second sits between the last empty record and the + // first full one, so this is the reading starting part-way through a second. + XCTAssertEqual(UInt16(framePartial[32]) | UInt16(framePartial[33]) << 8, 245) + XCTAssertEqual(u32le(framePartial, 15), 1786031625) + XCTAssertEqual(u32le(frame11, 15), 1786031627) + + XCTAssertNotEqual(framePartial[768], 0, "the last byte of sample 244") + XCTAssertTrue(framePartial[769..<1534].allSatisfy { $0 == 0 }, + "the region's unused capacity is zero-filled, not absent") + // Same fixed leads-off offsets as the two full records. + XCTAssertEqual(framePartial[1534], 10) + func slots(_ frame: [UInt8], _ start: Int) -> [UInt16] { + (0..<11).map { UInt16(frame[start + $0 * 2]) | UInt16(frame[start + $0 * 2 + 1]) << 8 } + } + XCTAssertEqual(slots(framePartial, 1535), [69, 68, 69, 69, 69, 69, 69, 69, 70, 71, 0]) + XCTAssertEqual(slots(framePartial, 1557), + [65529, 65529, 65530, 65530, 65530, 65530, 65530, 65530, 65531, 65532, 0]) + XCTAssertEqual(framePartial[1579], 0) + } + + /// Regression: the last populated record the decoder discarded. Reading the region as exactly + /// `n * bytesPerSample` put the count byte on payload[752] — a zero inside the fill — which yielded an + /// empty leads-off block and 766 trailing bytes against `defaultMaxPadding = 3`. + func testPartlyFilledRecordDecodesAtTheStreamWidth() { + guard let payload = Whoop5Ecg.innerPayload(framePartial, payloadStart: Self.ecgPayloadStart), + let packet = Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 3) else { + return XCTFail("decodeRaw failed on the partly-filled record") + } + XCTAssertEqual(payload.count, 1563, "same body length as a full record") + XCTAssertEqual(payload[752], 0, "where the region-is-exactly-full reading looked for the count") + + XCTAssertEqual(packet.header.numberOfECGSamples, 245) + XCTAssertEqual(packet.rawECGDataRaw.count, 735, "245 * 3 — only the VALID slots are carried") + XCTAssertEqual(packet.bytesPerSample, 3) + XCTAssertEqual(packet.unusedSampleBytes, 765) + XCTAssertEqual(packet.sampleRegionBytes, 1500, "the same region every full record carries") + XCTAssertEqual(packet.numberOfLeadsOffSamples, 10) + XCTAssertEqual(packet.leadsOffIRaw, [69, 68, 69, 69, 69, 69, 69, 69, 70, 71]) + XCTAssertEqual(packet.leadsOffQRaw, + [65529, 65529, 65530, 65530, 65530, 65530, 65530, 65530, 65531, 65532]) + XCTAssertEqual(packet.padding, [0x00], "accepted on its own remainder, not a widened tolerance") + + // The blob's ends, verbatim. No encoding claim is made for this record: it is the first half + // second of a reading, and its samples are still settling — see the tag census below. + XCTAssertEqual(Array(packet.rawECGDataRaw.prefix(3)), [0x83, 0xAE, 0x1E]) + XCTAssertEqual(Array(packet.rawECGDataRaw.suffix(3)), [0x40, 0x02, 0xA1]) + + // Every byte of the 1563-byte body is claimed by exactly one field, with BOTH fixed-size + // containers charged at their full size and the counts charged at what they declare. + let accounted = Whoop5Ecg.headerLength + + packet.rawECGDataRaw.count + + packet.unusedSampleBytes + + Whoop5Ecg.leadsOffBlockLength + + packet.padding.count + XCTAssertEqual(accounted, payload.count, "17 + 735 + 765 + 45 + 1") + } + + /// A partly-filled record does NOT determine its own sample width, and the enumerator says so rather + /// than picking one: 245 samples inside a 1500-byte region fits 3 bytes per sample and 4 alike. The + /// full records are where the width comes from — they resolve unaided. + func testPartlyFilledRecordReportsItsWidthAsAmbiguous() { + guard let payload = Whoop5Ecg.innerPayload(framePartial, payloadStart: Self.ecgPayloadStart) else { + return XCTFail("payload extraction failed") + } + XCTAssertEqual(Whoop5Ecg.rawBytesPerSampleCandidates(payload: payload), [3, 4]) + XCTAssertNil(Whoop5Ecg.decodeRaw(payload: payload), + "an ambiguous buffer must refuse to decode, not guess") + // 1 and 2 are ruled out by real sample bytes sitting in what they would call unused capacity. + XCTAssertNil(Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 1)) + XCTAssertNil(Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 2)) + } + + /// The tags this record's samples carry, as a census rather than an interpretation. The two full + /// records are uniformly 0b100000; this one is not, which is the reason no encoding claim is made + /// for its values here — the electrode input is still settling half a second into the reading. + func testPartlyFilledRecordSampleTagsAreNotUniform() { + guard let payload = Whoop5Ecg.innerPayload(framePartial, payloadStart: Self.ecgPayloadStart), + let packet = Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 3) else { + return XCTFail("decodeRaw failed") + } + var census = [UInt8: Int]() + for i in 0..> 2, default: 0] += 1 + } + XCTAssertEqual(census, [48: 176, 32: 65, 16: 4]) + } + /// The count=10 record accounts for every byte too — with the leads-off block charged at its FIXED /// size rather than at the count. func testCountTenRecordAccountsForEveryByte() { @@ -405,12 +552,14 @@ final class Whoop5EcgRawHardwareTests: XCTestCase { XCTAssertEqual(payload.count, 1563) XCTAssertEqual(packet.header.numberOfECGSamples, 500) XCTAssertEqual(packet.rawECGDataRaw.count, 1500) + XCTAssertEqual(packet.unusedSampleBytes, 0) let accounted = Whoop5Ecg.headerLength + packet.rawECGDataRaw.count + + packet.unusedSampleBytes + 1 + Whoop5Ecg.leadsOffSlotCount * 2 + Whoop5Ecg.leadsOffSlotCount * 2 + packet.padding.count - XCTAssertEqual(accounted, payload.count, "17 + 1500 + 1 + 22 + 22 + 1") + XCTAssertEqual(accounted, payload.count, "17 + 1500 + 0 + 1 + 22 + 22 + 1") } } diff --git a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift index 4735bd4e8f..dd6b3cc120 100644 --- a/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift +++ b/Packages/WhoopProtocol/Tests/WhoopProtocolTests/Whoop5EcgTests.swift @@ -226,6 +226,41 @@ final class Whoop5EcgTests: XCTestCase { XCTAssertEqual(packet?.header.numberOfECGSamples, 4) } + /// The SAMPLE region is fixed-size too: `numberOfECGSamples` says how many of its slots are valid, + /// and the rest is zero-filled capacity that still has to be stepped over to reach the leads-off + /// block. Only the valid bytes are carried; the unused ones are counted (see the real record in + /// `Whoop5EcgRawHardwareTests`, where the count is 245 inside a 500-slot region). + func testRawPartlyFilledSampleRegionIsSteppedOverNotCarried() { + let samples: [UInt8] = [1, 2, 3, 4, 5, 6] // 3 valid samples × 2 bytes + let unused = [UInt8](repeating: 0, count: 8) // 4 more slots the record did not fill + let payload = header(samples: 3) + samples + unused + leadsOffBlock(i: [5], q: [6]) + + let packet = Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 2) + XCTAssertEqual(packet?.rawECGDataRaw, samples, "only the valid slots are carried") + XCTAssertEqual(packet?.unusedSampleBytes, 8) + XCTAssertEqual(packet?.sampleRegionBytes, 14) + XCTAssertEqual(packet?.bytesPerSample, 2, "still count ÷ numberOfECGSamples") + XCTAssertEqual(packet?.numberOfLeadsOffSamples, 1) + XCTAssertEqual(packet?.leadsOffIRaw, [5]) + XCTAssertEqual(packet?.leadsOffQRaw, [6]) + XCTAssertEqual(packet?.padding, [], "the unused capacity is region, not trailing padding") + } + + /// The zero-fill is what makes the step-over safe. A non-zero byte between the declared samples and + /// the leads-off block means the block is not where this width says it is, so the decode fails closed + /// rather than skipping over bytes that might be data. + func testRawRejectsNonZeroBytesInTheUnusedSampleRegion() { + // The stray byte leads the unused span, so it stays inside that span at every end position the + // padding budget allows — a byte at the span's tail would simply become the count byte of a + // one-shorter record, which is a different (and legitimate) reading, not a violation. + let region: [UInt8] = [1, 2, 3, 4, 5, 6] + [0x09, 0, 0, 0, 0, 0, 0, 0] + let payload = header(samples: 3) + region + leadsOffBlock(i: [5], q: [6]) + XCTAssertNil(Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 2)) + // With the same bytes declared as samples the record is fine — it is the SKIPPING that is gated. + let full = header(samples: 7) + region + leadsOffBlock(i: [5], q: [6]) + XCTAssertEqual(Whoop5Ecg.decodeRaw(payload: full, bytesPerSample: 2)?.rawECGDataRaw, region) + } + /// A zero count still carries the full fixed block — the slots are simply all unused. func testRawWithNoLeadsOffSamples() { let payload = header(samples: 2) + [0xAA, 0xBB, 0xCC, 0xDD] + leadsOffBlock(i: [], q: []) @@ -258,9 +293,11 @@ final class Whoop5EcgTests: XCTestCase { /// The block holds a fixed number of slots, so a count above it cannot describe this layout and must /// fail closed rather than read past the block. func testRawRejectsLeadsOffCountAboveTheBlockCapacity() { + // The blob is exactly full, so the block can only be where the count byte is — this pins the + // capacity check itself rather than letting the end-of-record search reject the buffer earlier. let overCount = UInt8(Whoop5Ecg.leadsOffSlotCount + 1) let payload = header(samples: 2) + [0, 0, 0, 0] + [overCount] - + [UInt8](repeating: 0, count: Whoop5Ecg.leadsOffSlotCount * 4 + 8) + + [UInt8](repeating: 0, count: Whoop5Ecg.leadsOffSlotCount * 4) XCTAssertNil(Whoop5Ecg.decodeRaw(payload: payload, bytesPerSample: 2)) } diff --git a/android/app/src/main/java/com/noop/protocol/Whoop5Ecg.kt b/android/app/src/main/java/com/noop/protocol/Whoop5Ecg.kt index 55656c2a29..5fb9390153 100644 --- a/android/app/src/main/java/com/noop/protocol/Whoop5Ecg.kt +++ b/android/app/src/main/java/com/noop/protocol/Whoop5Ecg.kt @@ -117,13 +117,26 @@ data class FilteredLabradorPacket( * The raw blob is opaque: its bytes-per-sample is `rawECGDataRaw.size / numberOfECGSamples`, which means * the blob's LENGTH is not itself on the wire (see [Whoop5Ecg.rawBytesPerSampleCandidates]). * - * The leads-off block that follows is FIXED-SIZE — [Whoop5Ecg.LEADS_OFF_SLOT_COUNT] slots for I and the - * same again for Q, with [numberOfLeadsOffSamples] selecting how many leading slots are valid. - * [leadsOffIRaw] and [leadsOffQRaw] carry only the VALID values; unused slots are dropped. + * BOTH variable-length-looking regions are really FIXED-SIZE containers with a count that says how many + * leading slots are valid: the sample region (see [unusedSampleBytes] and [Whoop5Ecg.decodeRaw]) and the + * leads-off block ([Whoop5Ecg.LEADS_OFF_SLOT_COUNT] slots for I and the same again for Q, with + * [numberOfLeadsOffSamples] selecting how many are valid). In both, the unused slots are dropped rather + * than carried, so a packet's arrays only ever hold values the record says are real. */ data class RawLabradorPacket( val header: EcgStatusHeader, + /** + * Exactly `numberOfECGSamples * bytesPerSample` bytes — the VALID part of the sample region, which on + * a partly-filled record is shorter than the region itself. + */ val rawECGDataRaw: List, + /** + * Bytes of the fixed sample region that `numberOfECGSamples` did not fill. Zero for a full record. + * + * The strap zero-fills them, and the decoder requires that. They are counted, not carried, so + * `HEADER_LENGTH + rawECGDataRaw.size + unusedSampleBytes` is where the leads-off block begins. + */ + val unusedSampleBytes: Int, val numberOfLeadsOffSamples: Int, val leadsOffIRaw: List, val leadsOffQRaw: List, @@ -132,6 +145,10 @@ data class RawLabradorPacket( /** Bytes per raw sample, or null when the packet carried no samples to divide by. */ val bytesPerSample: Int? get() = if (header.numberOfECGSamples > 0) rawECGDataRaw.size / header.numberOfECGSamples else null + + /** The full sample region the record reserved, valid part plus unused capacity. */ + val sampleRegionBytes: Int + get() = rawECGDataRaw.size + unusedSampleBytes } object Whoop5Ecg { @@ -338,58 +355,101 @@ object Whoop5Ecg { // Decode — raw + /** + * Bytes the raw record's leads-off block occupies: the count byte, then a full I array, then a full Q + * array. Fixed — see [LEADS_OFF_SLOT_COUNT]. + */ + const val LEADS_OFF_BLOCK_LENGTH = 1 + LEADS_OFF_SLOT_COUNT * 4 + /** * Decode a raw packet from the inner record's PAYLOAD with an explicit sample width. * * The width has to be supplied because the raw blob's length is NOT on the wire. + * + * **The sample region is a fixed-size container, not `n * bytesPerSample` bytes.** MEASURED FROM + * HARDWARE, on the same capture and by the same argument as [LEADS_OFF_SLOT_COUNT]: the capture's one + * partly-filled record (`numberOfECGSamples == 245`, embedded in the Swift `Whoop5EcgRawHardwareTests` + * and its Kotlin twin) is the same 1584-byte frame as every full 500-sample record, its sample data + * stops after `245 * 3` bytes, the rest of the region is zeroed, and its leads-off block sits at the + * SAME frame offset as every other record's. So `numberOfECGSamples` says how many of the region's + * slots are valid in exactly the way `numberOfLeadsOffSamples` says how many leads-off slots are. + * + * That is why the block is located from the END of the payload rather than from + * `HEADER_LENGTH + n * bytesPerSample`: the region's LENGTH is no more on the wire than the width is, + * and a decoder that assumes the region is exactly full lands the count byte inside the zero fill. + * No region length is hardcoded; the block's own fixed size anchors it. + * + * Candidate ends are tried closest-to-the-end first, so the record claims the least padding it can — + * the puffin pad4 filler is minimal by construction. That ordering is a tie-break, not a proof: an + * all-zero tail can also validate a block placed a byte or two earlier, and the tightest placement is + * the one that explains the most bytes. Two things then have to hold: the region must be long enough + * for the declared samples, and every byte between them and the block must be ZERO. The zero-fill + * check is what keeps the width enumeration honest — a width that under-reads the samples leaves real + * sample bytes in the unused span and is rejected there. */ - fun decodeRaw(payload: List, bytesPerSample: Int): RawLabradorPacket? { - if (bytesPerSample <= 0) return null + fun decodeRaw( + payload: List, + bytesPerSample: Int, + maxPadding: Int = DEFAULT_MAX_PADDING, + ): RawLabradorPacket? { + if (bytesPerSample <= 0 || maxPadding < 0) return null val header = decodeHeader(payload) ?: return null // `bytesPerSample` is caller-supplied and `numberOfECGSamples` comes off the wire, so the product // is checked rather than assumed: a Kotlin Int overflow wraps silently to a NEGATIVE index, which // would throw on the subscript below. A decode failure is the correct outcome, not an exception. val blobLength = header.numberOfECGSamples.toLong() * bytesPerSample.toLong() if (blobLength > Int.MAX_VALUE - HEADER_LENGTH) return null - val rawEnd = HEADER_LENGTH + blobLength.toInt() - if (rawEnd < HEADER_LENGTH || rawEnd >= payload.size) return null // the leads-off count byte must fit - // Same domain check as decodeHeader: a negative count would make blockEnd negative, slip past the - // `blockEnd > size` bound, and throw on subList. On the wire this byte is always 0..255, and the - // block holds only LEADS_OFF_SLOT_COUNT slots — a count above that is not a record this layout can - // describe, so it fails closed rather than reading past the block. - val leadsOffCount = payload[rawEnd] - if (leadsOffCount !in 0..LEADS_OFF_SLOT_COUNT) return null - // Both arrays are FIXED-SIZE and always fully present — see [LEADS_OFF_SLOT_COUNT]. The count byte - // selects how many leading slots are VALID; it does not size the block. - val iStart = rawEnd + 1 - val qStart = iStart + LEADS_OFF_SLOT_COUNT * 2 - val blockEnd = qStart + LEADS_OFF_SLOT_COUNT * 2 - if (blockEnd > payload.size) return null - - val leadsOffI = (0 until leadsOffCount).map { payload[iStart + it * 2] or (payload[iStart + it * 2 + 1] shl 8) } - val leadsOffQ = (0 until leadsOffCount).map { payload[qStart + it * 2] or (payload[qStart + it * 2 + 1] shl 8) } - return RawLabradorPacket( - header = header, - rawECGDataRaw = payload.subList(HEADER_LENGTH, rawEnd).toList(), - numberOfLeadsOffSamples = leadsOffCount, - leadsOffIRaw = leadsOffI, - leadsOffQRaw = leadsOffQ, - padding = payload.subList(blockEnd, payload.size).toList(), - ) + val sampleEnd = HEADER_LENGTH + blobLength.toInt() + if (sampleEnd < HEADER_LENGTH || sampleEnd > payload.size) return null + + for (pad in 0..maxPadding) { + val blockEnd = payload.size - pad + val countIndex = blockEnd - LEADS_OFF_BLOCK_LENGTH + // The block must sit entirely after the declared samples. This also bounds `countIndex` from + // below, since `sampleEnd >= HEADER_LENGTH >= 0`. + if (countIndex < sampleEnd) continue + // Unused sample capacity is zero-filled on the wire. A non-zero byte here means the block is + // not at this offset — including the case where it holds samples a wider width would read. + if ((sampleEnd until countIndex).any { payload[it] != 0 }) continue + // Same domain check the Swift twin makes: on the wire this byte is 0..255, and the block holds + // only LEADS_OFF_SLOT_COUNT slots — a count above that is not a record this layout can + // describe, so it fails closed rather than reading past the block. + val leadsOffCount = payload[countIndex] + if (leadsOffCount !in 0..LEADS_OFF_SLOT_COUNT) continue + // Both arrays are FIXED-SIZE and always fully present — see [LEADS_OFF_SLOT_COUNT]. The count + // byte selects how many leading slots are VALID; it does not size the block. + val iStart = countIndex + 1 + val qStart = iStart + LEADS_OFF_SLOT_COUNT * 2 + + val leadsOffI = (0 until leadsOffCount).map { payload[iStart + it * 2] or (payload[iStart + it * 2 + 1] shl 8) } + val leadsOffQ = (0 until leadsOffCount).map { payload[qStart + it * 2] or (payload[qStart + it * 2 + 1] shl 8) } + return RawLabradorPacket( + header = header, + rawECGDataRaw = payload.subList(HEADER_LENGTH, sampleEnd).toList(), + unusedSampleBytes = countIndex - sampleEnd, + numberOfLeadsOffSamples = leadsOffCount, + leadsOffIRaw = leadsOffI, + leadsOffQRaw = leadsOffQ, + padding = payload.subList(blockEnd, payload.size).toList(), + ) + } + return null } /** * Every sample width in [widths] that yields a structurally consistent record leaving at most * [maxPadding] trailing bytes. A DISAMBIGUATION helper, not a claim. + * + * A PARTLY-FILLED record is genuinely ambiguous, and that is a fact about the record rather than a gap + * here: 245 samples inside a 1500-byte region fits 3 bytes per sample and 4 alike, so the capture's + * short record reports `[3, 4]` and the single-candidate [decodeRaw] below declines it. The width is a + * property of the STREAM — the 350 full records in the same capture resolve to `[3]` on their own. */ fun rawBytesPerSampleCandidates( payload: List, widths: List = listOf(1, 2, 3, 4), maxPadding: Int = DEFAULT_MAX_PADDING, - ): List = widths.filter { width -> - val packet = decodeRaw(payload, width) - packet != null && packet.padding.size <= maxPadding - } + ): List = widths.filter { width -> decodeRaw(payload, width, maxPadding) != null } /** Decode a raw record only when the buffer admits exactly ONE width; ambiguity returns null. */ fun decodeRaw( @@ -398,12 +458,17 @@ object Whoop5Ecg { maxPadding: Int = DEFAULT_MAX_PADDING, ): RawLabradorPacket? { val candidates = rawBytesPerSampleCandidates(payload, widths, maxPadding) - return if (candidates.size == 1) decodeRaw(payload, candidates[0]) else null + return if (candidates.size == 1) decodeRaw(payload, candidates[0], maxPadding) else null } /** CRC-gated raw decode straight off a complete 5/MG frame, with an explicit sample width. */ - fun decodeRawFrame(frame: ByteArray, bytesPerSample: Int, payloadStart: Int = PUFFIN_PAYLOAD_START): RawLabradorPacket? = - innerPayload(frame, payloadStart)?.let { decodeRaw(it, bytesPerSample) } + fun decodeRawFrame( + frame: ByteArray, + bytesPerSample: Int, + payloadStart: Int = PUFFIN_PAYLOAD_START, + maxPadding: Int = DEFAULT_MAX_PADDING, + ): RawLabradorPacket? = + innerPayload(frame, payloadStart)?.let { decodeRaw(it, bytesPerSample, maxPadding) } // Discovery diff --git a/android/app/src/test/java/com/noop/protocol/Whoop5EcgRawHardwareTest.kt b/android/app/src/test/java/com/noop/protocol/Whoop5EcgRawHardwareTest.kt index 65ac635ab6..3b3069621a 100644 --- a/android/app/src/test/java/com/noop/protocol/Whoop5EcgRawHardwareTest.kt +++ b/android/app/src/test/java/com/noop/protocol/Whoop5EcgRawHardwareTest.kt @@ -2,19 +2,22 @@ package com.noop.protocol import org.junit.Assert.assertEquals import org.junit.Assert.assertNotNull +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue import org.junit.Test /** * The Kotlin twin of WhoopProtocolTests/Whoop5EcgRawHardwareTests.swift, against the SAME real bytes. * - * The two complete 1584-byte type-47 layout-16 WHOOP MG flash records embedded here came off hardware on - * 2026-08-06 while the clasp electrodes were held; they are byte-identical to the Swift fixtures. The + * The three complete 1584-byte type-47 layout-16 WHOOP MG flash records embedded here came off hardware + * on 2026-08-06 while the clasp electrodes were held; they are byte-identical to the Swift fixtures. The * cross-platform parity contract says the two decoders must agree on stored values, and a synthetic * fixture can only prove they agree about an assumption — these bytes prove they agree about the strap. * - * Only the load-bearing assertions are mirrored: the fixed-size leads-off block, the resolved sample - * width, and that a populated record is not discarded. The full characterisation (including the header - * misalignment that is still open, and the 18-bit continuity argument) lives on the Swift side. + * Only the load-bearing assertions are mirrored: the two fixed-size containers (the leads-off block and + * the sample region), the resolved sample width, and that a populated record is not discarded. The full + * characterisation (including the header misalignment that is still open, and the 18-bit continuity + * argument) lives on the Swift side. * * No scale or unit is claimed for the sample values, and none is applied. */ @@ -95,11 +98,54 @@ class Whoop5EcgRawHardwareTest { "0b800589800dde8019cf8020168020a2800e2b8003ac800ba6801956801838800ebe800c68800d218009ee800e4b0a43" + "004300430043004300420042004200420042000000f7fff7fff7fff7fff7fff7fff6fff6fff6fff6ff000000fe104255" + /** + * unix 1786031625 — the capture's one PARTLY FILLED record, numberOfECGSamples == 245. + * + * The first record of the second reading: the empty record before it is at 1786031624 and the first + * full 500-sample one at 1786031626, so the strap began the reading part-way into this record's + * second and wrote 245 of the region's 500 slots. Same 1584-byte frame, same leads-off offsets. + */ + private val hexPartial245 = + "aa0128060100cde02f10036861cd0009ae746aeb51010a000100000000ffff00f50083ae1e8259e48210008210008210" + + "008210008210008210008210008210008210008210008210008210008210008210008210008210008210008210008210" + + "008210008210008210008210008210008210008210008210008210008210008210008210008210008210008210008210" + + "008210008210008210008210008210008210008210008210008210008210008210008210008210008210008210008210" + + "00821000821000821000821000821000821000821000821000821000821000821000821000c21000c21000c21000c210" + + "00c21085c25696c328a5c39fc1c3cda0c3e169c3e9d9c3ecc2c3ec8bc3ec7dc3efd1c3f2d9c3f434c3f49ac3f4c6c3f5" + + "b0c3f6f2c3f847c3f716c3f7dac3f776c3f66ec3f67fc3f8a3c3f980c3f8cac3f79dc3f85dc3f839c3f747c3f8b7c3f8" + + "fac3f829c3f964c3f923c3f7cec3fa53c3f88ac3f78dc3f8c2c3f8f0c3f73fc3f763c3f867c3f7d5c3f816c3f99bc3f7" + + "dfc3f7f2c3f8fbc3f830c3f777c3f800c3f7b3c3f864c3f6fdc3f79ec3f882c3f77ac3f748c3f88ec3f907c3f980c3f7" + + "f4c3f924c3f8c8c3f856c3f802c3f823c3f903c3f90fc3f950c3fab5c3fa82c3f98ec3f91ac3f8bdc3fb13c3fb7ec3fb" + + "3fc3fbd5c3fa08c3faf3c3fc74c3fa42c3fa90c3fbd5c3fc54c3fc01c3fcf6c3fb4dc3fc34c3fd5ac3fcccc3fdfac3fd" + + "40c3fbacc3fb30c3fc65c3fbf4c3fb99c3fb7dc3fbc4c3fc6fc3fcf4c3fbadc3fbcec3fb46c3fb58c3fc58c3fb0bc3f9" + + "45c3fa54c3fa75c3fa37c3fab2c3fa6cc3faf0c3fab2c3fad5c3fb46c3fb1ac3fc9dc3fbd2c3faa7c3fa98c3fb85c3fd" + + "b0c3fc78c3fb95c3fab7c3fc94c3fe67c3fe15c3fcffc3fa3bc3fd33c3ff86c00062c001dac0013cc3ff22c3fefcc3ff" + + "78c3fef1c00077c00148c000a1c00088c0011dc00263c0017ec000d6c002d9c0023cc001c3c00287c00322c00340c003" + + "18c00460c002eec003e1c0059bc00561c00479c0056cc00640c005e2c00670c0063bc006444005d440043a4002eb4002" + + "a10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + + "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a45" + + "004400450045004500450045004500460047000000f9fff9fffafffafffafffafffafffafffbfffcff0000009b973d34" + private fun bytes(h: String): ByteArray = ByteArray(h.length / 2) { ((h[it * 2].digitToInt(16) shl 4) or h[it * 2 + 1].digitToInt(16)).toByte() } private val frame11 get() = bytes(hex11) private val frame10 get() = bytes(hex10) + private val framePartial get() = bytes(hexPartial245) /** 24-bit big-endian field, low 18 bits signed. Measured from these bytes; not a vendor fact. */ private fun sample18(blob: List, i: Int): Int { @@ -167,12 +213,69 @@ class Whoop5EcgRawHardwareTest { listOf(-2948, -2535, -3301, -3030, -2720, -1860, -3162, -3830, -3752, -3371, -2282, -2985), (0 until 12).map { sample18(packet.rawECGDataRaw, it) }, ) - // 17 + 1500 + 1 + 22 + 22 + 1 = 1563, with the block charged at its FIXED size. - val accounted = Whoop5Ecg.HEADER_LENGTH + packet.rawECGDataRaw.size + 1 + - Whoop5Ecg.LEADS_OFF_SLOT_COUNT * 2 + Whoop5Ecg.LEADS_OFF_SLOT_COUNT * 2 + packet.padding.size + // 17 + 1500 + 0 + 1 + 22 + 22 + 1 = 1563, with the block charged at its FIXED size. + assertEquals(0, packet.unusedSampleBytes) + val accounted = Whoop5Ecg.HEADER_LENGTH + packet.rawECGDataRaw.size + packet.unusedSampleBytes + + 1 + Whoop5Ecg.LEADS_OFF_SLOT_COUNT * 2 + Whoop5Ecg.LEADS_OFF_SLOT_COUNT * 2 + + packet.padding.size assertEquals(payload.size, accounted) } + /** + * The SAMPLE region is fixed-size too. The partly-filled record is byte-for-byte the same size as a + * full one, its sample data stops at frame[769] = 34 + 245 * 3, everything from there to the + * leads-off block is zero, and the block is at the SAME frame offsets as both full records'. Reading + * the region as exactly `n * bytesPerSample` landed the count byte on a zero at payload[752], read an + * empty block and left 766 trailing bytes — the last populated record the decoder discarded. + */ + @Test + fun partlyFilledRecordDecodesAtTheStreamWidth() { + assertEquals(1584, framePartial.size) + assertTrue((769 until 1534).all { framePartial[it].toInt() == 0 }) + assertEquals(10, framePartial[1534].toInt() and 0xFF) + + val payload = Whoop5Ecg.innerPayload(framePartial, ecgPayloadStart)!! + assertEquals(1563, payload.size) + assertEquals(0, payload[752]) + + val packet = Whoop5Ecg.decodeRaw(payload, bytesPerSample = 3)!! + assertEquals(245, packet.header.numberOfECGSamples) + assertEquals(735, packet.rawECGDataRaw.size) // only the VALID slots are carried + assertEquals(3, packet.bytesPerSample) + assertEquals(765, packet.unusedSampleBytes) + assertEquals(1500, packet.sampleRegionBytes) // the region every full record carries + assertEquals(10, packet.numberOfLeadsOffSamples) + assertEquals(listOf(69, 68, 69, 69, 69, 69, 69, 69, 70, 71), packet.leadsOffIRaw) + assertEquals( + listOf(65529, 65529, 65530, 65530, 65530, 65530, 65530, 65530, 65531, 65532), + packet.leadsOffQRaw, + ) + assertEquals(listOf(0x00), packet.padding) + // The blob's ends, verbatim. No encoding claim is made for this record: it is the first half + // second of a reading and its samples are still settling. + assertEquals(listOf(0x83, 0xAE, 0x1E), packet.rawECGDataRaw.take(3)) + assertEquals(listOf(0x40, 0x02, 0xA1), packet.rawECGDataRaw.takeLast(3)) + + val accounted = Whoop5Ecg.HEADER_LENGTH + packet.rawECGDataRaw.size + packet.unusedSampleBytes + + Whoop5Ecg.LEADS_OFF_BLOCK_LENGTH + packet.padding.size + assertEquals(payload.size, accounted) // 17 + 735 + 765 + 45 + 1 + } + + /** + * A partly-filled record does NOT determine its own sample WIDTH, and the enumerator says so rather + * than picking one: 245 samples inside a 1500-byte region fits 3 bytes per sample and 4 alike. The + * full records are where the width comes from — they resolve unaided (see the test above). + */ + @Test + fun partlyFilledRecordReportsItsWidthAsAmbiguous() { + val payload = Whoop5Ecg.innerPayload(framePartial, ecgPayloadStart)!! + assertEquals(listOf(3, 4), Whoop5Ecg.rawBytesPerSampleCandidates(payload)) + assertNull(Whoop5Ecg.decodeRaw(payload)) + // 1 and 2 are ruled out by real sample bytes sitting in what they would call unused capacity. + assertNull(Whoop5Ecg.decodeRaw(payload, bytesPerSample = 1)) + assertNull(Whoop5Ecg.decodeRaw(payload, bytesPerSample = 2)) + } + /** * The block's geometry, read straight off the frame with no decoder involved: the count byte at * frame[1534], I slots at frame[1535], Q slots at frame[1557] — the SAME offsets under both counts, diff --git a/android/app/src/test/java/com/noop/protocol/Whoop5EcgTest.kt b/android/app/src/test/java/com/noop/protocol/Whoop5EcgTest.kt index 47171c61bc..2cbaa41a08 100644 --- a/android/app/src/test/java/com/noop/protocol/Whoop5EcgTest.kt +++ b/android/app/src/test/java/com/noop/protocol/Whoop5EcgTest.kt @@ -234,6 +234,45 @@ class Whoop5EcgTest { assertEquals(3, packet?.bytesPerSample) // count ÷ numberOfECGSamples } + /** + * The SAMPLE region is fixed-size too: numberOfECGSamples says how many of its slots are valid, and + * the rest is zero-filled capacity that still has to be stepped over to reach the leads-off block. + * Only the valid bytes are carried; the unused ones are counted. Twin of the Swift + * rawPartlyFilledSampleRegionIsSteppedOverNotCarried. + */ + @Test + fun rawPartlyFilledSampleRegionIsSteppedOverNotCarried() { + val samples = listOf(1, 2, 3, 4, 5, 6) // 3 valid samples × 2 bytes + val unused = List(8) { 0 } // 4 more slots the record did not fill + val payload = header(samples = 3) + samples + unused + leadsOffBlock(listOf(5), listOf(6)) + + val packet = Whoop5Ecg.decodeRaw(payload, bytesPerSample = 2) + assertEquals(samples, packet?.rawECGDataRaw) + assertEquals(8, packet?.unusedSampleBytes) + assertEquals(14, packet?.sampleRegionBytes) + assertEquals(2, packet?.bytesPerSample) + assertEquals(1, packet?.numberOfLeadsOffSamples) + assertEquals(listOf(5), packet?.leadsOffIRaw) + assertEquals(listOf(6), packet?.leadsOffQRaw) + assertEquals(emptyList(), packet?.padding) + } + + /** + * The zero-fill is what makes the step-over safe: a non-zero byte between the declared samples and + * the block means the block is not where this width says it is, so the decode fails closed rather + * than skipping over bytes that might be data. The stray byte LEADS the unused span so it stays + * inside that span at every end position the padding budget allows. + */ + @Test + fun rawRejectsNonZeroBytesInTheUnusedSampleRegion() { + val region = listOf(1, 2, 3, 4, 5, 6) + listOf(0x09) + List(7) { 0 } + val payload = header(samples = 3) + region + leadsOffBlock(listOf(5), listOf(6)) + assertNull(Whoop5Ecg.decodeRaw(payload, bytesPerSample = 2)) + // With the same bytes declared as samples the record is fine — it is the SKIPPING that is gated. + val full = header(samples = 7) + region + leadsOffBlock(listOf(5), listOf(6)) + assertEquals(region, Whoop5Ecg.decodeRaw(full, bytesPerSample = 2)?.rawECGDataRaw) + } + /** A zero count still carries the full fixed block — the slots are simply all unused. */ @Test fun rawWithNoLeadsOffSamples() { @@ -273,8 +312,10 @@ class Whoop5EcgTest { */ @Test fun rawRejectsLeadsOffCountAboveTheBlockCapacity() { + // The blob is exactly full, so the block can only be where the count byte is — this pins the + // capacity check itself rather than letting the end-of-record search reject the buffer earlier. val payload = header(samples = 2) + listOf(0, 0, 0, 0) + - listOf(Whoop5Ecg.LEADS_OFF_SLOT_COUNT + 1) + List(Whoop5Ecg.LEADS_OFF_SLOT_COUNT * 4 + 8) { 0 } + listOf(Whoop5Ecg.LEADS_OFF_SLOT_COUNT + 1) + List(Whoop5Ecg.LEADS_OFF_SLOT_COUNT * 4) { 0 } assertNull(Whoop5Ecg.decodeRaw(payload, bytesPerSample = 2)) } From ee029a8bc5a0f55cd540e076c993b4567b7c3129 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 24 Aug 2026 13:42:38 -0400 Subject: [PATCH 169/171] =?UTF-8?q?Sleep:=20HR-first=20detection=20(experi?= =?UTF-8?q?mental)=20=E2=80=94=20rescue=20restless=20nights,=20bridge=20br?= =?UTF-8?q?ief=20wakes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gravity-stillness spine reads a restless but genuinely asleep night as "active", so whole hours of physiologically unambiguous sleep (sustained sub-baseline HR) never become candidate runs: a real 5/MG user's 6-8 h nights collapsed to 1-3 h quiescent morning fragments (hourly HR 47-58 from ~01:00 to ~09:00 local scored nothing), and an ~8-min bathroom get-up — smeared past mergeMin by the centered 15-min rolling stillness window — split the remaining night into separate stored sessions at a 16-min seam. The dense path has no HR-led recovery at all (the #308 rescue is gated on gravity SPARSENESS, which a dense restless night never trips), and detection emits one stored session per still-run with no session-level re-merge (the #561/#861 bridges only group later, at selection/aggregation). New experimental toggle "HR-first sleep detection" (default OFF, PuffinExperiment.hrFirstSleep on both platforms), applied between the run spine and the gate ladder in SleepStager.detectSleep: 1. hrRescueActiveRuns — an "active" run whose MEDIAN in-run HR sits inside the SAME sleep band the ladder already trusts (confirmSleepWithHR's baseline x 1.05, same median statistic, same sleepHRBaseline override) is reclassified sleep: motion alone must not veto sleep the cardiac signal confirms. 2. bridgeWakeGaps — adjacent sleep runs separated by <= 45 min (wakeBridgeMaxMin, below the selector-side gapBridgeMaxMin of 60) are assembled into ONE candidate run, so a brief get-up yields one session with the wake staged inside it, and sub-minSleepMin fragments between arousals are no longer dropped outright. Every downstream gate (minSleepMin, H4 span cap, hrConfirm, off-wrist, daytime/morning guards) still applies to the assembled runs, so a rescued daytime stretch still needs a real cardiac dip. Default OFF keeps every existing caller and golden test byte-identical (locked by an explicit byte-identity test); the flag is folded into both platforms' detectSleep memo keys. Trace gates hrRescue/wakeBridge emit under the Sleep test mode. Cross-platform parity: identical constants, functions, and threading in Swift (SleepStager, AnalyticsEngine.analyzeDay, IntelligenceEngine, PuffinExperiment, SettingsView) and Kotlin (SleepStager, AnalyticsEngine, IntelligenceEngine, AppViewModel, WhoopBleClient, PuffinExperiment, SettingsScreen). Tests run without a strap and follow the varying-input rule (#194/#345): the rescue must TRACK injected sleep windows at different positions/lengths and refuse the identical motion pattern with elevated HR (SleepStagerHrFirstTests.swift / SleepStagerHrFirstTest.kt, same scenarios, same numbers). Verified: swift test (StrandAnalytics, 1342 passing incl. 9 new), macOS Strand build, iOS NOOPiOS build, Android compileFullDebugKotlin + testFullDebugUnitTest (all passing, 9 new). Co-Authored-By: Claude Fable 5 --- .../StrandAnalytics/AnalyticsEngine.swift | 10 + .../Sources/StrandAnalytics/SleepStager.swift | 118 ++++++++++ .../SleepStagerHrFirstTests.swift | 199 +++++++++++++++++ Strand/BLE/PuffinExperiment.swift | 17 ++ Strand/Data/IntelligenceEngine.swift | 6 + Strand/Screens/SettingsView.swift | 16 ++ .../com/noop/analytics/AnalyticsEngine.kt | 8 + .../com/noop/analytics/IntelligenceEngine.kt | 14 +- .../java/com/noop/analytics/SleepStager.kt | 117 +++++++++- .../java/com/noop/ble/PuffinExperiment.kt | 17 ++ .../main/java/com/noop/ble/WhoopBleClient.kt | 2 + .../src/main/java/com/noop/ui/AppViewModel.kt | 4 + .../main/java/com/noop/ui/SettingsScreen.kt | 46 ++++ .../noop/analytics/SleepStagerHrFirstTest.kt | 202 ++++++++++++++++++ 14 files changed, 772 insertions(+), 4 deletions(-) create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerHrFirstTests.swift create mode 100644 android/app/src/test/java/com/noop/analytics/SleepStagerHrFirstTest.kt diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift index 7cdd68f005..1c781e0e15 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift @@ -344,6 +344,15 @@ public enum AnalyticsEngine { // caller/test byte-identical; IntelligenceEngine threads // `PuffinExperiment.motionAwareWakeEnabled`. useMotionAwareWake: Bool = false, + // HR-first sleep detection (experimental, default OFF — see the + // section comment above `SleepStager.wakeBridgeMaxMin`). When true, + // detection weights HR over motion where the two disagree (an + // "active" run whose median HR sits in the sleep band is rescued as + // sleep) and assembles one session per sleep period across + // ≤ 45-min wake gaps. Default false keeps every pure-function + // caller/test byte-identical; IntelligenceEngine threads + // `PuffinExperiment.hrFirstSleepEnabled`. + useHrFirstSleep: Bool = false, // Sleep PROVENANCE for the per-day sleep trace (CAPTURE-C / #799). The // measured BLE path is `.measured` (the default); the caller passes // `.imported(...)` when a previously-imported sleep row WON the daily merge, @@ -385,6 +394,7 @@ public enum AnalyticsEngine { tzOffsetSeconds: tzOffsetSeconds, wristOff: wristOff, bandSleepState: bandSleepState, useSleepStagerV2: useSleepStagerV2, + hrFirstSleep: useHrFirstSleep, traceSink: traceSink) // Motion-aware wake refinement (#364 follow-up) runs AFTER V1/V2 staging, over every detected // session (naps included — the same eligibility gates apply). `steps` is the SAME calendar-day/ diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift index 4778b8609e..c218bedff3 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift @@ -190,6 +190,99 @@ public enum SleepStager { /// while still dropping an all-day desk strap (≈100% gap) or a session genuinely spent off-wrist. public static let maxOffWristSleepFraction: Double = 0.5 + // MARK: - HR-first sleep detection (experimental, default OFF) + + // The gravity-stillness spine reads a RESTLESS but genuinely asleep night — periodic limb + // movement, an injured/bedridden sleeper's tossing, or a strap whose motion channel over-reports + // — as "active", so whole hours of physiologically unambiguous sleep (sustained sub-baseline HR) + // never become candidate runs and the night collapses to whichever quiescent fragment survives. + // A real 5/MG user lost 6–8 h nights to 1–3 h morning fragments this way (hourly HR 47–58 bpm from + // ~01:00 through ~09:00 local scored nothing), while an ~8-min bathroom get-up — smeared past + // mergeMin by the CENTERED 15-min rolling stillness window — split the remaining night into + // separate stored sessions at a 16-min seam. The dense-gravity path has no HR-led recovery at all: + // HR only ever CONFIRMS gravity-derived runs (the #308 rescue is gated on gravity SPARSENESS, + // which a dense restless night never trips). + // + // The HR-first pass (default OFF; the live call site threads PuffinExperiment.hrFirstSleepEnabled) + // weights HR over motion when the two disagree, in two steps between the run spine and the ladder: + // 1. `hrRescueActiveRuns` — an "active" run whose MEDIAN in-run HR sits inside the SAME sleep + // band the ladder already trusts (`confirmSleepWithHR`'s ≤ baseline × hrSleepBaselineMult, + // same median statistic, same `sleepHRBaseline` override) is reclassified "sleep": motion + // alone must not veto sleep the cardiac signal confirms. + // 2. `bridgeWakeGaps` — adjacent sleep runs separated by ≤ `wakeBridgeMaxMin` are merged into + // ONE candidate run (the "same sleep period" convention — a brief mid-night wake belongs + // INSIDE the night), so a short get-up yields one session whose stager scores the wake + // inside it instead of two stored sessions — and so sub-minSleepMin sleep fragments between + // arousals are no longer dropped outright before the selector-side bridges ever see them. + // Every downstream gate (minSleepMin, the H4 span cap, hrConfirm, off-wrist, the daytime/morning + // guards) still applies to the ASSEMBLED runs, so a rescued daytime stretch still needs a real + // cardiac dip to survive, and an off-wrist block is still dropped. + // + // KNOWN LIMIT (why this defaults OFF, per the #194/#345 discipline): an awake-in-bed stretch whose + // HR genuinely sits inside the sleep band (deep pre-sleep relaxation) can be scored asleep — the + // same limit the still-sedentary gravity path already has, extended to fidgety-sedentary. The + // synthetic tests (SleepStagerHrFirstTests + the Kotlin twin) prove the pass TRACKS varying + // injected inputs — rescue follows wherever the sleep-band HR actually is, and refuses an + // identical motion pattern whose HR is elevated — not a single matched night. + + /// Adjacent sleep runs separated by at most this many minutes are assembled into ONE candidate + /// run (HR-first pass only). Below the selector-side `SleepStageTotals.gapBridgeMaxMin` (60) so + /// selection/aggregation behaviour is unchanged for anything this bridge leaves split; well above + /// the ~16-min smeared seam a brief real get-up produces. + public static let wakeBridgeMaxMin: Int = 45 + + /// HR-first step 1: reclassify an "active" run to "sleep" when its median in-run HR sits inside + /// the sleep band (≤ effective-baseline × `hrSleepBaselineMult` — the identical band + median + /// statistic `confirmSleepWithHR` trusts, with the same `sleepHRBaseline` personalised override). + /// Runs with fewer than `hrRefineMinSamples` in-run HR samples are left untouched (too little + /// cardiac evidence to overrule motion), as is everything when no baseline exists. Pure + + /// deterministic; mirrors Kotlin `hrRescueActiveRuns`. + static func hrRescueActiveRuns(_ periods: [Period], hr: [HRSample], baseline: Double?, + sleepHRBaseline: Double? = nil, + traceSink: ((String) -> Void)? = nil) -> [Period] { + guard let effBaseline = sleepHRBaseline ?? baseline else { return periods } + return periods.map { p in + guard p.stage == "active" else { return p } + let seg = rowsBetween(hr, start: p.start, end: p.end) { $0.ts } + guard seg.count >= hrRefineMinSamples else { return p } + let medHR = HRVAnalyzer.median(seg.map { Double($0.bpm) }) + guard medHR <= effBaseline * hrSleepBaselineMult else { return p } + traceSink?(GateTrace.runLine(index: -1, startTs: p.start, endTs: p.end, + verdict: .kept, gate: "hrRescue", + detail: "medHR=\(Int(medHR)) band=\(round2(effBaseline * hrSleepBaselineMult))")) + return Period(stage: "sleep", start: p.start, end: p.end) + } + } + + /// HR-first step 2: assemble adjacent sleep runs separated by at most `wakeBridgeMaxMin` minutes + /// (whatever lies between — an un-rescued active run, or a plain data hole) into one candidate + /// run; the intervening runs are absorbed and the stager later scores the wake INSIDE the single + /// session. A gap over the threshold (or a negative/overlapping gap — pinned `gap >= 0` + /// semantics, matching `bridgeSparseSleep`) never bridges. Pure + deterministic; mirrors Kotlin + /// `bridgeWakeGaps`. + static func bridgeWakeGaps(_ periods: [Period], + traceSink: ((String) -> Void)? = nil) -> [Period] { + let bridgeS = wakeBridgeMaxMin * 60 + var out: [Period] = [] + for p in periods { + if p.stage == "sleep", + let j = out.lastIndex(where: { $0.stage == "sleep" }), + p.start - out[j].end >= 0, p.start - out[j].end <= bridgeS { + let gapMin = (p.start - out[j].end) / 60 + if gapMin > 0 { + traceSink?(GateTrace.runLine(index: -1, startTs: out[j].end, endTs: p.start, + verdict: .kept, gate: "wakeBridge", + detail: "gapMin=\(gapMin) max=\(wakeBridgeMaxMin)")) + } + out[j] = Period(stage: "sleep", start: out[j].start, end: max(out[j].end, p.end)) + out.removeSubrange((j + 1)...) + } else { + out.append(p) + } + } + return out + } + /// Minimum average HR-stream density for the off-wrist HR-gap proxy to be trusted (#507). The proxy /// reads a >`offWristHRGapMin`-minute hole in HR as "off the wrist" — valid only when HR is otherwise /// dense (live 5/MG, or a worn night with continuous HR), so a real gap is anomalous. A WHOOP 4.0's @@ -849,6 +942,14 @@ public enum SleepStager { wristOff: [(start: Int, end: Int)] = [], bandSleepState: [(ts: Int, state: Int)] = [], useSleepStagerV2: Bool = false, + // HR-first sleep detection (experimental, default OFF — see the + // section comment above `wakeBridgeMaxMin`). When true, active runs + // whose median HR sits in the sleep band are rescued as sleep and + // adjacent sleep runs across ≤ wakeBridgeMaxMin wake gaps are + // assembled into one candidate run before the gate ladder. The + // default keeps every existing caller byte-identical; the live call + // site threads PuffinExperiment.hrFirstSleepEnabled. + hrFirstSleep: Bool = false, sleepHRBaseline: Double? = nil, traceSink: ((String) -> Void)? = nil) -> [SleepSession] { // Sleep & Rest test mode only: when a trace is requested we MUST run the live ladder, not a @@ -860,6 +961,7 @@ public enum SleepStager { return detectSleepUncached(hr: hr, rr: rr, resp: resp, gravity: gravity, tzOffsetSeconds: tzOffsetSeconds, wristOff: wristOff, bandSleepState: bandSleepState, useSleepStagerV2: useSleepStagerV2, + hrFirstSleep: hrFirstSleep, sleepHRBaseline: sleepHRBaseline, traceSink: traceSink) } // v7.0.2 perf (#707): the single heaviest analytics call — it sorts the dense full-day gravity @@ -879,11 +981,13 @@ public enum SleepStager { wristOff: StreamFingerprint.of(wristOff, ts: { $0.start }, quant: { $0.end }), band: StreamFingerprint.of(bandSleepState, ts: { $0.ts }, quant: { $0.state }), v2: useSleepStagerV2, + hrFirst: hrFirstSleep, sleepHRBaseline: sleepHRBaseline) return detectSleepCache.value(key) { detectSleepUncached(hr: hr, rr: rr, resp: resp, gravity: gravity, tzOffsetSeconds: tzOffsetSeconds, wristOff: wristOff, bandSleepState: bandSleepState, useSleepStagerV2: useSleepStagerV2, + hrFirstSleep: hrFirstSleep, sleepHRBaseline: sleepHRBaseline, traceSink: nil) } } @@ -894,6 +998,7 @@ public enum SleepStager { let tz: Int let wristOff: StreamFingerprint; let band: StreamFingerprint let v2: Bool + let hrFirst: Bool let sleepHRBaseline: Double? } /// ≈ the number of distinct days in a scoring window; FIFO-evicted, holds only small session arrays. @@ -908,6 +1013,7 @@ public enum SleepStager { wristOff: [(start: Int, end: Int)], bandSleepState: [(ts: Int, state: Int)], useSleepStagerV2: Bool, + hrFirstSleep: Bool = false, sleepHRBaseline: Double? = nil, traceSink: ((String) -> Void)? = nil) -> [SleepSession] { let grav = gravity.sorted { $0.ts < $1.ts } @@ -959,6 +1065,18 @@ public enum SleepStager { } } + // HR-first pass (experimental, default OFF — see the section comment above `wakeBridgeMaxMin`): + // weight HR over motion where the two disagree (an active run whose median HR is in the sleep + // band is rescued as sleep), then assemble one candidate run per sleep period across short + // (≤ wakeBridgeMaxMin) wake gaps, BEFORE the ladder — so sub-minSleepMin fragments between + // arousals survive as part of the night instead of being dropped, and a brief get-up stays + // INSIDE one session as staged wake. Every gate below still applies to the assembled runs. + if hrFirstSleep { + runs = hrRescueActiveRuns(runs, hr: hrS, baseline: baseline, + sleepHRBaseline: sleepHRBaseline, traceSink: traceSink) + runs = bridgeWakeGaps(runs, traceSink: traceSink) + } + let minSleepS = minSleepMin * 60 var sessions: [SleepSession] = [] diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerHrFirstTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerHrFirstTests.swift new file mode 100644 index 0000000000..be4b4d4aba --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerHrFirstTests.swift @@ -0,0 +1,199 @@ +import XCTest +@testable import StrandAnalytics +import WhoopProtocol + +/// HR-first sleep detection (experimental, default OFF): weight HR over motion when the two +/// disagree, and assemble one session per sleep period across short wake gaps. +/// +/// Per the "validate against varying inputs" rule (#194/#345), these tests inject the sleep window +/// at DIFFERENT positions and lengths and assert the rescue TRACKS it — and that the identical +/// motion pattern with an elevated (awake) HR is refused — rather than pinning one matched night. +/// The Kotlin twin (`SleepStagerHrFirstTest`) runs the same scenarios with the same numbers. +final class SleepStagerHrFirstTests: XCTestCase { + + // MARK: - Stream builders (1 Hz, mirroring SleepStagerTests) + + private func stillGravity(start: Int, durationS: Int) -> [GravitySample] { + (0.. [GravitySample] { + (0.. GravitySample in + let phase = Double(i % 2) * 0.5 + return GravitySample(ts: start + i, x: phase, y: 0, z: 1.0) + } + } + + private func hrStream(start: Int, durationS: Int, bpm: Int) -> [HRSample] { + (0.. Int { + 1_749_513_600 + hourUTC * 3_600 + min * 60 + } + + // MARK: - Restless-night rescue (the DROPPED-night failure) + + /// A night that is restless (motion says "active") but cardiac-unambiguous (HR 52 vs awake 75) + /// from 22:00 to 04:30, then still to 06:00. OFF: only the quiescent morning fragment scores — + /// the real-world symptom. ON: the whole night is one session. + func testRestlessNightRescuedToFullSession() { + let grav = restlessGravity(start: at(22), durationS: 6 * 3_600 + 1_800) // 22:00–04:30 + + stillGravity(start: at(28, min: 30), durationS: 90 * 60) // 04:30–06:00 + let hr = hrStream(start: at(20), durationS: 2 * 3_600, bpm: 75) // 20:00–22:00 awake + + hrStream(start: at(22), durationS: 8 * 3_600, bpm: 52) // 22:00–06:00 asleep + + hrStream(start: at(30), durationS: 2 * 3_600, bpm: 75) // 06:00–08:00 awake + + let off = SleepStager.detectSleep(hr: hr, gravity: grav) + XCTAssertEqual(off.count, 1, "OFF: only the still morning fragment should score") + XCTAssertGreaterThanOrEqual(off[0].start, at(28, min: 25), "OFF fragment starts near 04:30") + XCTAssertLessThanOrEqual(off[0].start, at(28, min: 45)) + + let on = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: true) + XCTAssertEqual(on.count, 1, "ON: one assembled session for the whole night") + XCTAssertLessThanOrEqual(on[0].start, at(22) + 120, "ON: onset tracks the injected 22:00 sleep start") + XCTAssertGreaterThanOrEqual(on[0].end, at(30) - 120, "ON: end tracks the injected 06:00 wake") + } + + /// Same rescue with the sleep window injected at a DIFFERENT position and length (01:00–07:30, + /// all restless, no still stretch at all): OFF finds nothing; ON tracks the injected bounds. + func testRestlessRescueTracksInjectedWindow() { + let grav = restlessGravity(start: at(25), durationS: 6 * 3_600 + 1_800) // 01:00–07:30 + let hr = hrStream(start: at(22), durationS: 3 * 3_600, bpm: 75) // 22:00–01:00 awake + + hrStream(start: at(25), durationS: 6 * 3_600 + 1_800, bpm: 52) // 01:00–07:30 asleep + + hrStream(start: at(31, min: 30), durationS: 2 * 3_600, bpm: 75) // 07:30–09:30 awake + + XCTAssertEqual(SleepStager.detectSleep(hr: hr, gravity: grav).count, 0, + "OFF: an all-restless night scores nothing") + + let on = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: true) + XCTAssertEqual(on.count, 1) + XCTAssertLessThanOrEqual(abs(on[0].start - at(25)), 120, "onset tracks the moved window") + XCTAssertLessThanOrEqual(abs(on[0].end - at(31, min: 30)), 120, "end tracks the moved window") + } + + /// The IDENTICAL motion pattern as the full-night rescue, but with the night HR elevated (78 vs + /// a 62 day median — awake-in-bed restlessness): the rescue must refuse the restless stretch. + /// (The STILL morning fragment is still accepted — by the pre-existing motion-corroborated + /// quiescent band (#462), which deliberately tolerates elevated-but-flat HR on a motionless + /// run — so the assertion is that HR-first adds NOTHING: same single fragment, not a full + /// night, with an onset still at the quiescent stretch.) + func testElevatedHrNightIsNotRescued() { + let grav = restlessGravity(start: at(22), durationS: 6 * 3_600 + 1_800) // 22:00–04:30 + + stillGravity(start: at(28, min: 30), durationS: 90 * 60) // 04:30–06:00 + let hr = hrStream(start: at(22), durationS: 8 * 3_600, bpm: 78) // 22:00–06:00 elevated + + hrStream(start: at(30), durationS: 10 * 3_600, bpm: 62) // 06:00–16:00 day median + + let off = SleepStager.detectSleep(hr: hr, gravity: grav) + let on = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: true) + XCTAssertEqual(on, off, "same motion, elevated HR: HR-first must change nothing") + XCTAssertEqual(on.count, 1) + XCTAssertGreaterThanOrEqual(on[0].start, at(28, min: 25), + "the restless 22:00–04:30 stretch must NOT be rescued") + } + + // MARK: - Wake-gap bridging (the SPLIT-night failure) + + /// A 16-min get-up (real movement + HR 80) mid-night. The centered rolling window smears it to a + /// >mergeMin active run, so OFF stores TWO sessions. ON bridges them into ONE session. + func testBriefGetUpBridgedIntoOneSession() { + let grav = stillGravity(start: at(23), durationS: 4 * 3_600) // 23:00–03:00 + + restlessGravity(start: at(27), durationS: 16 * 60) // 03:00–03:16 get-up + + stillGravity(start: at(27, min: 16), durationS: 3 * 3_600 + 44 * 60) // 03:16–07:00 + let hr = hrStream(start: at(21), durationS: 2 * 3_600, bpm: 75) // 21:00–23:00 awake + + hrStream(start: at(23), durationS: 4 * 3_600, bpm: 52) // 23:00–03:00 asleep + + hrStream(start: at(27), durationS: 16 * 60, bpm: 80) // get-up spike + + hrStream(start: at(27, min: 16), durationS: 3 * 3_600 + 44 * 60, bpm: 52) // back asleep + + hrStream(start: at(31), durationS: 2 * 3_600, bpm: 75) // 07:00–09:00 awake + + let off = SleepStager.detectSleep(hr: hr, gravity: grav) + XCTAssertEqual(off.count, 2, "OFF: the smeared get-up splits the night") + + let on = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: true) + XCTAssertEqual(on.count, 1, "ON: one session bridging the get-up") + XCTAssertLessThanOrEqual(abs(on[0].start - at(23)), 120) + XCTAssertLessThanOrEqual(abs(on[0].end - at(31)), 120) + } + + /// An 80-min genuinely-awake block (elevated HR, real movement) exceeds the 45-min bridge: + /// the night stays two sessions even with HR-first ON. + func testLongWakeGapStaysSplit() { + let grav = stillGravity(start: at(22), durationS: 3 * 3_600) // 22:00–01:00 + + restlessGravity(start: at(25), durationS: 80 * 60) // 01:00–02:20 awake + + stillGravity(start: at(26, min: 20), durationS: 3 * 3_600 + 40 * 60) // 02:20–06:00 + let hr = hrStream(start: at(20), durationS: 2 * 3_600, bpm: 75) + + hrStream(start: at(22), durationS: 3 * 3_600, bpm: 52) + + hrStream(start: at(25), durationS: 80 * 60, bpm: 78) + + hrStream(start: at(26, min: 20), durationS: 3 * 3_600 + 40 * 60, bpm: 52) + + hrStream(start: at(30), durationS: 2 * 3_600, bpm: 75) + + let on = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: true) + XCTAssertEqual(on.count, 2, "an over-threshold wake gap must not be bridged") + } + + // MARK: - Guards still hold on assembled runs + + /// A rescued + bridged DAYTIME stretch (in-band flat HR 60, no cardiac dip) must still be + /// dropped by the daytime false-sleep guard: the rescue may not create phantom daytime naps. + func testDaytimeGuardStillAppliesToRescuedRuns() { + let grav = stillGravity(start: at(11), durationS: 2 * 3_600) // 11:00–13:00 + + restlessGravity(start: at(13), durationS: 2 * 3_600) // 13:00–15:00 + + stillGravity(start: at(15), durationS: 2 * 3_600) // 15:00–17:00 + let hr = hrStream(start: at(10), durationS: 8 * 3_600, bpm: 60) // flat, no dip + + let on = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: true) + XCTAssertEqual(on.count, 0, "no resting-HR dip: the daytime guard must reject the rescue") + } + + /// Default-off byte-identity: omitting the flag and passing `false` produce the identical result. + func testDefaultOffIsByteIdentical() { + let grav = stillGravity(start: at(23), durationS: 4 * 3_600) + + restlessGravity(start: at(27), durationS: 16 * 60) + + stillGravity(start: at(27, min: 16), durationS: 3 * 3_600 + 44 * 60) + let hr = hrStream(start: at(23), durationS: 8 * 3_600, bpm: 52) + + hrStream(start: at(31), durationS: 4 * 3_600, bpm: 75) + + let implicitDefault = SleepStager.detectSleep(hr: hr, gravity: grav) + let explicitFalse = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: false) + XCTAssertEqual(implicitDefault, explicitFalse) + } + + // MARK: - Pure helpers + + func testBridgeWakeGapsMergesOnlyUnderThreshold() { + let a = SleepStager.Period(stage: "sleep", start: 0, end: 3_600) + let gap = SleepStager.Period(stage: "active", start: 3_600, end: 3_600 + 40 * 60) + let b = SleepStager.Period(stage: "sleep", start: 3_600 + 40 * 60, end: 3 * 3_600) + let merged = SleepStager.bridgeWakeGaps([a, gap, b]) + XCTAssertEqual(merged.count, 1) + XCTAssertEqual(merged[0].start, 0) + XCTAssertEqual(merged[0].end, 3 * 3_600) + + let farGap = SleepStager.Period(stage: "active", start: 3_600, end: 3_600 + 46 * 60) + let c = SleepStager.Period(stage: "sleep", start: 3_600 + 46 * 60, end: 4 * 3_600) + let unmerged = SleepStager.bridgeWakeGaps([a, farGap, c]) + XCTAssertEqual(unmerged.count, 3, "a 46-min gap is over wakeBridgeMaxMin (45) and must not merge") + } + + func testHrRescueRespectsBandAndSampleFloor() { + let p = SleepStager.Period(stage: "active", start: 0, end: 3_600) + let inBand = (0..<3_600).map { HRSample(ts: $0, bpm: 52) } + let rescued = SleepStager.hrRescueActiveRuns([p], hr: inBand, baseline: 52.0) + XCTAssertEqual(rescued[0].stage, "sleep") + + let outOfBand = (0..<3_600).map { HRSample(ts: $0, bpm: 60) } + let kept = SleepStager.hrRescueActiveRuns([p], hr: outOfBand, baseline: 52.0) + XCTAssertEqual(kept[0].stage, "active", "median above baseline × 1.05 must not rescue") + + // Below the hrRefineMinSamples floor: too little cardiac evidence to overrule motion. + let sparse = (0..<10).map { HRSample(ts: $0 * 60, bpm: 52) } + let untouched = SleepStager.hrRescueActiveRuns([p], hr: sparse, baseline: 52.0) + XCTAssertEqual(untouched[0].stage, "active") + + // No baseline at all: nothing to weigh HR against. + let noBase = SleepStager.hrRescueActiveRuns([p], hr: inBand, baseline: nil) + XCTAssertEqual(noBase[0].stage, "active") + } +} diff --git a/Strand/BLE/PuffinExperiment.swift b/Strand/BLE/PuffinExperiment.swift index 02592f50c0..532acf26f2 100644 --- a/Strand/BLE/PuffinExperiment.swift +++ b/Strand/BLE/PuffinExperiment.swift @@ -201,6 +201,23 @@ enum PuffinExperiment { static var motionAwareWakeEnabled: Bool { UserDefaults.standard.bool(forKey: motionAwareWakeKey) } + /// Opt-in "HR-first sleep detection" (default OFF): weight the cardiac signal over wrist motion when + /// the two disagree, inside `SleepStager.detectSleep`. Two effects, both BEFORE the gate ladder: + /// a motion-"active" run whose median HR sits inside the same sleep band the ladder already trusts is + /// rescued as sleep (a restless-but-asleep night — periodic limb movement, an injured sleeper, or a + /// strap whose motion channel over-reports — is no longer collapsed to its quiescent morning + /// fragment), and adjacent sleep runs across a ≤ 45-min wake gap are assembled into ONE session with + /// the wake staged inside it (a brief get-up no longer splits the night into separate stored + /// sessions). Every existing guard (min duration, span cap, HR confirmation, off-wrist, daytime nap + /// bars) still applies to the assembled runs. Default OFF per the varying-input validation + /// discipline (#194/#345): an awake-in-bed stretch whose HR genuinely sits in the sleep band can be + /// scored asleep, the same limit the still-sedentary path already has. Read at the scoring call site + /// (IntelligenceEngine) and threaded through `AnalyticsEngine.analyzeDay`. Mirrors the Android + /// `PuffinExperiment.KEY_HR_FIRST_SLEEP`. + static let hrFirstSleepKey = "noopHrFirstSleep" + + static var hrFirstSleepEnabled: Bool { UserDefaults.standard.bool(forKey: hrFirstSleepKey) } + /// Opt-in "WHOOP MG ECG (experimental)": unlock the gated, user-initiated ECG ("Labrador") probe that /// asks an MG strap to start its ECG subsystem. Default OFF, and OFF is not merely the default — with /// this key false the four ECG opcodes are dropped by the BLEManager allowlist, so no ECG byte can diff --git a/Strand/Data/IntelligenceEngine.swift b/Strand/Data/IntelligenceEngine.swift index 0054a0552a..eca3abb385 100644 --- a/Strand/Data/IntelligenceEngine.swift +++ b/Strand/Data/IntelligenceEngine.swift @@ -702,6 +702,10 @@ final class IntelligenceEngine: ObservableObject { // ever runs AFTER whichever stager above just ran, and self-gates on the night's observed // gravity + step density, so flipping it on is a no-op for any night too sparse to trust. let useMotionAwareWake = PuffinExperiment.motionAwareWakeEnabled + // HR-first sleep detection (experimental, default OFF): read once, off the detached + // executor, like the two flags above. Weights HR over motion in detection and bridges + // short wake gaps into one session — see `PuffinExperiment.hrFirstSleepEnabled`. + let useHrFirstSleep = PuffinExperiment.hrFirstSleepEnabled // Already OFF the main actor , score directly (the prior nested `Task.detached` here only // existed to hop off the main actor; the whole loop now runs off it, so the score is computed @@ -733,6 +737,8 @@ final class IntelligenceEngine: ObservableObject { // #364 follow-up: same threading for the motion-aware wake // refinement post-pass. useMotionAwareWake: useMotionAwareWake, + // HR-first sleep detection (experimental, default OFF). + useHrFirstSleep: useHrFirstSleep, traceSink: traceSink, hrvTraceSink: hrvTraceSink, // Per-window HRV detail ONLY for the most-recent night diff --git a/Strand/Screens/SettingsView.swift b/Strand/Screens/SettingsView.swift index 7e6125690b..9c6bcce341 100644 --- a/Strand/Screens/SettingsView.swift +++ b/Strand/Screens/SettingsView.swift @@ -90,6 +90,7 @@ struct SettingsView: View { /// minutes. Self-gates on OBSERVED gravity + step density (#345) — a no-op on a sparse night (e.g. /// WHOOP 4.0) regardless of this switch. See [PuffinExperiment.motionAwareWakeKey]. @AppStorage(PuffinExperiment.motionAwareWakeKey) private var motionAwareWakeEnabled = false + @AppStorage(PuffinExperiment.hrFirstSleepKey) private var hrFirstSleepEnabled = false // Imperial/Metric display preference (D#103). Stored data is always SI; this only changes how // distances/weights/heights/temperatures are SHOWN — and lets the profile fields below take @@ -1431,6 +1432,21 @@ struct SettingsView: View { .font(StrandFont.caption) .foregroundStyle(StrandPalette.textTertiary) .fixedSize(horizontal: false, vertical: true) + + Divider().overlay(StrandPalette.hairline) + + // MARK: HR-first sleep detection — default OFF. + Toggle(isOn: $hrFirstSleepEnabled) { + Text("HR-first sleep detection") + .font(StrandFont.subhead) + .foregroundStyle(StrandPalette.textPrimary) + } + .toggleStyle(.switch) + .tint(StrandPalette.accent) + Text("Trusts your heart rate over wrist movement when the two disagree about sleep. A restless night whose heart rate clearly sits at your sleep level is kept as sleep instead of being cut down to its calmest fragment, and a brief get-up (under ~45 minutes) stays inside one sleep with the wake scored inside it, rather than splitting the night into separate sleeps. Worth trying if your nights show up fragmented or mostly missing despite a normal sleeping heart rate. Trade-off: a long, deeply relaxed stretch awake in bed can be scored as sleep. Off by default; takes effect on the next nights scored.") + .font(StrandFont.caption) + .foregroundStyle(StrandPalette.textTertiary) + .fixedSize(horizontal: false, vertical: true) } } } diff --git a/android/app/src/main/java/com/noop/analytics/AnalyticsEngine.kt b/android/app/src/main/java/com/noop/analytics/AnalyticsEngine.kt index afbc5aca1c..64f07a41c2 100644 --- a/android/app/src/main/java/com/noop/analytics/AnalyticsEngine.kt +++ b/android/app/src/main/java/com/noop/analytics/AnalyticsEngine.kt @@ -241,6 +241,13 @@ object AnalyticsEngine { // flag. Default false keeps every pure-function caller/test byte-identical; IntelligenceEngine // threads PuffinExperiment.from(context).motionAwareWake. Mirrors Swift. useMotionAwareWake: Boolean = false, + // HR-first sleep detection (experimental, default OFF — see the section comment above + // [SleepStager.wakeBridgeMaxMin]). When true, detection weights HR over motion where the two + // disagree (an "active" run whose median HR sits in the sleep band is rescued as sleep) and + // assembles one session per sleep period across ≤ 45-min wake gaps. Default false keeps every + // pure-function caller/test byte-identical; IntelligenceEngine threads + // PuffinExperiment.from(context).hrFirstSleep. Mirrors Swift. + useHrFirstSleep: Boolean = false, // Sleep & Rest test-mode trace sink (E11). null = byte-identical default. When non-null the gate // trace from detectSleep and the Rest sub-score line are forwarded line-by-line. Mirrors Swift. traceSink: ((String) -> Unit)? = null, @@ -264,6 +271,7 @@ object AnalyticsEngine { hr = hr, rr = rr, resp = resp, gravity = gravity, tzOffsetSeconds = tzOffsetSeconds, wristOff = wristOff, bandSleepState = bandSleepState, useSleepStagerV2 = useSleepStagerV2, + hrFirstSleep = useHrFirstSleep, traceSink = traceSink, ) // Motion-aware wake refinement (#364 follow-up) runs AFTER V1/V2 staging, over every detected diff --git a/android/app/src/main/java/com/noop/analytics/IntelligenceEngine.kt b/android/app/src/main/java/com/noop/analytics/IntelligenceEngine.kt index f06e453420..1977ffee55 100644 --- a/android/app/src/main/java/com/noop/analytics/IntelligenceEngine.kt +++ b/android/app/src/main/java/com/noop/analytics/IntelligenceEngine.kt @@ -184,6 +184,10 @@ object IntelligenceEngine { // OWN observed gravity + step density (see [WakeMotionRefinement]), so existing callers / tests // and any night too sparse to trust (e.g. a WHOOP 4.0) are unaffected either way. useMotionAwareWake: Boolean = false, + // HR-first sleep detection (experimental, default OFF), threaded the same Context-free way: the + // Context-aware caller reads PuffinExperiment.from(context).hrFirstSleep and passes it down to + // AnalyticsEngine.analyzeDay. See [SleepStager.wakeBridgeMaxMin]'s section comment. Mirrors Swift. + useHrFirstSleep: Boolean = false, // Sleep & Rest test-mode trace sink (Test Centre E5). The analytics layer is Context-free, so the // Context-aware caller (AppViewModel / WhoopBleClient) reads TestCentre.active(SLEEP) and passes a // non-null sink ONLY when the mode is on, routing each line to the .sleep-tagged strap log. null (the @@ -231,7 +235,8 @@ object IntelligenceEngine { analyzeGate.withLock { val (out, healed) = analyzeRecentOnCpu(repo, profile, maxDays, importedDeviceId, maxHROverride, nowSeconds, ownerSource, manualStepCoefficient, persistStepsCalibration, baselineEpoch, - recoveryEpoch, diag, useExperimentalSleepV2, useMotionAwareWake, sleepTraceSink, recoveryTraceSink, + recoveryEpoch, diag, useExperimentalSleepV2, useMotionAwareWake, useHrFirstSleep, + sleepTraceSink, recoveryTraceSink, stepsTraceSink, universalSink, workoutsTraceSink, hrvTraceSink, deepHrvWindow) if (healed == 0) out // #899 heal re-pass: the pass above deleted overlapping duplicate sleep sessions AFTER its days @@ -241,7 +246,8 @@ object IntelligenceEngine { // are gone), so this can never loop. Mirrors the Swift pendingForcedRescore re-arm. else analyzeRecentOnCpu(repo, profile, maxDays, importedDeviceId, maxHROverride, nowSeconds, ownerSource, manualStepCoefficient, persistStepsCalibration, baselineEpoch, - recoveryEpoch, diag, useExperimentalSleepV2, useMotionAwareWake, sleepTraceSink, recoveryTraceSink, + recoveryEpoch, diag, useExperimentalSleepV2, useMotionAwareWake, useHrFirstSleep, + sleepTraceSink, recoveryTraceSink, stepsTraceSink, universalSink, workoutsTraceSink, hrvTraceSink, deepHrvWindow).first } } @@ -307,6 +313,8 @@ object IntelligenceEngine { useExperimentalSleepV2: Boolean = false, // Opt-in motion-aware wake refinement (#364 follow-up), threaded the same way. Default false. useMotionAwareWake: Boolean = false, + // HR-first sleep detection (experimental, default OFF), threaded the same way. Default false. + useHrFirstSleep: Boolean = false, // Sleep & Rest test-mode trace sink (Test Centre E5). null = byte-identical default; when non-null // each scored day threads it into AnalyticsEngine.analyzeDay so detectSleep's gate trace + the Rest // sub-score line forward line-by-line to the .sleep-tagged strap log. Mirrors Swift. @@ -592,6 +600,8 @@ object IntelligenceEngine { useSleepStagerV2 = useExperimentalSleepV2, // #364 follow-up: same threading for the motion-aware wake refinement post-pass. useMotionAwareWake = useMotionAwareWake, + // HR-first sleep detection (experimental, default OFF). + useHrFirstSleep = useHrFirstSleep, // Sleep & Rest test mode (Test Centre E5): thread the trace sink straight through. null (the // default) keeps analyzeDay's byte-identical untraced path; when the caller passed a non-null // sink (mode on), detectSleep's gate trace + the Rest sub-score line route to the .sleep-tagged diff --git a/android/app/src/main/java/com/noop/analytics/SleepStager.kt b/android/app/src/main/java/com/noop/analytics/SleepStager.kt index 177f2a1aa4..dac21bca1a 100644 --- a/android/app/src/main/java/com/noop/analytics/SleepStager.kt +++ b/android/app/src/main/java/com/noop/analytics/SleepStager.kt @@ -587,6 +587,97 @@ object SleepStager { return out } + // ── HR-first sleep detection (experimental, default OFF) ───────────────── + // + // The gravity-stillness spine reads a RESTLESS but genuinely asleep night — periodic limb + // movement, an injured/bedridden sleeper's tossing, or a strap whose motion channel + // over-reports — as "active", so whole hours of physiologically unambiguous sleep (sustained + // sub-baseline HR) never become candidate runs and the night collapses to whichever quiescent + // fragment survives. A real 5/MG user lost 6-8 h nights to 1-3 h morning fragments this way, + // while an ~8-min bathroom get-up — smeared past mergeMin by the CENTERED 15-min rolling + // stillness window — split the remaining night into separate stored sessions at a 16-min seam. + // The dense-gravity path has no HR-led recovery at all: HR only ever CONFIRMS gravity-derived + // runs (the #308 rescue is gated on gravity SPARSENESS, which a dense restless night never + // trips). The HR-first pass (default OFF; the live call site threads + // PuffinExperiment.hrFirstSleep) weights HR over motion when the two disagree: + // 1. [hrRescueActiveRuns] — an "active" run whose MEDIAN in-run HR sits inside the SAME sleep + // band the ladder already trusts ([confirmSleepWithHR]'s ≤ baseline × hrSleepBaselineMult) + // is reclassified "sleep": motion alone must not veto sleep the cardiac signal confirms. + // 2. [bridgeWakeGaps] — adjacent sleep runs separated by ≤ [wakeBridgeMaxMin] are merged into + // ONE candidate run, so a short get-up yields one session whose stager scores the wake + // INSIDE it instead of two stored sessions — and so sub-minSleepMin fragments between + // arousals are no longer dropped outright. + // Every downstream gate (minSleepMin, the H4 span cap, hrConfirm, off-wrist, the daytime and + // morning guards) still applies to the ASSEMBLED runs. KNOWN LIMIT (why default OFF, per the + // #194/#345 discipline): an awake-in-bed stretch whose HR genuinely sits inside the sleep band + // (deep pre-sleep relaxation) can be scored asleep — the same limit the still-sedentary gravity + // path already has, extended to fidgety-sedentary. Mirrors Swift. + + /** Adjacent sleep runs separated by at most this many minutes are assembled into ONE candidate + * run (HR-first pass only). Below the selector-side `SleepStageTotals.gapBridgeMaxMin` (60) so + * selection/aggregation behaviour is unchanged for anything this bridge leaves split; well + * above the ~16-min smeared seam a brief real get-up produces. Mirrors Swift `wakeBridgeMaxMin`. */ + const val wakeBridgeMaxMin: Int = 45 + + /** + * HR-first step 1: reclassify an "active" run to "sleep" when its median in-run HR sits inside + * the sleep band (≤ effective-baseline × [hrSleepBaselineMult] — the identical band + median + * statistic [confirmSleepWithHR] trusts, with the same [sleepHRBaseline] personalised override). + * Runs with fewer than [hrRefineMinSamples] in-run HR samples are left untouched (too little + * cardiac evidence to overrule motion), as is everything when no baseline exists. Pure + + * deterministic; mirrors Swift `hrRescueActiveRuns`. + */ + internal fun hrRescueActiveRuns( + periods: List, hr: List, baseline: Double?, + sleepHRBaseline: Double? = null, traceSink: ((String) -> Unit)? = null, + ): List { + val effBaseline = sleepHRBaseline ?: baseline ?: return periods + return periods.map { p -> + if (p.stage != "active") return@map p + val seg = rowsBetween(hr, p.start, p.end) { it.ts } + if (seg.size < hrRefineMinSamples) return@map p + val medHR = HrvAnalyzer.median(seg.map { it.bpm.toDouble() }) + if (medHR > effBaseline * hrSleepBaselineMult) return@map p + traceSink?.invoke(SleepStagerTrace.runLine(-1, p.start, p.end, + SleepStagerTrace.Verdict.KEPT, "hrRescue", + "medHR=${medHR.toInt()} band=${SleepStagerTrace.round2(effBaseline * hrSleepBaselineMult)}")) + Period(stage = "sleep", start = p.start, end = p.end) + } + } + + /** + * HR-first step 2: assemble adjacent sleep runs separated by at most [wakeBridgeMaxMin] minutes + * (whatever lies between — an un-rescued active run, or a plain data hole) into one candidate + * run; the intervening runs are absorbed and the stager later scores the wake INSIDE the single + * session. A gap over the threshold (or a negative/overlapping gap — pinned `gap >= 0` + * semantics, matching [bridgeSparseSleep]) never bridges. Pure + deterministic; mirrors Swift + * `bridgeWakeGaps`. + */ + internal fun bridgeWakeGaps( + periods: List, traceSink: ((String) -> Unit)? = null, + ): List { + val bridgeS = (wakeBridgeMaxMin * 60).toLong() + val out = ArrayList() + for (p in periods) { + val j = out.indexOfLast { it.stage == "sleep" } + if (p.stage == "sleep" && j >= 0 && + p.start - out[j].end >= 0 && p.start - out[j].end <= bridgeS + ) { + val gapMin = (p.start - out[j].end) / 60 + if (gapMin > 0) { + traceSink?.invoke(SleepStagerTrace.runLine(-1, out[j].end, p.start, + SleepStagerTrace.Verdict.KEPT, "wakeBridge", + "gapMin=$gapMin max=$wakeBridgeMaxMin")) + } + out[j] = Period(stage = "sleep", start = out[j].start, end = maxOf(out[j].end, p.end)) + while (out.size > j + 1) out.removeAt(out.size - 1) + } else { + out.add(p) + } + } + return out + } + // ── HR refinement ──────────────────────────────────────────────────────── private inline fun rowsBetween(rows: List, start: Long, end: Long, ts: (T) -> Long): List = @@ -876,6 +967,7 @@ object SleepStager { private data class DetectKey( val grav: Long, val hr: Long, val rr: Long, val resp: Long, val tz: Long, val wristOff: Long, val band: Long, val v2: Boolean, + val hrFirst: Boolean, val sleepHRBaseline: Double?, ) @@ -947,6 +1039,12 @@ object SleepStager { // #277, extended to every strap family in #351), so a normal user's nights are staged by V2. // Mirrors Swift. useSleepStagerV2: Boolean = false, + // HR-first sleep detection (experimental, default OFF — see the section comment above + // [wakeBridgeMaxMin]). When true, active runs whose median HR sits in the sleep band are + // rescued as sleep and adjacent sleep runs across ≤ wakeBridgeMaxMin wake gaps are assembled + // into one candidate run before the gate ladder. The default keeps every existing caller + // byte-identical; the live call site threads PuffinExperiment.hrFirstSleep. Mirrors Swift. + hrFirstSleep: Boolean = false, // Motion-corroborated wake (#462, directive b): the wearer's PERSONALISED overnight HR band // ([adaptiveOvernightHRBaseline]), used by [confirmSleepWithHR] in place of the day-median so a // supplement / fitness era self-calibrates the sleep band. Default null keeps the day-median @@ -964,7 +1062,7 @@ object SleepStager { // Swift detectSleep traceSink bypass (#707). if (traceSink != null) { return detectSleepUncached(hr, rr, resp, gravity, tzOffsetSeconds, wristOff, - bandSleepState, useSleepStagerV2, sleepHRBaseline, traceSink) + bandSleepState, useSleepStagerV2, hrFirstSleep, sleepHRBaseline, traceSink) } val key = DetectKey( // Fold the three gravity axes SEPARATELY (raw IEEE-754 bits, like StagerCache.fingerprint) @@ -985,6 +1083,7 @@ object SleepStager { wristOff = streamFingerprint(wristOff, { it.first }) { it.second }, band = streamFingerprint(bandSleepState, { it.first }) { it.second.toLong() }, v2 = useSleepStagerV2, + hrFirst = hrFirstSleep, sleepHRBaseline = sleepHRBaseline, ) synchronized(detectCacheLock) { detectCache[key] }?.let { return copyDetected(it) } @@ -992,7 +1091,7 @@ object SleepStager { // serialises another thread's lookup, and a racing duplicate compute just overwrites the entry // with an identical value — benign, the function is deterministic. val sessions = detectSleepUncached(hr, rr, resp, gravity, tzOffsetSeconds, wristOff, - bandSleepState, useSleepStagerV2, sleepHRBaseline, null) + bandSleepState, useSleepStagerV2, hrFirstSleep, sleepHRBaseline, null) synchronized(detectCacheLock) { detectCache[key] = copyDetected(sessions) } return sessions } @@ -1008,6 +1107,7 @@ object SleepStager { wristOff: List>, bandSleepState: List>, useSleepStagerV2: Boolean, + hrFirstSleep: Boolean, sleepHRBaseline: Double?, traceSink: ((String) -> Unit)?, ): List { @@ -1041,6 +1141,19 @@ object SleepStager { "sparseBridge", "sparse=true gapMin=$sparseBridgeGapMin runsBefore=$runsBeforeBridge runsAfter=$runsAfterBridge")) } + // HR-first pass (experimental, default OFF — see the section comment above [wakeBridgeMaxMin]): + // weight HR over motion where the two disagree (an active run whose median HR is in the sleep + // band is rescued as sleep), then assemble one candidate run per sleep period across short + // (≤ wakeBridgeMaxMin) wake gaps, BEFORE the ladder — so sub-minSleepMin fragments between + // arousals survive as part of the night instead of being dropped, and a brief get-up stays + // INSIDE one session as staged wake. Every gate below still applies to the assembled runs. + // Mirrors Swift. + if (hrFirstSleep) { + runs = hrRescueActiveRuns(runs, hrS, baseline, + sleepHRBaseline = sleepHRBaseline, traceSink = traceSink) + runs = bridgeWakeGaps(runs, traceSink = traceSink) + } + val minSleepS = (minSleepMin * 60).toLong() val sessions = ArrayList() diff --git a/android/app/src/main/java/com/noop/ble/PuffinExperiment.kt b/android/app/src/main/java/com/noop/ble/PuffinExperiment.kt index 8f070188c0..a6ff900395 100644 --- a/android/app/src/main/java/com/noop/ble/PuffinExperiment.kt +++ b/android/app/src/main/java/com/noop/ble/PuffinExperiment.kt @@ -96,6 +96,22 @@ class PuffinExperiment(private val prefs: SharedPreferences) { get() = prefs.getBoolean(KEY_MOTION_AWARE_WAKE, false) set(v) = prefs.edit().putBoolean(KEY_MOTION_AWARE_WAKE, v).apply() + /** Opt-in "HR-first sleep detection" (default OFF): weight the cardiac signal over wrist motion + * when the two disagree, inside [com.noop.analytics.SleepStager.detectSleep]. Two effects, both + * BEFORE the gate ladder: a motion-"active" run whose median HR sits inside the same sleep band + * the ladder already trusts is rescued as sleep (a restless-but-asleep night is no longer + * collapsed to its quiescent morning fragment), and adjacent sleep runs across a ≤ 45-min wake + * gap are assembled into ONE session with the wake staged inside it (a brief get-up no longer + * splits the night into separate stored sessions). Every existing guard (min duration, span cap, + * HR confirmation, off-wrist, daytime nap bars) still applies to the assembled runs. Default OFF + * per the varying-input validation discipline (#194/#345): an awake-in-bed stretch whose HR + * genuinely sits in the sleep band can be scored asleep, the same limit the still-sedentary path + * already has. Pure analysis flag (sends nothing to the strap), model-agnostic. Mirrors the + * macOS `PuffinExperiment.hrFirstSleepKey`. */ + var hrFirstSleep: Boolean + get() = prefs.getBoolean(KEY_HR_FIRST_SLEEP, false) + set(v) = prefs.edit().putBoolean(KEY_HR_FIRST_SLEEP, v).apply() + /** * Turn OFF every 5/MG-only experimental probe: protocol probes ([isEnabled]), raw capture * ([isCaptureEnabled]), the R22 deep-data strap write ([isDeepDataEnabled]) and broadcast-HR @@ -146,6 +162,7 @@ class PuffinExperiment(private val prefs: SharedPreferences) { /** "Motion-aware wake refinement" opt-in (mirrors macOS `PuffinExperiment.motionAwareWakeKey`). */ const val KEY_MOTION_AWARE_WAKE = "noopMotionAwareWake" + const val KEY_HR_FIRST_SLEEP = "noopHrFirstSleep" fun from(context: Context): PuffinExperiment = PuffinExperiment(context.getSharedPreferences(PREFS, Context.MODE_PRIVATE)) diff --git a/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt b/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt index 0184d05340..b75bdd0000 100644 --- a/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt +++ b/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt @@ -2126,6 +2126,8 @@ class WhoopBleClient( useExperimentalSleepV2 = PuffinExperiment.from(context).experimentalSleepV2, // Opt-in motion-aware wake refinement (#364 follow-up) — same Context-free threading. useMotionAwareWake = PuffinExperiment.from(context).motionAwareWake, + // HR-first sleep detection (experimental, default OFF) — same Context-free threading. + useHrFirstSleep = PuffinExperiment.from(context).hrFirstSleep, // Sleep & Rest test mode (Test Centre E5): when the SLEEP domain is on, route this // post-backfill pass's per-day sleep gate trace into the .sleep-tagged strap log, so a // shared report carries the staging proof from THIS scoring pass too, not only the UI diff --git a/android/app/src/main/java/com/noop/ui/AppViewModel.kt b/android/app/src/main/java/com/noop/ui/AppViewModel.kt index 12954c02f4..95ee62f26c 100644 --- a/android/app/src/main/java/com/noop/ui/AppViewModel.kt +++ b/android/app/src/main/java/com/noop/ui/AppViewModel.kt @@ -943,6 +943,8 @@ class AppViewModel(app: Application) : AndroidViewModel(app) { useExperimentalSleepV2 = PuffinExperiment.from(appContext).experimentalSleepV2, // Opt-in motion-aware wake refinement (#364 follow-up) — same Context-free threading. useMotionAwareWake = PuffinExperiment.from(appContext).motionAwareWake, + // HR-first sleep detection (experimental, default OFF) — same Context-free threading. + useHrFirstSleep = PuffinExperiment.from(appContext).hrFirstSleep, // Sleep & Rest test mode (Test Centre E5): when the SLEEP domain is on, route the // per-day sleep gate trace into the SAME shareable strap log, tagged .sleep so it // lands under the profile in the export. Zero-cost when off: the gate is one @@ -1562,6 +1564,8 @@ class AppViewModel(app: Application) : AndroidViewModel(app) { useExperimentalSleepV2 = PuffinExperiment.from(appContext).experimentalSleepV2, // Opt-in motion-aware wake refinement (#364 follow-up) — same flag the 15-min loop reads. useMotionAwareWake = PuffinExperiment.from(appContext).motionAwareWake, + // HR-first sleep detection (experimental, default OFF) — same flag the 15-min loop reads. + useHrFirstSleep = PuffinExperiment.from(appContext).hrFirstSleep, ) }.onFailure { if (it is kotlin.coroutines.cancellation.CancellationException) throw it } } diff --git a/android/app/src/main/java/com/noop/ui/SettingsScreen.kt b/android/app/src/main/java/com/noop/ui/SettingsScreen.kt index 469a2f735b..ed08f939f0 100644 --- a/android/app/src/main/java/com/noop/ui/SettingsScreen.kt +++ b/android/app/src/main/java/com/noop/ui/SettingsScreen.kt @@ -502,6 +502,9 @@ fun SettingsScreen( // "Motion-aware wake refinement" (#364 follow-up) — OFF by default. Self-gates on observed gravity + // step density, so it is a no-op on a sparse (e.g. WHOOP 4.0) night regardless of this switch. var motionAwareWake by remember { mutableStateOf(puffinExperiment.motionAwareWake) } + // "HR-first sleep detection" — OFF by default. Weights HR over wrist motion in sleep DETECTION and + // bridges short (≤45 min) wake gaps into one session; see PuffinExperiment.hrFirstSleep. + var hrFirstSleep by remember { mutableStateOf(puffinExperiment.hrFirstSleep) } // Whether to surface the WHOOP 5/MG-only probes (puffin / R22 / broadcast-HR / frame-capture). Gated // so a confident 4.0 owner never sees 5/MG controls that can't touch their strap (#22). The model @@ -2204,6 +2207,49 @@ fun SettingsScreen( color = Palette.textTertiary, ) + // --- HR-first sleep detection — OFF by default. --- + Row( + modifier = Modifier.fillMaxWidth(), + verticalAlignment = Alignment.CenterVertically, + horizontalArrangement = Arrangement.spacedBy(16.dp), + ) { + Text( + "HR-first sleep detection", + style = NoopType.subhead, + color = Palette.textPrimary, + modifier = Modifier.weight(1f), + ) + Switch( + checked = hrFirstSleep, + onCheckedChange = { + hrFirstSleep = it + puffinExperiment.hrFirstSleep = it + }, + colors = SwitchDefaults.colors( + checkedThumbColor = Palette.surfaceBase, + checkedTrackColor = Palette.accent, + uncheckedThumbColor = Palette.textSecondary, + uncheckedTrackColor = Palette.surfaceInset, + uncheckedBorderColor = Palette.hairline, + ), + modifier = Modifier.semantics { + contentDescription = "HR-first sleep detection" + }, + ) + } + Text( + "Trusts your heart rate over wrist movement when the two disagree about sleep. A " + + "restless night whose heart rate clearly sits at your sleep level is kept as sleep " + + "instead of being cut down to its calmest fragment, and a brief get-up (under ~45 " + + "minutes) stays inside one sleep with the wake scored inside it, rather than " + + "splitting the night into separate sleeps. Worth trying if your nights show up " + + "fragmented or mostly missing despite a normal sleeping heart rate. Trade-off: a " + + "long, deeply relaxed stretch awake in bed can be scored as sleep. Off by default; " + + "takes effect on the next nights scored.", + style = NoopType.caption, + color = Palette.textTertiary, + ) + // Diagnostics: dump the decoded per-sample sensor streams (last 24h) to one long-format // CSV so power users / external devs can prototype sleep/activity/VBT algorithms on real // data without a BLE stream (#308/#276/#322). On-device only; plain text, no BLE hex. diff --git a/android/app/src/test/java/com/noop/analytics/SleepStagerHrFirstTest.kt b/android/app/src/test/java/com/noop/analytics/SleepStagerHrFirstTest.kt new file mode 100644 index 0000000000..eccaa12d30 --- /dev/null +++ b/android/app/src/test/java/com/noop/analytics/SleepStagerHrFirstTest.kt @@ -0,0 +1,202 @@ +package com.noop.analytics + +import com.noop.data.GravitySample +import com.noop.data.HrSample +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test +import kotlin.math.abs + +/** + * HR-first sleep detection (experimental, default OFF): weight HR over motion when the two + * disagree, and assemble one session per sleep period across short wake gaps. + * + * Per the "validate against varying inputs" rule (#194/#345), these tests inject the sleep window + * at DIFFERENT positions and lengths and assert the rescue TRACKS it — and that the identical + * motion pattern with an elevated (awake) HR is refused — rather than pinning one matched night. + * Faithful Kotlin mirror of SleepStagerHrFirstTests.swift: same scenarios, same numbers. + */ +class SleepStagerHrFirstTest { + + private val dev = "test" + + /** 2026-06-10 00:00:00 UTC — same fixed midnight the Swift twin uses. */ + private val refMidnight = 1_749_513_600L + + private fun at(hourUTC: Int, min: Int = 0): Long = refMidnight + hourUTC * 3_600L + min * 60L + + private fun stillGravity(start: Long, durationS: Int): List = + (0 until durationS).map { GravitySample(deviceId = dev, ts = start + it, x = 0.0, y = 0.0, z = 1.0) } + + /** Oscillating orientation (0.5 g jumps per sample) — clearly "moving" to the stillness spine. */ + private fun restlessGravity(start: Long, durationS: Int): List = + (0 until durationS).map { i -> + val phase = (i % 2) * 0.5 + GravitySample(deviceId = dev, ts = start + i, x = phase, y = 0.0, z = 1.0) + } + + private fun hrStream(start: Long, durationS: Int, bpm: Int): List = + (0 until durationS).map { HrSample(deviceId = dev, ts = start + it, bpm = bpm) } + + // ── Restless-night rescue (the DROPPED-night failure) ───────────────────── + + @Test + fun restlessNightRescuedToFullSession() { + // Restless (motion "active") but cardiac-unambiguous (HR 52 vs awake 75) 22:00–04:30, + // then still to 06:00. OFF: only the quiescent morning fragment scores. ON: one session. + val grav = restlessGravity(at(22), 6 * 3_600 + 1_800) + // 22:00–04:30 + stillGravity(at(28, 30), 90 * 60) // 04:30–06:00 + val hr = hrStream(at(20), 2 * 3_600, 75) + // 20:00–22:00 awake + hrStream(at(22), 8 * 3_600, 52) + // 22:00–06:00 asleep + hrStream(at(30), 2 * 3_600, 75) // 06:00–08:00 awake + + val off = SleepStager.detectSleep(hr = hr, gravity = grav) + assertEquals("OFF: only the still morning fragment should score", 1, off.size) + assertTrue("OFF fragment starts near 04:30", off[0].start >= at(28, 25)) + assertTrue(off[0].start <= at(28, 45)) + + val on = SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = true) + assertEquals("ON: one assembled session for the whole night", 1, on.size) + assertTrue("ON: onset tracks the injected 22:00 sleep start", on[0].start <= at(22) + 120) + assertTrue("ON: end tracks the injected 06:00 wake", on[0].end >= at(30) - 120) + } + + @Test + fun restlessRescueTracksInjectedWindow() { + // Same rescue with the sleep window injected at a DIFFERENT position and length + // (01:00–07:30, all restless, no still stretch at all). + val grav = restlessGravity(at(25), 6 * 3_600 + 1_800) // 01:00–07:30 + val hr = hrStream(at(22), 3 * 3_600, 75) + // 22:00–01:00 awake + hrStream(at(25), 6 * 3_600 + 1_800, 52) + // 01:00–07:30 asleep + hrStream(at(31, 30), 2 * 3_600, 75) // 07:30–09:30 awake + + assertEquals("OFF: an all-restless night scores nothing", + 0, SleepStager.detectSleep(hr = hr, gravity = grav).size) + + val on = SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = true) + assertEquals(1, on.size) + assertTrue("onset tracks the moved window", abs(on[0].start - at(25)) <= 120) + assertTrue("end tracks the moved window", abs(on[0].end - at(31, 30)) <= 120) + } + + @Test + fun elevatedHrNightIsNotRescued() { + // IDENTICAL motion to the full-night rescue, night HR elevated (78 vs a 62 day median): + // the rescue must refuse the restless stretch. (The STILL morning fragment is still + // accepted by the pre-existing motion-corroborated quiescent band (#462), so the assertion + // is that HR-first adds NOTHING: same single fragment, onset still at the quiescent stretch.) + val grav = restlessGravity(at(22), 6 * 3_600 + 1_800) + // 22:00–04:30 + stillGravity(at(28, 30), 90 * 60) // 04:30–06:00 + val hr = hrStream(at(22), 8 * 3_600, 78) + // 22:00–06:00 elevated + hrStream(at(30), 10 * 3_600, 62) // 06:00–16:00 day median + + val off = SleepStager.detectSleep(hr = hr, gravity = grav) + val on = SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = true) + assertEquals("same motion, elevated HR: HR-first must change nothing", off, on) + assertEquals(1, on.size) + assertTrue("the restless 22:00–04:30 stretch must NOT be rescued", on[0].start >= at(28, 25)) + } + + // ── Wake-gap bridging (the SPLIT-night failure) ────────────────────────── + + @Test + fun briefGetUpBridgedIntoOneSession() { + // A 16-min get-up (real movement + HR 80) mid-night. The centered rolling window smears it + // past mergeMin, so OFF stores TWO sessions; ON bridges them into ONE. + val grav = stillGravity(at(23), 4 * 3_600) + // 23:00–03:00 + restlessGravity(at(27), 16 * 60) + // 03:00–03:16 get-up + stillGravity(at(27, 16), 3 * 3_600 + 44 * 60) // 03:16–07:00 + val hr = hrStream(at(21), 2 * 3_600, 75) + + hrStream(at(23), 4 * 3_600, 52) + + hrStream(at(27), 16 * 60, 80) + + hrStream(at(27, 16), 3 * 3_600 + 44 * 60, 52) + + hrStream(at(31), 2 * 3_600, 75) + + assertEquals("OFF: the smeared get-up splits the night", + 2, SleepStager.detectSleep(hr = hr, gravity = grav).size) + + val on = SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = true) + assertEquals("ON: one session bridging the get-up", 1, on.size) + assertTrue(abs(on[0].start - at(23)) <= 120) + assertTrue(abs(on[0].end - at(31)) <= 120) + } + + @Test + fun longWakeGapStaysSplit() { + // An 80-min genuinely-awake block (elevated HR, real movement) exceeds the 45-min bridge. + val grav = stillGravity(at(22), 3 * 3_600) + // 22:00–01:00 + restlessGravity(at(25), 80 * 60) + // 01:00–02:20 awake + stillGravity(at(26, 20), 3 * 3_600 + 40 * 60) // 02:20–06:00 + val hr = hrStream(at(20), 2 * 3_600, 75) + + hrStream(at(22), 3 * 3_600, 52) + + hrStream(at(25), 80 * 60, 78) + + hrStream(at(26, 20), 3 * 3_600 + 40 * 60, 52) + + hrStream(at(30), 2 * 3_600, 75) + + assertEquals("an over-threshold wake gap must not be bridged", + 2, SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = true).size) + } + + // ── Guards still hold on assembled runs ────────────────────────────────── + + @Test + fun daytimeGuardStillAppliesToRescuedRuns() { + // A rescued + bridged DAYTIME stretch (in-band flat HR 60, no cardiac dip) must still be + // dropped by the daytime false-sleep guard. + val grav = stillGravity(at(11), 2 * 3_600) + // 11:00–13:00 + restlessGravity(at(13), 2 * 3_600) + // 13:00–15:00 + stillGravity(at(15), 2 * 3_600) // 15:00–17:00 + val hr = hrStream(at(10), 8 * 3_600, 60) // flat, no dip + + assertEquals("no resting-HR dip: the daytime guard must reject the rescue", + 0, SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = true).size) + } + + @Test + fun defaultOffIsByteIdentical() { + val grav = stillGravity(at(23), 4 * 3_600) + + restlessGravity(at(27), 16 * 60) + + stillGravity(at(27, 16), 3 * 3_600 + 44 * 60) + val hr = hrStream(at(23), 8 * 3_600, 52) + hrStream(at(31), 4 * 3_600, 75) + + val implicitDefault = SleepStager.detectSleep(hr = hr, gravity = grav) + val explicitFalse = SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = false) + assertEquals(implicitDefault, explicitFalse) + } + + // ── Pure helpers ───────────────────────────────────────────────────────── + + @Test + fun bridgeWakeGapsMergesOnlyUnderThreshold() { + val a = SleepStager.Period(stage = "sleep", start = 0L, end = 3_600L) + val gap = SleepStager.Period(stage = "active", start = 3_600L, end = 3_600L + 40 * 60) + val b = SleepStager.Period(stage = "sleep", start = 3_600L + 40 * 60, end = 3 * 3_600L) + val merged = SleepStager.bridgeWakeGaps(listOf(a, gap, b)) + assertEquals(1, merged.size) + assertEquals(0L, merged[0].start) + assertEquals(3 * 3_600L, merged[0].end) + + val farGap = SleepStager.Period(stage = "active", start = 3_600L, end = 3_600L + 46 * 60) + val c = SleepStager.Period(stage = "sleep", start = 3_600L + 46 * 60, end = 4 * 3_600L) + assertEquals("a 46-min gap is over wakeBridgeMaxMin (45) and must not merge", + 3, SleepStager.bridgeWakeGaps(listOf(a, farGap, c)).size) + } + + @Test + fun hrRescueRespectsBandAndSampleFloor() { + val p = SleepStager.Period(stage = "active", start = 0L, end = 3_600L) + val inBand = (0 until 3_600).map { HrSample(deviceId = dev, ts = it.toLong(), bpm = 52) } + assertEquals("sleep", SleepStager.hrRescueActiveRuns(listOf(p), inBand, 52.0)[0].stage) + + val outOfBand = (0 until 3_600).map { HrSample(deviceId = dev, ts = it.toLong(), bpm = 60) } + assertEquals("median above baseline × 1.05 must not rescue", + "active", SleepStager.hrRescueActiveRuns(listOf(p), outOfBand, 52.0)[0].stage) + + // Below the hrRefineMinSamples floor: too little cardiac evidence to overrule motion. + val sparse = (0 until 10).map { HrSample(deviceId = dev, ts = it * 60L, bpm = 52) } + assertEquals("active", SleepStager.hrRescueActiveRuns(listOf(p), sparse, 52.0)[0].stage) + + // No baseline at all: nothing to weigh HR against. + assertEquals("active", SleepStager.hrRescueActiveRuns(listOf(p), inBand, null)[0].stage) + } +} From 56443e1dbab10b4340eea2e708e14881d48614f6 Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 24 Aug 2026 14:36:46 -0400 Subject: [PATCH 170/171] Sleep: withdraw the HR rescue after a real-data replay; keep the wake bridge, span-guarded A 5-night replay of the reporting wearer's raw mirror streams (Aug 2026) through detectSleep falsified the HR-rescue half of the previous commit: on a low-HR wearer the quiet awake EVENING median (~58 bpm) also sits inside the x1.05 sleep band off the ~64 window baseline, so whole evenings were rescued and chained through the wake bridge - the assembled run then either exceeded the 16 h H4 cap and dropped the night to ZERO (worse than baseline), or squeaked under it and scored a 15.4 h "night" starting at 18:00. The strap's own banked band sleep_state cannot arbitrate: it is motion-led too, and read "wake" across hours of HR-proven sleep. Median HR LEVEL cannot separate that wearer's quiet wakefulness from sleep; only dip DEPTH (sleep-only minima 42-48 bpm) looked separable, and deriving an onset boundary from dips needs a properly validated design (#194/#345). The restless-night DROPPING failure stays OPEN; a history note above `wakeBridgeMaxMin` records the evidence so it is not re-attempted naively. What remains is the safe half, renamed "Bridge brief wakes" (`noopSleepWakeBridge`, default OFF): `bridgeWakeGaps` assembles adjacent sleep runs <= 45 min apart into one candidate run before the gate ladder, now REFUSING any merge whose assembled span would exceed maxMainSleepSpanS - without that guard a pathological chain would assemble and then be dropped whole by the H4 cap, converting a partially-scored night into a zero-score night (the exact bimodal failure the replay exposed). The tests are rewritten around the surviving feature and pick up a REAL-DATA pin: the exact stored fragment layouts of five of the wearer's split nights, asserting which seams the bridge merges (the 16/12/28/25/ 19-min arousal seams) and which it leaves split (2.2-3.9 h genuine wake gaps; a 90-min morning gap left to the selector's #861 night-tail bridge). The bridge reproduces 7/7 of the sub-45-min seam-fills that an independent HR-evidence-led correction pass applied to the same nights on the wearer's mirror; the two disagreements (a kept-separate morning doze it folds in, and the 90-min restless gap it refuses to guess) are documented in the pin. A restless-night test now pins that the bridge deliberately does NOT reclassify motion-vetoed sleep. Cross-platform parity: identical rename (wakeBridge / useWakeBridge / PuffinExperiment.wakeBridgeEnabled / sleepWakeBridge, key noopSleepWakeBridge), identical span guard, identical tests (SleepStagerWakeBridgeTests.swift / SleepStagerWakeBridgeTest.kt). Verified: swift test (StrandAnalytics, 1332 passing incl. 8 rewritten), macOS Strand build, iOS NOOPiOS build, Android compileFullDebugKotlin + testFullDebugUnitTest (all passing incl. 8 rewritten). Co-Authored-By: Claude Fable 5 --- .../StrandAnalytics/AnalyticsEngine.swift | 19 +- .../Sources/StrandAnalytics/SleepStager.swift | 144 ++++++------- .../SleepStagerHrFirstTests.swift | 199 ----------------- .../SleepStagerWakeBridgeTests.swift | 194 +++++++++++++++++ Strand/BLE/PuffinExperiment.swift | 27 ++- Strand/Data/IntelligenceEngine.swift | 12 +- Strand/Screens/SettingsView.swift | 10 +- .../com/noop/analytics/AnalyticsEngine.kt | 15 +- .../com/noop/analytics/IntelligenceEngine.kt | 20 +- .../java/com/noop/analytics/SleepStager.kt | 123 ++++------- .../java/com/noop/ble/PuffinExperiment.kt | 31 ++- .../main/java/com/noop/ble/WhoopBleClient.kt | 4 +- .../src/main/java/com/noop/ui/AppViewModel.kt | 8 +- .../main/java/com/noop/ui/SettingsScreen.kt | 32 ++- .../noop/analytics/SleepStagerHrFirstTest.kt | 202 ------------------ .../analytics/SleepStagerWakeBridgeTest.kt | 202 ++++++++++++++++++ 16 files changed, 589 insertions(+), 653 deletions(-) delete mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerHrFirstTests.swift create mode 100644 Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerWakeBridgeTests.swift delete mode 100644 android/app/src/test/java/com/noop/analytics/SleepStagerHrFirstTest.kt create mode 100644 android/app/src/test/java/com/noop/analytics/SleepStagerWakeBridgeTest.kt diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift index 1c781e0e15..90c26d2407 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/AnalyticsEngine.swift @@ -344,15 +344,14 @@ public enum AnalyticsEngine { // caller/test byte-identical; IntelligenceEngine threads // `PuffinExperiment.motionAwareWakeEnabled`. useMotionAwareWake: Bool = false, - // HR-first sleep detection (experimental, default OFF — see the - // section comment above `SleepStager.wakeBridgeMaxMin`). When true, - // detection weights HR over motion where the two disagree (an - // "active" run whose median HR sits in the sleep band is rescued as - // sleep) and assembles one session per sleep period across - // ≤ 45-min wake gaps. Default false keeps every pure-function - // caller/test byte-identical; IntelligenceEngine threads - // `PuffinExperiment.hrFirstSleepEnabled`. - useHrFirstSleep: Bool = false, + // Wake-gap bridging (experimental, default OFF — see the section + // comment above `SleepStager.wakeBridgeMaxMin`). When true, + // detection assembles one session per sleep period across ≤ 45-min + // wake gaps, so a brief get-up stays inside one session as staged + // wake. Default false keeps every pure-function caller/test + // byte-identical; IntelligenceEngine threads + // `PuffinExperiment.wakeBridgeEnabled`. + useWakeBridge: Bool = false, // Sleep PROVENANCE for the per-day sleep trace (CAPTURE-C / #799). The // measured BLE path is `.measured` (the default); the caller passes // `.imported(...)` when a previously-imported sleep row WON the daily merge, @@ -394,7 +393,7 @@ public enum AnalyticsEngine { tzOffsetSeconds: tzOffsetSeconds, wristOff: wristOff, bandSleepState: bandSleepState, useSleepStagerV2: useSleepStagerV2, - hrFirstSleep: useHrFirstSleep, + wakeBridge: useWakeBridge, traceSink: traceSink) // Motion-aware wake refinement (#364 follow-up) runs AFTER V1/V2 staging, over every detected // session (naps included — the same eligibility gates apply). `steps` is the SAME calendar-day/ diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift index c218bedff3..e12c5c2b01 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift @@ -190,76 +190,54 @@ public enum SleepStager { /// while still dropping an all-day desk strap (≈100% gap) or a session genuinely spent off-wrist. public static let maxOffWristSleepFraction: Double = 0.5 - // MARK: - HR-first sleep detection (experimental, default OFF) - - // The gravity-stillness spine reads a RESTLESS but genuinely asleep night — periodic limb - // movement, an injured/bedridden sleeper's tossing, or a strap whose motion channel over-reports - // — as "active", so whole hours of physiologically unambiguous sleep (sustained sub-baseline HR) - // never become candidate runs and the night collapses to whichever quiescent fragment survives. - // A real 5/MG user lost 6–8 h nights to 1–3 h morning fragments this way (hourly HR 47–58 bpm from - // ~01:00 through ~09:00 local scored nothing), while an ~8-min bathroom get-up — smeared past - // mergeMin by the CENTERED 15-min rolling stillness window — split the remaining night into - // separate stored sessions at a 16-min seam. The dense-gravity path has no HR-led recovery at all: - // HR only ever CONFIRMS gravity-derived runs (the #308 rescue is gated on gravity SPARSENESS, - // which a dense restless night never trips). + // MARK: - Wake-gap bridging (experimental, default OFF) + + // A brief mid-night get-up splits the night into separate stored sessions: the CENTERED 15-min + // rolling stillness window smears an ~8-min bathroom trip past mergeMin into an "active" run (a + // real MG night split at a 16-min seam, 06:17 end / 06:33 start), detection emits one stored + // session per still-run, and nothing at the detection layer re-merges them — the #561/#861 + // bridges only group sessions later, at selection/aggregation, so sub-minSleepMin sleep + // fragments between arousals are dropped by the ladder before those bridges ever see them. + // Industry convention (and the selector's own <60-min "same sleep period" threshold) treats a + // brief wake as belonging INSIDE the night. // - // The HR-first pass (default OFF; the live call site threads PuffinExperiment.hrFirstSleepEnabled) - // weights HR over motion when the two disagree, in two steps between the run spine and the ladder: - // 1. `hrRescueActiveRuns` — an "active" run whose MEDIAN in-run HR sits inside the SAME sleep - // band the ladder already trusts (`confirmSleepWithHR`'s ≤ baseline × hrSleepBaselineMult, - // same median statistic, same `sleepHRBaseline` override) is reclassified "sleep": motion - // alone must not veto sleep the cardiac signal confirms. - // 2. `bridgeWakeGaps` — adjacent sleep runs separated by ≤ `wakeBridgeMaxMin` are merged into - // ONE candidate run (the "same sleep period" convention — a brief mid-night wake belongs - // INSIDE the night), so a short get-up yields one session whose stager scores the wake - // inside it instead of two stored sessions — and so sub-minSleepMin sleep fragments between - // arousals are no longer dropped outright before the selector-side bridges ever see them. - // Every downstream gate (minSleepMin, the H4 span cap, hrConfirm, off-wrist, the daytime/morning - // guards) still applies to the ASSEMBLED runs, so a rescued daytime stretch still needs a real - // cardiac dip to survive, and an off-wrist block is still dropped. + // When enabled (the live call site threads PuffinExperiment.wakeBridgeEnabled, default OFF), + // `bridgeWakeGaps` assembles adjacent sleep runs separated by ≤ `wakeBridgeMaxMin` into ONE + // candidate run between the run spine and the gate ladder; the stager then scores the wake + // INSIDE the single session, and every downstream gate (minSleepMin, the H4 span cap, hrConfirm, + // off-wrist, the daytime/morning guards) still applies to the ASSEMBLED run. A merge that would + // assemble a span over `maxMainSleepSpanS` is REFUSED: without that guard a pathological chain + // would assemble and then be dropped WHOLE by the H4 cap, converting a partially-scored night + // into a zero-score night. // - // KNOWN LIMIT (why this defaults OFF, per the #194/#345 discipline): an awake-in-bed stretch whose - // HR genuinely sits inside the sleep band (deep pre-sleep relaxation) can be scored asleep — the - // same limit the still-sedentary gravity path already has, extended to fidgety-sedentary. The - // synthetic tests (SleepStagerHrFirstTests + the Kotlin twin) prove the pass TRACKS varying - // injected inputs — rescue follows wherever the sleep-band HR actually is, and refuses an - // identical motion pattern whose HR is elevated — not a single matched night. + // HISTORY — an HR-led "rescue" of motion-vetoed runs was built beside this bridge and WITHDRAWN + // after a 5-night real-data replay against the reporting wearer's server mirror (2026-08-24). + // The rescue reclassified an "active" run as sleep when its median in-run HR sat inside the + // `confirmSleepWithHR` band, aiming at restless-but-asleep nights the stillness spine collapses + // to their quietest fragment (hours of HR 47–58 bpm scoring nothing). On the real wearer the + // discriminator failed: a low-HR wearer's quiet awake EVENING (median ~58 bpm vs a ~64 window + // baseline) also sits inside the ×1.05 band, so whole evenings were rescued and chained — the + // assembled run then either exceeded the H4 cap and dropped the night to ZERO, or squeaked + // under it and scored a 15 h "night" starting at 18:00. The strap's own banked band sleep_state + // could not arbitrate: it is motion-led too and read "wake" across the HR-proven sleep. Median + // HR LEVEL cannot separate that wearer's quiet wakefulness from sleep; only dip DEPTH + // (sleep-only minima 42–48 bpm) looked separable, and placing an onset BOUNDARY from dips needs + // a properly validated design (#194/#345). The restless-night DROPPING failure therefore + // remains OPEN — do not re-land an HR-band rescue without multi-night evidence that it refuses + // quiet awake evenings on low-HR wearers. /// Adjacent sleep runs separated by at most this many minutes are assembled into ONE candidate - /// run (HR-first pass only). Below the selector-side `SleepStageTotals.gapBridgeMaxMin` (60) so - /// selection/aggregation behaviour is unchanged for anything this bridge leaves split; well above - /// the ~16-min smeared seam a brief real get-up produces. + /// run (wake-bridge pass only). Below the selector-side `SleepStageTotals.gapBridgeMaxMin` (60) + /// so selection/aggregation behaviour is unchanged for anything this bridge leaves split; well + /// above the ~16-min smeared seam a brief real get-up produces. public static let wakeBridgeMaxMin: Int = 45 - /// HR-first step 1: reclassify an "active" run to "sleep" when its median in-run HR sits inside - /// the sleep band (≤ effective-baseline × `hrSleepBaselineMult` — the identical band + median - /// statistic `confirmSleepWithHR` trusts, with the same `sleepHRBaseline` personalised override). - /// Runs with fewer than `hrRefineMinSamples` in-run HR samples are left untouched (too little - /// cardiac evidence to overrule motion), as is everything when no baseline exists. Pure + - /// deterministic; mirrors Kotlin `hrRescueActiveRuns`. - static func hrRescueActiveRuns(_ periods: [Period], hr: [HRSample], baseline: Double?, - sleepHRBaseline: Double? = nil, - traceSink: ((String) -> Void)? = nil) -> [Period] { - guard let effBaseline = sleepHRBaseline ?? baseline else { return periods } - return periods.map { p in - guard p.stage == "active" else { return p } - let seg = rowsBetween(hr, start: p.start, end: p.end) { $0.ts } - guard seg.count >= hrRefineMinSamples else { return p } - let medHR = HRVAnalyzer.median(seg.map { Double($0.bpm) }) - guard medHR <= effBaseline * hrSleepBaselineMult else { return p } - traceSink?(GateTrace.runLine(index: -1, startTs: p.start, endTs: p.end, - verdict: .kept, gate: "hrRescue", - detail: "medHR=\(Int(medHR)) band=\(round2(effBaseline * hrSleepBaselineMult))")) - return Period(stage: "sleep", start: p.start, end: p.end) - } - } - - /// HR-first step 2: assemble adjacent sleep runs separated by at most `wakeBridgeMaxMin` minutes - /// (whatever lies between — an un-rescued active run, or a plain data hole) into one candidate - /// run; the intervening runs are absorbed and the stager later scores the wake INSIDE the single - /// session. A gap over the threshold (or a negative/overlapping gap — pinned `gap >= 0` - /// semantics, matching `bridgeSparseSleep`) never bridges. Pure + deterministic; mirrors Kotlin - /// `bridgeWakeGaps`. + /// Assemble adjacent sleep runs separated by at most `wakeBridgeMaxMin` minutes (whatever lies + /// between — an active run, or a plain data hole) into one candidate run; the intervening runs + /// are absorbed and the stager later scores the wake INSIDE the single session. A gap over the + /// threshold, a negative/overlapping gap (pinned `gap >= 0` semantics, matching + /// `bridgeSparseSleep`), or a merge whose assembled span would exceed `maxMainSleepSpanS` (the + /// H4 guard above) never bridges. Pure + deterministic; mirrors Kotlin `bridgeWakeGaps`. static func bridgeWakeGaps(_ periods: [Period], traceSink: ((String) -> Void)? = nil) -> [Period] { let bridgeS = wakeBridgeMaxMin * 60 @@ -267,7 +245,8 @@ public enum SleepStager { for p in periods { if p.stage == "sleep", let j = out.lastIndex(where: { $0.stage == "sleep" }), - p.start - out[j].end >= 0, p.start - out[j].end <= bridgeS { + p.start - out[j].end >= 0, p.start - out[j].end <= bridgeS, + max(out[j].end, p.end) - out[j].start <= maxMainSleepSpanS { let gapMin = (p.start - out[j].end) / 60 if gapMin > 0 { traceSink?(GateTrace.runLine(index: -1, startTs: out[j].end, endTs: p.start, @@ -942,14 +921,14 @@ public enum SleepStager { wristOff: [(start: Int, end: Int)] = [], bandSleepState: [(ts: Int, state: Int)] = [], useSleepStagerV2: Bool = false, - // HR-first sleep detection (experimental, default OFF — see the - // section comment above `wakeBridgeMaxMin`). When true, active runs - // whose median HR sits in the sleep band are rescued as sleep and - // adjacent sleep runs across ≤ wakeBridgeMaxMin wake gaps are - // assembled into one candidate run before the gate ladder. The - // default keeps every existing caller byte-identical; the live call - // site threads PuffinExperiment.hrFirstSleepEnabled. - hrFirstSleep: Bool = false, + // Wake-gap bridging (experimental, default OFF — see the section + // comment above `wakeBridgeMaxMin`). When true, adjacent sleep runs + // across ≤ wakeBridgeMaxMin wake gaps are assembled into one + // candidate run before the gate ladder, so a brief get-up stays + // INSIDE one session as staged wake. The default keeps every + // existing caller byte-identical; the live call site threads + // PuffinExperiment.wakeBridgeEnabled. + wakeBridge: Bool = false, sleepHRBaseline: Double? = nil, traceSink: ((String) -> Void)? = nil) -> [SleepSession] { // Sleep & Rest test mode only: when a trace is requested we MUST run the live ladder, not a @@ -961,7 +940,7 @@ public enum SleepStager { return detectSleepUncached(hr: hr, rr: rr, resp: resp, gravity: gravity, tzOffsetSeconds: tzOffsetSeconds, wristOff: wristOff, bandSleepState: bandSleepState, useSleepStagerV2: useSleepStagerV2, - hrFirstSleep: hrFirstSleep, + wakeBridge: wakeBridge, sleepHRBaseline: sleepHRBaseline, traceSink: traceSink) } // v7.0.2 perf (#707): the single heaviest analytics call — it sorts the dense full-day gravity @@ -981,13 +960,13 @@ public enum SleepStager { wristOff: StreamFingerprint.of(wristOff, ts: { $0.start }, quant: { $0.end }), band: StreamFingerprint.of(bandSleepState, ts: { $0.ts }, quant: { $0.state }), v2: useSleepStagerV2, - hrFirst: hrFirstSleep, + bridge: wakeBridge, sleepHRBaseline: sleepHRBaseline) return detectSleepCache.value(key) { detectSleepUncached(hr: hr, rr: rr, resp: resp, gravity: gravity, tzOffsetSeconds: tzOffsetSeconds, wristOff: wristOff, bandSleepState: bandSleepState, useSleepStagerV2: useSleepStagerV2, - hrFirstSleep: hrFirstSleep, + wakeBridge: wakeBridge, sleepHRBaseline: sleepHRBaseline, traceSink: nil) } } @@ -998,7 +977,7 @@ public enum SleepStager { let tz: Int let wristOff: StreamFingerprint; let band: StreamFingerprint let v2: Bool - let hrFirst: Bool + let bridge: Bool let sleepHRBaseline: Double? } /// ≈ the number of distinct days in a scoring window; FIFO-evicted, holds only small session arrays. @@ -1013,7 +992,7 @@ public enum SleepStager { wristOff: [(start: Int, end: Int)], bandSleepState: [(ts: Int, state: Int)], useSleepStagerV2: Bool, - hrFirstSleep: Bool = false, + wakeBridge: Bool = false, sleepHRBaseline: Double? = nil, traceSink: ((String) -> Void)? = nil) -> [SleepSession] { let grav = gravity.sorted { $0.ts < $1.ts } @@ -1065,15 +1044,12 @@ public enum SleepStager { } } - // HR-first pass (experimental, default OFF — see the section comment above `wakeBridgeMaxMin`): - // weight HR over motion where the two disagree (an active run whose median HR is in the sleep - // band is rescued as sleep), then assemble one candidate run per sleep period across short + // Wake-gap bridge (experimental, default OFF — see the section comment above + // `wakeBridgeMaxMin`): assemble one candidate run per sleep period across short // (≤ wakeBridgeMaxMin) wake gaps, BEFORE the ladder — so sub-minSleepMin fragments between // arousals survive as part of the night instead of being dropped, and a brief get-up stays // INSIDE one session as staged wake. Every gate below still applies to the assembled runs. - if hrFirstSleep { - runs = hrRescueActiveRuns(runs, hr: hrS, baseline: baseline, - sleepHRBaseline: sleepHRBaseline, traceSink: traceSink) + if wakeBridge { runs = bridgeWakeGaps(runs, traceSink: traceSink) } diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerHrFirstTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerHrFirstTests.swift deleted file mode 100644 index be4b4d4aba..0000000000 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerHrFirstTests.swift +++ /dev/null @@ -1,199 +0,0 @@ -import XCTest -@testable import StrandAnalytics -import WhoopProtocol - -/// HR-first sleep detection (experimental, default OFF): weight HR over motion when the two -/// disagree, and assemble one session per sleep period across short wake gaps. -/// -/// Per the "validate against varying inputs" rule (#194/#345), these tests inject the sleep window -/// at DIFFERENT positions and lengths and assert the rescue TRACKS it — and that the identical -/// motion pattern with an elevated (awake) HR is refused — rather than pinning one matched night. -/// The Kotlin twin (`SleepStagerHrFirstTest`) runs the same scenarios with the same numbers. -final class SleepStagerHrFirstTests: XCTestCase { - - // MARK: - Stream builders (1 Hz, mirroring SleepStagerTests) - - private func stillGravity(start: Int, durationS: Int) -> [GravitySample] { - (0.. [GravitySample] { - (0.. GravitySample in - let phase = Double(i % 2) * 0.5 - return GravitySample(ts: start + i, x: phase, y: 0, z: 1.0) - } - } - - private func hrStream(start: Int, durationS: Int, bpm: Int) -> [HRSample] { - (0.. Int { - 1_749_513_600 + hourUTC * 3_600 + min * 60 - } - - // MARK: - Restless-night rescue (the DROPPED-night failure) - - /// A night that is restless (motion says "active") but cardiac-unambiguous (HR 52 vs awake 75) - /// from 22:00 to 04:30, then still to 06:00. OFF: only the quiescent morning fragment scores — - /// the real-world symptom. ON: the whole night is one session. - func testRestlessNightRescuedToFullSession() { - let grav = restlessGravity(start: at(22), durationS: 6 * 3_600 + 1_800) // 22:00–04:30 - + stillGravity(start: at(28, min: 30), durationS: 90 * 60) // 04:30–06:00 - let hr = hrStream(start: at(20), durationS: 2 * 3_600, bpm: 75) // 20:00–22:00 awake - + hrStream(start: at(22), durationS: 8 * 3_600, bpm: 52) // 22:00–06:00 asleep - + hrStream(start: at(30), durationS: 2 * 3_600, bpm: 75) // 06:00–08:00 awake - - let off = SleepStager.detectSleep(hr: hr, gravity: grav) - XCTAssertEqual(off.count, 1, "OFF: only the still morning fragment should score") - XCTAssertGreaterThanOrEqual(off[0].start, at(28, min: 25), "OFF fragment starts near 04:30") - XCTAssertLessThanOrEqual(off[0].start, at(28, min: 45)) - - let on = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: true) - XCTAssertEqual(on.count, 1, "ON: one assembled session for the whole night") - XCTAssertLessThanOrEqual(on[0].start, at(22) + 120, "ON: onset tracks the injected 22:00 sleep start") - XCTAssertGreaterThanOrEqual(on[0].end, at(30) - 120, "ON: end tracks the injected 06:00 wake") - } - - /// Same rescue with the sleep window injected at a DIFFERENT position and length (01:00–07:30, - /// all restless, no still stretch at all): OFF finds nothing; ON tracks the injected bounds. - func testRestlessRescueTracksInjectedWindow() { - let grav = restlessGravity(start: at(25), durationS: 6 * 3_600 + 1_800) // 01:00–07:30 - let hr = hrStream(start: at(22), durationS: 3 * 3_600, bpm: 75) // 22:00–01:00 awake - + hrStream(start: at(25), durationS: 6 * 3_600 + 1_800, bpm: 52) // 01:00–07:30 asleep - + hrStream(start: at(31, min: 30), durationS: 2 * 3_600, bpm: 75) // 07:30–09:30 awake - - XCTAssertEqual(SleepStager.detectSleep(hr: hr, gravity: grav).count, 0, - "OFF: an all-restless night scores nothing") - - let on = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: true) - XCTAssertEqual(on.count, 1) - XCTAssertLessThanOrEqual(abs(on[0].start - at(25)), 120, "onset tracks the moved window") - XCTAssertLessThanOrEqual(abs(on[0].end - at(31, min: 30)), 120, "end tracks the moved window") - } - - /// The IDENTICAL motion pattern as the full-night rescue, but with the night HR elevated (78 vs - /// a 62 day median — awake-in-bed restlessness): the rescue must refuse the restless stretch. - /// (The STILL morning fragment is still accepted — by the pre-existing motion-corroborated - /// quiescent band (#462), which deliberately tolerates elevated-but-flat HR on a motionless - /// run — so the assertion is that HR-first adds NOTHING: same single fragment, not a full - /// night, with an onset still at the quiescent stretch.) - func testElevatedHrNightIsNotRescued() { - let grav = restlessGravity(start: at(22), durationS: 6 * 3_600 + 1_800) // 22:00–04:30 - + stillGravity(start: at(28, min: 30), durationS: 90 * 60) // 04:30–06:00 - let hr = hrStream(start: at(22), durationS: 8 * 3_600, bpm: 78) // 22:00–06:00 elevated - + hrStream(start: at(30), durationS: 10 * 3_600, bpm: 62) // 06:00–16:00 day median - - let off = SleepStager.detectSleep(hr: hr, gravity: grav) - let on = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: true) - XCTAssertEqual(on, off, "same motion, elevated HR: HR-first must change nothing") - XCTAssertEqual(on.count, 1) - XCTAssertGreaterThanOrEqual(on[0].start, at(28, min: 25), - "the restless 22:00–04:30 stretch must NOT be rescued") - } - - // MARK: - Wake-gap bridging (the SPLIT-night failure) - - /// A 16-min get-up (real movement + HR 80) mid-night. The centered rolling window smears it to a - /// >mergeMin active run, so OFF stores TWO sessions. ON bridges them into ONE session. - func testBriefGetUpBridgedIntoOneSession() { - let grav = stillGravity(start: at(23), durationS: 4 * 3_600) // 23:00–03:00 - + restlessGravity(start: at(27), durationS: 16 * 60) // 03:00–03:16 get-up - + stillGravity(start: at(27, min: 16), durationS: 3 * 3_600 + 44 * 60) // 03:16–07:00 - let hr = hrStream(start: at(21), durationS: 2 * 3_600, bpm: 75) // 21:00–23:00 awake - + hrStream(start: at(23), durationS: 4 * 3_600, bpm: 52) // 23:00–03:00 asleep - + hrStream(start: at(27), durationS: 16 * 60, bpm: 80) // get-up spike - + hrStream(start: at(27, min: 16), durationS: 3 * 3_600 + 44 * 60, bpm: 52) // back asleep - + hrStream(start: at(31), durationS: 2 * 3_600, bpm: 75) // 07:00–09:00 awake - - let off = SleepStager.detectSleep(hr: hr, gravity: grav) - XCTAssertEqual(off.count, 2, "OFF: the smeared get-up splits the night") - - let on = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: true) - XCTAssertEqual(on.count, 1, "ON: one session bridging the get-up") - XCTAssertLessThanOrEqual(abs(on[0].start - at(23)), 120) - XCTAssertLessThanOrEqual(abs(on[0].end - at(31)), 120) - } - - /// An 80-min genuinely-awake block (elevated HR, real movement) exceeds the 45-min bridge: - /// the night stays two sessions even with HR-first ON. - func testLongWakeGapStaysSplit() { - let grav = stillGravity(start: at(22), durationS: 3 * 3_600) // 22:00–01:00 - + restlessGravity(start: at(25), durationS: 80 * 60) // 01:00–02:20 awake - + stillGravity(start: at(26, min: 20), durationS: 3 * 3_600 + 40 * 60) // 02:20–06:00 - let hr = hrStream(start: at(20), durationS: 2 * 3_600, bpm: 75) - + hrStream(start: at(22), durationS: 3 * 3_600, bpm: 52) - + hrStream(start: at(25), durationS: 80 * 60, bpm: 78) - + hrStream(start: at(26, min: 20), durationS: 3 * 3_600 + 40 * 60, bpm: 52) - + hrStream(start: at(30), durationS: 2 * 3_600, bpm: 75) - - let on = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: true) - XCTAssertEqual(on.count, 2, "an over-threshold wake gap must not be bridged") - } - - // MARK: - Guards still hold on assembled runs - - /// A rescued + bridged DAYTIME stretch (in-band flat HR 60, no cardiac dip) must still be - /// dropped by the daytime false-sleep guard: the rescue may not create phantom daytime naps. - func testDaytimeGuardStillAppliesToRescuedRuns() { - let grav = stillGravity(start: at(11), durationS: 2 * 3_600) // 11:00–13:00 - + restlessGravity(start: at(13), durationS: 2 * 3_600) // 13:00–15:00 - + stillGravity(start: at(15), durationS: 2 * 3_600) // 15:00–17:00 - let hr = hrStream(start: at(10), durationS: 8 * 3_600, bpm: 60) // flat, no dip - - let on = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: true) - XCTAssertEqual(on.count, 0, "no resting-HR dip: the daytime guard must reject the rescue") - } - - /// Default-off byte-identity: omitting the flag and passing `false` produce the identical result. - func testDefaultOffIsByteIdentical() { - let grav = stillGravity(start: at(23), durationS: 4 * 3_600) - + restlessGravity(start: at(27), durationS: 16 * 60) - + stillGravity(start: at(27, min: 16), durationS: 3 * 3_600 + 44 * 60) - let hr = hrStream(start: at(23), durationS: 8 * 3_600, bpm: 52) - + hrStream(start: at(31), durationS: 4 * 3_600, bpm: 75) - - let implicitDefault = SleepStager.detectSleep(hr: hr, gravity: grav) - let explicitFalse = SleepStager.detectSleep(hr: hr, gravity: grav, hrFirstSleep: false) - XCTAssertEqual(implicitDefault, explicitFalse) - } - - // MARK: - Pure helpers - - func testBridgeWakeGapsMergesOnlyUnderThreshold() { - let a = SleepStager.Period(stage: "sleep", start: 0, end: 3_600) - let gap = SleepStager.Period(stage: "active", start: 3_600, end: 3_600 + 40 * 60) - let b = SleepStager.Period(stage: "sleep", start: 3_600 + 40 * 60, end: 3 * 3_600) - let merged = SleepStager.bridgeWakeGaps([a, gap, b]) - XCTAssertEqual(merged.count, 1) - XCTAssertEqual(merged[0].start, 0) - XCTAssertEqual(merged[0].end, 3 * 3_600) - - let farGap = SleepStager.Period(stage: "active", start: 3_600, end: 3_600 + 46 * 60) - let c = SleepStager.Period(stage: "sleep", start: 3_600 + 46 * 60, end: 4 * 3_600) - let unmerged = SleepStager.bridgeWakeGaps([a, farGap, c]) - XCTAssertEqual(unmerged.count, 3, "a 46-min gap is over wakeBridgeMaxMin (45) and must not merge") - } - - func testHrRescueRespectsBandAndSampleFloor() { - let p = SleepStager.Period(stage: "active", start: 0, end: 3_600) - let inBand = (0..<3_600).map { HRSample(ts: $0, bpm: 52) } - let rescued = SleepStager.hrRescueActiveRuns([p], hr: inBand, baseline: 52.0) - XCTAssertEqual(rescued[0].stage, "sleep") - - let outOfBand = (0..<3_600).map { HRSample(ts: $0, bpm: 60) } - let kept = SleepStager.hrRescueActiveRuns([p], hr: outOfBand, baseline: 52.0) - XCTAssertEqual(kept[0].stage, "active", "median above baseline × 1.05 must not rescue") - - // Below the hrRefineMinSamples floor: too little cardiac evidence to overrule motion. - let sparse = (0..<10).map { HRSample(ts: $0 * 60, bpm: 52) } - let untouched = SleepStager.hrRescueActiveRuns([p], hr: sparse, baseline: 52.0) - XCTAssertEqual(untouched[0].stage, "active") - - // No baseline at all: nothing to weigh HR against. - let noBase = SleepStager.hrRescueActiveRuns([p], hr: inBand, baseline: nil) - XCTAssertEqual(noBase[0].stage, "active") - } -} diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerWakeBridgeTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerWakeBridgeTests.swift new file mode 100644 index 0000000000..cd4798d6c0 --- /dev/null +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerWakeBridgeTests.swift @@ -0,0 +1,194 @@ +import XCTest +@testable import StrandAnalytics +import WhoopProtocol + +/// Wake-gap bridging (experimental, default OFF): assemble adjacent sleep runs across ≤ 45-min wake +/// gaps into ONE candidate run, so a brief get-up stays inside one session as staged wake. +/// +/// Includes a REAL-DATA pin: the exact seam layout of a reporting wearer's stored fragmented nights +/// (Aug 2026 mirror), asserting precisely which seams the bridge merges and which it leaves split. +/// An HR-led rescue of restless-but-asleep nights was prototyped alongside this bridge and WITHDRAWN +/// after a real-data replay — see the history note above `SleepStager.wakeBridgeMaxMin`; the +/// restless-night test below pins that the bridge deliberately does NOT fix that failure. +/// The Kotlin twin (`SleepStagerWakeBridgeTest`) runs the same scenarios with the same numbers. +final class SleepStagerWakeBridgeTests: XCTestCase { + + // MARK: - Stream builders (1 Hz, mirroring SleepStagerTests) + + private func stillGravity(start: Int, durationS: Int) -> [GravitySample] { + (0.. [GravitySample] { + (0.. GravitySample in + let phase = Double(i % 2) * 0.5 + return GravitySample(ts: start + i, x: phase, y: 0, z: 1.0) + } + } + + private func hrStream(start: Int, durationS: Int, bpm: Int) -> [HRSample] { + (0.. Int { + 1_749_513_600 + hourUTC * 3_600 + min * 60 + } + + // MARK: - Wake-gap bridging (the SPLIT-night failure) + + /// A 16-min get-up (real movement + HR 80) mid-night. The centered rolling window smears it to a + /// >mergeMin active run, so OFF stores TWO sessions. ON bridges them into ONE session. + func testBriefGetUpBridgedIntoOneSession() { + let grav = stillGravity(start: at(23), durationS: 4 * 3_600) // 23:00–03:00 + + restlessGravity(start: at(27), durationS: 16 * 60) // 03:00–03:16 get-up + + stillGravity(start: at(27, min: 16), durationS: 3 * 3_600 + 44 * 60) // 03:16–07:00 + let hr = hrStream(start: at(21), durationS: 2 * 3_600, bpm: 75) // 21:00–23:00 awake + + hrStream(start: at(23), durationS: 4 * 3_600, bpm: 52) // 23:00–03:00 asleep + + hrStream(start: at(27), durationS: 16 * 60, bpm: 80) // get-up spike + + hrStream(start: at(27, min: 16), durationS: 3 * 3_600 + 44 * 60, bpm: 52) // back asleep + + hrStream(start: at(31), durationS: 2 * 3_600, bpm: 75) // 07:00–09:00 awake + + let off = SleepStager.detectSleep(hr: hr, gravity: grav) + XCTAssertEqual(off.count, 2, "OFF: the smeared get-up splits the night") + + let on = SleepStager.detectSleep(hr: hr, gravity: grav, wakeBridge: true) + XCTAssertEqual(on.count, 1, "ON: one session bridging the get-up") + XCTAssertLessThanOrEqual(abs(on[0].start - at(23)), 120) + XCTAssertLessThanOrEqual(abs(on[0].end - at(31)), 120) + } + + /// An 80-min genuinely-awake block (elevated HR, real movement) exceeds the 45-min bridge: + /// the night stays two sessions even with the bridge ON. + func testLongWakeGapStaysSplit() { + let grav = stillGravity(start: at(22), durationS: 3 * 3_600) // 22:00–01:00 + + restlessGravity(start: at(25), durationS: 80 * 60) // 01:00–02:20 awake + + stillGravity(start: at(26, min: 20), durationS: 3 * 3_600 + 40 * 60) // 02:20–06:00 + let hr = hrStream(start: at(20), durationS: 2 * 3_600, bpm: 75) + + hrStream(start: at(22), durationS: 3 * 3_600, bpm: 52) + + hrStream(start: at(25), durationS: 80 * 60, bpm: 78) + + hrStream(start: at(26, min: 20), durationS: 3 * 3_600 + 40 * 60, bpm: 52) + + hrStream(start: at(30), durationS: 2 * 3_600, bpm: 75) + + let on = SleepStager.detectSleep(hr: hr, gravity: grav, wakeBridge: true) + XCTAssertEqual(on.count, 2, "an over-threshold wake gap must not be bridged") + } + + /// The withdrawn-rescue pin: a restless night (motion "active", HR at clear sleep levels) is NOT + /// rescued by the bridge — ON must change nothing versus OFF. The bridge only assembles runs the + /// unchanged stillness spine already accepted; the restless-night DROPPING failure is documented + /// open (see the history note above `wakeBridgeMaxMin`). + func testRestlessNightIsNotRescuedByBridge() { + let grav = restlessGravity(start: at(22), durationS: 6 * 3_600 + 1_800) // 22:00–04:30 + + stillGravity(start: at(28, min: 30), durationS: 90 * 60) // 04:30–06:00 + let hr = hrStream(start: at(20), durationS: 2 * 3_600, bpm: 75) + + hrStream(start: at(22), durationS: 8 * 3_600, bpm: 52) + + hrStream(start: at(30), durationS: 2 * 3_600, bpm: 75) + + let off = SleepStager.detectSleep(hr: hr, gravity: grav) + let on = SleepStager.detectSleep(hr: hr, gravity: grav, wakeBridge: true) + XCTAssertEqual(on, off, "the bridge must not reclassify restless stretches") + XCTAssertEqual(on.count, 1, "only the still morning fragment scores either way") + XCTAssertGreaterThanOrEqual(on[0].start, at(28, min: 25)) + } + + // MARK: - Guards still hold on assembled runs + + /// Two still DAYTIME stretches bridged across a 30-min gap (flat in-band HR, no cardiac dip) + /// must still be dropped by the daytime false-sleep guard on the ASSEMBLED window. + func testDaytimeGuardStillAppliesToBridgedRuns() { + let grav = stillGravity(start: at(12), durationS: 2 * 3_600) // 12:00–14:00 + + stillGravity(start: at(14, min: 30), durationS: 2 * 3_600) // 14:30–16:30 + let hr = hrStream(start: at(10), durationS: 8 * 3_600, bpm: 60) // flat, no dip + + let on = SleepStager.detectSleep(hr: hr, gravity: grav, wakeBridge: true) + XCTAssertEqual(on.count, 0, "no resting-HR dip: the daytime guard must reject the bridged window") + } + + /// Default-off byte-identity: omitting the flag and passing `false` produce the identical result. + func testDefaultOffIsByteIdentical() { + let grav = stillGravity(start: at(23), durationS: 4 * 3_600) + + restlessGravity(start: at(27), durationS: 16 * 60) + + stillGravity(start: at(27, min: 16), durationS: 3 * 3_600 + 44 * 60) + let hr = hrStream(start: at(23), durationS: 8 * 3_600, bpm: 52) + + hrStream(start: at(31), durationS: 4 * 3_600, bpm: 75) + + let implicitDefault = SleepStager.detectSleep(hr: hr, gravity: grav) + let explicitFalse = SleepStager.detectSleep(hr: hr, gravity: grav, wakeBridge: false) + XCTAssertEqual(implicitDefault, explicitFalse) + } + + // MARK: - Pure helper + + func testBridgeWakeGapsMergesOnlyUnderThreshold() { + let a = SleepStager.Period(stage: "sleep", start: 0, end: 3_600) + let gap = SleepStager.Period(stage: "active", start: 3_600, end: 3_600 + 40 * 60) + let b = SleepStager.Period(stage: "sleep", start: 3_600 + 40 * 60, end: 3 * 3_600) + let merged = SleepStager.bridgeWakeGaps([a, gap, b]) + XCTAssertEqual(merged.count, 1) + XCTAssertEqual(merged[0].start, 0) + XCTAssertEqual(merged[0].end, 3 * 3_600) + + let farGap = SleepStager.Period(stage: "active", start: 3_600, end: 3_600 + 46 * 60) + let c = SleepStager.Period(stage: "sleep", start: 3_600 + 46 * 60, end: 4 * 3_600) + let unmerged = SleepStager.bridgeWakeGaps([a, farGap, c]) + XCTAssertEqual(unmerged.count, 3, "a 46-min gap is over wakeBridgeMaxMin (45) and must not merge") + } + + /// H4 span guard: a merge whose ASSEMBLED span would exceed maxMainSleepSpanS (16 h) is refused, + /// so a pathological chain can never assemble and then be dropped whole by the ladder's cap — + /// the exact failure the withdrawn HR rescue exhibited on real data. + func testBridgeRefusesOverSpanCapMerge() { + let a = SleepStager.Period(stage: "sleep", start: 0, end: 8 * 3_600) + let b = SleepStager.Period(stage: "sleep", start: 8 * 3_600 + 30 * 60, end: 17 * 3_600) + let out = SleepStager.bridgeWakeGaps([a, b]) + XCTAssertEqual(out.count, 2, "a merge assembling a >16 h span must be refused") + } + + // MARK: - Real-data seam pin (Aug 2026 mirror, reporting wearer's stored fragments) + + /// The stored fragment layouts of five REAL split nights, as (start, end) unix pairs from the + /// wearer's server mirror. Asserts exactly which seams the 45-min bridge merges (16/12/28/25+19/ + /// 17-min arousal seams) and which it leaves split (2.2–3.9 h genuine wake gaps, and a 90-min + /// morning re-sleep gap — over the threshold, left to the selector's #861 night-tail bridge). + func testRealNightSeamsBridgeExactly() { + func sleeps(_ spans: [(Int, Int)]) -> [SleepStager.Period] { + spans.map { SleepStager.Period(stage: "sleep", start: $0.0, end: $0.1) } + } + // Aug 11→12 ET: 4 fragments; only the 16-min bathroom seam (06:17→06:33 ET) merges → 3 blocks. + let aug12 = sleeps([(1786491049, 1786496628), (1786510824, 1786514588), + (1786522512, 1786529853), (1786530794, 1786541952)]) + let aug12Out = SleepStager.bridgeWakeGaps(aug12) + XCTAssertEqual(aug12Out.map { [$0.start, $0.end] }, + [[1786491049, 1786496628], [1786510824, 1786514588], [1786522512, 1786541952]]) + + // Aug 17→18 ET: 3 fragments; the 12-min seam merges, the 3.8 h gap stays → 2 blocks. + let aug18 = sleeps([(1787032890, 1787037056), (1787050829, 1787054649), + (1787055394, 1787063445)]) + XCTAssertEqual(SleepStager.bridgeWakeGaps(aug18).map { [$0.start, $0.end] }, + [[1787032890, 1787037056], [1787050829, 1787063445]]) + + // Aug 18→19 ET: 2 fragments at a 28-min seam → 1 block. + let aug19 = sleeps([(1787126248, 1787132602), (1787134284, 1787141425)]) + XCTAssertEqual(SleepStager.bridgeWakeGaps(aug19).map { [$0.start, $0.end] }, + [[1787126248, 1787141425]]) + + // Aug 20→21 ET: 3 fragments at 25-min and 19-min seams → 1 block (11.4 h < the 16 h cap). + // NOTE: the mirror's correction pass (an HR-evidence-led edit session, not a human label) + // kept the third block (a 10:44–12:03 post-wake morning doze) as a SEPARATE doze; the + // 45-min rule cannot distinguish "brief wake, more sleep" from "woke, dozed again", so the + // bridge deliberately folds it in (wake staged inside) and a wearer who disagrees hand-edits + // that one seam. Pinned as-is, eyes open. + let aug21 = sleeps([(1787287220, 1787297404), (1787298936, 1787322337), + (1787323440, 1787328166)]) + XCTAssertEqual(SleepStager.bridgeWakeGaps(aug21).map { [$0.start, $0.end] }, + [[1787287220, 1787328166]]) + + // Aug 23→24 ET: 2 fragments at a 90-min gap → stays split (over the 45-min bridge). The + // mirror's correction pass filled this gap as SLEEP on HR evidence — it is the + // withdrawn-rescue (restless dropping) class, not a seam; the bridge must not guess it. + let aug24 = sleeps([(1787546768, 1787551261), (1787556701, 1787575851)]) + XCTAssertEqual(SleepStager.bridgeWakeGaps(aug24).count, 2) + } +} diff --git a/Strand/BLE/PuffinExperiment.swift b/Strand/BLE/PuffinExperiment.swift index 532acf26f2..c2b9f3ada1 100644 --- a/Strand/BLE/PuffinExperiment.swift +++ b/Strand/BLE/PuffinExperiment.swift @@ -201,22 +201,21 @@ enum PuffinExperiment { static var motionAwareWakeEnabled: Bool { UserDefaults.standard.bool(forKey: motionAwareWakeKey) } - /// Opt-in "HR-first sleep detection" (default OFF): weight the cardiac signal over wrist motion when - /// the two disagree, inside `SleepStager.detectSleep`. Two effects, both BEFORE the gate ladder: - /// a motion-"active" run whose median HR sits inside the same sleep band the ladder already trusts is - /// rescued as sleep (a restless-but-asleep night — periodic limb movement, an injured sleeper, or a - /// strap whose motion channel over-reports — is no longer collapsed to its quiescent morning - /// fragment), and adjacent sleep runs across a ≤ 45-min wake gap are assembled into ONE session with - /// the wake staged inside it (a brief get-up no longer splits the night into separate stored - /// sessions). Every existing guard (min duration, span cap, HR confirmation, off-wrist, daytime nap - /// bars) still applies to the assembled runs. Default OFF per the varying-input validation - /// discipline (#194/#345): an awake-in-bed stretch whose HR genuinely sits in the sleep band can be - /// scored asleep, the same limit the still-sedentary path already has. Read at the scoring call site + /// Opt-in "Bridge brief wakes" (default OFF): inside `SleepStager.detectSleep`, adjacent sleep runs + /// across a ≤ 45-min wake gap are assembled into ONE session with the wake staged inside it, so a + /// brief get-up (a bathroom trip smeared past the run-merge threshold by the centered rolling + /// stillness window) no longer splits the night into separate stored sessions, and sub-1h sleep + /// fragments between arousals are no longer dropped outright. Every existing guard (min duration, + /// the 16-h span cap — which the bridge also refuses to assemble past — HR confirmation, off-wrist, + /// daytime nap bars) still applies to the assembled run. Default OFF per the varying-input + /// validation discipline (#194/#345). NOTE: an HR-led rescue of restless-but-asleep nights was + /// prototyped behind this flag and WITHDRAWN after a real-data replay — see the history note above + /// `SleepStager.wakeBridgeMaxMin` before re-attempting it. Read at the scoring call site /// (IntelligenceEngine) and threaded through `AnalyticsEngine.analyzeDay`. Mirrors the Android - /// `PuffinExperiment.KEY_HR_FIRST_SLEEP`. - static let hrFirstSleepKey = "noopHrFirstSleep" + /// `PuffinExperiment.KEY_SLEEP_WAKE_BRIDGE`. + static let wakeBridgeKey = "noopSleepWakeBridge" - static var hrFirstSleepEnabled: Bool { UserDefaults.standard.bool(forKey: hrFirstSleepKey) } + static var wakeBridgeEnabled: Bool { UserDefaults.standard.bool(forKey: wakeBridgeKey) } /// Opt-in "WHOOP MG ECG (experimental)": unlock the gated, user-initiated ECG ("Labrador") probe that /// asks an MG strap to start its ECG subsystem. Default OFF, and OFF is not merely the default — with diff --git a/Strand/Data/IntelligenceEngine.swift b/Strand/Data/IntelligenceEngine.swift index eca3abb385..88dbaf927d 100644 --- a/Strand/Data/IntelligenceEngine.swift +++ b/Strand/Data/IntelligenceEngine.swift @@ -702,10 +702,10 @@ final class IntelligenceEngine: ObservableObject { // ever runs AFTER whichever stager above just ran, and self-gates on the night's observed // gravity + step density, so flipping it on is a no-op for any night too sparse to trust. let useMotionAwareWake = PuffinExperiment.motionAwareWakeEnabled - // HR-first sleep detection (experimental, default OFF): read once, off the detached - // executor, like the two flags above. Weights HR over motion in detection and bridges - // short wake gaps into one session — see `PuffinExperiment.hrFirstSleepEnabled`. - let useHrFirstSleep = PuffinExperiment.hrFirstSleepEnabled + // Wake-gap bridging (experimental, default OFF): read once, off the detached + // executor, like the two flags above. Assembles a night split by a brief get-up into + // one session with the wake staged inside — see `PuffinExperiment.wakeBridgeEnabled`. + let useWakeBridge = PuffinExperiment.wakeBridgeEnabled // Already OFF the main actor , score directly (the prior nested `Task.detached` here only // existed to hop off the main actor; the whole loop now runs off it, so the score is computed @@ -737,8 +737,8 @@ final class IntelligenceEngine: ObservableObject { // #364 follow-up: same threading for the motion-aware wake // refinement post-pass. useMotionAwareWake: useMotionAwareWake, - // HR-first sleep detection (experimental, default OFF). - useHrFirstSleep: useHrFirstSleep, + // Wake-gap bridging (experimental, default OFF). + useWakeBridge: useWakeBridge, traceSink: traceSink, hrvTraceSink: hrvTraceSink, // Per-window HRV detail ONLY for the most-recent night diff --git a/Strand/Screens/SettingsView.swift b/Strand/Screens/SettingsView.swift index 9c6bcce341..32e39f95f4 100644 --- a/Strand/Screens/SettingsView.swift +++ b/Strand/Screens/SettingsView.swift @@ -90,7 +90,7 @@ struct SettingsView: View { /// minutes. Self-gates on OBSERVED gravity + step density (#345) — a no-op on a sparse night (e.g. /// WHOOP 4.0) regardless of this switch. See [PuffinExperiment.motionAwareWakeKey]. @AppStorage(PuffinExperiment.motionAwareWakeKey) private var motionAwareWakeEnabled = false - @AppStorage(PuffinExperiment.hrFirstSleepKey) private var hrFirstSleepEnabled = false + @AppStorage(PuffinExperiment.wakeBridgeKey) private var wakeBridgeEnabled = false // Imperial/Metric display preference (D#103). Stored data is always SI; this only changes how // distances/weights/heights/temperatures are SHOWN — and lets the profile fields below take @@ -1435,15 +1435,15 @@ struct SettingsView: View { Divider().overlay(StrandPalette.hairline) - // MARK: HR-first sleep detection — default OFF. - Toggle(isOn: $hrFirstSleepEnabled) { - Text("HR-first sleep detection") + // MARK: Bridge brief wakes — default OFF. + Toggle(isOn: $wakeBridgeEnabled) { + Text("Bridge brief wakes") .font(StrandFont.subhead) .foregroundStyle(StrandPalette.textPrimary) } .toggleStyle(.switch) .tint(StrandPalette.accent) - Text("Trusts your heart rate over wrist movement when the two disagree about sleep. A restless night whose heart rate clearly sits at your sleep level is kept as sleep instead of being cut down to its calmest fragment, and a brief get-up (under ~45 minutes) stays inside one sleep with the wake scored inside it, rather than splitting the night into separate sleeps. Worth trying if your nights show up fragmented or mostly missing despite a normal sleeping heart rate. Trade-off: a long, deeply relaxed stretch awake in bed can be scored as sleep. Off by default; takes effect on the next nights scored.") + Text("Keeps a night interrupted by a brief get-up (under ~45 minutes) as one sleep, with the wake scored inside it, instead of splitting it into separate sleeps — and short sleep stretches between such wakes are no longer discarded for being under an hour on their own. Worth trying if your nights show up split into two or three pieces around bathroom trips or brief stirs. Off by default; takes effect on the next nights scored.") .font(StrandFont.caption) .foregroundStyle(StrandPalette.textTertiary) .fixedSize(horizontal: false, vertical: true) diff --git a/android/app/src/main/java/com/noop/analytics/AnalyticsEngine.kt b/android/app/src/main/java/com/noop/analytics/AnalyticsEngine.kt index 64f07a41c2..ac48302a6e 100644 --- a/android/app/src/main/java/com/noop/analytics/AnalyticsEngine.kt +++ b/android/app/src/main/java/com/noop/analytics/AnalyticsEngine.kt @@ -241,13 +241,12 @@ object AnalyticsEngine { // flag. Default false keeps every pure-function caller/test byte-identical; IntelligenceEngine // threads PuffinExperiment.from(context).motionAwareWake. Mirrors Swift. useMotionAwareWake: Boolean = false, - // HR-first sleep detection (experimental, default OFF — see the section comment above - // [SleepStager.wakeBridgeMaxMin]). When true, detection weights HR over motion where the two - // disagree (an "active" run whose median HR sits in the sleep band is rescued as sleep) and - // assembles one session per sleep period across ≤ 45-min wake gaps. Default false keeps every - // pure-function caller/test byte-identical; IntelligenceEngine threads - // PuffinExperiment.from(context).hrFirstSleep. Mirrors Swift. - useHrFirstSleep: Boolean = false, + // Wake-gap bridging (experimental, default OFF — see the section comment above + // [SleepStager.wakeBridgeMaxMin]). When true, detection assembles one session per sleep + // period across ≤ 45-min wake gaps, so a brief get-up stays inside one session as staged + // wake. Default false keeps every pure-function caller/test byte-identical; + // IntelligenceEngine threads PuffinExperiment.from(context).sleepWakeBridge. Mirrors Swift. + useWakeBridge: Boolean = false, // Sleep & Rest test-mode trace sink (E11). null = byte-identical default. When non-null the gate // trace from detectSleep and the Rest sub-score line are forwarded line-by-line. Mirrors Swift. traceSink: ((String) -> Unit)? = null, @@ -271,7 +270,7 @@ object AnalyticsEngine { hr = hr, rr = rr, resp = resp, gravity = gravity, tzOffsetSeconds = tzOffsetSeconds, wristOff = wristOff, bandSleepState = bandSleepState, useSleepStagerV2 = useSleepStagerV2, - hrFirstSleep = useHrFirstSleep, + wakeBridge = useWakeBridge, traceSink = traceSink, ) // Motion-aware wake refinement (#364 follow-up) runs AFTER V1/V2 staging, over every detected diff --git a/android/app/src/main/java/com/noop/analytics/IntelligenceEngine.kt b/android/app/src/main/java/com/noop/analytics/IntelligenceEngine.kt index 1977ffee55..0e166976fe 100644 --- a/android/app/src/main/java/com/noop/analytics/IntelligenceEngine.kt +++ b/android/app/src/main/java/com/noop/analytics/IntelligenceEngine.kt @@ -184,10 +184,10 @@ object IntelligenceEngine { // OWN observed gravity + step density (see [WakeMotionRefinement]), so existing callers / tests // and any night too sparse to trust (e.g. a WHOOP 4.0) are unaffected either way. useMotionAwareWake: Boolean = false, - // HR-first sleep detection (experimental, default OFF), threaded the same Context-free way: the - // Context-aware caller reads PuffinExperiment.from(context).hrFirstSleep and passes it down to - // AnalyticsEngine.analyzeDay. See [SleepStager.wakeBridgeMaxMin]'s section comment. Mirrors Swift. - useHrFirstSleep: Boolean = false, + // Wake-gap bridging (experimental, default OFF), threaded the same Context-free way: the + // Context-aware caller reads PuffinExperiment.from(context).sleepWakeBridge and passes it down + // to AnalyticsEngine.analyzeDay. See [SleepStager.wakeBridgeMaxMin]'s section comment. Mirrors Swift. + useWakeBridge: Boolean = false, // Sleep & Rest test-mode trace sink (Test Centre E5). The analytics layer is Context-free, so the // Context-aware caller (AppViewModel / WhoopBleClient) reads TestCentre.active(SLEEP) and passes a // non-null sink ONLY when the mode is on, routing each line to the .sleep-tagged strap log. null (the @@ -235,7 +235,7 @@ object IntelligenceEngine { analyzeGate.withLock { val (out, healed) = analyzeRecentOnCpu(repo, profile, maxDays, importedDeviceId, maxHROverride, nowSeconds, ownerSource, manualStepCoefficient, persistStepsCalibration, baselineEpoch, - recoveryEpoch, diag, useExperimentalSleepV2, useMotionAwareWake, useHrFirstSleep, + recoveryEpoch, diag, useExperimentalSleepV2, useMotionAwareWake, useWakeBridge, sleepTraceSink, recoveryTraceSink, stepsTraceSink, universalSink, workoutsTraceSink, hrvTraceSink, deepHrvWindow) if (healed == 0) out @@ -246,7 +246,7 @@ object IntelligenceEngine { // are gone), so this can never loop. Mirrors the Swift pendingForcedRescore re-arm. else analyzeRecentOnCpu(repo, profile, maxDays, importedDeviceId, maxHROverride, nowSeconds, ownerSource, manualStepCoefficient, persistStepsCalibration, baselineEpoch, - recoveryEpoch, diag, useExperimentalSleepV2, useMotionAwareWake, useHrFirstSleep, + recoveryEpoch, diag, useExperimentalSleepV2, useMotionAwareWake, useWakeBridge, sleepTraceSink, recoveryTraceSink, stepsTraceSink, universalSink, workoutsTraceSink, hrvTraceSink, deepHrvWindow).first } @@ -313,8 +313,8 @@ object IntelligenceEngine { useExperimentalSleepV2: Boolean = false, // Opt-in motion-aware wake refinement (#364 follow-up), threaded the same way. Default false. useMotionAwareWake: Boolean = false, - // HR-first sleep detection (experimental, default OFF), threaded the same way. Default false. - useHrFirstSleep: Boolean = false, + // Wake-gap bridging (experimental, default OFF), threaded the same way. Default false. + useWakeBridge: Boolean = false, // Sleep & Rest test-mode trace sink (Test Centre E5). null = byte-identical default; when non-null // each scored day threads it into AnalyticsEngine.analyzeDay so detectSleep's gate trace + the Rest // sub-score line forward line-by-line to the .sleep-tagged strap log. Mirrors Swift. @@ -600,8 +600,8 @@ object IntelligenceEngine { useSleepStagerV2 = useExperimentalSleepV2, // #364 follow-up: same threading for the motion-aware wake refinement post-pass. useMotionAwareWake = useMotionAwareWake, - // HR-first sleep detection (experimental, default OFF). - useHrFirstSleep = useHrFirstSleep, + // Wake-gap bridging (experimental, default OFF). + useWakeBridge = useWakeBridge, // Sleep & Rest test mode (Test Centre E5): thread the trace sink straight through. null (the // default) keeps analyzeDay's byte-identical untraced path; when the caller passed a non-null // sink (mode on), detectSleep's gate trace + the Rest sub-score line route to the .sleep-tagged diff --git a/android/app/src/main/java/com/noop/analytics/SleepStager.kt b/android/app/src/main/java/com/noop/analytics/SleepStager.kt index dac21bca1a..c7a9a048e6 100644 --- a/android/app/src/main/java/com/noop/analytics/SleepStager.kt +++ b/android/app/src/main/java/com/noop/analytics/SleepStager.kt @@ -587,71 +587,44 @@ object SleepStager { return out } - // ── HR-first sleep detection (experimental, default OFF) ───────────────── + // ── Wake-gap bridging (experimental, default OFF) ──────────────────────── // - // The gravity-stillness spine reads a RESTLESS but genuinely asleep night — periodic limb - // movement, an injured/bedridden sleeper's tossing, or a strap whose motion channel - // over-reports — as "active", so whole hours of physiologically unambiguous sleep (sustained - // sub-baseline HR) never become candidate runs and the night collapses to whichever quiescent - // fragment survives. A real 5/MG user lost 6-8 h nights to 1-3 h morning fragments this way, - // while an ~8-min bathroom get-up — smeared past mergeMin by the CENTERED 15-min rolling - // stillness window — split the remaining night into separate stored sessions at a 16-min seam. - // The dense-gravity path has no HR-led recovery at all: HR only ever CONFIRMS gravity-derived - // runs (the #308 rescue is gated on gravity SPARSENESS, which a dense restless night never - // trips). The HR-first pass (default OFF; the live call site threads - // PuffinExperiment.hrFirstSleep) weights HR over motion when the two disagree: - // 1. [hrRescueActiveRuns] — an "active" run whose MEDIAN in-run HR sits inside the SAME sleep - // band the ladder already trusts ([confirmSleepWithHR]'s ≤ baseline × hrSleepBaselineMult) - // is reclassified "sleep": motion alone must not veto sleep the cardiac signal confirms. - // 2. [bridgeWakeGaps] — adjacent sleep runs separated by ≤ [wakeBridgeMaxMin] are merged into - // ONE candidate run, so a short get-up yields one session whose stager scores the wake - // INSIDE it instead of two stored sessions — and so sub-minSleepMin fragments between - // arousals are no longer dropped outright. - // Every downstream gate (minSleepMin, the H4 span cap, hrConfirm, off-wrist, the daytime and - // morning guards) still applies to the ASSEMBLED runs. KNOWN LIMIT (why default OFF, per the - // #194/#345 discipline): an awake-in-bed stretch whose HR genuinely sits inside the sleep band - // (deep pre-sleep relaxation) can be scored asleep — the same limit the still-sedentary gravity - // path already has, extended to fidgety-sedentary. Mirrors Swift. + // A brief mid-night get-up splits the night into separate stored sessions: the CENTERED 15-min + // rolling stillness window smears an ~8-min bathroom trip past mergeMin into an "active" run (a + // real MG night split at a 16-min seam), detection emits one stored session per still-run, and + // nothing at the detection layer re-merges them — the #561/#861 bridges only group sessions + // later, at selection/aggregation, so sub-minSleepMin sleep fragments between arousals are + // dropped by the ladder before those bridges ever see them. When enabled (the live call site + // threads PuffinExperiment.sleepWakeBridge, default OFF), [bridgeWakeGaps] assembles adjacent + // sleep runs separated by ≤ [wakeBridgeMaxMin] into ONE candidate run between the run spine and + // the gate ladder; the stager then scores the wake INSIDE the single session, and every + // downstream gate still applies to the ASSEMBLED run. A merge that would assemble a span over + // [maxMainSleepSpanS] is REFUSED (without that guard a pathological chain would assemble, then + // be dropped WHOLE by the H4 cap — converting a partially-scored night into a zero-score night). + // + // HISTORY — an HR-led "rescue" of motion-vetoed runs was built beside this bridge and WITHDRAWN + // after a 5-night real-data replay against the reporting wearer's server mirror (2026-08-24): + // on a low-HR wearer the quiet awake EVENING median (~58 bpm vs a ~64 window baseline) also + // sits inside the ×1.05 sleep band, so whole evenings were rescued and chained — the assembled + // run then either exceeded the H4 cap (night dropped to ZERO) or squeaked under it (a 15 h + // "night" starting at 18:00). The strap's own banked band sleep_state could not arbitrate (it + // is motion-led too and read "wake" across the HR-proven sleep). The restless-night DROPPING + // failure remains OPEN — see the Swift history note above `wakeBridgeMaxMin` before + // re-attempting an HR-band rescue. Mirrors Swift. /** Adjacent sleep runs separated by at most this many minutes are assembled into ONE candidate - * run (HR-first pass only). Below the selector-side `SleepStageTotals.gapBridgeMaxMin` (60) so - * selection/aggregation behaviour is unchanged for anything this bridge leaves split; well + * run (wake-bridge pass only). Below the selector-side `SleepStageTotals.gapBridgeMaxMin` (60) + * so selection/aggregation behaviour is unchanged for anything this bridge leaves split; well * above the ~16-min smeared seam a brief real get-up produces. Mirrors Swift `wakeBridgeMaxMin`. */ const val wakeBridgeMaxMin: Int = 45 /** - * HR-first step 1: reclassify an "active" run to "sleep" when its median in-run HR sits inside - * the sleep band (≤ effective-baseline × [hrSleepBaselineMult] — the identical band + median - * statistic [confirmSleepWithHR] trusts, with the same [sleepHRBaseline] personalised override). - * Runs with fewer than [hrRefineMinSamples] in-run HR samples are left untouched (too little - * cardiac evidence to overrule motion), as is everything when no baseline exists. Pure + - * deterministic; mirrors Swift `hrRescueActiveRuns`. - */ - internal fun hrRescueActiveRuns( - periods: List, hr: List, baseline: Double?, - sleepHRBaseline: Double? = null, traceSink: ((String) -> Unit)? = null, - ): List { - val effBaseline = sleepHRBaseline ?: baseline ?: return periods - return periods.map { p -> - if (p.stage != "active") return@map p - val seg = rowsBetween(hr, p.start, p.end) { it.ts } - if (seg.size < hrRefineMinSamples) return@map p - val medHR = HrvAnalyzer.median(seg.map { it.bpm.toDouble() }) - if (medHR > effBaseline * hrSleepBaselineMult) return@map p - traceSink?.invoke(SleepStagerTrace.runLine(-1, p.start, p.end, - SleepStagerTrace.Verdict.KEPT, "hrRescue", - "medHR=${medHR.toInt()} band=${SleepStagerTrace.round2(effBaseline * hrSleepBaselineMult)}")) - Period(stage = "sleep", start = p.start, end = p.end) - } - } - - /** - * HR-first step 2: assemble adjacent sleep runs separated by at most [wakeBridgeMaxMin] minutes - * (whatever lies between — an un-rescued active run, or a plain data hole) into one candidate - * run; the intervening runs are absorbed and the stager later scores the wake INSIDE the single - * session. A gap over the threshold (or a negative/overlapping gap — pinned `gap >= 0` - * semantics, matching [bridgeSparseSleep]) never bridges. Pure + deterministic; mirrors Swift - * `bridgeWakeGaps`. + * Assemble adjacent sleep runs separated by at most [wakeBridgeMaxMin] minutes (whatever lies + * between — an active run, or a plain data hole) into one candidate run; the intervening runs + * are absorbed and the stager later scores the wake INSIDE the single session. A gap over the + * threshold, a negative/overlapping gap (pinned `gap >= 0` semantics, matching + * [bridgeSparseSleep]), or a merge whose assembled span would exceed [maxMainSleepSpanS] (the + * H4 guard) never bridges. Pure + deterministic; mirrors Swift `bridgeWakeGaps`. */ internal fun bridgeWakeGaps( periods: List, traceSink: ((String) -> Unit)? = null, @@ -661,7 +634,8 @@ object SleepStager { for (p in periods) { val j = out.indexOfLast { it.stage == "sleep" } if (p.stage == "sleep" && j >= 0 && - p.start - out[j].end >= 0 && p.start - out[j].end <= bridgeS + p.start - out[j].end >= 0 && p.start - out[j].end <= bridgeS && + maxOf(out[j].end, p.end) - out[j].start <= maxMainSleepSpanS ) { val gapMin = (p.start - out[j].end) / 60 if (gapMin > 0) { @@ -967,7 +941,7 @@ object SleepStager { private data class DetectKey( val grav: Long, val hr: Long, val rr: Long, val resp: Long, val tz: Long, val wristOff: Long, val band: Long, val v2: Boolean, - val hrFirst: Boolean, + val bridge: Boolean, val sleepHRBaseline: Double?, ) @@ -1039,12 +1013,12 @@ object SleepStager { // #277, extended to every strap family in #351), so a normal user's nights are staged by V2. // Mirrors Swift. useSleepStagerV2: Boolean = false, - // HR-first sleep detection (experimental, default OFF — see the section comment above - // [wakeBridgeMaxMin]). When true, active runs whose median HR sits in the sleep band are - // rescued as sleep and adjacent sleep runs across ≤ wakeBridgeMaxMin wake gaps are assembled - // into one candidate run before the gate ladder. The default keeps every existing caller - // byte-identical; the live call site threads PuffinExperiment.hrFirstSleep. Mirrors Swift. - hrFirstSleep: Boolean = false, + // Wake-gap bridging (experimental, default OFF — see the section comment above + // [wakeBridgeMaxMin]). When true, adjacent sleep runs across ≤ wakeBridgeMaxMin wake gaps + // are assembled into one candidate run before the gate ladder, so a brief get-up stays + // INSIDE one session as staged wake. The default keeps every existing caller byte-identical; + // the live call site threads PuffinExperiment.sleepWakeBridge. Mirrors Swift. + wakeBridge: Boolean = false, // Motion-corroborated wake (#462, directive b): the wearer's PERSONALISED overnight HR band // ([adaptiveOvernightHRBaseline]), used by [confirmSleepWithHR] in place of the day-median so a // supplement / fitness era self-calibrates the sleep band. Default null keeps the day-median @@ -1062,7 +1036,7 @@ object SleepStager { // Swift detectSleep traceSink bypass (#707). if (traceSink != null) { return detectSleepUncached(hr, rr, resp, gravity, tzOffsetSeconds, wristOff, - bandSleepState, useSleepStagerV2, hrFirstSleep, sleepHRBaseline, traceSink) + bandSleepState, useSleepStagerV2, wakeBridge, sleepHRBaseline, traceSink) } val key = DetectKey( // Fold the three gravity axes SEPARATELY (raw IEEE-754 bits, like StagerCache.fingerprint) @@ -1083,7 +1057,7 @@ object SleepStager { wristOff = streamFingerprint(wristOff, { it.first }) { it.second }, band = streamFingerprint(bandSleepState, { it.first }) { it.second.toLong() }, v2 = useSleepStagerV2, - hrFirst = hrFirstSleep, + bridge = wakeBridge, sleepHRBaseline = sleepHRBaseline, ) synchronized(detectCacheLock) { detectCache[key] }?.let { return copyDetected(it) } @@ -1091,7 +1065,7 @@ object SleepStager { // serialises another thread's lookup, and a racing duplicate compute just overwrites the entry // with an identical value — benign, the function is deterministic. val sessions = detectSleepUncached(hr, rr, resp, gravity, tzOffsetSeconds, wristOff, - bandSleepState, useSleepStagerV2, hrFirstSleep, sleepHRBaseline, null) + bandSleepState, useSleepStagerV2, wakeBridge, sleepHRBaseline, null) synchronized(detectCacheLock) { detectCache[key] = copyDetected(sessions) } return sessions } @@ -1107,7 +1081,7 @@ object SleepStager { wristOff: List>, bandSleepState: List>, useSleepStagerV2: Boolean, - hrFirstSleep: Boolean, + wakeBridge: Boolean, sleepHRBaseline: Double?, traceSink: ((String) -> Unit)?, ): List { @@ -1141,16 +1115,13 @@ object SleepStager { "sparseBridge", "sparse=true gapMin=$sparseBridgeGapMin runsBefore=$runsBeforeBridge runsAfter=$runsAfterBridge")) } - // HR-first pass (experimental, default OFF — see the section comment above [wakeBridgeMaxMin]): - // weight HR over motion where the two disagree (an active run whose median HR is in the sleep - // band is rescued as sleep), then assemble one candidate run per sleep period across short + // Wake-gap bridge (experimental, default OFF — see the section comment above + // [wakeBridgeMaxMin]): assemble one candidate run per sleep period across short // (≤ wakeBridgeMaxMin) wake gaps, BEFORE the ladder — so sub-minSleepMin fragments between // arousals survive as part of the night instead of being dropped, and a brief get-up stays // INSIDE one session as staged wake. Every gate below still applies to the assembled runs. // Mirrors Swift. - if (hrFirstSleep) { - runs = hrRescueActiveRuns(runs, hrS, baseline, - sleepHRBaseline = sleepHRBaseline, traceSink = traceSink) + if (wakeBridge) { runs = bridgeWakeGaps(runs, traceSink = traceSink) } diff --git a/android/app/src/main/java/com/noop/ble/PuffinExperiment.kt b/android/app/src/main/java/com/noop/ble/PuffinExperiment.kt index a6ff900395..6e0c1f2d14 100644 --- a/android/app/src/main/java/com/noop/ble/PuffinExperiment.kt +++ b/android/app/src/main/java/com/noop/ble/PuffinExperiment.kt @@ -96,21 +96,20 @@ class PuffinExperiment(private val prefs: SharedPreferences) { get() = prefs.getBoolean(KEY_MOTION_AWARE_WAKE, false) set(v) = prefs.edit().putBoolean(KEY_MOTION_AWARE_WAKE, v).apply() - /** Opt-in "HR-first sleep detection" (default OFF): weight the cardiac signal over wrist motion - * when the two disagree, inside [com.noop.analytics.SleepStager.detectSleep]. Two effects, both - * BEFORE the gate ladder: a motion-"active" run whose median HR sits inside the same sleep band - * the ladder already trusts is rescued as sleep (a restless-but-asleep night is no longer - * collapsed to its quiescent morning fragment), and adjacent sleep runs across a ≤ 45-min wake - * gap are assembled into ONE session with the wake staged inside it (a brief get-up no longer - * splits the night into separate stored sessions). Every existing guard (min duration, span cap, - * HR confirmation, off-wrist, daytime nap bars) still applies to the assembled runs. Default OFF - * per the varying-input validation discipline (#194/#345): an awake-in-bed stretch whose HR - * genuinely sits in the sleep band can be scored asleep, the same limit the still-sedentary path - * already has. Pure analysis flag (sends nothing to the strap), model-agnostic. Mirrors the - * macOS `PuffinExperiment.hrFirstSleepKey`. */ - var hrFirstSleep: Boolean - get() = prefs.getBoolean(KEY_HR_FIRST_SLEEP, false) - set(v) = prefs.edit().putBoolean(KEY_HR_FIRST_SLEEP, v).apply() + /** Opt-in "Bridge brief wakes" (default OFF): inside [com.noop.analytics.SleepStager.detectSleep], + * adjacent sleep runs across a ≤ 45-min wake gap are assembled into ONE session with the wake + * staged inside it, so a brief get-up no longer splits the night into separate stored sessions, + * and sub-1h sleep fragments between arousals are no longer dropped outright. Every existing + * guard (min duration, the 16-h span cap — which the bridge also refuses to assemble past — HR + * confirmation, off-wrist, daytime nap bars) still applies to the assembled run. Default OFF per + * the varying-input validation discipline (#194/#345). NOTE: an HR-led rescue of + * restless-but-asleep nights was prototyped behind this flag and WITHDRAWN after a real-data + * replay — see the Swift history note above `SleepStager.wakeBridgeMaxMin` before re-attempting + * it. Pure analysis flag (sends nothing to the strap), model-agnostic. Mirrors the macOS + * `PuffinExperiment.wakeBridgeKey`. */ + var sleepWakeBridge: Boolean + get() = prefs.getBoolean(KEY_SLEEP_WAKE_BRIDGE, false) + set(v) = prefs.edit().putBoolean(KEY_SLEEP_WAKE_BRIDGE, v).apply() /** * Turn OFF every 5/MG-only experimental probe: protocol probes ([isEnabled]), raw capture @@ -162,7 +161,7 @@ class PuffinExperiment(private val prefs: SharedPreferences) { /** "Motion-aware wake refinement" opt-in (mirrors macOS `PuffinExperiment.motionAwareWakeKey`). */ const val KEY_MOTION_AWARE_WAKE = "noopMotionAwareWake" - const val KEY_HR_FIRST_SLEEP = "noopHrFirstSleep" + const val KEY_SLEEP_WAKE_BRIDGE = "noopSleepWakeBridge" fun from(context: Context): PuffinExperiment = PuffinExperiment(context.getSharedPreferences(PREFS, Context.MODE_PRIVATE)) diff --git a/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt b/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt index b75bdd0000..bbc1ff2b68 100644 --- a/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt +++ b/android/app/src/main/java/com/noop/ble/WhoopBleClient.kt @@ -2126,8 +2126,8 @@ class WhoopBleClient( useExperimentalSleepV2 = PuffinExperiment.from(context).experimentalSleepV2, // Opt-in motion-aware wake refinement (#364 follow-up) — same Context-free threading. useMotionAwareWake = PuffinExperiment.from(context).motionAwareWake, - // HR-first sleep detection (experimental, default OFF) — same Context-free threading. - useHrFirstSleep = PuffinExperiment.from(context).hrFirstSleep, + // Wake-gap bridging (experimental, default OFF) — same Context-free threading. + useWakeBridge = PuffinExperiment.from(context).sleepWakeBridge, // Sleep & Rest test mode (Test Centre E5): when the SLEEP domain is on, route this // post-backfill pass's per-day sleep gate trace into the .sleep-tagged strap log, so a // shared report carries the staging proof from THIS scoring pass too, not only the UI diff --git a/android/app/src/main/java/com/noop/ui/AppViewModel.kt b/android/app/src/main/java/com/noop/ui/AppViewModel.kt index 95ee62f26c..a4006e0c17 100644 --- a/android/app/src/main/java/com/noop/ui/AppViewModel.kt +++ b/android/app/src/main/java/com/noop/ui/AppViewModel.kt @@ -943,8 +943,8 @@ class AppViewModel(app: Application) : AndroidViewModel(app) { useExperimentalSleepV2 = PuffinExperiment.from(appContext).experimentalSleepV2, // Opt-in motion-aware wake refinement (#364 follow-up) — same Context-free threading. useMotionAwareWake = PuffinExperiment.from(appContext).motionAwareWake, - // HR-first sleep detection (experimental, default OFF) — same Context-free threading. - useHrFirstSleep = PuffinExperiment.from(appContext).hrFirstSleep, + // Wake-gap bridging (experimental, default OFF) — same Context-free threading. + useWakeBridge = PuffinExperiment.from(appContext).sleepWakeBridge, // Sleep & Rest test mode (Test Centre E5): when the SLEEP domain is on, route the // per-day sleep gate trace into the SAME shareable strap log, tagged .sleep so it // lands under the profile in the export. Zero-cost when off: the gate is one @@ -1564,8 +1564,8 @@ class AppViewModel(app: Application) : AndroidViewModel(app) { useExperimentalSleepV2 = PuffinExperiment.from(appContext).experimentalSleepV2, // Opt-in motion-aware wake refinement (#364 follow-up) — same flag the 15-min loop reads. useMotionAwareWake = PuffinExperiment.from(appContext).motionAwareWake, - // HR-first sleep detection (experimental, default OFF) — same flag the 15-min loop reads. - useHrFirstSleep = PuffinExperiment.from(appContext).hrFirstSleep, + // Wake-gap bridging (experimental, default OFF) — same flag the 15-min loop reads. + useWakeBridge = PuffinExperiment.from(appContext).sleepWakeBridge, ) }.onFailure { if (it is kotlin.coroutines.cancellation.CancellationException) throw it } } diff --git a/android/app/src/main/java/com/noop/ui/SettingsScreen.kt b/android/app/src/main/java/com/noop/ui/SettingsScreen.kt index ed08f939f0..d05cc578cc 100644 --- a/android/app/src/main/java/com/noop/ui/SettingsScreen.kt +++ b/android/app/src/main/java/com/noop/ui/SettingsScreen.kt @@ -502,9 +502,9 @@ fun SettingsScreen( // "Motion-aware wake refinement" (#364 follow-up) — OFF by default. Self-gates on observed gravity + // step density, so it is a no-op on a sparse (e.g. WHOOP 4.0) night regardless of this switch. var motionAwareWake by remember { mutableStateOf(puffinExperiment.motionAwareWake) } - // "HR-first sleep detection" — OFF by default. Weights HR over wrist motion in sleep DETECTION and - // bridges short (≤45 min) wake gaps into one session; see PuffinExperiment.hrFirstSleep. - var hrFirstSleep by remember { mutableStateOf(puffinExperiment.hrFirstSleep) } + // "Bridge brief wakes" — OFF by default. Bridges short (≤45 min) wake gaps into one detected + // session with the wake staged inside; see PuffinExperiment.sleepWakeBridge. + var sleepWakeBridge by remember { mutableStateOf(puffinExperiment.sleepWakeBridge) } // Whether to surface the WHOOP 5/MG-only probes (puffin / R22 / broadcast-HR / frame-capture). Gated // so a confident 4.0 owner never sees 5/MG controls that can't touch their strap (#22). The model @@ -2207,23 +2207,23 @@ fun SettingsScreen( color = Palette.textTertiary, ) - // --- HR-first sleep detection — OFF by default. --- + // --- Bridge brief wakes — OFF by default. --- Row( modifier = Modifier.fillMaxWidth(), verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(16.dp), ) { Text( - "HR-first sleep detection", + "Bridge brief wakes", style = NoopType.subhead, color = Palette.textPrimary, modifier = Modifier.weight(1f), ) Switch( - checked = hrFirstSleep, + checked = sleepWakeBridge, onCheckedChange = { - hrFirstSleep = it - puffinExperiment.hrFirstSleep = it + sleepWakeBridge = it + puffinExperiment.sleepWakeBridge = it }, colors = SwitchDefaults.colors( checkedThumbColor = Palette.surfaceBase, @@ -2233,19 +2233,17 @@ fun SettingsScreen( uncheckedBorderColor = Palette.hairline, ), modifier = Modifier.semantics { - contentDescription = "HR-first sleep detection" + contentDescription = "Bridge brief wakes" }, ) } Text( - "Trusts your heart rate over wrist movement when the two disagree about sleep. A " + - "restless night whose heart rate clearly sits at your sleep level is kept as sleep " + - "instead of being cut down to its calmest fragment, and a brief get-up (under ~45 " + - "minutes) stays inside one sleep with the wake scored inside it, rather than " + - "splitting the night into separate sleeps. Worth trying if your nights show up " + - "fragmented or mostly missing despite a normal sleeping heart rate. Trade-off: a " + - "long, deeply relaxed stretch awake in bed can be scored as sleep. Off by default; " + - "takes effect on the next nights scored.", + "Keeps a night interrupted by a brief get-up (under ~45 minutes) as one sleep, " + + "with the wake scored inside it, instead of splitting it into separate sleeps " + + "-- and short sleep stretches between such wakes are no longer discarded for " + + "being under an hour on their own. Worth trying if your nights show up split " + + "into two or three pieces around bathroom trips or brief stirs. Off by " + + "default; takes effect on the next nights scored.", style = NoopType.caption, color = Palette.textTertiary, ) diff --git a/android/app/src/test/java/com/noop/analytics/SleepStagerHrFirstTest.kt b/android/app/src/test/java/com/noop/analytics/SleepStagerHrFirstTest.kt deleted file mode 100644 index eccaa12d30..0000000000 --- a/android/app/src/test/java/com/noop/analytics/SleepStagerHrFirstTest.kt +++ /dev/null @@ -1,202 +0,0 @@ -package com.noop.analytics - -import com.noop.data.GravitySample -import com.noop.data.HrSample -import org.junit.Assert.assertEquals -import org.junit.Assert.assertTrue -import org.junit.Test -import kotlin.math.abs - -/** - * HR-first sleep detection (experimental, default OFF): weight HR over motion when the two - * disagree, and assemble one session per sleep period across short wake gaps. - * - * Per the "validate against varying inputs" rule (#194/#345), these tests inject the sleep window - * at DIFFERENT positions and lengths and assert the rescue TRACKS it — and that the identical - * motion pattern with an elevated (awake) HR is refused — rather than pinning one matched night. - * Faithful Kotlin mirror of SleepStagerHrFirstTests.swift: same scenarios, same numbers. - */ -class SleepStagerHrFirstTest { - - private val dev = "test" - - /** 2026-06-10 00:00:00 UTC — same fixed midnight the Swift twin uses. */ - private val refMidnight = 1_749_513_600L - - private fun at(hourUTC: Int, min: Int = 0): Long = refMidnight + hourUTC * 3_600L + min * 60L - - private fun stillGravity(start: Long, durationS: Int): List = - (0 until durationS).map { GravitySample(deviceId = dev, ts = start + it, x = 0.0, y = 0.0, z = 1.0) } - - /** Oscillating orientation (0.5 g jumps per sample) — clearly "moving" to the stillness spine. */ - private fun restlessGravity(start: Long, durationS: Int): List = - (0 until durationS).map { i -> - val phase = (i % 2) * 0.5 - GravitySample(deviceId = dev, ts = start + i, x = phase, y = 0.0, z = 1.0) - } - - private fun hrStream(start: Long, durationS: Int, bpm: Int): List = - (0 until durationS).map { HrSample(deviceId = dev, ts = start + it, bpm = bpm) } - - // ── Restless-night rescue (the DROPPED-night failure) ───────────────────── - - @Test - fun restlessNightRescuedToFullSession() { - // Restless (motion "active") but cardiac-unambiguous (HR 52 vs awake 75) 22:00–04:30, - // then still to 06:00. OFF: only the quiescent morning fragment scores. ON: one session. - val grav = restlessGravity(at(22), 6 * 3_600 + 1_800) + // 22:00–04:30 - stillGravity(at(28, 30), 90 * 60) // 04:30–06:00 - val hr = hrStream(at(20), 2 * 3_600, 75) + // 20:00–22:00 awake - hrStream(at(22), 8 * 3_600, 52) + // 22:00–06:00 asleep - hrStream(at(30), 2 * 3_600, 75) // 06:00–08:00 awake - - val off = SleepStager.detectSleep(hr = hr, gravity = grav) - assertEquals("OFF: only the still morning fragment should score", 1, off.size) - assertTrue("OFF fragment starts near 04:30", off[0].start >= at(28, 25)) - assertTrue(off[0].start <= at(28, 45)) - - val on = SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = true) - assertEquals("ON: one assembled session for the whole night", 1, on.size) - assertTrue("ON: onset tracks the injected 22:00 sleep start", on[0].start <= at(22) + 120) - assertTrue("ON: end tracks the injected 06:00 wake", on[0].end >= at(30) - 120) - } - - @Test - fun restlessRescueTracksInjectedWindow() { - // Same rescue with the sleep window injected at a DIFFERENT position and length - // (01:00–07:30, all restless, no still stretch at all). - val grav = restlessGravity(at(25), 6 * 3_600 + 1_800) // 01:00–07:30 - val hr = hrStream(at(22), 3 * 3_600, 75) + // 22:00–01:00 awake - hrStream(at(25), 6 * 3_600 + 1_800, 52) + // 01:00–07:30 asleep - hrStream(at(31, 30), 2 * 3_600, 75) // 07:30–09:30 awake - - assertEquals("OFF: an all-restless night scores nothing", - 0, SleepStager.detectSleep(hr = hr, gravity = grav).size) - - val on = SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = true) - assertEquals(1, on.size) - assertTrue("onset tracks the moved window", abs(on[0].start - at(25)) <= 120) - assertTrue("end tracks the moved window", abs(on[0].end - at(31, 30)) <= 120) - } - - @Test - fun elevatedHrNightIsNotRescued() { - // IDENTICAL motion to the full-night rescue, night HR elevated (78 vs a 62 day median): - // the rescue must refuse the restless stretch. (The STILL morning fragment is still - // accepted by the pre-existing motion-corroborated quiescent band (#462), so the assertion - // is that HR-first adds NOTHING: same single fragment, onset still at the quiescent stretch.) - val grav = restlessGravity(at(22), 6 * 3_600 + 1_800) + // 22:00–04:30 - stillGravity(at(28, 30), 90 * 60) // 04:30–06:00 - val hr = hrStream(at(22), 8 * 3_600, 78) + // 22:00–06:00 elevated - hrStream(at(30), 10 * 3_600, 62) // 06:00–16:00 day median - - val off = SleepStager.detectSleep(hr = hr, gravity = grav) - val on = SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = true) - assertEquals("same motion, elevated HR: HR-first must change nothing", off, on) - assertEquals(1, on.size) - assertTrue("the restless 22:00–04:30 stretch must NOT be rescued", on[0].start >= at(28, 25)) - } - - // ── Wake-gap bridging (the SPLIT-night failure) ────────────────────────── - - @Test - fun briefGetUpBridgedIntoOneSession() { - // A 16-min get-up (real movement + HR 80) mid-night. The centered rolling window smears it - // past mergeMin, so OFF stores TWO sessions; ON bridges them into ONE. - val grav = stillGravity(at(23), 4 * 3_600) + // 23:00–03:00 - restlessGravity(at(27), 16 * 60) + // 03:00–03:16 get-up - stillGravity(at(27, 16), 3 * 3_600 + 44 * 60) // 03:16–07:00 - val hr = hrStream(at(21), 2 * 3_600, 75) + - hrStream(at(23), 4 * 3_600, 52) + - hrStream(at(27), 16 * 60, 80) + - hrStream(at(27, 16), 3 * 3_600 + 44 * 60, 52) + - hrStream(at(31), 2 * 3_600, 75) - - assertEquals("OFF: the smeared get-up splits the night", - 2, SleepStager.detectSleep(hr = hr, gravity = grav).size) - - val on = SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = true) - assertEquals("ON: one session bridging the get-up", 1, on.size) - assertTrue(abs(on[0].start - at(23)) <= 120) - assertTrue(abs(on[0].end - at(31)) <= 120) - } - - @Test - fun longWakeGapStaysSplit() { - // An 80-min genuinely-awake block (elevated HR, real movement) exceeds the 45-min bridge. - val grav = stillGravity(at(22), 3 * 3_600) + // 22:00–01:00 - restlessGravity(at(25), 80 * 60) + // 01:00–02:20 awake - stillGravity(at(26, 20), 3 * 3_600 + 40 * 60) // 02:20–06:00 - val hr = hrStream(at(20), 2 * 3_600, 75) + - hrStream(at(22), 3 * 3_600, 52) + - hrStream(at(25), 80 * 60, 78) + - hrStream(at(26, 20), 3 * 3_600 + 40 * 60, 52) + - hrStream(at(30), 2 * 3_600, 75) - - assertEquals("an over-threshold wake gap must not be bridged", - 2, SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = true).size) - } - - // ── Guards still hold on assembled runs ────────────────────────────────── - - @Test - fun daytimeGuardStillAppliesToRescuedRuns() { - // A rescued + bridged DAYTIME stretch (in-band flat HR 60, no cardiac dip) must still be - // dropped by the daytime false-sleep guard. - val grav = stillGravity(at(11), 2 * 3_600) + // 11:00–13:00 - restlessGravity(at(13), 2 * 3_600) + // 13:00–15:00 - stillGravity(at(15), 2 * 3_600) // 15:00–17:00 - val hr = hrStream(at(10), 8 * 3_600, 60) // flat, no dip - - assertEquals("no resting-HR dip: the daytime guard must reject the rescue", - 0, SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = true).size) - } - - @Test - fun defaultOffIsByteIdentical() { - val grav = stillGravity(at(23), 4 * 3_600) + - restlessGravity(at(27), 16 * 60) + - stillGravity(at(27, 16), 3 * 3_600 + 44 * 60) - val hr = hrStream(at(23), 8 * 3_600, 52) + hrStream(at(31), 4 * 3_600, 75) - - val implicitDefault = SleepStager.detectSleep(hr = hr, gravity = grav) - val explicitFalse = SleepStager.detectSleep(hr = hr, gravity = grav, hrFirstSleep = false) - assertEquals(implicitDefault, explicitFalse) - } - - // ── Pure helpers ───────────────────────────────────────────────────────── - - @Test - fun bridgeWakeGapsMergesOnlyUnderThreshold() { - val a = SleepStager.Period(stage = "sleep", start = 0L, end = 3_600L) - val gap = SleepStager.Period(stage = "active", start = 3_600L, end = 3_600L + 40 * 60) - val b = SleepStager.Period(stage = "sleep", start = 3_600L + 40 * 60, end = 3 * 3_600L) - val merged = SleepStager.bridgeWakeGaps(listOf(a, gap, b)) - assertEquals(1, merged.size) - assertEquals(0L, merged[0].start) - assertEquals(3 * 3_600L, merged[0].end) - - val farGap = SleepStager.Period(stage = "active", start = 3_600L, end = 3_600L + 46 * 60) - val c = SleepStager.Period(stage = "sleep", start = 3_600L + 46 * 60, end = 4 * 3_600L) - assertEquals("a 46-min gap is over wakeBridgeMaxMin (45) and must not merge", - 3, SleepStager.bridgeWakeGaps(listOf(a, farGap, c)).size) - } - - @Test - fun hrRescueRespectsBandAndSampleFloor() { - val p = SleepStager.Period(stage = "active", start = 0L, end = 3_600L) - val inBand = (0 until 3_600).map { HrSample(deviceId = dev, ts = it.toLong(), bpm = 52) } - assertEquals("sleep", SleepStager.hrRescueActiveRuns(listOf(p), inBand, 52.0)[0].stage) - - val outOfBand = (0 until 3_600).map { HrSample(deviceId = dev, ts = it.toLong(), bpm = 60) } - assertEquals("median above baseline × 1.05 must not rescue", - "active", SleepStager.hrRescueActiveRuns(listOf(p), outOfBand, 52.0)[0].stage) - - // Below the hrRefineMinSamples floor: too little cardiac evidence to overrule motion. - val sparse = (0 until 10).map { HrSample(deviceId = dev, ts = it * 60L, bpm = 52) } - assertEquals("active", SleepStager.hrRescueActiveRuns(listOf(p), sparse, 52.0)[0].stage) - - // No baseline at all: nothing to weigh HR against. - assertEquals("active", SleepStager.hrRescueActiveRuns(listOf(p), inBand, null)[0].stage) - } -} diff --git a/android/app/src/test/java/com/noop/analytics/SleepStagerWakeBridgeTest.kt b/android/app/src/test/java/com/noop/analytics/SleepStagerWakeBridgeTest.kt new file mode 100644 index 0000000000..76d5c29d73 --- /dev/null +++ b/android/app/src/test/java/com/noop/analytics/SleepStagerWakeBridgeTest.kt @@ -0,0 +1,202 @@ +package com.noop.analytics + +import com.noop.data.GravitySample +import com.noop.data.HrSample +import org.junit.Assert.assertEquals +import org.junit.Assert.assertTrue +import org.junit.Test +import kotlin.math.abs + +/** + * Wake-gap bridging (experimental, default OFF): assemble adjacent sleep runs across ≤ 45-min wake + * gaps into ONE candidate run, so a brief get-up stays inside one session as staged wake. + * + * Includes a REAL-DATA pin: the exact seam layout of a reporting wearer's stored fragmented nights + * (Aug 2026 mirror), asserting precisely which seams the bridge merges and which it leaves split. + * An HR-led rescue of restless-but-asleep nights was prototyped alongside this bridge and WITHDRAWN + * after a real-data replay — see the Swift history note above `SleepStager.wakeBridgeMaxMin`; the + * restless-night test below pins that the bridge deliberately does NOT fix that failure. + * Faithful Kotlin mirror of SleepStagerWakeBridgeTests.swift: same scenarios, same numbers. + */ +class SleepStagerWakeBridgeTest { + + private val dev = "test" + + /** 2026-06-10 00:00:00 UTC — same fixed midnight the Swift twin uses. */ + private val refMidnight = 1_749_513_600L + + private fun at(hourUTC: Int, min: Int = 0): Long = refMidnight + hourUTC * 3_600L + min * 60L + + private fun stillGravity(start: Long, durationS: Int): List = + (0 until durationS).map { GravitySample(deviceId = dev, ts = start + it, x = 0.0, y = 0.0, z = 1.0) } + + /** Oscillating orientation (0.5 g jumps per sample) — clearly "moving" to the stillness spine. */ + private fun restlessGravity(start: Long, durationS: Int): List = + (0 until durationS).map { i -> + val phase = (i % 2) * 0.5 + GravitySample(deviceId = dev, ts = start + i, x = phase, y = 0.0, z = 1.0) + } + + private fun hrStream(start: Long, durationS: Int, bpm: Int): List = + (0 until durationS).map { HrSample(deviceId = dev, ts = start + it, bpm = bpm) } + + // ── Wake-gap bridging (the SPLIT-night failure) ────────────────────────── + + @Test + fun briefGetUpBridgedIntoOneSession() { + // A 16-min get-up (real movement + HR 80) mid-night. The centered rolling window smears it + // past mergeMin, so OFF stores TWO sessions; ON bridges them into ONE. + val grav = stillGravity(at(23), 4 * 3_600) + // 23:00–03:00 + restlessGravity(at(27), 16 * 60) + // 03:00–03:16 get-up + stillGravity(at(27, 16), 3 * 3_600 + 44 * 60) // 03:16–07:00 + val hr = hrStream(at(21), 2 * 3_600, 75) + + hrStream(at(23), 4 * 3_600, 52) + + hrStream(at(27), 16 * 60, 80) + + hrStream(at(27, 16), 3 * 3_600 + 44 * 60, 52) + + hrStream(at(31), 2 * 3_600, 75) + + assertEquals("OFF: the smeared get-up splits the night", + 2, SleepStager.detectSleep(hr = hr, gravity = grav).size) + + val on = SleepStager.detectSleep(hr = hr, gravity = grav, wakeBridge = true) + assertEquals("ON: one session bridging the get-up", 1, on.size) + assertTrue(abs(on[0].start - at(23)) <= 120) + assertTrue(abs(on[0].end - at(31)) <= 120) + } + + @Test + fun longWakeGapStaysSplit() { + // An 80-min genuinely-awake block (elevated HR, real movement) exceeds the 45-min bridge. + val grav = stillGravity(at(22), 3 * 3_600) + // 22:00–01:00 + restlessGravity(at(25), 80 * 60) + // 01:00–02:20 awake + stillGravity(at(26, 20), 3 * 3_600 + 40 * 60) // 02:20–06:00 + val hr = hrStream(at(20), 2 * 3_600, 75) + + hrStream(at(22), 3 * 3_600, 52) + + hrStream(at(25), 80 * 60, 78) + + hrStream(at(26, 20), 3 * 3_600 + 40 * 60, 52) + + hrStream(at(30), 2 * 3_600, 75) + + assertEquals("an over-threshold wake gap must not be bridged", + 2, SleepStager.detectSleep(hr = hr, gravity = grav, wakeBridge = true).size) + } + + @Test + fun restlessNightIsNotRescuedByBridge() { + // The withdrawn-rescue pin: a restless night (motion "active", HR at clear sleep levels) is + // NOT rescued by the bridge — ON must change nothing versus OFF. The bridge only assembles + // runs the unchanged stillness spine already accepted. + val grav = restlessGravity(at(22), 6 * 3_600 + 1_800) + // 22:00–04:30 + stillGravity(at(28, 30), 90 * 60) // 04:30–06:00 + val hr = hrStream(at(20), 2 * 3_600, 75) + + hrStream(at(22), 8 * 3_600, 52) + + hrStream(at(30), 2 * 3_600, 75) + + val off = SleepStager.detectSleep(hr = hr, gravity = grav) + val on = SleepStager.detectSleep(hr = hr, gravity = grav, wakeBridge = true) + assertEquals("the bridge must not reclassify restless stretches", off, on) + assertEquals("only the still morning fragment scores either way", 1, on.size) + assertTrue(on[0].start >= at(28, 25)) + } + + // ── Guards still hold on assembled runs ────────────────────────────────── + + @Test + fun daytimeGuardStillAppliesToBridgedRuns() { + // Two still DAYTIME stretches bridged across a 30-min gap (flat in-band HR, no cardiac dip) + // must still be dropped by the daytime false-sleep guard on the ASSEMBLED window. + val grav = stillGravity(at(12), 2 * 3_600) + // 12:00–14:00 + stillGravity(at(14, 30), 2 * 3_600) // 14:30–16:30 + val hr = hrStream(at(10), 8 * 3_600, 60) // flat, no dip + + assertEquals("no resting-HR dip: the daytime guard must reject the bridged window", + 0, SleepStager.detectSleep(hr = hr, gravity = grav, wakeBridge = true).size) + } + + @Test + fun defaultOffIsByteIdentical() { + val grav = stillGravity(at(23), 4 * 3_600) + + restlessGravity(at(27), 16 * 60) + + stillGravity(at(27, 16), 3 * 3_600 + 44 * 60) + val hr = hrStream(at(23), 8 * 3_600, 52) + hrStream(at(31), 4 * 3_600, 75) + + val implicitDefault = SleepStager.detectSleep(hr = hr, gravity = grav) + val explicitFalse = SleepStager.detectSleep(hr = hr, gravity = grav, wakeBridge = false) + assertEquals(implicitDefault, explicitFalse) + } + + // ── Pure helper ────────────────────────────────────────────────────────── + + @Test + fun bridgeWakeGapsMergesOnlyUnderThreshold() { + val a = SleepStager.Period(stage = "sleep", start = 0L, end = 3_600L) + val gap = SleepStager.Period(stage = "active", start = 3_600L, end = 3_600L + 40 * 60) + val b = SleepStager.Period(stage = "sleep", start = 3_600L + 40 * 60, end = 3 * 3_600L) + val merged = SleepStager.bridgeWakeGaps(listOf(a, gap, b)) + assertEquals(1, merged.size) + assertEquals(0L, merged[0].start) + assertEquals(3 * 3_600L, merged[0].end) + + val farGap = SleepStager.Period(stage = "active", start = 3_600L, end = 3_600L + 46 * 60) + val c = SleepStager.Period(stage = "sleep", start = 3_600L + 46 * 60, end = 4 * 3_600L) + assertEquals("a 46-min gap is over wakeBridgeMaxMin (45) and must not merge", + 3, SleepStager.bridgeWakeGaps(listOf(a, farGap, c)).size) + } + + @Test + fun bridgeRefusesOverSpanCapMerge() { + // H4 span guard: a merge whose ASSEMBLED span would exceed maxMainSleepSpanS (16 h) is + // refused, so a pathological chain can never assemble and then be dropped whole by the + // ladder's cap — the exact failure the withdrawn HR rescue exhibited on real data. + val a = SleepStager.Period(stage = "sleep", start = 0L, end = 8 * 3_600L) + val b = SleepStager.Period(stage = "sleep", start = 8 * 3_600L + 30 * 60, end = 17 * 3_600L) + assertEquals("a merge assembling a >16 h span must be refused", + 2, SleepStager.bridgeWakeGaps(listOf(a, b)).size) + } + + // ── Real-data seam pin (Aug 2026 mirror, reporting wearer's stored fragments) ── + + @Test + fun realNightSeamsBridgeExactly() { + // The stored fragment layouts of five REAL split nights, as (start, end) unix pairs from + // the wearer's server mirror. Asserts exactly which seams the 45-min bridge merges + // (16/12/28/25+19-min arousal seams) and which it leaves split (2.2–3.9 h genuine wake + // gaps, and a 90-min morning re-sleep gap — over the threshold, left to the selector's + // #861 night-tail bridge). Same pairs as the Swift twin. + fun sleeps(spans: List>): List = + spans.map { SleepStager.Period(stage = "sleep", start = it.first, end = it.second) } + + // Aug 11→12 ET: 4 fragments; only the 16-min bathroom seam (06:17→06:33 ET) merges → 3 blocks. + val aug12 = sleeps(listOf(1786491049L to 1786496628L, 1786510824L to 1786514588L, + 1786522512L to 1786529853L, 1786530794L to 1786541952L)) + assertEquals(listOf(1786491049L to 1786496628L, 1786510824L to 1786514588L, + 1786522512L to 1786541952L), + SleepStager.bridgeWakeGaps(aug12).map { it.start to it.end }) + + // Aug 17→18 ET: 3 fragments; the 12-min seam merges, the 3.8 h gap stays → 2 blocks. + val aug18 = sleeps(listOf(1787032890L to 1787037056L, 1787050829L to 1787054649L, + 1787055394L to 1787063445L)) + assertEquals(listOf(1787032890L to 1787037056L, 1787050829L to 1787063445L), + SleepStager.bridgeWakeGaps(aug18).map { it.start to it.end }) + + // Aug 18→19 ET: 2 fragments at a 28-min seam → 1 block. + val aug19 = sleeps(listOf(1787126248L to 1787132602L, 1787134284L to 1787141425L)) + assertEquals(listOf(1787126248L to 1787141425L), + SleepStager.bridgeWakeGaps(aug19).map { it.start to it.end }) + + // Aug 20→21 ET: 3 fragments at 25-min and 19-min seams → 1 block (11.4 h < the 16 h cap). + // NOTE: the mirror's correction pass (an HR-evidence-led edit session, not a human label) + // kept the third block (a post-wake morning doze) as a SEPARATE doze; the 45-min rule cannot + // distinguish "brief wake, more sleep" from "woke, dozed again", so the bridge deliberately + // folds it in (wake staged inside). Pinned as-is. + val aug21 = sleeps(listOf(1787287220L to 1787297404L, 1787298936L to 1787322337L, + 1787323440L to 1787328166L)) + assertEquals(listOf(1787287220L to 1787328166L), + SleepStager.bridgeWakeGaps(aug21).map { it.start to it.end }) + + // Aug 23→24 ET: 2 fragments at a 90-min gap → stays split (over the 45-min bridge). The + // mirror's correction pass filled this gap as SLEEP on HR evidence — it is the + // withdrawn-rescue (restless dropping) class, not a seam; the bridge must not guess it. + val aug24 = sleeps(listOf(1787546768L to 1787551261L, 1787556701L to 1787575851L)) + assertEquals(2, SleepStager.bridgeWakeGaps(aug24).size) + } +} From 24ce6583e7bfebe12260b20cf34795c9154ee2fd Mon Sep 17 00:00:00 2001 From: vishk23 <119831996+vishk23@users.noreply.github.com> Date: Mon, 24 Aug 2026 20:31:52 -0400 Subject: [PATCH 171/171] Sleep: correct the withdrawn-rescue history note after the decode investigation The MG motion/gravity decode investigation (2026-08-24) closed with the decode verified CLEAN: the alarming server-side "postureVar" is a per-axis RANGE (max-min of the unit gravity vector, in g), not a variance, so the "RMS |g| ~1.5 g during sleep, physically implausible" inference that motivated a decode-bug suspicion was invalid. More importantly, it showed the rescue's premise was itself partly wrong: on the flagship "dropped" night the strap's own band sleep_state read wake across the disputed hours WITH corroborating 70-75 bpm stretches and continuous posture change - likely genuine in-bed wakefulness the stillness spine was RIGHT to reject (the wearer's awake-resting HR sits in the low 50s, RHR ~44, so "HR 47-58 = unambiguous sleep" does not hold for them). Comment-only: rewrite the history note above wakeBridgeMaxMin (both platforms) to drop the "band state was fooled / HR-proven sleep" framing and record the decode-clean finding, so the next attempt at the open restless-night problem starts from the corrected evidence. No code change. Co-Authored-By: Claude Fable 5 --- .../Sources/StrandAnalytics/SleepStager.swift | 28 +++++++++++-------- .../java/com/noop/analytics/SleepStager.kt | 16 ++++++----- 2 files changed, 25 insertions(+), 19 deletions(-) diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift index e12c5c2b01..4a26c27a6d 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStager.swift @@ -213,18 +213,22 @@ public enum SleepStager { // HISTORY — an HR-led "rescue" of motion-vetoed runs was built beside this bridge and WITHDRAWN // after a 5-night real-data replay against the reporting wearer's server mirror (2026-08-24). // The rescue reclassified an "active" run as sleep when its median in-run HR sat inside the - // `confirmSleepWithHR` band, aiming at restless-but-asleep nights the stillness spine collapses - // to their quietest fragment (hours of HR 47–58 bpm scoring nothing). On the real wearer the - // discriminator failed: a low-HR wearer's quiet awake EVENING (median ~58 bpm vs a ~64 window - // baseline) also sits inside the ×1.05 band, so whole evenings were rescued and chained — the - // assembled run then either exceeded the H4 cap and dropped the night to ZERO, or squeaked - // under it and scored a 15 h "night" starting at 18:00. The strap's own banked band sleep_state - // could not arbitrate: it is motion-led too and read "wake" across the HR-proven sleep. Median - // HR LEVEL cannot separate that wearer's quiet wakefulness from sleep; only dip DEPTH - // (sleep-only minima 42–48 bpm) looked separable, and placing an onset BOUNDARY from dips needs - // a properly validated design (#194/#345). The restless-night DROPPING failure therefore - // remains OPEN — do not re-land an HR-band rescue without multi-night evidence that it refuses - // quiet awake evenings on low-HR wearers. + // `confirmSleepWithHR` band, on the premise that a motion-vetoed night with HR 47–58 bpm was + // restless-but-asleep. On the real wearer the discriminator failed: a low-HR wearer's quiet + // awake HR (median ~58 bpm, awake-resting low-50s, RHR ~44) also sits inside the ×1.05 band off + // the ~64 window baseline, so whole awake evenings were rescued and chained — the assembled run + // then either exceeded the H4 cap and dropped the night to ZERO, or squeaked under it and + // scored a 15 h "night" starting at 18:00. A follow-up decode investigation (2026-08-24) then + // verified the MG gravity decode CLEAN (unit-norm throughout; the alarming server "postureVar" + // is a per-axis RANGE, not a variance) and showed the premise itself was partly wrong: on the + // flagship "dropped" night the strap's own band state read wake across the disputed hours WITH + // corroborating 70–75 bpm stretches and continuous posture change — likely genuine in-bed + // wakefulness the stillness spine was RIGHT to reject. Median HR LEVEL cannot separate that + // wearer's quiet wakefulness from sleep; only dip DEPTH (sleep-only minima 42–48 bpm) looked + // separable, and placing an onset BOUNDARY from dips needs a properly validated design + // (#194/#345) plus per-night ground truth that HR alone cannot supply. Do not re-land an + // HR-band rescue without multi-night evidence that it refuses quiet awake stretches on low-HR + // wearers. /// Adjacent sleep runs separated by at most this many minutes are assembled into ONE candidate /// run (wake-bridge pass only). Below the selector-side `SleepStageTotals.gapBridgeMaxMin` (60) diff --git a/android/app/src/main/java/com/noop/analytics/SleepStager.kt b/android/app/src/main/java/com/noop/analytics/SleepStager.kt index c7a9a048e6..84a48116fd 100644 --- a/android/app/src/main/java/com/noop/analytics/SleepStager.kt +++ b/android/app/src/main/java/com/noop/analytics/SleepStager.kt @@ -604,13 +604,15 @@ object SleepStager { // // HISTORY — an HR-led "rescue" of motion-vetoed runs was built beside this bridge and WITHDRAWN // after a 5-night real-data replay against the reporting wearer's server mirror (2026-08-24): - // on a low-HR wearer the quiet awake EVENING median (~58 bpm vs a ~64 window baseline) also - // sits inside the ×1.05 sleep band, so whole evenings were rescued and chained — the assembled - // run then either exceeded the H4 cap (night dropped to ZERO) or squeaked under it (a 15 h - // "night" starting at 18:00). The strap's own banked band sleep_state could not arbitrate (it - // is motion-led too and read "wake" across the HR-proven sleep). The restless-night DROPPING - // failure remains OPEN — see the Swift history note above `wakeBridgeMaxMin` before - // re-attempting an HR-band rescue. Mirrors Swift. + // on a low-HR wearer the quiet awake HR (median ~58 bpm, awake-resting low-50s, RHR ~44) also + // sits inside the ×1.05 sleep band off the ~64 window baseline, so whole awake evenings were + // rescued and chained — the assembled run then either exceeded the H4 cap (night dropped to + // ZERO) or squeaked under it (a 15 h "night" starting at 18:00). A follow-up decode + // investigation then verified the MG gravity decode CLEAN and showed the premise was partly + // wrong: on the flagship "dropped" night the strap's band state read wake across the disputed + // hours with corroborating 70–75 bpm stretches — likely genuine in-bed wakefulness the + // stillness spine was RIGHT to reject. See the Swift history note above `wakeBridgeMaxMin` + // before re-attempting an HR-band rescue. Mirrors Swift. /** Adjacent sleep runs separated by at most this many minutes are assembled into ONE candidate * run (wake-bridge pass only). Below the selector-side `SleepStageTotals.gapBridgeMaxMin` (60)