diff --git a/CLAUDE.md b/CLAUDE.md index ba4fc498cd..953110e220 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -144,6 +144,13 @@ Swift, you MUST build the app yourself: `xcodebuild … build` locally, or run ` canonical resolver (`DeviceFamily.forRegistryModel` on both platforms), never a scattered string compare — the wizard stores `"4.0"`, other paths `"WHOOP 4.0"`, and single-spelling checks silently miss straps. Reads must thread the registry's **active** strap id, not a raw BLE address. +- **Sleep stage labels — same trap, same rule:** the tree carries TWO stage vocabularies on purpose — + segment `stage` strings canonicalise to `"wake"`, minutes-dictionary keys to `"awake"`. Imports never + pass through `SleepStagerV2` (Oura's phase table is `["deep","light","rem","awake"]`), so a segment + comparison written `stage != "wake"` counts those as SLEEP. Never compare a segment stage to a bare + literal — use `SleepStageVocabulary.isWake` (`StrandAnalytics`, + Kotlin twin) to compare, and + `SleepStageVocabulary.canonicalStage` to key by stage; both are in-memory only and must never rewrite + a stored string. (#979) - **Design system is law:** UI uses only design tokens — `StrandPalette` / `StrandFont` / shared components on Apple, `Palette` / `Metrics` on Android. No hardcoded colors, fonts, or spacing. - **Migrations:** add a versioned migration + a test; never mutate an existing migration. Watch for diff --git a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStageVocabulary.swift b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStageVocabulary.swift index 57f59ccfd3..6d97cfe2a6 100644 --- a/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStageVocabulary.swift +++ b/Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStageVocabulary.swift @@ -15,8 +15,10 @@ /// Six sites already defended with `case "wake", "awake"` while five did not, which is what makes this /// a missing shared rule rather than a missing idea. /// -/// A PREDICATE, deliberately, not a canonicaliser: it fixes the comparisons without rewriting any -/// stored string, so no persisted hypnogram changes meaning and neither vocabulary above moves. +/// The alias rule (which spellings mean wake) has ONE definition — `isWake`. `canonicalStage` folds +/// through it for the consumers that must KEY by stage rather than merely compare. Neither rewrites a +/// stored string: both operate on in-memory values, so no persisted hypnogram changes meaning and +/// neither vocabulary above moves. public enum SleepStageVocabulary { /// True for either spelling of the wake stage, ignoring case and surrounding whitespace. @@ -27,4 +29,21 @@ public enum SleepStageVocabulary { let s = stage.trimmingCharacters(in: .whitespacesAndNewlines).lowercased() return s == "wake" || s == "awake" } + + /// A SEGMENT stage label folded to the segment vocabulary's one spelling per class: trimmed, + /// lowercased, and wake in either spelling becomes `"wake"`. Unknown labels pass through (trimmed, + /// lowercased) rather than being invented into the vocabulary. + /// + /// For in-memory comparison and bucketing ONLY — dictionary keys, confusion-matrix classes, group + /// keys. Never write its output back to a store: a persisted hypnogram keeps the spelling its + /// producer chose, for the reasons in the type comment. + /// + /// The Kotlin twin of this rule is `com.noop.ui.canonicalStage`, which folds the SAME alias set + /// (via `SleepStageVocabulary.isWake`) toward `"awake"` instead, because its consumers key the + /// minutes/colour vocabulary. One alias rule, one canonical spelling per vocabulary — the fold + /// target is a property of the vocabulary being keyed, not of the rule. + public static func canonicalStage(_ stage: String) -> String { + let s = stage.trimmingCharacters(in: .whitespacesAndNewlines).lowercased() + return isWake(s) ? "wake" : s + } } diff --git a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStageVocabularyTests.swift b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStageVocabularyTests.swift index 85ca6a720a..8d98fff1eb 100644 --- a/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStageVocabularyTests.swift +++ b/Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStageVocabularyTests.swift @@ -57,6 +57,36 @@ final class SleepStageVocabularyTests: XCTestCase { XCTAssertEqual(wake, 900) } + /// `canonicalStage` folds BOTH wake spellings onto the segment vocabulary's `"wake"`, and only + /// tidies (trims, lowercases) everything else. Swift-only cases below this line: the Kotlin twin of + /// this rule is the UI's `canonicalStage`, which folds the same alias set toward `"awake"` because + /// its consumers key the minutes/colour vocabulary — same rule, that vocabulary's spelling. + func testCanonicalStageFoldsWakeSpellings() { + XCTAssertEqual(SleepStageVocabulary.canonicalStage("wake"), "wake") + XCTAssertEqual(SleepStageVocabulary.canonicalStage("awake"), "wake") + XCTAssertEqual(SleepStageVocabulary.canonicalStage(" AWAKE\t"), "wake") + XCTAssertEqual(SleepStageVocabulary.canonicalStage("Wake"), "wake") + } + + /// Non-wake labels are tidied, not renamed — the canonicaliser must not invent vocabulary. + func testCanonicalStagePassesOtherLabelsThroughTidied() { + XCTAssertEqual(SleepStageVocabulary.canonicalStage("deep"), "deep") + XCTAssertEqual(SleepStageVocabulary.canonicalStage(" Light "), "light") + XCTAssertEqual(SleepStageVocabulary.canonicalStage("REM"), "rem") + XCTAssertEqual(SleepStageVocabulary.canonicalStage("restless"), "restless") + XCTAssertEqual(SleepStageVocabulary.canonicalStage(""), "") + } + + /// The two APIs must never disagree about what wake is: for every label, folding to "wake" and + /// satisfying the predicate are the same fact. + func testCanonicalStageAgreesWithIsWake() { + for label in ["wake", "awake", "Awake", " WAKE ", "deep", "light", "rem", "restless", ""] { + XCTAssertEqual(SleepStageVocabulary.canonicalStage(label) == "wake", + SleepStageVocabulary.isWake(label), + "canonicalStage and isWake disagree on \(label)") + } + } + /// INTEGRATION, not the predicate. The tests above pass whether or not the five call sites were /// actually changed — they exercise the rule, not its users. This one exercises a real caller, so /// it is the test that fails if a site is reverted. diff --git a/Tools/SleepBench/README.md b/Tools/SleepBench/README.md new file mode 100644 index 0000000000..53f69a8638 --- /dev/null +++ b/Tools/SleepBench/README.md @@ -0,0 +1,154 @@ +# SleepBench — an offline scoring harness for the sleep stagers + +Replays `SleepStager` (V1) and `SleepStagerV2` over real recorded nights held in a NOOP SQLite +database and scores each hypnogram against every independent reference that database carries. It +exists so a claim about a staging recipe ("kappa went up", "wake is over-called") can be **measured** +rather than asserted. + +The harness is read-only (`SQLITE_OPEN_READONLY` + `immutable=1`), takes its database path as a +command-line argument, and never writes anything back. **No health data lives in this repository and +none may be committed.** + +## Running it + +```bash +cd Tools/SleepBench && swift build +``` + +```bash +.build/debug/sleepbench --db /path/to/COPY-of-whoop.sqlite --device my-whoop-noop --stream-device my-whoop +``` + +| Flag | Default | Meaning | +|---|---|---| +| `--db` | *(required)* | Path to a **copy** of a NOOP SQLite database | +| `--device` | `my-whoop-noop` | The `sleepSession.deviceId` whose nights are scored | +| `--stream-device` | `my-whoop` | The `deviceId` the raw streams are stored under | +| `--pad` | `3600` | Seconds of stream padding either side of each session | +| `--csv` | *(none)* | Also write a per-night CSV | + +**Pass a COPY. Never point this at a device's live database.** `--device` and `--stream-device` are +separate arguments because they genuinely differ in a real database: session rows carry the app's own +device identity while decoded streams carry the strap's. + +## What it reports + +| Section | Question it answers | +|---|---| +| **0. Ground-truth provenance** | How many rows are `userEdited`, and how many of those carry a `stagelock` cursor (i.e. human-authored *stages*, not just human-corrected *bounds*) | +| **A. Which stager produced the stored nights?** | Replays V1 and V2 over the identical window and compares epoch-for-epoch against `stagesJSON`, identifying the live recipe | +| **B. Accuracy vs the wearer's manual restages** | Wake-minute error and 4-class / sleep-wake agreement against reference (a) | +| **C. Agreement with the strap's own band `sleep_state`** | Sleep-vs-wake agreement against reference (b), the v18 `@81` high nibble — the strap's own verdict, not a re-derivation of NOOP's. **In-session domain**: epochs inside stored session bounds only, so its kappa answers "how good is the wake detection inside a night already found" and is **not comparable to a whole-record kappa** ("how good is sleep/wake overall") quoted from anywhere else | +| **D. Does the wake over-call track overnight HR?** | Tests the standing "supplement HR" inference, both on human-labelled nights and on the far larger band-disputed-wake set | +| **E. Per-stage fraction calibration** | The guard kappa does **not** provide — how much of the night each recipe spends at each stage, as a signed bias in percentage points | +| **F. First-REM latency** | A stage-*timing* property kappa is also blind to | +| **G. Per-night detail** | Every session, one row each | + +Section E exists because of a specific failure. PR #348 fitted the stager to DREAMT, raised kappa on +all three benchmarks with a held-out gap of −0.027, and was reverted 48 hours later by PR #437 for +re-scoring a healthy night from 6% to 23% awake. The revert's own words: *"kappa doesn't guard +stage-fraction calibration."* + +## The two references, and what each is worth + +**(a) The wearer's manual restages.** `userEdited = 1` rows. A `userEdited` row alone does **not** +prove human stage labels: the local editor corrects bed/wake *bounds* and then re-derives the +hypnogram from raw, so its stages are machine output over a human-chosen window. Only a row carrying +a `stagelock::` cursor had its stages authored directly, via the cloud +`edit_sleep_stages` path. Section B additionally excludes an edited night whose stored hypnogram is a +byte-exact V2 replay — scoring V2 against a row V2 itself wrote is self-comparison. + +Section E deliberately uses the `stagelock` set rather than section B's set: B's exclusion is +version-dependent by construction, so a night can enter or leave it when the recipe changes, silently +swapping the denominator underneath a before/after comparison. The `stagelock` set comes from +`cursors` and does not move when the recipe moves. + +**(b) The strap's own band `sleep_state`.** Carries no light/deep/REM, so it is a sleep-vs-wake +comparison only. STRICT maps band 2 → sleep and band 0/3 → wake, and **excludes** band 1 ("still", +genuinely ambiguous). + +## Stage label vocabulary — read this before touching a comparison + +The tree carries **two** stage vocabularies on purpose, documented on `SleepStageVocabulary` +(`Packages/StrandAnalytics`): + +- **Segment `stage` strings** (`StageSegment.stage`, hypnogram rows) canonicalise to `"wake"`. + `SleepStagerV2` models its own states as `"awake"` internally and renames on the way out. +- **Minutes-dictionary keys** (`SleepStageTotals`, `SleepWindowReclip`) canonicalise to `"awake"`. + +The bug is the dictionary vocabulary reaching a **segment** comparison. Imports do not pass through +`SleepStagerV2`: Oura's phase table is `["deep","light","rem","awake"]`, generic wearable JSON carries +whatever the source app wrote, and the `edit_sleep_stages` vocabulary is `"awake"` too — so `"awake"` +can sit on exactly the stage-locked rows this harness uses as its human reference. + +Issue #979 fixed eleven such comparisons across the app. **It did not touch this harness**, which went +on comparing bare literals — so every `"awake"` epoch was scored as **sleep**, biasing wake down and +sleep up in precisely the quantities the harness exists to measure. Worse, an `"awake"` label matched +no class in `stageOrder`, so `Confusion.add` **dropped** those epochs: they left the denominator +instead of showing up as disagreement. + +Measured on one real 564-minute stage-locked night (119.5 reference wake minutes), stored under both +spellings with identical streams: + +``` + "awake" "wake" + truth wake minutes 0 -> 120 + V2 wake-minute error 0 -> -119.5 (sign flip) + E.1 wake stage bias 0.00 pp -> -21.19 pp + E.0 4-class kappa 0.145 -> 0.091 + E.0 sleep/wake accuracy 100.0% -> 78.8% + E.0 epochs scored 889 -> 1128 (239 epochs had been dropped) + C stored kappa 0.000 -> 0.029 + F human first-REM latency 138.0 -> 131.5 min +``` + +The E.1 line is the pointed one: the section added to *guard* stage-fraction calibration reported a +wake bias of **0.00 pp** where the true bias is **−21.2 pp** — larger in magnitude than the +17 pp +blowout that forced #437's revert. + +**The rule:** never compare a segment stage against a bare literal. Use +`SleepStageVocabulary.isWake(_:)` to compare (it folds both spellings and any case), and +`SleepStageVocabulary.canonicalStage(_:)` where a consumer must *key* by stage. Both operate on +in-memory values only — neither may rewrite a stored string, because both vocabularies above are live +and moving either would change what a persisted hypnogram means. + +Inside this harness, both are aliased in `Metrics.swift` (`isWake`, `bucketLabel`) and every comparison +goes through them. `bucketLabel` folds a label onto one spelling per class for the metrics that key by +stage (`stageMinutes`, `stagePercentages`, `Confusion`), applied once at the DB load point (`DB.swift`) +and once at epoch expansion (`epochLabels`). That is the use the canonicaliser permits: sleepbench opens +the database read-only and those arrays live only for the duration of a run. + +## Testing + +```bash +cd Tools/SleepBench && swift test +``` + +The scoring primitives are pure functions over 30-second label arrays, so the suite needs **no +database, no health data and no `--db` argument**. + +> **CI does not cover this directory.** `swift-packages.yml` runs on `Packages/**` only, and +> `app-build.yml` is disabled by default — nothing under `Tools/` is built or tested by any default CI +> job. That is exactly how #979 could fix eleven call sites in the app and leave this harness broken +> with every check green. If you change anything here, run `swift test` yourself. + +## Known limitations + +- **Imported nights are not scorable.** `stagesJSON` has two shapes: the segment array + `[{start,end,stage}]` (on-device stager, Oura, Fitbit, cloud stage edits) and the minute dict + `{"awake":N,"light":N,"deep":N,"rem":N}` (Apple Health / WHOOP CSV / Xiaomi importers). The harness + decodes only the former; the dict shape currently decodes to `[]` and the night is then scored as + **100% wake**. Latent for the default `--device my-whoop-noop`, since imported nights carry a + different `deviceId` — but `sleepbench --device apple-health` would report a confidently wrong + all-wake corpus with no warning. +- **Section D is underpowered on human-labelled nights**, which span only a narrow HR band by + construction. The band-disputed-wake test in the same section needs no human labels and covers a far + wider HR range; prefer it. +- **`bandVetoMirror` is a transcription**, not the shipped code. `SleepStager.applyBandStateWakeVeto` + is `internal` and unreachable from this executable. `SleepStagerTests` pins the real implementation. + +## Related + +- `docs/ANALYTICS.md` — what the stagers actually compute +- `Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStageVocabulary.swift` — the shared predicate (#979) +- `Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStageTotals.swift` — both `stagesJSON` shapes diff --git a/Tools/SleepBench/Sources/sleepbench/DB.swift b/Tools/SleepBench/Sources/sleepbench/DB.swift index 3cdcf8c79b..2056c1673d 100644 --- a/Tools/SleepBench/Sources/sleepbench/DB.swift +++ b/Tools/SleepBench/Sources/sleepbench/DB.swift @@ -83,8 +83,15 @@ extension ReadOnlyDB { """ try query(sql) { s in let stagesJSON = ReadOnlyDB.str(s, 5) ?? "[]" - let stages = (try? JSONDecoder().decode([StageSegment].self, - from: Data(stagesJSON.utf8))) ?? [] + // This column has several producers and they do NOT agree on how to spell wake: the on-device + // stager writes "wake", while Oura's phase table, generic wearable JSON and the + // `edit_sleep_stages` vocabulary write "awake". Fold to one spelling HERE, at the one point + // rows enter the harness, so no downstream comparison has to know which producer wrote the + // row. Safe because sleepbench is read-only and never writes a folded string back — the use + // `SleepStageVocabulary.canonicalStage` permits. + let stages = ((try? JSONDecoder().decode([StageSegment].self, + from: Data(stagesJSON.utf8))) ?? []) + .map { StageSegment(start: $0.start, end: $0.end, stage: bucketLabel($0.stage)) } var band: [Int] = [] if let bj = ReadOnlyDB.str(s, 6) { band = (try? JSONDecoder().decode([Int].self, from: Data(bj.utf8))) ?? [] diff --git a/Tools/SleepBench/Sources/sleepbench/Metrics.swift b/Tools/SleepBench/Sources/sleepbench/Metrics.swift index dd2af6de99..1f2f2ce6ed 100644 --- a/Tools/SleepBench/Sources/sleepbench/Metrics.swift +++ b/Tools/SleepBench/Sources/sleepbench/Metrics.swift @@ -2,8 +2,27 @@ import Foundation import StrandAnalytics /// The four scored classes, in a fixed order so every confusion matrix in the run is indexed alike. +/// These are the SEGMENT vocabulary's spellings, so wake is `"wake"` here — see `SleepStageVocabulary`. let stageOrder = ["wake", "light", "deep", "rem"] +/// Wake in either spelling, any case — `SleepStageVocabulary.isWake` under a short local name, because +/// every stage comparison in this harness goes through it rather than a bare `"wake"` literal. +/// +/// #979 fixed eleven segment comparisons in the app but not this harness, which still compared literals. +/// A hypnogram spelling wake `"awake"` (Oura's phase table, generic wearable JSON, or a stage-edited +/// night — the `edit_sleep_stages` vocabulary is `"awake"`, and its `stagelock` rows are section E's +/// human reference) had every wake epoch scored as SLEEP — biasing wake down and sleep up in precisely +/// the quantities this harness exists to measure. +func isWake(_ label: String) -> Bool { SleepStageVocabulary.isWake(label) } + +/// The label under which an epoch is BUCKETED by the metrics below (`stageMinutes`, `stagePercentages`, +/// `Confusion`), which key by stage and so need one spelling per class. +/// +/// `SleepStageVocabulary.canonicalStage` under a short local name. Nothing here is persisted — sleepbench +/// opens the database read-only and these arrays last one run — which is exactly the use the shared +/// canonicaliser permits: fold for comparison and bucketing, never write the folded string back. +func bucketLabel(_ label: String) -> String { SleepStageVocabulary.canonicalStage(label) } + /// 30 s epochs, matching `SleepStager.epochS` and the grid `stagesJSON` / `sleepStateJSON` are stored on. let epochSeconds = 30.0 @@ -24,28 +43,32 @@ func epochLabels(_ stages: [StageSegment], start: Int, end: Int) -> [String] { let t = start + Int(Double(i) * epochSeconds) while si + 1 < sorted.count && sorted[si].end <= t { si += 1 } let seg = sorted[si] - if seg.start <= t && t < seg.end { out[i] = seg.stage } - else if let hit = sorted.first(where: { $0.start <= t && t < $0.end }) { out[i] = hit.stage } - else if t >= sorted[sorted.count - 1].end { out[i] = sorted[sorted.count - 1].stage } + // Bucketed on expansion: this is the one place a `[StageSegment]` becomes a label array, so every + // label array downstream carries one spelling per class whichever producer wrote the row. + if seg.start <= t && t < seg.end { out[i] = bucketLabel(seg.stage) } + else if let hit = sorted.first(where: { $0.start <= t && t < $0.end }) { out[i] = bucketLabel(hit.stage) } + else if t >= sorted[sorted.count - 1].end { out[i] = bucketLabel(sorted[sorted.count - 1].stage) } } return out } -/// Minutes at each stage in a label array. +/// Minutes at each stage in a label array, keyed by BUCKET — so a night spelling wake `"awake"` adds to +/// the same bucket as one spelling it `"wake"` rather than opening a fifth key no caller reads. func stageMinutes(_ labels: [String]) -> [String: Double] { var m: [String: Double] = [:] - for l in labels { m[l, default: 0] += epochSeconds / 60.0 } + for l in labels { m[bucketLabel(l), default: 0] += epochSeconds / 60.0 } return m } func wakeMinutes(_ labels: [String]) -> Double { stageMinutes(labels)["wake"] ?? 0 } +/// Sleep is everything that is not wake — sound only while the label set is closed, which the tests pin. func sleepMinutes(_ labels: [String]) -> Double { - Double(labels.filter { $0 != "wake" }.count) * epochSeconds / 60.0 + Double(labels.filter { !isWake($0) }.count) * epochSeconds / 60.0 } /// Index of the first and last non-wake epoch — sleep onset and final wake on this grid. func onsetAndFinalWake(_ labels: [String]) -> (onset: Int?, final: Int?) { - (labels.firstIndex(where: { $0 != "wake" }), labels.lastIndex(where: { $0 != "wake" })) + (labels.firstIndex(where: { !isWake($0) }), labels.lastIndex(where: { !isWake($0) })) } // MARK: - Stage-fraction calibration @@ -62,7 +85,7 @@ func onsetAndFinalWake(_ labels: [String]) -> (onset: Int?, final: Int?) { func stagePercentages(_ labels: [String]) -> [String: Double] { var pct = [String: Double](uniqueKeysWithValues: stageOrder.map { ($0, 0.0) }) guard !labels.isEmpty else { return pct } - for l in labels { pct[l, default: 0] += 1 } + for l in labels { pct[bucketLabel(l), default: 0] += 1 } for k in pct.keys { pct[k]! = pct[k]! / Double(labels.count) * 100 } return pct } @@ -82,8 +105,8 @@ func stageBias(ref: [String], pred: [String]) -> [String: Double] { /// REM before onset cannot occur (REM is not wake, so onset is at or before it), but the guard is kept so /// a caller passing a hand-built label array cannot produce a negative latency. func firstRemLatencyMinutes(_ labels: [String]) -> Double? { - guard let onset = labels.firstIndex(where: { $0 != "wake" }), - let rem = labels.firstIndex(of: "rem"), rem >= onset else { return nil } + guard let onset = labels.firstIndex(where: { !isWake($0) }), + let rem = labels.firstIndex(where: { bucketLabel($0) == "rem" }), rem >= onset else { return nil } return Double(rem - onset) * epochSeconds / 60.0 } @@ -97,8 +120,12 @@ struct Confusion { self.classes = classes m = Array(repeating: Array(repeating: 0, count: classes.count), count: classes.count) } + /// Labels are bucketed before lookup. Without it an `"awake"` reference epoch matches no class and is + /// silently DROPPED from the matrix — not misfiled but absent, quietly shrinking the denominator + /// behind every accuracy and kappa in the run rather than showing up as disagreement. mutating func add(ref: String, pred: String) { - guard let r = classes.firstIndex(of: ref), let c = classes.firstIndex(of: pred) else { return } + guard let r = classes.firstIndex(of: bucketLabel(ref)), + let c = classes.firstIndex(of: bucketLabel(pred)) else { return } m[r][c] += 1 } mutating func merge(_ o: Confusion) { @@ -148,7 +175,9 @@ func confusion(ref: [String], pred: [String], classes: [String] = stageOrder) -> } /// Collapse a 4-class hypnogram to the 2-class sleep/wake problem the wake over-call lives in. -func toSleepWake(_ labels: [String]) -> [String] { labels.map { $0 == "wake" ? "wake" : "sleep" } } +/// Wake is matched in either spelling: a bare `== "wake"` here folded every `"awake"` epoch into "sleep", +/// producing a reference that claims the wearer never woke. +func toSleepWake(_ labels: [String]) -> [String] { labels.map { isWake($0) ? "wake" : "sleep" } } // MARK: - Descriptive statistics diff --git a/Tools/SleepBench/Sources/sleepbench/main.swift b/Tools/SleepBench/Sources/sleepbench/main.swift index 1207006723..5e3887d086 100644 --- a/Tools/SleepBench/Sources/sleepbench/main.swift +++ b/Tools/SleepBench/Sources/sleepbench/main.swift @@ -91,10 +91,10 @@ func nightLabel(_ ts: Int) -> String { /// veto's effect can be scored on the isolated staging path as well as end-to-end. func bandVetoMirror(_ labels: [String], bandEpochs: [Int]) -> [String] { guard !bandEpochs.isEmpty, labels.count == bandEpochs.count else { return labels } - guard let onset = labels.firstIndex(where: { $0 != "wake" }), - let final = labels.lastIndex(where: { $0 != "wake" }), onset <= final else { return labels } + guard let onset = labels.firstIndex(where: { !isWake($0) }), + let final = labels.lastIndex(where: { !isWake($0) }), onset <= final else { return labels } var out = labels - for i in onset...final where out[i] == "wake" && bandEpochs[i] == SleepStager.bandStateAsleep { + for i in onset...final where isWake(out[i]) && bandEpochs[i] == SleepStager.bandStateAsleep { out[i] = "light" } return out @@ -285,7 +285,7 @@ for (name, get) in [("V1", { (n: Night) in n.v1 }), ("V2", { (n: Night) in n.v2 for nt in banded { let pred = get(nt) for (i, b) in nt.band.enumerated() where i < pred.count { - if pred[i] == "wake" { + if isWake(pred[i]) { wakeEpochs += 1 if b == SleepStager.bandStateAsleep { disputed += 1 } } else { @@ -339,7 +339,7 @@ var bhr: [Double] = [], bdisp: [Double] = [] for nt in banded.sorted(by: { $0.meanHR < $1.meanHR }) { var wake = 0, disp = 0 for (i, b) in nt.band.enumerated() where i < nt.v2.count { - if nt.v2[i] == "wake" { wake += 1; if b == SleepStager.bandStateAsleep { disp += 1 } } + if isWake(nt.v2[i]) { wake += 1; if b == SleepStager.bandStateAsleep { disp += 1 } } } guard wake >= 10, !nt.meanHR.isNaN else { continue } let pct = Double(disp) / Double(wake) * 100 diff --git a/Tools/SleepBench/Tests/sleepbenchTests/StageVocabularyTests.swift b/Tools/SleepBench/Tests/sleepbenchTests/StageVocabularyTests.swift new file mode 100644 index 0000000000..29cd53f388 --- /dev/null +++ b/Tools/SleepBench/Tests/sleepbenchTests/StageVocabularyTests.swift @@ -0,0 +1,165 @@ +import StrandAnalytics +import XCTest + +@testable import sleepbench + +/// #979 fixed eleven segment comparisons in the app but left this harness comparing bare `"wake"` +/// literals, so a hypnogram spelling wake `"awake"` — Oura's phase table, generic wearable JSON, or a +/// stage-edited night's `stagelock` rows — had every wake epoch scored as SLEEP. That biases wake +/// DOWN and sleep UP in exactly the metrics the project has repeatedly re-measured and got wrong signs +/// on: wake-minute error, wake over-call, and sleep/wake kappa against the strap band. +/// +/// Every test here scores the SAME night written two ways and demands the two agree. No database, no +/// health data, no `--db` argument. +final class StageVocabularyTests: XCTestCase { + + private func run(_ stage: String, _ n: Int) -> [String] { [String](repeating: stage, count: n) } + + /// One night, two spellings: 40 epochs wake, 100 light, 20 deep, 20 rem, then 20 more wake. + private var deviceSpelling: [String] { + run("wake", 40) + run("light", 100) + run("deep", 20) + run("rem", 20) + run("wake", 20) + } + private var importSpelling: [String] { + run("awake", 40) + run("light", 100) + run("deep", 20) + run("rem", 20) + run("awake", 20) + } + + // MARK: - The vocabulary itself + + /// The harness classifies by "everything that is not wake is sleep", which is sound only while the + /// scored vocabulary is these four. A fifth label must break this test, not land in the sleep bucket. + func testStageOrderIsTheClosedScoredSet() { + XCTAssertEqual(stageOrder, ["wake", "light", "deep", "rem"]) + XCTAssertFalse(stageOrder.contains("awake"), "\"awake\" is an input spelling, never a scored class") + for s in stageOrder { XCTAssertEqual(bucketLabel(s), s, "\(s) must be its own bucket") } + } + + func testIsWakeAcceptsBothSpellingsAndRejectsSleepStages() { + for w in ["wake", "awake", "Awake", " WAKE "] { XCTAssertTrue(isWake(w), "\(w) should read as wake") } + for s in ["light", "deep", "rem"] { XCTAssertFalse(isWake(s)) } + } + + /// The harness bucket must agree with the shared predicate — one rule, not two. + func testBucketLabelAgreesWithTheSharedPredicate() { + for s in ["wake", "awake", "Awake", "light", "deep", "rem", "restless"] { + XCTAssertEqual(bucketLabel(s) == "wake", SleepStageVocabulary.isWake(s), "disagreement on \(s)") + } + XCTAssertEqual(bucketLabel("awake"), "wake") + XCTAssertEqual(bucketLabel("restless"), "restless", "an unknown label stays visible, not coerced") + } + + // MARK: - The two spellings must score identically + + func testWakeMinutesAreIdenticalAcrossSpellings() { + XCTAssertEqual(wakeMinutes(importSpelling), wakeMinutes(deviceSpelling), accuracy: 1e-9) + // 60 wake epochs x 30 s = 30 min. The pre-fix harness returned 0 for the "awake" spelling. + XCTAssertEqual(wakeMinutes(importSpelling), 30.0, accuracy: 1e-9) + } + + func testSleepMinutesAreIdenticalAcrossSpellings() { + XCTAssertEqual(sleepMinutes(importSpelling), sleepMinutes(deviceSpelling), accuracy: 1e-9) + // 140 sleep epochs x 30 s = 70 min. Pre-fix, all 200 epochs (100 min) counted as sleep. + XCTAssertEqual(sleepMinutes(importSpelling), 70.0, accuracy: 1e-9) + } + + func testToSleepWakeIsIdenticalAcrossSpellings() { + XCTAssertEqual(toSleepWake(importSpelling), toSleepWake(deviceSpelling)) + // Pre-fix this mapped every "awake" epoch to "sleep", producing a reference that claims the wearer + // never woke — which is what section C's sleep/wake kappa was then scored against. + XCTAssertEqual(toSleepWake(importSpelling).filter { $0 == "wake" }.count, 60) + } + + func testOnsetAndFinalWakeAreIdenticalAcrossSpellings() { + let imported = onsetAndFinalWake(importSpelling), device = onsetAndFinalWake(deviceSpelling) + XCTAssertEqual(imported.onset, device.onset) + XCTAssertEqual(imported.final, device.final) + // Onset is the first sleep epoch (40), final wake the last (179). Pre-fix onset landed on index 0 — + // the first "awake" epoch — collapsing the whole 20 min sleep latency to zero. + XCTAssertEqual(imported.onset, 40) + XCTAssertEqual(imported.final, 179) + } + + func testStagePercentagesAndBiasAreIdenticalAcrossSpellings() { + let imported = stagePercentages(importSpelling), device = stagePercentages(deviceSpelling) + for s in stageOrder { XCTAssertEqual(imported[s]!, device[s]!, accuracy: 1e-9, "stage \(s)") } + XCTAssertEqual(imported["wake"]!, 30.0, accuracy: 1e-9) // 60 of 200 epochs + // Pre-fix, "awake" opened a fifth key no caller reads while "wake" stayed 0 — so a night the human + // scored 30% awake read as a 30 pp UNDER-call of wake against any replay that got it right. + XCTAssertNil(imported["awake"]) + + // A recipe scored against the two spellings of the same reference must get the same bias. + let pred = run("wake", 60) + run("light", 140) + let bImported = stageBias(ref: importSpelling, pred: pred) + let bDevice = stageBias(ref: deviceSpelling, pred: pred) + for s in stageOrder { XCTAssertEqual(bImported[s]!, bDevice[s]!, accuracy: 1e-9, "bias \(s)") } + XCTAssertEqual(bImported["wake"]!, 0.0, accuracy: 1e-9, "same wake fraction ⇒ zero bias") + } + + func testFirstRemLatencyIsIdenticalAcrossSpellings() { + XCTAssertEqual(firstRemLatencyMinutes(importSpelling)!, + firstRemLatencyMinutes(deviceSpelling)!, accuracy: 1e-9) + // Onset at epoch 40, REM at 160 ⇒ 120 epochs x 30 s = 60 min. Pre-fix the "awake" spelling put + // onset at epoch 0 and reported 80 min. + XCTAssertEqual(firstRemLatencyMinutes(importSpelling)!, 60.0, accuracy: 1e-9) + } + + // MARK: - Agreement scoring + + /// The nastiest pre-fix failure mode: an `"awake"` reference epoch matched no class in `stageOrder`, + /// so `Confusion.add` dropped it. The epochs did not go to the wrong cell — they left the matrix, + /// silently shrinking the denominator under every accuracy and kappa in the run. + func testConfusionKeepsEveryEpochRegardlessOfSpelling() { + let pred = deviceSpelling + let imported = confusion(ref: importSpelling, pred: pred) + let device = confusion(ref: deviceSpelling, pred: pred) + XCTAssertEqual(imported.total, 200, "no epoch may be dropped for spelling wake \"awake\"") + XCTAssertEqual(imported.total, device.total) + XCTAssertEqual(imported.m, device.m) + XCTAssertEqual(imported.accuracy, 1.0, accuracy: 1e-9) + } + + /// Section C's kappa is computed on the 2-class collapse, so the fold decides it outright. + func testSleepWakeKappaIsIdenticalAcrossSpellings() { + let pred = toSleepWake(deviceSpelling) + let imported = confusion(ref: toSleepWake(importSpelling), pred: pred, classes: ["wake", "sleep"]) + let device = confusion(ref: toSleepWake(deviceSpelling), pred: pred, classes: ["wake", "sleep"]) + XCTAssertEqual(imported.kappa, device.kappa, accuracy: 1e-9) + XCTAssertEqual(imported.kappa, 1.0, accuracy: 1e-9, "a hypnogram scored against itself is perfect") + } + + // MARK: - The load path + + /// The fold has to happen on the way in from `StageSegment`, not only in the predicates — this is the + /// shape the harness actually reads out of `stagesJSON`. + func testEpochLabelsFoldImportSpelledSegments() { + let start = 1_700_000_000 + let imported = [ + StageSegment(start: start, end: start + 1800, stage: "awake"), + StageSegment(start: start + 1800, end: start + 3600, stage: "light"), + StageSegment(start: start + 3600, end: start + 5400, stage: "rem"), + ] + let device = imported.map { + StageSegment(start: $0.start, end: $0.end, stage: $0.stage == "awake" ? "wake" : $0.stage) + } + let li = epochLabels(imported, start: start, end: start + 5400) + let ld = epochLabels(device, start: start, end: start + 5400) + XCTAssertEqual(li, ld) + XCTAssertFalse(li.contains("awake"), "expansion must emit one spelling per class") + XCTAssertEqual(wakeMinutes(li), 30.0, accuracy: 1e-9) + XCTAssertEqual(firstRemLatencyMinutes(li)!, 30.0, accuracy: 1e-9) + } + + /// A JSON round-trip through the exact `[{start,end,stage}]` shape an import writes, pinning that the + /// decode path the harness uses carries the alias all the way to a scored metric. + func testImportSpelledStagesJSONDecodesToTheSameMetrics() throws { + let start = 1_700_000_000 + let json = """ + [{"start":\(start),"end":\(start + 1800),"stage":"awake"}, + {"start":\(start + 1800),"end":\(start + 5400),"stage":"light"}] + """ + let segs = try JSONDecoder().decode([StageSegment].self, from: Data(json.utf8)) + XCTAssertEqual(segs.first?.stage, "awake", "the decoder itself must not silently rewrite the label") + let labels = epochLabels(segs, start: start, end: start + 5400) + XCTAssertEqual(wakeMinutes(labels), 30.0, accuracy: 1e-9) + XCTAssertEqual(sleepMinutes(labels), 60.0, accuracy: 1e-9) + } +} diff --git a/docs/ANALYTICS.md b/docs/ANALYTICS.md index 81e7d8a821..6e7a75bfcd 100644 --- a/docs/ANALYTICS.md +++ b/docs/ANALYTICS.md @@ -286,6 +286,20 @@ Consecutive same-stage epochs are merged into `StageSegment`s tiling `[start, en - `SleepSession` — `start`, `end`, `efficiency` (AASM `asleep / in-bed`, where `asleep = in-bed − wake`), `stages`, per-session `restingHR` (lowest 5-min rolling-mean HR) and `avgHRV` (mean RMSSD over 5-min tumbling windows). - `hypnogramMetrics(_:)` — AASM-style roll-up: TIB / TST / SPT / SOL / REM latency / WASO / efficiency / disturbances, plus deep/REM/light minutes and percentages. +### Stage label vocabulary — wake has two spellings + +The tree carries **two** stage vocabularies, deliberately (see `SleepStageVocabulary`): **segment `stage` strings** (`StageSegment.stage`, hypnogram rows) canonicalise to `"wake"` — `SleepStagerV2` models its own states as `"awake"` internally and renames on the way out — while **minutes-dictionary keys** (`SleepStageTotals`, `SleepWindowReclip`) canonicalise to `"awake"`. + +The bug this closes is the dictionary vocabulary reaching a **segment** comparison. Imports never pass through `SleepStagerV2`: Oura's phase table is `["deep","light","rem","awake"]`, generic wearable JSON carries whatever the source app wrote, and the `edit_sleep_stages` vocabulary is `"awake"` too. A consumer written `stage == "wake"` silently misfiles those segments, and `stage != "wake"` — used to mean "asleep" — counts them as **sleep**, inflating efficiency and under-counting wake. + +Use `SleepStageVocabulary.isWake(_:)` to **compare** any segment stage string, and `SleepStageVocabulary.canonicalStage(_:)` where a consumer must **key** by stage (dictionary buckets, confusion-matrix classes) — it folds either wake spelling onto the segment vocabulary's `"wake"` and tidies everything else. Both operate on in-memory values only: never write `canonicalStage`'s output back to a store, so neither vocabulary moves and no persisted hypnogram changes meaning. Minutes dictionaries are keyed `"awake"` by construction and need neither. Issue #979 converted eleven comparisons; Android's UI-layer `canonicalStage` folds the same alias set (via the Kotlin twin's `isWake`) toward `"awake"`, because its consumers key the minutes/colour vocabulary — one alias rule, one canonical spelling per vocabulary. + +### Measuring the stagers — `Tools/SleepBench` + +`Tools/SleepBench` replays both stagers over real recorded nights and scores them against every independent reference a database carries: the wearer's manual restages (`userEdited = 1`, and specifically the `stagelock`-cursor subset whose *stages* a human authored), and the strap's own band `sleep_state`. It reports wake-minute error, 4-class and sleep/wake agreement, **per-stage fraction calibration** and first-REM latency — the last two because Cohen's kappa constrains neither how much of the night a recipe spends at each stage nor when it places it. That gap is not hypothetical: PR #348 raised kappa on all three benchmarks and was reverted 48 hours later by PR #437 for re-scoring a healthy night from 6% to 23% awake. + +Nothing under `Tools/` is built or tested by any default CI job, which is how #979 could fix eleven call sites in the app while leaving the harness comparing bare `"wake"` literals with every check green — on a night whose reference spells wake `"awake"`, that reported a wake-calibration bias of 0.00 pp where the truth was −21.2 pp, and dropped 239 reference epochs out of the confusion matrix entirely. Run `swift test` there yourself when you change it. See [`Tools/SleepBench/README.md`](../Tools/SleepBench/README.md). + ### Motion-corroborated wake — elevated-but-motionless HR is not wake (default ON) Both stagers (`SleepStager` V1 and the default `SleepStagerV2`) and the HR-led session confirmation (`confirmSleepWithHR`) previously called **wake** primarily off HR / HR-variability with no motion or posture cross-check. On a night whose resting HR is held elevated **without the wearer getting up** — a supplement protocol, a fever, a hot room, alcohol — that logic scored hot-but-motionless sleep as wake, over-calling WASO, mis-placing onset, and tanking efficiency and Rest. Two confirmed nights required manual relabeling (2026-07-13: 194 min WAKE vs ~67; 2026-07-14: onset 1:41 vs ~1:29 plus a 44-min WAKE block).