Repository navigation
sleep: lower the deep base prior to 0.15 on PSG evidence - #2576
Conversation
The variant table pools 26 773 epochs, so a few long nights can carry a row. Section 7 (--section priors) scores each of the 31 subjects once: their own kappa and how far each stage's share of their night is from their own PSG, for the shipped recipe, ryanbr#348-A's deep prior alone, its awake prior alone, and both, and counts the subjects each one improves. On sleep-accel the deep prior alone raises kappa for 21 of 31 subjects and cuts |deep bias| for 19 of 31 without moving wake or REM; the awake prior alone improves wake for 20 but worsens the mean |wake bias|, the ryanbr#437 shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SleepStagerV2's deep log-prior was log(0.18). Against human-scored PSG (PhysioNet sleep-accel, n = 31, Tools/SleepPSG) the shipped recipe over-called deep by +5.2 pp of the night pooled (18.94 % against 13.76 %). 0.15 is ryanbr#348's value, not a fit to this cohort. Alone it moves no wake or REM epoch and, pooled: kappa 0.363 -> 0.371, accuracy 59.67 -> 61.14 %, deep 18.94 -> 15.09 % (truth 13.76), light 50.47 -> 54.33 % (truth 55.19); deep F1 0.498 -> 0.484 (precision 0.430 -> 0.463, recall 0.592 -> 0.507). Per subject (section 7): kappa higher for 21 of 31, |deep bias| lower for 19 of 31. ryanbr#348's awake half is not taken: it fixes the pooled wake share but worsens the mean per-subject wake error, the ryanbr#437 shape. Swift and Kotlin move together and pin the priors in a test each; RecipeConfig.shipped follows (port validation), the variant list keeps the old value as "deep prior 0.18 (before)", and the README's shipped figures and section 7 are updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
), ryanbr#2569 merged, build 769113c; rescore-profile tools; STATE rewritten for a fresh session
|
One data point from the Oura side, since it lands in the regime the PR names as unmeasured: R-R present, gravity absent. Where this constant reaches an Oura night. A ring night with its own SleepNet hypnogram is not staged by What was run. 60 ring nights (2026-07-28 → 09-27, one wearer, each ≥ 4 h with a stored ring hypnogram and banked R-R), each re-staged with
Same direction and the same precision-for-recall shape as the PSG result, and no loss in how well deep tracks night to night. On this path it still leaves deep over-called (+6.3 pp mean signed bias; below the ring on 8 of 60 nights), so 0.15 does not overshoot here. REM moved on 5 of 60 nights by a few epochs, wake on none. Two things this regime shows that sleep-accel cannot, neither of them for this PR:
Nothing here changes the PR; it adds one no-gravity, R-R-present cohort to the "what this does not establish" list, pointing the same way. |
|
@UtkuDenizAltiok Reviewed. 17/17 green. This is careful work and the method is the part worth saying out loud. I checked the load-bearing claim rather than taking it. "0.15 is #348's value, not a fit to this cohort" holds, and it is stronger than the sentence suggests: #348 proposed The part that makes this safe is what you left out. #437 reverted all seven of #348's components 48 hours after they shipped, because one healthy night went from 6 % to 23 % awake. The component responsible was the awake prior, 0.10 → 0.34. You excluded exactly that one, and excluded it on the evidence that would have justified taking it: the pooled wake share moves to truth (4.15 % → 9.39 % against 9.07 %) while the mean per-subject wake error gets worse (5.50 → 5.55 pp). That is #437's own lesson, "kappa doesn't guard stage-fraction calibration", turned on the change that would have repeated it. Reporting per subject with every subject counted once, rather than pooled, is what let you see it. Also verified:
Two notes, neither blocking, neither needing a code change here. A visible discontinuity is coming. Walch is not a fresh held-out set. #348 already used it as one of its out-of-sample checks and reported 0.356 → 0.362 there. Your numbers are consistent with that and the per-subject breakdown is genuinely new, but the cohort is not independent of the reporting that first justified 0.15. The strongest version of this evidence would be a cohort neither #348 nor #987 has touched. Given the value itself is unchanged prior art and the awake half stays out, I do not think that blocks it. Taking it. Thank you for stating the trade-off (deep recall 0.592 → 0.507 against precision 0.430 → 0.463) instead of leading with kappa, and for saying plainly that sleep-accel has no R-R so the RSA term is silent and REM remains unjudged. |
`main` fails its own ratchet at 5de0519, on a clean tree, with no change applied: ERROR: Tools/parity_ledger_baseline.json: accepted finding group regressed test-only-callsite|Packages/StrandAnalytics ERROR: Tools/parity_ledger_baseline.json: accepted finding group regressed test-only-callsite|android/analytics I caused it by merging #2557. That change took the daily resting HR from the primary night's gated five-minute floor instead of the whole-session mean, which replaced the only production call to `primarySessionRestingHR` on both platforms: - ?? primarySessionRestingHR(physiologySessions, hr)?.roundToInt() + ?: primarySession?.restingHR The function is now reached from four test files and nothing else, so the lexical call graph emits one new `test-only-callsite` finding per platform. The checked-in baseline did not record them. This records them: 43 to 44 in `Packages/StrandAnalytics`, 64 to 65 in `android/analytics`, two lines, nothing else moved. Why nothing caught it: #2557 touched no `Tools/parity_*` path, so parity-governance never ran on it. Its seventeen green checks were real and did not include the gate that governs call-site structure. #2575 and #2576 landed the same way today and happened to be harmless. Why my review did not catch it: I checked that `primarySessionRestingHRWithCoverage` still had callers and concluded the old mean was not dead code. That is true of the `WithCoverage` variant and false of the plain function, which is the one #2557 orphaned. I verified the neighbouring symbol. Recording rather than deleting, deliberately. The findings are true, and the baseline exists to hold accepted findings. Whether `primarySessionRestingHR` should survive as a tested-but-unused function, or be deleted with its tests repointed at the `WithCoverage` variant, is a change to code merged an hour ago and belongs in its own review. A governance repair that also deletes product code gets neither part looked at properly. This does not make the debt go away, it makes it visible. The next change in either scope inherits 44 and 65 as the ceiling, so a second orphaned function there fires the gate again. Refs #2557, #2567
What this PR does
NOOP over-reports deep sleep. Against the human-scored PSG in
Tools/SleepPSG(PhysioNet sleep-accel, 31 subjects), the shippedSleepStagerV2puts 18.94 % of the night in deep against 13.76 % true (+5.2 pp, pooled). This lowers one constant, the deep base log-prior, fromlog(0.18)tolog(0.15), on both platforms. Nothing else in the recipe moves.0.15 is not fitted to this cohort. It is #348's value, one half of #348-A (base priors), and this PR takes that half and not the other one, for a reason the harness can now show.
Evidence (the harness's own report, before → after):
Per subject (new
--section priors, every subject counted once): mean kappa 0.351 → 0.357, higher for 21 of 31; mean |deep bias| 8.86 → 7.99 pp, lower for 19 of 31; wake and REM unchanged for every subject.Why not all of #348-A. Its awake half (0.10 → 0.34) brings the pooled wake share to truth (4.15 % → 9.39 % against 9.07 %), which looks like the fix. Per subject it is not: the mean |wake bias| gets worse (5.50 → 5.55 pp), because it over-calls wake for the subjects it does not help. That is the #437 failure, visible on truth this time, so it stays out.
The trade-off, stated. Deep recall drops (0.592 → 0.507) as precision rises (0.430 → 0.463): the recipe says "deep" less often and is right more often when it does; deep F1 is 0.014 lower, light F1 0.025 higher, kappa and accuracy up.
What this does not establish. sleep-accel has no R-R, so the RSA term is silent here; on a WHOOP night it is live. On one WHOOP 5.0 wearer's four nights (R-R present, staged through the harness's port with the wearer's own HR, gravity, R-R and respiration), deep moved 28.0 / 31.0 / 27.6 / 32.3 % → 27.1 / 27.2 / 27.1 / 26.0 % of sleep, same direction, smaller where the RSA term carries the deep call. No truth exists for those nights. REM (+4.5 pp here, and 31–40 % of sleep on that wearer) is untouched; it needs PSG with R-R to be judged.
Changes.
SleepStagerV2.baseLogPrior(Swift) and its Kotlin twin, with the evidence in the Swift doc; a pin of the four priors on each platform (testBaseLogPriorsArePinned/baseLogPriorsArePinned), because the frozen golden hypnogram does not move when a prior changes;RecipeConfig.shippedfollows (port validation); the variant list gainsdeep prior 0.18 (before)so the change stays re-checkable,#348-Ais annotated; the README's shipped figures and a section-7 description are updated. The first commit adds section 7 on its own, so the evidence can be reviewed before the change.Type of change
How it was tested
sleeppsgreport onmainand on this branch (numbers above).swift testinTools/SleepPSG(port validation, 33 tests) passes: the port and the shipped stager agree on every epoch label with the new value. With only the Swift prior put back to 0.18, port validation fails (3 of 5) and the pin fails; restored byte-identical (sha256).baseLogPriorsArePinnedfails.parity_ledger.py: no new findings;parity_ratchet.py: 0 errors.9482462b(on0c982899), every step passed: WhoopStore 629 · StrandAnalytics 2082 · StrandImport 327 · doc lint · i18n · parity ledger · ratchet · governance (124) · macOSStrandTests2,244 (only the two locale-dependentTodayCarryOverTests) · iOS build.Checklist
swift testinPackages/StrandAnalytics,Tools/SleepPSG)android/(fork CI)docs/CONTRIBUTING.mdStrand.xcodeproj/) or any secrets/keystoresRelated issues
Refs #348, #437, #987
🤖 Generated with Claude Code