Skip to content

sleep: lower the deep base prior to 0.15 on PSG evidence - #2576

Merged
ryanbr merged 2 commits into
ryanbr:mainfrom
UtkuDenizAltiok:psg-priors
Sep 28, 2026
Merged

ryanbr merged 2 commits into
ryanbr:mainfrom
UtkuDenizAltiok:psg-priors

Conversation

@UtkuDenizAltiok

Copy link
Copy Markdown

What this PR does

NOOP over-reports deep sleep. Against the human-scored PSG in Tools/SleepPSG (PhysioNet sleep-accel, 31 subjects), the shipped SleepStagerV2 puts 18.94 % of the night in deep against 13.76 % true (+5.2 pp, pooled). This lowers one constant, the deep base log-prior, from log(0.18) to log(0.15), on both platforms. Nothing else in the recipe moves.

0.15 is not fitted to this cohort. It is #348's value, one half of #348-A (base priors), and this PR takes that half and not the other one, for a reason the harness can now show.

Evidence (the harness's own report, before → after):

before (0.18) after (0.15) truth
kappa (pooled) 0.363 0.371
accuracy 59.67 % 61.14 %
deep, % of night (pooled) 18.94 15.09 13.76
light, % of night (pooled) 50.47 54.33 55.19
wake / REM, % of night 4.15 / 26.44 4.15 / 26.44 (identical epochs) 9.07 / 21.98
deep F1 (precision / recall) 0.498 (0.430 / 0.592) 0.484 (0.463 / 0.507)
light F1 0.659 0.684

Per subject (new --section priors, every subject counted once): mean kappa 0.351 → 0.357, higher for 21 of 31; mean |deep bias| 8.86 → 7.99 pp, lower for 19 of 31; wake and REM unchanged for every subject.

Why not all of #348-A. Its awake half (0.10 → 0.34) brings the pooled wake share to truth (4.15 % → 9.39 % against 9.07 %), which looks like the fix. Per subject it is not: the mean |wake bias| gets worse (5.50 → 5.55 pp), because it over-calls wake for the subjects it does not help. That is the #437 failure, visible on truth this time, so it stays out.

The trade-off, stated. Deep recall drops (0.592 → 0.507) as precision rises (0.430 → 0.463): the recipe says "deep" less often and is right more often when it does; deep F1 is 0.014 lower, light F1 0.025 higher, kappa and accuracy up.

What this does not establish. sleep-accel has no R-R, so the RSA term is silent here; on a WHOOP night it is live. On one WHOOP 5.0 wearer's four nights (R-R present, staged through the harness's port with the wearer's own HR, gravity, R-R and respiration), deep moved 28.0 / 31.0 / 27.6 / 32.3 % → 27.1 / 27.2 / 27.1 / 26.0 % of sleep, same direction, smaller where the RSA term carries the deep call. No truth exists for those nights. REM (+4.5 pp here, and 31–40 % of sleep on that wearer) is untouched; it needs PSG with R-R to be judged.

Changes. SleepStagerV2.baseLogPrior (Swift) and its Kotlin twin, with the evidence in the Swift doc; a pin of the four priors on each platform (testBaseLogPriorsArePinned / baseLogPriorsArePinned), because the frozen golden hypnogram does not move when a prior changes; RecipeConfig.shipped follows (port validation); the variant list gains deep prior 0.18 (before) so the change stays re-checkable, #348-A is annotated; the README's shipped figures and a section-7 description are updated. The first commit adds section 7 on its own, so the evidence can be reviewed before the change.

Type of change

  • Bug fix (a sleep-stage fraction measured wrong against PSG)

How it was tested

  • PSG: the full sleeppsg report on main and on this branch (numbers above). swift test in Tools/SleepPSG (port validation, 33 tests) passes: the port and the shipped stager agree on every epoch label with the new value. With only the Swift prior put back to 0.18, port validation fails (3 of 5) and the pin fails; restored byte-identical (sha256).
  • Swift: StrandAnalytics 2,081 tests pass unchanged (no golden output pinned the old prior), plus the new pin.
  • Kotlin: Android CI on the fork: green (6,535 tests); with only the Kotlin prior put back to 0.18 (a throwaway branch), exactly baseLogPriorsArePinned fails.
  • Parity: parity_ledger.py: no new findings; parity_ratchet.py: 0 errors.
  • Full local run on 9482462b (on 0c982899), every step passed: WhoopStore 629 · StrandAnalytics 2082 · StrandImport 327 · doc lint · i18n · parity ledger · ratchet · governance (124) · macOS StrandTests 2,244 (only the two locale-dependent TodayCarryOverTests) · iOS build.

Checklist

  • Swift package tests pass for any package I touched (swift test in Packages/StrandAnalytics, Tools/SleepPSG)
  • Android unit tests pass if I touched android/ (fork CI)
  • No new build warnings introduced
  • Follows the conventions in docs/CONTRIBUTING.md
  • I did not commit generated output (Strand.xcodeproj/) or any secrets/keystores

Related issues

Refs #348, #437, #987

🤖 Generated with Claude Code

UtkuDenizAltiok and others added 2 commits September 28, 2026 18:58
The variant table pools 26 773 epochs, so a few long nights can carry a row.
Section 7 (--section priors) scores each of the 31 subjects once: their own
kappa and how far each stage's share of their night is from their own PSG,
for the shipped recipe, ryanbr#348-A's deep prior alone, its awake prior alone,
and both, and counts the subjects each one improves.

On sleep-accel the deep prior alone raises kappa for 21 of 31 subjects and
cuts |deep bias| for 19 of 31 without moving wake or REM; the awake prior
alone improves wake for 20 but worsens the mean |wake bias|, the ryanbr#437 shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SleepStagerV2's deep log-prior was log(0.18). Against human-scored PSG
(PhysioNet sleep-accel, n = 31, Tools/SleepPSG) the shipped recipe
over-called deep by +5.2 pp of the night pooled (18.94 % against 13.76 %).

0.15 is ryanbr#348's value, not a fit to this cohort. Alone it moves no wake or
REM epoch and, pooled: kappa 0.363 -> 0.371, accuracy 59.67 -> 61.14 %, deep
18.94 -> 15.09 % (truth 13.76), light 50.47 -> 54.33 % (truth 55.19); deep F1
0.498 -> 0.484 (precision 0.430 -> 0.463, recall 0.592 -> 0.507). Per subject
(section 7): kappa higher for 21 of 31, |deep bias| lower for 19 of 31.
ryanbr#348's awake half is not taken: it fixes the pooled wake share but worsens
the mean per-subject wake error, the ryanbr#437 shape.

Swift and Kotlin move together and pin the priors in a test each;
RecipeConfig.shipped follows (port validation), the variant list keeps the
old value as "deep prior 0.18 (before)", and the README's shipped figures
and section 7 are updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UtkuDenizAltiok added a commit to UtkuDenizAltiok/noop that referenced this pull request Sep 28, 2026
), ryanbr#2569 merged, build 769113c; rescore-profile tools; STATE rewritten for a fresh session
@pipiche38

Copy link
Copy Markdown

One data point from the Oura side, since it lands in the regime the PR names as unmeasured: R-R present, gravity absent.

Where this constant reaches an Oura night. A ring night with its own SleepNet hypnogram is not staged by SleepStagerV2 at all (#804 Fix A hands the ring's hypnogram to analyzeDay), so for most ring nights this PR changes nothing. It does reach a ring night that has no stored hypnogram (a missed drain, a dark night): grav.count < 2 sends it to SleepStager.hrOnlySessions, which calls SleepStagerV2.stageSession(grav: []) with the ring's banked R-R. There the RSA term is live and there is no motion evidence at all.

What was run. 60 ring nights (2026-07-28 → 09-27, one wearer, each ≥ 4 h with a stored ring hypnogram and banked R-R), each re-staged with SleepStagerV2.stageSession(start, end, grav: [], hr, rr, resp: []) over the ring's own window, once with the deep prior at 0.18 and once at 0.15, scored per 30-s epoch against the ring's hypnogram. The ring hypnogram is not PSG: it is the vendor's on-device staging, used here only as an independent reference for deep. Its deep share is close to Oura's own app (median 16.2 % of asleep here; the app's server-side result for the same wearer over 280 nights is 12.8 %), whereas its REM is known to sit below the app's, so it is not used to judge REM.

deep prior 0.18 deep prior 0.15 ring
deep, % of window (pooled) 22.8 19.7 13.2
deep, % of asleep (median night) 22.8 19.4 16.2
mean |deep bias| per night 9.57 pp 6.91 pp (lower on 52 of 60)
deep precision / recall 0.314 / 0.543 0.334 / 0.498
deep F1 0.398 0.400
r(night deep %, ring deep %) 0.546 0.557

Same direction and the same precision-for-recall shape as the PSG result, and no loss in how well deep tracks night to night. On this path it still leaves deep over-called (+6.3 pp mean signed bias; below the ring on 8 of 60 nights), so 0.15 does not overshoot here. REM moved on 5 of 60 nights by a few epochs, wake on none.

Two things this regime shows that sleep-accel cannot, neither of them for this PR:

  • REM: with R-R present and no gravity, V2 puts ~39 % of the window in REM (median 39.5 %), against 12 % on the ring and ~20 % in Oura's app. That is consistent with the 31–40 % seen on the WHOOP 5.0 nights in the description, and points at the REM side of the RSA term rather than at the priors.
  • Wake: with grav: [] V2 calls no wake epoch on any of the 60 nights. On the app path this is partly masked, because hrOnlySessions trims the window from heart rate before staging, but inside the window every awakening becomes sleep.

Nothing here changes the PR; it adds one no-gravity, R-R-present cohort to the "what this does not establish" list, pointing the same way.

@ryanbr

ryanbr commented Sep 28, 2026

Copy link
Copy Markdown
Owner

@UtkuDenizAltiok Reviewed. 17/17 green. This is careful work and the method is the part worth saying out loud.

I checked the load-bearing claim rather than taking it. "0.15 is #348's value, not a fit to this cohort" holds, and it is stronger than the sentence suggests: #348 proposed deep 0.18 → 0.15 from an ascent on DREAMT (n = 100) with its own 80/20 held-out check, train kappa 0.328 against test 0.355. So the value was fitted on one dataset and is being judged here on another.

The part that makes this safe is what you left out. #437 reverted all seven of #348's components 48 hours after they shipped, because one healthy night went from 6 % to 23 % awake. The component responsible was the awake prior, 0.10 → 0.34. You excluded exactly that one, and excluded it on the evidence that would have justified taking it: the pooled wake share moves to truth (4.15 % → 9.39 % against 9.07 %) while the mean per-subject wake error gets worse (5.50 → 5.55 pp). That is #437's own lesson, "kappa doesn't guard stage-fraction calibration", turned on the change that would have repeated it. Reporting per subject with every subject counted once, rather than pooled, is what let you see it.

Also verified:

  • The twins are byte-identical (log(0.15) / ln(0.15)), the other three priors are untouched, and wake / REM moving for no subject follows from that rather than being asserted.
  • Not redistributing the freed three points is right. Only deep's weight relative to the others changes, which is the minimal intervention the evidence actually supports.
  • The pin on each platform earns its place: a frozen golden hypnogram does not move when a prior changes, so without it the constant had no guard at all.

Two notes, neither blocking, neither needing a code change here.

A visible discontinuity is coming. analyzeRecent re-scores a rolling 21-day window (a strap log from today shows re-score census: 21 night(s) 2026-09-08..2026-09-28). So after an update the last three weeks restage toward 15 % deep while everything older keeps its 19 %. Someone looking at a three-month deep-sleep trend sees a step at the window edge, not a gradual change, and the step is the fix rather than a bug. Worth a release-note line so it does not arrive as a report.

Walch is not a fresh held-out set. #348 already used it as one of its out-of-sample checks and reported 0.356 → 0.362 there. Your numbers are consistent with that and the per-subject breakdown is genuinely new, but the cohort is not independent of the reporting that first justified 0.15. The strongest version of this evidence would be a cohort neither #348 nor #987 has touched. Given the value itself is unchanged prior art and the awake half stays out, I do not think that blocks it.

Taking it. Thank you for stating the trade-off (deep recall 0.592 → 0.507 against precision 0.430 → 0.463) instead of leading with kappa, and for saying plainly that sleep-accel has no R-R so the RSA term is silent and REM remains unjudged.

@ryanbr
ryanbr merged commit bbeb20e into ryanbr:main Sep 28, 2026
17 checks passed
ryanbr added a commit that referenced this pull request Sep 29, 2026
`main` fails its own ratchet at 5de0519, on a clean tree, with no change
applied:

  ERROR: Tools/parity_ledger_baseline.json: accepted finding group regressed
  test-only-callsite|Packages/StrandAnalytics
  ERROR: Tools/parity_ledger_baseline.json: accepted finding group regressed
  test-only-callsite|android/analytics

I caused it by merging #2557. That change took the daily resting HR from the
primary night's gated five-minute floor instead of the whole-session mean,
which replaced the only production call to `primarySessionRestingHR` on both
platforms:

  -  ?? primarySessionRestingHR(physiologySessions, hr)?.roundToInt()
  +  ?: primarySession?.restingHR

The function is now reached from four test files and nothing else, so the
lexical call graph emits one new `test-only-callsite` finding per platform. The
checked-in baseline did not record them. This records them: 43 to 44 in
`Packages/StrandAnalytics`, 64 to 65 in `android/analytics`, two lines, nothing
else moved.

Why nothing caught it: #2557 touched no `Tools/parity_*` path, so
parity-governance never ran on it. Its seventeen green checks were real and did
not include the gate that governs call-site structure. #2575 and #2576 landed
the same way today and happened to be harmless.

Why my review did not catch it: I checked that
`primarySessionRestingHRWithCoverage` still had callers and concluded the old
mean was not dead code. That is true of the `WithCoverage` variant and false of
the plain function, which is the one #2557 orphaned. I verified the neighbouring
symbol.

Recording rather than deleting, deliberately. The findings are true, and the
baseline exists to hold accepted findings. Whether `primarySessionRestingHR`
should survive as a tested-but-unused function, or be deleted with its tests
repointed at the `WithCoverage` variant, is a change to code merged an hour ago
and belongs in its own review. A governance repair that also deletes product
code gets neither part looked at properly.

This does not make the debt go away, it makes it visible. The next change in
either scope inherits 44 and 65 as the ceiling, so a second orphaned function
there fires the gate again.

Refs #2557, #2567
@UtkuDenizAltiok
UtkuDenizAltiok deleted the psg-priors branch September 30, 2026 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants