sleep: lower the deep base prior to 0.15 on PSG evidence

Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift

@@ -157,10 +157,18 @@ public enum SleepStagerV2 {
 
     static let stageNames = ["deep", "rem", "light", "awake"]
 
-    /// Population sleep-architecture base rates as log-priors (adult TST ≈ light 50 / deep 18 / rem 22 /
-    /// waso 10 %). Calibrates the boundary so light wins weak-evidence epochs.
+    /// Population sleep-architecture base rates as log-priors (adult TST ≈ light 50 / rem 22 / waso 10 %,
+    /// deep 15). Calibrates the boundary so light wins weak-evidence epochs.
+    ///
+    /// Deep was 0.18. Against human-scored PSG (PhysioNet sleep-accel, n = 31, `Tools/SleepPSG` section 7)
+    /// that over-called deep by +5.2 pp of the night pooled (18.9 % against 13.8 %). 0.15 is #348's value, not
+    /// a fit to this cohort; alone it raises per-subject kappa for 21 of 31 subjects and cuts |deep bias| for
+    /// 19 of 31, and moves no wake or REM epoch. #348's other prior (awake 0.10 → 0.34) is deliberately not
+    /// taken: it fixes the pooled wake share but worsens the mean per-subject wake error, the #437 shape.
+    /// The three freed points are not redistributed; the evidence is for this one value.
+    /// Kotlin twin: `SleepStagerV2.baseLogPrior`.
     static let baseLogPrior: [String: Double] = [
-        "light": log(0.50), "deep": log(0.18), "rem": log(0.22), "awake": log(0.10)]
+        "light": log(0.50), "deep": log(0.15), "rem": log(0.22), "awake": log(0.10)]
 
     /// Deep is eligible only in the night's lowest ~25 % HR-flatness epochs (≈ deep base rate + margin).
     /// Widened 0.20 -> 0.25 by the multi-subject (AAUWSS + sleep-accel LOSO) deep-boundary tune, which

Packages/StrandAnalytics/Tests/StrandAnalyticsTests/SleepStagerV2Tests.swift

@@ -398,6 +398,14 @@ final class SleepStagerV2Tests: XCTestCase {
         }
     }
 
+    /// Pin the base log-priors, deep included. The golden hypnogram does not move when a prior changes, so
+    /// without this a one-sided edit would pass. Deep is 0.15, not 0.18: the one prior the sleep-accel PSG
+    /// comparison (`Tools/SleepPSG` section 7) supports moving. Twin: `SleepStagerV2Test.baseLogPriorsArePinned`.
+    func testBaseLogPriorsArePinned() {
+        XCTAssertEqual(SleepStagerV2.baseLogPrior,
+                       ["light": log(0.50), "deep": log(0.15), "rem": log(0.22), "awake": log(0.10)])
+    }
+
     /// Pin the AWAKE transition row directly, for the same reason the deep row is pinned: the end-to-end
     /// golden is only sensitive where its own input sits near a decision boundary, and it does NOT move when
     /// this row changes — so without this assertion the row is effectively unguarded. Twin:

Tools/SleepPSG/README.md

@@ -38,7 +38,7 @@ swift run -c release sleeppsg --dataset ~/datasets/motion-and-heart-rate-*-1.0.0
 | flag | meaning |
 |---|---|
 | `--dataset PATH` | the extracted `sleep-accel` root. Required for every section except `port`. |
-| `--section S` | `all` (default), or one of `port`, `baseline`, `strata`, `rem`, `variants`. |
+| `--section S` | `all` (default), or one of `port`, `baseline`, `strata`, `rem`, `variants`, `priors`. |
 | `--subjects N` | score only the first N subject ids — a fast smoke run. |
 | `--csv PATH` | write the per-variant table as CSV. |
 | `--seed N` | the port-validation corpus seed. |
@@ -106,8 +106,11 @@ What varies is how the 31 subjects are combined:
 
 | convention | what it is | deep, shipped recipe |
 |---|---|---|
-| **pooled over all scored epochs** | one denominator of 26 773 epochs for the whole cohort; a long night carries more weight than a short one | predicted **18.94 %**, truth **13.76 %**, bias **+5.18 pp** |
-| **mean of per-subject percentages** | each subject's own % of their own night, then averaged over 31 subjects; every subject counts once | predicted **19.24 %**, truth **14.76 %**, bias **+4.48 pp** |
+| **pooled over all scored epochs** | one denominator of 26 773 epochs for the whole cohort; a long night carries more weight than a short one | predicted **15.09 %**, truth **13.76 %**, bias **+1.32 pp** |
+| **mean of per-subject percentages** | each subject's own % of their own night, then averaged over 31 subjects; every subject counts once | predicted **15.16 %**, truth **14.76 %**, bias **+0.40 pp** |
+
+Those are the recipe with the deep prior at 0.15 (section 7). With the deep prior at its former 0.18 (the
+`deep prior 0.18 (before)` row) they were 18.94 % / 19.24 %, bias +5.18 / +4.48 pp.
 
 Section 3 prints both tables, one above the other, each labelled. They are the same epochs and the same
 denominator *within* a night; the ~1 pp gap between them is the weighting across subjects, nothing else.
@@ -150,12 +153,32 @@ Each row is the shipped recipe with **one** named change: PR #987's awake-transi
 #348's seven components measured alone, plus all seven together. Reported with kappa **and** the stage-
 fraction biases, because a component is an improvement only if it wins the first without wrecking the
 second. **Every percentage and every `bias pp` in this table is pooled over all 26 773 scored epochs** —
-so the incumbent's deep bias reads +5.18 pp here, not the +4.48 pp the per-subject mean gives.
+so the incumbent's deep bias reads +1.32 pp here, not the +0.40 pp the per-subject mean gives.
+
+`deep prior 0.18 (before)` is the deep prior as it stood before section 7's evidence moved it to 0.15.
 
 The last row is not a candidate. `pre-#930 REM guard (provenance)` runs the REM-latency guard as it stood
 before PR #930 — a hard `c < 0.12 ? 3.0 : 0.0` step in the session-fraction domain — and exists to explain
 the self-check below.
 
+### 7. Base priors, subject by subject
+
+The variant table pools 26 773 epochs, so a few long nights can carry a row. `--section priors` scores each
+of the 31 subjects once: their own kappa, and how far each stage's share of their night is from their own
+PSG, and counts the subjects a variant improves. This is the evidence the deep prior moved on:
+
+| deep prior | mean per-subject κ | subjects with κ higher at 0.15 | mean \|deep bias\| | subjects with \|deep bias\| lower at 0.15 | wake, REM |
+|---|---|---|---|---|---|
+| 0.18 (before) | 0.351 | — | 8.86 pp | — | — |
+| **0.15** | **0.357** | 21 / 31 | **7.99 pp** | 19 / 31 | unchanged, epoch for epoch |
+
+Pooled, deep goes from 18.94 % to 15.09 % of the night against 13.76 % true, light from 50.47 % to 54.33 %
+against 55.19 %, kappa 0.363 → 0.371; deep F1 0.498 → 0.484 (precision 0.430 → 0.463, recall 0.592 → 0.507).
+0.15 is #348's value, not a fit to this cohort. #348's awake half (0.10 → 0.34) is not taken: it brings the
+pooled wake share to truth (4.15 % → 9.39 % against 9.07 %) but worsens the mean per-subject wake error
+(5.50 → 5.55 pp) by over-calling wake for the subjects it does not fix, which is the #437 failure. The R-R
+stream is empty here, so the RSA term is silent in all of this; on a WHOOP night it is live.
+
 ---
 
 ## The self-check, and what it found
@@ -172,8 +195,9 @@ convention in its own header so a figure copied out of it carries its denominato
 
 Every **truth-side** figure reproduces exactly: 31 subjects, **26 773** PSG-scored epochs, deep **14.76 %**
 of each subject's own night averaged over the 31 (pooled: 13.76 %), truth median first-REM latency
-**88.5 min**. Predicted deep lands at **19.24 %** on the same convention against a reported 19.25 %
-(pooled: 18.94 %), and four-class kappa at **0.356** against a reported 0.349.
+**88.5 min**. Predicted deep, with the deep prior at the 0.18 that harness measured (the `deep prior 0.18
+(before)` row), lands at **19.24 %** on the same convention against a reported 19.25 % (pooled: 18.94 %), and
+four-class kappa at **0.356** against a reported 0.349.
 
 The prediction side of REM does not match: more REM (**26.96 %** per-subject mean, against a reported
 20.8 %), REM F1 0.569 vs 0.515, and a first REM period arriving much earlier (median 91.5 min vs 142.0).
@@ -190,9 +214,9 @@ it.
 What the residual is remains **unresolved**, and the tool is not tuned to close it. Two constraints on any
 future explanation, both from the table above:
 
-- It cannot be the session window. `clock` drives the deep prior and the REM ramp alike; predicted deep
-  reproduces to 0.01 pp (19.24 % measured against 19.25 % reported, per-subject mean), so `clock` is the
-  same quantity in both harnesses.
+- It cannot be the session window. `clock` drives the deep prior and the REM ramp alike; predicted deep at
+  the 0.18 prior reproduces to 0.01 pp (19.24 % measured against 19.25 % reported, per-subject mean), so
+  `clock` is the same quantity in both harnesses.
 - It is REM-specific and it is not the latency guard, which the variant just ruled out.
 
 One suggestive coincidence, offered as an observation and not a conclusion: the previously reported "REM

Tools/SleepPSG/Sources/sleeppsg/RecipePort.swift

@@ -87,7 +87,7 @@ struct RecipeConfig: Equatable {
 
     /// The shipped recipe, as of `Packages/StrandAnalytics/Sources/StrandAnalytics/SleepStagerV2.swift`.
     static let shipped = RecipeConfig(
-        priorLight: log(0.50), priorDeep: log(0.18), priorRem: log(0.22), priorAwake: log(0.10),
+        priorLight: log(0.50), priorDeep: log(0.15), priorRem: log(0.22), priorAwake: log(0.10),
         deepGateThresh: 0.25, deepGateSlope: 5.0,
         jerkFloorMoveMult: 38.0, jerkFloorGateMult: 55.0, motionGateBoost: 2.0,
         respWeight: 0.6,

Tools/SleepPSG/Sources/sleeppsg/Variants.swift

@@ -48,12 +48,21 @@ enum Variants {
     }
 
     /// #348 component 1 — base priors. Measured alone on band state: healthy wake fraction 9.43 % → 17.76 %,
-    /// i.e. the #437 blow-out reproduced.
+    /// i.e. the #437 blow-out reproduced. Its deep half (0.18 → 0.15) now ships, on this dataset's evidence, so
+    /// against the incumbent this row moves only the awake prior.
     static var p348Priors: Variant {
         var c = RecipeConfig.shipped
         c.priorDeep = log(0.15); c.priorAwake = log(0.34)
         return Variant(name: "#348-A base priors",
-                       note: "deep 0.18→0.15, awake 0.10→0.34", config: c)
+                       note: "deep 0.15 (shipped), awake 0.10→0.34", config: c)
+    }
+
+    /// The deep prior as it was before it moved to #348's 0.15 on this PSG evidence (section 7). Kept as the
+    /// "before" row, so the change stays re-checkable against the dataset rather than asserted.
+    static var deepPriorBefore: Variant {
+        var c = RecipeConfig.shipped
+        c.priorDeep = log(0.18)
+        return Variant(name: "deep prior 0.18 (before)", note: "deep 0.15→0.18, the value before", config: c)
     }
 
     /// #348 component 2 — motion gate. A second wake channel: alone, healthy wake 9.43 % → 15.92 %.
@@ -153,7 +162,7 @@ enum Variants {
     }
 
     static var all: [Variant] {
-        [incumbent, pr987, p348Priors, p348Motion, p348Emissions, p348DeepGate,
+        [incumbent, deepPriorBefore, pr987, p348Priors, p348Motion, p348Emissions, p348DeepGate,
          p348Deadzone, p348OtherRows, p348All, preNine30Guard]
     }
 }

Tools/SleepPSG/Sources/sleeppsg/main.swift

@@ -27,7 +27,7 @@ while let k = it.next() {
     case "--seed": a.seed = UInt64(it.next() ?? "") ?? a.seed
     case "-h", "--help":
         print("""
-        usage: sleeppsg --dataset <sleep-accel root> [--section all|port|baseline|strata|rem|variants]
+        usage: sleeppsg --dataset <sleep-accel root> [--section all|port|baseline|strata|rem|variants|priors]
                         [--subjects N] [--csv <path>] [--seed <n>]
 
           --dataset   PhysioNet sleep-accel v1.0.0, extracted. See README.md for the download step and the
@@ -478,4 +478,37 @@ if want("variants") {
     }
 }
 
+if want("priors") {
+    print("""
+
+    ================================================================================
+    7. BASE PRIORS, SUBJECT BY SUBJECT
+    ================================================================================
+    The pooled table above can be carried by a few long nights. Here every subject counts once: their own
+    kappa, and how far each stage's share of THEIR night is from their own PSG. The deep prior moved 0.18 →
+    0.15 on this section's evidence; the "before" row is that change undone, and #348-A adds the awake half
+    the recipe does not take. `better` counts subjects whose value improved against the incumbent (kappa up,
+    |bias| down); a tie counts as neither.
+    """)
+    func perSubject(_ rows: [Scored]) -> [(kappa: Double, bias: [String: Double])] {
+        rows.map { r in
+            let bias = stageBias(ref: r.truth, pred: r.pred)
+            return (confusion(ref: r.truth, pred: r.pred).kappa, bias)
+        }
+    }
+    let base = perSubject(shipped)
+    print("    variant                     mean κ  better κ   mean|bias| wake  deep   REM  light   better |bias| wake deep  REM light")
+    for v in [Variants.incumbent, Variants.deepPriorBefore, Variants.p348Priors] {
+        let rows = v.name.hasPrefix("incumbent") ? shipped : subjects.map { score($0, using: stageVariant(v.config)) }
+        let ps = perSubject(rows)
+        let n = ps.count
+        let kBetter = zip(ps, base).filter { $0.0.kappa > $0.1.kappa }.count
+        func meanAbs(_ st: String) -> Double { mean(ps.map { abs($0.bias[st] ?? 0) }) }
+        func better(_ st: String) -> Int { zip(ps, base).filter { abs($0.0.bias[st] ?? 0) < abs($0.1.bias[st] ?? 0) }.count }
+        print("    \(v.name.padding(toLength: 26, withPad: " ", startingAt: 0))\(f(mean(ps.map { $0.kappa }), 7, 3))   \(kBetter)/\(n)     "
+              + "\(f(meanAbs("wake"), 5, 2)) \(f(meanAbs("deep"), 5, 2)) \(f(meanAbs("rem"), 5, 2)) \(f(meanAbs("light"), 5, 2))      "
+              + "\(better("wake"))/\(n) \(better("deep"))/\(n) \(better("rem"))/\(n) \(better("light"))/\(n)")
+    }
+}
+
 print("")

android/app/src/main/java/com/noop/analytics/SleepStagerV2.kt

@@ -161,10 +161,13 @@ object SleepStagerV2 {
 
     private val stageNames = listOf("deep", "rem", "light", "awake")
 
-    /** Population sleep-architecture base rates as log-priors (adult TST ≈ light 50 / deep 18 / rem 22 /
-     *  waso 10 %). Calibrates the boundary so light wins weak-evidence epochs. */
-    private val baseLogPrior: Map<String, Double> = mapOf(
-        "light" to ln(0.50), "deep" to ln(0.18), "rem" to ln(0.22), "awake" to ln(0.10))
+    /** Population sleep-architecture base rates as log-priors (adult TST ≈ light 50 / rem 22 / waso 10 %,
+     *  deep 15). Calibrates the boundary so light wins weak-evidence epochs. Deep was 0.18, which over-called
+     *  deep against PSG (PhysioNet sleep-accel, n = 31: +5.2 pp pooled); 0.15 alone raises per-subject kappa for
+     *  21 of 31 subjects and moves no wake or REM epoch. Swift twin, with the full evidence:
+     *  `SleepStagerV2.baseLogPrior`. */
+    internal val baseLogPrior: Map<String, Double> = mapOf(
+        "light" to ln(0.50), "deep" to ln(0.15), "rem" to ln(0.22), "awake" to ln(0.10))
 
     /** Deep is eligible only in the night's lowest ~25 % HR-flatness epochs (≈ deep base rate + margin).
      *  Widened 0.20 -> 0.25 by the multi-subject (AAUWSS + sleep-accel LOSO) deep-boundary tune, which

android/app/src/test/java/com/noop/analytics/SleepStagerV2Test.kt

@@ -428,6 +428,18 @@ class SleepStagerV2Test {
         }
     }
 
+    /**
+     * Pin the base log-priors, deep included. The golden hypnogram does not move when a prior changes, so without
+     * this a one-sided edit would pass. Deep is 0.15, not 0.18: the one prior the sleep-accel PSG comparison
+     * (`Tools/SleepPSG` section 7) supports moving. Twin: `SleepStagerV2Tests.testBaseLogPriorsArePinned`.
+     */
+    @Test
+    fun baseLogPriorsArePinned() {
+        assertEquals(
+            mapOf("light" to ln(0.50), "deep" to ln(0.15), "rem" to ln(0.22), "awake" to ln(0.10)),
+            SleepStagerV2.baseLogPrior)
+    }
+
     /**
      * Pin the AWAKE transition row directly, for the same reason the deep row is pinned: the end-to-end golden
      * is only sensitive where its own input sits near a decision boundary, and it does NOT move when this row