Repository navigation
fix(diagnostics): keep the night's raw Oura sidecar in the export — session-scoped, ring-buffered, fairly capped, generations merged - #2154
Conversation
…al-steps sidecars TestBundleAssembler.normalizedOuraEntryName() hardcoded the Oura Diagnostics-dir sidecar kinds it recognizes to the original three (raw/ibihr/activity). The cva-ppg, motion, and real-steps dump writers shipped afterward and were writing real files on-device the whole time, but the export bundler silently dropped all three from every Test Centre bundle since it didn't know their kind - confirmed against two real exports where the strap log showed the writers firing but the files never made it into the bundle. Collapsed the 4 places that independently listed sidecar kinds down to one source of truth (ouraSidecarKinds), so a future dump writer needs exactly one line added to be picked up by normalizedOuraEntryName/ouraDiagnosticEntries/ trimmableNames/ouraSidecarNames together instead of silently missing export. (The original commit also carried a hardcoded test-branch build marker for the strap-log header, explicitly flagged "must never ship". It is dropped here along with its two call sites, and the follow-up commit that existed only to remove it is dropped with it.) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EHFxGQzockMYJqDXYB3S1
TestBundleAssembler.capEntries trimmed oversized bundle files to a raw byte count (keeping the most-recent tail), which is safe for a single freeform text file but not for the newline-delimited JSONL sidecars: a byte-count cut can land mid-record. Confirmed in production - a real overnight export (2026-07-30/31) shipped oura-cva-ppg.jsonl, oura-real-steps.jsonl, and oura-motion.jsonl each with a corrupted first line once the sidecars' combined size crossed the 20 MB cap for the first time. Adds trimToLineBoundary: after the proportional byte-count trim, drop bytes up to and including the first newline, so every kept line is a complete record. Every trimmable name is newline-delimited JSONL, so this is safe across the board; a single very-long line (no newline found) is returned unchanged, never worse than before. Tested: new testCapSnapsTrimmedTailToLineBoundary (every kept line parses as valid JSON after a forced mid-file trim) and testTrimToLineBoundaryHandlesNoNewline; existing cap tests unaffected (their fixtures have no embedded newlines, so the snap is a no-op for them). macOS Strand target builds; TestBundleAssemblerTests 13/13 pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013EHFxGQzockMYJqDXYB3S1
…ion to size
`TestBundleAssembler.capEntries` held the bundle to 20 MB by splitting the budget
across `trimmableNames` IN PROPORTION to each file's size. That hands the budget to
the bulkiest stream, and for the Oura sidecars bulk is close to inversely correlated
with diagnostic value.
Measured on a real export (`noop-master-iOS-v9.3.1-260809-0716.zip`, six trimmable
sidecars over a 20,863,648-byte budget):
oura-spo2.jsonl 11,242,865 53.9 %
oura-real-steps.jsonl 3,782,590 18.1 %
oura-motion.jsonl 2,827,017 13.5 %
oura-cva-ppg.jsonl 2,417,008 11.6 %
oura-raw.jsonl 388,064 1.9 %
oura-activity.jsonl 206,921 1.0 %
Trimmable total 20,864,465 against the computed budget of 20,863,648 — agreeing to
817 bytes of integer-floor rounding, so this is the cap firing exactly as written.
`oura-raw.jsonl` is the UNDECODED wire bytes: the one file in the bundle from which a
protocol fact can be re-derived independently of NOOP's own decoder, which is the
entire reason `OuraRawDump` exists. It shipped 3,123 records — 8 min 43 s of an 8.4 h
night whose full history drain had just completed. The verifiers that read the raw
sidecar ON PURPOSE (so a decode bug cannot launder itself into the number checking it)
could not run at all that night, while 11 MB of SpO2 samples, also present in the
SQLite the same bundle ships, rode along whole.
Replaced with max-min fair (water-filling) allocation: walk the streams smallest-first
offering each an equal share of what is left, so a stream under its share is kept WHOLE
and its surplus rolls forward to the ones still over. It makes no editorial claim about
which sidecar matters — the small file wins because it is small, not because it is
favoured. Integer floor keeps the sum at or under budget, so the cap can never breach.
Because each shipped size IS its old proportional share, the on-disk sizes are the
shipped sizes times one unknown scale k, and the outcome follows exactly: with six
streams the fair share is 3.48 MB, so `oura-raw.jsonl` ships WHOLE for any k up to ~9
(a sidecar directory under ~190 MB) and still gets 9x its current bytes beyond that.
A single trimmable stream still receives the whole remainder, so the original
raw-capture-only behaviour is unchanged.
`fairAllowances` is pure and separately tested. One existing test pinned the old rule
(bigger stream gets the bigger share) and now pins the new one; two tests are added —
the water-filling contract including determinism and degenerate budgets, and the real
bug shape, a modest wire capture beside a bulky neighbour.
Verification: StrandTests 1083/0 (1 skipped) including all 15 TestBundleAssembler
tests; `Strand` macOS and `NOOPiOS` iOS both built locally, which no default CI covers.
Apple-only — `TestBundleAssembler` has no Android twin, so there is no parity half.
Diagnostics only: no BLE writes, no schema change, no analytics, nothing user-facing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E7i8DiChrWDDbppKpywaSA
…never roll mid-drain `OuraRawDump` rolled its sidecar to a `.1` sibling whenever the file passed 25 MB, checked per record. A size threshold can fire MID-DRAIN, and on 2026-08-09 it did: it rolled at 07:06:58 during the morning offload, carrying 32,005 records of that drain and 71 of the night's 75 0x5D records into the `.1` — which the strap-log bundle does not collect. The bundle shipped 3,123 records covering 8 min 43 s of an 8.4 h night, and the verifiers that read the raw sidecar ON PURPOSE (so a decode bug cannot launder itself into the number checking it) could not run. The evidence was recoverable only by pulling the file off the phone by hand. The live file now holds exactly ONE session. It is rolled aside when the object resolves its file for the first time — the START of a session — and never again while that session runs. One OuraRawDump is built per OuraLiveSource, and a source survives BLE reconnects (SourceCoordinator returns early when the active strap is unchanged), so a drain that disconnects and re-engages several times still lands whole in one file: the five reconnects of that same morning would have been a single session. Lowering the threshold instead would make the failure MORE frequent, not less. Only a session boundary removes it, and the corpus stays bounded anyway because a session is bounded: an overnight drain is ~32,000 records, about 4 MB. The multi-day corpus is deliberately given up, because the bundle replaces it — every capture now carries its own COMPLETE raw sidecar, and bundles are what get archived. One previous session is kept as `.1` so a session that ends without an export is not lost. The per-record `stat` goes with it: the file starts empty each session, so a running byte counter is authoritative and costs no filesystem call. That counter enforces a 25 MB SESSION ceiling which STOPS appending and logs once rather than rolling — rolling mid-session is the exact failure being removed, so it must not reappear at a new threshold. A real drain is nowhere near it; it exists so a pathological backlog cannot fill the disk. `directory` is injectable (defaulting to Application Support) purely so the roll is testable: this class had no coverage at all, because it hardcoded its path. Six tests now pin the session contract — a whole session in one file with nothing rolled aside, a new session rolling the previous one aside and starting empty, exactly two generations retained, an empty leftover NOT spending the retained slot, a silent session touching nothing, and the ceiling bounding a session WITHOUT rolling it. Verification: StrandTests 1089/0 (1 skipped); `Strand` macOS and `NOOPiOS` iOS both built locally, which no default CI covers. Apple-only — OuraRawDump has no Android twin, so there is no parity half. Diagnostics only: no BLE writes, no schema change, no analytics, nothing user-facing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E7i8DiChrWDDbppKpywaSA
…the one that holds the capture Session scoping (c9fd883) fixed the mid-drain roll and then lost a night a different way. Measured on the 2026-08-10 capture: the app relaunched twice after wake (07:45 and 08:17 local), so both retained slots — the live file and its single `.1` — held post-wake sessions of 87 KB and 44 KB, and the night's 4 MB was gone before it could be pulled off the phone. Rotation ate a night every ~5-6 days at random. One-deep session scoping ate it every morning the app restarted twice, which is the normal morning. An iOS app is relaunched whenever the user opens it, so the number of session boundaries between a night and its export is not something OuraRawDump can predict; it can only keep enough of them. Two changes, and the second is what makes the first useful: * OuraRawDump keeps a numbered generation ring (`.1` ... `.8`) instead of one `.1`. A session under `minRolledBytes` (16 KB) is discarded rather than rolled, so a connect that captured essentially nothing cannot evict a real overnight capture, and the retained generations are pruned oldest-first to a 64 MB budget so the ring is bounded in BYTES, not merely in file count. * TestBundleAssembler.normalizedOuraEntryName now maps `.jsonl.<n>` to the same entry name as the live file. Its existing largest-wins rule then ships the generation that actually holds the capture. Without this the export could only ever carry the live file — a 30-second morning session — and keeping generations on disk would help nobody without a USB cable. Largest-wins means a morning export can ship a sidecar from an earlier session than the live one. That is the point, and it is safe to choose blind: every line carries its own `utc`/`iso`, so an analysis scoped to a night either finds that night's records or honestly finds none. Diagnostics only - no BLE writes, no protocol change, nothing scored. Apple-only (Android has no raw sidecar writer), so there is no parity half. Verified: StrandTests 1092/0 (1 skipped), including 4 new OuraRawDump tests (the regression test replays the two post-wake relaunches) and a new normalizer test. `Strand` macOS and `NOOPiOS` iOS both built locally - no default CI compiles either.
…ing the largest The bundle picked, per sidecar kind, whichever generation file was biggest and shipped only that one. Size is not recency: a backfill burst rolls a huge generation that then wins every export until it ages out, so the export can carry a stale window while the night it was taken for sits in a smaller, newer file that is dropped. Measured on the 2026-09-05 export: the shipped `oura-raw.jsonl` spanned 2026-09-04 10:58:22 -> 12:03:47 UTC, the previous day's post-reflash backfill, and the hypnogram ledger then reported a plausible-looking 2026-09-04 row for a bundle exported on 09-05 - a wrong answer, not a missing one. Merge the generations chronologically instead of choosing between them. The entry cap already keeps the tail, so "the newest records survive" falls out of the two rules composed rather than being a third rule to get right. The selection logic is pure, so it lives in WhoopStore where `swift test` covers it (`app-build.yml` is disabled, so nothing in the app target is CI-validated); `TestBundleAssembler` keeps only the file I/O. Shipping every generation as separate entries was the other option and is worse: extra entries compete for the same byte budget under the fair-share allocator and would starve the very file being protected. Verification: WhoopStore `swift test` 503 tests / 0 failures (14 new); StrandTests/TestBundleAssemblerTests 19/19 including three end-to-end cases over real files, one reproducing the 09-05 generation layout; macOS app builds (`xcodebuild -scheme Strand -destination 'platform=macOS'`); doc_comment_lint clean. Not yet verified against a device export - that needs this build on the phone and one morning capture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YKP96uMkGnnFhrp4VuXnPD
…ump this branch does not carry The four preceding commits were cherry-picked from integration/oura-full, where an SpO2 raw-sample research sidecar exists. It was deliberately dropped from the upstream split (fix/oura-spo2-drop-dc-raw / ryanbr#2152 carries only the persist-time filter), so on this branch "spo2" is not a registered sidecar kind: the fixture name "oura-spo2.jsonl" fell through normalizedOuraEntryName (nil) and was treated as untrimmable by capEntries, failing two tests for a reason unrelated to what they assert. Swap the fixture to the CVA-PPG dump, which this branch does register and which is the bulky sidecar in practice. The rules under test (generation normalisation, max-min fair capping, merge-plan ordering) are kind-agnostic - nothing about the assertions changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Thanks @pipiche38, and welcome. This is careful work: three distinct failure modes, each measured against a real corpus rather than reasoned about, and each fixed separately. The one I would have missed is the third, because a largest-wins bundler does not merely drop a night, it ships the WRONG one and says nothing. One blocker, and it is not visible from your PRThe parity ledger fails on this branch:
You could not have known, and your 15 green checks are not wrong: that gate is path-filtered to The good news: your rationale is already the right one, it just needs recording
Yours is the third, and I verified the claim rather than taking it:
"The Android bundler has nothing to merge yet" is exactly true, and that is a concrete platform rationale rather than a shrug. What is needed is an entry in Happy to add it if you would rather not touch the governance registry on a first PR, or to review yours. Notes, none blocking
The scope note is accurate and worth keeping in the commit. I cannot build the Apple app target on this machine, so the Swift side of my review is reading plus your CI: 15 of 15 green, including |
|
Correcting my own comment: the disposition I told you to write does not clear this, and there is nothing for you to do. I quoted The reason is in if finding.rule not in BASELINE_REASONS:
raise ValueError(f"finding {finding.identity} has no reviewed compact-baseline disposition")It raises for any rule outside a fixed set of five, and consults no registry on the way. What actually settles itRe-deriving the authority, which is a maintainer action. Once the three functions are part of the base inventory they stop being NEW, and 257 files to 258, 1870 functions to 1873, and findings unchanged at 295. So nothing is being waved through: your declarations enter the recorded inventory, and no new debt is accepted. So the plan is to merge this and re-derive behind it, the same way main was re-derived yesterday. Your PR does not need another commit. Worth saying, since it is not your problem but it cost you a review roundThe documented process and the code disagree for this rule, and the error text points at the registry that cannot help. That is a governance bug rather than a fact about your change, and I will file it. Your Apple-only rationale is still exactly right and belongs in the merge message, it simply is not expressed as a disposition entry. Everything else in my review stands. Nice work. |
#2154 adds OuraSidecarGenerations.swift under Packages/WhoopStore, which is inside the scan, so the authority moves: 257 Swift files to 258 and 1870 Swift functions to 1873. Findings are unchanged at 295, so nothing new is accepted as debt. The three declarations simply enter the recorded inventory. The change is Apple-only for a verified reason rather than an unexamined one: TestBundleAssembler.kt names Oura zero times, and OuraMotionDump and OuraRealStepsDump are built by OuraLiveSource but referenced by no exporter, so Android writes those files and ships none of them. Its bundler has nothing to merge yet. The wider gap is #2151. This commit rather than a disposition because the documented route does not work for `add-unpaired-function`: build_compact_baseline raises for any rule outside BASELINE_REASONS and consults no registry on the way. I wrote the three platform_specific entries in the documented shape, with exact identities and matching identity_sha256, and the failure did not move. Tracked in #2163. Regenerated from a PRISTINE clone. parity_ledger_baseline.json came back byte-identical and is untouched here. Fifth staleness event, and the first that was NOT inherited by an unrelated PR: the daily run added yesterday would have caught this tomorrow morning, and doing it now means it never reaches anyone else.
The problem
A Test Centre export is capped at 20 MB and the Oura raw-wire sidecar (
oura-raw.jsonl, the onlyfile a protocol fact can be re-derived from) kept losing the night to it, three separate ways —
measured across the August capture corpus, where 11 of 31 nights' sidecars did not contain the
morning drain and the offline reframer refused them (
analysis/oura-hypnogram-ledger.md):raw wire capture with 1.9 % — 8 min of an 8.4 h night (
…-260809-0716.zip).files, and the export only ever shipped the live one — a 30-second session, while the night sat in a
generation nothing looked at (
…-260810).which is not the newest — a morning export could ship a stale, older night while the current one
was smaller and got dropped (worse than the omission count suggests: the tool silently reports the
wrong night when the sidecar happens to hold an older drain).
The fix, in the order the commits landed
capEntriesshares the cap max-min fairly (water-filling), not in proportion to size: a modeststream ships WHOLE, the bytes come off the stream that is merely bulky. Pure, tested
(
testFairAllowancesIsWaterFillingAndNeverBreachesBudget,testSmallHighValueSidecarSurvivesABulkyNeighbour).OuraRawDumpis scoped to ONE capture session and never rolls mid-drain — a roll happens onlyat session boundaries, so a drain is always in one file.
.jsonl.1,.jsonl.2, …) instead of one live file,and
normalizedOuraEntryNamerecognises a numeric generation suffix so the bundler can see them.OuraSidecarGenerations.mergePlan(new, pure, package-level inWhoopStore, 14 tests): thegenerations behind one entry name are merged oldest → newest under the ceiling, so the cap's
keep-the-tail trim preserves the newest capture. Replaces largest-wins. Deterministic regardless of
directory-listing order; ordered by
kinds, not discovery.Scope
Apple-only. Android's
TestBundleAssembler.ktexports no Oura sidecars at all today (itsOuraMotionDump/OuraRealStepsDumpwrite files nobody exports) — a wider, pre-existing gap noted on#2151, not a twin of this.
OuraSidecarGenerationsis package-level and has no Kotlin twin becausethe Android bundler has nothing to merge yet.
Verification
Strand:BUILD SUCCEEDED+StrandTests1767/0 (1 pre-existing skip) — app-targetSwift, verified via
xcodebuild … test;TestBundleAssemblerTestsandOuraRawDumpSessionTestsgreen.
WhoopStoreswift test560/0,OuraSidecarGenerationsTests14/14.doc_comment_lint.pyOK.integration/oura-fullsince 2026-09-05. The09-05/09-06 morning export was the first off the fixed build and carried the right night's drain,
closing the ledger's "limitation 4"; every nightly
raw==storedledger row since (09-06 → 09-12)is computed from sidecars this fix kept.
Relates to #1716 (the holed-hypnogram investigation this capture-coverage defect was found under).