Skip to content

fix(diagnostics): keep the night's raw Oura sidecar in the export — session-scoped, ring-buffered, fairly capped, generations merged - #2154

Merged
ryanbr merged 8 commits into
ryanbr:mainfrom
pipiche38:fix/oura-sidecar-generations
Sep 13, 2026
Merged

ryanbr merged 8 commits into
ryanbr:mainfrom
pipiche38:fix/oura-sidecar-generations

Conversation

@pipiche38

Copy link
Copy Markdown

The problem

A Test Centre export is capped at 20 MB and the Oura raw-wire sidecar (oura-raw.jsonl, the only
file a protocol fact can be re-derived from) kept losing the night to it, three separate ways —
measured across the August capture corpus, where 11 of 31 nights' sidecars did not contain the
morning drain
and the offline reframer refused them (analysis/oura-hypnogram-ledger.md):

  1. Proportional-to-size capping handed 53.9 % of the budget to a bulky research dump and left the
    raw wire capture with 1.9 % — 8 min of an 8.4 h night (…-260809-0716.zip).
  2. The sidecar rolled mid-drain: a size-based roll could split one capture session across two
    files, and the export only ever shipped the live one — a 30-second session, while the night sat in a
    generation nothing looked at (…-260810).
  3. Largest-wins between generations: once generations were kept, the bundler picked the biggest,
    which is not the newest — a morning export could ship a stale, older night while the current one
    was smaller and got dropped (worse than the omission count suggests: the tool silently reports the
    wrong night when the sidecar happens to hold an older drain).

The fix, in the order the commits landed

  1. capEntries shares the cap max-min fairly (water-filling), not in proportion to size: a modest
    stream ships WHOLE, the bytes come off the stream that is merely bulky. Pure, tested
    (testFairAllowancesIsWaterFillingAndNeverBreachesBudget, testSmallHighValueSidecarSurvivesABulkyNeighbour).
  2. OuraRawDump is scoped to ONE capture session and never rolls mid-drain — a roll happens only
    at session boundaries, so a drain is always in one file.
  3. A RING of raw-sidecar generations is kept (.jsonl.1, .jsonl.2, …) instead of one live file,
    and normalizedOuraEntryName recognises a numeric generation suffix so the bundler can see them.
  4. OuraSidecarGenerations.mergePlan (new, pure, package-level in WhoopStore, 14 tests): the
    generations behind one entry name are merged oldest → newest under the ceiling, so the cap's
    keep-the-tail trim preserves the newest capture. Replaces largest-wins. Deterministic regardless of
    directory-listing order; ordered by kinds, not discovery.
  5. Test-fixture adaptation for this branch (see the header note).

Scope

Apple-only. Android's TestBundleAssembler.kt exports no Oura sidecars at all today (its
OuraMotionDump/OuraRealStepsDump write files nobody exports) — a wider, pre-existing gap noted on
#2151, not a twin of this. OuraSidecarGenerations is package-level and has no Kotlin twin because
the Android bundler has nothing to merge yet.

Verification

  • macOS Strand: BUILD SUCCEEDED + StrandTests 1767/0 (1 pre-existing skip) — app-target
    Swift, verified via xcodebuild … test; TestBundleAssemblerTests and OuraRawDumpSessionTests
    green. WhoopStore swift test 560/0, OuraSidecarGenerationsTests 14/14.
  • doc_comment_lint.py OK.
  • Hardware: this exact behaviour has shipped on integration/oura-full since 2026-09-05. The
    09-05/09-06 morning export was the first off the fixed build and carried the right night's drain,
    closing the ledger's "limitation 4"; every nightly raw==stored ledger row since (09-06 → 09-12)
    is computed from sidecars this fix kept.

Relates to #1716 (the holed-hypnogram investigation this capture-coverage defect was found under).

pipiche38 and others added 7 commits September 12, 2026 21:09
…al-steps sidecars

TestBundleAssembler.normalizedOuraEntryName() hardcoded the Oura Diagnostics-dir
sidecar kinds it recognizes to the original three (raw/ibihr/activity). The
cva-ppg, motion, and real-steps dump writers shipped afterward and were writing
real files on-device the whole time, but the export bundler silently dropped
all three from every Test Centre bundle since it didn't know their kind -
confirmed against two real exports where the strap log showed the writers
firing but the files never made it into the bundle.

Collapsed the 4 places that independently listed sidecar kinds down to one
source of truth (ouraSidecarKinds), so a future dump writer needs exactly one
line added to be picked up by normalizedOuraEntryName/ouraDiagnosticEntries/
trimmableNames/ouraSidecarNames together instead of silently missing export.

(The original commit also carried a hardcoded test-branch build marker for the
strap-log header, explicitly flagged "must never ship". It is dropped here along
with its two call sites, and the follow-up commit that existed only to remove it
is dropped with it.)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013EHFxGQzockMYJqDXYB3S1
TestBundleAssembler.capEntries trimmed oversized bundle files to a raw byte
count (keeping the most-recent tail), which is safe for a single freeform
text file but not for the newline-delimited JSONL sidecars: a byte-count cut
can land mid-record. Confirmed in production - a real overnight export
(2026-07-30/31) shipped oura-cva-ppg.jsonl, oura-real-steps.jsonl, and
oura-motion.jsonl each with a corrupted first line once the sidecars' combined
size crossed the 20 MB cap for the first time.

Adds trimToLineBoundary: after the proportional byte-count trim, drop bytes up
to and including the first newline, so every kept line is a complete record.
Every trimmable name is newline-delimited JSONL, so this is safe across the
board; a single very-long line (no newline found) is returned unchanged,
never worse than before.

Tested: new testCapSnapsTrimmedTailToLineBoundary (every kept line parses as
valid JSON after a forced mid-file trim) and testTrimToLineBoundaryHandlesNoNewline;
existing cap tests unaffected (their fixtures have no embedded newlines, so
the snap is a no-op for them). macOS Strand target builds; TestBundleAssemblerTests
13/13 pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013EHFxGQzockMYJqDXYB3S1
…ion to size

`TestBundleAssembler.capEntries` held the bundle to 20 MB by splitting the budget
across `trimmableNames` IN PROPORTION to each file's size. That hands the budget to
the bulkiest stream, and for the Oura sidecars bulk is close to inversely correlated
with diagnostic value.

Measured on a real export (`noop-master-iOS-v9.3.1-260809-0716.zip`, six trimmable
sidecars over a 20,863,648-byte budget):

    oura-spo2.jsonl        11,242,865   53.9 %
    oura-real-steps.jsonl   3,782,590   18.1 %
    oura-motion.jsonl       2,827,017   13.5 %
    oura-cva-ppg.jsonl      2,417,008   11.6 %
    oura-raw.jsonl            388,064    1.9 %
    oura-activity.jsonl       206,921    1.0 %

Trimmable total 20,864,465 against the computed budget of 20,863,648 — agreeing to
817 bytes of integer-floor rounding, so this is the cap firing exactly as written.

`oura-raw.jsonl` is the UNDECODED wire bytes: the one file in the bundle from which a
protocol fact can be re-derived independently of NOOP's own decoder, which is the
entire reason `OuraRawDump` exists. It shipped 3,123 records — 8 min 43 s of an 8.4 h
night whose full history drain had just completed. The verifiers that read the raw
sidecar ON PURPOSE (so a decode bug cannot launder itself into the number checking it)
could not run at all that night, while 11 MB of SpO2 samples, also present in the
SQLite the same bundle ships, rode along whole.

Replaced with max-min fair (water-filling) allocation: walk the streams smallest-first
offering each an equal share of what is left, so a stream under its share is kept WHOLE
and its surplus rolls forward to the ones still over. It makes no editorial claim about
which sidecar matters — the small file wins because it is small, not because it is
favoured. Integer floor keeps the sum at or under budget, so the cap can never breach.

Because each shipped size IS its old proportional share, the on-disk sizes are the
shipped sizes times one unknown scale k, and the outcome follows exactly: with six
streams the fair share is 3.48 MB, so `oura-raw.jsonl` ships WHOLE for any k up to ~9
(a sidecar directory under ~190 MB) and still gets 9x its current bytes beyond that.

A single trimmable stream still receives the whole remainder, so the original
raw-capture-only behaviour is unchanged.

`fairAllowances` is pure and separately tested. One existing test pinned the old rule
(bigger stream gets the bigger share) and now pins the new one; two tests are added —
the water-filling contract including determinism and degenerate budgets, and the real
bug shape, a modest wire capture beside a bulky neighbour.

Verification: StrandTests 1083/0 (1 skipped) including all 15 TestBundleAssembler
tests; `Strand` macOS and `NOOPiOS` iOS both built locally, which no default CI covers.
Apple-only — `TestBundleAssembler` has no Android twin, so there is no parity half.
Diagnostics only: no BLE writes, no schema change, no analytics, nothing user-facing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E7i8DiChrWDDbppKpywaSA
…never roll mid-drain

`OuraRawDump` rolled its sidecar to a `.1` sibling whenever the file passed 25 MB,
checked per record. A size threshold can fire MID-DRAIN, and on 2026-08-09 it did: it
rolled at 07:06:58 during the morning offload, carrying 32,005 records of that drain and
71 of the night's 75 0x5D records into the `.1` — which the strap-log bundle does not
collect. The bundle shipped 3,123 records covering 8 min 43 s of an 8.4 h night, and the
verifiers that read the raw sidecar ON PURPOSE (so a decode bug cannot launder itself
into the number checking it) could not run. The evidence was recoverable only by pulling
the file off the phone by hand.

The live file now holds exactly ONE session. It is rolled aside when the object resolves
its file for the first time — the START of a session — and never again while that session
runs. One OuraRawDump is built per OuraLiveSource, and a source survives BLE reconnects
(SourceCoordinator returns early when the active strap is unchanged), so a drain that
disconnects and re-engages several times still lands whole in one file: the five
reconnects of that same morning would have been a single session.

Lowering the threshold instead would make the failure MORE frequent, not less. Only a
session boundary removes it, and the corpus stays bounded anyway because a session is
bounded: an overnight drain is ~32,000 records, about 4 MB.

The multi-day corpus is deliberately given up, because the bundle replaces it — every
capture now carries its own COMPLETE raw sidecar, and bundles are what get archived. One
previous session is kept as `.1` so a session that ends without an export is not lost.

The per-record `stat` goes with it: the file starts empty each session, so a running byte
counter is authoritative and costs no filesystem call. That counter enforces a 25 MB
SESSION ceiling which STOPS appending and logs once rather than rolling — rolling
mid-session is the exact failure being removed, so it must not reappear at a new
threshold. A real drain is nowhere near it; it exists so a pathological backlog cannot
fill the disk.

`directory` is injectable (defaulting to Application Support) purely so the roll is
testable: this class had no coverage at all, because it hardcoded its path. Six tests now
pin the session contract — a whole session in one file with nothing rolled aside, a new
session rolling the previous one aside and starting empty, exactly two generations
retained, an empty leftover NOT spending the retained slot, a silent session touching
nothing, and the ceiling bounding a session WITHOUT rolling it.

Verification: StrandTests 1089/0 (1 skipped); `Strand` macOS and `NOOPiOS` iOS both built
locally, which no default CI covers. Apple-only — OuraRawDump has no Android twin, so
there is no parity half. Diagnostics only: no BLE writes, no schema change, no analytics,
nothing user-facing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01E7i8DiChrWDDbppKpywaSA
…the one that holds the capture

Session scoping (c9fd883) fixed the mid-drain roll and then lost a night a
different way. Measured on the 2026-08-10 capture: the app relaunched twice
after wake (07:45 and 08:17 local), so both retained slots — the live file and
its single `.1` — held post-wake sessions of 87 KB and 44 KB, and the night's
4 MB was gone before it could be pulled off the phone.

Rotation ate a night every ~5-6 days at random. One-deep session scoping ate it
every morning the app restarted twice, which is the normal morning. An iOS app
is relaunched whenever the user opens it, so the number of session boundaries
between a night and its export is not something OuraRawDump can predict; it can
only keep enough of them.

Two changes, and the second is what makes the first useful:

* OuraRawDump keeps a numbered generation ring (`.1` ... `.8`) instead of one
  `.1`. A session under `minRolledBytes` (16 KB) is discarded rather than
  rolled, so a connect that captured essentially nothing cannot evict a real
  overnight capture, and the retained generations are pruned oldest-first to a
  64 MB budget so the ring is bounded in BYTES, not merely in file count.

* TestBundleAssembler.normalizedOuraEntryName now maps `.jsonl.<n>` to the same
  entry name as the live file. Its existing largest-wins rule then ships the
  generation that actually holds the capture. Without this the export could only
  ever carry the live file — a 30-second morning session — and keeping
  generations on disk would help nobody without a USB cable.

Largest-wins means a morning export can ship a sidecar from an earlier session
than the live one. That is the point, and it is safe to choose blind: every line
carries its own `utc`/`iso`, so an analysis scoped to a night either finds that
night's records or honestly finds none.

Diagnostics only - no BLE writes, no protocol change, nothing scored. Apple-only
(Android has no raw sidecar writer), so there is no parity half.

Verified: StrandTests 1092/0 (1 skipped), including 4 new OuraRawDump tests (the
regression test replays the two post-wake relaunches) and a new normalizer test.
`Strand` macOS and `NOOPiOS` iOS both built locally - no default CI compiles
either.
…ing the largest

The bundle picked, per sidecar kind, whichever generation file was biggest and
shipped only that one. Size is not recency: a backfill burst rolls a huge
generation that then wins every export until it ages out, so the export can carry
a stale window while the night it was taken for sits in a smaller, newer file that
is dropped. Measured on the 2026-09-05 export: the shipped `oura-raw.jsonl`
spanned 2026-09-04 10:58:22 -> 12:03:47 UTC, the previous day's post-reflash
backfill, and the hypnogram ledger then reported a plausible-looking 2026-09-04
row for a bundle exported on 09-05 - a wrong answer, not a missing one.

Merge the generations chronologically instead of choosing between them. The entry
cap already keeps the tail, so "the newest records survive" falls out of the two
rules composed rather than being a third rule to get right.

The selection logic is pure, so it lives in WhoopStore where `swift test` covers
it (`app-build.yml` is disabled, so nothing in the app target is CI-validated);
`TestBundleAssembler` keeps only the file I/O.

Shipping every generation as separate entries was the other option and is worse:
extra entries compete for the same byte budget under the fair-share allocator and
would starve the very file being protected.

Verification: WhoopStore `swift test` 503 tests / 0 failures (14 new);
StrandTests/TestBundleAssemblerTests 19/19 including three end-to-end cases over
real files, one reproducing the 09-05 generation layout; macOS app builds
(`xcodebuild -scheme Strand -destination 'platform=macOS'`); doc_comment_lint
clean. Not yet verified against a device export - that needs this build on the
phone and one morning capture.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YKP96uMkGnnFhrp4VuXnPD
…ump this branch does not carry

The four preceding commits were cherry-picked from integration/oura-full, where
an SpO2 raw-sample research sidecar exists. It was deliberately dropped from the
upstream split (fix/oura-spo2-drop-dc-raw / ryanbr#2152 carries only the persist-time
filter), so on this branch "spo2" is not a registered sidecar kind: the fixture
name "oura-spo2.jsonl" fell through normalizedOuraEntryName (nil) and was
treated as untrimmable by capEntries, failing two tests for a reason unrelated
to what they assert. Swap the fixture to the CVA-PPG dump, which this branch
does register and which is the bulky sidecar in practice. The rules under test
(generation normalisation, max-min fair capping, merge-plan ordering) are
kind-agnostic - nothing about the assertions changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ryanbr

ryanbr commented Sep 13, 2026

Copy link
Copy Markdown
Owner

Thanks @pipiche38, and welcome. This is careful work: three distinct failure modes, each measured against a real corpus rather than reasoned about, and each fixed separately. The one I would have missed is the third, because a largest-wins bundler does not merely drop a night, it ships the WRONG one and says nothing.

One blocker, and it is not visible from your PR

The parity ledger fails on this branch:

FAIL compact baseline drift in finding add-unpaired-function|...
  OuraSidecarGenerations.swift:39: new one-sided swift function ...::entryName/1#1
  OuraSidecarGenerations.swift:51: new one-sided swift function ...::classify/2#1
  OuraSidecarGenerations.swift:93: new one-sided swift function ...::mergePlan/3#1

Packages/WhoopStore/Sources is inside the ledger's Swift scan, so a new file there with no Kotlin twin is debt the governance gate wants declared.

You could not have known, and your 15 green checks are not wrong: that gate is path-filtered to Tools/parity_* and Tools/tests/**, so it does not run on product changes. It is the same filter that has put a red check on three other contributors' PRs this week for drift they did not cause. A daily run on main landed yesterday (#2142), so the first thing to see this would be that, the morning after merge.

The good news: your rationale is already the right one, it just needs recording

Tools/PARITY_GOVERNANCE.md gives exactly three ways to resolve a new one-sided declaration: implement the twin, an experimental disposition with an issue and an expiry, or a durable platform_specific disposition with a concrete rationale.

Yours is the third, and I verified the claim rather than taking it:

  • android/app/src/main/java/com/noop/testcentre/TestBundleAssembler.kt mentions Oura zero times
  • OuraMotionDump and OuraRealStepsDump are constructed by OuraLiveSource, so they write, and are referenced by no exporter, so nothing ships them

"The Android bundler has nothing to merge yet" is exactly true, and that is a concrete platform rationale rather than a shrug.

What is needed is an entry in Tools/parity_dispositions.json naming the three identities, the platform, and that reason. The registry is manually reviewed on purpose and regeneration cannot populate it, so it has to be written rather than generated.

Happy to add it if you would rather not touch the governance registry on a first PR, or to review yours.

Notes, none blocking

capEntries is the widest part of this. Water-filling changes every export, not only the Oura path, so it is the piece most worth a second reader. The property your test asserts, that a modest stream ships whole and the bytes come off the merely bulky one, is the right shape, and CI ran those tests.

The scope note is accurate and worth keeping in the commit. TestBundleAssembler.kt exporting no Oura sidecars is a real, wider gap and belongs to #2151 rather than here.

I cannot build the Apple app target on this machine, so the Swift side of my review is reading plus your CI: 15 of 15 green, including test (WhoopStore) and both app builds.

@ryanbr

ryanbr commented Sep 13, 2026

Copy link
Copy Markdown
Owner

Correcting my own comment: the disposition I told you to write does not clear this, and there is nothing for you to do.

I quoted PARITY_GOVERNANCE.md, which lists three ways to settle a new one-sided declaration, and assumed the third applied without testing it. I wrote all three entries in the documented platform_specific shape, exact identities, matching identity_sha256, and re-ran the ledger. Same failure, unchanged.

The reason is in build_compact_baseline:

if finding.rule not in BASELINE_REASONS:
    raise ValueError(f"finding {finding.identity} has no reviewed compact-baseline disposition")

It raises for any rule outside a fixed set of five, and consults no registry on the way. add-unpaired-function is not among the five, so a typed disposition is never reached. The message says "has no reviewed compact-baseline disposition", which is what led me to a registry that this path does not read.

What actually settles it

Re-deriving the authority, which is a maintainer action. Once the three functions are part of the base inventory they stop being NEW, and _new_unpaired_diagnostics has nothing to report. Verified on your branch:

OK no NEW parity ledger findings (295 baselined; 258 Swift files ...; 1873 Swift functions ...)

257 files to 258, 1870 functions to 1873, and findings unchanged at 295. So nothing is being waved through: your declarations enter the recorded inventory, and no new debt is accepted.

So the plan is to merge this and re-derive behind it, the same way main was re-derived yesterday. Your PR does not need another commit.

Worth saying, since it is not your problem but it cost you a review round

The documented process and the code disagree for this rule, and the error text points at the registry that cannot help. That is a governance bug rather than a fact about your change, and I will file it. Your Apple-only rationale is still exactly right and belongs in the merge message, it simply is not expressed as a disposition entry.

Everything else in my review stands. Nice work.

@ryanbr
ryanbr merged commit 5fc6c7e into ryanbr:main Sep 13, 2026
15 checks passed
ryanbr added a commit that referenced this pull request Sep 13, 2026
#2154 adds OuraSidecarGenerations.swift under Packages/WhoopStore, which
is inside the scan, so the authority moves: 257 Swift files to 258 and
1870 Swift functions to 1873. Findings are unchanged at 295, so nothing
new is accepted as debt. The three declarations simply enter the recorded
inventory.

The change is Apple-only for a verified reason rather than an unexamined
one: TestBundleAssembler.kt names Oura zero times, and OuraMotionDump and
OuraRealStepsDump are built by OuraLiveSource but referenced by no
exporter, so Android writes those files and ships none of them. Its
bundler has nothing to merge yet. The wider gap is #2151.

This commit rather than a disposition because the documented route does
not work for `add-unpaired-function`: build_compact_baseline raises for
any rule outside BASELINE_REASONS and consults no registry on the way. I
wrote the three platform_specific entries in the documented shape, with
exact identities and matching identity_sha256, and the failure did not
move. Tracked in #2163.

Regenerated from a PRISTINE clone. parity_ledger_baseline.json came back
byte-identical and is untouched here.

Fifth staleness event, and the first that was NOT inherited by an
unrelated PR: the daily run added yesterday would have caught this
tomorrow morning, and doing it now means it never reaches anyone else.
@pipiche38
pipiche38 deleted the fix/oura-sidecar-generations branch September 24, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants