Skip to content

fix(diagnostics): Test Centre export dropped the newer Oura sidecars, and trimmed JSONL files mid-record - #2151

Closed
pipiche38 wants to merge 2 commits into
ryanbr:mainfrom
pipiche38:fix/oura-testcentre-export-sidecars
Closed

pipiche38 wants to merge 2 commits into
ryanbr:mainfrom
pipiche38:fix/oura-testcentre-export-sidecars

Conversation

@pipiche38

Copy link
Copy Markdown

Two export-bundler bugs, one file

1. Three Oura sidecars never made it into the bundle. TestBundleAssembler.normalizedOuraEntryName()
hard-coded the Oura diagnostics kinds it recognises to the original three (raw/ibihr/activity).
The cva-ppg, motion and real-steps dump writers shipped later and have been writing real files
on-device the whole time — but the bundler didn't know their kind, so it silently dropped all three
from every Test Centre export. Confirmed against two real exports where the strap log shows the writers
firing and the files never appear in the zip.

Fix: collapse the four places that independently listed sidecar kinds down to one source of truth
(ouraSidecarKinds), so a future dump writer needs exactly one line added to be picked up by
normalizedOuraEntryName / ouraDiagnosticEntries / trimmableNames / ouraSidecarNames together,
instead of silently missing export.

2. Oversized JSONL sidecars were trimmed to a raw byte count. capEntries keeps the most-recent tail
of an oversized file by byte count — fine for a freeform text log, but a byte-count cut on a
newline-delimited JSONL file can land mid-record. Confirmed in production: a real overnight export
(2026-07-30/31) shipped oura-cva-ppg.jsonl, oura-real-steps.jsonl and oura-motion.jsonl each with a
corrupted first line, the first time their combined size crossed the 20 MB cap.

Fix: trimToLineBoundary — after the proportional byte-count trim, drop bytes up to and including
the first newline, so every kept line is a complete record. Every trimmable name is JSONL, so this is
safe across the board; a single very-long line with no newline is returned unchanged, never worse than
before.

Verification

  • StrandTests 1753/0 (1 pre-existing skip) on the rebased tip — app-target Swift, so verified via
    xcodebuild … test, not just swift test. TestBundleAssemblerTests 14/14, including the new
    testCapSnapsTrimmedTailToLineBoundary (every kept line parses as valid JSON after a forced mid-file
    trim) and a normalisation case for each newly-recognised sidecar kind.
  • Apple-only, and deliberately so. Android does have OuraMotionDump/OuraRealStepsDump writers, but
    its TestBundleAssembler.kt exports no Oura sidecars at all today (only raw-capture.jsonl) — so
    the Android situation is a wider, pre-existing gap ("Oura sidecars are never exported"), not a twin of
    this bug ("three of six sidecars are dropped"). Worth its own issue; not smuggled into this PR.

pipiche38 and others added 2 commits September 12, 2026 21:09
…al-steps sidecars

TestBundleAssembler.normalizedOuraEntryName() hardcoded the Oura Diagnostics-dir
sidecar kinds it recognizes to the original three (raw/ibihr/activity). The
cva-ppg, motion, and real-steps dump writers shipped afterward and were writing
real files on-device the whole time, but the export bundler silently dropped
all three from every Test Centre bundle since it didn't know their kind -
confirmed against two real exports where the strap log showed the writers
firing but the files never made it into the bundle.

Collapsed the 4 places that independently listed sidecar kinds down to one
source of truth (ouraSidecarKinds), so a future dump writer needs exactly one
line added to be picked up by normalizedOuraEntryName/ouraDiagnosticEntries/
trimmableNames/ouraSidecarNames together instead of silently missing export.

(The original commit also carried a hardcoded test-branch build marker for the
strap-log header, explicitly flagged "must never ship". It is dropped here along
with its two call sites, and the follow-up commit that existed only to remove it
is dropped with it.)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013EHFxGQzockMYJqDXYB3S1
TestBundleAssembler.capEntries trimmed oversized bundle files to a raw byte
count (keeping the most-recent tail), which is safe for a single freeform
text file but not for the newline-delimited JSONL sidecars: a byte-count cut
can land mid-record. Confirmed in production - a real overnight export
(2026-07-30/31) shipped oura-cva-ppg.jsonl, oura-real-steps.jsonl, and
oura-motion.jsonl each with a corrupted first line once the sidecars' combined
size crossed the 20 MB cap for the first time.

Adds trimToLineBoundary: after the proportional byte-count trim, drop bytes up
to and including the first newline, so every kept line is a complete record.
Every trimmable name is newline-delimited JSONL, so this is safe across the
board; a single very-long line (no newline found) is returned unchanged,
never worse than before.

Tested: new testCapSnapsTrimmedTailToLineBoundary (every kept line parses as
valid JSON after a forced mid-file trim) and testTrimToLineBoundaryHandlesNoNewline;
existing cap tests unaffected (their fixtures have no embedded newlines, so
the snap is a no-op for them). macOS Strand target builds; TestBundleAssemblerTests
13/13 pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013EHFxGQzockMYJqDXYB3S1
ryanbr pushed a commit that referenced this pull request Sep 13, 2026
A Test Centre export is capped at 20 MB and the Oura raw-wire sidecar,
the only file a protocol fact can be re-derived from, kept losing the
night to it three separate ways. Measured across the August capture
corpus rather than reasoned about: 11 of 31 nights' sidecars did not
contain the morning drain, and the offline reframer refused them.

  1. Proportional capping handed 53.9% of the budget to a bulky research
     dump and left the raw wire 1.9%, 8 minutes of an 8.4 hour night.
  2. A size-based roll could split one capture session across two files,
     and the export only ever shipped the live one, so a 30-second
     session went out while the night sat in a generation nothing looked
     at.
  3. Largest-wins between generations is not newest-wins, so a morning
     export could ship a stale night. Worse than an omission: the tool
     reports the WRONG night and says nothing.

The fixes are separable. capEntries shares the cap max-min fairly by
water-filling, so a modest stream ships whole and the bytes come off the
merely bulky one. OuraRawDump is scoped to one capture session and rolls
only at session boundaries. Generations are kept as a ring and
normalizedOuraEntryName recognises the suffix. OuraSidecarGenerations
merges the generations behind one entry name oldest to newest under the
ceiling, so the cap's keep-the-tail trim preserves the newest capture.

Apple-only, and the rationale was verified rather than taken:
TestBundleAssembler.kt names Oura zero times, and OuraMotionDump and
OuraRealStepsDump are constructed by OuraLiveSource but referenced by no
exporter, so they write files nothing ships. The Android bundler has
nothing to merge yet. That wider gap belongs to #2151.

capEntries is the widest part of this, since water-filling changes every
export rather than only the Oura path.

The three new functions in OuraSidecarGenerations are one-sided, and the
authority is re-derived in the commit after this one. The documented
route for that, a typed disposition, does not work for this rule; that is
tracked in #2163 and is not a fact about this change.

15 of 15 CI checks green, including every package test suite and both app
builds. StrandTests 1767 with 0 failures, reported by the contributor.

Reported, measured and fixed by @pipiche38.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ryanbr added a commit that referenced this pull request Sep 13, 2026
#2154 adds OuraSidecarGenerations.swift under Packages/WhoopStore, which
is inside the scan, so the authority moves: 257 Swift files to 258 and
1870 Swift functions to 1873. Findings are unchanged at 295, so nothing
new is accepted as debt. The three declarations simply enter the recorded
inventory.

The change is Apple-only for a verified reason rather than an unexamined
one: TestBundleAssembler.kt names Oura zero times, and OuraMotionDump and
OuraRealStepsDump are built by OuraLiveSource but referenced by no
exporter, so Android writes those files and ships none of them. Its
bundler has nothing to merge yet. The wider gap is #2151.

This commit rather than a disposition because the documented route does
not work for `add-unpaired-function`: build_compact_baseline raises for
any rule outside BASELINE_REASONS and consults no registry on the way. I
wrote the three platform_specific entries in the documented shape, with
exact identities and matching identity_sha256, and the failure did not
move. Tracked in #2163.

Regenerated from a PRISTINE clone. parity_ledger_baseline.json came back
byte-identical and is untouched here.

Fifth staleness event, and the first that was NOT inherited by an
unrelated PR: the daily run added yesterday would have caught this
tomorrow morning, and doing it now means it never reaches anyone else.
@ryanbr

ryanbr commented Sep 13, 2026

Copy link
Copy Markdown
Owner

This conflicts with main now, and the reason is friendly rather than a problem with it.

#2154 landed, which rewrote the same export path: capEntries shares the cap by water-filling instead of proportionally, OuraRawDump is scoped to one capture session, generations are kept as a ring, and OuraSidecarGenerations.mergePlan merges them oldest to newest under the ceiling. Your own #2154 description named this issue as the wider, pre-existing gap, so the two were always adjacent; the larger one simply went first and moved the ground.

Worth rebasing and then deciding what is left rather than replaying it. Some of what this addresses may already be covered by the merge, and the part that is not will read differently against the new shape.

The seven other Oura PRs merged clean, so main now also carries the 5 minute fetch cadence, the freshly-offloaded scoring, the onset-rekey guard, the SpO2 column change, the 10 second anchor gap and both documentation corrections. That is a lot of adjacent movement in one evening, which is another reason to rebase before re-reading this one.

No review yet on the substance: I would rather read it against the tree it will actually live in.

@pipiche38

Copy link
Copy Markdown
Author

Rebased onto main@2a4c157 to see what was left: nothing. Both commits come out empty — #2154 was
built on top of this branch and carried the whole of it (ouraSidecarKinds, trimToLineBoundary,
the line-boundary test and the per-kind normalisation cases are all on main). The only conflicts
were the two doc-comment lines #2154 rewrote when it replaced the proportional trim with the
fair-share one.

Closing this as superseded by #2154 rather than re-landing an empty diff.

@pipiche38 pipiche38 closed this Sep 13, 2026
@pipiche38
pipiche38 deleted the fix/oura-testcentre-export-sidecars branch September 24, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants