[bhelm]

WHOOP 5/MG finally has a working 100 Hz raw-IMU capture workflow

This closes the reverse-engineering loop opened in #423 and turns the result into something developers can build on.

NOOP already understood the WHOOP 5/MG's 100 Hz, 6-axis IMU format, but it could not reliably start and record that stream on demand. Several individually plausible commands acknowledged successfully without producing data.

Hardware validation established that the producer requires an ordered sequence:

1. START_RAW_DATA (81)
2. TOGGLE_IMU_MODE (106) with [1, 1]

With that sequence, a regular WHOOP 5/MG emits continuous type-43/type-51 packets containing 100 Hz accelerometer and gyroscope data. No research-band firmware or developer-only hardware is required.

This PR does more than expose that command sequence. It introduces a complete, bounded raw-data capture workflow for collecting, annotating, retaining and exporting labeled sensor data.

A reusable Raw Data Collector

The new Raw Data Collector lets developers create intentional recording sessions instead of relying on incidental diagnostic logs.

A recording session can:

• explicitly start and stop the 100 Hz IMU producer;
• remain available across temporary Bluetooth interruptions;
• accept later packets delivered through history synchronization;
• show connection, synchronization and IMU coverage while recording;
• show the recording duration and stored IMU size;
• edit the selected start and end time;
• attach a general session comment;
• add timestamped markers with quick types and free-text notes;
• adjust marker timestamps by ±10 seconds or return them to the current recording time;
• edit and delete markers after recording;
• retain recordings for repeated exports;
• show when a session was last exported;
• delete an individual session or all sessions with confirmation;
• export the selected interval as a self-contained ZIP archive.

Markers make the collector useful for gathering labeled datasets while an activity is happening. Starts, stops, transitions, mistakes and notable moments can be recorded without modifying the sensor stream itself.

This provides the missing foundation for future motion-based features. A module could, for example, use the existing low-rate motion stream for inexpensive continuous activity detection and enable 100 Hz capture only around a candidate activity.

High-rate windows could then support swimming detection, lap and stroke counting, gait analysis, workout classification, exercise and repetition recognition, posture or movement analysis, and locally evaluated activity models.

Those recognizers are deliberately not part of this PR. This PR provides the capture, labeling and export infrastructure required to develop and validate them.

Storage designed for longer recordings

High-rate IMU samples are stored outside the operational SQLite sample tables.

Each session owns a set of decoded .imus files:

• recordings are divided into fixed 30-minute UTC segments;
• each segment contains independently compressed blocks of up to 30 seconds;
• samples are stored in a decoded, documented 100 Hz, 6-axis representation;
• timestamps and receive times are retained for coverage and transport analysis;
• segment boundaries make long recordings easier to inspect and process;
• interrupted final blocks remain recoverable;
• late history data can still be associated with the applicable recording window;
• duplicate strap seconds are rejected;
• exports select the requested full-second interval;
• source recordings remain available for repeated export;
• explicit session deletion removes the associated files.

The 30-minute segmentation limits the amount of data that future analysis modules must open at once, while the internal 30-second blocks keep recording append-oriented and resilient to interruption.

Exports remain convenient for users: the selected .imus segments, metadata, markers and accompanying sensor data are delivered together in one ZIP archive.

The design intentionally stops short of implementing an always-on rolling buffer today. It nevertheless supports later features such as bounded workout recordings, deferred processing while the phone is charging, or a rolling activity-detection buffer without requiring a replacement storage format.

Observed storage and battery use

One uncontrolled Android test recorded approximately 5 hours and 47 minutes of continuous 100 Hz IMU data:

• 2,078,300 IMU samples were retained;
• the decoded IMU values represented approximately 24.9 MB before compression;
• timestamps and per-second record metadata increased that to approximately 25.4 MB;
• the compressed .imus segments occupied approximately 11.5 MB;
• the complete ZIP export, including IMU segments, regular sensor CSV files, metadata and markers, occupied approximately 13.0 MB;
• compressed IMU storage was approximately 2 MB per hour in this recording.

During the same session, the band battery indication decreased from 76% to 72%. A later database-backed discharge trace provides a more useful order of magnitude across three recordings:

• a 5-hour-47-minute overnight capture averaged approximately 0.71 battery percentage points per hour;
• two later 42–45 minute captures each averaged approximately 0.57 percentage points per hour;
• nearby non-capture periods in the same discharge averaged approximately 0.31 percentage points per hour;
• a separate 5.6-day pre-capture baseline averaged approximately 0.46 percentage points per hour.

These are observations from one band under uncontrolled conditions, not storage or battery benchmarks. Depending on the comparison baseline, active 100 Hz capture used roughly 1.5–2.3 times the normal drain in this trace. Compression depends on movement, and the battery series does not isolate IMU production, BLE transport, or subsequent history repair. The measurements provide a planning estimate for bounded recordings, not a continuous-capture runtime guarantee.

Honest capture coverage

Coverage is calculated from the IMU seconds actually stored, rather than inferred from connection state or ordinary low-rate sensor freshness.

The collector and export metadata distinguish between the requested interval, the first and last stored IMU second, the number of covered seconds, startup delay, internal gaps, and complete or incomplete recordings.

A recent 5-hour-47-minute test retained 20,783 of 20,803 requested seconds, corresponding to 99.904% coverage. The export was correctly marked incomplete because of startup, tail and isolated internal gaps.

This does not claim that Bluetooth transport can guarantee lossless capture. It ensures that downstream developers can determine exactly what was recorded and do not unknowingly treat partial data as complete.

Test Centre cleanup and platform parity

The existing WHOOP 5/MG raw-data controls were experimental developer tooling, but they were spread through ordinary Settings and did not provide a coherent recording workflow.

This PR:

• moves the experimental WHOOP 5/MG and raw-data surfaces into Test Centre;
• updates outdated wording around the current state of WHOOP 5/MG support;
• replaces the dormant rawImuSample path with session-owned file storage;
• keeps raw-session storage separate from the general diagnostics capture path;
• avoids duplicate .imuc, .imus, raw-outbox or SQLite storage;
• removes step-count, clicker and ground-truth-specific behavior from the Raw Data Collector;
• aligns the Android and Apple feature surfaces through the existing parity contract and test toolkit;
• keeps developer-oriented controls away from ordinary user settings.

The collector also includes an off-session fail-safe. If unexpected realtime type-43/type-51 packets arrive while no recording or intentional diagnostics capture is active, NOOP disarms the producer after a short grace period. History packets are not stopped, and cooldown handling prevents a delayed packet from creating a stop-command loop.

Platform status

Android

The complete workflow has been exercised on a physical Android device with a regular WHOOP 5/MG.

Validation included starting and stopping the producer, multi-hour recording, temporary Bluetooth interruption, later packet delivery, editable bounds, markers, repeated export, deletion, compressed segment parsing, sample-rate and payload checks, coverage and gap detection, and verification that only the canonical .imus path stores session IMU data.

Apple platforms

The macOS application and iOS Simulator target compile successfully, and the shared Strand test suite passes in CI.

Android and Apple implement the same session model, marker workflow, editable time ranges, coverage reporting, deletion behavior and export structure. Their parity is checked with the project's parity contract and test toolkit.

The implementation has not been exercised on a physical iPhone or against a real strap. In particular, iOS BLE capture, background behavior, filesystem behavior, Share Sheet export and the final on-device user experience remain unvalidated until someone tests them on real hardware.

History

This work builds on:

• [ble] WHOOP 5/MG raw accel/IMU: community-verified live 0x2B raw-accel stream (cmd 81) that the 5/MG allowlist never arms — gated probe + decode proposal #423, which coordinated the WHOOP 5/MG raw-IMU investigation;
• feat(ble): capture WHOOP 5/MG high-rate R22 deep buffers for research (#423) #454, which retained large R22 buffers for offline research;
• feat: decode WHOOP 5/MG raw 6-axis IMU offload buffer + activity features (#423) #455, which decoded the 1244-byte buffer as 100 Hz, 6-axis IMU;
• rawImuSample has never held a row: the writer post-dates the only capture window, and Swift has no reader at all #978 and docs(store): mark rawImuSample as intentionally-dormant capture (#978) #1345, which documented that the previous rawImuSample path remained dormant.

The earlier work established that the data existed and could be decoded. This PR supplies the missing on-demand producer sequence and turns those findings into a usable, cross-platform recording workflow.

Builds on #454 and #455.

Supersedes the dormant raw-IMU storage documented in #978 and #1345.

Related to the broader WHOOP 5/MG deep-data roadmap in #761.

Addresses #423.

[ryanbr]

@bhelm — I pushed two commits to this branch (maintainer edits were on) and dispatched the app-build gate against it. All 15 normal checks are green; the app-build macOS leg is not, and the one failure is yours to settle rather than mine.

What I pushed

Merged current main in. Only conflict was Strand/Resources/Localizable.xcstrings: main had added two strings for the strap readback check while this branch added sixty-six, disjoint keys, so both sets are kept and the entry that had been last gained its trailing comma. Resolved as text — parsing and re-emitting that catalogue expands every hand-compacted entry and turns it into a several-thousand-line diff.

I did not rebase. The branch is 44 commits and the first one already conflicted against current main, so a rebase means resolving 43 intermediate states that the squash-merge discards anyway. Merging main in resolves once and leaves your commits intact.

Localized the collector into the five ungated iOS locales. 52 of the new iOS strings carried only de/es/fr/pt-PT, so Italian, Polish, Russian and both Chinese locales would have rendered the English source. Your Android side was already complete — all seven locale files gain the same 67 keys.

Worth knowing why this passed review: Tools/i18n_audit.py --ci hard-gates only the focus set (de/es/fr/pt-PT), and the remaining five sit behind a ratchet that forbids only NEW gaps against a tracked baseline. A block this size lands green. The audit's OK does not mean the catalogue is complete.

Terminology follows what the catalogue already uses rather than anything I coined — strap is fascia/pasek/браслет/手环/手環, Session is Sessione/Sesja/Сессия/会话/工作階段, Test Centre matches the existing entries. The bare "0" entry I left exactly as you wrote it: no localizations, shouldTranslate:false, and five identical numerals would be noise.

The app-build failure — a question for you

app-build is disabled by default here, so nothing in normal CI ever compiles the app targets or runs StrandTests. I enabled it, dispatched against this PR, and re-disabled it. iOS built clean. macOS failed on one assertion in your own parity test:

RawDataCollectorParityTests.testAppleSurfaceStillImplementsEveryDeclaredCapability
Apple collector lost raw_data_removed_with_session marker: capturedAt >= ? AND capturedAt <= ?

Not caused by the merge — I checked that marker across every scanned Swift file at your original head da9e2b218 and at the merged head: zero occurrences at either. The test had simply never run.

Exactly one marker is missing across all capabilities, and the two readings point at different fixes:

marker	present
swift deleteFiles(sessionId	yes
swift capturedAt >= ? AND capturedAt <= ?	no
kotlin deleteFiles(sessionId)	yes
kotlin noop-5mg-raw-$sessionId.zip	yes

Neither platform deletes rows on session delete; both delete files. So either the marker is stale, left from the SQLite-backed design before session storage moved to files, and the fix is to drop it from both oracle copies — or it is aspirational, raw-outbox rows whose capturedAt falls in the session window genuinely should be purged, neither platform does it, and raw_data_removed_with_session is currently a promise the code does not keep.

I did not want to delete a requirement you may have meant, so I left it. If it is the second reading, note the Kotlin markers do not require that deletion either, so the oracle is holding the two platforms to different standards.

One other thing

The migration comment says deployed installations are "known not to contain usable rows", and on main WhoopBleClient still writes rawImuSample whenever raw capture is on — it is the reader that has zero callers, not the writer. The drop still looks right to me: the table is capped at RAW_IMU_RETENTION_ROWS, nothing can read it, and file-backed sessions supersede it. But DROP TABLE is irreversible and the v28 comment it replaces was explicit ("do NOT 'clean up' either side as dead code — the retained rows are the deliverable. Audit: #978"), so the reasoning is worth stating accurately: capped, write-only and superseded, rather than empty.

Separately, your note that iOS is unvalidated against real hardware still stands — the app-build gate is a compile-and-unit-test check, not a device test.

[bhelm]

Thanks for catching these — both points are fixed in 27e82ed73.

• The parity oracle no longer expects the stale SQLite time-range deletion. That marker belonged to the abandoned raw-outbox design. Both platforms now declare the actual file-backed lifecycle symmetrically: delete the session IMU files, then remove the durable session metadata.
• The migration comments no longer claim that rawImuSample is known-empty. They now accurately describe it as a bounded, opt-in, write-only legacy cache with no analytics/UI/export consumer, intentionally retired after session-owned IMU moved to the file-backed store. The corresponding Android test was renamed as well.

Verification after the fix:

• targeted Android collector/parity/migration tests: pass
• all 15 regular PR checks: pass
• macOS app build: pass
• full StrandTests, including RawDataCollectorParityTests: pass
• iOS simulator build: pass

The upstream app-build workflow is currently disabled_manually, so I could not dispatch it there without admin rights. I ran the identical workflow against the exact PR head on the fork instead: https://github.com/bhelm/noop/actions/runs/33243286625

One process note: keeping this kind of cross-platform contract current would be a little easier once the parity guard itself lives on main. At the moment the raw-data collector and its guard are introduced together in this feature PR, so there is no independent mainline guard against which the branch can continuously converge. This is not a blocker for this PR—the corrected guard has now run green—but having the guard infrastructure on main makes future parity work easier to maintain and harder to accidentally skip.