Skip to content

Fix Data Loss and Stuck Collection Found in the FHIR Stack Review - #117

Open
lukaskollmer wants to merge 10 commits into
lukas/disable-lowered-targets-by-defaultfrom
lukas/fhir-review-fixes
Open

lukaskollmer wants to merge 10 commits into
lukas/disable-lowered-targets-by-defaultfrom
lukas/fhir-review-fixes

Conversation

@lukaskollmer

@lukaskollmer lukaskollmer commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

♻️ Current situation & Problem

Stack: 9/9, following #116. Consumed by SchmiedmayerLab/MyHeartCounts-iOS#220 through SchmiedmayerLab/MyHeartCounts-StudyDefinitions#52. A review of the FHIR stack (#67, #94, #104, #116) together with its consumer, SchmiedmayerLab/MyHeartCounts-iOS#204, found problems that lose or distort participant data or leave a collection permanently stuck. This PR fixes the ones that belong in Grove:

  • Every HealthKit State of Mind sample was refused. applyEffective rejected any .period contract whose sample has endDate == startDate. State of Mind samples carry a single date, and so do manual point-in-time entries (dietary values, insulin, water, falls, alcoholic beverages). The IG only requires a strictly positive interval for step count and wheelchair push count (grove-step-count-period-1) and, through corpus edge rules, for basal energy and mindful sessions.
  • A SensorKit cursor could wedge forever. After an unacknowledged batch, any SensorKit content drift makes the retry throw pendingBatchMismatch on every fetch, and reset() refused to run while a batch was pending. There was no API to recover, so the sensor stopped collecting until reinstall.
  • Skipping one optional marked question dropped every extracted reading of the response. For example, a participant who enters their glucose but skips the blood-pressure panel got no Observation at all.
  • Sleeping breathing disturbances were divided by the night's length. HealthKit already stores this value per hour, so the wire value was about 7x too small.
  • The ECG uniform-timing check measured offsets from the .begin marker instead of the first voltage chunk and compared Doubles exactly, which can reject real sessions.
  • Instants in the repeated DST hour were still serialized one hour early on the SensorKit, questionnaire authored, HealthKit ECG and issued paths. Fix HealthKit Export Completion and Retries #104 had fixed this only for HealthKit effective.
  • The first -apple15 release would fail. apple15-release.py expects a watchOS 9 floor, but the lowered platforms have been watchOS 8 since Generate the Swift FHIR Contract from the Implementation Guides #67.
  • CI pinned grove-fhir to 9c5eff45, a commit that exists only as the head of the squash-merged Align the Producer Contract Across Swift, Kotlin and TypeScript grove-fhir#42.

⚙️ Release Notes

  • A zero-length HealthKit sample now converts to an effectivePeriod with equal start and end, unless its measurement requires a non-zero period (step count, wheelchair push count, basal energy, mindful session). Reversed intervals are still refused.
  • New SensorKit.discardPendingBatches(for:) abandons delivered-but-unacknowledged batches and keeps the committed cursor. Records fetched afterwards get fresh acquisition coordinates. Consumers call it after pendingBatchMismatch.
  • resetQueryAnchors(for:) now succeeds when a batch is pending, abandons it, and no longer aborts at the first pending device partition.
  • Questionnaire extraction: an unanswered marked item, or a panel with no answered component, extracts nothing instead of refusing the whole response. This applies even to required items, because enablement is the pair validator's job. A half-answered panel still refuses, now with componentIncomplete. Integer and decimal panel components with a fixed questionnaire-unit now extract.
  • appleSleepingBreathingDisturbances is emitted unchanged as events per hour.
  • ECG timing is measured from the first voltage chunk, with a 1 µs tolerance for Date representation noise. Real gaps and shifts are still rejected.
  • New TimeZone.fixedOffset(at:) in FHIRModelsExtensions, now used by every FHIR date-time Grove builds from a named zone.
  • Apple 15 release checks, docs and comments now agree on the watchOS 8 floor. CI validates against grove-fhir eff5af8, the squash commit of grove-fhir#42 on main.

Behavior changes consumers may notice:

  • ObservationExtractionError.answerMissing is no longer thrown for unanswered items.
  • Breathing-disturbance values are about 7x larger than before.
  • An ECG's effectivePeriod.start moves to its first voltage chunk.
  • FHIR date-times built by Grove now carry GMT-offset zones instead of named zones. Serialized output only changes inside the repeated DST hour.

📚 Documentation

The SensorKit docs list discardPendingBatches(for:) and link it from pendingBatchMismatch. The ExtractingObservations article and the extractor's doc comments describe the unanswered-item rules. APPLE15_RELEASES.md and the README describe standard targets as the default and lowered targets as opt-in. Inline comments cite the IG invariant and corpus rules behind the non-zero-period set and justify the ECG tolerance.

Follow-up for grove-fhir, not changed here: mobile/input/fsh/terminology.fsh defines #session-rate as "a count of qualifying events divided by the session duration". That reads as if the producer must divide, which is how this bug came about. The definition should allow a platform-computed rate.

✅ Testing

Run locally on macOS (Xcode 27, standard targets, all traits), all passing:

  • FHIRModelsExtensions test plan
  • GroveHealthKitFHIR test plan (183 tests)
  • GroveSensorKitFHIR test plan (83 tests)
  • GroveQuestionnaire test plan (218 + 28 + 14 tests)
  • Scripts/Tests/test_apple15_release.py

The GroveSensorKit tests only run on iOS. They compile (build-for-testing, generic iOS Simulator), but they were not run locally, so CI runs them. That covers the new reset/discardPendingBatch tests.

The ECG fix has no fixture from a real device yet. If real voltage chunks jitter by more than 1 µs, those sessions will still be rejected.

Code of Conduct & Contributing Guidelines

By creating and submitting this pull request, you agree to follow our Code of Conduct and Contributing Guidelines:

🤖 Generated with Claude Code

lukaskollmer and others added 10 commits September 28, 2026 17:06
The anchored fetchers persist a pending batch before yielding it and refuse to
continue when a retry does not reproduce it exactly (pendingBatchMismatch).
SensorKit does not promise exact replay, and reset() refused to run while a
batch was pending, so one unacknowledged batch plus any content drift wedged
the sensor cursor permanently, with no API to recover.

- reset() now abandons a pending batch together with the cursor; the new reset
  generation guarantees its acquisition coordinates are never reused.
- New SensorKit.discardPendingBatches(for:) abandons only the pending batches
  and keeps the committed cursors, so the range is fetched again under fresh
  coordinates. Consumers call it after pendingBatchMismatch.
- resetQueryAnchors(for:) no longer aborts at the first pending partition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FHIRModels' DateTime re-derives a named zone's offset from its wall-clock
components when serialized, so an instant in the second occurrence of a
DST fall-back hour is written one hour early. #104 fixed this only in
Observation.setEffective. Move its fixed-offset zone into a shared
TimeZone.fixedOffset(at:) helper in FHIRModelsExtensions and use it for
SensorKitConverter.exactDateTime/exactInstant/period,
SensorConverter.period and QuestionnaireResponse.authored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6b30921 lowered the compatibility manifest to watchOS 8, but
apple15-release.py validate-defaults still required watchOS 9.0, so the
first -apple15 release would fail at "Check the release defaults". Expect
8.0 in the script and its test fixture, and say watchOS 8 in
APPLE15_RELEASES.md, the README note and the Package.swift comment.

Also drop the stale "temporary lowered default" wording now that the
standard targets are the default again (lowered is opt-in via
GROVE_LOWERED_DEPLOYMENT_TARGETS=1), and point the Package.swift comments
at Scripts/build-floor.py instead of the removed build-floor.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9c5eff45 is reachable only from grove-fhir's refs/pull/42/head; #42 was
squash-merged as eff5af8, which is on main. Re-pin the workflow defaults
and validate-fhir-conformance.sh to eff5af8. The two commits differ only
in grove-fhir's guide QA script, the Health Connect producer validator,
its tests and one questionnaire example; Grove's generators read only
catalog/ and Conformance/corpora, so the generated sources are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every Period measurement refused a sample whose start equals its end, so
each HKStateOfMind (HealthKit gives it a single date) and every
point-in-time manual entry (dietary, insulin, falls, ...) was a permanent
conversion refusal. FHIR per-1 admits start == end, so such samples now
emit an equal-endpoint effectivePeriod. Reversed intervals stay invalid,
and zero-width intervals stay invalid for step count and wheelchair push
count (grove-step-count-period-1) and for basal energy and mindfulness
sessions (the mobile-semantics corpus's non-zero Period edge rule).
Session rates now refuse a zero-width interval explicitly rather than
dividing by zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HealthKit already stores HKQuantityTypeIdentifierAppleSleepingBreathingDisturbances
as disturbance events per hour (a Discrete (Arithmetic) type that
HKAppleSleepingBreathingDisturbancesClassification classifies directly), so
dividing it by the sample duration in hours shrank the wire value by the
length of the night (~7x). Emit the platform rate unchanged under the
contract's /h unit and session-rate method, and make the test assert
passthrough instead of the division.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
validateECG required every batch offset to equal sampleCount * period
exactly, measured from SensorKitECGSession.startDate, which is the
.begin marker's date rather than the first data chunk's. A begin marker
that precedes the first chunk, or the ~0.1 µs Double rounding of
8e8-second Dates, failed the whole session with nonUniformTiming, and
MHC then never acknowledged the batch.

Offsets and the duration are now measured relative to the first batch
and compared with a 1 µs tolerance that only absorbs floating-point
representation noise (a few Date ulps, three orders of magnitude below
a sample period), so gaps and shifted samples still fail closed. The
effective Period now starts at the first chunk's own Date, keeping
effectivePeriod.end = start + (frames - 1) x period exact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One unanswered observationExtract-marked item threw answerMissing and the
whole extraction produced nothing, so a HeartRisk response answering the
fasting glucose but skipping the optional blood-pressure panel (or vice
versa) lost every extracted reading.

- An unanswered marked item that is not `required` now extracts nothing
  instead of refusing; a panel with no component answered likewise.
- A `required` marked item or panel left unanswered still refuses with
  answerMissing.
- A partially answered panel refuses with componentIncomplete (previously
  answerMissing on the child); an instrument that marks no item for a
  declared component still refuses regardless of answers.
- Component answers go through the same numeric path as standalone ones,
  so integer/decimal answers with a fixed questionnaire-unit extract and
  other answer families refuse with unsupportedAnswer instead of a
  misleading answerMissing.

Grounding in grove-fhir 9c5eff45: the questionnaire IG defers to standard
SDC Observation-based extraction and states no rule for unanswered items
(questionnaire/input/pagecontent/measurements.md); responses.md requires a
completed response to answer every enabled required item, so a missing
required answer is a defect; the mobile blood-pressure profile requires
`component contains systolic 1..1 MS and diastolic 1..1 MS`
(mobile/input/fsh/generated-measurement-profiles.fsh), and conformance.md
forbids silently discarding an answer, so a half-filled panel refuses
rather than emitting or dropping a partial reading; measurements.md "Unit
declarations" applies the fixed questionnaire-unit to integer and decimal
items without restricting it to standalone items.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Extraction refused a required marked item left unanswered, but `required`
binds only while the item is enabled: a required item disabled by enableWhen,
or nested under a group the participant left out, is legitimately absent. The
extractor cannot evaluate enablement, so that refusal dropped every other
reading of a conforming response, the failure the previous commit set out to
remove. An unanswered item now always extracts nothing; the pair validator
enforces required answers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous DST commit left two places that still hand FHIRModels a named
zone: the HealthKit ECG date-times (effective period and ECG extension
periods) and Observation.setIssued(on:), which MHC calls for every
self-modelled sample. In the repeated fall-back hour both serialized the
instant one hour early. Both now use TimeZone.fixedOffset(at:).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lukaskollmer lukaskollmer added this to the 0.3.0 milestone Sep 28, 2026
@lukaskollmer lukaskollmer added the bug Something isn't working label Sep 28, 2026
@lukaskollmer lukaskollmer self-assigned this Sep 28, 2026
@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 2dfc6fd9-5351-4a6e-83c2-2db1c88e1c32


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

lukaskollmer added a commit to SchmiedmayerLab/MyHeartCounts-iOS that referenced this pull request Sep 28, 2026
Grove moves to 611d0cd8 (SchmiedmayerLab/Grove#117), which adds
SensorKit.discardPendingBatches(for:) used by the SensorKit recovery above and
fixes the State of Mind, partial-extraction, breathing-disturbance, ECG timing
and DST issues. StudyDefinitions moves to d1cbe285
(SchmiedmayerLab/MyHeartCounts-StudyDefinitions#52), which only re-pins Grove,
because SwiftPM rejects two different revision requirements for Grove. Pinned
in the project, MyHeartCountsShared, Package.resolved and the submodule.

A half-answered blood-pressure panel still refuses, but Grove now reports it
as componentIncomplete instead of answerMissing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lukaskollmer
lukaskollmer added this pull request to stack #112 September 28, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants