Repository navigation
Fix Data Loss and Stuck Collection Found in the FHIR Stack Review - #117
Open
lukaskollmer wants to merge 10 commits into
Open
lukaskollmer wants to merge 10 commits into
lukaskollmer wants to merge 10 commits into
Conversation
The anchored fetchers persist a pending batch before yielding it and refuse to continue when a retry does not reproduce it exactly (pendingBatchMismatch). SensorKit does not promise exact replay, and reset() refused to run while a batch was pending, so one unacknowledged batch plus any content drift wedged the sensor cursor permanently, with no API to recover. - reset() now abandons a pending batch together with the cursor; the new reset generation guarantees its acquisition coordinates are never reused. - New SensorKit.discardPendingBatches(for:) abandons only the pending batches and keeps the committed cursors, so the range is fetched again under fresh coordinates. Consumers call it after pendingBatchMismatch. - resetQueryAnchors(for:) no longer aborts at the first pending partition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FHIRModels' DateTime re-derives a named zone's offset from its wall-clock components when serialized, so an instant in the second occurrence of a DST fall-back hour is written one hour early. #104 fixed this only in Observation.setEffective. Move its fixed-offset zone into a shared TimeZone.fixedOffset(at:) helper in FHIRModelsExtensions and use it for SensorKitConverter.exactDateTime/exactInstant/period, SensorConverter.period and QuestionnaireResponse.authored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6b30921 lowered the compatibility manifest to watchOS 8, but apple15-release.py validate-defaults still required watchOS 9.0, so the first -apple15 release would fail at "Check the release defaults". Expect 8.0 in the script and its test fixture, and say watchOS 8 in APPLE15_RELEASES.md, the README note and the Package.swift comment. Also drop the stale "temporary lowered default" wording now that the standard targets are the default again (lowered is opt-in via GROVE_LOWERED_DEPLOYMENT_TARGETS=1), and point the Package.swift comments at Scripts/build-floor.py instead of the removed build-floor.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9c5eff45 is reachable only from grove-fhir's refs/pull/42/head; #42 was squash-merged as eff5af8, which is on main. Re-pin the workflow defaults and validate-fhir-conformance.sh to eff5af8. The two commits differ only in grove-fhir's guide QA script, the Health Connect producer validator, its tests and one questionnaire example; Grove's generators read only catalog/ and Conformance/corpora, so the generated sources are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every Period measurement refused a sample whose start equals its end, so each HKStateOfMind (HealthKit gives it a single date) and every point-in-time manual entry (dietary, insulin, falls, ...) was a permanent conversion refusal. FHIR per-1 admits start == end, so such samples now emit an equal-endpoint effectivePeriod. Reversed intervals stay invalid, and zero-width intervals stay invalid for step count and wheelchair push count (grove-step-count-period-1) and for basal energy and mindfulness sessions (the mobile-semantics corpus's non-zero Period edge rule). Session rates now refuse a zero-width interval explicitly rather than dividing by zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HealthKit already stores HKQuantityTypeIdentifierAppleSleepingBreathingDisturbances as disturbance events per hour (a Discrete (Arithmetic) type that HKAppleSleepingBreathingDisturbancesClassification classifies directly), so dividing it by the sample duration in hours shrank the wire value by the length of the night (~7x). Emit the platform rate unchanged under the contract's /h unit and session-rate method, and make the test assert passthrough instead of the division. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
validateECG required every batch offset to equal sampleCount * period exactly, measured from SensorKitECGSession.startDate, which is the .begin marker's date rather than the first data chunk's. A begin marker that precedes the first chunk, or the ~0.1 µs Double rounding of 8e8-second Dates, failed the whole session with nonUniformTiming, and MHC then never acknowledged the batch. Offsets and the duration are now measured relative to the first batch and compared with a 1 µs tolerance that only absorbs floating-point representation noise (a few Date ulps, three orders of magnitude below a sample period), so gaps and shifted samples still fail closed. The effective Period now starts at the first chunk's own Date, keeping effectivePeriod.end = start + (frames - 1) x period exact. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One unanswered observationExtract-marked item threw answerMissing and the whole extraction produced nothing, so a HeartRisk response answering the fasting glucose but skipping the optional blood-pressure panel (or vice versa) lost every extracted reading. - An unanswered marked item that is not `required` now extracts nothing instead of refusing; a panel with no component answered likewise. - A `required` marked item or panel left unanswered still refuses with answerMissing. - A partially answered panel refuses with componentIncomplete (previously answerMissing on the child); an instrument that marks no item for a declared component still refuses regardless of answers. - Component answers go through the same numeric path as standalone ones, so integer/decimal answers with a fixed questionnaire-unit extract and other answer families refuse with unsupportedAnswer instead of a misleading answerMissing. Grounding in grove-fhir 9c5eff45: the questionnaire IG defers to standard SDC Observation-based extraction and states no rule for unanswered items (questionnaire/input/pagecontent/measurements.md); responses.md requires a completed response to answer every enabled required item, so a missing required answer is a defect; the mobile blood-pressure profile requires `component contains systolic 1..1 MS and diastolic 1..1 MS` (mobile/input/fsh/generated-measurement-profiles.fsh), and conformance.md forbids silently discarding an answer, so a half-filled panel refuses rather than emitting or dropping a partial reading; measurements.md "Unit declarations" applies the fixed questionnaire-unit to integer and decimal items without restricting it to standalone items. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Extraction refused a required marked item left unanswered, but `required` binds only while the item is enabled: a required item disabled by enableWhen, or nested under a group the participant left out, is legitimately absent. The extractor cannot evaluate enablement, so that refusal dropped every other reading of a conforming response, the failure the previous commit set out to remove. An unanswered item now always extracts nothing; the pair validator enforces required answers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous DST commit left two places that still hand FHIRModels a named zone: the HealthKit ECG date-times (effective period and ECG extension periods) and Observation.setIssued(on:), which MHC calls for every self-modelled sample. In the repeated fall-back hour both serialized the instant one hour early. Both now use TimeZone.fixedOffset(at:). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
PaulGoldschmidt
approved these changes
Sep 28, 2026
1 task done
lukaskollmer
added a commit
to SchmiedmayerLab/MyHeartCounts-iOS
that referenced
this pull request
Sep 28, 2026
Grove moves to 611d0cd8 (SchmiedmayerLab/Grove#117), which adds SensorKit.discardPendingBatches(for:) used by the SensorKit recovery above and fixes the State of Mind, partial-extraction, breathing-disturbance, ECG timing and DST issues. StudyDefinitions moves to d1cbe285 (SchmiedmayerLab/MyHeartCounts-StudyDefinitions#52), which only re-pins Grove, because SwiftPM rejects two different revision requirements for Grove. Pinned in the project, MyHeartCountsShared, Package.resolved and the submodule. A half-answered blood-pressure panel still refuses, but Grove now reports it as componentIncomplete instead of answerMissing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 task done
lukaskollmer
added this pull request to stack #112
September 28, 2026 15:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
♻️ Current situation & Problem
Stack: 9/9, following #116. Consumed by SchmiedmayerLab/MyHeartCounts-iOS#220 through SchmiedmayerLab/MyHeartCounts-StudyDefinitions#52. A review of the FHIR stack (#67, #94, #104, #116) together with its consumer, SchmiedmayerLab/MyHeartCounts-iOS#204, found problems that lose or distort participant data or leave a collection permanently stuck. This PR fixes the ones that belong in Grove:
applyEffectiverejected any.periodcontract whose sample hasendDate == startDate. State of Mind samples carry a single date, and so do manual point-in-time entries (dietary values, insulin, water, falls, alcoholic beverages). The IG only requires a strictly positive interval for step count and wheelchair push count (grove-step-count-period-1) and, through corpus edge rules, for basal energy and mindful sessions.pendingBatchMismatchon every fetch, andreset()refused to run while a batch was pending. There was no API to recover, so the sensor stopped collecting until reinstall..beginmarker instead of the first voltage chunk and compared Doubles exactly, which can reject real sessions.authored, HealthKit ECG andissuedpaths. Fix HealthKit Export Completion and Retries #104 had fixed this only for HealthKiteffective.-apple15release would fail.apple15-release.pyexpects a watchOS 9 floor, but the lowered platforms have been watchOS 8 since Generate the Swift FHIR Contract from the Implementation Guides #67.9c5eff45, a commit that exists only as the head of the squash-merged Align the Producer Contract Across Swift, Kotlin and TypeScript grove-fhir#42.⚙️ Release Notes
effectivePeriodwith equal start and end, unless its measurement requires a non-zero period (step count, wheelchair push count, basal energy, mindful session). Reversed intervals are still refused.SensorKit.discardPendingBatches(for:)abandons delivered-but-unacknowledged batches and keeps the committed cursor. Records fetched afterwards get fresh acquisition coordinates. Consumers call it afterpendingBatchMismatch.resetQueryAnchors(for:)now succeeds when a batch is pending, abandons it, and no longer aborts at the first pending device partition.requireditems, because enablement is the pair validator's job. A half-answered panel still refuses, now withcomponentIncomplete. Integer and decimal panel components with a fixedquestionnaire-unitnow extract.appleSleepingBreathingDisturbancesis emitted unchanged as events per hour.Daterepresentation noise. Real gaps and shifts are still rejected.TimeZone.fixedOffset(at:)in FHIRModelsExtensions, now used by every FHIR date-time Grove builds from a named zone.eff5af8, the squash commit of grove-fhir#42 onmain.Behavior changes consumers may notice:
ObservationExtractionError.answerMissingis no longer thrown for unanswered items.effectivePeriod.startmoves to its first voltage chunk.📚 Documentation
The SensorKit docs list
discardPendingBatches(for:)and link it frompendingBatchMismatch. TheExtractingObservationsarticle and the extractor's doc comments describe the unanswered-item rules.APPLE15_RELEASES.mdand the README describe standard targets as the default and lowered targets as opt-in. Inline comments cite the IG invariant and corpus rules behind the non-zero-period set and justify the ECG tolerance.Follow-up for grove-fhir, not changed here:
mobile/input/fsh/terminology.fshdefines#session-rateas "a count of qualifying events divided by the session duration". That reads as if the producer must divide, which is how this bug came about. The definition should allow a platform-computed rate.✅ Testing
Run locally on macOS (Xcode 27, standard targets, all traits), all passing:
FHIRModelsExtensionstest planGroveHealthKitFHIRtest plan (183 tests)GroveSensorKitFHIRtest plan (83 tests)GroveQuestionnairetest plan (218 + 28 + 14 tests)Scripts/Tests/test_apple15_release.pyThe
GroveSensorKittests only run on iOS. They compile (build-for-testing, generic iOS Simulator), but they were not run locally, so CI runs them. That covers the newreset/discardPendingBatchtests.The ECG fix has no fixture from a real device yet. If real voltage chunks jitter by more than 1 µs, those sessions will still be rejected.
Code of Conduct & Contributing Guidelines
By creating and submitting this pull request, you agree to follow our Code of Conduct and Contributing Guidelines:
🤖 Generated with Claude Code