Repository navigation
[Onyx] Investigate DataError: Failed to write blobs (InvalidBlob) storage error #87871
Description
Coming from #87844.
Summary
Let's investigate the following Onyx storage error: DataError: Failed to write blobs (InvalidBlob), which accounts for 63,281 (1.9%) of the 3,364,645 analysed errors during a time span of 1 month. Full details about the investigation and methodology are in #87844.
| Platforms | Top Methods |
|---|---|
| Desktop Chrome (61K), Desktop Edge (2K) | multiSetWithRetry (48K), setWithRetry (15K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 9,537 | 0 | 0 | 0 | — | — | 0 |
- Chromium-exclusive — blob data becoming invalid between when it's created and when it's written to IDB
- Notable that
multiSetWithRetrydominates (76%) — batch writes are more susceptible - Not a capacity error, so no eviction — retries are unlikely to help since the blob data is fundamentally invalid
Action Items
- Investigate with Sentry, VictoriaLogs and Fullstory to understand user scenarios — what data is being written when blobs become invalid? Are there specific pages or actions that produce blob-containing data?
- Investigate why
multiSetWithRetryis disproportionately affected (76%) — blob references may become invalid during the time it takes to batch multiple writes - Consider validating blob data before writing, or catching this specific error and retrying with fresh blob references
Activity
melvin-bot commentedon Jun 29, 2026
@fabioh8010, this Monthly task hasn't been acted upon in 6 weeks; closing.
If you disagree, feel encouraged to reopen it -- but pick your least important issue to close instead.
fabioh8010 commentedon Jun 30, 2026
@mountiny Could you reopen? Thanks!
fabioh8010 commentedon Jul 8, 2026
@Julesssss
@elirangoshen will work on this one, could you assign him instead? Thanks!
elirangoshen commentedon Jul 20, 2026
Hi, there is potential lead, and Im working with @fabioh8010 to validate the idea before opening the pr. (for now the pr is drafted)
elirangoshen commentedon Jul 21, 2026
I still validating the fix by trying to reproduce it locally
elirangoshen commentedon Jul 23, 2026
Update: I was able to reproduce it locally and now investigating solutions
fabioh8010 commentedon Aug 17, 2026
Updates:
Waiting for Bump PR of Expensify/react-native-onyx#814 to land before @elirangoshen can check logs and confirm issue is mitigated.
elirangoshen commentedon Aug 31, 2026
Confirmed mitigated — safe to close
Following up on the blocker in this comment: the wait was on the bump PR for react-native-onyx#814 to land. That bump PR, #96957, is still open and titled
[TESTING ONLY]— it was a pin for verification and was never intended to merge. The change reached production anyway through the normal version bumps, so the blocker is resolved.Both fixes are in production
Files (the root fix)Failed to write blobsasINVALID_DATA, never retryce8b8ed5a19on Aug 20; main is on 3.0.101 todayI unpacked the published tarballs to confirm the onyx side: the
failed to write blobsbranch inclassifyError.tsis present in 3.0.95 and every version after it. App carried^3.0.95from Jul 31 and an exact pin from Aug 20, so #814 has been live for weeks without #96957.Log verification
VictoriaLogs, query
"Failed to write blobs", Aug 24-30:The 1,784-event spike on Aug 25 is almost entirely a single client, and the
sourcefield explains it.sourceis<platform-token><app-version>concatenated, soecash…is NewDot web,iphone…is iOS,android…is Android:ecash9.4.40-0ecash9.4.62-4ecash9.4.58-4ecash9.4.59-5ecash9.4.55-4ecash9.4.63-1ecash9.4.57-3ecash9.4.60-2ecash9.4.61-4Every source is
ecash, i.e. web only — expected, since this is an IndexedDB write path.That 1,750-event burst came from
ecash9.4.40-0— a NewDot web client on App 9.4.40-0, which was tagged Jul 20. The first release carrying #96940 was 9.4.46-10, tagged Jul 30, so that client has neither #96940 nor #814.On web the reported version comes from the JS bundle loaded at page load, so a tab opened on Jul 20 and never reloaded keeps reporting 9.4.40-0 indefinitely. That population is real but small —
source:ecash9.4.40-0logged 3,360 lines from 21 distinct users on Aug 29 — and all 1,750InvalidBlobevents came from one of those users. It is not a live regression on current code, and it clears itself once those tabs reload.Across all current releases combined the residual is ~226 hits/week over ~9 users, single-to-double digits per release. Those remaining events are also the macOS
IOErrordialect (Failed to write blobs (IOError)), while theInvalidBlobvariant named in this issue's title is zero on every current release — all 1,750InvalidBlobevents trace to the one stale client.Conclusion
~99.99% reduction, and what remains is a stale-bundle tail rather than an active failure mode. Closing this as completed.
Also closing #96957, which is superseded — the change it was pinning shipped through the regular onyx bump.
What do you think @Julesssss @fabioh8010 ?
fabioh8010 commentedon Aug 31, 2026
I agree we can wrap up this one
Julesssss commentedon Aug 31, 2026
Agree