Repository navigation
[Onyx] Investigate the most common storage errors #87844
Description
Coming from #85252 (comment).
Onyx Storage Errors — Summary Report
Time span: 2026-03-02 to 2026-03-30 (~4 weeks)
Total
Failed to save to storageentries: 3,364,645Distinct error types: 52
Full list of all 52 error types
| # | Error Type | Count | % |
|---|---|---|---|
| 1 | UnknownError: Internal error opening backing store for indexedDB.open. |
884,955 | 26.3% |
| 2 | QuotaExceededError: The current transaction exceeded its quota limitations. |
865,703 | 25.7% |
| 3 | UnknownError: Connection to Indexed Database server lost. Refresh the page to try again |
637,821 | 19.0% |
| 4 | QuotaExceededError: Encountered full disk while opening backing store for indexedDB.open. |
234,697 | 7.0% |
| 5 | InvalidStateError: Failed to execute 'transaction' on 'IDBDatabase': The database connection is closing. |
216,981 | 6.4% |
| 6 | null |
90,661 | 2.7% |
| 7 | NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] database or disk is full |
89,174 | 2.7% |
| 8 | NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] disk I/O error |
89,137 | 2.6% |
| 9 | DataError: Failed to write blobs (InvalidBlob) |
63,281 | 1.9% |
| 10 | QuotaExceededError |
51,900 | 1.5% |
| 11 | UnknownError: Internal error. |
51,743 | 1.5% |
| 12 | UnknownError: Connection is closing because of: IO error: <indexedDB file path> |
19,053 | 0.6% |
| 13 | InvalidStateError: IDBDatabase.transaction: Can't start a transaction on a closed database |
16,428 | 0.5% |
| 14 | NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] cannot rollback - no transaction is active |
11,925 | 0.4% |
| 15 | AbortError: The connection was closed. |
5,898 | 0.2% |
| 16 | UnknownError: Database deleted by request of the user |
5,243 | 0.2% |
| 17 | NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] database is locked |
4,097 | 0.1% |
| 18 | AbortError: The operation was aborted. |
3,894 | 0.1% |
| 19 | InvalidStateError: A mutation operation was attempted on a database that did not allow mutations. |
3,385 | 0.1% |
| 20 | UnknownError: Connection is closing because of: Failed to remove blob file. |
2,991 | 0.1% |
| 21 | UnknownError: Unable to open database file on disk |
2,820 | 0.1% |
| 22 | UnknownError: An internal error was encountered in the Indexed Database server |
2,531 | 0.1% |
| 23 | AbortError: The transaction was aborted, so the request cannot be fulfilled. |
2,399 | 0.1% |
| 24 | QuotaExceededError: The quota has been exceeded. |
1,534 | 0.0% |
| 25 | ReferenceError: Can't find variable: indexedDB |
1,356 | 0.0% |
| 26 | DataCloneError: Failed to execute 'put' on 'IDBObjectStore': function arrayBuffer() { [native code] } could not be cloned. |
956 | 0.0% |
| 27 | DataCloneError: Failed to store record in an IDBObjectStore: BlobURLs are not yet supported. |
726 | 0.0% |
| 28 | TimeoutError: Transaction timed out due to inactivity. |
721 | 0.0% |
| 29 | NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] not an error |
480 | 0.0% |
| 30 | UnknownError: Connection is closing because of: Force close delete origin |
467 | 0.0% |
| 31 | UnknownError: The operation failed for reasons unrelated to the database itself and not covered by any other error code. |
384 | 0.0% |
| 32 | DataError: Failed to write blobs (IOError) |
379 | 0.0% |
| 33 | AbortError: A request was aborted, for example through a call to IDBTransaction.abort. |
252 | 0.0% |
| 34 | UnknownError: Internal error opening database with no version specified. |
132 | 0.0% |
| 35 | NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] another row available |
89 | 0.0% |
| 36 | DataCloneError: The object can not be cloned. |
84 | 0.0% |
| 37 | DataCloneError: Function object could not be cloned. |
84 | 0.0% |
| 38 | UnknownError: Connection is closing. |
78 | 0.0% |
| 39 | UnknownError: Internal error committing transaction. |
73 | 0.0% |
| 40 | QuotaExceededError: Encountered disk full while committing transaction. |
48 | 0.0% |
| 41 | NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] no more rows available |
33 | 0.0% |
| 42 | UnknownError: Version change transaction on cached page is aborted to unblock other connections |
18 | 0.0% |
| 43 | UnknownError: Attempt to get a record from database without an in-progress transaction |
9 | 0.0% |
| 44 | UnknownError: Connection is closing because of: The database is deleted. |
9 | 0.0% |
| 45 | UnknownError: Error looking up record in object store by key range |
5 | 0.0% |
| 46 | Error: [SqlExecutionError] cannot rollback - no transaction is active |
4 | 0.0% |
| 47 | TransactionInactiveError: A request was placed against a transaction which is currently not active, or which is finished. |
2 | 0.0% |
| 48 | UnknownError: Connection is closing because of: Corruption: block checksum mismatch |
1 | 0.0% |
| 49 | DataCloneError: Failed to execute 'put' on 'IDBObjectStore': error => { ... } could not be cloned. |
1 | 0.0% |
| 50 | NitroSQLiteError: NitroSQLite.executeBatchAsync(...) threw an unknown St13runtime_error error. |
1 | 0.0% |
| 51 | DataCloneError: Failed to execute 'put' on 'IDBObjectStore': Data cannot be cloned, out of memory. |
1 | 0.0% |
| 52 | NotReadableError: Data lost due to missing file. Affected record should be considered irrecoverable |
1 | 0.0% |
Methodology
Seven log messages were analyzed from OnyxUtils.retryOperation and OnyxUtils.reportStorageQuota in react-native-onyx:
| Message Pattern | Entries |
|---|---|
Failed to save to storage. Error: ${error}. onyxMethod: ${method}. retryAttempt: N/5 |
3,364,645 |
Storage operation failed after 5 retries. Error: ${error}. onyxMethod: ${method}. |
354,111 |
Out of storage. Evicting least recently accessed key (${key}) and retrying. |
1,238,436 |
Storage Quota Check -- bytesUsed: ${used} bytesRemaining: ${remaining} |
1,065,476 |
Attempted to set invalid data set in Onyx. Please ensure all data is serializable. |
958 |
Out of storage. But found no acceptable keys to remove. |
895 |
Unable to get database size. Error: ${dbSizeError} |
2,083 |
Cross-referencing: auxiliary events were matched to the nearest Failed to save to storage entry sharing the same request_id within a 2-second window. This approach was validated by measuring time gaps between events: 99.5%+ of related events occur within 1 second of each other.
Platform inference: derived from the userAgent string in each log entry. Detection relies on unique tokens: Edge includes Edg/, Opera includes OPR/, Firefox includes Firefox/ (no Chrome/ token), and Chrome includes Chrome/ without Edge/Opera markers. These are checked in priority order to avoid misclassification (e.g. Edge's UA also contains Chrome/).
Storage Quota Check interpretation: on web, bytesUsed and bytesRemaining come from navigator.storage.estimate() which reports the browser's overall storage quota for the origin (not just the IndexedDB database). On native, bytesUsed is the SQLite database file size (via PRAGMA page_size * page_count) and bytesRemaining is the device's free disk space (via getFreeDiskStorage()).
Top 10 Errors
1. UnknownError: Internal error opening backing store for indexedDB.open. — 884,955 (26.3%)
| Platforms | Top Methods |
|---|---|
| Desktop Chrome (664K), Desktop Edge (220K) | setWithRetry (682K), multiSetWithRetry (181K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 147,944 | 0 | 0 | 0 | — | — | 0 |
- No eviction or quota check events — this is an IndexedDB internal failure unrelated to storage space
- This error is not classified as a storage capacity error in Onyx (
STORAGE_ERRORSonly includesQuotaExceededErrorfor IDB), so it never triggers the eviction flow — it just retries immediately and fails repeatedly - Likely caused by corrupted IndexedDB state, browser extensions interfering, or Chromium-level bugs
2. QuotaExceededError: The current transaction exceeded its quota limitations. — 865,703 (25.7%)
| Platforms | Top Methods |
|---|---|
| Desktop Firefox (865K — 100%) | setWithRetry (865K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 6,089 | 859,566 | 0 | 859,276 | 0 | 0.0% | 0 |
- 859,276 quota checks with mean 0.0% full — Firefox reports quota exceeded even when overall usage is negligible. The exact quota model (per-transaction, per-write, or other) is unclear from the data alone
- Firefox-exclusive error — different quota behavior from Chromium and Safari
- Eviction is triggered frequently, and relatively few requests end up exhausting retries (6,089), suggesting eviction generally helps resolve this error
3. UnknownError: Connection to Indexed Database server lost. — 637,821 (19.0%)
| Platforms | Top Methods |
|---|---|
| Desktop Safari (558K), iOS Safari (72K), iOS Chrome (8K) | setWithRetry (578K), multiSetWithRetry (34K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 106,394 | 0 | 0 | 0 | — | — | 0 |
- Safari/WebKit-exclusive — the IDB connection drops unexpectedly
- No eviction or quota involvement — not recognized as a capacity error, so Onyx retries immediately without eviction
- 106K exhausted all retries (16.7% of this error) — immediate retries don't help because the connection itself is gone
4. QuotaExceededError: Encountered full disk while opening backing store for indexedDB.open. — 234,697 (7.0%)
| Platforms | Top Methods |
|---|---|
| Desktop Chrome (205K), Desktop Edge (21K), Desktop Opera (9K) | setWithRetry (232K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 6 | 234,222 | 471 | 16,327 | 0 | 0.0% | 475 |
- 0 at >= 95% and mean 0.0% full despite "full disk" in the error — this reflects IndexedDB's per-origin quota being exhausted, not the actual disk being full. The browser allocates a fraction of total storage to each origin, so "full disk" is a misnomer from Chromium
- Very few requests exhausted retries (6), suggesting eviction generally resolves the issue
5. InvalidStateError: Failed to execute 'transaction' on 'IDBDatabase': The database connection is closing. — 216,981 (6.4%)
| Platforms | Top Methods |
|---|---|
| Desktop Safari (103K), Desktop Chrome (75K), iOS Safari (29K), iOS Chrome (7K) | setWithRetry (181K), multiSetWithRetry (32K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 37,303 | 0 | 0 | 0 | — | — | 0 |
- Cross-browser but Safari-dominant — the DB connection is being closed (page navigation, tab discard, or browser cleanup) while writes are in-flight
- Similar to
#3: not a capacity error, so no eviction is triggered. Immediate retries are unlikely to help if the DB connection is closing
Issue was handled here: #84192
6. null — 90,661 (2.7%)
| Platforms | Top Methods |
|---|---|
| Desktop Safari (40K), Desktop Chrome (26K), iOS Safari (15K), Desktop Edge (7K) | setWithRetry (80K), multiSetWithRetry (9K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 13,089 | 23 | 0 | 254 | 0 | 0.1% | 0 |
- The error object was null — no useful error message was provided by the storage backend
- Cross-browser, slightly Safari-leaning — may be a Safari-specific issue where the error object is not properly populated
7. NitroSQLiteError: database or disk is full — 89,174 (2.7%)
| Platforms | Top Methods |
|---|---|
| Other/Native (89K), Android Native (153) | setWithRetry (89K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 10 | 88,998 | 106 | 60,468 | 0 | 2.4% | 0 |
- 60,468 quota checks with mean 2.4% of device disk used — the SQLite database itself is relatively small compared to overall device storage, but the device's free space is exhausted by other apps/data
- Native-only (NitroSQLite) — very few requests exhausted retries (10), suggesting eviction generally resolves the issue
8. NitroSQLiteError: disk I/O error — 89,137 (2.6%)
| Platforms | Top Methods |
|---|---|
| Other/Native (89K) | setWithRetry (89K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 14,882 | 0 | 0 | 0 | — | — | 0 |
- No eviction or quota events despite
"disk I/O error"being explicitly listed inSQLITE_STORAGE_ERRORS. This is unexpected — the eviction path should be triggered but isn't. Needs investigation (see action items) disk I/O erroris arguably not a storage capacity error and should probably be removed fromSQLITE_STORAGE_ERRORS. It indicates filesystem corruption, SD card issues, or device under stress — evicting keys won't help with I/O failures- Native-only
9. DataError: Failed to write blobs (InvalidBlob) — 63,281 (1.9%)
| Platforms | Top Methods |
|---|---|
| Desktop Chrome (61K), Desktop Edge (2K) | multiSetWithRetry (48K), setWithRetry (15K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 9,537 | 0 | 0 | 0 | — | — | 0 |
- Chromium-exclusive — blob data becoming invalid between when it's created and when it's written to IDB
- Notable that
multiSetWithRetrydominates (76%) — batch writes are more susceptible - Not a capacity error, so no eviction — retries are unlikely to help since the blob data is fundamentally invalid
10. QuotaExceededError (bare, no detail) — 51,900 (1.5%)
| Platforms | Top Methods |
|---|---|
| Desktop Chrome (51K), Desktop Edge (915) | setWithRetry (51K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 259 | 51,322 | 307 | 48,022 | 1,990 | 4.2% | 1,196 |
- The only error with significant >= 95% full cases (1,990) and the highest mean storage usage (4.2%)
- 1,196 failed to even get database size —
navigator.storage.estimate()itself threw an error - Chromium — generic quota exceeded with real storage pressure in some cases
Cross-Reference Summary
How many auxiliary events matched to a nearby Failed to save to storage (same request_id, within 2s):
| Event | Total | Matched | Unmatched |
|---|---|---|---|
| Attempted to set invalid data | 958 | 957 | 1 |
| Failed after 5 retries | 354,111 | 353,969 | 142 |
| Out of storage — no keys to remove | 895 | 895 | 0 |
| Out of storage — evicting key | 1,238,436 | 1,235,763 | 2,673 |
| Storage Quota Check | 1,065,476 | 987,856 | 77,620 |
| Unable to get database size | 2,083 | 1,671 | 412 |
Key Takeaways
Error Landscape
-
The top 3 errors account for 71% of all failures — and each is exclusive to a specific browser engine:
- Chrome/Edge:
Internal error opening backing store(26.3%) - Firefox:
QuotaExceededError: transaction exceeded quota(25.7%) - Safari:
Connection to Indexed Database server lost(19.0%)
- Chrome/Edge:
-
Errors fall into two distinct categories with very different behaviors:
- Quota/capacity errors (
#2,#4,#7,#10— ~36.9%): trigger eviction, and relatively few of these requests end up exhausting retries, suggesting eviction generally helps. - Connection/state errors (
#1,#3,#5— ~51.7%): do NOT trigger eviction, just retry immediately and often exhaust all retries. These represent data loss since the write is silently skipped.
- Quota/capacity errors (
-
354K operations exhausted all 5 retries (10.5% of all errors), and
retryOperationresolves withPromise.resolve()— the caller has no way of knowing data was lost.
Quota Check Insights
-
Storage usage is extremely low across all quota-exceeding errors. Mean usage never exceeds 4.2% even for genuine quota errors. This is because:
- On web,
navigator.storage.estimate()reports the browser's total origin quota (often tens of GB), and IndexedDB only uses a small fraction - On native,
getFreeDiskStorage()reports total device free space, dwarfing the SQLite DB size - The actual per-origin storage limits enforced by browsers are much smaller than what
estimate()reports as the total quota
- On web,
-
Firefox's quota behavior is different from other browsers: it reports
QuotaExceededErroreven when overall storage usage is at 0.0%. The exact quota model Firefox uses (per-transaction, per-write, per-database, or something else) is unclear from the data alone — but the pattern is 100% Firefox-exclusive and accounts for 25.7% of all errors. -
Only 895 requests (0.027%) hit the "no evictable keys" dead end — it's rare for the eviction mechanism to run out of keys to remove.
Platform Distribution
-
Every major browser engine has its own dominant failure mode:
- Chromium (Chrome/Edge): internal backing store errors, blob write failures
- Gecko (Firefox): per-transaction quota limits
- WebKit (Safari): connection drops and database state errors
-
Native (NitroSQLite) errors account for ~5.3% — split between disk full (which triggers eviction effectively) and I/O errors (which don't trigger eviction at all).
Action Items
- Create GH issues to investigate each of the top 10 errors (except number 5) separately, where Sentry, VictoriaLogs and Fullstory could be used together to investigate some user scenarios with these errors to understand better why/when they happen, and look for a way to fix or lower this amount of errors.
- Improve
OnyxUtils.retryOperationandOnyxUtils.reportStorageQuotalogging to always include the original error in the different possible logs we can have there, so it's easier to query and analyse them in Sentry/VictoriaLogs instead of having to check nearby logs in the same timestamp to discover what was the true storage error (that is logged here). - Remove
disk I/O errorandout of memoryfromSQLITE_STORAGE_ERRORS, as the data shows they never trigger eviction in practice (0 evictions, 0 quota checks out of 89K occurrences) and they shouldn't be related to "storage" errors anyway. - Explore retry backoff or different mechanism for connection errors in
OnyxUtils.retryOperation. Errors#1,#3,#5account for 51.7% of all failures. Currently retries are immediate with no delay, which is ineffective for transient connection issues — the DB connection needs time to recover.
Activity
fabioh8010 commentedon Apr 14, 2026
@Julesssss @mountiny Could you add all these issues as sub-issues? Thanks!
UnknownError: Internal error opening backing store for indexedDB.open.storage error #87862QuotaExceededError: The current transaction exceeded its quota limitations.storage error #87863UnknownError: Connection to Indexed Database server lost.storage error #87864QuotaExceededError: Encountered full disk while opening backing store for indexedDB.open.storage error #87866nullstorage error #87867NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] database or disk is fullstorage error #87868NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] disk I/O errorstorage error #87869DataError: Failed to write blobs (InvalidBlob)storage error #87871QuotaExceededErrorstorage error #87873UnknownError: Internal error opening backing store for indexedDB.open.storage error #87862QuotaExceededError: The current transaction exceeded its quota limitations.storage error #87863UnknownError: Connection to Indexed Database server lost.storage error #87864QuotaExceededError: Encountered full disk while opening backing store for indexedDB.open.storage error #87866nullstorage error #87867NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] database or disk is fullstorage error #87868NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] disk I/O errorstorage error #87869DataError: Failed to write blobs (InvalidBlob)storage error #87871QuotaExceededErrorstorage error #87873mountiny commentedon Apr 15, 2026
done
13 remaining items
Julesssss commentedon Aug 5, 2026
Subtask here, coming from @elirangoshen's proposal
mountiny commentedon Aug 30, 2026
Fresh data for this investigation from the 14 day production crash scan on 2026-08-30. These are the storage errors that surfaced as unresolved unhandled crashes, ranked by unique users:
UnknownError: Database deleted by request of the userUnknownError: Internal error opening backing store for indexedDB.openAbortError: IDB write transaction aborted without an errorQuotaExceededErrorAbortError: The transaction was abortedTimeoutError: Transaction timed out due to inactivityUnknownError: Attempt to generate key in database without an in-progress transactionQuotaExceededError: Encountered full diskDataError: Failed to write blobs (IOError)ConstraintError: Key already exists in the object storeAbortError: IDB write transaction aborted without an errorAbortError: The connection was closedRoughly 326 unique users and 2670 events across the cluster in 14 days.
Two stand out as more than plain transients. APP-E0 is the only one Sentry rates
mediumactionability, and aConstraintError: Key already existssuggests a write path that does not treat a re-insert as an upsert. APP-KG (Attempt to generate key in database without an in-progress transaction) points at a transaction that has already committed or aborted being written to, which is a lifecycle bug rather than a disk or quota condition.Filed via the 14 day production crash scan; these are all treated as noise for crash triage purposes and left to this issue.
fabioh8010 commentedon Aug 31, 2026
I agree that APP-E0 and APP-KG seem to be the only ones worth to investigate, will create internal issues for them.
UnknownError: Attempt to generate key in database without an in-progress transactionstorage error #100046ConstraintError: Key already exists in the object storestorage error #100047Julesssss commentedon Sep 10, 2026
2 open issues here. One had a fix deployed to prod yesterday, another PR was just flipped from draft today.