Skip to content

[Onyx] Investigate the most common storage errors #87844

Open
16 of 16 issues completed
Open

Description

@fabioh8010

Coming from #85252 (comment).

Onyx Storage Errors — Summary Report

Time span: 2026-03-02 to 2026-03-30 (~4 weeks)

Total Failed to save to storage entries: 3,364,645

Distinct error types: 52

Full list of all 52 error types
# Error Type Count %
1 UnknownError: Internal error opening backing store for indexedDB.open. 884,955 26.3%
2 QuotaExceededError: The current transaction exceeded its quota limitations. 865,703 25.7%
3 UnknownError: Connection to Indexed Database server lost. Refresh the page to try again 637,821 19.0%
4 QuotaExceededError: Encountered full disk while opening backing store for indexedDB.open. 234,697 7.0%
5 InvalidStateError: Failed to execute 'transaction' on 'IDBDatabase': The database connection is closing. 216,981 6.4%
6 null 90,661 2.7%
7 NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] database or disk is full 89,174 2.7%
8 NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] disk I/O error 89,137 2.6%
9 DataError: Failed to write blobs (InvalidBlob) 63,281 1.9%
10 QuotaExceededError 51,900 1.5%
11 UnknownError: Internal error. 51,743 1.5%
12 UnknownError: Connection is closing because of: IO error: <indexedDB file path> 19,053 0.6%
13 InvalidStateError: IDBDatabase.transaction: Can't start a transaction on a closed database 16,428 0.5%
14 NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] cannot rollback - no transaction is active 11,925 0.4%
15 AbortError: The connection was closed. 5,898 0.2%
16 UnknownError: Database deleted by request of the user 5,243 0.2%
17 NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] database is locked 4,097 0.1%
18 AbortError: The operation was aborted. 3,894 0.1%
19 InvalidStateError: A mutation operation was attempted on a database that did not allow mutations. 3,385 0.1%
20 UnknownError: Connection is closing because of: Failed to remove blob file. 2,991 0.1%
21 UnknownError: Unable to open database file on disk 2,820 0.1%
22 UnknownError: An internal error was encountered in the Indexed Database server 2,531 0.1%
23 AbortError: The transaction was aborted, so the request cannot be fulfilled. 2,399 0.1%
24 QuotaExceededError: The quota has been exceeded. 1,534 0.0%
25 ReferenceError: Can't find variable: indexedDB 1,356 0.0%
26 DataCloneError: Failed to execute 'put' on 'IDBObjectStore': function arrayBuffer() { [native code] } could not be cloned. 956 0.0%
27 DataCloneError: Failed to store record in an IDBObjectStore: BlobURLs are not yet supported. 726 0.0%
28 TimeoutError: Transaction timed out due to inactivity. 721 0.0%
29 NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] not an error 480 0.0%
30 UnknownError: Connection is closing because of: Force close delete origin 467 0.0%
31 UnknownError: The operation failed for reasons unrelated to the database itself and not covered by any other error code. 384 0.0%
32 DataError: Failed to write blobs (IOError) 379 0.0%
33 AbortError: A request was aborted, for example through a call to IDBTransaction.abort. 252 0.0%
34 UnknownError: Internal error opening database with no version specified. 132 0.0%
35 NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] another row available 89 0.0%
36 DataCloneError: The object can not be cloned. 84 0.0%
37 DataCloneError: Function object could not be cloned. 84 0.0%
38 UnknownError: Connection is closing. 78 0.0%
39 UnknownError: Internal error committing transaction. 73 0.0%
40 QuotaExceededError: Encountered disk full while committing transaction. 48 0.0%
41 NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] no more rows available 33 0.0%
42 UnknownError: Version change transaction on cached page is aborted to unblock other connections 18 0.0%
43 UnknownError: Attempt to get a record from database without an in-progress transaction 9 0.0%
44 UnknownError: Connection is closing because of: The database is deleted. 9 0.0%
45 UnknownError: Error looking up record in object store by key range 5 0.0%
46 Error: [SqlExecutionError] cannot rollback - no transaction is active 4 0.0%
47 TransactionInactiveError: A request was placed against a transaction which is currently not active, or which is finished. 2 0.0%
48 UnknownError: Connection is closing because of: Corruption: block checksum mismatch 1 0.0%
49 DataCloneError: Failed to execute 'put' on 'IDBObjectStore': error => { ... } could not be cloned. 1 0.0%
50 NitroSQLiteError: NitroSQLite.executeBatchAsync(...) threw an unknown St13runtime_error error. 1 0.0%
51 DataCloneError: Failed to execute 'put' on 'IDBObjectStore': Data cannot be cloned, out of memory. 1 0.0%
52 NotReadableError: Data lost due to missing file. Affected record should be considered irrecoverable 1 0.0%

Methodology

Seven log messages were analyzed from OnyxUtils.retryOperation and OnyxUtils.reportStorageQuota in react-native-onyx:

Message Pattern Entries
Failed to save to storage. Error: ${error}. onyxMethod: ${method}. retryAttempt: N/5 3,364,645
Storage operation failed after 5 retries. Error: ${error}. onyxMethod: ${method}. 354,111
Out of storage. Evicting least recently accessed key (${key}) and retrying. 1,238,436
Storage Quota Check -- bytesUsed: ${used} bytesRemaining: ${remaining} 1,065,476
Attempted to set invalid data set in Onyx. Please ensure all data is serializable. 958
Out of storage. But found no acceptable keys to remove. 895
Unable to get database size. Error: ${dbSizeError} 2,083

Cross-referencing: auxiliary events were matched to the nearest Failed to save to storage entry sharing the same request_id within a 2-second window. This approach was validated by measuring time gaps between events: 99.5%+ of related events occur within 1 second of each other.

Platform inference: derived from the userAgent string in each log entry. Detection relies on unique tokens: Edge includes Edg/, Opera includes OPR/, Firefox includes Firefox/ (no Chrome/ token), and Chrome includes Chrome/ without Edge/Opera markers. These are checked in priority order to avoid misclassification (e.g. Edge's UA also contains Chrome/).

Storage Quota Check interpretation: on web, bytesUsed and bytesRemaining come from navigator.storage.estimate() which reports the browser's overall storage quota for the origin (not just the IndexedDB database). On native, bytesUsed is the SQLite database file size (via PRAGMA page_size * page_count) and bytesRemaining is the device's free disk space (via getFreeDiskStorage()).


Top 10 Errors

1. UnknownError: Internal error opening backing store for indexedDB.open. — 884,955 (26.3%)

Platforms Top Methods
Desktop Chrome (664K), Desktop Edge (220K) setWithRetry (682K), multiSetWithRetry (181K)
Retries Exhausted Evictions No Eviction Quota Checks >= 95% Full Mean % Full Quota Check Failed
147,944 0 0 0 — — 0
  • No eviction or quota check events — this is an IndexedDB internal failure unrelated to storage space
  • This error is not classified as a storage capacity error in Onyx (STORAGE_ERRORS only includes QuotaExceededError for IDB), so it never triggers the eviction flow — it just retries immediately and fails repeatedly
  • Likely caused by corrupted IndexedDB state, browser extensions interfering, or Chromium-level bugs

2. QuotaExceededError: The current transaction exceeded its quota limitations. — 865,703 (25.7%)

Platforms Top Methods
Desktop Firefox (865K — 100%) setWithRetry (865K)
Retries Exhausted Evictions No Eviction Quota Checks >= 95% Full Mean % Full Quota Check Failed
6,089 859,566 0 859,276 0 0.0% 0
  • 859,276 quota checks with mean 0.0% full — Firefox reports quota exceeded even when overall usage is negligible. The exact quota model (per-transaction, per-write, or other) is unclear from the data alone
  • Firefox-exclusive error — different quota behavior from Chromium and Safari
  • Eviction is triggered frequently, and relatively few requests end up exhausting retries (6,089), suggesting eviction generally helps resolve this error

3. UnknownError: Connection to Indexed Database server lost. — 637,821 (19.0%)

Platforms Top Methods
Desktop Safari (558K), iOS Safari (72K), iOS Chrome (8K) setWithRetry (578K), multiSetWithRetry (34K)
Retries Exhausted Evictions No Eviction Quota Checks >= 95% Full Mean % Full Quota Check Failed
106,394 0 0 0 — — 0
  • Safari/WebKit-exclusive — the IDB connection drops unexpectedly
  • No eviction or quota involvement — not recognized as a capacity error, so Onyx retries immediately without eviction
  • 106K exhausted all retries (16.7% of this error) — immediate retries don't help because the connection itself is gone

4. QuotaExceededError: Encountered full disk while opening backing store for indexedDB.open. — 234,697 (7.0%)

Platforms Top Methods
Desktop Chrome (205K), Desktop Edge (21K), Desktop Opera (9K) setWithRetry (232K)
Retries Exhausted Evictions No Eviction Quota Checks >= 95% Full Mean % Full Quota Check Failed
6 234,222 471 16,327 0 0.0% 475
  • 0 at >= 95% and mean 0.0% full despite "full disk" in the error — this reflects IndexedDB's per-origin quota being exhausted, not the actual disk being full. The browser allocates a fraction of total storage to each origin, so "full disk" is a misnomer from Chromium
  • Very few requests exhausted retries (6), suggesting eviction generally resolves the issue

5. InvalidStateError: Failed to execute 'transaction' on 'IDBDatabase': The database connection is closing. — 216,981 (6.4%)

Platforms Top Methods
Desktop Safari (103K), Desktop Chrome (75K), iOS Safari (29K), iOS Chrome (7K) setWithRetry (181K), multiSetWithRetry (32K)
Retries Exhausted Evictions No Eviction Quota Checks >= 95% Full Mean % Full Quota Check Failed
37,303 0 0 0 — — 0
  • Cross-browser but Safari-dominant — the DB connection is being closed (page navigation, tab discard, or browser cleanup) while writes are in-flight
  • Similar to #3: not a capacity error, so no eviction is triggered. Immediate retries are unlikely to help if the DB connection is closing

Issue was handled here: #84192

6. null — 90,661 (2.7%)

Platforms Top Methods
Desktop Safari (40K), Desktop Chrome (26K), iOS Safari (15K), Desktop Edge (7K) setWithRetry (80K), multiSetWithRetry (9K)
Retries Exhausted Evictions No Eviction Quota Checks >= 95% Full Mean % Full Quota Check Failed
13,089 23 0 254 0 0.1% 0
  • The error object was null — no useful error message was provided by the storage backend
  • Cross-browser, slightly Safari-leaning — may be a Safari-specific issue where the error object is not properly populated

7. NitroSQLiteError: database or disk is full — 89,174 (2.7%)

Platforms Top Methods
Other/Native (89K), Android Native (153) setWithRetry (89K)
Retries Exhausted Evictions No Eviction Quota Checks >= 95% Full Mean % Full Quota Check Failed
10 88,998 106 60,468 0 2.4% 0
  • 60,468 quota checks with mean 2.4% of device disk used — the SQLite database itself is relatively small compared to overall device storage, but the device's free space is exhausted by other apps/data
  • Native-only (NitroSQLite) — very few requests exhausted retries (10), suggesting eviction generally resolves the issue

8. NitroSQLiteError: disk I/O error — 89,137 (2.6%)

Platforms Top Methods
Other/Native (89K) setWithRetry (89K)
Retries Exhausted Evictions No Eviction Quota Checks >= 95% Full Mean % Full Quota Check Failed
14,882 0 0 0 — — 0
  • No eviction or quota events despite "disk I/O error" being explicitly listed in SQLITE_STORAGE_ERRORS. This is unexpected — the eviction path should be triggered but isn't. Needs investigation (see action items)
  • disk I/O error is arguably not a storage capacity error and should probably be removed from SQLITE_STORAGE_ERRORS. It indicates filesystem corruption, SD card issues, or device under stress — evicting keys won't help with I/O failures
  • Native-only

9. DataError: Failed to write blobs (InvalidBlob) — 63,281 (1.9%)

Platforms Top Methods
Desktop Chrome (61K), Desktop Edge (2K) multiSetWithRetry (48K), setWithRetry (15K)
Retries Exhausted Evictions No Eviction Quota Checks >= 95% Full Mean % Full Quota Check Failed
9,537 0 0 0 — — 0
  • Chromium-exclusive — blob data becoming invalid between when it's created and when it's written to IDB
  • Notable that multiSetWithRetry dominates (76%) — batch writes are more susceptible
  • Not a capacity error, so no eviction — retries are unlikely to help since the blob data is fundamentally invalid

10. QuotaExceededError (bare, no detail) — 51,900 (1.5%)

Platforms Top Methods
Desktop Chrome (51K), Desktop Edge (915) setWithRetry (51K)
Retries Exhausted Evictions No Eviction Quota Checks >= 95% Full Mean % Full Quota Check Failed
259 51,322 307 48,022 1,990 4.2% 1,196
  • The only error with significant >= 95% full cases (1,990) and the highest mean storage usage (4.2%)
  • 1,196 failed to even get database size — navigator.storage.estimate() itself threw an error
  • Chromium — generic quota exceeded with real storage pressure in some cases

Cross-Reference Summary

How many auxiliary events matched to a nearby Failed to save to storage (same request_id, within 2s):

Event Total Matched Unmatched
Attempted to set invalid data 958 957 1
Failed after 5 retries 354,111 353,969 142
Out of storage — no keys to remove 895 895 0
Out of storage — evicting key 1,238,436 1,235,763 2,673
Storage Quota Check 1,065,476 987,856 77,620
Unable to get database size 2,083 1,671 412

Key Takeaways

Error Landscape

  1. The top 3 errors account for 71% of all failures — and each is exclusive to a specific browser engine:

    • Chrome/Edge: Internal error opening backing store (26.3%)
    • Firefox: QuotaExceededError: transaction exceeded quota (25.7%)
    • Safari: Connection to Indexed Database server lost (19.0%)
  2. Errors fall into two distinct categories with very different behaviors:

    • Quota/capacity errors (#2, #4, #7, #10 — ~36.9%): trigger eviction, and relatively few of these requests end up exhausting retries, suggesting eviction generally helps.
    • Connection/state errors (#1, #3, #5 — ~51.7%): do NOT trigger eviction, just retry immediately and often exhaust all retries. These represent data loss since the write is silently skipped.
  3. 354K operations exhausted all 5 retries (10.5% of all errors), and retryOperation resolves with Promise.resolve() — the caller has no way of knowing data was lost.

Quota Check Insights

  1. Storage usage is extremely low across all quota-exceeding errors. Mean usage never exceeds 4.2% even for genuine quota errors. This is because:

    • On web, navigator.storage.estimate() reports the browser's total origin quota (often tens of GB), and IndexedDB only uses a small fraction
    • On native, getFreeDiskStorage() reports total device free space, dwarfing the SQLite DB size
    • The actual per-origin storage limits enforced by browsers are much smaller than what estimate() reports as the total quota
  2. Firefox's quota behavior is different from other browsers: it reports QuotaExceededError even when overall storage usage is at 0.0%. The exact quota model Firefox uses (per-transaction, per-write, per-database, or something else) is unclear from the data alone — but the pattern is 100% Firefox-exclusive and accounts for 25.7% of all errors.

  3. Only 895 requests (0.027%) hit the "no evictable keys" dead end — it's rare for the eviction mechanism to run out of keys to remove.

Platform Distribution

  1. Every major browser engine has its own dominant failure mode:

    • Chromium (Chrome/Edge): internal backing store errors, blob write failures
    • Gecko (Firefox): per-transaction quota limits
    • WebKit (Safari): connection drops and database state errors
  2. Native (NitroSQLite) errors account for ~5.3% — split between disk full (which triggers eviction effectively) and I/O errors (which don't trigger eviction at all).


Action Items

  1. Create GH issues to investigate each of the top 10 errors (except number 5) separately, where Sentry, VictoriaLogs and Fullstory could be used together to investigate some user scenarios with these errors to understand better why/when they happen, and look for a way to fix or lower this amount of errors.
  2. Improve OnyxUtils.retryOperation and OnyxUtils.reportStorageQuota logging to always include the original error in the different possible logs we can have there, so it's easier to query and analyse them in Sentry/VictoriaLogs instead of having to check nearby logs in the same timestamp to discover what was the true storage error (that is logged here).
  3. Remove disk I/O error and out of memory from SQLITE_STORAGE_ERRORS, as the data shows they never trigger eviction in practice (0 evictions, 0 quota checks out of 89K occurrences) and they shouldn't be related to "storage" errors anyway.
  4. Explore retry backoff or different mechanism for connection errors in OnyxUtils.retryOperation. Errors #1, #3, #5 account for 51.7% of all failures. Currently retries are immediate with no delay, which is ineffective for transient connection issues — the DB connection needs time to recover.

Activity

  1. fabioh8010 commented on Apr 14, 2026

    @fabioh8010
    ContributorAuthor
  2. mountiny commented on Apr 15, 2026

    @mountiny
    Contributor

    done

  3. 13 remaining items

  4. Julesssss commented on Aug 5, 2026

    @Julesssss
    Contributor

    Subtask here, coming from @elirangoshen's proposal

  5. mountiny commented on Aug 30, 2026

    @mountiny
    Contributor

    Fresh data for this investigation from the 14 day production crash scan on 2026-08-30. These are the storage errors that surfaced as unresolved unhandled crashes, ranked by unique users:

    Sentry ID Users (14d) Events (14d) Error
    APP-F1 39 226 UnknownError: Database deleted by request of the user
    APP-EF 38 128 UnknownError: Internal error opening backing store for indexedDB.open
    APP-G2H 37 1427 AbortError: IDB write transaction aborted without an error
    APP-F9 35 111 QuotaExceededError
    APP-EM 35 126 AbortError: The transaction was aborted
    APP-4R 25 40 TimeoutError: Transaction timed out due to inactivity
    APP-KG 22 37 UnknownError: Attempt to generate key in database without an in-progress transaction
    APP-11A 22 71 QuotaExceededError: Encountered full disk
    APP-12V 20 34 DataError: Failed to write blobs (IOError)
    APP-E0 18 137 ConstraintError: Key already exists in the object store
    APP-G0Z 17 145 AbortError: IDB write transaction aborted without an error
    APP-15T 18 88 AbortError: The connection was closed

    Roughly 326 unique users and 2670 events across the cluster in 14 days.

    Two stand out as more than plain transients. APP-E0 is the only one Sentry rates medium actionability, and a ConstraintError: Key already exists suggests a write path that does not treat a re-insert as an upsert. APP-KG (Attempt to generate key in database without an in-progress transaction) points at a transaction that has already committed or aborted being written to, which is a lifecycle bug rather than a disk or quota condition.

    Filed via the 14 day production crash scan; these are all treated as noise for crash triage purposes and left to this issue.

  6. fabioh8010 commented on Aug 31, 2026

    @fabioh8010
    ContributorAuthor

    I agree that APP-E0 and APP-KG seem to be the only ones worth to investigate, will create internal issues for them.

  7. Julesssss commented on Sep 10, 2026

    @Julesssss
    Contributor

    2 open issues here. One had a fix deployed to prod yesterday, another PR was just flipped from draft today.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions