Repository navigation
[Due for payment 2026-07-28] [Onyx] Investigate UnknownError: Connection to Indexed Database server lost. storage error #87864
Description
Coming from #87844.
Summary
Let's investigate the following Onyx storage error: UnknownError: Connection to Indexed Database server lost., which accounts for 637,821 (19.0%) of the 3,364,645 analysed errors during a time span of 1 month. Full details about the investigation and methodology are in #87844.
| Platforms | Top Methods |
|---|---|
| Desktop Safari (558K), iOS Safari (72K), iOS Chrome (8K) | setWithRetry (578K), multiSetWithRetry (34K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 106,394 | 0 | 0 | 0 | — | — | 0 |
- Safari/WebKit-exclusive — the IDB connection drops unexpectedly
- No eviction or quota involvement — not recognized as a capacity error, so Onyx retries immediately without eviction
- 106K exhausted all retries (16.7% of this error) — immediate retries don't help because the connection itself is gone
Action Items
- Investigate with Sentry, VictoriaLogs and Fullstory to understand user scenarios — is this caused by tab backgrounding, memory pressure, or Safari's aggressive IDB cleanup?
Consider adding retry backoff for this error. Immediate retries are ineffective when the connection is gone — a delay could allow the browser to re-establish it.(being handled here)- Consider detecting when the IDB connection is lost and reopening it before retrying
- Consider adding a
visibilitychangelistener to proactively reconnect when the tab returns to foreground
Activity
leshniak commentedon Apr 28, 2026
I'll investigate this too.
leshniak commentedon May 15, 2026
Investigation Report
Error Overview
UnknownError: Connection to Indexed Database server lost. Refresh the page to try againThe 2,300x gap between Sentry (275) and VL (121K) is expected — Sentry captures only the unhandled rejection that bubbles to
window.onunhandledrejection, while VL captures everyFailed to save to storageretry attempt (up to 5 per operation).Hypothesis: Safari Kills IDB Connections for Backgrounded Tabs
Sentry breadcrumbs show a consistent pattern across sampled events:
UnknownErrorFor example, in session #1 (Sentry event), the breadcrumbs show a 5h22m gap between last activity (05:30 UTC) and the error (10:52 UTC). After the gap: version.json check, Ping, PusherPing, ReconnectApp — all API fetches succeed (status 200), but the subsequent Onyx IDB writes fail because the connection was severed during the background period.
Safari/WebKit aggressively terminates IDB connections for backgrounded tabs to conserve resources. When the tab comes back, the JavaScript context is still alive but the underlying IDB connection to the browser's database process has been severed. All subsequent IDB operations fail immediately.
Sampled Sessions
All sampled events show
lowMemoryThreat: falseandhandled: no(unhandled rejection viaonunhandledrejection). No stacktrace is available — the error originates from IDB internals and surfaces as a promise rejection.Evidence
1. Retries Are Completely Futile
Retry attempt distribution (VL, 7d):
Almost perfectly flat. Every operation that fails attempt 1 also fails 2, 3, 4, and 5. The connection is permanently dead within the retry window. Immediate retries (current behavior) cannot help because there is no delay or reconnection logic — the same dead connection is reused.
97.5% of sessions that encounter this error exhaust all 5 retries, resulting in silent data loss (
retryOperationresolves withPromise.resolve()— the caller never knows).2. Bursty — Few Sessions Generate Massive Log Volume
Top hourly spikes (VL, 7d, UTC):
A single backgrounded tab returning can generate thousands of failed retry attempts as ReconnectApp triggers a cascade of Onyx writes (
setWithRetryandmultiSetWithRetry) against a dead IDB connection — consistent with the reconnection burst pattern observed in session #1's breadcrumbs.Average: ~5.6 retry entries per session, but distribution is heavily skewed — some sessions produce hundreds of entries.
3. 100% WebKit — No Other Browser Engine Affected
Sentry breakdown (30d):
All browsers in the list use WebKit under the hood (Chrome/Edge/DuckDuckGo on iOS are required to use WebKit by Apple's App Store policy). Zero events from Chromium or Gecko engines. This is visible across the sampled sessions — ranging from Edge Mobile on iOS (session #1) to desktop Safari 18.x-26.x (sessions #2-#5).
OS distribution: macOS (360 events), iOS (297+ events). Both desktop and mobile Safari are affected.
4. Affects Landing Pages During Reconnection
Top pages (Sentry):
These are the pages a user lands on when returning to a backgrounded tab — exactly where ReconnectApp fires. The sampled sessions confirm this: /home (#1, #5), /search (#2), /e/{reportID} (#3), /onboarding/purpose (#4).
5. Not Memory or Storage Related
lowMemoryThreat: falsein all 5 sampled Sentry events6. Broadly Distributed Across Users and Releases
7. Operations That Fail
VL breakdown by onyxMethod (7d):
setWithRetrymultiSetWithRetryCurrent Handling in Onyx
STORAGE_ERRORS(which only containsquotaexceedederroranddatabase or disk is full), so it is treated as a generic errorretryOperationretries immediately 5 times with zero delay, reusing the same dead connectionretryOperationresolves withPromise.resolve()— silent data lossdbpand reopens, this error does not trigger any reconnection attemptonclosehandler exists (createStore.ts:30-33): clearsdbpwhen the browser firesoncloseon the IDB connection — but Safari may not always fire this event before the connection is dead, leaving a staledbpcachedPreliminary Recommendations
"connection to indexed database server lost"to a newIDB_CONNECTION_ERRORSconstant inOnyxUtils.ts(already specced in the retry-backoff design doc)executeTransaction(similar to backing store heal atcreateStore.ts:133-142), drop the cacheddbpand open a fresh IDB connection before retrying. This is the critical missing piece — without it, retries and backoff are both useless.visibilitychangelistener — proactively detect tab foregrounding and test/refresh the IDB connection before Onyx writes land, preventing the error from occurring at allNON_RETRIABLE_ERRORSapplicability — if the reconnect-and-retry strategy doesn't work (i.e., the IDB server is truly gone and cannot be reopened), this error should be added toNON_RETRIABLE_ERRORSto avoid generating 5x futile retry logs per operationOpen Questions (next steps)
leshniak commentedon May 18, 2026
Final Investigation Report
Error Overview
UnknownError: Connection to Indexed Database server lost. Refresh the page to try againThe gap between Sentry (275) and VL (121K) is expected — Sentry captures unhandled rejections that bubble to
window.onunhandledrejection(plus logger integration and sampling), while VL captures everyFailed to save to storageretry attempt (up to 5 per operation). The Sentry events likely originate from initialization paths (getAll,multiGet,getAllKeys) and logger integration rather than from the retry chain itself, which resolves withPromise.resolve().Root Cause: Safari Terminates IDB for Backgrounded Tabs
Safari/WebKit aggressively terminates IDB connections for backgrounded tabs. When the tab returns, the JavaScript context is still alive but the underlying IDB connection is severed. This is a documented WebKit behavior (WebKit #197050, WebKit #201483).
Sentry breadcrumbs and Fullstory session replays confirm the pattern:
UnknownErrorCaveat: Tab backgrounding is the primary trigger, but not the only one — WebKit networking process crashes and memory pressure can also sever the IDB connection without a clean background/foreground cycle.
Sampled Sessions
Sentry Events
S1 breadcrumb analysis: 5h22m gap between last activity (05:30 UTC) and error (10:52 UTC). After the gap: version.json, Ping, PusherPing, ReconnectApp — all API fetches succeed (status 200), but subsequent Onyx IDB writes fail.
All sampled events show
lowMemoryThreat: falseandhandled: no.Fullstory Sessions
InvalidStateErrorheal fires → fresh connection also fails with connection lost.error-clickevents.InvalidStateError→ connection lost → user works 40 min with errors recurring at 15:32, 16:05, 16:30, 17:17. Connection permanently dead.Evidence
1. Retries Are Futile (with current code)
Retry attempt distribution (VL, 7d):
Almost perfectly flat. The connection is dead within the retry window. But critically: the current code never clears the cached
dbpfor this error — all 5 retries reuse the same stale connection. This doesn't prove that a freshindexedDB.open()would also fail (see "Important nuance" below).97.5% of sessions exhaust all 5 retries, resulting in silent data loss (
retryOperationresolves withPromise.resolve()).2. Bursty — Few Sessions Generate Massive Log Volume
Top hourly spikes (VL, 7d, UTC):
A single backgrounded tab returning triggers a cascade of Onyx writes against a dead IDB connection — consistent with S1 breadcrumbs and F1 replay.
3. 100% WebKit
Sentry breakdown (30d):
Safari version breakdown (VL, 7d):
Reflects market share — no version is immune. Long-standing WebKit behavior.
4. IDB Never Recovers (without intervention)
Fullstory F5 and F6: users worked 30-40+ minutes after the first error. Errors recurred at every navigation point. The connection is permanently dead until page refresh.
The app remains usable from in-memory cache — users navigate, browse, submit expenses. But all writes silently fail.
5. Important Nuance: Reconnection Was Never Tested for 99% of Cases
"Connection to Indexed Database server lost" is an
UnknownErroronIDBRequest— it hits the async rejection path inexecuteTransaction. The existing heal mechanism increateStore.tsonly triggers for:InvalidStateError(sync throw ondb.transaction())UnknownErrorwith "Internal error opening backing store")For this error, the cached
dbpis never cleared andindexedDB.open()is never re-attempted. The flat retry counts prove the stale connection is reused, not that a fresh connection would also fail.In the ~1% of cases where
InvalidStateErrorheal did fire first (F4, F6), the fresh connection also failed — but WebKit #197050 comment #5 suggests this is expected when the IDB server process was terminated. Whether a delayed reopen (e.g., aftervisibilitychange+ short wait) could succeed is untested.6. Top Pages = Landing Pages
Exactly where ReconnectApp fires on tab return.
7. Not Memory or Storage Related
lowMemoryThreat: falsein all sampled events8. Operations That Fail
setWithRetrymultiSetWithRetryCurrent Handling in Onyx
STORAGE_ERRORS, treated as generic errorretryOperationretries immediately 5 times with zero delay, reusing the same stale cacheddbpcreateStore.tsheal only triggers forInvalidStateErrorand backing store errors — this error bypasses it entirelyPromise.resolve()— silent data lossonclosehandler: Exists increateStore.ts— clearsdbpwhen browser firesonclose. Safari may not fireonclosereliably when it kills the IDB server, leaving staledbpcached.Proposed Fix
Layer 1: Extend createStore heal + fail-fast (immediate win)
In
createStore.ts: ExtendexecuteTransactionerror handling to detect"connection to indexed database server lost"(and"connection is closing"). On match:dbp(set toundefined)indexedDB.open()) + retry the transactionstoragePermanentlyDegradedflagIn
OnyxUtils.ts: WhenretryOperationencounters this error after the createStore heal has already failed, stop retrying immediately (fail-fast). Emit onelogAlertper session instead of 5×logInfoper operation.Impact: Eliminates ~120K VL entries/week. Tests whether a fresh
indexedDB.open()can actually recover some sessions (currently unknown — we've never tried for 99% of cases).Layer 2: Proactive detection via visibilitychange
Add a
visibilitychangelistener in the IDB provider. When tab returns to foreground:storagePermanentlyDegradedbefore ReconnectApp triggers the Onyx write stormThis prevents the entire error cascade. Note:
document.onfreeze/document.onresume(Page Lifecycle API) are not supported in Safari —visibilitychangeis the correct event.Layer 3: User notification
When
storagePermanentlyDegradedis set, expose a callback (e.g.,Onyx.onStorageDegraded) so the App can show a persistent banner prompting the user to refresh. A full page reload is the documented recovery path per WebKit #197050.Alternative: Call
degradePerformance()to switch toMemoryOnlyProviderfor the rest of the session — this makes the silent failure explicit and matches existing degradation infrastructure instorage/index.ts. However, this doesn't solve the data loss on next refresh (IDB is stale).Scoping
Layers are independent and can ship separately. Layer 1 should go first — it both fixes the immediate noise problem and answers the open question of whether
indexedDB.open()can recover these sessions.Note on #87782 (retry backoff)
The retry backoff proposal in #87782 targets connection errors broadly with exponential delay. For this specific Safari error class, backoff alone won't help — the connection is permanently dead, not transiently unavailable. The fix here (heal + fail-fast) should be treated as a separate path from general backoff, which remains useful for other error classes.
leshniak commentedon May 18, 2026
Proposed Solution
The core problem
When Safari kills the IDB connection, Onyx keeps using the stale cached connection (
dbpincreateStore.ts). Every retry hits the same dead connection —indexedDB.open()is never re-attempted. This is why retries are 100% futile (flat distribution across all 5 attempts).The fix: detect the error and reopen the connection
In
createStore.ts, the.catchhandler aroundexecuteTransactioncurrently only handlesInvalidStateError. We extend it to also detect"Connection to Indexed Database server lost"and"Connection is closing". On match:dbp(forcing a freshindexedDB.open()on the next call)Note: this is the same heal pattern being built for the sibling issue #87862 (Chromium backing store corruption) in react-native-onyx#780. Both errors have the same structural problem (stale
dbp, never cleared) and the same fix shape — we're handling them together in one mechanism.What this tells us
The investigation found that 99% of connection lost cases never attempted a fresh
indexedDB.open()— the heal path was bypassed entirely. Only ~1% triggeredInvalidStateErrorfirst (where the existing heal fired, but the fresh connection also failed).This fix directly tests what happens for the other 99%:
Either outcome is a win — we either recover sessions or stop wasting 120K+ log entries/week on retries that can never succeed.
Follow-up layers (separate PRs, independent)
NON_RETRIABLE_ERRORSto stop 5× futile retries inretryOperationvisibilitychangelistener + lightweight IDB probe when tab returns to foregroundOnyx.onStorageDegradedcallback so the App can show a "please refresh" bannerfabioh8010 commentedon May 19, 2026
The
Proactive detectionidea sounds interesting to me and I think it can help in these cases.However I don't think we should do
User notification– I don't think it's good UX to notify users about it, mainly becauseI think we should focus in efforts of healing and preventing the DB gets to this state, while keeping the failing situations invisible to the user.
feat: extend IDB heal to detect Safari connection lost error
feat: add visibilitychange probe for proactive IDB health check
feat: extend IDB heal to detect Safari connection lost error
feat: add visibilitychange probe for proactive IDB health check
leshniak commentedon May 19, 2026
@fabioh8010 extended Expensify/react-native-onyx#780 to handle this kind of error.
leshniak commentedon May 20, 2026
Extracted proactive detection to Expensify/react-native-onyx#788
feat: extend IDB heal to detect Safari connection lost error
15 remaining items
melvin-bot commentedon Jul 6, 2026
Reviewinglabel has been removed, please complete the "BugZero Checklist".melvin-bot commentedon Jul 6, 2026
The solution for this issue has been 🚀 deployed to production 🚀 in version 9.4.27-2 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:
If no regressions arise, payment will be issued on 2026-07-13. 🎊
For reference, here are some details about the assignees on this issue:
[-][Due for payment 2026-07-13] [Onyx] Investigate `UnknownError: Connection to Indexed Database server lost.` storage error[/-][+][Due for payment 2026-07-28] [Onyx] Investigate `UnknownError: Connection to Indexed Database server lost.` storage error[/+]melvin-bot commentedon Jul 21, 2026
Reviewinglabel has been removed, please complete the "BugZero Checklist".melvin-bot commentedon Jul 21, 2026
The solution for this issue has been 🚀 deployed to production 🚀 in version 9.4.40-0 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:
If no regressions arise, payment will be issued on 2026-07-28. 🎊
For reference, here are some details about the assignees on this issue: