Repository navigation
[Onyx] Investigate QuotaExceededError: Encountered full disk while opening backing store for indexedDB.open. storage error #87866
Description
Coming from #87844.
Summary
Let's investigate the following Onyx storage error: QuotaExceededError: Encountered full disk while opening backing store for indexedDB.open., which accounts for 234,697 (7.0%) of the 3,364,645 analysed errors during a time span of 1 month. Full details about the investigation and methodology are in #87844.
| Platforms | Top Methods |
|---|---|
| Desktop Chrome (205K), Desktop Edge (21K), Desktop Opera (9K) | setWithRetry (232K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 6 | 234,222 | 471 | 16,327 | 0 | 0.0% | 475 |
- 0 at >= 95% and mean 0.0% full despite "full disk" in the error — this reflects IndexedDB's per-origin quota being exhausted, not the actual disk being full. The browser allocates a fraction of total storage to each origin, so "full disk" is a misnomer from Chromium
- Very few requests exhausted retries (6), suggesting eviction generally resolves the issue
Action Items
- Investigate with Sentry, VictoriaLogs and Fullstory to understand user scenarios — are these users with genuinely large Onyx stores, or is something else consuming the origin's quota?
- 471 requests hit the "no evictable keys" dead end — review whether more Onyx keys could be added to the
evictableKeyslist for these cases
Activity
sosek108 commentedon May 26, 2026
I've checked in VictoriaLogs and 100% of
Out of storage. But found no acceptable keys to removeerrors are not connected to any email. Query"Out of storage. But found no acceptable keys to remove" and emailreturned with zero hits over last 30 days.leshniak commentedon May 26, 2026
I'll take it.
From what I saw in the internet, it can be another bug in chrome - sometimes restart of the browser helps. Of course there can be also a real scenario where there is no more space on the disk, but I think it should be quite rare.
I'll go deeper.
leshniak commentedon May 27, 2026
Investigation Report
Root Cause Analysis
The error message
QuotaExceededError: Encountered full disk while opening backing store for indexedDB.open.is misleading — it does not indicate actual disk or quota exhaustion in the majority of cases. Our telemetry confirms this: 0% mean storage usage and 0 users at ≥95% full across 234,697 occurrences.How Chromium generates this error
Chromium uses LevelDB as the backing store for IndexedDB. When
indexedDB.open()is called, Chrome invokesIndexedDBBackingStore::Open()→leveldb_factory->OpenLevelDB(). If LevelDB returns an IOError, Chrome checks it withleveldb_env::IndicatesDiskFull(status)— a heuristic on the LevelDB status code, not an actual disk space check. When this heuristic fires, Chrome setsdisk_full = trueand surfaces theQuotaExceededErrorto JavaScript (source: indexed_db_factory.cc, source: indexed_db_backing_store.cc).This means LevelDB-level issues (stale lock files, corrupted metadata, QuotaManager inconsistency) are misreported as quota errors.
Known Chromium bugs contributing to false positives
Browser restart as the universal fix
Multiple sources confirm that restarting Chrome resolves this error because it:
This is documented by PixieBrix and confirmed in Chromium bug discussions.
Can Onyx's heal mechanism help?
Unlikely for most cases. Unlike the backing store corruption error (
UnknownError: Internal error opening backing store) where the problem is a stale cacheddbpin our code, this error originates from Chrome's IDB infrastructure actively refusingindexedDB.open(). A fresh JS-levelindexedDB.open()call hits the same Chrome-level state (corrupted QuotaManager, held LevelDB locks).However, the current eviction path resolves this 99.997% of the time (only 6 out of 234,697 exhausted retries). Two hypotheses for why:
Impact Assessment
navigator.storage.estimate()itself threw)Recommendations
Instrument to distinguish real vs false-positive quota errors. Add logging that captures:
navigator.storage.estimate()at the time of error (bytes used vs quota)Consider a retry-with-delay before eviction. If the async delay during eviction is what actually helps (hypothesis Fix spaces #2), a simple delayed retry before triggering eviction could avoid unnecessary data deletion. This could be implemented in
OnyxUtils.retryOperationas a short backoff (e.g. 100–500ms) specifically for this error pattern before falling back to the eviction path.Expand
evictableKeysfor the 471 dead-end cases. These users hit the error, eviction was triggered, but no keys were available to remove. Review which Onyx keys could be added to the evictable list.Do not add to the heal mechanism in
createStore.ts. The heal (dropdbp, immediate reopen) would likely fail for the same reason, waste a budget attempt, and the error would propagate to eviction anyway. ThecacheOpenPromiserejection cleanup in PR Update version to 1.0.1-177 #780 already prevents stale rejected promises from blocking subsequent operations, which is sufficient at thecreateStorelevel.sosek108 commentedon May 29, 2026
Hi, I'll be continuing investigation here
leshniak commentedon May 29, 2026
Investigation Update
Error Identification
The error
QuotaExceededError: Encountered full disk while opening backing store for indexedDB.open.accounts for 234,697 (7.0%) of all Onyx storage errors per month. It is 100% Chromium-based (Chrome, Edge, Opera) and occurs duringindexedDB.open(), not during write transactions.Sentry Analysis (90-day window)
Two Sentry issues capture this error as an unhandled rejection (after all Onyx retries + eviction failed):
Platform breakdown (90d):
~95% Windows. 100% Chromium-based.
Several users hit this error 3–5 times over weeks (e.g. user
10634764on May 20 and May 27), consistent with a persistent corrupted browser/profile state that recurs until something forces a reset.Fullstory Session Analysis
Built a Fullstory segment matching sessions with this console error (last 90 days, 20+ sessions returned). Examined 4 sessions in detail:
Finding 1: The "full disk" error is always preceded by another IDB error
In every session examined, the
QuotaExceededErrorwas a secondary/cascade error, never the root cause:/home): Error appears at 10.6s with no prior user action — immediately hits"Out of storage. But found no acceptable keys to remove", suggesting the IDB was already in a broken state from a previous page load or session.AbortError: The transaction was aborted→ thenUnknownError: Internal error opening backing store(error Some initial fixes and code style updates #1, backing store corruption) — the full-disk error never appears here, but the same LevelDB corruption is the root cause.UnknownError: Connection is closing→IDB InvalidStateError, retrying with fresh connection→ thenQuotaExceededError: Encountered full disk. Clear cascade: connection issue corrupts LevelDB state, which then reports false "full disk".DataError: Failed to write blobs (IOError)→nullerror → thenQuotaExceededError: Encountered full disk. Another cascade: blob write failure leads to corrupted backing store state.Finding 2: Infinite background error loop
Once the error starts, it never self-heals within the session. The pattern in every affected session:
retryOperationresolves silently (write skipped)This means a single occurrence of the underlying Chrome bug generates dozens to hundreds of error log entries per session. The 234K monthly error count likely represents far fewer actual affected sessions than it appears.
Finding 3: Users don't know their data is being lost
In Sessions C and D, users continued actively working — submitting expenses, editing fields, navigating between reports — for 10+ minutes after the errors started. Every Onyx write during this time silently failed. The app remained visually functional because it reads from the in-memory cache, but nothing was persisted to IDB. Any data entered during this window is lost on page refresh.
Finding 4: Co-occurrence with backing store corruption (APP-17G breadcrumbs)
In the latest APP-17G Sentry event, the breadcrumb trail shows
UnknownError: Internal error opening backing store for indexedDB.open.occurring 200ms before theQuotaExceededError. Both errors in the same session, same user — strongly confirming they share the same root cause: LevelDB backing store in a corrupted state.Root Cause: Chromium Bug, Not Real Quota Exhaustion
Evidence summary
disk_fullflag comes fromleveldb_env::IndicatesDiskFull(status)— a heuristic on LevelDB's status code, not an actual disk space checkWhy the current eviction path works 99.997% of the time
Only 6 out of 234,697 occurrences exhausted all retries. Two hypotheses:
Can the heal mechanism (PR #780) help?
Not directly. Unlike backing store corruption where the problem is a stale cached
dbp, this error originates from Chrome's IDB infrastructure actively refusingindexedDB.open(). A fresh JS-levelindexedDB.open()call hits the same Chrome-level state. ThecacheOpenPromiserejection cleanup in #780 already prevents stale rejected promises from blocking subsequent operations, which is sufficient at thecreateStorelevel.However, the Fullstory data shows that the "full disk" error frequently co-occurs with or follows the backing store corruption error (#1) and connection closing error (#3) — both of which #780 does heal. Fixing those upstream errors may reduce the cascade into false "full disk" errors.
Key Takeaways
The "full disk" error is a symptom, not the disease. The real root cause is LevelDB backing store corruption in Chromium. The
QuotaExceededErroris a misclassified cascade error.The 234K monthly count is inflated. A single affected session generates 30-80+ error entries due to the infinite retry loop (~30s interval). The actual number of affected user-sessions is orders of magnitude smaller.
Silent data loss is the real impact. Users continue working while every write silently fails. The app looks functional (reads from memory cache) but nothing persists to IDB.
Existing infrastructure can help filter false positives. Onyx already calls
navigator.storage.estimate()viagetDatabaseSize()on every eviction — but only logs the result, never uses it for decision-making. The data consistently shows 0% usage for these errors. Using this signal to skip eviction and retry with a delay instead could avoid unnecessary data deletion.The infinite retry loop needs a circuit breaker. Once
"no acceptable keys to remove"fires for a given error, subsequent retries for the same root cause will also fail. A mechanism to stop retrying the same failing operation (or at least back off exponentially) would reduce log noise and wasted CPU.sosek108 commentedon Jun 1, 2026
Update posted here: #87873 (comment)
sosek108 commentedon Jun 3, 2026
Let's close this as duplicate of #87873 (comment)
cc: @Julesssss