Repository navigation
[Due for payment 2026-07-28] [Onyx] Investigate QuotaExceededError storage error #87873
Description
Coming from #87844.
Summary
Let's investigate the following Onyx storage error: QuotaExceededError (bare, no detail), which accounts for 51,900 (1.5%) of the 3,364,645 analysed errors during a time span of 1 month. Full details about the investigation and methodology are in #87844.
| Platforms | Top Methods |
|---|---|
| Desktop Chrome (51K), Desktop Edge (915) | setWithRetry (51K) |
| Retries Exhausted | Evictions | No Eviction | Quota Checks | >= 95% Full | Mean % Full | Quota Check Failed |
|---|---|---|---|---|---|---|
| 259 | 51,322 | 307 | 48,022 | 1,990 | 4.2% | 1,196 |
- The only error with significant >= 95% full cases (1,990) and the highest mean storage usage (4.2%)
- 1,196 failed to even get database size —
navigator.storage.estimate()itself threw an error - Chromium — generic quota exceeded with real storage pressure in some cases
Action Items
- Investigate with Sentry, VictoriaLogs and Fullstory to understand user scenarios — these are the users with the highest actual storage pressure. What does their Onyx store look like? Are there specific keys consuming disproportionate space?
- 307 requests hit the "no evictable keys" dead end — review whether more Onyx keys could be added to the
evictableKeyslist - Investigate why 1,196 requests failed to get database size —
navigator.storage.estimate()throwing errors may indicate browsers in private/incognito mode or restrictive privacy settings
Activity
sosek108 commentedon May 26, 2026
I'll be investigating this
sosek108 commentedon May 27, 2026
We're seeing ~52k QuotaExceededError events/month on web when Onyx writes to IndexedDB, and 100% of the affected population is on Chromium (Chrome + Edge). The brief framed this as a storage-pressure problem ("heavy users hitting the origin quota cap"), but log analysis killed that story: across 23k VictoriaLogs lines over 7 days, the maximum observed IDB usage was 6.26% of origin quota — every affected user has gigabytes of headroom.
The current best hypothesis is that Chromium surfaces JS-heap allocation failures during IDB writes as a bare QuotaExceededError with no message. Three reproducer users in Sentry were all at 87–97% JS heap when the error fired, on screens that issue large bulk writes (/search, /home, /transition, /workspaces). MDN and web.dev confirm QuotaExceededError is an umbrella class covering disk-quota, per-transaction sub-quota, and memory-constraint failures — Chromium throws the bare error name without a message by design, so the empty message is normal, not a logging bug. Example Sentry to support this hypothesis. (FullStory)
Proposed next steps:
sosek108 commentedon May 28, 2026
What a ride this investigation is.
Update
Out of 28k events 80% of them are from 4 users (where one of them is our test account) so this is user specific:
These users are reporting high Storage usage (
Storage Quota Check -- bytesUsed:logs):I've been able to reproduce this issue by setting low storage quota:
When storage is overused browser throws
QuotaExceededErrorwithout any specific explanation. Our logsStorage Quota Check -- bytesUsed: ${bytesUsed} bytesRemaining: ${bytesRemaining}. Original error: ${error}are misleading because they don't show correct bytesRemaining. Browser reports full 10G availability, but assigned storage can be much much lower (eg. 150M).Observation
when user has low storage assigned (weaker laptop, small disk availability) storage may be totally used and we get this error. I don't think we can do anything about this.
Maybe we can see why users has so much storage used at all?
I've open staging app on my second computer (old M1 with 9G disk space availability) and Chrome assinged 147 MB storage.
60M of it was used by CACHE before I've even logged in!. After switch to Applause QA perf account I've used 128MB out of 147MB.
Action steps
IDBKeyValProvidermodifygetDatabaseSizeto return alsousageDetailsprops which tells us how is the storage used.sosek108 commentedon May 28, 2026
Follow up:
after 4h of total inactivity there is 132M used and 124 by cache and 7,5M by IndexedDB. So inactivity does not add much, but it's still 3M out of thin air
sosek108 commentedon May 29, 2026
Immediately after application is open Workbox Precache is caching ~314 elements and this weights around 125M. I did not notice this value to rise, but prod user sessions may be very long. And cache more things such as receipt images.
The final root cause of this error is: user's application uses all of available space. This is either for cache, IndexedDB or both. This is hard to reproduce on dev as Precache is disabled for dev environment and we use quite capable machines I guess. For low end machines or machines with low disk space availability storage allowed for Exfy is very low. And we cannot do much about it.
After I've logged in to
applausetester@account I've used 137M in no time. But it was hard to go to cap 147 as database clearing was working.Impact
There are plenty of log messages on ViktoriaLog because there is retry method after
QuotaExceededErroris very aggresive but after taking closer look only few (like less than 10) users are really affected by this.Sentry data backups this claim as
QuotaExceededErrorsare much much less occuring in sentry.Proposed next steps
Storage Quota Check -- bytesUsed:to log also what is using the bytes - cache, DB or other. This will give us more debug information for future issues.sosek108 commentedon May 29, 2026
If we decide to push these logs here is PR: Expensify/react-native-onyx#795
cc: @mountiny @fabioh8010
sosek108 commentedon Jun 1, 2026
Bump PR in E/App: #92085
sosek108 commentedon Jun 1, 2026
I'd propose to merge investigations of this, #87863 and #87866 as their main root cause is the same - no space left.
Summary of the problem
Real root cause of all issues is that there is no space left for the application on user's machine. There are two types of this situation:
There is nothing we can do about user's disk space.
Impact
For last 30 days this impacted 700 users where vast majority (~ 95%) was hit by this only once. The issue was reproducible only for 4 real users and they was responsible for >95% of log volume.
Volume of logging comes from situation that for many of user's session problem was not visible. Application operated normally (through cache) from user's perspective so they were using it as usually. Every user action (eg. OpenReport) causes multiple Onyx write and each onyx write create few (~5-7) log lines.
Potential action steps
It seems like application operates normally and as expected in unfriendly environment. For most cases it silently discards write issues and user is able to recover from this by restarting browser or computer so no immediate action is required.
I don't see any reason to continue investigation here.
63 remaining items
abzokhattab commentedon Aug 8, 2026
looks like payment is missing here ... i reviewed these two PRs #95044 #96268 (comment)
can we reopen for payment ... cc @mountiny
melvin-bot commentedon Aug 9, 2026
Triggered auto assignment to @mallenexpensify (
Awaiting Payment)melvin-bot commentedon Aug 9, 2026
Payment Summary
Resolving PRs:
Bump Onyx to 3.0.83 #93280
[Onyx bump] Bump react-native-onyx from 3.0.86 to 3.0.88 #95044
[Onyx bump] Bump react-native-onyx from 3.0.86 to 3.0.88 #96268
Contributor: @sosek108 does not require payment (Contractor)
Reviewer: @abzokhattab owed $250 via NewDot
BugZero Checklist (@mallenexpensify)
mallenexpensify commentedon Aug 10, 2026
Payment Summary
Contributor+: @abzokhattab due $250 via NewDot
@abzokhattab @mountiny I'm assuming a single payment for both PRs here, please comment if you disagree. I'll leave it open for a round then close next time I see it in my inbox.
abzokhattab commentedon Aug 10, 2026
@mallenexpensify thanks for picking this up. Would you consider two payments here?
These were two separate reviews rather than one .. #95044 was reverted, and #96268 re-landed it two weeks later on a much newer main. Both carried the HybridApp warning, so each needed separate review, and on the second one I also tested the SAML flow that caused the revert
Happy to defer if you see it differently.
cc @mountiny
mallenexpensify commentedon Aug 11, 2026
@roryabraham 👀 above plz since you reviewed both PRs. Thx
roryabraham commentedon Aug 11, 2026
Hmmmm I don't think that a revert of the first PR justifies an independent payment for the second PR.
quinthar commentedon Aug 19, 2026
Approved $250 to @abzokhattab