[Onyx] Investigate NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] disk I/O error storage error

Coming from #87844.

Summary

Let's investigate the following Onyx storage error: NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] disk I/O error, which accounts for 89,137 (2.6%) of the 3,364,645 analysed errors during a time span of 1 month. Full details about the investigation and methodology are in #87844.

Platforms	Top Methods
Other/Native (89K)	`setWithRetry` (89K)

Retries Exhausted	Evictions	No Eviction	Quota Checks	>= 95% Full	Mean % Full	Quota Check Failed
14,882	0	0	0	—	—	0

No eviction or quota events despite "disk I/O error" being explicitly listed in SQLITE_STORAGE_ERRORS. This is unexpected — the eviction path should be triggered but isn't. Needs investigation (see action items)
disk I/O error is arguably not a storage capacity error and should probably be removed from SQLITE_STORAGE_ERRORS. It indicates filesystem corruption, SD card issues, or device under stress — evicting keys won't help with I/O failures
Native-only

Action Items

Investigate with Sentry, VictoriaLogs and Fullstory to understand user scenarios — what devices/OS versions are affected? Is there a pattern (e.g., SD card storage, specific Android vendors)?
~~Remove "disk I/O error" from SQLITE_STORAGE_ERRORS in OnyxUtils.ts — it's not a capacity error and evicting keys won't resolve filesystem I/O failures. Consider adding retry backoff instead.~~ (being handled here)

ViktoriaLogs analysis
Although many errors are reported over last 30 days - 50K events, only 36 users are affected. And query grouping by time

_time:30d "[NativeNitroSQLiteException]" "[SqlExecutionError]" "disk I/O error"
  | stats by (email, _time:30m) count() as errors_in_window
  | stats by (email) count() as incidents, sum(errors_in_window) as total_errors
  | sort by (incidents desc)

shows that 23 of the affected users encountered that error only once. Many repeated error logs for single session are reported.
13 users encounter this error more than once and 1 user saw this 6 times.
A lot of incidents were reported for sessions without email (email is null or we@dont.know)

Sentry analysis
Sentry analysis shows that this issue hits only iOS users.

Fullstory

User session shows that this error occurs when he tries to download PDF with transaction (10:36 AM UTC, 10:53 AM UTC)
Same here (10:44 PM UTC)
Same here (2:54 AM UTC)

TL;DR; I've confirmed flow that leads to this error but wasn't able to reproduce this on simulator.

SQLite disk I/O error Burst — Investigation Summary

Background

iOS (HybridApp) users hit recurring bursts of [NativeNitroSQLiteException][SqlExecutionError] disk I/O error. A few users generate a large burst without any specific repetition. up to 7,569 log lines in one hour for a single user.
More closer look into ViktoriaLogs shoved that only few users were affected by this error, but due to excessive logging and write heavy situation one instance of error was logged many times.

37 users affected for last 30 days
only 4 users encountered this more than once

Approach

in logs found 4 users that encounter this more than once and cross referenced logs, Sentry data and Fullstroy replays
- 8 separate incidents were analyzed

Observations and conclusions

7 out of 8 sessions showed that this error were reported during heavy write, upload and download action. Users did:

scan multiple receipts into report
open the report
tried to download report as PDF or export report
unfortunately due to Fullstory limitations it’s hard to tell if the download were successfull but probably not as some sessions show multiple trials

3 times application was backgrounded when this happened. This may be related to iOS Data Protection. SQLITE_IOERR here means that application cannot access it’s SQLIte file when it’s in background. Application resumed to operate

Two app-side amplifiers turn a transient hiccup into a storm:

Futile retry loop. setWithRetry retries 6× with no backoff and logs every attempt → ~7 lines per failed write = the 70× gap (7,569 logs vs 109 events). It then resolves silently → the write is lost from disk (still in cache) → this is already resolved Remove disk I/O and out of memory from SQLITE_STORAGE_ERRORS #87781
OnyxDerived write storm. reportAttributes/todos recompute on any item change in any dep collection, each a full-blob Onyx.set. At one session’s peak minute: 818 IOERRs + 239 reportAttributes writes — same event.

Reproduction (the workflow every affected user ran)

All bursts trace to the same admin flow on SearchMoneyRequestReport:

iOS HybridApp, workspace with many reports.
Create an expense report; add many expenses via SmartScan ("Create expense → Choose from gallery"), ~4-10 receipts, sometimes resolving duplicates ("Review duplicates → Keep selected"), editing category/description.
Open the report → Submit.
From the report More menu → Download as PDF (often repeated, plus Print / Export → All Data).
PDF generation stalls for seconds–minutes (one session: 255s gap)
User locks the phone / backgrounds the app during/after generation.
→ burst of disk I/O error as the deferred Onyx write flush hits the now-locked SQLite file.

Exact instances:

(May 30): report (~8 expenses) → Download as PDF (×3) → Print → Submit another expense → Print again; errors 23:16–23:33Z while backgrounded.
(Jun 12): draft report, 21 expenses → repeated Export / Download as PDF / Print; ManualOpenReport 19.3s.
(May 25): Scan 4 receipts → Submit → Download as PDF → backgrounded on Workspaces_List; errors fire 45–63 min later on background flush, free memory ~73 MB.
(Jun 8): SmartScan loop → Submit → More → Download as PDF; tight 25s foreground burst.

Next steps

Investigate the reproduction steps more carefully to ensure that PDF Download / Export works properly for users with large reports
Investigate SQLITE_IOERR in background. Wheter it’s possible to defer actions until app is foregrounded?

@fabioh8010

@fabioh8010, this Monthly task hasn't been acted upon in 6 weeks; closing.

If you disagree, feel encouraged to reopen it -- but pick your least important issue to close instead.

@mountiny

@mountiny Could you reopen? Thanks

@elirangoshen

Updates:

Waiting for Bump PR of Classify SQLite disk-pressure errors and stop retrying them react-native-onyx#816 to land before @elirangoshen can check logs and confirm issue is mitigated.

Prod verification of #97954 (nitro-sqlite patch so failed ROLLBACKs stop masking real SQLite errors, prod 9.4.52-11, Aug 12):

cannot rollback - no transaction is active ran ~70/day before Aug 12, down to single digits by Aug 20+ (VictoriaLogs, [OnyxUpdateManagerError] stream).
The real underlying errors now surface instead — e.g. database or disk is full (77 hits / 35 users since Aug 1) and IDB/SQLite disk-pressure variants — so this issue's disk I/O error class can now be investigated from actual error strings rather than masked rollback noise.

Follow-up: how much free disk do devices actually have when SQLite throws database or disk is full?

Since onyx classifies these as DISK_PRESSURE, every occurrence now logs a paired Storage Quota Check line with the DB size (PRAGMA page_size * page_count) and the device's free bytes (getFreeDiskStorage()). Pulled the distribution from VictoriaLogs for Aug 1–24 — 7,180 events across 105 users, all iOS (per userAgent):

Free disk at throw	Events	Users
<1MB (literally full)	47	1
1–50MB	411	5
50–200MB	5,732 (80%)	82
200–500MB	582	18
>500MB (up to 3GB)	17	3

(391 events had an unparsable field; median free = 122MB, p90 = 204MB.)

Takeaways:

"Disk is full" almost never means zero bytes free. The dominant class is a device with 50–200MB free — consistent with iOS denying app writes once free space drops below its low-space reserve (~200MB), while NSFileSystemFreeSize (what getFreeDiskStorage reads) still reports space the OS won't actually grant. So on iOS, SQLITE_FULL effectively means "below the iOS reserve threshold", not literal exhaustion.
The 21 users throwing with >200MB free are an anomaly worth a separate look. SQLITE_FULL can also fire on max_page_count/container-level limits, and free space can rebound between the failed write and the quota check (iOS purges caches under pressure) — but 3GB-free cases don't fit the disk-pressure story at all.
This supports the no-retry/no-evict classification for DISK_PRESSURE: evicting rows inside our own DB can't fix a device-level reserve threshold.

Caveat: bytesRemaining is sampled after the write fails (throttled, once per burst), so individual readings can drift from the instant of the throw; the distribution shape is the signal, not any single value.

@fabioh8010

Wrapping up the investigation. Summary of where this landed, then the case for closing.

The blocker in my last update is resolved

The wait was on #97944 to bump onyx#816. #97944 is still open and titled [TESTING ONLY] — it was a verification pin, never meant to merge. #816 reached production through the regular version bump instead, though later than assumed:

step	date
onyx#816 merged	Aug 7
first onyx version containing it	3.0.97 (verified by unpacking the published tarballs; 3.0.96 does not have it)
App main on 3.0.98	Aug 20 (`ce8b8ed5a19`)
first prod release carrying #816	9.4.60-2, Aug 25

Release tags confirm the earlier builds were not covered: 9.4.57-3 and 9.4.58-4 both ship onyx 3.0.95, 9.4.60-2 ships 3.0.98, 9.4.62-4 ships 3.0.100.

Also already in prod: #97954 (nitro-sqlite patch so failed ROLLBACKs stop masking real SQLite errors), merged Aug 10, prod 9.4.52-11 Aug 12, verified in the Aug 24 update.

What the investigation established

This is device disk pressure, not an App defect. From the free-disk distribution in my Aug 24 comment, 80% of throws happen with 50-200MB free — below the iOS low-space reserve. On iOS SQLITE_FULL / disk I/O error effectively means "below the OS reserve threshold", not literal exhaustion. Nothing in our code can raise a device-level threshold.
The classification work is shipped and correct. DISK_PRESSURE is never retried and never triggers eviction — evicting rows from our own DB cannot fix a device reserve threshold.
Amplification, not incidence, was the loud part. disk I/O error logged 35,149 events between Aug 17-30, but from only 1-9 unique users per day, ~30 users total. The single loudest user produced 8,389 lines in one day. The headline volume was one broken DB handle being re-reported by many Onyx call sites, not a broad failure.
Platform is 100% iOS native — every event carries an Expensify/<version> CFNetwork/… Darwin/… userAgent with stacks inside Expensify.app/main.jsbundle. (The source field reads ecash… here; that token is not web-specific, so platform has to be read from the userAgent.)

Early signal from #816

Splitting Aug 25-30 by whether the client carries #816:

client	events	users	events/user
releases ≤ 9.4.59-5 (no #816)	22,694	17	~1,335
9.4.60-2 / 9.4.61-4 / 9.4.63-1 (#816)	331	4	~83

~16x less amplification per affected user, and 9.4.62-4 produced zero events in the window. Being straight about the limits: that is 6 days of exposure across 4 affected users, and absolute counts are not comparable mid-rollout, so treat it as directional rather than proof.

Why close rather than hold

The remaining loose end is the group throwing SQLITE_FULL with >200MB free. That is 21 users, and SQLITE_FULL has known non-disk causes (max_page_count, container-level limits) plus free space can rebound between the failed write and the quota sample. It is not a large enough population to justify keeping an investigation issue open, and it is a different question from the one this issue asks.

So: root cause is understood and environmental, the classification fix is shipped, the affected population is ~30 users over two weeks, and the amplification is trending down on the releases that carry the fix. I do not think further open-ended investigation here buys anything.

Proposing we close this as complete. If the >200MB group is worth chasing later it should be its own issue with its own scope. Happy to be overruled if anyone wants to wait for 9.4.60+ to reach the bulk of iOS traffic before calling it — that would be roughly another two weeks.

#97944 should be closed as superseded regardless.

what do you think @fabioh8010 ?

@Julesssss

I agree, can we wrap up this one @Julesssss ?

This issue has not been updated in over 15 days. eroding to Monthly issue.

P.S. Is everyone reading this sure this is really a near-term priority? Be brave: if you disagree, go ahead and close it out. If someone disagrees, they'll reopen it, and if they don't: one less thing to do!

Navigation Menu

Repository navigation

[Onyx] Investigate `NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] disk I/O error` storage error #87869

Summary

Action Items

SQLite disk I/O error Burst — Investigation Summary

Background

Approach

Observations and conclusions

Reproduction (the workflow every affected user ran)

Next steps

The blocker in my last update is resolved

What the investigation established

Early signal from #816

Why close rather than hold

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Onyx] Investigate NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] disk I/O error storage error #87869

Description

Summary

Action Items

Activity

sosek108 commented on Jun 16, 2026

sosek108 commented on Jun 18, 2026

SQLite disk I/O error Burst — Investigation Summary

Background

Approach

Observations and conclusions

Reproduction (the workflow every affected user ran)

Next steps

melvin-bot commented on Jun 29, 2026

fabioh8010 commented on Jun 30, 2026

fabioh8010 commented on Aug 17, 2026

elirangoshen commented on Aug 24, 2026

elirangoshen commented on Aug 24, 2026

elirangoshen commented on Aug 31, 2026

The blocker in my last update is resolved

What the investigation established

Early signal from #816

Why close rather than hold

fabioh8010 commented on Aug 31, 2026

melvin-bot commented on Aug 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[Onyx] Investigate `NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] disk I/O error` storage error #87869