[Due for payment 2026-08-31] [Onyx] Classify and measure disk-pressure storage failures on iOS #97908

Slack discussion: https://expensify.slack.com/archives/C08CZDJFJ77/p1785931079074959

Problem

Background: Expensify is offline-first: on iOS, every read and write goes through Onyx, backed by a NitroSQLite database on device storage. Several common flows also write large files to that same storage: receipt scanning, attachment downloads, and "Download as PDF" export. Over the last 30 days, Sentry issue APP-19Q (Expensify/App#87869) shows iOS-only bursts of NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] disk I/O error — ~92K log lines from ~50 users (1,747 iOS events vs 4 on Android), with the top user producing 16K lines in one day and one-hour bursts up to 7,569 lines. A June analysis correlated the bursts with heavy receipt-scan, report, and PDF-export activity, and found our storage retry loop amplifies each failure ~70x in logs.

On Jul 29 we reproduced the exact mechanism on an iOS simulator by putting the app database on a deliberately full disk volume:

• With the disk full at database (re)open, SQLite fails to size the OnyxDB-shm WAL-index file to 32KB (ENOSPC → SQLITE_IOERR_SHMSIZE), and every subsequent read and write returns "disk I/O error" until space frees — then it recovers on its own, matching the transient prod bursts.
• The same condition explains the other prod error shapes: "unable to open database file" (the shm file can't be created) and "cannot rollback - no transaction is active" (a batch-write failure masked by the rollback itself failing on the same full disk). A disk that fills after the DB is already open surfaces as SQLITE_FULL / masked rollback errors instead — explaining the mix of error strings in prod.
• PDF export on a full disk: the backend generates the PDF, but the download to device fails ("Attachment error: Attachment can't be downloaded") — matching prod users who retry downloads repeatedly.
• Most severe: if the disk is full at app boot, Onyx logs "Failed to load data from storage during init. The app will boot with default key states only." — the user sees the sign-in screen — and once disk space frees, those default values get persisted over the real database. Verified: the session record in SQLite was clobbered to {"loading":false,"creationDate":...}, authToken gone. That is a permanent logout and silent data loss, and a plausible root cause for unexplained session-loss reports in prod.

When an iOS device's storage is full or nearly full, if the user opens or uses Expensify, then every local read and write fails with NitroSQLiteError: [NativeNitroSQLiteException][SqlExecutionError] disk I/O error — breaking the app in-session, flooding our logs ~70x per failure, and, if it happens at app launch, silently signing the user out and permanently overwriting their real local data with blank defaults.

Solution

Classify these failures for what they are and make them measurable. Add a new DISK_PRESSURE storage error class in Onyx: classifySQLiteError routes the three error strings above to it, and retryOperation drops the write (the in-memory cache stays authoritative), skips retries and eviction (neither can free OS-level disk space), and logs one throttled alert per 60s window plus one storage quota snapshot per burst — on native including getFreeDiskStorage() bytes. This immediately kills the ~70x log amplification, and the free-disk telemetry answers the one thing we do NOT yet know: what fraction of prod IOERR bursts are truly disk-full vs other I/O causes (e.g. file-lock / Data Protection). If VictoriaLogs shows near-zero free-disk bytes on these alerts, the theory is confirmed at scale; if not, we've cheaply learned it's something else.