[OlamideAdesina]

What does this PR do?

Closes #645 by hardening the one background worker task processor that still contradicts the issue's goal: src/worker/tasks/balance-sync.ts.

A prior assessment (my evidence comment on #645) found the worker cycle retry/dead-letter machinery already on main via #708/#783 (withRetry, classifyJobFailure, recordCycleFailure). That part is genuinely done. But balance-sync.ts — a sibling worker task, explicitly in the issue's scope ("any associated worker task processors"; "prevents dropped tasks") — had the exact failure mode #645 describes, and zero tests:

• syncAccountBalances wrapped its whole body in try/catch and swallowed every error (including a loadAccount that had already exhausted its internal Horizon read-retry budget, and any transient throw from the accountBalance upsert). Because nothing rethrew, the outer loop's catch could never fire, so synced++ ran and failed was dead code pinned at 0. A Horizon/RPC outage reported { synced: N, failed: 0 } — tasks dropped with a success-shaped result.
• No task-level bounded retry over the DB-write step, and no structured "this failed permanently, here's the category" log.

Change — reuse the repo's existing primitives rather than invent new ones:

• Route each account's sync through the shared withRetry wrapper (src/worker/health.ts) with a small BALANCE_SYNC_RETRY_POLICY. withRetry classifies via classifyJobFailure and re-attempts only demonstrably-transient failures, so a 4xx fails fast instead of burning attempts; indeterminate and permanent failures throw straight to the accounting.
• syncAccountBalances no longer swallows — it lets failures propagate to the wrapper/caller, and keeps the legitimate "account not found" case as a non-failure early-return (preserving existing semantics).
• The loop now counts honestly: a permanent / indeterminate / budget-exhausted failure increments failed and emits a structured pino error with category, accountId, type, and a safeFailureMessage-scrubbed reason (no secrets/signed XDR in logs).
• Injectable retryPolicy / retrySleep / onRetry seams (the same pattern withRetry already exposes for cycle tasks) so the behaviour is testable offline with deterministic timing.

Tests — tests/worker/balance-sync.test.ts (8 cases, src/db + stellar + pino mocked, the real withRetry exercised):

• transient failure → retried, succeeds, counted synced, onRetry fired, error never logged;
• transient, budget exhausted → counted failed, loadAccount called exactly maxAttempts (bounded, not infinite), structured error with category: "transient";
• permanent (4xx) → fails fast, loadAccount called once (the "don't waste retries" guarantee);
• missing account → non-failure, nothing upserted;
• no active accounts → {0,0}, no network call;
• sibling isolation — one account's permanent failure doesn't starve another; counts split {1,1};
• runBalanceSync wires production defaults.

Non-vacuity (the tests actually catch the bug)

Reverting only src/worker/tasks/balance-sync.ts to main makes 5 of the 8 tests fail with exactly the dropped-task symptom — e.g. expected { synced: 1, failed: 0 } to deeply equal { synced: 0, failed: 1 }. The fix flips them green.

Checklist

• Read and followed the Contributing Guidelines
• npm run lint passes (0 errors, 33 warnings — identical to base 063571b)
• npm run build passes (tsc -p tsconfig.json, exit 0)
• npm test passes (offline — DB/network mocked)
• New endpoints validated with Zod + permission checks — N/A, no endpoint added
• Audit logs added for state-changing actions — N/A, this is a worker task; failure visibility is the structured pino dead-letter log
• Contract changes mirrored in mergepay-web — N/A, no HTTP contract touched
• No secrets committed — log reasons go through safeFailureMessage (scrubs bearer/token/xdr)

Gates — base 063571b vs 063571b + this branch

Measured locally (shell Node 26; CI pins 20.x). The change is isolated: balance-sync.ts is imported by nothing else in src, and its only test is this new file, so the integration suite is unaffected (no integration test references it).

                       base 063571b          branch (this)         delta
npm run build          exit 0                exit 0                —
npm run lint           0 err / 33 warn       0 err / 33 warn       —
npm test (unit)        184 files / 3717 p    185 files / 3725 p    +1 file / +8 tests, 0 regressions

Related issues

Closes #645

(On the worker cycle side, #645 was already satisfied by #708/#783 — see my evidence comment on #645; this PR closes the remaining gap in the balance-sync task processor, which #708/#783 did not cover.)

[drips-wave]

@OlamideAdesina Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits