INCIDENT: crypto payments cannot reconcile stuck provider state

[dzianisv]

Symptom\nOpenClawBot payment id=226 has remained pending for about 33 hours. Production readiness cannot determine whether funds moved because CryptoPayments exposes no authenticated provider-status or reconciliation surface.\n\n## Impact\nOne known payment is stuck and customer fulfillment cannot be safely decided. Any lost/delayed callback can leave future payments non-terminal indefinitely.\n\n## Safety\nDo not mutate id=226 until the OpenClawBot row identity, on-chain/provider record, and entitlement state are proven. A paid but unfulfilled customer must be provisioned or refunded, never silently closed.\n\n## Status\nInvestigating both databases, provider webhook transitions, and scheduled-job delivery. Design will be posted before money-path implementation.

[dzianisv]

Ground-truth tracing underway. Confirmed implementation split: OpenClawBot creates local pending rows at checkout/hold; CryptoPayments owns Supabase payment_intents and chain verification. CryptoPayments has authenticated list/read APIs but no tx-hash status endpoint, no callback replay endpoint, no scheduler, and no alert transport. Existing public GET /api/payment/:id only works when caller already knows provider intent id. Also confirmed the real-chain E2E workflow has failed on every main run since 2026-07-24; CI unit suite remains green, so current safety signal is false-green for the live rail. No payment state has been mutated.

[dzianisv]

Money-path design (before implementation)

Ground truth

• OpenClawBot row 226: user_id 22 / dedicated E2E identity raccoonfriendly (Telegram 6668889106), max, 10000 AUSD, tx 0xb02dab84c633b51a2b3081f6f1ddcb90c65c4157f6d6c0905d895c5c49b93b90, created 2026-08-11T14:11:07Z, no subscription.
• Live row changed concurrently from pending to refunded at 2026-08-13T06:34:46Z; this investigation did not mutate it. On-chain receipt proves a successful 100 aUSD Ethereum Sepolia transfer from the dedicated test wallet to the configured recipient. A DB refunded flag alone is not refund proof; reverse-transfer verification is required.
• Transition that never happened while stuck: local pending -> completed with subscription, because no fulfillment was attached. Bot retained logs contain no callback in the last 48h. Provider key present in the live bot currently returns 401, so current operator status access is broken.

API

• Add authenticated GET /v1/payment_intents/by_tx/:chain/:txHash. Return redacted provider intent, chain-derived status, callback delivery state/attempt count, and a needs_action classification. Never expose secrets or arbitrary metadata.
• Add authenticated POST /v1/payment_intents/:id/reconcile. Re-read chain truth. Pending chain tx remains non-terminal and reports retryable. Confirmed valid transfer becomes succeeded, then missing/failed callback is replayed from server-persisted signed intent metadata. Invalid/reverted/expired transfer becomes failed. Already succeeded+delivered is idempotent. Refund status is only accepted with a verified reverse transaction; no DB-only refund.

Scheduled reconciler

• Add a repository script that scans stale processing/legacy-pending provider intents plus succeeded intents whose callback is absent/failed, calls the same reconciliation function, and exits nonzero if any item remains unknown or undeliverable.
• Add a scheduled GitHub workflow using pnpm (never npm ci), existing Supabase service-role + callback secrets, with workflow_dispatch for immediate proof. The workflow creates/updates a GitHub incident on failure, so a broken monitor is itself visible. It must print counts/ids only, never secrets.

State ownership

• CryptoPayments terminalizes provider state and callback delivery. OpenClawBot remains entitlement owner: successful callback consumes its matching pending row and attaches fulfillment. A compatibility-held payment remains explicit REFUND-OR-APPLY debt; provider must not call it refunded without reverse-chain evidence.

Coverage

• API integration tests with real Hono request/auth and persistent DB behavior.
• Child-process reconciler test proving pnpm wiring and nonzero alert result.
• Real Sepolia reconciliation against a known confirmed transaction plus callback receiver, in the existing live E2E workflow; repair the currently red workflow instead of accepting unit-only green.

Rollback/safety

• Additive endpoints and job; no migration required for callback replay state (existing metadata). No automatic outbound wallet signing. Reconciliation uses conditional/idempotent transitions and callback replay only.

[dzianisv]

Current resolution evidence

Payment 226 source of truth: direct read from live OpenClaw control-plane Postgres on vibe-k3s-prod (not local .env, not inferred from a sweep). Final local state is refunded, refunded_at=2026-08-13T06:34:46.349Z, user_id=22 / dedicated E2E identity raccoonfriendly, no subscription, amount 10000 AUSD, tx 0xb02dab84c633b51a2b3081f6f1ddcb90c65c4157f6d6c0905d895c5c49b93b90.

Chain source of truth: Ethereum Sepolia receipt succeeded and transferred 100 aUSD from the dedicated test sender to recipient 0xa6a5...15ef; reverse-transfer query found none. Therefore no production customer paid and received nothing, but the local refunded label is administrative E2E cleanup, not proof of an economic refund.

Implementation PRs:

• CryptoPayments feat: reconcile stuck crypto payments #45: authenticated tx status, reconciliation, fail-loud pnpm scheduler + incident creation, Edge deploy workflow.
• OpenClawBot #3916: authenticated GET /admin/api/payments/:id, POST /admin/api/payments/:id/reconcile, explicit provider unavailable/missing states, docs/admin.md.

Privileged blockers proven by live CI:

• stored SUPABASE_SERVICE_ROLE_KEY is rejected with Legacy API keys are disabled;
• SUPABASE_ACCESS_TOKEN is absent, so Edge deployment cannot run;
• current bot CRYPTO_PAYMENTS_API_KEY gets provider 401 and needs parity rotation.
  No secret values were printed.

[dzianisv]

Provider state is observable again — and 226 was already terminal

Credential blocker (#47) is cleared: CRYPTO_PAYMENTS_API_KEY rotated and fingerprint-verified across Supabase / k8s / Bitwarden / both repos' Actions, and the disabled legacy service_role JWT is replaced by the project's sb_secret_* key. Details and evidence in #47 — no values anywhere.

Payment 226 is not stuck. With provider reads working:

Side	State
provider payment_intents 8198354c-…	succeeded, block 11466686, succeeded_at 2026-08-11T14:11:06Z
callback	delivered, attempts=1, last_error=null, 2026-08-11T14:11:07.777Z
OpenClawBot local	refunded, refundedAt 2026-08-13T06:34:46Z

The callback succeeded on the first attempt 1.7s after settlement and our row was written at 14:11:07.328Z. It settled → notified → recorded → refunded. Terminal, consistent, nothing owed.

Sweep of the full provider table (the thing this incident actually wanted):

succeeded 192 | failed 49 | requires_payment_method 44

succeeded by chain:
  eth_sepolia  / ausd  189   $11,120
  base_sepolia / usdc    2       $20
  eth_sepolia  / usdc    1       $20

There has never been a mainnet payment. All 94 succeeded-without-delivered-callback rows come from one wallet (0x64cd…1a30, 189 payments over ~7 weeks) — the automated E2E wallet.

So the "stuck provider state" this incident describes is real as a mechanism gap, but it has never touched customer funds. Suggest re-scoping this from an active incident to a pre-mainnet correctness requirement, gated on:

• e2e-topup has been red on main since 2026-07-24 (pre-dates the deploymentType fix) #42 (e2e-topup UI failure — now the only thing blocking PR feat: reconcile stuck crypto payments #45)
• PR feat: reconcile stuck crypto payments #45's /api/status + /v1/payments/reconcile routes, currently 404 / not deployed