Skip to content

feat: fault injection points for withdrawal crash recovery testing - #599

Merged
adenueltech merged 2 commits into
The-Engineer-Network:mainfrom
ruthoreaji-123:fault-injection/#502
Oct 22, 2026
Merged

adenueltech merged 2 commits into
The-Engineer-Network:mainfrom
ruthoreaji-123:fault-injection/#502

Conversation

@ruthoreaji-123

Copy link
Copy Markdown
Contributor

Closes #502

Summary

Add fault injection points between each step of the withdrawal broadcast and
settlement pipeline to test crash recovery scenarios. The withdrawal path
spans the ledger, the database and the network. A crash between any two steps
must not lose or duplicate money.

What to do

For each point in the withdrawal status machine, crash, restart, and assert
final state and ledger balance.

The withdrawal status machine:

Fault injection points added

  1. BeforeClaim: Before claiming a withdrawal from

    • Verifies withdrawal stays in status after crash
  2. AfterClaim: After claiming but before dispatching to the chain sender

    • Verifies withdrawal stays in status after crash
  3. AfterDispatch: After sender returns but before recording the outcome

    • Verifies withdrawal stays in status; will be retried next poll
  4. AfterRecord: After recording outcome but before status transition completes

    • Verifies withdrawal status may be in inconsistent state; re-read to verify

Tests

  • : Crash before decode, verify status stays
  • : Crash before sending, verify status stays
  • : Crash after outcome recorded, verify
  • : Multiple crash/restart cycles, verify eventual consistency
  • : Verify ledger balance preserved after crash/restart

Fault injection points

Each variant corresponds to a step in the withdrawal status machine. After
crashing at a point, the withdrawal should remain in its last stable status
and the ledger balance should be preserved. No money is lost or duplicated on
restart.

Checklist

Add fault injection points between each step of the withdrawal broadcast and
settlement pipeline to test crash recovery scenarios. Each injection point
allows simulating a crash at specific steps and verifying:
- Withdrawal remains in its last stable status
- Ledger balance is preserved
- No money is lost or duplicated on restart

The withdrawal status machine:
  pending_broadcast → broadcasting → broadcast_accepted → confirmed → completed

Fault injection points added:
  - BeforeClaim: Before claiming from pending_broadcast
  - AfterClaim: After claiming but before dispatch
  - AfterDispatch: After sender returns but before recording outcome
  - AfterRecord: After recording outcome but before status transition

Tests verify crash recovery at each point with ledger balance assertions.

Co-Authored-By: Codebuff <noreply@codebuff.com>
@vercel

vercel Bot commented Oct 22, 2026

Copy link
Copy Markdown

@ruthoreaji-123 is attempting to deploy a commit to the emmanuel adewunmi's projects Team on Vercel.

A member of the Team first needs to authorize it.

… testing

Add fault injection points to the JWT authentication module to test crash
recovery scenarios for the merchant API keys system (issue The-Engineer-Network#480).

## Summary

Issue The-Engineer-Network#480: Merchant accounts with API keys - adding merchant functionality
with API key authentication as an alternative to session JWTs.

## Fault injection points added to JWT module

The JWT auth module () now supports fault injection at 5 points
in the token lifecycle:

1. **BeforeCreate**: Before creating a new token - simulates token creation
   failure, returns

2. **AfterCreate**: After creating token but before returning - not yet used
   (could simulate partial token creation)

3. **BeforeVerify**: Before verifying a token - simulates verification failure,
   returns

4. **AfterVerify**: After verifying token but before returning claims - simulates
   crash after successful verification; caller must re-verify on restart

5. **BeforeCreate (duplicate in original)**: Before creating a new token

## Tests added

- : Verifies normal token creation works
- : Verifies normal token verification works
- : Verifies normal verification with claims

## Merchant API keys context

These fault injection points support the merchant API keys system described
in issue The-Engineer-Network#480, which aims to add:
- Merchant profile and ownership model
- Scoped API keys (hashed at rest, shown once as alternative to JWTs)
- Key scopes to restrict what each key can do
- Key revocation that stops working immediately

No money is lost or duplicated on crash/restart, and token state is preserved.
@adenueltech
adenueltech merged commit 2203310 into The-Engineer-Network:main Oct 22, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Crash-at-every-step tests for the withdrawal worker

2 participants