Skip to content

fix: offload bolt11_decode in list_transactions to worker thread#37

Open
BenGWeeks wants to merge 1 commit into
lnbits:mainfrom
BenGWeeks:fix/list-transactions-event-loop
Open

fix: offload bolt11_decode in list_transactions to worker thread#37
BenGWeeks wants to merge 1 commit into
lnbits:mainfrom
BenGWeeks:fix/list-transactions-event-loop

Conversation

@BenGWeeks
Copy link
Copy Markdown

Take the proposed fix with a pinch of salt — drafted quickly with AI assistance. The diagnosis (synchronous bolt11_decode blocking the event loop) is grounded in reading the handler code, but the exact fix shape deserves maintainer review.

Summary

Fixes #35

_on_list_transactions in tasks.py calls the synchronous bolt11_decode() inside a Python for loop in an async handler. On wallets with non-trivial history this blocks the asyncio event loop long enough that:

  • NWC clients hit their reply timeout (~60s) and see list_transactions fail
  • Relay keep-alives, publish tasks, and concurrent NWC requests are starved
  • LNbits logs show nothing useful — the handler is simply still running

Other NWC methods (get_info, get_balance, make_invoice, lookup_invoice) are unaffected because they don't do per-record synchronous CPU work in a loop.

Status

Draft — currently testing this fix in production (LNbits v1.5.3, LNDRest). Will mark ready for review once verified over a sustained period.

Fix

Offload each bolt11_decode call to a worker thread via asyncio.to_thread, so the event loop can keep running while each invoice is parsed. This is the same pattern used in lnbits/lnbits#3925 for analogous synchronous websocket calls in send_nostr_dm.

# before
invoice_data = bolt11_decode(p.bolt11)

# after
invoice_data = await asyncio.to_thread(bolt11_decode, p.bolt11)

asyncio is already imported; no new dependencies.

Why this matters

Together with the other two fixes that are already in production testing, this closes the last of three event-loop-starvation paths that collectively rendered NWC unusable on busy LNDRest-backed wallets:

Fix Repo Path
IN_FLIGHT polling hot-loop lnbits/lnbits#3918 lnbits/core/services/payments.py
Blocking relay publishes lnbits/lnbits#3925 lnbits/core/services/nostr.py
This PR lnbits/nwcprovider nwcprovider/tasks.py

Alternatives considered

  • Defer decode to the client: pass raw bolt11 through and let the client decode. Smaller payload, no server CPU. But it changes the response schema (fields description, description_hash, and the invoice_data.date fallback for timestamp) and would break clients that rely on the current shape. Rejected for backward compatibility.
  • Periodic asyncio.sleep(0) yields: works, but still runs all the CPU on the event-loop thread. asyncio.to_thread is cleaner and scales better.

Related

Changes

  • tasks.py: wrap bolt11_decode in asyncio.to_thread inside _on_list_transactions

🤖 Generated with Claude Code

Closes lnbits#35

`_on_list_transactions` decodes each payment's BOLT11 invoice with the
synchronous `bolt11_decode()` inside the asyncio event loop. On wallets
with non-trivial history this blocks the event loop long enough that
NWC clients hit their reply timeout (~60s) and LNbits' other async
work (relay keep-alives, publishes, concurrent NWC requests) stalls.

Move the decode call to a worker thread via `asyncio.to_thread` so the
event loop keeps running while each invoice is parsed.

Sibling event-loop-starvation fixes:
- lnbits/lnbits#3918 (IN_FLIGHT payment polling backoff)
- lnbits/lnbits#3925 (non-blocking relay publishes in send_nostr_dm)

Together these three changes restore NWC responsiveness on busy
LNDRest-backed wallets.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BenGWeeks added a commit to BenGWeeks/lightning-piggy-mobile that referenced this pull request Apr 12, 2026
Swap recovery service:
- Parse lockup tx hex to extract vout+amount (Boltz v2 /swap/{id} doesn't
  return these fields; derive them from transaction.hex by matching the
  lockup address script).
- initEccLib so bitcoinjs-lib can parse bech32m (taproot) addresses.
- Surface progress via react-native-toast-message: info toasts when
  checking/claiming, success when recovered, error with full message when
  recovery fails. Custom renderer lets body text wrap so long Electrum
  errors (e.g. script-verify-flag-failed) are readable.

Boltz claim (work-in-progress, witness-hash mismatch debugging):
- Attempt MuSig2 key aggregation via @scure/btc-signer (MIT-licensed
  alternative to AGPL boltz-core) for Taproot internal key.
- Diagnostic logging compares computed tweaked key vs actual lockup
  output script. Currently doesn't match — internal-key derivation
  still needs alignment with Boltz v2 (their docs say "Boltz key first"
  but our sorted/claim_refund/refund_claim variants all diverge from
  on-chain output key; needs boltz-core reference comparison).

LNbits container hot-fix patches (docker/patches/):
- README + apply.sh to re-apply patches after image upgrades.
- lnbits_wallets_lndrest.py — IN_FLIGHT payment polling backoff
  (lnbits/lnbits#3918).
- lnbits_core_services_nostr.py — non-blocking relay publishes
  (lnbits/lnbits#3925).
- lnbits_extensions_nwcprovider_tasks.py — bolt11_decode in worker
  thread (lnbits/nwcprovider#37).
All three fixes target independent event-loop starvation paths that
together rendered NWC unusable on busy wallets. apply.sh requires
LNBITS_HOST to be set (no hard-coded hostname, safe for public repo).

Docs:
- TERMS.adoc: expand Taproot entry with bech32m/bc1p, add bech32m,
  P2TR, swap-in/out, lockup/claim/refund tx, preimage, key-path vs
  script-path, control block, tapleaf. Add @scure/btc-signer and
  react-native-toast-message to dependent libraries table.

Ignore:
- .claude/ runtime state directory.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@BenGWeeks BenGWeeks marked this pull request as ready for review April 15, 2026 10:17
@motorina0
Copy link
Copy Markdown
Collaborator

please drop the AI slop description

This is not a blocker operation, it runs in constant time.

invoice_data = bolt11_decode(p.bolt11)

Starting a new thread for each decode might actually degrade the overall performance.

invoice_data = await asyncio.to_thread(bolt11_decode, p.bolt11)

Is there a before/after performance test done?

@BenGWeeks
Copy link
Copy Markdown
Author

Feel free to fix how ever you guys see fit - you know your solution better than I do.

The issue was that list_transactions would timeout and fail.

BenGWeeks added a commit to BenGWeeks/nwcprovider that referenced this pull request May 7, 2026
@riccardobl
Copy link
Copy Markdown
Collaborator

riccardobl commented May 19, 2026

I see, the TL;DR is basically that you want to yield the event loop, not to make _on_list_transactions faster.
Actually makes sense. But as @motorina0 said, spamming threads is not the right solution, the proposed alternative await asyncio.sleep(0) is the right approach.

Could you please change the PR to reflect that? Or if you prefer i can commit the fix to main branch.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] list_transactions blocks asyncio event loop in synchronous bolt11_decode loop

3 participants