Skip to content

perf(address): incremental Calls list and upsert-by-hash persistence - #96

Merged
amanusk merged 1 commit into
masterfrom
perf/address-calls-incremental
Sep 14, 2026
Merged

amanusk merged 1 commit into
masterfrom
perf/address-calls-incremental

Conversation

@amanusk

@amanusk amanusk commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Summary

Closes #92.

Per WS event on a viewed contract, the Calls tab did O(n) work and a full SQLite rewrite of the address's rows: a linear dedup scan, push + sort of the whole list, a linear lookup per enriched row, then a clone of the entire list into PersistAddressCalls, which deleted and re-inserted every row. Gap rows were located with a linear position() per frame.

Changes

  • Indexed, sorted Calls list (AddressInfoState): a tx_hash → block map gives O(1) dedup; insert_call uses partition_point into the block-descending order (same placement as the old stable sort of a push); find_call_mut goes hash → block → binary search into that block's few rows. Gap render positions for both tabs binary-search the sorted key (block for calls, nonce for txs) instead of position().
  • Upsert-by-hash persistence: address_calls is re-keyed by (address, tx_hash) with a block_number column. save_address_calls now upserts only the rows it is given. The reducer sends just the rows a batch created or changed.
  • Shape-keyed migration: detected via pragma_table_info, not user_version, so it runs on any existing DB regardless of which version bumps it has seen. Old rows are copied in one transaction with json_extract, invalid JSON skipped.
  • One merge helper (merge_call_into) replaces the duplicated field-upgrade logic in deduplicate_contract_calls and the enrichment reducer. Enrichment now also fills inner_targets on streamed rows, so the Prv column no longer needs a restart.
  • Network-side saves all go through persist_calls_merged (field-merge with cached rows, then upsert the touched hashes). Two non-Dune call builders previously never saved their rows and relied on the reducer's full-list write; they persist now. fetch_more_address_txs no longer overwrites the address's other rows.
  • Stubs persist on insert, so a failed enrichment still leaves a row for the load-time repair pass.

Verification

  • cargo test: 296 lib tests pass, including four new ones (insert order + dedup, hash lookup, gap positions for both tabs, migration + upsert against an old-shape table). cargo clippy --all-targets adds no warnings on the touched lines. Release build passes.
  • Smoke run with a scratch config whose cache.db held a copy of a real address_calls table in the old shape, then streaming the public STRK token contract:
Check Result
Re-key migration, 259,659 rows 1.46 s, one transaction
Writes during 90 s of streaming 133 upserts of 1 row + one 355-row Dune batch
Final run, in-memory vs DB Calls count 3194 vs 3194
Search paste visible while streaming 12 to 38 ms
50k-row microbench (call_insert_and_lookup_timing, debug) insert 2.2 ms → 137 µs, lookup 1.4 µs

Notes

  • First start after upgrade migrates the cache once (about 1.5 s per 260k rows). Nothing else changes for the user.
  • Behaviour deltas to be aware of: enrichment joins endpoint names instead of only filling an empty one (same rule dedup already used), and deduplicate_contract_calls now also replaces a "?" status with a known one.

🤖 Generated with Claude Code

Per WS event on a viewed contract, the Calls tab did O(n) work and a
full SQLite rewrite of the address's rows:

- dedup via a linear scan, then push + sort of the whole list
- enrichment looked each row up linearly, then cloned the entire list
  into PersistAddressCalls, which deleted and re-inserted every row
- gap rows were located with a linear position() per frame

Now the list keeps a tx_hash -> block index: inserts use partition_point
into the block-descending order, lookups are hash -> block -> binary
search, and gap positions binary-search the sorted key (block for calls,
nonce for txs).

address_calls is re-keyed by (address, tx_hash) with a block_number
column; saves upsert only the touched rows. The migration is keyed on
the table shape, not user_version, and re-keyed 260k rows in 1.5 s.
Network-side saves go through one merge-then-upsert helper so a sparser
row never overwrites a richer cached one, and the two non-Dune call
builders now persist their rows too instead of relying on the reducer's
full-list write. Enrichment and dedup share merge_call_into, which also
fills inner_targets on streamed rows. WS stubs are persisted on insert
so a failed enrichment still leaves a row for the load-time repair pass.

Measured at 50k rows (debug): insert 2.2 ms -> 137 us, hash lookup 1.4 us.
Streaming STRK for 90 s: 133 single-row upserts and one 355-row batch
instead of a ~2,500-row rewrite per event.

Closes #92

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@amanusk amanusk added performance Performance: latency, throughput, resource usage snbeat Main snbeat TUI binary labels Sep 14, 2026
@amanusk
amanusk merged commit 106cf3d into master Sep 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Performance: latency, throughput, resource usage snbeat Main snbeat TUI binary

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(address): per-WS-event work on large Calls lists is O(n): full re-sort, linear dedup, and a full SQLite rewrite per enrichment

1 participant