Repository navigation
perf(address): incremental Calls list and upsert-by-hash persistence - #96
Merged
Merged
Conversation
Per WS event on a viewed contract, the Calls tab did O(n) work and a full SQLite rewrite of the address's rows: - dedup via a linear scan, then push + sort of the whole list - enrichment looked each row up linearly, then cloned the entire list into PersistAddressCalls, which deleted and re-inserted every row - gap rows were located with a linear position() per frame Now the list keeps a tx_hash -> block index: inserts use partition_point into the block-descending order, lookups are hash -> block -> binary search, and gap positions binary-search the sorted key (block for calls, nonce for txs). address_calls is re-keyed by (address, tx_hash) with a block_number column; saves upsert only the touched rows. The migration is keyed on the table shape, not user_version, and re-keyed 260k rows in 1.5 s. Network-side saves go through one merge-then-upsert helper so a sparser row never overwrites a richer cached one, and the two non-Dune call builders now persist their rows too instead of relying on the reducer's full-list write. Enrichment and dedup share merge_call_into, which also fills inner_targets on streamed rows. WS stubs are persisted on insert so a failed enrichment still leaves a row for the load-time repair pass. Measured at 50k rows (debug): insert 2.2 ms -> 137 us, hash lookup 1.4 us. Streaming STRK for 90 s: 133 single-row upserts and one 355-row batch instead of a ~2,500-row rewrite per event. Closes #92 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #92.
Per WS event on a viewed contract, the Calls tab did O(n) work and a full SQLite rewrite of the address's rows: a linear dedup scan, push + sort of the whole list, a linear lookup per enriched row, then a clone of the entire list into
PersistAddressCalls, which deleted and re-inserted every row. Gap rows were located with a linearposition()per frame.Changes
AddressInfoState): atx_hash → blockmap gives O(1) dedup;insert_callusespartition_pointinto the block-descending order (same placement as the old stable sort of a push);find_call_mutgoes hash → block → binary search into that block's few rows. Gap render positions for both tabs binary-search the sorted key (block for calls, nonce for txs) instead ofposition().address_callsis re-keyed by(address, tx_hash)with ablock_numbercolumn.save_address_callsnow upserts only the rows it is given. The reducer sends just the rows a batch created or changed.pragma_table_info, notuser_version, so it runs on any existing DB regardless of which version bumps it has seen. Old rows are copied in one transaction withjson_extract, invalid JSON skipped.merge_call_into) replaces the duplicated field-upgrade logic indeduplicate_contract_callsand the enrichment reducer. Enrichment now also fillsinner_targetson streamed rows, so the Prv column no longer needs a restart.persist_calls_merged(field-merge with cached rows, then upsert the touched hashes). Two non-Dune call builders previously never saved their rows and relied on the reducer's full-list write; they persist now.fetch_more_address_txsno longer overwrites the address's other rows.Verification
cargo test: 296 lib tests pass, including four new ones (insert order + dedup, hash lookup, gap positions for both tabs, migration + upsert against an old-shape table).cargo clippy --all-targetsadds no warnings on the touched lines. Release build passes.cache.dbheld a copy of a realaddress_callstable in the old shape, then streaming the public STRK token contract:call_insert_and_lookup_timing, debug)Notes
deduplicate_contract_callsnow also replaces a"?"status with a known one.🤖 Generated with Claude Code