tool: add full-scan EVM logical digest#3611
Conversation
Add an `evm-logical-digest` seidb operation for comparing EVM state across FlatKV and memIAVL at the same height. The command normalizes both backends into FlatKV physical keys, strips height-dependent value metadata, reports per-bucket `bucket_digest` values, and emits one `FINAL_DIGEST` line for backend comparison.
PR SummaryLow Risk Overview The command normalizes both backends to FlatKV-style physical keys, strips height-dependent value headers, and builds order-independent per-bucket digests (XOR of per-entry Diagnostics include Reviewed by Cursor Bugbot for commit 3855d28. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is
❌ Your patch check has failed because the patch coverage (24.27%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #3611 +/- ##
==========================================
- Coverage 59.01% 57.98% -1.04%
==========================================
Files 2224 2151 -73
Lines 182699 174851 -7848
==========================================
- Hits 107823 101388 -6435
+ Misses 65197 64454 -743
+ Partials 9679 9009 -670
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b7bb0fc7a5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…sk panic isolation (#211) Adds the **full-keyspace digest gate** ([sei-chain#3611](sei-protocol/sei-chain#3611 `seidb evm-logical-digest`) as a discrete sidecar task — the per-segment boundary seal that closes the touched-key comparator's **cold-state blind spot** (a key migrated wrong and never touched again is invisible to per-block Layer 2). Plus three seams the systems-engineering review called for. No "ShadowResultProducer" abstraction — that's deferred to the 3rd producer (YAGNI). ### What's here - **`sidecar/s3/emit.go`** — one S3 emission helper (`StreamGzipNDJSON`/`StreamGzipJSON`/`StreamGzipFunc`), collapsing 3 duplicated gzip-pipe paths. Twofold integrity seal: an aws-chunked SHA-256 **wire** checksum over the compressed body (io.Pipe streaming/backpressure preserved) + an **uncompressed-payload** SHA-256 surfaced via `EmitResult` for out-of-band verification. `result_compare`/`result_export` refactored onto it (no behavior change). - **`sidecar/engine`** — `recover()` in `runTask` turns a handler panic into a failed `TaskResult` (+ `seictl_task_panics_total`) instead of crashing the sidecar. - **`sidecar/tasks/evm_logical_digest.go`** — the discrete task: shells out to `seidb` for flatkv + memiavl (`semantic` + `translator`), asserts **both** backends' opened version `== height` (fail-closed — no wrong-height false match), parses the `FINAL_DIGEST`/per-bucket contract, publishes an `EndpointDigestRecord`. `axes_proved` deliberately omits **balance** (the semantic account digest zeroes it — that axis stays the per-block comparator's job). ### Cross-review (systems-engineer + idiomatic-reviewer) — applied - **Symmetric memiavl version assertion** (the flatkv-only check left a wrong-height false-match hole if seidb clamps to the nearest snapshot). - **`recover()` inside the s3 writer goroutine** — a panic there (e.g. `MarshalJSON` over chain data) runs on a task-spawned goroutine *outside* the engine's handler recover; converted to a returned error so the upload aborts (no truncated-but-valid object) and the process survives. - **Dropped the empty-by-construction `uncompressed_sha256`** from the published record (a record can't carry the hash of its own bytes; the seal is out-of-band in the log/TaskResult). - comment-precision fixes (memory bound is the uploader part-pool, not "gzip window"; S3 checksum is per-part-composite for multipart) + a `version:`-line length guard. ### Notes - `seidb` is **shelled out to** (configurable `seidbPath`), not vendored. #3611 also needs a one-line registration fix (`EvmLogicalDigestCmd()` isn't in `seidb`'s root `AddCommand`) — flagged to the author. - Trigger is the **out-of-band task API**; no controller/CRD change (consistent with the avoided `ResultExportConfig` one-way door). - **One-way-door surfaces for confirmation before a consumer reads them:** the task-type string `"evm-logical-digest"`, the param field names, and the `EndpointDigestRecord` schema. - `GOWORK=off go build ./...` clean; `go test ./sidecar/...` green (incl. new memiavl-version-mismatch + writer-panic regression tests); `gofmt -s` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit d34312a. Configure here.
cody-littley
left a comment
cody-littley
left a comment
There was a problem hiding this comment.
It looks like this code reads all account data into memory. If we have too much account data that risks OOMing the node we're using to compare. But since this tooling is temporary and won't be needed after the migration, I think it's probably ok for now as long as we can fit mainnet account state into memory on the machine where we are doing the testing.
good point. this does not keep all evm state in memory, it does keep the account merged state keyed by unique account address so nonce/codehash can be normalized into the flatkv account buckets. storage/code/legacy are streamed into the digest. this is a good call out and we shuld watch memory usage during the pacific-1 comparision |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
2 participants
Summary
Add an
evm-logical-digestseidb operation for comparing EVM state across FlatKV and memIAVL at the same height. The command normalizes both backends into FlatKV physical keys, strips height-dependent value metadata, reports per-bucketbucket_digestvalues, and emits oneFINAL_DIGESTline for backend comparison.sei-db/tools/cmd/seidb/operations/evm_logical_digest.go: Adds theevm-logical-digestcommand with FlatKV native scanning and memIAVL snapshot scanning. FlatKV reads useRawGlobalIterator; memIAVL reads stream snapshotkvsrecords sequentially so scan order does not affect correctness.sei-db/tools/cmd/seidb/operations/evm_logical_digest.go: Enforces an order-independent bucket accumulator oversha256(len(key)||key||len(value)||value). The final digest combines account, code, storage, and marker-adjusted legacy bucket digests so a FlatKV-only migration-version row does not create a false mismatch.sei-db/tools/cmd/seidb/operations/evm_logical_digest.go: Supports semantic memIAVL normalization by default and an opt-in translator mode through--memiavl-normalization translator. Semantic mode decodes raw EVM leaves directly; translator mode routes leaves throughflatkv.ImportTranslatorto validate the migration mapping.sei-db/tools/cmd/seidb/operations/evm_logical_digest.go: Adds--inspect-bucket, prefix sharding, row listing, backend metadata details, and--find-hashsupport for isolating mismatched entries. memIAVL inspect honors the same normalization flag as the global digest, so diagnostics match the selected digest path.Test plan
sei-db/tools/cmd/seidb/operations/evm_logical_digest_test.go:TestSemanticMemiavlDigestMatchesTranslatorForCoreEVMKeysverifies semantic normalization matches translator normalization for account, code, storage, and legacy buckets, including delete-equivalent zero storage and empty code rows.sei-db/tools/cmd/seidb/operations/evm_logical_digest_test.go:TestSemanticMemiavlInspectMatchesTranslatorForCoreEVMKeysverifies inspect bucket results match translator output for all normalized buckets.sei-db/tools/cmd/seidb/operations/evm_logical_digest_test.go:TestInspectMemiavlRejectsUnknownNormalizationBeforeOpeningSnapshotguarantees invalid--memiavl-normalizationvalues are rejected before filesystem access.go test ./sei-db/tools/cmd/seidb/operations.