Skip to content

fix(logs-source): follow the heads of upstreams that can serve logs - #398

Merged
l0gun0v merged 1 commit into
mainfrom
logs-source-no-silent-skip
Oct 2, 2026
Merged

l0gun0v merged 1 commit into
mainfrom
logs-source-no-silent-skip

Conversation

@l0gun0v

@l0gun0v l0gun0v commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

The local logs subscription source announces blocks from the heads of the upstreams that can serve them, instead of the chain's merged head, and backfills heights that head jumps over.

Seen on ethereum drpc-core: the merged head came from the faster of two upstreams, which has eth_getLogs disabled (infra-bcn #1190). At that moment the upstream with the method did not have the block yet, so 80–95% of blocks were skipped and every logs subscriber on the instance lost those events.

Spec

  • Logs head. StreamBlockUpdates follows the highest head among the upstreams that fetchBlockLogs may select — Available and with eth_getLogs, the same matchers — not the merged head. Heads are re-read on every chain state event and every 50 ms (the chain state publishes only when the merged head moves, not when such an upstream catches up). On a tie at the top height the head already followed is kept. Logs therefore lag the chain head by the lag of the best upstream with eth_getLogs; blocks it serves are no longer skipped.
  • Backfill. When a head does not link to what was announced (it jumped over heights, or its parent at an announced height has another hash), its ancestors are fetched by parentHash with eth_getBlockByHash (up to 32, from upstreams at that height) and announced oldest-first through the regular tracker logic. This also announces the new canonical block below a reorged tip, which was previously never emitted. A deeper gap or a failed fetch announces the head with the gap, as before, with a warning and backfill_failed_total.
  • Unchanged. fetchBlockLogs (3 attempts down the rating list, then the block is skipped), reorg handling, fan-out. The source still terminates when the chain loses LogsCap; it now also checks that every second, since blocks stop arriving once no upstream has eth_getLogs.
  • Metrics. New nodecore_logs_source_backfill_failed_total{chain} and nodecore_logs_source_head_lag_blocks{chain} (merged head minus logs head).
  • Compatibility. No config changes. Only the local logs source is affected; newHeads, pending-tx sources and node-backed subscriptions are untouched.

Changes

File Change
internal/upstreams/flow/subengine/blockupdates.go StreamBlockUpdates follows bestHead of eligible upstreams (event + 50 ms recheck); needsParent, advanceWithAncestors, BlockResolver; new metrics
internal/upstreams/flow/logs_source.go canServeLogs (status + eth_getLogs matchers), blockByHashResolver, periodic LogsCap check
internal/upstreams/flow/subengine/blockupdates_test.go backfill: forward gap, reorged-in parent, reorg behind a gap, no fetch when linked, resolve error, wrong parent, gap too deep; bestHead
internal/upstreams/flow/logs_source_internal_test.go head from an upstream without eth_getLogs (end to end), backfill and its failure, reorg with the reorged-in parent, termination when eth_getLogs is gone
docs/nodecore/08-prometheus-metrics.md, docs/nodecore/13-subscriptions.md logs head, backfill, new metrics

@l0gun0v l0gun0v changed the title fix(logs-source): never skip a block silently fix(logs-source): wait for an upstream that can serve the block Oct 2, 2026
@l0gun0v
l0gun0v marked this pull request as ready for review October 2, 2026 09:54
The logs source announced blocks from the chain's merged head, which may come
from an upstream without eth_getLogs. At that moment no upstream with the
method had the block yet, so its logs were skipped for every subscriber
(80-95% of ethereum blocks on drpc-core after eth_getLogs was disabled on the
faster of two upstreams).

- The block-update stream follows the highest head among the upstreams that
  fetchBlockLogs may select (available, with eth_getLogs), re-read on every
  chain state event and every 50 ms.
- Heights that head jumps over, and the new chain after a reorg, are fetched
  by parent hash (eth_getBlockByHash, up to 32 blocks) and announced in order;
  a deeper gap or a failed fetch announces the head with the gap.
- The source checks LogsCap every second as well, since blocks stop arriving
  once no upstream has eth_getLogs.
- Metrics: logs_source_backfill_failed_total, logs_source_head_lag_blocks.
@l0gun0v
l0gun0v force-pushed the logs-source-no-silent-skip branch from ef40f1c to 645f727 Compare October 2, 2026 10:27
@l0gun0v l0gun0v changed the title fix(logs-source): wait for an upstream that can serve the block fix(logs-source): follow the heads of upstreams that can serve logs Oct 2, 2026
@l0gun0v
l0gun0v merged commit 3da5aa0 into main Oct 2, 2026
5 checks passed
@l0gun0v
l0gun0v deleted the logs-source-no-silent-skip branch October 2, 2026 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant