[v0.51] [AN] Backport graceful shutdown fix (#8662, #8664)

[vishalchangrani]

Backports the graceful shutdown fix for the Access Node gRPC server from master to v0.51, so we can cut a v0.51 release with the fix.

Cherry-picked commits, in order:

From #8662 (closes #8666):

• 4ad464c fix the graceful shutdown by adding a timeout (Leo)
• 71fde5c make tidy (Leo)
• b090d15 cancel streaming RPCs on shutdown via interceptor
• b2d22af drop unnecessary godoc and whitespace changes
• 3a4c15e address review comments (Leo)

From #8664:

• 2b2caad Remove entirely blocking on gracefulDone channel readiness (Ardit)
• c05db3c Update comment regarding time.Sleep on Stream() (Ardit)
• 5402e2b Fix linting issue around comment (Ardit)

No merge conflicts — all cherry-picks applied cleanly. module/grpcserver/ matches master exactly after the port (verified via git diff origin/master -- module/grpcserver/). go build, go vet, and go test ./module/grpcserver/... all pass.

Nothing else from master is included beyond the commits in these two PRs.

Suggested labels: Bugfix, S-Access — please confirm.

Produced in collaboration with claude.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 049f5e86-6d45-4251-980d-d484be638d9c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 70.45455% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
module/grpcserver/server_builder.go	0.00%	8 Missing ⚠️
module/grpcserver/interceptor_shutdown.go	80.00%	2 Missing and 1 partial ⚠️
module/grpcserver/server.go	90.47%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[vishalchangrani]

End-to-end validation on foundation testnet AN

Ran a before/after test against access-001.devnet54.nodes.onflow.org:9000 (foundation testnet AN) using a small probe that opens 3 concurrent SubscribeBlockDigestsFromLatest gRPC streams and measures how long they take to close after systemctl stop on the AN.

Fix under test: PR #8662 + PR #8664 ported to v0.51 via PR #8676 (tag v0.51.0-graceful-an-shutdown).

Before (unfixed build)

Client side — streams did not close until Docker's 10s stop grace expired and the container was SIGKILLed:

elapsed from operator-signalled shutdown to first close: 2.98s
                            (10.24s from last block digest received)
stream 0: closed with error: rpc error: code = Internal desc = stream
          terminated by RST_STREAM with error code: NO_ERROR

The RST_STREAM NO_ERROR is HTTP/2 transport-level tear-down — i.e. the AN process was killed, not gracefully shut down.

Server side — matches the original bug report verbatim:

22:07:25  libp2p-node: component shutdown complete
22:07:30  admin: failed to shutdown http server gracefully; forcing close
          (no shutdown-complete log for `unsecure grpc server` or `RPC engine`)
22:07:35  systemd: flow-access.service: Main process exited, code=exited, status=137/n/a
          systemd: flow-access.service: Failed with result 'exit-code'

status=137 = SIGKILL. Service exited as Failed.

After (with fix, tag v0.51.0-graceful-an-shutdown)

Client side:

elapsed from last block digest to first close: 60.518ms
stream 0: closed with error: rpc error: code = Canceled desc = handle
          subscription error: stream encountered an error: client
          disconnected: context canceled

context canceled is a clean application-level cancel from ShutdownStreamInterceptor — the stream's context was cancelled the moment the node's SignalerContext was cancelled. The 5s force-stop backstop was not needed.

Server side:

23:03:55.301  blockstore closed
23:03:55.466  database has been closed
23:03:55.466  node shutdown complete
              systemd: flow-access.service: Deactivated successfully.
              systemd: Stopped flow-access.service.

Clean self-reported node shutdown complete, systemd reports Deactivated successfully, no SIGKILL, no Failed.

Summary

	Before	After
Stream close latency	~10s	~60ms
Stream close cause	Docker SIGKILL → RST_STREAM	Interceptor cancel → context canceled
node shutdown complete log	never emitted	emitted
systemd result	Failed, status=137	Deactivated successfully

Comment produced in collaboration with claude.

[m-Peter]

LGTM!

[vishalchangrani]

Mainnet validation

Repeated the before/after test against the foundation mainnet AN access-001.mainnet28.nodes.onflow.org:9000. Same probe, same procedure as the testnet run above.

Before (v0.51.0-rc.1 — the exact build QN reported the bug against)

Server side:

16:47:10  libp2p-node: component shutdown complete
16:47:15  admin: failed to shutdown http server gracefully; forcing close
          (no shutdown-complete log for `unsecure grpc server` or `RPC engine`)
16:47:21  systemd: flow-access.service: Main process exited, code=exited, status=137/n/a
          systemd: flow-access.service: Failed with result 'exit-code'
          systemd: flow-access.service: Consumed 26.553s CPU time.

status=137 = SIGKILL by Docker's stop-grace expiry. Service exited as Failed.

Client side: streams closed within ~250ms via RST_STREAM NO_ERROR — but this was the container's network being torn down at a lower layer than the AN's gRPC server, not the AN cleanly closing my streams. On mainnet the client-side view is muddied by the container network dropping first, so the systemd result is the definitive signal here.

After (v0.51.0-graceful-an-shutdown)

Server side:

16:51:07.921  blockstore closed
16:51:08.101  database has been closed
16:51:08.101  node shutdown complete
              systemd: flow-access.service: Deactivated successfully.
              systemd: Stopped flow-access.service.

node shutdown complete emitted, systemd reports Deactivated successfully, no SIGKILL, no Failed. Total server-side shutdown: <1s (blockstore closed and node shutdown complete in the same second).

Client side:

elapsed from last response to first close: 71.4 ms
stream 0: closed with error: rpc error: code = Canceled desc = handle
          subscription error: stream encountered an error: client
          disconnected: context canceled

context canceled — clean application-level cancel from the ShutdownStreamInterceptor (the fix's interceptor path). The 5s force-stop backstop was not needed.

Summary

	Before (v0.51.0-rc.1)	After (v0.51.0-graceful-an-shutdown)
Client-side close cause	RST_STREAM (container net drop)	context canceled (interceptor)
Client-side latency	~250ms	71ms
Server: node shutdown complete log	never emitted	emitted
Server: shutdown duration	~10s (bounded by Docker SIGKILL)	<1s
systemd result	Failed, status=137	Deactivated successfully
Docker SIGKILL required?	yes	no

Fix is confirmed working on both foundation testnet AN (access-001.devnet54) and foundation mainnet AN (access-001.mainnet28) under real network traffic. Safe to roll out to external operators.

Comment produced in collaboration with claude.