Skip to content

Access node hangs when shutting down#8666

Description

@vishalchangrani

馃悶 Bug Report

Reported by QuickNode:

We're seeing a shutdown hang on our mainnet access node gcr.io/flow-container-registry/access:v0.51.0-rc.1. The process never exits and has to be killed or until graceful shutdown timeout.

Looking at the logs, ~25 components report component shutdown complete almost immediately, but two never do: RPC engine and unsecure grpc server.

We sent SIGQUIT to get a goroutine dump. It shows GracefulStop (grpcserver/server.go:110) blocked. The logs are attached.

What is the severity of this bug?

Should Have

Critical - Urgent: We can't do anything if this isn't actioned immediately (product doesn't function without this, it's blocking us or users, or it resolves a high severity security issue). Whole team should drop what they're doing and work on this.

Critical: We can't do anything if this isn't actioned immediately (product doesn't function without this, it's blocking us or users, or it resolves a high severity security issue). One person should look at this right now.

Important: * We have to do this before we ship, but it can wait until the next sprint (product or feature won't function without it, but it's not blocking us or users right now). Team should do this in the next sprint.

Should have: * It would be better if we do this before we ship, but it's OK if we don't (product functions without this, but it's a better user experience). Consider adding to a future sprint.

Could have: It really doesn't matter if we do this (product functions without this, impact to user is minimal).

Expected behaviour

Access node should shutdown cleanly

Metadata

Metadata

Labels

BugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions