zero reconciliation loop

[bputt-e]

We're having intermittent connectivity with the zero and we believe it's due to the recent improvements where the zero will update its internal host name based on the "my" flag, might be related to #9680

Logs

# zero seeing this every few secs
raft.go:801] Zero address reconciliation complete: all addresses up to date

# alpha logs, at least the 3rd alpha seems to be having connectivity issues
groups.go:735] Found connection to leader: fqdn.mydomain.com:5080
groups.go:689] No healthy zero leader found. Trying to find a Zero leader...
groups.go:735] Found connection to leader: fqdn.mydomain.com:5080

Expected behavior

I expect the zero to not consistently produce the log message raft.go:801] Zero address reconciliation complete: all addresses up to date and for the 3rd alpha to have a stable connection

Environment

• OS: RHEL 9.8
• Version 25.3.7

Additional context

We're running dgraph via podman
Configuration: 1 zero, 3 alphas(3 different groups) on the same instance, using the same local ip address and just using different ports

Configs:

# zero
my: "fqdn.mydomain.com:5080"
bindall: true
replicas: 1
cid: xxx

# alphas[1-3], only diff is increasing my port by 1 and port_offset by 1
my: "fqdn.mydomain.com:7080"
zero: "fqdn.mydomain.com:5080"
bindall: true
port_offset: 0

[shiva-istari]

Hi @bputt-e , Thanks for the detailed report. I looked into this the reconciliation loop isn't causing your connectivity issue, and the log messages you're seeing are debug-level output. Details below.

About raft.go:801] Zero address reconciliation complete: all addresses up to date

This message is logged at verbosity level 2 (glog.V(2)), so it only appears because your deployment is running with --v=2. At the default verbosity it never prints.

It's emitted by a periodic loop (every 10s) introduced in #9680 that checks whether the addresses stored in Zero's MembershipState still match the live --my / transport addresses, and proposes a Raft ConfChangeUpdateNode if they've drifted (e.g., after a restart with a changed --my flag). The message you're seeing is the no-op path: it means the loop ran, found every address correct, and proposed nothing. It doesn't touch Raft state or any connections in that case, so it cannot cause connection flaps seeing it every ~10s at -v=2 is expected and is actually confirmation that your Zero addresses are stable.

About the Alpha 3 logs

groups.go:689] No healthy Zero leader found. Trying to find a Zero leader...
groups.go:735] Found connection to leader: fqdn.mydomain.com:5080

These are also verbosity-level-1 messages (glog.V(1)) from Alpha's long-standing Zero-leader retry loop — they predate #9680 and are unrelated to the reconciliation change. They indicate that Alpha 3's gRPC connection pool to fqdn.mydomain.com:5080 intermittently failed its health check and was immediately re-established. Since the Zero-side loop confirms addresses never changed, this points to a transient network-level issue between that container and Zero rather than anything in the membership/address logic.

To dig into the Alpha 3 instability, could you share:

• a longer -v=2 log window from alpha3 and zero around one of the flaps, including any gRPC transport errors
• whether alphas 1 and 2 ever log the same No healthy Zero leader found / Found connection to leader pair
• how frequently the flap occurs, and whether it correlates with load or anything else on the host