Make Ctrl-C work when an SSH connection stalls - #5085
Draft
Vandit1604 wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Summary
What and Why:
SSHConnectpassedcontext.Background()to bothConnectandShell(internal/command/ssh/ssh_terminal.go:99 and :118), even thoughSSHParams.Ctxholds the caller's context and the same function already uses it 26 lines earlier to fetch the certificate (:73).That matters more than it first looks.
main.go:61builds the root context withsignal.NotifyContext, which traps Ctrl-C instead of killing the process. It cancels the context and expects the code to unwind. Withcontext.Background()nothing was listening, so a stalled SSH handshake ignored every interrupt. I checked this rather than assuming it, with a small program using the same signal setup: after the first signal, five more interrupts were swallowed and the process stayed alive. The only way out is to kill flyctl from another terminal.ssh/client.gowas already written to handle cancellation. The comment at :82 says "ssh.NewClientConn doesn't take a context, so we need to handle cancelation on our end", and there is a select onctx.Done()below it. The call site made that select unreachable.Commands that reach this, directly or through
RunSSHCommand(flypg/cmd.go:67,88,115):fly postgres connect(postgres/connect.go:110)fly postgres import(postgres/import.go:193)fly postgres failover(postgres/failover.go:240,415)fly postgres config update(postgres/config_update.go:202)fly postgres events(postgres/events.go:135)fly machine destroyon a Postgres machine, which unregisters the member over SSH in the deletion hook (machine/lifecycle_hooks.go:32)fly ssh consolewas never affected. console.go:257 already passes the real context. This change brings ssh_terminal.go in line with it.How:
Both call sites now pass
p.Ctx.On its own that would trade a hang for a leak, so
Connectneeded two more lines:tcpConn.Close()on thectx.Done()path (ssh/client.go:100). Closing the socket is what unblocksssh.NewClientConn, which has no other way to learn the caller gave up. Before this the socket was a local variable that nothing ever closed.respChis now buffered (ssh/client.go:80). Otherwise the handshake goroutine blocks forever sending a result nobody is left to read.Both are needed. Closing the socket alone only moves the leak to the channel send.
I also removed a
forthat wrapped a select whose every branch returns, so it could never run twice. That is why the diff looks bigger than it is. The real change is 4 lines and the rest is indentation.One thing worth flagging: on the cancel path the socket is closed twice, once by us and once by
ssh.NewClientConnwhen the handshake fails as a result (x/crypto/ssh v0.54.0, ssh/client.go:84). That is safe. The secondClosereturns "use of closed network connection", and neither caller checks it.Testing:
TestConnectStopsWhenTheContextIsCanceledin ssh/client_test.go. The far end is anet.Pipethat never speaks SSH, so the handshake blocks until something closes the socket under it.It fails on master:
go build ./...,go vet,gofmt,go test -race ./ssh/...and golangci-lint v2.11.3 are all clean.What I did not test: the behaviour against a real unreachable machine. I have no app to break. The evidence is the code path and the unit test.
Documentation