Skip to content

Make Ctrl-C work when an SSH connection stalls - #5085

Draft
Vandit1604 wants to merge 1 commit into
superfly:masterfrom
Vandit1604:ssh-connect-context
Draft

Make Ctrl-C work when an SSH connection stalls#5085
Vandit1604 wants to merge 1 commit into
superfly:masterfrom
Vandit1604:ssh-connect-context

Conversation

@Vandit1604

Copy link
Copy Markdown

Change Summary

What and Why:

SSHConnect passed context.Background() to both Connect and Shell (internal/command/ssh/ssh_terminal.go:99 and :118), even though SSHParams.Ctx holds the caller's context and the same function already uses it 26 lines earlier to fetch the certificate (:73).

That matters more than it first looks. main.go:61 builds the root context with signal.NotifyContext, which traps Ctrl-C instead of killing the process. It cancels the context and expects the code to unwind. With context.Background() nothing was listening, so a stalled SSH handshake ignored every interrupt. I checked this rather than assuming it, with a small program using the same signal setup: after the first signal, five more interrupts were swallowed and the process stayed alive. The only way out is to kill flyctl from another terminal.

ssh/client.go was already written to handle cancellation. The comment at :82 says "ssh.NewClientConn doesn't take a context, so we need to handle cancelation on our end", and there is a select on ctx.Done() below it. The call site made that select unreachable.

Commands that reach this, directly or through RunSSHCommand (flypg/cmd.go:67,88,115):

  • fly postgres connect (postgres/connect.go:110)
  • fly postgres import (postgres/import.go:193)
  • fly postgres failover (postgres/failover.go:240,415)
  • fly postgres config update (postgres/config_update.go:202)
  • fly postgres events (postgres/events.go:135)
  • fly machine destroy on a Postgres machine, which unregisters the member over SSH in the deletion hook (machine/lifecycle_hooks.go:32)

fly ssh console was never affected. console.go:257 already passes the real context. This change brings ssh_terminal.go in line with it.

How:

Both call sites now pass p.Ctx.

On its own that would trade a hang for a leak, so Connect needed two more lines:

  • tcpConn.Close() on the ctx.Done() path (ssh/client.go:100). Closing the socket is what unblocks ssh.NewClientConn, which has no other way to learn the caller gave up. Before this the socket was a local variable that nothing ever closed.
  • respCh is now buffered (ssh/client.go:80). Otherwise the handshake goroutine blocks forever sending a result nobody is left to read.

Both are needed. Closing the socket alone only moves the leak to the channel send.

I also removed a for that wrapped a select whose every branch returns, so it could never run twice. That is why the diff looks bigger than it is. The real change is 4 lines and the rest is indentation.

One thing worth flagging: on the cancel path the socket is closed twice, once by us and once by ssh.NewClientConn when the handshake fails as a result (x/crypto/ssh v0.54.0, ssh/client.go:84). That is safe. The second Close returns "use of closed network connection", and neither caller checks it.

Testing:

TestConnectStopsWhenTheContextIsCanceled in ssh/client_test.go. The far end is a net.Pipe that never speaks SSH, so the handshake blocks until something closes the socket under it.

It fails on master:

--- FAIL: TestConnectStopsWhenTheContextIsCanceled
    client_test.go:80: expected the socket to be closed, it still reads

go build ./..., go vet, gofmt, go test -race ./ssh/... and golangci-lint v2.11.3 are all clean.

What I did not test: the behaviour against a real unreachable machine. I have no app to break. The evidence is the code path and the unit test.


Documentation

  • Fresh Produce
  • In superfly/docs, or asked for help from docs team
  • n/a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant