[Bug]: Direct reseed from master can get stuck in direct state after restore failure

[caffeinated92]

Component

Backup/Restore

Bug Description

A direct reseed from master (RejoinDirectDump / direct mysqldump-based reseed) can get stuck in reseeding state when the restore side fails.

The dump keeps running from the source, but the destination restore has already hit an SQL error. After that, the reseed remains stuck in direct instead of failing cleanly and clearing the reseed state.

Steps to Reproduce

1. Trigger a direct reseed from master.
2. Ensure the restore stream contains a statement that fails on the destination, for example INSTALL PLUGIN DISKS SONAME 'disks.so' when the plugin is already installed.
3. Observe that the destination restore logs an SQL error while the source dump continues streaming.
4. Observe that the server remains stuck in reseeding state direct.

Expected Behavior

If the destination restore fails, the direct reseed should fail promptly, stop/cancel the source dump, clear IsReseeding="direct", and surface the failure clearly in runtime/job state.

Actual Behavior

The destination restore reports an SQL error, the source mysqldump continues running, and the direct reseed remains stuck in direct. The reseed does not unwind cleanly, so the server appears permanently stuck in reseeding.

Relevant Logs

2026/08/10 07:53:07	DEBUG	[backup] [db1] ERROR 1968 (HY000) at line 44307111: Plugin 'DISKS' already installed
2026/08/10 07:53:07	DEBUG	[backup] [db1] --------------
2026/08/10 07:53:07	DEBUG	[backup] [db1] INSTALL PLUGIN DISKS SONAME 'disks.so'
2026/08/10 07:53:07	DEBUG	[backup] [db1] --------------
2026/08/10 07:53:06	DEBUG	[backup] [db2] -- Retrieving rows...
2026/08/10 07:53:06	DEBUG	[backup] [db2] -- Sending SELECT query...

Replication Manager Version

Unknown from the incident report; reproduced from the current working tree investigation.

Database System

MariaDB

Database Version

Unknown

Replication Mode

Unknown

Proxy Type

Unknown

Operating System

Unknown

Configuration

# Relevant config values were not included in the incident report.
# The issue affects direct reseed from master / RejoinDirectDump.

Additional Context

This appears to be a pipeline teardown / wait-order bug in the direct reseed path:

• cluster/srv_rejoin.go -> RejoinDirectDump() arms the async reseed and sets IsReseeding="direct".
• cluster/srv_job_backup.go -> JobRejoinMysqldumpFromSource(...) runs mysqldump | mysql.
• When the destination restore fails first, the source dump can keep running and the function may never unwind cleanly.
• Because the function does not return, the deferred SetInReseedBackup("") never runs, leaving the server stuck in direct.

This issue is specifically about the direct reseed getting stuck after a restore failure, not about whether INSTALL PLUGIN ... already installed should be tolerated semantically.

[svaroqui]

Process-level fallout to cover in the fix: the stuck reseed also leaks a zombie client + a runaway mysqldump.

On a live monitor, hours after this exact restore-error scenario, the process table showed both:

• a mysqldump --single-transaction --all-databases … still running, and
• a <defunct> (zombie) mysql/mariadb client.

So "stuck in direct" has two concrete side effects worth handling explicitly, not just the state flag:

1. Unreaped client → zombie. In JobRejoinMysqldumpFromSource (cluster/srv_job_backup.go) the two ends are waited sequentially — dumpCmd.Wait() then clientCmd.Wait() — and dumpCmd had no cancel context. When the client errors and exits first, it stops draining the pipe, so mysqldump keeps running and dumpCmd.Wait() never returns → clientCmd.Wait() is never reached → the exited client is never reaped. If every failed reseed leaks one zombie, PID/handle exhaustion can eventually stop the monitor from forking — i.e. it degrades monitoring itself.
2. Runaway dump on the source. The orphaned --single-transaction --all-databases dump holds a long-open consistent read on the source DB → undo/history-list growth and sustained load on the monitored server. A failed reseed should never keep an hours-long transaction open against a production source.

This is already handled by 078a15eae ("fix(cluster): unstick direct reseed failures", branch reseed-from-master): it moves both commands to exec.CommandContext(ctx, …) with a shared cancel() (either side dying SIGKILLs the other) and waits on them concurrently (dumpErrCh/clientErrCh) instead of sequentially — which both reaps the client and kills the dump.

Flagging so the merge/verification explicitly confirms no leftover <defunct> client and no still-running mysqldump after an induced restore failure, in addition to the reseed state clearing. Fix still needs to reach develop.