高可用3节点，运行一段时间后有一台报错no rows in result set,mysql: getstamp failed

[zhongshao666]

报错时ccr.syncers也只能看到两个节点了

[w41ter]

能否提供第一次报错的日志，我看看什么原因引起的。

[zhongshao666]

@w41ter

[zhongshao666]

[2025-11-10 14:06:19.916] INFO ccr start, version: 3.0.6-rc03:a836ab1 line=ccr_syncer/ccr_syncer.go:142
[2025-11-10 14:06:19.947] INFO Server listening on :9190 line=service/http_service.go:1111
[2025-11-10 14:06:20.972] INFO update jobs [] line=ccr/checker.go:135
[2025-11-10 14:07:20.952] INFO GOROUTINE: Total = 19 line=ccr_syncer/monitor.go:45
[2025-11-10 14:07:20.953] INFO MEMORY STATS: Alloc = 5 MiB, TotalAlloc = 8 MiB, Sys = 14 MiB, NumGC = 4, LiveObjects = 17248 line=ccr_syncer/monitor.go:55
[2025-11-10 14:07:20.953] INFO JOB STATS: Total = 0, Running = 0, DBSync = 0, TableSync = 0 line=ccr_syncer/monitor.go:84
[2025-11-10 14:07:20.953] INFO JOB STATUS: FullSync = 0, PartialSync = 0, IncrementalSync = 0 line=ccr_syncer/monitor.go:86
[2025-11-10 14:08:20.955] INFO GOROUTINE: Total = 19 line=ccr_syncer/monitor.go:45
[2025-11-10 14:08:20.965] INFO MEMORY STATS: Alloc = 8 MiB, TotalAlloc = 10 MiB, Sys = 18 MiB, NumGC = 4, LiveObjects = 30979 line=ccr_syncer/monitor.go:55
[2025-11-10 14:08:20.966] INFO JOB STATS: Total = 0, Running = 0, DBSync = 0, TableSync = 0 line=ccr_syncer/monitor.go:84
[2025-11-10 14:08:20.966] INFO JOB STATUS: FullSync = 0, PartialSync = 0, IncrementalSync = 0 line=ccr_syncer/monitor.go:86
[2025-11-10 14:09:20.953] INFO GOROUTINE: Total = 19 line=ccr_syncer/monitor.go:45
[2025-11-10 14:09:20.953] INFO MEMORY STATS: Alloc = 6 MiB, TotalAlloc = 12 MiB, Sys = 22 MiB, NumGC = 5, LiveObjects = 23840 line=ccr_syncer/monitor.go:55
[2025-11-10 14:09:20.953] INFO JOB STATS: Total = 0, Running = 0, DBSync = 0, TableSync = 0 line=ccr_syncer/monitor.go:84
[2025-11-10 14:09:20.953] INFO JOB STATUS: FullSync = 0, PartialSync = 0, IncrementalSync = 0 line=ccr_syncer/monitor.go:86
[2025-11-10 14:10:20.958] INFO GOROUTINE: Total = 19 line=ccr_syncer/monitor.go:45
[2025-11-10 14:10:20.981] INFO MEMORY STATS: Alloc = 8 MiB, TotalAlloc = 14 MiB, Sys = 22 MiB, NumGC = 5, LiveObjects = 36653 line=ccr_syncer/monitor.go:55
[2025-11-10 14:10:20.982] INFO JOB STATS: Total = 0, Running = 0, DBSync = 0, TableSync = 0 line=ccr_syncer/monitor.go:84
[2025-11-10 14:10:20.982] INFO JOB STATUS: FullSync = 0, PartialSync = 0, IncrementalSync = 0 line=ccr_syncer/monitor.go:86
[2025-11-10 14:11:20.957] INFO GOROUTINE: Total = 19 line=ccr_syncer/monitor.go:45
[2025-11-10 14:11:20.967] INFO MEMORY STATS: Alloc = 10 MiB, TotalAlloc = 17 MiB, Sys = 22 MiB, NumGC = 6, LiveObjects = 25507 line=ccr_syncer/monitor.go:55
[2025-11-10 14:11:20.967] INFO JOB STATS: Total = 0, Running = 0, DBSync = 0, TableSync = 0 line=ccr_syncer/monitor.go:84
[2025-11-10 14:11:20.967] INFO JOB STATUS: FullSync = 0, PartialSync = 0, IncrementalSync = 0 line=ccr_syncer/monitor.go:86
[2025-11-10 14:12:20.952] INFO GOROUTINE: Total = 19 line=ccr_syncer/monitor.go:45
[2025-11-10 14:12:20.952] INFO MEMORY STATS: Alloc = 12 MiB, TotalAlloc = 20 MiB, Sys = 26 MiB, NumGC = 6, LiveObjects = 37943 line=ccr_syncer/monitor.go:55
[2025-11-10 14:12:20.953] INFO JOB STATS: Total = 0, Running = 0, DBSync = 0, TableSync = 0 line=ccr_syncer/monitor.go:84
[2025-11-10 14:12:20.953] INFO JOB STATUS: FullSync = 0, PartialSync = 0, IncrementalSync = 0 line=ccr_syncer/monitor.go:86
[2025-11-10 14:13:20.952] INFO GOROUTINE: Total = 19 line=ccr_syncer/monitor.go:45
[2025-11-10 14:13:20.952] INFO MEMORY STATS: Alloc = 8 MiB, TotalAlloc = 21 MiB, Sys = 26 MiB, NumGC = 7, LiveObjects = 23704 line=ccr_syncer/monitor.go:55
[2025-11-10 14:13:20.952] INFO JOB STATS: Total = 0, Running = 0, DBSync = 0, TableSync = 0 line=ccr_syncer/monitor.go:84
[2025-11-10 14:13:20.952] INFO JOB STATUS: FullSync = 0, PartialSync = 0, IncrementalSync = 0 line=ccr_syncer/monitor.go:86
[2025-11-10 14:14:20.953] INFO GOROUTINE: Total = 19 line=ccr_syncer/monitor.go:45
[2025-11-10 14:14:20.953] INFO MEMORY STATS: Alloc = 10 MiB, TotalAlloc = 23 MiB, Sys = 26 MiB, NumGC = 7, LiveObjects = 36741 line=ccr_syncer/monitor.go:55
[2025-11-10 14:14:20.953] INFO JOB STATS: Total = 0, Running = 0, DBSync = 0, TableSync = 0 line=ccr_syncer/monitor.go:84
[2025-11-10 14:14:20.953] INFO JOB STATUS: FullSync = 0, PartialSync = 0, IncrementalSync = 0 line=ccr_syncer/monitor.go:86
[2025-11-10 14:15:20.953] INFO GOROUTINE: Total = 19 line=ccr_syncer/monitor.go:45
[2025-11-10 14:15:20.953] INFO MEMORY STATS: Alloc = 7 MiB, TotalAlloc = 24 MiB, Sys = 26 MiB, NumGC = 8, LiveObjects = 23662 line=ccr_syncer/monitor.go:55
[2025-11-10 14:15:20.953] INFO JOB STATS: Total = 0, Running = 0, DBSync = 0, TableSync = 0 line=ccr_syncer/monitor.go:84
[2025-11-10 14:15:20.953] INFO JOB STATUS: FullSync = 0, PartialSync = 0, IncrementalSync = 0 line=ccr_syncer/monitor.go:86
[2025-11-10 14:16:05.290] ERROR rebalance load failed and rollback, err: sql: no rows in result set line=storage/mysql.go:250
[2025-11-10 14:16:05.383] INFO update jobs [] line=ccr/checker.go:135
[2025-11-10 14:16:05.402] ERROR checker failed, host info: 172.16.17.137:9190, err: [db] sql: no rows in result set
mysql: get stamp failed.
github.com/selectdb/ccr_syncer/pkg/storage.(*MysqlDB).GetStampAndJobs
/root/workspace/ccr-syncer/pkg/storage/mysql.go:257
github.com/selectdb/ccr_syncer/pkg/ccr.(*Checker).handleUpdate
/root/workspace/ccr-syncer/pkg/ccr/checker.go:131
github.com/selectdb/ccr_syncer/pkg/ccr.(*Checker).check
/root/workspace/ccr-syncer/pkg/ccr/checker.go:156
github.com/selectdb/ccr_syncer/pkg/ccr.(*Checker).run
/root/workspace/ccr-syncer/pkg/ccr/checker.go:206
github.com/selectdb/ccr_syncer/pkg/ccr.(*Checker).Start
/root/workspace/ccr-syncer/pkg/ccr/checker.go:188
github.com/avast/retry-go.Do
/root/go/pkg/mod/github.com/avast/retry-go@v3.0.0+incompatible/retry.go:127
main.retryWithAttempts
/root/workspace/ccr-syncer/cmd/ccr_syncer/ccr_syncer.go:124
main.main.func3
/root/workspace/ccr-syncer/cmd/ccr_syncer/ccr_syncer.go:210
runtime.goexit
/root/workspace/go/src/runtime/asm_amd64.s:1695 line=ccr/checker.go:207

[w41ter]

我分析一下看看什么原因。