Skip to content

fix(jobs): avoid orphan SST listeners for API log fetches - #1694

Merged
caffeinated92 merged 1 commit into
developfrom
db-log-state
Aug 11, 2026
Merged

fix(jobs): avoid orphan SST listeners for API log fetches#1694
caffeinated92 merged 1 commit into
developfrom
db-log-state

Conversation

@caffeinated92

@caffeinated92 caffeinated92 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • stop API-mode DB-log scheduling from pre-opening unused SST receivers
  • keep SQL-mode receiver behavior unchanged
  • add regression coverage for API-mode no-preopen and SQL-mode compatibility

Problem

In API mode, scheduled DB-log tasks were opening an SST receiver during scheduling, but dbjobs later requested a fresh receiver through receive-task/{taskname} and used that one instead. The first listener was left idle until its accept deadline expired, producing delayed accept tcp ... i/o timeout log noise.

Fix

For the four scheduler DB-log paths (errorlog, auditlog, sqlerrorlog, slowquery), return early in API mode and queue the task with port "0" instead of pre-opening an SST receiver. SQL mode still opens the receiver up front.

Testing

  • go test ./cluster -run 'TestJobBackupDBLog_'

Related to #1693

@claude

claude Bot commented Aug 11, 2026

Copy link
Copy Markdown

Review

Traced the fix end-to-end against the API-mode dispatch path (jobInsertTask in cluster/srv_job.go) and the on-demand receiver (handlerMuxServerReceiveTask in server/api_database.go), and it holds up:

  • In API mode, jobInsertTask's if cluster.Conf.SchedulerJobsMode == "api" branch never reads the port argument at all — it only claims the task slot and writes a cookie. So swapping the real SSTRunReceiverToDBLogFile(...) call for a literal "0" is behaviorally inert from JobInsertTask's perspective; nothing downstream persists or depends on that value in API mode.
  • The real receiver is opened later, on demand, by handlerMuxServerReceiveTask, which for errorlog/auditlog/sqlerrorlog/slowquery calls the same SSTRunReceiverToDBLogFile with the same DBLogKindFromTaskName mapping — so API-mode and scheduler-mode still land in the same canonical log file. This confirms the orphaned first listener really was pure waste (the accept-deadline-timeout noise described in the linked issue), not a listener anything relied on.
  • All four touched task names (errorlog, slowquery, sqlerrorlog, auditlog) are registered as TaskCapRemoteOnly in config/config.go, so IsRemoteTask correctly routes them through the cookie-write path both before and after this change — no behavior change there.
  • SQL mode is untouched (the early return is gated strictly behind SchedulerJobsMode == "api"), matching the PR description that SQL-mode's polling-based dbjobs script still needs the pre-opened receiver's port written into the jobs table.

Tests: TestJobBackupDBLog_APIMode_DoesNotPreOpenReceiver and TestJobBackupDBLog_SQLMode_StillOpensReceiver are solid regression coverage — they snapshot the real SSTs.SSTconnections map before/after, assert on the diff (not "whatever's in the map now," which avoids cross-test pollution), and use a locked poll-with-timeout to wait for the listener's own deferred cleanup before asserting, rather than racing it. The SstAvailablePorts nil-map footgun in the SQL-mode test setup (SSTSenderFreePort writes into it during cleanup) is called out and handled correctly. I traced SSTGetSenderPort/SSTRunReceiverToFile/tcp_con_handle_to_file to confirm the map keying and cleanup assumptions the tests make are accurate for both the DBLogRotate and non-rotate code paths.

I wasn't able to actually execute go test in this sandboxed session (shell commands beyond read-only inspection required approval that wasn't available here), so this review is based on static tracing of the full call graph rather than an observed test run — the author's own go test ./cluster -run 'TestJobBackupDBLog_' in CI is the authoritative check.

Minor nit (non-blocking): the four call sites repeat an identical 3-4 line comment + if cluster.Conf.SchedulerJobsMode == "api" { return server.JobInsertTask(task, "0", cluster.Conf.MonitorAddress) } block verbatim. Given all four already share the task/early-return shape, this could be a small helper (e.g. server.queueAPIModeLogTask(task string) (int64, error)) to avoid four copies drifting out of sync if the API-mode dispatch logic ever changes. Purely stylistic — I would not block on it.

No security, performance, or test-coverage concerns beyond the above. The fix is narrowly scoped, matches the stated problem, and the new tests would have caught the regression this PR fixes.

Verdict: Approve.

@caffeinated92
caffeinated92 merged commit ac11e35 into develop Aug 11, 2026
1 check passed
@caffeinated92
caffeinated92 deleted the db-log-state branch August 12, 2026 04:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant