Skip to content

fix(node_info): reap zombie pve_mod_worker and orphaned collector processes - #307

Merged
Meliox merged 2 commits into
mainfrom
reap-zombie-worker-processes
Aug 22, 2026
Merged

fix(node_info): reap zombie pve_mod_worker and orphaned collector processes#307
Meliox merged 2 commits into
mainfrom
reap-zombie-worker-processes

Conversation

@Meliox

@Meliox Meliox commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Problem

Follow-up to #306. and an attempt to resolve side note in #305. Two related issues remained:

  1. Old pve_mod_worker processes stay behind as <defunct> zombies. _pve_mod_worker() did a single fork(), so the worker's parent was whatever process called pve_mod_starter() (e.g. a pveproxy pool worker). That process never reaps the PID, and since pveproxy dispatches requests across a pool, the process that later notices the worker is gone is often not its actual parent — so the zombie can only be cleaned up incidentally, if ever.
  2. If the worker is killed with SIGKILL (uncatchable) or crashes, $SIG{TERM}/$SIG{INT} and the END block never run, so _stop_child_collectors() is skipped. Its collector children (e.g. collector-temperature-sensors) get reparented to init and keep running orphaned indefinitely.

Fix

  • Double-fork daemonization in _pve_mod_worker(): an intermediate process forks the real worker and exits immediately, reparenting the worker to init (PID 1). Init always reaps its children, so the worker can never outlive its own exit as a zombie, regardless of which process later calls pve_mod_starter(). The original caller waitpid()s the short-lived intermediate process so that one doesn't zombie either.
  • _reap_orphaned_collectors(): runs at the top of _pve_mod_keep_alive() on every new worker startup. Scans /proc for collector-* processes whose parent is init (ppid == 1), and terminates them (SIGTERM, then SIGKILL after a short grace period) — cleaning up stragglers left by a non-gracefully-killed previous worker.
  • _worker_lock_file_exists(): opportunistically calls waitpid($worker_pid, WNOHANG) before removing a stale/zombie lock, as a cheap supplemental reap for cases where the caller does happen to be the actual parent.
  • New get_process_ppid() helper in Utils.pm (parses /proc/<pid>/stat, same care around the comm field as the existing zombie-detection regex).

Meliox added 2 commits August 22, 2026 23:41
…cesses

Double-forks pve_mod_worker so it is reparented to init instead of the calling process, which reaps it immediately on exit and prevents it lingering as a <defunct> zombie. Also adds startup-time reaping of orphaned collector-* processes (ppid==1) left behind if a worker was previously killed with SIGKILL, and an opportunistic waitpid(WNOHANG) in _worker_lock_file_exists as a cheap supplemental reap.
@Meliox
Meliox force-pushed the reap-zombie-worker-processes branch from 86fefc5 to 103181d Compare August 22, 2026 21:45
@Meliox
Meliox merged commit 972c9ae into main Aug 22, 2026
2 checks passed
@Meliox
Meliox deleted the reap-zombie-worker-processes branch August 22, 2026 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sensors disappear permanently after pveproxy restart: stale worker lock with zombie PID is never detected

1 participant