Skip to content

fix(runtime): fail fast when proxy listeners terminate - #16

Merged
kovawx merged 17 commits into
developfrom
fix/supervise-proxy-runtime
Aug 10, 2026
Merged

fix(runtime): fail fast when proxy listeners terminate#16
kovawx merged 17 commits into
developfrom
fix/supervise-proxy-runtime

Conversation

@kovawx

@kovawx kovawx commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Related

Problem

The Zero application starts proxy orchestration in a detached Tokio task, then the root task waits only for Ctrl-C. If proxy/listener orchestration terminates unexpectedly, inbound sockets are dropped but IPC/control services and the process remain alive. This produces a false-healthy state: systemd sees a running Zero process while protocol ports have disappeared.

This matches a field report from one VPS where Zero remained responsive through its control plane after the configured listener ports stopped listening.

Changes

Supervise the data plane from the root application lifecycle

  • Run proxy.run_until(wait_for_shutdown_signal()) directly under the application root instead of proxy.spawn() + an unsupervised join handle.
  • Log any unexpected proxy runtime error immediately.
  • Continue orderly shutdown of status monitoring, event dispatcher and control servers.
  • Emit engine.stopped with runtime_error for unexpected data-plane termination and signal for normal shutdown.
  • Return the proxy error after cleanup so the process exits non-zero and systemd/container restart policy can recover it.

Distinguish QUIC endpoint failure from per-client failure

The QUIC stream listener previously awaited both the client handshake and first bidirectional stream inline. A failed or abandoned client could therefore terminate the listener and, once root supervision was added, force the whole process to restart.

Now QUIC listener lifecycle and connection lifecycle are separated:

  • the endpoint accept loop yields incoming connection attempts and remains runtime-owned;
  • client handshake, authentication and first-stream establishment run inside per-connection tasks;
  • a failed or abandoned client is logged with inbound identity and remote address, then the listener continues accepting;
  • only unexpected endpoint termination returns an EngineError and triggers the application supervisor;
  • normal explicit listener shutdown remains a clean Ok(()) path.

Add structured listener/runtime diagnostics

The field incident cannot be reproduced reliably, so this PR also makes the next occurrence actionable instead of leaving only a generic task-exit error.

Listener lifecycle logs now carry stable fields such as:

  • inbound_tag;
  • protocol;
  • transport where known;
  • configured/listening address and port where available;
  • explicit reason values including shutdown_signal, shutdown_channel_closed, listener_endpoint_closed, connection_accept_error, connection-task panic and listener-task error;
  • the original error text.

The listener task wrapper logs the configured inbound identity both when a task returns cleanly and when it fails. This covers protocols whose inner listener loop does not expose all configured address metadata itself.

The orchestration supervisor logs unexpected listener and URLTest termination with:

  • core_instance_id;
  • config_revision;
  • active listener/URLTest task counts;
  • pending expected listener exits during reconciliation;
  • structured termination reason and root error.

TCP accept() errors retain the listener instead of terminating it, but now include inbound tag, transport and local listen address so host-specific resource failures can be diagnosed from the journal. System-stack closure and shutdown-channel loss are also logged explicitly.

URLTest runtime tasks log their group ID and task result so a URLTest task cannot collapse into an untraceable UrlTestTaskExited if it becomes the orchestration failure source.

Scope

This PR intentionally does not add an in-process listener restart loop. The safe first recovery boundary is the Zero process: an unexpected data-plane failure must become visible and cause a non-zero exit rather than leave a partially alive process.

The exact host-specific trigger on the affected VPS still requires runtime evidence. This change makes that evidence observable and prevents the trigger from being hidden behind a false-healthy process.

Validation

Added focused regressions for application-level error supervision, expected versus unexpected listener/URLTest exits, QUIC handshake failure isolation, QUIC client close before first stream, and runtime ownership boundaries.

Repository formatting, workspace checks/tests, all-target/all-feature clippy, optional surfaces, musl compatibility, and GitHub CI pass. Follow-up host-side diagnostics in Zboard are tracked separately in zerodenet/zboard#42.

@kovawx
kovawx marked this pull request as ready for review August 10, 2026 07:20
@kovawx
kovawx merged commit 581d109 into develop Aug 10, 2026
15 checks passed
@kovawx
kovawx deleted the fix/supervise-proxy-runtime branch August 10, 2026 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(runtime): supervise proxy orchestration so listener failure cannot leave a false-healthy process

1 participant