Skip to content

reactor serve: one reactor's rejected poll takes every hosted reactor down #4

Description

@samuelmbabhazi

What happens

With a reactors: list, reactor serve runs every reactor's continuity poll, gateway poll and shutdown through Promise.all (packages/reactor-cli/src/run/host.ts: pollAll, pollGatewaysAll, shutdown). When one reactor's task rejects, the combined promise rejects as soon as that happens: the other reactors' tasks keep running in the pool, but their results are discarded, any second failure in the same cycle is never surfaced, and the driver loop in commands/serve.ts has no handler around await host.pollGatewaysAll(now) / await host.pollAll(now), so the command exits with code 1 without draining anyone. One reactor's failure stops the daemon for all of them.

The spec describes the host as serving N isolated reactors, and the offline gate proves that a trigger of one reactor never moves another's ledger. That isolation does not hold for failures.

What can reject a reactor's task in practice

Render failures are not the issue: a throwing or failing render is mapped to a RenderFailure and committed as a failed receipt (mounted-dag.ts, runMountedRender / runAsyncMountedRender), so the poll resolves. The rejections come from I/O around the render:

  • A gateway connector failing at poll time. The gateway poller (run/connectors.ts, poll) has no error handling around pollGatewayAsync, and the http connector throws on a non 2xx response (reactor connector(http): GET <url> → <status>) or on a network error, so one unreachable feed on one reactor takes the whole host down every cycle.
  • Storage writes: writeRegistry on the filesystem adapter throws (read only storage by design, or any fs error), inside the same poll.

Reproduction (offline gate)

Boot a two reactor host from the serve-host.test.ts fixture where alpha has a freshness reader that arms the monitor at boot and throws on the read the continuity poll performs, while beta is healthy. await host.pollAll(now) rejects with alpha's error; in the daemon the loop exits with code 1. The same happens through pollGatewaysAll with a gateway whose fetch throws.

Expected

Every reactor's task settles; the failing reactor is reported with its name and phase, and the other reactors keep being polled, drained and shut down. shutdown has the same shape: one reactor's failed shutdown should not skip the pool drain or another reactor's shutdown.

I have a fix with regression tests in the offline gate; PR to follow.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions