Skip to content

test: eliminate TOCTOU port collisions in multi-process harnesses #324

Description

@NeaBouli

Context\n\nThe exact-head protected Go CI for PR #320 failed once in the governed-upgrade job even though the cumulative dependency/security diff does not change this harness.\n\n## Reproduction evidence\n\n- Workflow run: https://github.com/NeaBouli/TrueRepublic/actions/runs/35462066576\n- Job: governed-upgrade\n- Test: TestGovernedUpgradeMultiValidatorHaltFailureRecovery\n- Failure: validator-3 could not bind 127.0.0.1:46433 because the address was already in use.\n- Consequence: only three validators started; waitForSmokeHeight then timed out waiting for all four at height 2.\n- The surviving validators continued producing blocks, confirming this was startup port contention rather than a consensus or dependency regression.\n\n## Root cause\n\nfreeTCPPort in server_lifecycle_test.go opens 127.0.0.1:0, records the assigned port, and immediately closes the listener. The governed-upgrade and other multi-process harnesses allocate many ports this way before starting subprocesses. This creates a time-of-check/time-of-use window in which the OS or another harness allocation can reuse a supposedly free port.\n\n## Acceptance criteria\n\n- [ ] Port allocation for process harnesses cannot return duplicate or stolen ports between allocation and child bind.\n- [ ] The fix is shared by governed-upgrade and the other smoke/recovery harnesses that use freeTCPPort.\n- [ ] Add a deterministic regression test for unique/reserved allocation and cleanup.\n- [ ] Run governed-upgrade repeatedly and all affected multi-validator/recovery targets.\n- [ ] Preserve production code and network behavior; this is test infrastructure only.\n- [ ] Document real commands/results in BRIDGE.md.\n\n## Scope boundary\n\nDo not weaken timeouts, skip validators, ignore bind failures, or mask a failed child process. The harness must remain fail-closed. No production deployment, migration, live-network or rollout action.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions