Skip to content

Make the integration test deterministic#381

Merged
stepchowfun merged 1 commit into
mainfrom
deterministic-integration-test
Jul 2, 2026
Merged

Make the integration test deterministic#381
stepchowfun merged 1 commit into
mainfrom
deterministic-integration-test

Conversation

@stepchowfun

Copy link
Copy Markdown
Owner

toast validate_release was flaky because the integration test waited for Docuum to finish processing events by sampling the process state in /proc, which can't distinguish "idle, waiting for the next event" from "blocked on I/O mid-vacuum", and which also raced against Docker's asynchronous event delivery.

The test now synchronizes on Docuum's own logs instead:

  • Docuum runs with LOG_LEVEL=trace, so it logs every incoming event before processing it.
  • After each step, the test emits a sentinel event by creating and removing a uniquely named Docker network, then waits for Docuum to log its arrival. Docker delivers events to a subscriber in order and Docuum processes them strictly sequentially, so receipt of the sentinel proves all earlier events were fully processed. Network events are irrelevant to Docuum, so the sentinels don't perturb the state under test.
  • Containers are removed explicitly rather than with --rm, whose asynchronous removal could emit events after the sentinel.
  • At startup, the test probes with network events until Docuum reports receiving one, closing the race between spawning docker system events and its subscription being established.

The two trace/info log lines the test greps for are now tagref-tagged ([tag:incoming_event], [tag:listening_for_docker_events]), so changing them without updating the test fails toast lint. The src/run.rs changes are comments only.

Verified by running toast validate_release locally end to end.

Status: Ready

Fixes: N/A

🤖 Generated with Claude Code

The integration test previously waited for Docuum to finish processing
events by sampling the process state in /proc, which couldn't
distinguish "idle, waiting for the next event" from "blocked on I/O
mid-vacuum" and also raced against Docker's asynchronous event
delivery, causing flaky failures.

The test now synchronizes on Docuum's own logs: after each step, it
emits a uniquely named Docker network event as a sentinel and waits for
Docuum to log its arrival at trace level. Since Docker delivers events
in order and Docuum processes them sequentially, receipt of the
sentinel proves all earlier events were fully processed. Network events
are irrelevant to Docuum, so the sentinels don't perturb the state
under test. Containers are removed explicitly rather than with --rm,
whose asynchronous removal could emit events after the sentinel.
Startup similarly probes with network events until Docuum reports
receiving one, closing the race between spawning `docker system events`
and its subscription being established.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@stepchowfun stepchowfun merged commit 989214c into main Jul 2, 2026
6 checks passed
@stepchowfun stepchowfun deleted the deterministic-integration-test branch July 2, 2026 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant