Repository navigation
fix(tests): make two flaky test suites deterministic - #1771
Merged
Merged
Conversation
generate-cve-json: `main(["--config", ...])` reloads the module's config globals and nothing restored them, so a test on the same xdist worker that ran after the release-vote gating tests inherited `release_vote_gating = true` and got DRAFT where it expected REVIEW. An autouse fixture now puts the config back after any test that changed it. Run in that order, the two tests failed every time before this change. adversarial-review: two runner tests raced a wall-clock window against stub start-up, which takes seconds on a loaded machine (the full parallel pre-commit run). Both now use a handshake instead: the stubs wait for each other to start, and the timeout test reports gemini's timeout only after codex has answered. Each still fails when `run_all` runs the reviewers one after the other. Generated-by: Claude Code (Opus 5.5)
github-actions
Bot
added
family:tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three tests failed intermittently under the full parallel run (the commit hook's
workspace-pytest, and CI). They blocked commits that had nothing to do with them.generate-cve-json:
test_normalised_body.pysawDRAFTwhere it expectedREVIEWmain(["--config", ...])reloads the module-level config globals, and nothing restored them.TestReleaseVoteGatingintest_cli.pypasses a config withrelease_vote_gating = true. Any later test on the same xdist worker that calledmain()without--configran under that config. Run in that order, the pair failed every time:An autouse fixture in
tests/conftest.pynow puts the config back after any test that changed it. The CLI itself is unchanged: a real process exits after one run, so the leak only exists inside the test process.adversarial-review: two runner tests raced the clock
test_one_timeout_does_not_block_the_othersgave codex's stub a shared 5 s budget, already raised once from 1 s.test_reviewers_run_in_parallel_and_keep_orderrequired two 3 s sleeps to overlap.On a loaded machine a fresh stub can start several seconds late. In one failure, one stub started almost 5 s after the other, after the other had already finished its 3 s sleep. So both failed now and then.
Both now use a handshake instead of a time window:
As a check, I made
run_allsequential (max_workers=1) and confirmed both tests fail. They still test concurrency, not just the absence of a race.Verification
mainand passes here; the full generate-cve-json suite passed 8 of 8 parallel runs.tools/adversarial-review/tests/test_runner.pypassed 10 of 10 parallel runs.prek run --all-files --skip workspace-pytestis clean. The commit hook'sworkspace-pytestpassed.Generative AI disclosure: this change was prepared with Claude Code (Opus 5.5) and reviewed by me before submission.
🤖 Generated with Claude Code