Repository navigation
Fix flaky retention test: resume launched inside the delete's commit window - #10
Merged
Merged
Conversation
"resume vs delete: a resume launched INSIDE the commit window still wins" failed on CI (root suite, node 18.17.1, PR #7 at 79f04fd) with `the resumed engine took ownership of the run` false after reapResumer's 8 s budget. The launch in beforeCommit installs `.resuming` and spawns a real detached engine. removeRun then finishes a synchronous tail (audit, rename, diff, restore, release), and the test assumed that tail always beats a node boot. If this process gets preempted for longer than that, the child arrives early. It finds the run in the trash ("no journal") or the delete's own lock, held by the live test pid ("already being executed"), and exits, because the engine does not retry. Stalling the test 1.5 s right after the launch reproduces it every time: 0/3 on macOS and 0/5 on Linux, both on node 18.17.1. The child's run.log says `run is already being executed by pid <test pid>`. The spawned engine is now preloaded (NODE_OPTIONS=--require) with test/fixtures/spawn-gate.cjs. That parks it before any flowition code runs, until the test opens the gate after removeRun has returned. It is still the same real launcher and the same accepted handoff, and the same engine has to take ownership. Only its start is pinned, to "after the delete lost". Nothing the test asserted is dropped. Two things that were implicit before are now checked directly while the engine is held: the delete released its lock, and the handoff marker came back with the run. Mutation checks: if the delete keeps its lock, the test fails at `the delete released its lock`. If the post-rename marker diff is removed, it fails at `Missing expected rejection`.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test/retention.test.js› resume vs delete: a resume launched INSIDE the commit window still wins failed once on CI: run 37995281494, job 114039619367 (root suite · node 18.17.1), on PR #7 at 79f04fd. That PR touches nosrc/,test/orscripts/files, and the rerun passed. The failure is a timing assumption in the test, not a production regression.Cause (test)
All earlier assertions passed: the delete was refused (
live), the run was restored, the trash was empty, and the audit readok→rolled-back/resume_raced. Only the last step failed.Inside
beforeCommit, the real launcher (cli run --resume --detach) installs.resumingand spawns a detached engine.removeRunthen finishes a synchronous tail: audit, rename, marker diff, restore,lock.release(). The test assumed that tail always finishes before a freshly spawned node process boots and reaches the run. When the test process is preempted for longer than a node boot, the child arrives first and hits one of two cases:run.lockis still there, held by the live test pid, so it fails withrun is already being executed by pid <test pid>.The engine's
acquireRunLockdoes not retry, so the child exits. No foreign pid ever appears inrun.lock, andreapResumertimes out.Repro. I busy-stalled the test process for 1.5 s right after the launch:
node:18.17.1-slim)The child's
run.login the failing runs saysrun is already being executed by pid <test pid>.Fix (test only)
The spawned engine is held at process start until
removeRunhas returned:test/fixtures/spawn-gate.cjsis preloaded throughNODE_OPTIONS=--require.--requireis used because Node 18.17.1 has no--import. It parks the child before any flowition code runs and waits until a gate file exists. If the gate is never opened, it exits after 60 s without having touched anything.launchHeld()setsNODE_OPTIONSandFLOWITION_TEST_GATEonly around the launch.detachRunpassesprocess.envat spawn, so only that one spawn inherits them, and both are restored before it returns. The gate also deletes its variable, so the engine's own children run unheld.Nothing changes in the launcher, the handoff, or the engine. The launch is still accepted inside the window, and the same engine has to take ownership. The only difference is that its start is pinned to after the delete lost, which is the interleaving the test is about. That replaces relying on a node boot to be slower than a sync tail.
Not weakened. Every original assertion is kept. Two things that were only implied before are now asserted directly, while the engine is held and can't interfere:
run.lockis gone, so the delete released its lock..resumingcame back with the restored run.Mutation checks:
src/retention.jsthe delete released its lockMissing expected rejectionVerification
Target test, 50× each on node 18.17.1 and node 24 (macOS) under heavy CPU contention (36 busy-loop processes on 16 cores): 50/50 and 50/50. Also 30/30 on Linux, node 18.17.1,
--cpus=1with 4 hogs. The unfixed test also passed 30/30 under that natural load, because the race needs a stall longer than a node boot. That is why the stall repro above is the evidence.test/retention.test.js: 31/31 on node 18.17.1 and node 24.Full root suite on node 24: 550/551. Under node 18.17.1, two tests in other files failed locally:
viewer-perfP2 (a 120 ms perf budget) failed while the machine's load average was about 100.viewer-e2e"viewer spine" (control socket disappeared after listen) fails the same way on untouchedmainon this Mac with node 18.Neither file is touched here, and both pass in CI.
Note: possible production follow-up (not changed here)
The same ordering exists outside tests. Suppose a resume is accepted during a delete's under-lock derivation, and its engine reaches
acquireRunLockbefore the delete's rollback releases the lock. That engine refuses with "already being executed" and exits. Safety still holds: the run is intact and the delete refused, so you never get "accepted and deleted". But the accepted resume quietly fails, and its marker ages out after 30 s. Fixing that would need a design decision, such as the engine retrying briefly while it holds a fresh handoff marker, or retention marking its lock so the engine can tell it apart from another engine. That is out of scope for a test de-flake.