Skip to content

fix(ladder): teardown guard hung case asserts the event, not a 6-tick clock (K1) #5832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
noahgift wants to merge 1 commit into from
Closed

fix(ladder): teardown guard hung case asserts the event, not a 6-tick clock (K1) #5832

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions scripts/check_ladder_serve_teardown.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,15 +222,20 @@ extract_wait() {
printf '%s\n' "$body"
}

wait_case() { # <name> <mode> <secs> <stall> <ceiling> <want-verdict> <max-seconds>
local name="$1" mode="$2" secs="$3" stall="$4" ceil="$5" want="$6" maxs="$7" port pid out verdict took
# <took> is the health wait's own tick count (one tick per loop turn), not wall time. For `hung` it is NOT
# a fixed number: the quiet counter restarts on any CPU-jiffy the tree burns, and a loaded host burns a few
# (python start-up, scheduler wake-ups), so 3 quiet ticks took 3..7 turns (9 of 40 runs read 4 under load, #5798).
# The contract is the EVENT: verdict `stalled` (not ceiling, not ready, not died) after at least <stall> quiet
# ticks (never early) and before the ceiling. The bound for `hung` is therefore the ceiling, not stall+3.
wait_case() { # <name> <mode> <secs> <stall> <ceiling> <want-verdict> <max-ticks> [<min-ticks>]
local name="$1" mode="$2" secs="$3" stall="$4" ceil="$5" want="$6" maxs="$7" mins="${8:-0}" port pid out verdict took
port=$(( 20000 + (RANDOM % 20000) ))
flock -w 5 "$LOCK" python3 "$TMP/fake_loader.py" "$port" "$mode" "$secs" "$TMP/$name.loader.pid" > "$TMP/$name.log" 2>&1 & pid=$!
out=$(ladder_serve_wait_health "$pid" "$port" "$TMP/$name.log" "$stall" "$ceil") || true
verdict=${out%% *}; took=${out##* }
reap_pidfile "$TMP/$name.loader.pid"; kill -9 "$pid" 2>/dev/null || true; wait "$pid" 2>/dev/null || true
if [ "$verdict" != "$want" ] || [ "$took" -gt "$maxs" ]; then
echo " FAIL $name: '$out' — want '$want' within ${maxs}s"; return 1
if [ "$verdict" != "$want" ] || [ "$took" -gt "$maxs" ] || [ "$took" -lt "$mins" ]; then
echo " FAIL $name: '$out' — want '$want' in ${mins}..${maxs} ticks"; return 1
fi
echo " ok $name: '$out'"
}
Expand All @@ -242,7 +247,7 @@ run_wait_cases() { # <wait-function-bodies> -> 0 if every case lands
eval "$body"
wait_case slow-log slow-log 6 3 30 ready 10 || fails=1
wait_case slow-cpu slow-cpu 6 3 30 ready 10 || fails=1
wait_case hung hung 60 3 30 stalled 6 || fails=1
wait_case hung hung 60 3 30 stalled 29 3 || fails=1
wait_case dies dies 2 5 30 died 4 || fails=1
wait_case forever forever 0 3 6 ceiling 9 || fails=1
return "$fails"
Expand Down
Loading