Skip to content

Commit 7228d81

Browse files
TMHSDigitalclaude
andcommitted
fix(ci): shard the weekly showcase falsifier sweep and report a red cron
The 2026-10-12 cron cancelled the 5.2 smoke job at its 75-minute limit inside the serial showcase sweep (80 pieces, 831 falsifiers), so no showcase falsifier had run end to end since 2026-10-05, and nothing said so. The sweep is now its own job, fanned out over four shards by piece (check_falsifier_targets.py --shard K/N) with a 60-minute budget each, and prints each run's wall time so slow pieces are visible. workflow_dispatch gains a sweep input to run it by hand. A failed or cancelled scheduled run opens or comments on a tracking issue. Closes #482 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: TMHSDigital <154358121+TMHSDigital@users.noreply.github.com>
1 parent d4824bb commit 7228d81

5 files changed

Lines changed: 206 additions & 16 deletions

File tree

‎.github/workflows/blender-smoke.yml‎

Lines changed: 127 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,11 @@ on:
3030
- "5.1"
3131
- "4.5"
3232
default: "5.2"
33+
sweep:
34+
description: Also run the full showcase falsifier sweep (5.2, sharded)
35+
required: false
36+
type: boolean
37+
default: false
3338
schedule:
3439
- cron: "0 7 * * 1" # weekly, Monday 07:00 UTC
3540
push:
@@ -391,17 +396,130 @@ jobs:
391396
--xvfb \
392397
--catalog tests/smoke/catalog.json
393398
394-
- name: Full showcase falsifier sweep (weekly, 5.2)
395-
# The catalog runs one falsifier per row on every smoke run. The weekly
396-
# cron also runs every falsifier in every showcase README table and
397-
# asserts each lands on the exit it declares (~5 min).
398-
if: ${{ github.event_name == 'schedule' && matrix.series == '5.2' }}
399-
run: |
400-
set -euo pipefail
401-
python3 tests/check_falsifier_targets.py --run "$BLENDER"
402-
403399
- name: Smoke summary
404400
if: ${{ always() && env.BLENDER != '' }}
405401
run: |
406402
set -euo pipefail
407403
python3 tests/smoke/summarize.py --status "$BDT_SMOKE_STATUS"
404+
405+
showcase-sweep:
406+
# The catalog runs one falsifier per row on every smoke run. The weekly
407+
# cron also runs every falsifier in every showcase README table and asserts
408+
# each lands on the exit it declares. At 80 pieces / 800+ falsifiers one
409+
# serial sweep no longer fit beside the happy paths in the smoke job's 75
410+
# minutes (cancelled on 2026-10-12, #482), so it is its own job, fanned out
411+
# over four jobs by piece (check_falsifier_targets.py --shard K/N).
412+
name: Showcase falsifier sweep ${{ matrix.shard }}/4 (5.2)
413+
if: github.event_name == 'schedule' || (github.event_name == 'workflow_dispatch' && inputs.sweep)
414+
runs-on: ubuntu-latest
415+
timeout-minutes: 60
416+
strategy:
417+
fail-fast: false
418+
matrix:
419+
shard: [0, 1, 2, 3] # N is 4: keep the "/4" in the job name and the run step in step
420+
steps:
421+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
422+
423+
- name: Install Blender runtime libraries
424+
run: |
425+
set -euo pipefail
426+
sudo apt-get update
427+
sudo apt-get install -y --no-install-recommends \
428+
xvfb libgl1 libegl1 libxrender1 libxxf86vm1 libxfixes3 libxi6 \
429+
libxkbcommon0 libsm6 libice6
430+
431+
- name: Resolve Blender 5.2
432+
id: resolve
433+
run: |
434+
set -euo pipefail
435+
base="https://download.blender.org/release/Blender5.2/"
436+
file=$(curl -fsSL "$base" \
437+
| grep -oE "blender-5\.2\.[0-9]+-linux-x64\.tar\.xz" \
438+
| sort -V | uniq | tail -1)
439+
if [ -z "$file" ]; then
440+
echo "::error::Could not resolve a linux-x64 build for Blender 5.2 at $base"
441+
exit 1
442+
fi
443+
version="${file#blender-}"
444+
version="${version%-linux-x64.tar.xz}"
445+
echo "base=$base" >> "$GITHUB_OUTPUT"
446+
echo "file=$file" >> "$GITHUB_OUTPUT"
447+
echo "version=$version" >> "$GITHUB_OUTPUT"
448+
449+
- name: Restore Blender ${{ steps.resolve.outputs.version }} from cache
450+
id: bl-cache
451+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
452+
with:
453+
path: ${{ runner.temp }}/bl
454+
key: blender-${{ steps.resolve.outputs.version }}-linux-x64
455+
456+
- name: Download Blender and verify its published sha256
457+
if: steps.bl-cache.outputs.cache-hit != 'true'
458+
run: |
459+
set -euo pipefail
460+
base="${{ steps.resolve.outputs.base }}"
461+
file="${{ steps.resolve.outputs.file }}"
462+
version="${{ steps.resolve.outputs.version }}"
463+
mkdir -p "$RUNNER_TEMP/bl"
464+
curl -fSL --retry 3 -o "$RUNNER_TEMP/bl.tar.xz" "${base}${file}"
465+
curl -fsSL --retry 3 -o "$RUNNER_TEMP/bl.sha256" "${base}blender-${version}.sha256"
466+
expected=$(grep -E " ${file}$" "$RUNNER_TEMP/bl.sha256" | awk '{print $1}')
467+
if [ -z "$expected" ]; then
468+
echo "::error::no published sha256 for ${file}"
469+
exit 1
470+
fi
471+
echo "${expected} $RUNNER_TEMP/bl.tar.xz" | sha256sum -c - \
472+
|| { echo "::error::sha256 mismatch for ${file}; refusing to extract"; exit 1; }
473+
tar -xf "$RUNNER_TEMP/bl.tar.xz" -C "$RUNNER_TEMP/bl"
474+
475+
- name: Locate Blender binary
476+
run: |
477+
set -euo pipefail
478+
bl=$(find "$RUNNER_TEMP/bl" -maxdepth 2 -type f -name blender | head -1)
479+
if [ -z "$bl" ]; then
480+
echo "::error::blender binary not found"
481+
exit 1
482+
fi
483+
"$bl" --version | head -1
484+
echo "BLENDER=$bl" >> "$GITHUB_ENV"
485+
486+
- name: Run shard ${{ matrix.shard }}/4
487+
run: |
488+
set -euo pipefail
489+
python3 tests/check_falsifier_targets.py --run "$BLENDER" --shard "${{ matrix.shard }}/4"
490+
491+
report-scheduled-failure:
492+
# A red or cancelled weekly run used to go unnoticed (#482): open an issue,
493+
# or comment on the one already open, so the cron is never silently red.
494+
name: Report a failed weekly run
495+
needs: [smoke, showcase-sweep]
496+
if: >-
497+
always() && github.event_name == 'schedule' &&
498+
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
499+
runs-on: ubuntu-latest
500+
timeout-minutes: 5
501+
permissions:
502+
issues: write
503+
steps:
504+
- name: Open or update the tracking issue
505+
env:
506+
GH_TOKEN: ${{ github.token }}
507+
GH_REPO: ${{ github.repository }}
508+
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
509+
SMOKE: ${{ needs.smoke.result }}
510+
SWEEP: ${{ needs.showcase-sweep.result }}
511+
run: |
512+
set -euo pipefail
513+
title="Weekly Blender smoke run failed"
514+
body="Scheduled run: $RUN_URL
515+
- smoke matrix: \`$SMOKE\`
516+
- showcase falsifier sweep: \`$SWEEP\`
517+
518+
Opened by blender-smoke.yml (report-scheduled-failure). Close it once a weekly run is green."
519+
num=$(gh issue list --state open --search "\"$title\" in:title" --json number --jq '.[0].number // empty')
520+
if [ -n "$num" ]; then
521+
gh issue comment "$num" --body "$body"
522+
echo "commented on #$num"
523+
else
524+
gh issue create --title "$title" --body "$body" --label "bug,ci,priority: high"
525+
fi

‎.github/workflows/validate.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -438,6 +438,9 @@ jobs:
438438
- name: Exit-code gate resolves fail() helpers and tuple returns
439439
run: python3 tests/test_exit_code_readme.py -v
440440

441+
- name: Showcase falsifier sweep shards cover every piece once
442+
run: python3 tests/test_falsifier_shard.py -v
443+
441444
- name: Hero-drift tool exit-code tests
442445
run: |
443446
pip install -r requirements-dev.txt

‎CLAUDE.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,7 +189,9 @@ The host runner is `tests/smoke/run_example.py`. Shipped examples are listed in
189189
`run_catalog.py` runs each after the row's happy path; it passes only on
190190
exactly that exit. `tests/check_smoke_catalog.py` rejects a row without one
191191
and any unknown key. The weekly cron also runs the full showcase sweep
192-
(`tests/check_falsifier_targets.py --run`).
192+
(`tests/check_falsifier_targets.py --run --shard K/4`, four jobs; dispatch
193+
with `sweep: true` to run it by hand), and a red or cancelled weekly run
194+
opens or comments on a "Weekly Blender smoke run failed" issue.
193195
- **Summary:** `tests/smoke/summarize.py` prints passed/skipped/failed. Zero
194196
PASSes makes the job red even if every example skipped cleanly. Falsifier
195197
runs are tabled separately and never count toward that PASS.

‎tests/check_falsifier_targets.py‎

Lines changed: 35 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,11 @@
3333
falsifier tripping an earlier check. It costs one Blender launch per
3434
falsifier: measured at 276 s for the whole showcase tree on one version
3535
(160 falsifiers, 25 pieces, Blender 5.2.1). That is an authoring and cron
36-
tool, not a per-PR smoke step.
36+
tool, not a per-PR smoke step. At 80 pieces and 800+ falsifiers one serial
37+
sweep outgrew the smoke job's budget (#482), so ``--shard K/N`` runs only
38+
every N-th piece starting at K (0-based, sorted by name), and the weekly cron
39+
fans the sweep out across N jobs. Each run prints its wall time so the slow
40+
pieces are visible.
3741
3842
Exit codes: 0 clean, 1 a mapping disagrees, 2 usage.
3943
"""
@@ -46,6 +50,7 @@
4650
import re
4751
import subprocess
4852
import sys
53+
import time
4954

5055
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
5156
SHOWCASE = os.path.join(ROOT, "showcase")
@@ -222,14 +227,32 @@ def check_static():
222227
RUN_TIMEOUT = 900 # seconds per Blender run, matching tests/smoke/run_example.py
223228

224229

225-
def check_runtime(blender, only=None):
230+
def parse_shard(text):
231+
"""'K/N' -> (K, N) with 0 <= K < N; argparse-friendly."""
232+
try:
233+
k, n = (int(x) for x in text.split("/"))
234+
except ValueError:
235+
raise argparse.ArgumentTypeError(f"--shard wants K/N, got {text!r}")
236+
if n < 1 or not 0 <= k < n:
237+
raise argparse.ArgumentTypeError(f"--shard {text!r}: need 0 <= K < N")
238+
return k, n
239+
240+
241+
def in_shard(index, shard):
242+
return shard is None or index % shard[1] == shard[0]
243+
244+
245+
def check_runtime(blender, only=None, shard=None):
226246
failures, checked = [], 0
227-
for name, readme, script in piece_paths():
247+
for index, (name, readme, script) in enumerate(piece_paths()):
228248
if only and name not in only:
229249
continue
250+
if not in_shard(index, shard):
251+
continue
230252
falsifiers, _exits = parse_readme(readme)
231253
for flag, (want, _target) in sorted(falsifiers.items()):
232254
checked += 1
255+
started = time.monotonic()
233256
try:
234257
proc = subprocess.run(
235258
[blender, "--background", "--python-exit-code", "1", "--python", script,
@@ -243,7 +266,9 @@ def check_runtime(blender, only=None):
243266
# one hang cannot stall the weekly sweep.
244267
got = f"timeout after {RUN_TIMEOUT}s"
245268
status = "ok" if got == want else "MISMATCH"
246-
print(f" {status:9} {name} {flag}: want {want}, got {got}")
269+
secs = time.monotonic() - started
270+
print(f" {status:9} {name} {flag}: want {want}, got {got} ({secs:.0f}s)",
271+
flush=True)
247272
if got != want:
248273
failures.append(
249274
(name, f"{flag} declared exit {want} but exited {got}")
@@ -257,6 +282,8 @@ def main(argv=None):
257282
help="also execute each falsifier with this Blender binary")
258283
p.add_argument("--only", action="append", default=None,
259284
help="limit the runtime mode to these piece names")
285+
p.add_argument("--shard", type=parse_shard, default=None, metavar="K/N",
286+
help="runtime mode: only every N-th piece, starting at K (0-based)")
260287
args = p.parse_args(argv)
261288

262289
failures, undocumented, checked, pieces, skipped = check_static()
@@ -271,8 +298,10 @@ def main(argv=None):
271298
)
272299

273300
if args.run:
274-
print(f"runtime falsifier sweep with {args.run}")
275-
rt_failures, rt_checked = check_runtime(args.run, set(args.only or []) or None)
301+
shard = f" (shard {args.shard[0]}/{args.shard[1]})" if args.shard else ""
302+
print(f"runtime falsifier sweep with {args.run}{shard}", flush=True)
303+
rt_failures, rt_checked = check_runtime(args.run, set(args.only or []) or None,
304+
args.shard)
276305
failures.extend(rt_failures)
277306
checked += rt_checked
278307

‎tests/test_falsifier_shard.py‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
"""Tests for tests/check_falsifier_targets.py --shard (the weekly sweep fan-out, #482).
2+
3+
Run: python tests/test_falsifier_shard.py -v
4+
"""
5+
from __future__ import annotations
6+
7+
import argparse
8+
import sys
9+
import unittest
10+
from pathlib import Path
11+
12+
sys.path.insert(0, str(Path(__file__).resolve().parent))
13+
import check_falsifier_targets as cft # noqa: E402
14+
15+
16+
class ShardTest(unittest.TestCase):
17+
def test_shards_partition_every_piece_exactly_once(self):
18+
pieces = [name for name, _r, _s in cft.piece_paths()]
19+
self.assertGreater(len(pieces), 0)
20+
for n in (1, 2, 4, 7):
21+
seen = []
22+
for k in range(n):
23+
seen += [p for i, p in enumerate(pieces) if cft.in_shard(i, (k, n))]
24+
self.assertEqual(sorted(seen), sorted(pieces), f"N={n}")
25+
26+
def test_no_shard_keeps_everything(self):
27+
self.assertTrue(all(cft.in_shard(i, None) for i in range(10)))
28+
29+
def test_parse_shard(self):
30+
self.assertEqual(cft.parse_shard("0/4"), (0, 4))
31+
self.assertEqual(cft.parse_shard("3/4"), (3, 4))
32+
for bad in ("4/4", "-1/4", "1/0", "x", "1/2/3"):
33+
with self.assertRaises(argparse.ArgumentTypeError, msg=bad):
34+
cft.parse_shard(bad)
35+
36+
37+
if __name__ == "__main__":
38+
unittest.main()

0 commit comments

Comments
 (0)