Skip to content

ci: fix alignment setup and runner preparation - #11

Draft
zrr1999 wants to merge 9 commits into
PFCCLab:mainfrom
zrr1999:fix/alignment-single-setup
Draft

zrr1999 wants to merge 9 commits into
PFCCLab:mainfrom
zrr1999:fix/alignment-single-setup

Conversation

@zrr1999

@zrr1999 zrr1999 commented Sep 14, 2026 •

Copy link
Copy Markdown

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

Repair CI setup failures that prevent alignment and unit tests from reaching model execution. Use the CUDA 12.9 image and matching PaddleFleet ops wheel, following PFCCLab/Megatron-LM#19. Keep run_alignment_test.sh as the single owner of dependency installation and propagate setup failures.

GPU unit tests now check out into a directory unique to the workflow run and attempt. Docker mounts that checkout read-only and copies it into a private container directory before running the unchanged test command. Build products and Git writes therefore cannot leave root-owned files in the host checkout. Cleanup removes only this run's checkout; it does not require sudo or alter older workspaces. CI-script changes now trigger GPU tests.

The hosted lint workflow uses pinned Node 24 actions. The GPU runner has modern runner software but an older glibc, so its workflow checks out the exact GitHub event commit using native Git, retaining PR merge depth and LFS behavior. Full-repository formatting fixes retain the same Python ASTs.

Experiment results

  • Full-repository pre-commit and actual commit hooks pass; Git signing alone is disabled per user authorization.
  • Actionlint, the new wrapper's ShellCheck and shell syntax checks pass.
  • Four executed filesystem regressions verify private writable copies, unchanged host input and Git metadata, literal argument preservation, cleanup after success and failure, exact nonzero exit propagation, and refusal to run after a copy failure. The lint job also runs these regressions.
  • The previous head 4b473796 passed GPU model-accuracy alignment. The current head 9bc18303 also passed GPU model-accuracy alignment (job 111320507250); its GPU unit and NPU jobs still need to pass. Local wrapper tests do not establish native hardware acceptance.

Git LFS is initialized only in the disposable checkout. If the binary is missing, the job verifies the official v3.8.0 archive SHA-256 before using a temporary binary, then fetches, materializes and checks the LFS objects. The current GPU unit job fails closed because the host has Git 1.8.3.1, while Git LFS requires Git >= 2.0. An earlier diagnostic run also established that the required /mnt/modelscope/ci_env.sh is absent. A compatible, correctly configured runner is required; these are infrastructure blockers, not passing unit tests. The NPU job remains required.

zrr1999 and others added 3 commits September 14, 2026 02:46
Let run_alignment_test.sh own environment setup and propagate setup failures.

Signed-off-by: Zhan Rongrui <me@zrr.dev>
Signed-off-by: Zhan Rongrui <me@zrr.dev>
@zrr1999 zrr1999 changed the title ci: install alignment environments only once ci: fix alignment setup and runner preparation Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant