Skip to content

ci: install alignment environments only once - #16

Draft
zrr1999 wants to merge 2 commits into
PFCCLab:mainfrom
zrr1999:fix/alignment-single-setup
Draft

zrr1999 wants to merge 2 commits into
PFCCLab:mainfrom
zrr1999:fix/alignment-single-setup

Conversation

@zrr1999

@zrr1999 zrr1999 commented Sep 14, 2026 •

Copy link
Copy Markdown
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

The workflow calls setup_venvs.sh before run_alignment_test.sh, which already calls it. This installs the environment twice and starts a second attempt after the first setup fails. Remove the redundant workflow call so the test entrypoint owns setup; its existing set -euo pipefail and the workflow exit-status check propagate installation failures.

Validation: after a normal merge of current PFCCLab main, the PR diff remains one removed workflow command. Repository pre-commit checks and tools/autoformat.sh in check mode passed. The actual workflow shell tail and test entrypoint passed four isolated command replays: success, setup failure, model failure, and comparison failure. Setup runs exactly once; its failure starts no models, and all failure paths return nonzero. The unchanged upstream baseline reproduces duplicate setup. Real GPU alignment passed in the current CI run: MinimaxV2.5 EP2, two ranks and ten training steps, with identical per-token and final-loss MD5 values for every rank and step.

This removes duplicate setup invocations; it does not change the package manager's own download retries or fix network connection resets.

Issue tracking

For PRs from open-source community contributors:

  • New features: a linked issue is required. Please open a feature request and reference it here before submitting the PR.
  • Small updates (bug fixes, minor improvements): a linked issue is recommended and will accelerate the PR review process.

Related failure: https://github.com/PFCCLab/Megatron-LM/actions/runs/34825280415/job/103915866438

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment @NVIDIA/mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

  1. When your PR is ready, click Ready for Review.
  2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
    • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

zrr1999 and others added 2 commits September 14, 2026 02:45
Let run_alignment_test.sh own environment setup and propagate setup failures.

Signed-off-by: Zhan Rongrui <me@zrr.dev>
Signed-off-by: zrr1999 <2742392377@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant