[v4.1.x] pml/cm, pml/ob1: pack buffered sends on every MPI_Start - #14366
Open
yinliaws wants to merge 2 commits into
Open
[v4.1.x] pml/cm, pml/ob1: pack buffered sends on every MPI_Start#14366yinliaws wants to merge 2 commits into
yinliaws wants to merge 2 commits into
Conversation
This patch fixes a bug exposed by MTT MPI_Bsend_init_overtake_c, where - MPI_Bsend_init(buf, ..., req) - MPI_Start(req) - MPI_Wait(req, stat) - modify(buf) - MPI_Start(req) // this does not send the updated data - MPI_Wait(buf) This is because PML/CM uses a bounce buffer send_req.req_buff to pack data from the application buffer in MPI_Bsend_int but not MPI_Start. This patch adds a step to pack data from the application buffer in each MPI_Start call. Signed-off-by: Wenduo Wang <wenduwan@amazon.com> (cherry picked from commit 3226c6c) (cherry picked from commit b124df4) Signed-off-by: Yin Li <yinliq@amazon.com>
Contributor
Author
|
@hppritcha @janjust Could you help approve and merge this backport? Thanks! |
bosilca
approved these changes
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of two buffered-send fixes to v4.1.x.
PML/CM packs the user buffer into its bounce buffer at MPI_Bsend_init time but not on each MPI_Start, so a restarted persistent buffered send retransmits the payload of the previous start. The MPI_Bsend_init_overtake test exercises exactly that sequence, and on v4.1.x every message after
the first carries stale data.
3226c6c / b124df4 pml/cm: pack data from application buffer in successive MPI_Start calls
429c7b7 / 78bfdae Fix buffered sends for OB1 as well.
Cherry-picked from the v5.0.x versions rather than the main originals. The main version of the first commit does apply to v4.1.x, but produces slightly different content; the v5.0.x version is the one verified here. Order matters: the OB1 commit also touches pml_cm_sendreq.h and will not apply to a pristine v4.1.x tree on its own.
Verified on v4.1.x with a two-rank reduction of that test, which needs no fabric and no scale: before the change 19 of 20 messages carry a stale payload, after it all 20 are correct. pml/ob1 over shared memory passes both before and after, so the OB1 commit is here for parity with v5.0.x rather than because this test needs it.
Scope: this is only to get the MTT runs on this branch clean.