During Github Actions CI/CD testing, the first test occasionally hangs. Nick reproduced this in his own Docker environment. Bumping the number of containers up to 32 in Github more or less guarantees multiple ranks will hang. No OOM activity observed in dmesg logs. gdb backtraces isolated the problem to certain ranks not seeing the decorated shell.init message arriving in the event log.
During Github Actions CI/CD testing, the first test occasionally hangs. Nick reproduced this in his own Docker environment. Bumping the number of containers up to 32 in Github more or less guarantees multiple ranks will hang. No OOM activity observed in dmesg logs. gdb backtraces isolated the problem to certain ranks not seeing the decorated
shell.initmessage arriving in the event log.