Should hpc-stack updates happen during off-hours?

[edwardhartnett]

During the recent hpc-stack-1.2.0 update there were problems caused by updating hpc-stack during work time:

@Hang-Lei-NOAA writes:

I try to update the hpc-stack with v1.2.0 release during work time, which made some users interrupted on dell.
So, I stopped and recovered all from the backup.

@Hang-Lei-NOAA also suggests:

Perhaps, future updates should be made during machine start up before or after maintenance.

This is a good suggestion. Also it would be good if we could find some way to announce to everyone before the update.

Hang also reports:
for hpc-stack, the libs under v1.1.0 and v1.2.0 for the same compiler version are installed in the same place, and shared. Updating some modules may interrupt the other.

[Hang-Lei-NOAA]

Yes, the hpc/1.1.0 are still available after adding v1.2.0. Both 1.1.0 and 1.2.0 code for the same compiler are in the same directory. Therefore, update some libs in v1.2.0 may interrupt the installation in v1.1.0 for a short time. After the installation, under the same compiler, the v1.1.0 and v1.2.0 are equal. All the modules under the same directory are accessible by both hpc/1.1.0 and hpc/1.2.0

…

On Mon, Aug 16, 2021 at 9:07 AM Edward Hartnett ***@***.***> wrote: Assigned #300 <#300> to @Hang-Lei-NOAA <https://github.com/Hang-Lei-NOAA>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#300 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKWSMFA26VYREZNE4RJOAXTT5EESHANCNFSM5CHWMNNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[aerorahul]

@Hang-Lei-NOAA
If the installation of v1.2.0 is in the same directory as v1.1.0, why is there an interruption expected?
When installing v1.2.0 of the hpc-stack in the same location as v1.1.0 and for the same compiler+mpi, were packages that did not need an update e.g. netcdf/4.7.4, hdf5/1.10.6, etc get overwritten?

[arunchawla-NOAA]

when we move to v1.2.0 are we updating libraries that are otherwise unchanged ?

[Hang-Lei-NOAA]

@Ruhual yes, most of libs are not necessary to update This time that ncio is needed for the bug. Boi also find issues for grib associated libs, these need to be updated.

…

On Mon, Aug 16, 2021 at 9:21 AM arun chawla ***@***.***> wrote: when we move to v1.2.0 are we updating libraries that are otherwise unchanged ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#300 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKWSMFGT52VO7BVYLYNIIMLT5EGDTANCNFSM5CHWMNNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[aerorahul]

@arunchawla-NOAA
They should not.
The installation path of the stack is like a growing tree.
Every subsequent installation is an addition to the installation path, and not a replacement (except where a replacement of a package is warranted; e.g. corrupted installation, faulty installation, etc).
Otherwise, this path should keep on growing and the entire limbs should be pruned at the mpi or compiler level when the compiler is labeled as obsolete or unusable.

The versioning of hpc-stack pertains the the software of hpc-stack (modulefiles, build scripts etc), and not that of the software it installs (netcdf, hdf5, nceplibs, etc).

[aerorahul]

@Hang-Lei-NOAA
I understand that most libs are not necessary to be updated.
The question is, is that what was done or was the whole tree replaced?
And if @BoiVuong-NOAA found issues or bugs were found in ncio, then they need to make a release with the bug fix and that bugfix release should be installed before "silently" updating the installation.

[arunchawla-NOAA]

should we just have one stack location ? hpc-stack. The software itself can be versioned and grow as a function of time but the entire stack does not need to be reinstalled whenever the software is updated, only if the underlying library changes ?

[Hang-Lei-NOAA]

Boi reported the issue in grib_util. For the grib series, the problem in grib_util was associated with g2. The g2 was updated several times. This include changing the jasper lib v2.2.5, and adding new functions. Now, the new function was failed. We need to recompile to make it right.

[Hang-Lei-NOAA]

not a whole tree need to be updated