Hardware information through Groovy/Java: The OSHI library

[JosuaCarl]

Hi everyone,

As I have now repeatedly sunken my teeth into the Nextflow core code I know that the trace value extraction is currently handled by the command-trace.txt file which consists of multiple shell commands that are then integrated into every .command.run file and called by inserting the nxf_trace command here:

nextflow/modules/nextflow/src/main/groovy/nextflow/executor/BashWrapperBuilder.groovy

Line 610 in 0285c3b

launcher = "${shellPath()} ${fileStr(wrapperFile)} nxf_trace"

While we recently developed our own trace value reconstruction at the session level for the nf-co2footprint plugin, we encountered a library that made this job so much easier and the resulting code a lot more readable:

The OS hardware information (OSHI) library

I have done a short consideration of upsides and downsides of a possible switch:

✅ Upsides

Better compatibility with different operating systems

The library is well maintained and support a myriad of OS. The current solution provides only limited integration with MacOS (which is not necessary because of containerization, I know).

No change to methodology on OSX

The library calls the same registers in /proc if you are on Linux. So while the code may change, the results do not.

Better maintainability

The code I encountered in command-trace.txt looks frankly speaking a bit like a relic. Most of it was written more than eight years ago with little changes since its conception. It could be documented more thoroughly, the variable names are often less than verbose, it contains duplicates, ... That is not to say that it is all bad, I just think there is a reason why we use Java/Groovy for larger code segments, as it provides so many more ways to document stuff, to construct logical links, and make the code readable.

Save storage space

250 lines are written for each task that is run by any pipeline because the script is inserted in .command.run.
For a pipeline like rnaseq, which has 303 tasks when executing the test_full profile, this amounts to about 75000 lines. By comparison, the complete Groovy code within the Nextflow repo (all files with .groovy ending) are 277148 lines, so roughly 1/4th of that is stored in the work directory every time this pipeline is executed. We all know that the size of the work directory can explode quickly and this would provide one step to mitigate that problem by approximately 2.5MB for each similar run.

A replacement of command-trace.txt could also make some parts in command-run.txt superfluous. For instance the nxf_date method.

❌ Downsides

Work overhead

You have to implement it, write tests, ...

Additional dependency

While I think of the library as a solid piece of code that is maintained since 2015 by a core of dedicated developers, nobody can guarantee that it will not introduce unforeseen consequences.

⏳ TLDR

I think the OSHI libary is a better solution for obtaining trace values than the current Bash scripting approach. At least I would have preferred to work with that.

If you want to have a look at our approach: https://github.com/nextflow-io/nf-co2footprint/blob/master/src/main/nextflow/co2footprint/Recorders/SessionTraceRecorder.groovy

[bentsherman]

So it's a Java library? How do you collect traces from tasks that are running on a different node?

[JosuaCarl]

That's a very good question I have not thought of. In our case (Session level) this was never an issue, but I can see how this might make things difficult. In general, the library recognizes the OS and accesses the registers that would also be accessed by the bash code. Since the rest of Nextflow also runs on said node, I would assume that the library is no exception, but one would need to test that assumption.

[bentsherman]

Any solution has to run wherever the task is running, which is why we put the trace metrics in the task wrapper script instead of Nextflow itself

Ideally, the task wrapper should only use basic Linux tools so that users don't have to remember to include custom tooling in their environments/containers. And we still document the tools that it does use here just in case

That's why we just grab the metrics from proc, because it's very portable

[JosuaCarl]

Alright, then consider myself humbled once again. I guess we will keep the library for our purpose (tracking the metrics of Nextflow as an orchestrator on the head node), but I can see why other use cases are not covered by the proposed approach.