I believe many projects (and many cluster quotas) would profit from having easy access to profiling for their experiments. In my seml projects I used this code: https://gist.github.com/siboehm/bf69a17cc9bca71c37a2fae0214a1eeb
It profiles the whole experiment using py-spy. It works well even for long running experiments (many hours). The size of the profile is a few MB and the CPU overhead of the profiler is pretty low. It runs in a subprocess and doesn't require changes to the main experiment.
Would you be interested in getting a PR that integrates profiling via py-spy into seml directly? If not, should I add the Gist as an example or can we link to it somehow?
I believe many projects (and many cluster quotas) would profit from having easy access to profiling for their experiments. In my
semlprojects I used this code: https://gist.github.com/siboehm/bf69a17cc9bca71c37a2fae0214a1eebIt profiles the whole experiment using
py-spy. It works well even for long running experiments (many hours). The size of the profile is a few MB and the CPU overhead of the profiler is pretty low. It runs in a subprocess and doesn't require changes to the main experiment.Would you be interested in getting a PR that integrates profiling via
py-spyinto seml directly? If not, should I add the Gist as an example or can we link to it somehow?