Use ParallelTestRunner.jl

[SoongNoonien]

Today at JuliaCon we attended a talk about a package allowing for running tests in parallel.

This is a first attempt to getting this into Oscar perhaps replacing our custom parallel tests. @fingolfin and I discussed this briefly and came to the conclusion that AbstractAlgebra.jl might be just the ideal testing ground for this endeavour. So, let's see what CI has to say.

[aaruni96]

We (@fingolfin and I, that is) looked into this for Oscar during the coding sprint this February in Berlin.

I no longer remember the exact details, or the "why" behind the behaviour, but the summary was (if I remember correctly), that this calls using Oscar too many times and that bogs down the total testing time.

This is unfortunately not documented anywhere. @fingolfin and I had discussed this in his office, it had gone into my pen-and-paper todo list, and then scratched off from my todo list when we decided in Berlin that it is probably not worth it, for Oscar, at least.

[SoongNoonien]

Yes, it seems like it is calling using AbstractAlgebra quite often. I'm working on fixing the failing CI locally and for every test it has to call using AbstractAlgebra first. Not sure how this performs when I get to fix all failures.

[SoongNoonien]

I don't quite understand the remaining error but for the runners with 4 cores we get at least a speedup by a factor of two.

[aaruni96]

I don't quite understand the remaining error

I tried digging at it, but didn't get anywhere. I think it might be a bug in ParallelTestRunner.jl : looks like it doesn't properly evaluate / expand the "Expression" part of the test?

runners with 4 cores we get at least a speedup by a factor of two.

It might be worth to also compare against a similar strategy which Oscar currently uses ?

[fingolfin]

@aaruni96 hehe I had forgotten all about that 🤦

Anyway, for AA a using AbstractAlgebra isn't so bad. But for OSCAR it's indeed a problem.

[SoongNoonien]

But for OSCAR it's indeed a problem.

Perhaps there might be a way to improve ParallelTestRunner in that regard. Maybe by adopting some of the approach we are using for our custom parallel tests? But I'm not familiar with that code.

[aaruni96]

But I'm not familiar with that code.

The setup for Oscar's parallel tests is fairly simple, as in, it doesn't do too many clever things :

Based on an environment variable, or a test argument, a number of worker processes are created, and Oscar is initialized once on all of them, along with some limited amount of common code (similar to init_code here). A list of test files is created, and we distribute jobs to all the worker processes one at a time, as and when they become free ( We use https://docs.julialang.org/en/v1/stdlib/Distributed/#Distributed.pmap ).

So, if you have 2 workers: worker1 and worker2, and 3 test files: testfile1, testfile2, and testfile3, the oscar setup will distribute testfile1 to worker1, and testfile2 to worker2 (or the other way round). Then, testfile3 is given to whichever worker finishes testing first.

In contrast to ParallelTestRunner.jl, we don't try to run each test file in total isolation from other test files, so we don't have to run init_code as often.

One downside to our approach is, each worker holds on to a large chunk of memory until all the tests are finished. E.g., if you run using 8 workers, and you have 49 long tests, the first 48 tests are roughly evenly distributed in terms of time / cpu / memory usage. But for the last test, in theory, you should have 7 workers doing nothing but still holding onto RAM, while they wait for the last worker to finish, and then all the workers exit together, and relinquish RAM. But upon surface level investigation, it didn't seem like much of a problem in practice, so I didn't dig too deeply into this.