Using multiple threads/cores on OSX

[daducci]

Is it possible to use multiple threads/cores under OSX? It works on Linux, but I'm not able to use this feature on the Mac.

[gdurif]

It should be possible and it should be working.

Could try to run the function check_openmp() defined in the setup.py file here.

It returns 0 if linking against OpenMP works and 1 if not.

[samuelstjean]

It is, but you need to compile with not the default mac toolchain since that does not work (I think they have a non standard implementation or something similar). I am personally using another module to check for it automatically here https://github.com/samuelstjean/spams-python/blob/master/setup.py#L114 Seems like it was also added to a special package to do just that if it's cleaner to use https://github.com/astropy/extension-helpers

[gdurif]

Thanks @samuelstjean, I'll look into what you coded, we should reuse it.

extension-helpers seems great, the github repository seems to be still active but there has not been any release on PyPI since december 2019, and that worries me a little.

[samuelstjean]

I am personally using the older version, directly stolen from astropy, so just including this one file could be an idea. If stuff breaks we would have to fix it though, or copy an updated one, so it should not require too much work if it works as is and nobody touches it.

[gdurif]

I asked them about releasing on PyPI more often (c.f. astropy/extension-helpers#38).

Depending on their answer, we will either directly integrate their file as you did or just use extension-helpers as a requirement (which would be less maintenance on our side if possible).

[gdurif]

Should be fixed by #20

@daducci could you try with the corresponding version (c.f. below) and tell me if the mutli-threading works?

pip install git+https://github.com/getspams/spams-python.git@fix_openmp_req_macos

[daducci]

Sorry for the delay. I just tested this version, but it does not work, only 1 core is used. I found that I could fix the issue with OpenMP on OSX if I change the following in the version on this repository:

• setup.py:34 from

['cc', '-fopenmp' ,'-o',

to

['cc', '-Xpreprocessor', '-fopenmp', '-lomp', '-o',

• setup.py:133 from

if check_openmp() == 0:
    cc_flags.append('-fopenmp')
    link_flags.append('-fopenmp')

to

if check_openmp() == 0:
    cc_flags.append('-Xpreprocessor')
    cc_flags.append('-fopenmp')
    link_flags.append('-lomp')

[gdurif]

@daducci Thanks for the report. I'll do some tests (and eventually report to https://github.com/astropy/extension-helpers if needed).

[samuelstjean]

Seems like someone also found it astropy/extension-helpers#40, my guess is that those flags are for gcc, and apple is using clang as the default compiler. For a very long time, their version had issues or something else, and people had to install a third party gcc to get stuff working (like by using homebrew for example).

Seems like the easy way here is to let extension helper do the job and put the correct flag for the correct compiler, or hack it in somehow, but that sounds like trouble in the future to make it work properly, and on old versions also at the same time

[gdurif]

I am digging a bit on this. As long as extension-helpers does not support this (no answer on the issue), it will be tricky on our part.

setuptools.command.build_ext.new_compiler does not discriminate^[1] between gcc and clang (even less between Apple clang and LLVM clang I would guess). So we will have to do the check manually, and edit the flags accordingly (after calling extension-helpers) for the moment.

[1] See

$ python setup.py build_ext --help-compiler
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -c test_openmp.c -o objects/test_openmp.o -fopenmp
x86_64-linux-gnu-gcc -pthread objects/test_openmp.o -o test_openmp -fopenmp
Compiling Cython/C/C++ extension with OpenMP support
List of available compilers:
  --compiler=bcpp     Borland C++ Compiler
  --compiler=cygwin   Cygwin port of GNU C Compiler for Win32
  --compiler=mingw32  Mingw32 port of GNU C Compiler for Win32
  --compiler=msvc     Microsoft Visual C++
  --compiler=unix     standard UNIX-style compiler

[gdurif]

I am trying to fix extension-helpers regarding this matter, c.f. https://github.com/gdurif/extension-helpers/tree/support_apple_clang

I'll keep you updated (I will propose a PR when it is working).

Update: PR on the way astropy/extension-helpers#42

[gdurif]

@daducci could you tell me if installing the modified version of extension-helpers before installing spams solves the problem?

pip install git+https://github.com/gdurif/extension-helpers.git@support_apple_clang
pip install git+https://github.com/getspams/spams-python.git@fix_openmp_req_macos

[daducci]

No, in my case only 1 core is used. :-( Multiple cores are used only when installing the version with the manual patch I suggested few days ago. Dunno why this one does not work, as it seems all the modifications in there match mine.