Problems with Linux (Arch)

[cvalencia09]

It seems to be that the Julia package have a problem when running in a Linux machine. I have the stackoverflow error after running the newuoa algorithm, this problem doesn't occur in my Windows partition. In Windows I have the Intel compiler for Fortran, in Linux just the gcc compiler. Which libraries do you require to run the package on Linux?

Regards,

[amontoison]

When you use PRIMA.jl, a precompiled version of PRIMA is provided with the artifact PRIMA_jll.jl so you don't need any compiler to use this Julia interface.
Can you run the Julia tests with the following commands and provide the error(s) that you encounter?

julia> ]
pkg> test PRIMA

[soldasim]

Hello, I have also encountered the StackOverflowError on vairous Linux devices.

I have tested the issue on multiple devices. This is what information I could gather;

• It appears the error only occurs on Linux. (At least I have not encountered it on Windows yet.)
• On some Linux devices, the error only occurs when runnning PRIMA in parallel. On some devices it occurs even when running PRIMA in serial.
• On the devices where the error occurs only when running in parallel, it does not matter if I parallelize using tasks (Threads.@spawn), parallel for loop (Threads.@threads for) or even if I run PRIMA in a distributed manner (using @distributed). All of these options behave the same.

Unfortunately, the error message does not provide any information (not even a stacktrace), so this may be difficult to debug. I will try to provide as much information as I can.

MWE

Consider the following MWE;

using PRIMA

function prima_serial(; tasks=1)
    obj = (x) -> abs(5. - x[1])
    start = [0.]

    results = [newuoa(obj, start)[1] for _ in 1:tasks]
end

function prima_parallel(; tasks=1)
    obj = (x) -> abs(5. - x[1])
    start = [0.]

    tasks = [Threads.@spawn newuoa(obj, start)[1] for _ in 1:tasks]
    results = fetch.(tasks)
end

Note that I am running only a single task when parallelizing. So there actually are not multiple PRIMA instances running in parallel. But somehow it causes errors on some devices anyway.

Test Results

The following table summarizes the results of running the two functions prima_serial and prima_parallel from the MWE on various devices that I have access to:

Device	prima_serial	prima_parallel
PC-1 (Win)	✅	✅
PC-2 (Win)	✅	✅
PC-2 (Linux)	✅	StackOverflowError
PC-2 (Linux) [VSCode REPL]	StackOverflowError	StackOverflowError
Cluster-1 (Linux)	✅	StackOverflowError
Cluster-2 (Linux)	✅	StackOverflowError
PC-3 (MacOS)	✅	✅

(Note that PC-2 (Win) and PC-2 (Linux) is the same exact computer with dualboot.)

Correct output:

julia> prima_serial()
1-element Vector{Vector{Float64}}:
 [5.0000000000000115]

julia> 

Error message for prima_serial:

julia> prima_serial()
ERROR: StackOverflowError:

julia> 

Error message for prima_parallel:

julia> prima_parallel()
ERROR: TaskFailedException
Stacktrace:
  [1] wait
    @ ./task.jl:352 [inlined]
  [2] fetch
    @ ./task.jl:372 [inlined]
  [3] _broadcast_getindex_evalf
    @ ./broadcast.jl:709 [inlined]
  [4] _broadcast_getindex
    @ ./broadcast.jl:682 [inlined]
  [5] getindex
    @ ./broadcast.jl:636 [inlined]
  [6] copy
    @ ./broadcast.jl:942 [inlined]
  [7] materialize
    @ ./broadcast.jl:903 [inlined]
  [8] prima_parallel(; tasks::Int64)
    @ Main ~/julia-sandbox/prima_parallel/test.jl:15
  [9] prima_parallel()
    @ Main ~/julia-sandbox/prima_parallel/test.jl:10
 [10] top-level scope
    @ REPL[3]:1

    nested task error: StackOverflowError:

julia> 

Device Specifications

PC-1 and PC-2 are my personal computers. PC-2 has both Windows and Linux on dualboot. Cluster-1 and Cluster-2 are academic clusters that I have access to. The information below contains specs of both the "login" and "work" nodes from the clusters. I've tested the MWE on both the login and work nodes and the behavior does not differ between them.

PC-1

OS: Microsoft Windows 10 Pro
Version: 10.0.19045 Build 19045
System Type: x64-based PC
Processor: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz, 3401 Mhz, 4 Core(s), 4 Logical Processor(s)

Julia version: 1.10.2

PC-2 (Windows)

OS: Microsoft Windows 10 Home
Version: 10.0.19045 Build 19045
System Type: x64-based PC
Processor: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz, 2001 Mhz, 4 Core(s), 8 Logical Processor(s)

Julia version: 1.10.2

PC-2 (Linux)

OS: Ubuntu 20.04.6 LTS
Kernel: Linux 5.4.0-171-generic
Architecture: x86-64
Processor: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz

Julia version: 1.10.2

Cluster-1

Login Node:
OS: CentOS Linux 7 (Core)
Kernel: Linux 3.10.0-1127.13.1.el7.x86_64
Architecture: x86-64
Processor: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz

Work Node:
Kernel: Linux 4.18.0-425.13.1.el8_7.x86_64
Architecture: x86-64
Processor: Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz

Julia version: 1.10.0

Cluster-2

Login Node:
OS: Ubuntu 20.04.6 LTS
Kernel: Linux 5.15.0-94-generic
Architecture: x86-64
Processor: Common KVM processor

Work Node:
OS: Ubuntu 22.04.3 LTS
Kernel: Linux 5.15.0-91-generic
Architecture: x86-64
Processor: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz

Julia version: 1.10.2

PC-3 (MacOS)

OS: macOS (arm64-apple-darwin22.4.0)
CPU: 12 × Apple M3 Pro

Julia version: 1.11.1

---

Let me know if I can help with any additional information or testing. :)

EDIT-1: Added PC-2 (Linux) VSCode and "non-VSCode" versions to the test result table.

EDIT-2: Added PC-3 (MacOS)

[soldasim]

I have run ] test PRIMA on all of the devices mentioned above and all tests succeed on all devices.

Test Summary: | Pass  Total   Time
PRIMA.jl      |   81     81  12.9s
     Testing PRIMA tests passed 

[emmt]

Thank you for all these details. I have tested your examples on my Linux laptop (Ubuntu 23.10 with 6.0.0 kernel) with the following results:

julia> prima_serial()
1-element Vector{Vector{Float64}}:
 [5.0000000000000115]

julia> prima_parallel()
ERROR: TaskFailedException
Stacktrace:
  [1] wait
    @ ./task.jl:352 [inlined]
  [2] fetch
    @ ./task.jl:372 [inlined]
  [3] _broadcast_getindex_evalf
    @ ./broadcast.jl:709 [inlined]
  [4] _broadcast_getindex
    @ ./broadcast.jl:682 [inlined]
  [5] getindex
    @ ./broadcast.jl:636 [inlined]
  [6] copy
    @ ./broadcast.jl:942 [inlined]
  [7] materialize
    @ ./broadcast.jl:903 [inlined]
  [8] prima_parallel(; tasks::Int64)
    @ Main ./REPL[3]:6
  [9] prima_parallel()
    @ Main ./REPL[3]:1
 [10] top-level scope
    @ REPL[7]:1

    nested task error: StackOverflowError:

So the serial version worked, not the parallel one. Note that the serial version also worked for tasks=2 or more, not the parallel one (with always the same stack overflow error).

Are you sure that the serial version failed on your PC-2 (Linux)?

For the parallel version, I can see some questions that need to be answered:

1. Are the functions in libprima (the Fortran90 and the C versions) thread safe or not?
2. The StackOverflowError seems to indicate a problem on the side of the Julia interface. So the same question arises for the Julia version. In principle, this interface allows for having an objective function that itself calls one of the PRIMA optimizers (hierarchical optimization). But it may not have been fully tested.

[amontoison]

In Julia, the use of @ccall is thread-safe if it can help to isolate the issue.

[emmt]

Yes @ccall is used but the problem I can see, is that the Julia interface uses a per-thread stack of contexts (stored in the global variable _objfun_stack) to be at the same time thread-safe and to allow for hierarchical optimization and this has not been thoroughly tested. With the new C API of libprima (see #28) this management would no longer be necessary and the problem may be solved (provided the C and Fortran code are thread-safe).

[amontoison]

Ok, I see 👍
It increases the priority to do an unoffical build of PRIMA_jll.jl v0.8.0 asap.

[soldasim]

Are you sure that the serial version failed on your PC-2 (Linux)?

I have tested it again to be sure. The serial version really fails on my Linux PC but only in Julia started by VSCode.

When I run the two functions from Julia REPL started by the VSCode's Julia extension (Ctrl+Shift+P -> Julia: Start REPL), both the serial and the parallel version throw StackOverflowError.

When I start Julia REPL from bash myself, only the parallel version fails and the serial works fine as on other linux devices.

I don't know what to make of this, but at least it is consistent when tried multiple times.

Version info

The only difference in versioninfo() is that the REPL started by VSCode has an additional line JULIA_EDITOR = code in the Environment.

julia> versioninfo()
Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 8 × Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, skylake)
Threads: 8 default, 0 interactive, 4 GC (on 8 virtual cores)
Environment:
  LD_LIBRARY_PATH = :/opt/gurobi10.0.0_linux64/gurobi1000/linux64/lib
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 8

julia> 

[emmt]

Ok that's really puzzling...

We have started to figure out a way to deal with thread-safety (and hierarchical optimization) differntly than currently done in PRIMA.jl. I hope this will solve the issue. Your MWE should definitely be part of the test suite of PRIMA.jl.

[renatomatz]

Hi, I have the same issue in my Linux machine, with prima_serial working just fine but not prima_parallel. Other than that, all tests from test PRIMA seem to pass.

Has there been any progress on this bug?

OS: Ubuntu 22.04.4 LTS
Kernel: Linux 6.5.0-41-generic
Architecture: x86-64
Processor: Intel i7-7700HQ (8) @ 3.800GHz
Julia version: 1.10.4

[soldasim]

Hi, got my hands on a Mac, so I've tested the parallelization on it as well.

The prima_parallel runs fine on MacOS (at least in my case). I've added the results to my original comment.

So the issue seems to be purely Linux-related.

[soldasim]

Interestingly, some tests do not pass on my Mac.

I believe this to be unrelated to this parallelization issue, so I've created a new issue.

[andreasnoack]

@vchuravy based on the info here, do you have any idea if this is because you aren't supposed to mix task_local_storage with C callbacks of this is more likely to be an issue with the wrapped C/Fortran code? The key parts of the code are

PRIMA.jl/src/PRIMA.jl

Lines 441 to 443 in af79578

	status = task_local_storage(:objfun, fw) do
	prima_newuoa(fp, n, x, fx, nf, rhobeg, rhoend, ftarget, maxfun, npt, iprint)
	end

where libprima is called with a function pointer while setting task_local_storage and then

PRIMA.jl/src/PRIMA.jl

Lines 738 to 744 in af79578

	function _objfun(x_ptr::Ptr{Cdouble}, # (input) variables
	f_ptr::Ptr{Cdouble}) # (output) function value
	# Retrieve objective function object and dispatch on its type to compute f(x).
	f = task_local_storage(:objfun)
	unsafe_store!(f_ptr, unsafe_call(f, x_ptr))
	return nothing
	end

where the Julia function is read from task_local_storage.

FWIW here is the immediate debugger output when running prima_parallel

julia> prima_parallel()
Process 156924 stopped
* thread #1, name = 'julia', stop reason = signal SIGSEGV: invalid permissions for mapped object (fault address: 0x7ffec69986d8)
    frame #0: 0x00007ffec69986d8
->  0x7ffec69986d8: movabsq $0x7ffec06eebd0, %r11 ; imm = 0x7FFEC06EEBD0
    0x7ffec69986e2: movabsq $0x7ffec69986d0, %r10 ; imm = 0x7FFEC69986D0
    0x7ffec69986ec: jmpq   *%r11
    0x7ffec69986ef: nop

with backtrace

    (lldb) bt
* thread #1, name = 'julia', stop reason = signal SIGSEGV: invalid permissions for mapped object (faul
t address: 0x7ffec69986d8)
  * frame #0: 0x00007ffec69986d8
    frame #1: 0x00007ffebf4e98f9 libprimaf.so`__evaluate_mod_MOD_evaluatef + 425
    frame #2: 0x00007ffebf587cad libprimaf.so`__initialize_newuoa_mod_MOD_initxf + 2477
    frame #3: 0x00007ffebf5969b1 libprimaf.so`__newuob_mod_MOD_newuob + 1969
    frame #4: 0x00007ffebf59bb96 libprimaf.so`__newuoa_mod_MOD_newuoa + 1350
    frame #5: 0x00007ffec06eee13 libprimac.so`newuoa_c + 451
    frame #6: 0x00007ffec06ef1d1 libprimac.so`prima_newuoa + 33
    frame #7: 0x00007ffec60a8f33 JIT(0x14fe960)`julia_prima_newuoa_1517 at wrappers.jl:46
    frame #8: 0x00007ffec60b37f4 JIT(0x16a19f0)`julia_task_local_storage_2204 at PRIMA.jl:442
    frame #9: 0x00007ffec60b3e9c JIT(0x1721e30)`julia_#newuoa!#13_2160 at PRIMA.jl:441
    frame #10: 0x00007ffec60b480d JIT(0x175c540)`julia_#9_2123 at PRIMA.jl:418
    frame #11: 0x00007ffec60b4930 JIT(0x175c540)`jfptr_#9_2124 + 32
    frame #12: 0x00007ffff655fd99 libjulia-internal-debug.so.1.11`_jl_invoke(F=0x00007fffdf5a7200, arg
s=0x00007fffe17d2438, nargs=0, mfunc=0x00007fffe06689d0, world=26724) at gf.c:2948:23

I'm not really sure what to look for here but I noticed that 0x7ffec69986d8 is located in a memory region that doesn't have the execution bit

(lldb) memory region 0x7ffec69986d8
[0x00007ffec65a1000-0x00007ffec6999000) rw-

In contrast, if I set a breakpoint at __evaluate_mod_MOD_evaluatef and step to the same location when running prima_serial, the region is executable

(lldb) si
Process 156613 stopped
* thread #1, name = 'julia', stop reason = instruction step into
    frame #0: 0x00007fffffffaa28
->  0x7fffffffaa28: movabsq $0x7ffec06eebd0, %r11 ; imm = 0x7FFEC06EEBD0
    0x7fffffffaa32: movabsq $0x7fffffffaa20, %r10 ; imm = 0x7FFFFFFFAA20
    0x7fffffffaa3c: jmpq   *%r11
    0x7fffffffaa3f: nop
(lldb) memory region 0x7fffffffaa28
[0x00007ffffff9a000-0x00007fffffffe000) rwx [stack]

[pclins]

I’m encountering the same issue when using the BOBYQA routine.

I initially created a post on the Julia Discourse forum before realizing that this GitHub issue already existed. For completeness, here’s the link to that discussion: https://discourse.julialang.org/t/stackoverflowerror-when-using-prima-jl-on-ubuntu-works-fine-on-macos-and-windows/131012

The problem occurs on Ubuntu; the same code runs without issues on macOS and Windows.

[vchuravy]

@andreasnoack as far as I can tell from this issue the issue is a StackOverflow? iIRC tasks run with less stack space that the root-task, so could it be that we are quite literally just running out of space?

From your GDB session the other question I would ask is: Are we sure that we are executing the callback on the same thread as the one where we set the TLS? Or could it be that we are executing the callback on a second thread and thus we get the wrong task?