Conversation
Compile HiGHS + HiPO from source everywhere instead of downloading the
prebuilt static-apache archive. On Linux and Windows, the HiPO BLAS
dependency is satisfied through libblastrampoline (lbt) rather than by
linking OpenBLAS directly: lbt is fetched as a prebuilt binary and staged
under the "openblas" name that HiGHS's own find_package(BLAS)/
BLA_VENDOR=OpenBLAS lookup resolves (confirmed by reading HiGHS's actual
cmake/FindHipoDeps.cmake), so no upstream HiGHS change is needed. OpenBLAS
is no longer compiled by this project (that's what caused the AVX2/AVX512
build breakage this project moved away from before): it's fetched prebuilt
from conda-forge and used purely as the default lbt-forwarded backend,
selected at import time in _blas_backend.py rather than at build time.
On macOS, none of this applies. Reading FindHipoDeps.cmake showed HiGHS
unconditionally links Apple's Accelerate framework there regardless of
anything found via find_package/find_library, so lbt is skipped entirely
and cyhighs's macOS wheels use Accelerate directly. musllinux (Alpine)
wheels are the other exception: conda-forge has no musl builds, so the
default OpenBLAS backend there comes from Alpine's own apk package instead,
discovered via find_library the same way this project sourced OpenBLAS
before adopting lbt.
Getting this working across both the wheel builds (cibuildwheel, with an
auditwheel/delvewheel repair step) and the plain editable build that
tests.yml / uv sync use (no repair step) took several real-CI-driven
fixes, all for the same underlying reason -- the editable build and the
wheel repair each need _core to be able to find its bundled libraries, and
nothing was arranging that:
* HiGHS's own CLI executable statically links libhighs_extras.a in full,
including a call to the OpenBLAS-specific (not standard BLAS/LAPACK ABI)
openblas_set_num_threads() that lbt doesn't provide, so it failed to
link. It's excluded from the build (FetchContent EXCLUDE_FROM_ALL, CMake
3.28+); a no-op stub of that symbol is compiled into _core so the same
latent undefined reference can't crash at runtime either.
* The bundled OpenBLAS and libblastrampoline libraries are installed into
the package directory next to _core, and _core is pointed at them: an
$ORIGIN INSTALL_RPATH on Linux (with FOLLOW_SYMLINK_CHAIN so the real
SONAME target libblastrampoline.so.5 and the OpenBLAS/libgfortran
symlink chains are all installed), and a next-to-the-module copy on
Windows, which has no RPATH equivalent. On Windows the MinGW-only lbt
import library is regenerated for MSVC, locating dumpbin/lib.exe next to
the compiler rather than on PATH.
* _blas_backend.py locates the bundled OpenBLAS relative to _core (via its
import spec, without importing it) rather than relative to its own source
file, so LBT_DEFAULT_LIBS is set correctly even in the editable build
where the compiled extension and the .py sources live in different
directories.
This also adds an optional cyhighs[mkl] extra (Linux x86_64 / Windows
x86_64 only, since Intel doesn't publish MKL for macOS or ARM): if
installed, its bundled MKL runtime is used as the BLAS backend instead of
the bundled OpenBLAS, with no cyhighs rebuild required.
Since every wheel now builds the same way, wheels.yml collapses the old
prebuilt-archive/from-source split into a single cibuildwheel job with a
4-way OS matrix.
The libblastrampoline release tag/version and the per-triplet asset
filenames and hashes are confirmed against a live release listing of
JuliaBinaryWrappers/libblastrampoline_jll.jl; the conda-forge
OpenBLAS/libgfortran coordinates, SONAMEs, transitive dependency chains,
and package layouts are all confirmed against real downloaded and
extracted packages.
Add a section to the bundling guide explaining how to force a specific BLAS backend and how to enable verbose backend-selection diagnostics.
Add a second benchmark run using the cyhighs[mkl] extra and CYHIGHS_LBT_PREFER=mkl, tracked as its own regression series. Skipped on macOS, where MKL is not published and the extra has no effect.
HiGHS calls openblas_get_num_threads/openblas_set_num_threads directly, believing it is linked against real OpenBLAS. Forward both to lbt's own lbt_get_num_threads/lbt_set_num_threads instead of a no-op stub, so HiGHS's thread-count control reaches whichever backend lbt is actually forwarding to, OpenBLAS or MKL.
Off links the bundled OpenBLAS directly with no libblastrampoline forwarding layer in between, for local A/B testing against the default lbt-forwarded build. Both HiGHS's own BLAS detection and _core link straight to the real backend, so no thread-count stub is needed and _blas_backend.py no-ops (no lbt to configure). Real OpenBLAS needs libgfortran resolvable at link time, not just at runtime, unlike lbt which has no such dependency, so the no-lbt shim directory stages it there too. Verified end to end with a real local build: linked, imported, and solved an LP via IPM correctly with no lbt anywhere in the dependency tree (confirmed via ldd), and the full test suite passes against it.
Rebuild with CYHIGHS_USE_LBT=OFF in a separate build-dir and track it as its own regression series, alongside the existing OpenBLAS and MKL runs. Skipped on macOS, where the flag has no effect.
Build a second wheel variant per Linux/Windows target with CYHIGHS_USE_LBT=OFF, tagged with a PEP 440 +static local version segment so it never collides with or gets picked up in place of the default lbt-forwarded wheel on PyPI. These are attached directly to the GitHub release instead, since PyPI has no way for pip to choose between two wheels of the same name, version, and platform tag. Document the static wheels and the existing CYHIGHS_LBT_PREFER/ CYHIGHS_LBT_DEBUG environment variables in the bundling guide.
Windows runners default run: steps to pwsh, which parses the whole
script block as PowerShell rather than passing it through, and pwsh's
backslash escaping rules differ from bash's. That broke the embedded
Python one-liner's quoting, confirmed by a real CI failure ("The
string is missing the terminator") on Wheels on windows-latest
(static).
Replace the scikit-build-core/CMake build that compiled HiGHS + HiPO from source on every platform with a meson-python build that links prebuilt JLL binaries (HiGHS_jll, OpenBLAS32_jll) resolved through meson-jll. The committed subprojects/*.wrap files and meson-jll.lock pin those binaries, so no HiGHS source compile, conda/apk OpenBLAS staging, MSVC import-lib generation, or rpath bundling machinery is needed. meson-python folds the JLL runtime libraries into the wheel automatically. BLAS: libblastrampoline still forwards HiPO's BLAS/LAPACK calls at runtime. _blas_backend.py now selects a backend by preference order (Apple Accelerate on macOS, then MKL, then the bundled OpenBLAS), which is bundled on every platform as the zero-config fallback. OpenBLAS32_jll (LP64), not the plain OpenBLAS_jll, is used deliberately: the plain build is ILP64-only on Windows, and lbt's "64_" suffix mangling breaks the CBLAS-prefixed entry points some HiPO paths call, degrading into an effectively hung solve. Windows needs add_windows_dll_directories() because it has no RPATH equivalent (a meson-python limitation), and lbt's own LoadLibrary calls consult only PATH, so both os.add_dll_directory and PATH are extended before _core is imported. Drop the static (CYHIGHS_USE_LBT=OFF) wheel variant: HiGHS_jll always links lbt, so a no-lbt build cannot be reproduced from it. Remove its CI matrix leg, release-attach job, and benchmark leg.
- Install meson-python before `uv build --sdist`: no-build-isolation-package in pyproject.toml means uv no longer provisions an isolated build env for cyhighs, so the sdist job needs it installed explicitly. - Switch test jobs to `uv sync --no-editable` and add patchelf as a Linux dev dependency: meson-python only relocates the bundled JLL shared libraries and rewrites the extension's rpath to point at them during a real install (mesonpy._rpath.fix_rpath), never for editable installs, which is why the compiled extension couldn't find libblastrampoline at import time. - Fix the macOS and Windows CompilerSupportLibraries packagefiles: some of the bundled libraries (libgcc_s on Darwin, libssp-0 on Windows) ship only under a versioned/DLL-only name with no unversioned symlink or import library, so the plain find_library() lookup used for every other library in the same file fails for these. Fall back to linking the exact file directly when that lookup comes up empty.
- `uv run` re-syncs the environment by default, which was silently redoing
the build in editable mode right after the previous commit's `uv sync
--no-editable`, undoing that fix. Pass --no-sync to every `uv run`
invocation in the test job so it reuses the non-editable install as-is.
- Bundle zlib (dependency('Zlib')) on every platform: libhighs is
dynamically linked against it everywhere, and unlike Linux, macOS and
Windows have no guarantee a compatible copy is already present.
- Fix the same versioned-symlink gap in the macOS Zlib packagefiles that
CompilerSupportLibraries had: the tarball ships libz.1.3.2.dylib but
consumers reference @rpath/libz.1.dylib.
- Bundle libquadmath on macOS when present, needed by libgfortran but not
declared as one of CompilerSupportLibraries' own products.
- Point Linux wheel builds at manylinux_2_34 instead of cibuildwheel's
default manylinux_2_28: the JLL binaries were built with a newer
toolchain than that container's auditwheel policy allows ("too-recent
versioned symbols").
…lly run
- The auditwheel/runtime error narrowed down to an exact, confirmed name
each time: libhighs.so.1, and (from the very first CI run)
libblastrampoline.so.5. Same root cause as the earlier darwin Zlib fix:
these Linux packagefiles install only the unversioned lib*.so dev name,
not the versioned name matching the library's own DT_SONAME that
consumers actually reference. Add install_symlink for libhighs.so.1,
libopenblas.so.0, and libblastrampoline.so.5.
- The previous commit's dependency('Zlib') never actually bundled zlib on
any platform: build environments here all have some system zlib already
(an SDK stub on macOS, the system package on Linux), and dependency()
only falls back to the wrap-provided subproject once a plain system
search comes up empty, which it doesn't. Force the subproject directly
with subproject('Zlib').get_variable('zlib_dep') so its install_data/
install_symlink calls actually run regardless of what's already on the
system.
install_symlink() targets never actually made it into any wheel: I verified locally (a small standalone meson project, no JLL binaries needed) that meson's own intro-install_plan.json --- what meson-python reads to decide which files go in the wheel --- has no entry at all for install_symlink() targets, only for install_data()/install target entries. That's why the previous commit's libhighs.so.1 (and the other soname-compat) symlinks had no effect: they were silently invisible to meson-python's wheel manifest. fs.copyfile(..., install: true, install_dir: ...) creates a real, independently install-plan-tracked file instead of a symlink, so it shows up correctly. Switch every previously-added install_symlink() call (HiGHS/OpenBLAS32/libblastrampoline soname names on Linux, libz.1.dylib on macOS) to that.
…enBLAS32/lbt) Same gap as the last two commits, now confirmed for libssp.so.0 on Linux (the Test job got past libhighs/libopenblas/libblastrampoline once those were fixed, and hit the next one in CompilerSupportLibraries) and libhighs.1.dylib on macOS (delocate, now that zlib bundles correctly). Fixed the confirmed ones plus, proactively, every other library in the same two packagefiles that has the identical full-version-vs-soname naming gap (libatomic/libgcc_s/libgfortran/libgomp/libstdc++ on Linux; OpenBLAS32 and libblastrampoline on macOS), using the same well-established GCC/glibc soname conventions already confirmed for the others, to avoid re-discovering each one via another CI round trip.
…rash
- macOS Test/Wheels jobs now fail with "duplicate LC_RPATH
'@loader_path/../.cyhighs.mesonpy.libs'" once _core actually loads far
enough to hit dyld. meson-python's fix_rpath rewrites every distinct
original @loader_path-relative rpath entry to that same final string,
so any two build-tree directories that collapse onto it produce a
duplicate. _core's own dependencies list linked the Zlib subproject
directly, on top of it already coming in transitively through HiGHS's
own declare_dependency() -- forcing subproject('Zlib') to configure
it (for its bundling side effects) doesn't require also linking it a
second time. Drop the redundant direct link.
- The Linux wheel job now gets past auditwheel's symbol-version check
entirely, but crashes inside its own dependency-tree walk with
"ELFError: Magic number does not match" -- it's resolving some SONAME
to a file that isn't a real ELF (auditwheel's non-verbose output
doesn't say which). Add -v to the repair command to find out before
guessing at a fix.
- The previous commit's fix (removing the redundant direct Zlib link)
didn't touch the actual cause: _core links against several JLL
libraries that each live in their own distinct subproject directory
(HiGHS's own, OpenBLAS32's, Zlib's, ...), so meson gives it one
distinct @loader_path-relative build-tree RPATH entry per directory
regardless. meson-python's own install-time RPATH fix-up rewrites
every one of those to the identical final destination, so they
collapse into duplicate LC_RPATH commands that dyld refuses to load.
Add a meson.add_install_script() that de-dupes them after every
`meson install`, covering both the wheel-build path and the
`uv sync --no-editable` dev path uniformly, since both go through the
same underlying meson install step.
- auditwheel's -v is a global flag on the auditwheel command itself, not
the repair subcommand ("auditwheel -v repair ...", not
"auditwheel repair -v ...") -- the previous commit had it in the wrong
place and cibuildwheel's invocation just errored on it directly.
meson.add_install_script() runs as part of `meson install` itself, which happens *before* meson-python's own install-time RPATH rewrite (mesonpy._rpath.fix_rpath, invoked from its own Python code afterward, not a subprocess meson can hook into). The duplicate entries this is meant to clean up don't exist yet at that point -- each subproject directory's original, distinct rpath entry is still just that, distinct -- so the previous commit's install script had nothing to do and ran as a no-op. Move the fix to run after the fact instead, wherever a `_core` extension module actually ends up: - wheels.yml: as an extra step in CIBW_REPAIR_WHEEL_COMMAND_MACOS, after delocate, operating on the wheel it just wrote (delocate does its own rpath/vendoring work, so this has to run last). The script now handles wheels directly (extract, fix the bundled _core*.so, rezip). - tests.yml: directly against the installed .venv copy, right after the `uv sync --no-editable` step that produces it and before pytest tries to import it.
…eel diagnostic
- The macOS delocate repair itself now succeeds (all libquadmath/libz.1/
libhighs.1 relocations land correctly); the only failure was that
cibuildwheel's repair command has no {project} placeholder, so the
fix_duplicate_macos_rpaths.py path expanded to a literal "{project}/...".
The macOS build runs natively (no container) and inherits the runner
environment, so reference the checkout root through $GITHUB_WORKSPACE
instead.
- On Linux, auditwheel still crashes in its own dependency walk with
"ELFError: Magic number does not match" and even -v does not name the
offending file. Prepend a temporary diagnostic that lists every ".so"
member of the built wheel with its first four bytes, to catch a bundled
member that is a text linker script rather than a real ELF. Both the
diagnostic and -v come back out once the culprit is identified.
On Windows and Linux, cibuildwheel repairs the built wheel with delvewheel or auditwheel, which copies the libraries the extension links (including libblastrampoline) into a `cyhighs.libs` directory, name-mangled, and rewrites the extension to load those copies. That mangled copy is the lbt instance actually resident at runtime. _library_search_dirs only looked in `.cyhighs.mesonpy.libs`, so _open_loaded_lbt found the original unmangled name there, queried GetModuleHandleW/RTLD_NOLOAD for it, got nothing (the loaded module is the mangled name in cyhighs.libs), and returned None. configure_blas_backend then skipped lbt_forward entirely, leaving HiPO with no BLAS backend, which segfaulted on the first solve in the Windows CI test step. Restore `cyhighs.libs` to the search directories, which an earlier simplification of this module had dropped. Verified against a locally delvewheel-repaired wheel: lbt now binds the mangled instance, forwards to the bundled OpenBLAS, and the full test suite passes.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'benchmarks-macos-latest'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: ddd362a | Previous: 8a22b41 | Ratio |
|---|---|---|---|
benchmarks/test_benchmarks.py::test_array_solve[lp-100] |
738.1436778980014 iter/sec (stddev: 0.00014339458124977864) |
1321.1563235840813 iter/sec (stddev: 0.00007036664582816934) |
1.79 |
benchmarks/test_benchmarks.py::test_array_solve[lp-1000] |
25.534614978134197 iter/sec (stddev: 0.006320657185484664) |
41.32705316964579 iter/sec (stddev: 0.0010590375713438864) |
1.62 |
benchmarks/test_benchmarks.py::test_array_solve[mip-1000] |
0.0733727677610343 iter/sec (stddev: 1.5617864888631714) |
0.10026955485086086 iter/sec (stddev: 1.3656391960338892) |
1.37 |
benchmarks/test_benchmarks.py::test_sparse_solve[mip-100] |
136.35031121963732 iter/sec (stddev: 0.0011220303721090413) |
179.34901222742988 iter/sec (stddev: 0.0009568028078864596) |
1.32 |
benchmarks/test_benchmarks.py::test_linprog_solve[lp-100] |
357.29863934701405 iter/sec (stddev: 0.0007876761750761731) |
617.1346573555685 iter/sec (stddev: 0.00014554380954616688) |
1.73 |
benchmarks/test_benchmarks.py::test_linprog_solve[lp-1000] |
28.17451335268248 iter/sec (stddev: 0.004881213066703258) |
39.63002738003352 iter/sec (stddev: 0.0009300394151651562) |
1.41 |
benchmarks/test_benchmarks.py::test_linprog_solve[mip-100] |
100.82427779225 iter/sec (stddev: 0.001999870020800495) |
188.05240629409806 iter/sec (stddev: 0.0006986071112258156) |
1.87 |
This comment was automatically generated by workflow using github-action-benchmark.
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'benchmarks-ubuntu-latest'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: db901df | Previous: 8a22b41 | Ratio |
|---|---|---|---|
benchmarks/test_benchmarks.py::test_array_solve[lp-100] |
638.5393896760701 iter/sec (stddev: 0.00005284364230169796) |
777.233101456788 iter/sec (stddev: 0.00007331256211535615) |
1.22 |
benchmarks/test_benchmarks.py::test_array_solve[lp-1000] |
28.57114441098931 iter/sec (stddev: 0.0014731183893328039) |
34.82034632086863 iter/sec (stddev: 0.0011065957488331692) |
1.22 |
benchmarks/test_benchmarks.py::test_array_solve[lp-10000] |
0.10698878589671228 iter/sec (stddev: 0.025402952744289088) |
0.13090608970390633 iter/sec (stddev: 0.0732322434031961) |
1.22 |
benchmarks/test_benchmarks.py::test_array_solve[mip-100] |
154.18994761409365 iter/sec (stddev: 0.0002412686943724364) |
192.67692428504841 iter/sec (stddev: 0.00031427658116649925) |
1.25 |
benchmarks/test_benchmarks.py::test_sparse_solve[lp-10000] |
0.10699259535562138 iter/sec (stddev: 0.03030978330707275) |
0.1315023074896695 iter/sec (stddev: 0.08971279272085499) |
1.23 |
benchmarks/test_benchmarks.py::test_sparse_solve[mip-100] |
150.37083628612544 iter/sec (stddev: 0.00031468023871623213) |
183.30126233304014 iter/sec (stddev: 0.0000602145582712725) |
1.22 |
benchmarks/test_benchmarks.py::test_linprog_solve[lp-100] |
577.2244324830652 iter/sec (stddev: 0.000022513868262059065) |
766.8958074624117 iter/sec (stddev: 0.0000230580133124934) |
1.33 |
benchmarks/test_benchmarks.py::test_linprog_solve[lp-10000] |
0.10469901326677938 iter/sec (stddev: 0.10722690148307003) |
0.131812978927733 iter/sec (stddev: 0.09057955846089433) |
1.26 |
benchmarks/test_benchmarks.py::test_linprog_solve[mip-100] |
151.2858640353746 iter/sec (stddev: 0.00008707891899115422) |
198.77732070009353 iter/sec (stddev: 0.000047917507329365796) |
1.31 |
This comment was automatically generated by workflow using github-action-benchmark.
The temporary wheel-member diagnostic pinpointed it: in the bundled libs
dir, every library was b'\x7fELF' except libgcc_s.so and libgcc_s.so.1,
which both began b'/* G' -- a GNU ld linker script
(GROUP(libgcc_s.so.1 -lgcc), the standard GCC layout), not an ELF. In this
JLL tarball lib/libgcc_s.so is that text script; the real shared object is
lib/libgcc_s.so.1. The generated packagefile's install_data('lib/libgcc_s.so')
bundled the script, and the earlier soname-copy fix propagated it into
libgcc_s.so.1 as well, so the wheel shipped two text files where shared
objects belong -- which is what crashed auditwheel's ELF walk (and would
have failed the loader at runtime). Source both bundled copies from the real
lib/libgcc_s.so.1 instead. Every other library in the tarball ships lib/*.so
as a real ELF, so this is the one special case.
Also drops the now-diagnosed temporary auditwheel -v / member-listing repair
command, returning Linux to cibuildwheel's default (non-verbose output still
carries tracebacks if anything else surfaces).
cibuildwheel builds both manylinux (glibc) and musllinux wheels for Linux, using separate per-triplet packagefiles. The earlier fixes only touched the -linux-gnu ones, so the musllinux build still shipped only unversioned lib*.so files and auditwheel failed with "required library libhighs.so.1 could not be located". Mirror every fix onto the -linux-musl x86_64 and aarch64 packagefiles: versioned soname copies for libhighs/libopenblas/ libblastrampoline and the CompilerSupportLibraries runtime libs, plus sourcing libgcc_s from the real lib/libgcc_s.so.1 ELF rather than its GNU ld linker script. musl's CompilerSupportLibraries ships no libssp, so that one is simply absent here (five libs, not six).
…egen
The meson-jll v0.0.3 regeneration replaced the previous hand-patched
CompilerSupportLibraries packagefiles with mechanical cc.find_library()
calls that assume every declared library has an unversioned dev symlink
(or, on Windows MinGW, an import library) sitting in lib/. Two products
in this JLL never ship one:
- libgcc_s on macOS (both darwin arches): only the versioned
libgcc_s.1.1.dylib exists, no libgcc_s.dylib symlink, so
find_library('gcc_s') failed outright.
- libssp on Windows (MinGW, non-MSVC path): only bin/libssp-0.dll ships,
no lib/libssp-0.dll.a import library, so find_library('ssp-0') failed.
Restore the required:false + declare_dependency(link_args: [...]) fallback
for every CompilerSupportLibraries product on both macOS variants, and
the bin/*.dll direct-link fallback for libssp specifically on Windows,
matching what was already proven working before the regen.
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'benchmarks-windows-latest'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: ddd362a | Previous: 8a22b41 | Ratio |
|---|---|---|---|
benchmarks/test_benchmarks.py::test_sparse_solve[mip-100] |
87.997416396014 iter/sec (stddev: 0.002988882823651534) |
110.79080259088542 iter/sec (stddev: 0.00027728557200366205) |
1.26 |
This comment was automatically generated by workflow using github-action-benchmark.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.