From cbbcfa8d10def634bd6f00d1c4e49170854b6058 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 16 Jul 2026 19:24:35 +0000 Subject: [PATCH 1/9] Build HiGHS's BLAS through libblastrampoline on all platforms Compile HiGHS + HiPO from source everywhere instead of downloading the prebuilt static-apache archive. On Linux and Windows, the HiPO BLAS dependency is satisfied through libblastrampoline (lbt) rather than by linking OpenBLAS directly: lbt is fetched as a prebuilt binary and staged under the "openblas" name that HiGHS's own find_package(BLAS)/ BLA_VENDOR=OpenBLAS lookup resolves (confirmed by reading HiGHS's actual cmake/FindHipoDeps.cmake), so no upstream HiGHS change is needed. OpenBLAS is no longer compiled by this project (that's what caused the AVX2/AVX512 build breakage this project moved away from before): it's fetched prebuilt from conda-forge and used purely as the default lbt-forwarded backend, selected at import time in _blas_backend.py rather than at build time. On macOS, none of this applies. Reading FindHipoDeps.cmake showed HiGHS unconditionally links Apple's Accelerate framework there regardless of anything found via find_package/find_library, so lbt is skipped entirely and cyhighs's macOS wheels use Accelerate directly. musllinux (Alpine) wheels are the other exception: conda-forge has no musl builds, so the default OpenBLAS backend there comes from Alpine's own apk package instead, discovered via find_library the same way this project sourced OpenBLAS before adopting lbt. Getting this working across both the wheel builds (cibuildwheel, with an auditwheel/delvewheel repair step) and the plain editable build that tests.yml / uv sync use (no repair step) took several real-CI-driven fixes, all for the same underlying reason -- the editable build and the wheel repair each need _core to be able to find its bundled libraries, and nothing was arranging that: * HiGHS's own CLI executable statically links libhighs_extras.a in full, including a call to the OpenBLAS-specific (not standard BLAS/LAPACK ABI) openblas_set_num_threads() that lbt doesn't provide, so it failed to link. It's excluded from the build (FetchContent EXCLUDE_FROM_ALL, CMake 3.28+); a no-op stub of that symbol is compiled into _core so the same latent undefined reference can't crash at runtime either. * The bundled OpenBLAS and libblastrampoline libraries are installed into the package directory next to _core, and _core is pointed at them: an $ORIGIN INSTALL_RPATH on Linux (with FOLLOW_SYMLINK_CHAIN so the real SONAME target libblastrampoline.so.5 and the OpenBLAS/libgfortran symlink chains are all installed), and a next-to-the-module copy on Windows, which has no RPATH equivalent. On Windows the MinGW-only lbt import library is regenerated for MSVC, locating dumpbin/lib.exe next to the compiler rather than on PATH. * _blas_backend.py locates the bundled OpenBLAS relative to _core (via its import spec, without importing it) rather than relative to its own source file, so LBT_DEFAULT_LIBS is set correctly even in the editable build where the compiled extension and the .py sources live in different directories. This also adds an optional cyhighs[mkl] extra (Linux x86_64 / Windows x86_64 only, since Intel doesn't publish MKL for macOS or ARM): if installed, its bundled MKL runtime is used as the BLAS backend instead of the bundled OpenBLAS, with no cyhighs rebuild required. Since every wheel now builds the same way, wheels.yml collapses the old prebuilt-archive/from-source split into a single cibuildwheel job with a 4-way OS matrix. The libblastrampoline release tag/version and the per-triplet asset filenames and hashes are confirmed against a live release listing of JuliaBinaryWrappers/libblastrampoline_jll.jl; the conda-forge OpenBLAS/libgfortran coordinates, SONAMEs, transitive dependency chains, and package layouts are all confirmed against real downloaded and extracted packages. --- .github/workflows/wheels.yml | 140 ++----- CMakeLists.txt | 729 +++++++++++++++++++++++++---------- docs/guide/bundling.md | 93 +++-- docs/guide/installation.md | 56 ++- pyproject.toml | 12 +- src/cyhighs/__init__.py | 19 +- src/cyhighs/_blas_backend.py | 204 ++++++++++ src/cyhighs/_core.pyx | 4 + uv.lock | 87 +++++ 9 files changed, 959 insertions(+), 385 deletions(-) create mode 100644 src/cyhighs/_blas_backend.py diff --git a/.github/workflows/wheels.yml b/.github/workflows/wheels.yml index 53da0b3..43a9aed 100644 --- a/.github/workflows/wheels.yml +++ b/.github/workflows/wheels.yml @@ -20,109 +20,12 @@ concurrency: cancel-in-progress: true jobs: - # Linux wheels embed the prebuilt HiGHS static-apache archive, which is built - # against a recent glibc and needs glibc >= 2.38 to link. No manylinux - # container image provides that for x86_64, so Linux wheels are built directly - # on the native runners (Ubuntu 24.04 ships glibc 2.39) and repaired with - # auditwheel, which tags them manylinux_2_38+. - build_linux: - name: Linux ${{ matrix.os }} cp${{ matrix.python }} - runs-on: ${{ matrix.os }} - strategy: - fail-fast: false - matrix: - os: - - ubuntu-24.04 # Linux x86_64 - - ubuntu-24.04-arm # Linux aarch64 - python: ["3.11", "3.12", "3.13", "3.14"] - steps: - - uses: actions/checkout@v4 - - - name: Set up Python ${{ matrix.python }} - uses: actions/setup-python@v5 - with: - python-version: ${{ matrix.python }} - - - name: Build and repair wheel - run: | - set -euxo pipefail - python -m pip install --upgrade pip build auditwheel patchelf - python -m build --wheel --outdir dist - # auditwheel bundles the external libraries (libstdc++, libgcc_s, - # libz) into the wheel and applies the highest compatible manylinux - # tag for the referenced glibc symbols. - auditwheel repair dist/*.whl --wheel-dir wheelhouse - - - name: Test the repaired wheel - run: | - set -euxo pipefail - python -m pip install wheelhouse/*.whl - python -m pip install pytest scipy pytest-markdown-docs - pytest "${GITHUB_WORKSPACE}/tests" "${GITHUB_WORKSPACE}/docs" - - - name: Upload wheel artifacts - uses: actions/upload-artifact@v4 - with: - name: wheels-linux-${{ matrix.os }}-cp${{ matrix.python }} - path: ./wheelhouse/*.whl - - # Portable Linux wheels built from source inside cibuildwheel's default - # containers, which are manylinux_2_28 (x86_64/aarch64) and musllinux_1_2. - # Compiling HiGHS in the container links against the container glibc/musl, so - # these wheels carry the portable manylinux_2_28 / musllinux tags that the - # prebuilt-archive job above (glibc >= 2.38) cannot produce. A prebuilt - # OpenBLAS is installed into the container so only the HiGHS solver and HiPO - # are compiled from source. - build_linux_source: - name: Linux source ${{ matrix.os }} - runs-on: ${{ matrix.os }} - strategy: - fail-fast: false - matrix: - os: - - ubuntu-24.04 # x86_64 -> manylinux_2_28 + musllinux_1_2 - - ubuntu-24.04-arm # aarch64 -> manylinux_2_28 + musllinux_1_2 - steps: - - uses: actions/checkout@v4 - - - name: Build wheels - uses: pypa/cibuildwheel@v3.2.0 - env: - # Build CPython 3.11 through 3.14. The free threaded builds are left - # out for now while Cython support for them is still settling. - CIBW_BUILD: "cp311-* cp312-* cp313-* cp314-*" - # Skip 32-bit targets. HiGHS is not viable on 32-bit and there are no - # 32-bit SciPy wheels for the test step. - CIBW_SKIP: "*_i686" - # Install a prebuilt OpenBLAS so only the HiGHS solver and HiPO are - # compiled from source. manylinux_2_28 is AlmaLinux 8 (dnf, with - # openblas-devel from EPEL); musllinux_1_2 is Alpine (apk, openblas-dev - # from the community repo). One command covers both container types. - CIBW_BEFORE_ALL: >- - if command -v dnf; then dnf install -y epel-release && dnf install -y openblas-devel; - elif command -v yum; then yum install -y epel-release && yum install -y openblas-devel; - elif command -v apk; then apk add --no-cache openblas-dev; - fi - # Build HiGHS from source (linking the prebuilt OpenBLAS above). - # scikit-build-core forwards CMAKE_ARGS to the cmake configure step. - CIBW_ENVIRONMENT: 'CMAKE_ARGS="-DCYHIGHS_HIGHS_FROM_SOURCE=ON"' - # Verify each freshly built wheel by running the test suite against it. - CIBW_TEST_REQUIRES: "pytest scipy pytest-markdown-docs" - CIBW_TEST_COMMAND: "pytest {project}/tests {project}/docs" - # scipy does not publish musllinux wheels for every CPython yet, so the - # in-container test install can fail there. The wheels are still built; - # drop this once scipy musllinux coverage is confirmed for all targets. - CIBW_TEST_SKIP: "*-musllinux_*" - - - name: Upload wheel artifacts - uses: actions/upload-artifact@v4 - with: - name: wheels-linux-source-${{ matrix.os }} - path: ./wheelhouse/*.whl - - # macOS and Windows wheels are built with cibuildwheel, which handles the - # per-Python builds and the delocate/delvewheel repair. glibc is not a concern - # on these platforms, so the prebuilt archives build in the standard images. + # Every wheel flavor builds the same way now (see CMakeLists.txt): HiGHS + + # HiPO compile from source, and OpenBLAS + libblastrampoline are fetched as + # prebuilt binaries. That means a single cibuildwheel job with an OS matrix + # covers every platform, with cibuildwheel picking the right container + # (manylinux_2_28/musllinux_1_2 on Linux) or native toolchain (macOS, + # Windows) per matrix entry. build_wheels: name: Wheels on ${{ matrix.os }} runs-on: ${{ matrix.os }} @@ -130,6 +33,8 @@ jobs: fail-fast: false matrix: os: + - ubuntu-24.04 # Linux x86_64 -> manylinux_2_28 + musllinux_1_2 + - ubuntu-24.04-arm # Linux aarch64 -> manylinux_2_28 + musllinux_1_2 - macos-latest # macOS arm64 - windows-latest # Windows AMD64 steps: @@ -142,16 +47,31 @@ jobs: # out for now while Cython support for them is still settling. CIBW_BUILD: "cp311-* cp312-* cp313-* cp314-*" # Skip 32-bit targets. HiGHS is not viable on 32-bit and there are no - # 32-bit SciPy wheels for the test step. + # 32-bit SciPy wheels for the test step. "*-win32" is a no-op outside + # the Windows matrix entry. CIBW_SKIP: "*_i686 *-win32" - # HiGHS is C++17. macOS arm64 requires a deployment target of at least - # 11.0 for the C++17 standard library features HiGHS relies on. + # HiGHS is C++17. macOS arm64 requires a deployment target of at + # least 11.0 for the C++17 standard library features HiGHS relies + # on. Unused outside the macOS matrix entry. MACOSX_DEPLOYMENT_TARGET: "11.0" - # Verify each freshly built wheel by running the test suite against it. - # pytest-markdown-docs is required because the project addopts enable it, - # and it lets the documentation examples run against the built wheel too. + # Verify each freshly built wheel by running the test suite against + # it. pytest-markdown-docs is required because the project addopts + # enable it, and it lets the documentation examples run against the + # built wheel too. CIBW_TEST_REQUIRES: "pytest scipy pytest-markdown-docs" CIBW_TEST_COMMAND: "pytest {project}/tests {project}/docs" + # scipy does not publish musllinux wheels for every CPython yet, so + # the in-container test install can fail there. The wheels are + # still built; drop this once scipy musllinux coverage is confirmed + # for all targets. Unused outside the Linux matrix entries. + CIBW_TEST_SKIP: "*-musllinux_*" + # conda-forge (CMakeLists.txt's default OpenBLAS source) only ships + # glibc binaries, so the musllinux_1_2 container needs its own, + # Alpine-native OpenBLAS instead, found via find_library. The + # _LINUX suffix means cibuildwheel only runs this for the Linux + # matrix entries; within those, it's a no-op in the manylinux_2_28 + # container, which has no apk. + CIBW_BEFORE_ALL_LINUX: "command -v apk && apk add --no-cache openblas-dev || true" - name: Upload wheel artifacts uses: actions/upload-artifact@v4 @@ -176,7 +96,7 @@ jobs: publish: name: Publish to PyPI - needs: [build_linux, build_linux_source, build_wheels, build_sdist] + needs: [build_wheels, build_sdist] if: github.event_name == 'release' runs-on: ubuntu-latest environment: diff --git a/CMakeLists.txt b/CMakeLists.txt index 3ac1b24..da4fc1a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,22 +1,58 @@ # CMake build for the cyhighs Cython extension. # -# There are two ways to acquire HiGHS, selected by CYHIGHS_HIGHS_FROM_SOURCE: +# HiGHS + HiPO are always compiled from source (FetchContent) on every platform. # -# * OFF (default): download a prebuilt HiGHS "static-apache" release archive -# for the target platform, extract it, and link the static libraries it -# ships (libhighs, libhighs_extras and the bundled OpenBLAS). The Linux -# archives are built against glibc >= 2.38, so this path is used on the -# native runners (producing manylinux_2_39) and on macOS/Windows. +# On Linux and Windows, HiPO's BLAS dependency is satisfied by +# libblastrampoline (lbt) rather than by linking OpenBLAS directly: lbt is a +# small MIT-licensed shared library that exports the standard BLAS/LAPACK ABI +# and forwards each call, at runtime, to whichever real BLAS implementation is +# named by the LBT_DEFAULT_LIBS environment variable (this is the same +# mechanism Julia's LinearAlgebra/HiGHS.jl use to switch BLAS backends without +# recompiling). cyhighs's own Python `__init__.py` sets that variable before +# importing the compiled extension: # -# * ON: compile the HiGHS solver and HiPO from source with FetchContent, -# linking a prebuilt OpenBLAS already installed in the build environment -# (HiGHS's default BUILD_OPENBLAS=OFF). This path is used inside the -# manylinux_2_28 / musllinux_1_2 containers, where the prebuilt archive -# cannot link, to produce the more portable manylinux_2_28 / musllinux -# wheels. +# * by default, to a prebuilt OpenBLAS bundled inside the wheel (so the +# package remains fully self contained with zero configuration, matching +# the project's historical "no external system dependencies" promise); +# * to Intel MKL instead, if the optional `cyhighs[mkl]` extra is installed +# and its shared library can be located (Linux x86_64 / Windows x86_64 +# only -- Intel has never published MKL for macOS or for ARM). # -# Either way the result is a single self contained extension module. -cmake_minimum_required(VERSION 3.24) +# On macOS, none of this applies. Reading HiGHS's own cmake/FindHipoDeps.cmake +# shows it unconditionally links Apple's Accelerate framework on Apple +# platforms (in highs_link_blas()), regardless of anything found via +# find_package/find_library, so lbt would have nothing to intercept there. +# cyhighs's macOS wheels use Accelerate directly, the same as before this +# project adopted lbt. +# +# HiGHS's FindHipoDeps.cmake does not call find_library(NAMES openblas) +# directly (also confirmed by reading that file). With no BLAS_LIBRARIES/ +# BLA_VENDOR set, it tries find_package(OpenBLAS CONFIG) first (finds nothing, +# since neither lbt nor the bundled OpenBLAS ships a CMake package config), +# then sets BLA_VENDOR=OpenBLAS and calls find_package(BLAS). CMake's own +# FindBLAS.cmake module handles that vendor with an internal +# find_library(NAMES openblas) that does respect CMAKE_LIBRARY_PATH/ +# CMAKE_PREFIX_PATH, so this file copies the fetched lbt library into a +# directory prepended to those two variables, under the "openblas" name, so +# that lookup resolves to lbt instead of a real BLAS -- with no upstream +# HiGHS source change needed, just via CMake's own module rather than a +# literal find_library() call in HiGHS's own code. +# +# musllinux (Alpine) wheels are the one exception on Linux: conda-forge only +# ships glibc binaries, so the *default* OpenBLAS backend there comes from +# Alpine's own package (installed via apk in CI, see wheels.yml) instead of +# conda-forge, found via find_library the same way this project found +# OpenBLAS before adopting lbt. lbt itself is still fetched from +# JuliaBinaryWrappers, which does publish musl-tagged assets. +# +# The libblastrampoline release tag/version and the individual asset +# filenames and hashes below are all confirmed against a live release listing +# of JuliaBinaryWrappers/libblastrampoline_jll.jl (as is the MinGW-only +# import library situation handled by cyhighs_generate_msvc_import_lib()). +# The conda-forge OpenBLAS/libgfortran5 coordinates and the Windows/Linux +# package layouts are pinned and confirmed the same way, against real +# repodata.json hashes and real extracted packages. +cmake_minimum_required(VERSION 3.28) project(cyhighs LANGUAGES C CXX) @@ -26,7 +62,7 @@ set(CMAKE_CXX_STANDARD 17) set(CMAKE_CXX_STANDARD_REQUIRED ON) # Position independent code is required to link the static HiGHS libraries into -# the shared extension module. The release archives are already built PIC. +# the shared extension module. set(CMAKE_POSITION_INDEPENDENT_CODE ON) # --------------------------------------------------------------------------- @@ -37,215 +73,431 @@ find_package( COMPONENTS Interpreter Development.Module REQUIRED) -# The HiGHS release the wheel is built against. Used as the prebuilt archive -# version and as the FetchContent git tag (v${HIGHS_VERSION}). +# The HiGHS release built from source. Used as the FetchContent git tag +# (v${HIGHS_VERSION}). set(HIGHS_VERSION 1.15.1) -# Build HiGHS from source instead of using the prebuilt static-apache archive. -# Enabled inside the manylinux_2_28 / musllinux containers so that the resulting -# wheel links against the container glibc/musl and carries the portable -# manylinux_2_28 / musllinux tag. -option(CYHIGHS_HIGHS_FROM_SOURCE - "Build HiGHS from source (manylinux_2_28 / musllinux) instead of the prebuilt static-apache archive" - OFF) - -# These are filled in by whichever acquisition path runs below and consumed by -# the common extension-build section that follows. -set(_highs_include_dir "") -set(_highs_link_libraries "") -set(_highs_license_file "") +# The libblastrampoline release fetched as a prebuilt binary. The version, +# the release tag's "+" suffix, and the per-triplet asset hashes below +# are all confirmed real against a live release listing of +# https://github.com/JuliaBinaryWrappers/libblastrampoline_jll.jl +# (tag libblastrampoline-v5.15.0+0). +set(CYHIGHS_LBT_VERSION 5.15.0) +set(CYHIGHS_LBT_BUILD 0) + +set(_downloads_dir "${CMAKE_BINARY_DIR}/_cyhighs_downloads") +file(MAKE_DIRECTORY "${_downloads_dir}") -if(CYHIGHS_HIGHS_FROM_SOURCE) - # ------------------------------------------------------------------------- - # Compile HiGHS + HiPO from source, linking a prebuilt OpenBLAS. - # ------------------------------------------------------------------------- - include(FetchContent) - - # Build settings applied to the HiGHS subproject. These are set as cache - # variables before the fetch so that HiGHS picks them up during configuration. - set(BUILD_SHARED_LIBS OFF CACHE BOOL "" FORCE) - set(FAST_BUILD ON CACHE BOOL "" FORCE) - set(BUILD_TESTING OFF CACHE BOOL "" FORCE) - set(BUILD_EXAMPLES OFF CACHE BOOL "" FORCE) - set(ZLIB OFF CACHE BOOL "" FORCE) - - # HiPO is the interior point solver added in HiGHS 1.15. It requires FAST_BUILD - # and a BLAS. BUILD_OPENBLAS is deliberately left at HiGHS's default of OFF: we - # link the prebuilt OpenBLAS installed in the build environment (openblas-devel - # on manylinux, openblas-dev on musllinux). HiGHS's cmake/FindHipoDeps.cmake - # discovers it with find_library(NAMES openblas) on the default system paths, - # so no BLAS source compile and no NO_AVX2/NO_AVX512 workaround are needed here. - # - # BUILD_SHARED_EXTRAS_LIB must be OFF. HiGHS places the HiPO numerical - # dependencies (AMD, BLAS, METIS, RCM) in a separate highs_extras library. When - # that library is shared, HiGHS loads it at runtime from a highs_extras.so that - # would have to be shipped alongside the wheel. Building it static instead links - # the HiPO code directly into our extension, so the wheel stays self contained. - set(HIPO ON CACHE BOOL "" FORCE) - set(BUILD_SHARED_EXTRAS_LIB OFF CACHE BOOL "" FORCE) - - FetchContent_Declare( - highs - GIT_REPOSITORY https://github.com/ERGO-Code/HiGHS.git - GIT_TAG v${HIGHS_VERSION} - GIT_SHALLOW ON) - FetchContent_MakeAvailable(highs) - - # The C API header lives in the highs/interfaces directory of the source tree. - set(_highs_include_dir "${highs_SOURCE_DIR}/highs") - # The exported target transitively pulls highs_extras and the discovered - # OpenBLAS. The prebuilt libopenblas.so becomes a NEEDED entry on the - # extension, which auditwheel then vendors into the wheel. - set(_highs_link_libraries "highs::highs") - - # Bundle the upstream HiGHS license from the fetched source tree. - foreach(_candidate LICENSE.txt LICENSE) - if(EXISTS "${highs_SOURCE_DIR}/${_candidate}") - set(_highs_license_file "${highs_SOURCE_DIR}/${_candidate}") - break() +# --------------------------------------------------------------------------- +# Small helper: download (with a pinned hash when we have one) and extract an +# archive exactly once per build directory. +# --------------------------------------------------------------------------- +function(cyhighs_fetch_and_extract url sha256 dest_dir) + if(EXISTS "${dest_dir}/.cyhighs_extracted") + return() + endif() + get_filename_component(_local_name "${url}" NAME) + set(_local_path "${_downloads_dir}/${_local_name}") + if(NOT EXISTS "${_local_path}") + message(STATUS "Downloading ${url}") + if(sha256 STREQUAL "") + message(WARNING "No pinned SHA256 for ${_local_name}; downloading without an integrity check.") + file(DOWNLOAD "${url}" "${_local_path}" TLS_VERIFY ON STATUS _dl) + else() + file(DOWNLOAD "${url}" "${_local_path}" EXPECTED_HASH SHA256=${sha256} TLS_VERIFY ON STATUS _dl) endif() + list(GET _dl 0 _dl_code) + if(NOT _dl_code EQUAL 0) + list(GET _dl 1 _dl_msg) + file(REMOVE "${_local_path}") + message(FATAL_ERROR "Failed to download ${url}: ${_dl_msg}") + endif() + endif() + file(MAKE_DIRECTORY "${dest_dir}") + file(ARCHIVE_EXTRACT INPUT "${_local_path}" DESTINATION "${dest_dir}") + file(WRITE "${dest_dir}/.cyhighs_extracted" "") +endfunction() + +# A .conda package is a zip container holding pkg-*.tar.zst (the payload) and +# info-*.tar.zst (metadata, discarded here). This unwraps both layers so +# dest_dir ends up with the normal lib/, bin/, Library/ layout. +function(cyhighs_fetch_conda_package url sha256 dest_dir) + if(EXISTS "${dest_dir}/.cyhighs_extracted") + return() + endif() + set(_conda_layer "${dest_dir}_conda_layer") + cyhighs_fetch_and_extract("${url}" "${sha256}" "${_conda_layer}") + file(GLOB _pkg_tarball "${_conda_layer}/pkg-*.tar.zst") + if(NOT _pkg_tarball) + message(FATAL_ERROR "No pkg-*.tar.zst payload found after extracting ${url}") + endif() + list(GET _pkg_tarball 0 _pkg_tarball) + file(MAKE_DIRECTORY "${dest_dir}") + file(ARCHIVE_EXTRACT INPUT "${_pkg_tarball}" DESTINATION "${dest_dir}") + file(WRITE "${dest_dir}/.cyhighs_extracted" "") +endfunction() + +# Generate an MSVC-compatible import library from a DLL's own export table, +# using dumpbin + lib.exe (both ship with MSVC). Needed on Windows because +# BinaryBuilder cross-compiles JLL artifacts with MinGW, which produces a +# *.dll.a import library that MSVC cannot consume directly. +# +# cibuildwheel's Windows build runs in a plain shell, not an MSVC developer +# command prompt, so dumpbin/lib.exe are not on PATH (confirmed by a real CI +# failure: "dumpbin/lib.exe not found"). CMAKE_C_COMPILER is always the +# resolved absolute path to cl.exe once project() has run, though, and MSVC +# always installs dumpbin.exe/lib.exe in that exact same toolset directory +# (.../VC/Tools/MSVC//bin/Hostx64/x64/), so look there directly +# instead of relying on PATH. +function(cyhighs_generate_msvc_import_lib dll_path def_path out_lib_path) + get_filename_component(_msvc_bin_dir "${CMAKE_C_COMPILER}" DIRECTORY) + find_program(_dumpbin dumpbin HINTS "${_msvc_bin_dir}") + find_program(_implib lib HINTS "${_msvc_bin_dir}") + if(NOT _dumpbin OR NOT _implib) + message(FATAL_ERROR + "dumpbin/lib.exe not found (looked next to the MSVC compiler at " + "${_msvc_bin_dir} and on PATH); cannot generate an MSVC import library " + "for ${dll_path}.") + endif() + execute_process(COMMAND "${_dumpbin}" /exports "${dll_path}" OUTPUT_VARIABLE _exports_output) + get_filename_component(_dll_name "${dll_path}" NAME) + file(WRITE "${def_path}" "LIBRARY ${_dll_name}\nEXPORTS\n") + string(REGEX MATCHALL "[ \t]+[0-9]+[ \t]+[0-9A-Fa-f]+[ \t]+[0-9A-Fa-f]+[ \t]+([A-Za-z_][A-Za-z0-9_@]*)" + _export_lines "${_exports_output}") + foreach(_line IN LISTS _export_lines) + string(REGEX REPLACE ".*[ \t]([A-Za-z_][A-Za-z0-9_@]*)$" "\\1" _symbol "${_line}") + file(APPEND "${def_path}" "${_symbol}\n") endforeach() + execute_process(COMMAND "${_implib}" "/def:${def_path}" "/out:${out_lib_path}" /machine:x64) +endfunction() + +# Stage a single shared library into the runtime bundle under its real SONAME, +# as one plain file with no symlinks. The prebuilt lbt/OpenBLAS/libgfortran +# archives ship each library as a dev-symlink -> SONAME-symlink -> real-file +# chain (e.g. libopenblas.so -> libopenblas.so.0 -> libopenblasp-r0.3.33.so). +# _core's NEEDED entry only ever refers to the SONAME (confirmed with +# objdump/readelf against a real built wheel), so the other chain members are +# never loaded and don't need to ship at all. A real CI-built wheel showed the +# wheel-packaging step does not preserve symlinks (each chain member was +# expanded into its own full-size copy), roughly tripling the wheel size; +# writing exactly one real file under the SONAME sidesteps that regardless of +# how the packaging step handles symlinks, since there is none left to expand. +function(cyhighs_stage_runtime_lib candidate_paths dest_dir) + list(GET candidate_paths 0 _first) + file(REAL_PATH "${_first}" _real_file) + execute_process( + COMMAND "${CMAKE_OBJDUMP}" -p "${_real_file}" + OUTPUT_VARIABLE _objdump_output + RESULT_VARIABLE _objdump_result) + if(NOT _objdump_result EQUAL 0) + message(FATAL_ERROR "cyhighs_stage_runtime_lib: objdump failed on ${_real_file}") + endif() + string(REGEX MATCH "SONAME[ \t]+([^ \t\r\n]+)" _ "${_objdump_output}") + if(NOT CMAKE_MATCH_1) + message(FATAL_ERROR "cyhighs_stage_runtime_lib: no SONAME found in ${_real_file}") + endif() + set(_dest_file "${dest_dir}/${CMAKE_MATCH_1}") + file(COPY_FILE "${_real_file}" "${_dest_file}") + # These are prebuilt binaries from conda-forge/JuliaBinaryWrappers, shipped + # with full DWARF debug info still attached (confirmed against a real built + # wheel: stripping libgfortran alone cut it from 11.4 MB to 3.6 MB). None of + # it is reachable from Python, so strip it here rather than relying on + # auditwheel, which does not strip by default. + execute_process(COMMAND "${CMAKE_STRIP}" --strip-unneeded "${_dest_file}") +endfunction() + +# --------------------------------------------------------------------------- +# Work out which platform/arch we are building for. Reused below for both the +# lbt asset name and the conda-forge subdir. +# --------------------------------------------------------------------------- +string(TOLOWER "${CMAKE_SYSTEM_PROCESSOR}" _raw_arch) +if(_raw_arch MATCHES "amd64|x86_64|x64") + set(_arch "x86_64") +elseif(_raw_arch MATCHES "aarch64|arm64") + set(_arch "aarch64") else() - # ------------------------------------------------------------------------- - # Acquire the prebuilt HiGHS static-apache archive. - # - # The static-apache variant bundles the HiPO interior point solver together - # with its numerical dependencies (AMD, BLAS via OpenBLAS, METIS, RCM), ships - # headers and a CMake package config, and is distributed under Apache 2.0. - # ------------------------------------------------------------------------- - - # Normalise the architecture name to match the release asset naming. - string(TOLOWER "${CMAKE_SYSTEM_PROCESSOR}" _highs_arch) - if(_highs_arch MATCHES "amd64|x86_64|x64") - set(_highs_arch "x86_64") - elseif(_highs_arch MATCHES "aarch64|arm64") - set(_highs_arch "aarch64") + message(FATAL_ERROR "Unsupported CPU architecture for cyhighs: ${CMAKE_SYSTEM_PROCESSOR}") +endif() + +if(APPLE) + if(_arch STREQUAL "aarch64") + set(_conda_subdir "osx-arm64") + set(_lbt_triplet "aarch64-apple-darwin") + else() + set(_conda_subdir "osx-64") + set(_lbt_triplet "x86_64-apple-darwin") + endif() +elseif(WIN32) + if(NOT _arch STREQUAL "x86_64") + message(FATAL_ERROR "cyhighs only supports Windows x86_64.") + endif() + set(_conda_subdir "win-64") + set(_lbt_triplet "x86_64-w64-mingw32") +elseif(UNIX) + # conda-forge only ships glibc binaries, so musllinux (Alpine) wheels can't + # use the conda-forge OpenBLAS fetch below and need their own musl-tagged + # lbt asset. Ask the compiler which libc it targets rather than assuming + # any particular container, matching the actual triplet suffixes + # JuliaBinaryWrappers publishes (e.g. x86_64-linux-musl). + execute_process( + COMMAND "${CMAKE_C_COMPILER}" -dumpmachine + OUTPUT_VARIABLE _target_triple + OUTPUT_STRIP_TRAILING_WHITESPACE + ERROR_QUIET) + set(_libc "gnu") + if(_target_triple MATCHES "musl") + set(_libc "musl") endif() - # Select the release asset and its pinned SHA256 for the target platform. Only - # the platforms with a published static-apache archive are supported; anything - # else (musllinux, Intel macOS, 32 bit) must build against a system HiGHS. - set(_highs_sha "") - if(APPLE) - if(NOT _highs_arch STREQUAL "aarch64") - message(FATAL_ERROR - "No prebuilt HiGHS static-apache binary is published for Intel macOS. " - "Only Apple Silicon (arm64) is supported.") - endif() - set(_highs_asset "highs-${HIGHS_VERSION}-arm-apple-static-apache.tar.gz") - # TODO: pin the SHA256 of the arm-apple archive. - elseif(WIN32) - if(NOT _highs_arch STREQUAL "x86_64") - message(FATAL_ERROR "No prebuilt HiGHS static-apache binary for Windows ${_highs_arch}.") + if(_arch STREQUAL "aarch64") + set(_conda_subdir "linux-aarch64") + set(_lbt_triplet "aarch64-linux-${_libc}") + else() + set(_conda_subdir "linux-64") + set(_lbt_triplet "x86_64-linux-${_libc}") + endif() +else() + message(FATAL_ERROR "Unsupported platform for cyhighs: ${CMAKE_SYSTEM_NAME}/${_arch}") +endif() + +# --------------------------------------------------------------------------- +# BLAS/LAPACK for HiPO. Skipped entirely on macOS: reading HiGHS's own +# cmake/FindHipoDeps.cmake shows highs_link_blas() unconditionally links +# Apple's Accelerate framework on Apple platforms, regardless of anything +# found via find_package/find_library, so there is nothing here for lbt to +# intercept. cyhighs's macOS wheels use Accelerate directly, the same as +# before this project adopted lbt. +# --------------------------------------------------------------------------- +set(_bundled_backend_libs "") +set(_needs_openblas_extension_stub FALSE) +if(NOT APPLE) + set(_needs_openblas_extension_stub TRUE) + # Fetch libblastrampoline (lbt): the BLAS/LAPACK ABI HiPO links against. + # HiGHS's FindHipoDeps.cmake does not call find_library(NAMES openblas) + # directly (confirmed by reading that file). With BLAS_LIBRARIES/BLA_VENDOR + # unset, it tries find_package(OpenBLAS CONFIG) first (finds nothing here, + # since neither lbt nor the bundled OpenBLAS ships a CMake package config), + # then sets BLA_VENDOR=OpenBLAS and calls find_package(BLAS). CMake's own + # FindBLAS.cmake module handles that vendor with an internal + # find_library(NAMES openblas) that does respect + # CMAKE_LIBRARY_PATH/CMAKE_PREFIX_PATH, so staging lbt under the "openblas" + # name in a directory prepended to those two variables (below) still works, + # just through CMake's own module rather than a literal find_library() call + # in HiGHS's own code as originally assumed. + set(_lbt_hash "") + if(_lbt_triplet STREQUAL "aarch64-linux-gnu") + set(_lbt_hash 9cca820658a7206b3324bf4fb309154bf2427c81ebd0f6a448a761183a553cdb) + elseif(_lbt_triplet STREQUAL "aarch64-linux-musl") + set(_lbt_hash a50cd8b2cf54df66c324480e66a563d0a756173914f0399d6b17bd1f627def28) + elseif(_lbt_triplet STREQUAL "x86_64-linux-gnu") + set(_lbt_hash 76ddd4223122d9664f827fc940f7696bb4225cfd1cf6bb00b3acbeb524bcc609) + elseif(_lbt_triplet STREQUAL "x86_64-linux-musl") + set(_lbt_hash 759c0699e8675b1ca13d3f574dfab339f62825cae168a525549b669535207c92) + elseif(_lbt_triplet STREQUAL "x86_64-w64-mingw32") + set(_lbt_hash 4d301c454f1259d50db44a2a0af83cb441d3cbc790963c375ed27686d884624f) + endif() + set(_lbt_asset "libblastrampoline.v${CYHIGHS_LBT_VERSION}.${_lbt_triplet}.tar.gz") + set(_lbt_url + "https://github.com/JuliaBinaryWrappers/libblastrampoline_jll.jl/releases/download/libblastrampoline-v${CYHIGHS_LBT_VERSION}+${CYHIGHS_LBT_BUILD}/${_lbt_asset}") + set(_lbt_root "${CMAKE_BINARY_DIR}/lbt-prebuilt") + cyhighs_fetch_and_extract("${_lbt_url}" "${_lbt_hash}" "${_lbt_root}") + + if(WIN32) + file(GLOB _lbt_dll "${_lbt_root}/bin/libblastrampoline*.dll") + else() + file(GLOB _lbt_shared "${_lbt_root}/lib/libblastrampoline.*") + endif() + + # The directory the find_package(BLAS)/BLA_VENDOR=OpenBLAS lookup above is + # pointed at, via CMAKE_LIBRARY_PATH/CMAKE_PREFIX_PATH (prepended below). A + # copy of lbt lives here under the "openblas" name so that lookup resolves + # to lbt instead of a real BLAS, with no upstream HiGHS source change. + set(_blas_shim_dir "${CMAKE_BINARY_DIR}/blas-shim") + file(MAKE_DIRECTORY "${_blas_shim_dir}") + + if(WIN32) + if(NOT _lbt_dll) + message(FATAL_ERROR "No libblastrampoline DLL found in ${_lbt_root}/bin") endif() - set(_highs_asset "highs-${HIGHS_VERSION}-x86_64-windows-static-apache.zip") - # TODO: pin the SHA256 of the x86_64-windows archive. - elseif(UNIX) - set(_highs_asset "highs-${HIGHS_VERSION}-${_highs_arch}-linux-gnu-static-apache.tar.gz") - if(_highs_arch STREQUAL "x86_64") - set(_highs_sha "b5c5d25cfbb66d438a3b40b6919c50c3c82341f74062f5f44371a52e65d6bd2c") + list(GET _lbt_dll 0 _lbt_dll) + file(COPY_FILE "${_lbt_dll}" "${_blas_shim_dir}/blastrampoline.dll") + # CMake's FindBLAS.cmake looks for an import library (openblas.lib) on + # Windows, not the DLL itself. The JLL release only ships a MinGW-style + # *.dll.a import library (confirmed by extracting the real archive), + # which MSVC cannot consume directly, so generate a fresh one. + cyhighs_generate_msvc_import_lib( + "${_blas_shim_dir}/blastrampoline.dll" + "${_blas_shim_dir}/openblas.def" + "${_blas_shim_dir}/openblas.lib") + # The JLL release is MinGW-built and ships with DWARF debug sections still + # attached (confirmed against a real built wheel: stripping it cut the DLL + # from 2.8 MB to 1.2 MB). cibuildwheel's Windows build runs outside an + # MSYS/MinGW shell (the same reason cyhighs_generate_msvc_import_lib above + # cannot assume dumpbin/lib.exe are on PATH), so a MinGW strip.exe is not + # guaranteed to be present; skip stripping rather than fail the build if + # none is found. + find_program(_mingw_strip strip) + if(_mingw_strip) + execute_process(COMMAND "${_mingw_strip}" --strip-unneeded "${_blas_shim_dir}/blastrampoline.dll") endif() - # TODO: pin the SHA256 of the aarch64-linux archive. else() - message(FATAL_ERROR "Unsupported platform for prebuilt HiGHS: ${CMAKE_SYSTEM_NAME}/${_highs_arch}.") + if(NOT _lbt_shared) + message(FATAL_ERROR "No libblastrampoline shared library found in ${_lbt_root}/lib") + endif() + # Preserve the whole libblastrampoline.so -> .so.N -> .so.N.N.N chain + # under its own names, not just one file renamed to "libopenblas.so". + # The linker embeds whatever SONAME is baked into the real versioned + # file (e.g. libblastrampoline.so.5) as _core's NEEDED entry, regardless + # of what name/path we link it under -- confirmed by a real CI failure + # at import time: "libblastrampoline.so.5: cannot open shared object + # file". Renaming only the copy find_package(BLAS) is pointed at left + # that actual SONAME target missing from the shim directory entirely. + file(COPY ${_lbt_shared} DESTINATION "${_blas_shim_dir}") + list(GET _lbt_shared 0 _lbt_shared_one) + file(COPY_FILE "${_lbt_shared_one}" "${_blas_shim_dir}/libopenblas.so") endif() - # The extracted tree lands here. HIGHS_ARCHIVE lets a caller point at a local - # copy of the archive (air gapped builds, CI caches, offline development) so - # that no download happens. - set(HIGHS_ARCHIVE "" CACHE FILEPATH "Local HiGHS static-apache archive to use instead of downloading") - set(_highs_root "${CMAKE_BINARY_DIR}/highs-prebuilt") + # The *default* lbt-forwarded backend (not linked directly -- see + # LBT_DEFAULT_LIBS in src/cyhighs/_blas_backend.py). Compiling OpenBLAS + # ourselves is deliberately avoided: it's what caused the AVX2/AVX512 build + # breakage this project hit and moved away from previously. conda-forge + # provides a prebuilt, plain-symbol OpenBLAS for every platform reached + # here except musl Linux (conda-forge is glibc-only); musllinux instead + # uses the Alpine package installed via apk in CI (see wheels.yml), + # discovered via find_library the same way this project found OpenBLAS + # before adopting lbt. + if(_libc STREQUAL "musl") + find_library(_openblas_backend NAMES openblas REQUIRED) + list(APPEND _bundled_backend_libs "${_openblas_backend}") + else() + # Hashes pinned against real conda.anaconda.org/conda-forge repodata.json + # entries. The pthreads (non-OpenMP) build variant avoids an additional + # OpenMP runtime dependency. + if(_conda_subdir STREQUAL "linux-64") + set(_openblas_conda "libopenblas-0.3.33-pthreads_h94d23a6_0.conda") + set(_openblas_sha256 3d9aa85648e5e18a6d66db98b8c4317cc426721ad7a220aa86330d1ccedc8903) + set(_libgfortran_conda "libgfortran5-15.2.0-h68bc16d_19.conda") + set(_libgfortran_sha256 057978bb69fea29ed715a9b98adf71015c31baecc4aeb2bfc20d4fd5d83579d4) + elseif(_conda_subdir STREQUAL "linux-aarch64") + set(_openblas_conda "libopenblas-0.3.33-pthreads_h9d3fd7e_0.conda") + set(_openblas_sha256 b018ecfb05e75a8eea3f21f6b5c5c2a54b5178bdcf19e2e2df2735740214a8c8) + set(_libgfortran_conda "libgfortran5-15.2.0-h1b7bec0_19.conda") + set(_libgfortran_sha256 af8e9bdcaa77f133a8ee4c1ef57ef564d9c45aa262abf9f5ef9b50eb99d96407) + elseif(_conda_subdir STREQUAL "win-64") + # The Windows build has no libgfortran dependency (statically linked). + set(_openblas_conda "libopenblas-0.3.33-pthreads_h877e47f_0.conda") + set(_openblas_sha256 104bba89eaed090ab7dd6de17376ba7c14be7f4d44807346012e50a493f18ec9) + set(_libgfortran_conda "") + endif() - if(NOT EXISTS "${_highs_root}/lib") - if(HIGHS_ARCHIVE AND EXISTS "${HIGHS_ARCHIVE}") - set(_highs_local "${HIGHS_ARCHIVE}") - message(STATUS "Using local HiGHS archive: ${_highs_local}") - else() - set(_highs_local "${CMAKE_BINARY_DIR}/${_highs_asset}") - set(_highs_url - "https://github.com/ERGO-Code/HiGHS/releases/download/v${HIGHS_VERSION}/${_highs_asset}") - message(STATUS "Downloading HiGHS: ${_highs_url}") - if(_highs_sha STREQUAL "") - message(WARNING - "No pinned SHA256 for ${_highs_asset}; downloading without an integrity check.") - file(DOWNLOAD "${_highs_url}" "${_highs_local}" TLS_VERIFY ON STATUS _highs_dl) - else() - file(DOWNLOAD "${_highs_url}" "${_highs_local}" - EXPECTED_HASH SHA256=${_highs_sha} TLS_VERIFY ON STATUS _highs_dl) + set(_openblas_url "https://conda.anaconda.org/conda-forge/${_conda_subdir}/${_openblas_conda}") + set(_openblas_root "${CMAKE_BINARY_DIR}/openblas-prebuilt") + cyhighs_fetch_conda_package("${_openblas_url}" "${_openblas_sha256}" "${_openblas_root}") + + if(WIN32) + # conda-forge's Windows package ships OpenBLAS as a bare DLL with no + # "lib" prefix and no import library: Library/bin/openblas.dll. + # Confirmed by extracting the real package; there is no + # "libopenblas*.dll" name at all. + file(GLOB _openblas_lib "${_openblas_root}/Library/bin/openblas.dll") + if(NOT _openblas_lib) + file(GLOB _openblas_lib "${_openblas_root}/Library/bin/openblas*.dll") endif() - list(GET _highs_dl 0 _highs_dl_code) - if(NOT _highs_dl_code EQUAL 0) - list(GET _highs_dl 1 _highs_dl_msg) - message(FATAL_ERROR "Failed to download ${_highs_url}: ${_highs_dl_msg}") + else() + # Prefer the canonical libopenblas.so(.N) name over the + # differently-prefixed internal real filename it's a symlink to (e.g. + # libopenblas.so.0 -> libopenblasp-r0.3.33.so, confirmed by a real + # extraction), since file(GLOB) result order is not guaranteed to be + # deterministic across platforms. + file(GLOB _openblas_lib "${_openblas_root}/lib/libopenblas.so*") + if(NOT _openblas_lib) + file(GLOB _openblas_lib "${_openblas_root}/lib/libopenblas*.so*") endif() endif() - file(MAKE_DIRECTORY "${_highs_root}") - file(ARCHIVE_EXTRACT INPUT "${_highs_local}" DESTINATION "${_highs_root}") - endif() + if(NOT _openblas_lib) + message(FATAL_ERROR "No OpenBLAS shared library found after extracting ${_openblas_conda}") + endif() + list(GET _openblas_lib 0 _openblas_backend) - # Collect the static libraries in link order: highs depends on highs_extras, - # which depends on OpenBLAS. Match both the Unix (lib*.a) and Windows (*.lib) - # naming so the same logic works everywhere. - set(_highs_static_libs "") - foreach(_name highs highs_extras) - file(GLOB _found - "${_highs_root}/lib/lib${_name}.a" - "${_highs_root}/lib/${_name}.lib" - "${_highs_root}/lib/lib${_name}.lib") - if(NOT _found) - message(FATAL_ERROR "Prebuilt HiGHS library '${_name}' not found in ${_highs_root}/lib") + if(WIN32) + # MSVC's linker needs a .lib import library, not the bare DLL, and + # conda-forge ships none for OpenBLAS (it's only linked here to force + # delvewheel to bundle it -- see the target_link_libraries comment + # below). Generate one the same way as for libblastrampoline above. + cyhighs_generate_msvc_import_lib( + "${_openblas_backend}" + "${_blas_shim_dir}/openblas-backend.def" + "${_blas_shim_dir}/openblas-backend.lib") + list(APPEND _bundled_backend_libs "${_blas_shim_dir}/openblas-backend.lib") + else() + list(APPEND _bundled_backend_libs "${_openblas_backend}") endif() - list(GET _found 0 _lib) - list(APPEND _highs_static_libs "${_lib}") - endforeach() - # HiPO needs a BLAS. The Linux and Windows archives bundle OpenBLAS as a static - # library (its name may carry a target or version suffix, so match with a - # wildcard). The macOS archive ships no OpenBLAS and instead relies on the system - # Accelerate framework, so fall back to that there. - file(GLOB _openblas_found - "${_highs_root}/lib/libopenblas.a" - "${_highs_root}/lib/libopenblas*.a" - "${_highs_root}/lib/openblas.lib" - "${_highs_root}/lib/openblas*.lib" - "${_highs_root}/lib/libopenblas*.lib") - if(_openblas_found) - list(GET _openblas_found 0 _lib) - list(APPEND _highs_static_libs "${_lib}") - elseif(APPLE) - find_library(_accelerate Accelerate REQUIRED) - list(APPEND _highs_static_libs "${_accelerate}") - else() - file(GLOB _lib_contents "${_highs_root}/lib/*") - message(FATAL_ERROR - "No BLAS library found in ${_highs_root}/lib. Contents: ${_lib_contents}") + if(_libgfortran_conda) + set(_libgfortran_url "https://conda.anaconda.org/conda-forge/${_conda_subdir}/${_libgfortran_conda}") + set(_libgfortran_root "${CMAKE_BINARY_DIR}/libgfortran-prebuilt") + cyhighs_fetch_conda_package("${_libgfortran_url}" "${_libgfortran_sha256}" "${_libgfortran_root}") + file(GLOB _libgfortran_lib "${_libgfortran_root}/lib/libgfortran*.so*") + if(_libgfortran_lib) + list(GET _libgfortran_lib 0 _libgfortran_backend) + list(APPEND _bundled_backend_libs "${_libgfortran_backend}") + endif() + endif() endif() - # The prebuilt libraries reference the system zlib (the archives are built with - # ZLIB on) and pthreads. On Windows the Windows archive may bundle zlib, so it - # is looked up but not required there. - find_package(Threads REQUIRED) - if(WIN32) - find_package(ZLIB) - else() - find_package(ZLIB REQUIRED) - endif() + list(PREPEND CMAKE_LIBRARY_PATH "${_blas_shim_dir}") + list(PREPEND CMAKE_PREFIX_PATH "${_blas_shim_dir}") +endif() - # The C API header lives at include/highs/interfaces/highs_c_api.h in the - # archive, so include/highs is the directory to add. - set(_highs_include_dir "${_highs_root}/include/highs") - set(_highs_link_libraries ${_highs_static_libs} Threads::Threads) - if(ZLIB_FOUND) - list(APPEND _highs_link_libraries ZLIB::ZLIB) - endif() - if(CMAKE_DL_LIBS) - list(APPEND _highs_link_libraries ${CMAKE_DL_LIBS}) - endif() +# --------------------------------------------------------------------------- +# Compile HiGHS + HiPO from source. On non-Apple platforms this links the lbt +# shim staged above in place of OpenBLAS; on Apple it links Accelerate. +# --------------------------------------------------------------------------- +include(FetchContent) + +set(BUILD_SHARED_LIBS OFF CACHE BOOL "" FORCE) +set(FAST_BUILD ON CACHE BOOL "" FORCE) +set(BUILD_TESTING OFF CACHE BOOL "" FORCE) +set(BUILD_EXAMPLES OFF CACHE BOOL "" FORCE) +set(ZLIB OFF CACHE BOOL "" FORCE) - if(EXISTS "${_highs_root}/share/doc/HIGHS/LICENSE.txt") - set(_highs_license_file "${_highs_root}/share/doc/HIGHS/LICENSE.txt") +# HiPO is the interior point solver added in HiGHS 1.15. It requires +# FAST_BUILD and a BLAS. BUILD_OPENBLAS stays OFF: OpenBLAS is never compiled +# by this project (see the BLAS/LAPACK section above, which also prepends +# CMAKE_LIBRARY_PATH/CMAKE_PREFIX_PATH with the lbt shim directory on +# non-Apple platforms before this FetchContent runs). +set(HIPO ON CACHE BOOL "" FORCE) +set(BUILD_SHARED_EXTRAS_LIB OFF CACHE BOOL "" FORCE) + +# EXCLUDE_FROM_ALL (CMake 3.28+, hence the cmake_minimum_required above) keeps +# HiGHS's own standalone "highs" CLI executable out of the default build. +# cyhighs never needs that target, and it cannot link on non-Apple platforms +# anyway: it statically links libhighs_extras.a in full, including +# myblas.cpp's highs_openblas_set_num_threads(), which calls the real +# OpenBLAS-specific (not standard BLAS/LAPACK ABI) openblas_set_num_threads() +# -- undefined against our lbt shim, confirmed by a real CI link failure. +# _core only pulls in the highs/highs_extras static libraries themselves +# (via highs::highs below), which still build normally either way. +FetchContent_Declare( + highs + GIT_REPOSITORY https://github.com/ERGO-Code/HiGHS.git + GIT_TAG v${HIGHS_VERSION} + GIT_SHALLOW ON + EXCLUDE_FROM_ALL) +FetchContent_MakeAvailable(highs) + +set(_highs_include_dir "${highs_SOURCE_DIR}/highs") +set(_highs_link_libraries "highs::highs") + +set(_highs_license_file "") +foreach(_candidate LICENSE.txt LICENSE) + if(EXISTS "${highs_SOURCE_DIR}/${_candidate}") + set(_highs_license_file "${highs_SOURCE_DIR}/${_candidate}") + break() endif() -endif() +endforeach() + +find_package(Threads REQUIRED) # --------------------------------------------------------------------------- # Transpile the Cython source to C. @@ -262,13 +514,86 @@ add_custom_command( "${CMAKE_CURRENT_SOURCE_DIR}/src/cyhighs/_highs_c_api.pxd" VERBATIM) +set(_core_extra_sources "") +if(_needs_openblas_extension_stub) + # HiGHS's myblas.cpp calls the real OpenBLAS-specific extension function + # openblas_set_num_threads() (not part of the standard BLAS/LAPACK ABI lbt + # forwards), because HiGHS's build believes "openblas" is a real OpenBLAS + # (see the BLAS/LAPACK section above). A static executable that links + # libhighs_extras.a in full cannot leave this undefined (confirmed by a + # real CI failure linking HiGHS's own CLI, excluded from the build above); + # a shared library like _core tolerates it at link time, but would crash + # if this ever got called at runtime. Supply a no-op definition so _core + # can never hit that crash. This only affects OpenBLAS's own thread-count + # tuning, not correctness or which real BLAS lbt forwards to. + set(_openblas_stub_c "${CMAKE_BINARY_DIR}/openblas_extension_stub.c") + file(WRITE "${_openblas_stub_c}" "void openblas_set_num_threads(int num_threads) { (void)num_threads; }\n") + list(APPEND _core_extra_sources "${_openblas_stub_c}") +endif() + # --------------------------------------------------------------------------- # Build the extension module. # --------------------------------------------------------------------------- -python_add_library(_core MODULE "${generated_c}" WITH_SOABI) +python_add_library(_core MODULE "${generated_c}" ${_core_extra_sources} WITH_SOABI) target_include_directories(_core PRIVATE "${_highs_include_dir}") -target_link_libraries(_core PRIVATE ${_highs_link_libraries}) +target_link_libraries(_core PRIVATE ${_highs_link_libraries} Threads::Threads) +if(CMAKE_DL_LIBS) + target_link_libraries(_core PRIVATE ${CMAKE_DL_LIBS}) +endif() + +# Link the bundled OpenBLAS (+ libgfortran) into _core, so that +# auditwheel/delvewheel see it as a normal dependency to relocate and vendor, +# and so its transitive deps (libgfortran -> libquadmath) get pulled into the +# wheel too. Empty (a no-op) on Apple, where HiGHS links Accelerate directly. +target_link_libraries(_core PRIVATE ${_bundled_backend_libs}) + +# Make _core's own runtime dependencies loadable. This has to work in two +# very different situations: the wheel builds (cibuildwheel, where +# auditwheel/delvewheel repair the wheel afterward) and the plain editable +# build that tests.yml / `uv sync` / local development use, which has no +# repair step at all. In both, the bundled libraries are installed into the +# package directory next to _core, and _core is pointed at that directory. +if(NOT APPLE) + if(WIN32) + # Windows has no RPATH-equivalent: dependent DLLs are found next to the + # loading module. Copy them next to _core in the build tree (for the + # unrepaired build) and install them into the package directory (for the + # wheel, which a real CI failure showed delvewheel does not otherwise + # populate from the build-tree shim location on its own). + add_custom_command(TARGET _core POST_BUILD + COMMAND "${CMAKE_COMMAND}" -E copy_if_different + "${_blas_shim_dir}/blastrampoline.dll" + "${_openblas_backend}" + "$") + install(FILES "${_blas_shim_dir}/blastrampoline.dll" "${_openblas_backend}" + DESTINATION cyhighs) + else() + # On Linux, _core's NEEDED entries are libblastrampoline.so.5 (via HiGHS's + # shim link) and the bundled OpenBLAS SONAME. Stage just those SONAME + # files (via cyhighs_stage_runtime_lib, see above) into a bundle directory + # and install it next to _core, then give _core an $ORIGIN INSTALL_RPATH + # so it resolves them as siblings. This is what a real CI failure showed + # was missing: with an empty install RPATH, the editable build could not + # import _core (libblastrampoline.so.5 not found) and auditwheel could not + # locate the same libraries to vendor them into the repaired wheel. The + # BUILD_RPATH keeps the in-build-tree _core loadable too. On the wheel + # path, auditwheel still runs afterward and pulls the deeper transitive + # deps (e.g. libquadmath) into cyhighs.libs; on the editable path those + # come from the system toolchain. + set(_runtime_bundle_dir "${CMAKE_BINARY_DIR}/runtime-bundle") + file(REMOVE_RECURSE "${_runtime_bundle_dir}") + file(MAKE_DIRECTORY "${_runtime_bundle_dir}") + cyhighs_stage_runtime_lib("${_lbt_shared}" "${_runtime_bundle_dir}") + foreach(_backend_lib IN LISTS _bundled_backend_libs) + cyhighs_stage_runtime_lib("${_backend_lib}" "${_runtime_bundle_dir}") + endforeach() + install(DIRECTORY "${_runtime_bundle_dir}/" DESTINATION cyhighs) + set_target_properties(_core PROPERTIES + INSTALL_RPATH "$ORIGIN" + BUILD_RPATH "${_blas_shim_dir}") + endif() +endif() # The Cython source compiles as C, but the HiGHS libraries are C++. Linking with # the C++ driver pulls in the C++ runtime (libstdc++ / libc++). @@ -277,7 +602,7 @@ set_target_properties(_core PROPERTIES LINKER_LANGUAGE CXX) # Install the compiled module next to the Python sources inside the package. install(TARGETS _core DESTINATION cyhighs) -# Bundle the upstream HiGHS license into the package for attribution, since the +# Bundle the upstream HiGHS license from the fetched source tree, since the # solver is statically linked into the extension. if(_highs_license_file) install(FILES "${_highs_license_file}" diff --git a/docs/guide/bundling.md b/docs/guide/bundling.md index 886adf4..ce039d0 100644 --- a/docs/guide/bundling.md +++ b/docs/guide/bundling.md @@ -11,50 +11,77 @@ sync. The build is driven by [scikit-build-core](https://scikit-build-core.readthedocs.io/), which runs a -CMake build behind the standard Python packaging interface. By default, instead -of compiling HiGHS from source, CMake downloads the official prebuilt -**`static-apache`** release archive for the target platform at a pinned version, -currently 1.15.1, verifies it against a pinned checksum, extracts it, and links -the static libraries it ships directly into the extension. Because the version -is pinned, every wheel is built against a known solver, and the Python -enumerations in `cyhighs` are transcribed from that same release. - -The one exception is the portable Linux wheels (`manylinux_2_28` and -`musllinux_1_2`). The prebuilt archive is built against a recent glibc and -cannot link inside the older manylinux/musllinux build containers, so for those -wheels CMake compiles HiGHS and HiPO from source at the same pinned version -(`-DCYHIGHS_HIGHS_FROM_SOURCE=ON`), linking a prebuilt OpenBLAS installed in the -container. The result is the same statically bundled solver, just built in place. - -The `static-apache` archive already contains the HiPO interior point solver and -its bundled OpenBLAS linear algebra kernels, so the wheels carry their own linear -algebra and still need nothing from the host system. The Cython source is -transpiled to C and compiled against the prebuilt HiGHS headers. The binding -deliberately avoids the NumPy C API. It moves array data across the boundary -using typed memoryviews and the buffer protocol only, which keeps the compiled -surface small and the dependency on NumPy loose. +CMake build behind the standard Python packaging interface. CMake compiles +HiGHS and HiPO from source (via `FetchContent`) at a pinned version, currently +1.15.1, on every platform. Because the version is pinned, every wheel is built +against a known solver, and the Python enumerations in `cyhighs` are +transcribed from that same release. + +The Cython source is transpiled to C and compiled against the HiGHS headers. +The binding deliberately avoids the NumPy C API. It moves array data across the +boundary using typed memoryviews and the buffer protocol only, which keeps the +compiled surface small and the dependency on NumPy loose. + +## The BLAS backend: libblastrampoline + +HiPO, HiGHS's interior-point solver, needs a BLAS/LAPACK implementation. On +Linux and Windows, instead of linking one directly, `cyhighs` links HiGHS +against +[libblastrampoline](https://github.com/JuliaLinearAlgebra/libblastrampoline) +(lbt). This is a small shared library, originally built for the Julia +ecosystem, that implements the standard BLAS/LAPACK ABI and forwards every +call, at runtime, to whichever real implementation lbt has been pointed at. +Once the compiled extension (and therefore lbt) is loaded, `cyhighs`'s own +`__init__.py` calls lbt's `lbt_forward` C API to register a backend: + +- by default, a prebuilt OpenBLAS bundled inside the wheel, so installing + `cyhighs` still gives you a fully working solver with no configuration and no + external dependencies +- Intel MKL instead, if the optional `cyhighs[mkl]` extra is installed and + MKL's runtime library can be found + +Because the switch happens at import time rather than at compile time, going +from the bundled OpenBLAS to MKL (or back) never requires reinstalling +`cyhighs`. If the `LBT_DEFAULT_LIBS` environment variable is already set when +`cyhighs` is imported, it is left alone, so you can also point HiGHS at any +other lbt-compatible BLAS build yourself. + +CMake never compiles OpenBLAS itself. Both lbt and the bundled OpenBLAS are +fetched as prebuilt binaries and verified against pinned checksums, the same +way the HiGHS source is fetched at a pinned version. lbt comes from +JuliaBinaryWrappers' releases on every platform. The bundled OpenBLAS comes +from conda-forge, except on the portable `musllinux_1_2` (Alpine) Linux +wheels, where conda-forge has no build to offer and Alpine's own OpenBLAS +package is used instead. + +On macOS, none of this applies. HiGHS links directly against Apple's +Accelerate framework there, which ships with every macOS system, so lbt has +nothing to forward and the `cyhighs[mkl]` extra has no effect (Intel has also +never published MKL for macOS anyway). ## Platform coverage Wheels are published for Linux x86_64 and aarch64, macOS on Apple Silicon, and -Windows on x86_64. On Linux both a `manylinux_2_39` wheel (from the prebuilt -archive, requiring **glibc 2.39 or newer**) and the more portable -`manylinux_2_28` (**glibc 2.28 or newer**) and `musllinux_1_2` (Alpine and other -musl distros) wheels are shipped; pip installs whichever best matches the host. -There are no 32-bit wheels; that platform builds from source instead. +Windows on x86_64. There are no 32-bit wheels, since HiGHS is not viable on +32-bit targets. The `cyhighs[mkl]` extra is only available on Linux x86_64 and +Windows x86_64, since Intel does not publish MKL for macOS or for any ARM +target. ## Licensing Because the bundled build includes HiPO, which carries Apache-licensed -dependencies, the `static-apache` archive is distributed under the Apache License -2.0, and `cyhighs` is released under the same license. The upstream HiGHS license -and notices are included in the wheels for attribution. +dependencies, `cyhighs` is released under the Apache License 2.0. The upstream +HiGHS license and notices are included in the wheels for attribution. +libblastrampoline is MIT licensed, and the bundled OpenBLAS is BSD licensed. +Both are dynamically linked, and neither imposes further obligations on +`cyhighs` itself. ## What this means for you -Because everything is static, installing `cyhighs` gives you a working solver -immediately. You can confirm which HiGHS version you have by asking the library -directly. +Because HiGHS is statically linked and the BLAS backend it needs is bundled +alongside it, installing `cyhighs` gives you a working solver immediately, with +nothing further to install. You can confirm which HiGHS version you have by +asking the library directly. ```python from cyhighs import highs_version diff --git a/docs/guide/installation.md b/docs/guide/installation.md index fb5d92a..b96ea84 100644 --- a/docs/guide/installation.md +++ b/docs/guide/installation.md @@ -15,20 +15,32 @@ which pip installs for you. Wheels are published for CPython 3.11 through 3.14 on: -- **Linux** x86_64 and aarch64. Two flavours are shipped and pip picks the best - match automatically: - - `manylinux_2_28` (glibc 2.28 or newer, for example CentOS/RHEL 8, Debian - 10+, Ubuntu 18.10+) and `musllinux_1_2` (Alpine and other musl distros), - built by compiling HiGHS from source in the manylinux/musllinux containers. - - `manylinux_2_39` (glibc 2.39 or newer, for example Ubuntu 24.04+), built - from the official prebuilt HiGHS archive. On new-enough systems pip prefers - this one. +- **Linux** x86_64 and aarch64, as `manylinux_2_28` (glibc 2.28 or newer, for + example CentOS/RHEL 8, Debian 10+, Ubuntu 18.10+) and `musllinux_1_2` (Alpine + and other musl distros). - **macOS** on Apple Silicon (arm64). - **Windows** on x86_64. If a matching wheel exists for your platform, pip uses it and no compiler is needed. +### Using Intel MKL instead of the bundled OpenBLAS + +By default, on Linux and Windows, the wheel's HiPO interior-point solver runs +on a bundled OpenBLAS, so there is nothing further to install. On macOS, HiPO +always uses Apple's Accelerate framework instead, and the rest of this section +does not apply there. + +If you have Intel MKL available on Linux or Windows and want HiGHS to use it +instead of the bundled OpenBLAS, install the optional extra: + +```bash +pip install cyhighs[mkl] +``` + +This takes effect automatically the next time `cyhighs` is imported, with no +rebuild. See [How HiGHS is bundled](bundling.md) for how the switch works. + ## Verifying the install After installing, you can check that the extension imports and that the solver @@ -48,21 +60,9 @@ for a platform without a published wheel, you need a C and C++ compiler. CMake and Ninja are pulled in automatically as build dependencies, so you do not have to install them yourself. -By default, rather than compiling HiGHS, the build downloads the official -prebuilt HiGHS release archive for your platform and links it in, so it is quick -and needs network access at build time. Because the prebuilt Linux archive -targets a recent glibc, this default source build on Linux also requires -**glibc 2.38 or newer**. - -To instead compile HiGHS itself from source — needed on older glibc or on musl, -and how the `manylinux_2_28` / `musllinux` wheels are produced — pass -`-DCYHIGHS_HIGHS_FROM_SOURCE=ON`. That path links a prebuilt OpenBLAS that must -already be installed (for example `openblas-devel` on RHEL/Fedora, -`libopenblas-dev` on Debian/Ubuntu, or `openblas-dev` on Alpine): - -```bash -pip wheel . -C cmake.define.CYHIGHS_HIGHS_FROM_SOURCE=ON -``` +The build always compiles HiGHS and HiPO from source at a pinned version, and +needs network access to fetch the HiGHS source and prebuilt +libblastrampoline/OpenBLAS binaries (see [How HiGHS is bundled](bundling.md)). The project uses [uv](https://docs.astral.sh/uv/) for development. Cloning the repository and running a sync builds the extension. @@ -72,13 +72,3 @@ git clone https://github.com/nardi/cyhighs cd cyhighs uv sync ``` - -### Building offline or from a local archive - -If you already have the HiGHS `static-apache` archive for your platform (or want -to avoid the download), point the build at it with the `HIGHS_ARCHIVE` CMake -variable and no download happens: - -```bash -pip wheel . -C cmake.define.HIGHS_ARCHIVE=/path/to/highs-1.15.1-x86_64-linux-gnu-static-apache.tar.gz -``` diff --git a/pyproject.toml b/pyproject.toml index 3dfc3bc..fa90030 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,7 +6,7 @@ requires = [ "scikit-build-core>=0.10", "cython>=3.0", - "cmake>=3.24", + "cmake>=3.28", "ninja>=1.11", ] build-backend = "scikit_build_core.build" @@ -37,6 +37,16 @@ classifiers = [ # bindings deliberately avoid the numpy C API so there is no tighter coupling. dependencies = ["numpy>=1.23.2", "scipy>=1.9.2"] +[project.optional-dependencies] +# HiGHS's BLAS calls are forwarded through libblastrampoline at runtime (see +# src/cyhighs/_blas_backend.py), so swapping in Intel MKL is just a matter of +# having its runtime library present -- no cyhighs rebuild needed. Intel's +# `mkl` wheel is only published for Linux x86_64 and Windows x86_64 (Intel has +# never shipped MKL for macOS or for any ARM target), so the environment +# marker below makes `pip install cyhighs[mkl]` fail clearly on unsupported +# platforms instead of silently installing nothing. +mkl = ["mkl; (platform_system == 'Linux' or platform_system == 'Windows') and platform_machine in 'x86_64 AMD64'"] + [project.urls] Homepage = "https://github.com/nardilam/cyhighs" Repository = "https://github.com/nardilam/cyhighs" diff --git a/src/cyhighs/__init__.py b/src/cyhighs/__init__.py index fb0d5a7..61de9ef 100644 --- a/src/cyhighs/__init__.py +++ b/src/cyhighs/__init__.py @@ -22,13 +22,20 @@ from __future__ import annotations +from ._blas_backend import configure_blas_backend from ._core import highs_infinity, highs_version -from .enumerations import ModelStatus, ObjectiveSense, PresolveRule, VariableType -from .linprog_interface import linprog -from .options import HighsOption -from .result import LinearProblemSolution, OptimizeResult -from .sparse_interface import solve_linear_problem_sparse -from .validation import HIGHS_INFINITY, solve_linear_problem + +# Importing _core loads libblastrampoline; point it at cyhighs's BLAS backend +# (bundled OpenBLAS, or MKL if the cyhighs[mkl] extra is installed) before any +# solve calls into it. +configure_blas_backend() + +from .enumerations import ModelStatus, ObjectiveSense, PresolveRule, VariableType # noqa: E402 +from .linprog_interface import linprog # noqa: E402 +from .options import HighsOption # noqa: E402 +from .result import LinearProblemSolution, OptimizeResult # noqa: E402 +from .sparse_interface import solve_linear_problem_sparse # noqa: E402 +from .validation import HIGHS_INFINITY, solve_linear_problem # noqa: E402 __all__ = [ "HIGHS_INFINITY", diff --git a/src/cyhighs/_blas_backend.py b/src/cyhighs/_blas_backend.py new file mode 100644 index 0000000..4bc80d7 --- /dev/null +++ b/src/cyhighs/_blas_backend.py @@ -0,0 +1,204 @@ +"""Select the BLAS/LAPACK backend HiGHS's HiPO solver forwards to. + +On Linux and Windows, the compiled `_core` extension links HiGHS/HiPO against +libblastrampoline (lbt), a small shared library that forwards BLAS/LAPACK +calls, at runtime, to whichever real implementation lbt has been pointed at. +After `_core` (and therefore lbt) is loaded, this module calls lbt's own +`lbt_forward` C API to register a backend: + +- Intel MKL, if the optional `cyhighs[mkl]` extra is installed and its shared + library can be found (Linux x86_64 and Windows x86_64 only, since Intel has + never published MKL for macOS or for ARM) +- otherwise, the OpenBLAS bundled alongside cyhighs, so cyhighs keeps working + with zero configuration and no external dependencies by default. + +`lbt_forward` is used in preference to lbt's `LBT_DEFAULT_LIBS` environment +variable because a value set through Python's `os.environ` is not reliably +seen by lbt's own `getenv` on Windows (Python and the MinGW-built lbt can use +different C runtimes), which leaves HiPO with no backend and crashing at solve +time. Calling the C API configures the exact lbt instance already loaded into +the process, with no dependence on environment-variable propagation. + +If `LBT_DEFAULT_LIBS` is already set in the environment, it is respected and +this module does nothing, so advanced users can still point HiGHS at any other +lbt-compatible BLAS themselves. + +On macOS, this module does nothing. HiGHS unconditionally links Apple's +Accelerate framework there instead of going through lbt (see the BLAS/LAPACK +section of the top-level CMakeLists.txt), so there is no backend to register. +""" + +from __future__ import annotations + +import ctypes +import glob +import importlib.util +import os +import sys +from pathlib import Path + + +def _core_dir() -> Path: + """Return the directory that holds the compiled `_core` extension. + + The bundled BLAS libraries are installed next to `_core`, which is not + necessarily next to this source file: in an editable install the compiled + extension lives in the build directory while these `.py` sources are + imported from `src/cyhighs`. Locating `_core` by its import spec finds the + right directory in both the editable and the installed-wheel layouts. + """ + spec = importlib.util.find_spec("cyhighs._core") + if spec is not None and spec.origin: + return Path(spec.origin).parent + return Path(__file__).parent + + +def _first_match(directories: tuple[Path, ...], pattern_name: str) -> str | None: + for directory in directories: + matches = sorted(glob.glob(str(directory / pattern_name))) + if matches: + return matches[0] + return None + + +def _locate_mkl() -> str | None: + """Return the path to Intel MKL's runtime library, if installed. + + The `mkl` PyPI package ships no importable Python module. It is a + data-only wheel that drops shared libraries directly under the + environment's install prefix (`/lib` on Linux, + `/Library/bin` on Windows). + """ + if sys.platform == "win32": + pattern = str(Path(sys.prefix, "Library", "bin", "mkl_rt*.dll")) + else: + pattern = str(Path(sys.prefix, "lib", "libmkl_rt.so*")) + + matches = sorted(glob.glob(pattern)) + return matches[0] if matches else None + + +def _locate_bundled_openblas() -> str | None: + """Return the path to the OpenBLAS shared library bundled with cyhighs. + + OpenBLAS is installed next to `_core`. The wheel-repair tools + (auditwheel/delvewheel) may additionally relocate a copy into a + hash-suffixed sibling directory, so both locations are searched. + """ + core_dir = _core_dir() + # conda-forge's OpenBLAS DLL has no "lib" prefix (openblas.dll) on Windows, + # unlike its Linux build, confirmed by extracting the real packages. + pattern_name = "openblas*.dll" if sys.platform == "win32" else "libopenblas*.so*" + return _first_match((core_dir, core_dir.parent / "cyhighs.libs"), pattern_name) + + +def _open_loaded_lbt() -> ctypes.CDLL | None: + """Return a handle to the libblastrampoline `_core` already loaded. + + Importing `_core` pulls lbt into the process as a dependency. It is + essential to operate on that exact instance rather than load another copy: + a wheel can contain two lbt files (one installed next to `_core`, another + that auditwheel/delvewheel vendored into the `*.libs` directory that + `_core` actually links), and loading a second lbt corrupts its PLT + trampoline resolution and segfaults on x86_64 (confirmed by a real CI + crash at the `lbt_forward` call). + + On Linux, `RTLD_NOLOAD` returns the resident library's handle without + loading anything new. On Windows, `GetModuleHandleW` returns the handle of + an already-loaded module by name (without a fresh `LoadLibrary`), which is + then wrapped as a ctypes handle. + """ + core_dir = _core_dir() + search_dirs = (core_dir, core_dir.parent / "cyhighs.libs") + + if sys.platform == "win32": + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + kernel32.GetModuleHandleW.restype = ctypes.c_void_p + kernel32.GetModuleHandleW.argtypes = [ctypes.c_wchar_p] + # The DLL may be loaded under its plain name (unrepaired build) or the + # mangled name delvewheel gives it in the wheel, so match against + # whatever blastrampoline DLLs are actually present next to _core or + # in *.libs. + names: list[str] = [] + for directory in search_dirs: + for match in sorted(glob.glob(str(directory / "*blastrampoline*.dll"))): + name = os.path.basename(match) + if name not in names: + names.append(name) + for name in (*names, "blastrampoline.dll", "libblastrampoline-5.dll"): + handle = kernel32.GetModuleHandleW(name) + if handle: + try: + return ctypes.CDLL(name, handle=handle) + except OSError: + continue + return None + + # Linux: bind to the already-resident instance by its soname, never a copy. + sonames: list[str] = [] + for directory in search_dirs: + for match in sorted(glob.glob(str(directory / "libblastrampoline.so*"))): + name = os.path.basename(match) + if name not in sonames: + sonames.append(name) + # Prefer a versioned soname (libblastrampoline.so.5) over a bare .so symlink, + # since the versioned name is what the loader tracks the library under. + sonames.sort(key=lambda name: (".so." not in name, name)) + for name in (*sonames, "libblastrampoline.so.5", "libblastrampoline.so"): + try: + return ctypes.CDLL(name, mode=os.RTLD_NOLOAD) + except OSError: + continue + return None + + +def configure_blas_backend() -> None: + """Point the already-loaded libblastrampoline at cyhighs's BLAS backend. + + Called after `_core` is imported, so lbt is already in the process. Does + nothing on macOS (no lbt), or if the user has set `LBT_DEFAULT_LIBS` + (respecting their choice), or if lbt or a backend cannot be located. + """ + if sys.platform == "darwin" or "LBT_DEFAULT_LIBS" in os.environ: + return + + debug = bool(os.environ.get("CYHIGHS_LBT_DEBUG")) + + backend = _locate_mkl() or _locate_bundled_openblas() + lbt = _open_loaded_lbt() + if debug: + print(f"cyhighs: BLAS backend={backend!r} lbt={lbt!r}", file=sys.stderr) + + if backend is None or lbt is None: + # A silent missing backend would surface as a crash the first time HiPO + # calls BLAS, so warn instead of failing quietly. + print( + f"cyhighs: could not configure a BLAS backend " + f"(backend={backend!r}, lbt={lbt!r}), the HiPO solver may be unavailable.", + file=sys.stderr, + ) + return + + try: + # int32_t lbt_forward(const char* libname, int32_t clear, int32_t verbose, + # const char* suffix_hint) + # All four arguments are required. Passing only three left the final + # (pointer) argument as an uninitialized register: harmless by luck on + # the System V AMD64 ABI (Linux/macOS), but a garbage non-NULL pointer + # that lbt dereferenced and crashed on the Windows x64 ABI. suffix_hint + # is NULL so lbt auto-detects the symbol suffix. + lbt.lbt_forward.argtypes = [ + ctypes.c_char_p, + ctypes.c_int32, + ctypes.c_int32, + ctypes.c_char_p, + ] + lbt.lbt_forward.restype = ctypes.c_int32 + # clear=1 replaces any existing forwards. verbose is on under CYHIGHS_LBT_DEBUG. + nforwarded = lbt.lbt_forward(os.fsencode(backend), 1, 1 if debug else 0, None) + if debug: + print(f"cyhighs: lbt_forward returned {nforwarded}", file=sys.stderr) + except (OSError, AttributeError): + # lbt does not export lbt_forward. Leave it on its own compiled-in + # fallback rather than failing the import. + pass diff --git a/src/cyhighs/_core.pyx b/src/cyhighs/_core.pyx index 0e323f0..008400a 100644 --- a/src/cyhighs/_core.pyx +++ b/src/cyhighs/_core.pyx @@ -17,6 +17,8 @@ particular NumPy binary version. import numpy as np +cimport cython + from ._highs_c_api cimport ( Highs_create, Highs_destroy, @@ -99,6 +101,8 @@ def highs_infinity(): Highs_destroy(highs) +@cython.boundscheck(False) +@cython.wraparound(False) def merge_constraint_matrices_csc( HighsInt number_of_columns, double[::1] inequality_values not None, diff --git a/uv.lock b/uv.lock index 44f51b9..95822e1 100644 --- a/uv.lock +++ b/uv.lock @@ -38,6 +38,11 @@ dependencies = [ { name = "scipy", version = "1.18.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.12'" }, ] +[package.optional-dependencies] +mkl = [ + { name = "mkl", marker = "(platform_machine in 'x86_64 AMD64' and sys_platform == 'linux') or (platform_machine in 'x86_64 AMD64' and sys_platform == 'win32')" }, +] + [package.dev-dependencies] dev = [ { name = "mkdocstrings-python" }, @@ -52,9 +57,11 @@ dev = [ [package.metadata] requires-dist = [ + { name = "mkl", marker = "(platform_machine in 'x86_64 AMD64' and sys_platform == 'linux' and extra == 'mkl') or (platform_machine in 'x86_64 AMD64' and sys_platform == 'win32' and extra == 'mkl')" }, { name = "numpy", specifier = ">=1.23.2" }, { name = "scipy", specifier = ">=1.9.2" }, ] +provides-extras = ["mkl"] [package.metadata.requires-dev] dev = [ @@ -141,6 +148,30 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, ] +[[package]] +name = "intel-cmplr-lib-ur" +version = "2026.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "umf" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/8e/06/da0fcd62ee4672489ede80f322eec61b48a38695b0a5072d6d1075b37197/intel_cmplr_lib_ur-2026.1.0-py2.py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:74b64ace8277b031aa36328dc8f87f7f959a6ca4da5b68c69923d37283e01664", size = 31555105, upload-time = "2026-07-01T15:14:58.416Z" }, + { url = "https://files.pythonhosted.org/packages/c3/74/6f8bed53ec47a1159f3cd92de0ee4cbeb1eabb6d29accf97c5dc3983a9bb/intel_cmplr_lib_ur-2026.1.0-py2.py3-none-win_amd64.whl", hash = "sha256:538e1135ee1801fb19fa1a580838a1f666f05534d35350d82736d959373d3b15", size = 1308879, upload-time = "2026-07-01T15:10:19.657Z" }, +] + +[[package]] +name = "intel-openmp" +version = "2026.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "intel-cmplr-lib-ur" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/72/23/60aeb428e6b1fb34fb81d4970d91ff8b5deeeeb446e1628bd78f9e3d1f8b/intel_openmp-2026.1.0-py2.py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:e6887f701b7d2323ed147008893b9af22e511aaa97e0fc9c37eb8be24a5366b0", size = 52547733, upload-time = "2026-07-01T15:14:42.64Z" }, + { url = "https://files.pythonhosted.org/packages/6e/cc/b4cb4ed4a1cca9dcc63cec93f7813ef3840f37d8bd30c06b8013948edc00/intel_openmp-2026.1.0-py2.py3-none-win_amd64.whl", hash = "sha256:8f9f86d93da9d549eec7bae26d65d6188ed748792ce655b5cdbc2e1a15cb7206", size = 26355044, upload-time = "2026-07-01T15:10:22.582Z" }, +] + [[package]] name = "isort" version = "8.0.1" @@ -358,6 +389,20 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d1/fc/10ab7e80650a9c9e8f4f1105f8c8e73567f88ed0c06ada589ab81d38687c/mkdocstrings_python-2.0.5-py3-none-any.whl", hash = "sha256:30c837bbff016549f659fcba6539ac351303f0fd7e713c89a040611072236e9d", size = 104951, upload-time = "2026-06-19T10:41:07.378Z" }, ] +[[package]] +name = "mkl" +version = "2026.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "intel-openmp" }, + { name = "onemkl-license" }, + { name = "tbb" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/61/da/4921e17b1f455f7fed30d5cc0964f3289eee6a6cb03cdf7d5e20c14bd025/mkl-2026.1.0-py2.py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:4d5a26449818a8aebd4b2aaf6b291831e691f9cf74b152df49b99cf48bbd5360", size = 224034262, upload-time = "2026-07-01T15:11:10.038Z" }, + { url = "https://files.pythonhosted.org/packages/d6/bd/e35bd5477278c91b0f5d4733b13009d183c284d636d68c649204858c207a/mkl-2026.1.0-py2.py3-none-win_amd64.whl", hash = "sha256:8194a8270d70622eff00eec84c793efc382b14848b98c3ace6bacdda42170a80", size = 180806880, upload-time = "2026-07-01T15:14:02.845Z" }, +] + [[package]] name = "numpy" version = "2.4.6" @@ -494,6 +539,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a1/5a/4d2b1601df3602dba7a14f3348ba9bfe94a18adb428e693df6154c293831/numpy-2.5.1-cp314-cp314t-win_arm64.whl", hash = "sha256:5a6db61f9aaa57e369905c67d852045d3c4f7126405b29d09b19dec118e9c9cb", size = 10697674, upload-time = "2026-07-04T17:07:58.506Z" }, ] +[[package]] +name = "onemkl-license" +version = "2026.1.0" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e3/ef/8437c187319e779a76f4dbb468a1863d729297d79a1b5f44b10a58c96ec2/onemkl_license-2026.1.0-py2.py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:39fd829648af92c9e03c22ba228f07174d829aaa2fe3c5f6cd7073b8eb8a9805", size = 56161, upload-time = "2026-07-01T15:11:45.836Z" }, + { url = "https://files.pythonhosted.org/packages/66/0d/82581d2048b5a01f3145fa48cc9c461f34443499420dcdaa6ecbb444ce4d/onemkl_license-2026.1.0-py2.py3-none-win_amd64.whl", hash = "sha256:425b599af279e6bc03e27319d75aa156ca7dd8645dbcd0c324ef894d3fae6be3", size = 56185, upload-time = "2026-07-01T15:13:49.098Z" }, +] + [[package]] name = "packaging" version = "26.2" @@ -857,6 +911,27 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/97/33/215bab639a10887870672257cc477a4d3e331a4adae6b14e48f22a631639/stubgen_pyx-0.2.16-py3-none-any.whl", hash = "sha256:09a149ecfdfea0fc3241d2be4eb9159fd945764f43300e032a88ddb9ab17a4ed", size = 44331, upload-time = "2026-06-26T20:00:26.867Z" }, ] +[[package]] +name = "tbb" +version = "2023.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "tcmlib" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/25/0c/0266c71e3fa50a71db5ce8a1d0807863df3215c5f7b5fe7c98b257561138/tbb-2023.1.0-py2.py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:64ad35241c736a595498f5343abec8eaaa203e9fe0dbdbf4b86d37c5a3ab1d9c", size = 6840282, upload-time = "2026-07-01T15:10:33.348Z" }, + { url = "https://files.pythonhosted.org/packages/ac/ff/168d3498762069b0cd2dff20ab34d53fa48a0a35ffd1965cf456c603f292/tbb-2023.1.0-py3-none-win_amd64.whl", hash = "sha256:27df1315202defc67a73800c667ba4fbd03cf8924f73949373d4a34d1443fca2", size = 428501, upload-time = "2026-07-01T15:12:55.771Z" }, +] + +[[package]] +name = "tcmlib" +version = "1.5.0" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/60/24/aa409bb20703acc70cf4d3bc620a55c789639c2995b2667fb44ae7236ec9/tcmlib-1.5.0-py2.py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:9d7c01cff35aae9bf5390b620680ebdf10a7d211c22d6488a27a029502e7d0aa", size = 2812669, upload-time = "2026-04-24T14:14:18.557Z" }, + { url = "https://files.pythonhosted.org/packages/67/6d/9095c93326d0f8a5469ab22480d02795f24c08f1e7f383c73316ff106347/tcmlib-1.5.0-py2.py3-none-win_amd64.whl", hash = "sha256:f7b62787214083d490b39d7650f1e477eeacc875e7f86799feb9aa1fd34460d4", size = 366719, upload-time = "2026-04-24T14:09:45.977Z" }, +] + [[package]] name = "tomli" version = "2.4.1" @@ -936,6 +1011,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d4/8f/ac36fde77e223297454c1e0aeb8888c169eaacf3163bb609e3af942c88cb/ty-0.0.59-py3-none-win_arm64.whl", hash = "sha256:987043ee9e021f49493d9135891ac69c1affeee0d4ad4480c5fa4d9c975fc91b", size = 11650921, upload-time = "2026-07-12T20:22:00.348Z" }, ] +[[package]] +name = "umf" +version = "1.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "tcmlib" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/c4/72/2e0182f4e6a727a15d0a8a99a82182a4f5bdec1a4f5767acfd2abdc72070/umf-1.1.0-py2.py3-none-manylinux_2_28_x86_64.whl", hash = "sha256:567152c5ee6b8e16cc56b29a8a9a6b918de1febf89877f71ea2e8247ef39fc32", size = 424455, upload-time = "2026-04-24T14:13:51.436Z" }, + { url = "https://files.pythonhosted.org/packages/2d/43/d89fee46bed22a461714a8786b833a89bff2f0a5e058b02eed3be842752e/umf-1.1.0-py2.py3-none-win_amd64.whl", hash = "sha256:a8c5ff84901fe6348715b6c1c85de4d195a6c69185d99ce4ebf7449b5d04011c", size = 289901, upload-time = "2026-04-24T14:08:31.647Z" }, +] + [[package]] name = "watchdog" version = "6.0.0" From d8b8038afaf368424e4c4f182b2faa8a835c437a Mon Sep 17 00:00:00 2001 From: Nardi Lam Date: Fri, 17 Jul 2026 11:05:23 +0200 Subject: [PATCH 2/9] feat: Allow the user to choose a preferred BLAS backend --- src/cyhighs/_blas_backend.py | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/src/cyhighs/_blas_backend.py b/src/cyhighs/_blas_backend.py index 4bc80d7..711f5c9 100644 --- a/src/cyhighs/_blas_backend.py +++ b/src/cyhighs/_blas_backend.py @@ -163,8 +163,20 @@ def configure_blas_backend() -> None: return debug = bool(os.environ.get("CYHIGHS_LBT_DEBUG")) + preferred_backend = str(os.environ.get("CYHIGHS_LBT_PREFER")) + + mkl = _locate_mkl() + openblas = _locate_bundled_openblas() + + # Select the backend based on user preference, or by default, prefer MKL if + # installed. + if preferred_backend.lower() == "mkl": + backend = mkl or openblas + elif preferred_backend.lower() == "openblas": + backend = openblas + else: + backend = mkl or openblas - backend = _locate_mkl() or _locate_bundled_openblas() lbt = _open_loaded_lbt() if debug: print(f"cyhighs: BLAS backend={backend!r} lbt={lbt!r}", file=sys.stderr) From a20189d92498e3251e3eb183e557714cc338352f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 09:33:35 +0000 Subject: [PATCH 3/9] docs: Document CYHIGHS_LBT_PREFER and CYHIGHS_LBT_DEBUG Add a section to the bundling guide explaining how to force a specific BLAS backend and how to enable verbose backend-selection diagnostics. --- docs/guide/bundling.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/guide/bundling.md b/docs/guide/bundling.md index ce039d0..0191e0d 100644 --- a/docs/guide/bundling.md +++ b/docs/guide/bundling.md @@ -59,6 +59,23 @@ Accelerate framework there, which ships with every macOS system, so lbt has nothing to forward and the `cyhighs[mkl]` extra has no effect (Intel has also never published MKL for macOS anyway). +### Choosing a backend explicitly + +By default `cyhighs` prefers MKL when it can find it and falls back to the +bundled OpenBLAS otherwise. Two environment variables let you override this, +read once when `cyhighs` is imported. + +`CYHIGHS_LBT_PREFER` picks a backend explicitly instead of relying on the +default preference. Set it to `mkl` to prefer MKL, falling back to OpenBLAS if +MKL cannot be found, the same as the default. Set it to `openblas` to force +the bundled OpenBLAS even if the `cyhighs[mkl]` extra is installed. + +`CYHIGHS_LBT_DEBUG` prints diagnostic information to stderr during backend +selection, including which backend was found, whether lbt itself was located, +and the result of the `lbt_forward` call. Set it to any non-empty value to +enable it. This is useful for confirming which backend is active or for +diagnosing why HiPO has no backend at all. + ## Platform coverage Wheels are published for Linux x86_64 and aarch64, macOS on Apple Silicon, and From 9314767dab2779ae5580e171c7a7e16aa8019f56 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 09:34:24 +0000 Subject: [PATCH 4/9] ci: Benchmark both OpenBLAS and MKL on Linux and Windows Add a second benchmark run using the cyhighs[mkl] extra and CYHIGHS_LBT_PREFER=mkl, tracked as its own regression series. Skipped on macOS, where MKL is not published and the extra has no effect. --- .github/workflows/tests.yml | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index c56c6fe..59d1282 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -53,7 +53,9 @@ jobs: # Run the performance benchmarks once per operating system, reusing the # extension already built above. One Python version is enough, since the # benchmarks measure the compiled solver rather than the Python layer. - - name: Run benchmarks + # This first run uses the bundled OpenBLAS, since the mkl extra is not + # installed yet. + - name: Run benchmarks (OpenBLAS) if: matrix.python-version == '3.12' run: uv run pytest benchmarks --benchmark-json=benchmark.json # Compare against the stored history and comment on regressions. Each @@ -61,7 +63,7 @@ jobs: # absolute timings differ per platform and a shared branch would race when # the three jobs push on main. The alert is advisory and never fails the # build, because hosted runners are too noisy for a hard threshold. - - name: Track benchmark regressions + - name: Track benchmark regressions (OpenBLAS) if: matrix.python-version == '3.12' uses: benchmark-action/github-action-benchmark@v1 with: @@ -76,3 +78,30 @@ jobs: fail-on-alert: false auto-push: ${{ github.event_name == 'push' }} save-data-file: ${{ github.event_name == 'push' }} + # Also benchmark against MKL, on the platforms it is published for (the + # cyhighs[mkl] extra has no effect on macOS, see the bundling guide). + # Reinstalling does not rebuild the extension, since the backend is + # chosen at import time rather than at compile time. + - name: Install the mkl extra + if: matrix.python-version == '3.12' && matrix.os != 'macos-latest' + run: uv sync --extra mkl + - name: Run benchmarks (MKL) + if: matrix.python-version == '3.12' && matrix.os != 'macos-latest' + env: + CYHIGHS_LBT_PREFER: mkl + run: uv run pytest benchmarks --benchmark-json=benchmark-mkl.json + - name: Track benchmark regressions (MKL) + if: matrix.python-version == '3.12' && matrix.os != 'macos-latest' + uses: benchmark-action/github-action-benchmark@v1 + with: + name: benchmarks-${{ matrix.os }}-mkl + tool: pytest + output-file-path: benchmark-mkl.json + github-token: ${{ secrets.GITHUB_TOKEN }} + gh-pages-branch: benchmarks-${{ matrix.os }}-mkl + benchmark-data-dir-path: dev/bench + alert-threshold: "120%" + comment-on-alert: true + fail-on-alert: false + auto-push: ${{ github.event_name == 'push' }} + save-data-file: ${{ github.event_name == 'push' }} From 00addba38275279d0864714e57dda9a659479d95 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 12:25:26 +0000 Subject: [PATCH 5/9] feat: Forward OpenBLAS thread-count control through lbt HiGHS calls openblas_get_num_threads/openblas_set_num_threads directly, believing it is linked against real OpenBLAS. Forward both to lbt's own lbt_get_num_threads/lbt_set_num_threads instead of a no-op stub, so HiGHS's thread-count control reaches whichever backend lbt is actually forwarding to, OpenBLAS or MKL. --- CMakeLists.txt | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index da4fc1a..a518398 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -521,13 +521,30 @@ if(_needs_openblas_extension_stub) # forwards), because HiGHS's build believes "openblas" is a real OpenBLAS # (see the BLAS/LAPACK section above). A static executable that links # libhighs_extras.a in full cannot leave this undefined (confirmed by a - # real CI failure linking HiGHS's own CLI, excluded from the build above); - # a shared library like _core tolerates it at link time, but would crash - # if this ever got called at runtime. Supply a no-op definition so _core - # can never hit that crash. This only affects OpenBLAS's own thread-count - # tuning, not correctness or which real BLAS lbt forwards to. + # real CI failure linking HiGHS's own CLI, excluded from the build above), + # and a shared library like _core would crash if this ever got called at + # runtime with no definition at all. Rather than a no-op, forward both + # openblas_get_num_threads() and openblas_set_num_threads() to lbt's own + # lbt_get_num_threads()/lbt_set_num_threads(), which lbt already routes to + # whichever real backend (OpenBLAS or MKL) is currently forwarded to (lbt + # has built-in knowledge of both libraries' thread-control functions, see + # lbt_register_thread_interface() in libblastrampoline.h), so HiGHS's + # thread-count control keeps working regardless of the active backend. set(_openblas_stub_c "${CMAKE_BINARY_DIR}/openblas_extension_stub.c") - file(WRITE "${_openblas_stub_c}" "void openblas_set_num_threads(int num_threads) { (void)num_threads; }\n") + file(WRITE "${_openblas_stub_c}" [[ +#include + +extern int32_t lbt_get_num_threads(void); +extern void lbt_set_num_threads(int32_t num_threads); + +int openblas_get_num_threads(void) { + return (int)lbt_get_num_threads(); +} + +void openblas_set_num_threads(int num_threads) { + lbt_set_num_threads((int32_t)num_threads); +} +]]) list(APPEND _core_extra_sources "${_openblas_stub_c}") endif() From 7a98f1937dfc6a0e40ea01dba3219a7d968554a8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 13:03:11 +0000 Subject: [PATCH 6/9] feat: Add CYHIGHS_USE_LBT to link OpenBLAS directly, bypassing lbt Off links the bundled OpenBLAS directly with no libblastrampoline forwarding layer in between, for local A/B testing against the default lbt-forwarded build. Both HiGHS's own BLAS detection and _core link straight to the real backend, so no thread-count stub is needed and _blas_backend.py no-ops (no lbt to configure). Real OpenBLAS needs libgfortran resolvable at link time, not just at runtime, unlike lbt which has no such dependency, so the no-lbt shim directory stages it there too. Verified end to end with a real local build: linked, imported, and solved an LP via IPM correctly with no lbt anywhere in the dependency tree (confirmed via ldd), and the full test suite passes against it. --- CMakeLists.txt | 290 ++++++++++++++++++++--------------- src/cyhighs/_blas_backend.py | 27 +++- 2 files changed, 192 insertions(+), 125 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index a518398..042cc68 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -5,19 +5,23 @@ # On Linux and Windows, HiPO's BLAS dependency is satisfied by # libblastrampoline (lbt) rather than by linking OpenBLAS directly: lbt is a # small MIT-licensed shared library that exports the standard BLAS/LAPACK ABI -# and forwards each call, at runtime, to whichever real BLAS implementation is -# named by the LBT_DEFAULT_LIBS environment variable (this is the same -# mechanism Julia's LinearAlgebra/HiGHS.jl use to switch BLAS backends without -# recompiling). cyhighs's own Python `__init__.py` sets that variable before -# importing the compiled extension: +# and forwards each call, at runtime, to whichever real BLAS implementation it +# has been pointed at (this is the same mechanism Julia's LinearAlgebra/ +# HiGHS.jl use to switch BLAS backends without recompiling). Once the compiled +# extension (and therefore lbt) is loaded, cyhighs's own +# src/cyhighs/_blas_backend.py calls lbt's `lbt_forward` C API to register: # -# * by default, to a prebuilt OpenBLAS bundled inside the wheel (so the +# * by default, a prebuilt OpenBLAS bundled inside the wheel (so the # package remains fully self contained with zero configuration, matching # the project's historical "no external system dependencies" promise); -# * to Intel MKL instead, if the optional `cyhighs[mkl]` extra is installed +# * Intel MKL instead, if the optional `cyhighs[mkl]` extra is installed # and its shared library can be located (Linux x86_64 / Windows x86_64 # only -- Intel has never published MKL for macOS or for ARM). # +# CYHIGHS_USE_LBT (below) turns this off, linking the bundled OpenBLAS +# directly with no lbt in the process at all. It exists for local A/B testing +# against the default lbt-forwarded build, not as a shipped configuration. +# # On macOS, none of this applies. Reading HiGHS's own cmake/FindHipoDeps.cmake # shows it unconditionally links Apple's Accelerate framework on Apple # platforms (in highs_link_blas()), regardless of anything found via @@ -77,6 +81,17 @@ find_package( # (v${HIGHS_VERSION}). set(HIGHS_VERSION 1.15.1) +# Off links the bundled OpenBLAS directly instead of routing it through +# libblastrampoline, with no forwarding layer in between. This exists for +# local A/B testing against the default lbt-forwarded build, not as a shipped +# configuration: with it off, the cyhighs[mkl] extra and the CYHIGHS_LBT_PREFER/ +# CYHIGHS_LBT_DEBUG environment variables (see src/cyhighs/_blas_backend.py) +# have no effect, since there is no lbt in the process to configure. Toggle it +# with, for example, `uv sync --config-settings=cmake.define.CYHIGHS_USE_LBT=OFF`, +# in a clean build directory since scikit-build-core otherwise reuses the same +# build directory (see build-dir in pyproject.toml) across runs. +option(CYHIGHS_USE_LBT "Route HiPO's BLAS through libblastrampoline" ON) + # The libblastrampoline release fetched as a prebuilt binary. The version, # the release tag's "+" suffix, and the per-triplet asset hashes below # are all confirmed real against a live release listing of @@ -271,101 +286,15 @@ endif() set(_bundled_backend_libs "") set(_needs_openblas_extension_stub FALSE) if(NOT APPLE) - set(_needs_openblas_extension_stub TRUE) - # Fetch libblastrampoline (lbt): the BLAS/LAPACK ABI HiPO links against. - # HiGHS's FindHipoDeps.cmake does not call find_library(NAMES openblas) - # directly (confirmed by reading that file). With BLAS_LIBRARIES/BLA_VENDOR - # unset, it tries find_package(OpenBLAS CONFIG) first (finds nothing here, - # since neither lbt nor the bundled OpenBLAS ships a CMake package config), - # then sets BLA_VENDOR=OpenBLAS and calls find_package(BLAS). CMake's own - # FindBLAS.cmake module handles that vendor with an internal - # find_library(NAMES openblas) that does respect - # CMAKE_LIBRARY_PATH/CMAKE_PREFIX_PATH, so staging lbt under the "openblas" - # name in a directory prepended to those two variables (below) still works, - # just through CMake's own module rather than a literal find_library() call - # in HiGHS's own code as originally assumed. - set(_lbt_hash "") - if(_lbt_triplet STREQUAL "aarch64-linux-gnu") - set(_lbt_hash 9cca820658a7206b3324bf4fb309154bf2427c81ebd0f6a448a761183a553cdb) - elseif(_lbt_triplet STREQUAL "aarch64-linux-musl") - set(_lbt_hash a50cd8b2cf54df66c324480e66a563d0a756173914f0399d6b17bd1f627def28) - elseif(_lbt_triplet STREQUAL "x86_64-linux-gnu") - set(_lbt_hash 76ddd4223122d9664f827fc940f7696bb4225cfd1cf6bb00b3acbeb524bcc609) - elseif(_lbt_triplet STREQUAL "x86_64-linux-musl") - set(_lbt_hash 759c0699e8675b1ca13d3f574dfab339f62825cae168a525549b669535207c92) - elseif(_lbt_triplet STREQUAL "x86_64-w64-mingw32") - set(_lbt_hash 4d301c454f1259d50db44a2a0af83cb441d3cbc790963c375ed27686d884624f) - endif() - set(_lbt_asset "libblastrampoline.v${CYHIGHS_LBT_VERSION}.${_lbt_triplet}.tar.gz") - set(_lbt_url - "https://github.com/JuliaBinaryWrappers/libblastrampoline_jll.jl/releases/download/libblastrampoline-v${CYHIGHS_LBT_VERSION}+${CYHIGHS_LBT_BUILD}/${_lbt_asset}") - set(_lbt_root "${CMAKE_BINARY_DIR}/lbt-prebuilt") - cyhighs_fetch_and_extract("${_lbt_url}" "${_lbt_hash}" "${_lbt_root}") - - if(WIN32) - file(GLOB _lbt_dll "${_lbt_root}/bin/libblastrampoline*.dll") - else() - file(GLOB _lbt_shared "${_lbt_root}/lib/libblastrampoline.*") - endif() - - # The directory the find_package(BLAS)/BLA_VENDOR=OpenBLAS lookup above is - # pointed at, via CMAKE_LIBRARY_PATH/CMAKE_PREFIX_PATH (prepended below). A - # copy of lbt lives here under the "openblas" name so that lookup resolves - # to lbt instead of a real BLAS, with no upstream HiGHS source change. - set(_blas_shim_dir "${CMAKE_BINARY_DIR}/blas-shim") - file(MAKE_DIRECTORY "${_blas_shim_dir}") - - if(WIN32) - if(NOT _lbt_dll) - message(FATAL_ERROR "No libblastrampoline DLL found in ${_lbt_root}/bin") - endif() - list(GET _lbt_dll 0 _lbt_dll) - file(COPY_FILE "${_lbt_dll}" "${_blas_shim_dir}/blastrampoline.dll") - # CMake's FindBLAS.cmake looks for an import library (openblas.lib) on - # Windows, not the DLL itself. The JLL release only ships a MinGW-style - # *.dll.a import library (confirmed by extracting the real archive), - # which MSVC cannot consume directly, so generate a fresh one. - cyhighs_generate_msvc_import_lib( - "${_blas_shim_dir}/blastrampoline.dll" - "${_blas_shim_dir}/openblas.def" - "${_blas_shim_dir}/openblas.lib") - # The JLL release is MinGW-built and ships with DWARF debug sections still - # attached (confirmed against a real built wheel: stripping it cut the DLL - # from 2.8 MB to 1.2 MB). cibuildwheel's Windows build runs outside an - # MSYS/MinGW shell (the same reason cyhighs_generate_msvc_import_lib above - # cannot assume dumpbin/lib.exe are on PATH), so a MinGW strip.exe is not - # guaranteed to be present; skip stripping rather than fail the build if - # none is found. - find_program(_mingw_strip strip) - if(_mingw_strip) - execute_process(COMMAND "${_mingw_strip}" --strip-unneeded "${_blas_shim_dir}/blastrampoline.dll") - endif() - else() - if(NOT _lbt_shared) - message(FATAL_ERROR "No libblastrampoline shared library found in ${_lbt_root}/lib") - endif() - # Preserve the whole libblastrampoline.so -> .so.N -> .so.N.N.N chain - # under its own names, not just one file renamed to "libopenblas.so". - # The linker embeds whatever SONAME is baked into the real versioned - # file (e.g. libblastrampoline.so.5) as _core's NEEDED entry, regardless - # of what name/path we link it under -- confirmed by a real CI failure - # at import time: "libblastrampoline.so.5: cannot open shared object - # file". Renaming only the copy find_package(BLAS) is pointed at left - # that actual SONAME target missing from the shim directory entirely. - file(COPY ${_lbt_shared} DESTINATION "${_blas_shim_dir}") - list(GET _lbt_shared 0 _lbt_shared_one) - file(COPY_FILE "${_lbt_shared_one}" "${_blas_shim_dir}/libopenblas.so") - endif() - - # The *default* lbt-forwarded backend (not linked directly -- see - # LBT_DEFAULT_LIBS in src/cyhighs/_blas_backend.py). Compiling OpenBLAS - # ourselves is deliberately avoided: it's what caused the AVX2/AVX512 build - # breakage this project hit and moved away from previously. conda-forge - # provides a prebuilt, plain-symbol OpenBLAS for every platform reached - # here except musl Linux (conda-forge is glibc-only); musllinux instead - # uses the Alpine package installed via apk in CI (see wheels.yml), - # discovered via find_library the same way this project found OpenBLAS - # before adopting lbt. + # The *default* backend: real OpenBLAS, forwarded to by lbt at runtime when + # CYHIGHS_USE_LBT is on (the default), or linked directly when it is off. + # Fetched either way. Compiling OpenBLAS ourselves is deliberately avoided: + # it's what caused the AVX2/AVX512 build breakage this project hit and + # moved away from previously. conda-forge provides a prebuilt, plain-symbol + # OpenBLAS for every platform reached here except musl Linux (conda-forge + # is glibc-only); musllinux instead uses the Alpine package installed via + # apk in CI (see wheels.yml), discovered via find_library the same way + # this project found OpenBLAS before adopting lbt. if(_libc STREQUAL "musl") find_library(_openblas_backend NAMES openblas REQUIRED) list(APPEND _bundled_backend_libs "${_openblas_backend}") @@ -419,17 +348,7 @@ if(NOT APPLE) endif() list(GET _openblas_lib 0 _openblas_backend) - if(WIN32) - # MSVC's linker needs a .lib import library, not the bare DLL, and - # conda-forge ships none for OpenBLAS (it's only linked here to force - # delvewheel to bundle it -- see the target_link_libraries comment - # below). Generate one the same way as for libblastrampoline above. - cyhighs_generate_msvc_import_lib( - "${_openblas_backend}" - "${_blas_shim_dir}/openblas-backend.def" - "${_blas_shim_dir}/openblas-backend.lib") - list(APPEND _bundled_backend_libs "${_blas_shim_dir}/openblas-backend.lib") - else() + if(NOT WIN32) list(APPEND _bundled_backend_libs "${_openblas_backend}") endif() @@ -445,6 +364,125 @@ if(NOT APPLE) endif() endif() + # The directory the find_package(BLAS)/BLA_VENDOR=OpenBLAS lookup below is + # pointed at, via CMAKE_LIBRARY_PATH/CMAKE_PREFIX_PATH (prepended below). + # HiGHS's FindHipoDeps.cmake does not call find_library(NAMES openblas) + # directly (confirmed by reading that file). With BLAS_LIBRARIES/BLA_VENDOR + # unset, it tries find_package(OpenBLAS CONFIG) first (finds nothing here, + # since neither lbt nor the bundled OpenBLAS ships a CMake package config), + # then sets BLA_VENDOR=OpenBLAS and calls find_package(BLAS). CMake's own + # FindBLAS.cmake module handles that vendor with an internal + # find_library(NAMES openblas) that does respect + # CMAKE_LIBRARY_PATH/CMAKE_PREFIX_PATH, so whatever is staged here under the + # "openblas" name is what HiGHS links, with no upstream HiGHS source change. + set(_blas_shim_dir "${CMAKE_BINARY_DIR}/blas-shim") + file(REMOVE_RECURSE "${_blas_shim_dir}") + file(MAKE_DIRECTORY "${_blas_shim_dir}") + + if(CYHIGHS_USE_LBT) + set(_needs_openblas_extension_stub TRUE) + # Fetch libblastrampoline (lbt): the BLAS/LAPACK ABI HiPO links against, + # staged here under the "openblas" name so it is what find_package(BLAS) + # above resolves to, instead of the real backend fetched above. + set(_lbt_hash "") + if(_lbt_triplet STREQUAL "aarch64-linux-gnu") + set(_lbt_hash 9cca820658a7206b3324bf4fb309154bf2427c81ebd0f6a448a761183a553cdb) + elseif(_lbt_triplet STREQUAL "aarch64-linux-musl") + set(_lbt_hash a50cd8b2cf54df66c324480e66a563d0a756173914f0399d6b17bd1f627def28) + elseif(_lbt_triplet STREQUAL "x86_64-linux-gnu") + set(_lbt_hash 76ddd4223122d9664f827fc940f7696bb4225cfd1cf6bb00b3acbeb524bcc609) + elseif(_lbt_triplet STREQUAL "x86_64-linux-musl") + set(_lbt_hash 759c0699e8675b1ca13d3f574dfab339f62825cae168a525549b669535207c92) + elseif(_lbt_triplet STREQUAL "x86_64-w64-mingw32") + set(_lbt_hash 4d301c454f1259d50db44a2a0af83cb441d3cbc790963c375ed27686d884624f) + endif() + set(_lbt_asset "libblastrampoline.v${CYHIGHS_LBT_VERSION}.${_lbt_triplet}.tar.gz") + set(_lbt_url + "https://github.com/JuliaBinaryWrappers/libblastrampoline_jll.jl/releases/download/libblastrampoline-v${CYHIGHS_LBT_VERSION}+${CYHIGHS_LBT_BUILD}/${_lbt_asset}") + set(_lbt_root "${CMAKE_BINARY_DIR}/lbt-prebuilt") + cyhighs_fetch_and_extract("${_lbt_url}" "${_lbt_hash}" "${_lbt_root}") + + if(WIN32) + file(GLOB _lbt_dll "${_lbt_root}/bin/libblastrampoline*.dll") + if(NOT _lbt_dll) + message(FATAL_ERROR "No libblastrampoline DLL found in ${_lbt_root}/bin") + endif() + list(GET _lbt_dll 0 _lbt_dll) + file(COPY_FILE "${_lbt_dll}" "${_blas_shim_dir}/blastrampoline.dll") + # CMake's FindBLAS.cmake looks for an import library (openblas.lib) on + # Windows, not the DLL itself. The JLL release only ships a MinGW-style + # *.dll.a import library (confirmed by extracting the real archive), + # which MSVC cannot consume directly, so generate a fresh one. + cyhighs_generate_msvc_import_lib( + "${_blas_shim_dir}/blastrampoline.dll" + "${_blas_shim_dir}/openblas.def" + "${_blas_shim_dir}/openblas.lib") + # The JLL release is MinGW-built and ships with DWARF debug sections + # still attached (confirmed against a real built wheel: stripping it + # cut the DLL from 2.8 MB to 1.2 MB). cibuildwheel's Windows build runs + # outside an MSYS/MinGW shell (the same reason + # cyhighs_generate_msvc_import_lib above cannot assume dumpbin/lib.exe + # are on PATH), so a MinGW strip.exe is not guaranteed to be present; + # skip stripping rather than fail the build if none is found. + find_program(_mingw_strip strip) + if(_mingw_strip) + execute_process(COMMAND "${_mingw_strip}" --strip-unneeded "${_blas_shim_dir}/blastrampoline.dll") + endif() + # A separate import library for the real backend, purely so delvewheel + # treats it as a normal dependency to vendor (see the + # target_link_libraries comment below). openblas.lib above points at + # blastrampoline.dll instead, for HiGHS's own BLAS resolution. + cyhighs_generate_msvc_import_lib( + "${_openblas_backend}" + "${_blas_shim_dir}/openblas-backend.def" + "${_blas_shim_dir}/openblas-backend.lib") + list(APPEND _bundled_backend_libs "${_blas_shim_dir}/openblas-backend.lib") + else() + file(GLOB _lbt_shared "${_lbt_root}/lib/libblastrampoline.*") + if(NOT _lbt_shared) + message(FATAL_ERROR "No libblastrampoline shared library found in ${_lbt_root}/lib") + endif() + # Preserve the whole libblastrampoline.so -> .so.N -> .so.N.N.N chain + # under its own names, not just one file renamed to "libopenblas.so". + # The linker embeds whatever SONAME is baked into the real versioned + # file (e.g. libblastrampoline.so.5) as _core's NEEDED entry, regardless + # of what name/path we link it under -- confirmed by a real CI failure + # at import time: "libblastrampoline.so.5: cannot open shared object + # file". Renaming only the copy find_package(BLAS) is pointed at left + # that actual SONAME target missing from the shim directory entirely. + file(COPY ${_lbt_shared} DESTINATION "${_blas_shim_dir}") + list(GET _lbt_shared 0 _lbt_shared_one) + file(COPY_FILE "${_lbt_shared_one}" "${_blas_shim_dir}/libopenblas.so") + endif() + else() + # No lbt: point find_package(BLAS) straight at the real backend fetched + # above, so HiGHS links it directly with no forwarding layer in between. + # This exists for local A/B testing (see the CYHIGHS_USE_LBT docstring + # near the top of this file), not as a shipped configuration. + if(WIN32) + cyhighs_generate_msvc_import_lib( + "${_openblas_backend}" + "${_blas_shim_dir}/openblas.def" + "${_blas_shim_dir}/openblas.lib") + list(APPEND _bundled_backend_libs "${_blas_shim_dir}/openblas.lib") + else() + # Stage the real SONAME (for _core's BUILD_RPATH to resolve it in the + # editable/build-tree case, using cyhighs_stage_runtime_lib defined + # above) and the "libopenblas.so" alias find_package(BLAS) looks for. + cyhighs_stage_runtime_lib("${_openblas_backend}" "${_blas_shim_dir}") + file(COPY_FILE "${_openblas_backend}" "${_blas_shim_dir}/libopenblas.so") + if(_libgfortran_backend) + # Unlike lbt (which only depends on libdl/libc), real OpenBLAS's + # Fortran-compiled kernels need libgfortran resolvable at link time, + # not just at runtime -- confirmed by a real link failure here + # ("undefined reference to `_gfortran_etime'") when it was missing + # from this directory, since HiGHS's own BLAS detection performs a + # real link check, not just a file-existence check. + cyhighs_stage_runtime_lib("${_libgfortran_backend}" "${_blas_shim_dir}") + endif() + endif() + endif() + list(PREPEND CMAKE_LIBRARY_PATH "${_blas_shim_dir}") list(PREPEND CMAKE_PREFIX_PATH "${_blas_shim_dir}") endif() @@ -559,10 +597,12 @@ if(CMAKE_DL_LIBS) target_link_libraries(_core PRIVATE ${CMAKE_DL_LIBS}) endif() -# Link the bundled OpenBLAS (+ libgfortran) into _core, so that -# auditwheel/delvewheel see it as a normal dependency to relocate and vendor, -# and so its transitive deps (libgfortran -> libquadmath) get pulled into the -# wheel too. Empty (a no-op) on Apple, where HiGHS links Accelerate directly. +# Link the bundled OpenBLAS (+ libgfortran) into _core. With CYHIGHS_USE_LBT +# on (the default), this is only so auditwheel/delvewheel see it as a normal +# dependency to relocate and vendor, and so its transitive deps +# (libgfortran -> libquadmath) get pulled into the wheel too; the actual BLAS +# calls go through lbt instead. With it off, this is the real, direct BLAS +# link. Empty (a no-op) on Apple, where HiGHS links Accelerate directly. target_link_libraries(_core PRIVATE ${_bundled_backend_libs}) # Make _core's own runtime dependencies loadable. This has to work in two @@ -578,13 +618,15 @@ if(NOT APPLE) # unrepaired build) and install them into the package directory (for the # wheel, which a real CI failure showed delvewheel does not otherwise # populate from the build-tree shim location on its own). + if(CYHIGHS_USE_LBT) + set(_windows_runtime_dlls "${_blas_shim_dir}/blastrampoline.dll" "${_openblas_backend}") + else() + set(_windows_runtime_dlls "${_openblas_backend}") + endif() add_custom_command(TARGET _core POST_BUILD COMMAND "${CMAKE_COMMAND}" -E copy_if_different - "${_blas_shim_dir}/blastrampoline.dll" - "${_openblas_backend}" - "$") - install(FILES "${_blas_shim_dir}/blastrampoline.dll" "${_openblas_backend}" - DESTINATION cyhighs) + ${_windows_runtime_dlls} "$") + install(FILES ${_windows_runtime_dlls} DESTINATION cyhighs) else() # On Linux, _core's NEEDED entries are libblastrampoline.so.5 (via HiGHS's # shim link) and the bundled OpenBLAS SONAME. Stage just those SONAME @@ -601,7 +643,9 @@ if(NOT APPLE) set(_runtime_bundle_dir "${CMAKE_BINARY_DIR}/runtime-bundle") file(REMOVE_RECURSE "${_runtime_bundle_dir}") file(MAKE_DIRECTORY "${_runtime_bundle_dir}") - cyhighs_stage_runtime_lib("${_lbt_shared}" "${_runtime_bundle_dir}") + if(CYHIGHS_USE_LBT) + cyhighs_stage_runtime_lib("${_lbt_shared}" "${_runtime_bundle_dir}") + endif() foreach(_backend_lib IN LISTS _bundled_backend_libs) cyhighs_stage_runtime_lib("${_backend_lib}" "${_runtime_bundle_dir}") endforeach() diff --git a/src/cyhighs/_blas_backend.py b/src/cyhighs/_blas_backend.py index 711f5c9..bea617a 100644 --- a/src/cyhighs/_blas_backend.py +++ b/src/cyhighs/_blas_backend.py @@ -26,6 +26,12 @@ On macOS, this module does nothing. HiGHS unconditionally links Apple's Accelerate framework there instead of going through lbt (see the BLAS/LAPACK section of the top-level CMakeLists.txt), so there is no backend to register. + +A build compiled with CMake's CYHIGHS_USE_LBT off (see CMakeLists.txt) links +OpenBLAS directly with no lbt in the process at all, for local A/B testing +against the default lbt-forwarded build. This module also does nothing there, +since there is no lbt to configure and HiGHS already has a working BLAS +backend linked in directly. """ from __future__ import annotations @@ -92,6 +98,21 @@ def _locate_bundled_openblas() -> str | None: return _first_match((core_dir, core_dir.parent / "cyhighs.libs"), pattern_name) +def _lbt_present() -> bool: + """Return whether this build links against libblastrampoline at all. + + A CYHIGHS_USE_LBT=OFF build (see CMakeLists.txt) links OpenBLAS directly, + so no lbt files exist next to `_core` at all. That is a normal, expected + build configuration and not a broken lbt install, so it is checked + separately from `_open_loaded_lbt` returning None, which does mean lbt + should be present but could not be located. + """ + core_dir = _core_dir() + search_dirs = (core_dir, core_dir.parent / "cyhighs.libs") + pattern_name = "*blastrampoline*.dll" if sys.platform == "win32" else "libblastrampoline.so*" + return _first_match(search_dirs, pattern_name) is not None + + def _open_loaded_lbt() -> ctypes.CDLL | None: """Return a handle to the libblastrampoline `_core` already loaded. @@ -156,11 +177,13 @@ def configure_blas_backend() -> None: """Point the already-loaded libblastrampoline at cyhighs's BLAS backend. Called after `_core` is imported, so lbt is already in the process. Does - nothing on macOS (no lbt), or if the user has set `LBT_DEFAULT_LIBS` - (respecting their choice), or if lbt or a backend cannot be located. + nothing on macOS (no lbt), on a CYHIGHS_USE_LBT=OFF build (also no lbt), + or if the user has set `LBT_DEFAULT_LIBS` (respecting their choice). """ if sys.platform == "darwin" or "LBT_DEFAULT_LIBS" in os.environ: return + if not _lbt_present(): + return debug = bool(os.environ.get("CYHIGHS_LBT_DEBUG")) preferred_backend = str(os.environ.get("CYHIGHS_LBT_PREFER")) From 8b63104851dd1fb2868f006b1bc502012b09c43e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 14:08:13 +0000 Subject: [PATCH 7/9] ci: Benchmark the static (no lbt) variant on Linux and Windows Rebuild with CYHIGHS_USE_LBT=OFF in a separate build-dir and track it as its own regression series, alongside the existing OpenBLAS and MKL runs. Skipped on macOS, where the flag has no effect. --- .github/workflows/tests.yml | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 59d1282..63d81e7 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -105,3 +105,35 @@ jobs: fail-on-alert: false auto-push: ${{ github.event_name == 'push' }} save-data-file: ${{ github.event_name == 'push' }} + # Also benchmark the "static" variant (CYHIGHS_USE_LBT=OFF, see + # CMakeLists.txt): OpenBLAS linked directly, with no libblastrampoline + # forwarding layer in between. This is a compile-time flag, unlike MKL + # above, so it needs a real rebuild; a separate build-dir keeps that + # from reusing the lbt-enabled build's CMake cache. Not run on macOS, + # where CYHIGHS_USE_LBT has no effect at all (HiGHS always links + # Accelerate directly there), so it would just duplicate the OpenBLAS + # benchmark above. + - name: Install the static (no lbt) variant + if: matrix.python-version == '3.12' && matrix.os != 'macos-latest' + run: > + uv sync + --config-settings=cmake.define.CYHIGHS_USE_LBT=OFF + --config-settings=build-dir=build-static + - name: Run benchmarks (static) + if: matrix.python-version == '3.12' && matrix.os != 'macos-latest' + run: uv run pytest benchmarks --benchmark-json=benchmark-static.json + - name: Track benchmark regressions (static) + if: matrix.python-version == '3.12' && matrix.os != 'macos-latest' + uses: benchmark-action/github-action-benchmark@v1 + with: + name: benchmarks-${{ matrix.os }}-static + tool: pytest + output-file-path: benchmark-static.json + github-token: ${{ secrets.GITHUB_TOKEN }} + gh-pages-branch: benchmarks-${{ matrix.os }}-static + benchmark-data-dir-path: dev/bench + alert-threshold: "120%" + comment-on-alert: true + fail-on-alert: false + auto-push: ${{ github.event_name == 'push' }} + save-data-file: ${{ github.event_name == 'push' }} From 599149df73747f1d34f8dd612d9af74b2d5b0d87 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 14:13:04 +0000 Subject: [PATCH 8/9] feat: Attach static (no lbt) wheels to GitHub releases Build a second wheel variant per Linux/Windows target with CYHIGHS_USE_LBT=OFF, tagged with a PEP 440 +static local version segment so it never collides with or gets picked up in place of the default lbt-forwarded wheel on PyPI. These are attached directly to the GitHub release instead, since PyPI has no way for pip to choose between two wheels of the same name, version, and platform tag. Document the static wheels and the existing CYHIGHS_LBT_PREFER/ CYHIGHS_LBT_DEBUG environment variables in the bundling guide. --- .github/workflows/wheels.yml | 58 ++++++++++++++++++++++++++++++++++-- docs/guide/bundling.md | 26 ++++++++++++++++ 2 files changed, 82 insertions(+), 2 deletions(-) diff --git a/.github/workflows/wheels.yml b/.github/workflows/wheels.yml index 43a9aed..8e35222 100644 --- a/.github/workflows/wheels.yml +++ b/.github/workflows/wheels.yml @@ -27,7 +27,7 @@ jobs: # (manylinux_2_28/musllinux_1_2 on Linux) or native toolchain (macOS, # Windows) per matrix entry. build_wheels: - name: Wheels on ${{ matrix.os }} + name: Wheels on ${{ matrix.os }}${{ matrix.variant == 'static' && ' (static)' || '' }} runs-on: ${{ matrix.os }} strategy: fail-fast: false @@ -37,9 +37,40 @@ jobs: - ubuntu-24.04-arm # Linux aarch64 -> manylinux_2_28 + musllinux_1_2 - macos-latest # macOS arm64 - windows-latest # Windows AMD64 + # The "static" variant builds with CYHIGHS_USE_LBT=OFF (see + # CMakeLists.txt): OpenBLAS linked directly, with no libblastrampoline + # forwarding layer in between. It is excluded on macOS, where the flag + # has no effect at all (HiGHS always links Accelerate directly there), + # so it would just duplicate the default build. + variant: [default, static] + exclude: + - os: macos-latest + variant: static steps: - uses: actions/checkout@v4 + # The static variant is not published to PyPI (see "Static (no lbt) + # wheels" in docs/guide/bundling.md): PyPI does not let two wheels with + # the same name, version, and platform tag coexist for users to choose + # between, pip would just pick one. Give it a PEP 440 local version + # segment instead, the same mechanism PyTorch uses for its CUDA/CPU + # variant wheels, so it is only ever installed when asked for by exact + # file or URL. Matching this in pyproject.toml itself, rather than + # renaming the wheel after the fact, keeps the wheel's filename and its + # internal metadata consistent. + - name: Tag the static variant's version + if: matrix.variant == 'static' + run: | + python3 -c " + import pathlib, re + path = pathlib.Path('pyproject.toml') + text = path.read_text() + new_text, count = re.subn( + r'(?m)^version = \"(\d[^\"]*)\"', r'version = \"\1+static\"', text, count=1) + assert count == 1, 'expected exactly one project.version line to update' + path.write_text(new_text) + " + - name: Build wheels uses: pypa/cibuildwheel@v3.2.0 env: @@ -72,11 +103,12 @@ jobs: # matrix entries; within those, it's a no-op in the manylinux_2_28 # container, which has no apk. CIBW_BEFORE_ALL_LINUX: "command -v apk && apk add --no-cache openblas-dev || true" + CIBW_CONFIG_SETTINGS: ${{ matrix.variant == 'static' && 'cmake.define.CYHIGHS_USE_LBT=OFF' || '' }} - name: Upload wheel artifacts uses: actions/upload-artifact@v4 with: - name: wheels-${{ matrix.os }} + name: ${{ matrix.variant == 'static' && 'static-wheels' || 'wheels' }}-${{ matrix.os }} path: ./wheelhouse/*.whl build_sdist: @@ -119,3 +151,25 @@ jobs: merge-multiple: true - name: Publish to PyPI uses: pypa/gh-action-pypi-publish@release/v1 + + publish_static_wheels: + name: Attach static wheels to the release + needs: build_wheels + if: github.event_name == 'release' + runs-on: ubuntu-latest + permissions: + contents: write + steps: + - name: Download static wheel artifacts + uses: actions/download-artifact@v4 + with: + pattern: static-wheels-* + path: static-wheels + merge-multiple: true + - name: Upload to the release + env: + GH_TOKEN: ${{ github.token }} + run: > + gh release upload "${{ github.event.release.tag_name }}" + static-wheels/*.whl + --repo "${{ github.repository }}" diff --git a/docs/guide/bundling.md b/docs/guide/bundling.md index 0191e0d..36bac25 100644 --- a/docs/guide/bundling.md +++ b/docs/guide/bundling.md @@ -76,6 +76,32 @@ and the result of the `lbt_forward` call. Set it to any non-empty value to enable it. This is useful for confirming which backend is active or for diagnosing why HiPO has no backend at all. +### Static (no lbt) wheels + +The published wheels always go through lbt on Linux and Windows, as described +above. A second, unpublished build exists for comparison, controlled by the +CMake option `CYHIGHS_USE_LBT`. With it off, HiGHS links the bundled OpenBLAS +directly and there is no lbt anywhere in the process at all. This is meant for +measuring whether lbt's forwarding adds any overhead, not for everyday use. +`CYHIGHS_LBT_PREFER`, `CYHIGHS_LBT_DEBUG`, and the `cyhighs[mkl]` extra all +have no effect on this build, since there is no lbt left to configure. It is +also skipped on macOS, where `CYHIGHS_USE_LBT` has no effect at all, since +HiGHS already links Accelerate directly there. + +These static wheels are not published to PyPI. PyPI does not let two wheels +with the same name, version, and platform tag coexist for `pip` to choose +between, so a plain `pip install cyhighs` always gets the default, +lbt-forwarded build. Instead, static wheels are attached directly to each +[GitHub release](https://github.com/nardilam/cyhighs/releases) for Linux and +Windows, with a `+static` local version segment in the filename (the same +scheme PyTorch uses for its CUDA and CPU wheel variants) so they never get +picked up by a normal install. Install one by pointing `pip` at its release +asset URL directly. + +```bash +pip install https://github.com/nardilam/cyhighs/releases/download/v0.2.2/cyhighs-0.2.2+static-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl +``` + ## Platform coverage Wheels are published for Linux x86_64 and aarch64, macOS on Apple Silicon, and From 5f55cb2585d8cec7512901e186d3083372ec89c0 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 17 Jul 2026 15:17:00 +0000 Subject: [PATCH 9/9] fix: Force bash for the static variant's version-tagging step Windows runners default run: steps to pwsh, which parses the whole script block as PowerShell rather than passing it through, and pwsh's backslash escaping rules differ from bash's. That broke the embedded Python one-liner's quoting, confirmed by a real CI failure ("The string is missing the terminator") on Wheels on windows-latest (static). --- .github/workflows/wheels.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/workflows/wheels.yml b/.github/workflows/wheels.yml index 8e35222..8e45548 100644 --- a/.github/workflows/wheels.yml +++ b/.github/workflows/wheels.yml @@ -60,6 +60,13 @@ jobs: # internal metadata consistent. - name: Tag the static variant's version if: matrix.variant == 'static' + # Explicit bash, not the default pwsh on Windows runners: pwsh parses + # this whole block as PowerShell script rather than passing it + # through, and its escaping rules differ from bash's (a real CI + # failure: pwsh treated \" inside the python -c string as a literal + # backslash followed by a real string terminator, not an escaped + # quote, and failed to parse the script at all). + shell: bash run: | python3 -c " import pathlib, re