Skip to content

Commit fc20b50

Browse files
LGroLGro
committed
Focus on sdist-only distribution, defer wheel building
- Remove cibuildwheel configuration from pyproject.toml - Update build-wheels.yml to sdist-only with Python 3.14 install tests - Add sdist install verification on Ubuntu and macOS before PyPI publish - Document wheel building issues and lessons learned in docs/ - Update README to reflect sdist-only installation - Simplify CMakeLists.txt (remove flatc workaround)
1 parent 764e752 commit fc20b50

4 files changed

Lines changed: 86 additions & 117 deletions

File tree

‎.github/workflows/build-wheels.yml‎

Lines changed: 19 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Build Wheels and sdist
1+
name: Build and publish sdist
22

33
on:
44
push:
@@ -31,55 +31,41 @@ jobs:
3131
name: sdist
3232
path: dist/*.tar.gz
3333

34-
build_wheels:
35-
name: Build wheels on ${{ matrix.os }}
34+
test_sdist_install:
35+
name: Test sdist install on ${{ matrix.os }}
36+
needs: build_sdist
3637
runs-on: ${{ matrix.os }}
3738
strategy:
3839
fail-fast: false
3940
matrix:
40-
os: [ubuntu-latest, macos-13, macos-14, windows-latest]
41-
41+
os: [ubuntu-latest, macos-latest]
4242
steps:
43-
- uses: actions/checkout@v4
44-
with:
45-
submodules: recursive
46-
47-
- uses: actions/setup-python@v5
43+
- uses: actions/download-artifact@v4
4844
with:
49-
python-version: '3.11'
50-
51-
- name: Install cibuildwheel
52-
run: pip install cibuildwheel
53-
54-
- name: Build wheels
55-
run: cibuildwheel --output-dir wheelhouse
56-
env:
57-
CIBW_ARCHS_LINUX: x86_64
58-
CIBW_ARCHS_MACOS: x86_64 arm64
59-
CIBW_ARCHS_WINDOWS: AMD64
60-
61-
- uses: actions/upload-artifact@v4
45+
name: sdist
46+
path: dist
47+
- name: Set up Python 3.14
48+
uses: actions/setup-python@v5
6249
with:
63-
name: wheels-${{ matrix.os }}
64-
path: wheelhouse/*.whl
50+
python-version: '3.14'
51+
- name: Install from sdist
52+
run: pip install dist/*.tar.gz
53+
- name: Install test dependencies
54+
run: pip install pytest
55+
- name: Run tests
56+
run: |
57+
python -c "from apsi import LabeledServer, LabeledClient; print('Import successful')"
6558
6659
publish:
6760
name: Publish to PyPI
68-
needs: [build_sdist, build_wheels]
61+
needs: [build_sdist, test_sdist_install]
6962
runs-on: ubuntu-latest
7063
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
7164
permissions:
7265
id-token: write
7366
steps:
74-
- uses: actions/download-artifact@v4
75-
with:
76-
pattern: wheels-*
77-
merge-multiple: true
78-
path: dist
79-
8067
- uses: actions/download-artifact@v4
8168
with:
8269
name: sdist
8370
path: dist
84-
8571
- uses: pypa/gh-action-pypi-publish@release/v1

‎README.md‎

Lines changed: 8 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# PyAPSI
22

3-
[![Actions Status](https://github.com/LGro/PyAPSI/workflows/Build%20Wheels%20and%20sdist/badge.svg)](https://github.com/LGro/PyAPSI/actions)
4-
[![PyPI - Wheel](https://img.shields.io/pypi/wheel/apsi)](https://pypi.org/project/apsi/)
3+
[![Actions Status](https://github.com/LGro/PyAPSI/workflows/Build%20and%20publish%20sdist/badge.svg)](https://github.com/LGro/PyAPSI/actions)
4+
[![PyPI](https://img.shields.io/pypi/v/apsi)](https://pypi.org/project/apsi/)
55
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/apsi)](https://pypi.org/project/apsi/)
66
[![License: MIT](https://img.shields.io/github/license/LGro/PyAPSI)](https://github.com/LGro/PyAPSI/blob/main/LICENSE)
77
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
@@ -11,34 +11,22 @@ Python wrapper for labeled and unlabeled asynchronous private set intersection
1111

1212
## Installation
1313

14-
### From PyPI (pre-built wheel)
14+
### From PyPI (builds from source)
1515

1616
```bash
1717
pip install apsi
1818
```
1919

20-
Pre-built wheels are available for:
21-
- **Linux**: x86_64 (manylinux_2_28)
22-
- **macOS**: x86_64 (Intel) and arm64 (Apple Silicon)
23-
- **Windows**: x86_64
24-
25-
These wheels are built with conservative CPU flags (`-march=x86-64 -mtune=generic`) for maximum compatibility across different CPU generations. This means they may not use AVX2/AVX-512 optimizations even if your CPU supports them.
26-
27-
### From source (optimized for your CPU)
28-
29-
To build with native CPU optimizations (AVX2, AVX-512, etc.):
30-
31-
```bash
32-
pip install apsi --no-binary apsi
33-
```
34-
35-
This compiles APSI and all dependencies from source, automatically selecting the best optimizations for your CPU. Build time is approximately 5-15 minutes.
20+
This downloads the source distribution and builds APSI + all dependencies locally.
21+
The build automatically selects the best CPU optimizations for your machine.
3622

3723
**Requirements:**
3824
- C++ compiler (GCC >= 9, Clang >= 10, or MSVC >= 2019)
3925
- CMake >= 3.13.4
4026
- Python >= 3.11
41-
- Internet access (dependencies are fetched during build)
27+
- Internet access (vcpkg downloads dependencies during build)
28+
29+
**Build time:** Approximately 5-15 minutes depending on your machine.
4230

4331
## Example
4432

@@ -95,13 +83,6 @@ pip install -e .
9583
pytest tests/
9684
```
9785

98-
### Building a Wheel Locally
99-
100-
```bash
101-
pip install build cibuildwheel
102-
cibuildwheel --platform linux --output-dir wheelhouse
103-
```
104-
10586
### Building a Source Distribution
10687

10788
```bash

‎docs/README.md‎

Lines changed: 59 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -4,35 +4,74 @@ This directory contains implementation plans for distributing PyAPSI across plat
44

55
## Plans
66

7-
- [Strategy 4: Hybrid — Source Distribution + Pre-built Wheels](./strategy-4-hybrid-wheels-sdist.md) — **IMPLEMENTED**
7+
- [Strategy 4: Hybrid — Source Distribution + Pre-built Wheels](./strategy-4-hybrid-wheels-sdist.md) — **PARTIALLY IMPLEMENTED (sdist only)**
88
- [Strategy 5: Conda-forge Distribution](./strategy-5-conda-forge.md) — Not started
99

10+
## Current State
11+
12+
### Implemented: Source Distribution (sdist)
13+
14+
The sdist workflow is active and:
15+
- Builds a source tarball on every push/PR
16+
- Tests installation from the tarball on Python 3.14 (Ubuntu + macOS)
17+
- Publishes to PyPI on tags only after successful install tests
18+
19+
Users install with:
20+
```bash
21+
pip install apsi --no-binary apsi
22+
```
23+
24+
This triggers a full local build with native CPU optimizations. Build time is ~5-15 minutes.
25+
26+
### Deferred: Pre-built Wheels
27+
28+
Wheel building was attempted but blocked by several issues. The work is documented below for future reference.
29+
30+
#### Issues encountered
31+
32+
1. **flatc segfault in manylinux containers**: The flatbuffers compiler (`flatc`) from vcpkg segfaults when running in manylinux Docker containers, particularly on ARM64 hosts running x86_64 emulation. This happens during APSI's CMake configuration when it tries to generate C++ headers from `.fbs` schema files.
33+
34+
2. **SEAL version mismatch**: vcpkg provides SEAL 4.3.0, but APSI's CMakeLists.txt requires `find_package(SEAL 4.1)`. Workaround: patch the version check at build time.
35+
36+
3. **pybind11 CMake compatibility**: pybind11 v2.9.2 is incompatible with CMake 3.31+. Fixed by upgrading to v2.13.6.
37+
38+
4. **vcpkg bootstrap permissions**: Zip extraction doesn't preserve execute permissions on Unix. Fixed with `os.chmod()`.
39+
40+
5. **macOS `/opt/vcpkg` permissions**: CI runners can't write to `/opt/`. Fixed by using `/tmp/vcpkg`.
41+
42+
6. **Windows vcpkg pre-installed**: Windows runners already have vcpkg at `C:\vcpkg`. Fixed by checking for existence before cloning.
43+
44+
#### Potential solutions for future wheel builds
45+
46+
1. **Pre-generate flatbuffers headers**: Commit the `*_generated.h` files to the APSI submodule or generate them in a separate build step before the manylinux container runs. This eliminates the need for `flatc` during the wheel build.
47+
48+
2. **Use a custom manylinux image**: Pre-build vcpkg + all dependencies + flatbuffers headers into a custom Docker image, avoiding the bootstrap and flatc issues entirely.
49+
50+
3. **Build wheels on native runners**: Avoid cross-architecture emulation by using native x86_64 runners for Linux wheels.
51+
52+
4. **Consider scikit-build-core**: A modern alternative to setuptools + CMake that handles many of these edge cases better.
53+
54+
#### Files to restore for wheel building
55+
56+
The following files contain wheel-building configuration that can be reactivated:
57+
58+
- `pyproject.toml`: Contains `[tool.cibuildwheel]` sections (currently kept but not used)
59+
- `setup.py`: Contains self-bootstrapping vcpkg logic (works for sdist installs)
60+
- `.github/workflows/build-wheels.yml`: Currently sdist-only, can be extended back to wheels
61+
1062
## Quick Comparison
1163

1264
| | Strategy 4 (Hybrid) | Strategy 5 (Conda-forge) |
1365
|---|---|---|
1466
| **Target users** | pip users (broadest) | conda users (data science, HPC) |
15-
| **Install command** | `pip install apsi` | `conda install -c conda-forge pyapsi` |
16-
| **CPU optimization** | sdist only (`--no-binary`) | Always (per-platform CI build) |
67+
| **Install command** | `pip install apsi --no-binary apsi` | `conda install -c conda-forge pyapsi` |
68+
| **CPU optimization** | Always (builds locally) | Always (per-platform CI build) |
1769
| **Build system** | vcpkg (self-bootstrapping) | conda-forge packages |
18-
| **Effort** | ~3-4 days | ~2 weeks + review time |
70+
| **Effort** | sdist: done, wheels: blocked | ~2 weeks + review time |
1971
| **Prerequisite work** | APSI git submodule (done) | Dependency feedstocks (SEAL, Kuku) |
20-
| **Can be done independently** | Yes (done) | Partially (needs APSI on conda-forge first) |
2172

2273
## Recommended Order
2374

24-
1. **Strategy 4** — DONE. Self-bootstrapping vcpkg works immediately for pip users.
25-
2. **Strategy 5** — Future. Once APSI and its dependencies are available on conda-forge.
26-
27-
## Current State
28-
29-
Strategy 4 has been implemented with the following changes:
30-
31-
- `setup.py`: Self-bootstrapping vcpkg logic
32-
- `CMakeLists.txt`: Simplified, uses vcpkg toolchain
33-
- `pyproject.toml`: cibuildwheel configuration for Linux/macOS/Windows
34-
- `MANIFEST.in`: sdist completeness
35-
- `.github/workflows/build-wheels.yml`: Wheel + sdist CI
36-
- `.github/workflows/cicd.yaml`: Multi-platform testing
37-
- `external/apsi`: Git submodule pointing to APSI v0.12.0
38-
- `README.md`: Updated installation instructions
75+
1. **Strategy 4 sdist** — DONE. Works for all platforms with native optimizations.
76+
2. **Strategy 4 wheels** — Deferred until flatc/pre-generation issues are resolved.
77+
3. **Strategy 5** — Future. Once APSI and its dependencies are available on conda-forge.

‎pyproject.toml‎

Lines changed: 0 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -15,48 +15,11 @@ pytest = "^7.1.2"
1515
pydocstyle = "^6.1.1"
1616
toml = "^0.10.2"
1717
build = "^0.8.0"
18-
cibuildwheel = "^2.16"
1918

2019
[tool.pydocstyle]
2120
match-dir = "apsi"
2221
convention = "google"
2322

24-
[tool.cibuildwheel]
25-
build = "cp311-* cp312-* cp313-* cp314-*"
26-
skip = "*-musllinux_*"
27-
archs = "auto64"
28-
test-requires = "pytest"
29-
test-command = "pytest {project}/tests"
30-
31-
[tool.cibuildwheel.linux]
32-
manylinux-x86_64-image = "manylinux_2_28"
33-
manylinux-aarch64-image = "manylinux_2_28"
34-
before-all = """
35-
yum install -y git zip && \
36-
git clone https://github.com/microsoft/vcpkg /opt/vcpkg && \
37-
/opt/vcpkg/bootstrap-vcpkg.sh && \
38-
/opt/vcpkg/vcpkg install seal[no-throw-tran]:x64-linux kuku:x64-linux log4cplus:x64-linux cppzmq:x64-linux flatbuffers:x64-linux jsoncpp:x64-linux
39-
"""
40-
environment = "VCPKG_ROOT_DIR=/opt/vcpkg"
41-
42-
[tool.cibuildwheel.macos]
43-
archs = "x86_64 arm64"
44-
before-all = """
45-
brew install git && \
46-
git clone https://github.com/microsoft/vcpkg /tmp/vcpkg && \
47-
/tmp/vcpkg/bootstrap-vcpkg.sh && \
48-
/tmp/vcpkg/vcpkg install seal[no-throw-tran] kuku log4cplus cppzmq flatbuffers jsoncpp
49-
"""
50-
environment = "VCPKG_ROOT_DIR=/tmp/vcpkg"
51-
52-
[tool.cibuildwheel.windows]
53-
archs = "AMD64"
54-
before-all = """
55-
if (!(Test-Path C:\\vcpkg\\vcpkg.exe)) { git clone https://github.com/microsoft/vcpkg C:\\vcpkg; C:\\vcpkg\\bootstrap-vcpkg.bat }
56-
C:\\vcpkg\\vcpkg install seal[no-throw-tran]:x64-windows-static kuku:x64-windows-static log4cplus:x64-windows-static cppzmq:x64-windows-static flatbuffers:x64-windows-static jsoncpp:x64-windows-static
57-
"""
58-
environment = "VCPKG_ROOT_DIR=C:\\vcpkg"
59-
6023
[build-system]
6124
requires = [
6225
"setuptools>=42",

0 commit comments

Comments
 (0)