Skip to content

Replace Boost.Python with pybind11 - #176

Merged
vespakoen merged 31 commits into
aewallin:masterfrom
CalaW:pybind11
Oct 16, 2026
Merged

vespakoen merged 31 commits into
aewallin:masterfrom
CalaW:pybind11

Conversation

@CalaW

@CalaW CalaW commented Apr 19, 2025 •

Copy link
Copy Markdown
Contributor

This PR replaces Boost.Python with pybind11, which has become the de facto standard for Python bindings in C++.

Key improvements:

  • Simplified binding code, thanks to pybind11’s automatic handling of standard containers.
  • Easier build process, as pybind11 is a lightweight, header-only library.
  • Additionally, this PR includes fixes for existing CI errors. I'm happy to split those into a separate PR if preferred.

This should resolve #164 and close #180.

@CalaW
CalaW marked this pull request as ready for review April 25, 2025 10:02
@CalaW

CalaW commented May 24, 2025

Copy link
Copy Markdown
Contributor Author

Hi @aewallin @vespakoen, just following up on this PR. If either of you happens to have some spare time, would you mind giving it a look or testing it out? I’d be happy to make any necessary changes. Thanks in advance!

@SpectralVectors

Copy link
Copy Markdown

Hi @CalaW - not sure what you needed from this lib, but I built a set of wheels from your branch for Python 3.13 and uploaded them here if you want them: https://github.com/SpectralVectors/opencamlib/releases/tag/Python313_Wheels
Thanks!

@vespakoen

Copy link
Copy Markdown
Collaborator

Cool! I will take a look at this next week, I hope to also upgrade Boost to the modern version which should remove a lot of the install script.
I think we still use boost for other stuff, right?
Anyways, i'll take a look soon, thanks for all this effort!

@CalaW

CalaW commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Hi @vespakoen, Thanks for taking a look!

Yes, Boost is still required. This PR only replaces Boost.Python with pybind11. We still use Boost.Graph, for example boost/graph/adjacency_list.hpp.

I have already replaced some uses of BOOST_FOREACH, boost::tie, and Boost.Math with the standard library. I can also make another PR to replace the remaining ones.

@vespakoen

Copy link
Copy Markdown
Collaborator

Removing boost completely would be great, it saves a lot of hassle with their somewhat weird build system.

Boost did gain some better compatibility with CMake though, which might help.
And Boost.Python specifically, was a pain to cross compile, so this will already help a lot!

Maybe getting rid of Boost completely should be a follow up PR, I'd focus on getting this to work on all the platforms.

Will look into this soon...

@vespakoen

Copy link
Copy Markdown
Collaborator

Did you also consider nanobind? (https://github.com/wjakob/nanobind) it seems to be very similar in syntax so might be worth a try?

@CalaW

CalaW commented Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

Yes, I did consider nanobind. At that time, it was still relatively new, while pybind11 was already the de facto standard and had a much larger user base (3.4k vs 18k star).

From what I have seen, nanobind does have some advantages, especially smaller binaries and lower binding overhead. However, for ocl, I believe most of the heavy computation happens in C++, so I don’t expect the Python binding overhead to be a bottleneck.

@vespakoen

vespakoen commented Jul 22, 2026 •

Copy link
Copy Markdown
Collaborator

I opened a pull request to your fork, If that lands I think I am fine with merging this.
Do you think it's better to do a major release since we changed the binding generation from Boost.Python to pybind11? I guess the API actually stayed the same?
I think it might be a good signal to package maintainers that something bigger changed, since the packagers will have to swap out Boost.Python for pybind11 as well...

Let me know what you all think ;)

EDIT: Just saw that the README also needs some updating

@CalaW

CalaW commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Thanks! I will check your PR and update the README as well.

I am not fully sure that the Python API is completely stable, because the project currently does not have a complete test framework for checking API compatibility. I run all python examples and test file and they looked fine. I tried to keep the exposed classes, functions, and method names the same, but there may still be some behavior differences.

For example, AdaptiveWaterline.getXFibers, behaves differently in the old Boost.Python bindings. The old binding drops all empty elements before returning the list:

/// return a list of xfibers to python
boost::python::list getXFibers() const {
boost::python::list flist;
BOOST_FOREACH( Fiber f, xfibers ) {
if (!f.empty()) {
Fiber_py f2(f);
flist.append(f2);
}
}
return flist;
}

However, the underlying C++ implementation inherits the behavior from Waterline, which returns the complete list:

boost::python::list py_getXFibers() const {
boost::python::list flist;
std::vector<Fiber> xfibers = *( subOp[0]->getFibers() );
BOOST_FOREACH( Fiber f, xfibers ) {
Fiber_py f2(f);
flist.append(f2);
}
return flist;
}

In the pybind11 version, I preserved the behavior of the C++ implementation instead but it's still easy to modify.

https://github.com/CalaW/opencamlib/blob/5470a8977e2b0540a863c61a3c7b038edfdcf61a/src/algo/waterline.hpp#L54-L55

Speaking of versioning, I noticed that the current versioning seems to be date-based, with the latest release being 2023.01.11, rather than semantic versioning. In that case, maybe we can make a new dated release and clearly mention that it contains a major binding and packaging change.

CalaW and others added 8 commits July 31, 2026 22:06
The previous Boost.Python override support for `MillingCutter` was incomplete and effectively unusable:
- `vertexDrop` was not virtual, so Python overrides were never called by `dropCutter()`.
- Several required virtual functions (e.g. height/width queries) were not exposed, making it impossible to implement a complete custom cutter.
- `offsetCutter()` returns an owning raw pointer, which is incompatible with Python object lifetime and could lead to dangling pointers or crashes.

Even if these issues were addressed, overriding the low-level contact routines from Python would not be a practical or useful extension mechanism.

The API was introduced incorrectly and is not known to have been used, so removing it does not introduce a compatibility concern.
Stop creating and activating a project-local virtual environment.
Install and test with the currently selected Python executable instead.
CalaW added 10 commits August 3, 2026 02:44
- add GIL release
- remove CCType package level export
- change some python constructor signature to match cpp
- Use a local TSP graph to prevent leaks and double frees
- Reset tour state between runs and handle empty/singleton inputs
- Restore EllipsePosition's default constructor and type annotation
- Make offset cutter ownership explicit and adjust fiber return policy
- Normalize binding source formatting
@CalaW

CalaW commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor Author

Here's a full review by GPT-6 Astra:

This PR replaces Boost.Python with pybind 11, simplifies packaging, and fixes several build and lifetime issues. This summary includes the current local follow-ups.

  • Python build: removes compiled Boost.Python and its wrapper classes. Boost headers remain required; pip installs pybind 11 and scikit-build-core automatically.
  • Python support: raises the minimum to 3.8, adds wheel configurations through 3.14, and includes type stubs plus py.typed.
  • Installer: respects the selected Python environment, improves argument validation and dependency installation, and forwards build options correctly to pip.
  • Native fixes: repairs TSP initialization/reset/repeated runs, fixes the triangle-query memory leak, and moves Python-specific result conversion out of core C++.
  • CI/CD: updates runners/actions, fixes ARM Boost discovery and macOS Intel/OpenMP setup, and repairs Emscripten compilation and inheritance registration. The full local Emscripten release build and runtime smoke test pass.

The Python migration includes observable API changes, beyond replacing the binding framework:

Area Behavior change
Threading Long-running bound operations release the GIL, allowing other Python threads to run. This does not make concurrent access to the same operation safe.
Object lifetimes Operations retain their supplied surfaces, cutters, and paths. Temporary Python inputs therefore remain alive while referenced by the operation.
Constructors Zero-argument construction is no longer exposed for Line, Triangle, Arc, Fiber, Ellipse, Bbox, or MillingCutter. Supported defaults such as Point(), Path(), STLSurf(), Interval(), and EllipsePosition() remain.
Custom cutters Python overrides of MillingCutter methods no longer receive callbacks from native operations. Custom cutting behavior must use a native implementation.
Wrapper types Public *_base scaffolding disappears. Returned triangles and fibers use the directly bound types.
Point APIs CCPoint and CLPoint now inherit Python-visible Point operations. Point.isInside() gains triangle and segment overloads.
Path results getTypeSpanPairs() returns a list of tuples, replacing inner lists. Unpacking works; modifying a pair in place does not.
Minimum Z minimumZ becomes a real property forwarding to getZ() / setZ(). Previously, assignment could merely create an unrelated Python attribute.
Adaptive waterlines getXFibers() and getYFibers() now include empty fibers. Callers requiring intersections should filter using fiber.getInts().
Additional APIs Exposes SimpleWeave, SmartWeave, and TSPSolver; TSP handles empty/singleton input and repeated run/reset safely.
Representations Enum text, generated signatures, and exception messages change with pybind 11; avoid relying on their exact historical strings.

Compatibility checks found matching geometry in the tested BlenderCAM and FreeCAD scenarios. Waterline loop starting points and component order can differ, which may affect entry/travel positions even when contours match; exact historical ordering is not guaranteed.

@CalaW

CalaW commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

Hi @vespakoen I think this PR is ready to merge. I’ve run extensive compatibility tests on the Python bindings and documented the intentional behavior changes. I’ve also updated the README and CI/CD workflows, fixed the build issues we encountered.

@CalaW

CalaW commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

The python compatibility tests includes all 255 inventoried public Python API members, including methods, constructors, and properties, plus object lifetimes and representative BlenderCAM and FreeCAD usage patterns.

See this commit.

@vespakoen

Copy link
Copy Markdown
Collaborator

Great stuff! I am fine with merging this in.

@aewallin any objections to this?

@sliptonic

Copy link
Copy Markdown
Contributor

FWIW, I built and tested the FreeCAD unit test suite against the PR branch. Everything is green on our side.
Really looking forward to getting this merged. I think if we could merge this and #177, #180, and #170, it would be good to tag a release.

Going beyond this, @aewallin might not have time or interest to maintain this library. If he's interested in some help, we could suggest moving to a github organization that he controls where he could add contributors to help.

I'm pretty weak on the C++ but willing to test, advise, and cheer people on. :-)

@vespakoen
vespakoen merged commit 064bb87 into aewallin:master Oct 16, 2026
12 of 13 checks passed
@CalaW

CalaW commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

Hey @sliptonic thanks for testing on FreeCAD. And thank you @vespakoen for reviewing this, glad to see this finally merged!
I'd also be happy to help maintain OpenCAMLib! I'm Chen, a robotics PhD student at Tsinghua University. ocl is an important dependency in my research, and we expect to keep using it for years to come. I'm planning to graduate in 2030, so I'll be able to dedicate time to maintaining and improving the library over the next few years.
I think moving to a GitHub organization could be a good idea for long-term maintenance, if @aewallin is open to it. Either way, I'd be happy to take on a maintainer role and help keep the project moving forward!

@CalaW
CalaW deleted the pybind11 branch October 16, 2026 19:01
@vespakoen

Copy link
Copy Markdown
Collaborator

I would be happy to have you onboard as a contributer, this is for aewallin to decide though, you got my vote anyways ;)

I was added as a contributor at some point to this repository, which seems to work fine.

Are there any advantages to having a github organisation for this?

@sliptonic

Copy link
Copy Markdown
Contributor

An organization can outlive an individual account. Multiple owners are possible. Basically, It reduces the bus factor. Otherwise, not necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support python3.12, Blender 4.0.2 uses python3.12

4 participants