BLD/RFC: compatibility with Python 3.15's Limited API

[neutrinoceros]

Context: there are currently 2 competing PEPs (803 and 809) that aim to specify a new Limited API for Python 3.15 that would be compatible with both GIL-enabled and free-threaded builds. Since astropy is already compliant with Python 3.11's Limited API my hope is that it would not be a great deal more work to also get compatibility with this other API, but of course it's too early to tell and I'm merely speculating here. This issue is intended to help keep track of progress on this matter, both locally and upstream

Upstream issues:

• Support for the new Limited API and Stable ABI in Python 3.15 (PEP 803/809) numpy/numpy#30704
• [INFO] Limited API and freethreading Python 3.15 cython/cython#7399
• Implement PEP 803 -- abi3t python/cpython#146636

[ngoldbaum]

I didn't realize astropy and yt had both been converted to use abi3 builds. Since astropy has more nontrivial use of the threading module, I thought I'd try getting a build working.

tl;dr: this is going to take some pretty substantial surgery in astropy C internals to get working.

In order to get everything to build, you'll need to install the branches of packaging tools that have experimental support for the opaque PyObject build. You'll also need to compile NumPy, SciPy, and PyErfa from source. For NumPy you'll need to use the experimental opaque branch on @kumaraditya303's fork of NumPy.

Once you do that, you'll also need to patch extension-helpers:

diff --git a/extension_helpers/_setup_helpers.py b/extension_helpers/_setup_helpers.py
index 3d27971..fe56596 100644
--- a/extension_helpers/_setup_helpers.py
+++ b/extension_helpers/_setup_helpers.py
@@ -9,6 +9,7 @@ import os
 import shutil
 import subprocess
 import sys
+import sysconfig
 from collections import defaultdict
 
 from setuptools import Extension, find_packages
@@ -25,6 +26,8 @@ __all__ = ["get_compiler", "get_extensions", "pkg_config"]
 
 log = logging.getLogger(__name__)
 
+FREE_THREADED_BUILD = bool(sysconfig.get_config_var("Py_GIL_DISABLED"))
+
 
 def get_compiler():
     """
@@ -154,6 +157,8 @@ def get_extensions(srcdir="."):
         for ext in ext_modules:
             ext.py_limited_api = True
             ext.define_macros.append(("Py_LIMITED_API", version_hex))
+            if FREE_THREADED_BUILD:
+                ext.define_macros.append(("_Py_OPAQUE_PYOBJECT", 1))
 
     return ext_modules
 

And astropy itself:

diff --git a/setup.py b/setup.py
index 71e7fcf65e..dddd107ad0 100755
--- a/setup.py
+++ b/setup.py
@@ -46,7 +46,7 @@ class EditableInstallWithStubs(editable_wheel):
 # Specify the minimum version for the Numpy C-API
 for ext in ext_modules:
     if ext.include_dirs and "numpy" in ext.include_dirs[0]:
-        ext.define_macros.append(("NPY_TARGET_VERSION", "NPY_1_25_API_VERSION"))
+        ext.define_macros.append(("NPY_TARGET_VERSION", "NPY_2_5_API_VERSION"))
         ext.define_macros.append(("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION"))
 
 setup(

That's sufficient to start a build with the correct NumPy API version, but unfortunately it leads to errors like this:

Details

astropy/wcs/include/astropy_wcs/astropy_wcs.h:14:3: error: unknown type name 'PyObject_HEAD'
   14 |   PyObject_HEAD
      |   ^
astropy/wcs/include/astropy_wcs/astropy_wcs.h:15:13: error: expected ';' at end of declaration list
   15 |   pipeline_t x;
      |             ^
      |             ;
In file included from astropy/wcs/src/astropy_wcs.c:7:
astropy/wcs/include/astropy_wcs/wcslib_wrap.h:14:3: error: unknown type name 'PyObject_HEAD'
   14 |   PyObject_HEAD
      |   ^
In file included from astropy/wcs/src/astropy_wcs.c:8:
astropy/wcs/include/astropy_wcs/wcslib_tabprm_wrap.h:15:3: error: unknown type name 'PyObject_HEAD'
   15 |   PyObject_HEAD
      |   ^
In file included from astropy/wcs/src/astropy_wcs.c:9:
astropy/wcs/include/astropy_wcs/wcslib_auxprm_wrap.h:10:3: error: unknown type name 'PyObject_HEAD'
   10 |   PyObject_HEAD
      |   ^
In file included from astropy/wcs/src/astropy_wcs.c:10:
astropy/wcs/include/astropy_wcs/wcslib_prjprm_wrap.h:10:5: error: unknown type name 'PyObject_HEAD'
   10 |     PyObject_HEAD
      |     ^
In file included from astropy/wcs/src/astropy_wcs.c:11:
astropy/wcs/include/astropy_wcs/wcslib_celprm_wrap.h:10:5: error: unknown type name 'PyObject_HEAD'
   10 |     PyObject_HEAD
      |     ^
In file included from astropy/wcs/src/astropy_wcs.c:12:
astropy/wcs/include/astropy_wcs/wcslib_units_wrap.h:15:3: error: unknown type name 'PyObject_HEAD'
   15 |   PyObject_HEAD
      |   ^
In file included from astropy/wcs/src/astropy_wcs.c:13:
astropy/wcs/include/astropy_wcs/wcslib_wtbarr_wrap.h:15:3: error: unknown type name 'PyObject_HEAD'
   15 |   PyObject_HEAD
      |   ^
In file included from astropy/wcs/src/astropy_wcs.c:14:
astropy/wcs/include/astropy_wcs/distortion_wrap.h:15:3: error: unknown type name 'PyObject_HEAD'
   15 |   PyObject_HEAD
      |   ^
astropy/wcs/include/astropy_wcs/distortion_wrap.h:16:22: error: expected ';' at end of declaration list
   16 |   distortion_lookup_t                    x;
      |                      ^
      |                      ;
In file included from astropy/wcs/src/astropy_wcs.c:15:
astropy/wcs/include/astropy_wcs/sip_wrap.h:15:3: error: unknown type name 'PyObject_HEAD'
   15 |   PyObject_HEAD
      |   ^
astropy/wcs/include/astropy_wcs/sip_wrap.h:16:8: error: expected ';' at end of declaration list
   16 |   sip_t x;
      |        ^
      |        ;
astropy/wcs/src/astropy_wcs.c:92:24: error: no member named 'x' in 'struct (unnamed at astropy/wcs/include/astropy_wcs/astropy_wcs.h:13:9)'
   92 |   pipeline_free(&self->x);
      |                  ~~~~  ^
astropy/wcs/src/astropy_wcs.c:109:27: error: no member named 'x' in 'struct (unnamed at astropy/wcs/include/astropy_wcs/astropy_wcs.h:13:9)'
  109 |     pipeline_clear(&self->x);
      |                     ~~~~  ^
astropy/wcs/src/astropy_wcs.c:155:13: error: no member named 'x' in 'struct (unnamed at astropy/wcs/include/astropy_wcs/astropy_wcs.h:13:9)'
  155 |       self->x.det2im[i] = &(((PyDistLookup*)py_det2im[i])->x);
      |       ~~~~  ^
astropy/wcs/src/astropy_wcs.c:155:60: error: no member named 'x' in 'struct (unnamed at astropy/wcs/include/astropy_wcs/distortion_wrap.h:14:9)'
  155 |       self->x.det2im[i] = &(((PyDistLookup*)py_det2im[i])->x);
      |                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  ^
astropy/wcs/src/astropy_wcs.c:170:11: error: no member named 'x' in 'struct (unnamed at astropy/wcs/include/astropy_wcs/astropy_wcs.h:13:9)'
  170 |     self->x.sip = &(((PySip*)py_sip)->x);
      |     ~~~~  ^
astropy/wcs/src/astropy_wcs.c:170:39: error: no member named 'x' in 'struct (unnamed at astropy/wcs/include/astropy_wcs/sip_wrap.h:14:9)'
  170 |     self->x.sip = &(((PySip*)py_sip)->x);
      |                     ~~~~~~~~~~~~~~~~  ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]

All of the types that rely on extending PyObject statically will need to be updated to use heap types. Additionally, all the C extensions defined with PyModuleDef_HEAD_INIT will need to be converted to use the new PEP 793 initialization API.

I didn't realize astropy had raw C extensions as well as cython-generated extensions. Packages using bindings generators with no "manual" C extensions will have an easier time supporting the new stable ABI.

[pllim]

I don't think removing raw C extensions is feasible request. I think some are bundled and beyond our control.

[neutrinoceros]

Thanks for plugging in, Nathan.

To some extent, the patch you proposed for extension helpers would be a minimal version of astropy/extension-helpers#132, and I'd support upstreaming it since my PR is stuck in limbo.

Packages using bindings generators with no "manual" C extensions will have an easier time supporting the new stable ABI.

Yeah, that's been my experience with complying to the current abi3. Astropy took much longer to adapt than any other packages I worked on for precisely this reason. That said I'm still unsure that it'd be worth converting to generated low level code, but I'm considering exploring this idea, FWIW. In the mean time I can do the manual conversion to heap types as I already did recently.

[ngoldbaum]

I don't think removing raw C extensions is feasible request.

Didn't mean to imply that.

and I'd support upstreaming it since my PR is stuck in limbo

It shouldn't be upstreamed until/if PEP 803 is accepted by the Python steering council. This is all to facilitate testing in support of PEP 803.

In the mean time I can do the manual conversion to heap types as I already did recently.

Nice, that's the hardest refactor. Supporting PEP 793 module initalization is a lot easier.

[neutrinoceros]

It shouldn't be upstreamed until/if PEP 803 is accepted by the Python steering council.

good point. I'll keep your patch to a branch then.

[neutrinoceros]

I checked everything you described to my fork of astropy under a exp/pep803 branch, and tried building locally with

uv sync --only-group build -p 3.15 && EXTENSION_HELPERS_PY_LIMITED_API=cp315 uv sync --inexact

and to my surprise, the builds appears successful. I must be missing something. Do you see any relevant difference with what you did ?

[da-woods]

I think it's _Py_OPAQUE_PYOBJECT that you're missing

[neutrinoceros]

Ah, of course, I forgot it was only enabled with free-threaded builds. I'll try again with 3.15t

[da-woods]

Ah, of course, I forgot it was only enabled with free-threaded builds. I'll try again with 3.15t

No - that shouldn't actually be necessary. The idea is that it works the same on both freethreaded and non-freethreaded builds. Of course, eventually you'll want to test it on both, but you can build it with either.

[da-woods]

[...] Ah, but Nathan's set up extensionhelpers to only add that flag on freethreaded builds.... That isn't required though - it's just the way he's set it up.

[ngoldbaum]

setuptools needs new UI to enable that and I didn't want to bikeshed over it until I get the attention of a setuptools maintainer

[neutrinoceros]

I tried again building my branch with

uv venv -p 3.15t
uv pip install --group build pyproject.toml # automatically use patched numpy and extension-helpers
EXTENSION_HELPERS_PY_LIMITED_API=cp315 uv pip install -e . --no-build-isolation

but this immediately stops from setuptools:

ValueError: `py_limited_api='cp315'` not supported. `Py_LIMITED_API` is currently incompatible with `Py_GIL_DISABLED`. See https://github.com/python/cpython/issues/111506.

@ngoldbaum, did you also patch setuptools in your attempt ? or am I still missing something ?