[nanvix] E: Build array as .so (Phase 0 of .a -> .so migration)#690
Open
esaurez wants to merge 7 commits into
Open
[nanvix] E: Build array as .so (Phase 0 of .a -> .so migration)#690esaurez wants to merge 7 commits into
array as .so (Phase 0 of .a -> .so migration)#690esaurez wants to merge 7 commits into
Conversation
Updates `Makefile.nanvix` so that `python.elf` correctly serves as the "main module" against which extension `.so`s (numpy, ssl, lxml, future pip-installed wheels, ...) resolve their C and C++ runtime symbols at dlopen() time. This is the consumer-side companion to the Nanvix loader's STB_WEAK support (esaurez/nanvix#22) and is gated on the new libposix `pathconf` / `fpathconf` stubs (esaurez/nanvix#23) for the configure conftest to even produce an executable. Three coordinated link-flag changes to the `CONFIGURE_ENV` block: 1. `LIBS` segment 1 -- new `--whole-archive ... --no-whole-archive` block ahead of the existing `--start-group`. Forces every object from libposix, libc, libm, libstdc++, and libgcc into python.elf so the runtime symbols extension `.so`s depend on are embedded (and re-exported via `-Wl,--export-dynamic`, already present). Without this, the static linker drops unreferenced objects (e.g. `fscanf`, `longjmp`, `strtold_l` for numpy; `operator new/delete[]`, `__cxa_*`, `_Unwind_*`, `std::type_info` vtables for any C++ extension) and subsequent dlopen() of those `.so`s fails with "symbol not found". 2. `LIBS` segment 2 -- the existing `--start-group` is trimmed to just the external add-on libraries (sqlite3, ssl, crypto, z, bz2, lzma, ffi). It no longer re-lists libposix / libc / libm: those archives are already fully included by segment 1, so the external libs can resolve their references against the already-embedded objects. 3. Two new top-level Makefile vars `LIBSTDCXX := -lstdc++` and `LIBGCC := -lgcc`. The GCC driver resolves them against its built- in search paths (libgcc lives under a versioned `lib/gcc/i686- nanvix/<gcc-version>/` directory, which would be fragile to hardcode). Defined once at top level because the `-l` form is identical between the docker and host build paths. `LDFLAGS` is unchanged. The existing `-Wl,--allow-multiple-definition` flag is kept and the surrounding comment is expanded to honestly enumerate the duplicate-symbol categories the flag is masking (newlib long-double math helpers, libposix/libc env+isatty overlaps, libc/libm math helper overlaps, libgcc internal `__x86.get_pc_thunk.*` duplicates, etc.) -- the set is large and toolchain-build-version- dependent, and is the only practical workaround until the contributing upstreams are fixed. `.nanvix/config.py::configure_env()` -- an unused helper that mirrors `Makefile.nanvix`'s `CONFIGURE_ENV` -- is kept in sync (same `--whole-archive` LIBS, same LDFLAGS) and gains a docstring calling out the dead-code status. A separate small cleanup PR can delete the helper entirely. Validated end-to-end on the Nanvix microvm: CPython 3.12 + numpy 1.26.4 runs `import numpy`, `np.arange`, `np.dot`, `reshape`, `flatten`, broadcasting, all producing `NUMPY_TEST_OK`. Hello.py and the existing single-process / multi-process / standalone modes are unaffected by the change because the linker flags are not mode-conditional. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Stacked on top of the prior `[nanvix] E: Embed C/C++ runtime in
python.elf for .so dlopen support` PR.
Adds `libnvx_crt0.a` -- the executable-startup archive introduced by
the Nanvix `nvx-crt0` crate split -- to python.elf's `--whole-archive`
LIBS segment. This makes python.elf forward-compatible with the
planned Nanvix upstream change that removes `_start` / `_do_start` /
`c_trampoline` from `libposix.a`: once libposix drops the duplicate,
`libnvx_crt0.a` becomes the sole provider of those symbols.
What changed in `Makefile.nanvix`:
- New `LIBNVX_CRT0` variable (defined in both the docker and
host-build branches): absolute path to the sysroot copy of
`libnvx_crt0.a`.
- Existence check: parse-time error if `$(LIBNVX_CRT0)` is missing
from the sysroot, with a clear "update your sysroot" hint. Gated
on `MAKECMDGOALS` so `clean` and `distclean` still work against
older sysroots.
- `LIBS` line: prepend `$(LIBNVX_CRT0)` to the `--whole-archive`
group. Listed FIRST so that under the existing
`-Wl,--allow-multiple-definition` LDFLAG, the linker picks
`libnvx_crt0`'s `_start` over the duplicate copy `libposix.a`
currently still ships. This is an intentional behaviour change:
python.elf today and python.elf against a future stripped-down
libposix.a both use the same `_start` source (the standalone crt0
crate), avoiding subtle differences in startup behaviour across
the migration window.
- Comment block extended to document the ordering rationale and the
expected future state where libposix no longer carries startup
symbols.
`.nanvix/config.py::configure_env()` (the dead-code helper kept in
sync as documented in PR-10) is updated to mirror the new LIBS line.
Validated end-to-end:
- `./z build` succeeds; python.elf grows by ~500 bytes (the crt0
objects added under `--whole-archive`).
- `nm python.elf` shows `T _start` and `T _do_start` resolved.
- numpy 1.26.4 import + arithmetic + broadcasting test produces
`NUMPY_TEST_OK` on the Nanvix microvm.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds `ninja-build`, `python3-pip`, and `Cython<3` to the
toolchain-python docker image so that meson- and Cython-based Python
extension cross-builds (numpy, scipy, ...) work out-of-the-box,
without an apt/pip preamble on every `docker run` invocation.
What changed in `.nanvix/docker/Dockerfile`:
- Added `python3-pip` and `ninja-build` to the apt install list.
- Added `pip3 install --break-system-packages 'Cython<3'` (pinned for
numpy 1.26.x compatibility; lift the pin when bumping numpy).
- Added `rm -rf /usr/include/python3.12` after the install. The
`python3-pip`/`ninja-build` apt packages transitively pull in
`libpython3.12-dev`, whose headers under `/usr/include/python3.12`
would otherwise be picked up by meson's regen step ahead of the
Nanvix cross sysroot headers and silently corrupt the cross-build.
- Comment block explaining the rationale for each addition and the
`/usr/include/python3.12` purge.
Why this matters:
The numpy `.so` cross-build (validated end-to-end on 2026-05-27 with
the STB_WEAK loader fix landed) requires two tools that were not
present in the image as shipped:
- `ninja` — meson's default backend; missing it makes every
meson-based Python extension build fail immediately.
- `Cython` — used by `numpy/_build_utils/tempita.py` to template
`.pyx.in` files; without it the `numpy.random` codegen step fails.
Before this change, the workaround was to inject:
```bash
apt-get update -qq
apt-get install -qq -y --no-install-recommends ninja-build python3-pip
pip3 install --quiet --break-system-packages 'Cython<3'
rm -rf /usr/include/python3.12
```
into every numpy build invocation, which (a) was fragile, (b) required
the docker container to have outbound network access on every build
(non-hermetic), and (c) re-paid the apt install cost in CI every run.
Validated locally:
- `docker build -f .nanvix/docker/Dockerfile -t toolchain-python:pr13
.nanvix/docker/` succeeds.
- `docker run --rm <image> bash -c 'ninja --version'` → `1.11.1`.
- `docker run --rm <image> bash -c 'python3 -c "import Cython;
print(Cython.__version__)"'` → `0.29.37`.
- `docker run --rm <image> bash -c 'ls /usr/include/python3.12'` →
exits non-zero / "No such file or directory".
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… and 'origin/feat/toolchain-python-bake-ninja-cython' into phase0-array
Adds a small bash wrapper that sits in front of the real `i686-nanvix-gcc` and `i686-nanvix-g++` driver binaries in the toolchain-python docker image. The wrapper detects whether the invocation is producing an executable or a shared library and applies the correct linker flags for each case.
This mirrors the well-established `emcc` pattern used by emscripten and Pyodide, where the compiler-driver wrapper is what knows the difference between main-module and side-module builds, so consumers (cpython's Makefile.nanvix, future numpy / scipy / lxml meson builds, etc.) can use a single `LDFLAGS` env var without needing to encode the exe-vs-shared distinction themselves.
## Why this exists
`Makefile.nanvix` sets a single `LDFLAGS` on `./configure` that contains executable-specific linker flags:
- `-T <sysroot>/lib/user.ld` — executable layout script.
- `-no-pie` — disable PIE for executables.
- `-Wl,--no-dynamic-linker` — executable-only marker.
- `-Wl,--export-dynamic` — populate the main executable's `.dynsym`.
cpython propagates that same `LDFLAGS` to BOTH the main `python.elf` link AND every extension-module `.so` link via `PY_LDFLAGS`. For `.so` outputs these exe-only flags are wrong:
- `-T user.ld` tells `ld` to use an executable layout. When applied to a `-shared` link, `ld` treats the output as an exe and rejects undefined symbols, even those that should resolve at `dlopen()` time against the main exe's `.dynsym` (the Python C API symbols every extension references).
- `-no-pie` is incompatible with `-shared` (shared libraries must be PIC).
- `-Wl,--no-dynamic-linker` is meaningless for `.so`.
- `-Wl,--export-dynamic` is exe-only.
There is no clean place in cpython 3.12's `Modules/Setup` system to express "use this LDFLAGS for the exe link and that LDFLAGS for shared modules". cpython expects a single `LDFLAGS` shared between both.
The cleanest fix is to install a compiler-driver wrapper that does the split transparently: forwards exe builds unchanged, and strips exe-only flags + adds `-fPIC` for shared builds. That way `LDFLAGS` stays the same for everyone and the wrapper does the right thing per-invocation.
## What changed in `.nanvix/docker/`
- **New file `cc-wrapper.sh`** (~110 lines bash). Detects compile-only mode (`-c` / `-S` / `-E`), exe-link mode (no `-shared`), or shared-link mode (`-shared` present). Compile-only and exe modes forward to the real binary unchanged via `exec "$real_bin" "$@"`. Shared mode iterates the args and:
- Strips `-T <script>` (both `-T <script>` and `-T<script>` forms), `-no-pie`, `-Wl,--no-dynamic-linker`, `-Wl,--export-dynamic`, `-Wl,-T,<script>`, and bare `*.ld` argument files.
- Adds `-fPIC` if not already present.
- Uses a bash `filtered=()` array (not a string) so argv boundaries and quoting are preserved across the rewrite — `exec "$real_bin" "${filtered[@]}"` is argv-preserving for arguments containing spaces, empty strings, or glob metacharacters.
- **Dockerfile updated** to install the wrapper. The script is copied to `/opt/nanvix/bin/i686-nanvix-cc-wrapper.sh`, made executable, and the real `i686-nanvix-gcc` / `i686-nanvix-g++` binaries are renamed to `<name>.real`. Symlinks `<name>` -> `i686-nanvix-cc-wrapper.sh` are then installed. The wrapper picks the right `.real` binary based on `argv[0]`.
The install pattern is defensive: if the upstream base image already has a wrapper symlinked (some local-build flavours of `toolchain-python` carry an earlier wrapper), the symlink is removed and replaced — but only after asserting that the matching `<name>.real` file already exists, so a missing `.real` aborts the install rather than stranding the toolchain. If the real binary has not yet been renamed, it is moved to `<name>.real` in the same step.
## Validation
Built the image locally with the additions and smoke-tested all three wrapper modes against a trivial `int main(){return 0;}`:
- **Test 1 (exe link, no `-shared`):** wrapper transparent; real gcc handles normally.
- **Test 2 (simple `-shared`):** wrapper detects shared mode, adds `-fPIC`, link succeeds.
- **Test 3 (`-shared` plus the toxic exe-only flags `-T <script>` `-no-pie` `-Wl,--no-dynamic-linker`):** wrapper strips the toxic flags and the link succeeds with no spurious "cannot create executables" error.
Build-side end-to-end smoke test with cpython: applied a Phase-0-style change to `Modules/Setup.local` adding `*shared*\narray arraymodule.c`. With the wrapper in place, `array.cpython-312-i686-nanvix.so` (~190 KB) is produced and installed at the canonical `lib/python3.12/lib-dynload/` location; `PyInit_array` no longer appears in `python.elf`. Full `./z build` succeeds.
## What this unblocks
This wrapper is the foundational prerequisite for the broader `.a` -> `.so` migration documented in `nanvix-todo/cpython-static-to-shared-migration.md`. Every subsequent phase of that plan (peeling stdlib extension modules off `python.elf` and shipping them as `.so` files loaded via dlopen, exactly as upstream CPython works) depends on this wrapper being in place. Without it, every per-module conversion would need a per-call LDFLAGS hack.
## Backward compatibility
This change is purely additive at the image level — it adds the wrapper script and rewrites the gcc/g++ entry points to point at it. Existing build paths that don't pass `-shared` see no behavioural change (the wrapper falls through to the real binary unchanged). Existing `.so` build paths that DID work before (e.g., when LDFLAGS happened not to include `-T user.ld`) continue to work because the wrapper's strip-then-add-fPIC produces a strict superset of what bare `gcc -shared` would have done.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
First step of the .a -> .so migration plan documented in
`nanvix-todo/cpython-static-to-shared-migration.md`. Converts `array`
from a static built-in module into a shared `.so` loaded via dlopen
at runtime, matching upstream CPython's layout for stdlib extension
modules.
## Changes
* `.nanvix/docker.py` + `.nanvix/lxml.py` — `Setup.local` generation:
- Group the existing static modules under an explicit `*static*`
directive (was the implicit `#` comment that makesetup tolerated).
- Add a `*shared*` group with `array arraymodule.c` so cpython's
`setup.py build_ext` produces
`array.cpython-312-i686-nanvix.so` at
`lib/python3.12/lib-dynload/`.
* `.nanvix/test.py` — extend the `test_hello.py` smoke test to
`import array`, assert it is NOT in `sys.builtin_module_names`, and
exercise a tiny array operation. This proves the dlopen path
end-to-end on every test run; if the `.so` fails to load (loader
bug, missing PIE relocation, symbol-resolution issue, ...) the
smoke test fails immediately rather than silently regressing.
## Validation
`./z build` and `./z test`:
- `array.cpython-312.so` is produced at
`<sysroot>/lib/python3.12/lib-dynload/array.cpython-312.so`.
- Hello smoke prints
`CPYTHON_TEST_ARRAY_SO: array loaded via dlopen from
/lib/python3.12/lib-dynload/array.cpython-312.so`.
- lxml + HTTP server smoke continue to PASS.
## Dependencies
This PR depends on the following prereq PRs landing first:
- #682 — `--whole-archive` + `--export-dynamic` in
python.elf LIBS.
- #683 — bake `ninja` + `Cython` into the toolchain
image (needed by future meson-based extensions; harmless for
`array`).
- #684 — link `libnvx_crt0.a` into python.elf (will
become unnecessary once nanvix/nanvix#2453 lands but harmless
meanwhile).
- #687 — cc-wrapper for `-shared` vs exe link mode
(the `.so` link uses it).
- nanvix/nanvix#2450 — loader honours `STB_WEAK` undefined symbols.
- nanvix/nanvix#2453 — `libposix.a` is the sole owner of `sysalloc`
state (Fix 2 v3); without this, the first `dlopen` collides with
the heap.
Reviewers: please wait for those PRs (and the matching
`ghcr.io/nanvix/toolchain-python:latest` rebuild) before testing
this one.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR advances the Nanvix “static stdlib modules (.a in python.elf) → shared extensions (.so via dlopen)” migration by converting array into a shared extension module and updating the Nanvix build/test harness so the shared-module path is exercised during smoke testing. In the current stacked diff, it also includes supporting toolchain/build changes needed for reliable .so linking/loading (wrapper + link flags), aligning Nanvix’s layout more closely with upstream CPython’s lib-dynload model.
Changes:
- Generate
Modules/Setup.localwith explicit*static*and*shared*groups, addingarray arraymodule.cunder*shared*. - Extend the Nanvix smoke test to import
array, assert it’s not built-in, and run a small runtime check (verifying dlopen path). - Add/propagate toolchain/linker support for shared extensions (cc-wrapper in the toolchain Dockerfile and updated link inputs for
python.elfsymbol export/availability).
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
Makefile.nanvix |
Updates Nanvix link inputs and adds an existence guard for libnvx_crt0.a to support dlopen’d extensions resolving against python.elf. |
.nanvix/test.py |
Extends the hello/smoke script to validate array loads as a shared extension and performs a minimal correctness check. |
.nanvix/lxml.py |
Generates Modules/Setup.local with explicit *static*/*shared* groups and adds array as a shared module entry. |
.nanvix/docker/Dockerfile |
Installs host build helpers (pip/ninja/Cython) and installs a gcc/g++ wrapper to strip exe-only flags for -shared links. |
.nanvix/docker/cc-wrapper.sh |
New compiler-driver wrapper to distinguish exe vs shared links and filter toxic LDFLAGS for .so builds. |
.nanvix/docker.py |
Updates Docker-side Setup.local generation to use *static* and add the *shared* array entry. |
.nanvix/config.py |
Keeps the (currently unused) configure_env() helper in sync with updated linker inputs and adds libnvx_crt0 to toolchain paths. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+137
to
+143
| ifdef CONFIG_NANVIX | ||
| ifneq ($(filter clean distclean,$(MAKECMDGOALS)),$(MAKECMDGOALS)) | ||
| ifeq ($(wildcard $(LIBNVX_CRT0)),) | ||
| $(error libnvx_crt0.a not found at $(LIBNVX_CRT0). Update the Nanvix sysroot to one that ships libnvx_crt0.a (the nvx-crt0 crate must be present and built into the sysroot lib/ directory).) | ||
| endif | ||
| endif | ||
| endif |
Comment on lines
+451
to
+455
| # Phase 0 of the .a -> .so migration: `array` is now a shared | ||
| # extension at lib/python3.12/lib-dynload/array.cpython-312.so | ||
| # (built from `*shared* array arraymodule.c` in Setup.local). | ||
| # Asserting it is NOT in `sys.builtin_module_names` proves the | ||
| # dlopen path is exercised end-to-end; if the .so failed to load, |
This was referenced Jun 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First step of the
.a→.somigration plan documented innanvix-todo/cpython-static-to-shared-migration.md. Convertsarrayfrom a static built-in module into a shared extension loaded viadlopenat runtime, matching upstream CPython's layout for stdlib extension modules.The diff is small (3 files, 32 / -2 lines) and is intentionally a proof-of-concept for one module so the round-trip (configure →
setup.py build_ext→ install → import via dlopen) can be validated end-to-end before the bulk Tier-1 conversion lands.Changes
.nanvix/docker.py+.nanvix/lxml.py—Setup.localgeneration:*static*directive (was an implicit#comment thatmakesetuptolerated).*shared*group witharray arraymodule.cso CPython'ssetup.py build_extproducesarray.cpython-312-i686-nanvix.soatlib/python3.12/lib-dynload/..nanvix/test.py— extend thetest_hello.pysmoke test toimport array, assert it is NOT insys.builtin_module_names, and exercise a tiny array operation. This proves the dlopen path end-to-end on every test run; if the.sofails to load (loader bug, missing PIE relocation, symbol-resolution issue, ...) the smoke test fails immediately rather than silently regressing.Validation
./z buildand./z test:array.cpython-312.sois produced at<sysroot>/lib/python3.12/lib-dynload/array.cpython-312.so.CPYTHON_TEST_ARRAY_SO: array loaded via dlopen from /lib/python3.12/lib-dynload/array.cpython-312.so.regrtest: 39 of 40 batches pass. The remaining failure istest_structwith 32 real subtest failures (not import errors); this is investigated in a follow-up as it appears unrelated to the.a→.somigration itself.Dependencies
This PR depends on the following prereq PRs landing first. Reviewers: please wait for these (and the matching
ghcr.io/nanvix/toolchain-python:latestrebuild) before testing this one.--whole-archive+--export-dynamicinpython.elfLIBS, so dlopen'd.sos can resolve C/C++ runtime symbols against the main executable.ninjaandCythoninto the toolchain image (needed by future meson-based extensions; harmless forarray).libnvx_crt0.aintopython.elf(becomes unnecessary once [nvx] Cut over to nvx-crt0 startup crate nanvix#2453 lands, but harmless meanwhile).i686-nanvix-gccwrapper that strips exe-only LDFLAGS for.solinks. The.solink uses it.STB_WEAKundefined symbols. Without this, any.sowith a weak UND fails to dlopen.libposix.ais the sole owner ofsysallocstate ("Fix 2 v3"). Without this, the firstdlopencollides with the heap.dlfcnkeeps the leading/on absolute paths. Without this,dlopen()ofarray.cpython-312.sofails inside CPython'sregrtest(whichchdirs into a scratch dir).