[nanvix] E: Phase 1A — build 11 Tier-1 data primitives as .so#691
Closed
esaurez wants to merge 8 commits into
Closed
[nanvix] E: Phase 1A — build 11 Tier-1 data primitives as .so#691esaurez wants to merge 8 commits into
esaurez wants to merge 8 commits into
Conversation
Updates `Makefile.nanvix` so that `python.elf` correctly serves as the "main module" against which extension `.so`s (numpy, ssl, lxml, future pip-installed wheels, ...) resolve their C and C++ runtime symbols at dlopen() time. This is the consumer-side companion to the Nanvix loader's STB_WEAK support (esaurez/nanvix#22) and is gated on the new libposix `pathconf` / `fpathconf` stubs (esaurez/nanvix#23) for the configure conftest to even produce an executable. Three coordinated link-flag changes to the `CONFIGURE_ENV` block: 1. `LIBS` segment 1 -- new `--whole-archive ... --no-whole-archive` block ahead of the existing `--start-group`. Forces every object from libposix, libc, libm, libstdc++, and libgcc into python.elf so the runtime symbols extension `.so`s depend on are embedded (and re-exported via `-Wl,--export-dynamic`, already present). Without this, the static linker drops unreferenced objects (e.g. `fscanf`, `longjmp`, `strtold_l` for numpy; `operator new/delete[]`, `__cxa_*`, `_Unwind_*`, `std::type_info` vtables for any C++ extension) and subsequent dlopen() of those `.so`s fails with "symbol not found". 2. `LIBS` segment 2 -- the existing `--start-group` is trimmed to just the external add-on libraries (sqlite3, ssl, crypto, z, bz2, lzma, ffi). It no longer re-lists libposix / libc / libm: those archives are already fully included by segment 1, so the external libs can resolve their references against the already-embedded objects. 3. Two new top-level Makefile vars `LIBSTDCXX := -lstdc++` and `LIBGCC := -lgcc`. The GCC driver resolves them against its built- in search paths (libgcc lives under a versioned `lib/gcc/i686- nanvix/<gcc-version>/` directory, which would be fragile to hardcode). Defined once at top level because the `-l` form is identical between the docker and host build paths. `LDFLAGS` is unchanged. The existing `-Wl,--allow-multiple-definition` flag is kept and the surrounding comment is expanded to honestly enumerate the duplicate-symbol categories the flag is masking (newlib long-double math helpers, libposix/libc env+isatty overlaps, libc/libm math helper overlaps, libgcc internal `__x86.get_pc_thunk.*` duplicates, etc.) -- the set is large and toolchain-build-version- dependent, and is the only practical workaround until the contributing upstreams are fixed. `.nanvix/config.py::configure_env()` -- an unused helper that mirrors `Makefile.nanvix`'s `CONFIGURE_ENV` -- is kept in sync (same `--whole-archive` LIBS, same LDFLAGS) and gains a docstring calling out the dead-code status. A separate small cleanup PR can delete the helper entirely. Validated end-to-end on the Nanvix microvm: CPython 3.12 + numpy 1.26.4 runs `import numpy`, `np.arange`, `np.dot`, `reshape`, `flatten`, broadcasting, all producing `NUMPY_TEST_OK`. Hello.py and the existing single-process / multi-process / standalone modes are unaffected by the change because the linker flags are not mode-conditional. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Stacked on top of the prior `[nanvix] E: Embed C/C++ runtime in
python.elf for .so dlopen support` PR.
Adds `libnvx_crt0.a` -- the executable-startup archive introduced by
the Nanvix `nvx-crt0` crate split -- to python.elf's `--whole-archive`
LIBS segment. This makes python.elf forward-compatible with the
planned Nanvix upstream change that removes `_start` / `_do_start` /
`c_trampoline` from `libposix.a`: once libposix drops the duplicate,
`libnvx_crt0.a` becomes the sole provider of those symbols.
What changed in `Makefile.nanvix`:
- New `LIBNVX_CRT0` variable (defined in both the docker and
host-build branches): absolute path to the sysroot copy of
`libnvx_crt0.a`.
- Existence check: parse-time error if `$(LIBNVX_CRT0)` is missing
from the sysroot, with a clear "update your sysroot" hint. Gated
on `MAKECMDGOALS` so `clean` and `distclean` still work against
older sysroots.
- `LIBS` line: prepend `$(LIBNVX_CRT0)` to the `--whole-archive`
group. Listed FIRST so that under the existing
`-Wl,--allow-multiple-definition` LDFLAG, the linker picks
`libnvx_crt0`'s `_start` over the duplicate copy `libposix.a`
currently still ships. This is an intentional behaviour change:
python.elf today and python.elf against a future stripped-down
libposix.a both use the same `_start` source (the standalone crt0
crate), avoiding subtle differences in startup behaviour across
the migration window.
- Comment block extended to document the ordering rationale and the
expected future state where libposix no longer carries startup
symbols.
`.nanvix/config.py::configure_env()` (the dead-code helper kept in
sync as documented in PR-10) is updated to mirror the new LIBS line.
Validated end-to-end:
- `./z build` succeeds; python.elf grows by ~500 bytes (the crt0
objects added under `--whole-archive`).
- `nm python.elf` shows `T _start` and `T _do_start` resolved.
- numpy 1.26.4 import + arithmetic + broadcasting test produces
`NUMPY_TEST_OK` on the Nanvix microvm.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds `ninja-build`, `python3-pip`, and `Cython<3` to the
toolchain-python docker image so that meson- and Cython-based Python
extension cross-builds (numpy, scipy, ...) work out-of-the-box,
without an apt/pip preamble on every `docker run` invocation.
What changed in `.nanvix/docker/Dockerfile`:
- Added `python3-pip` and `ninja-build` to the apt install list.
- Added `pip3 install --break-system-packages 'Cython<3'` (pinned for
numpy 1.26.x compatibility; lift the pin when bumping numpy).
- Added `rm -rf /usr/include/python3.12` after the install. The
`python3-pip`/`ninja-build` apt packages transitively pull in
`libpython3.12-dev`, whose headers under `/usr/include/python3.12`
would otherwise be picked up by meson's regen step ahead of the
Nanvix cross sysroot headers and silently corrupt the cross-build.
- Comment block explaining the rationale for each addition and the
`/usr/include/python3.12` purge.
Why this matters:
The numpy `.so` cross-build (validated end-to-end on 2026-05-27 with
the STB_WEAK loader fix landed) requires two tools that were not
present in the image as shipped:
- `ninja` — meson's default backend; missing it makes every
meson-based Python extension build fail immediately.
- `Cython` — used by `numpy/_build_utils/tempita.py` to template
`.pyx.in` files; without it the `numpy.random` codegen step fails.
Before this change, the workaround was to inject:
```bash
apt-get update -qq
apt-get install -qq -y --no-install-recommends ninja-build python3-pip
pip3 install --quiet --break-system-packages 'Cython<3'
rm -rf /usr/include/python3.12
```
into every numpy build invocation, which (a) was fragile, (b) required
the docker container to have outbound network access on every build
(non-hermetic), and (c) re-paid the apt install cost in CI every run.
Validated locally:
- `docker build -f .nanvix/docker/Dockerfile -t toolchain-python:pr13
.nanvix/docker/` succeeds.
- `docker run --rm <image> bash -c 'ninja --version'` → `1.11.1`.
- `docker run --rm <image> bash -c 'python3 -c "import Cython;
print(Cython.__version__)"'` → `0.29.37`.
- `docker run --rm <image> bash -c 'ls /usr/include/python3.12'` →
exits non-zero / "No such file or directory".
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… and 'origin/feat/toolchain-python-bake-ninja-cython' into phase0-array
Adds a small bash wrapper that sits in front of the real `i686-nanvix-gcc` and `i686-nanvix-g++` driver binaries in the toolchain-python docker image. The wrapper detects whether the invocation is producing an executable or a shared library and applies the correct linker flags for each case.
This mirrors the well-established `emcc` pattern used by emscripten and Pyodide, where the compiler-driver wrapper is what knows the difference between main-module and side-module builds, so consumers (cpython's Makefile.nanvix, future numpy / scipy / lxml meson builds, etc.) can use a single `LDFLAGS` env var without needing to encode the exe-vs-shared distinction themselves.
## Why this exists
`Makefile.nanvix` sets a single `LDFLAGS` on `./configure` that contains executable-specific linker flags:
- `-T <sysroot>/lib/user.ld` — executable layout script.
- `-no-pie` — disable PIE for executables.
- `-Wl,--no-dynamic-linker` — executable-only marker.
- `-Wl,--export-dynamic` — populate the main executable's `.dynsym`.
cpython propagates that same `LDFLAGS` to BOTH the main `python.elf` link AND every extension-module `.so` link via `PY_LDFLAGS`. For `.so` outputs these exe-only flags are wrong:
- `-T user.ld` tells `ld` to use an executable layout. When applied to a `-shared` link, `ld` treats the output as an exe and rejects undefined symbols, even those that should resolve at `dlopen()` time against the main exe's `.dynsym` (the Python C API symbols every extension references).
- `-no-pie` is incompatible with `-shared` (shared libraries must be PIC).
- `-Wl,--no-dynamic-linker` is meaningless for `.so`.
- `-Wl,--export-dynamic` is exe-only.
There is no clean place in cpython 3.12's `Modules/Setup` system to express "use this LDFLAGS for the exe link and that LDFLAGS for shared modules". cpython expects a single `LDFLAGS` shared between both.
The cleanest fix is to install a compiler-driver wrapper that does the split transparently: forwards exe builds unchanged, and strips exe-only flags + adds `-fPIC` for shared builds. That way `LDFLAGS` stays the same for everyone and the wrapper does the right thing per-invocation.
## What changed in `.nanvix/docker/`
- **New file `cc-wrapper.sh`** (~110 lines bash). Detects compile-only mode (`-c` / `-S` / `-E`), exe-link mode (no `-shared`), or shared-link mode (`-shared` present). Compile-only and exe modes forward to the real binary unchanged via `exec "$real_bin" "$@"`. Shared mode iterates the args and:
- Strips `-T <script>` (both `-T <script>` and `-T<script>` forms), `-no-pie`, `-Wl,--no-dynamic-linker`, `-Wl,--export-dynamic`, `-Wl,-T,<script>`, and bare `*.ld` argument files.
- Adds `-fPIC` if not already present.
- Uses a bash `filtered=()` array (not a string) so argv boundaries and quoting are preserved across the rewrite — `exec "$real_bin" "${filtered[@]}"` is argv-preserving for arguments containing spaces, empty strings, or glob metacharacters.
- **Dockerfile updated** to install the wrapper. The script is copied to `/opt/nanvix/bin/i686-nanvix-cc-wrapper.sh`, made executable, and the real `i686-nanvix-gcc` / `i686-nanvix-g++` binaries are renamed to `<name>.real`. Symlinks `<name>` -> `i686-nanvix-cc-wrapper.sh` are then installed. The wrapper picks the right `.real` binary based on `argv[0]`.
The install pattern is defensive: if the upstream base image already has a wrapper symlinked (some local-build flavours of `toolchain-python` carry an earlier wrapper), the symlink is removed and replaced — but only after asserting that the matching `<name>.real` file already exists, so a missing `.real` aborts the install rather than stranding the toolchain. If the real binary has not yet been renamed, it is moved to `<name>.real` in the same step.
## Validation
Built the image locally with the additions and smoke-tested all three wrapper modes against a trivial `int main(){return 0;}`:
- **Test 1 (exe link, no `-shared`):** wrapper transparent; real gcc handles normally.
- **Test 2 (simple `-shared`):** wrapper detects shared mode, adds `-fPIC`, link succeeds.
- **Test 3 (`-shared` plus the toxic exe-only flags `-T <script>` `-no-pie` `-Wl,--no-dynamic-linker`):** wrapper strips the toxic flags and the link succeeds with no spurious "cannot create executables" error.
Build-side end-to-end smoke test with cpython: applied a Phase-0-style change to `Modules/Setup.local` adding `*shared*\narray arraymodule.c`. With the wrapper in place, `array.cpython-312-i686-nanvix.so` (~190 KB) is produced and installed at the canonical `lib/python3.12/lib-dynload/` location; `PyInit_array` no longer appears in `python.elf`. Full `./z build` succeeds.
## What this unblocks
This wrapper is the foundational prerequisite for the broader `.a` -> `.so` migration documented in `nanvix-todo/cpython-static-to-shared-migration.md`. Every subsequent phase of that plan (peeling stdlib extension modules off `python.elf` and shipping them as `.so` files loaded via dlopen, exactly as upstream CPython works) depends on this wrapper being in place. Without it, every per-module conversion would need a per-call LDFLAGS hack.
## Backward compatibility
This change is purely additive at the image level — it adds the wrapper script and rewrites the gcc/g++ entry points to point at it. Existing build paths that don't pass `-shared` see no behavioural change (the wrapper falls through to the real binary unchanged). Existing `.so` build paths that DID work before (e.g., when LDFLAGS happened not to include `-T user.ld`) continue to work because the wrapper's strip-then-add-fPIC produces a strict superset of what bare `gcc -shared` would have done.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
First step of the .a -> .so migration plan documented in
`nanvix-todo/cpython-static-to-shared-migration.md`. Converts `array`
from a static built-in module into a shared `.so` loaded via dlopen
at runtime, matching upstream CPython's layout for stdlib extension
modules.
## Changes
* `.nanvix/docker.py` + `.nanvix/lxml.py` — `Setup.local` generation:
- Group the existing static modules under an explicit `*static*`
directive (was the implicit `#` comment that makesetup tolerated).
- Add a `*shared*` group with `array arraymodule.c` so cpython's
`setup.py build_ext` produces
`array.cpython-312-i686-nanvix.so` at
`lib/python3.12/lib-dynload/`.
* `.nanvix/test.py` — extend the `test_hello.py` smoke test to
`import array`, assert it is NOT in `sys.builtin_module_names`, and
exercise a tiny array operation. This proves the dlopen path
end-to-end on every test run; if the `.so` fails to load (loader
bug, missing PIE relocation, symbol-resolution issue, ...) the
smoke test fails immediately rather than silently regressing.
## Validation
`./z build` and `./z test`:
- `array.cpython-312.so` is produced at
`<sysroot>/lib/python3.12/lib-dynload/array.cpython-312.so`.
- Hello smoke prints
`CPYTHON_TEST_ARRAY_SO: array loaded via dlopen from
/lib/python3.12/lib-dynload/array.cpython-312.so`.
- lxml + HTTP server smoke continue to PASS.
## Dependencies
This PR depends on the following prereq PRs landing first:
- #682 — `--whole-archive` + `--export-dynamic` in
python.elf LIBS.
- #683 — bake `ninja` + `Cython` into the toolchain
image (needed by future meson-based extensions; harmless for
`array`).
- #684 — link `libnvx_crt0.a` into python.elf (will
become unnecessary once nanvix/nanvix#2453 lands but harmless
meanwhile).
- #687 — cc-wrapper for `-shared` vs exe link mode
(the `.so` link uses it).
- nanvix/nanvix#2450 — loader honours `STB_WEAK` undefined symbols.
- nanvix/nanvix#2453 — `libposix.a` is the sole owner of `sysalloc`
state (Fix 2 v3); without this, the first `dlopen` collides with
the heap.
Reviewers: please wait for those PRs (and the matching
`ghcr.io/nanvix/toolchain-python:latest` rebuild) before testing
this one.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Phase 1A of the .a -> .so migration (see nanvix-todo/cpython-static-to-shared-migration.md section 5). Builds on Phase 0 (#690 array) by promoting the remaining 11 Tier-1 "data primitives" stdlib extension modules from statically linked into python.elf to dlopen-loaded shared objects under lib/python3.12/lib-dynload/. Modules moved to *shared* in Modules/Setup.local generation (.nanvix/docker.py): - _bisect, _heapq, _struct, _random, _opcode (5 simple pure-C) - _queue, _csv, binascii (3 medium) - _json, _pickle, _zoneinfo (3 larger) All eleven modules have no external library dependencies (verified by grepping their .c sources for #include <zlib|expat|openssl|sqlite| mpdec|bzlib|lzma|hacl> — none match), so they need nothing beyond the same -lc / -lm symbols that array.so already pulls from python.elf's --whole-archive --export-dynamic main binary. Test coverage (.nanvix/test.py): - New phase1a_snippet imports each module, asserts it is NOT in sys.builtin_module_names, exercises one trivial API call to confirm dlopen + PyInit_<name> succeeded, and prints the resolved __file__ path so the host-side regex sees the .so came from lib-dynload/. - Smoke test, lxml import, HTTP server smoke, and full regrtest (160/160 modules) all continue to pass. Validation on local toolchain (phase0-llfix overlay of phase0-stable with newlib %lld printf fix applied; see nanvix-todo/newlib-z-missing-io-long-long-flag.md): - All 11 new .so files produced (~30-230 KB each) and installed under lib/python3.12/lib-dynload/. - nm python.elf no longer shows PyInit_<name> for any of the 11 modules; the symbols moved to their respective .so files. - python.elf size: 19.97 MB (Phase 0) -> 19.31 MB (Phase 1A), ~660 KB net reduction. - Hello-world + Phase 1A import probe + lxml + HTTP smoke + full regrtest 160/160 PASS in standalone mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR advances Nanvix’s staged .a → .so migration by moving 11 additional Tier‑1 “data primitive” stdlib extension modules from being statically linked into python.elf to being built/installed as dlopen-loaded shared objects under lib/python3.12/lib-dynload/. It also updates the Nanvix build/link strategy to ensure python.elf exports the runtime symbols needed by dlopen’d extensions.
Changes:
- Extend
Modules/Setup.localgeneration to markarray+ 11 additional modules as*shared*, producing.sofiles inlib-dynload/. - Strengthen Nanvix link configuration for
python.elf(whole-archive + export-dynamic) and add a toolchain wrapper to strip exe-only link flags when linking shared libraries. - Extend the Nanvix smoke test to import and lightly exercise each newly-shared module to validate dlopen +
PyInit_*end-to-end.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
Makefile.nanvix |
Adds libnvx_crt0 checks and updates link flags/LIBS to embed/export symbols for dlopen’d .sos. |
.nanvix/test.py |
Extends the smoke test script to assert modules are not built-in and to exercise trivial APIs for Phase 0 + Phase 1A modules. |
.nanvix/lxml.py |
Updates the generated Modules/Setup.local template to use *static*/*shared* directives (but needs Phase 1A parity). |
.nanvix/docker/Dockerfile |
Installs build helpers (pip/ninja/Cython) and installs the cc-wrapper into the toolchain image. |
.nanvix/docker/cc-wrapper.sh |
New wrapper that strips exe-only linker flags for -shared links. |
.nanvix/docker.py |
Updates Docker-side Modules/Setup.local generation to include the Phase 1A shared modules list. |
.nanvix/config.py |
Updates toolchain path mapping and keeps configure_env() LIBS in sync with the Makefile’s new link strategy. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+137
to
+142
| ifdef CONFIG_NANVIX | ||
| ifneq ($(filter clean distclean,$(MAKECMDGOALS)),$(MAKECMDGOALS)) | ||
| ifeq ($(wildcard $(LIBNVX_CRT0)),) | ||
| $(error libnvx_crt0.a not found at $(LIBNVX_CRT0). Update the Nanvix sysroot to one that ships libnvx_crt0.a (the nvx-crt0 crate must be present and built into the sysroot lib/ directory).) | ||
| endif | ||
| endif |
Comment on lines
+26
to
32
| # Phase 0 of the .a -> .so migration: array as proof-of-concept shared module. | ||
| # See nanvix-todo/cpython-static-to-shared-migration.md section 4. | ||
| # Listed BEFORE Setup.stdlib's static declaration so makesetup's | ||
| # "first rule wins" semantics make this shared variant take precedence. | ||
| *shared* | ||
| array arraymodule.c | ||
| """ |
Author
|
Closing per esaurez request — Phase 1A work will live on the esaurez/cpython fork for now while the migration plan is iterated; will reopen against nanvix/cpython once the full migration sequence is reviewed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1A of the
.a→.somigration (seenanvix-todo/cpython-static-to-shared-migration.mdsection 5). Builds on Phase 0 (#690 —array) by promoting the remaining 11 Tier-1 "data primitive" stdlib extension modules from statically linked intopython.elfto dlopen-loaded shared objects underlib/python3.12/lib-dynload/.Modules moved to
*shared*Eleven new
.sofiles, edited in.nanvix/docker.py'sModules/Setup.localgeneration:_bisect,_heapq,_struct,_random,_opcode._queue,_csv,binascii._json,_pickle,_zoneinfo.All eleven have no external library dependencies — verified by grepping their
.csources for#include <zlib|expat|openssl|sqlite|mpdec|bzlib|lzma|hacl>, none match. They need nothing beyond the same-lc/-lmsymbols thatarray.soalready pulls frompython.elf's--whole-archive--export-dynamicmain binary.Test coverage
phase1a_snippetin.nanvix/test.pyimports each module, asserts it is NOT insys.builtin_module_names, exercises one trivial API call to confirmdlopen+PyInit_<name>succeeded, and prints the resolved__file__path so the host-side regex sees the.socame fromlib-dynload/.Validation
Tested locally on the
phase0-llfixtoolchain overlay (phase0-stable+ the newlib%lldprintf fix from nanvix/newlib#14; seenanvix-todo/newlib-z-missing-io-long-long-flag.md):.sofiles produced (~30–230 KB each) and installed underlib/python3.12/lib-dynload/.nm python.elfno longer showsPyInit_<name>for any of the 11 modules; the symbols moved to their respective.sofiles (12/12 OK includingarray).python.elfsize: 19.97 MB (Phase 0) → 19.31 MB (Phase 1A), ~660 KB net reduction.Prerequisites
Same as Phase 0 (#690) — this PR depends on it. See
nanvix-todo/open-pr-merge-order.mdfor the full cross-repo merge order. No additional newlib / gcc / nanvix PRs are needed beyond what Phase 0 requires.Risk
Mechanical configuration change only — adds 11 entries to the
*shared*block ofSetup.local. Each module's source is unchanged. If any module fails to load at runtime, the failure is isolated (anImportErroron that specific module); the rest of the interpreter is unaffected.