[nanvix] E: Link libnvx_crt0.a into python.elf#684
Open
esaurez wants to merge 2 commits into
Open
Conversation
Updates `Makefile.nanvix` so that `python.elf` correctly serves as the "main module" against which extension `.so`s (numpy, ssl, lxml, future pip-installed wheels, ...) resolve their C and C++ runtime symbols at dlopen() time. This is the consumer-side companion to the Nanvix loader's STB_WEAK support (esaurez/nanvix#22) and is gated on the new libposix `pathconf` / `fpathconf` stubs (esaurez/nanvix#23) for the configure conftest to even produce an executable. Three coordinated link-flag changes to the `CONFIGURE_ENV` block: 1. `LIBS` segment 1 -- new `--whole-archive ... --no-whole-archive` block ahead of the existing `--start-group`. Forces every object from libposix, libc, libm, libstdc++, and libgcc into python.elf so the runtime symbols extension `.so`s depend on are embedded (and re-exported via `-Wl,--export-dynamic`, already present). Without this, the static linker drops unreferenced objects (e.g. `fscanf`, `longjmp`, `strtold_l` for numpy; `operator new/delete[]`, `__cxa_*`, `_Unwind_*`, `std::type_info` vtables for any C++ extension) and subsequent dlopen() of those `.so`s fails with "symbol not found". 2. `LIBS` segment 2 -- the existing `--start-group` is trimmed to just the external add-on libraries (sqlite3, ssl, crypto, z, bz2, lzma, ffi). It no longer re-lists libposix / libc / libm: those archives are already fully included by segment 1, so the external libs can resolve their references against the already-embedded objects. 3. Two new top-level Makefile vars `LIBSTDCXX := -lstdc++` and `LIBGCC := -lgcc`. The GCC driver resolves them against its built- in search paths (libgcc lives under a versioned `lib/gcc/i686- nanvix/<gcc-version>/` directory, which would be fragile to hardcode). Defined once at top level because the `-l` form is identical between the docker and host build paths. `LDFLAGS` is unchanged. The existing `-Wl,--allow-multiple-definition` flag is kept and the surrounding comment is expanded to honestly enumerate the duplicate-symbol categories the flag is masking (newlib long-double math helpers, libposix/libc env+isatty overlaps, libc/libm math helper overlaps, libgcc internal `__x86.get_pc_thunk.*` duplicates, etc.) -- the set is large and toolchain-build-version- dependent, and is the only practical workaround until the contributing upstreams are fixed. `.nanvix/config.py::configure_env()` -- an unused helper that mirrors `Makefile.nanvix`'s `CONFIGURE_ENV` -- is kept in sync (same `--whole-archive` LIBS, same LDFLAGS) and gains a docstring calling out the dead-code status. A separate small cleanup PR can delete the helper entirely. Validated end-to-end on the Nanvix microvm: CPython 3.12 + numpy 1.26.4 runs `import numpy`, `np.arange`, `np.dot`, `reshape`, `flatten`, broadcasting, all producing `NUMPY_TEST_OK`. Hello.py and the existing single-process / multi-process / standalone modes are unaffected by the change because the linker flags are not mode-conditional. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Stacked on top of the prior `[nanvix] E: Embed C/C++ runtime in
python.elf for .so dlopen support` PR.
Adds `libnvx_crt0.a` -- the executable-startup archive introduced by
the Nanvix `nvx-crt0` crate split -- to python.elf's `--whole-archive`
LIBS segment. This makes python.elf forward-compatible with the
planned Nanvix upstream change that removes `_start` / `_do_start` /
`c_trampoline` from `libposix.a`: once libposix drops the duplicate,
`libnvx_crt0.a` becomes the sole provider of those symbols.
What changed in `Makefile.nanvix`:
- New `LIBNVX_CRT0` variable (defined in both the docker and
host-build branches): absolute path to the sysroot copy of
`libnvx_crt0.a`.
- Existence check: parse-time error if `$(LIBNVX_CRT0)` is missing
from the sysroot, with a clear "update your sysroot" hint. Gated
on `MAKECMDGOALS` so `clean` and `distclean` still work against
older sysroots.
- `LIBS` line: prepend `$(LIBNVX_CRT0)` to the `--whole-archive`
group. Listed FIRST so that under the existing
`-Wl,--allow-multiple-definition` LDFLAG, the linker picks
`libnvx_crt0`'s `_start` over the duplicate copy `libposix.a`
currently still ships. This is an intentional behaviour change:
python.elf today and python.elf against a future stripped-down
libposix.a both use the same `_start` source (the standalone crt0
crate), avoiding subtle differences in startup behaviour across
the migration window.
- Comment block extended to document the ordering rationale and the
expected future state where libposix no longer carries startup
symbols.
`.nanvix/config.py::configure_env()` (the dead-code helper kept in
sync as documented in PR-10) is updated to mirror the new LIBS line.
Validated end-to-end:
- `./z build` succeeds; python.elf grows by ~500 bytes (the crt0
objects added under `--whole-archive`).
- `nm python.elf` shows `T _start` and `T _do_start` resolved.
- numpy 1.26.4 import + arithmetic + broadcasting test produces
`NUMPY_TEST_OK` on the Nanvix microvm.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ppenna
force-pushed
the
feat/makefile-nanvix-so-link-flags
branch
2 times, most recently
from
3c4d98c to
741c061
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacks on top of nanvix/cpython#682. Adds
libnvx_crt0.a-- the executable-startup archive introduced by the Nanvixnvx-crt0crate split -- topython.elf's--whole-archiveLIBS segment.This makes
python.elfforward-compatible with the planned Nanvix upstream change that removes_start/_do_start/c_trampolinefromlibposix.a: once libposix drops the duplicates,libnvx_crt0.abecomes the sole provider of those symbols. Today both archives ship them; with the existing-Wl,--allow-multiple-definitionLDFLAG and the new ordering (libnvx_crt0 first), the linker picks the crt0 copy, sopython.elfuses the same_startsource across both the current and the future Nanvix state.What changed in
Makefile.nanvixLIBNVX_CRT0 := .../lib/libnvx_crt0.ain both the docker and host-build branches.$(error ...)if$(LIBNVX_CRT0)is missing, gated onMAKECMDGOALSsocleananddistcleanstill work against older sysroots.LIBSline$(LIBNVX_CRT0)to the--whole-archivegroup:-Wl,--whole-archive $(LIBNVX_CRT0) $(LIBPOSIX) $(LIBC) $(LIBM) $(LIBSTDCXX) $(LIBGCC) -Wl,--no-whole-archive .....nanvix/config.py::configure_env()(the dead-code helper kept in sync per the prior PR's docstring) is also updated to mirror the new LIBS line.Ordering rationale (the subtle bit)
libnvx_crt0is listed first inside--whole-archiveso that under-Wl,--allow-multiple-definition, the linker picks the crt0-side copy of_start/_do_start/c_trampolineover the libposix-side copy. This is an intentional behaviour change:_startcomes fromlibnvx_crt0.a. Functionally identical to the old libposix-provided_start, but the source archive is different._startcomes fromlibnvx_crt0.a-- now the sole provider.Listing crt0 first means
python.elf's startup behaviour does not silently change when the upstream cutover happens. The alternative (list crt0 last to preserve today's exact provider) would mean a subtle behaviour shift the day libposix changes.Existence guard
The new guard fails the make-parse step with an actionable message if the sysroot lacks
libnvx_crt0.a:Without the guard, a stale sysroot would lead to a confusing linker error ("file not found: libnvx_crt0.a") inside a long python configure log. With it, the user gets the message the moment they invoke
make build.The guard is gated:
so it only fires when CPython is actually being built;
make cleanagainst an older sysroot still works.Validation
./z buildsucceeds;python.elfgrows by ~500 bytes (the small handful of crt0 objects added under--whole-archive; most of nvx-crt0's content was already inpython.elfvia libposix transitively).i686-nanvix-nm python.elfshowsT _startandT _do_startresolved.np.arange+np.dot+ reshape + broadcasting all work; the test harness printsNUMPY_TEST_OKon the Nanvix microvm.Compatibility
libnvx_crt0.a-- this is enforced by the existence guard.Makefile.nanvix+.nanvix/config.py).Follow-ups
A separate Nanvix-side PR (next in the stack) will remove the startup symbols from
libposix.aitself, after whichlibnvx_crt0.abecomes the only_startprovider. The link line in this PR is already correct for that future state -- no further cpython change is needed.