Replace docker-compose with a Debian package and systemd units#49
Draft
robertodauria wants to merge 11 commits into
Draft
Replace docker-compose with a Debian package and systemd units#49robertodauria wants to merge 11 commits into
robertodauria wants to merge 11 commits into
Conversation
Convert the autonode deployment from docker-compose to an mlab-node Debian package whose components run as hardened systemd services (systemd sandboxing instead of Docker isolation). No Docker at build or run time. - .build/fetch-binaries.sh: extract component binaries from the pinned M-Lab images via skopeo (no Docker daemon); fail the build on musl-linked binaries. - systemd/: mlab-node.target plus per-component services and oneshots (schema generation, uuid.prefix, BBR, metadata). Atomic lifecycle: every component BindsTo register-node, so if registration goes down the whole stack goes down; uniform Restart=always. - Per-service sandboxing (ProtectSystem=strict, dropped capabilities, RestrictAddressFamilies, etc.); CAP_NET_BIND_SERVICE for ndt-server, CAP_NET_RAW for scamper, DynamicUser for node-exporter. - debian/: native debhelper packaging (control, rules, install, sysusers, tmpfiles, postinst, conffiles). - config/: /etc/mlab env templates, verify.pub, modules-load.d + sysctl.d BBR. - Remove docker-compose.yml and env; rewrite README for the package workflow. Filesystem moves to FHS: binaries /usr/lib/mlab, config /etc/mlab, state /var/lib/mlab, sockets/metadata /run/mlab.
Real extraction on Ubuntu 24.04 showed 9/11 binaries static, scamper glibc-dynamic, and only generate-schemas-ndt7 musl-linked (needs just libc.musl-x86_64.so.1). Stop aborting on musl: warn instead, add Depends: musl, and exclude that one binary from dh_shlibdeps (no musl shlibs info) while keeping strict resolution for scamper's glibc libs. Harden temp cleanup against root-owned image entries.
The ndt-server image builds cmd/generate-schemas on Alpine with CGO (musl-dev), so the extracted copy is musl-linked. cmd/generate-schemas is a nested Go module that pins ndt-server@v0.20.17, whose bbr package needs CGO on Linux (so a static build fails). Instead, build that nested module from the pinned source with CGO on the glibc build host -> a glibc-dynamic binary (NEEDED: libc.so.6 only, resolved by base libc6) that produces a schema byte-identical to the image's. - fetch-binaries.sh: stop extracting generate-schemas-ndt7; shallow-clone ndt-server@v0.25.2 and 'go build' the nested cmd/generate-schemas with CGO_ENABLED=1 and a pinned GOTOOLCHAIN. Require go+git. - control: drop Depends: musl; add golang-go/git/ca-certificates/gcc/libc6-dev to Build-Depends. - rules: drop the dh_shlibdeps -X exclude (binary is now glibc-resolvable).
On debhelper 13.14 (Ubuntu 24.04) dh_installsysusers is not in the default dh sequence, so the mlab-node system user was never created. Invoke it via execute_after_dh_install (before dh_installtmpfiles) so the user exists before the tmpfiles dirs are chowned to it.
The uuid-annotator image provides the AS-names data via 'ENV ASNAME_URL file:///data/asnames.ipinfo.csv', which the binary picks up through flagx.ArgsFromEnv. systemd does not inherit image ENVs, so without it the binary panics (asname.url is required). Extract /data/asnames.ipinfo.csv from the pinned image, install it to /usr/share/mlab, and pass -asname.url=file:// to it from the unit. Also add AF_NETLINK to the uuid-annotator sandbox (needed to enumerate local addresses).
The binary defaults -scamper.bin to /usr/local/bin/scamper; ours is at /usr/lib/mlab/scamper. Set the flag explicitly and drop the PATH override (traceroute-caller uses the absolute flag, not a PATH lookup).
…ge extraction Replace fetch-binaries.sh (skopeo-based extraction from the released container images) with build-binaries.sh, which shallow-clones each component repository at the same pinned release tags and compiles from source. This gives verifiable provenance (git tag + go.sum + the Go checksum database), removes the skopeo/jq machinery and all musl special-casing, and produces glibc-compatible binaries by construction. Every Go component builds with CGO_ENABLED=0 (static) except ndt-server and its schema generator, whose bbr package requires cgo on Linux; those two are glibc-dynamic. Version ldflags mirror each upstream image build so Prometheus build-info metrics keep reporting versions. scamper is built from the snapshot tarball vendored by traceroute-caller, with --disable-shared so it ships self-contained. heartbeat builds from m-lab/locate (cmd/heartbeat), where it actually lives.
Add a GitHub Actions workflow that builds the mlab-node package on ubuntu-24.04 (amd64) and ubuntu-24.04-arm (arm64) runners — both native, no QEMU. Build dependencies come from debian/control via mk-build-deps so the workflow cannot drift from the package metadata. Packages are uploaded as workflow artifacts on every run and attached to the GitHub Release on v* tag pushes. Extend the package to Architecture: amd64 arm64 and relax the build-binaries.sh host check accordingly; all components compile natively for the host architecture.
main bumped ndt-server to v0.25.3 in docker-compose.yml (#48); the compose file is deleted on this branch, so the bump moves to the VER_NDT_SERVER pin in .build/build-binaries.sh.
Go marks module cache files read-only, so the EXIT trap's plain rm -rf failed with permission errors and tanked the build exit code after all artifacts had been staged. Use -modcacherw and chmod in the trap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the docker-compose BYOS deployment with a native Debian package (
mlab-node) running the measurement stack as hardened systemd units, so partner nodes get unattended upgrades and standard systemd/journald operations.Highlights:
mlab-node.target; registration is the keystone unit (BindsTo=)..build/build-binaries.sh): verifiable provenance (git tag + go.sum + Go checksum DB), no Docker/skopeo at build or run time. ndt-server and its schema generator build with cgo (theirbbrpackage requires it); everything else is static.ubuntu-24.04/ubuntu-24.04-arm) and uploads the.debs as artifacts;v*tags attach them to the GitHub Release.Draft until the package is validated end-to-end on the sandbox VM.
This change is