Feat/release pipeline overhaul and improvements - #110
Merged
darrendignam merged 17 commits intoAug 31, 2026
Conversation
The anti-spoofing rule negated the address instead of the interface, rendering as "-i lo ! -s 127.0.0.0/8". That sits behind an unconditional "-i lo -j ACCEPT", so it never matched and there was no anti-spoofing rule in effect. The IPv6 form also carried a space after the bang, which ip6tables rejects outright. The rule logging forwarded traffic was on the INPUT chain while its log prefix said FORWARD. Both match the intent recorded in files/etc/iptables/rules.v4.
The playbook set opf_server_hostname, opf_server_hostdomain and opf_server_fqdn, but the role reads viper_env_*. Nothing consumed the values, so the hostname task fell back to its default and every built machine came out called "localhost".
tool.yml cloned the full git history of JHOVE, DROID, Tika and veraPDF into /usr/local/src. Tika alone is around a gigabyte, and the cost is paid twice: once uploading each release to the artifact server, and again by every user who downloads an image. Nobody opens those trees. The provenance that mattered was which upstream tag each tool was built from, so that is now recorded in /usr/local/share/viper/manifest.json alongside the versions of the Debian and MediaArea packages. Also drops openjdk-17-doc and openjdk-17-source, which are several hundred megabytes of reference material on an appliance with no offline Java development story.
Every published image carried the build identity: the vagrant account with password "vagrant", a NOPASSWD sudo rule, the well-known Vagrant public key, and a set of SSH host keys identical across every download. cleanup.sh now runs as the last provisioner and strips that, along with apt caches, provisioning leftovers and logs. The account lock and the sudoers removal happen in shutdown_command instead, because cleanup.sh still needs sudo and Packer still needs to authenticate as vagrant to issue the shutdown. sshd ships installed but disabled, with no host keys; viper-regenerate-host-keys.service gives an installation its own set if the user enables it. Freed blocks are returned to the image with fstrim and disk_discard, rather than zeroed with dd, which on a 50G disk would inflate the qcow2 to 50G along the way and not fit on a CI runner. smoke-test.sh asserts the bundled tools actually run before anything is published, so a broken image fails here instead of after a multi-hour upload. disk_size moves to 50G, which the README and release notes have claimed all along, and the image name is now derived from a version variable rather than hardcoded. The security role turns off password authentication mid-build, which is what Packer authenticates with, so a build-only sshd drop-in keeps our own connections working and is removed at shutdown.
The descriptor hardcoded a 50 GB capacity, 2 CPUs, 4096 MB and version "v1.2" regardless of what was built. Against a 25 GB image that meant the OVF described a machine that did not exist. A capacity that disagrees with the VMDK descriptor is what makes strict OVF importers such as VMware ovftool reject an appliance. Capacity is now read from the image with qemu-img, and CPUs, memory and product version are passed in by the caller. With no arguments the script picks up the only QCOW2 in the output directory. Refs openpreserve#75
build-ova.yml built the VM, uploaded the QCOW2, converted to OVA and uploaded that, all from one job. At the throughput we see to the artifact server that needs roughly eight hours, and GitHub cancels any job at six, so the OVA upload was being killed at around 80% on every release. The tag shipped with only the QCOW2. The image is still built once, so the two formats are genuinely the same machine rather than two builds of one tag, and is then handed to two publish jobs that run concurrently. Each is a separate job and so gets its own six hour budget. Images move between jobs as split GitHub artifacts. Splitting keeps the hand-off clear of the per-artifact size ceiling, which sits right about where these images land. A manifest.env travels with the parts carrying the filename, type, checksums and destination, so the publish workflow needs almost no inputs and republish.yml can drive it against an artifact from an earlier run. The upload budgets the remaining job time so it never starts an attempt it cannot finish, and the published file is size-checked afterwards with a HEAD request. Previously a truncated upload looked exactly like a successful one. Retained artifacts mean a failed upload no longer needs a rebuild: "Re-run failed jobs", or the new Republish artifact workflow, both cost only the upload. Also removes test-api.yml, a debugging workflow one uncommented line away from printing ARTIFACT_UPLOAD_TOKEN into the log.
Nothing ran on a pull request, so the first sign that a change had broken the build was a failed release run hours into a tag. yamllint, ansible-lint, a syntax check, shellcheck and packer validate take a couple of minutes and would have caught the iptables defects fixed earlier in this branch. ansible-lint starts at the "min" profile so it reports genuine errors without failing over the conventions of a codebase that predates it. requirements.yml declares the community.general dependency that the roles have always had and never stated.
The maintainer guide described only the Vagrant flow and never mentioned Packer, which is how releases are actually produced. PACKER_BUILD.md still pointed at hardcoded v1.2-alpha paths. Also corrects the README download section, which linked to a single retired OVA URL, and records that the appliance ships without a remote access account.
Locking the vagrant build account removed the only path to root on the appliance: viper was in vboxsf alone, and the preseed sets passwd/root-login false so root has no password. A research environment whose users cannot install a package is not much use. viper joins the sudo group, which is what its account comment has claimed all along. sudo prompts and the blank password satisfies it via the nullok in Debian's common-auth. smoke-test.sh exercises that path during the build rather than trusting it, because the failure mode is an published image nobody can administer.
viper.setup already runs an apt dist-upgrade as its first task, so the preseed copy was duplicated work. It is also the copy that runs inside the installer, where it counts against Packer's ssh_timeout rather than against the provisioning phase. Because the ISO is a pinned point release, the gap this had to close grew every month until the build stopped fitting in the timeout.
vboxguest and vboxsf ship in Debian's stock kernel, and neither virtualbox-guest-utils nor virtualbox-guest-x11 depends on dkms, so the toolchain install bought nothing. It also broke the build outright once the preseed stopped upgrading: linux-headers-$(uname -r) tied the build to a kernel version that only stays in the archive while the pinned point release is current.
Promote the hardcoded 30m to an ssh_timeout variable defaulting to 45m, so a slow mirror can be ridden out without editing the template. Also record why the ISO stays on 12.8.0. Bumping to 12.15.0 was tried and reverted: it made the unattended install roughly ten times slower, reproducibly, at a sustained ~174 KB/s. The cause was not identified and is probably mirror side, but the measurements belong next to the pin so the next person does not rediscover them.
The lint job added earlier on this branch fails on its own repository, because `yamllint .` exits 1 on three errors that nothing had run before. GitHub Actions requires a top-level `on:` key, which the truthy rule reads as the boolean true, so key checking is now disabled for it. The other two are pre-existing whitespace defects the new job surfaces for the first time: a missing newline at end of file, and a trailing space.
The local reference build takes 1h42m, against a 90 minute cap. Runners fetch far faster than the ~225 KB/s link that dominates that figure, so the cap should not be approached, but the margin was thinner than the cost of hitting it: a killed job loses the whole build and upload cycle.
prerelease was hardcoded false, so tagging v0.0.1-rc1 would publish a release candidate as the latest stable download. Derive it in the metadata step instead. Only a bare semver triple counts as a full release; everything else, including the rc tags and the release-* pattern, is published as a prerelease.
Tagging v0.0.1-rc1 failed at startup before any runner began: The nested job 'publish' is requesting 'actions: read', but is only allowed 'actions: none'. publish-artifact.yml needs actions: read because actions/download-artifact reads from another run when given a run-id. Both callers declared only contents: read at workflow level, which leaves every other scope at none, and a reusable workflow cannot request more than its caller grants. Granted on the calling jobs rather than at workflow level, so build and release keep the narrower default. republish.yml had the same defect and would have failed the first time it was used.
Both publish jobs failed after a successful upload. The check read Content-Length from a HEAD of the download URL, but the artifact server answers HEAD with 405 and a 31 byte JSON body, so every upload looked truncated to 31 bytes: Published file is 31 bytes, expected 2370375680. Upload was truncated. The server also ignores Range, so the stored object cannot be sized from outside without downloading it again, which costs as much as the upload. Check the server's own upload response instead. It reports status, the size it stored and the checksums it recorded, so this now compares size and sha256 rather than size alone. It confirms what the server recorded and cannot detect corruption the server is itself unaware of, which is the best available until the server supports HEAD.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tagged releases ran as one job that built the VM and uploaded both images, needing roughly 8 hours against the artifact server. GitHub kills any job at 6, so the OVA upload died around 80% and releases shipped with only the QCOW2. This builds the image once and fans the two uploads out into separate jobs, so each gets its own budget.
Measured on a real tagged run
Why build once rather than two independent workflows
Verification
ovf:capacity53687091200 matches the VMDK extent,createType="streamOptimized", descriptor first in the tar,vbox:uuidmatches<Image uuid>, disk attached to the declared AHCI controllerNotes for review
grub-pcremoval is deliberate so a single-OS appliance shows no boot menu. The MBR and/boot/grub/i386-pc/are not package-owned so it still boots, but a future kernel upgrade would leave grub.cfg stale. Worth a second opinionIssues