Add content hash to exported artifacts, and PersistentVolumeClaims for k8s hosting - #421
Open
michaelkirk wants to merge 6 commits into
Open
Add content hash to exported artifacts, and PersistentVolumeClaims for k8s hosting#421michaelkirk wants to merge 6 commits into
michaelkirk wants to merge 6 commits into
Conversation
Artifacts are named <stem>-<date>-<hash>.<ext> across the board:
Bogota-valhalla-2026-08-19-a2b786e2.tar.zst
Bogota-bogota-graph-2026-08-19-c1689471.obj.zst
terrain-2026-08-19-d961012b.mbtiles
so a rebuild lands beside what it replaced rather than overwriting it. That's
what lets a deploy tell "same data" from "new data" by name alone, and it makes
a rollback a matter of pointing back at a file that's still there.
The pieces:
- An Artifact carries its Stem, Date, Ext and content, rather than being an
anonymous *dagger.File that the caller re-derives the name of:
DirectoryArtifact("Bogota-valhalla", tiles).Compress().AddTo(ctx, out)
Compress returns an Artifact and extends the extension (".zst", or
".tar.zst" for a directory) instead of making the caller spell out the
result. The name stays split because both hashing and compressing insert
text - one splices into the middle, one appends - and neither is reliable as
string surgery on a whole filename.
- Artifacts are named by what they are rather than by extension, so
`Bogota.valhalla.tar.zst` becomes `Bogota-valhalla-<date>-<hash>.tar.zst`.
The extension is now just the extension.
- The date was previously only on transit artifacts, spliced into the middle
of the stem. It's a field of its own now, in the same place for everything.
Producers leave it unset and Build/BuildTransit stamp it once: the producers
are cacheable, so a date baked in down there would be whatever day the step
first ran. The hash is 8 chars - it only has to separate builds sharing a
stem and a date.
- buildAll. Resolving a hash into a filename blocks, and dagger can only
overlap work it has already been handed, so hashing artifacts one at a time
would also *build* them one at a time. Getting every query in flight first
restores the parallelism lazy evaluation was providing.
Consumers can no longer spell an artifact's name, so they ask for it:
bin/artifacts collapses to one VERSION_RE and one latest_match (a behaviour
change as well as fewer lines - newest-date-wins and same-day-tie-errors now
apply to every artifact, where before the area-level ones refused to choose
between two builds at all), the compose files take a HEADWAY_*_FILE from it,
and bin/k8s/generate renders the URLs from the manifest instead of the
templates interpolating a name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
michaelkirk
changed the title
Ephemeral volumes are destroyed when a pod terminates, so every restart - a node drain, an OOM, a config tweak - re-downloaded and re-extracted the whole dataset before the service came back. Named PVCs let the data outlive the pod, and the init containers skip the download when it's already there. The claim name carries the version of what's in it: area tag, data tag, and a `<date>-<hash>` computed by volume_version() from the names of the artifacts unpacked into it - newest date, hash over the exact set, so one artifact or three works the same way. A claim therefore changes exactly when its contents change, which is the whole trick: a rebuild names a new claim and gets a fresh download, while everything else keeps the volume it already has. (Getting this wrong is silent - the init containers short-circuit on a volume that already has data, so the pod comes up green serving the previous build.) That means volumes accumulate, one per version, which is what makes a rollback instant. bin/k8s/unused-volumes collects them: it asks which claims are still spoken for - by a workload's pod template, or by a pod that exists right now, which matters mid-rollout - and offers the rest. bin/k8s/show-volumes is the read-only view. Rollout strategy had to move too. Deployments that only read their volume keep RollingUpdate with maxSurge: 1 - on a version bump the two pods claim different PVCs, so the new one downloads while the old one serves; on a same-version restart they share one ReadWriteOnce claim, which the scheduler only allows where it can attach. Elasticsearch is the exception and uses Recreate: it takes an exclusive node.lock on its data directory. Finally, the pelias inits had to become atomic. With ephemeral volumes an init container killed mid-install self-healed, since the next pod got a fresh volume; with a persistent one the partial data survives and passes the "already have data" check, permanently serving a truncated dataset. Both scripts staged in /tmp and finished with a cross-filesystem mv, which is a recursive copy: - placeholder stages on the volume and installs with a single same-filesystem rename. The compose file mounts the volume at /data rather than /data/placeholder so there's a parent to rename into. - elasticsearch needs its data at the volume root, so no rename is possible; instead the staging dir doubles as an in-progress marker - still present on the next run means the previous init was interrupted, and the volume is wiped and re-downloaded. valhalla, OTP and tileserver already download to a .download path and finish with a same-filesystem rename, so they were fine. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.