diff --git a/AGENTS.md b/AGENTS.md index 5c90cdf..ec76bfa 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -162,7 +162,7 @@ go test ./... # Standard tests - **Entrypoint is always named `plugin`** — build output and migrate scan require `plugins/{group}/{name}/{version}/plugin`; Dockerfiles must `COPY` the entrypoint to `/plugin` (sidecars optional) - **A plugin runs outside its image** — the service unpacks the image filesystem and execs `plugin` as a plain process on its own base (`debian:trixie-slim`). Wrappers resolve paths from `DIR=${0%/*}`, never from `/`; absolute symlinks are skipped on unpack. `plugins build` proves it per version: it relativises in-bundle absolute links, round-trips the archive through `plugarchive`, and smoke-runs `plugin` in the runtime image as uid 65532 with an empty env — a failure leaves only `build.log`. `TestSmokeImageMatchesServiceBase` keeps the smoke image equal to the Dockerfile's runtime stage - **`easyp-svc plugins build `** — uses Docker multi-stage builds to extract image filesystems to the output directory (`plugins/` by default) and verifies each version; `--archive` keeps only the verified `{g}/{n}/{v}/plugin.tgz` instead. Supports `--filter`, `--parallel`, `--force`, `--dry-run`. Nothing is uploaded -- **`easyp-svc plugins publish [registry-path]`** — per version: build → verify → upload with sha256 metadata → `CreatePlugin` (or `UpdatePlugin` when it exists) → delete the local copy. Sources: a registry, `--archives ` (a `build --archive` tree), or with neither, `--plugins-dir` itself. Destination: S3 (`--cfg` supplies `registry.s3`, flags override it) or `--plugins-dir` (no object storage, same host). Skips what is registered and in storage, only registers what is in storage but not registered, skips uploads of an identical archive; `--force` republishes. Preflight checks storage and the token before building. `--build-parallel` 4, `--parallel` 8, backing off on the rate and concurrency limits — the SDK deliberately does not retry `ResourceExhausted` — and stopping new versions at the licence tier's plugin cap. Every config under `deploy/` sets `rate_limit.max_concurrent_per_ip: 10` for it; against a server left at the default of 2, pass `--parallel 2`. Needs a v1.2.0+ server +- **`easyp-svc plugins publish [registry-path]`** — per version: build → verify → upload with sha256 metadata → `CreatePlugin` (or `UpdatePlugin` when it exists) → delete the local copy. Sources: a registry, `--archives ` (a `build --archive` tree), or with neither, `--plugins-dir` itself. Destination: S3 (`--cfg` supplies `registry.s3`, flags override it) or `--plugins-dir` (no object storage, same host). Skips what is registered and in storage, only registers what is in storage but not registered, skips uploads of an identical archive; `--force` republishes. `plugarchive.PackDir` is reproducible (epoch mtimes, no owners, modes reduced to 0755/0644/0777), so an unchanged rebuild is an identical archive Preflight checks storage and the token before building. `--build-parallel` 4, `--parallel` 8, backing off on the rate and concurrency limits — the SDK deliberately does not retry `ResourceExhausted` — and stopping new versions at the licence tier's plugin cap. Every config under `deploy/` sets `rate_limit.max_concurrent_per_ip: 10` for it; against a server left at the default of 2, pass `--parallel 2`. Needs a v1.2.0+ server - **Port 5432 conflict** — if postgres already runs locally, minimal stack uses port 5433 (`EASYP_POSTGRES_PORT`) - **Plugin binaries must exist** — publish them (`task publish-plugins`, or `task build-plugins` + `task run-local` without S3) before the service can generate code - **License:** `PasetoLicenseClient` verifies the token offline against `license.public_keys` (kid → hex; an entry under `"*"` verifies any key id). No key configured means community mode; a key that does not decode stops startup. Anything else that goes wrong resolves to community, never to an error diff --git a/README.md b/README.md index e2bcc75..e01721b 100644 --- a/README.md +++ b/README.md @@ -573,7 +573,7 @@ plugins publish registry Key properties: - **One command, in the right order.** `plugins publish` builds, verifies, uploads, registers and deletes each version before the next one needs the disk; a catalogue never has to fit on the publishing machine, and nothing is registered before its archive is in place. -- **Idempotent.** A version that is registered and in storage is skipped; one in storage but not registered is only registered; an upload is skipped whenever storage already holds the same archive. An interrupted run is resumed by re-running it. `--force` rebuilds and re-registers everything it matches. +- **Idempotent.** A version that is registered and in storage is skipped; one in storage but not registered is only registered; an upload is skipped whenever storage already holds the same archive. An interrupted run is resumed by re-running it. `--force` rebuilds and re-registers everything it matches; archives are reproducible — the same plugin packs to the same bytes, wherever and whenever it is built — so a rebuild of an unchanged version uploads nothing and no replica refetches it. - **Republishing takes effect everywhere.** `--force` on a registered version replaces its config with the new checksum (`UpdatePlugin`), and every replica refetches its unpacked copy on next use — each keeps a `.sha256` marker beside it. A registration that names a checksum the stored archive does not have fails with `FAILED_PRECONDITION` / `CHECKSUM_MISMATCH`, as does a download that does not match the registry. - **The service owns the command.** It is always `{plugins_dir}/{group}/{name}/{version}/plugin`, derived on registration; clients need not know the server's `plugins_dir`. An explicit `command` is still accepted for a plugin that needs arguments. - **Credentials split:** publishing needs S3 write access; the service only needs read (plus delete for `DeletePlugin`). Clients of the gRPC API need no S3 access at all. diff --git a/internal/plugarchive/plugarchive.go b/internal/plugarchive/plugarchive.go index cae4dda..c264bbf 100644 --- a/internal/plugarchive/plugarchive.go +++ b/internal/plugarchive/plugarchive.go @@ -14,6 +14,7 @@ import ( "path/filepath" "strings" "syscall" + "time" ) // EntrypointName is the required executable name inside every plugin archive. @@ -37,6 +38,13 @@ const ( // ownerAccess is OR-ed into restored directory modes so the extracting // process can always descend into what it just created. ownerAccess = 0o700 + // Modes recorded in an archive. A file keeps only whether it is + // executable: the rest of its mode is the building machine's umask. + packedDirMode = 0o755 + packedExecMode = 0o755 + packedFileMode = 0o644 + packedSymlinkMode = 0o777 + anyExecBits = 0o111 ) // Package errors. @@ -46,8 +54,14 @@ var ( ) // PackDir writes a tar.gz archive of the contents of dirPath to w. -// Entry names are relative to dirPath; file modes (including exec bits) are -// preserved and symlinks are stored verbatim. +// Entry names are relative to dirPath; exec bits are preserved and symlinks are +// stored verbatim. +// +// The same contents always pack to the same bytes. Times, owners and the +// umask's share of a mode depend on where and when the version was built, not +// on what it is, so they are left out. The archive's sha256 then names the +// plugin: a rebuild of an unchanged version matches what storage holds, its +// upload is skipped, and no replica refetches it. func PackDir(dirPath string, w io.Writer) error { gzWriter := gzip.NewWriter(w) tarWriter := tar.NewWriter(gzWriter) @@ -102,6 +116,7 @@ func packEntry(tarWriter *tar.Writer, dirPath string, path string, info os.FileI if info.IsDir() { header.Name += "/" } + normalizeHeader(header, info) err = tarWriter.WriteHeader(header) if err != nil { @@ -129,6 +144,32 @@ func packEntry(tarWriter *tar.Writer, dirPath string, path string, info os.FileI return nil } +// normalizeHeader drops from header everything that depends on the machine and +// moment of the build rather than on the plugin. +// +// Every entry gets the epoch as its modification time. Nothing reads it back — +// Unpack lets files take the time they are written — and a real one is the +// build's clock, which made two builds of the same binary two different +// archives. +func normalizeHeader(header *tar.Header, info os.FileInfo) { + header.ModTime = time.Unix(0, 0).UTC() + header.AccessTime = time.Time{} + header.ChangeTime = time.Time{} + header.Uid, header.Gid = 0, 0 + header.Uname, header.Gname = "", "" + + switch { + case info.IsDir(): + header.Mode = packedDirMode + case info.Mode()&os.ModeSymlink != 0: + header.Mode = packedSymlinkMode + case info.Mode()&anyExecBits != 0: + header.Mode = packedExecMode + default: + header.Mode = packedFileMode + } +} + // Unpack extracts the tar.gz archive at archivePath into destDir, replacing // destDir atomically: entries are extracted into a unique sibling temp // directory which is then renamed over destDir. The archive must contain the diff --git a/internal/plugarchive/plugarchive_test.go b/internal/plugarchive/plugarchive_test.go index 0d5e288..e738e01 100644 --- a/internal/plugarchive/plugarchive_test.go +++ b/internal/plugarchive/plugarchive_test.go @@ -4,9 +4,12 @@ import ( "archive/tar" "bytes" "compress/gzip" + "errors" + "io" "os" "path/filepath" "testing" + "time" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -79,6 +82,48 @@ func TestPackUnpackRoundTrip(t *testing.T) { assert.Equal(t, "app/tool", linkTarget) } +func TestPackDirIsReproducible(t *testing.T) { + t.Parallel() + + pack := func(fileMode os.FileMode, mtime time.Time) []byte { + t.Helper() + + dir := t.TempDir() + require.NoError(t, os.WriteFile(filepath.Join(dir, "plugin"), []byte("binary"), 0o755)) + require.NoError(t, os.MkdirAll(filepath.Join(dir, "etc"), 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(dir, "etc", "passwd"), []byte("nobody"), fileMode)) + require.NoError(t, os.Symlink("plugin", filepath.Join(dir, "link"))) + for _, name := range []string{"plugin", "etc/passwd", "etc"} { + require.NoError(t, os.Chtimes(filepath.Join(dir, name), mtime, mtime)) + } + + var buf bytes.Buffer + require.NoError(t, PackDir(dir, &buf)) + + return buf.Bytes() + } + + first := pack(0o644, time.Date(2026, 7, 27, 23, 53, 0, 0, time.UTC)) + second := pack(0o664, time.Date(2026, 10, 30, 16, 49, 0, 0, time.UTC)) + assert.Equal(t, first, second, "the same contents built at another time, under another umask") + + gzReader, err := gzip.NewReader(bytes.NewReader(first)) + require.NoError(t, err) + modes := map[string]int64{} + tarReader := tar.NewReader(gzReader) + for { + header, err := tarReader.Next() + if errors.Is(err, io.EOF) { + break + } + require.NoError(t, err) + modes[header.Name] = header.Mode + } + assert.Equal(t, map[string]int64{ + "etc/": 0o755, "etc/passwd": 0o644, "link": 0o777, "plugin": 0o755, + }, modes) +} + func TestUnpackReplacesExistingDir(t *testing.T) { t.Parallel()