security: member builds can read the shared registry push credential and push to any member's images #113

Activity

1. sre-helmcode

   added

   bugSomething isn't workingSomething isn't working

   on Sep 29, 2026
2. sre-helmcode

   assigned

   barckcode

   on Sep 29, 2026
3. sre-helmcode commented on Sep 30, 2026

   

   sre-helmcode

   on Sep 30, 2026

   ContributorAuthor

   More actions

   Phase B design: per-space registry token auth

   Phase A is in helmcode/nan-cloud-api#271. After it, member build code holds no push credential and no GitHub token. Two residuals remain:

   • kaniko reads the shared puller, which can pull every repository;
   • builder and puller are static, all-repo credentials.

   Phase B scopes every registry credential to one space. Design only. Nothing is implemented here.

   1. Registry: distribution token auth

   Replace auth: htpasswd in the registry-docker-registry config:

   auth:
     token:
       realm: https://api.nan.builders/registry/token   # in-cluster callers use the Service URL via the CoreDNS rewrite
       service: registry.nan.builders
       issuer: nan-cloud-api
       rootcertbundle: /etc/registry/token/ca.pem         # public cert only, mounted from a Secret/ConfigMap

   Distribution 3.0 verifies RS256 or ES256 JWTs against rootcertbundle. It checks iss, aud (= service), exp/nbf, and the access claim ([{type: repository, name, actions}]). Catalog (registry:catalog:*) is never granted.

   2. cloud-api: GET /registry/token

   Docker token protocol: basic-auth in, {"token","access_token","expires_in","issued_at"} out.

   • Input: service, one or more scope=repository:<name>:<actions>, and Authorization: Basic <account>:<password>.
   • Accounts live in a new table registry_accounts(id, kind, space_namespace, repo_prefix, password_hash, actions, expires_at, revoked_at, created_at). Passwords are random 32 bytes, stored as SHA-256 (high-entropy secrets, so no bcrypt cost), and compared in constant time.
   • Authorization is the intersection of requested and allowed. It never errors on an over-broad scope; it just issues less (Docker semantics).
     • repository:<space>/<anything>: name must satisfy path.Clean(name) == name and strings.HasPrefix(name, space_namespace+"/").
     • actions ⊆ account.actions.
     • Any other type (for example registry:catalog) is dropped.
   • Token: signed with a key from a k8s Secret (registry-token-signing, never in git), with exp = 5 min, a kid header, and jti. Rotation: publish the new cert in the bundle first, then switch the signer.
   • Limits: rate limit per account and per IP (auth path). Unknown or revoked account → 401 with the same body and timing as a bad password. The endpoint is served only on the in-cluster Service plus api.nan.builders, with no CORS. Tests cover the prefix check with .., space-a vs space-ab/, a mixed-case name, and multiple scopes.

   3. Credentials

   Account	Where	Allowed	Lifetime
   per-build push	Minted at Builder.Trigger, one per Job. Stored as a per-Job Secret with ownerReference → Job (GC'd with it). Mounted only in push.	push,pull on exactly <space>/<app>	expires_at = now + activeDeadline + 5m; revoked by the buildwatcher on terminal state
   per-build pull	Same Secret, separate key, mounted in kaniko	pull on <space>/*	same
   per-space pull	registry-puller in each space-* namespace (replaces the copied template), created at space provisioning, and referenced by the SA imagePullSecrets as today	pull on <space>/*	long-lived, rotatable, revoked on space delete/churn
   platform	nan, tagoro, nan-smoke, test repos	explicit prefixes	long-lived, operator-managed

   This removes the shared registry-builder-creds and registry-puller-template. A leaked build credential can only push to that member's own app, for at most about 20 minutes. A leaked space pull credential can only read that space.

   4. Rollout (no downtime)

   1. Ship the token endpoint and account minting behind a flag. The registry is still on htpasswd, so nothing changes yet.
   2. Backfill per-space pull accounts. Write the new registry-puller Secret next to the old one (a new name, then switch the SA imagePullSecrets).
   3. Switch builds to per-build Secrets (flag on).
   4. Flip the registry to auth: token. Keep the htpasswd users only as token-endpoint accounts during the transition (kind=legacy, all-repo), then revoke them.
   5. Verify:
      • a space pull account gets 401/403 on another space's manifest;
      • a build token cannot push outside <space>/<app>;
      • _catalog is denied.

   5. Open questions

   • Private FROM across spaces (member A building from member B's public image) is not supported unless we add an explicit share. Proposal: not supported.
   • Marketplace images that live outside space-*: they need an operator-managed platform account.
   • Whether to also move the GitHub token scoping (repositories:[repo], contents:read) into the same PR. It is independent and small.

   Required regardless of phase B: rotate builder and puller now (listed as a follow-up in #271), because every build before #271 could read builder.
4. sre-helmcode commented on Sep 30, 2026

   

   sre-helmcode

   on Sep 30, 2026

   ContributorAuthor

   More actions

   Correction to the phase B design above: in distribution 3.0 with htpasswd and no authz rules, the htpasswd module ignores the action, so puller can push as well. As a result:

   • helmcode/nan-cloud-api#271 no longer mounts any registry credential in kaniko. A private FROM registry.nan.builders/... now gets 401. The "kaniko reads the shared puller" residual above no longer applies.
   • In phase B, the "pull-only" guarantee comes only from the token access claim. No htpasswd user should survive the switch to auth: token.
   • Rotating both builder and puller is required, not optional.