feat: five runtime knobs are named, kept and shipped (ADR 0047)

docs/adr/0046-the-cache-is-the-source.md

@@ -6,6 +6,8 @@ Status: Accepted
 
 Amends: 0031 §7
 
+Amended by: 0047
+
 Moves the drop of `env`, `args` and `command` from the payload seam to the
 informer's transform, so the values are never held rather than never shipped.
 No payload changes.

docs/adr/0047-runtime-knobs-are-named-and-kept.md

@@ -0,0 +1,92 @@
+# 0047. Five environment variables are named, kept and shipped; every other one is still dropped
+
+Date: 2026-08-28
+
+Status: Accepted
+
+Amends: 0046 §2
+
+Narrows ADR 0046's blanket drop of `env` to a closed list of Go runtime knobs,
+and ships them in `workload_metadata`. `envFrom`, `args` and `command` are
+unchanged, and so is every variable not on the list.
+
+## Context
+
+ADR 0046 removed `env` at the informer transform, which was right for the reason
+it gave: environment values are where connection strings and tokens get written
+by hand. It removed something else along with them.
+
+`GOMAXPROCS` left unset under a CPU limit is the most common waste in Go on
+Kubernetes: the runtime counts the node's cores, not the cgroup's quota, so a
+container limited to half a core starts as many scheduler threads as the machine
+has CPUs and spends its quota being descheduled. `GOMEMLIMIT` unset under a
+memory limit is the second: the collector has no idea a ceiling exists, so the
+container is killed where the garbage collector would have run. Both are stated
+entirely in the environment. Neither is visible anywhere else — the module list
+can show that `go.uber.org/automaxprocs` is absent, which is evidence that the
+default was *not* fixed, but not evidence about what was set instead.
+
+Both are readable from a single pass over the cluster, so a report that names
+them needs no profiling and no accumulated history.
+
+There is a way to get this wrong that looks reasonable: keep `env` and let the
+payload builder pick. That is the shape ADR 0046 exists to refuse — the values
+would be held in the cache for the lifetime of the process and dropped at
+serialization, which is the outcome CLAUDE.md invariant 4 ranks second.
+
+## Decision
+
+**1. A closed list of names, applied at the same transform.** `GOMAXPROCS`,
+`GOGC`, `GOMEMLIMIT`, `GODEBUG`, `GOTRACEBACK`. Everything else is removed before
+the object is cached, exactly as ADR 0046 has it. The list is in code, not in
+configuration: a customer-editable list of variables to collect would be a
+setting whose wrong value is a disclosure.
+
+**2. A name is not a promise about the value's origin.** A variable called
+`GOGC` whose value comes from a `secretKeyRef` or a `configMapKeyRef` is a read
+of a Secret or a ConfigMap, which the agent does not do (invariant 4, and it
+holds no RBAC for either). Those are dropped whatever their name.
+
+**3. A knob read from the container's own limits is kept as the field path it
+reads.** `valueFrom.resourceFieldRef` is the canonical way to set `GOMAXPROCS`
+from a CPU limit, and it is the *good* configuration this finding is looking
+for. The spec holds no value in that case — the kubelet resolves it at start —
+so the payload carries `resource:limits.cpu` rather than a number. Naming the
+source answers the question the finding asks: does this knob follow the limit?
+
+**4. `envFrom` stays dropped entirely.** It imports a Secret or ConfigMap whole.
+There is no name to filter on before the import happens, so there is no version
+of this decision that applies to it.
+
+**5. The values ship.** `workload_metadata` records carry `runtime_env`, absent
+when nothing on the list is set — which is the state most of these findings are
+about. Keeping a field in the cache that no payload carries would be collection
+without a consumer, which is the thing this ADR was careful not to do.
+
+## Consequences
+
+**Easier.** The first-hour report can state the two findings above with the
+configuration in hand rather than inferring it from a missing module. `GODEBUG`
+and `GOTRACEBACK` come along because they are the same class of fact and cost
+nothing extra: a fleet-wide view of which `GODEBUG` switches are set is
+occasionally the whole explanation for a performance question.
+
+**Harder / given up.** The promise in `docs/security.md` is no longer "no
+environment variable is read". It is now a list, and a list is a thing a reader
+must check rather than a thing they can take on faith. That is a real reduction
+in how easy the guarantee is to verify, bought for a finding we believe is worth
+it. The mitigation is that the list is short, is in one function, and is
+enumerated in the customer document by name.
+
+A value on this list is still operator-authored free text. `GODEBUG` in
+particular is a comma-separated set of switches, and nothing stops someone
+writing something else into it. The agent does not bound or truncate these
+values, which is the same treatment the image reference already gets and is
+called out for (ADR 0039).
+
+**Not changed.** `args`, `command` and `envFrom` are dropped as before, and so is
+every variable whose name is not one of the five. No new RBAC, no new watch, no
+new privilege: this is a narrowing of a filter the agent already runs.
+
+This ADR records decisions implemented in the same pull request, per the process
+in [`README.md`](README.md).

docs/adr/README.md

@@ -104,3 +104,4 @@ line to paste.
 - [0044. One fact in one place, and the places have ceilings](0044-prose-has-a-ceiling.md)
 - [0045. A kubelet read is bounded in time and in size, and one node's silence costs one node](0045-a-kubelet-read-is-bounded.md)
 - [0046. The informer cache is the source, so the fields the agent promises not to collect are removed there](0046-the-cache-is-the-source.md)
+- [0047. Five environment variables are named, kept and shipped; every other one is still dropped](0047-runtime-knobs-are-named-and-kept.md)

docs/security.md

@@ -464,7 +464,7 @@ VCS hostname and the layout of your source tree. Line numbers are kept.
 
 ## 8. Data collected and data leaving the cluster
 
-This is usually the most important section for review. Data falls into three
+This is usually the most important section for review. Data falls into four
 classes with different sensitivity and different default policies:
 
 | Data class | Examples | Sensitivity | Default policy |
@@ -491,7 +491,7 @@ appearing here (ADR 0022).
 | `pod_disruptions` | Per hour: the pods the *cluster* removed — preempted, evicted under node pressure, drained, or removed via the eviction API — with the node and the instant | **yes** |
 | `job_runs` | Per hour: each Job that finished — when it started and finished, whether it succeeded, the failure reason, its pod success/failure counts, and its declared `parallelism`/`completions`/`backoffLimit` | **yes** |
 | `deployment_revisions` | Per flush: each ReplicaSet of a collected Deployment — its revision number, when it was created, its desired/current/ready replica counts, and each container's image reference | **yes** |
-| `workload_metadata` | Declared shape per (namespace, workload, container, image digest): image, requests, limits, ports, QoS, replica counts by phase, by node, and by unscheduled reason, and the pod's reduced placement constraints (ADR 0031) | no |
+| `workload_metadata` | Declared shape per (namespace, workload, container, image digest): image, requests, limits, ports, the five runtime knobs of the field-minimization list below, QoS, replica counts by phase, by node, and by unscheduled reason, and the pod's reduced placement constraints (ADR 0031) | no |
 | `workload_policy` | Per collected workload, what bounds it from outside its own spec: disruption budgets covering it, autoscalers driving it, and the volume claims its pods mount (ADR 0032); plus `unavailable_sources`, naming any of those the agent could not read — never granted, or no longer being fed (ADR 0033, ADR 0035) | not directly — but a claim is named, and a StatefulSet's claim is `<template>-<set>-<ordinal>`, so `data-db-0` identifies pod `db-0` (ADR 0039) |
 | `cluster_policy` | Per collected namespace, its LimitRanges and ResourceQuotas; plus the cluster's PriorityClasses and StorageClasses, which workloads reference by name (ADR 0032), and `unavailable_sources` for any of those the agent could not read — never granted, or no longer being fed (ADR 0033, ADR 0035). StorageClass `parameters` are never read | no |
 | `node_metadata` | Per node: name, size, instance and capacity type, zone, region, kernel version, CPU architecture. The **name** is your cluster's, and on EKS and on GKE's legacy naming it encodes the node's private address — `ip-10-42-13-201.eu-west-1.compute.internal`. The agent reads no address field; the name it does read may be one (ADR 0039) | no |
@@ -500,11 +500,9 @@ appearing here (ADR 0022).
 | `ebpf_profile` | One capture: allow-list-filtered symbolized pprof bytes, keyed by workload, image digest and capture window | no |
 
 Each declares its provenance in a `source` field — `structural` (read from a
-spec), `measured` (polled from the kubelet), `journal` (from object history), or
-`sampled` (a profiler's estimate) — so the backend can never merge
-epistemically different data under one key
-([ADR 0012](adr/0012-payload-registry-and-provenance.md)). The rest of this
-section describes each in detail.
+spec), `measured` (polled from the kubelet), `journal` (from object history) or
+`sampled` (a profiler's estimate)
+([ADR 0012](adr/0012-payload-registry-and-provenance.md)).
 
 ### Field minimization
 
@@ -521,14 +519,20 @@ section describes each in detail.
   carries a cloud account identifier, a region, your repository taxonomy, and
   whatever your CI writes into a tag.
 - From pod specs: `metadata`, `spec.containers[].resources` and `[].ports`, the
-  nine placement fields below, `ownerReferences`, and `status`. `env`, `envFrom`,
-  `args` and `command` are removed as each object arrives, before it enters the
-  agent's cache, because they frequently contain inline credentials — so they are
-  not held in memory either, not merely kept out of payloads
+  nine placement fields below, `ownerReferences`, and `status`. `envFrom`, `args`
+  and `command` are removed as each object arrives, before it enters the agent's
+  cache, so they are not held in memory either, not merely kept out of payloads
   ([ADR 0046](adr/0046-the-cache-is-the-source.md)). The
-  `kubectl.kubernetes.io/last-applied-configuration` annotation is dropped with
-  them: it is a verbatim copy of the object as applied, environment included.
-  From `status`: the image digest, the
+  `kubectl.kubernetes.io/last-applied-configuration` annotation goes with them —
+  a verbatim copy of the object as applied, environment included.
+- **Five environment variables are kept by name, and no other one is**
+  ([ADR 0047](adr/0047-runtime-knobs-are-named-and-kept.md)): `GOMAXPROCS`,
+  `GOGC`, `GOMEMLIMIT`, `GODEBUG`, `GOTRACEBACK`. They ship in
+  `workload_metadata` as `runtime_env`. The list is in code, not in your values
+  file. One of them whose value comes from a `secretKeyRef` or `configMapKeyRef`
+  is **not** read; one read from the container's own limits ships as the field
+  path (`resource:limits.cpu`), never as a resolved value.
+- From `status`: the image digest, the
   restart counter, a terminated state's reason and exit code, the reason and
   transition time of exactly two conditions — `PodScheduled` and
   `DisruptionTarget` — and `status.reason` when it says `Evicted`. Nothing else.

internal/collector/cachetransform.go

@@ -57,9 +57,44 @@ func dropPodSpecFields(spec *corev1.PodSpec) {
 	}
 }
 
+// runtimeEnvNames is the closed list of environment variables the agent keeps.
+// Every one is a Go runtime knob whose value is a number, a size or a
+// comma-separated set of switches, and whose setting is the whole content of
+// several findings (ADR 0047).
+var runtimeEnvNames = map[string]bool{
+	"GOMAXPROCS":  true,
+	"GOGC":        true,
+	"GOMEMLIMIT":  true,
+	"GODEBUG":     true,
+	"GOTRACEBACK": true,
+}
+
 func dropContainerFields(c *corev1.Container) {
-	c.Env = nil
+	c.Env = keptEnv(c.Env)
+	// envFrom names a Secret or ConfigMap and imports it whole. There is no
+	// name to filter on before the import happens, so it is never kept.
 	c.EnvFrom = nil
 	c.Command = nil
 	c.Args = nil
 }
+
+// keptEnv returns the allow-listed variables, dropping any whose value the
+// agent would have to resolve through a Secret or ConfigMap to read — a
+// runtime knob is written inline or derived from the container's own limits,
+// and one that is not is not worth reaching for (ADR 0047).
+func keptEnv(env []corev1.EnvVar) []corev1.EnvVar {
+	var kept []corev1.EnvVar
+	for _, v := range env {
+		if !runtimeEnvNames[v.Name] {
+			continue
+		}
+		if v.ValueFrom != nil && v.ValueFrom.SecretKeyRef == nil && v.ValueFrom.ConfigMapKeyRef == nil {
+			kept = append(kept, v)
+			continue
+		}
+		if v.ValueFrom == nil {
+			kept = append(kept, v)
+		}
+	}
+	return kept
+}

internal/collector/cachetransform_test.go

@@ -24,7 +24,10 @@ func loadedContainer(name string) corev1.Container {
 		Image:   "example.com/app:1.2.3",
 		Command: []string{"/bin/sh", "-c"},
 		Args:    []string{"exec /app --token=t0ps3cret"},
-		Env:     []corev1.EnvVar{{Name: "DATABASE_URL", Value: secretish}},
+		Env: []corev1.EnvVar{
+			{Name: "DATABASE_URL", Value: secretish},
+			{Name: "GOMAXPROCS", Value: "2"},
+		},
 		EnvFrom: []corev1.EnvFromSource{{
 			SecretRef: &corev1.SecretEnvSource{LocalObjectReference: corev1.LocalObjectReference{Name: "db"}},
 		}},
@@ -85,6 +88,47 @@ func containersIn(obj any) []corev1.Container {
 	return out
 }
 
+func envNames(env []corev1.EnvVar) []string {
+	var names []string
+	for _, v := range env {
+		names = append(names, v.Name)
+	}
+	return names
+}
+
+// The allow-list is a list of names, and a name is not a promise about where
+// the value came from: a variable called GOGC that reads a Secret is a Secret
+// read (ADR 0047).
+func TestARuntimeKnobIsKeptOnlyWhenItsValueIsNotASecret(t *testing.T) {
+	secretRef := &corev1.EnvVarSource{SecretKeyRef: &corev1.SecretKeySelector{
+		LocalObjectReference: corev1.LocalObjectReference{Name: "db"}, Key: "url",
+	}}
+	configRef := &corev1.EnvVarSource{ConfigMapKeyRef: &corev1.ConfigMapKeySelector{
+		LocalObjectReference: corev1.LocalObjectReference{Name: "tuning"}, Key: "gogc",
+	}}
+	limitRef := &corev1.EnvVarSource{ResourceFieldRef: &corev1.ResourceFieldSelector{Resource: "limits.cpu"}}
+
+	c := corev1.Container{Name: "app", Env: []corev1.EnvVar{
+		{Name: "GOGC", ValueFrom: secretRef},
+		{Name: "GOTRACEBACK", ValueFrom: configRef},
+		{Name: "GOMAXPROCS", ValueFrom: limitRef},
+		{Name: "GOMEMLIMIT", Value: "900MiB"},
+		{Name: "DATABASE_URL", Value: secretish},
+		{Name: "GOPATH", Value: "/go"},
+	}}
+	dropContainerFields(&c)
+
+	if got, want := envNames(c.Env), []string{"GOMAXPROCS", "GOMEMLIMIT"}; !reflect.DeepEqual(got, want) {
+		t.Fatalf("kept %v, want %v", got, want)
+	}
+	if got := runtimeEnvOf(&c); !reflect.DeepEqual(got, map[string]string{
+		"GOMAXPROCS": "resource:limits.cpu",
+		"GOMEMLIMIT": "900MiB",
+	}) {
+		t.Errorf("runtime env = %v; a knob read from the container's own limits must name the field", got)
+	}
+}
+
 // "Not collected" beats "collected and dropped": the cache is the source, so
 // the fields have to go before the object is stored, not before it is
 // serialized (CLAUDE.md invariant 4, ADR 0046).
@@ -120,9 +164,12 @@ func TestNoWatchedKindEntersTheCacheCarryingEnvArgsOrCommand(t *testing.T) {
 			t.Fatalf("%s: %v", kind, err)
 		}
 		for _, c := range containersIn(stored) {
-			if c.Env != nil || c.EnvFrom != nil || c.Command != nil || c.Args != nil {
-				t.Errorf("%s: container %q reached the cache with env=%v envFrom=%v command=%v args=%v",
-					kind, c.Name, c.Env, c.EnvFrom, c.Command, c.Args)
+			if c.EnvFrom != nil || c.Command != nil || c.Args != nil {
+				t.Errorf("%s: container %q reached the cache with envFrom=%v command=%v args=%v",
+					kind, c.Name, c.EnvFrom, c.Command, c.Args)
+			}
+			if names := envNames(c.Env); !reflect.DeepEqual(names, []string{"GOMAXPROCS"}) {
+				t.Errorf("%s: container %q kept env %v, want only the allow-listed knob", kind, c.Name, names)
 			}
 			if c.Image == "" || c.Name == "" {
 				t.Errorf("%s: container %q lost a field the payloads are built from", kind, c.Name)

internal/collector/pods.go

@@ -59,8 +59,9 @@ type ContainerPort struct {
 }
 
 // Container is the collected view of a container: name, image, the image
-// digest once the container has started, declared resources, and declared
-// ports. Env, args, and command are deliberately never read (filter early).
+// digest once the container has started, declared resources, declared ports,
+// and the runtime knobs named in ADR 0047. Args and command are never read, and
+// neither is any other environment variable (filter early).
 type Container struct {
 	Name  string `json:"name"`
 	Image string `json:"image"`
@@ -71,6 +72,13 @@ type Container struct {
 	Init        bool            `json:"init,omitempty"`
 	Resources   Resources       `json:"resources"`
 	Ports       []ContainerPort `json:"ports,omitempty"`
+	// RuntimeEnv holds the Go runtime knobs from the container's environment,
+	// and only those: the names are a closed list, and a variable whose value
+	// comes from a Secret or ConfigMap is not read (ADR 0047). A knob set from
+	// the container's own limits carries the field path it derives from rather
+	// than a value, because the value does not exist until the kubelet resolves
+	// it.
+	RuntimeEnv map[string]string `json:"runtime_env,omitempty"`
 }
 
 // PodInfo is the collected view of one pod.
@@ -850,12 +858,14 @@ func (w *PodWatcher) describe(pod *corev1.Pod) PodInfo {
 		info.Containers = append(info.Containers, Container{
 			Name: c.Name, Image: c.Image, Init: true,
 			ImageDigest: digests[c.Name], Resources: resourcesOf(&c), Ports: portsOf(&c),
+			RuntimeEnv: runtimeEnvOf(&c),
 		})
 	}
 	for _, c := range pod.Spec.Containers {
 		info.Containers = append(info.Containers, Container{
 			Name: c.Name, Image: c.Image,
 			ImageDigest: digests[c.Name], Resources: resourcesOf(&c), Ports: portsOf(&c),
+			RuntimeEnv: runtimeEnvOf(&c),
 		})
 	}
 	return info
@@ -954,6 +964,31 @@ func portsOf(c *corev1.Container) []ContainerPort {
 	return ports
 }
 
+// runtimeEnvOf reads the runtime knobs the cache transform kept. A knob set
+// through the downward API has no value in the spec, so the field path it reads
+// is recorded instead — "limits.cpu" is the answer to whether GOMAXPROCS
+// follows the limit, and it is the answer the finding needs (ADR 0047).
+func runtimeEnvOf(c *corev1.Container) map[string]string {
+	var env map[string]string
+	for _, v := range c.Env {
+		value := v.Value
+		switch {
+		case v.ValueFrom == nil:
+		case v.ValueFrom.ResourceFieldRef != nil:
+			value = "resource:" + v.ValueFrom.ResourceFieldRef.Resource
+		case v.ValueFrom.FieldRef != nil:
+			value = "field:" + v.ValueFrom.FieldRef.FieldPath
+		default:
+			continue
+		}
+		if env == nil {
+			env = make(map[string]string, len(c.Env))
+		}
+		env[v.Name] = value
+	}
+	return env
+}
+
 // resourcesOf normalizes a container's declared requests and limits:
 // CPU quantities to millicores, memory to bytes. Absent entries stay nil.
 func resourcesOf(c *corev1.Container) Resources {