Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/adr/0019-build-settings-by-allow-list.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
Date: 2026-08-22
Status: Accepted
Amends: 0017, 0018
Amended by: 0050

Extends [ADR 0017](0017-build-facts-keyed-by-digest.md): the payload it
introduced as `go_dependencies` is renamed `go_build` and gains a `settings`
Expand Down
2 changes: 2 additions & 0 deletions docs/adr/0047-runtime-knobs-are-named-and-kept.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ Status: Accepted

Amends: 0046 §2

Amended by: 0050

Narrows ADR 0046's blanket drop of `env` to a closed list of Go runtime knobs,
and ships them in `workload_metadata`. `envFrom`, `args` and `command` are
unchanged, and so is every variable not on the list.
Expand Down
109 changes: 109 additions & 0 deletions docs/adr/0050-godebug-defaults-are-a-build-fact.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# 0050. The GOMAXPROCS finding rests on a build fact, not on the toolchain version

Date: 2026-08-29

Status: Accepted

Amends: 0019, 0047

Adds two allow-listed GODEBUG defaults to `go_build`, parsed out of the compound
`DefaultGODEBUG` build setting. Corrects [ADR 0047](0047-runtime-knobs-are-named-and-kept.md)'s
Context, which states a runtime behaviour that stopped being universal in Go 1.25.

## Context

ADR 0047 justified keeping `GOMAXPROCS` and `GOMEMLIMIT` with a claim about the
Go runtime: "the runtime counts the node's cores, not the cgroup's quota". That
was true when it was written about every Go binary. It is now true about some of
them.

Go 1.25 made the runtime read the cgroup CPU quota when sizing `GOMAXPROCS`, and
made it follow a quota that changes while the process runs. Both are GODEBUG
settings with a changed default — `containermaxprocs` and `updatemaxprocs`,
`Old: "0"`, `Changed: 25` in the toolchain's own `internal/godebugs` table. So the
waste ADR 0047 calls the most common in Go on Kubernetes is, on a current
toolchain, already absent.

The version that decides this is not the one the agent collects. GODEBUG defaults
follow the `go` directive of the **main module**, not the toolchain that ran the
build: a binary compiled by Go 1.26 from a module declaring `go 1.21` gets the
pre-1.25 behaviour. `go_build.go_version` is the toolchain. Nothing the agent
ships distinguishes the two builds, so the finding fires on both — and the fleet
where it is wrong is the fleet that already did the work.

The toolchain records the difference. When a build's GODEBUG defaults deviate
from the toolchain's own, it writes them into a `DefaultGODEBUG` build setting,
which is exactly the fact the finding needs and is already sitting in the build
info the node scanner reads and discards.

## Decision

**1. `DefaultGODEBUG` is parsed on the node, and only named settings survive.**
The kept names are `containermaxprocs` and `updatemaxprocs`. They ship as a
`godebug` object on `go_build`, beside `settings`.

Parsing is the decision, not a convenience. ADR 0019 keeps build settings by an
allow-list of whole values, and `DefaultGODEBUG` is a comma-joined list of every
deviating setting — six of them for a module on `go 1.21`, most of them about
TLS, `net/http` and `go/types`. Allow-listing it whole would ship all of those,
and would then drop the value entirely against ADR 0019's 128-character bound —
which lands on long lists, which are the old `go` directives, which are the only
builds this field exists to find. The failure mode would be silence on exactly
the population the finding is about.

The two names satisfy ADR 0019's own test for the list: a token the toolchain
chose from a fixed vocabulary, holding `0` or `1`. Nothing operator-written
enters through them. The other sixty names in the table decide nothing about
resources and are not collected.

**2. An absent name is not a value, and the backend is told so.** The toolchain
records only what deviates from its own defaults, so `containermaxprocs` absent
means either "this build gets the current default" or "this toolchain predates
the setting". `go_build.go_version` separates them, and the two fields are read
together or not at all — the discipline of [ADR 0048](0048-findings-name-the-fields-they-need.md) §2,
applied to a field whose absence is the common case.

There is a third input, already collected: `GODEBUG` set in the container's
environment overrides the compiled default at startup, and ADR 0047 keeps that
variable in `workload_metadata.runtime_env`. Build fact and runtime override are
different claims about different objects; the agent ships both and joins
neither.

**3. ADR 0047's Context is corrected here rather than edited there.** An accepted
ADR's body is immutable ([`README.md`](README.md)), so this paragraph is the
forwarding address: ADR 0047 §1's decision — five names, closed list, applied at
the informer transform — stands unchanged. What does not stand is the sentence in
its Context that describes the pre-1.25 runtime as the runtime.

## Consequences

**Easier.** The finding names the builds where it is true. A cluster whose
services are all on a current `go` directive gets silence, which is the answer,
and the report can say what it checked rather than listing every Go service.
`updatemaxprocs` adds a finding that did not exist: a workload whose CPU limit
was resized in place, whose runtime never noticed.

The `/proc/<pid>/status` read that was to supply `Cpus_allowed_list` for this
same finding is not needed for it. That read may still be worth making for
`VmHWM`, on its own argument, and would arrive with its own ADR and its own
extension of ADR 0009's node scope.

**Harder / given up.** A second allow-list, in a second shape, in the same
function's neighbourhood. `buildSettingsAllowList` keeps whole values by key;
this one keeps parsed pairs out of one value. Two mechanisms doing similar work
is a cost, accepted because collapsing them would mean either shipping the
compound string or teaching the first list about structure inside a value.

The list will need revisiting when the runtime grows another knob that reads the
cgroup. That is a maintenance obligation on a file, not on a payload: adding a
name changes what is collected and needs `security.md` in the same PR, which is
the mechanism working.

**Not changed.** No new RBAC, no new read, no new privilege: `DefaultGODEBUG` is
in build info the node scanner already opens, and the parse happens before the
node→controller channel, so nothing outside the list crosses the wire (CLAUDE.md
invariant 4). `go_build` remains write-once and keyed by image digest; the new
object is another immutable property of the build (ADR 0017).

This ADR records decisions implemented in the same pull request, per the process
in [`README.md`](README.md).
1 change: 1 addition & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,3 +107,4 @@ line to paste.
- [0047. Five environment variables are named, kept and shipped; every other one is still dropped](0047-runtime-knobs-are-named-and-kept.md)
- [0048. Four first-hour findings name fields the agent did not collect](0048-findings-name-the-fields-they-need.md)
- [0049. Revisions belong to whatever controls the ReplicaSet, and the payload is renamed for it](0049-revisions-are-not-only-deployments.md)
- [0050. The GOMAXPROCS finding rests on a build fact, not on the toolchain version](0050-godebug-defaults-are-a-build-fact.md)
11 changes: 11 additions & 0 deletions docs/backend-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -349,6 +349,17 @@ build had a VCS working tree, which containerized builds frequently do not: thei
absence is the common case and MUST NOT be surfaced as missing data or as a
coverage gap. The whole `settings` object is omitted when nothing was kept.

**`go_build.godebug` is read together with `go_build.go_version`, never alone**
([ADR 0050](adr/0050-godebug-defaults-are-a-build-fact.md)). It carries at most
two names, `containermaxprocs` and `updatemaxprocs`, and only when the build
deviates from the defaults of the toolchain that produced it. An absent name
therefore MUST NOT be read as `0`: it means the build takes its toolchain's
default, and which default that is follows from `go_version` — from Go 1.25 both
are on, before it neither exists. The object is omitted when the build deviates
in nothing. A `GODEBUG` variable in `workload_metadata.runtime_env` overrides
both at startup; it is a fact about a workload, this is a fact about a build, and
the backend MUST NOT merge them into one value.

**`node_metadata.nodes[].architecture` is what `go_build.settings.GOARCH` is
compared against.** A build's target architecture and microarchitecture level
mean nothing on their own; the comparison is the finding. The field is the
Expand Down
15 changes: 8 additions & 7 deletions docs/prose_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,19 @@ const maxCommentRun = 8

// The two documents that grow with every payload kind, at the size each landed
// once the restated ADRs came out (ADR 0044). `security.md` is deliberately not
// back at ADR 0024's 750: five kinds have been added since, each a table row and
// a paragraph. `backend-requirements.md` restates decisions on purpose — its
// reader implements ingest and reads neither our ADRs nor `security.md` — so
// what is bounded there is growth, not duplication. The file ceiling moved
// 920 → 930 for obligations the document did not have before (ADR 0048).
// back at ADR 0024's 750: kinds have been added since, each a table row and a
// paragraph. `backend-requirements.md` restates decisions on purpose — its
// reader implements ingest and reads neither our ADRs nor `security.md`.
// Ceilings rose for obligations the documents did not have before: 920 → 930
// (ADR 0048), then 930 → 940 and 460 → 475 for a field whose absence the
// backend would otherwise read as a value (ADR 0050).
var ceilings = []struct {
file string
lines int
section int
}{
{"security.md", 930, 290},
{"backend-requirements.md", 460, 320},
{"security.md", 940, 300},
{"backend-requirements.md", 475, 330},
}

func TestNoCommentRunIsLongerThanAPointerToAnADR(t *testing.T) {
Expand Down
10 changes: 9 additions & 1 deletion docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -495,7 +495,7 @@ test fails if a kind ships without appearing here (ADR 0022).
| `cluster_policy` | Per collected namespace, its LimitRanges and ResourceQuotas; plus the cluster's PriorityClasses and StorageClasses, which workloads reference by name (ADR 0032), and `unavailable_sources` for any of those the agent could not read — never granted, or no longer being fed (ADR 0033, ADR 0035). StorageClass `parameters` are never read | no |
| `node_metadata` | Per node: name, size, instance and capacity type, zone, region, kernel version, CPU architecture. The **name** is your cluster's, and on EKS and on GKE's legacy naming it encodes the node's private address — `ip-10-42-13-201.eu-west-1.compute.internal`. The agent reads no address field; the name it does read may be one (ADR 0039) | no |
| `go_inventory` | Per (namespace, workload, container): Go version, main module, image digest, PGO flag, plus a fleet-coverage block | no |
| `go_build` | Per image digest, written once: Go version, main module, each dependency module's **path and version**, allow-listed build settings. This is a bill of materials for the build — see [§10.4](#104-the-build-inventory-is-a-bill-of-materials) | no |
| `go_build` | Per image digest, written once: Go version, main module, each dependency module's **path and version**, allow-listed build settings, and two allow-listed GODEBUG defaults. This is a bill of materials for the build — see [§10.4](#104-the-build-inventory-is-a-bill-of-materials) | no |
| `ebpf_profile` | One capture: allow-list-filtered symbolized pprof bytes, keyed by workload, image digest and capture window | no |

Each declares its provenance in a `source` field — `structural` (read from a
Expand Down Expand Up @@ -707,6 +707,14 @@ the filter-early rule there, before any record is formed.
| `vcs.time` | that commit's timestamp (not the build time — Go does not record it) |
| `vcs.modified` | whether the working tree was dirty |

Two GODEBUG defaults are read out of the toolchain's compound `DefaultGODEBUG`
setting and shipped as `godebug`
([ADR 0050](adr/0050-godebug-defaults-are-a-build-fact.md)): `containermaxprocs`
and `updatemaxprocs`, each `0` or `1`, which say whether the binary's runtime
sizes `GOMAXPROCS` from the container's CPU quota. The compound value is parsed,
never shipped whole: the rest of what it carries is TLS, HTTP and type-checker
switches, and none of it is collected.

Everything outside it is discarded on the node, including **`-ldflags`,
`-gcflags`, `-asmflags` and `-tags`** — free-form flags that routinely carry
build-machine paths, internal hostnames and injected version strings — and the
Expand Down
6 changes: 6 additions & 0 deletions internal/inventory/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,10 @@ type BuildFacts struct {
// scanner sent: the filtering happened there, before the channel, so nothing
// outside the allow-list ever reached this struct (ADR 0019).
Settings map[string]string `json:"settings,omitempty"`
// GoDebug is the allow-listed GODEBUG defaults compiled into the build. It
// is read beside GoVersion, never alone: absence is the toolchain's own
// default, not a missing value (ADR 0050 §2).
GoDebug map[string]string `json:"godebug,omitempty"`
}

// Resolved is what a ContainerResolver returns for a (pod UID, container ID)
Expand Down Expand Up @@ -194,6 +198,7 @@ func (s *Store) ingestBuild(b nodescan.BinaryInfo, digest string) {
MainModule: b.MainModule,
Modules: modules,
Settings: maps.Clone(b.Settings),
GoDebug: maps.Clone(b.GoDebug),
}
}

Expand All @@ -209,6 +214,7 @@ func (s *Store) PendingBuilds() []BuildFacts {
// no aliasing of store state past the lock
d.Modules = slices.Clone(d.Modules)
d.Settings = maps.Clone(d.Settings)
d.GoDebug = maps.Clone(d.GoDebug)
out = append(out, d)
}
}
Expand Down
25 changes: 25 additions & 0 deletions internal/inventory/store_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,31 @@ func TestIngestCarriesBuildSettings(t *testing.T) {
}
}

// The GODEBUG defaults ride the same way, and need their own aliasing check:
// they are a second map on a struct copied out of the store (ADR 0050).
func TestIngestCarriesGoDebugDefaults(t *testing.T) {
resolver := fakeResolver{
"pod-web/cid-web": {Namespace: "shop", WorkloadKind: "Deployment", WorkloadName: "web", Container: "app", ImageDigest: "sha256:web"},
}
b := binary("pod-web", "cid-web", "go1.26.1", "github.com/acme/web", false)
b.GoDebug = map[string]string{"containermaxprocs": "0", "updatemaxprocs": "0"}
s := NewStore(testSince)
s.Ingest(nodescan.Report{Node: "node-1", Binaries: []nodescan.BinaryInfo{b}}, resolver)

pending := s.PendingBuilds()
if len(pending) != 1 {
t.Fatalf("got %d build fact sets, want 1", len(pending))
}
if !maps.Equal(pending[0].GoDebug, b.GoDebug) {
t.Errorf("godebug = %v, want %v", pending[0].GoDebug, b.GoDebug)
}

pending[0].GoDebug["containermaxprocs"] = "tampered"
if again := s.PendingBuilds(); again[0].GoDebug["containermaxprocs"] != "0" {
t.Errorf("containermaxprocs = %q after the caller mutated its copy, want 0", again[0].GoDebug["containermaxprocs"])
}
}

func TestBuildFactsDedupAcrossReplicasAndSurviveUntilMarked(t *testing.T) {
// Two replicas of the same build on two nodes, plus a second build: one
// dependency set per digest, not per replica.
Expand Down
56 changes: 56 additions & 0 deletions internal/nodescan/scan.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,12 @@ type BinaryInfo struct {
// did not record them, which for vcs.* is the common case in container
// builds (ADR 0019).
Settings map[string]string `json:"settings,omitempty"`
// GoDebug is the allow-listed subset of the GODEBUG defaults compiled into
// the binary. An absent name is not a missing value: the toolchain records
// only what differs from its own default, so absence means the default for
// the main module's `go` directive applies, and which default that is
// follows from GoVersion (ADR 0050).
GoDebug map[string]string `json:"godebug,omitempty"`
// PGO is true when the binary was built with profile-guided optimization
// (a "-pgo" build setting with a non-empty value).
PGO bool `json:"pgo"`
Expand Down Expand Up @@ -164,6 +170,7 @@ func (s *Scanner) scanPID(pid int, scope Scope, res *Result) {
MainModule: mainModule,
Dependencies: dependencyModules(info),
Settings: buildSettings(info),
GoDebug: goDebugDefaults(info),
PGO: hasPGO(info),
PodUID: binding.PodUID,
ContainerID: binding.ContainerID,
Expand Down Expand Up @@ -257,6 +264,55 @@ func buildSettings(info *buildinfo.BuildInfo) map[string]string {
return out
}

// goDebugAllowList is the exhaustive set of GODEBUG names kept out of the
// compound `DefaultGODEBUG` build setting. Both decide how the runtime sizes
// GOMAXPROCS against the cgroup's CPU quota, both hold "0" or "1", and both
// changed their default in Go 1.25 — so a binary's behaviour here follows from
// the main module's `go` directive, not from the toolchain that built it
// (ADR 0050).
var goDebugAllowList = map[string]struct{}{
"containermaxprocs": {},
"updatemaxprocs": {},
}

// maxGoDebugValue bounds a kept value. Every allowed name holds a single digit;
// anything longer is not the setting this list assumes and is dropped whole, for
// the reason maxSettingValue exists.
const maxGoDebugValue = 8

// goDebugDefaults returns the allow-listed GODEBUG defaults compiled into the
// binary. `DefaultGODEBUG` is parsed here rather than allow-listed whole because
// it is a comma-joined list that runs past maxSettingValue for exactly the old
// `go` directives this field exists to find (ADR 0050 §1). Later duplicates win,
// as they do in the runtime's own parse.
func goDebugDefaults(info *buildinfo.BuildInfo) map[string]string {
var raw string
for _, s := range info.Settings {
if s.Key == "DefaultGODEBUG" {
raw = s.Value
break
}
}
var out map[string]string
for pair := range strings.SplitSeq(raw, ",") {
name, value, ok := strings.Cut(pair, "=")
if !ok {
continue
}
if _, allowed := goDebugAllowList[name]; !allowed {
continue
}
if value == "" || len(value) > maxGoDebugValue {
continue
}
if out == nil {
out = make(map[string]string, len(goDebugAllowList))
}
out[name] = value
}
return out
}

// hasPGO reports whether the binary was built with profile-guided optimization.
// The toolchain records this as a "-pgo" build setting whose value is the
// profile path ("off" or empty means no PGO). Only the boolean survives: the
Expand Down
Loading