feat: a state says when it began (ADR 0087)

docs/adr/0054-coverage-says-how-much-was-hidden-never-what.md

@@ -4,7 +4,7 @@ Date: 2026-08-30
 
 Status: Accepted
 
-Amended by: 0057, 0058, 0060
+Amended by: 0057, 0058, 0060, 0087
 
 Amends: 0012 §2, 0033
 

docs/adr/0067-a-record-carries-the-age-of-its-source.md

@@ -4,6 +4,8 @@ Date: 2026-09-04
 
 Status: Accepted
 
+Amended by: 0087
+
 Amends: 0052, 0056, 0062
 
 The node-fed payloads gain `asserted_at`, `go_inventory.coverage` gains the list

docs/adr/0087-a-state-says-when-it-began.md

@@ -0,0 +1,101 @@
+# 0087. A state says when it began
+
+Date: 2026-09-18
+
+Status: Accepted
+
+Amends: 0054, 0067
+
+`sources[].failing` and `shipping.halted` gain a start instant beside them, and
+the metrics endpoint gains one gauge. No new collection, no new privilege, no
+new read: both instants are already held in memory and discarded at payload
+time.
+
+## Context
+
+ADR 0054 made `collection_coverage` the payload the agent answers for itself
+with, so that an empty report and a broken agent are not the same bytes. ADR
+0067 established that a record carries the instant its source last stated it,
+and applied it to the node-fed records — not to the agent's own state.
+
+Every number in that payload is a counter, cumulative from `since`, and two
+fields are not numbers at all:
+
+```
+sources[].failing   bool
+shipping.halted     bool
+```
+
+A boolean describes the present, and this kind supersedes — the backend upserts,
+last write wins, and one row is all it is obliged to keep. So a state that ended
+leaves nothing behind. A watch that failed for nineteen minutes and recovered
+produces, on the next flush, bytes identical to a watch that never failed; an
+outage shorter than the flush interval is never stated at all. The agent knew
+and did not say.
+
+`halted` is the sharper case, and its own field comment already half-admitted
+it: a halted agent ships nothing, so the payload carrying the fact never
+travels. Its readers are whoever holds the spool and whoever scrapes the metrics
+endpoint, and to both of them "shipping stopped" without "at 04:31 nine days
+ago" is nearly the same as silence.
+
+The third fact is what makes this cheap. Both instants already exist.
+`watchHealth` records where each run of failures began — that is what ADR 0035's
+streak is measured from — and `SourceHealths` reduced it to a boolean on the way
+out. The halt had no clock at all, because nothing had ever asked it for one.
+
+What is *not* in scope here is history. A state that ended is absent from the
+next payload by design, and no field added below changes that.
+
+## Decision
+
+**1. Two states carry the instant they began.** `sources[].failing_since` and
+`shipping.halted_since`. Absent is the claim that the agent is not in that
+state, so a source that recovers drops the field rather than restating it.
+
+**2. The instant is where the run began, not where it last failed.** What
+stopped being true is that the cache was fed, and it stopped when the run
+started. The latest failure says only when the agent last looked, which is a
+fact about the agent's schedule and not about the source.
+
+**3. The boolean stays for now, derived from the instant.** §6 of
+`backend-requirements.md` requires both forms across a deprecation period, and
+the schema gate enforces the same: `buf breaking` runs the `FILE` category,
+which refuses the deletion of a field even with its number reserved — measured,
+not assumed. Both booleans are therefore computed from the instant beside them
+in one place each, so no payload can carry a state with no start or a start with
+no state.
+
+**4. `synced` stays a boolean and is not amended.** It is a latch rather than a
+state that ends: it goes true once and never back. The instant it would carry
+for the false case is the payload's own `since` — the cache has not filled since
+this process began counting — and for the true case it answers a question about
+startup that nothing asks.
+
+**5. One gauge, not two.** `shipping_halted_since_seconds` joins
+`shipping_halted`, because for a halt the metrics endpoint is the only channel
+that works. `source_failing` gets no companion: its payload arrives on every
+pass, and a scrape of the boolean is dated by the scrape.
+
+## Consequences
+
+**Easier.** "When did this stop?" becomes answerable for the two states where it
+was not, and answerable from the payload rather than from a customer's logs. The
+spool file a halted agent leaves behind stops being a fact about the file's own
+timestamp and becomes a fact about the halt.
+
+**Harder / given up.** The payload carries two fields where it carried one, and
+will until the booleans are removed — which needs either a released agent to
+deprecate against or a decision about the `FILE` gate, and is neither taken nor
+pre-judged here. Nothing else grows: both instants were already in memory.
+
+A state that ended is still invisible, and this ADR does not pretend otherwise.
+Recovering that needs occupancy within a window, which is a different mechanism
+with a different cost, and is not decided here.
+
+**Not changed.** What is collected, what is filtered, what is read, any cadence,
+any natural key, or the one-way protocol. No identity of any object reaches
+either new field: a source is a resource class, and a halt is about this agent.
+
+This ADR records decisions implemented in the same pull request, per the process
+in [`README.md`](README.md).

docs/adr/README.md

@@ -144,3 +144,4 @@ line to paste.
 - [0084. What was excluded is named, inside the cluster and only there](0084-what-was-excluded-is-named-inside-the-cluster.md)
 - [0085. The image is built and scanned where the code is, and pinned to what it was built from](0085-the-image-is-built-and-scanned-by-ci.md)
 - [0086. A toolchain move is a review of what the toolchain reports](0086-a-toolchain-move-reviews-what-it-reports.md)
+- [0087. A state says when it began](0087-a-state-says-when-it-began.md)

docs/backend-requirements.md

@@ -453,7 +453,13 @@ a hash of it; `agent.config.since` is when that configuration took effect, and i
 is what separates "the data changed" from "the filters changed". `sources[]`
 reports what the agent's reads actually did, so a source that is `synced: false`
 or `failing: true` bounds every finding resting on it — the backend MUST NOT
-present such a finding as complete. The scan block is a fleet aggregate over the
+present such a finding as complete. `failing_since` is when the run of failures
+that left a cache unfed began, and `failing` is the same fact as a boolean,
+derived from it, deprecated and retained for the window of section 6. An absent
+instant means **not failing now**, never "never failed": a source that failed
+and recovered between two flushes states nothing at all, so the backend MUST NOT
+read a clean payload as evidence that these reads were healthy throughout the
+period since `since` ([ADR 0087](adr/0087-a-state-says-when-it-began.md)). The scan block is a fleet aggregate over the
 latest pass of each reporting node and MUST NOT be summed across flushes.
 
 The `filter` counters count **decisions, not pods**. A customer who annotates a

docs/prose_test.go

@@ -35,23 +35,23 @@ var ceilings = []struct {
 	section int
 }{
 	// Each raise names the obligation that bought it: payloads leave the cluster
-	// at all (0075); a journal window states when it was written (0078); an
-	// endpoint confirmed once per image is now read once a minute per process
-	// (0080); twelve more GODEBUG names come out of the build (0081); every
-	// request names the agent that made it (0083); a listener answers inside the
-	// cluster what the filters excluded, by name (0084); and the image a customer
-	// trusts with a node states what it is made of, and what does not yet back
-	// it (0085).
-	{"security.md", 1135, 366},
+	// at all (0075); a journal window states when it was written (0078); a
+	// confirmed endpoint is read once a minute per process (0080); twelve more
+	// GODEBUG names come out of the build (0081); a request names the agent that
+	// made it (0083); a listener names inside the cluster what the filters
+	// excluded (0084); the image states what it is made of and what does not yet
+	// back it (0085); a failing read says since when (0087), whose diff also
+	// corrects the startup-read list from five kinds to the six it reads.
+	{"security.md", 1138, 366},
 	// Each raise names its obligation here too. The body is encoded, so a
-	// payload's size is no longer the request's (ADR 0076); a window states
-	// whether it is finished (ADR 0078); gauge samples arrive, where adding
-	// them up is the wrong ingest (ADR 0080); `godebug` carries names where
-	// neither value is safe (ADR 0081); the filter counters count decisions
-	// (ADR 0054); a port's `pprof` is absent when nothing asked, not `absent`
-	// (ADR 0082); and a request may name an agent that states no version
-	// (ADR 0083).
-	{"backend-requirements.md", 823, 644},
+	// payload's size is no longer the request's (0076); a window states whether
+	// it is finished (0078); gauge samples arrive, where adding them up is the
+	// wrong ingest (0080); `godebug` carries names where neither value is safe
+	// (0081); the filter counters count decisions (0054); a port's `pprof` is
+	// absent when nothing asked, not `absent` (0082); a request may name an
+	// agent that states no version (0083); and an absent instant is told apart
+	// from an absent state (0087) — the raise that also moved the section.
+	{"backend-requirements.md", 829, 651},
 }
 
 func TestNoCommentRunIsLongerThanAPointerToAnADR(t *testing.T) {

docs/security.md

@@ -272,8 +272,9 @@ and 20000 files, dropping the oldest first; and the `emptyDir` declares a 1 GiB
 - **The agent reads it once, at startup**, so a restart resumes its open windows
   rather than starting them over ([ADR 0072](adr/0072-a-restart-resumes-the-window-the-spool-holds.md),
   [ADR 0077](adr/0077-a-restart-resumes-every-open-window.md)). It is a pure read
-  of five kinds and no others — `usage_snapshot`, `container_restarts`,
-  `pod_disruptions`, `node_lifecycle`, `job_runs` — and it deletes nothing.
+  of six kinds and no others — `usage_snapshot`, `container_restarts`,
+  `pod_disruptions`, `node_lifecycle`, `job_runs`, `goroutine_counts` — and it
+  deletes nothing.
 - **Configuration is never cached on it.** Filters are read from the ConfigMap at
   every start and never reread while running; a stale filter set cannot resurrect
   from disk, and a filter change takes a restart.
@@ -1092,7 +1093,9 @@ cluster at `/metrics` ([§5](#5-network-access)) — so filtering behavior is
 verifiable by you and from what we receive, not only from what you configured.
 Both report which of the agent's own reads worked, measured from its caches
 rather than read back from the rules, so a grant defeated by a webhook reads as
-failing. No `SelfSubjectRulesReview` self-audit is performed: that is a `create`
+failing — and says since when, so an outage you have already fixed is not
+reported as one you still have (ADR 0087). No `SelfSubjectRulesReview`
+self-audit is performed: that is a `create`
 call (ADR 0054 §3). A 403 degrades with a log line, never a crash-loop.
 
 **And named, to you, inside your cluster.** The counts are what *we* receive. You can read the same decisions resolved to objects at `/collection` on the controller's loopback address ([§5](#5-network-access)): one line per namespace, pod and Job, whether it is collected, whether it may be profiled, which of your five controls refused it, and whether its workload was one the agent cannot read. It is computed on request from the same caches and the same filter code the agent collects with, so it cannot disagree with what it does. Nothing of it is transmitted, and nothing of it is remembered.

internal/collector/pods.go

@@ -309,10 +309,12 @@ func (w *PodWatcher) SourceHealths() []model.SourceHealth {
 	}
 	out := make([]model.SourceHealth, 0, len(w.policySources))
 	for _, source := range w.policySources {
+		since := source.health.failingSince(now, window)
 		out = append(out, model.SourceHealth{
-			Name:    source.name,
-			Synced:  source.synced != nil && source.synced(),
-			Failing: source.health != nil && source.health.failedWithin(now, window),
+			Name:         source.name,
+			Synced:       source.synced != nil && source.synced(),
+			Failing:      since != nil,
+			FailingSince: since,
 		})
 	}
 	sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })

internal/collector/watchhealth.go

@@ -129,14 +129,30 @@ func (h *watchHealth) failingFor(now time.Time) (time.Duration, string) {
 	return h.last.Sub(h.first), h.lastErr
 }
 
-// failedWithin reports whether this cache failed at any point in the last d.
-func (h *watchHealth) failedWithin(now time.Time, d time.Duration) bool {
+// failingSince is when the run of failures that leaves this cache unfed began,
+// and nil when it is not failing by that test.
+//
+// The instant is the run's first failure rather than its latest: what stopped
+// being true is that the cache was fed, and that stopped when the run started.
+// The latest failure is only when the agent last looked (ADR 0087).
+func (h *watchHealth) failingSince(now time.Time, d time.Duration) *time.Time {
 	if h == nil {
-		return false
+		return nil
 	}
 	h.mu.Lock()
 	defer h.mu.Unlock()
-	return !h.last.IsZero() && now.Sub(h.last) <= d
+	if h.last.IsZero() || now.Sub(h.last) > d {
+		return nil
+	}
+	first := h.first
+	return &first
+}
+
+// failedWithin reports whether this cache failed at any point in the last d.
+// Answered from failingSince so that the boolean and the instant beside it in
+// the payload cannot disagree.
+func (h *watchHealth) failedWithin(now time.Time, d time.Duration) bool {
+	return h.failingSince(now, d) != nil
 }
 
 // watchdog blocks until ctx is canceled, or until one of the gating caches has

internal/collector/watchhealth_test.go

@@ -352,3 +352,50 @@ func waitFor(t *testing.T, within time.Duration, what string, cond func() bool)
 	}
 	t.Fatalf("timed out waiting for %s", what)
 }
+
+// The instant a source carries is where its run of failures began, not where it
+// last failed. A reader asking when the cache stopped being fed is asking about
+// the start; the latest failure only says when the agent last looked (ADR 0087).
+func TestAFailingSourceIsDatedFromTheStartOfTheRun(t *testing.T) {
+	t0 := time.Date(2026, 9, 18, 12, 0, 0, 0, time.UTC)
+	h := newWatchHealth(2 * time.Minute)
+	h.record(t0, errors.New("forbidden"))
+	h.record(t0.Add(time.Minute), errors.New("forbidden"))
+	h.record(t0.Add(2*time.Minute), errors.New("forbidden"))
+
+	since := h.failingSince(t0.Add(3*time.Minute), 10*time.Minute)
+	if since == nil {
+		t.Fatal("a cache failing three times running is not reported as failing")
+	}
+	if !since.Equal(t0) {
+		t.Errorf("failing since %s, want the first failure at %s", since, t0)
+	}
+}
+
+// A run that ended leaves nothing behind: the field is absent, and the claim
+// the payload makes is that the cache is fed again. What the outage was is the
+// window's to answer, not this field's.
+func TestASourceThatRecoveredCarriesNoInstant(t *testing.T) {
+	t0 := time.Date(2026, 9, 18, 12, 0, 0, 0, time.UTC)
+	h := newWatchHealth(2 * time.Minute)
+	h.record(t0, errors.New("connection reset"))
+
+	if since := h.failingSince(t0.Add(time.Hour), 10*time.Minute); since != nil {
+		t.Errorf("a cache fed again for an hour is dated %s", since)
+	}
+}
+
+// The boolean and the instant are one answer read twice, so no payload can
+// carry a source that is failing with no start, or a start with no failure.
+func TestTheBooleanAndTheInstantAgree(t *testing.T) {
+	t0 := time.Date(2026, 9, 18, 12, 0, 0, 0, time.UTC)
+	h := newWatchHealth(2 * time.Minute)
+	h.record(t0, errors.New("forbidden"))
+
+	for _, at := range []time.Time{t0, t0.Add(time.Minute), t0.Add(time.Hour)} {
+		since, failing := h.failingSince(at, 10*time.Minute), h.failedWithin(at, 10*time.Minute)
+		if (since != nil) != failing {
+			t.Errorf("at %s: failing=%v, instant=%v", at, failing, since)
+		}
+	}
+}

internal/metrics/collect.go

@@ -243,6 +243,14 @@ func CollectController(c Controller) *Set {
 		s.Gauge("shipping_halted",
 			"Whether shipping has stopped on an identity failure the agent will not retry into.",
 			b2f(sh.Halted))
+		// And when, which is the half that matters here: nothing clears a halt
+		// but a restart, so the question a scrape answers is how long this
+		// agent has been silent (ADR 0087).
+		if sh.HaltedSince != nil {
+			s.Gauge("shipping_halted_since_seconds",
+				"When shipping stopped, in seconds since the epoch. Absent while shipping runs.",
+				float64(sh.HaltedSince.Unix()))
+		}
 	}
 	return s
 }

internal/metrics/collect_test.go

@@ -229,6 +229,7 @@ func TestAnUnconfiguredControllerOmitsWhatItDoesNotHave(t *testing.T) {
 		// An agent with no backend has not failed to ship: it was told to ship
 		// nowhere, and a delivered count of zero would claim otherwise.
 		"shipments_total", "shipments_rejected_total", "shipping_halted",
+		"shipping_halted_since_seconds",
 	} {
 		if present[Prefix+absent] {
 			t.Errorf("%s%s is exposed by an installation that has no such component", Prefix, absent)

internal/model/coverage.go

@@ -1,5 +1,7 @@
 package model
 
+import "time"
+
 // Coverage is an aggregate snapshot of what the filter admitted and
 // excluded, counted once per pod appearance. It is the seed of the coverage
 // report (docs/security.md §11): full information about what is collected,
@@ -46,11 +48,18 @@ type SourceHealth struct {
 	// Name is the resource class, not a customer object: "services",
 	// "endpoint_slices", and so on.
 	Name string `json:"name"`
-	// Synced is whether the cache ever filled.
+	// Synced is whether the cache ever filled. It is a latch rather than a
+	// state that ends, and the instant it would carry is the payload's own
+	// `since` — so it stays a boolean where the two below do not (ADR 0087).
 	Synced bool `json:"synced"`
 	// Failing is whether its watch has been erroring recently enough to treat
-	// the cache as no longer fed (ADR 0035).
+	// the cache as no longer fed (ADR 0035). Derived from FailingSince so the
+	// two cannot disagree, and kept only for the compatibility window.
 	Failing bool `json:"failing,omitempty"`
+	// FailingSince is when the run of failures that left the cache unfed began.
+	// Nil is the claim that it is still being fed: a source that recovers drops
+	// the field, and what it was doing before that is the window's to say.
+	FailingSince *time.Time `json:"failing_since,omitempty"`
 }
 
 // NodeDrops is what the node reductions refused to carry, cumulative since the
@@ -129,11 +138,15 @@ type Shipping struct {
 	Unreadable uint64 `json:"unreadable"`
 	// Halted is whether shipping has stopped on an identity failure. Nothing
 	// retries into a rejected credential, so the agent stops rather than turning
-	// a fleet's spools into a retry storm against it.
+	// a fleet's spools into a retry storm against it. Derived from HaltedSince,
+	// and kept only for the compatibility window.
 	//
 	// A halted agent ships nothing, this payload included, so its readers are
 	// whoever holds the spool or scrapes the metrics endpoint.
 	Halted bool `json:"halted"`
+	// HaltedSince is when that halt began. A halt has no end — it is cleared by
+	// a restart, which moves `since` and starts the counters over.
+	HaltedSince *time.Time `json:"halted_since,omitempty"`
 	// Rejected is what the backend refused permanently, by reason.
 	Rejected ShippingRejections `json:"rejected"`
 	// PayloadBytes and TransmittedBytes are the same deliveries measured on