feat: finished Job runs ship as a windowed journal (ADR 0029)

[kfeofantov]

Slice C1b. Adds the payload kind job_runs; nothing existing changes shape.

Why

Usage rollups systematically misrepresent short-lived workloads. A Job that runs ninety seconds inside an hour-long window reports a tiny average, because the average is over the window and the workload existed for a fortieth of it. ADR 0013's covered_nanoseconds says the coverage was low; it does not say the workload was a batch run that started, did its work, and finished.

That gap lands where it costs most. CronJob requests are routinely set from the peak run and then left, so a batch fleet is where overprovisioning accumulates quietly — and it is the one workload class the agent measures but cannot explain.

What ships

job_runs · source journal · key (window start, window length) · supersedes — the same shape as container_restarts and pod_disruptions, for ADR 0021's reason: a fleet of CronJobs can finish hundreds of runs an hour, and one file per run would put the spool's file count under the cluster's control.

Per record: namespace, the scheduling CronJob (or the Job itself when bare), the run's own name, started_at/finished_at, result, failure reason, the run's pod success/failure counts, and the declared parallelism/completions/backoffLimit.

The last two groups are there because without them the outcome is unreadable. A run that succeeded with failed: 2 retried twice, and those retries consumed resources nothing else in the protocol reports. "Three succeeded" is a complete run against completions: 3 and a third of one against completions: 9.

Failure reasons come from Kubernetes' vocabulary — BackoffLimitExceeded and DeadlineExceeded are different capacity facts — and the free-text message beside them is never read (ADR 0020 §6).

Reversal from the plan: no baseline

The plan said to baseline already-terminal Jobs on the ADR 0020 §5 precedent. Building it showed the precedent does not transfer.

Restarts are baselined because a restart counter carries no time, so reporting a pre-existing count would file history in whichever window happens to be open at startup. A Job carries status.startTime, status.completionTime and its condition's transition time — nothing has to be invented, and an already-finished run files itself in the window where it actually finished. This is the distinction ADR 0021 drew for disruptions.

So a freshly started agent picks up the cluster's recent batch history, correctly dated, instead of discarding it. The burst is bounded by the cluster's own retention: three successful and one failed Job per CronJob by default, less where ttlSecondsAfterFinished is set.

Four filter steps, not three

Namespace allow/deny → the namespace's annotation → the owning CronJob's annotation (ADR 0028's workload step) → the Job's own annotations or its pod template's.

That last one is what keeps the payload honest. Before ADR 0028 the documented way to opt a workload out was an annotation in its pod template, so a customer who did that has excluded pods — and if job_runs read only the Job object, the agent would ship facts about a workload whose pods it refuses to measure. Collected and sent is worse than collected and dropped.

Template metadata only. The env, args and command beneath it are never read, on a Job no less than on a pod.

Job counters are kept apart from pod counters: one Job makes many pods, so a shared observed number would answer neither question.

Cost

No new RBAC and no new informer. The Job handler rides the watch already running for owner resolution.

Changes

• docs/adr/0029-job-runs.md
• internal/collector/jobs.go — JobRun, the terminal-condition read, dedup by UID, admitJob
• internal/collector/filter.go — AdmitJob, ExcludedByObjectAnnotation, five Job counters
• internal/collector/pods.go — the handler on the existing Job informer
• internal/journal/jobs.go — the windowed accumulator
• internal/sink/ — payload, WriteJobRuns, registry row, golden
• cmd/agent/main.go — wiring, flusher, coverage log
• docs/security.md — payload table, the Job field-minimization rule, the RBAC why, and what job_runs names
• docs/plans/job-and-rollout-facts.md — C1a ticked, C1b's reporting rule corrected in place

Verification

make test and make lint pass.

Golden covers the three cases a consumer must tell apart: a run that succeeded after retries, one that exhausted its backoff limit, and a bare Job that failed before starting (no started_at — omitted rather than rendered as the epoch).

Collector tests run the real Job informer against a fake cluster: instants and outcome come from the cluster not the observation, a running Job is not a run, a pod-template opt-out excludes the run, and a CronJob opt-out excludes its runs. The pod-template step was negative-tested by removing it — TestPodTemplateOptOutExcludesTheRun fails on both the report and the counter.

Journal tests cover filing by finish instant (including a run three hours old), idempotent re-observation, unfinished runs, and window closing.

🤖 Generated with Claude Code