diff --git a/contracts/tag-step-milestone-gate-v1.yaml b/contracts/tag-step-milestone-gate-v1.yaml new file mode 100644 index 0000000000..8e18d272fc --- /dev/null +++ b/contracts/tag-step-milestone-gate-v1.yaml @@ -0,0 +1,104 @@ +metadata: + version: 1.0.0 + created: '2026-09-20' + author: PAIML Engineering + description: "The release train's tag step cannot be reached without a clean milestone. v0.68.1 was tagged 2026-09-17T15:06:29Z by an autopilot copy living outside the repository, six minutes after #3455 merged the milestone gate; `grep -c check_milestone_cut autopilot.sh` was 0. The spec required the read and no code path that cuts a tag made it. This contract is the property that the requirement and the mechanism are the same object: the gate call lives INSIDE cut_tag(), ahead of `git tag`, and a guard runs that function against each gate outcome and asserts on whether a tag was cut." + kind: pattern + references: + - "scripts/release/autopilot.sh — cut_tag(); the gate call precedes `git tag` inside it" + - "scripts/check_milestone_cut.sh — the gate (#3445, #3455); exit 0 clean, 1 open items, 2 cannot judge" + - "scripts/check_tag_step_gated.sh — the behavioural guard and its mutants" + - "docs/specifications/APR-RELEASE-001-train-and-build-kaizen.md — §4 T-0/T-3" + - "docs/specifications/06x-release-schedule.md — §4 steps 1 and 4" + - "contracts/release-schedule-06x-v1.yaml — milestone_settled_at_cut, the schedule-level statement of the same rule" + +five_whys: + symptom: "v0.68.1 was tagged while nothing had read its milestone, six minutes after the gate that would have read it merged." + why_1: "The script that cuts the tag was not in the repository — it was a per-train copy under /mnt/nvme-raid0/agent-wt/, so merging a gate into the repo could not reach it." + why_2: "The gate was wired into specifications and into CI, and the tag is cut by neither: the one caller that matters was the one surface nobody enumerated." + why_3: "A rule landed without its enforcer in the same change, so the enforcer's absence was invisible from the rule's side." + why_4: "The only evidence that the gate ran would have been a log line, and a missing log line reads as 'nothing to report' rather than as 'the gate did not run'." + why_5: "Nothing could refuse the tag: `git tag` was reachable from a code path that made no claim about the milestone at all." + root_cause: "The requirement lived in prose and the tag lived in an unversioned script, so there was no object in which both could be false together. The remedy is that the gate call is structurally inside the function that tags, and a guard proves by execution — not by grep — that no gate outcome other than 0 reaches `git tag`." + +equations: + tag_requires_clean_milestone: + formula: "cut_tag(v, t, mc) reaches `git tag` iff check_milestone_cut.sh(v) exited 0; rc 1 and rc 2 both die before it" + domain: "the gate's exit codes {0, 1, 2+}" + codomain: "a tag is cut, or the train stops with a STOP line naming which code refused it" + invariants: + - "rc 1 (the milestone holds open items) cuts no tag and publishes nothing" + - "rc 2 (the gate could not judge) cuts no tag — Unknown is not a pass" + - "the gate call precedes `git tag` inside one function, so removing it is a diff to that function and not to a distant caller" + + guard_is_behavioural_not_textual: + formula: "check_tag_step_gated.sh extracts cut_tag() from the autopilot, runs it against a stubbed gate once per outcome, and asserts on the presence of a GIT-TAG in the transcript" + domain: "scripts/release/autopilot.sh and three mutants of it" + codomain: "PASS only when rc 0 tags and rc 1 and rc 2 do not" + invariants: + - "a guard that greps for the gate's NAME is satisfied by a comment; this one is not" + - "a mutant with the gate call deleted is RED" + - "a mutant whose gate verdict is discarded with `|| true` is RED" + - "a subject with no cut_tag() is ENV (rc 2), never a pass" + +proof_obligations: + - id: TSMG-INV-001 + type: invariant + property: "No gate outcome other than 0 reaches `git tag`" + formal: "scripts/check_tag_step_gated.sh exit 0" + applies_to: tag_requires_clean_milestone + - id: TSMG-INV-002 + type: invariant + property: "The guard still fails on the defect it names" + formal: "scripts/check_tag_step_gated.sh --self-test exit 0 — three mutants killed and the real subject green" + applies_to: guard_is_behavioural_not_textual + - id: TSMG-INV-003 + type: invariant + property: "The autopilot that cuts the tag is in the repository and reviewed" + formal: "test -f scripts/release/autopilot.sh" + applies_to: tag_requires_clean_milestone + +falsification_tests: + - id: FALSIFY-TSMG-001 + rule: "The tag path is in the repository" + prediction: "scripts/release/autopilot.sh is a regular file" + test: "test -f scripts/release/autopilot.sh" + if_fails: "The tag is cut by a script no pull request has ever reviewed — the v0.68.1 condition, restored" + + - id: FALSIFY-TSMG-002 + rule: "The tag step names the milestone gate" + prediction: "cut_tag() in the autopilot calls scripts/check_milestone_cut.sh" + test: "awk '/^cut_tag\\(\\) \\{/,/^\\}/' scripts/release/autopilot.sh | grep -q 'check_milestone_cut\\.sh'" + if_fails: "`grep -c check_milestone_cut autopilot.sh` is 0 again, which is exactly what was true when v0.68.1 was tagged" + + - id: FALSIFY-TSMG-003 + rule: "The gate call precedes `git tag` inside cut_tag()" + prediction: "In cut_tag(), the check_milestone_cut.sh line is at a lower line number than the `git tag` line" + test: "b=$(awk '/^cut_tag\\(\\) \\{/,/^\\}/' scripts/release/autopilot.sh); g=$(printf '%s\\n' \"$b\" | grep -n 'check_milestone_cut\\.sh' | head -1 | cut -d: -f1); t=$(printf '%s\\n' \"$b\" | grep -n 'git tag -a' | head -1 | cut -d: -f1); [ -n \"$g\" ] && [ -n \"$t\" ] && [ \"$g\" -lt \"$t\" ]" + if_fails: "A gate that runs after the tag is a receipt, not a gate" + + - id: FALSIFY-TSMG-004 + rule: "No gate outcome other than 0 reaches `git tag`" + prediction: "The guard runs cut_tag() against a stubbed gate at rc 0, 1 and 2 and exits 0" + test: "bash scripts/check_tag_step_gated.sh" + if_fails: "A tag can be cut while the milestone holds open items, or while the gate could not judge at all" + + - id: FALSIFY-TSMG-005 + rule: "The guard can still fail on the defect it names" + prediction: "Three mutants — gate call deleted, gate verdict discarded, cut_tag() removed — are each RED, and the real subject is GREEN" + test: "bash scripts/check_tag_step_gated.sh --self-test" + if_fails: "The guard is theater: it would have passed the tree that tagged v0.68.1" + + - id: FALSIFY-TSMG-006 + rule: "The gate it calls is itself present and self-testing" + prediction: "scripts/check_milestone_cut.sh is a regular file — the repo invokes guards as `bash scripts/...`, so the executable bit is not the property (guard_tree.sh says so of its whole universe)" + test: "test -f scripts/check_milestone_cut.sh" + if_fails: "cut_tag() calls a script that is not there; every cut becomes rc 2 and the train stops — fail-closed, but the gate is gone" + +qa_gate: + id: F-TSMG-001 + name: "The release train's tag step cannot be reached without a clean milestone" + description: "The gate call lives inside cut_tag() ahead of `git tag`, and check_tag_step_gated.sh proves by execution that rc 1 and rc 2 both refuse the tag. guard_tree.sh's universe is `git ls-files 'scripts/check_*.sh'`, so the guard runs in the guard-tree job, which `ci / gate` reads through needs.guard-tree.result." + checks: + - "tag_requires_clean_milestone" + - "guard_is_behavioural_not_textual" diff --git a/scripts/check_tag_step_gated.sh b/scripts/check_tag_step_gated.sh new file mode 100755 index 0000000000..4545b2f011 --- /dev/null +++ b/scripts/check_tag_step_gated.sh @@ -0,0 +1,128 @@ +#!/usr/bin/env bash +# check_tag_step_gated.sh -- the release train's tag step must call +# scripts/check_milestone_cut.sh, and must not be reachable without it (PMAT-3459). +# +# THE DEFECT, measured 2026-09-17. v0.68.1 was tagged at 15:06:29Z by a per-train +# autopilot copy living OUTSIDE the repository, six minutes after #3455 merged the +# milestone gate. `grep -c check_milestone_cut autopilot.sh` was 0: the tag step read +# no milestone at all. The spec required the read; no code path that cuts a tag made it. +# +# WHY THIS IS BEHAVIOURAL AND NOT A grep. "The file mentions the gate" is satisfied by +# a comment. What must hold is that `git tag` is UNREACHABLE unless the gate returned +# 0, so this guard EXTRACTS cut_tag() from the autopilot and RUNS it against stubs, +# once per gate outcome, and asserts on whether a tag was cut: +# gate rc 0 -> tag is cut +# gate rc 1 -> no tag, no publish (the milestone holds open items) +# gate rc 2 -> no tag (Unknown; never a silent pass) +# --self-test then removes the gate call to build a MUTANT and requires this guard to +# go RED on it. A guard that cannot fail on the defect it names is theater. +# +# check_tag_step_gated.sh judge scripts/release/autopilot.sh +# check_tag_step_gated.sh --self-test case table + the gate-removed mutant +set -uo pipefail +ROOT="$(cd "$(dirname "$0")/.." && pwd)" || exit 2 + +# SEC011: never let an empty or '/' value reach `rm -rf`. One validated helper, used +# by every call site in this file, rather than the check repeated at each one. +rmtree() { case "${1:-}" in ''|/) return 0 ;; *) [ -d "$1" ] && rm -rf -- "$1" ;; esac; return 0; } +SUBJECT="$ROOT/scripts/release/autopilot.sh" + +# run_cut_tag -- extract cut_tag(), run it with stubs, print a +# transcript (SAY/DIE/GIT-TAG/GIT-PUSH lines). Returns 2 if the function is missing. +run_cut_tag() { + local ap=$1 grc=$2 d fn + d=$(mktemp -d) || return 2 + fn=$(awk '/^cut_tag\(\) \{/,/^\}/' "$ap") + [ -n "$fn" ] || { rmtree "$d"; return 2; } + mkdir -p "$d/scripts" + printf '#!/usr/bin/env bash\nexit %s\n' "$grc" > "$d/scripts/check_milestone_cut.sh" + { + printf 'set -uo pipefail\n' + printf 'REPO_ROOT=%q\nLOG=%q\n' "$d" "$d/log" + printf 'say() { printf "SAY %%s\\n" "$*"; }\n' + printf 'die() { printf "DIE %%s\\n" "$*"; exit 1; }\n' + printf 'git() { printf "GIT-%%s %%s\\n" "$(printf %%s "$1" | tr "a-z" "A-Z")" "$*"; }\n' + printf '%s\n' "$fn" + printf 'cut_tag 0.0.0 v0.0.0 deadbeef\n' + } > "$d/harness.sh" + bash "$d/harness.sh" 2>&1 + cat "$d/log" 2>/dev/null + rmtree "$d" +} + +# judge -- 0 when every gate outcome behaves; 1 otherwise. Prints rows. +judge() { + local ap=$1 out bad=0 + if ! grep -q '^cut_tag() {' "$ap"; then + printf 'FAIL %s has no cut_tag() -- the tag path moved and this guard is judging nothing\n' "$ap" >&2 + return 2 + fi + out=$(run_cut_tag "$ap" 0) || true + if grep -q 'GIT-TAG' <<< "$out"; then printf 'ok gate rc=0 -> the tag is cut\n' + else printf 'FAIL gate rc=0 -> NO tag was cut\n%s\n' "$out" >&2; bad=1; fi + + out=$(run_cut_tag "$ap" 1) || true + if grep -q 'GIT-TAG' <<< "$out"; then + printf 'FAIL gate rc=1 (milestone holds open items) -> A TAG WAS CUT ANYWAY\n%s\n' "$out" >&2; bad=1 + else printf 'ok gate rc=1 -> no tag, no publish\n'; fi + + out=$(run_cut_tag "$ap" 2) || true + if grep -q 'GIT-TAG' <<< "$out"; then + printf 'FAIL gate rc=2 (Unknown) -> A TAG WAS CUT ANYWAY\n%s\n' "$out" >&2; bad=1 + else printf 'ok gate rc=2 (Unknown) -> no tag\n'; fi + return "$bad" +} + +# guard_tree.sh decides a guard "advertises --self-test" by looking for the literal +# substring `self-test` in its OWN `--help` output (guard_tree.sh:47, advertises_self_test). +# Without this handler the self-test below is discovered by nothing and runs nowhere -- +# a facility with a self-test and no caller. With it, guard_tree gives this guard TWO +# rows, `[self-test]` and `[run]`, inside the guard-tree job. +case "${1:-}" in -h|--help) sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; esac + +if [ "${1:-}" = "--self-test" ]; then + echo "=== the tag-gate guard must still turn RED (mutants) ===" + d=$(mktemp -d) || exit 2 + trap 'rmtree "${d:-}"' EXIT + bad=0 + ok() { printf 'ok %s\n' "$*"; } + nok() { printf 'FAIL %s\n' "$*" >&2; bad=1; } + + # M1: the gate call deleted -- the #3459 defect exactly, restored. + sed '/check_milestone_cut\.sh/d' "$SUBJECT" > "$d/m1.sh" + if judge "$d/m1.sh" > "$d/m1.out" 2>&1; then + nok "MUTANT 1 (gate call deleted) PASSED -- this guard cannot see its own defect" + else + ok "mutant 1: gate call deleted -> RED ($(grep -c '^FAIL' "$d/m1.out") failing row(s))" + fi + + # M2: the gate runs but its verdict is discarded (`|| true`) -- absence-as-consent. + sed 's#\(bash "$REPO_ROOT/scripts/check_milestone_cut.sh" "$v" >> "$LOG" 2>&1\) || rc=$?#\1 || true#' "$SUBJECT" > "$d/m2.sh" + if ! grep -q '|| true' "$d/m2.sh"; then + nok "MUTANT 2 could not be built -- the gate-call line did not match; this self-test is vacuous" + elif judge "$d/m2.sh" > "$d/m2.out" 2>&1; then + nok "MUTANT 2 (verdict discarded) PASSED -- a gate whose result is thrown away reads as gated" + else + ok "mutant 2: gate verdict discarded -> RED" + fi + + # M3: cut_tag() removed entirely -> ENV (2), never a pass. + awk '/^cut_tag\(\) \{/,/^\}/ {next} {print}' "$SUBJECT" > "$d/m3.sh" + judge "$d/m3.sh" > "$d/m3.out" 2>&1; rc=$? + [ "$rc" -eq 2 ] && ok "mutant 3: cut_tag() removed -> ENV rc=2, not a pass" \ + || nok "mutant 3: cut_tag() removed gave rc=$rc, wanted 2" + + # and the real subject must be GREEN, or a red subject masquerades as a killed mutant + if judge "$SUBJECT" > "$d/real.out" 2>&1; then + ok "the real subject is GREEN, so the REDs above are the mutants'" + else + nok "the real subject is itself RED -- the mutations prove nothing"; cat "$d/real.out" >&2 + fi + [ "$bad" -eq 0 ] && { echo "SELF-TEST PASSED"; exit 0; } + echo "SELF-TEST FAILED" >&2; exit 1 +fi + +echo "=== the tag step cannot be reached without the milestone gate (check_tag_step_gated.sh) ===" +judge "$SUBJECT"; rc=$? +[ "$rc" -eq 0 ] && echo "PASS" || echo "FAIL: the tag step is reachable without a clean milestone (rc=$rc)" >&2 +exit "$rc" diff --git a/scripts/release/autopilot.sh b/scripts/release/autopilot.sh index 9a89ba36c3..d1de1b4bc1 100755 --- a/scripts/release/autopilot.sh +++ b/scripts/release/autopilot.sh @@ -107,11 +107,30 @@ if run_step dogfood; then fi # 3. tag + release (binary-release.yml fires on release: published, from the TAG's workflow file) +# cut_tag -- PMAT-3459. The milestone gate lives INSIDE the +# function that tags, ahead of `git tag`, so the tag cannot be cut without it: there is +# no path through cut_tag() that reaches `git tag` with the gate unsatisfied. v0.68.1 +# was tagged 15:06:29Z by a copy of this script that read no milestone at all, six +# minutes after #3455 merged the gate (`grep -c check_milestone_cut autopilot.sh` = 0). +# Fail-closed on BOTH non-zero codes, and they are different failures: +# 1 = the milestone holds open item(s) -> no tag, no publish +# 2 = the gate could not judge (Unknown) -> no tag. Never a silent pass. +# scripts/check_tag_step_gated.sh runs this function against stubs and requires each +# of those three paths, plus a gate-call-removed MUTANT, to behave as stated. +cut_tag() { + local v=$1 t=$2 mc=$3 rc=0 + case "$rc" in + 0) say "MILESTONE-GATE $v clean at the cut (check_milestone_cut.sh rc=0)" ;; + 1) die "milestone $v still holds open item(s) -- no tag, no publish (check_milestone_cut.sh rc=1)" ;; + *) die "milestone $v could not be judged (check_milestone_cut.sh rc=$rc) -- no tag; Unknown is not a pass" ;; + esac + git tag -a "$t" -m "aprender $t" "$mc" >> "$LOG" 2>&1 || die "tag failed" + git push origin "$t" >> "$LOG" 2>&1 || die "tag push failed" +} if run_step tag; then git rev-parse -q --verify "refs/tags/$T" > /dev/null && die "tag $T already exists locally" [ -f "$AP/release_notes.md" ] || die "no $AP/release_notes.md (prepare_bump.sh writes it from CHANGELOG [$V])" - git tag -a "$T" -m "aprender $T" "$MC" >> "$LOG" 2>&1 || die "tag failed" - git push origin "$T" >> "$LOG" 2>&1 || die "tag push failed" + cut_tag "$V" "$T" "$MC" say "TAGGED $T at $MC" gh release create "$T" --repo $REPO --verify-tag --title "aprender $V" --notes-file "$AP/release_notes.md" >> "$LOG" 2>&1 || die "gh release create failed" say "RELEASED $(gh release view "$T" --repo $REPO --json url -q .url)"