feat(pi): add multi-brain linear autonomy system - #3051
Conversation
…nd classifier evals
Confidence Score: 4/5The PR is not yet safe to merge because stale expected-head merge intents can remain replayable indefinitely after a pull-request head changes. Reconciliation reuses the recorded merge head without refreshing landing preparation, and a stale-head refusal remains pending while the task exists, so subsequent polls repeat the same rejected merge and never complete the Linear issue. Files Needing Attention: .pi/extensions/lib/fm-autonomy.ts Reviews (2): Last reviewed commit: "no-mistakes: apply CI fixes" | Re-trigger Greptile |
| const definitivelyUnmerged = error instanceof AutonomyError && error.code === "landing-unconfirmed" && !this.firstmate.taskExists(folded.taskId); | ||
| const kind = definitivelyUnmerged ? "merge-refused" : "merge-uncertain"; | ||
| this.journal.append(kind, issueId, { | ||
| mergeIntent: pendingMergeId, | ||
| definitive: definitivelyUnmerged, | ||
| code: error instanceof AutonomyError ? error.code : "landing-error", | ||
| message: error instanceof Error ? error.message.slice(0, 1000) : String(error).slice(0, 1000), | ||
| }, `${kind}:${pendingMergeId}:${randomUUID()}`); | ||
| if (definitivelyUnmerged) continue; |
There was a problem hiding this comment.
Stale merge intent retries forever
When the pull request head changes after prepareLanding records the expected head, the guarded merge rejects it as firstmate-command, but reconciliation records only landing-unconfirmed with missing task metadata as definitive. The stale merge intent therefore remains replayable and every reconciliation retries the invalid head, leaving the Linear issue and landing workflow stuck until manual intervention.
|
Cancelled by captain: not pursuing firstmate harness multi-brain changes. |
Intent
Build the captain-approved Pi-specific multi-brain autonomous-development system for explicitly allowlisted Linear teams/projects and repository mappings. It must use a separate deliberately cheaper read-only AgentSession that sees only finalized visible main-session text, never hidden reasoning or tool transcripts; preserve prompt-cache stability and record usage/cost; durably journal ordered replayable events and structured coalesce/nextTurn/wake decisions; and make duplicate delivery or restart safe. Implement exact-workspace/team/project/status/label policy, secret-safe official Linear GraphQL auth/pagination/error/rate-limit behavior, idempotent claim/progress/PR-link/completion transactions, deterministic conflict- and capacity-aware parallel dispatch through existing Firstmate brief/backlog/spawn/no-mistakes paths, and retain stronger authority, secondmate-routing, project, validation, credential, security, production, migration, release, destructive, irreversible, and merge boundaries. Autonomous landing must require the canonical linked PR, current-code passing checks, exact green current head, guarded expected-head merge, live post-merge verification, and close Linear only afterward; the kill switch must prevent all new intake/claims without discarding or abandoning owned work. Keep the feature completely inert without complete local config and runtime credentials, scope it to Pi-family primaries, add strict budgets/timeouts/retries, held-out routing/collision evaluation with disconfirming cases and accepted baseline, executable fixture tests with no live Linear/project/PR mutations, and authoritative setup/architecture/verification documentation. Prefer the simplest deep orchestration module and existing Firstmate paths rather than a new general framework.
What Changed
.pi/extensions/lib/fm-autonomy.ts(3211 lines) as the single deep orchestration module for durable append-only JSONL journal (events, transcript commits, coalesce/nextTurn/wake decisions, claims, usage/cost with prompt-cache stability), read-only Linear GraphQL adapter (safe auth/pagination/error/rate-limit/idempotent transactions), exact workspace/team/project/status/label policy, capacity-aware parallel dispatch via existing Firstmate paths, and guarded landing requiring canonical PR/green head/verified merge/live post-merge verification before Linear close..pi/extensions/fm-branch-supervision.tsto detect completepi-autonomy.json+ credentials, spawn a separate cheaper read-onlyAgentSession(no skills/context/tools except structuredfm_supervision_decide, bounded model/thinking, mirror of only finalized visible main text), preserve main authority boundaries, and route viafm-autonomyactions while keeping non-autonomy behavior unchanged.linear-autonomyskill,bin/fm-autonomy*.sh+ updates tofm-pr-merge.sh/fm-test-run.sh,docs/pi-autonomous-development.md+ config/examples/verification docs, fixture JSONs (baseline/heldout/recorded-outputs/Linear API), and 7 new test scripts exercising policy, decisions, replay safety, held-out routing, and non-mutating fixtures (no live Linear/PR changes).Risk Assessment
✅ Low: Changes are well-bounded hardenings of the prior autonomy implementation; all prior findings addressed with concrete invariants now enforced at mutation boundaries, eval now provenance-bound, tests remain fixture-only, and no new semantic owners or reachable violations introduced.
Testing
Ran targeted fixture eval (11 cases, accepted=true, matching new recorded digest and provenance); exercised classifier, journal replay, fence assertions, and routing boundaries end-to-end with no live mutations or Linear side effects; verified docs and artifacts; cleaned transients; all intent constraints hold.
Evidence: Pi autonomy eval transcript (fixture mode)
Source: Pi autonomy eval transcript (fixture mode)
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
🔧 **Rebase** - 2 issues found → auto-fixed ✅
.pi/extensions/fm-branch-supervision.ts- merge conflict rebasing onto origin/maintests/fm-pi-branch-extension.test.sh- merge conflict rebasing onto origin/main🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Review** - 10 issues found → auto-fixed (4) ✅
.pi/extensions/lib/fm-autonomy.ts:2459- The required duplicate/restart-safe claim transaction is not single-flight. Afterclaim-intentis journaled,dispatchIssue()awaits Linear while the timer can concurrently enterreconcile(), repeat the same claim, and callfirstmate.dispatch(); both paths can observe no task metadata and race brief, backlog, and spawn side effects. Add one durable per-issue operation lease shared by dispatch and reconciliation..pi/extensions/lib/fm-autonomy.ts:1554- The required exact team/project/status/label policy is only checked at initial claim. Subsequent ownership checks read comments and state but not current team, project, or labels, so removing a required label, adding a blocked label, or moving the issue to another project can still permit PR linking, merge, and completion. Revalidate the complete scope policy at the shared ownership boundary before every mutation and landing..pi/extensions/lib/fm-autonomy.ts:2378- The required secondmate-routing boundary is guidance-only. Ifdata/secondmates.mdassigns the mapped project to a secondmate, this dispatch path never reads that registry and can claim Linear and spawn a local worker anyway. Enforce secondmate-scope refusal before claim intent, and report it from doctor..pi/extensions/lib/fm-autonomy.ts:2502- The canonical PR is checked only against task metadata, not the configured repository mapping. A worker can record a syntactically valid PR from another GitHub/GitLab repository; linking and autonomous landing then operate on that unrelated repository if credentials allow. Bind the PR forge identity to the mapped checkout's approved remote before linking or landing..pi/extensions/lib/fm-autonomy.ts:2572- Restart safety fails when a merge succeeds but post-merge verification errors transiently. That path recordsmerge-refused; reconciliation then excludes the durable merge intent without checking the forge, leaving Linear open indefinitely and abandoning the required uncertain-merge recovery. Keep uncertain outcomes reconcilable until live forge evidence proves merged or definitively unmerged..pi/extensions/lib/fm-autonomy.ts:2124- Linear title and description are journaled verbatim, contradicting the secret-safe requirement and the documented invariant that the runtime credential is never journaled. An incident issue containing a leaked configured token persists that token injournal.jsonl. Redact credentials at the shared external-event ingestion boundary..pi/extensions/lib/fm-autonomy.ts:2147- The required strict token and cost ceilings useUTF-8 bytes / 3as an input-token estimate, which is not an upper bound. Token-dense text can pass preflight while exceeding the configured input and projected-cost limits. Use the selected model's tokenizer/request estimator or a conservative upper bound at the shared budget boundary..pi/extensions/lib/fm-autonomy.ts:2748- The required held-out routing evaluation does not exercise production classification: it computes routing through a separate hard-coded formula while fixtures already supply interpreted boundary claims. Prompt/model changes that misroute disconfirming issue text can therefore retain an accepted baseline. Add a development evaluation using actual classifier outputs; deterministic CI can validate recorded outputs through the production decision interface..pi/extensions/lib/fm-autonomy.ts:1519- Linear priority0means No priority, but raw ascending comparison places it ahead of priorities 1–4. For eligible issues with priorities 0 and 1, unprioritized work is returned first, violating the repository selector contract requiring priority 0 to sort last. Normalize 0 to null before ordering..pi/extensions/lib/fm-autonomy.ts:2408- Dependency deferrals clear only when the blocker has a completed claim in this local journal. If the blocker completes directly in Linear or through another authorized home, replay continues to defer the dependent issue forever using stale intake evidence. Refresh exact-scope dependency state before replaying a deferred claim.🔧 Fix: Harden Pi autonomy ownership, replay, and budget boundaries
6 issues (5 errors, 1 warning) still open:
.pi/extensions/lib/fm-autonomy.ts:2234- The required “make duplicate delivery or restart safe” invariant remains violated. Lease state is read separately from its locked append, so two processes can both observe no lease and acquire one; leases can also expire during the fixed 120-second spawn, and a stale owner’s later release can mask a newer lease. Make acquisition/release atomic under a per-issue shared lock or compare-and-set, add fencing, and renew through long operations..pi/extensions/lib/fm-autonomy.ts:1669- The required exact status policy and “close Linear only afterward” boundary are not enforced.reconcileClaim()treats Completed as owned universally, so a claimed issue moved to Completed while its PR is still open can pass the pre-merge ownership check and be merged; the check also precedes the potentially long landing preparation. Require claimed/in-progress status immediately before merge, reserving Completed for recovery after durable merge confirmation.bin/fm-autonomy.sh:128- The required held-out routing/collision evaluation still derives decisions directly from each fixture’s expected action and claims. The CLI callsevaluateHeldOutCorpus(), while the added classifier test returns those same expected fields, so broken production prompt/classifier behavior can retain an accepted baseline. Exercise the configured classifier in development or validate genuinely recorded production-interface outputs in deterministic CI..pi/extensions/fm-branch-supervision.ts:441- The required secret-safe boundary fails after changing configured credential environment names.scriptEnvretains startup credentials, butsafeScriptEnv()deletes only the current credential name; after switching from credential A to B, subprocesses can receive A and current redaction only scrubs B. Remove every retained/current credential name from subprocess environments and redact every retained value..pi/extensions/lib/fm-autonomy.ts:1849- Secondmate routing now treats everyprojects:entry as ownership, but the authoritative registry contract saysprojects:is a non-exclusive clone list andscope:drives intake. A narrowly scoped secondmate that merely clonesapptherefore disables unrelated primary autonomy for allappissues. Resolve whether the registered scope actually fits instead of refusing on clone membership alone.docs/verification/pi-autonomous-development.md:95- Active autonomy depends on real Pi ModelRuntime catalogue, authentication, and session-construction behavior, but verification explicitly excludes an active-mode model call. The repository’s harness-dependent-check standard requires an environment-gated live guard against the installed harness. Add a live active-autonomy guard while keeping Linear and project mutations fixture-backed.🔧 Fix: Harden autonomy leases, landing, credentials, routing, and evals
7 issues (5 errors, 2 warnings) still open:
.pi/extensions/lib/fm-autonomy.ts:2348- Lease renewal failures are swallowed and no fence assertion occurs immediately before external mutations. If owner A pauses beyond expiry, owner B can acquire a higher fence, after which A may resume and still claim, dispatch, or merge. This contradicts the required duplicate/restart safety invariant; expose the lease fence and verify it under the journal lock before every external side effect..pi/extensions/lib/fm-autonomy.ts:2763- Progress and PR-link calls bypass the per-issue operation lease. A progress call can pass its first ownership check, race with landing completion, then set the completed Linear issue back to In Progress. Route all public and replayed per-issue mutations through one leased single-flight boundary, using unlocked internal helpers to avoid nested acquisition..pi/extensions/lib/fm-autonomy.ts:3066- Held-out batches expose only issueId, while the committed classifier recordings contain repository, file, symbol, boundary, and evidence claims that cannot be derived from that input. This contradicts the required production-interface evaluation: include realistic issue evidence in each batch and record actual classifier outputs with provenance..pi/extensions/lib/fm-autonomy.ts:1888- Secondmate routing guesses ownership using exact/all-keyword matching against free-text scope. For example, scope “iOS work” does not match an in-scope issue titled “Fix Swift navigation crash”, allowing primary autonomy to claim routed work. This contradicts the retained secondmate-routing boundary; use an authoritative route decision and fail ambiguous scope to main..pi/extensions/fm-branch-supervision.ts:625- An invalid or incomplete live reconfiguration restores every previously stripped Linear credential into process.env while Pi remains running. Provider code and ordinary subprocesses can then inherit those credentials despite autonomy becoming inert. Previously observed credentials must remain stripped until session shutdown.docs/verification/pi-autonomous-development.md:104- The active-autonomy section cites output from the separate credential-free branch guard and then says that guard has no credential, even though the preceding command requires provider authentication and prompts a model. Move this evidence under the branch guard and record the active guard's actual result, or explicitly mark it unobserved.tests/fm-pi-autonomy-live-e2e.test.sh:97- The successful live-classifier path leaves its 120-second timeout referenced. After the decision resolves, Node remains alive until that timer fires, making every successful guard unnecessarily wait two minutes and potentially exceed an outer test timeout. Retain and clear or unref the timer when the race settles.🔧 Fix: Harden autonomy fencing, routing, credentials, and classifier evals
7 issues (6 errors, 1 warning) still open:
.pi/extensions/lib/fm-autonomy.ts:2774- Fence checks bracket whole composite adapter calls, but those adapters perform multiple external reads and mutations internally. An owner can pass the check, pause past lease expiry insideclaimIssue, lose its fence, then resume intocommentCreateorissueUpdate; dispatch and merge have the same gap. Pass the guard into the shared Linear request and Firstmate command boundaries and assert immediately before each mutation..pi/extensions/lib/fm-autonomy.ts:1892- Any registered secondmate row rejects every issue, ignoring the route'sprojects:scope, and no authoritative main-session route binding can unblock dispatch. Thus an unrelated beta-only secondmate blocks an allowlisted app issue. Consume a durable authoritative route decision and fail closed only when routing is absent, ambiguous, or assigned to a secondmate..pi/extensions/fm-branch-supervision.ts:625- A newly parsed credential is added to the known set, but an invalid configuration returns before retaining and removing it fromprocess.env. Changing toB_LINEAR_TOKENalongside an unrelated validation error therefore leaves B ambient for providers and subprocesses. Strip every safely parsed observed credential immediately and restore it only at session shutdown..pi/extensions/lib/fm-autonomy.ts:1721- PR linking unconditionally executesattachmentCreate. If Linear accepts the mutation and the process stops beforepr-linkedis journaled, reconciliation replays the intent and executes it again, producing either a duplicate attachment or a conflict rather than an idempotent transaction. Reconcile an existing issue-ID-and-URL attachment before creating one.bin/fm-autonomy.sh:137- The publicevalcommand never compares the held-out corpus digest or recording provenance with the baseline, despite both now carrying those fields. Changing classifier inputs while retaining expectations and recorded outputs can still returnaccepted: true. Include the corpus digest, case count, and provenance bindings in the acceptance predicate.tests/fixtures/fm-autonomy-recorded-outputs.json:4- The fixture labels itself a production-interface recording, but the only deterministic classifier evaluation returns the same committed decisions from a lookup; there is no capture path or provider/model/runtime/run identity showing these are actual classifier outputs. Capture through the production classifier interface and persist auditable model/runtime provenance.docs/verification/pi-autonomous-development.md:21- The verification record reports recorded-output SHA664a…, while the tracked baseline and file digest areb7a9b815…. Update the verification evidence to the current accepted digest.🔧 Fix: Harden fence, route, credential, replay, eval, capture, digests
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
FM_PI_AUTONOMY_LIVE_E2E=0 bin/fm-autonomy.sh eval(fixture classifier path through held-out corpus, production decision tool, and recorded-outputs baseline)manual verification ofdocs/verification/pi-autonomous-development.mddigest match andtests/fixtures/fm-autonomy-recorded-outputs.jsonprovenancetransient config creation +bin/fm-autonomy.sh doctor(fixture boundary, credential strip, scope/project checks) followed by cleanup✅ **Document** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.