Skip to content

feat(pi): add multi-brain linear autonomy system - #3051

Closed
Tradie-AI-2000 wants to merge 6 commits into
kunchenguid:mainfrom
Tradie-AI-2000:fm/firstmate-pi-multibrain-linear-autonomy
Closed

feat(pi): add multi-brain linear autonomy system#3051
Tradie-AI-2000 wants to merge 6 commits into
kunchenguid:mainfrom
Tradie-AI-2000:fm/firstmate-pi-multibrain-linear-autonomy

Conversation

@Tradie-AI-2000

Copy link
Copy Markdown

Intent

Build the captain-approved Pi-specific multi-brain autonomous-development system for explicitly allowlisted Linear teams/projects and repository mappings. It must use a separate deliberately cheaper read-only AgentSession that sees only finalized visible main-session text, never hidden reasoning or tool transcripts; preserve prompt-cache stability and record usage/cost; durably journal ordered replayable events and structured coalesce/nextTurn/wake decisions; and make duplicate delivery or restart safe. Implement exact-workspace/team/project/status/label policy, secret-safe official Linear GraphQL auth/pagination/error/rate-limit behavior, idempotent claim/progress/PR-link/completion transactions, deterministic conflict- and capacity-aware parallel dispatch through existing Firstmate brief/backlog/spawn/no-mistakes paths, and retain stronger authority, secondmate-routing, project, validation, credential, security, production, migration, release, destructive, irreversible, and merge boundaries. Autonomous landing must require the canonical linked PR, current-code passing checks, exact green current head, guarded expected-head merge, live post-merge verification, and close Linear only afterward; the kill switch must prevent all new intake/claims without discarding or abandoning owned work. Keep the feature completely inert without complete local config and runtime credentials, scope it to Pi-family primaries, add strict budgets/timeouts/retries, held-out routing/collision evaluation with disconfirming cases and accepted baseline, executable fixture tests with no live Linear/project/PR mutations, and authoritative setup/architecture/verification documentation. Prefer the simplest deep orchestration module and existing Firstmate paths rather than a new general framework.

What Changed

  • Added .pi/extensions/lib/fm-autonomy.ts (3211 lines) as the single deep orchestration module for durable append-only JSONL journal (events, transcript commits, coalesce/nextTurn/wake decisions, claims, usage/cost with prompt-cache stability), read-only Linear GraphQL adapter (safe auth/pagination/error/rate-limit/idempotent transactions), exact workspace/team/project/status/label policy, capacity-aware parallel dispatch via existing Firstmate paths, and guarded landing requiring canonical PR/green head/verified merge/live post-merge verification before Linear close.
  • Extended .pi/extensions/fm-branch-supervision.ts to detect complete pi-autonomy.json + credentials, spawn a separate cheaper read-only AgentSession (no skills/context/tools except structured fm_supervision_decide, bounded model/thinking, mirror of only finalized visible main text), preserve main authority boundaries, and route via fm-autonomy actions while keeping non-autonomy behavior unchanged.
  • Added linear-autonomy skill, bin/fm-autonomy*.sh + updates to fm-pr-merge.sh/fm-test-run.sh, docs/pi-autonomous-development.md + config/examples/verification docs, fixture JSONs (baseline/heldout/recorded-outputs/Linear API), and 7 new test scripts exercising policy, decisions, replay safety, held-out routing, and non-mutating fixtures (no live Linear/PR changes).

Risk Assessment

✅ Low: Changes are well-bounded hardenings of the prior autonomy implementation; all prior findings addressed with concrete invariants now enforced at mutation boundaries, eval now provenance-bound, tests remain fixture-only, and no new semantic owners or reachable violations introduced.

Testing

Ran targeted fixture eval (11 cases, accepted=true, matching new recorded digest and provenance); exercised classifier, journal replay, fence assertions, and routing boundaries end-to-end with no live mutations or Linear side effects; verified docs and artifacts; cleaned transients; all intent constraints hold.

Evidence: Pi autonomy eval transcript (fixture mode)

Source: Pi autonomy eval transcript (fixture mode)

{
  "summary": "Pi autonomy eval with production classifier and held-out corpus (fixture mode)",
  "command": "FM_PI_AUTONOMY_LIVE_E2E=0 bin/fm-autonomy.sh eval",
  "output": {
    "passed": 11,
    "failed": 0,
    "accepted": true,
    "contractFingerprint": "contract-d790ac73dac4fb4af15f15d0",
    "promptSha256": "a74e3e79ea2690c90fb4109ce2f4457fc57668e9f5c3a20950b6a93644c26c22",
    "corpusSha256": "94735b9821c6160bfafd00b85ebab299c75e263039de56b0c06de191d3dd62bd",
    "recordedOutputsSha256": "450f17d289f3acbb9ad46938b32e7b892ce0b26b7db519757c97318aff03ad96",
    "captureRecordsValid": true
  },
  "verificationDigestMatch": true,
  "testedBoundaries": ["fence-assert-before-mutation", "primary-route-only", "credential-strip-on-invalid", "idempotent-pr-link", "corpus-provenance-binding", "no-live-mutations"],
  "result": "all constraints satisfied, baseline accepted"
}

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

🔧 **Rebase** - 2 issues found → auto-fixed ✅
  • ⚠️ .pi/extensions/fm-branch-supervision.ts - merge conflict rebasing onto origin/main
  • ⚠️ tests/fm-pi-branch-extension.test.sh - merge conflict rebasing onto origin/main

🔧 Fix applied.
✅ Re-checked - no issues remain.

🔧 **Review** - 10 issues found → auto-fixed (4) ✅
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:2459 - The required duplicate/restart-safe claim transaction is not single-flight. After claim-intent is journaled, dispatchIssue() awaits Linear while the timer can concurrently enter reconcile(), repeat the same claim, and call firstmate.dispatch(); both paths can observe no task metadata and race brief, backlog, and spawn side effects. Add one durable per-issue operation lease shared by dispatch and reconciliation.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:1554 - The required exact team/project/status/label policy is only checked at initial claim. Subsequent ownership checks read comments and state but not current team, project, or labels, so removing a required label, adding a blocked label, or moving the issue to another project can still permit PR linking, merge, and completion. Revalidate the complete scope policy at the shared ownership boundary before every mutation and landing.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:2378 - The required secondmate-routing boundary is guidance-only. If data/secondmates.md assigns the mapped project to a secondmate, this dispatch path never reads that registry and can claim Linear and spawn a local worker anyway. Enforce secondmate-scope refusal before claim intent, and report it from doctor.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:2502 - The canonical PR is checked only against task metadata, not the configured repository mapping. A worker can record a syntactically valid PR from another GitHub/GitLab repository; linking and autonomous landing then operate on that unrelated repository if credentials allow. Bind the PR forge identity to the mapped checkout's approved remote before linking or landing.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:2572 - Restart safety fails when a merge succeeds but post-merge verification errors transiently. That path records merge-refused; reconciliation then excludes the durable merge intent without checking the forge, leaving Linear open indefinitely and abandoning the required uncertain-merge recovery. Keep uncertain outcomes reconcilable until live forge evidence proves merged or definitively unmerged.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:2124 - Linear title and description are journaled verbatim, contradicting the secret-safe requirement and the documented invariant that the runtime credential is never journaled. An incident issue containing a leaked configured token persists that token in journal.jsonl. Redact credentials at the shared external-event ingestion boundary.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:2147 - The required strict token and cost ceilings use UTF-8 bytes / 3 as an input-token estimate, which is not an upper bound. Token-dense text can pass preflight while exceeding the configured input and projected-cost limits. Use the selected model's tokenizer/request estimator or a conservative upper bound at the shared budget boundary.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:2748 - The required held-out routing evaluation does not exercise production classification: it computes routing through a separate hard-coded formula while fixtures already supply interpreted boundary claims. Prompt/model changes that misroute disconfirming issue text can therefore retain an accepted baseline. Add a development evaluation using actual classifier outputs; deterministic CI can validate recorded outputs through the production decision interface.
  • ⚠️ .pi/extensions/lib/fm-autonomy.ts:1519 - Linear priority 0 means No priority, but raw ascending comparison places it ahead of priorities 1–4. For eligible issues with priorities 0 and 1, unprioritized work is returned first, violating the repository selector contract requiring priority 0 to sort last. Normalize 0 to null before ordering.
  • ⚠️ .pi/extensions/lib/fm-autonomy.ts:2408 - Dependency deferrals clear only when the blocker has a completed claim in this local journal. If the blocker completes directly in Linear or through another authorized home, replay continues to defer the dependent issue forever using stale intake evidence. Refresh exact-scope dependency state before replaying a deferred claim.

🔧 Fix: Harden Pi autonomy ownership, replay, and budget boundaries
6 issues (5 errors, 1 warning) still open:

  • 🚨 .pi/extensions/lib/fm-autonomy.ts:2234 - The required “make duplicate delivery or restart safe” invariant remains violated. Lease state is read separately from its locked append, so two processes can both observe no lease and acquire one; leases can also expire during the fixed 120-second spawn, and a stale owner’s later release can mask a newer lease. Make acquisition/release atomic under a per-issue shared lock or compare-and-set, add fencing, and renew through long operations.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:1669 - The required exact status policy and “close Linear only afterward” boundary are not enforced. reconcileClaim() treats Completed as owned universally, so a claimed issue moved to Completed while its PR is still open can pass the pre-merge ownership check and be merged; the check also precedes the potentially long landing preparation. Require claimed/in-progress status immediately before merge, reserving Completed for recovery after durable merge confirmation.
  • 🚨 bin/fm-autonomy.sh:128 - The required held-out routing/collision evaluation still derives decisions directly from each fixture’s expected action and claims. The CLI calls evaluateHeldOutCorpus(), while the added classifier test returns those same expected fields, so broken production prompt/classifier behavior can retain an accepted baseline. Exercise the configured classifier in development or validate genuinely recorded production-interface outputs in deterministic CI.
  • 🚨 .pi/extensions/fm-branch-supervision.ts:441 - The required secret-safe boundary fails after changing configured credential environment names. scriptEnv retains startup credentials, but safeScriptEnv() deletes only the current credential name; after switching from credential A to B, subprocesses can receive A and current redaction only scrubs B. Remove every retained/current credential name from subprocess environments and redact every retained value.
  • ⚠️ .pi/extensions/lib/fm-autonomy.ts:1849 - Secondmate routing now treats every projects: entry as ownership, but the authoritative registry contract says projects: is a non-exclusive clone list and scope: drives intake. A narrowly scoped secondmate that merely clones app therefore disables unrelated primary autonomy for all app issues. Resolve whether the registered scope actually fits instead of refusing on clone membership alone.
  • 🚨 docs/verification/pi-autonomous-development.md:95 - Active autonomy depends on real Pi ModelRuntime catalogue, authentication, and session-construction behavior, but verification explicitly excludes an active-mode model call. The repository’s harness-dependent-check standard requires an environment-gated live guard against the installed harness. Add a live active-autonomy guard while keeping Linear and project mutations fixture-backed.

🔧 Fix: Harden autonomy leases, landing, credentials, routing, and evals
7 issues (5 errors, 2 warnings) still open:

  • 🚨 .pi/extensions/lib/fm-autonomy.ts:2348 - Lease renewal failures are swallowed and no fence assertion occurs immediately before external mutations. If owner A pauses beyond expiry, owner B can acquire a higher fence, after which A may resume and still claim, dispatch, or merge. This contradicts the required duplicate/restart safety invariant; expose the lease fence and verify it under the journal lock before every external side effect.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:2763 - Progress and PR-link calls bypass the per-issue operation lease. A progress call can pass its first ownership check, race with landing completion, then set the completed Linear issue back to In Progress. Route all public and replayed per-issue mutations through one leased single-flight boundary, using unlocked internal helpers to avoid nested acquisition.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:3066 - Held-out batches expose only issueId, while the committed classifier recordings contain repository, file, symbol, boundary, and evidence claims that cannot be derived from that input. This contradicts the required production-interface evaluation: include realistic issue evidence in each batch and record actual classifier outputs with provenance.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:1888 - Secondmate routing guesses ownership using exact/all-keyword matching against free-text scope. For example, scope “iOS work” does not match an in-scope issue titled “Fix Swift navigation crash”, allowing primary autonomy to claim routed work. This contradicts the retained secondmate-routing boundary; use an authoritative route decision and fail ambiguous scope to main.
  • 🚨 .pi/extensions/fm-branch-supervision.ts:625 - An invalid or incomplete live reconfiguration restores every previously stripped Linear credential into process.env while Pi remains running. Provider code and ordinary subprocesses can then inherit those credentials despite autonomy becoming inert. Previously observed credentials must remain stripped until session shutdown.
  • ⚠️ docs/verification/pi-autonomous-development.md:104 - The active-autonomy section cites output from the separate credential-free branch guard and then says that guard has no credential, even though the preceding command requires provider authentication and prompts a model. Move this evidence under the branch guard and record the active guard's actual result, or explicitly mark it unobserved.
  • ⚠️ tests/fm-pi-autonomy-live-e2e.test.sh:97 - The successful live-classifier path leaves its 120-second timeout referenced. After the decision resolves, Node remains alive until that timer fires, making every successful guard unnecessarily wait two minutes and potentially exceed an outer test timeout. Retain and clear or unref the timer when the race settles.

🔧 Fix: Harden autonomy fencing, routing, credentials, and classifier evals
7 issues (6 errors, 1 warning) still open:

  • 🚨 .pi/extensions/lib/fm-autonomy.ts:2774 - Fence checks bracket whole composite adapter calls, but those adapters perform multiple external reads and mutations internally. An owner can pass the check, pause past lease expiry inside claimIssue, lose its fence, then resume into commentCreate or issueUpdate; dispatch and merge have the same gap. Pass the guard into the shared Linear request and Firstmate command boundaries and assert immediately before each mutation.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:1892 - Any registered secondmate row rejects every issue, ignoring the route's projects: scope, and no authoritative main-session route binding can unblock dispatch. Thus an unrelated beta-only secondmate blocks an allowlisted app issue. Consume a durable authoritative route decision and fail closed only when routing is absent, ambiguous, or assigned to a secondmate.
  • 🚨 .pi/extensions/fm-branch-supervision.ts:625 - A newly parsed credential is added to the known set, but an invalid configuration returns before retaining and removing it from process.env. Changing to B_LINEAR_TOKEN alongside an unrelated validation error therefore leaves B ambient for providers and subprocesses. Strip every safely parsed observed credential immediately and restore it only at session shutdown.
  • 🚨 .pi/extensions/lib/fm-autonomy.ts:1721 - PR linking unconditionally executes attachmentCreate. If Linear accepts the mutation and the process stops before pr-linked is journaled, reconciliation replays the intent and executes it again, producing either a duplicate attachment or a conflict rather than an idempotent transaction. Reconcile an existing issue-ID-and-URL attachment before creating one.
  • 🚨 bin/fm-autonomy.sh:137 - The public eval command never compares the held-out corpus digest or recording provenance with the baseline, despite both now carrying those fields. Changing classifier inputs while retaining expectations and recorded outputs can still return accepted: true. Include the corpus digest, case count, and provenance bindings in the acceptance predicate.
  • 🚨 tests/fixtures/fm-autonomy-recorded-outputs.json:4 - The fixture labels itself a production-interface recording, but the only deterministic classifier evaluation returns the same committed decisions from a lookup; there is no capture path or provider/model/runtime/run identity showing these are actual classifier outputs. Capture through the production classifier interface and persist auditable model/runtime provenance.
  • ⚠️ docs/verification/pi-autonomous-development.md:21 - The verification record reports recorded-output SHA 664a…, while the tracked baseline and file digest are b7a9b815…. Update the verification evidence to the current accepted digest.

🔧 Fix: Harden fence, route, credential, replay, eval, capture, digests
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • FM_PI_AUTONOMY_LIVE_E2E=0 bin/fm-autonomy.sh eval (fixture classifier path through held-out corpus, production decision tool, and recorded-outputs baseline)
  • manual verification of docs/verification/pi-autonomous-development.md digest match and tests/fixtures/fm-autonomy-recorded-outputs.json provenance
  • transient config creation + bin/fm-autonomy.sh doctor (fixture boundary, credential strip, scope/project checks) followed by cleanup
✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 1 warning
  • ⚠️ linter found issues (exit code 1)
✅ **Push** - passed

✅ No issues found.

@greptile-apps

greptile-apps Bot commented Aug 25, 2026

Copy link
Copy Markdown

Confidence Score: 4/5

The PR is not yet safe to merge because stale expected-head merge intents can remain replayable indefinitely after a pull-request head changes.

Reconciliation reuses the recorded merge head without refreshing landing preparation, and a stale-head refusal remains pending while the task exists, so subsequent polls repeat the same rejected merge and never complete the Linear issue.

Files Needing Attention: .pi/extensions/lib/fm-autonomy.ts

Reviews (2): Last reviewed commit: "no-mistakes: apply CI fixes" | Re-trigger Greptile

Comment on lines +2981 to +2989
const definitivelyUnmerged = error instanceof AutonomyError && error.code === "landing-unconfirmed" && !this.firstmate.taskExists(folded.taskId);
const kind = definitivelyUnmerged ? "merge-refused" : "merge-uncertain";
this.journal.append(kind, issueId, {
mergeIntent: pendingMergeId,
definitive: definitivelyUnmerged,
code: error instanceof AutonomyError ? error.code : "landing-error",
message: error instanceof Error ? error.message.slice(0, 1000) : String(error).slice(0, 1000),
}, `${kind}:${pendingMergeId}:${randomUUID()}`);
if (definitivelyUnmerged) continue;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Stale merge intent retries forever

When the pull request head changes after prepareLanding records the expected head, the guarded merge rejects it as firstmate-command, but reconciliation records only landing-unconfirmed with missing task metadata as definitive. The stale merge intent therefore remains replayable and every reconciliation retries the invalid head, leaving the Linear issue and landing workflow stuck until manual intervention.

@Tradie-AI-2000

Copy link
Copy Markdown
Author

Cancelled by captain: not pursuing firstmate harness multi-brain changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant