Skip to content

[bamboo] feat: unify SubAgent as a distributed persistent Actor runtime #791

Description

@bigduu

Summary

Create one authoritative, location-transparent SubAgent model for Bamboo.

A SubAgent is a durable logical ActorSession with its own transcript, parent/root/Project identity, inbox, lifecycle, policy, diagnostics, and stable ActorId. A local process, warm worker, Docker container, SSH host, remote resident worker, or scheduled pool slot is only a WorkerHost used by one ActorActivation. It is never the agent identity.

Every placement, including local execution, uses the broker transport. The LLM sees one compact SubAgent tool and never sees deploy_agent, ask_agent, broker mailbox ids, worker ids, container ids, endpoints, or scheduling mechanics.

This is an umbrella architecture and acceptance issue. It coordinates rather than replaces the scoped implementation issues listed under Related work.

Motivation and confirmed current split

Bamboo currently exposes two semantic models even though normal local SubAgent execution already uses the broker:

  1. Session-backed SubAgent children:

    • create a durable child Session first;
    • preserve parent_session_id, root_session_id, Project, spawn_depth, transcript, and lifecycle;
    • execute by sending a Run to an actor worker over the broker;
    • write the final answer back to Session storage;
    • can be observed through Bamboo session history and agent.{SessionId} streams.
  2. Broker-deployed workers:

    • deploy_agent returns a physical worker/mailbox id;
    • DeployedRegistry is scoped to server memory;
    • ask_agent addresses that mailbox directly;
    • accumulated context is worker-local rather than an authoritative Bamboo Session;
    • restart recovery, nested ownership, policy, history, diagnostics, and Lotus Next visibility are incomplete.

The target is not to add a second Session implementation inside broker workers. The target is to make every broker execution belong to the one control-plane ActorSession model.

Non-negotiable invariants

  • Session is the durable actor truth; a worker is replaceable capacity.
  • ActorId is stable across activation, restart, passivation, retry, migration, and placement changes.
  • Every owned actor is persisted before activation or event publication.
  • Local and remote execution use the same broker protocol and activation contract.
  • The LLM sees exactly one SubAgent tool surface.
  • Parent/root/Project/depth and effective policy come from the authoritative runtime, never from worker claims.
  • All messages and parent requests are addressed by logical Session id.
  • All undecided child permission requests and blocking questions are delegated to the direct parent Session.
  • No permission, question, delivery, wakeup, or parent/child wait can block indefinitely.
  • The browser uses Bamboo's authenticated gateway and never connects directly to the broker or a worker.
  • High-frequency delivery events are not automatically admitted into LLM history.
  • Capability authority only narrows down the actor tree; a child can never gain authority its ancestors do not possess.

Target architecture

Root/Parent ActorSession
        |
        | one SubAgent facade
        v
ActorRuntime / ActorDirectory ---------------- Session store + SessionInbox
        |                                               |
        | PlacementRequest                              | history/diagnostics
        v                                               |
PlacementScheduler ---- HostRegistry                    |
        |                                               |
        | ActorActivation + lease                       |
        v                                               |
Broker <--------------------> Host Agent / WorkerHost ---+
                               |
                               +-- local process
                               +-- Docker/container
                               +-- SSH/remote resident
                               +-- schedulable pool
                               +-- future orchestrator
                               |
                               +-- EnvironmentLease
                                   workspace/tools/secrets/network

ActorSession

The durable logical identity:

  • actor_id equal to Session.id;
  • parent_session_id, root_session_id, Project identity, and spawn_depth;
  • transcript/history and durable inbox;
  • role/profile and effective policy revision;
  • lifecycle and current state;
  • current activation attempt/version;
  • placement intent, not physical endpoint;
  • diagnostic and audit state.

All SubAgent Sessions remain inspectable after completion. Physical oneshot/resident differences become activation temperature: Active, Idle, Cold, Suspended, Retired.

ActorActivation

One bounded execution attempt:

  • actor_id;
  • activation_id, monotonically increasing attempt, and run_id;
  • worker/placement lease and lease epoch;
  • inbox generation/wakeup epoch;
  • start/finish/heartbeat timestamps;
  • status, checkpoint revision, error, retry reason;
  • permission/question correlation scope;
  • cancellation and stale-frame fencing.

Only one activation may mutate an ActorSession transcript at a time.

WorkerHost and HostRegistry

A WorkerHost advertises capacity rather than identity:

  • trusted host id and health lease;
  • OS/architecture/runtime/container support;
  • GPU/CPU/memory and load;
  • available workspace/capability labels;
  • network/trust zone;
  • supported executor/provider capabilities.

"Any host" means any registered and authorized WorkerHost. It must not mean an arbitrary unvalidated IP or shell command supplied by a model.

A host or warm worker may serve multiple ActorSessions sequentially, but must scrub transcript, events, approvals, cancellation state, workspace, and secrets between activations.

PlacementScheduler

Support two semantic modes without expanding the LLM schema:

  • pinned: user/session policy requests a specific trusted host, pool, or locality;
  • auto: the scheduler chooses from eligible capacity.

Scheduling first filters hard constraints, then scores soft preferences:

  1. Project/data/security boundary;
  2. required workspace, OS, GPU, tools, credentials, and network zone;
  3. workspace/data affinity and transfer cost;
  4. local/remote preference;
  5. load, latency, cost, and stickiness.

A task requiring current uncommitted files, a local screen, or host-bound tools should remain local unless an explicit environment bridge exists. A portable Git snapshot, research task, build, or GPU task may run remotely.

Fallback behavior is explicit. A pinned or sensitive task must not silently cross a host/data boundary when its preferred placement is unavailable.

One LLM-facing SubAgent facade

Replace the current multi-tool/multi-parameter mental model with one compact tool:

{
  "intent": "chat | inspect | control",
  "target": "optional ActorId",
  "message": "natural-language request",
  "reply_to": "optional ParentRequest id"
}

intent defaults to chat.

Semantics

  • chat with no target: create a durable child ActorSession and send its first task.
  • chat with a target: continue the same child Session.
  • inspect with no target: return the owned descendant tree and health summary.
  • inspect with a target: return authoritative status, diagnostics, and requested history slices.
  • control with a target: apply a bounded lifecycle command such as cancel current activation, retry, reassign, or retire.
  • reply_to: resolve a pending permission or clarification request through the same chat facade.

Runtime behavior hidden from the model:

  • create/update/run/send_message/ask_agent converge into chat;
  • list/get/history/status converge into inspect;
  • wait is owned by the runtime and completion coordinator;
  • cancel/retry/stop converge into control;
  • delete remains a user/admin operation by default;
  • deploy_agent becomes internal capacity management or a compatibility adapter;
  • model selection, lifecycle temperature, workspace, host, container, and scheduling parameters move to profile, policy, configuration, and runtime inference;
  • list_models is not part of SubAgent orchestration.

The facade returns ActorId, logical state, response/observation, and actionable diagnostics. It never returns a broker token or physical mailbox address.

EnvironmentLease and local-environment access

Remote actors cannot magically read a path on the parent's host. Every activation receives an explicit EnvironmentLease using one or more modes:

  1. Host-local workspace:

    • schedule onto the host that owns the workspace;
    • use an isolated worktree/branch lease for writes.
  2. Scoped container mount:

    • mount only the selected workspace/artifact paths;
    • never mount all of ~/.bamboo or unrelated credentials by default.
  3. Remote snapshot:

    • materialize a Git SHA plus optional uncommitted patch/content snapshot;
    • return changes as a patch, branch, or artifact.
  4. Capability proxy:

    • execute explicitly allowlisted host-bound tools on the owning host;
    • existing MCP proxy behavior is a starting point;
    • default to read-only and audit every proxied action.

Large logs, patches, binaries, screenshots, and build outputs belong in a content-addressed Artifact Store. Broker and SessionInbox messages carry references and hashes, not unbounded file payloads.

Credentials use secret-free capability references and are resolved only on the selected trusted host. They never enter Session history, model-visible diagnostics, argv, broker ids, or browser state.

Nested delegation

Every ActorSession receives the same internal runtime capability to:

  • spawn a child;
  • assign/chat;
  • inspect owned descendants;
  • await/collect results;
  • cancel/retry/retire;
  • answer parent requests.

Nested creation must:

  • register the durable child before activation;
  • use an idempotent command_id;
  • derive parent/root/Project/depth from ActorDirectory;
  • enforce configurable depth, child concurrency, token/time/cost budgets;
  • attenuate file/tool/credential/network/placement authority;
  • deliver semantic ChildOutcome messages to the parent inbox;
  • preserve recovery when a parent or child passivates or restarts.

Owned children, resident actors, and shared service actors use the same ActorSession identity contract. Shared service actors additionally require explicit caller ACLs and must not silently share one transcript among unrelated callers.

Parent-owned permission and question handling

All permission requests that still require a decision, and all blocking child questions, become typed durable ParentRequests delivered to the direct parent Session.

Already-allowed actions execute without a request. Explicit hard-deny, Guardian/Plan, authentication, OS, and sandbox boundaries reject immediately and cannot be overridden by a parent. Auto mode remains zero-prompt: it executes allowed actions and preserves hard denials rather than opening an approval loop.

ParentRequest

At minimum:

  • request_id;
  • origin child ActorId and activation attempt;
  • direct parent ActorId;
  • kind: permission or clarification;
  • exact blocked operation/resource digest;
  • rationale, options, and recommended safe default;
  • deadline;
  • escalation path/hop count;
  • policy revision and maximum delegable authority.

ParentResolution

At minimum:

  • request_id;
  • decision: approve_once, deny, answer, or escalate;
  • answer/reason;
  • exact scope and expiry for a grant;
  • decided_by ActorId/activation/policy;
  • persisted resolution version.

The request enters the actual parent Session and its audit/history. It must not exist only as a hidden reviewer LLM call or an in-memory live bridge.

The parent may inspect the child's history and diagnostics before replying:

{
  "target": "child-123",
  "reply_to": "permission-7",
  "message": "approve_once"
}

A parent may approve only within its inherited delegation envelope. If it lacks authority, it may deny or escalate to its own parent. Every escalation records its path and is bounded against cycles.

Breaking parent/child wait cycles

The runtime must explicitly break this cycle:

Parent WaitingForChild
Child AwaitingParent(request)

When a child emits ParentRequest:

  1. persist it in the parent SessionInbox;
  2. mark the child Suspended/AwaitingParent, not terminal;
  3. interrupt or temporarily release the parent's WaitingForChild state;
  4. wake/activate the parent to process the request;
  5. persist ParentResolution;
  6. deliver and wake the child;
  7. re-arm the parent's original wait if still applicable.

No Session lock, placement lock, activation reservation, or parent wait lease may be held while waiting for a model decision. A warm worker may be kept briefly for latency, then checkpointed and released after a grace period.

Permission deadlines fail closed with deny. Clarification deadlines use an explicit safe default or transition to a recoverable blocked_needs_input state. They never remain falsely Running.

Durable queue, retry, and wakeup semantics

Use one canonical SessionInbox per logical Session. Do not create a second in-memory source of truth.

Queue states

READY
  | claim
  v
IN_FLIGHT ----------------------------> ACKED
  | transient failure / lease timeout
  v
DELAYED -- next_visible_at -----------> READY

permanent failure / retry limit ------> DEAD_LETTER

Each delivery record needs:

  • stable message_id and generation;
  • delivery_attempt;
  • next_visible_at;
  • claim/lease owner;
  • lease epoch and lease_expires_at;
  • last error;
  • activation eligibility and wakeup epoch.

Retry metadata is queue metadata and must not change the semantic message id or payload.

Claim/ack contract

  • claim moves a ready item into an in-flight lease; it does not delete it;
  • the active consumer renews the lease while making progress;
  • ack is legal only after the exact message id is durably represented in the authoritative Session transcript/admission cursor;
  • worker loss before durable admission causes lease expiry and requeue;
  • crash after transcript checkpoint but before ack is reconciled from the admitted receipt and completes ack without duplicate admission;
  • stale workers cannot ack or emit after their lease epoch is fenced;
  • invalid, unauthorized, or malformed messages are quarantined/dead-lettered, not hot-looped.

Distinguish delivery failure from execution failure

  • Activation/host launch failed before admission: keep/requeue the message and retry wakeup.
  • Worker died after claim but before transcript checkpoint: requeue the same message after lease expiry.
  • Input was already admitted and the LLM/tool execution later failed: retry ActorActivation against the existing transcript; do not append the same input again.
  • Permission denied or a permanent domain error: produce a terminal semantic result; do not retry the original request.
  • Delivery of that final result may itself use normal queue retry semantics.

WakeReconciler

Whenever eligible work exists:

  • if the receiver is Active, notify its current activation;
  • if it is Cold, reserve exactly one successor activation;
  • coalesce multiple messages into one activation;
  • if activation/placement fails, retain the queue item and schedule retry with exponential backoff plus jitter;
  • when next_visible_at arrives, try again;
  • reclaim expired claims;
  • on process restart, scan eligible durable backlog;
  • detect READY messages with no owner/heartbeat and repair the wakeup;
  • stop poison-message hot loops with bounded attempts and dead-letter state.

A business deadline is different from a visibility lease. A delivery timeout requeues. A permission/question decision deadline produces a deterministic deny/default/blocked resolution rather than restarting an infinite approval conversation.

History, diagnostics, and authority

The root Session can inspect its full descendant tree. A direct parent can inspect its owned subtree. Siblings cannot read each other by default. Shared/service actors require ACLs.

Inspection is on demand: child histories remain isolated and are not automatically merged into the parent LLM context. The tool returns paginated or query-focused transcript slices/summaries and redacts credentials or protected tool output.

Chat self-report and runtime diagnosis remain distinct:

  • chat asks the child what it believes;
  • inspect reads authoritative control-plane state even when the child is dead, cold, disconnected, or stuck.

Diagnostics must include:

  • logical lifecycle and current activation;
  • placement class without sensitive endpoint/token details;
  • Host/worker heartbeat and lease;
  • last event, LLM round, tool name and tool phase;
  • pending inbox, claimed/delayed/dead counts;
  • claim owner/expiry, attempts, next retry, and last wake error;
  • pending permission/question and its deadline;
  • parent/child wait graph and cycle detection;
  • latest error, retry history, and checkpoint;
  • health classification: healthy, waiting, stalled, failed, orphaned, blocked_needs_input, or lost;
  • evidence and suggested recovery action.

Event and Lotus Next projection

Every worker event is wrapped in an authoritative ActorEventEnvelope:

  • actor_id;
  • root/parent identity from ActorDirectory;
  • activation attempt and run_id;
  • per-attempt monotonic sequence;
  • timestamp;
  • typed event payload.

Bamboo validates the lease/attempt and projects:

  • durable low-volume topology/lifecycle;
  • agent.{ActorId} high-volume live content;
  • authoritative transcript/runtime changes;
  • metrics for duplicate, stale, missing, and rejected events.

Lotus Next consumes the same ActorDirectory/SessionDiagnostics projection as the model-facing inspect capability:

  • recursive persistent ActorSession tree;
  • history and activation inspector;
  • local/remote/container placement badge;
  • queue, permission/question, retry, and health state;
  • one authenticated shared /v2/stream connection;
  • lazy reference-counted subscriptions only for visible/expanded/previewed actors;
  • transcript/snapshot recovery after sequence gaps.

Lotus Next never receives broker credentials, worker endpoints, PIDs, container ids, or secret values.

Reliability and security requirements

  • At-least-once transport plus message-id deduplication; do not claim globally exactly-once side effects.
  • Tool actions that may be retried use idempotency keys or transactional side-effect records.
  • One transcript writer per ActorSession.
  • Attempt/lease fencing rejects stale events, replies, approvals, cancellations, and acks.
  • ParentRequest and ParentResolution are idempotent and exactly one terminal resolution wins.
  • mTLS or equivalent authenticated host identity for remote workers.
  • Short-lived, actor/activation-scoped broker credentials; no one shared token as an authorization boundary.
  • Effective capability equals the intersection of root policy, parent delegation envelope, Project policy, EnvironmentLease, and Host capabilities.
  • Every grant, denial, escalation, placement, retry, and recovery is auditable.
  • Queue and history limits apply backpressure without silently dropping durable semantic messages.
  • Retire preserves history; destructive deletion remains a user/admin operation.

Migration plan

Phase 0: contracts and invariants

Phase 1: identity convergence

Phase 2: durable requests and queue recovery

  • Extend SessionInbox with lease/renew/nack, delayed retry, retry wakeup epoch, dead-letter, and reconciliation.
  • Converge child permissions and questions into ParentRequest/Resolution.
  • Add wait-graph cycle breaking and parent wakeup.
  • Preserve Auto/hard-deny semantics.

Phase 3: placement and environment

  • Add authoritative HostRegistry and capability-aware PlacementScheduler.
  • Move local, Docker, SSH, remote, and schedulable execution behind one WorkerHost lease interface.
  • Add EnvironmentLease, isolated workspace/snapshot, artifact, and capability-proxy contracts.

Phase 4: one tool and nested delegation

Phase 5: lifecycle, diagnostics, and Lotus Next

Phase 6: removal and documentation

  • Remove raw mailbox identity from model/UI contracts.
  • Remove the in-memory deployment registry as actor truth.
  • Delete compatibility tools only after migration telemetry and tests prove no active callers.
  • Update actor runtime, broker, remote placement, tool, permissions, queue, recovery, and Lotus Next documentation.

Acceptance criteria

Identity and placement

  • Every owned local/remote/container/scheduled actor has a durable ActorSession before execution.
  • ActorId never equals or depends on PID, worker id, mailbox id, endpoint, container id, or host id.
  • Reassignment to a different WorkerHost preserves ActorId, history, inbox, parent/root/Project, and policy.
  • Local and remote paths use the same broker-mediated activation contract.
  • No warm worker leaks transcript, events, permissions, cancellation, workspace, or secrets between actors.
  • Pinned and auto placement honor hard constraints and never silently cross a security/data boundary.
  • Remote work receives an explicit EnvironmentLease; no remote actor assumes a parent-host absolute path exists.

Model-facing facade

  • The LLM receives one SubAgent tool and no deploy_agent/ask_agent tool.
  • Common delegation requires only a natural-language message; target is optional.
  • Existing Session chat, inspect/history, lifecycle control, and ParentRequest replies use the same ActorId.
  • Waiting, model selection, lifecycle temperature, workspace resolution, and placement mechanics are runtime-owned.
  • Physical topology and credentials never appear in model-visible tool results.

Parent requests and deadlock freedom

  • Every undecided child permission and blocking question is durably delivered to the direct parent.
  • Parent can inspect child history/diagnostics before resolving.
  • Parent decisions cannot widen inherited authority or override hard deny/Guardian/Plan/auth/OS/sandbox boundaries.
  • Auto mode emits no ordinary approval wait.
  • A Parent WaitingForChild is woken when that child needs a decision; the original wait is safely re-armed afterward.
  • Parent/child/grandchild escalation is bounded, cycle-safe, restart-safe, and auditable.
  • Every request has a deterministic timeout result; no actor remains falsely Running.

Queue and recovery

  • Durable messages survive process, worker, host, and broker reconnect failures.
  • Claim leases expire and safely requeue messages without allowing stale consumers to ack.
  • Queue retry uses backoff/jitter and bounded poison-message handling.
  • Transcript checkpoint before ack prevents duplicate provider admission.
  • Same-generation failed wakeups can reserve a new activation attempt.
  • Eligible queued work wakes an active receiver or reserves exactly one cold activation.
  • Startup and periodic reconciliation repair stranded eligible backlog.
  • Delivery failure, execution failure, permanent rejection, and business deadline have distinct outcomes.
  • Dead-letter items are visible, diagnosable, and manually retryable without changing semantic ids.

History, diagnostics, and UI

  • Root can inspect the authorized descendant tree and requested transcript slices without automatic history merging.
  • Diagnostics work for Active, Cold, Suspended, Failed, Lost, and disconnected actors.
  • Inspection exposes queue, activation, wait, permission/question, heartbeat, error, and retry evidence.
  • Every observable actor publishes lifecycle before live content.
  • Stale/duplicate/out-of-order ActorEventEnvelopes cannot regress state or pollute a newer activation.
  • Lotus Next renders a recursive persistent tree and subscribes lazily through one authenticated Bamboo stream.
  • Browser clients never connect to broker/worker endpoints or receive their credentials.

Deterministic test plan

  • Create equivalent actors through SubAgent and legacy deploy_agent compatibility; compare durable identity and policy.
  • Restart between Session persistence, queue delivery, placement reservation, activation start, transcript checkpoint, and ack.
  • Lose a local process, container, SSH connection, remote resident worker, and schedulable host; recover on another host with the same ActorId.
  • Reuse one warm worker across unrelated root/Project actors and prove complete isolation.
  • Claim a message, crash before admission, expire lease, requeue, and admit once.
  • Crash after transcript checkpoint but before ack; reconcile without duplicate history.
  • Fail placement repeatedly; verify backoff, wake retry, attempt fencing, and dead-letter behavior.
  • Deliver several messages while Cold; verify one activation drains a bounded ordered batch.
  • Deliver while Active; verify notification without a second transcript writer.
  • Parent waits for child while child requests permission; verify parent wake, resolution, child resume, and wait re-arm.
  • Repeat the previous case at grandchild depth and across restart.
  • Parent absent/retired and root unavailable; verify bounded escalation and deterministic deny/blocked outcome.
  • Auto, Default, Plan, Guardian, hard-deny, and forced-ask policy matrices.
  • Late permission reply, stale activation event, old cancellation, old ack, and broker redelivery.
  • Local dirty workspace task stays local; portable snapshot task schedules remote; sensitive task refuses unsafe fallback.
  • Container gets only scoped mounts; remote artifact/patch returns through references.
  • Root inspect finds a deliberately stalled queue/activation and proposes a valid recovery.
  • Recursive Lotus Next tree with 100+ actors proves bounded high-volume channel count and correct gap recovery.

Documentation

Update:

  • docs/design/subagent-actor-runtime-design.md
  • docs/design/remote-actor-plan.md
  • docs/design/remote-mailbox-broker-design.md
  • docs/design/architecture-overview.md
  • SubAgent tool documentation
  • permission/Auto/hard-deny documentation
  • SessionInbox delivery/recovery documentation
  • Lotus Next actor-tree and streaming documentation

Terminology must be consistent:

  • ActorSession: durable logical agent;
  • ActorActivation: one execution attempt;
  • WorkerHost: physical capacity;
  • EnvironmentLease: authorized environment/data/tool access;
  • ParentRequest: permission or clarification delegated to an ancestor;
  • broker: transport only;
  • ActorDirectory/Runtime: identity, authority, activation, placement, and recovery control plane.

Non-goals

  • No direct browser-to-broker or browser-to-worker protocol.
  • No second worker-local Session source of truth.
  • No magical remote access to a parent-host filesystem.
  • No promise of lossless persistence for every token event.
  • No global exactly-once guarantee for arbitrary external side effects.
  • No unbounded actor recursion.
  • No child authority escalation.
  • No cross-root/cloud peer Handoff redesign; that remains [cloud] epic: peer Handoff and authoritative actor discovery #224.
  • No replacement of existing placement launcher work where [bamboo] feat: remote actor execution (远程拉起 sub-agent) #9 already owns it.
  • No immediate removal of compatibility surfaces before migration evidence.

Related work

Bamboo:

Lotus Next:

Delivery guidance

This Epic is not a single-agent implementation slice. Split work into scoped, dependency-ordered Issues and isolated branches with local review. Keep it agent:blocked until the first ready implementation slice has reconciled closed #681 and the authoritative contracts above are accepted.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent:blockedBlocked by dependency or conflictpriority:P1High — this sprintscope:cross-moduleAffects multiple modules — coordinate carefullytype:featureNew feature or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions