Repository navigation
feat(harness): add devin_cloud_agent harness on the Devin API v3 contract - #235
devin-ai-integration[bot] wants to merge 14 commits into
Conversation
Adds the Devin cloud agent as a fourth capability-discoverable harness kind, advertised as unavailable with explicit feature gaps, plus the internal opensymphony-devin adapter, client primitives, and event normalization. Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9d7becad1d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
Devin Review found 4 potential issues.
3 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
- reject query/fragment suffixes in devin endpoint and repository URLs - disable reqwest redirects and validate config before bearer use - preserve events lacking a discriminator as unknown raw payloads - keep devin.api.api_key_env a literal env var name - resolve devin config only for devin-routed workflows - reject unavailable routes before ensure_workspace in dispatch - allowlist devin_cloud_agent in frontend model profiles - document devin.api in docs/configuration.md Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
|
|
|
|
|
Repository URL validation now rejects query strings and fragments alongside embedded userinfo, so token-decorated remotes ( |
|
Redirects are now disabled ( |
|
Events without |
|
Devin config is now resolved only when |
|
|
|
|
|
Fixed: route decision moved ahead of |
|
|
Summary: - Vendor the official Devin v3 OpenAPI document and rebuild the client request builder, response models, and lifecycle against it, replacing the previously invented endpoints and event vocabulary. - Address v3 organization-scoped routes: /v3/self, session create/get/list (qs query document), cursor-paginated messages, operator message send, delete/archive, attachments, tag replacement. - Drive sessions by polling with event_id dedupe, status/status-detail transitions, and PR/ACU/structured-output/attachment run evidence. - Replace v1-only workflow config with devin.api and devin.session blocks (org id, playbook, knowledge/secret ids, ACU cap, tags, title, devin_mode, platform, resumable) and validate them. - Add tests/devin_cloud_agent_live.rs, an opt-in live suite that runs the real API, plus contract tests asserting every request path and body against the vendored spec. - Document the live verification, its cost, and what remains unverified. Rationale: - The first pass guessed the API surface; the endpoints, payload shape and event discriminators did not exist. Only the published contract plus a real request/response round trip can establish compatibility. - v3 only, with no v1 fallback, so there is a single implementation. - Capability stays available: false. Live protocol compatibility is not hosted-security, tenancy, or remote-workspace evidence. Tests: - cargo fmt --all; cargo check --all-targets; cargo clippy --all-targets -- -D warnings - cargo test --test devin_cloud_agent (11 passed) - cargo test --lib devin; cargo test --test scheduler unavailable; cargo test --test gateway; cargo test --test gateway_schema - OPENSYMPHONY_DEVIN_LIVE=1 cargo test --test devin_cloud_agent_live -- --ignored: identity/listing and a real session create, poll, message, terminate+archive against https://api.devin.ai Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
Summary: - Wire `devin_cloud_agent` through the CLI orchestrator route: bind tenancy, resolve secret references, create the session, stream normalized events, import evidence, and settle the worker outcome. - Bind every client to one organization via `GET /v3/self`, rejecting a configured foreign org and any session/list/message payload owned by another org. - Inject secrets by organization-scoped Devin secret id or key resolved from the bound org; raw values never enter OpenSymphony. - Import remote evidence into the issue workspace under `.opensymphony/devin/<run-id>` (session, event journal, manifest, attachments) with origin, size, count, and path-containment limits. - Terminate and archive the remote session on poll timeout, transport failure, scheduler interrupt, and worker task drop. - Flip the capability to available with `devin-api-v3` and explicit feature gaps; update gateway, desktop, scheduler, and schema tests. - Rewrite the Devin sections of AGENTS.md and the configuration, harness-compatibility, and host-client-architecture docs. Rationale: - The harness was a boundary-only scaffold: the route rejected dispatch, so nothing exercised the client. The hosted gaps (tenancy, secrets, artifact retrieval) had to close before the route could be honest. - Availability is now backed by live evidence rather than fixtures, and the remaining limitations are contract limits of Devin API v3, so they belong in `feature_gaps` rather than blocking the capability. - A Devin session outlives the worker process and bills ACUs, so every abandoned path had to terminate and archive explicitly. Tests: - cargo fmt --all - cargo clippy --all-targets -- -D warnings - cargo test --test devin_cloud_agent (17), --test gateway_schema (58), --test gateway (77), --test scheduler (94), --lib devin (5) - cargo test --lib: 1237 passed, 1 pre-existing sandbox failure (memory::tests::configured_memory_checkout_must_match_canonical_remote, git remotes rewritten to a proxy URL in this environment) - OPENSYMPHONY_DEVIN_LIVE=1 cargo test --test devin_cloud_agent_live -- --ignored: 4 passed against api.devin.ai (identity/listing, tenancy + secret scope, full lifecycle with evidence import, poll-timeout cleanup verified archived) Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
Summary: - Settle `--dry-run` before credential resolution or session creation so a dry run never submits a task or consumes ACUs. - Persist `run.json` and the Devin conversation binding before reporting the launch, and rebuild a tenant-bound client when an interrupt finds no tracked session, so cancellation and restart recovery reach the live session. - Settle `WaitingOnOperator` as a paused `Detached` run instead of a retryable `Stalled` one, keeping the session tracked so the operator's answer continues the same run. - Keep the cleanup guard armed and settle as `CancelFailed` when terminating after a polling failure fails, instead of releasing ownership for a retry while the original session may still bill. - Prepare the issue workspace through a new evidence-only path for remote-only harness capabilities, skipping checkout clone/verify and repository hooks. - Compare complete attachment origins including effective ports, and read attachment bodies incrementally so an unbounded chunked response cannot exhaust daemon memory before the size ceiling applies. Rationale: - Every finding was a way for the route to lose ownership of a remote session that keeps executing and billing, or to start a second one. Ownership is now released only once the session is provably gone. - Devin clones the repository itself, so requiring a local checkout on the orchestrator host blocked launches the remote agent could serve. Tests: - cargo fmt --all; cargo check --all-targets; cargo clippy --all-targets -- -D warnings - cargo test --lib (1241 pass; the pre-existing configured_memory_checkout_must_match_canonical_remote failure is environmental: the sandbox rewrites git remotes to a proxy URL) - cargo test --test devin_cloud_agent --test scheduler --test gateway --test gateway_schema Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
|
All seven findings are addressed in [P1] Dry-run started a real session. The route now settles the dry run before [P1] Missing run manifests. [P1] Waiting for human input. [P1] Failed termination released ownership. [P2] Remote execution required a local checkout. Added [P2] Attachment-origin validation ignored ports. Validation now compares scheme, host, and [P2] Unbounded buffering. The download reads Validation: |
Summary: - Recovery reattaches to the Devin session recorded in the conversation manifest and follows it instead of calling create_session; a session that cannot be looked up settles as non-retrying CancelFailed. - RunManifest persists terminal_worker_outcome, RecoveryRecord carries it, and scheduler bootstrap restores the outcome (new IssueExecution::restore_worker_outcome) before any dispatch decision. - Workspace finalization and evidence import preserve Detached and CancelFailed whenever remote termination is unconfirmed, and keep the cleanup guard armed. - Terminal workspace cleanup detects a Devin conversation binding, rebuilds a tenant-bound client from workflow config, terminates and archives the session, and retains the workspace when that fails. - Evidence-only workspaces skip every repository hook: ensure, start, and finish run through a hooks-skipped path with no checkout, no after_create, and no before_run/after_run. Rationale: - Paused and Failed alone are ambiguous across a restart, so a detached or cancel-failed run was becoming a replacement session that billed ACUs while the original kept running. - Local persistence failures must not downgrade a safeguard into an ordinary retryable failure. - A detached session had no interruptible run left, so terminal cleanup was the only remaining owner able to stop it. - Remote execution must not require local repository access, which the repository's hooks (clone plus git commands) still forced. Tests: - cargo fmt --all; cargo clippy --all-targets -- -D warnings - cargo test --test devin_cloud_agent; cargo test --lib devin - cargo test --test scheduler; --test gateway; --test gateway_schema; --test workspace_manager - cargo test --lib (only the pre-existing sandbox failure configured_memory_checkout_must_match_canonical_remote, caused by the environment rewriting git remotes to a proxy URL) Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
|
All five findings at [P1] Recovery creates a new session instead of reattaching. The Devin route now takes a [P1] Non-retrying outcomes become retryable after restart. [P1] Local persistence failures undo the cancellation safeguards. [P1] Terminal cleanup does not stop detached Devin sessions. [P2] Evidence-only workspaces still execute repository hooks. Validation: |
Summary: - Report a failed reattachment as a successful launch bound to the persisted session, then settle CancelFailed, so the scheduler never turns the lost session into a retry that dispatches a replacement. - Keep the repository binding in the evidence-only issue manifest; only the checkout is skipped, so recovery no longer compares a live binding against None and removes a repository-bound detached workspace. - Skip workspace removal during recovery whenever a non-retrying outcome is persisted, since something outside the process is still bound to the workspace and its evidence. - Add WorkspaceManager::cleanup_evidence_only and route Devin terminal cleanup through it so before_remove no longer runs repository commands in a directory that has no checkout. Rationale: - LaunchReport::Failed becomes a launch error, which is retried; the non-retrying outcome has to survive the launch protocol to have any effect. - Repository identity and checkout preparation are separate concerns: dropping the binding made the workspace look unbound to every later comparison and cost the waiting session its evidence. - All four hooks are now uniformly skipped for evidence-only workspaces. Tests: - cargo clippy --all-targets -- -D warnings - cargo test --test devin_cloud_agent - cargo test --lib devin - cargo test --test scheduler - cargo test --test workspace_manager - cargo test --test gateway - cargo test --test gateway_schema Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
|
All three findings at [P1] Failed reattachment still triggers automatic replacement — the route no longer sends [P1] Restart terminates repository-bound detached sessions — [P2] Evidence-only cleanup still runs Validation: |
Central `config.yaml` could select `routing.harness: devin_cloud_agent` but had no place for the `devin:` block, and the repository-local front-matter merge dropped `devin` entirely, so the Devin route ran on inert defaults (no org/api env overrides, no ACU cap, no tags). - Accept an optional top-level `devin:` block in the central config, include it in the versioned shared-key set, and convert it into `WorkflowFrontMatter.devin`. - Merge repository-local `devin:` settings only when the central block is absent/default; central configuration wins otherwise. - Reject reuse of `devin.api.api_key_env` / `org_id_env` as a repository checkout credential variable. - Re-export `DevinFrontMatter` types from `opensymphony-workflow`. - Document a complete central `project_set` + Devin live-testing configuration in docs/configuration.md, including the dry-run guard, evidence location, and Ctrl-C/session-termination semantics. Validation: cargo fmt --all; cargo clippy --all-targets -D warnings; cargo test --lib orchestrator_run::config (77 passed); the documented sample config resolves through `opensymphony run --dry-run` up to the Linear authentication step with placeholder credentials. Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2e3d270f0f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ites Summary: - Terminate tracked Devin sessions (awaited, 30s bound) from WorkerBackend::abort_worker and a new WorkerBackend::shutdown called before runtime teardown; DevinSessionGuard::drop is opportunistic only. - Persist devin-route.json (session id, API origin, org, credential env NAME) beside the conversation manifest; terminal cleanup and the pre-creation fence address the bound session through that identity. - Reconcile a failed create_session through the correlation tag (list_sessions) before any retry; an unreconcilable create settles as non-retrying CancelFailed via the launch protocol. - DevinCloudClient::requests() requires verified tenancy (TenantUnbound), not a configured org id; identity() stays the only pre-binding call. - Evidence writes validate every path component (no symlinks) and commit via exclusive temp file + rename; symlinked dirs/files refused. - decide_issue_route rejects parent issues with sub-issues for remote-only harnesses; capability lists the gap. - Reserve one of 50 session tags for the correlation tag (49 operator tags, validated); send_user_message is advertised as false. - Box::pin the Devin route future inside the worker task like the other routes; the enlarged inline future overflowed the 1.5 MiB worker stack exercised by tests/run.rs. - Keep the guard armed through evidence persistence; failed import archives the settled session before reporting the retryable failure. Rationale: - Codex review at 2e3d270: a Drop-spawned task may never run during runtime shutdown, a lost create response could orphan a billing session, and cleanup reconstructed from mutable workflow settings could mistake another tenant's 404 for proof the session is gone. - Durable bindings plus persisted route identity make cleanup ownership independent of the worker task and of later config edits. Tests: - cargo fmt; cargo clippy --all-targets -- -D warnings - cargo test --test devin_cloud_agent (20), --test gateway_schema (59), --test scheduler (155), --lib devin (17); full cargo test - OPENSYMPHONY_DEVIN_LIVE=1 cargo test --test devin_cloud_agent_live -- --ignored (see PR comment) Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
Review at
|
| Finding | Fix | |
|---|---|---|
| P1 | Keep the cleanup guard armed until the session is archived | The guard now stays armed through evidence persistence. If the import fails after a settled outcome, the route archives the session (awaited) before reporting the retryable failure, and keeps CancelFailed ownership if even that fails. |
| P1 | Await task-drop termination before releasing the worker | WorkerBackend::abort_worker and a new WorkerBackend::shutdown (called in orchestrator_run before runtime teardown) terminate the sessions tracked for the affected worker(s) and await the acknowledgement (30 s bound) before aborting the task. DevinSessionGuard::drop is explicitly opportunistic; the durable fence is the persisted binding below. |
| P1 | Reconcile an uncertain session creation before retrying | A failed create_session first calls list_sessions for unarchived sessions carrying the run's correlation tag (select_reconciled_session, with epoch s/ms normalization and a creation-time floor). A match is followed as if the create had succeeded; a confirmed miss is retryable; a lookup failure settles as non-retrying CancelFailed through the launch protocol (LaunchReport::Conversation, never Failed) and names the tag to search in Devin. |
| P1 | Require tenant binding before building session requests | DevinCloudClient::requests() now needs tenancy (set only by bind_tenant/bind_verified_tenancy) and returns TenantUnbound otherwise; a configured org id alone no longer suffices. identity() stays the only pre-binding call. Tests cover unconfigured, configured-but-unbound, bound, and mismatched tenancy. |
| P1 | Prevent evidence writes from following symlinks | create_dir walks every component with symlink_metadata and refuses symlinks/non-directories; write_file refuses a symlinked parent or destination and commits via an exclusive (create_new) temp file + rename. New tests: symlinked evidence directory and symlinked events.jsonl, both refused with nothing written through. |
| P1 | Reject unsupported Devin parent integrations | decide_issue_route rejects a parent issue with sub-issues for any remote-only harness (InvalidConfiguration, names the issue and the missing local checkout); the gap is listed in the capability's feature_gaps. Scheduler test covers parent-rejected / leaf-ok / local-harness-ok. |
| P2 | Use the persisted Devin route for terminal cleanup | The route persists devin-route.json (session id, API origin, org id, credential env name) beside the conversation manifest before reporting launch. Terminal cleanup and the pre-creation fence (stop a session a dropped worker left bound) rebuild the client from that identity, so a changed workflow origin/org cannot produce a 404 that looks like proof of termination. |
| P2 | Reserve a tag slot for the correlation identifier | Operators get DEVIN_MAX_OPERATOR_TAGS = 49 (validated in DevinCloudConfig::validate); session_create_request appends the correlation tag last and never truncates it. |
| P2 | Stop advertising an unwired user-message action | send_user_message: false, with a feature gap explaining that the route follows the Devin session but does not forward operator messages. Gateway-schema tests updated. |
Validation
cargo fmt
cargo clippy --all-targets -- -D warnings clean
cargo test --test devin_cloud_agent 20 passed
cargo test --test gateway_schema 59 passed
cargo test --test scheduler 155 passed
cargo test --no-fail-fast all targets pass except 8 tests that fail identically on origin/main in this sandbox (see below)
OPENSYMPHONY_DEVIN_LIVE=1 cargo test --test devin_cloud_agent_live -- --ignored
4 passed (identity/listing, tenant+secret scope,
poll-timeout termination, full session lifecycle)
Sandbox-only failures: The sandbox's global gitconfig rewrites https://github.com/ to a git-manager.devin.ai/proxy/ URL via insteadOf, so configured_memory_checkout_must_match_canonical_remote (lib) and the 7 init::* tests that assert the literal clone URL in the scaffolded WORKFLOW.md fail here. Verified on an origin/main (d755ca27) worktree with the same toolchain: the same 8 tests fail there with the same assertions, and nothing else does. tests/run.rs::run_dispatches_gateway_cancel_to_openhands_interrupt (1.5 MiB worker stack) did fail on this branch before the Box::pin fix and passes on both branch and main now.
Also fixed while running the full suite: the enlarged Devin route future was created inline in the worker task and overflowed the 1.5 MiB worker stack exercised by tests/run.rs (deterministic on this branch, passes on main); it is now Box::pin'd like the ACP/Codex/OpenHands routes.
Summary:
- Add `DevinCloudClient::stop_session` with `DevinStopOutcome`
(`Terminated` / `AlreadyStopped` / `Missing`) and the documented
`POST .../sessions/{id}/archive` request. DELETE on an already-exited
session is rejected by the real API with HTTP 400 "Devin session
already exited"; the client now re-reads the session, treats terminal
statuses as stopped, archives them if needed, and treats 404 as gone.
A rejection while the session is still live keeps the original error.
- Route every cleanup path (tracked-session shutdown, worker drop,
polling failure, suspended session, evidence-import failure,
scheduler interrupt, terminal workspace cleanup) through
`stop_session`; interrupt acknowledgements report which outcome
occurred. A session guard dropped after `abort_worker`/`shutdown`
already stopped the session no longer spawns a redundant stop.
- Production `DevinRunOptions` set `stop_when_waiting_on_operator`, so
`waiting_for_user` / `waiting_for_approval` settle as `Detached`
instead of being polled until Devin suspends for inactivity.
- Scheduler recovery of an in-flight run with a retry attempt now passes
through `RetryQueued` (`restore_retry` before `claim`), fixing
"retry attempt mismatch: expected None, got Some(RetryAttempt(2))".
- Evidence manifest gains `message_pull_request_urls`: `/pull/<n>` and
`/merge_requests/<n>` https URLs that appear only in Devin-authored
messages (deduplicated against the API-reported `pull_request_urls`);
the settlement summary lists them.
Rationale:
- Observed during the live StackPerf dispatch: the first run's session
was polled through inactivity suspension and treated as a stall, the
retry's cleanup DELETE failed with "already exited", and a restarted
daemon failed to rebuild its scheduler from the recovered retry
attempt. The second run opened its PR through the GitHub API (Devin's
built-in PR tool lacked permission) so the API's `pull_requests` was
empty although the PR URL was in the journal.
Tests:
- cargo fmt; git diff --check
- cargo clippy --all-targets -- -D warnings
- cargo test --test devin_cloud_agent (24 passed)
- cargo test --lib -- devin
- cargo test --test scheduler terminal_recovery (5 passed)
- Live: `opensymphony run` against Linear COE-733/COE-734 routed to
`devin_cloud_agent`; restart recovery restored `Detached` for COE-734
without creating a replacement session; terminal cleanup of COE-733
archived its exited session.
Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
Summary: - Fetch the repository's configured instruction file (WORKFLOW.md) from the GitHub remote at the target branch through the contents API, using the repository checkout credential, and render it into the `## Repository Instructions` section of the Devin session prompt. - Share the workspace `instruction_body` helper and `MAX_INSTRUCTION_FILE_BYTES` so remote instructions get the same front matter stripping and size bound as local checkouts. - Record path, git ref, SHA-256 and size as `repository_instructions` in `evidence.json` / `session.json`. - Document that `devin.session.secret_ids` must reference the Linear and GitHub secrets the repository's agent-side attach/push steps need. Rationale: - Remote-only routes have an evidence-only issue workspace with no checkout, so Devin sessions received "No repository-specific instructions were selected." and never saw the repository workflow step that attaches the PR to the Linear issue (observed on COE-733 and COE-734: PRs opened, no Linear attachments). - A configured file that cannot be fetched fails the launch (retryable), matching the local route, instead of running without instructions. Tests: - cargo test --lib -- devin (22 passed, 4 new) - cargo test --test devin_cloud_agent (24 passed) - cargo test --test scheduler terminal_recovery (5 passed) - cargo test --test workspace_manager (45 passed) - cargo clippy --all-targets -- -D warnings; cargo fmt --check; git diff --check Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
…stall Summary: - `DevinSessionRunner::follow` emits a `session.heartbeat` liveness event when an executing session has produced no message or status change for `DevinRunOptions::heartbeat_interval` (default 60s); the route derives that interval from the scheduler stall timeout (`stall / 3`, capped at 60s) so it always lands inside the stall window. - `SessionResponse::is_executing` names the liveness condition (not settled, not waiting on the operator); `heartbeat_event` carries the status, status detail, ACUs and `updated_at` of the snapshot. - `DevinEvidenceCollector::record` keeps heartbeats out of `events.jsonl`; they are orchestrator observations, not remote events. - Docs and AGENTS.md record the rule; the live suite's option literals use `..DevinRunOptions::default()`. Rationale: - The first real StackPerf dispatch (COE-373) was aborted by the scheduler with `scheduler stall timeout reached` (`agent.stall_timeout_ms` default 300000) about five minutes after Devin posted its plan: Devin reports no message or status change while it implements, so the polling route went silent and the stall policy terminated the session and created a replacement, which failed the same way. The stall policy is activity based (`observe_runtime_event`), so the fix is to make genuine remote execution observable as activity rather than to disable stall detection for the harness. Tests: - cargo fmt --check; git diff --check - cargo clippy --all-targets -- -D warnings - cargo test --test devin_cloud_agent (25 passed, incl. quiet_executing_sessions_publish_heartbeats_outside_the_evidence_journal) - cargo test --lib -- devin_run_options (2 passed) Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
…orted Summary: - `abort_worker` and `shutdown` still terminate the tracked Devin sessions first, but a worker that owns a Devin session now gets a bounded window (`DEVIN_SETTLE_TIMEOUT`, 30s) in which its route task observes the archived session, imports the evidence, and finishes `run.json` as cancelled; only a task that does not settle in time is aborted. - A route task that panics while settling is reported through the normal `WorkerUpdate::Finished` boundary (failed, harness stopped), mirroring the ACP abort path, so `poll_updates` never joins a consumed handle. - `settle_devin_tasks` takes the settle budget as a parameter so the unit test covers the settled, wedged, and panicking cases without waiting out the production timeout. Rationale: - The first real StackPerf dispatch (COE-373) ended with Devin moving the issue to `Human Review` after attaching PR #53; the scheduler released the worker as tracker-inactive, the backend terminated the session and immediately aborted the task, so the evidence-only workspace was left with `run.json` still `running` and no imported evidence for a run that had in fact produced a PR. Letting the route see the termination it was handed is cheaper and more faithful than finalizing the manifest from the abort path, which would have to duplicate the settle logic. Tests: - cargo fmt --check; git diff --check - cargo clippy --all-targets -- -D warnings - cargo test --lib -- settling_terminated_devin_tasks (1 passed) - cargo test --lib -- devin (24 passed) - cargo test --test devin_cloud_agent (25 passed) - cargo test --test run (9 passed, 4 ignored) Co-Authored-By: Leonardo Gonzalez <Leonardo@EncryptedCommerce.net>
Summary
Adds
devin_cloud_agentas a runnable harness on the Devin API v3 contract. Devin owns the execution workspace; the scheduler-bound issue workspace is evidence-only (manifests, journals, imported attachments) and is never Devin's cwd. The route reattaches to a bound session on restart, keeps polling a session that is waiting on an operator, and treats remote termination as something that must be confirmed before any local state is released or any replacement session is created.Changes
crates/opensymphony-devin):GET /v3/selftenant binding is required before any org-scoped request (TenantUnboundotherwise); cross-tenant payloads are rejected; secrets go in as org-scopedsecret_ids; HTTPS/TLS 1.2+, no redirects, attachment downloads pinned to the API origin (incl. port) and read incrementally under a size cap; evidence writes refuse symlinked directories/files and commit via exclusive temp + rename.stop_session -> DevinStopOutcome {Terminated, AlreadyStopped, Missing}: Devin answersDELETEon an already-exited session with HTTP 400"Devin session already exited", so the client falls back toGET+POST /archiveand only reports failure when the session is still live. Every cleanup path (tracked-session cleanup, worker drop, polling error, suspended session, evidence-import failure, scheduler interrupt, terminal cleanup, shutdown) goes through it.backends.rs): dry-run never reaches Devin;run.json, the conversation binding anddevin-route.json(session id, API origin, org id, credential env name) are persisted beforeLaunchReport; a failedcreate_sessionis reconciled through the per-run correlation tag before any retry;WaitingOnOperator→ non-retryingDetached; unconfirmed termination → non-retryingCancelFailed, both durable inRunManifest.terminal_worker_outcomeand restored on recovery; local evidence/finalization failures never downgrade a non-retrying outcome.evidence.json) separatespull_request_urls(Devin APIpull_requests) frommessage_pull_request_urls(absolute HTTPS/pull/Nor/merge_requests/NURLs found only in Devin-authored message events; operator messages are not mined; query/fragment stripped, deduped). Message-only PRs are appended to the settlement summary.abort_workerand the newWorkerBackend::shutdown(called before runtime teardown) terminate+archive tracked sessions with awaited acknowledgement before aborting tasks; terminal cleanup and the pre-creation fence address the bound session through the persisted route identity, so later config edits cannot turn another tenant's 404 into proof of termination.DevinSessionGuard::dropis opportunistic only and skips sessions another path already removed from the registry.transport.remote && !local) get the evidence-only workspace path (no checkout, none of the four hooks); parent issues with sub-issues are rejected for remote-only harnesses; recovery restores a persisted retry attempt throughrestore_retrybeforeclaim, so a recovered attempt-N run no longer fails scheduler construction withretry attempt mismatch.available: true,runtime_contract_version: devin-api-v3,send_user_message: false, with explicit feature gaps (polling, terminate-as-cancel, no pause/resume/approvals, fixed model, no operator forwarding, no parent integration). 49 operator tags max; the 50th slot is the correlation tag.config.yamlnow propagates thedevin:block;docs/configuration.mdhas the end-to-end walkthrough.instructions.path, e.g.WORKFLOW.md) from the configured remote at the target branch through the GitHub contents API (configured token, 1 MiB bound enforced onContent-Lengthand on the incremental read, SHA-256 over the raw bytes recorded with path/ref/byte count), stripsWORKFLOW.mdfront matter, and renders it into the Devin prompt under## Repository Instructions; launch fails if the configured file cannot be fetched. Together withdevin.session.secret_ids: [LINEAR_API_KEY, GH_TOKEN_TG], Devin can run the repository workflow itself, including the agent-sideattachmentLinkGitHubPRstep.session.heartbeatnormalized event (everystall_timeout_ms / 3, capped at 60s) while the sessionis_executing()and no remote event arrived; it travels throughWorkerUpdate::RuntimeEventso the scheduler sees activity, and is excluded fromevents.jsonlso it is never mistaken for Devin-authored evidence.abort_worker/shutdownstill terminate the tracked sessions first, but a worker that owns a Devin session then gets a bounded window (DEVIN_SETTLE_TIMEOUT, 30s) in which its route task observes the archived session, imports evidence, and finishesrun.jsonas cancelled; only a task that does not settle in time is aborted (a panicking task is reported as failed/harness-stopped like the ACP path).Evidence
Test output
Checks on the current head (
2d355e6e):The sandbox's global gitconfig rewrites
https://github.com/to agit-manager.devin.ai/proxy/URL viainsteadOf, soconfigured_memory_checkout_must_match_canonical_remote(lib) and the 7init::*tests that assert the literal clone URL in the scaffolded WORKFLOW.md fail here. Verified on anorigin/main(d755ca27) worktree with the same toolchain: the same 8 tests fail there with the same assertions, and nothing else does; this branch only adds struct fields tomemory.rstest fixtures.tests/run.rs::run_dispatches_gateway_cancel_to_openhands_interrupt(1.5 MiB worker stack) did fail on this branch before theBox::pinfix and passes on both branch and main now.Live orchestrator dispatch (StackPerf / Linear)
Synthetic smoke issues first (
max_acu_limit: 3), then a real issue (max_acu_limit: 15), all viaopensymphony runwith centralproject_setrouting against the StackPerf Linear project:--dry-runresolved routing and wroterun.jsonwithout creating a Devin session.Succeeded/ non-retryingDetached), terminated+archived at 0 ACUs; StackPerf Add configurable OpenHands context condenser support #51/Support workflow-owned OpenHands conversation reuse policy at runtime #52 surfaced viamessage_pull_request_urls. These runs exposed theretry attempt mismatchrecovery bug and the HTTP 400 onDELETEof an exited session (both fixed), and that the prompt carried no repository instructions and no Linear/GitHub secrets, so the PRs were never attached in Linear (fixed by the remote-instructions +secret_idschange).d47f42d5…,94efb0d9…) exited without a PR — root cause was the false stall abort fixed by the heartbeat. With the heartbeat, sessionf45298b55bc5440dbe5aae4c2c6d845cfollowed StackPerf'sWORKFLOW.md: branchleonardogonzalez/coe-373-implement-sessionless-litellm-usage-collector-and-normalizer, feat(collectors): sessionless LiteLLM usage collector and normalizer trilogy-group/StackPerf#53 (2 commits, 8 files, +1,883/-3), attached in Linear as an API attachment (attachments.nodes[0].url == PR #53,sourceType: api) and the issue movedIn Progress -> Human Review.Human Reviewis outside the configured active states, so the scheduler released the worker as tracker-inactive and the backend terminated+archived the session.run.jsonasrunningwith no imported evidence for a run that had produced a PR. Fixed by the settle window above (the COE-373 workspace from that run keeps the stale manifest; the fix applies to subsequent runs).COE-373 blocks COE-374relation, and COE-373 is inHuman Review, notDone.Verification
Live against
api.devin.aiwith the org service-user token: sessions created, tagged, polled, messaged, heartbeated, and terminated+archived; the poll-timeout path archived its session; tenant binding rejected a foreign org id; secret-id resolution was scoped to the bound org; the rendered prompt for COE-373 contained the StackPerfWORKFLOW.mdbody and theattachmentLinkGitHubPRstep with no raw secret values. Offline tests cover dry-run, launch-before-manifest ordering, Detached/CancelFailed persistence and recovery, correlation-tag reconciliation, verified-tenancy gating, symlinked evidence paths, remote-only parent rejection, the persisted route identity, the archive fallback for exited sessions, message-authored PR URL extraction, remote instruction fetch bounds/hashing/front-matter stripping, heartbeat shape/is_executing/journal exclusion, and the settle window (settled, wedged, panicking task).Checklist
cargo test)cargo fmt)cargo clippy)AGENTS.mdupdated (if repo knowledge changed)Type of change
Breaking changes
None.
devin_cloud_agentis an additional harness route;WorkerBackend::shutdownhas a default no-op implementation;DevinRunOptionsgainedheartbeat_interval(defaulted).Related issues
None.
Additional notes
Needs
COG_SERVICE_USER_TOKENandDEVIN_ORG_ID(names configurable); for the agent-side Linear attach step the Devin org must hold the Linear/GitHub secrets referenced bydevin.session.secret_ids. Not implemented and not claimed: TLS certificate pinning, push events (v3 has none), operator-message forwarding, 429/Retry-Afterbackoff on the v3 client, Devin CLI/ACP (separate local harness, separate PR).Link to Devin session: https://app.devin.ai/sessions/fd803159e50d4eb5983e73cb91691f82
Open in Devin Desktop: https://app.devin.ai/desktop/session/fd803159e50d4eb5983e73cb91691f82?variant=devin
Requested by: @kumanday