Skip to content

test: stabilize e2e approval and cancellation tests - #624

Merged
EugeneTheDev merged 4 commits into
mainfrom
eugenethedev/fix-e2e-tests
Oct 13, 2026
Merged

EugeneTheDev merged 4 commits into
mainfrom
eugenethedev/fix-e2e-tests

Conversation

@EugeneTheDev

Copy link
Copy Markdown
Collaborator

Summary

The E2E run on #620 (run) failed on two model-dependent tests. #620 didn't cause it: the run before it shows the same behavior and passed only by luck. This PR changes test code only.

  • Default e2e model gpt-5.2 → gpt-6-luna[low]. gpt-5.2 isn't in the Codex model list anymore, so Codex used fallback metadata. In 0.160.1 that has apply_patch_tool_type: None, so the model ran apply_patch through the shell. A malformed patch came in as an execute approval, not edit, and broke applies an approved workspace file edit.

  • Shell approval prompts ask for elevated permissions on the first try. In read-only mode the model sometimes ran the command in the sandbox, hit Read-only file system, and ended the turn without asking for approval (cancels every command when cancel is selected got end_turn). This applies only to the tests that expect an approval prompt.

  • Cancellation test follows how Codex handles interrupts. gpt-6-luna runs commands through unified exec, and turn/interrupt doesn't kill those processes: they keep running as background terminals. The test now:

    • waits until the client has seen the command's tool call;
    • cancels and asserts the turn is cancelled and the process is still running;
    • stops the process with _session/async_task/stop and asserts it exits.

    The ACP SDK client rejects the async_task_spawned update because its schema doesn't include it, so the test uses the tool call id as the task id. For a root command they're the same.

  • Fixture helpers: the fixture records tool calls, and createAuthenticatedFixture can declare AIR capabilities.

Test plan

  • npx tsc --noEmit
  • E2E workflow on this branch: run 38406199843. 33 passed, 2 skipped. One retry, in the exec-policy amendment test: the model mistyped the long temp path, so the saved approval rule didn't match.

Switch the default e2e model to gpt-6-luna: gpt-5.2 is no longer in the
Codex catalog, so it ran on fallback metadata without the apply_patch tool
and edited files through the shell instead.

Ask for escalated permissions up front in the shell approval prompts so
the model does not run the command sandboxed and skip the approval.
Reuse the escalation hint for read-only shell tests that expect an
approval, and wait for the execute tool call before cancelling so the
interrupt does not race Codex registering the process.
Codex keeps a running unified exec command as a background terminal
when the turn is interrupted. Assert that cancel ends the turn, then
stop the command with _session/async_task/stop and check it exits.
The SDK client rejects the async_task_spawned update, so the test cannot
read the task id from it. A root command's async task id is its tool
call id, so retry the stop request with it until the task is published.
@EugeneTheDev
EugeneTheDev merged commit 05cc8a4 into main Oct 13, 2026
7 checks passed
@EugeneTheDev
EugeneTheDev deleted the eugenethedev/fix-e2e-tests branch October 13, 2026 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant