Skip to content

Add binary E2E coverage and fix exposed runtime regressions - #226

Merged
capy-ai[bot] merged 4 commits into
mainfrom
capy/add-binary-e2e-coverage
Sep 21, 2026
Merged

capy-ai[bot] merged 4 commits into
mainfrom
capy/add-binary-e2e-coverage

Conversation

@CuriosityOS

Copy link
Copy Markdown
Collaborator

What and why

Adds a standalone black-box suite that accepts a BetterWright executable or tests betterwright on PATH. The suite imports no target source/package internals, so it can exercise a built checkout or a separately installed release.

  • 100 independently isolated cases cover CLI input/configuration, browser actions and observations, sessions/profiles, artifacts/recording, network and sandbox restrictions, credentials, MCP, local mock-model agent execution, stealth, provider fallback, timeouts, and live view.
  • Each case uses a fresh home/profile/vault and local fixtures. Inherited credentials and preload hooks are excluded; synthetic secrets are redacted from diagnostics. No real cloud accounts, native personal cookie stores, installers, or paid services are used.
  • Adds command/case deadlines, cleanup, group/filter selection, --require-all, and atomic JSON reports with explicit failures, skips, incomplete runs, and coverage limits. The existing Node tests job runs the strict suite and uploads its report.
  • Documents usage and limitations in docs/binary-e2e.md; the suite complements rather than replaces unit, type, platform, and Electron tests.
bun run test:e2e
bun run test:e2e /absolute/path/to/betterwright
bun run test:e2e --binary bun --binary-arg "$PWD/dist/bin/betterwright.js" --require-all

Regressions exposed and fixed

  • human.type(..., { clear: false }) now positions the caret at the end for normal typing as well as fallback insertion.
  • History navigation waits for completed document readiness after commit, including BFCache restores that do not emit a new DOMContentLoaded/load event.
  • MCP explicitly observes stdin EOF/close and closes both its transport and browser instead of depending on an SDK close callback that EOF never invoked.
  • Live-view warm-up preserves setContent pages whose URL remains about:blank.
  • The stealth worker directly selects patchright-core; Bun 1.4's handling of Node loader registration no longer silently leaves the stock driver active.

Verification

  • Standalone copy of tests/e2e/, run outside the checkout against betterwright on PATH with Bun 1.4.0 and --require-all: 102 passed, 0 failed, 0 skipped (100 cases plus 2 prerequisite checks).
  • Related existing real-browser regressions: 17 passed, 0 failed.
  • Focused harness, navigation, MCP, human-input, and stealth unit tests: 72 passed, 0 failed.
  • Credential redaction case additionally passed three consecutive synchronized runs.
  • bun run lint, bun run typecheck, bun run test:types, bun run build:harness, bun run check:versions, and bun run check:build pass. Lint retains existing warnings.
  • Final rebuild produced byte-identical dist/ files to those exercised by the standalone run.

Public API declarations and dependency pins are unchanged. Required CI display names are preserved; the new artifact action is SHA-pinned. No test imports the worker entrypoint. The separately requested local worker-refactor plan is excluded from Git and is not part of this PR.

Open in Capy

@greptile-apps

greptile-apps Bot commented Sep 21, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

The reviewed runtime changes have focused execution evidence for browser history and editable-control behavior, and no confirmed merge-blocking issue remains.

What we checked:

  • Exercised the MCP server startup with input already closed and verified that the server cleanup and browser cleanup handlers ran, and the process exited successfully. T-Rex
  • Built the browser validation distribution and exercised same-document navigation with BFCache restoration, confirming readiness states completed and no timeout occurred. T-Rex
  • Rendered editable controls with carets at the beginning and ran the typing harness with clearing disabled, confirming text was appended to all target controls. T-Rex
  • Validated the post-connection setup and observed server-close and browser-close events with exit code 0, noting that a real StdioServerTransport is required for a complete stdin EOF event. T-Rex
  • Ran the focused same-document BFCache validation and observed ok:true, readyState 'complete', and nonzero navigation timings with timeout 0. T-Rex

Summary

  • Adds a standalone black-box binary E2E framework and runs its strict suite from CI with uploaded reports.
  • Updates editable-control typing, browser-history readiness, MCP shutdown cleanup, populated about:blank live-view handling, and stealth driver selection.
  • Focused browser checks confirmed history traversal and end-append typing behavior in real Chromium. MCP cleanup was confirmed when standard input was already closed before connection.

No issues requiring changes before merging were found.

Reviews (1) · Last reviewed commit: "Keep E2E additions under Unreleased afte..."

@capy-ai
capy-ai Bot merged commit 029b639 into main Sep 21, 2026
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant