test(onboarding): end-to-end funnel telemetry harness - #1050
Conversation
Opt-in via ALTIMATE_E2E=1; skipped otherwise, so it cannot flake anyone's
commit. Two scenarios, ~27s total.
Every other telemetry test in this repo spies inside one process, which cannot
see the thing that actually matters: the TUI renders on the main thread and the
server runs in a Bun Worker, each with its own Telemetry module instance and its
own buffer. This spawns the real CLI in a PTY with its telemetry endpoint
pointed at a local sink, drives it with keystrokes, and asserts on the envelopes
that arrive over HTTP.
It has already earned its place — it caught `launch_id` differing between the
two threads, which made the correlation id useless for the exact join it exists
to provide. No single-process test could have seen that.
Covered:
- A full first run through Big Pickle: the eight Part 1 events, their
properties, and that every event across both threads and two different session
ids shares one launch_id.
- Quitting at the picker: `onboarding_abandoned{last_stage: model_picker}`
arrives after process exit, which is the only proof the flush-on-exit path
delivers.
Notes for anyone extending it:
- A throwaway HOME is essential. A developer machine has credentials, so the
first-run gate never opens and the funnel never starts.
- The CLI must run from packages/opencode, as the `dev` script does — the JSX
runtime comes from the workspace bunfig.toml, and bun only picks it up from
there. The project under test is passed as the positional argument.
- Steps are paced with sleeps, not by waiting on events or screen text. Events
arrive on the flush interval and lag the UI by seconds; terminal output is
per-cell ANSI, so a visible label is split across escape sequences and never
matches a substring search.
- bun-pty, not @lydell/node-pty: the latter loads a platform-specific native
package that is absent in this workspace and fails silently — no output, no
error, no exit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
1 participant
Adds an opt-in end-to-end test for the onboarding funnel telemetry. Stacked on #1049 — based on that branch, so the diff here is just the harness.
Why this exists
Every other telemetry test in the repo spies inside a single process. That cannot see the thing that actually matters:
altimate-code tuirenders the TUI on the process main thread and runs the HTTP server in a BunWorker, and each thread loads its own instance of the telemetry module with its own buffer.This spawns the real CLI in a PTY with its telemetry endpoint pointed at a local sink, drives it with keystrokes, and asserts on the envelopes that arrive over HTTP.
It has already paid for itself: it caught
launch_iddiffering between the two threads, which made the correlation id useless for the exact join it was added to provide (fixed in #1049). No single-process test could have seen it, and two plausible fixes were wrong before the third worked — each ruled out by running this, not by reasoning.Coverage
launch_id. Big Pickle needs no signup, so the run stays entirely local.onboarding_abandoned{last_stage: model_picker}arrives after process exit. This is the only proof that the flush-on-exit path delivers.Running it
Skipped without that flag, so it cannot flake anyone's commit. Two scenarios, ~27s.
Notes for anyone extending it
Four things cost me time here and are documented in the code so they don't cost the next person the same:
HOMEis essential — a developer machine has credentials, so the first-run gate never opens and the funnel never starts.packages/opencode, as thedevscript does. The JSX runtime comes from the workspacebunfig.tomland bun only picks it up from that directory; the project under test goes in as the positional argument.bun-pty, not@lydell/node-pty— the latter loads a platform-specific native package that is absent in this workspace and fails completely silently: no output, no error, no exit.What it does not cover
Gateway OAuth (needs a real browser and account), the activation events beyond the menu (need a live model), and rare failure paths unless deliberately induced.
🤖 Generated with Claude Code
https://claude.ai/code/session_018SLUQF3xgZHsGZHSjxe7vb
Summary by cubic
Add an opt-in end-to-end telemetry test harness for the onboarding funnel by launching the real CLI in a PTY and capturing events via a local HTTP sink. It verifies a single
launch_idacross threads and thatonboarding_abandonedis flushed on exit.New Features
launch_idacross TUI and Worker sessions.Dependencies
bun-pty@0.4.8.Written for commit 0d7bc17. Summary will update on new commits.