Skip to content

Interactions SDK 1.1.0: keep customer data out of test IDs - #6

Merged
lightninglu10 merged 3 commits into
mainfrom
sdk-testid-filter-d67809d2
Nov 5, 2026
Merged

lightninglu10 merged 3 commits into
mainfrom
sdk-testid-filter-d67809d2

Conversation

@lightninglu10

@lightninglu10 lightninglu10 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Built

  • testId?: (value: string, element: Element) => string | null option. It runs wherever a test ID or interaction ID is read: the snapshot test_id dictionary, a click's control_id, control_id on UI state events, and so a finding's stable_id and fingerprint.
  • The option is on InteractionOptions. The browser package has no CaptureOptions type; that name belongs to the Playwright helper, which gets the option too. The signature is exactly as specified.
  • defaultTestIdFilter, exported and used when there is no hook. It drops any value containing a ., and any value with a segment that has a digit or is longer than 32 characters. A hook replaces it.
  • data-interaction-id follows the same rule. When it is dropped, the element's data-testid is tried next, matching how an invalid data-interaction-id already behaved.
  • A hook that throws, or returns a non-string, non-null value, drops the ID. Nothing is logged: the SDK has no debug channel, so this path writes nothing at all.
  • A hook's result must still pass the existing identifier shape check. The hook is also part of the shared-instance identity, like routeTemplate.
  • @codepress/interactions-playwright: testId on captureSnapshot, findIssues and expectNoFindings. The function is serialized and runs in the page, so it must be self-contained, like a page.evaluate callback. A hook that uses a name missing from the page fails the capture instead of silently dropping every ID, and it can call globalThis.__codepressInteractions.defaultTestIdFilter. Its code changed, so it is bumped to 1.1.0.
  • @codepress/interactions-react passes options through whole, so it needs no code change and stays at 1.0.0. Because that ends the "all three share one version" rule, CHANGELOG and CONTRIBUTING now say a release bumps only the packages that changed.
  • Fixes from independent QA of the first head:
    • A hook that writes to the DOM no longer freezes the page through the mutation observer. When the callback has run a hook, it discards the records pending at its end, which can only be the hook's own writes. The hook still runs on every state event, so a hook that starts dropping an ID, for example after consent is withdrawn, takes effect at once.
    • A rejecting async hook no longer reaches the page as an unhandled rejection, which had also produced false click.error_after_click findings.
    • A click's tag path control_id is cut to the protocol's 128 characters. The long path existed before, but dropping an ID made more controls hit it, and an invalid event can cost the whole batch.
    • initInteractions no longer joins a runtime from an older SDK copy on the page, which would ignore the hook.
  • packages/browser bumped to 1.1.0 (VERSION too), CHANGELOG entry added, and a "Test IDs and privacy" section added to the browser README. Root, protocol and Playwright READMEs updated.
  • Golden samples regenerated. Their dotted IDs (checkout.pay) became hyphenated (checkout-pay) because the default now drops dots. No schema change.
  • Test IDs in existing tests that the default would drop were renamed. That includes private.indicator, which would otherwise have passed for the wrong reason in the data-interactions-exclude test.

Differs from the brief:

  • The brief says a dropped ID falls back to "role, name and path". The existing fallback is the tag path only (body/button, html/body/button, or #ancestor/div/button), and this PR keeps it. See the note under Could not verify.
  • The brief's domain fixture is replaced with domain-row-acme.com, which has the same shape, because this repo's examples must stay generic.

Verified

Run locally on the current code: browser build, pnpm typecheck, pnpm test (87 passed) and pnpm test:browser (browser 74 passed and 1 skipped, the opt-in golden regeneration; Playwright helpers 10 passed).

The first head was also checked independently against packed tarballs in real Chromium. That run and a rerun failed, finding the defects fixed above. See the QA reports on this PR for the result on the current head.

  • Default fixtures: row-3f2a9c1e-…, domain-row-acme.com and item-42 are absent from the snapshot dictionary, click control_id and finding stable_id; save-button and nav:settings are kept. Unit tests are in test/test-id.test.ts. test/e2e/privacy.spec.ts proves it in real Chromium against the ui.finding events the SDK actually sends.

  • Negative, hook returns null for everything: no test ID in any payload, and stable_id and fingerprints match across two different snapshots of the same elements (unit test).

  • Negative, hook throws: capture continues, the ID is dropped, and console.log/info/warn/error/debug are never called (unit test). A non-string return is also covered.

  • Hook replaces default: a hook returning the value unchanged keeps item-42 (unit test, plus Chromium and the Playwright helper).

  • npm publish --dry-run: passed locally and in the CI Publish dry run step of publish-browser.yml on the first head, printing + @codepress/interactions-browser@1.1.0. It runs again on every push.

  • Regression tests catch the fixes: reverting the rejection, tag path and older-runtime fixes one at a time made their tests fail. Removing takeRecords failed all 6 Chromium freeze tests.

  • Per-record split fixed: in Chromium the three data-bearing buttons now produce one finding instead of three.

  • Overhead (pnpm --filter @codepress/interactions-browser bench, 5 runs, same machine, before → after). Body size falls because the bench page's card-N IDs are now dropped, and that also lightens the walk, so the table is not a pure measure of the filter. On a page where every ID is kept, independent QA measured about 2% more walk time (27.1 → 27.7 ms at 1x CPU, 115.6 → 117.9 ms at 4x, 15 runs each), and a slow-ish hook costs about 20%:

    Before After
    Walk, 5,000 elements, 1x CPU 40.6 ms 31.9 ms
    Walk, 5,000 elements, 4x CPU 144.5 ms 125.3 ms
    Longest slice, 5,000 elements, 4x CPU 7.32 ms 7.9 ms
    Body, 5,000 elements 350,995 B 347,484 B
    Main bundle, minified and gzipped 24,641 B 24,779 B

Could not verify

  • npm view @codepress/interactions-browser version showing 1.1.0: this can only be checked after merge.
  • That the Playwright package publishes on the same merge. publish-playwright.yml skips with a warning if it runs before browser 1.1.0 is on npm, and then needs a re-run. That is the documented process, but the order is not guaranteed.
  • That fingerprints stay distinct for similar elements once IDs are dropped. Same-shaped siblings now share one tag path, so they share one fingerprint. That merges per-record duplicates as intended, but it also merges two genuinely different elements with the same path when their IDs are dropped.
  • That the default catches every data-bearing ID. It cannot: a slug (project-acme-launch), a name (row-JaneDoe) or a random ID with no digit is kept. The README says so and shows an allowlist hook.
  • A hook that writes to the DOM later than its own call. takeRecords only sees writes made during the callback. In Chromium a write from queueMicrotask or a promise .then still freezes the page, and one from setTimeout keeps calling the hook without freezing it. Hooks must be free of side effects, and the README says so.
  • A hook that adds an element with an ID on every call. The observer stops, but the walk keeps finding the new elements until its element limit. In QA the page stayed responsive, the snapshot came back truncated and 761 events were dropped.
  • That every Playwright hook mistake is reported. A name the page does not have fails the capture. These still drop the ID with no error, and are accepted because they send less: a test variable named after a page global (status, name, location), a method that loses this, a property of a missing page object, an async hook, and a generator hook. The README says to check the snapshot's IDs when writing a hook.
  • Follow-ups left open: globalThis.__codepressInteractions has no global type, so the documented call gives TS7017 in a strictly typed test file, and a cut tag path with no slash reports structural_id: false, which needs a control tag of 128 or more characters.
  • Browsers other than Chromium, and how a real receiver treats batches. The tests use a memory sink and intercepted requests.

@codepress-reviewer-dev

Copy link
Copy Markdown

Starting initial review of 3640dcad from baseline b12f53b6... 👀

View automation run

@codepress-reviewer-dev codepress-reviewer-dev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR adds a testId hook and a privacy-conservative defaultTestIdFilter. Every read of data-interaction-id and data-testid now goes through one helper, elementTestId. That covers the snapshot dictionary, click control_id, UI state events and, through them, finding stable_id and fingerprints.

Architecture

The shape is sound (KEEP_CURRENT). Replacing the three separate safeIdentifier(...) ?? safeIdentifier(...) read sites with one helper makes it hard for a new code path to skip the filter. A hook that throws or returns a non-string drops the ID silently. The result still has to pass the existing identifier shape check. The hook is part of the shared-instance identity, so two callers cannot silently share a runtime with different filters.

Checks

  • The Playwright helper serializes the function and evaluates it in the page, with a fallback for method shorthand. A non-function result throws a clear error. This is documented as page.evaluate-style self-containment.
  • Tests cover the default fixtures, a null-returning hook, a throwing hook with no console output, a non-string return, and a hook that replaces the default. There is also a Chromium e2e against the real ui.finding events. These guard real invariants.
  • The golden samples and protocol files change only because the dotted IDs became hyphenated. There is no schema change.
  • The version bump and the new rule that a release bumps only the packages that changed are consistent across CHANGELOG, CONTRIBUTING and package.json.

One nonblocking suggestion is attached: the default filter is broad on digits, and an allowlist example in the README would help apps with static IDs like step2.

The PR is cohesive and does not need splitting.

Reviewed `b12f53b6`...`3640dcad`. Diff collection and GitHub posting were handled by the automation.

View automation run

@codepress-reviewer-dev

codepress-reviewer-dev Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Review head 29be39460722c4f01757782c7d92b641e39a0c55; round 3; 24 reviewed file(s), 0 pending/skipped file(s). 1 subsystem record(s) retained; 0 omitted.

View automation run

@codepress-reviewer-dev

Copy link
Copy Markdown

Starting new changes review of da5e309c from baseline 3640dcad... 👀

View automation run

@codepress-reviewer-dev codepress-reviewer-dev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This follow-up (3640dcad...da5e309c) addresses the defects independent QA found on the first head. I re-read each change against the protocol schema and the surrounding code.

Architecture

KEEP_CURRENT. The single elementTestId helper remains the one place IDs are read. The new observedId cache is a small WeakMap keyed by the two attribute values, which stops a hook that writes aria-busy or disabled from re-queuing mutation records forever. It adds no new coordination machinery.

Checks

  • Observer freeze: the hook now runs once per element and attribute-value pair. Changed IDs are still picked up, because the cache key includes both attributes.
  • Async hook: attaching a no-op rejection handler to a thenable result stops an unhandled rejection reaching the page. The result is still dropped as a non-string.
  • Tag path control_id: slice(-128) matches the schema's maxLength: 128 (schema lines 808-811), and keeping the tail keeps the part nearest the control.
  • Older runtime: the identity length comparison closes the gap where some() over a shorter array would match. The second caller gets an inactive handle, as documented.
  • Playwright helper: a ReferenceError from the hook now fails the capture with a clear message, instead of every ID being silently dropped. Bound and native functions get an explicit error. defaultTestIdFilter is exposed on the page global for hooks to call. A hook that catches its own ReferenceError would hide the failure, which is a narrow edge case.
  • Tests: the new tests exercise the behaviors fixed (observer termination, rejection, long path, older-runtime join, Chromium privacy, helper failure), and the PR description says they fail when each fix is reverted.

No actionable findings. Docs and CHANGELOG match the code.

Reviewed `3640dcad`...`da5e309c`. Diff collection and GitHub posting were handled by the automation.

View automation run

@lightninglu10

Copy link
Copy Markdown
Contributor Author

QA re-verification: PR #6, quantfive/codepress-interactions

PR Head SHA: da5e309caf6bdfabe5b4e7e7ad02e7280eebd304

  • Previous verdict: FAIL on 3640dca. Base: origin/main b12f53b. PR head unchanged at the end of the run.
  • Plane: local, real Chromium (Playwright pinned to 1.64.1, chromium_headless_shell-1248, the same as the first run), consumers built from packed tarballs. Node 26.13.0, pnpm 10.28.2, uv 0.9.28, npm 11.20.0.
  • The diff 3640dca..da5e309 was read directly and matches the coordinator's summary. One addition: filteredId gained a .then check, so the walk's code path did change slightly (see B12).
  • Nothing posted, committed, pushed or published.
  • Verifier: the qa-verifier agent type is not installed in this environment, so a general-purpose agent ran as an isolated verifier with the same contract. report_verify_result is not available here either.
  • Posted as the verifier wrote it, except that one internal fixture string in the D4 row is redacted because this repository is public. Artifact paths are local to the verification machine.

DB Performance

DB performance verifier skipped: no backend DB-backed performance surface. Inventory of the diff: packages/browser, packages/playwright, protocol docs and generated schema descriptions, README and CHANGELOG files. No backend API routes, WebSocket handlers, Celery tasks, ORM or query helpers, migrations, SQL, queue consumers, session persistence or aggregation code.

Screenshot Status

Not applicable: the diff has no frontend or UI change. This is an SDK with no rendered interface, so no screenshots were captured.

Overall verdict: FAIL

Much narrower than last time. B1 to B10 and B12 pass; B10 was a FAIL on 3640dca. Two things still fail, and the contract needs every row to pass:

  • B11: the rewritten Playwright README says using a variable from the test file fails the capture. That is false when the variable's name matches a page global such as status, name or location: the capture resolves with no IDs and no error.
  • Named edge "hook that mutates the element": the original D2 hooks are fixed, but 7 other hook shapes still freeze the page.

There is also one decision for the owner: N1, a behaviour change introduced by the D2 fix.

Status of D1 to D7

ID Status Evidence
D1 Partly fixed Fixed: the closure now fails the capture (testId runs in the page and must be self-contained: allow is not defined), on all three helpers, when only one element reaches the name, and on every repeat. A good hook works straight after, and the default after that. The README rule written out now matches production: 2dddd95e 7aa0f5c4 e652fd1c on both sides. Not fixed: only ReferenceError is caught, so 6 other mistakes still return ids: [] with no error (list below).
D2 Partly fixed Fixed: all 4 original hooks stay responsive with 1 hook call. Also safe: writing to another element, toggling another element, writing to every ID element, re-appending itself, constant data-testid plus aria-busy, and writes during a click. Not fixed: 7 shapes still hang (list below). All 3 sampled also hung at 3640dca, so they are remaining cases, not regressions.
D3 Fixed Same control: 158 chars and invalid at 3640dca, 128 chars and valid now. Boundaries at path lengths 94, 122, 126, 128+ all valid. All head payloads validate. Side effect in N3.
D4 Fixed The old fixture string has 0 matches in the checkout. README, util.ts and tests use domain-row-*. Limit: this only proves that string is gone. Per the new rule I did not compare against the private repo, so other internal IDs were not audited.
D5 Fixed 8-case matrix below. A hook is no longer ignored in any order tested.
D6 Fixed 10 of 10 promise-like shapes: async throw, Promise.reject, subclass, iframe-realm promise, 5 thenable variants, throwing Proxy. 0 unhandled rejections, 0 pageerror, 0 console, 0 false error_after_click, ID dropped, capture continues. Was 14 pageerrors.
D7 Partly fixed Fixed: bound and native functions reach the friendly error. Class: limitation as agreed, since tsc --strict rejects it. Generator: a typed path exists. function* (): any { ... } compiles against both initInteractions and captureSnapshot, then silently drops every ID. Without the : any annotation it is rejected. By the coordinator's rule that makes it a defect, though a very weak one, since any is an explicit escape hatch.

D1: what still drops every ID silently

All of these resolve with ids: [], 0 errors. Repro: probes/b8/attack.spec.ts; the compile column is from probes/ts/attack2.mts under tsc --strict.

Hook mistake Why it escapes Compiles without any?
Test variable named status Page global exists, so .has is a TypeError Yes
Test variable named name Same Yes
Test variable named location Same Yes
Object method using this this is lost, TypeError Yes
Property of a missing page object (window.appRules.has) TypeError Yes, with a typed cast on window
Async hook closing over a test variable The rejection never reaches the wrapper's try No, needs a cast

D2: what still hangs

Each case ran in its own browser and was killed by its own handle. Repro: probes/attack-memo.mjs.

Hook Trigger Result
Alternates own data-testid between two values, plus aria-busy Snapshot Hangs
Changes own data-testid every call, plus aria-busy Snapshot Hangs
Alternates own data-interaction-id, plus disabled Snapshot Hangs
Alternates another element's data-testid, plus aria-busy Snapshot Hangs
Adds a new busy element with an ID on each call Snapshot Hangs
Replaces itself with a busy clone Snapshot Hangs
Alternates own data-testid, plus aria-busy, armed only for a click Real click Hangs

Cause: the memo keeps one key per element, so a changing ID always misses, and a new element has no entry at all.

These are proven unbounded, not slow. With a cap added to the hook, the page ran 20,002 hook calls in 38 ms (alternate), 21,629 in 92 ms (spawn) and 20,002 in 79 ms (clone), and recovered only because of the cap (artifacts/attack-compare.jsonl).

All of these break the README's "free of side effects", so the trigger is narrower than the original D2, where a single constant attribute write was enough. A per-task limit on hook calls from the observer would close every shape at once, and could also replace the memo, which would remove N1.

D5 matrix

Runtime on page Caller Second handle Result
1.0.0 1.1.0, drop-all hook Inactive Fixed. item-42 still recorded by the 1.0.0 caller's own runtime, which this PR cannot change
1.0.0 1.1.0, no hook Inactive As the CHANGELOG says
1.1.0, no hook 1.0.0 Shared Safe: default applies, item-42 dropped
1.1.0, hook 1.0.0 Inactive Not shared
1.1.0, no hook 1.1.0, no hook Shared Unchanged
1.1.0, hook 1.1.0, same hook Shared Unchanged
1.1.0, hook 1.1.0, different hook Inactive Unchanged
1.1.0, no hook 1.1.0, hook Inactive Unchanged

In every case an inactive handle did not throw, shutting it down did not stop the first caller, and re-init after shutdown applied the right rule. Repro: probes/attack-d5.mjs.

New findings

N1 (medium, decision needed): the D2 memo can keep sending an ID the hook now drops

The memo keys only on the two ID attributes. A hook whose answer depends on anything else keeps its first answer on state events, while clicks and snapshots use the current answer. This is new in da5e309.

Hook Busy events at 3640dca Busy events at da5e309
(v, el) => (el.closest("[data-private]") ? null : v), element moved into the private section save-button, then null save-button on every event
(v) => (window.consent === false ? null : v), consent withdrawn save-button, null save-button, save-button

On the same page the click recorded the tag path and the snapshot dictionary was empty, so the three surfaces disagree. It fails toward sending, and the hook's element argument invites exactly this kind of check.

The new README line "return the same answer for the same element" does cover this, so it can be accepted as a disclosed trade-off. It tells authors what to do but not what happens otherwise, and "on clicks and UI state changes" is now imprecise. The default filter is not affected: it depends only on the value, and 8 attribute changes tracked correctly with nothing dropped reaching the wire.

Repro: probes/attack-stale.mjs, probes/attack-stale-old.mjs.

N2 (low): the documented Playwright pattern does not type-check

globalThis.__codepressInteractions.defaultTestIdFilter(value), as the README writes it, gives TS7017 under strict in a TS test file, because the helper declares no global type. It runs fine, since Playwright does not typecheck, so this only affects projects that run tsc over tests. Repro: probes/ts/attack.mts:18.

N3 (low): a cut path with no slash is reported as non-structural

structural is controlId.includes("/") (timeline.ts:48). When the control's own tag is 128 characters or more, the cut leaves no slash, so structural_id is false for a tag path. Observed on a 145-character tag. The batch stays valid and the subject is null, so only the evidence flag is wrong. Very unlikely in practice. Repro: probes/attack-d3.mjs.

N4 (cosmetic): memo key collision on the literal value null

A missing attribute prints as null in the key, so data-testid="null" followed by removing the attribute keeps emitting null. Nothing sensitive can reach the wire this way.

Per-row results

Row Evidence class Verdict Observed Artifact
B1 Default LIVE, wire + memory PASS 26/26. domain-row-acme.com, row-3f2a…, item-42, acme absent from raw and gunzipped bytes; keep-all control sends all five artifacts/b1-results.jsonl, artifacts/b1/
B2 Hook null LIVE + packed /detect PASS Part of 57/57 in b2to6; no IDs anywhere, snapshots differ, fingerprints identical artifacts/b2to6-results.jsonl, artifacts/b2/
B3 Hook failure LIVE, console + pageerror PASS 20 failing hooks, 0 console, 0 pageerror. Rejecting async hook now clean (D6) artifacts/b2to6-results.jsonl, artifacts/attack-d6.jsonl
B4 Replace LIVE, README block verbatim PASS order-row collapse, (v) => v keeps item-42 on all three surfaces artifacts/b4/
B5 interaction-id LIVE PASS All 7 cases artifacts/b5/
B6 Every read site LIVE, raw bytes + keep-all PASS, see N1 Zero fixtures in default bytes; b6b 5/5. The two observer sites now go through observedId, which still calls elementTestId. N1 applies only to hooks that change their answer artifacts/b6/, artifacts/b6b-results.jsonl, artifacts/b6-read-site-grep.txt
B7 Consumers LIVE + tsc --strict PASS tsc passes under bundler, nodenext, node10; wrong usages rejected; ESM and CJS OK; React StrictMode 4/4 with packed and published react artifacts/b7/, artifacts/b7-react-results.jsonl
B8 Playwright helper LIVE, scratch project PASS, D1 partly fixed Default and all hook styles honoured on all three helpers. CSP identical to origin/main below. Named edges reported above artifacts/b8-observations.jsonl, artifacts/b8-attack.jsonl
B9 Artifacts Tarball inspection PASS browser 1.1.0, playwright 1.1.0, react 1.0.0; VERSION and sdk_version 1.1.0; LICENSE, README, entry points present; workspace: rewritten to ^1.1.0; no env or credential files; npm publish --dry-run exit 0 for browser and playwright artifacts/b9/
B10 Schema and regression Commands + LIVE PASS Head payloads all valid. Schemas and generator unchanged. Goldens unchanged since 3640dca and differ from main only by IDs, version, timings and derived hashes artifacts/b10/
B11 Docs truth Executed claims FAIL Playwright README claim false for page-global names (D1). The rest verified, list below artifacts/b8-attack.jsonl, artifacts/claims2-results.jsonl
B12 Overhead LIVE bench, 5 rounds PASS Below artifacts/b12-bench.log
Named edge cases LIVE FAIL "Hook that mutates the element" fails (D2). All others pass: edges 47/47, ancestor 2/2, older runtime now passes artifacts/edges-results.jsonl, artifacts/attack-memo-hang.jsonl

Checks across all probes: 198 pass, 9 fail, plus 201 observation rows. The 9 are D2 (7), N4 (1), N3 (1).

B10 counts

Command Result
pnpm test 83 passed
pnpm typecheck exit 0
pnpm build exit 0 at both revisions
pnpm test:browser, 3 runs each 71 passed, 1 skipped, plus 10 Playwright; no flake
pack:check, 3 packages exit 0
npm publish --dry-run --ignore-scripts --access public, browser exit 0
uv run pytest -q 13 passed
uv run ruff check src tests exit 0
uv build exit 0
CI on this head all pass, publish jobs skipped

Schema validation of every dumped payload: 85 batches, 100 snapshots, 1 invalid. That one is extra/longpath-main-none/batch-0.json, the deliberate origin/main control, which shows the original bug still exists at base. Head has 0 invalid, including all long-path and emoji-tag cases.

B11 claims

True when executed:

  • The README allowlist block, verbatim, keeps exactly save-button, nav:settings, step2.
  • project-acme-launch, row-JaneDoe and a digit-free random ID are kept by the default; step2 is dropped.
  • CHANGELOG: the tag path is cut to 128 keeping the end nearest the control; an older runtime is not joined; the second caller gets an inactive handle; a name missing from the page fails the capture.
  • claims.mjs 9/9, covering the first run's table and CHANGELOG claims.

False: "using a variable from the test file fails the capture", for variables named after page globals.

B8 CSP comparison

Policy origin/main, no hook head, no hook head, hook
script-src 'self' works works works
default-src 'none' works works works
Trusted Types works works works

findIssues counts match at both revisions (3, 0, 3). No new limitation. The helper's new code runs inside page.evaluate, as before.

B12 numbers

One short rerun as agreed: 5 fresh-page runs per variant, interleaved, median of 5 timed walks after 3 warm-ups, 4,996 kept static IDs, ms. The runtime-path bench was not rerun. Other agent sessions were active on the machine.

Variant 1x CPU 4x CPU
origin/main, kept IDs 28.3 119.1
head default, kept IDs 28.5 123.6
head, identity hook 27.7 119.4
head, hook with inline default rule 28.6 121.7
head, slow-ish hook 32.2 138.5
origin/main, no IDs (floor) 27.0 115.4
head default, all dropped 28.5 122.2

The default adds 0.2 ms at 1x and 4.5 ms (about 3.8%) at 4x. The first run measured 2% at 4x with 15 runs. Five runs cannot show whether the .then check contributed, but neither figure is meaningful for a sliced background walk.

Limitations (not defects)

  • Class hooks drop every ID silently; TypeScript rejects them.
  • Cut paths can merge. Controls with different ancestors can cut to the same control_id. Same family as the disclosed sibling merge.
  • Cut can split a surrogate pair with an emoji tag name over 128 units. control_id then starts with a lone surrogate. JSON and the schema accept it; a strict UTF-8 receiver might not. Not tested against one.
  • Older walker already in the page. If a 1.0.0 PAGE_SCRIPT is present, the helper reuses it, so hooks and the default are both ignored and all 6 IDs come back. This is D5's analogue in the helper, and needs two helper versions on one page. The result stays local to the test.
  • Rejections the SDK cannot see still surface, such as a floating Promise.reject() inside the hook or one inside a returned array. Those are ordinary app bugs.
  • Inactive handles are silent; getDiagnostics().enabled is the only signal.
  • The first run's limitations stand, and the README now names mixed-case and digit-free random IDs.

Gaps

  • qa/roundtrip.py not run, as declared.
  • Firefox and WebKit not run.
  • D4: not compared against the private repo, per the new rule.
  • A real receiver's handling of a lone surrogate is untested.

Corrections to my own probes

No failure below came from the SDK:

  • The first full pass exited 1 because zsh does not word-split env vars.
  • npm install picked Playwright 1.65.0; pinned back to 1.64.1.
  • B4's README extraction took the new allowlist block first; now targets the order-row block.
  • parity.spec.ts closed over the import, which now fails loudly as intended; rewritten to the README's written-out rule.
  • attack-stale.mjs first counted structural_change events, producing false FAILs; now filters to busy.
  • My ignore attack assumed ignored elements skip the hook. They are walked and flagged, as at main.
  • bench.mjs exited 1 after its last table because QA_ARTIFACTS was unset, so b12-bench.json was not written. The log has the numbers.
  • The first run's "writes hidden" hang label is corrected to aria-expanded.

Hard rules

  • No pkill, and no pattern-based kill of any kind. hang.mjs's old pkill -P was removed.
  • Every probe browser came from launchServer with its PID logged: 44 started, 44 killed by handle (artifacts/browser-pids.log). No recorded PID or process group remains.
  • Browsers started by the Playwright test runner (B8, pnpm test:browser) are managed by that runner and were not logged individually. All runners exited and none hung.
  • Nothing read outside the checkouts and $TMPDIR, apart from the Playwright browser cache that Chromium runs from, and GitHub for PR status.

Cleanup status

  • $TMPDIR/qa-pr6b and $TMPDIR/qa-pr6b-main removed with git worktree remove --force, then git worktree prune. All exited 0; neither is in git worktree list.
  • Author worktree: at da5e309, clean, untouched by this run.
  • No process references this run. Repo-commands PID 52569 has exited.
  • Kept: this run directory (143 MB), and the first run's directory. probes/ needs QA_REPO pointing at a fresh checkout; schemas used are in artifacts/protocol/.

@codepress-reviewer-dev

Copy link
Copy Markdown

Starting new changes review of 29be3946 from baseline da5e309c... 👀

View automation run

@codepress-reviewer-dev codepress-reviewer-dev Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This follow-up (da5e309c...29be3946) replaces the observedId memo with a takeRecords() call at the end of the mutation observer callback whenever a custom hook ran.

Architecture

KEEP_CURRENT, and the change is a simplification. The memo was a second source of truth keyed on two attributes. It kept sending an ID after the hook began dropping it, which is the stale-answer problem QA raised. It also missed the changing-ID, spawn and clone hang shapes. The callback is synchronous, so any record pending at its end was written by the hook itself. Discarding those records breaks the feedback loop without caching any answers. The hook now runs on every state event, so a hook that starts dropping an ID takes effect immediately. The README line that required the same answer for the same element is correctly removed.

Checks

  • ranHook is set only when a custom hook exists. The default filter is pure and never triggers the discard, so there is no change for hook-free users.
  • Only records pending inside the synchronous callback are discarded, so page mutations are not lost.
  • The README's "free of side effects" caveat is kept, and the PR description discloses the residual case of deferred writes (microtask or promise).
  • The Chromium privacy and freeze tests and the new unit tests cover the behaviour. The PR description reports that removing takeRecords fails the freeze tests.

No actionable findings.

Reviewed `da5e309c`...`29be3946`. Diff collection and GitHub posting were handled by the automation.

View automation run

@lightninglu10

Copy link
Copy Markdown
Contributor Author

@codepress /judge-verification can you judge this verification?

Final narrow QA: PR #6, quantfive/codepress-interactions

PR Head SHA: 29be39460722c4f01757782c7d92b641e39a0c55

  • Scope: the five checks set by the Orchestrator. Rows outside them stand from the da5e309 report.
  • Previous verdict: FAIL on da5e309, for B11 and the named edge "hook that mutates the element", with N1 open.
  • Plane: real Chromium (Playwright 1.64.1, chromium_headless_shell-1248), consumer built from tarballs packed at this head.
  • git diff da5e309..29be394 read directly. It matches the description: the memo is gone, takeRecords() runs at the end of the observer callback when a hook ran, both READMEs changed, and there is no Playwright source change. The diff also adds tests.
  • Nothing posted, committed, pushed or published.
  • Verifier: the qa-verifier agent type is not installed in this environment, so a general-purpose agent ran as an isolated verifier with the same contract. report_verify_result is not available here either.
  • Baseline: this round is a scoped delta, set by the delivery owner. The other rows are in the FAIL report for da5e309 on this PR (Interactions SDK 1.1.0: keep customer data out of test IDs #6 (comment)), where B1 to B10 and B12 passed.
  • Added after the report was written: "Typecheck, unit and browser tests" has since passed on this head, so every required GitHub check is green. Artifact paths are local to the verification machine.

DB Performance

DB performance verifier skipped: no backend DB-backed performance surface. Inventory of the diff: packages/browser, packages/playwright, protocol docs and generated schema descriptions, README and CHANGELOG files. No backend API routes, WebSocket handlers, Celery tasks, ORM or query helpers, migrations, SQL, queue consumers, session persistence or aggregation code.

Screenshot Status

Not applicable: the diff has no frontend or UI change. This is an SDK with no rendered interface, so no screenshots were captured.

Verdict: PASS for this scope

All five checks pass on the required evidence. Both previous failures and N1 are resolved, and no new defect was found.

# Check Verdict Observed Artifact
1 D2: 7 shapes + 4 original hooks PASS 11/11 responsive, hook count stops moving, a later real write is recorded, 0 pageerrors. Control: all 7 shapes hang on da5e309 artifacts/final-d2.jsonl, artifacts/final-d2-shape5.jsonl, artifacts/control-d2-da5e309.jsonl
2 N1, both directions PASS 8/8. ID stops on state events when denied and returns when allowed; stays on an element allowed throughout. Control: 4 of the 8 fail on da5e309 artifacts/final-n1.jsonl, artifacts/final/, artifacts/control-n1-da5e309.jsonl
3 B6: every read site PASS 14/14 B6 checks. 0 dropped fixtures in raw default bytes; 19 distinct fixtures reach the wire under keep-all artifacts/b2to6-results.jsonl, artifacts/b6b-results.jsonl, artifacts/b6-read-site-grep.txt
4 Normal paths and repo commands PASS 22/22 state checks; 53 state events identical event for event across all 5 variants. All commands exit 0 artifacts/final-state.jsonl, artifacts/b10/
5 B11: rewritten README sentence PASS Each clause executed and true; 7/7 Playwright tests artifacts/b8-final.jsonl, artifacts/b8/readme-testid-row.txt

Totals: 104 checks pass and 0 fail across the live probes, plus the 7 Playwright tests. That includes all of b2to6.mjs, of which only the B6 rows are in scope. All 42 batches and 46 snapshots in this run directory, control payloads included, validate against the schemas.

1. D2

Each case ran in its own browser, killed by its own handle. Hook count was read twice, 800 ms apart, after the capture finished. The queue was then drained, a real aria-busy write made on an element with no ID, and the event looked for.

Hook Trigger Hook calls Elements after Later real write
Alternates own data-testid + aria-busy Snapshot 4, stopped 12 Recorded
Changes own data-testid every call + aria-busy Snapshot 4, stopped 12 Recorded
Alternates own data-interaction-id + disabled Snapshot 4, stopped 12 Recorded
Alternates another element's data-testid + aria-busy Snapshot 4, stopped 12 Recorded
Adds a new busy element with an ID each call Snapshot 6,776, stopped 6,788 Recorded
Replaces itself with a busy clone Snapshot 4, stopped 12 Recorded
Alternates own data-testid + aria-busy, armed for a click Real click 6, stopped 12 Recorded
Toggle aria-busy Snapshot 4, stopped 12 Recorded
Constant aria-busy="false" Snapshot 4, stopped 12 Recorded
Toggle disabled Snapshot 4, stopped 12 Recorded
Constant disabled Snapshot 4, stopped 12 Recorded

The real click landed: page.click resolved, and interaction.started carried alpha-id on the wire.

The spawning hook is the one exception to the small counts. takeRecords closes the observer loop, but the walker keeps finding the elements the hook adds until it reaches its 5,000-element limit. The case took 6.8 s, including 1.2 s of deliberate waits, and no poll of the page hit its 4 s limit. In the dedicated run the snapshot came back truncated and torn after 252 slices, the longest 12.8 ms, and the 1,000-event queue limit dropped 761 events. It is bounded and recoverable, and it needs a hook that breaks the README's "free of side effects".

Negative control. The same probe against the da5e309 tarballs: all 7 shapes stop responding, including the real click, and the 4 original hooks pass, as they did on that head. So the 7 shape results above come from this change. The originals were fixed one head earlier, and their control is the first-round hang.

2. N1

State events taken from the wire in HTTP mode, across allowed, denied and re-allowed phases. T is save-button on the element that changes; K is nav-settings on a second element; - is no ID.

Scenario Busy events, in order Disabled Added status element
Moved into [data-private] and back T K T K, - K - K, T K T K T, -, T ID, none, ID
Consent withdrawn and re-granted T K T K, - - - -, T K T K T, -, T ID, none, ID

In the first scenario K shows the allowed-throughout case next to a changing element. Separately, an element that never left the allowed state carried its ID on all 12 state events. Clicks agreed across the same three phases.

Negative control. The same probe against the da5e309 tarballs fails 4 of 8: busy and disabled keep save-button through the denied phase in both scenarios. The other 4 pass there, because a newly added element, an unchanging answer and clicks were never affected by the memo.

3. B6

Grep at this head: one attribute read of an ID (util.ts:38), with elementTestId called from the click path, both observer sites and the snapshot walk. No dataset or selector-based read. Live coverage: aria-busy, disabled, added and removed nodes, submit, label forward, shadow host, ancestor anchors, and dead and rage findings.

4. Normal paths

The same page script ran against 5 variants: da5e309 and 29be394 with no hook, the same two with (v) => v, and 29be394 with an inline rule. All gave the same 53 events in the same order: busy 20, structural_change 20, status_region 6, progress_indicator 4, disabled 3.

Real writes confirmed recorded after a hook-running callback:

  • In the next task.
  • In the next microtask.
  • Several kinds in one batch.
  • From an app MutationObserver notified before the SDK's.
  • From an app MutationObserver notified after the SDK's.
Repo command Result
pnpm test 87 passed
pnpm typecheck exit 0
pnpm build exit 0
pnpm test:browser, 2 runs each 74 passed, 1 skipped, plus 10 Playwright; no flake
pack:check, 3 packages exit 0

GitHub checks at report time, recorded in artifacts/pr-state-final.json: "Test and pack" and "Python tests and lint" succeeded and the npm publish jobs were skipped. "Typecheck, unit and browser tests" was still in progress, so that check is not part of this verdict; the same commands passed locally as above.

5. B11

The sentence: "a name the page does not have fails the capture. A name the page does have, such as a test variable called status, or any other runtime error in the hook, drops the ID with no error".

Clause Result
Name the page lacks All three helpers reject: testId runs in the page and must be self-contained: allow is not defined
Test variable named status Resolves, no IDs, 0 pageerrors
Hook throws Error Resolves, no IDs, 0 pageerrors
Hook hits TypeError Resolves, no IDs, 0 pageerrors
Hook throws for one element That ID dropped, the other 5 kept, 0 pageerrors
(value) => value keeps every ID All 6 kept
Writing out the rule matches production Fingerprints match for the default, a self-contained hook, and the README rule

Status of earlier findings

ID Status
D2 Fixed
N1 Fixed
N4 Gone with the memo; not separately re-tested
D1, D7, N2, N3 Accepted by the Orchestrator; not re-tested

Limitations

  • Disclosed, not measured by me in this run: a hook that writes later than its own call, through queueMicrotask, .then or setTimeout, is not covered by takeRecords. The coordinator measured this, and it is listed under Could not verify in the PR.
  • Spawning hook: bounded by the walker limit rather than the observer, as described under D2.
  • Observer record budget, unchanged: 20 elements written in one task record 8 busy events at the default maxMutationRecords, the same with no hook, with a hook, and at da5e309. With the budget at 1000, all 20 are recorded.

Corrections to my own probes

The first run showed failures with three causes. None came from the SDK, and each was resolved by evidence rather than by removing the check:

  • Shape 5 "later write not recorded": my write sat behind a full queue and was never flushed. With the queue drained first, it is recorded.
  • N1 sequence failures: moving an aria-busy="true" element fires its own removed and appeared events, which broke my indexing. The scenario now ends idle before moving, and the full ordered sequence is asserted.
  • "All 20 elements in one task": this is the unchanged record budget under Limitations, now asserted directly with the budget at default and at 1000.

Before finalizing I also found that the first D2 run did not record whether the real click landed, and had no negative control. The D2 and N1 artifacts here come from the rerun that adds both.

Rules followed

  • No pattern kill. Every probe browser came from launchServer with its PID logged: 55 started, 55 killed by handle, none running (artifacts/browser-pids.log). Browsers started by the Playwright test runner are managed by that runner; all runners exited.
  • Reads stayed within the checkout and $TMPDIR, apart from the Playwright browser cache that Chromium runs from, and GitHub for PR status.

Cleanup status

  • $TMPDIR/qa-pr6c removed with git worktree remove --force, then git worktree prune. Both exited 0, and it no longer appears in git worktree list.
  • Author worktree at 29be394, clean, untouched by this run.
  • No process references this run.
  • Kept: this run directory. probes/ needs QA_REPO pointing at a fresh checkout.
  • Controls ran after the worktree was removed, against tarballs already in $RUN2/consumer, with the schemas copied to artifacts/protocol/.

@codepress-dev

codepress-dev Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

👋 Hey @lightninglu10! Spawning a verification judge session to help with your request. View session

@codepress-dev

codepress-dev Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

@Terminal @lightninglu10

Dispatched by judge-verification.

Work the Path to merge below. Reuse healthy exact-head verification environments; do not redeploy or restart unless stale or unhealthy.
Don't answer a verification-gap or environment row with new behavior: produce same-head evidence or repair the environment. Don't fix more than the named defect for a code-defect row. Don't commit for advisory findings; file a follow-up.
If this judgment names a repeated finding class on one surface, the path is a restructure of that ownership boundary, not another local guard against the same class. Say which you are doing and why.

Continue only when there is meaningful work to clear the blocker. If the path is already satisfied, the same blocker remains, or another judge pass would add no useful evidence, stop and report that instead of submitting another verification report.

Verification Judgment — PR #6 (local)

Verdict: ❌ NOT READY TO MERGE

PR Head SHA: 29be39460722c4f01757782c7d92b641e39a0c55

Summary: The five rows this report re-ran at the live head (D2, N1, B6, normal paths and repo commands, B11) are credible, and the fix in runtime.ts reads as sound. The block is binding, not a code defect I found. The report is a scoped delta: it says every other row "stands from the da5e309 report", which was itself a FAIL and carries no authenticated verification manifest. With no manifest, no row can be carried, and the delta da5e309..29be394 rewrote both mutation-observer read sites that rows B2 to B5 and B12 exercise. Those rows have no evidence at the live head, so the stale-binding rule applies. The path to green is a current-head report that carries the untouched rows through the composer and supplies fresh rows for the touched ones.

Revision binding

  • Report head: 29be39460722c4f01757782c7d92b641e39a0c55
  • Live PR head: 29be39460722c4f01757782c7d92b641e39a0c55
  • Audited checkout: clean detached checkout at 29be39460722c4f01757782c7d92b641e39a0c55
  • Delta assessment: The report says it re-ran only five checks and that the other rows stand from the da5e309 report. That earlier report (comment 6669815313) ends in verdict=FAIL and neither it nor the triggering report embeds a codepress-verification-manifest, so there is no CARRIED row, no carried_from_head and no carry_justification to authenticate. I fail closed on carrying. For da5e309..29be394 I read the diff directly: 5 files. packages/browser/src/runtime.ts removes the observedIds memo, restores direct elementTestId calls at the three observer sites, and adds ranHook plus this.observer?.takeRecords() at the end of the observer callback. Both READMEs and two test files change. There is no packages/playwright source change, no dependency, lockfile, workflow or schema change. B1 (default filter), B7 (types/consumers), B8 (Playwright helper source) and B9 (artifact contents, apart from README text) are not touched by that delta, but still cannot be carried without a manifest. B2, B3, B4, B5 and B12 run through the changed observer path, and the report did not re-run them at this head, so they are rerun-required.
  • Rows carried forward: none
  • Carried evidence heads: none
  • Rows reverified at live head: 1, 2, 3, 4, 5
  • Rows still requiring verification: B1, B2, B3, B4, B5, B7, B8, B9, B12
  • Publication timing: The report comment names the live head and carries a codepress-verify-result marker at that head. The head is unchanged since.

Diff trigger inventory

Surface Location Coverage
Test IDs are kept out of events and snapshots by default, with a testId hook to replace the rule PR title/body; packages/browser/src/util.ts:38 B1, B2, B4, 3
Hook throwing, rejecting or returning non-strings must not break the page or the capture packages/browser/src/util.ts, runtime.ts B3
data-interaction-id is read through the same filter packages/browser/src/util.ts:38 B5
Hook runs inside the mutation observer without freezing the page (memo removed, takeRecords added) packages/browser/src/runtime.ts:490-603 1, 2
State events keep or drop the ID as the hook's current answer changes runtime.ts:541-577 2
Normal state, structural and status events unchanged by the observer change runtime.ts:490-603 4
Overhead of the default filter and hooks runtime.ts, snapshot walk B12
Consumers: types, ESM/CJS, React packages/browser, packages/react B7
Playwright helper honours default and hook packages/playwright B8
Released artifacts: versions, tarball contents package manifests, READMEs B9
README claim about hooks and Playwright packages/browser/README.md, packages/playwright/README.md 5

Per-item

# Verifier said Judgment Evidence
1 PASS (D2, 7 shapes + 4 original hooks) ✅ PASS Each case in its own browser: 11 of 11 stay responsive, hook count read twice 800 ms apart stops moving, a later real aria-busy write is recorded, 0 pageerrors, and the real click case lands with alpha-id on the wire. The negative control against the da5e309 tarballs hangs all 7 shapes. This matches the takeRecords change in the diff. Advisory: the spawning-hook shape runs to the 5,000-element walker limit (6.8 s) and needs a hook that breaks the README's side-effect rule.
2 PASS (N1, both directions) ✅ PASS 8 of 8 on the wire in HTTP mode: ID stops on busy and disabled events when the hook denies and returns when allowed, and stays on an element allowed throughout. The control on da5e309 fails 4 of 8. This directly answers the prior N1 finding.
3 PASS (B6, every read site) ✅ PASS 14 of 14. Zero dropped fixtures in raw default bytes, 19 distinct fixtures on the wire under keep-all. My own read of runtime.ts at the live head shows the observer sites call elementTestId directly.
4 PASS (normal paths, repo commands) ✅ PASS 53 state events identical across 5 variants in the same order. Five write-timing cases recorded after a hook-running callback, including app observers notified before and after the SDK's. pnpm test 87, pnpm typecheck, pnpm build, pnpm test:browser twice (74 passed, 1 skipped, plus 10 Playwright), pack:check all exit 0. The report added after posting that the CI check "Typecheck, unit and browser tests" passed.
5 PASS (B11, rewritten README sentence) ✅ PASS Each clause of the sentence executed: undefined names reject, page-global names such as status resolve with no IDs and no pageerror, a throw drops that one ID, and a written-out rule matches production. 7 of 7 Playwright tests. This clears the earlier B11 FAIL.
B1 stands from da5e309 ❌ GAP No row at the live head and no authenticated manifest to carry it. The delta does not touch the default filter, so this clears by composition alone.
B2 stands from da5e309 ❌ GAP Hook returning null on state events ran through the sites the delta rewrote. The N1 probe covers part of this, but the cross-surface check (packed detect, fingerprints, snapshots differ) is not re-run.
B3 stands from da5e309 ❌ GAP Failing hooks (throw, reject, non-string) were last run when the observer sites called the memoised helper. The new direct call plus the takeRecords path was not run with a throwing hook, and the report's totals say 0 pageerrors only for the D2 and B11 cases.
B4 stands from da5e309 ❌ GAP The README replacement block (order-row collapse, (v) => v) was last run before the observer change; state-event surface not re-run.
B5 stands from da5e309 ❌ GAP data-interaction-id cases were last run before the observer change; only one D2 shape touches it at the live head.
B7 stands from da5e309 ❌ GAP Delta does not touch types or entry points, so it is carry-eligible, but there is no manifest to carry it.
B8 stands from da5e309 ❌ GAP No Playwright source change; the 7 Playwright tests in row 5 partly cover it. Carry-eligible once a manifest exists.
B9 stands from da5e309 ❌ GAP Both READMEs inside the tarballs changed; versions and manifests did not. Carry-eligible, or a cheap fresh pack:check already run in row 4.
B12 stands from da5e309 ❌ GAP The delta changes per-event work in the observer. The da5e309 report itself did not rerun the runtime-path bench, and the live head has none.

Overall status check

  • Verifier said: PASS
  • Result: NOT READY TO MERGE

Path to merge

Row Class Owner Action that clears it
B1 verification-gap delivery owner Compose a current-head report with the installed Factory CLI (carry-qa, or qa-stamp --carry-from with the recorded da5e309 source). B1 is a CARRIED row with carried_from_head da5e309caf6bdfabe5b4e7e7ad02e7280eebd304, empty path and slug intersections.
B2 verification-gap verifier Fresh row at 29be394: hook returning null, on click, snapshot and both observer sites; packed detect and fingerprints match, snapshots differ.
B3 verification-gap verifier Fresh row at 29be394: throwing, rejecting and non-string hooks fired from a busy or disabled mutation; 0 console errors, 0 pageerrors, capture continues.
B4 verification-gap verifier Fresh row at 29be394: the README replacement block verbatim, including (v) => v, observed on state events as well as clicks and snapshots.
B5 verification-gap verifier Fresh row at 29be394: the data-interaction-id cases, including state events, through the default and a hook.
B7 verification-gap delivery owner Carry through the composer as for B1, carried_from_head da5e309caf6bdfabe5b4e7e7ad02e7280eebd304.
B8 verification-gap delivery owner Carry through the composer as for B1, or include a fresh row from the existing 7 Playwright tests.
B9 verification-gap delivery owner Carry through the composer, or add a fresh pack:check row (already exit 0 at this head in row 4).
B12 verification-gap verifier Fresh row at 29be394: runtime-path bench with the default filter and with a hook on state events against origin/main.

Becomes ✅ VERIFICATION PASS when: a current-head report with an authenticated manifest at 29be39460722c4f01757782c7d92b641e39a0c55 carries B1, B7, B8 and B9 with justifications that re-derive clean against da5e309..29be394, supplies fresh PASS rows for B2, B3, B4, B5 and B12, and keeps rows 1 to 5 as above. No code change is requested.

Dispatch: yes

What stood up

  • The takeRecords fix in runtime.ts is small and does what the README now says. D2's seven shapes and N1's eight cases were run with negative controls against da5e309, so they are evidence of this change.
  • The earlier B11 failure is resolved by an executed clause-by-clause check.
  • Advisory, not blocking: a hook that writes later than its own call (queueMicrotask, .then, setTimeout) is not covered by takeRecords. It is disclosed in the PR, and it needs a hook that breaks the documented no-side-effects rule.
  • Advisory: the report's "Could not verify" items and the unchanged observer record budget (8 of 20 busy events at the default) are pre-existing and unchanged.

@lightninglu10

Copy link
Copy Markdown
Contributor Author

@codepress /judge-verification can you judge this verification?

QA report: PR #6, quantfive/codepress-interactions

PR Head SHA: 29be39460722c4f01757782c7d92b641e39a0c55

  • Answers the judgment of this head: B1, B2, B3, B4, B5, B7, B8, B9 and B12 are fresh rows at this head, as are the earlier rows. No code changed and the head did not move.
  • Manifest: this report has no codepress-verification-manifest and no row is carried. The installed Factory CLI reports carry-qa and qa-stamp as unsupported commands, so the composer could not run here, and none was hand-written.
  • Verifier: the qa-verifier agent type is not installed in this environment, so a general-purpose agent ran as an isolated verifier with the same contract. report_verify_result is not available here either.
  • Artifact paths are local to the verification machine.

DB Performance

DB performance verifier skipped: no backend DB-backed performance surface. Inventory of the diff: packages/browser, packages/playwright, protocol docs and generated schema descriptions, README and CHANGELOG files. No backend API routes, WebSocket handlers, Celery tasks, ORM or query helpers, migrations, SQL, queue consumers, session persistence or aggregation code.

Screenshot Status

Not applicable: the diff has no frontend or UI change. This is an SDK with no rendered interface, so no screenshots were captured.

Overall verdict: PASS

Every row in the contract passes at this head, and there are no defects. origin/main for comparison is b12f53b65060e8998595ccacf3daca39e26331a5.

How to read this report:

  • Evidence classes. LIVE means real Chromium (Playwright 1.64.1) driving a consumer built from packed tarballs, with real input and payloads read from the wire or the memory sink. ARTIFACT means inspection of the packed tarballs. COMMAND means repo or compiler commands.
  • Tarballs. Packed fresh for this report from a detached checkout of this head (artifacts/b10/head-build.log, install 0, build 0).
  • Same-head evidence. D2, N1, normal paths and B11 ran earlier today at this same head and are in artifacts/rows1-5/. The three tarballs from that run and from this one have identical contents, by SHA-256 of the extracted contents (artifacts/tarball-identity.txt). B6 was run again here.
  • Nothing was posted, committed, pushed or published. The author worktree was not touched.

Per-row results

Row Evidence class Verdict Observed Artifacts
B1 Default filter LIVE, wire and memory PASS 42/42. Fixtures absent from raw and gunzipped bytes; save-button and nav:settings kept; keep-all control holds every fixture artifacts/b1-results.jsonl, artifacts/b1/
B2 Hook returns null LIVE, wire, packed /detect PASS 23/23. No ID on click, snapshot or either observer site; same state events as keep-all; snapshots differ, fingerprints match artifacts/b2to6-results.jsonl, artifacts/rows-observer.jsonl, artifacts/b2/
B3 Hook failure LIVE, console and pageerror PASS 41/41. 19 failing hooks reached through the observer first: 0 console, 0 pageerror, 0 unhandled rejections, capture continued artifacts/rows-observer.jsonl, artifacts/b2to6-results.jsonl, artifacts/b3/
B4 Replacement hook LIVE, README verbatim PASS 15/15. README block and (value) => value observed on state events, clicks and snapshots artifacts/rows-observer.jsonl, artifacts/b4-readme-state.js, artifacts/b4/
B5 data-interaction-id LIVE, wire PASS 18/18. All 7 cases on both observer sites, clicks and snapshot, through the default and a hook artifacts/rows-observer.jsonl, artifacts/b5/
B6 Every read site LIVE, raw bytes, keep-all control PASS 14/14, rerun here. No dropped fixture in default bytes; 19 distinct fixtures reach the wire under keep-all artifacts/b2to6-results.jsonl, artifacts/b6b-results.jsonl, artifacts/b6/
B7 Consumers LIVE and COMMAND PASS tsc strict passes under bundler, nodenext and node10; ESM and CJS run; 7 wrong usages rejected; React StrictMode 4/4 artifacts/b7-tsc.log, artifacts/b7-wrong-usage.log, artifacts/b7-esm-cjs.log, artifacts/b7-react-results.jsonl, artifacts/b7-pubreact-tsc.log
B8 Playwright helper LIVE, scratch project PASS 23/23 tests at head, 6/6 at origin/main. Default and hook honoured on all three helpers; CSP outcomes match artifacts/b8-head.jsonl, artifacts/b8-main.jsonl, artifacts/b8/
B9 Artifacts ARTIFACT and COMMAND PASS 26/26 tarball checks; pack:check exit 0 x3; npm publish --dry-run exit 0 for browser and playwright artifacts/b9/
B10 Schema and regression LIVE and COMMAND PASS 75 batches and 81 snapshots validate, 0 invalid. No schema file changed; golden differences are only IDs, version and derived values artifacts/b10/
B11 Docs truth LIVE, scratch project PASS Each clause of the Playwright README sentence executed and true artifacts/rows1-5/b8-final.jsonl, artifacts/rows1-5/b8/
B12 Overhead LIVE bench PASS Observer path, runtime path and walk measured against origin/main. Default filter per callback: +0.24 µs at 1x and +0.84 µs at 4x on attribute writes, up to +1.58 µs on indicators at 4x. See the criterion under B12 artifacts/b12-observer.json, artifacts/b12-runtime.jsonl, artifacts/b12-bench.json, artifacts/b12-load.log
D2 Hook that mutates the element LIVE, real click, own browser per case PASS 7 shapes and 4 original hooks stay responsive and stop calling the hook artifacts/rows1-5/final-d2.jsonl, artifacts/rows1-5/final-d2-shape5.jsonl, artifacts/rows1-5/control-d2-da5e309.jsonl
N1 Hook answer changes LIVE, wire PASS 8/8. The ID stops on state events and returns, in both scenarios; stays on an always-allowed element artifacts/rows1-5/final-n1.jsonl, artifacts/rows1-5/control-n1-da5e309.jsonl, artifacts/rows1-5/final/
Normal paths LIVE, wire PASS 22/22. 53 state events identical in order across 5 variants; real writes after a hook-running callback recorded artifacts/rows1-5/final-state.jsonl
Repo commands COMMAND PASS pnpm test 87 passed; pnpm test:browser 74 passed, 1 skipped, plus 10 Playwright, twice; typecheck and build exit 0 artifacts/rows1-5/b10/, artifacts/b10/head-build.log

GitHub checks at report time: "Typecheck, unit and browser tests", "Test and pack" and "Python tests and lint" succeeded, and the npm publish jobs were skipped (artifacts/pr-state-final.json).

Defects

None.

Details by row

B1

The page holds row-3f2a9c1e-7b4d-4e8a-9c21-5d6e7f809a1b, domain-row-acme.com, item-42, save-button and nav:settings.

  • Default: the three data-bearing fixtures are absent from the bytes as sent and from the gunzipped snapshot.
  • Kept: save-button and nav:settings are in the snapshot dictionary, clicks, findings and raw batches.
  • Control: keep-all holds all five, in both raw batches and gunzipped snapshots, so both searches can hit.
  • Fingerprints: findings touching a dropped ID change; findings on kept IDs do not.

B2

() => null against a keep-all control running the same script. On the attribute site (aria-busy, disabled) and the indicator site (status, progress and a busy element, each added and removed), the control shows the ID and the null hook gives the same events in the same order with none. Clicks fall to tag paths and the dictionary is empty. After every ID value is rotated on the page, the two snapshots differ and packed checkSnapshot gives identical fingerprints.

B3

Each hook was wrapped in a counter and fired by mutations before any walk or click. All 19 hooks ran 6 times from the observer alone, then a real click, a snapshot and one more state write followed.

Class Hooks Result for every hook
Throwing Error, null, an object whose toString throws 7 state events, no ID, 0 console, 0 pageerror, 0 unhandled rejections, runtime still enabled, later click and snapshot recorded
Rejecting Async throw, Promise.reject(), a rejection 5 ms later Same
Non-string undefined, 42, object, array, "", true, symbol, bigint, String object, evil toString, promise of a string, async identity, invalid shape Same

Control: an identity hook has the ID on all 7 state events. A further 20 failing hooks through the click and walk path pass in b2to6.

B4

The README's replacement block was extracted from the checkout and bundled unchanged, apart from assigning the handle.

Surface order-row-8812 item-42 save-button
README block: state events, click, snapshot order-row dropped (tag path on click) kept
(value) => value: state events, click, snapshot order-row-8812 kept kept

The indicator site gives the same answers: order-row-77 collapses and item-42 drops under the block, and both stay under (value) => value.

B5

Expected and observed agree on 4 attribute events and 2 indicator events per case, plus the click and the snapshot dictionary.

data-interaction-id data-testid Default Hook
item-42 save-button save-button save-button (hook returns null)
checkout-pay other-name checkout-pay pay (rewrite)
row-acme.com item-43 No ID row-acme.com
Empty empty-first empty-first empty-first
bad value shape-fallback shape-fallback shape-fallback
only-interaction None only-interaction only-interaction
item-44 None No ID item-44

B6

One attribute read of an ID (util.ts:38), reached only through elementTestId from the click path, both observer sites and the snapshot walk (artifacts/rows1-5/b6-read-site-grep.txt). No dataset or selector-based read. Live coverage: aria-busy, disabled, added and removed nodes, submit, label forward, shadow host, ancestor anchors, dead and rage findings.

B7

  • tsc --strict exit 0 under bundler, nodenext and node10, with skipLibCheck off, across the ESM, React and Playwright fixtures. The CJS fixture is compiled under the configs that list it (consumer/ts/), and bundler does not.
  • With the @ts-expect-error lines removed, 7 wrong usages fail with TS2322: number, undefined and async returns, wrong parameter, a string, null, and a wrong return through the Playwright helper.
  • ESM and CJS run: VERSION 1.1.0, defaultTestIdFilter("item-42") null, defaultTestIdFilter("save-button") kept; CJS also loads /detect and /page-script.
  • React StrictMode passes with the packed react and with published @codepress/interactions-react 1.0.0: the hook reaches the runtime with one instance, and two components sharing a hook share one live instance. Swapping one to a different hook leaves exactly one live. Published react's types also compile against browser 1.1.0.

B8

  • Default on all three helpers: save-button and nav:settings only.
  • Keep-all hook in 7 function forms on all three helpers: all 6 IDs. An element-aware hook works too.
  • Drop-all hook leaves no ID; a hook does not persist to the next capture (6, 0, 2 IDs).
  • Fingerprints match production for the default, a self-contained hook and the README rule.
Policy origin/main, no hook Head, no hook Head, hook findIssues, both revisions
script-src 'self' Works Works Works 3
default-src 'none' Works Works Works 0
Trusted Types Works Works Works 3

bypassCSP works in all three. A no-hook capture followed by a hook capture on one page also works at head.

B9

Check Result
Versions browser 1.1.0, playwright 1.1.0, react 1.0.0; browser dist states 1.1.0
Contents dist, README.md, LICENSE, package.json only; Apache-2.0
Entry points Every exports, main, module and types target exists
workspace: None left; playwright depends on browser ^1.1.0
Env and credentials No env, key or credential files; no token or private key text
pack:check Exit 0 for all three
npm publish --dry-run --ignore-scripts --access public Exit 0 for browser and playwright, from the package directory and from the tarball; the workflows use the directory for browser and the tarball for playwright

B10

All 75 batches and 81 snapshots in this run directory validate against the head schemas. Only protocol/README.md and the two golden JSON files changed from origin/main. Golden differences are IDs (checkout.save to checkout-save, because the default drops a .), sdk_version, derived hashes and timing values (artifacts/b10/golden-diff.log).

B11

The sentence: "a name the page does not have fails the capture. A name the page does have, such as a test variable called status, or any other runtime error in the hook, drops the ID with no error".

Clause Result
Name the page lacks All three helpers reject: testId runs in the page and must be self-contained: allow is not defined
Test variable named status Resolves, no IDs, 0 pageerrors
Hook throws Error, or hits TypeError Resolves, no IDs, 0 pageerrors
Hook throws for one element That ID dropped, the other 5 kept
(value) => value All 6 kept

B12

Other agent sessions were active on the machine throughout: load averages ran from 3.8 to 8.6 (artifacts/b12-load.log). Variants were interleaved on fresh pages, with the order reversed on odd rounds. The reported observer bench ran with nothing else of mine running; a small file copy overlapped the start of the runtime and walk benches.

Criterion. The contract sets no numeric budget, so I applied this: the default must add a negligible absolute cost per state change, and no variant may produce a callback long enough to affect the page. The default and identity hook meet it. The cost of a slower hook is the hook's own, and the README tells authors to keep it fast.

Observer path. MutationObserver was wrapped before init, so each SDK callback is timed in full, including the hook and takeRecords. 1,000 buttons with kept IDs, 31 runs per variant, cross-origin isolated pages for 5 µs timers. Values are the median µs per callback (p25-p75), then the difference from origin/main. Workloads:

  • One write: 3,000 aria-busy or disabled writes, one per callback.
  • Indicator: 600 status elements added and removed, one per callback.
  • Burst, cut: 60 tasks of 40 writes at the default budget, so each callback stops early.
  • Burst, full: 100 tasks of 40 writes with maxMutationRecords: 1000, all processed.

1x CPU:

Variant One write Indicator Burst, cut Burst, full Longest callback, full burst (ms)
origin/main 4.07 (3.93-4.14) 4.82 (4.71-4.98) 18.08 (17.50-18.33) 101.45 (99.75-102.90) 0.48
head default 4.31 (4.21-4.37), +0.24 4.99 (4.87-5.13), +0.17 26.33 (24.75-27.58), +8.25 107.25 (105.85-109.30), +5.80 0.45
head identity hook 4.15 (4.05-4.23), +0.08 4.92 (4.82-5.06), +0.10 18.42 (17.42-18.67), +0.34 101.45 (99.90-102.40), +0.00 0.47
head inline default rule 4.45 (4.37-4.52), +0.38 5.12 (5.04-5.28), +0.30 25.58 (24.33-27.17), +7.50 108.85 (107.00-111.25), +7.40 0.47
head closest hook 4.24 (4.19-4.31), +0.17 4.99 (4.88-5.12), +0.17 18.58 (18.25-19.08), +0.50 106.70 (104.55-107.95), +5.25 0.48
head slow-ish hook 5.18 (5.08-5.23), +1.11 6.26 (6.10-6.35), +1.44 23.83 (23.08-24.33), +5.75 142.50 (140.45-145.25), +41.05 0.53

4x CPU:

Variant One write Indicator Burst, cut Burst, full Longest callback, full burst (ms)
origin/main 16.54 (16.27-17.09) 19.15 (17.87-19.85) 76.17 (62.58-85.00) 421.40 (403.95-428.65) 1.73
head default 17.38 (16.78-18.18), +0.84 20.73 (19.10-21.24), +1.58 102.75 (86.08-119.83), +26.58 451.90 (431.85-462.05), +30.50 1.85
head identity hook 16.94 (16.11-17.37), +0.40 19.65 (18.78-20.02), +0.50 75.08 (64.92-87.58), -1.09 415.40 (398.30-426.45), -6.00 1.79
head inline default rule 18.19 (17.49-18.92), +1.65 20.22 (19.41-22.00), +1.07 94.92 (78.92-106.83), +18.75 450.80 (439.05-457.50), +29.40 1.78
head closest hook 17.64 (16.92-18.28), +1.10 20.45 (19.90-21.17), +1.30 74.58 (67.08-86.08), -1.59 431.65 (416.50-443.45), +10.25 1.85
head slow-ish hook 20.98 (20.53-21.68), +4.44 24.84 (23.70-26.10), +5.69 94.00 (85.67-100.33), +17.83 565.15 (541.35-581.10), +143.75 2.00

Reading these:

  • Default filter: 0.24 µs per write at 1x and 0.84 µs at 4x, about 5 to 6%. On the indicator site it is 0.17 µs at 1x and 1.58 µs at 4x. A full 40-write burst pays 5.8 µs at 1x and 30.5 µs at 4x, about 6 to 7%.
  • Identity hook: within noise of origin/main, so the hook call and takeRecords cost almost nothing.
  • Slow-ish hook (400-iteration loop, closest, regex): about 27 to 30% per callback on one write or one indicator, and 34 to 41% on a full burst. The longest callback stays near 2 ms at 4x.
  • Floor: with no SDK the same one-write loop costs 0.33 µs per write at 1x and 1.25 µs at 4x.
  • Events: each one-write run produced 6,000 events and dropped none, on every variant. Each full burst produced 4,100 and dropped none. The cut burst shows 60 truncation drops on every variant, from the budget and identical at origin/main.

One unexplained spike. On the cut burst the default rule costs 26.3 µs per callback against 18.1 at origin/main at 1x, and 102.8 against 76.2 at 4x. Most of that is a single slow callback per run: 0.46 ms against 0.03 ms at 1x, and 1.6 ms against 0.86 ms at 4x. Without it, the 1x figures are 19.0 against 17.7. It appears only with the default rule and its inline copy, not with the identity or closest hooks, and it did not appear on the full burst. I did not profile it. It costs under 2 ms once per run even at 4x, so it does not change the verdict.

Runtime path. A snapshot through the real runtime and packed bundle, 4,996 kept IDs, ui.snapshot values, median of 9 runs:

Variant Runs Walk, ms Longest slice, ms Slices
main 9 25.6 6 5
head 9 26.3 6 5
head slow-ish hook 9 29.7 6 5
head 0.2ms/ID hook 9 1140.8 7.7 157

Walk. 4,996 kept IDs, 15 runs per variant, median of 5 timed walks after 3 warm-ups, ms (p25-p75):

Variant 1x CPU 4x CPU
main, kept IDs 26.7 (26.5-27.3) 115 (114.1-115.6)
head default, kept IDs 28.1 (27.7-28.3) 118.5 (117.6-119.7)
head hook identity 26.9 (26.8-27.6) 115.1 (113.8-115.7)
head hook, inline default rule 27.6 (27.1-27.9) 117.6 (115.9-118.5)
head hook slow-ish 32.2 (31.7-32.4) 135.2 (134.4-136.9)
main, no IDs at all (floor) 26.3 (26-26.5) 112.7 (111.9-113.2)
head default, all dropped 28 (27.5-28.3) 119.3 (117.4-120)

The default adds 1.4 ms at 1x and 3.5 ms at 4x, about 3 to 5%. The identity hook shows none, which is consistent with this being the cost of the default's checks, though I did not profile it. Above, a 0.2 ms per ID hook stretches the walk to 1.1 s, but it stays sliced at a longest slice of 7.7 ms.

D2

Own browser per case, killed by its own handle. Hook count read twice, 800 ms apart, after the capture finished. The queue was then drained, a real aria-busy write made on an element with no ID, and the event looked for.

Hook Trigger Hook calls Later real write On da5e309
Alternates own data-testid + aria-busy Snapshot 4, stopped Recorded Hangs
Changes own data-testid every call + aria-busy Snapshot 4, stopped Recorded Hangs
Alternates own data-interaction-id + disabled Snapshot 4, stopped Recorded Hangs
Alternates another element's data-testid + aria-busy Snapshot 4, stopped Recorded Hangs
Adds a new busy element with an ID each call Snapshot 6,776, stopped Recorded Hangs
Replaces itself with a busy clone Snapshot 4, stopped Recorded Hangs
Alternates own data-testid + aria-busy, armed for a click Real click 6, stopped Recorded Hangs
Toggle aria-busy Snapshot 4, stopped Recorded Passes
Constant aria-busy="false" Snapshot 4, stopped Recorded Passes
Toggle disabled Snapshot 4, stopped Recorded Passes
Constant disabled Snapshot 4, stopped Recorded Passes

The real click landed: page.click resolved and interaction.started shows alpha-id on the wire. All 11 cases had 0 pageerrors. The da5e309 column is a negative control from the same probe; those 4 original hooks were already fixed on that head, so this round has no failing control for them.

N1

State events from the wire, across allowed, denied and re-allowed phases. T is save-button on the element that changes, K is nav-settings on a second element, - is no ID.

Scenario Busy, in order Disabled Added status element
Moved into [data-private] and back T K T K, - K - K, T K T K T, -, T ID, none, ID
Consent withdrawn and re-granted T K T K, - - - -, T K T K T, -, T ID, none, ID

An element that stayed allowed kept its ID on all 12 state events, and clicks agreed across the same phases. Negative control: the same probe on da5e309 fails 4 of 8, keeping save-button through the denied phase.

Normal paths

The same page script ran against 5 variants: da5e309 and this head with no hook, the same two with (v) => v, and this head with an inline rule. All gave the same 53 events in order: busy 20, structural_change 20, status_region 6, progress_indicator 4, disabled 3. Real writes after a hook-running callback were recorded in the next task, in the next microtask, in a mixed batch, and from app MutationObservers notified before and after the SDK's.

Limitations and disclosed trade-offs

  • Later-than-call hook writes: a hook that writes through queueMicrotask, .then or setTimeout is not covered by takeRecords. Measured by the Orchestrator, listed under Could not verify in the PR, not re-tested.
  • Spawning hook: bounded by the walker's 5,000-element limit, not the observer. In the D2 run the case took 6.8 s including 1.2 s of deliberate waits, and the page kept responding. In a dedicated run the snapshot came back truncated and torn after 252 slices, longest 12.8 ms, with 761 events dropped at the queue limit.
  • Accepted by the Orchestrator, not re-tested: D1, D7, N2, N3.
  • Observer record budget, unchanged: 20 elements written in one task record 8 busy events at the default maxMutationRecords, with or without a hook and at da5e309. At 1000, all 20 are recorded.

Gaps

  • B12 has no contract budget, so its PASS rests on the criterion stated under B12.
  • The cut-burst spike under B12 is measured, not explained.
  • The 4 original D2 hooks have no failing control in this round.

Corrections to my own work

  • Observer bench, three passes. The first pass had a timer of about 100 µs and a "burst" that the budget cut short, and the second filled the event queue partway through. Only the third is reported; the earlier output is in artifacts/b12-first-pass/.
  • Stray file. A leftover cp wrote /tmp/nonexistent during this run, which breaks the no-/tmp rule. I confirmed it was byte-identical to my own probe file and removed that one file.

Rules followed

  • No pattern kill. Every probe browser came from launchServer with its PID logged: 10 started and 10 killed by handle in this run (artifacts/browser-pids.log), and 55 and 55 for the D2, N1 and normal-path probes (artifacts/rows1-5/browser-pids.log). None are running. Browsers started by the Playwright test runner are managed by that runner, and all runners exited.
  • Reads stayed within the checkouts and $TMPDIR, apart from the Playwright browser cache. Network use was GitHub for PR state, and the npm registry for dependencies and the dry runs.

Cleanup status

  • $TMPDIR/qa-pr6d and $TMPDIR/qa-pr6d-main removed with git worktree remove --force, then git worktree prune. All exited 0, and neither appears in git worktree list.
  • Author worktree at 29be394, clean, untouched.
  • No process references this run.
  • Kept: this run directory. probes/ holds every probe, and probes/rows1-5-probes/ the D2, N1, normal-path and B11 probes; they need QA_REPO pointing at a fresh checkout.

@codepress-dev

codepress-dev Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

👋 Hey @lightninglu10! Spawning a verification judge session to help with your request. View session

@codepress-dev

codepress-dev Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

@Terminal @lightninglu10

Dispatched by judge-verification.

This judgment is a VERIFICATION PASS at the current head. Run the guarded readiness transaction: codepress factory delivery ready-for-review --repo . --pr 6. It reads live GitHub state, re-validates current-head QA and judge evidence, reconciles the delivery-status marker, promotes only a currently draft PR, and preserves an already-ready PR. A draft-to-ready transition emits GitHub's native ready_for_review webhook for the initial review. Preserving an already-ready PR or returning already_ready emits no new readiness webhook and does not request a follow-up review. For a declared agent-owned PR with a prior review that does not approve the current head, fixes committed and pushed at a clean newer head, and no current-head review request or review round, use stage3-request-review: request the follow-up immediately with codepress factory delivery request-review --repo . --pr 6. Do not wait for QA or the judge or use readiness as a substitute for that request; QA and the judge continue independently. If a current-head request or round exists, do not duplicate it. A later current-head trusted PASS may invoke readiness again after a manual redraft; do not redraft to trigger re-review, and a ready state alone is not verification evidence. Do not call gh pr ready directly, and do not merge; merge authorization still requires the reviewer gate.

Verification Judgment — PR #6 (local)

Verdict: ✅ VERIFICATION PASS

PR Head SHA: 29be39460722c4f01757782c7d92b641e39a0c55

Summary: Every contract row was run fresh at the live head in real Chromium against tarballs packed from this head, with fixtures checked in raw wire bytes and keep-all controls so the searches can hit. Nothing is carried, so no manifest is needed. The earlier gaps (rows B1 to B5, B7 to B9, B12) are now covered, D2 and N1 carry negative controls against da5e309, and the earlier B11 failure is resolved. All required GitHub checks are green on this head. Remaining items are advisory.

Revision binding

  • Report head: 29be39460722c4f01757782c7d92b641e39a0c55
  • Live PR head: 29be39460722c4f01757782c7d92b641e39a0c55
  • Audited checkout: clean detached checkout at 29be39460722c4f01757782c7d92b641e39a0c55
  • Delta assessment: Report head equals live head and the report states that no row is carried and no manifest is attached. Every row is fresh at this head, so each row's evidence-head-to-live diff is empty. The report says D2, N1, normal paths and B11 ran earlier the same day at this same head and shows the packed tarballs from both runs have identical contents by SHA-256, so I accept that evidence as bound to this head.
  • Rows carried forward: none
  • Carried evidence heads: none
  • Rows reverified at live head: B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, B12, D2, N1, NP, RC
  • Rows still requiring verification: none
  • Publication timing: GitHub shows the head unchanged since the report. Typecheck/unit/browser tests, Test and pack (x3) and Python tests and lint all succeeded on this head; the publish jobs were skipped.

Diff trigger inventory

Surface Location Coverage
Test IDs kept out of events and snapshots by default PR title/body; packages/browser/src/util.ts:38 B1, B6
testId hook returning null or a replacement util.ts, runtime.ts observer, click and walk sites B2, B4
Hook failure of any kind never breaks the page or capture util.ts, runtime.ts B3
data-interaction-id read through the same filter util.ts:38 B5
Hook that mutates the element runs in the observer without freezing the page runtime.ts (ranHook, takeRecords) D2
State events follow the hook's current answer runtime.ts observer sites N1
State, structural and status events unchanged for normal pages runtime.ts NP
Consumers: types, ESM, CJS, React packages/browser, packages/react B7
Playwright helper honours default and hook packages/playwright B8
Released artifacts package manifests, tarballs B9
Schemas and goldens protocol/, goldens B10
README claims both package READMEs B11
Overhead of the default filter and hooks observer, walk B12
Repo commands workspace scripts RC

Per-item

# Verifier said Judgment Evidence
B1 PASS ✅ PASS 42/42 live. row-3f2a…, domain-row-acme.com, item-42 absent from raw and gunzipped bytes; save-button and nav:settings kept; keep-all control holds all five fixtures.
B2 PASS ✅ PASS 23/23. A null hook yields the same state events as keep-all with no ID, clicks fall to tag paths, snapshots differ, packed /detect fingerprints match.
B3 PASS ✅ PASS 41/41. 19 failing hooks reached through the observer first: 0 console, 0 pageerror, 0 unhandled rejections, capture continued, with an identity-hook control showing the ID on all 7 events.
B4 PASS ✅ PASS 15/15. The README block extracted from the checkout and (value) => value observed on state events, clicks and snapshots.
B5 PASS ✅ PASS 18/18. Seven data-interaction-id cases on both observer sites, clicks and snapshots, through default and hook.
B6 PASS ✅ PASS 14/14 rerun at this head. One attribute read of an ID, reached only via elementTestId; 19 fixtures reach the wire under keep-all and none under default.
B7 PASS ✅ PASS tsc --strict passes under bundler, nodenext and node10; 7 wrong usages rejected; ESM/CJS run; React StrictMode 4/4 including published react 1.0.0.
B8 PASS ✅ PASS 23/23 at head versus 6/6 at origin/main; default and hooks honoured on all three helpers; CSP outcomes identical to main.
B9 PASS ✅ PASS 26/26 tarball checks, versions 1.1.0/1.1.0/1.0.0, no workspace: left, no credential files, pack:check and npm publish --dry-run exit 0.
B10 PASS ✅ PASS 75 batches and 81 snapshots validate, 0 invalid; no schema file changed; golden differences explained.
B11 PASS ✅ PASS Each clause of the README sentence executed and true, run earlier at the same head with identical tarball contents.
B12 PASS ✅ PASS (FORM CAVEAT) Observer, runtime and walk benches against origin/main: default adds 0.24 µs per write at 1x and 0.84 µs at 4x; identity hook within noise; the walk stays sliced. Caveat: the contract sets no numeric budget, so the verifier applied a stated criterion, and one slow callback (under 2 ms once per run) on the cut burst is measured but unexplained. Advisory only.
D2 PASS ✅ PASS 11/11 hook shapes stay responsive and stop calling the hook, a later real write is recorded, real click lands; the 7 new shapes hang on the da5e309 control.
N1 PASS ✅ PASS 8/8: the ID stops on state events when denied and returns when allowed; the da5e309 control fails 4 of 8.
NP PASS ✅ PASS 22/22. 53 state events identical in order across 5 variants; real writes after a hook-running callback are recorded.
RC PASS ✅ PASS pnpm test 87 passed, pnpm test:browser 74 passed plus 10 Playwright twice, typecheck and build exit 0; CI green on this head.

Overall status check

  • Verifier said: PASS
  • Result: VERIFICATION PASS

What stood up

  • Fresh live evidence for every row at the live head, with controls that show the probes can fail.
  • Advisory, not blocking: a hook that writes later than its own call (queueMicrotask, .then, setTimeout) is not covered by takeRecords; it needs a hook that breaks the README's no-side-effects rule and is disclosed in the PR.
  • Advisory: a spawning hook runs to the walker's 5,000-element limit rather than being stopped by the observer; the page stays responsive.
  • Advisory: the report was posted without a manifest because the installed Factory CLI lacked the composer commands. Nothing is carried, so this is a form note only.

@lightninglu10
lightninglu10 merged commit c66401e into main Nov 5, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant