ci(node): integrate shared benchmark reports by dustinbyrne · Pull Request #5454 · PostHog/posthog-js

dustinbyrne · 2026-10-27T22:04:39Z

Changes

Stacked on the Node benchmark producer PR #5453.

Invoke the shared harness action in offline render mode inside the existing read-only benchmark job. Upload rendered comparisons separately from the three-file raw artifact.
Add a separate default-branch workflow_run consumer using the shared trusted publishing workflow. No SDK/PR code executes in the writable job.
Pin the harness action and shared workflow to real published immutable commits and document activation/measurement semantics.

Dependencies:

feat: add shared SDK benchmark reporting posthog-sdk-test-harness#93 (a558711336bd3b81be4a820ed10f26372fc568ed)
ci: add trusted SDK benchmark report workflow .github#84 (ced4b9694b90c9221bc277b1c8d9f013eafe56c5)

Both shared components and this caller configuration need normal review. Automatic PR comments activate only after the workflow_run caller lands on the default branch; summaries can render on this PR already.

Validation

Actionlint, immutable-pin/path verification through GitHub, and diff checks passed.
Real Node/Python result pairs rendered using the shared action code.
Shared harness: 185 focused tests; full suite 1,501 passed / 502 skipped. Shared workflow: five boundary/pin checks and 40 existing tests passed.
Fresh integrated read-only review covered this wiring and the shared implementation.

CI/tooling only; no SDK runtime, workload, or release changes in this layer. Performance differences remain informational.

Agent context

Autonomy: Human-driven (agent-assisted).
Implemented with Pi and reviewed independently. The stack keeps producer measurements separate from shared reporting integration.

stamphog

Not approved — this change needs a human reviewer.

Re-adding the stamphog label gives the same result unless the changed files change.

stamphog can't auto-approve this pull request because two gates failed. The deny-list gate flagged it as infra_cicd, since it adds .github/workflows/node-benchmark-report.yml and changes .github/workflows/node-benchmarks.yml. The tier gate classified it as T2-never, because it touches CI workflows across two areas. The size gate passed, so splitting the change wouldn't clear the refusal. Please ask a human reviewer, such as a CI or infra owner, to review and merge it.

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✗	matches: infra_cicd
size	✓	41L, 2F substantive, 47L/3F incl. docs/generated/snapshots — within ceiling
tier	✗	classified as T2-never: T2-never (47L, 3F, two-areas, ci)
stamphog 2.6.1		`.stamphog/policy.yml` @ `unknown` · reviewed head `8783dd4`

github-actions · 2026-10-27T22:05:05Z

📝 No Changeset Found

This PR doesn't include a changeset. A changeset is required to release a new version.

How to add a changeset

Run this command and follow the prompts:

pnpm changeset

Remember: Never use major version bumps for posthog-js as it's autoloaded by clients.

github-actions · 2026-10-27T22:05:11Z

⚠️ `posthog-node` is modified but this PR has no changeset

This is informational — the PR is not blocked. Click the triangle above to collapse, or push a fix and this comment will auto-delete.

Modified in this PR but no changeset added:

posthog-node

If this change should ship, run pnpm changeset and select a bump level.
If it isn't user-facing (refactor with no behavior change, internal tooling, generated files), no action needed.

posthog

PostHog Review · Standard · reviewed 8783dd4: 2 Should fix.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

posthog · 2026-10-27T22:07:43Z

+            contents: read
+            actions: read
+            pull-requests: write
+        uses: PostHog/.github/.github/workflows/sdk-benchmark-report.yml@ced4b9694b90c9221bc277b1c8d9f013eafe56c5 # SDK benchmark workflow (PostHog/.github PR #84)


P2 · Preserve queued publishers for different pull requests

When three benchmark runs finish close together, the pinned reusable workflow puts all Node publishers in one concurrency group. GitHub replaces the pending job even with cancel-in-progress: false, so a different PR loses its report. Queue publishers without dropping them or serialize by verified PR.

posthog · 2026-10-27T22:07:43Z

                    --repetitions 6

+            - name: Render benchmark comparison
+              if: ${{ !cancelled() && (steps.measure.outcome == 'success' || steps.measure.outcome == 'failure') }}


P2 · Carry measurement failures into partial reports

If run.mjs throws after saving a successful pair, such as on an invalid metric, this step renders the last saved documents without the exception. The publisher also ignores the source-run conclusion, so it posts a normal-looking comparison for an aborted suite. Propagate the measurement failure into both reports.

github-actions · 2026-10-27T22:08:26Z

Coherence

0 spec problems · 0 failing chokepoints · 2 spec gaps · 0 guard failures

44 components, 7 bullets (5 invariants, 2 requirements), 0 problems

Chokepoints: none declared.

Spec gaps: 2 entrances with outside or unknown trust and no traced control, and 10 more held by the adoption baseline

packages/web/Web.spec.md entrance public api (trust host-app): packages/web
packages/webpack-plugin/Webpack-plugin.spec.md entrance webpack plugin (trust build): packages/webpack-plugin

Close one with guard: <chokepoint>, an invariant that names it on its entrances: line, or control: none — <reason>; scaffold control proposes it.

github-actions · 2026-10-27T22:11:41Z

Node SDK acceptance v2

Commit: f58c2d7
Workflow run and artifacts

Capture v0

✅ 55 passed / ❌ 1 non-passing / ⚠️ 101 not selected

Drill-down: run artifacts → node-v2-v0 → acceptance/report.json.diagnostics.json (match case_id below).

❌ migration/yaml-parity-v1/remote-flags-v1.feature::Omitted GeoIP settings default to enabled in flag requests — failed_assertion

Flags field differs

Code: flag_request_field

Failed step: migration/yaml-parity-v1/remote-flags-v1.feature:134

the first flags request field "geoip_disable" should equal JSON false

Operation: /flags/

Field: geoip_disable

Expected:

false

Actual value:

true

Harness exit: 1; report validation exit: 1.

Capture v1

✅ 120 passed / ❌ 1 non-passing / ⚠️ 36 not selected

Drill-down: run artifacts → node-v2-v1 → acceptance/report.json.diagnostics.json (match case_id below).

❌ migration/yaml-parity-v1/remote-flags-v1.feature::Omitted GeoIP settings default to enabled in flag requests — failed_assertion

Flags field differs

Code: flag_request_field

Failed step: migration/yaml-parity-v1/remote-flags-v1.feature:134

the first flags request field "geoip_disable" should equal JSON false

Operation: /flags/

Field: geoip_disable

Expected:

false

Actual value:

true

Harness exit: 1; report validation exit: 1.

posthog

PostHog Review · Standard · reviewed ee2a194: 2 Should fix.

posthog · 2026-10-28T21:23:25Z

                    --repetitions 6

+            - name: Render benchmark comparison
+              if: ${{ !cancelled() && (steps.measure.outcome == 'success' || steps.measure.outcome == 'failure') }}


P2 · Publish failures that occur before measurement

When a build or test:benchmarks fails before measure, this condition skips rendering and the raw artifact contains at most revisions.json. The pinned publisher requires all three result files, so it cannot post the failure and leaves any previous benchmark comment in place. Publish an explicit pre-measurement failure instead.

posthog · 2026-10-28T21:23:25Z

+    ...['steady', 'burst'].flatMap((drain_strategy) =>
+      [32, 4096].map((payload_bytes) => ({
+        id: `capture.drain.${drain_strategy}.${payload_bytes}`,
+        mode: 'drain',
+        payload_bytes,
+        operations: smoke ? 200 : 1000,
+        concurrency: 1,
+        drain_strategy,
+        flush_every: drain_strategy === 'steady' ? 100 : smoke ? 200 : 1000,
+        completion_boundary: 'completed_delivery',
+        passes: ['timing', 'allocation', 'memory'],


P2 · Reserve time to upload results before the job timeout

The four drain workloads add 144 worker executions to a suite whose prior CI runs already took roughly 20 minutes under a 30-minute job timeout. A slower SDK or runner can exhaust that deadline, cancelling the job before partial results are rendered or uploaded. Bound measurement below the job deadline or increase the timeout with a separate upload reserve.

posthog

PostHog Review · Standard · reviewed f58c2d7: 2 Should fix.

This is review round 3 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

posthog · 2026-10-28T22:43:25Z

+            - name: Collect and render benchmark comparison
+              uses: PostHog/.github/.github/actions/benchmark-report@9b34fde6b26df7d874b85d2de09904b4ab701234 # SDK benchmark reporting (PostHog/.github PR #84)
+              with:
+                  mode: collect
+                  source-status: ${{ needs.benchmark.result }}
+                  shard-manifest: packages/node/benchmarks/shards.json


P2 · Publish a failure when a shard stops after timeouts

If two workers time out before a shard finishes its inventory, run.mjs saves partial results and stops, but collect rejects their incomplete workload coverage before producing bundle.json. The publisher then has no canonical artifact and leaves any previous PR comment in place. Provide a verified failure-report path for incomplete shards.

posthog · 2026-10-28T22:43:25Z

+        strategy:
+            fail-fast: false
+            matrix:
+                shard: [flags-single, flags-scaling, flags-memory, capture]


P2 · Skip benchmark shards for unrelated changes

Every pull request update now starts all four shards, including changes unrelated to posthog-node. Each shard independently installs and builds both revisions without caching, multiplying runner usage and queue pressure across the monorepo. Restore an affected-package guard, including benchmark workflow paths, before starting the matrix.

ci(node): integrate shared benchmark reports

8783dd4

dustinbyrne added the stamphog label Oct 27, 2026

dustinbyrne requested a review from a team as a code owner October 27, 2026 22:04

dustinbyrne self-assigned this Oct 27, 2026

stamphog Bot removed the stamphog label Oct 27, 2026

stamphog Bot reviewed Oct 27, 2026

View reviewed changes

dustinbyrne added this pull request to stack #5455 October 27, 2026 22:04

posthog Bot reviewed Oct 27, 2026

View reviewed changes

dustinbyrne added 3 commits October 27, 2026 18:19

fix(ci): report aborted benchmark suites and retain pending publishers

787a375

fix(ci): run every PR benchmark with organization-owned reporting

ea7e9f7

chore(ci): advance benchmark reporting pins

4f6854e

dustinbyrne mentioned this pull request Oct 28, 2026

ci: add .NET SDK performance reports PostHog/posthog-dotnet#358

Open

5 tasks done

test(benchmarks): align with shared workload specification

ee2a194

posthog Bot reviewed Oct 28, 2026

View reviewed changes

ci(benchmarks): shard paired Node workloads

f58c2d7

posthog Bot reviewed Oct 28, 2026

View reviewed changes

Conversation

dustinbyrne commented Oct 27, 2026

Changes

Validation

Agent context

Uh oh!

stamphog Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 27, 2026

📝 No Changeset Found

How to add a changeset

Uh oh!

github-actions Bot commented Oct 27, 2026

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

Uh oh!

posthog Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

posthog Bot Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 27, 2026

Coherence

Uh oh!

github-actions Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Node SDK acceptance v2

Capture v0

Capture v1

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

Uh oh!

posthog Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

posthog Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

Uh oh!

posthog Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

posthog Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Oct 27, 2026 •

edited

Loading