Skip to content

ci: add .NET SDK performance reports - #358

Open
dustinbyrne wants to merge 4 commits into
mainfrom
feat/dotnet-performance-benchmarks
Open

dustinbyrne wants to merge 4 commits into
mainfrom
feat/dotnet-performance-benchmarks

Conversation

@dustinbyrne

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

Extend SDK PR performance reporting to .NET, alongside Node and Python, using the shared reporting action and trusted publisher from PostHog/.github#84.

  • Run on every opened, reopened or updated PR. Compare its merge base and exact head using the same head-side benchmark sources, isolated processes and six alternating paired repetitions.
  • Add 17 public-API workloads: single/all local flags, concurrent evaluation, reloads, enqueueing and queue draining. Measure elapsed time, process CPU, managed allocation volume, retained heap and RSS separately. Exact allocation counts are explicitly unsupported.
  • Keep performance deltas informational. Fail on build/execution, fixture or delivery errors; stop after two worker timeouts while preserving diagnostics.
  • Keep PR jobs read-only. A separate pinned workflow_run publisher produces sticky comments once the caller workflow lands on the default branch.

The C# workers use .NET 9.0.8 with workstation GC and non-tiered JIT; the Python coordinator uses only the standard library. SDK dependencies restore in locked mode. Offline HTTP injection retains the SDK's public lifecycle without requiring credentials. Documentation lives in the existing contributing guide.

No SDK runtime/public API changes or NuGet release are included; no changeset is needed.

💚 How did you test it?

  • Final same-revision run: 372 successful workers, 17 workloads, six pairs, 160.94 seconds on macOS ARM64. Largest timing/CPU median delta was 8.85% with unchanged SDK code: this is variability, not a performance improvement claim.
  • Native controls passed for fixture correctness, concurrent/reload workloads, delivery, managed allocation churn, retained heap, RSS and strictly increasing checkpoints.
  • 9 coordinator tests passed, including timeout termination/budget, alternating pairs, wrong-build rejection, invalid measurements and dependency metadata classification.
  • Real shared reporter accepted the resulting schema-v1 documents and produced 57 numeric comparisons. Integration controls verified changed SDK/dependency identities remain comparable, different runtime configurations do not, and failed sources suppress numeric comparisons. Reproduced and fixed the dependency-identity issue found in independent review.
  • dotnet restore --locked-mode, bin/fmt --check, and the existing net8.0 unit suite: 1,403 passed, 2 skipped.
  • Python Ruff/Black checks, Actionlint and git diff --check passed. All action/workflow SHA references were verified through GitHub.
  • Linux execution is covered by this PR's benchmark workflow. Live comment publication still requires default-branch rollout.

📝 Checklist

  • All commits are signed.
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

🤖 Agent context

Autonomy: Human-driven (agent-assisted).

Implemented with Pi and checked by a fresh read-only reviewer. Validation used .NET, Python, Actionlint and the shared benchmark reporter. The design keeps instrumentation native to .NET and reporting shared across SDKs; this PR requires human review.

@dustinbyrne dustinbyrne self-assigned this Oct 28, 2026
@dustinbyrne
dustinbyrne requested a review from a team as a code owner October 28, 2026 18:09

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 166d3e7: 2 Should fix.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

Comment thread benchmarks/Program.cs Outdated
workloads = Fixtures.Workloads(args.Contains("--smoke")).Select(spec => new
{
spec,
fixture_digest = Fixtures.Digest(Fixtures.Definitions(spec.FlagCount) + JsonSerializer.Serialize(Fixtures.Options, Json) + JsonSerializer.Serialize(spec, Json)),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Hash explicit fixture inputs instead of the SDK options schema

When a PR adds a serializable property to FeatureFlagOptions or its base class, serializing Fixtures.Options includes that property only in the head build, even with its default value. The resulting fixture_digest mismatch makes the coordinator reject both inventories before running benchmarks. Hash a benchmark-owned representation of the explicit inputs instead.

Comment thread benchmarks/run.py
Comment on lines +88 to +92
subprocess.check_output(
["dotnet", str(worker), "describe", *(["--smoke"] if smoke else [])],
env=environment(),
text=True,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Apply a deadline to the describe subprocess

When SDK construction or disposal hangs, describe() blocks indefinitely because this check_output has no timeout. The describe command constructs and disposes PostHogClient, so lifecycle regressions bypass the worker deadline and timeout budget, consuming the workflow's 30-minute limit before either results document exists. Bound this subprocess with TIMEOUT.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

posthog-dotnet Compliance Report

Date: 2026-10-28T22:43:13.270566+00:00
Duration: 124998ms

✅ All Tests Passed!

47/47 tests passed


Capture Tests

✅ 30/30 tests passed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 258ms
Format Validation.Event Has Uuid ✅ 128ms
Format Validation.Event Has Lib Properties ✅ 111ms
Format Validation.Distinct Id Is String ✅ 118ms
Format Validation.Token Is Present ✅ 111ms
Format Validation.Custom Properties Preserved ✅ 118ms
Format Validation.Event Has Timestamp ✅ 111ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ✅ 121ms
Retry Behavior.Retries On 503 ✅ 8121ms
Retry Behavior.Does Not Retry On 400 ✅ 2119ms
Retry Behavior.Does Not Retry On 401 ✅ 2114ms
Retry Behavior.Respects Retry After Header ✅ 8119ms
Retry Behavior.Implements Backoff ✅ 22135ms
Retry Behavior.Retries On 500 ✅ 6118ms
Retry Behavior.Retries On 502 ✅ 6119ms
Retry Behavior.Retries On 504 ✅ 6119ms
Retry Behavior.Max Retries Respected ✅ 22121ms
Deduplication.Generates Unique Uuids ✅ 118ms
Deduplication.Preserves Uuid On Retry ✅ 6118ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 13129ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 6121ms
Deduplication.No Duplicate Events In Batch ✅ 118ms
Deduplication.Different Events Have Different Uuids ✅ 110ms
Compression.Sends Gzip When Enabled ✅ 110ms
Batch Format.Uses Proper Batch Structure ✅ 109ms
Batch Format.Flush With No Events Sends Nothing ✅ 106ms
Batch Format.Multiple Events Batched Together ✅ 115ms
Error Handling.Does Not Retry On 403 ✅ 2112ms
Error Handling.Does Not Retry On 413 ✅ 2112ms
Error Handling.Retries On 408 ✅ 6115ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 147ms
Request Payload.Flags Request Uses V2 Query Param ✅ 110ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 109ms
Request Payload.Flags Request Omits Authorization Header ✅ 109ms
Request Payload.Token In Flags Body Matches Init ✅ 110ms
Request Payload.Groups Round Trip ✅ 113ms
Request Payload.Groups Default To Empty Object ✅ 108ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 111ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 110ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 110ms
Request Lifecycle.No Flags Request On Init Alone ✅ 4ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 108ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 214ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 113ms
Retry Behavior.Retries Flags On 502 ✅ 1116ms
Retry Behavior.Retries Flags On 504 ✅ 1111ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 212ms

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 9497641: 1 Should fix. This pull request is large, so the review ran in 4 parts with less depth than usual.

Comment thread benchmarks/Workload.cs
if (request.Method == HttpMethod.Get && request.RequestUri?.AbsolutePath.TrimEnd('/') == "/flags/definitions")
{
Interlocked.Increment(ref DefinitionRequests);
return Response(Fixtures.Definitions(spec.FlagCount, Generation));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Prepare definition responses outside the measured reloads

During flags.refresh.1000, every measured reload calls Fixtures.Definitions here, constructing the 1,000-flag object graph and serializing JSON inside the worker. These mock-server allocations inflate measured RSS and GC pressure, obscuring changes in the SDK's reload memory usage. Precompute generation responses before measurement and serve those fixtures.

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 4bc67e1: no findings. This pull request is large, so the review ran in 4 parts with less depth than usual.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant