Skip to content

test: add spike, soak and breakpoint load tests - #170

Merged
Dejmenek merged 5 commits into
mainfrom
feat/165-loadtests-spike-soak-breakpoint
Sep 29, 2026
Merged

Dejmenek merged 5 commits into
mainfrom
feat/165-loadtests-spike-soak-breakpoint

Conversation

@Dejmenek

@Dejmenek Dejmenek commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Summary

This is the last PR for #165 and it is stacked on #169. It adds the three long running scenario types from the issue: a spike, a soak and a breakpoint test. It also records the design in a new ADR.

Tests

  • The spike test runs a heavy list query at 25 requests per second, jumps to 150 for 10 seconds, drains, and returns to 25. It checks the error budget, tail latency and payload size in each phase, and requires the p95 after the spike to stay within twice the p95 before it. A cheap list query showed a p95 of 22 ms even at 200 requests per second, so it could not stress the API.
  • The soak test runs the mixed browsing load for about 12 minutes as a warm up and six two minute segments. Each step is checked against its Medium baseline per segment, and the last segment must not drift past twice the first. Managed memory is sampled after forced collections at the end of the first segment and at the end of the run.
  • The breakpoint test steps a heavy query through 25 to 150 requests per second after a warm up, and finds the highest rate whose p95 stays under 800 ms. It measured 125 requests per second in two runs. The test asserts a floor of 50, and a safety abort stops a runaway run without failing the test.

The spike test carries Category=Load. Soak and breakpoint carry Category=Soak only, so the normal load run skips them. dotnet test TournamentAPI.LoadTests --filter "Category=Soak" runs them, and CLAUDE.md documents it.

Helpers and docs

LoadPhases turns named phases into Inject simulations and names each step after the phase running when it starts. LoadThreshold takes an optional abort count for the breakpoint safety abort. ADR 0009 records why the breakpoint is read from per step latency after the run instead of from an abort on errors, and why the soak is one run with named segments. ADR 0008 now lists the headroom rules the budgets use.

Things worth knowing

  • The 150 MB memory growth limit in the soak test has not been calibrated against a measured run. It stays as a guess for now.
  • The soak passed once. A second run failed on tail latency in a few segments while the machine had many other programs open, so it needs a quiet machine.
  • Spike budgets come from the worst of three runs. Run to run variance on this machine is large, and the first phase of any run carries cold start latency, which is why each test has an unasserted warm up.
  • Pinning the azure-sql-edge image from the issue's task list is not part of this stack. The image is still on latest on purpose.

Testing

Unit tests (160) and integration tests (252) pass locally. The spike test passed three times and the breakpoint test twice on the final code. This PR only changes the load test project and the docs, and CI runs the unit and integration tests. Load tests need Docker and run manually.

Closes #165.

Stack created with GitHub Stacks CLI • Give Feedback 💬

🤖 Generated with Claude Code

Dejmenek and others added 5 commits September 29, 2026 21:35
LoadPhases turns a list of named phases into Inject simulations and names each step after the phase that is running when it starts. LoadThreshold takes an optional abort count, so a runaway breakpoint run can stop early through AbortOnErrors. LoadTestBudgets gets Spike, Soak and Breakpoint groups. The spike budgets come from the worst of three runs with the usual headroom.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A heavy list query runs at 25 requests per second, jumps to 150 for 10 seconds, drains, and returns to 25. The test checks the error budget, tail latency and payload size per phase. It also requires the p95 after the spike to stay within twice the p95 before it, with a 100 ms floor. A cheap list query showed a p95 of 22 ms even at 200 requests per second, so it could not stress the API.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The mixed browsing load runs for about 12 minutes as a warm up and six named two minute segments. Every step is checked against its Medium baseline per segment, and the last segment must not drift past twice the first. Managed memory is sampled after forced collections at the end of the first segment and at the end of the run. The test carries Category=Soak only, so the normal load run skips it. The 150 MB growth limit has not been calibrated against a measured run.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A heavy query steps through 25, 50, 75, 100, 125 and 150 requests per second after a warm up phase. The breakpoint is the highest rate whose p95 stays under 800 ms with no failures, and the test asserts it is at least 50 requests per second. It measured 125 in two runs. A safety abort on a very high error rate stops a runaway run without failing the test.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ADR 0009 records how the breakpoint is found and how the soak is run, with the measured breakpoint and spike figures. ADR 0008 gets the headroom rules the budgets use and a link to the new ADR. CLAUDE.md gets the command for the soak and breakpoint tests.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@Dejmenek
Dejmenek added this pull request to stack #168 September 29, 2026 19:39
@Dejmenek Dejmenek changed the title feat/165 loadtests spike soak breakpoint test: add spike, soak and breakpoint load tests Sep 29, 2026
@Dejmenek
Dejmenek marked this pull request as ready for review September 29, 2026 19:44
Base automatically changed from feat/165-loadtests-write-contention to main September 29, 2026 19:51
@Dejmenek
Dejmenek merged commit 7c4d778 into main Sep 29, 2026
3 checks passed
@Dejmenek
Dejmenek deleted the feat/165-loadtests-spike-soak-breakpoint branch September 29, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve load test quality and expand scenario coverage

1 participant