Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
adb9c73
Add cache performance benchmarks with category filtering and quick mode
AlecAivazis Jun 14, 2026
b16ef90
Add benchmark serialization, quick-mode fix, and regression compariso…
AlecAivazis Jun 14, 2026
497416b
Update baseline on successful comparison
AlecAivazis Jun 14, 2026
785fd57
Add benchmark CI workflow
AlecAivazis Jun 14, 2026
c0012b3
Add initial benchmark snapshot
AlecAivazis Jun 14, 2026
3d3b45b
Optimize cache hot paths (write, read, subscriptions)
AlecAivazis Jun 14, 2026
107f78a
Architectural cache improvements + stable CI benchmarks
AlecAivazis Jun 14, 2026
8420c70
Use RME-adaptive thresholds in benchmark comparison
AlecAivazis Jun 14, 2026
37c75c1
Simplify CI benchmarks: 1 run per branch, 2000ms measurement window
AlecAivazis Jun 14, 2026
d4e9342
style: auto-format
github-actions[bot] Jun 14, 2026
e7c5b20
Add watch-bench script for fast feedback during cache development
AlecAivazis Jun 14, 2026
e201860
Trim bench scripts down to the four that matter
AlecAivazis Jun 14, 2026
8661ac3
Fix CI: restore benchmark suite when checking out base branch
AlecAivazis Jun 14, 2026
7b2b4d9
Fix TS errors in benchmark file
AlecAivazis Jun 14, 2026
7754701
style: auto-format
github-actions[bot] Jun 14, 2026
b2c4cca
Fix remaining TS errors in benchmark file
AlecAivazis Jun 14, 2026
2c3ac43
style: auto-format
github-actions[bot] Jun 14, 2026
e6084b8
Fix CI: remove copied benchmark dir before checking out PR branch
AlecAivazis Jun 14, 2026
86884db
Use git stash instead of hardcoded rm path when switching branches
AlecAivazis Jun 14, 2026
f7d3670
Revert bench time override — vitest defaults are fine, adaptive thres…
AlecAivazis Jun 14, 2026
f2dd1da
style: auto-format
github-actions[bot] Jun 14, 2026
35b67a7
Fix implicit any in QUICK bench wrapper
AlecAivazis Jun 14, 2026
64af6b7
Fix stray paren from rebase conflict
AlecAivazis Jun 14, 2026
df05979
style: auto-format
github-actions[bot] Jun 14, 2026
9c15c9b
Rewrite b as a plain function to stop fighting Biome's formatter
AlecAivazis Jun 14, 2026
15a16eb
style: auto-format
github-actions[bot] Jun 14, 2026
3956430
Skip 10000-size benchmarks in CI to avoid timeout on O(n²) base branch
AlecAivazis Jun 14, 2026
4622e69
Raise floor threshold to 7% to reduce false positive regressions
AlecAivazis Jun 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions .github/workflows/benchmarks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
name: Cache Benchmarks

on:
pull_request:
paths:
- 'packages/houdini/src/runtime/cache/**'

jobs:
benchmark:
name: Benchmark
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- uses: actions/setup-node@v4
with:
node-version: 24.9.0

- uses: pnpm/action-setup@v4.1.0

- name: Get pnpm store directory
id: pnpm-cache
run: echo "dir=$(pnpm store path)" >> $GITHUB_OUTPUT

- uses: actions/cache@v4
with:
path: ${{ steps.pnpm-cache.outputs.dir }}
key: pnpm-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: pnpm-

# ── PR branch ────────────────────────────────────────────────────
- run: pnpm install --frozen-lockfile --prefer-offline

- name: Benchmark PR branch
run: BENCH_MAX_N=1000 npx vitest bench packages/houdini/src/runtime/cache/benchmarks/ --outputJson /tmp/benchmark.current.json

# ── Base branch ──────────────────────────────────────────────────
- name: Save benchmark suite from PR
run: cp -r packages/houdini/src/runtime/cache/benchmarks /tmp/houdini-benchmarks

- name: Checkout base branch
run: git checkout ${{ github.base_ref }}

- name: Restore benchmark suite onto base branch
run: |
mkdir -p packages/houdini/src/runtime/cache/benchmarks
cp -r /tmp/houdini-benchmarks/. packages/houdini/src/runtime/cache/benchmarks/

- run: pnpm install --frozen-lockfile --prefer-offline

- name: Benchmark base branch
run: BENCH_MAX_N=1000 npx vitest bench packages/houdini/src/runtime/cache/benchmarks/ --outputJson /tmp/benchmark.baseline.json

# ── Compare ──────────────────────────────────────────────────────
- name: Checkout PR branch
run: |
git stash --include-untracked
git checkout ${{ github.head_ref }}

- name: Compare
run: node perf/compare.js /tmp/benchmark.baseline.json /tmp/benchmark.current.json
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,7 @@ vite.config.*.timestamp-*
.netlify

mise.toml

# Benchmark outputs — only benchmark.json (the full baseline) is committed
perf/benchmark.*.json
perf/benchmark.quick.json
20 changes: 20 additions & 0 deletions docs/shared/03-meta/03-contributing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,26 @@ pnpm compile:react # recompile houdini-react only (react app)

Each of these has `:dev`, `:generate`, and `:dev` suffixes that chain into the next step after compiling.

### Cache benchmarks

The cache has a benchmark suite in `packages/houdini/src/runtime/cache/benchmarks/`. Benchmarks are grouped into categories (`core`, `subscriptions`, `lists`, `multi-doc`, `optimistic`, `gc`, `ssr`) and can be run selectively.

For active work on the cache, `watch-bench` re-runs a category in fast mode (minimal iterations, just enough to catch regressions) every time a file changes:

```sh
pnpm watch-bench gc # re-run gc suite on every save
pnpm watch-bench core # re-run core suite on every save
```

To run the full suite at full statistical precision:

```sh
pnpm bench # all categories, full run
BENCH=core pnpm bench # single category, full run
```

A CI job runs automatically on any PR that touches `packages/houdini/src/runtime/cache/`. It benchmarks the base branch and the PR branch on the same runner and flags any benchmark that regressed beyond its noise band.

## Piecing It All Together

When thinking about adding a feature, a few questions help frame the work:
Expand Down
4 changes: 4 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@
"tests": "vitest",
"tests:ui": "vitest --ui --coverage",
"test": "pnpm run tests",
"bench": "vitest bench packages/houdini/src/runtime/cache/benchmarks/ --outputJson perf/benchmark.json",
"bench:check": "vitest bench packages/houdini/src/runtime/cache/benchmarks/ --outputJson perf/benchmark.current.json && node perf/compare.js",
"bench:quick": "BENCH_QUICK=1 vitest bench packages/houdini/src/runtime/cache/benchmarks/ --outputJson perf/benchmark.quick.json",
"watch-bench": "sh perf/watch-bench.sh",
"build:all": "turbo build",
"build": "turbo run build --filter=\"./packages/*\"",
"dev": "turbo dev --filter=\"./packages/*\"",
Expand Down
Loading
Loading