ci: replace the flaky absolute perf gate with a same-run A/B gate

[kevincostner17]

Addresses #162.

Summary

The nightly Performance regression workflow has failed roughly every other day, most recently with GATE FAIL: runtime slowdown 0.2701 > 20% vs baseline. It wasn't catching regressions; it was measuring noise:

• T5 timed a ~0.4s clean with median-of-3 and no warmup (runner.py:84-92).
• It compared that absolute time against baseline_v1.json, recorded once on a different machine (a Mac).
• An actions/cache entry overrode the committed baseline. A cache key can never be overwritten, so the update_baseline re-pin input silently did nothing.

New gate: python -m benchmarks.cleanbench.ab

Base and head are measured in the same job, on the same runner, and gated on their ratio:

• Per-side isolation: every measurement runs in a fresh worker subprocess whose PYTHONPATH puts that side's src first. The harness and T5 fixture always come from HEAD, and the worker exits with an error if freshdata was imported from anywhere else.
• Interleaved: base/head, then head/base across pairs (default 5), so runner drift hits both sides.
• Statistics: 200k-row T5 frame, 2 timed runs per worker after warmup. Runtime uses the fastest run per side. Memory uses the median peak-RSS delta of each worker's first (cold) clean, because after a clean the allocator keeps its pages and later deltas read as ~0.
• Confirmation: a breach of the existing thresholds (FULL_GATE_RUNTIME_SLOWDOWN 20%, FULL_GATE_MEMORY_OVERHEAD 15%) triggers a full re-measurement. A metric fails only if it breaches both times.
• Output: writes latest.ab.json and appends a markdown table to the job summary.

Workflow (perf-regression.yml)

Trigger	Base	Mode
pull_request touching src/freshdata/**, benchmarks/cleanbench/** or this workflow	PR base commit	blocking
daily schedule	main as of ~26h earlier; skipped when unchanged	blocking + nightly-failure alert issue
weekly schedule	latest v* release tag	report only
workflow_dispatch	base_ref input	accept_regression input switches to report only

Also:

• The checkout uses fetch-depth: 0, so base commits and tags resolve.
• The actions/cache baseline step is removed.
• Event inputs reach shell only through env:.

run_t5 and baseline_v1.json stay as they are for cleanbench-full reporting.

Verification

• New tests/test_cleanbench_ab.py (12 tests), which covers:

  • interleaving order and fastest-run / median-RSS statistics
  • memory gate not applicable for tiny deltas
  • passes within thresholds; a non-reproducing breach passes; reproduced runtime and memory breaches fail; a metric must breach on both runs
  • CLI exit codes, JSON and summary output
  • a real worker subprocess loads freshdata from the requested src, and refuses a src without it

  All 12 pass on Python 3.12 / pandas 2.3.3 and on Python 3.9 / pandas 1.5.3. ruff check is clean.

• Real local run, base 8a9394c vs this branch at 200k rows with 3 pairs: runtime +1.1%, peak RSS delta 32.9 MiB vs 32.9 MiB (+0.1%), PASS, about 20s total.

I'll close #162 once the new gate has been green for a week of scheduled runs.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: b82f7a9f-8d8e-4a2c-af6b-b48c73b67a8a

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

FreshData benchmark report — performance

• freshdata: ?
• python: ?
• platform: ?

fixture	n_rows	n_cols	p50 s	p95 s	peak MB	repair %	false-repair %	preserve %	trust	monotonic	export %

Authored-code reduction (Metric 6)