Skip to content

Unify aggregation, real RSS, Bencher prep (#40) #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
m-hu wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Unify aggregation, real RSS, Bencher prep (#40) #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 37 additions & 7 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,28 +97,57 @@ jobs:
- name: Install Valgrind
uses: awalsh128/cache-apt-pkgs-action@v1.6.3
with:
packages: valgrind
version: 1.0
# time: GNU time wraps every native run (cpu/maxrss); the
# memory signal is unreconcilable without it (issue #40).
packages: time valgrind
version: 1.1

- name: Measure performance (Callgrind instructions)
# Lean recipe mirroring the clojure twin: one GNU-timed
# native run (stdout gate + wall/cpu/maxrss) plus one
# Callgrind run (Ir) per case.
run: |
python3 tools/measure.py --resources tests/resources \
--repeat 1 --jobs 2 --timeout 300 \
--repeat 1 --callgrind-repeat 1 --jobs 2 --timeout 300 \
--shard ${{ matrix.shard }}/4 \
--output perf-results.json --bmf-output bencher-bmf.json \
--summary-file "$GITHUB_STEP_SUMMARY"

- name: Warn when Bencher history is disabled
if: env.BENCHER_API_KEY == ''
env:
BENCHER_API_KEY: ${{ secrets.BENCHER_API_KEY }}
run: |
echo "::warning::BENCHER_API_KEY secret is not set: Bencher history stays empty (maintainer action: create the python-code-snippet project on Bencher and set the secret)"

- name: Install Bencher CLI
if: env.BENCHER_API_KEY != ''
env:
BENCHER_API_KEY: ${{ secrets.BENCHER_API_KEY }}
uses: bencherdev/bencher@main

# Threshold policy (issue #40): callgrind_instructions
# (percentage) only, same shape as the clojure twin's sweep
# (min/max sample 10/64, 1% upper boundary over the ~zero
# deterministic-Ir wobble). The cpu_time/peak_memory entries
# are stripped before reporting: the sweep is repeat-1
# timing, not a timing signal; the full BMF stays in the
# uploaded artifacts for RSS reconciliation.
- name: Strip BMF to tracked measures
run: |
jq 'with_entries(select(.value.callgrind_instructions != null) | .value |= {callgrind_instructions})' \
bencher-bmf.json > bencher-bench.json
echo "tracked benchmarks: $(jq 'length' bencher-bench.json)"

- name: Track benchmarks with Bencher
if: env.BENCHER_API_KEY != ''
env:
BENCHER_API_KEY: ${{ secrets.BENCHER_API_KEY }}
run: |
if [ "$(jq 'length' bencher-bench.json)" = "0" ]; then
echo "no callgrind data in shard; skipping Bencher report"
exit 0
fi
if [ "${{ github.event_name }}" = "pull_request" ]; then
bencher run \
--project python-code-snippet \
Expand All @@ -130,21 +159,21 @@ jobs:
--start-point-clone-thresholds \
--testbed ubuntu-latest-python \
--adapter json \
--file bencher-bmf.json \
--file bencher-bench.json \
--github-actions "${{ secrets.GITHUB_TOKEN }}"
else
bencher run \
--project python-code-snippet \
--key '${{ secrets.BENCHER_API_KEY }}' \
--branch "${{ github.ref_name }}" \
--testbed ubuntu-latest-python \
--threshold-measure instructions \
--threshold-measure callgrind_instructions \
--threshold-test percentage \
--threshold-min-sample-size 10 \
--threshold-max-sample-size 64 \
--threshold-upper-boundary 0.10 \
--threshold-upper-boundary 0.01 \
--adapter json \
--file bencher-bmf.json \
--file bencher-bench.json \
--github-actions "${{ secrets.GITHUB_TOKEN }}"
fi

Expand All @@ -156,6 +185,7 @@ jobs:
path: |
perf-results.json
bencher-bmf.json
bencher-bench.json

report:
needs: perf-callgrind
Expand Down
13 changes: 8 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,16 +53,19 @@ See `tools/build_bundle.sh` for rotation instructions when using a fresh passphr

## Performance Tracking (Callgrind)

`tools/measure.py` measures load-independent algorithmic complexity using Valgrind Callgrind:
`tools/measure.py` measures load-independent algorithmic complexity using Valgrind Callgrind. Each case runs natively once under GNU time (required: `apt install time`) plus once under Callgrind per repeat:

- `instructions` (`Ir` — simulated instruction reads) — completely load-invariant and deterministic across runs and runners.
- `cpu_time` (user + sys CPU time)
- `wall_time`
- `callgrind_instructions` (`Ir` — simulated instruction reads) — completely load-invariant and deterministic across runs and runners.
- `cpu_time` (user + sys CPU time from GNU time)
- `peak_memory` (max RSS in bytes from GNU time)
- `wall_s`

Comparison rule ([clojure-code-snippet#40](https://github.com/hhh-pa/clojure-code-snippet/issues/40)): Ir is the sum of per-case minimums (least-disturbed run) on CPython exactly as on JVM and clojure-rs, so BMF values compare directly across runtimes. cpu_time sums per-case medians; peak_memory takes the max per-case median. Benchmark names stay stable.

```sh
python3 tools/measure.py --resources tests/resources --repeat 1 \
--output perf.json --bmf-output bencher.json
python3 tools/measure.py --self-test
```

CI runs correctness across sharded runners in the `test-shard` jobs, aggregated by `test`. Then the `perf-callgrind` job measures instruction counts under Valgrind Callgrind across 4 shards, reporting to Bencher under testbed `ubuntu-latest-python` and measure `instructions` (gated by `BENCHER_API_KEY`). A fan-in `report` job merges all shard results, verifies equality against the plain BMF union (`jq -s 'add'`), and publishes a combined Markdown summary table to the GitHub Actions job summary.
CI runs correctness across sharded runners in the `test-shard` jobs, aggregated by `test`. Then the `perf-callgrind` job measures instruction counts under Valgrind Callgrind across 4 shards, reporting to Bencher under testbed `ubuntu-latest-python` and measure `callgrind_instructions` (gated by `BENCHER_API_KEY`; a maintainer must create the `python-code-snippet` Bencher project and set the secret before history starts). A fan-in `report` job merges all shard results, verifies equality against the plain BMF union (`jq -s 'add'`), and publishes a combined Markdown summary table to the GitHub Actions job summary.
Loading
Loading