Skip to content

Commit 2a84755

Browse files
mdboommdboom
authored
Merge branch 'main' into per-runner-loops
2 parents 9144dc1 + 09b911f commit 2a84755

11 files changed

Lines changed: 148 additions & 20 deletions

File tree

‎.github/workflows/pypi.yml‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
name: Publish to PyPI
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
tags:
8+
- 'v*.*.*'
9+
10+
jobs:
11+
build:
12+
name: Build distribution 📦
13+
runs-on: ubuntu-latest
14+
15+
steps:
16+
- uses: actions/checkout@v4
17+
with:
18+
persist-credentials: false
19+
- name: Set up Python
20+
uses: actions/setup-python@v5
21+
with:
22+
python-version: "3.x"
23+
- name: Install pypa/build
24+
run: |
25+
python3 -m pip install build --user
26+
- name: Build a binary wheel and a source tarball
27+
run: |
28+
python3 -m build
29+
- name: Store the distribution packages
30+
uses: actions/upload-artifact@v4
31+
with:
32+
name: python-package-distributions
33+
path: dist/
34+
35+
publish-to-pypi:
36+
name: Publish to PyPI
37+
if: startsWith(github.ref, 'refs/tags/') # only publish to PyPI on tag pushes
38+
needs:
39+
- build
40+
runs-on: ubuntu-latest
41+
environment:
42+
name: pypi
43+
url: https://pypi.org/p/bench_runner
44+
permissions:
45+
id-token: write
46+
47+
steps:
48+
- name: Download all the dists
49+
uses: actions/download-artifact@v4
50+
with:
51+
name: python-package-distributions
52+
path: dist/
53+
- name: Publish distribution 📦 to PyPI
54+
uses: pypa/gh-action-pypi-publish@release/v1
55+
56+
github-release:
57+
name: >-
58+
Sign the Python 🐍 distribution 📦 with Sigstore
59+
and upload them to GitHub Release
60+
needs:
61+
- publish-to-pypi
62+
runs-on: ubuntu-latest
63+
64+
permissions:
65+
contents: write # IMPORTANT: mandatory for making GitHub Releases
66+
id-token: write # IMPORTANT: mandatory for sigstore
67+
68+
steps:
69+
- name: Download all the dists
70+
uses: actions/download-artifact@v4
71+
with:
72+
name: python-package-distributions
73+
path: dist/
74+
- name: Sign the dists with Sigstore
75+
uses: sigstore/gh-action-sigstore-python@v3.0.0
76+
with:
77+
inputs: >-
78+
./dist/*.tar.gz
79+
./dist/*.whl
80+
- name: Create GitHub Release
81+
env:
82+
GITHUB_TOKEN: ${{ github.token }}
83+
run: >-
84+
gh release create
85+
"$GITHUB_REF_NAME"
86+
--repo "$GITHUB_REPOSITORY"
87+
--notes ""
88+
- name: Upload artifact signatures to GitHub Release
89+
env:
90+
GITHUB_TOKEN: ${{ github.token }}
91+
# Upload to GitHub Release using the `gh` CLI.
92+
# `dist/` contains the built packages, and the
93+
# sigstore-produced signatures and certificates.
94+
run: >-
95+
gh release upload
96+
"$GITHUB_REF_NAME" dist/**
97+
--repo "$GITHUB_REPOSITORY"

‎PROVISIONING.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@ sudo apt install python3 build-essential ccache gdb lcov pkg-config \
1919
lzma lzma-dev tk-dev uuid-dev zlib1g-dev linux-perf
2020
```
2121

22+
The self-hosted runner needs to have `python` be Python 3.9 or later.
23+
2224
### Enable ccache
2325

2426
```bash session

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,10 @@ For example, you can see [the Faster CPython team's benchmarking results](https:
1010

1111
Create a new empty repository on Github and clone it locally.
1212

13-
Add bench_runner to your `requirements.txt`. Since there are no PyPI releases (yet), you can install it from a tag in the git repo:
13+
Add bench_runner to your `requirements.txt`.
1414

1515
```text
16-
git+https://github.com/faster-cpython/bench_runner@{VERSION}#egg=bench_runner
16+
bench_runner=={VERSION}
1717
```
1818

1919
Replace the {VERSION} above with the latest version tag of `bench_runner`.

‎bench_runner/gh.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,10 @@ def send_notification(body):
7474
conf = config.get_bench_runner_config()
7575
notification_issue = conf.get("notify", {}).get("notification_issue", 0)
7676

77+
if notification_issue == 0:
78+
print("Not sending Github notification.")
79+
return
80+
7781
print("Sending Github notification:")
7882
print("---")
7983
print(body)

‎bench_runner/plot.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,9 @@ class Options:
6565
plt.close("all")
6666

6767
if output_filename.suffix == ".svg":
68-
with tempfile.NamedTemporaryFile(delete=False) as tmp:
68+
with tempfile.NamedTemporaryFile(
69+
dir=output_filename.parent, delete=False
70+
) as tmp:
6971
with open(output_filename) as fd:
7072
scour.start(Options(), fd, tmp)
7173
output_filename.unlink()

‎bench_runner/scripts/run_benchmarks.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,9 @@ def run_benchmarks(
8686
if loops_file := os.environ.get(LOOPS_FILE_ENV_VAR):
8787
extra_args.append("--same-loops")
8888
extra_args.append(loops_file)
89+
90+
if affinity := os.environ.get("CPU_AFFINITY"):
91+
extra_args.append(f"--affinity={affinity}")
8992

9093
args = [
9194
sys.executable,

‎bench_runner/templates/_benchmark.src.yml‎

Lines changed: 8 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,7 @@ jobs:
112112
shell: cmd
113113
run: |
114114
cd cpython
115-
PCbuild\build.bat %BUILD_FLAGS% ${{ (inputs.pgo == true) && '--pgo' || '' }} ${{ inputs.jit == true && '--experimental-jit' || '' }} ${{ inputs.tier2 == true && '--experimental-jit-interpreter' || '' }} ${{ inputs.nogil == true && '--disable-gil' || '' }} -c Release ${{ inputs.clang == true && '"/p:PlatformToolset=clangcl"' || '' }} ${{ inputs.clang == true && '"/p:LLVMInstallDir=C:\Program Files\LLVM"' || '' }} ${{ inputs.clang == true && '"/p:LLVMToolsVersion=19.1.6"' || '' }} ${{ inputs.clang == true && '--tail-call-interp' || '' }}
115+
PCbuild\build.bat %BUILD_FLAGS% ${{ (inputs.pgo == true) && '--pgo' || '' }} ${{ inputs.clang == true && '--tail-call-interp' || '' }} ${{ inputs.jit == true && '--experimental-jit' || '' }} ${{ inputs.tier2 == true && '--experimental-jit-interpreter' || '' }} ${{ inputs.nogil == true && '--disable-gil' || '' }} -c Release ${{ inputs.clang == true && '"/p:PlatformToolset=clangcl"' || '' }} ${{ inputs.clang == true && '"/p:LLVMInstallDir=C:\Program Files\LLVM"' || '' }} ${{ inputs.clang == true && '"/p:LLVMToolsVersion=19.1.6"' || '' }}
116116
- name: Copy Python to different location
117117
if: ${{ steps.should_run.outputs.should_run != 'false' }}
118118
run: |
@@ -161,11 +161,6 @@ jobs:
161161
run: |
162162
git gc
163163
- uses: fregante/setup-git-user@v2
164-
- name: Setup system Python
165-
if: ${{ runner.arch == 'X64' }}
166-
uses: actions/setup-python@v5
167-
with:
168-
python-version: "3.11"
169164
- name: Checkout CPython
170165
uses: actions/checkout@v4
171166
with:
@@ -211,7 +206,7 @@ jobs:
211206
if: ${{ steps.should_run.outputs.should_run != 'false' }}
212207
run: |
213208
cd cpython
214-
./configure ${{ inputs.pgo == true && '--enable-optimizations --with-lto=full' || '' }} ${{ inputs.tier2 == true && '--enable-experimental-jit=interpreter' || '' }} ${{ inputs.jit == true && '--enable-experimental-jit=yes' || '' }} ${{ inputs.nogil == true && '--disable-gil' || '' }} ${{ inputs.clang == true && '--with-tail-call-interp' || '' }}
209+
./configure --enable-option-checking=fatal ${{ inputs.pgo == true && '--enable-optimizations --with-lto=full' || '' }} ${{ inputs.tier2 == true && '--enable-experimental-jit=interpreter' || '' }} ${{ inputs.jit == true && '--enable-experimental-jit=yes' || '' }} ${{ inputs.nogil == true && '--disable-gil' || '' }} ${{ inputs.clang == true && '--with-tail-call-interp' || '' }} ${PYTHON_CONFIGURE_FLAGS:-}
215210
make ${{ runner.arch == 'ARM64' && '-j' || '-j4' }}
216211
./python -VV
217212
- name: Install pyperformance
@@ -221,7 +216,7 @@ jobs:
221216
- name: Tune system
222217
if: ${{ steps.should_run.outputs.should_run != 'false' }}
223218
run: |
224-
sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH venv/bin/python -m pyperf system ${{ inputs.perf && 'reset' || 'tune' }}
219+
sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH venv/bin/python -m pyperf system ${{ inputs.perf && 'reset' || 'tune ${CPU_AFFINITY:+--affinity="$CPU_AFFINITY"}' }}
225220
- name: Tune for (Linux) perf
226221
if: ${{ steps.should_run.outputs.should_run != 'false' && inputs.perf }}
227222
run: |
@@ -232,6 +227,10 @@ jobs:
232227
run: |
233228
rm -rf ~/.debug/*
234229
venv/bin/python -m bench_runner run_benchmarks ${{ inputs.perf && 'perf' || 'benchmark' }} cpython/python ${{ inputs.fork }} ${{ inputs.ref }} ${{ inputs.benchmarks || 'all' }} ${{ env.flags }} --run_id ${{ github.run_id }}
230+
- name: Untune system
231+
if: ${{ steps.should_run.outputs.should_run != 'false' }}
232+
run: |
233+
sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH venv/bin/python -m pyperf system reset
235234
# Pull again, since another job may have committed results in the meantime
236235
- name: Pull benchmarking
237236
if: ${{ steps.should_run.outputs.should_run != 'false' && !inputs.perf }}
@@ -321,11 +320,9 @@ jobs:
321320
if: ${{ steps.should_run.outputs.should_run != 'false' }}
322321
run: |
323322
cd cpython
324-
./configure ${{ inputs.pgo == true && '--enable-optimizations --with-lto=full' || '' }} ${{ inputs.tier2 == true && '--enable-experimental-jit=interpreter' || '' }} ${{ inputs.jit == true && '--enable-experimental-jit=yes' || '' }} ${{ inputs.nogil == true && '--disable-gil' || '' }} ${{ inputs.clang == true && '--with-tail-call-interp' || '' }}
323+
./configure --enable-option-checking=fatal ${{ inputs.pgo == true && '--enable-optimizations --with-lto=full' || '' }} ${{ inputs.tier2 == true && '--enable-experimental-jit=interpreter' || '' }} ${{ inputs.jit == true && '--enable-experimental-jit=yes' || '' }} ${{ inputs.nogil == true && '--disable-gil' || '' }} ${{ inputs.clang == true && '--with-tail-call-interp' || '' }} ${PYTHON_CONFIGURE_FLAGS:-}
325324
make -j4
326325
./python.exe -VV
327-
# On macos ARM64, actions/setup-python isn't available, so we rely on a
328-
# pre-installed homebrew one, used through a venv
329326
- name: Install pyperformance
330327
if: ${{ steps.should_run.outputs.should_run != 'false' }}
331328
run: |

‎bench_runner/templates/_pystats.src.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ jobs:
100100
if: ${{ steps.should_run.outputs.should_run != 'false' }}
101101
run: |
102102
cd cpython
103-
./configure --enable-pystats --prefix=$PWD/install ${{ inputs.tier2 == true && '--enable-experimental-jit=interpreter' || '' }} ${{ inputs.jit == true && '--enable-experimental-jit=yes' || '' }} ${{ inputs.nogil == true && '--disable-gil' || '' }}
103+
./configure --enable-option-checking=fatal --enable-pystats --prefix=$PWD/install ${{ inputs.tier2 == true && '--enable-experimental-jit=interpreter' || '' }} ${{ inputs.jit == true && '--enable-experimental-jit=yes' || '' }} ${{ inputs.nogil == true && '--disable-gil' || '' }}
104104
make -j4
105105
make install
106106
- name: Install pyperformance into the system python

‎bench_runner/templates/bench_runner.toml‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,24 @@ notification_issue = 182
2424
# socket.gethostname()).
2525
# available: True if the machine is available (we need to keep the
2626
# metadata around for old machines for the generated results). (Default: True)
27-
27+
# include_in_all: Whether to include the runner when choosing to run a
28+
# benchmark on all runners.
29+
# env: a table of extra environment variables to pass to the runs.
30+
#
31+
# A few notable environment variables for runners:
32+
# CC, CXX: used by Python's configure to determine the compiler to use.
33+
# LLVM_AR, LLVM_PROFDATA: used for PGO optimization passes with clang, when
34+
# using a version of clang that isn't the default. These must be full
35+
# paths for configure to handle them correctly.
36+
# LLVM_BOLT, MERGE_FDATA: like AR/PROFDATA, but for BOLT.
37+
# LLVM_BOLT may need to point to, e.g., /usr/lib/llvm-19/bin/llvm-bolt
38+
# rather than /usr/bin/llvm-bolt-19 (see
39+
# https://github.com/python/cpython/issues/127047).
40+
# PYTHON_CONFIGURE_FLAGS: extra flags to pass to Python's configure. These
41+
# can override all earlier flags, which by default include
42+
# --enable-optimizations and --with-lto=full.
43+
# BUILD_DEST: (windows only) where to make build.bat install. This is
44+
# required for the Windows build to succeed.
2845
[[runners]]
2946

3047
[runners.linux]
@@ -33,6 +50,13 @@ arch = "x86_64"
3350
hostname = "pyperf"
3451
available = true
3552

53+
[runners.windows]
54+
os = "windows"
55+
arch = "x86_64"
56+
hostname = "winperf"
57+
available = true
58+
env.BUILD_DEST = "PCbuild/amd64"
59+
3660
[plot]
3761
bases = ["3.11.0"]
3862
versions = [[3, 12]]

‎pyproject.toml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ classifiers = [
1717
dependencies = [
1818
"matplotlib==3.10.1",
1919
"pyperf==2.9.0",
20-
"rich==13.9.4",
20+
"rich==14.0.0",
2121
"rich-argparse==1.7.0",
2222
"ruamel.yaml==0.18.10",
2323
"scour==0.38.2",
@@ -30,8 +30,8 @@ dynamic = ["version"]
3030
test = [
3131
"black==25.1.0",
3232
"filelock==3.18.0",
33-
"flake8==7.1.2",
34-
"pyright==1.1.397",
33+
"flake8==7.2.0",
34+
"pyright==1.1.398",
3535
"pytest==8.3.5",
3636
"pytest-xdist==3.6.1",
3737
]

0 commit comments

Comments
 (0)