Skip to content

Commit 5698fa8

Browse files
authored
Store results one file per framework (#1034)
Closes #751. Results lived in 52 shared arrays — site/data/<profile>-<conns>.json, each holding every framework — so a saved run rewrote files that every other framework also occupies. They now live one file per framework: site/data/baseline-512.json [ {actix…}, {hyper…}, … ] -> site/data/results/actix.json { framework, results: { "baseline-512": {…} } } 108 files, 1941 rows, same rows verbatim. The conflict is real but narrower than "any two saves": git merges non-adjacent hunks fine, so two frameworks updating existing rows usually merge. It bites when rows are *inserted* — two contributors each adding a framework, which is exactly when a rebase is most annoying. Reproduced both ways against the real data: old layout, actix + hyper update rows merges cleanly old layout, two new frameworks added CONFLICT in baseline-512 and -4096 new layout, the same two new frameworks merges cleanly, both files present Rows are copied verbatim rather than reshaped, which makes the migration provable: regenerating data.js before and after gives a byte-identical file. That is the main guarantee here — the board, the composite and every archived round see exactly what they saw before. Consumers: rebuild_site_data.py writes each framework's file; a run touches only the frameworks it benchmarked (verified: benchmarking actix + hyper modified 2 of 108 files) gen_leaderboard_data loads site/data/results/*.json once and rebuilds the per-profile view, sorted by framework name so the emitted data.js does not churn compare.sh one read of the framework's own file instead of a scan of every profile file archive.sh regroups into the existing round shape, so rounds/*.json keeps its current format benchmark-pr.yml stages only site/data/results/<slug>.json The filename is the display_name lowercased with anything outside [A-Za-z0-9._-] replaced by '-', because two gateway entries are named "aspnet-minimal + nginx" and "+ caddy". .gitignore's `results/` rule was unanchored and silently swallowed the new directory — the first attempt committed the code and none of the data. It is now `/results/`, which is the raw benchmark output it was always meant to cover; the root directory is still ignored. scripts/migrate_results_layout.py is kept rather than deleted: it re-derives the layout from a flat checkout, which is what a branch predating this needs in order to rebase. Note for #1020, which edits rebuild_site_data.py and deletes rows from the flat files: it will need a rebase, and running the migration script after merging main is the mechanical way to do it.
1 parent f8c6c13 commit 5698fa8

169 files changed

Lines changed: 39508 additions & 38908 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/benchmark-pr.yml

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,8 @@ jobs:
134134
135135
- name: Commit saved results
136136
if: inputs.save == 'true'
137+
env:
138+
FRAMEWORK: ${{ inputs.framework }}
137139
run: |
138140
PR_DATA=$(gh api "/repos/${{ github.repository }}/pulls/${{ inputs.pr }}" --jq '.head.ref + " " + .head.repo.full_name')
139141
PR_BRANCH=$(echo "$PR_DATA" | awk '{print $1}')
@@ -143,14 +145,11 @@ jobs:
143145
git config user.email "github-actions[bot]@users.noreply.github.com"
144146
find site/static/logs -name "${{ inputs.framework }}.log" -exec git add -f {} +
145147
git add -f site/data/frameworks.json site/data/current.json 2>/dev/null || true
146-
# Only add leaderboard files for the profiles that were actually benchmarked
147-
for f in results/*/*/${{ inputs.framework }}.json; do
148-
[ -f "$f" ] || continue
149-
dir=$(dirname "$f")
150-
conns=$(basename "$dir")
151-
prof=$(basename "$(dirname "$dir")")
152-
git add -f "site/data/${prof}-${conns}.json" 2>/dev/null || true
153-
done
148+
# Results are one file per framework (#751), so a run only ever stages
149+
# its own — two PRs benchmarking different frameworks no longer touch
150+
# the same file. The filename is the slugified display_name.
151+
SLUG=$(python3 -c "import json,re,sys;n=json.load(open('frameworks/%s/meta.json'%sys.argv[1])).get('display_name',sys.argv[1]);print((re.sub(r'[^A-Za-z0-9._-]+','-',n).strip('-') or 'unnamed').lower())" "$FRAMEWORK")
152+
git add -f "site/data/results/${SLUG}.json" 2>/dev/null || true
154153
if git diff --cached --quiet; then
155154
echo "No results to commit"
156155
else

.gitignore

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
1-
results/
1+
# Raw benchmark output at the repo root. Anchored: site/data/results/ holds
2+
# the published per-framework result files and must stay tracked.
3+
/results/
24
site/public/
35
site/resources/
46

scripts/archive.sh

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -87,16 +87,25 @@ import json, glob, os, sys
8787
site_data = sys.argv[1]
8888
round_file = sys.argv[2]
8989
90+
# Rounds keep their existing shape - {"<profile>-<conns>": [row, ...]} - so
91+
# archived rounds stay readable. Results now live one file per framework
92+
# (#751), so regroup them back into per-profile arrays here.
9093
bundle = {}
91-
for f in sorted(glob.glob(os.path.join(site_data, '*.json'))):
92-
name = os.path.basename(f)
93-
if name in ('frameworks.json', 'langcolors.json'):
94+
for f in sorted(glob.glob(os.path.join(site_data, 'results', '*.json'))):
95+
try:
96+
with open(f) as fh:
97+
entry = json.load(fh)
98+
except Exception:
9499
continue
95-
if name.startswith('rounds'):
96-
continue
97-
key = os.path.splitext(name)[0]
98-
with open(f) as fh:
99-
bundle[key] = json.load(fh)
100+
for key, row in (entry.get('results') or {}).items():
101+
bundle.setdefault(key, []).append(row)
102+
for key in bundle:
103+
bundle[key].sort(key=lambda r: (r.get('framework') or '').lower())
104+
105+
current = os.path.join(site_data, 'current.json')
106+
if os.path.exists(current):
107+
with open(current) as fh:
108+
bundle['current'] = json.load(fh)
100109
101110
# Include frameworks metadata
102111
fw_path = os.path.join(site_data, 'frameworks.json')

scripts/compare.sh

Lines changed: 18 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ SITE_DATA="$ROOT_DIR/site/data"
6464

6565
# Collect all profiles with results for this framework
6666
python3 -c "
67-
import json, os, sys, glob
67+
import json, os, re, sys, glob
6868
6969
framework = sys.argv[1]
7070
display_name = sys.argv[2]
@@ -106,21 +106,24 @@ if not new_results:
106106
print(f'No results found for \`{framework}\`')
107107
sys.exit(0)
108108
109-
# Find old results from site/data (published on main)
109+
# Find old results from site/data (published on main). Results are one file
110+
# per framework keyed by <profile>-<conns> (#751), so this is a single read.
111+
def _slug(name):
112+
return (re.sub(r'[^A-Za-z0-9._-]+', '-', name).strip('-') or 'unnamed').lower()
113+
110114
old_results = {}
111-
for key, new_data in new_results.items():
112-
profile, conns = key.split('/')
113-
site_file = f'{site_data}/{profile}-{conns}.json'
114-
if os.path.exists(site_file):
115-
try:
116-
with open(site_file) as f:
117-
entries = json.load(f)
118-
for entry in entries:
119-
if entry.get('framework') == baseline_name:
120-
old_results[key] = entry
121-
break
122-
except:
123-
pass
115+
baseline_file = f'{site_data}/results/{_slug(baseline_name)}.json'
116+
if os.path.exists(baseline_file):
117+
try:
118+
with open(baseline_file) as f:
119+
published = (json.load(f).get('results') or {})
120+
for key in new_results:
121+
profile, conns = key.split('/')
122+
entry = published.get(f'{profile}-{conns}')
123+
if entry:
124+
old_results[key] = entry
125+
except:
126+
pass
124127
125128
# Format helpers
126129
def fmt_num(n):

scripts/gen_new_leaderboard_data.py

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,37 @@
123123
}
124124

125125

126+
RESULTS: dict[str, list] = {}
127+
128+
129+
def load_results():
130+
"""Index site/data/results/*.json as {"<profile>-<conns>": [row, ...]}.
131+
132+
Results used to live in one array per profile-conns, which meant every
133+
framework's PR wrote the same files and collided (#751). They are now one
134+
file per framework; this rebuilds the per-profile view the rest of the
135+
generator expects.
136+
137+
Rows are sorted by framework name because that is the order the flat files
138+
were written in, and the emitted data.js must not churn.
139+
"""
140+
idx: dict[str, list] = {}
141+
rdir = DATA / "results"
142+
if not rdir.is_dir():
143+
return idx
144+
for f in sorted(rdir.glob("*.json")):
145+
try:
146+
entry = json.loads(f.read_text(encoding="utf-8"))
147+
except Exception as e:
148+
print(f"[warn] {f.name}: {e}")
149+
continue
150+
for key, row in (entry.get("results") or {}).items():
151+
idx.setdefault(key, []).append(row)
152+
for key in idx:
153+
idx[key].sort(key=lambda r: (r.get("framework") or "").lower())
154+
return idx
155+
156+
126157
def load(name):
127158
p = DATA / name
128159
if not p.exists():
@@ -514,6 +545,8 @@ def build_rounds():
514545

515546

516547
def main():
548+
global RESULTS
549+
RESULTS = load_results()
517550
frameworks = load("frameworks.json") or {}
518551
langcolors = load("langcolors.json") or {}
519552
current = load("current.json") or {}
@@ -533,7 +566,7 @@ def main():
533566
for pid, label, blurb, explorer, scored, s, es in entries:
534567
present = []
535568
for c in explorer:
536-
rows = load(f"{pid}-{c}.json")
569+
rows = RESULTS.get(f"{pid}-{c}")
537570
if not rows:
538571
continue
539572
trimmed = []

scripts/migrate_results_layout.py

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
#!/usr/bin/env python3
2+
"""One-off: split site/data/<profile>-<conns>.json into per-framework files.
3+
4+
Every framework's results used to live in 52 shared arrays, so two pull
5+
requests saving results touched the same files and conflicted (#751). After
6+
this each framework owns exactly one file, and concurrent runs cannot collide.
7+
8+
site/data/baseline-512.json [ {actix...}, {hyper...}, ... ] ->
9+
site/data/results/actix.json { framework, results: { "baseline-512": {...} } }
10+
11+
Rows are copied verbatim, so the generated data.js is unchanged. Run once;
12+
rebuild_site_data.py writes the new layout from then on.
13+
"""
14+
15+
from __future__ import annotations
16+
import argparse
17+
import json
18+
import re
19+
import sys
20+
from pathlib import Path
21+
22+
# Files in site/data that are not per-profile result arrays.
23+
NON_RESULT = {"frameworks.json", "current.json", "langcolors.json"}
24+
25+
26+
def slug(name: str) -> str:
27+
"""Filename for a display name. Two entries ('aspnet-minimal + nginx' and
28+
'+ caddy') contain spaces and a plus, so names cannot be used directly."""
29+
s = re.sub(r"[^A-Za-z0-9._-]+", "-", name).strip("-")
30+
return s.lower() or "unnamed"
31+
32+
33+
def collect(site_data: Path) -> dict[str, dict]:
34+
"""{slug: {framework, results: {profile-conns: row}}} from the flat files."""
35+
out: dict[str, dict] = {}
36+
for f in sorted(site_data.glob("*.json")):
37+
if f.name in NON_RESULT:
38+
continue
39+
try:
40+
rows = json.loads(f.read_text(encoding="utf-8"))
41+
except Exception as e:
42+
print(f"[warn] skipping {f.name}: {e}", file=sys.stderr)
43+
continue
44+
if not isinstance(rows, list):
45+
continue
46+
key = f.stem # e.g. "baseline-512"
47+
for row in rows:
48+
if not isinstance(row, dict):
49+
continue
50+
name = row.get("framework")
51+
if not name:
52+
continue
53+
entry = out.setdefault(slug(name), {"framework": name, "results": {}})
54+
if entry["framework"] != name:
55+
print(f"[warn] slug collision: {entry['framework']!r} vs {name!r}", file=sys.stderr)
56+
entry["results"][key] = row
57+
return out
58+
59+
60+
def main() -> None:
61+
ap = argparse.ArgumentParser()
62+
ap.add_argument("--root", default=str(Path(__file__).resolve().parent.parent))
63+
ap.add_argument("--dry-run", action="store_true")
64+
args = ap.parse_args()
65+
66+
site_data = Path(args.root) / "site" / "data"
67+
results_dir = site_data / "results"
68+
69+
frameworks = collect(site_data)
70+
n_rows = sum(len(e["results"]) for e in frameworks.values())
71+
print(f"{len(frameworks)} frameworks, {n_rows} rows")
72+
73+
if args.dry_run:
74+
for s, e in sorted(frameworks.items())[:5]:
75+
print(f" {s}.json <- {e['framework']} ({len(e['results'])} rows)")
76+
return
77+
78+
results_dir.mkdir(parents=True, exist_ok=True)
79+
for s, entry in sorted(frameworks.items()):
80+
entry["results"] = dict(sorted(entry["results"].items()))
81+
(results_dir / f"{s}.json").write_text(json.dumps(entry, indent=2) + "\n", encoding="utf-8")
82+
83+
removed = 0
84+
for f in sorted(site_data.glob("*.json")):
85+
if f.name in NON_RESULT:
86+
continue
87+
f.unlink()
88+
removed += 1
89+
print(f"wrote {len(frameworks)} files to {results_dir.relative_to(Path(args.root))}, "
90+
f"removed {removed} flat files")
91+
92+
93+
if __name__ == "__main__":
94+
main()

0 commit comments

Comments
 (0)