From 59d2365441fb5b4f90762b6bb81b8bffeb534cb4 Mon Sep 17 00:00:00 2001 From: Hao Hu Date: Tue, 3 Nov 2026 02:14:36 +0100 Subject: [PATCH 1/6] Issue #1: perf-callgrind JVM-vs-rs spec (mirror snippet T1211) Per-namespace callgrind sweep over all 1351 test namespaces, both runtimes, Bencher overlay. Key adaptations: SUMMARY-line gate instead of stdout diff, vendored measure recipe with native discovery, new perf-callgrind.yml (ci.yml untouched). --- docs/1/specification.md | 233 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 233 insertions(+) create mode 100644 docs/1/specification.md diff --git a/docs/1/specification.md b/docs/1/specification.md new file mode 100644 index 0000000..fa25857 --- /dev/null +++ b/docs/1/specification.md @@ -0,0 +1,233 @@ +# 1: callgrind perf CI with JVM vs clojure-rs comparison + +## Part 1 — goal and acceptance criteria + +Goal: measure every test namespace under deterministic callgrind +for both JVM Clojure and clojure-rs +(`clojure-rs`, https://github.com/hhh-pa/clojure) — so +`callgrind_instructions` and `callgrind_allocated_bytes` for the +two runtimes overlay per benchmark on Bencher's perf page (same +benchmark names, distinct testbeds). This mirrors the snippet +repo's T1211 sweep (hhh-pa/clojure-code-snippet#37), adapted +from script-x-case benchmarks to test-namespace benchmarks. + +Acceptance criteria: + +- New `perf-callgrind.yml` runs a shard(8) x + runtime{jvm, clojure-rs} matrix; `report-callgrind` merges + per runtime; `ci.yml` `test` is unchanged. +- `algoperf measure --runtime clojure-rs --command-template + 'clojure-rs run perf/driver.clj {ns} src test' --callgrind` + reports both gungraun-run measures per namespace (valgrind + Ir + allocator-shim total); the JVM recipe behaves exactly + like T1211 (same flags, same parsers, same BMF shape). +- The per-namespace correctness gate is the SUMMARY line: + `:fail`/`:error` zero on both legs, and the report fails + when any namespace's counts differ between legs. +- `algoperf measure --self-test` reports 0 failed; `go test + ./...` is green; `python3 -c yaml.safe_load` passes on the + workflow; `git diff --check` is clean. +- The pushed branch shows green `test` + all 16 sweep legs + + both `report-callgrind` merges, with per-shard Bencher + reports for both runtimes and merged tables in the report + summaries. +- Every per-shard Bencher check stays under GitHub's ~64KB + summary limit. + +BDD scenarios (user-observable): + +- Given a push to the branch, when CI runs, then the + correctness gate in `test` stays green, and 16 sweep legs + measure deterministic instruction and allocation counts + for both runtimes over all 1351 test namespaces. +- Given Bencher's perf page, when the `ubuntu-latest-jvm` + and `ubuntu-latest-clojure-rs` testbeds are overlaid, then + each namespace shows JVM vs clojure-rs Ir and allocated + bytes side by side. + +Non-goals: + +- No `--slope` in the sweep (unchanged from T1211: slope + fits timed runs; the sweep stays lean) and no AOT leg for + clojure-rs (`clojure-rs build` comparison is future work). +- No timed `allocated_bytes` for the native runtime: it + stays null, exactly like the JVM sweep under `--no-alloc`. + Native allocation is reported only as + `callgrind_allocated_bytes`. +- No full-stdout diff gate: namespaces print + nondeterministically (timing, random samples, hash order), + so the SUMMARY line is the gate (see design). +- No changes to `src/`, `test/`, `deps.edn`, or `build.clj`: + this is a pure CI addition (`perf/`, `scripts/`, + `.github/workflows/`, `docs/` only). +- No Bencher-side threshold tuning past the T1211 1% + `callgrind_instructions` policy; no branch-protection edit + on this branch — both are merge-time admin steps (below). + +Merge prerequisites (repo admin, at merge time — not on this +branch): + +- Create the Bencher project `clojure-algorithms` and set + the `BENCHER_API_KEY` repository secret (per-shard + `bencher run` reports against it; without the secret the + Bencher steps fail — keep the PR in draft until the + secret exists, or accept red Bencher steps as known). +- Optionally require the new `perf-callgrind (...)` / + `report-callgrind (...)` checks in branch protection + (exact names taken from this PR's checks). + +## Part 2 — design + +### Benchmarks: one per test namespace + +1351 files match `test/**/*_test.clj`, and every file's `ns` +decl equals its path mapped by `test/`-strip, `/` -> `.`, +`_` -> `-` (verified programmatically over all 1351). +Benchmark names are the ns symbols — identical on both legs. + +### Driver (`perf/driver.clj [load-path...]`) + +Same driver both legs (identical workload): + +```clojure +(when-let [lp (seq (rest *command-line-args*))] + (alter-var-root #'clojure.core/*load-path* + (constantly (vec lp)))) +(require (symbol (first *command-line-args*))) +(let [sum (clojure.test/run-tests + (symbol (first *command-line-args*)))] + (println "SUMMARY" (select-keys sum [:test :pass :fail :error]))) +``` + +- JVM leg passes no load-path (the `-cp` covers `src`/`test`); + the native leg passes `src test` (verified: require + + run-tests + SUMMARY all work under `clojure-rs run`). +- No `System/exit` (unemulated on the native leg): the gate + parses the SUMMARY line and fails the case on `:fail`/`:error` + > 0 or a missing SUMMARY (require crashes exit non-zero + with no SUMMARY — same treatment). +- Proven outputs match: `{:test 2, :pass 10, :fail 0, + :error 0}` on both runtimes for the probe namespace. + +Invocations: + +- JVM: `java -cp ':src:test' clojure.main + perf/driver.clj ` (jars: clojure 1.12.0 + spec.alpha + 0.5.238 + core.specs.alpha 0.4.74, Maven-cached like the + snippet CI — no Clojure CLI needed, test-runner not + needed since the driver calls `clojure.test` directly). +- Native: `clojure-rs run perf/driver.clj src test`. + +### SUMMARY gate + cross-leg comparison + +- Per leg, per namespace: exit 0 + SUMMARY parsed + + `:fail`/`:error` zero, else the case fails loudly. +- At `report-callgrind` time: diff the per-namespace + `(test,pass,fail,error)` tuples JVM vs clojure-rs; any + mismatch fails the report (behavioral drift can never + hide behind perf numbers). Randomized tests keep + assertion counts stable, so equality is expected. +- Timeout degrades to `callgrind_skipped` with a reason + (visible in JSON, excluded from BMF) — the T1211 policy. + The timeout starts at 600s per namespace (heavy + namespaces under callgrind are uncalibrated; tune from + CI data). + +### `algoperf measure`: vendored recipe, native discovery + +New Go module `github.com/hhh-pa/clojure-algorithms/ +scripts/ci/go` with `cmd/algoperf` + a vendored +`internal/measure` subset, ported line-for-line from the +snippet repo's `scripts/ci/go/internal/measure`: + +- Keep (recipe-identical): valgrind callgrind flags, the + JVM `-Xint` + EpsilonGC splice, the LD_PRELOAD + counting-shim source and parser, the callgrind summary + parser, BMF emission, shard selection. +- Drop: `--aot` / `AOTCache` (aot-only), `--slope` / + complexity BMF (slope-only), snippet discovery/resources. +- Vendor alongside: `pyjson` (whole, small), + `snipcore` helpers actually used (`Run`/`RunEnv`, + `Normalize`, `PyRepr`, `TruncateChars`, `LastLine`, + `TimeoutError`, `ExitCode`). +- New: namespace discovery (walk `test/`, ns mapping), + SUMMARY parser + gate, thin CLI (`measure`, `--merge`, + `--self-test`), own self-tests (SUMMARY parser checks + first, following the `ParseCallgrindSummary` pattern). +- Semantic note (documented in README): the JVM number is + EpsilonGC heap-used-at-exit, the native number is + malloc-request bytes — ratio-meaningful per benchmark, + not bit-comparable absolutes (T1211 wording). + +### Workflow (`perf-callgrind.yml`) + +- `resolve-rs`: resolve clojure-rs master HEAD once per + run (float; manual dispatch may pin `clojure_rs_sha`, + 40-char SHA validated) — same as the snippet repo. +- `perf-callgrind` matrix: `shard: [0..7]` x + `runtime: [jvm, clojure-rs]` (16 legs). + - JVM leg: temurin 21 + cached Maven jars. + - Native leg: stable Rust (`dtolnay/rust-toolchain`), + checkout `hhh-pa/clojure` at the resolved SHA, `cargo + build --release --locked --no-default-features` in + `rust/` (interpreter only, no LLVM), `Swatinem/rust-cache`, + binary to `$GITHUB_WORKSPACE/cljrs-bin` + `$GITHUB_PATH`. + - Measure step: `--repeat 1 --no-alloc --callgrind + --callgrind-repeat 1 --jobs 2 --timeout 600 + --shard N/8`, runtime template per leg. + - Bencher step: T1211 threshold flags verbatim, project + `clojure-algorithms`, testbeds + `ubuntu-latest-{jvm,clojure-rs}`; `jq` strip + + artifact upload runtime-agnostic (names carry + runtime+shard). +- `report-callgrind` matrix over runtimes: merge + + `jq -s add` verify + SUMMARY cross-leg diff + upload, + merged tables in the summary. + +Shard math: namespaces are ms-scale test logic (max 175 +lines/file) behind runtime startup (rs 0.07s, JVM 0.37s +native locally), the same shape as snippet cases +(~21s all-in): 1351 x ~25s / 2 jobs / 8 shards ~= 35 +min/shard. 16 shards stay available as a future lever; +per-shard BMFs (~170 benchmarks x 2 measures) sit far +below the check-summary limit. + +### TDD plan + +- SUMMARY parser + gate checks first (valid line amid + noise, missing/failing SUMMARY -> error), then the + vendored recipe files with the snippet self-tests + adapted (parser checks stay subprocess-free; live + shim/valgrind checks `t.Skip` where `cc`/Linux/valgrind + are absent). +- Local gates: `go build`, `go vet`, `go test ./...`, + `algoperf measure --self-test`, plus a native-leg + end-to-end over a few namespaces (`--only`, no + callgrind — this host has no valgrind) and the JVM + driver run (java + jars work locally). +- Linux CI on the PR is the proof for the valgrind/shim + paths (same as T1211). +- Workflow validated by `python3 -c yaml.safe_load` and + a matrix/condition review. + +### CI impact + +- New workflow file only; `ci.yml` untouched, no + Jenkinsfile (this repo uses GitHub Actions). +- New CI dependencies on the native leg only: Rust + stable toolchain + `Swatinem/rust-cache` (no LLVM); + JVM leg adds a Maven jar cache. + +### Risks + +- `BENCHER_API_KEY` does not exist yet: Bencher steps + fail until the admin step lands (draft PR meanwhile). +- Heavy namespaces (blockchain, SMO, primelib) may blow + the 600s timeout under callgrind on the first sweep — + they degrade to skips with reasons, then the timeout + tunes from data. +- clojure-rs nondeterminism (hash seeds) surfaces as + SUMMARY-mismatch failures or Bencher wobble, never as + silent drift; the 1% threshold arms after 10 samples. +- If native callgrind runs slower than estimated, the + lever is shard count (8 -> 16), not recipe changes. From e66d4a298c8821e274c6888065073a1a34a72841 Mon Sep 17 00:00:00 2001 From: Hao Hu Date: Tue, 3 Nov 2026 02:36:35 +0100 Subject: [PATCH 2/6] Issue #1: algoperf tool, driver, workflow + 4 test-isolation fixes Perf-callgrind sweep for JVM vs clojure-rs (mirrors snippet T1211): per-namespace driver with SUMMARY gate, algoperf measure tool (ported recipe core: valgrind flags, JVM -Xint+EpsilonGC, LD_PRELOAD shim, BMF), perf-callgrind.yml (shard x runtime matrix, Bencher, merge + cross-leg count diff). The first full sweep surfaced 4 test-isolation bugs, fixed here: 3 missing :requires (passed only by ambient load order) and a fail-fast negative-exponent guard (replaces stack-exhaustion dependence; port gap tracked at hhh-pa/clojure#220). --- .github/actions/algoperf-tools/action.yml | 37 ++ .github/workflows/perf-callgrind.yml | 326 ++++++++++ README.md | 21 + docs/1/specification.md | 41 +- perf/driver.clj | 17 + scripts/ci/go/cmd/algoperf/discover.go | 81 +++ scripts/ci/go/cmd/algoperf/discover_test.go | 58 ++ scripts/ci/go/cmd/algoperf/flags.go | 102 ++++ scripts/ci/go/cmd/algoperf/main.go | 40 ++ scripts/ci/go/cmd/algoperf/measure.go | 364 +++++++++++ scripts/ci/go/go.mod | 3 + scripts/ci/go/internal/algocore/algocore.go | 21 + .../ci/go/internal/algocore/algocore_test.go | 33 + scripts/ci/go/internal/algocore/check.go | 8 + scripts/ci/go/internal/algocore/classpath.go | 59 ++ scripts/ci/go/internal/algocore/exec.go | 71 +++ scripts/ci/go/internal/algocore/normalize.go | 27 + .../ci/go/internal/algocore/pgroup_unix.go | 20 + .../ci/go/internal/algocore/pgroup_windows.go | 14 + scripts/ci/go/internal/algocore/pyrepr.go | 49 ++ scripts/ci/go/internal/algocore/shard.go | 36 ++ scripts/ci/go/internal/algocore/text.go | 25 + scripts/ci/go/internal/measure/allocshim.go | 320 ++++++++++ scripts/ci/go/internal/measure/bmf.go | 89 +++ scripts/ci/go/internal/measure/exec.go | 568 ++++++++++++++++++ scripts/ci/go/internal/measure/measure.go | 33 + scripts/ci/go/internal/measure/parsers.go | 138 +++++ scripts/ci/go/internal/measure/report.go | 206 +++++++ scripts/ci/go/internal/measure/selftest.go | 111 ++++ .../ci/go/internal/measure/selftest_test.go | 55 ++ scripts/ci/go/internal/measure/shlex.go | 70 +++ scripts/ci/go/internal/measure/shlex_test.go | 43 ++ scripts/ci/go/internal/measure/summarize.go | 157 +++++ scripts/ci/go/internal/measure/summary.go | 78 +++ .../ci/go/internal/measure/summary_test.go | 64 ++ scripts/ci/go/internal/measure/table.go | 191 ++++++ .../go/internal/measure/testdata/shlex.json | 184 ++++++ scripts/ci/go/internal/pyjson/pyjson.go | 317 ++++++++++ src/clojure_algorithms/ciphers/xor_cipher.clj | 3 +- src/clojure_algorithms/graphs/karger.clj | 3 +- .../maths/power_using_recursion.clj | 12 +- .../disjoint_set/disjoint_set_test.clj | 1 + 42 files changed, 4073 insertions(+), 23 deletions(-) create mode 100644 .github/actions/algoperf-tools/action.yml create mode 100644 .github/workflows/perf-callgrind.yml create mode 100644 perf/driver.clj create mode 100644 scripts/ci/go/cmd/algoperf/discover.go create mode 100644 scripts/ci/go/cmd/algoperf/discover_test.go create mode 100644 scripts/ci/go/cmd/algoperf/flags.go create mode 100644 scripts/ci/go/cmd/algoperf/main.go create mode 100644 scripts/ci/go/cmd/algoperf/measure.go create mode 100644 scripts/ci/go/go.mod create mode 100644 scripts/ci/go/internal/algocore/algocore.go create mode 100644 scripts/ci/go/internal/algocore/algocore_test.go create mode 100644 scripts/ci/go/internal/algocore/check.go create mode 100644 scripts/ci/go/internal/algocore/classpath.go create mode 100644 scripts/ci/go/internal/algocore/exec.go create mode 100644 scripts/ci/go/internal/algocore/normalize.go create mode 100644 scripts/ci/go/internal/algocore/pgroup_unix.go create mode 100644 scripts/ci/go/internal/algocore/pgroup_windows.go create mode 100644 scripts/ci/go/internal/algocore/pyrepr.go create mode 100644 scripts/ci/go/internal/algocore/shard.go create mode 100644 scripts/ci/go/internal/algocore/text.go create mode 100644 scripts/ci/go/internal/measure/allocshim.go create mode 100644 scripts/ci/go/internal/measure/bmf.go create mode 100644 scripts/ci/go/internal/measure/exec.go create mode 100644 scripts/ci/go/internal/measure/measure.go create mode 100644 scripts/ci/go/internal/measure/parsers.go create mode 100644 scripts/ci/go/internal/measure/report.go create mode 100644 scripts/ci/go/internal/measure/selftest.go create mode 100644 scripts/ci/go/internal/measure/selftest_test.go create mode 100644 scripts/ci/go/internal/measure/shlex.go create mode 100644 scripts/ci/go/internal/measure/shlex_test.go create mode 100644 scripts/ci/go/internal/measure/summarize.go create mode 100644 scripts/ci/go/internal/measure/summary.go create mode 100644 scripts/ci/go/internal/measure/summary_test.go create mode 100644 scripts/ci/go/internal/measure/table.go create mode 100644 scripts/ci/go/internal/measure/testdata/shlex.json create mode 100644 scripts/ci/go/internal/pyjson/pyjson.go diff --git a/.github/actions/algoperf-tools/action.yml b/.github/actions/algoperf-tools/action.yml new file mode 100644 index 0000000..764599a --- /dev/null +++ b/.github/actions/algoperf-tools/action.yml @@ -0,0 +1,37 @@ +name: algoperf-tools +description: > + Provide the algoperf binary from a prebuilt binary kept in the + Actions cache (rebuilt on miss); no external dependencies beyond + setup-go. Callers invoke algoperf-bin/algoperf themselves. +inputs: + cache-key: + description: > + Full cache key for the prebuilt algoperf binary, shared by + every job on the OS: algoperf--, where the hash + covers this action's sources plus exactly the Go inputs the + build reads (go.mod, cmd/algoperf, the internal packages + algoperf imports). + required: true +runs: + using: composite + steps: + - id: cache + uses: actions/cache@v5 + with: + path: algoperf-bin + key: ${{ inputs.cache-key }} + - if: steps.cache.outputs.cache-hit != 'true' + uses: actions/setup-go@v6 + with: + go-version-file: scripts/ci/go/go.mod + cache: false + # NO working-directory KEY anywhere in this manifest: a build + # step with an explicit working-directory key can silently + # produce no file, while the identical command with an + # in-script cd works. Every path uses the quoted + # $GITHUB_WORKSPACE env. + - if: steps.cache.outputs.cache-hit != 'true' + shell: bash + run: | + cd "$GITHUB_WORKSPACE/scripts/ci/go" + CGO_ENABLED=0 go build -o "$GITHUB_WORKSPACE/algoperf-bin/algoperf" ./cmd/algoperf diff --git a/.github/workflows/perf-callgrind.yml b/.github/workflows/perf-callgrind.yml new file mode 100644 index 0000000..eb325e0 --- /dev/null +++ b/.github/workflows/perf-callgrind.yml @@ -0,0 +1,326 @@ +name: perf-callgrind + +on: + push: + branches: [main, master] + pull_request: + workflow_dispatch: + inputs: + clojure_rs_sha: + description: 'Pin clojure-rs to this revision instead of master HEAD (40-char SHA; empty = float)' + required: false + default: '' + +# Full-matrix callgrind sweep: every test namespace's gungraun-run +# measures (callgrind_instructions + callgrind_allocated_bytes) on +# every push, for JVM Clojure and clojure-rs alike. This is the +# only perf signal. Lean recipe (--repeat 1, --no-alloc since the +# callgrind phase itself yields allocation): ~25s/namespace all-in +# on CI hardware x ~170 namespaces/shard / 2 jobs ~= 35min per +# shard. A valgrind/toolchain failure fails the push: the sweep is +# a required signal. +# +# The perf-callgrind native leg always builds clojure-rs master HEAD +# (interpreter-only build; `run` needs no LLVM), resolved once per +# run by the resolve-rs job below. A bad upstream commit fails +# loudly (the sweep SUMMARY-gates every namespace), and Bencher +# shows the delta; to bisect or repro, dispatch manually with +# `clojure_rs_sha`. + +jobs: + # Resolve the clojure-rs revision ONCE per run so all 16 sweep + # legs build the same commit even if upstream master moves + # mid-sweep. Default: master HEAD (float); a manual dispatch may + # pin `clojure_rs_sha` instead. + resolve-rs: + runs-on: ubuntu-latest + permissions: + contents: read + outputs: + sha: ${{ steps.resolve.outputs.sha }} + steps: + - id: resolve + run: | + if [ -n "${{ inputs.clojure_rs_sha }}" ]; then + sha="${{ inputs.clojure_rs_sha }}" + echo "using dispatch-pinned revision" + else + sha=$(git ls-remote https://github.com/hhh-pa/clojure HEAD | awk '{print $1}') + fi + if ! printf '%s' "$sha" | grep -Eq '^[0-9a-f]{40}$'; then + echo "error: resolved clojure-rs revision is not a 40-char SHA: '$sha'" >&2 + exit 1 + fi + echo "sha=$sha" >> "$GITHUB_OUTPUT" + echo "clojure-rs revision: \`$sha\`" >> "$GITHUB_STEP_SUMMARY" + + # Correctness gate for the sweep (same steps as ci.yml): never + # sweep a red tree. (A cross-workflow `needs` is impossible, so + # the job is duplicated here; keep the two in sync.) + test: + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - uses: actions/checkout@v4 + + - name: Set up Java + uses: actions/setup-java@v4 + with: + distribution: temurin + java-version: "21" + + - name: Set up Clojure + uses: DeLaGuardo/setup-clojure@13.4 + with: + cli: latest + + - name: Fetch dependencies + run: clojure -P + + - name: Check formatting + run: clojure -M:cljfmt + + - name: Run tests + run: clojure -M:test + + perf-callgrind: + needs: [test, resolve-rs] + runs-on: ubuntu-latest + permissions: + contents: read + checks: write + pull-requests: write + strategy: + # Let every shard finish so partial artifacts upload for diagnosis. + fail-fast: false + matrix: + # One entry per compiler/runtime; benchmark names stay + # identical across runtimes (Bencher compares via testbeds, + # derived below as ubuntu-latest-). Keep the + # runtime list in sync with the report-callgrind job below. + shard: [0, 1, 2, 3, 4, 5, 6, 7] + runtime: [jvm, clojure-rs] + steps: + - uses: actions/checkout@v4 + - uses: ./.github/actions/algoperf-tools + with: + cache-key: algoperf-${{ runner.os }}-${{ hashFiles('.github/actions/algoperf-tools/**', 'scripts/ci/go/go.mod', 'scripts/ci/go/cmd/algoperf/**', 'scripts/ci/go/internal/algocore/**', 'scripts/ci/go/internal/pyjson/**', 'scripts/ci/go/internal/measure/**') }} + + - name: Set up JDK 21 + if: matrix.runtime == 'jvm' + uses: actions/setup-java@v4 + with: + distribution: temurin + java-version: '21' + + - name: Cache Clojure jars + if: matrix.runtime == 'jvm' + id: jars-cache + uses: actions/cache@v5 + with: + path: .ci/jars + key: clojure-jars-1.12.0-spec0.5.238-core0.4.74-v1 + + - name: Download Clojure jars + if: matrix.runtime == 'jvm' && steps.jars-cache.outputs.cache-hit != 'true' + run: | + mkdir -p .ci/jars + base=https://repo1.maven.org/maven2 + curl -fsSL -o .ci/jars/clojure.jar "$base/org/clojure/clojure/1.12.0/clojure-1.12.0.jar" + curl -fsSL -o .ci/jars/spec.alpha.jar "$base/org/clojure/spec.alpha/0.5.238/spec.alpha-0.5.238.jar" + curl -fsSL -o .ci/jars/core.specs.alpha.jar "$base/org/clojure/core.specs.alpha/0.4.74/core.specs.alpha-0.4.74.jar" + + # clojure-rs leg: interpreter-only release build (no LLVM: + # `run` needs just the interpreter crates). The revision is + # master HEAD resolved once per run (see resolve-rs); the + # sweep SUMMARY-gates every namespace, so a bad upstream + # commit fails loudly, and Bencher shows the perf delta. + - name: Install Rust toolchain + if: matrix.runtime == 'clojure-rs' + uses: dtolnay/rust-toolchain@stable + + - name: Check out clojure-rs + if: matrix.runtime == 'clojure-rs' + uses: actions/checkout@v4 + with: + repository: hhh-pa/clojure + ref: ${{ needs.resolve-rs.outputs.sha }} + path: clojure-rs + + - name: Cache cargo build + if: matrix.runtime == 'clojure-rs' + uses: Swatinem/rust-cache@v2 + with: + workspaces: clojure-rs/rust + + - name: Build clojure-rs + if: matrix.runtime == 'clojure-rs' + run: | + cd clojure-rs/rust + cargo build --release --locked --no-default-features + mkdir -p "$GITHUB_WORKSPACE/cljrs-bin" + cp target/release/clojure-rs "$GITHUB_WORKSPACE/cljrs-bin/" + echo "$GITHUB_WORKSPACE/cljrs-bin" >> "$GITHUB_PATH" + + - name: Install measurement tools + uses: awalsh128/cache-apt-pkgs-action@v1.6.3 + with: + # gcc builds algoperf's LD_PRELOAD allocation-counting + # shim on demand (native legs); preinstalled on the + # hosted image, listed here so the dependency is + # explicit. + packages: time linux-tools-common linux-tools-generic valgrind gcc + version: 1.0 + + - name: Measure callgrind sweep (deterministic metrics) + # Lean recipe: one timed run (SUMMARY gate) + the callgrind + # phase per namespace (JVM: one -Xint+EpsilonGC run; native: + # one plain callgrind run plus one LD_PRELOAD counting run). + # No --alloc (the callgrind phase itself yields + # allocation). + run: | + flags=(--runtime "${{ matrix.runtime }}") + if [ "${{ matrix.runtime }}" = "clojure-rs" ]; then + flags+=(--command-template 'clojure-rs run perf/driver.clj {ns} src test resources') + else + jars=".ci/jars/clojure.jar:.ci/jars/spec.alpha.jar:.ci/jars/core.specs.alpha.jar" + flags+=(--cp "$jars:src:test:resources") + fi + "$GITHUB_WORKSPACE/algoperf-bin/algoperf" measure \ + --repeat 1 --no-alloc --jobs 2 --timeout 1800 \ + --callgrind-timeout 600 \ + "${flags[@]}" \ + --shard ${{ matrix.shard }}/8 \ + --callgrind --callgrind-repeat 1 \ + --output sweep-results.json --bmf-output sweep-bmf.json \ + --summary-file "$GITHUB_STEP_SUMMARY" + + - name: Install Bencher CLI + uses: bencherdev/bencher@main + + # Threshold policy: callgrind_instructions (percentage) only, + # same flags on both testbeds (the clojure-rs testbed starts + # with empty history and arms after 10 samples). Both + # gungraun-run measures are tracked; the overlapping + # cpu_time/peak_memory entries are stripped before reporting + # (the sweep is repeat-1 timing, not a timing signal; no + # allocated_bytes here: --no-alloc). A fixed 1% upper + # boundary sits far above callgrind wobble; min/max sample + # 10/64. callgrind_allocated_bytes rides threshold-free + # until its own wobble is characterized. Each shard reports + # its own stripped subset (~170 benchmarks x 2 measures). + - name: Track callgrind benchmarks with Bencher + run: | + jq 'with_entries(select(.value.callgrind_instructions != null and .value.callgrind_allocated_bytes != null) | .value |= {callgrind_instructions, callgrind_allocated_bytes})' \ + sweep-bmf.json > sweep-bench.json + if [ "$(jq 'length' sweep-bench.json)" = "0" ]; then + echo "no callgrind data in shard; skipping Bencher report" + exit 0 + fi + if [ "${{ github.event_name }}" = "pull_request" ]; then + bencher run \ + --project clojure-algorithms \ + --key '${{ secrets.BENCHER_API_KEY }}' \ + --branch "${{ github.head_ref }}" \ + --hash "${{ github.event.pull_request.head.sha }}" \ + --start-point "${{ github.base_ref }}" \ + --start-point-hash "${{ github.event.pull_request.base.sha }}" \ + --start-point-clone-thresholds \ + --testbed ubuntu-latest-${{ matrix.runtime }} \ + --adapter json \ + --file sweep-bench.json \ + --github-actions "${{ secrets.GITHUB_TOKEN }}" + else + bencher run \ + --project clojure-algorithms \ + --key '${{ secrets.BENCHER_API_KEY }}' \ + --branch "${{ github.ref_name }}" \ + --testbed ubuntu-latest-${{ matrix.runtime }} \ + --threshold-measure callgrind_instructions \ + --threshold-test percentage \ + --threshold-min-sample-size 10 \ + --threshold-max-sample-size 64 \ + --threshold-upper-boundary 0.01 \ + --adapter json \ + --file sweep-bench.json \ + --github-actions "${{ secrets.GITHUB_TOKEN }}" + fi + + - name: Upload sweep results + if: always() + uses: actions/upload-artifact@v4 + with: + name: sweep-results-${{ matrix.runtime }}-shard-${{ matrix.shard }} + path: | + sweep-results.json + sweep-bmf.json + sweep-bench.json + + report-callgrind: + needs: perf-callgrind + runs-on: ubuntu-latest + # No Bencher run here by design: one merged ~1351-benchmark + # file would re-truncate the check summary. + permissions: + contents: read + # One merge per runtime; keep in sync with the perf-callgrind + # job's runtime list above. + strategy: + matrix: + runtime: [jvm, clojure-rs] + steps: + - uses: actions/checkout@v4 + - uses: ./.github/actions/algoperf-tools + with: + cache-key: algoperf-${{ runner.os }}-${{ hashFiles('.github/actions/algoperf-tools/**', 'scripts/ci/go/go.mod', 'scripts/ci/go/cmd/algoperf/**', 'scripts/ci/go/internal/algocore/**', 'scripts/ci/go/internal/pyjson/**', 'scripts/ci/go/internal/measure/**') }} + + - name: Download sweep results + uses: actions/download-artifact@v4 + with: + pattern: sweep-results-${{ matrix.runtime }}-shard-* + + - name: Merge sweep reports + run: | + "$GITHUB_WORKSPACE/algoperf-bin/algoperf" measure --merge sweep-results-*/sweep-results.json \ + --output sweep-results-merged.json \ + --bmf-output sweep-bmf-merged.json \ + --summary-file "$GITHUB_STEP_SUMMARY" + + # The merge must equal the plain BMF union: sharding is + # namespace-cohesive, so no benchmark straddles two shards. + - name: Verify merge matches per-shard BMFs + run: | + jq -s 'add' sweep-results-*/sweep-bmf.json > sweep-bmf-jq.json + diff <(jq -S . sweep-bmf-merged.json) <(jq -S . sweep-bmf-jq.json) + + - name: Upload merged sweep results + if: always() + uses: actions/upload-artifact@v4 + with: + name: sweep-results-${{ matrix.runtime }}-merged + path: | + sweep-results-merged.json + sweep-bmf-merged.json + + # Behavioral drift gate: every namespace's SUMMARY counts must + # match between the two runtimes. Perf numbers mean nothing if + # the two legs ran different tests. + report-compare: + needs: report-callgrind + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Download merged sweep results + uses: actions/download-artifact@v4 + with: + pattern: sweep-results-*-merged + + - name: Diff JVM vs native test counts + run: | + for leg in jvm clojure-rs; do + jq -S '{(.results[] | select(.ok) | .label): .test_counts}' \ + sweep-results-$leg-merged/sweep-results-merged.json > $leg-counts.json + done + diff jvm-counts.json clojure-rs-counts.json diff --git a/README.md b/README.md index f8d681b..48f6f20 100644 --- a/README.md +++ b/README.md @@ -115,6 +115,27 @@ resources/ | web_programming | `web-programming` | 35 | 35 | | **Total** | | **1,350** | **1,350** | +## Perf (JVM vs clojure-rs) + +Every push runs a deterministic callgrind sweep over all test +namespaces under both JVM Clojure and +[clojure-rs](https://github.com/hhh-pa/clojure) +(`.github/workflows/perf-callgrind.yml`), reporting +`callgrind_instructions` and `callgrind_allocated_bytes` per +namespace to Bencher (same benchmark names, distinct testbeds — +see `docs/1/specification.md`). + +- Each namespace runs `perf/driver.clj`, which prints one + `SUMMARY {:test :pass :fail :error}` line; the sweep gates on + it (failures fail loudly), and a report job diffs the counts + between runtimes so behavioral drift can never hide behind + perf numbers. +- Semantic note: the JVM allocation number is EpsilonGC + heap-used-at-exit, the native number is malloc-request bytes + (LD_PRELOAD counting shim). Both are "total requested + allocation" for the same program run — ratio-meaningful per + benchmark, not bit-comparable absolutes. + ## License MIT — see [LICENSE](LICENSE). diff --git a/docs/1/specification.md b/docs/1/specification.md index fa25857..6e1b23f 100644 --- a/docs/1/specification.md +++ b/docs/1/specification.md @@ -17,7 +17,7 @@ Acceptance criteria: runtime{jvm, clojure-rs} matrix; `report-callgrind` merges per runtime; `ci.yml` `test` is unchanged. - `algoperf measure --runtime clojure-rs --command-template - 'clojure-rs run perf/driver.clj {ns} src test' --callgrind` + 'clojure-rs run perf/driver.clj {ns} src test resources' --callgrind` reports both gungraun-run measures per namespace (valgrind Ir + allocator-shim total); the JVM recipe behaves exactly like T1211 (same flags, same parsers, same BMF shape). @@ -57,9 +57,14 @@ Non-goals: - No full-stdout diff gate: namespaces print nondeterministically (timing, random samples, hash order), so the SUMMARY line is the gate (see design). -- No changes to `src/`, `test/`, `deps.edn`, or `build.clj`: - this is a pure CI addition (`perf/`, `scripts/`, - `.github/workflows/`, `docs/` only). +- No changes to `deps.edn` or `build.clj`. The `src/` + + `test/` changes on this branch are limited to the four + test-isolation fixes the sweep itself surfaced (three + missing `:require`s that only passed by ambient load + order, plus a fail-fast negative-exponent guard replacing + a stack-exhaustion dependency — the underlying + abort-vs-`StackOverflowError` port gap is tracked at + hhh-pa/clojure#220). - No Bencher-side threshold tuning past the T1211 1% `callgrind_instructions` policy; no branch-protection edit on this branch — both are merge-time admin steps (below). @@ -100,7 +105,7 @@ Same driver both legs (identical workload): ``` - JVM leg passes no load-path (the `-cp` covers `src`/`test`); - the native leg passes `src test` (verified: require + + the native leg passes `src test resources` (verified: require + run-tests + SUMMARY all work under `clojure-rs run`). - No `System/exit` (unemulated on the native leg): the gate parses the SUMMARY line and fails the case on `:fail`/`:error` @@ -111,27 +116,31 @@ Same driver both legs (identical workload): Invocations: -- JVM: `java -cp ':src:test' clojure.main +- JVM: `java -cp ':src:test:resources' clojure.main perf/driver.clj ` (jars: clojure 1.12.0 + spec.alpha 0.5.238 + core.specs.alpha 0.4.74, Maven-cached like the snippet CI — no Clojure CLI needed, test-runner not needed since the driver calls `clojure.test` directly). -- Native: `clojure-rs run perf/driver.clj src test`. +- Native: `clojure-rs run perf/driver.clj src test resources`. ### SUMMARY gate + cross-leg comparison - Per leg, per namespace: exit 0 + SUMMARY parsed + `:fail`/`:error` zero, else the case fails loudly. - At `report-callgrind` time: diff the per-namespace - `(test,pass,fail,error)` tuples JVM vs clojure-rs; any - mismatch fails the report (behavioral drift can never - hide behind perf numbers). Randomized tests keep - assertion counts stable, so equality is expected. -- Timeout degrades to `callgrind_skipped` with a reason - (visible in JSON, excluded from BMF) — the T1211 policy. - The timeout starts at 600s per namespace (heavy - namespaces under callgrind are uncalibrated; tune from - CI data). + `(test,pass,fail,error)` tuples JVM vs clojure-rs (a + dedicated `report-compare` job diffs the recorded + `test_counts` maps with jq); any mismatch fails the + report (behavioral drift can never hide behind perf + numbers). Randomized tests keep assertion counts stable, + so equality is expected. +- Timeouts are split: `--timeout` (default 1800s) budgets + the native-speed runs (whose SUMMARY gate must stay + decisive — the slowest namespace needs ~11min on the + native leg), while `--callgrind-timeout` (default 600s) + budgets the instrumented phase, which degrades to + `callgrind_skipped` with a reason (visible in JSON, + excluded from BMF) — the T1211 policy. Tune from CI data. ### `algoperf measure`: vendored recipe, native discovery diff --git a/perf/driver.clj b/perf/driver.clj new file mode 100644 index 0000000..5fbac53 --- /dev/null +++ b/perf/driver.clj @@ -0,0 +1,17 @@ +;; Per-namespace benchmark driver for the perf-callgrind sweep (issue #1). +;; Same driver on both legs (identical workload): +;; JVM: java -cp ':src:test' clojure.main perf/driver.clj +;; native: clojure-rs run perf/driver.clj src test +;; The optional load-path args exist for the native leg only (the JVM +;; leg resolves via -cp). Prints one SUMMARY line; the sweep gates on +;; it (missing SUMMARY or :fail/:error > 0 fails the case). No +;; System/exit (unemulated natively) — the gate reads stdout. +(when-let [lp (seq (rest *command-line-args*))] + ;; Resolved dynamically: the JVM leg never passes load-path + ;; args, and JVM Clojure has no `*load-path*` var at all (a + ;; static `#'` reference would fail the whole script there). + (when-let [v (resolve 'clojure.core/*load-path*)] + (alter-var-root v (constantly (vec lp))))) +(require (symbol (first *command-line-args*))) +(let [sum (clojure.test/run-tests (symbol (first *command-line-args*)))] + (println "SUMMARY" (select-keys sum [:test :pass :fail :error]))) diff --git a/scripts/ci/go/cmd/algoperf/discover.go b/scripts/ci/go/cmd/algoperf/discover.go new file mode 100644 index 0000000..9789503 --- /dev/null +++ b/scripts/ci/go/cmd/algoperf/discover.go @@ -0,0 +1,81 @@ +package main + +import ( + "fmt" + "os" + "path/filepath" + "sort" + "strings" +) + +// Namespace is one benchmark: a test file plus its ns symbol. +type Namespace struct { + // Ns is the full ns symbol, e.g. + // clojure-algorithms.searches.linear-search-test. + Ns string + // Stem is the file stem, e.g. linear_search_test. + Stem string + // File is the test file path, repo-root-relative. + File string +} + +// DiscoverNamespaces walks test/ for *_test.clj files and maps each +// to its ns symbol (test/-strip, '/' -> '.', '_' -> '-'), verifying +// the mapping against the file's ns decl. Results sort by ns. +func DiscoverNamespaces(root string) ([]Namespace, error) { + testDir := filepath.Join(root, "test") + var out []Namespace + err := filepath.WalkDir(testDir, func(path string, d os.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() || !strings.HasSuffix(d.Name(), "_test.clj") { + return nil + } + rel, err := filepath.Rel(testDir, path) + if err != nil { + return err + } + mapped := strings.TrimSuffix(rel, ".clj") + mapped = strings.ReplaceAll(mapped, string(os.PathSeparator), ".") + mapped = strings.ReplaceAll(mapped, "_", "-") + decl, err := readNsDecl(path) + if err != nil { + return fmt.Errorf("%s: %v", rel, err) + } + if decl != mapped { + return fmt.Errorf("%s: ns decl %q does not match path mapping %q", rel, decl, mapped) + } + out = append(out, Namespace{ + Ns: mapped, + Stem: strings.TrimSuffix(d.Name(), ".clj"), + File: filepath.ToSlash(rel), + }) + return nil + }) + if err != nil { + return nil, err + } + sort.Slice(out, func(i, j int) bool { return out[i].Ns < out[j].Ns }) + return out, nil +} + +// readNsDecl reads the ns symbol from a file's ns decl (the first +// `(ns ` form near the top of the file). +func readNsDecl(path string) (string, error) { + raw, err := os.ReadFile(path) + if err != nil { + return "", err + } + text := string(raw) + idx := strings.Index(text, "(ns ") + if idx < 0 { + return "", fmt.Errorf("no (ns ...) decl found") + } + rest := strings.TrimSpace(text[idx+len("(ns "):]) + end := strings.IndexAny(rest, " \t\r\n)]") + if end < 0 { + return "", fmt.Errorf("malformed (ns ...) decl") + } + return rest[:end], nil +} diff --git a/scripts/ci/go/cmd/algoperf/discover_test.go b/scripts/ci/go/cmd/algoperf/discover_test.go new file mode 100644 index 0000000..4bfc645 --- /dev/null +++ b/scripts/ci/go/cmd/algoperf/discover_test.go @@ -0,0 +1,58 @@ +package main + +import ( + "os" + "path/filepath" + "testing" +) + +func TestDiscoverNamespaces(t *testing.T) { + root := t.TempDir() + write := func(rel, ns string) { + p := filepath.Join(root, "test", rel) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + body := "(ns " + ns + "\n (:require [clojure.test :refer [deftest]]))\n" + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + write("clojure_algorithms/searches/linear_search_test.clj", + "clojure-algorithms.searches.linear-search-test") + write("clojure_algorithms/core_test.clj", "clojure-algorithms.core-test") + // Non-test files are ignored. + other := filepath.Join(root, "test", "clojure_algorithms", "helper.clj") + if err := os.WriteFile(other, []byte("(ns helper)\n"), 0o644); err != nil { + t.Fatal(err) + } + + found, err := DiscoverNamespaces(root) + if err != nil { + t.Fatal(err) + } + if len(found) != 2 { + t.Fatalf("got %d namespaces, want 2: %+v", len(found), found) + } + // Sorted by ns. + if found[0].Ns != "clojure-algorithms.core-test" || found[0].Stem != "core_test" { + t.Fatalf("first: %+v", found[0]) + } + if found[1].Ns != "clojure-algorithms.searches.linear-search-test" || found[1].Stem != "linear_search_test" { + t.Fatalf("second: %+v", found[1]) + } +} + +func TestDiscoverNamespacesRejectsMismatch(t *testing.T) { + root := t.TempDir() + p := filepath.Join(root, "test", "clojure_algorithms", "odd_test.clj") + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte("(ns something.else)\n"), 0o644); err != nil { + t.Fatal(err) + } + if _, err := DiscoverNamespaces(root); err == nil { + t.Fatal("ns/path mismatch must error") + } +} diff --git a/scripts/ci/go/cmd/algoperf/flags.go b/scripts/ci/go/cmd/algoperf/flags.go new file mode 100644 index 0000000..ba28d01 --- /dev/null +++ b/scripts/ci/go/cmd/algoperf/flags.go @@ -0,0 +1,102 @@ +package main + +import ( + "errors" + "fmt" + "strconv" + "strings" +) + +// flagSpec declares one subcommand's flags. Values land in Strs, Ints, +// and Bools; Positionals collects the rest. +type flagSpec struct { + Strs map[string]*string + Ints map[string]*int + Bools map[string]*bool +} + +// parseFlags parses argv in long-option shape: --name value, +// --name=value, and the single-dash spellings, plus --no-name +// negation for default-true bools. Unknown flags, missing values, +// and bad ints are errors. +func parseFlags(sub string, argv []string, spec flagSpec) (positionals []string, err error) { + fail := func(format string, args ...any) ([]string, error) { + return nil, fmt.Errorf("algoperf "+sub+": "+format, args...) + } + + i := 0 + for i < len(argv) { + arg := argv[i] + if len(arg) < 2 || arg[0] != '-' { + positionals = append(positionals, arg) + i++ + continue + } + if arg == "--" { + positionals = append(positionals, argv[i+1:]...) + break + } + name := strings.TrimLeft(arg, "-") + var value string + hasValue := false + if idx := strings.IndexByte(name, '='); idx >= 0 { + value = name[idx+1:] + name = name[:idx] + hasValue = true + } + if name == "h" || name == "help" { + return nil, errHelp + } + if target, ok := spec.Bools[name]; ok { + if hasValue { + return fail("argument --%s: ignored explicit argument %q", name, value) + } + *target = true + i++ + continue + } + if strings.HasPrefix(name, "no-") { + if target, ok := spec.Bools[name[len("no-"):]]; ok { + if hasValue { + return fail("argument --%s: ignored explicit argument %q", name, value) + } + *target = false + i++ + continue + } + } + if target, ok := spec.Strs[name]; ok { + if !hasValue { + i++ + if i >= len(argv) { + return fail("argument --%s: expected one argument", name) + } + value = argv[i] + } + *target = value + i++ + continue + } + if target, ok := spec.Ints[name]; ok { + if !hasValue { + i++ + if i >= len(argv) { + return fail("argument --%s: expected one argument", name) + } + value = argv[i] + } + n, convErr := strconv.Atoi(strings.TrimSpace(value)) + if convErr != nil { + return fail("argument --%s: invalid int value: %q", name, value) + } + *target = n + i++ + continue + } + return fail("unknown flag --%s", name) + } + return positionals, nil +} + +// errHelp signals -h/--help; callers print their usage and exit 0. +var errHelp = errors.New("help") diff --git a/scripts/ci/go/cmd/algoperf/main.go b/scripts/ci/go/cmd/algoperf/main.go new file mode 100644 index 0000000..79a4805 --- /dev/null +++ b/scripts/ci/go/cmd/algoperf/main.go @@ -0,0 +1,40 @@ +// Command algoperf measures the algorithms repo's test namespaces +// under load-independent instruments (GNU time, perf, valgrind +// callgrind) for JVM-vs-native comparison: +// +// algoperf measure per-namespace perf sweep (see measure.go) +// +// Run from the repository root. +package main + +import ( + "fmt" + "os" +) + +const usage = `Usage: algoperf [flags] + +Subcommands: + measure per-namespace perf sweep +` + +func main() { + os.Exit(dispatch(os.Args[1:])) +} + +func dispatch(argv []string) int { + if len(argv) == 0 { + fmt.Print(usage) + return 2 + } + switch argv[0] { + case "-h", "--help", "help": + fmt.Print(usage) + return 0 + case "measure": + return runMeasure(argv[1:]) + default: + fmt.Fprintf(os.Stderr, "algoperf: unknown subcommand %q\n%s", argv[0], usage) + return 2 + } +} diff --git a/scripts/ci/go/cmd/algoperf/measure.go b/scripts/ci/go/cmd/algoperf/measure.go new file mode 100644 index 0000000..a24cf8d --- /dev/null +++ b/scripts/ci/go/cmd/algoperf/measure.go @@ -0,0 +1,364 @@ +package main + +import ( + "errors" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "sync" + "sync/atomic" + "time" + + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/algocore" + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/measure" + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/pyjson" +) + +const measureUsage = `Load-independent performance measurement for the algorithms repo. + +Runs perf/driver.clj once per test namespace and records cpu_s, +maxrss_kb, allocated_bytes, instructions/cycles, plus (under the +new flags) callgrind instruction counts. + +Usage: algoperf measure [--cp CP] [--repeat N] [--jobs N] [--timeout S] + [--only SUBSTR] [--output F] [--bmf-output F] + [--summary-file F] [--time-bin BIN] [--no-perf] + [--runtime NAME] [--command-template T] [--shard K/N] + [--alloc-repeat N | --no-alloc] + [--callgrind] [--callgrind-repeat N] + [--callgrind-timeout S] + + algoperf measure --merge INPUT... [--output F] [--bmf-output F] + [--summary-file F] + + algoperf measure --self-test +` + +func runMeasure(argv []string) int { + root := algocore.RepoRoot() + var cp, only, output, bmfOutput, summaryFile, timeBin, shard string + runtimeName := "jvm" + template := measure.DefaultCommandTemplate + jobs := 1 + timeout := 1800 + repeat := 1 + allocRepeat := measure.AllocRepeatDefault + callgrindRepeat := measure.CallgrindRepeatDefault + callgrindTimeout := 600 + perf := true + alloc := true + merge := false + selfTest := false + callgrind := false + positionals, err := parseFlags("measure", argv, flagSpec{ + Strs: map[string]*string{ + "cp": &cp, "only": &only, + "output": &output, "bmf-output": &bmfOutput, + "summary-file": &summaryFile, "time-bin": &timeBin, + "runtime": &runtimeName, "command-template": &template, + "shard": &shard, + }, + Ints: map[string]*int{ + "jobs": &jobs, "timeout": &timeout, "repeat": &repeat, + "alloc-repeat": &allocRepeat, + "callgrind-repeat": &callgrindRepeat, + "callgrind-timeout": &callgrindTimeout, + }, + Bools: map[string]*bool{ + "perf": &perf, "alloc": &alloc, "merge": &merge, + "self-test": &selfTest, "callgrind": &callgrind, + }, + }) + if err != nil { + if errors.Is(err, errHelp) { + fmt.Print(measureUsage) + return 0 + } + fmt.Fprintln(os.Stderr, err) + return 2 + } + if selfTest { + return measure.SelfTest(os.Stdout) + } + if merge { + return runMerge(positionals, output, bmfOutput, summaryFile) + } + if repeat < 1 { + fmt.Fprintln(os.Stderr, "algoperf measure: --repeat must be >= 1") + return 2 + } + if allocRepeat < 1 { + fmt.Fprintln(os.Stderr, "algoperf measure: --alloc-repeat must be >= 1") + return 2 + } + if jobs < 1 { + fmt.Fprintln(os.Stderr, "algoperf measure: --jobs must be >= 1") + return 2 + } + if callgrindRepeat < 1 { + fmt.Fprintln(os.Stderr, "algoperf measure: --callgrind-repeat must be >= 1") + return 2 + } + if callgrindTimeout < 1 { + fmt.Fprintln(os.Stderr, "algoperf measure: --callgrind-timeout must be >= 1") + return 2 + } + var shardParsed *[2]int + if shard != "" { + idx, total, err := algocore.ParseMeasureShard(shard) + if err != nil { + fmt.Fprintln(os.Stderr, "algoperf measure:", err) + return 2 + } + shardParsed = &[2]int{idx, total} + } + + timeBinPath, err := measure.ResolveTimeBin(timeBin) + if err != nil { + fmt.Fprintln(os.Stderr, err) + return 1 + } + usePerf := perf && measure.PerfUsable() + if perf && !usePerf { + fmt.Fprintln(os.Stderr, "warning: perf counters unavailable; instructions/cycles will be null") + } + useCallgrind := callgrind && measure.CallgrindUsable() + if callgrind && !useCallgrind { + fmt.Fprintln(os.Stderr, "warning: valgrind unavailable; callgrind_instructions will be null") + } + classpath, err := algocore.ResolveClasspath(root, cp) + if err != nil { + if !algocore.TemplateNeedsClasspath(template) { + fmt.Fprintln(os.Stderr, "warning: no Clojure classpath; continuing without one (template has no {cp} hole)") + classpath = "" + } else { + fmt.Fprintln(os.Stderr, err) + return 1 + } + } + found, err := DiscoverNamespaces(root) + if err != nil { + fmt.Fprintln(os.Stderr, err) + return 1 + } + + type task struct { + ns, stem string + } + var tasks []task + for _, entry := range found { + if only != "" && !strings.Contains(entry.Ns, only) { + continue + } + tasks = append(tasks, task{entry.Ns, entry.Stem}) + } + if shardParsed != nil { + tasks = measure.SelectShard(tasks, func(t task) string { + return t.ns + }, shardParsed[0], shardParsed[1]) + } + + effectiveAlloc := 0 + if alloc { + effectiveAlloc = allocRepeat + } + if effectiveAlloc > 0 && len(tasks) > 0 { + t0 := tasks[0] + if !measure.AllocSupported(template, classpath, t0.ns) { + fmt.Fprintln(os.Stderr, "warning: allocation measurement needs a java command template; skipping (BMF will omit allocated_bytes)") + effectiveAlloc = 0 + } + } + // Callgrind runs for JVM and native legs alike: the recipe + // branches on the resolved argv0 (EpsilonGC log vs LD_PRELOAD + // counting). Only valgrind's absence disables it. + effectiveCg := 0 + if useCallgrind { + effectiveCg = callgrindRepeat + } + + driver := filepath.Join(root, "perf", "driver.clj") + results := make([]measure.CaseResult, len(tasks)) + work := make(chan int) + total := len(tasks) + var done atomic.Int64 + var printMu sync.Mutex + var workers sync.WaitGroup + for w := 0; w < jobs; w++ { + workers.Add(1) + go func() { + defer workers.Done() + for i := range work { + t := tasks[i] + results[i] = measure.MeasureCase(measure.CaseOpts{ + TimeBin: timeBinPath, UsePerf: usePerf, Template: template, + Classpath: classpath, Label: t.ns, Stem: t.stem, + Script: driver, Case: "run", Ns: t.ns, Repeat: repeat, + AllocRepeat: effectiveAlloc, CallgrindRepeat: effectiveCg, + Timeout: timeout, CallgrindTimeout: callgrindTimeout, Root: root, + }) + n := done.Add(1) + printMu.Lock() + fmt.Println(formatCaseProgress(int(n), total, results[i])) + printMu.Unlock() + } + }() + } + go func() { + for i := range tasks { + work <- i + } + close(work) + }() + workers.Wait() + sort.Slice(results, func(i, j int) bool { + if results[i].Label != results[j].Label { + return results[i].Label < results[j].Label + } + if results[i].Stem != results[j].Stem { + return results[i].Stem < results[j].Stem + } + return results[i].Case < results[j].Case + }) + failures := []string{} + for _, r := range results { + if !r.Ok && r.Error != nil { + failures = append(failures, *r.Error) + } + } + sort.Strings(failures) + + javaLeg := false + nativeBin := "" + if len(tasks) > 0 { + t0 := tasks[0] + if inner, err := measure.BuildInnerCommand(template, classpath, t0.ns); err == nil && len(inner) > 0 { + if measure.InnerIsJava(inner) { + javaLeg = true + } else { + nativeBin = inner[0] + } + } + } else { + javaLeg = measure.TemplateHeadIsJava(template) + } + var javaVersion, runtimeVersion *string + if javaLeg { + javaVersion = measure.JavaVersion() + } else if nativeBin != "" { + runtimeVersion = measure.RuntimeVersion(nativeBin) + } + bmf := measure.BuildBMF(results) + report := measure.Report{ + Tool: "algoperf measure", Version: measure.ToolVersion, + TimestampUTC: time.Now().UTC().Format("2006-01-02T15:04:05.000000-07:00"), + Host: measure.HostInfo(), + Runtime: measure.Runtime{ + Name: runtimeName, CommandTemplate: template, + JavaVersion: javaVersion, RuntimeVersion: runtimeVersion, + Classpath: classpath, + TimeBin: timeBinPath, PerfUsed: usePerf, CallgrindUsed: effectiveCg > 0, + }, + Config: measure.Config{ + Repeat: repeat, Jobs: jobs, Timeout: timeout, Only: only, + Shard: strOrNil(shard), AllocRepeat: &effectiveAlloc, + CallgrindRepeat: effectiveCg, CallgrindTimeout: callgrindTimeout, + }, + Results: results, Failures: failures, + } + if output != "" { + if err := writeJSON(output, report); err != nil { + fmt.Fprintln(os.Stderr, err) + return 1 + } + } + if bmfOutput != "" { + if err := writeJSON(bmfOutput, bmf); err != nil { + fmt.Fprintln(os.Stderr, err) + return 1 + } + } + if summaryFile != "" { + if err := measure.AppendSummary(summaryFile, report, bmf); err != nil { + fmt.Fprintln(os.Stderr, err) + return 1 + } + } + measure.PrintTable(os.Stdout, results, failures) + if len(failures) > 0 { + return 1 + } + return 0 +} + +func strOrNil(s string) *string { + if s == "" { + return nil + } + return &s +} + +// formatCaseProgress renders one live progress line for a finished +// namespace. +func formatCaseProgress(done, total int, res measure.CaseResult) string { + outcome := "ok" + if !res.Ok { + outcome = "failed" + if res.Error != nil { + outcome = *res.Error + } + } else if res.CallgrindSkipped != nil { + outcome = "ok (callgrind skipped: " + *res.CallgrindSkipped + ")" + } + return fmt.Sprintf("measure [%d/%d] %s/%s/%s: %s", + done, total, res.Label, res.Stem, res.Case, outcome) +} + +func runMerge(inputs []string, output, bmfOutput, summaryFile string) int { + if len(inputs) == 0 { + fmt.Fprintln(os.Stderr, "algoperf measure: --merge needs at least one input report") + return 2 + } + report, bmf, err := measure.MergeReports(inputs) + if err != nil { + fmt.Fprintln(os.Stderr, err) + return 1 + } + if output != "" { + if err := writeJSON(output, report); err != nil { + fmt.Fprintln(os.Stderr, err) + return 1 + } + } + if bmfOutput != "" { + if err := writeJSON(bmfOutput, bmf); err != nil { + fmt.Fprintln(os.Stderr, err) + return 1 + } + } + if summaryFile != "" { + if err := measure.AppendSummary(summaryFile, report, bmf); err != nil { + fmt.Fprintln(os.Stderr, err) + return 1 + } + } + fmt.Printf("merge: %d results, %d benchmarks, %d failures\n", + len(report.Results), len(bmf), len(report.Failures)) + if len(report.Failures) > 0 { + return 1 + } + return 0 +} + +func writeJSON(path string, v any) error { + raw, err := pyjson.MarshalIndent2(v) + if err != nil { + return fmt.Errorf("cannot write %s: %v", path, err) + } + if err := os.WriteFile(path, raw, 0o644); err != nil { + return fmt.Errorf("cannot write %s: %v", path, err) + } + return nil +} diff --git a/scripts/ci/go/go.mod b/scripts/ci/go/go.mod new file mode 100644 index 0000000..06671fc --- /dev/null +++ b/scripts/ci/go/go.mod @@ -0,0 +1,3 @@ +module github.com/hhh-pa/clojure-algorithms/scripts/ci/go + +go 1.27 diff --git a/scripts/ci/go/internal/algocore/algocore.go b/scripts/ci/go/internal/algocore/algocore.go new file mode 100644 index 0000000..3648f15 --- /dev/null +++ b/scripts/ci/go/internal/algocore/algocore.go @@ -0,0 +1,21 @@ +// Package algocore holds the logic shared by the algoperf +// subcommands: repository discovery, classpath resolution, output +// normalization, subprocess execution, and shard selection. +// Ported from the snippet repo's snipcore package (same behavior: +// messages, ordering, quirks); snippet-case discovery stays behind. +package algocore + +import ( + "os" +) + +// RepoRoot returns the repository root, which for the Go tools is the +// process working directory: algoperf runs from the repo root (CI +// checks out and runs there; local runs do the same). +func RepoRoot() string { + wd, err := os.Getwd() + if err != nil { + return "." + } + return wd +} diff --git a/scripts/ci/go/internal/algocore/algocore_test.go b/scripts/ci/go/internal/algocore/algocore_test.go new file mode 100644 index 0000000..4c2f642 --- /dev/null +++ b/scripts/ci/go/internal/algocore/algocore_test.go @@ -0,0 +1,33 @@ +package algocore + +import ( + "testing" +) + +func TestParseMeasureShard(t *testing.T) { + idx, total, err := ParseMeasureShard("0/8") + if err != nil || idx != 0 || total != 8 { + t.Fatalf("parse 0/8: %d %d %v", idx, total, err) + } + idx, total, err = ParseMeasureShard("7/8") + if err != nil || idx != 7 || total != 8 { + t.Fatalf("parse 7/8: %d %d %v", idx, total, err) + } + for _, bad := range []string{"", "8/8", "0/0", "x/y", "1"} { + if _, _, err := ParseMeasureShard(bad); err == nil { + t.Fatalf("shard %q must error", bad) + } + } +} + +func TestTemplateNeedsClasspath(t *testing.T) { + if !TemplateNeedsClasspath("java -cp {cp} clojure.main {ns}") { + t.Fatal("{cp} hole not detected") + } + if TemplateNeedsClasspath("clojure-rs run perf/driver.clj {ns} src test") { + t.Fatal("false positive on native template") + } + if TemplateNeedsClasspath("echo {{cp}}") { + t.Fatal("escaped hole must not count") + } +} diff --git a/scripts/ci/go/internal/algocore/check.go b/scripts/ci/go/internal/algocore/check.go new file mode 100644 index 0000000..b6109c3 --- /dev/null +++ b/scripts/ci/go/internal/algocore/check.go @@ -0,0 +1,8 @@ +package algocore + +// Check is one named self-test assertion. +type Check struct { + Name string + Ok bool + Detail string +} diff --git a/scripts/ci/go/internal/algocore/classpath.go b/scripts/ci/go/internal/algocore/classpath.go new file mode 100644 index 0000000..4ea5003 --- /dev/null +++ b/scripts/ci/go/internal/algocore/classpath.go @@ -0,0 +1,59 @@ +package algocore + +import ( + "errors" + "os" + "path/filepath" + "sort" + "strings" +) + +// ResolveClasspath resolves the JVM classpath: explicit --cp wins, +// then $CLOJURE_CP, then .ci/jars/*.jar, then $CLOJURE_JAR plus its +// sibling spec jars; otherwise the same fatal message. +func ResolveClasspath(root, explicit string) (string, error) { + if explicit != "" { + return explicit, nil + } + if env := strings.TrimSpace(os.Getenv("CLOJURE_CP")); env != "" { + return env, nil + } + jars, err := filepath.Glob(filepath.Join(root, ".ci", "jars", "*.jar")) + if err == nil && len(jars) > 0 { + sort.Strings(jars) + return strings.Join(jars, string(os.PathListSeparator)), nil + } + if jar := strings.TrimSpace(os.Getenv("CLOJURE_JAR")); jar != "" { + dir := filepath.Dir(jar) + candidates := []string{jar} + for _, pattern := range []string{"spec.alpha*.jar", "core.specs*.jar"} { + extra, err := filepath.Glob(filepath.Join(dir, pattern)) + if err == nil { + sort.Strings(extra) + candidates = append(candidates, extra...) + } + } + return strings.Join(candidates, string(os.PathListSeparator)), nil + } + return "", errors.New("no Clojure classpath: pass --cp, set CLOJURE_CP/CLOJURE_JAR, or put jars under .ci/jars/") +} + +// TemplateNeedsClasspath reports whether a command template +// references the {cp} hole, honoring {{/}} escapes like +// ExpandTemplate. algoperf measure tolerates a missing Clojure +// classpath when the template never uses it (native legs). +func TemplateNeedsClasspath(template string) bool { + for i := 0; i < len(template); { + if template[i] == '{' { + if i+1 < len(template) && template[i+1] == '{' { + i += 2 + continue + } + if strings.HasPrefix(template[i:], "{cp}") { + return true + } + } + i++ + } + return false +} diff --git a/scripts/ci/go/internal/algocore/exec.go b/scripts/ci/go/internal/algocore/exec.go new file mode 100644 index 0000000..d8005f1 --- /dev/null +++ b/scripts/ci/go/internal/algocore/exec.go @@ -0,0 +1,71 @@ +package algocore + +import ( + "bytes" + "errors" + "os" + "os/exec" + "time" +) + +// TimeoutError marks a subprocess that exceeded its deadline, so +// callers can format the TIMEOUT message. +type TimeoutError struct { + Seconds int +} + +func (e *TimeoutError) Error() string { + return "timeout" +} + +// Run executes argv with stdin bytes on stdin, capturing stdout and +// stderr, killing the whole process group after timeoutSecs (so +// perf/time/valgrind grandchildren never outlive a timeout). It +// returns stdout, stderr, and either nil, *TimeoutError, or +// *exec.ExitError. +// +// The group kill fires from the deadline timer itself rather than +// after Wait: grandchildren inherit the capture pipes, and Wait +// blocks for pipe EOF, so killing after Wait would hang until the +// grandchildren exit on their own. +func Run(argv []string, stdin []byte, timeoutSecs int) (stdout, stderr []byte, err error) { + return RunEnv(argv, stdin, timeoutSecs, nil) +} + +// RunEnv is Run with extra child-process environment entries +// (KEY=value strings, appended after os.Environ so they win on +// duplicates). The native callgrind recipe uses it to inject +// LD_PRELOAD for the allocation-counting run. +func RunEnv(argv []string, stdin []byte, timeoutSecs int, extraEnv []string) (stdout, stderr []byte, err error) { + cmd := exec.Command(argv[0], argv[1:]...) + if len(extraEnv) > 0 { + cmd.Env = append(os.Environ(), extraEnv...) + } + setProcessGroup(cmd) + if stdin != nil { + cmd.Stdin = bytes.NewReader(stdin) + } + var outBuf, errBuf bytes.Buffer + cmd.Stdout = &outBuf + cmd.Stderr = &errBuf + timer := time.AfterFunc(time.Duration(timeoutSecs)*time.Second, func() { + if cmd.Process != nil { + KillProcessGroup(cmd.Process.Pid) + } + }) + + runErr := cmd.Run() + if !timer.Stop() { + return nil, nil, &TimeoutError{Seconds: timeoutSecs} + } + return outBuf.Bytes(), errBuf.Bytes(), runErr +} + +// ExitCode extracts the process exit code from a Run error (0 when nil). +func ExitCode(err error) int { + var exitErr *exec.ExitError + if errors.As(err, &exitErr) { + return exitErr.ExitCode() + } + return 0 +} diff --git a/scripts/ci/go/internal/algocore/normalize.go b/scripts/ci/go/internal/algocore/normalize.go new file mode 100644 index 0000000..0abdca7 --- /dev/null +++ b/scripts/ci/go/internal/algocore/normalize.go @@ -0,0 +1,27 @@ +package algocore + +import ( + "strings" +) + +// asciiSpace is the bytes.strip() set: space, tab, newline, +// carriage return, vertical tab, form feed. +const asciiSpace = " \t\n\r\x0b\x0c" + +// Normalize compares program output line-wise: split on \r\n, \r, +// \n with ASCII-whitespace stripping of every line. +func Normalize(content []byte) []byte { + text := string(content) + text = strings.ReplaceAll(text, "\r\n", "\n") + text = strings.ReplaceAll(text, "\r", "\n") + lines := strings.Split(text, "\n") + // A trailing line break ends the last line without adding an + // empty one ("a\n" -> ["a"], "a\n\n" -> ["a", ""]). + if strings.HasSuffix(text, "\n") { + lines = lines[:len(lines)-1] + } + for i, line := range lines { + lines[i] = strings.Trim(line, asciiSpace) + } + return []byte(strings.Join(lines, "\n")) +} diff --git a/scripts/ci/go/internal/algocore/pgroup_unix.go b/scripts/ci/go/internal/algocore/pgroup_unix.go new file mode 100644 index 0000000..b1bb276 --- /dev/null +++ b/scripts/ci/go/internal/algocore/pgroup_unix.go @@ -0,0 +1,20 @@ +//go:build !windows + +package algocore + +import ( + "os/exec" + "syscall" +) + +// setProcessGroup makes the child a group leader so KillProcessGroup +// can reap valgrind/perf/time grandchildren on timeout. +func setProcessGroup(cmd *exec.Cmd) { + cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} +} + +// KillProcessGroup kills the whole group of a timed-out child. +// Errors (already dead) are ignored by design. +func KillProcessGroup(pid int) { + _ = syscall.Kill(-pid, syscall.SIGKILL) +} diff --git a/scripts/ci/go/internal/algocore/pgroup_windows.go b/scripts/ci/go/internal/algocore/pgroup_windows.go new file mode 100644 index 0000000..322b410 --- /dev/null +++ b/scripts/ci/go/internal/algocore/pgroup_windows.go @@ -0,0 +1,14 @@ +//go:build windows + +package algocore + +import ( + "os/exec" +) + +// setProcessGroup is a no-op on Windows (no process groups); +// timeouts there kill only the direct child. +func setProcessGroup(cmd *exec.Cmd) {} + +// KillProcessGroup is a no-op on Windows. +func KillProcessGroup(pid int) {} diff --git a/scripts/ci/go/internal/algocore/pyrepr.go b/scripts/ci/go/internal/algocore/pyrepr.go new file mode 100644 index 0000000..5210f7e --- /dev/null +++ b/scripts/ci/go/internal/algocore/pyrepr.go @@ -0,0 +1,49 @@ +package algocore + +import ( + "fmt" + "strings" + "unicode" + "unicode/utf8" +) + +// PyRepr formats text the way Python's repr() formats str: single +// quotes unless the text contains a quote but no double quote, with +// short escapes for whitespace and \x/\u/\U escapes for the rest. +func PyRepr(text string) string { + quote := "'" + if strings.Contains(text, "'") && !strings.Contains(text, `"`) { + quote = `"` + } + var sb strings.Builder + sb.WriteString(quote) + for _, r := range text { + switch r { + case '\\': + sb.WriteString(`\\`) + case '\n': + sb.WriteString(`\n`) + case '\t': + sb.WriteString(`\t`) + case '\r': + sb.WriteString(`\r`) + default: + if rune(quote[0]) == r { + sb.WriteByte('\\') + sb.WriteRune(r) + } else if r < utf8.RuneSelf && (r < 0x20 || r == 0x7f) { + fmt.Fprintf(&sb, `\x%02x`, r) + } else if unicode.IsPrint(r) { + sb.WriteRune(r) + } else if r < 0x100 { + fmt.Fprintf(&sb, `\x%02x`, r) + } else if r < 0x10000 { + fmt.Fprintf(&sb, `\u%04x`, r) + } else { + fmt.Fprintf(&sb, `\U%08x`, r) + } + } + } + sb.WriteString(quote) + return sb.String() +} diff --git a/scripts/ci/go/internal/algocore/shard.go b/scripts/ci/go/internal/algocore/shard.go new file mode 100644 index 0000000..099ea21 --- /dev/null +++ b/scripts/ci/go/internal/algocore/shard.go @@ -0,0 +1,36 @@ +package algocore + +import ( + "errors" + "strconv" + "strings" +) + +// ParseMeasureShard parses the 0-based K/N --shard. +func ParseMeasureShard(text string) (index, total int, err error) { + picked, total, ok := splitShard(text) + if !ok { + return 0, 0, errors.New("bad --shard '" + text + "': want K/N") + } + if total < 1 || picked < 0 || picked >= total { + return 0, 0, errors.New("bad --shard '" + text + "': need 0 <= K < N") + } + return picked, total, nil +} + +func splitShard(text string) (picked, total int, ok bool) { + parts := strings.Split(text, "/") + if len(parts) != 2 { + return 0, 0, false + } + picked, err1 := parsePyInt(parts[0]) + total, err2 := parsePyInt(parts[1]) + if err1 != nil || err2 != nil { + return 0, 0, false + } + return picked, total, true +} + +func parsePyInt(text string) (int, error) { + return strconv.Atoi(strings.TrimSpace(text)) +} diff --git a/scripts/ci/go/internal/algocore/text.go b/scripts/ci/go/internal/algocore/text.go new file mode 100644 index 0000000..3160a78 --- /dev/null +++ b/scripts/ci/go/internal/algocore/text.go @@ -0,0 +1,25 @@ +package algocore + +import ( + "strings" +) + +// LastLine takes the last non-blank stderr line for subprocess +// failure diagnostics ("no stderr" when empty). +func LastLine(stderr []byte) string { + text := strings.TrimSpace(string(stderr)) + if text == "" { + return "no stderr" + } + lines := strings.Split(text, "\n") + return strings.TrimSpace(lines[len(lines)-1]) +} + +// TruncateChars cuts text to max runes. +func TruncateChars(text string, max int) string { + runes := []rune(text) + if len(runes) > max { + return string(runes[:max]) + } + return text +} diff --git a/scripts/ci/go/internal/measure/allocshim.go b/scripts/ci/go/internal/measure/allocshim.go new file mode 100644 index 0000000..d202291 --- /dev/null +++ b/scripts/ci/go/internal/measure/allocshim.go @@ -0,0 +1,320 @@ +package measure + +import ( + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "runtime" + "strconv" + "strings" + "sync" + + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/algocore" +) + +// allocShimSource is the LD_PRELOAD counting shim (Linux/glibc +// only). It interposes the libc allocation entry points, sums +// requested bytes, and a destructor reports the total on fd 2 as +// "snip_alloc_total_bytes=N". Freed bytes are never subtracted, so +// the total matches EpsilonGC heap-used-at-exit semantics: +// cumulative allocation for the whole process run. Deterministic +// programs yield deterministic totals; exit-path allocations after +// the destructor (if any) are missed identically every run, so +// they shift the total by a constant, never by noise. +// +// The marker keeps its historical `snip_` name: this file is a +// byte-identical port of the snippet repo's shim, and the marker +// is internal wire protocol between the C source and +// ParseAllocShimTotal below. +const allocShimSource = ` +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include + +static atomic_ullong total_bytes = 0; + +/* Bootstrap pool: dlsym(RTLD_NEXT, ...) itself allocates, so + * allocations arriving before init completes are served from here + * (uncounted) instead of recursing. 64KB dwarfs dlsym's needs. */ +#define POOL_SIZE (64u * 1024u) +_Alignas(64) static char bootstrap_pool[POOL_SIZE]; +static atomic_size_t pool_used = 0; +static volatile int resolving = 0; +static void *(*real_malloc)(size_t); +static void (*real_free)(void *); +static void *(*real_calloc)(size_t, size_t); +static void *(*real_realloc)(void *, size_t); +static void *(*real_reallocarray)(void *, size_t, size_t); +static int (*real_posix_memalign)(void **, size_t, size_t); +static void *(*real_aligned_alloc)(size_t, size_t); + +static void *pool_alloc(size_t size) { + if (size == 0) + size = 1; + size = (size + 15) & ~(size_t)15; + size_t off = atomic_fetch_add(&pool_used, size); + if (off + size > POOL_SIZE) + return NULL; + return bootstrap_pool + off; +} + +static int in_pool(const void *p) { + const char *c = (const char *)p; + return c >= bootstrap_pool && c < bootstrap_pool + POOL_SIZE; +} + +static size_t pool_tail(const void *p) { + return (size_t)(bootstrap_pool + POOL_SIZE - (const char *)p); +} + +static void resolve_all(void) { + real_malloc = dlsym(RTLD_NEXT, "malloc"); + real_free = dlsym(RTLD_NEXT, "free"); + real_calloc = dlsym(RTLD_NEXT, "calloc"); + real_realloc = dlsym(RTLD_NEXT, "realloc"); + real_reallocarray = dlsym(RTLD_NEXT, "reallocarray"); + real_posix_memalign = dlsym(RTLD_NEXT, "posix_memalign"); + real_aligned_alloc = dlsym(RTLD_NEXT, "aligned_alloc"); +} + +/* True once the real entry points are resolved. Reentrant calls + * during resolution (dlsym allocates) take the pool path. */ +static int ensure_init(void) { + if (real_malloc) + return 1; + if (resolving) + return 0; + resolving = 1; + resolve_all(); + resolving = 0; + return real_malloc != 0; +} + +__attribute__((constructor)) static void shim_init(void) { + resolving = 1; + resolve_all(); + resolving = 0; +} + +void *malloc(size_t size) { + if (!ensure_init()) + return pool_alloc(size); + void *p = real_malloc(size); + if (p) + atomic_fetch_add(&total_bytes, (unsigned long long)size); + return p; +} + +void free(void *p) { + if (!p || in_pool(p)) + return; + if (!ensure_init()) + return; + real_free(p); +} + +void *calloc(size_t n, size_t size) { + if (!ensure_init()) { + if (size != 0 && n > POOL_SIZE / size) + return NULL; + void *p = pool_alloc(n * size); + if (p) + memset(p, 0, n * size); + return p; + } + void *p = real_calloc(n, size); + if (p) + atomic_fetch_add(&total_bytes, (unsigned long long)n * (unsigned long long)size); + return p; +} + +void *realloc(void *ptr, size_t size) { + if (ptr && in_pool(ptr)) { + /* Bootstrap block outliving init: re-home it. The old + * size is unknown, so the copy clamps to the pool tail; + * glibc never reallocs its bootstrap blocks, so this + * path is unreachable in practice. */ + void *p = ensure_init() ? real_malloc(size ? size : 1) : pool_alloc(size); + if (p) { + size_t tail = pool_tail(ptr); + memcpy(p, ptr, size < tail ? size : tail); + if (real_malloc) + atomic_fetch_add(&total_bytes, (unsigned long long)size); + } + return p; + } + if (!ensure_init()) + return pool_alloc(size); + void *p = real_realloc(ptr, size); + if (p) + atomic_fetch_add(&total_bytes, (unsigned long long)size); + return p; +} + +void *reallocarray(void *ptr, size_t n, size_t size) { + if (size != 0 && n > (size_t)-1 / size) { + errno = ENOMEM; + return NULL; + } + if (real_reallocarray || ensure_init()) { + if (real_reallocarray && !(ptr && in_pool(ptr))) { + void *p = real_reallocarray(ptr, n, size); + if (p) + atomic_fetch_add(&total_bytes, (unsigned long long)n * (unsigned long long)size); + return p; + } + } + return realloc(ptr, n * size); +} + +int posix_memalign(void **memptr, size_t alignment, size_t size) { + if (!ensure_init() || !real_posix_memalign) { + if (alignment < sizeof(void *) || (alignment & (alignment - 1)) != 0) + return 22; /* EINVAL */ + if (size + alignment < size) + return 12; /* ENOMEM */ + void *raw = pool_alloc(size + alignment); + if (!raw) + return 12; /* ENOMEM */ + uintptr_t a = ((uintptr_t)raw + alignment - 1) & ~(uintptr_t)(alignment - 1); + *memptr = (void *)a; + return 0; + } + int r = real_posix_memalign(memptr, alignment, size); + if (r == 0) + atomic_fetch_add(&total_bytes, (unsigned long long)size); + return r; +} + +void *aligned_alloc(size_t alignment, size_t size) { + if (!ensure_init() || !real_aligned_alloc) { + void *p = NULL; + int r = posix_memalign(&p, alignment, size); + if (r != 0) { + errno = r; + return NULL; + } + return p; + } + void *p = real_aligned_alloc(alignment, size); + if (p) + atomic_fetch_add(&total_bytes, (unsigned long long)size); + return p; +} + +__attribute__((destructor)) static void shim_fini(void) { + static const char marker[] = "snip_alloc_total_bytes="; + char buf[64]; + unsigned long long v = atomic_load(&total_bytes); + size_t n = sizeof(marker) - 1; + memcpy(buf, marker, n); + char rev[24]; + int r = 0; + if (v == 0) + rev[r++] = '0'; + while (v > 0) { + rev[r++] = (char)('0' + (v % 10)); + v /= 10; + } + while (r > 0) + buf[n++] = rev[--r]; + buf[n++] = '\n'; + ssize_t w = write(2, buf, n); + (void)w; +} +` + +// InnerIsJava reports whether a resolved driver argv runs the JVM. +func InnerIsJava(inner []string) bool { + return len(inner) > 0 && filepath.Base(inner[0]) == "java" +} + +// TemplateHeadIsJava reports whether a command template's argv0 is +// the JVM. Used only for provenance when no namespace resolved an +// argv. +func TemplateHeadIsJava(template string) bool { + fields := strings.Fields(template) + if len(fields) == 0 { + return false + } + return filepath.Base(strings.Trim(fields[0], `"'`)) == "java" +} + +var ( + allocShimOnce sync.Once + allocShimPath string + allocShimErr error +) + +// EnsureAllocShim compiles the counting shim once per process and +// returns the .so path. Linux-only: LD_PRELOAD/glibc +// interposition has no macOS equivalent (and valgrind, the shim's +// partner, is Linux-only in practice too). +func EnsureAllocShim() (string, error) { + if runtime.GOOS != "linux" { + return "", errors.New("alloc shim needs Linux (LD_PRELOAD)") + } + allocShimOnce.Do(func() { + allocShimPath, allocShimErr = buildAllocShim() + }) + return allocShimPath, allocShimErr +} + +func buildAllocShim() (string, error) { + cc, err := exec.LookPath("cc") + if err != nil { + return "", errors.New("alloc shim needs a C compiler (cc not found)") + } + dir, err := os.MkdirTemp("", "algoperf-alloc-shim-") + if err != nil { + return "", err + } + src := filepath.Join(dir, "alloc_shim.c") + if err := os.WriteFile(src, []byte(allocShimSource), 0o644); err != nil { + return "", err + } + out := filepath.Join(dir, "alloc_shim.so") + _, stderr, err := algocore.Run([]string{cc, "-shared", "-fPIC", "-O2", "-o", out, src, "-ldl"}, nil, 120) + if err != nil { + return "", fmt.Errorf("alloc shim build failed: %s", algocore.LastLine(stderr)) + } + return out, nil +} + +// ParseAllocShimTotal extracts the shim's total from child stderr. +// The destructor prints exactly one line; the last marker match +// wins, so a program that echoes the marker cannot shadow the +// real total. +func ParseAllocShimTotal(text string) (int64, error) { + const marker = "snip_alloc_total_bytes=" + var value int64 + var ok bool + var bad string + for _, line := range strings.Split(text, "\n") { + trimmed := strings.TrimSpace(line) + if !strings.HasPrefix(trimmed, marker) { + continue + } + n, err := strconv.ParseInt(strings.TrimPrefix(trimmed, marker), 10, 64) + if err != nil || n < 0 { + bad = trimmed + continue + } + value, ok = n, true + } + if !ok { + if bad != "" { + return 0, fmt.Errorf("alloc-shim total unreadable: %s", algocore.PyRepr(bad)) + } + return 0, fmt.Errorf("alloc-shim total unreadable: no snip_alloc_total_bytes line") + } + return value, nil +} diff --git a/scripts/ci/go/internal/measure/bmf.go b/scripts/ci/go/internal/measure/bmf.go new file mode 100644 index 0000000..6fed7c6 --- /dev/null +++ b/scripts/ci/go/internal/measure/bmf.go @@ -0,0 +1,89 @@ +package measure + +import ( + "sort" + + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/pyjson" +) + +// Measure is one BMF measure (value with lower/upper bounds). +type Measure struct { + Value pyjson.Num `json:"value"` + LowerValue pyjson.Num `json:"lower_value"` + UpperValue pyjson.Num `json:"upper_value"` +} + +// Benchmark is one namespace's BMF entry. +type Benchmark struct { + CpuTime *Measure `json:"cpu_time,omitempty"` + PeakMemory *Measure `json:"peak_memory,omitempty"` + AllocatedBytes *Measure `json:"allocated_bytes,omitempty"` + CallgrindInstructions *Measure `json:"callgrind_instructions,omitempty"` + CallgrindAllocatedBytes *Measure `json:"callgrind_allocated_bytes,omitempty"` +} + +// BMF maps benchmark name (the ns symbol) to measures. +type BMF map[string]Benchmark + +// BuildBMF builds one Benchmark per namespace with +// cpu_time/peak_memory plus allocated_bytes, +// callgrind_instructions, and callgrind_allocated_bytes when the +// namespace reports them. +func BuildBMF(results []CaseResult) BMF { + out := BMF{} + for i := range results { + res := &results[i] + if !res.Ok || res.Summary == nil { + continue + } + s := res.Summary + bench := Benchmark{ + CpuTime: &Measure{ + Value: s.CpuS.Median, + LowerValue: s.CpuS.Min, + UpperValue: s.CpuS.Max, + }, + PeakMemory: &Measure{ + Value: s.MaxRSSKb.Median.Mul(pyjson.IntNum(1024)), + LowerValue: s.MaxRSSKb.Min.Mul(pyjson.IntNum(1024)), + UpperValue: s.MaxRSSKb.Max.Mul(pyjson.IntNum(1024)), + }, + } + if s.AllocatedBytes != nil { + bench.AllocatedBytes = &Measure{ + Value: pyjson.IntNum(s.AllocatedBytes.Min), + LowerValue: pyjson.IntNum(s.AllocatedBytes.Min), + UpperValue: pyjson.IntNum(s.AllocatedBytes.Max), + } + } + // Callgrind noise is one-sided like allocation (cold + // paths only add instructions), so the minimum is the + // point estimate here too. + if s.CallgrindInstructions != nil { + bench.CallgrindInstructions = &Measure{ + Value: s.CallgrindInstructions.Min, + LowerValue: s.CallgrindInstructions.Min, + UpperValue: s.CallgrindInstructions.Max, + } + } + if s.CallgrindAllocatedBytes != nil { + bench.CallgrindAllocatedBytes = &Measure{ + Value: pyjson.IntNum(s.CallgrindAllocatedBytes.Min), + LowerValue: pyjson.IntNum(s.CallgrindAllocatedBytes.Min), + UpperValue: pyjson.IntNum(s.CallgrindAllocatedBytes.Max), + } + } + out[res.Label] = bench + } + return out +} + +// SortedNames returns BMF benchmark names in order. +func SortedNames(bmf BMF) []string { + names := make([]string, 0, len(bmf)) + for name := range bmf { + names = append(names, name) + } + sort.Strings(names) + return names +} diff --git a/scripts/ci/go/internal/measure/exec.go b/scripts/ci/go/internal/measure/exec.go new file mode 100644 index 0000000..e059ec0 --- /dev/null +++ b/scripts/ci/go/internal/measure/exec.go @@ -0,0 +1,568 @@ +package measure + +import ( + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "sort" + "strings" + + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/algocore" + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/pyjson" +) + +// ExpandTemplate fills the {cp}/{ns} holes (plus {{/}} escapes). +// Unknown holes and malformed templates are errors. +func ExpandTemplate(template, cp, ns string) (string, error) { + holes := map[string]string{"cp": cp, "ns": ns} + var sb strings.Builder + i := 0 + for i < len(template) { + c := template[i] + switch c { + case '{': + if i+1 < len(template) && template[i+1] == '{' { + sb.WriteByte('{') + i += 2 + continue + } + end := strings.IndexByte(template[i:], '}') + if end < 0 { + return "", errors.New("Single '{' encountered in format string") + } + name := template[i+1 : i+end] + if name == "" || strings.ContainsAny(name, "!:{") { + return "", fmt.Errorf("unsupported hole {%s}", name) + } + value, ok := holes[name] + if !ok { + return "", fmt.Errorf("%s", algocore.PyRepr(name)) + } + sb.WriteString(value) + i += end + 1 + case '}': + if i+1 < len(template) && template[i+1] == '}' { + sb.WriteByte('}') + i += 2 + continue + } + return "", errors.New("Single '}' encountered in format string") + default: + sb.WriteByte(c) + i++ + } + } + return sb.String(), nil +} + +// BuildInnerCommand resolves the driver argv for one namespace. +func BuildInnerCommand(template, cp, ns string) ([]string, error) { + expanded, err := ExpandTemplate(template, cp, ns) + if err != nil { + return nil, fmt.Errorf("bad command template: %v", err) + } + argv, err := SplitCommand(expanded) + if err != nil { + return nil, fmt.Errorf("bad command template: %v", err) + } + if len(argv) == 0 { + return nil, errors.New("bad command template: expands to empty argv") + } + return argv, nil +} + +// AllocSupported reports whether template+ns resolve to a JVM argv +// (EpsilonGC allocation measurement is JVM-only). +func AllocSupported(template, cp, ns string) bool { + inner, err := BuildInnerCommand(template, cp, ns) + if err != nil || len(inner) == 0 { + return false + } + return filepath.Base(inner[0]) == "java" +} + +// ResolveTimeBin finds a GNU time binary. +func ResolveTimeBin(explicit string) (string, error) { + path := explicit + if path == "" { + if _, err := os.Stat("/usr/bin/time"); err == nil { + path = "/usr/bin/time" + } else if found, err := exec.LookPath("time"); err == nil { + path = found + } + } + if path == "" { + return "", errors.New("GNU time not found: install it (Debian/Ubuntu: apt install time) or pass --time-bin") + } + stdout, _, err := algocore.Run([]string{path, "--version"}, nil, 10) + if err != nil { + return "", fmt.Errorf("cannot execute time binary %s: %v", algocore.PyRepr(path), err) + } + if !strings.Contains(string(stdout), "GNU") { + return "", fmt.Errorf("%s is not GNU time (need -f/-o support); install GNU time or pass --time-bin", algocore.PyRepr(path)) + } + return path, nil +} + +// PerfUsable reports whether perf counters run on this host. +func PerfUsable() bool { + if _, err := exec.LookPath("perf"); err != nil { + return false + } + tmp, err := os.CreateTemp("", "perf-probe-") + if err != nil { + return false + } + name := tmp.Name() + tmp.Close() + defer os.Remove(name) + _, _, err = algocore.Run([]string{"perf", "stat", "-e", PerfEvents, "-x,", "-o", name, "--", "true"}, nil, 30) + if err != nil { + return false + } + raw, err := os.ReadFile(name) + if err != nil { + return false + } + counters := ParsePerfText(string(raw)) + return counters.Instructions != nil || counters.Cycles != nil +} + +// CallgrindUsable reports whether valgrind callgrind runs can execute. +func CallgrindUsable() bool { + path, err := exec.LookPath("valgrind") + if err != nil { + return false + } + _, _, err = algocore.Run([]string{path, "--version"}, nil, 30) + return err == nil +} + +// BuildCommand wraps the driver directly (perf outside, time inside). +func BuildCommand(timeBin, timeFile string, usePerf bool, perfFile string, inner []string) []string { + timed := append([]string{timeBin, "-f", TimeFormat, "-o", timeFile}, inner...) + if usePerf { + return append([]string{"perf", "stat", "-e", PerfEvents, "-x,", "-o", perfFile, "--"}, timed...) + } + return timed +} + +// MeasureOnce runs the driver once under GNU time (+perf). The +// driver reads no stdin. +func MeasureOnce(timeBin string, usePerf bool, inner []string, timeout int) ([]byte, *RunMetrics, string) { + timeFile, err := tempFile("algoperf-time-") + if err != nil { + return nil, nil, "time output unreadable: " + err.Error() + } + defer os.Remove(timeFile) + var perfFile string + if usePerf { + perfFile, err = tempFile("algoperf-perf-") + if err != nil { + return nil, nil, "perf output unreadable: " + err.Error() + } + defer os.Remove(perfFile) + } + cmd := BuildCommand(timeBin, timeFile, usePerf, perfFile, inner) + stdout, stderr, err := algocore.Run(cmd, nil, timeout) + var timeoutErr *algocore.TimeoutError + if errors.As(err, &timeoutErr) { + return nil, nil, fmt.Sprintf("TIMEOUT (>%ds)", timeout) + } + if err != nil { + return nil, nil, fmt.Sprintf("exit %d: %s", algocore.ExitCode(err), algocore.LastLine(stderr)) + } + timeRaw, err := os.ReadFile(timeFile) + if err != nil { + return nil, nil, "time output unreadable: " + err.Error() + } + parsed, err := ParseTimeText(string(timeRaw)) + if err != nil { + return nil, nil, "time output unreadable: " + err.Error() + } + metrics := &RunMetrics{ + WallS: pyjson.FloatNum(parsed.WallS), UserS: pyjson.FloatNum(parsed.UserS), + SysS: pyjson.FloatNum(parsed.SysS), CpuS: pyjson.FloatNum(parsed.CpuS), + MaxRSSKb: parsed.MaxRSSKb, CpuPct: pyjson.FloatNum(parsed.CpuPct), + MajorFaults: parsed.MajorFault, MinorFaults: parsed.MinorFault, + } + if usePerf { + perfRaw, err := os.ReadFile(perfFile) + if err != nil { + return nil, nil, "perf output unreadable: " + err.Error() + } + counters := ParsePerfText(string(perfRaw)) + metrics.Instructions = counters.Instructions + metrics.Cycles = counters.Cycles + } + return stdout, metrics, "" +} + +// MeasureAllocOnce runs the driver once under EpsilonGC. +func MeasureAllocOnce(inner []string, timeout int) ([]byte, int64, string) { + gcFile, err := tempFile("algoperf-gc-") + if err != nil { + return nil, 0, "epsilon log unreadable: " + err.Error() + } + defer os.Remove(gcFile) + cmd := append([]string{inner[0]}, EpsilonFlags...) + cmd = append(cmd, "-Xlog:gc:"+gcFile) + cmd = append(cmd, inner[1:]...) + stdout, stderr, err := algocore.Run(cmd, nil, timeout) + var timeoutErr *algocore.TimeoutError + if errors.As(err, &timeoutErr) { + return nil, 0, fmt.Sprintf("TIMEOUT (>%ds)", timeout) + } + if err != nil { + errText := string(stderr) + msg := fmt.Sprintf("exit %d: %s", algocore.ExitCode(err), algocore.LastLine(stderr)) + if strings.Contains(errText, "OutOfMemoryError") { + msg += " (Epsilon heap exhausted; raise -Xmx in EPSILON_FLAGS)" + } + return nil, 0, msg + } + gcRaw, err := os.ReadFile(gcFile) + if err != nil { + return nil, 0, "epsilon log unreadable: " + err.Error() + } + used, err := ParseEpsilonUsed(string(gcRaw)) + if err != nil { + return nil, 0, "epsilon log unreadable: " + err.Error() + } + return stdout, used, "" +} + +// ResolveArgv0 finds the real binary behind PATH/symlinks: under +// valgrind the /bin/java alternatives symlink loses libjli.so +// (exit 127), so callgrind runs need the resolved path. Used for +// JVM and native legs alike. +func ResolveArgv0(argv0 string) string { + path := argv0 + if !strings.Contains(path, string(os.PathSeparator)) { + if found, err := exec.LookPath(path); err == nil { + path = found + } + } + if resolved, err := filepath.EvalSymlinks(path); err == nil { + return resolved + } + return argv0 +} + +// IsCallgrindTimeout reports whether a CallgrindOnce error text is a +// deadline timeout rather than a crash or gate failure: timeouts +// mean the namespace exceeds the instrument's budget, so it +// degrades to a skip instead of failing the run. +func IsCallgrindTimeout(runErr string) bool { + return strings.Contains(runErr, "TIMEOUT") +} + +// IsAllocShimSkip reports whether a CallgrindOnce error text is a +// native-leg counter problem (missing toolchain, timeout, unreadable +// total) rather than a program failure: the Ir leg already gated +// correctness, so counter problems degrade to a skip. +func IsAllocShimSkip(runErr string) bool { + return strings.HasPrefix(runErr, "alloc-shim ") +} + +// CallgrindOnce runs inner once under valgrind callgrind and +// returns stdout plus total Ir and total allocation, dispatching on +// the resolved argv0: the JVM recipe (below) for java, the native +// recipe (no JVM flags, LD_PRELOAD-counted allocation) otherwise. +func CallgrindOnce(inner []string, timeout int) ([]byte, int64, int64, string) { + if !InnerIsJava(inner) { + return nativeCallgrindOnce(inner, timeout) + } + return jvmCallgrindOnce(inner, timeout) +} + +// jvmCallgrindOnce runs inner once under valgrind callgrind with the +// deterministic recipe (-Xint, EpsilonGC, fixed heap, no perfdata) +// and returns stdout plus total Ir and EpsilonGC total allocation. +// The allocation comes from the JVM's own GC log, not from valgrind: +// valgrind-side heap profilers (Massif/DHAT) see only HotSpot's +// bulk-mmaped heap reservation, never Java objects. +func jvmCallgrindOnce(inner []string, timeout int) ([]byte, int64, int64, string) { + outFile, err := tempFile("algoperf-cg-") + if err != nil { + return nil, 0, 0, "callgrind output unreadable: " + err.Error() + } + defer os.Remove(outFile) + gcFile, err := tempFile("algoperf-cg-gc-") + if err != nil { + return nil, 0, 0, "epsilon log unreadable: " + err.Error() + } + defer os.Remove(gcFile) + jvmFlags := append([]string{"-Xint"}, EpsilonFlags...) + jvmFlags = append(jvmFlags, "-XX:-UsePerfData", "-Xlog:gc:"+gcFile) + cmd := []string{"valgrind", "--tool=callgrind", + "--cache-sim=no", "--branch-sim=no", "--collect-jumps=no", + "--callgrind-out-file=" + outFile, ResolveArgv0(inner[0])} + cmd = append(cmd, jvmFlags...) + cmd = append(cmd, inner[1:]...) + stdout, stderr, err := algocore.Run(cmd, nil, timeout) + var timeoutErr *algocore.TimeoutError + if errors.As(err, &timeoutErr) { + return nil, 0, 0, fmt.Sprintf("TIMEOUT (>%ds)", timeout) + } + if err != nil { + return nil, 0, 0, fmt.Sprintf("exit %d: %s", algocore.ExitCode(err), algocore.LastLine(stderr)) + } + raw, err := os.ReadFile(outFile) + if err != nil { + return nil, 0, 0, "callgrind output unreadable: " + err.Error() + } + ir, err := ParseCallgrindSummary(string(raw)) + if err != nil { + return nil, 0, 0, err.Error() + } + gcRaw, err := os.ReadFile(gcFile) + if err != nil { + return nil, 0, 0, "epsilon log unreadable: " + err.Error() + } + used, err := ParseEpsilonUsed(string(gcRaw)) + if err != nil { + return nil, 0, 0, "epsilon log unreadable: " + err.Error() + } + return stdout, ir, used, "" +} + +// nativeCallgrindOnce runs a native binary once under valgrind +// callgrind (same no-sim flags, no JVM flags) for Ir, plus one +// near-native-speed LD_PRELOAD counting run for allocation. The Ir +// leg is authoritative — its crashes fail like the JVM path — while +// every counter-leg problem degrades to an "alloc-shim ..." skip +// reason (a broken counter must never fail correct runs). +func nativeCallgrindOnce(inner []string, timeout int) ([]byte, int64, int64, string) { + outFile, err := tempFile("algoperf-cg-") + if err != nil { + return nil, 0, 0, "callgrind output unreadable: " + err.Error() + } + defer os.Remove(outFile) + cmd := []string{"valgrind", "--tool=callgrind", + "--cache-sim=no", "--branch-sim=no", "--collect-jumps=no", + "--callgrind-out-file=" + outFile, ResolveArgv0(inner[0])} + cmd = append(cmd, inner[1:]...) + stdout, stderr, err := algocore.Run(cmd, nil, timeout) + var timeoutErr *algocore.TimeoutError + if errors.As(err, &timeoutErr) { + return nil, 0, 0, fmt.Sprintf("TIMEOUT (>%ds)", timeout) + } + if err != nil { + return nil, 0, 0, fmt.Sprintf("exit %d: %s", algocore.ExitCode(err), algocore.LastLine(stderr)) + } + raw, err := os.ReadFile(outFile) + if err != nil { + return nil, 0, 0, "callgrind output unreadable: " + err.Error() + } + ir, err := ParseCallgrindSummary(string(raw)) + if err != nil { + return nil, 0, 0, err.Error() + } + used, skip := MeasureShimAllocOnce(inner, stdout, timeout) + if skip != "" { + return nil, 0, 0, skip + } + return stdout, ir, used, "" +} + +// MeasureShimAllocOnce runs inner once under the LD_PRELOAD +// counting shim and returns total allocated bytes. wantStdout is +// the Ir leg's stdout: the two runs must agree on the SUMMARY +// line (full stdout is nondeterministic), else the counter run +// diverged, which degrades like any other counter problem. Every +// failure returns a skip reason (never a hard error); "" means +// success. +func MeasureShimAllocOnce(inner []string, wantStdout []byte, timeout int) (int64, string) { + so, err := EnsureAllocShim() + if err != nil { + return 0, "alloc-shim unavailable: " + err.Error() + } + stdout, stderr, err := algocore.RunEnv(inner, nil, timeout, []string{"LD_PRELOAD=" + so}) + var timeoutErr *algocore.TimeoutError + if errors.As(err, &timeoutErr) { + return 0, fmt.Sprintf("alloc-shim TIMEOUT (>%ds)", timeout) + } + if err != nil { + return 0, fmt.Sprintf("alloc-shim exit %d: %s", algocore.ExitCode(err), algocore.LastLine(stderr)) + } + wantSum, wantErr := ParseSummary(string(wantStdout)) + gotSum, gotErr := ParseSummary(string(stdout)) + if wantErr != nil || gotErr != nil || wantSum != gotSum { + return 0, "alloc-shim output mismatch: counter run diverged from the callgrind run" + } + used, err := ParseAllocShimTotal(string(stderr)) + if err != nil { + return 0, err.Error() + } + return used, "" +} + +func tempFile(prefix string) (string, error) { + fd, err := os.CreateTemp("", prefix) + if err != nil { + return "", err + } + name := fd.Name() + fd.Close() + return name, nil +} + +// CaseOpts bundles MeasureCase inputs. Timeout budgets the +// native-speed runs (timed + alloc); CallgrindTimeout budgets the +// instrumented callgrind phase, which degrades to a skip on +// expiry instead of failing. +type CaseOpts struct { + TimeBin string + UsePerf bool + Template string + Classpath string + Label string + Stem string + Script string + Case string + Ns string + Repeat int + AllocRepeat int + CallgrindRepeat int + Timeout int + CallgrindTimeout int + Root string +} + +// MeasureCase measures one namespace: timed runs, EpsilonGC +// allocation runs, then the callgrind phase. Every phase's first +// run passes the SUMMARY gate (missing SUMMARY or :fail/:error +// fails the namespace loudly). +func MeasureCase(o CaseOpts) CaseResult { + res := CaseResult{ + Label: o.Label, Stem: o.Stem, Case: o.Case, + Runs: []RunMetrics{}, AllocRuns: []int64{}, + } + rel, err := filepath.Rel(o.Root, o.Script) + if err != nil { + rel = o.Script + } + res.Script = rel + inner, err := BuildInnerCommand(o.Template, o.Classpath, o.Ns) + if err != nil { + msg := fmt.Sprintf("%s/%s: %v", o.Label, o.Case, err) + res.Error = &msg + return res + } + gate := func(phase string, stdout []byte) string { + prefix := o.Label + "/" + o.Case + if phase != "" { + prefix = phase + ": " + prefix + } + sum, err := ParseSummary(string(stdout)) + if err != nil { + return fmt.Sprintf("%s: %v", prefix, err) + } + if reason := CheckSummary(sum); reason != "" { + return fmt.Sprintf("%s: %s", prefix, reason) + } + if phase == "" && res.TestCounts == nil { + res.TestCounts = &TestCounts{Test: sum.Test, Pass: sum.Pass, Fail: sum.Fail, Error: sum.Error} + } + return "" + } + for i := 0; i < o.Repeat; i++ { + stdout, metrics, runErr := MeasureOnce(o.TimeBin, o.UsePerf, inner, o.Timeout) + if runErr != "" { + msg := fmt.Sprintf("%s/%s: %s", o.Label, o.Case, runErr) + res.Error = &msg + return res + } + if i == 0 { + if msg := gate("", stdout); msg != "" { + res.Error = &msg + return res + } + } + res.Runs = append(res.Runs, *metrics) + } + res.Summary = SummarizeCase(res.Runs) + if o.AllocRepeat > 0 { + for i := 0; i < o.AllocRepeat; i++ { + stdout, used, runErr := MeasureAllocOnce(inner, o.Timeout) + if runErr != "" { + msg := fmt.Sprintf("%s/%s: alloc: %s", o.Label, o.Case, runErr) + res.Error = &msg + return res + } + if i == 0 { + if msg := gate("alloc", stdout); msg != "" { + res.Error = &msg + return res + } + } + res.AllocRuns = append(res.AllocRuns, used) + } + res.Summary.AllocatedBytes = SummarizeAlloc(res.AllocRuns) + } + if o.CallgrindRepeat > 0 { + var irs []int64 + var cgAllocs []int64 + for i := 0; i < o.CallgrindRepeat; i++ { + stdout, ir, used, runErr := CallgrindOnce(inner, o.CallgrindTimeout) + if runErr != "" { + if IsCallgrindTimeout(runErr) || IsAllocShimSkip(runErr) { + skip := runErr + res.CallgrindSkipped = &skip + fmt.Fprintf(os.Stderr, "warning: %s/%s/%s: callgrind skipped (%s)\n", + o.Label, o.Stem, o.Case, runErr) + break + } + msg := fmt.Sprintf("%s/%s: callgrind: %s", o.Label, o.Case, runErr) + res.Error = &msg + return res + } + if i == 0 { + if msg := gate("callgrind", stdout); msg != "" { + res.Error = &msg + return res + } + } + irs = append(irs, ir) + cgAllocs = append(cgAllocs, used) + } + if res.CallgrindSkipped == nil { + s := SummarizeInts(irs) + res.Summary.CallgrindInstructions = &s + res.Summary.CallgrindAllocatedBytes = SummarizeAlloc(cgAllocs) + } + } + res.Ok = true + return res +} + +// SelectShard filters tasks deterministically by group: groups are +// sorted, and group i lands in shard i%total. +func SelectShard[T any](tasks []T, key func(T) string, index, total int) []T { + groups := map[string][]T{} + var order []string + for _, t := range tasks { + k := key(t) + if _, seen := groups[k]; !seen { + order = append(order, k) + } + groups[k] = append(groups[k], t) + } + sort.Strings(order) + keep := map[string]bool{} + for i, g := range order { + if i%total == index { + keep[g] = true + } + } + var out []T + for _, t := range tasks { + if keep[key(t)] { + out = append(out, t) + } + } + return out +} diff --git a/scripts/ci/go/internal/measure/measure.go b/scripts/ci/go/internal/measure/measure.go new file mode 100644 index 0000000..cad816a --- /dev/null +++ b/scripts/ci/go/internal/measure/measure.go @@ -0,0 +1,33 @@ +// Package measure runs the algorithms perf sweep: one driver run +// per test namespace under GNU time (+perf), plus the gungraun-style +// callgrind instruction backend. Ported from the snippet repo's +// measure package (recipe-identical valgrind/JVM/shim/BMF); the +// snippet case discovery and full-stdout gating are replaced by +// namespace discovery and the SUMMARY gate (see summary.go). +package measure + +// TimeFormat is the GNU time -f layout (TIME_FORMAT). +const TimeFormat = "wall=%e user=%U sys=%S maxrss_kb=%M cpu_pct=%P major_faults=%F minor_faults=%R" + +// PerfEvents are the user-space counters (PERF_EVENTS). +const PerfEvents = "instructions:u,cycles:u" + +// ToolVersion tags the report. +const ToolVersion = 1 + +// DefaultCommandTemplate runs one namespace on the JVM. +const DefaultCommandTemplate = "java -cp {cp} clojure.main perf/driver.clj {ns}" + +// EpsilonFlags make heap-used-at-exit equal total allocation. +var EpsilonFlags = []string{ + "-XX:+UnlockExperimentalVMOptions", + "-XX:+UseEpsilonGC", + "-Xms2g", + "-Xmx2g", +} + +// AllocRepeatDefault mirrors the snippet default. +const AllocRepeatDefault = 3 + +// CallgrindRepeatDefault mirrors the one-shot gungraun default. +const CallgrindRepeatDefault = 1 diff --git a/scripts/ci/go/internal/measure/parsers.go b/scripts/ci/go/internal/measure/parsers.go new file mode 100644 index 0000000..d07e9ba --- /dev/null +++ b/scripts/ci/go/internal/measure/parsers.go @@ -0,0 +1,138 @@ +package measure + +import ( + "fmt" + "regexp" + "strconv" + "strings" + + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/algocore" +) + +// TimeMetrics is one parsed GNU time line. +type TimeMetrics struct { + WallS float64 + UserS float64 + SysS float64 + CpuS float64 + MaxRSSKb int64 + CpuPct float64 + MajorFault int64 + MinorFault int64 +} + +// ParseTimeText parses one GNU time -f line. +func ParseTimeText(text string) (TimeMetrics, error) { + var m TimeMetrics + fields := map[string]string{} + for _, token := range strings.Fields(text) { + key, value, found := strings.Cut(token, "=") + if !found { + continue + } + fields[key] = value + } + wall, err1 := strconv.ParseFloat(fields["wall"], 64) + user, err2 := strconv.ParseFloat(fields["user"], 64) + sysS, err3 := strconv.ParseFloat(fields["sys"], 64) + maxrss, err4 := strconv.ParseInt(fields["maxrss_kb"], 10, 64) + cpuPct, err5 := strconv.ParseFloat(strings.TrimSuffix(fields["cpu_pct"], "%"), 64) + major, err6 := strconv.ParseInt(fields["major_faults"], 10, 64) + minor, err7 := strconv.ParseInt(fields["minor_faults"], 10, 64) + if err1 != nil || err2 != nil || err3 != nil || err4 != nil || err5 != nil || err6 != nil || err7 != nil { + return m, fmt.Errorf("unparseable GNU time output: %s", algocore.PyRepr(strings.TrimSpace(text))) + } + m.WallS = wall + m.UserS = user + m.SysS = sysS + m.CpuS = user + sysS + m.MaxRSSKb = maxrss + m.CpuPct = cpuPct + m.MajorFault = major + m.MinorFault = minor + return m, nil +} + +// PerfCounters holds user-space perf counts (nil when unavailable). +type PerfCounters struct { + Instructions *int64 + Cycles *int64 +} + +// ParsePerfText parses `perf stat -x,` output. +func ParsePerfText(text string) PerfCounters { + var out PerfCounters + for _, line := range strings.Split(text, "\n") { + line = strings.TrimSpace(line) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + parts := strings.Split(line, ",") + if len(parts) < 3 { + continue + } + raw := strings.TrimSpace(parts[0]) + base, _, _ := strings.Cut(strings.TrimSpace(parts[2]), ":") + if base != "instructions" && base != "cycles" { + continue + } + var value *int64 + if raw != "" && raw != "not counted" && raw != "" && raw != "" { + if n, err := strconv.ParseInt(raw, 10, 64); err == nil { + value = &n + } + } + if base == "instructions" { + out.Instructions = value + } else { + out.Cycles = value + } + } + return out +} + +var unitMultipliers = map[string]int64{"B": 1, "K": 1024, "M": 1024 * 1024, "G": 1024 * 1024 * 1024, "T": 1024 * 1024 * 1024 * 1024} + +var epsilonUsedRe = regexp.MustCompile(`(\d+(?:\.\d+)?)\s*([BKMGT])(?:\s*\([^)]*\))?\s+used\b`) + +// ParseEpsilonUsed parses heap-used-at-exit in bytes from the LAST +// Heap: line of an EpsilonGC log. +func ParseEpsilonUsed(text string) (int64, error) { + var found []string + for _, line := range strings.Split(text, "\n") { + if !strings.Contains(line, "Heap:") { + continue + } + for _, match := range epsilonUsedRe.FindAllStringSubmatch(line, -1) { + found = match + } + } + if found == nil { + tail := strings.TrimSpace(text) + if runes := []rune(tail); len(runes) > 200 { + tail = string(runes[len(runes)-200:]) + } + return 0, fmt.Errorf("no Epsilon Heap used value in: %s", algocore.PyRepr(tail)) + } + amount, err := strconv.ParseFloat(found[1], 64) + if err != nil { + return 0, fmt.Errorf("no Epsilon Heap used value in: %s", algocore.PyRepr(found[0])) + } + return int64(amount * float64(unitMultipliers[found[2]])), nil +} + +// ParseCallgrindSummary extracts the total Ir from a callgrind.out +// file (the `summary: N` line). +func ParseCallgrindSummary(text string) (int64, error) { + for _, line := range strings.Split(text, "\n") { + if !strings.HasPrefix(line, "summary:") { + continue + } + n, err := strconv.ParseInt(strings.TrimSpace(strings.TrimPrefix(line, "summary:")), 10, 64) + if err != nil { + return 0, fmt.Errorf("callgrind summary unreadable: %s", algocore.PyRepr(strings.TrimSpace(line))) + } + return n, nil + } + return 0, fmt.Errorf("callgrind summary unreadable: no summary line") +} diff --git a/scripts/ci/go/internal/measure/report.go b/scripts/ci/go/internal/measure/report.go new file mode 100644 index 0000000..4498fb2 --- /dev/null +++ b/scripts/ci/go/internal/measure/report.go @@ -0,0 +1,206 @@ +package measure + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "os/exec" + "runtime" + "sort" + "strconv" + "strings" + + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/algocore" +) + +// Host describes the measuring machine. +type Host struct { + Hostname string `json:"hostname"` + Os string `json:"os"` + Nproc int `json:"nproc"` + CpuModel *string `json:"cpu_model"` + MemTotalKb *int64 `json:"mem_total_kb"` +} + +// Runtime describes the measured runtime. +type Runtime struct { + Name string `json:"name"` + CommandTemplate string `json:"command_template"` + JavaVersion *string `json:"java_version"` + RuntimeVersion *string `json:"runtime_version"` + Classpath string `json:"classpath"` + TimeBin string `json:"time_bin"` + PerfUsed bool `json:"perf_used"` + CallgrindUsed bool `json:"callgrind_used"` +} + +// Config records the run configuration. +type Config struct { + Repeat int `json:"repeat"` + Jobs int `json:"jobs"` + Timeout int `json:"timeout"` + Only string `json:"only"` + Shard *string `json:"shard"` + AllocRepeat *int `json:"alloc_repeat"` + CallgrindRepeat int `json:"callgrind_repeat"` + CallgrindTimeout int `json:"callgrind_timeout"` +} + +// Report is the full perf-report JSON document. +type Report struct { + Tool string `json:"tool"` + Version int `json:"version"` + TimestampUTC string `json:"timestamp_utc"` + Host Host `json:"host"` + Runtime Runtime `json:"runtime"` + Config Config `json:"config"` + Results []CaseResult `json:"results"` + Failures []string `json:"failures"` +} + +// HostInfo describes the measuring machine. +func HostInfo() Host { + out := Host{Nproc: runtime.NumCPU()} + if name, err := os.Hostname(); err == nil { + out.Hostname = name + } + out.Os = runtime.GOOS + "/" + runtime.GOARCH + if raw, err := os.ReadFile("/proc/cpuinfo"); err == nil { + for _, line := range strings.Split(string(raw), "\n") { + for _, key := range []string{"model name", "Model", "Hardware"} { + if strings.HasPrefix(line, key) { + _, value, _ := strings.Cut(line, ":") + model := strings.TrimSpace(value) + out.CpuModel = &model + break + } + } + if out.CpuModel != nil { + break + } + } + } + if raw, err := os.ReadFile("/proc/meminfo"); err == nil { + for _, line := range strings.Split(string(raw), "\n") { + if strings.HasPrefix(line, "MemTotal:") { + fields := strings.Fields(line) + if len(fields) >= 2 { + if kb, err := strconv.ParseInt(fields[1], 10, 64); err == nil { + out.MemTotalKb = &kb + } + } + break + } + } + } + return out +} + +// JavaVersion probes `java -version` (first line or nil). Only a +// spawn failure or timeout yields nil; the exit code is ignored +// and java's stderr is parsed (that is where -version writes). +func JavaVersion() *string { + _, stderr, err := algocore.Run([]string{"java", "-version"}, nil, 30) + if err != nil { + var timeout *algocore.TimeoutError + var exitErr *exec.ExitError + if errors.As(err, &timeout) || !errors.As(err, &exitErr) { + return nil + } + } + for _, line := range strings.Split(string(stderr), "\n") { + if trimmed := strings.TrimSpace(line); trimmed != "" { + return &trimmed + } + } + return nil +} + +// RuntimeVersion probes a native runtime binary's ` +// --version`, returning the first non-empty output line (stdout, +// then stderr) or nil when the probe fails. Best-effort +// provenance for non-JVM legs, mirroring JavaVersion's tolerance. +func RuntimeVersion(argv0 string) *string { + stdout, stderr, err := algocore.Run([]string{argv0, "--version"}, nil, 30) + if err != nil { + var timeout *algocore.TimeoutError + var exitErr *exec.ExitError + if errors.As(err, &timeout) || !errors.As(err, &exitErr) { + return nil + } + } + for _, raw := range [][]byte{stdout, stderr} { + for _, line := range strings.Split(string(raw), "\n") { + if trimmed := strings.TrimSpace(line); trimmed != "" { + return &trimmed + } + } + } + return nil +} + +// MergeReports concatenates full-report JSONs into one (report, +// bmf) pair with dedupe and sorted failures. +func MergeReports(paths []string) (Report, BMF, error) { + merged := []CaseResult{} + seen := map[resultKey]bool{} + var failures []string + var base Report + haveBase := false + for _, path := range paths { + raw, err := os.ReadFile(path) + if err != nil { + return Report{}, nil, fmt.Errorf("cannot read report %s: %v", path, err) + } + var rep Report + if err := json.Unmarshal(raw, &rep); err != nil { + return Report{}, nil, fmt.Errorf("cannot read report %s: %v", path, err) + } + if !haveBase { + base = rep + haveBase = true + } + for _, res := range rep.Results { + key := resultKey{res.Label, res.Stem, res.Case} + if seen[key] { + continue + } + seen[key] = true + merged = append(merged, res) + } + failures = append(failures, rep.Failures...) + } + sort.Slice(merged, func(i, j int) bool { + if merged[i].Label != merged[j].Label { + return merged[i].Label < merged[j].Label + } + if merged[i].Stem != merged[j].Stem { + return merged[i].Stem < merged[j].Stem + } + return merged[i].Case < merged[j].Case + }) + report := base + report.Results = merged + report.Failures = sortedUnique(failures) + shard := "merged" + report.Config.Shard = &shard + return report, BuildBMF(merged), nil +} + +type resultKey struct { + label, stem, cas string +} + +func sortedUnique(in []string) []string { + set := map[string]bool{} + for _, v := range in { + set[v] = true + } + out := make([]string, 0, len(set)) + for v := range set { + out = append(out, v) + } + sort.Strings(out) + return out +} diff --git a/scripts/ci/go/internal/measure/selftest.go b/scripts/ci/go/internal/measure/selftest.go new file mode 100644 index 0000000..7d25458 --- /dev/null +++ b/scripts/ci/go/internal/measure/selftest.go @@ -0,0 +1,111 @@ +package measure + +import ( + "fmt" + "io" + "strings" + + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/algocore" +) + +// SelfTest runs the subprocess-free check suite (no JVM, time, +// perf, valgrind, or C compiler): parser checks for every text +// format the sweep reads, template checks, and gate checks. +// Returns the process exit code (0 when all checks pass). +func SelfTest(out io.Writer) int { + var checks []algocore.Check + add := func(name string, ok bool, detail string) { + checks = append(checks, algocore.Check{Name: name, Ok: ok, Detail: detail}) + } + + // SUMMARY gate. + sum, err := ParseSummary("Testing x\nSUMMARY {:test 2, :pass 10, :fail 0, :error 0}\n") + add("summary-valid", err == nil && sum == (Summary{Test: 2, Pass: 10}), fmt.Sprint(sum, err)) + _, err = ParseSummary("no marker here\n") + add("summary-missing", err != nil, fmt.Sprint(err)) + _, err = ParseSummary("SUMMARY {:test 2, :pass x, :fail 0, :error 0}\n") + add("summary-malformed", err != nil, fmt.Sprint(err)) + add("summary-gate-clean", CheckSummary(Summary{Test: 1, Pass: 1}) == "", "") + add("summary-gate-fail", CheckSummary(Summary{Test: 1, Fail: 1}) != "", "") + add("summary-gate-error", CheckSummary(Summary{Test: 1, Error: 1}) != "", "") + + // Callgrind summary. + ir, err := ParseCallgrindSummary("events: Ir\nsummary: 1234567\n") + add("callgrind-valid", err == nil && ir == 1234567, fmt.Sprint(ir, err)) + _, err = ParseCallgrindSummary("events: Ir\n") + add("callgrind-missing", err != nil, fmt.Sprint(err)) + _, err = ParseCallgrindSummary("summary: nope\n") + add("callgrind-malformed", err != nil, fmt.Sprint(err)) + + // Alloc-shim total. + used, err := ParseAllocShimTotal("noise\nsnip_alloc_total_bytes=987654\n") + add("allocshim-valid", err == nil && used == 987654, fmt.Sprint(used, err)) + _, err = ParseAllocShimTotal("nothing here\n") + add("allocshim-missing", err != nil, fmt.Sprint(err)) + _, err = ParseAllocShimTotal("snip_alloc_total_bytes=zzz\n") + add("allocshim-malformed", err != nil, fmt.Sprint(err)) + + // EpsilonGC log. + used, err = ParseEpsilonUsed("[0.1s] Heap: 12M reserved, 3M (25%) committed, 1536K (4%) used\n") + add("epsilon-valid", err == nil && used == 1536*1024, fmt.Sprint(used, err)) + _, err = ParseEpsilonUsed("no heap line\n") + add("epsilon-missing", err != nil, fmt.Sprint(err)) + + // GNU time. + tm, err := ParseTimeText("wall=1.5 user=1.2 sys=0.3 maxrss_kb=1024 cpu_pct=99% major_faults=1 minor_faults=2") + add("time-valid", err == nil && tm.WallS == 1.5 && tm.CpuS == 1.5 && tm.MaxRSSKb == 1024, fmt.Sprint(tm, err)) + _, err = ParseTimeText("wall=1.5") + add("time-truncated", err != nil, fmt.Sprint(err)) + + // Templates. + argv, err := BuildInnerCommand(DefaultCommandTemplate, "a.jar:b", "foo.bar-test") + want := []string{"java", "-cp", "a.jar:b", "clojure.main", "perf/driver.clj", "foo.bar-test"} + add("template-jvm", err == nil && equalStrs(argv, want), fmt.Sprint(argv, err)) + argv, err = BuildInnerCommand("clojure-rs run perf/driver.clj {ns} src test", "", "foo.bar-test") + want = []string{"clojure-rs", "run", "perf/driver.clj", "foo.bar-test", "src", "test"} + add("template-native", err == nil && equalStrs(argv, want), fmt.Sprint(argv, err)) + _, err = BuildInnerCommand("java -cp {bogus} x", "", "n") + add("template-bad-hole", err != nil, fmt.Sprint(err)) + + // Leg detection. + add("leg-java", InnerIsJava([]string{"java", "-cp", "x"}) && InnerIsJava([]string{"/usr/bin/java"}), "") + add("leg-native", !InnerIsJava([]string{"clojure-rs", "run"}) && !InnerIsJava(nil), "") + add("leg-template", TemplateHeadIsJava(DefaultCommandTemplate) && !TemplateHeadIsJava("clojure-rs run {ns}"), "") + + // Skip classifiers. + add("skip-timeout", IsCallgrindTimeout("TIMEOUT (>600s)"), "") + add("skip-shim-timeout", IsCallgrindTimeout("alloc-shim TIMEOUT (>600s)"), "") + add("skip-shim", IsAllocShimSkip("alloc-shim unavailable: x"), "") + add("skip-no-false-positive", !IsCallgrindTimeout("exit 1: boom") && !IsAllocShimSkip("exit 1: boom"), "") + + failed := 0 + for _, c := range checks { + status := "ok" + if !c.Ok { + status = "FAIL" + failed++ + } + if c.Detail != "" { + fmt.Fprintf(out, "self-test [%s] %s: %s\n", status, c.Name, strings.TrimSpace(c.Detail)) + } else { + fmt.Fprintf(out, "self-test [%s] %s\n", status, c.Name) + } + } + fmt.Fprintf(out, "self-test: %d failed, %d passed\n", failed, len(checks)-failed) + if failed > 0 { + return 1 + } + return 0 +} + +func equalStrs(a, b []string) bool { + if len(a) != len(b) { + return false + } + for i := range a { + if a[i] != b[i] { + return false + } + } + return true +} diff --git a/scripts/ci/go/internal/measure/selftest_test.go b/scripts/ci/go/internal/measure/selftest_test.go new file mode 100644 index 0000000..9c0ac22 --- /dev/null +++ b/scripts/ci/go/internal/measure/selftest_test.go @@ -0,0 +1,55 @@ +package measure + +import ( + "bytes" + "os/exec" + "runtime" + "strings" + "testing" + + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/algocore" +) + +// TestSelfTestGreen runs the full CLI self-test suite in-process: +// every check must pass with no JVM, time, perf, or valgrind. +func TestSelfTestGreen(t *testing.T) { + var buf bytes.Buffer + if code := SelfTest(&buf); code != 0 { + t.Fatalf("SelfTest exit %d:\n%s", code, buf.String()) + } + if !strings.Contains(buf.String(), "0 failed") { + t.Fatalf("missing summary:\n%s", buf.String()) + } +} + +// TestAllocShimLive builds the counting shim and runs a shell under +// it: stdout must pass through intact and the total must parse. +// Linux-only (LD_PRELOAD/glibc). +func TestAllocShimLive(t *testing.T) { + if runtime.GOOS != "linux" { + t.Skip("alloc shim needs Linux (LD_PRELOAD)") + } + if _, err := exec.LookPath("cc"); err != nil { + t.Skip("needs cc to build the shim") + } + so, err := EnsureAllocShim() + if err != nil { + t.Fatalf("build shim: %v", err) + } + stdout, stderr, err := algocore.RunEnv( + []string{"sh", "-c", "echo hi"}, nil, 60, + []string{"LD_PRELOAD=" + so}) + if err != nil { + t.Fatalf("run under shim: %v\n%s", err, stderr) + } + if string(stdout) != "hi\n" { + t.Fatalf("stdout mangled: %q", stdout) + } + total, err := ParseAllocShimTotal(string(stderr)) + if err != nil { + t.Fatalf("parse total: %v\n%s", err, stderr) + } + if total <= 0 { + t.Fatalf("non-positive total: %d", total) + } +} diff --git a/scripts/ci/go/internal/measure/shlex.go b/scripts/ci/go/internal/measure/shlex.go new file mode 100644 index 0000000..7fa2a8b --- /dev/null +++ b/scripts/ci/go/internal/measure/shlex.go @@ -0,0 +1,70 @@ +package measure + +import ( + "errors" + "strings" +) + +// SplitCommand splits POSIX shell words with single/double quotes +// and backslash escapes, no comment or expansion handling. +func SplitCommand(text string) ([]string, error) { + var tokens []string + var current strings.Builder + inToken := false + i := 0 + flush := func() { + if inToken { + tokens = append(tokens, current.String()) + current.Reset() + inToken = false + } + } + for i < len(text) { + c := text[i] + switch { + case c == ' ' || c == '\t' || c == '\n' || c == '\r': + flush() + i++ + case c == '\'' || c == '"': + quote := c + i++ + closed := false + for i < len(text) { + d := text[i] + if d == quote { + closed = true + i++ + break + } + if quote == '"' && d == '\\' && i+1 < len(text) && + (text[i+1] == '"' || text[i+1] == '\\') { + current.WriteByte(text[i+1]) + inToken = true + i += 2 + continue + } + current.WriteByte(d) + inToken = true + i++ + } + if !closed { + return nil, errors.New("No closing quotation") + } + // A quote pair starts a token even when empty. + inToken = true + case c == '\\': + if i+1 >= len(text) { + return nil, errors.New("No escaped character") + } + current.WriteByte(text[i+1]) + inToken = true + i += 2 + default: + current.WriteByte(c) + inToken = true + i++ + } + } + flush() + return tokens, nil +} diff --git a/scripts/ci/go/internal/measure/shlex_test.go b/scripts/ci/go/internal/measure/shlex_test.go new file mode 100644 index 0000000..f463ba1 --- /dev/null +++ b/scripts/ci/go/internal/measure/shlex_test.go @@ -0,0 +1,43 @@ +package measure + +import ( + "encoding/json" + "os" + "path/filepath" + "testing" +) + +func TestSplitCommandOracle(t *testing.T) { + raw, err := os.ReadFile(filepath.Join("testdata", "shlex.json")) + if err != nil { + t.Fatal(err) + } + var cases []struct { + In string `json:"in"` + Tokens []string `json:"tokens"` + Error string `json:"error"` + } + if err := json.Unmarshal(raw, &cases); err != nil { + t.Fatal(err) + } + for _, c := range cases { + got, err := SplitCommand(c.In) + if c.Error != "" { + if err == nil || err.Error() != c.Error { + t.Fatalf("SplitCommand(%q) err = %v, want %q", c.In, err, c.Error) + } + continue + } + if err != nil { + t.Fatalf("SplitCommand(%q) err = %v", c.In, err) + } + if len(got) != len(c.Tokens) { + t.Fatalf("SplitCommand(%q) = %q, want %q", c.In, got, c.Tokens) + } + for i := range got { + if got[i] != c.Tokens[i] { + t.Fatalf("SplitCommand(%q) = %q, want %q", c.In, got, c.Tokens) + } + } + } +} diff --git a/scripts/ci/go/internal/measure/summarize.go b/scripts/ci/go/internal/measure/summarize.go new file mode 100644 index 0000000..f751699 --- /dev/null +++ b/scripts/ci/go/internal/measure/summarize.go @@ -0,0 +1,157 @@ +package measure + +import ( + "sort" + + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/pyjson" +) + +// RunMetrics is one measured execution. +type RunMetrics struct { + WallS pyjson.Num `json:"wall_s"` + UserS pyjson.Num `json:"user_s"` + SysS pyjson.Num `json:"sys_s"` + CpuS pyjson.Num `json:"cpu_s"` + MaxRSSKb int64 `json:"maxrss_kb"` + CpuPct pyjson.Num `json:"cpu_pct"` + MajorFaults int64 `json:"major_faults"` + MinorFaults int64 `json:"minor_faults"` + Instructions *int64 `json:"instructions"` + Cycles *int64 `json:"cycles"` +} + +// MinMedMax aggregates one metric across repeats. +type MinMedMax struct { + Min pyjson.Num `json:"min"` + Median pyjson.Num `json:"median"` + Max pyjson.Num `json:"max"` +} + +// AllocSummary aggregates allocated-byte counts. +type AllocSummary struct { + Min int64 `json:"min"` + Median pyjson.Num `json:"median"` + Max int64 `json:"max"` + Deterministic bool `json:"deterministic"` +} + +// CaseSummary aggregates one namespace's runs. +type CaseSummary struct { + CpuS MinMedMax `json:"cpu_s"` + MaxRSSKb MinMedMax `json:"maxrss_kb"` + WallS MinMedMax `json:"wall_s"` + Runs int `json:"runs"` + Instructions *MinMedMax `json:"instructions"` + Cycles *MinMedMax `json:"cycles"` + AllocatedBytes *AllocSummary `json:"allocated_bytes"` + CallgrindInstructions *MinMedMax `json:"callgrind_instructions"` + CallgrindAllocatedBytes *AllocSummary `json:"callgrind_allocated_bytes"` +} + +// CaseResult is one measured namespace. Error is nil (null) on +// success. CallgrindSkipped records a degraded callgrind phase: +// the timed data is valid (Ok stays true) but the namespace +// exceeded the callgrind budget, so no callgrind summaries were +// recorded. TestCounts records the driver's SUMMARY counts so the +// report can diff JVM vs native behavior. +type CaseResult struct { + Label string `json:"label"` + Stem string `json:"stem"` + Script string `json:"script"` + Case string `json:"case"` + Ok bool `json:"ok"` + Error *string `json:"error"` + Runs []RunMetrics `json:"runs"` + AllocRuns []int64 `json:"alloc_runs"` + Summary *CaseSummary `json:"summary"` + TestCounts *TestCounts `json:"test_counts"` + CallgrindSkipped *string `json:"callgrind_skipped,omitempty"` +} + +// TestCounts is the driver's SUMMARY counts for one namespace. +type TestCounts struct { + Test int64 `json:"test"` + Pass int64 `json:"pass"` + Fail int64 `json:"fail"` + Error int64 `json:"error"` +} + +// SummarizeFloats aggregates floats (odd counts keep the middle +// value; even counts average, like statistics.median). +func SummarizeFloats(values []float64) MinMedMax { + sorted := append([]float64(nil), values...) + sort.Float64s(sorted) + return MinMedMax{ + Min: pyjson.FloatNum(sorted[0]), + Median: pyjson.MedianFloat64(sorted), + Max: pyjson.FloatNum(sorted[len(sorted)-1]), + } +} + +// SummarizeInts aggregates ints (odd-count medians stay integral). +func SummarizeInts(values []int64) MinMedMax { + sorted := append([]int64(nil), values...) + sort.Slice(sorted, func(i, j int) bool { return sorted[i] < sorted[j] }) + return MinMedMax{ + Min: pyjson.IntNum(sorted[0]), + Median: pyjson.MedianInt64(sorted), + Max: pyjson.IntNum(sorted[len(sorted)-1]), + } +} + +// SummarizeCase aggregates runs (nil for no runs). +func SummarizeCase(runs []RunMetrics) *CaseSummary { + if len(runs) == 0 { + return nil + } + cpu := make([]float64, 0, len(runs)) + rss := make([]int64, 0, len(runs)) + wall := make([]float64, 0, len(runs)) + var insn, cyc []int64 + for _, r := range runs { + cpu = append(cpu, r.CpuS.Float64()) + rss = append(rss, r.MaxRSSKb) + wall = append(wall, r.WallS.Float64()) + if r.Instructions != nil { + insn = append(insn, *r.Instructions) + } + if r.Cycles != nil { + cyc = append(cyc, *r.Cycles) + } + } + out := &CaseSummary{ + CpuS: SummarizeFloats(cpu), + MaxRSSKb: SummarizeInts(rss), + WallS: SummarizeFloats(wall), + Runs: len(runs), + } + if len(insn) > 0 { + s := SummarizeInts(insn) + out.Instructions = &s + } + if len(cyc) > 0 { + s := SummarizeInts(cyc) + out.Cycles = &s + } + return out +} + +// SummarizeAlloc aggregates allocated-byte counts (one-sided +// noise: the BMF uses the minimum as the point estimate). +func SummarizeAlloc(values []int64) *AllocSummary { + sorted := append([]int64(nil), values...) + sort.Slice(sorted, func(i, j int) bool { return sorted[i] < sorted[j] }) + det := true + for _, v := range sorted[1:] { + if v != sorted[0] { + det = false + break + } + } + return &AllocSummary{ + Min: sorted[0], + Median: pyjson.MedianInt64(sorted), + Max: sorted[len(sorted)-1], + Deterministic: det, + } +} diff --git a/scripts/ci/go/internal/measure/summary.go b/scripts/ci/go/internal/measure/summary.go new file mode 100644 index 0000000..a05279c --- /dev/null +++ b/scripts/ci/go/internal/measure/summary.go @@ -0,0 +1,78 @@ +package measure + +import ( + "fmt" + "regexp" + "strconv" + "strings" + + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/algocore" +) + +// Summary is one parsed driver SUMMARY line: the clojure.test +// counts for a namespace run. +type Summary struct { + Test int64 + Pass int64 + Fail int64 + Error int64 +} + +var summaryKV = regexp.MustCompile(`:(test|pass|fail|error)\s+(\d+)`) + +// ParseSummary extracts the driver's SUMMARY counts from child +// stdout. Namespaces print nondeterministically (timing, random +// samples, hash order), so the SUMMARY line — not full stdout — +// is the correctness gate. The last marker wins, so a test that +// echoes the marker cannot shadow the driver's line; the line +// itself must be exactly `SUMMARY {}`. +func ParseSummary(text string) (Summary, error) { + var sum Summary + var ok bool + for _, line := range strings.Split(text, "\n") { + trimmed := strings.TrimSpace(line) + if !strings.HasPrefix(trimmed, "SUMMARY ") { + continue + } + body := strings.TrimSpace(strings.TrimPrefix(trimmed, "SUMMARY")) + if !strings.HasPrefix(body, "{") || !strings.HasSuffix(body, "}") { + continue + } + inner := body[1 : len(body)-1] + found := map[string]int64{} + for _, m := range summaryKV.FindAllStringSubmatch(inner, -1) { + n, err := strconv.ParseInt(m[2], 10, 64) + if err != nil { + continue + } + found[m[1]] = n + } + if len(found) != 4 { + continue + } + stripped := summaryKV.ReplaceAllString(inner, "") + stripped = strings.ReplaceAll(stripped, ",", "") + if strings.TrimSpace(stripped) != "" { + continue + } + sum = Summary{Test: found["test"], Pass: found["pass"], Fail: found["fail"], Error: found["error"]} + ok = true + } + if !ok { + tail := strings.TrimSpace(text) + if runes := []rune(tail); len(runes) > 200 { + tail = string(runes[len(runes)-200:]) + } + return Summary{}, fmt.Errorf("no SUMMARY line in: %s", algocore.PyRepr(tail)) + } + return sum, nil +} + +// CheckSummary gates one namespace run: failures or errors fail, +// anything else passes. Returns "" on success. +func CheckSummary(sum Summary) string { + if sum.Fail > 0 || sum.Error > 0 { + return fmt.Sprintf("namespace failed: :fail %d :error %d (of :test %d)", sum.Fail, sum.Error, sum.Test) + } + return "" +} diff --git a/scripts/ci/go/internal/measure/summary_test.go b/scripts/ci/go/internal/measure/summary_test.go new file mode 100644 index 0000000..a594fd8 --- /dev/null +++ b/scripts/ci/go/internal/measure/summary_test.go @@ -0,0 +1,64 @@ +package measure + +import ( + "testing" +) + +// The SUMMARY gate is new in algoperf (namespaces print +// nondeterministically, so full-stdout comparison cannot work). +// TDD: these checks first, implementation in summary.go. +func TestParseSummary(t *testing.T) { + sum, err := ParseSummary("Testing foo\n\nRan 2 tests containing 10 assertions.\n0 failures, 0 errors.\nSUMMARY {:test 2, :pass 10, :fail 0, :error 0}\n") + if err != nil { + t.Fatalf("parse valid: %v", err) + } + if sum.Test != 2 || sum.Pass != 10 || sum.Fail != 0 || sum.Error != 0 { + t.Fatalf("wrong counts: %+v", sum) + } +} + +func TestParseSummaryAmidNoise(t *testing.T) { + // Surrounding noise is fine; the last marker wins (a test + // echoing the marker cannot shadow the driver's line). + out := "random line\nSUMMARY {:test 9, :pass 9, :fail 0, :error 0}\nmore noise\nSUMMARY {:test 1, :pass 3, :fail 0, :error 0}\n" + sum, err := ParseSummary(out) + if err != nil { + t.Fatalf("parse amid noise: %v", err) + } + if sum.Test != 1 || sum.Pass != 3 { + t.Fatalf("wrong counts: %+v", sum) + } +} + +func TestParseSummaryMissing(t *testing.T) { + if _, err := ParseSummary("Testing foo\nboom\n"); err == nil { + t.Fatal("missing SUMMARY must error") + } +} + +func TestParseSummaryMalformed(t *testing.T) { + for _, bad := range []string{ + "SUMMARY {:test 2, :pass x, :fail 0, :error 0}\n", + "SUMMARY {:test 2}\n", + "SUMMARY not-a-map\n", + "SUMMARY {:test -1, :pass 0, :fail 0, :error 0}\n", + "SUMMARY {:test 1, :pass 3, :fail 0, :error 0} trailing\n", + } { + if _, err := ParseSummary(bad); err == nil { + t.Fatalf("malformed SUMMARY must error: %q", bad) + } + } +} + +func TestCheckSummary(t *testing.T) { + ok := Summary{Test: 2, Pass: 10} + if err := CheckSummary(ok); err != "" { + t.Fatalf("clean summary must pass: %q", err) + } + if err := CheckSummary(Summary{Test: 2, Pass: 9, Fail: 1}); err == "" { + t.Fatal("failures must fail the gate") + } + if err := CheckSummary(Summary{Test: 2, Pass: 9, Error: 1}); err == "" { + t.Fatal("errors must fail the gate") + } +} diff --git a/scripts/ci/go/internal/measure/table.go b/scripts/ci/go/internal/measure/table.go new file mode 100644 index 0000000..aaba32b --- /dev/null +++ b/scripts/ci/go/internal/measure/table.go @@ -0,0 +1,191 @@ +package measure + +import ( + "fmt" + "io" + "os" + "strconv" + "strings" + + "github.com/hhh-pa/clojure-algorithms/scripts/ci/go/internal/pyjson" +) + +// PrintTable prints one line per result on out. +func PrintTable(out io.Writer, results []CaseResult, failures []string) { + for _, res := range results { + tag := res.Label + "/" + res.Stem + "/" + res.Case + if !res.Ok { + errText := "" + if res.Error != nil { + errText = *res.Error + } + fmt.Fprintf(out, "FAIL %s\n", errText) + continue + } + s := res.Summary + insn := "None" + if s.Instructions != nil { + insn = numPlain(s.Instructions.Median) + } + alloc := "n/a" + if s.AllocatedBytes != nil { + alloc = fmt.Sprintf("%dB", s.AllocatedBytes.Min) + } + fmt.Fprintf(out, "%s: ok cpu_min=%.3fs cpu_med=%.3fs rss_max=%dK wall_min=%.3fs insn_med=%s alloc_min=%s\n", + tag, s.CpuS.Min.Float64(), s.CpuS.Median.Float64(), + s.MaxRSSKb.Max.I, s.WallS.Min.Float64(), insn, alloc) + } + ok := 0 + for _, r := range results { + if r.Ok { + ok++ + } + } + fmt.Fprintf(out, "measure: %d ok, %d failed\n", ok, len(failures)) +} + +// numPlain renders a Num the way Python str() renders its int/float: +// bare digits for ints, repr for floats (11.0 keeps its .0). +func numPlain(n pyjson.Num) string { + if n.IsInt { + return strconv.FormatInt(n.I, 10) + } + return pyjson.FloatRepr(n.F) +} + +// AppendSummary appends the Markdown results table, with the +// conditional cg_insn / cg_alloc columns (present only when some +// benchmark reports callgrind data). +func AppendSummary(path string, report Report, bmf BMF) error { + showCg := false + showCgAlloc := false + for _, m := range bmf { + if m.CallgrindInstructions != nil { + showCg = true + } + if m.CallgrindAllocatedBytes != nil { + showCgAlloc = true + } + } + header := "| namespace | cpu total (s) | peak RSS (MB) | alloc total (MB) | instructions |" + sep := "| --- | ---: | ---: | ---: | ---: |" + if showCg { + header += " cg_insn |" + sep += " ---: |" + } + if showCgAlloc { + header += " cg_alloc (MB) |" + sep += " ---: |" + } + header += " wall total (s) |" + sep += " ---: |" + cpuModel := "unknown cpu" + if report.Host.CpuModel != nil { + cpuModel = *report.Host.CpuModel + } + javaVersion := "unknown" + if report.Runtime.JavaVersion != nil { + javaVersion = *report.Runtime.JavaVersion + } + perfUsed := "no" + if report.Runtime.PerfUsed { + perfUsed = "yes" + } + allocRepeat := "n/a" + if report.Config.AllocRepeat != nil { + allocRepeat = fmt.Sprintf("%d", *report.Config.AllocRepeat) + } + runtimeVersion := "" + if report.Runtime.RuntimeVersion != nil { + runtimeVersion = fmt.Sprintf(", runtime: %s", *report.Runtime.RuntimeVersion) + } + ok := 0 + for _, r := range report.Results { + if r.Ok { + ok++ + } + } + lines := []string{ + "## Perf measurement (`algoperf measure`)", + "", + fmt.Sprintf("- %s (%s), java: %s%s", report.TimestampUTC, cpuModel, javaVersion, runtimeVersion), + fmt.Sprintf("- runtime: %s, perf counters: %s, repeat: %d, jobs: %d, alloc_repeat: %s", + report.Runtime.Name, perfUsed, report.Config.Repeat, report.Config.Jobs, allocRepeat), + fmt.Sprintf("- %d ok, %d failed", ok, len(report.Failures)), + "- Primary signals: min CPU time, max peak RSS, min allocated " + + "bytes, median instructions (load-independent); wall time is " + + "secondary. Compare same-machine ratios only.", + "", + header, + sep, + } + perNsWall := map[string]float64{} + perNsInsn := map[string][]*pyjson.Num{} + for _, res := range report.Results { + if !res.Ok { + continue + } + key := res.Label + perNsWall[key] += res.Summary.WallS.Median.Float64() + if res.Summary.Instructions != nil { + median := res.Summary.Instructions.Median + perNsInsn[key] = append(perNsInsn[key], &median) + } else { + perNsInsn[key] = append(perNsInsn[key], nil) + } + } + for _, name := range SortedNames(bmf) { + m := bmf[name] + if m.CpuTime == nil || m.PeakMemory == nil { + panic(fmt.Sprintf("measure: BMF entry %q has no cpu_time/peak_memory; cannot render summary", name)) + } + parts := perNsInsn[name] + insn := "n/a" + if len(parts) > 0 { + var nums []pyjson.Num + complete := true + for _, p := range parts { + if p == nil { + complete = false + break + } + nums = append(nums, *p) + } + if complete { + insn = numPlain(pyjson.Sum(nums)) + } + } + alloc := "n/a" + if m.AllocatedBytes != nil { + alloc = fmt.Sprintf("%.1f", m.AllocatedBytes.Value.Float64()/1048576) + } + row := fmt.Sprintf("| %s | %.3f | %.1f | %s | %s |", + name, + m.CpuTime.Value.Float64(), m.PeakMemory.Value.Float64()/1048576, + alloc, insn) + if showCg { + cg := "n/a" + if m.CallgrindInstructions != nil { + cg = numPlain(m.CallgrindInstructions.Value) + } + row += fmt.Sprintf(" %s |", cg) + } + if showCgAlloc { + cga := "n/a" + if m.CallgrindAllocatedBytes != nil { + cga = fmt.Sprintf("%.1f", m.CallgrindAllocatedBytes.Value.Float64()/1048576) + } + row += fmt.Sprintf(" %s |", cga) + } + row += fmt.Sprintf(" %.3f |", perNsWall[name]) + lines = append(lines, row) + } + lines = append(lines, "") + fd, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o666) + if err != nil { + return fmt.Errorf("cannot append summary to %s: %v", path, err) + } + defer fd.Close() + _, err = fd.WriteString(strings.Join(lines, "\n")) + return err +} diff --git a/scripts/ci/go/internal/measure/testdata/shlex.json b/scripts/ci/go/internal/measure/testdata/shlex.json new file mode 100644 index 0000000..dd84c3c --- /dev/null +++ b/scripts/ci/go/internal/measure/testdata/shlex.json @@ -0,0 +1,184 @@ +[ + { + "in": "java -cp a.jar clojure.main x.clj", + "tokens": [ + "java", + "-cp", + "a.jar", + "clojure.main", + "x.clj" + ] + }, + { + "in": "./build/{stem}", + "tokens": [ + "./build/{stem}" + ] + }, + { + "in": "java -cp '/path with/spaces.jar' clojure.main", + "tokens": [ + "java", + "-cp", + "/path with/spaces.jar", + "clojure.main" + ] + }, + { + "in": "say \"hello world\"", + "tokens": [ + "say", + "hello world" + ] + }, + { + "in": "a'b'c\"d\"e", + "tokens": [ + "abcde" + ] + }, + { + "in": "''", + "tokens": [ + "" + ] + }, + { + "in": "\"\"", + "tokens": [ + "" + ] + }, + { + "in": "a''b", + "tokens": [ + "ab" + ] + }, + { + "in": "x\\ y", + "tokens": [ + "x y" + ] + }, + { + "in": "trailing\\", + "error": "No escaped character" + }, + { + "in": "'unterminated", + "error": "No closing quotation" + }, + { + "in": "\"unterminated", + "error": "No closing quotation" + }, + { + "in": "a'b\"c", + "error": "No closing quotation" + }, + { + "in": "'a\\b'", + "tokens": [ + "a\\b" + ] + }, + { + "in": "\"a\\b\"", + "tokens": [ + "a\\b" + ] + }, + { + "in": "\"a\\\"b\"", + "tokens": [ + "a\"b" + ] + }, + { + "in": "\"a\\\\b\"", + "tokens": [ + "a\\b" + ] + }, + { + "in": "\"a\\nb\"", + "tokens": [ + "a\\nb" + ] + }, + { + "in": "a\\nb", + "tokens": [ + "anb" + ] + }, + { + "in": " spaced out ", + "tokens": [ + "spaced", + "out" + ] + }, + { + "in": "tab\there", + "tokens": [ + "tab", + "here" + ] + }, + { + "in": "'it''s'", + "tokens": [ + "its" + ] + }, + { + "in": "\"$HOME\"", + "tokens": [ + "$HOME" + ] + }, + { + "in": "'$HOME'", + "tokens": [ + "$HOME" + ] + }, + { + "in": "\\$HOME", + "tokens": [ + "$HOME" + ] + }, + { + "in": "semi;colon", + "tokens": [ + "semi;colon" + ] + }, + { + "in": "hash#tag", + "tokens": [ + "hash#tag" + ] + }, + { + "in": "dash--x=1", + "tokens": [ + "dash--x=1" + ] + }, + { + "in": "back\\slash", + "tokens": [ + "backslash" + ] + }, + { + "in": "'a'b'c'", + "tokens": [ + "abc" + ] + } +] diff --git a/scripts/ci/go/internal/pyjson/pyjson.go b/scripts/ci/go/internal/pyjson/pyjson.go new file mode 100644 index 0000000..a636e6c --- /dev/null +++ b/scripts/ci/go/internal/pyjson/pyjson.go @@ -0,0 +1,317 @@ +// Package pyjson renders JSON exactly like Python's +// json.dump(obj, indent=2) plus the caller's trailing newline: +// 2-space indent, no HTML escaping, ensure_ascii (non-ASCII as +// lowercase \uXXXX with surrogate pairs), and Python float repr +// (shortest round-trip, .0 for integral floats, e+XX exponents). +package pyjson + +import ( + "bytes" + "encoding/json" + "math" + "strconv" + "strings" + "unicode/utf8" +) + +// MarshalIndent2 renders v like json.dumps(v, indent=2) + "\n". +func MarshalIndent2(v any) ([]byte, error) { + var buf bytes.Buffer + enc := json.NewEncoder(&buf) + enc.SetEscapeHTML(false) + enc.SetIndent("", " ") + if err := enc.Encode(v); err != nil { + return nil, err + } + return JSONEscape(buf.Bytes()), nil +} + +// Num is a JSON number with Python int-or-float semantics: values +// that Python keeps integral (counts, odd-count int medians, int +// arithmetic) render as bare digits; everything else renders with +// Python float repr. +type Num struct { + IsInt bool + I int64 + F float64 +} + +// IntNum builds an integral number. +func IntNum(v int64) Num { return Num{IsInt: true, I: v} } + +// FloatNum builds a floating number. +func FloatNum(v float64) Num { return Num{F: v} } + +// Float64 returns the numeric value. +func (n Num) Float64() float64 { + if n.IsInt { + return float64(n.I) + } + return n.F +} + +// Add mirrors Python + on ints/floats (int+int stays integral). +func (n Num) Add(o Num) Num { + if n.IsInt && o.IsInt { + return IntNum(n.I + o.I) + } + return FloatNum(n.Float64() + o.Float64()) +} + +// Mul mirrors Python * on ints/floats. +func (n Num) Mul(o Num) Num { + if n.IsInt && o.IsInt { + return IntNum(n.I * o.I) + } + return FloatNum(n.Float64() * o.Float64()) +} + +// Cmp compares numerically (-1, 0, +1). +func (n Num) Cmp(o Num) int { + if n.IsInt && o.IsInt { + switch { + case n.I < o.I: + return -1 + case n.I > o.I: + return 1 + } + return 0 + } + a, b := n.Float64(), o.Float64() + switch { + case a < b: + return -1 + case a > b: + return 1 + } + return 0 +} + +// Sum mirrors Python sum() (starts at integral 0). +func Sum(nums []Num) Num { + acc := IntNum(0) + for _, n := range nums { + acc = acc.Add(n) + } + return acc +} + +// Max mirrors Python max() (returns the winning element as-is). +func Max(nums []Num) Num { + best := nums[0] + for _, n := range nums[1:] { + if n.Cmp(best) > 0 { + best = n + } + } + return best +} + +// MedianInt64 mirrors statistics.median over sorted ints: the +// middle value stays integral for odd counts; even counts average +// to a float, exactly like (a+b)/2. +func MedianInt64(sorted []int64) Num { + mid := len(sorted) / 2 + if len(sorted)%2 == 1 { + return IntNum(sorted[mid]) + } + return FloatNum((float64(sorted[mid-1]) + float64(sorted[mid])) / 2) +} + +// MedianFloat64 mirrors statistics.median over sorted floats. +func MedianFloat64(sorted []float64) Num { + mid := len(sorted) / 2 + if len(sorted)%2 == 1 { + return FloatNum(sorted[mid]) + } + return FloatNum((sorted[mid-1] + sorted[mid]) / 2) +} + +// MarshalJSON renders the number with Python formatting. +func (n Num) MarshalJSON() ([]byte, error) { + if n.IsInt { + return []byte(strconv.FormatInt(n.I, 10)), nil + } + return []byte(FloatRepr(n.F)), nil +} + +// UnmarshalJSON restores a number, keeping bare-digit literals +// integral and parsing the rest as float64 (including the JSON +// Infinity/NaN tokens Python emits). +func (n *Num) UnmarshalJSON(raw []byte) error { + text := strings.TrimSpace(string(raw)) + if text == "Infinity" || text == "+Infinity" { + *n = FloatNum(math.Inf(1)) + return nil + } + if text == "-Infinity" { + *n = FloatNum(math.Inf(-1)) + return nil + } + if text == "NaN" { + *n = FloatNum(math.NaN()) + return nil + } + if !strings.ContainsAny(text, ".eEnN") { + if v, err := strconv.ParseInt(text, 10, 64); err == nil { + *n = IntNum(v) + return nil + } + } + f, err := strconv.ParseFloat(text, 64) + if err != nil { + return err + } + *n = FloatNum(f) + return nil +} + +// FloatRepr formats f exactly like Python repr(float): shortest +// round-trip digits, ".0" for integral values, and e+XX exponents +// (signed, at least two digits) outside [1e-4, 1e16). +func FloatRepr(f float64) string { + if f != f { + return "NaN" + } + if f > 1.7976931348623157e308 { + return "Infinity" + } + if f < -1.7976931348623157e308 { + return "-Infinity" + } + if f == 0 { + if 1/f < 0 { + return "-0.0" + } + return "0.0" + } + // Shortest digits via scientific form: mantissa digits plus + // decimal-point position. + sci := strconv.FormatFloat(f, 'e', -1, 64) + mant, expStr, _ := strings.Cut(sci, "e") + exp, _ := strconv.Atoi(expStr) + neg := false + if strings.HasPrefix(mant, "-") { + neg = true + mant = mant[1:] + } + digits := strings.Replace(mant, ".", "", 1) + decpt := exp + 1 + var out strings.Builder + if neg { + out.WriteByte('-') + } + useExp := decpt > 16 || decpt <= -4 + if !useExp { + switch { + case decpt <= 0: + out.WriteString("0.") + out.WriteString(strings.Repeat("0", -decpt)) + out.WriteString(digits) + case decpt >= len(digits): + out.WriteString(digits) + out.WriteString(strings.Repeat("0", decpt-len(digits))) + out.WriteString(".0") + default: + out.WriteString(digits[:decpt]) + out.WriteByte('.') + out.WriteString(digits[decpt:]) + } + return out.String() + } + out.WriteByte(digits[0]) + if len(digits) > 1 { + out.WriteByte('.') + out.WriteString(digits[1:]) + } + out.WriteByte('e') + point := decpt - 1 + if point < 0 { + out.WriteByte('-') + point = -point + } else { + out.WriteByte('+') + } + expDigits := strconv.Itoa(point) + if len(expDigits) < 2 { + out.WriteByte('0') + } + out.WriteString(expDigits) + return out.String() +} + +const hexd = "0123456789abcdef" + +// JSONEscape rewrites encoder output the way ensure_ascii does, +// plus the two short escapes Go omits: every non-ASCII rune inside +// a string becomes lowercase \uXXXX (surrogate pairs above the +// BMP), and Go-emitted \u0008/\u000c become \b/\f. The scan is +// string- and backslash-aware, so literal "\u0008" source text +// (arriving double-escaped) is never rewritten. +func JSONEscape(in []byte) []byte { + var out []byte + inString := false + for i := 0; i < len(in); { + c := in[i] + if !inString { + if c == '"' { + inString = true + } + out = append(out, c) + i++ + continue + } + // Inside a string. + switch { + case c == '\\': + if i+1 < len(in) && in[i+1] == 'u' && i+6 <= len(in) { + seq := string(in[i+2 : i+6]) + if seq == "0008" { + out = append(out, '\\', 'b') + i += 6 + continue + } + if seq == "000c" { + out = append(out, '\\', 'f') + i += 6 + continue + } + } + // Copy the escape (or lone backslash) verbatim; + // a doubled backslash keeps the parity honest. + out = append(out, c) + if i+1 < len(in) { + out = append(out, in[i+1]) + i++ + } + i++ + case c == '"': + inString = false + out = append(out, c) + i++ + case c < 0x80: + out = append(out, c) + i++ + default: + r, w := utf8.DecodeRune(in[i:]) + if r == utf8.RuneError && w == 1 { + r = 0xFFFD + } + if r > 0xFFFF { + r -= 0x10000 + appendEscape(&out, 0xD800+r>>10) + appendEscape(&out, 0xDC00+r&0x3FF) + } else { + appendEscape(&out, r) + } + i += w + } + } + return out +} + +func appendEscape(out *[]byte, r rune) { + *out = append(*out, '\\', 'u', + hexd[byte(r>>12)&0xF], hexd[byte(r>>8)&0xF], + hexd[byte(r>>4)&0xF], hexd[byte(r)&0xF]) +} diff --git a/src/clojure_algorithms/ciphers/xor_cipher.clj b/src/clojure_algorithms/ciphers/xor_cipher.clj index 95e7c35..45c4528 100644 --- a/src/clojure_algorithms/ciphers/xor_cipher.clj +++ b/src/clojure_algorithms/ciphers/xor_cipher.clj @@ -1,5 +1,6 @@ (ns clojure-algorithms.ciphers.xor-cipher - "XOR cipher over characters.") + "XOR cipher over characters." + (:require clojure.java.io)) (defn- effective-key [key] diff --git a/src/clojure_algorithms/graphs/karger.clj b/src/clojure_algorithms/graphs/karger.clj index 5429156..34b93fd 100644 --- a/src/clojure_algorithms/graphs/karger.clj +++ b/src/clojure_algorithms/graphs/karger.clj @@ -1,5 +1,6 @@ (ns clojure-algorithms.graphs.karger - "Graph partitioning via Karger's randomized contraction.") + "Graph partitioning via Karger's randomized contraction." + (:require clojure.set)) (defn partition-graph "Contracts random edges until two nodes remain; returns the cutset diff --git a/src/clojure_algorithms/maths/power_using_recursion.clj b/src/clojure_algorithms/maths/power_using_recursion.clj index f6c523b..4884b3e 100644 --- a/src/clojure_algorithms/maths/power_using_recursion.clj +++ b/src/clojure_algorithms/maths/power_using_recursion.clj @@ -27,8 +27,10 @@ by an integer concatenates; anything else raises a TypeError." (defn power "Calculate the power of a base raised to an exponent using recursion." [base exponent] - (if (and (number? exponent) (zero? exponent)) - 1 - (if (number? exponent) - (mul-strict base (power base (dec-strict exponent))) - (dec-strict exponent)))) + (if (and (number? exponent) (neg? exponent)) + (throw (ex-info "negative exponent" {:type :IllegalArgumentException})) + (if (and (number? exponent) (zero? exponent)) + 1 + (if (number? exponent) + (mul-strict base (power base (dec-strict exponent))) + (dec-strict exponent))))) diff --git a/test/clojure_algorithms/data_structures/disjoint_set/disjoint_set_test.clj b/test/clojure_algorithms/data_structures/disjoint_set/disjoint_set_test.clj index 9b17f4f..8869f9b 100644 --- a/test/clojure_algorithms/data_structures/disjoint_set/disjoint_set_test.clj +++ b/test/clojure_algorithms/data_structures/disjoint_set/disjoint_set_test.clj @@ -1,5 +1,6 @@ (ns clojure-algorithms.data-structures.disjoint-set.disjoint-set-test (:require [clojure.test :refer [deftest is]] + clojure.set [clojure-algorithms.data-structures.disjoint-set.disjoint-set :refer [make-set find-set union-set find-python-set]])) From 48513c5d0292556fd2c57ad9ded84b1c7bf8ec55 Mon Sep 17 00:00:00 2001 From: Hao Hu Date: Tue, 3 Nov 2026 02:49:18 +0100 Subject: [PATCH 3/6] Issue #1: workflow review fixes (gate name, JDK pin, action versions, count-diff jq) - gate job renamed (was test: collides with ci.yml's check name) - JVM leg pins temurin 21.0.13 (deterministic Bencher history) - conservative action majors (v4/v5) - report-compare jq: single-object counts map (verified locally) --- .github/actions/algoperf-tools/action.yml | 4 ++-- .github/workflows/perf-callgrind.yml | 16 ++++++++++------ docs/1/specification.md | 3 ++- 3 files changed, 14 insertions(+), 9 deletions(-) diff --git a/.github/actions/algoperf-tools/action.yml b/.github/actions/algoperf-tools/action.yml index 764599a..3f8a44b 100644 --- a/.github/actions/algoperf-tools/action.yml +++ b/.github/actions/algoperf-tools/action.yml @@ -16,12 +16,12 @@ runs: using: composite steps: - id: cache - uses: actions/cache@v5 + uses: actions/cache@v4 with: path: algoperf-bin key: ${{ inputs.cache-key }} - if: steps.cache.outputs.cache-hit != 'true' - uses: actions/setup-go@v6 + uses: actions/setup-go@v5 with: go-version-file: scripts/ci/go/go.mod cache: false diff --git a/.github/workflows/perf-callgrind.yml b/.github/workflows/perf-callgrind.yml index eb325e0..0fff297 100644 --- a/.github/workflows/perf-callgrind.yml +++ b/.github/workflows/perf-callgrind.yml @@ -56,8 +56,9 @@ jobs: # Correctness gate for the sweep (same steps as ci.yml): never # sweep a red tree. (A cross-workflow `needs` is impossible, so - # the job is duplicated here; keep the two in sync.) - test: + # the job is duplicated here; keep the two in sync. Named `gate` + # so its check never collides with ci.yml's `test`.) + gate: runs-on: ubuntu-latest permissions: contents: read @@ -85,7 +86,7 @@ jobs: run: clojure -M:test perf-callgrind: - needs: [test, resolve-rs] + needs: [gate, resolve-rs] runs-on: ubuntu-latest permissions: contents: read @@ -107,17 +108,20 @@ jobs: with: cache-key: algoperf-${{ runner.os }}-${{ hashFiles('.github/actions/algoperf-tools/**', 'scripts/ci/go/go.mod', 'scripts/ci/go/cmd/algoperf/**', 'scripts/ci/go/internal/algocore/**', 'scripts/ci/go/internal/pyjson/**', 'scripts/ci/go/internal/measure/**') }} + # Pinned (not floating "21" like ci.yml): JDK micro + # versions move instruction counts, and the sweep's Bencher + # history must stay comparable. Bump deliberately. - name: Set up JDK 21 if: matrix.runtime == 'jvm' uses: actions/setup-java@v4 with: distribution: temurin - java-version: '21' + java-version: '21.0.13' - name: Cache Clojure jars if: matrix.runtime == 'jvm' id: jars-cache - uses: actions/cache@v5 + uses: actions/cache@v4 with: path: .ci/jars key: clojure-jars-1.12.0-spec0.5.238-core0.4.74-v1 @@ -320,7 +324,7 @@ jobs: - name: Diff JVM vs native test counts run: | for leg in jvm clojure-rs; do - jq -S '{(.results[] | select(.ok) | .label): .test_counts}' \ + jq -S '[.results[] | select(.ok) | {(.label): .test_counts}] | add' \ sweep-results-$leg-merged/sweep-results-merged.json > $leg-counts.json done diff jvm-counts.json clojure-rs-counts.json diff --git a/docs/1/specification.md b/docs/1/specification.md index 6e1b23f..b24163e 100644 --- a/docs/1/specification.md +++ b/docs/1/specification.md @@ -175,7 +175,8 @@ snippet repo's `scripts/ci/go/internal/measure`: 40-char SHA validated) — same as the snippet repo. - `perf-callgrind` matrix: `shard: [0..7]` x `runtime: [jvm, clojure-rs]` (16 legs). - - JVM leg: temurin 21 + cached Maven jars. + - JVM leg: pinned temurin 21.0.13 (deterministic + history) + cached Maven jars. - Native leg: stable Rust (`dtolnay/rust-toolchain`), checkout `hhh-pa/clojure` at the resolved SHA, `cargo build --release --locked --no-default-features` in From 6be8623ba2784f6411e0eb196a9f5a31d2f89b53 Mon Sep 17 00:00:00 2001 From: Hao Hu Date: Tue, 3 Nov 2026 03:01:43 +0100 Subject: [PATCH 4/6] Issue #1: locale pin, count-diff exceptions, review fixes - JVM template pins -Duser.language/country (comma-decimal hosts failed 13 namespaces; also stabilizes Ir) - report-compare normalizes :pass for the 2 random-graph namespaces (proven the only varying ones over full 1351x2 sweeps) --- .github/workflows/perf-callgrind.yml | 7 ++++++- docs/1/specification.md | 14 ++++++++++---- scripts/ci/go/internal/measure/measure.go | 7 +++++-- scripts/ci/go/internal/measure/selftest.go | 2 +- 4 files changed, 22 insertions(+), 8 deletions(-) diff --git a/.github/workflows/perf-callgrind.yml b/.github/workflows/perf-callgrind.yml index 0fff297..b6cfec7 100644 --- a/.github/workflows/perf-callgrind.yml +++ b/.github/workflows/perf-callgrind.yml @@ -323,8 +323,13 @@ jobs: - name: Diff JVM vs native test counts run: | + # Two namespaces assert per edge of a random graph, so + # their :pass counts vary run to run by design + # (:test/:fail/:error still compare exactly). + vary='["clojure-algorithms.graphs.karger-test","clojure-algorithms.graphs.random-graph-generator-test"]' for leg in jvm clojure-rs; do - jq -S '[.results[] | select(.ok) | {(.label): .test_counts}] | add' \ + jq --argjson vary "$vary" -S \ + '[.results[] | select(.ok) | .label as $l | {($l): (if ($l | IN($vary[])) then (.test_counts | .pass = 0) else .test_counts end)}] | add' \ sweep-results-$leg-merged/sweep-results-merged.json > $leg-counts.json done diff jvm-counts.json clojure-rs-counts.json diff --git a/docs/1/specification.md b/docs/1/specification.md index b24163e..c72e36d 100644 --- a/docs/1/specification.md +++ b/docs/1/specification.md @@ -116,8 +116,11 @@ Same driver both legs (identical workload): Invocations: -- JVM: `java -cp ':src:test:resources' clojure.main - perf/driver.clj ` (jars: clojure 1.12.0 + spec.alpha +- JVM: `java -Duser.language=en -Duser.country=US -cp + ':src:test:resources' clojure.main perf/driver.clj + ` (the locale pin keeps `format`-based assertions and + instruction counts host-independent; jars: clojure 1.12.0 + + spec.alpha 0.5.238 + core.specs.alpha 0.4.74, Maven-cached like the snippet CI — no Clojure CLI needed, test-runner not needed since the driver calls `clojure.test` directly). @@ -132,8 +135,11 @@ Invocations: dedicated `report-compare` job diffs the recorded `test_counts` maps with jq); any mismatch fails the report (behavioral drift can never hide behind perf - numbers). Randomized tests keep assertion counts stable, - so equality is expected. + numbers). Two namespaces (`graphs.karger-test`, + `graphs.random-graph-generator-test`) assert per edge of + a random graph, so their `:pass` counts vary by design + and are normalized before the diff (`:test`/`:fail`/`:error` + still compare exactly). - Timeouts are split: `--timeout` (default 1800s) budgets the native-speed runs (whose SUMMARY gate must stay decisive — the slowest namespace needs ~11min on the diff --git a/scripts/ci/go/internal/measure/measure.go b/scripts/ci/go/internal/measure/measure.go index cad816a..06d53b0 100644 --- a/scripts/ci/go/internal/measure/measure.go +++ b/scripts/ci/go/internal/measure/measure.go @@ -15,8 +15,11 @@ const PerfEvents = "instructions:u,cycles:u" // ToolVersion tags the report. const ToolVersion = 1 -// DefaultCommandTemplate runs one namespace on the JVM. -const DefaultCommandTemplate = "java -cp {cp} clojure.main perf/driver.clj {ns}" +// DefaultCommandTemplate runs one namespace on the JVM. The +// locale is pinned: several namespaces format doubles with +// `format` (locale-sensitive), so a comma-decimal host locale +// would fail assertions and perturb instruction counts. +const DefaultCommandTemplate = "java -Duser.language=en -Duser.country=US -cp {cp} clojure.main perf/driver.clj {ns}" // EpsilonFlags make heap-used-at-exit equal total allocation. var EpsilonFlags = []string{ diff --git a/scripts/ci/go/internal/measure/selftest.go b/scripts/ci/go/internal/measure/selftest.go index 7d25458..71581d3 100644 --- a/scripts/ci/go/internal/measure/selftest.go +++ b/scripts/ci/go/internal/measure/selftest.go @@ -59,7 +59,7 @@ func SelfTest(out io.Writer) int { // Templates. argv, err := BuildInnerCommand(DefaultCommandTemplate, "a.jar:b", "foo.bar-test") - want := []string{"java", "-cp", "a.jar:b", "clojure.main", "perf/driver.clj", "foo.bar-test"} + want := []string{"java", "-Duser.language=en", "-Duser.country=US", "-cp", "a.jar:b", "clojure.main", "perf/driver.clj", "foo.bar-test"} add("template-jvm", err == nil && equalStrs(argv, want), fmt.Sprint(argv, err)) argv, err = BuildInnerCommand("clojure-rs run perf/driver.clj {ns} src test", "", "foo.bar-test") want = []string{"clojure-rs", "run", "perf/driver.clj", "foo.bar-test", "src", "test"} From 4c2d268f742181fc61cedf9b398c324dcab84034 Mon Sep 17 00:00:00 2001 From: Hao Hu Date: Tue, 3 Nov 2026 04:12:12 +0100 Subject: [PATCH 5/6] Issue #1: sweep skip-list, relaxed shim check, OOM hint CI round 1 evidence (all 8 native shards harvested): - matrix-exponentiation exceeds both legs' budgets (native timed run >1800s on CI; JVM callgrind OOMs the 2g Epsilon heap, reproduced locally). perf/sweep_skips.json excludes it with a reviewed reason (reported, never silent). - shim cross-check compares :test/:fail/:error + :pass>0: the 2 random-graph namespaces tally different assertion counts per run, which flaked the exact comparison. - JVM callgrind OOM now reports the heap hint (was cryptic 'exit 3'). AI-Tool: Muse Code 1.4.14-R7309.1 (Meta Muse Spark) --- docs/1/specification.md | 11 +++++ perf/sweep_skips.json | 3 ++ scripts/ci/go/cmd/algoperf/discover.go | 29 +++++++++++++ scripts/ci/go/cmd/algoperf/discover_test.go | 41 +++++++++++++++++++ scripts/ci/go/cmd/algoperf/measure.go | 16 +++++++- scripts/ci/go/internal/measure/exec.go | 8 +++- scripts/ci/go/internal/measure/summarize.go | 12 ++++++ scripts/ci/go/internal/measure/summary.go | 12 ++++++ .../ci/go/internal/measure/summary_test.go | 23 +++++++++++ scripts/ci/go/internal/measure/table.go | 17 ++++++-- 10 files changed, 166 insertions(+), 6 deletions(-) create mode 100644 perf/sweep_skips.json diff --git a/docs/1/specification.md b/docs/1/specification.md index c72e36d..97a3d05 100644 --- a/docs/1/specification.md +++ b/docs/1/specification.md @@ -147,6 +147,17 @@ Invocations: budgets the instrumented phase, which degrades to `callgrind_skipped` with a reason (visible in JSON, excluded from BMF) — the T1211 policy. Tune from CI data. +- `perf/sweep_skips.json` excludes named namespaces with a + reviewed reason (recorded in the report as `skipped`, never + silent). Seeded with one entry: + `maths.matrix-exponentiation-test` exceeds both legs' + budgets (native run >1800s on CI, JVM EpsilonGC >2g heap) + while yielding no Bencher signal (its callgrind phase + would always skip); `ci.yml` still covers it. +- The shim cross-check compares `:test`/`:fail`/`:error` + plus `:pass > 0` (not exact `:pass`): the two random-graph + namespaces legitimately tally different assertion counts + run to run. ### `algoperf measure`: vendored recipe, native discovery diff --git a/perf/sweep_skips.json b/perf/sweep_skips.json new file mode 100644 index 0000000..0eaea6f --- /dev/null +++ b/perf/sweep_skips.json @@ -0,0 +1,3 @@ +{ + "clojure-algorithms.maths.matrix-exponentiation-test": "exceeds the sweep budgets: native run needs >1800s on CI (big-int fib benchmark), JVM EpsilonGC needs >2g heap. Still covered by ci.yml." +} diff --git a/scripts/ci/go/cmd/algoperf/discover.go b/scripts/ci/go/cmd/algoperf/discover.go index 9789503..06252fc 100644 --- a/scripts/ci/go/cmd/algoperf/discover.go +++ b/scripts/ci/go/cmd/algoperf/discover.go @@ -1,6 +1,7 @@ package main import ( + "encoding/json" "fmt" "os" "path/filepath" @@ -60,6 +61,34 @@ func DiscoverNamespaces(root string) ([]Namespace, error) { return out, nil } +// LoadSkips reads perf/sweep_skips.json (ns symbol -> reason), +// returning an empty map when the file is absent. Unknown +// namespaces are an error (a skip must name a real benchmark). +func LoadSkips(root string, known []Namespace) (map[string]string, error) { + path := filepath.Join(root, "perf", "sweep_skips.json") + raw, err := os.ReadFile(path) + if err != nil { + if os.IsNotExist(err) { + return map[string]string{}, nil + } + return nil, err + } + var skips map[string]string + if err := json.Unmarshal(raw, &skips); err != nil { + return nil, fmt.Errorf("%s: %v", path, err) + } + knownSet := map[string]bool{} + for _, n := range known { + knownSet[n.Ns] = true + } + for ns := range skips { + if !knownSet[ns] { + return nil, fmt.Errorf("%s: unknown namespace %q", path, ns) + } + } + return skips, nil +} + // readNsDecl reads the ns symbol from a file's ns decl (the first // `(ns ` form near the top of the file). func readNsDecl(path string) (string, error) { diff --git a/scripts/ci/go/cmd/algoperf/discover_test.go b/scripts/ci/go/cmd/algoperf/discover_test.go index 4bfc645..5642c3f 100644 --- a/scripts/ci/go/cmd/algoperf/discover_test.go +++ b/scripts/ci/go/cmd/algoperf/discover_test.go @@ -43,6 +43,47 @@ func TestDiscoverNamespaces(t *testing.T) { } } +func TestLoadSkips(t *testing.T) { + root := t.TempDir() + p := filepath.Join(root, "test", "clojure_algorithms", "odd_test.clj") + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte("(ns clojure-algorithms.odd-test)\n"), 0o644); err != nil { + t.Fatal(err) + } + found, err := DiscoverNamespaces(root) + if err != nil { + t.Fatal(err) + } + // Absent file: no skips. + skips, err := LoadSkips(root, found) + if err != nil || len(skips) != 0 { + t.Fatalf("absent skips: %v %v", skips, err) + } + // Known ns: loads. + perf := filepath.Join(root, "perf") + if err := os.MkdirAll(perf, 0o755); err != nil { + t.Fatal(err) + } + body := `{"clojure-algorithms.odd-test": "too slow"}` + if err := os.WriteFile(filepath.Join(perf, "sweep_skips.json"), []byte(body), 0o644); err != nil { + t.Fatal(err) + } + skips, err = LoadSkips(root, found) + if err != nil || skips["clojure-algorithms.odd-test"] != "too slow" { + t.Fatalf("load skips: %v %v", skips, err) + } + // Unknown ns: error. + body = `{"clojure-algorithms.nope-test": "x"}` + if err := os.WriteFile(filepath.Join(perf, "sweep_skips.json"), []byte(body), 0o644); err != nil { + t.Fatal(err) + } + if _, err := LoadSkips(root, found); err == nil { + t.Fatal("unknown ns must error") + } +} + func TestDiscoverNamespacesRejectsMismatch(t *testing.T) { root := t.TempDir() p := filepath.Join(root, "test", "clojure_algorithms", "odd_test.clj") diff --git a/scripts/ci/go/cmd/algoperf/measure.go b/scripts/ci/go/cmd/algoperf/measure.go index a24cf8d..5eaa7ae 100644 --- a/scripts/ci/go/cmd/algoperf/measure.go +++ b/scripts/ci/go/cmd/algoperf/measure.go @@ -143,15 +143,26 @@ func runMeasure(argv []string) int { fmt.Fprintln(os.Stderr, err) return 1 } + skips, err := LoadSkips(root, found) + if err != nil { + fmt.Fprintln(os.Stderr, err) + return 1 + } type task struct { ns, stem string } var tasks []task + var skipped []measure.CaseResult + driver := filepath.Join(root, "perf", "driver.clj") for _, entry := range found { if only != "" && !strings.Contains(entry.Ns, only) { continue } + if reason, excluded := skips[entry.Ns]; excluded { + skipped = append(skipped, measure.SkippedResult(entry.Ns, entry.Stem, "perf/driver.clj", "run", reason)) + continue + } tasks = append(tasks, task{entry.Ns, entry.Stem}) } if shardParsed != nil { @@ -179,7 +190,6 @@ func runMeasure(argv []string) int { effectiveCg = callgrindRepeat } - driver := filepath.Join(root, "perf", "driver.clj") results := make([]measure.CaseResult, len(tasks)) work := make(chan int) total := len(tasks) @@ -213,6 +223,10 @@ func runMeasure(argv []string) int { close(work) }() workers.Wait() + for _, s := range skipped { + fmt.Printf("measure [skip] %s/%s/%s: %s\n", s.Label, s.Stem, s.Case, *s.Skipped) + } + results = append(results, skipped...) sort.Slice(results, func(i, j int) bool { if results[i].Label != results[j].Label { return results[i].Label < results[j].Label diff --git a/scripts/ci/go/internal/measure/exec.go b/scripts/ci/go/internal/measure/exec.go index e059ec0..8b874d4 100644 --- a/scripts/ci/go/internal/measure/exec.go +++ b/scripts/ci/go/internal/measure/exec.go @@ -308,7 +308,11 @@ func jvmCallgrindOnce(inner []string, timeout int) ([]byte, int64, int64, string return nil, 0, 0, fmt.Sprintf("TIMEOUT (>%ds)", timeout) } if err != nil { - return nil, 0, 0, fmt.Sprintf("exit %d: %s", algocore.ExitCode(err), algocore.LastLine(stderr)) + msg := fmt.Sprintf("exit %d: %s", algocore.ExitCode(err), algocore.LastLine(stderr)) + if strings.Contains(string(stderr), "OutOfMemoryError") { + msg += " (Epsilon heap exhausted; raise -Xmx in EPSILON_FLAGS)" + } + return nil, 0, 0, msg } raw, err := os.ReadFile(outFile) if err != nil { @@ -390,7 +394,7 @@ func MeasureShimAllocOnce(inner []string, wantStdout []byte, timeout int) (int64 } wantSum, wantErr := ParseSummary(string(wantStdout)) gotSum, gotErr := ParseSummary(string(stdout)) - if wantErr != nil || gotErr != nil || wantSum != gotSum { + if wantErr != nil || gotErr != nil || !summariesAgree(wantSum, gotSum) { return 0, "alloc-shim output mismatch: counter run diverged from the callgrind run" } used, err := ParseAllocShimTotal(string(stderr)) diff --git a/scripts/ci/go/internal/measure/summarize.go b/scripts/ci/go/internal/measure/summarize.go index f751699..1548be0 100644 --- a/scripts/ci/go/internal/measure/summarize.go +++ b/scripts/ci/go/internal/measure/summarize.go @@ -66,6 +66,18 @@ type CaseResult struct { Summary *CaseSummary `json:"summary"` TestCounts *TestCounts `json:"test_counts"` CallgrindSkipped *string `json:"callgrind_skipped,omitempty"` + Skipped *string `json:"skipped,omitempty"` +} + +// SkippedResult records a sweep_skips.json exclusion: Ok stays +// true (the exclusion is deliberate and reviewed), no runs exist, +// and BMF/summary builders skip it. +func SkippedResult(label, stem, script, cas, reason string) CaseResult { + return CaseResult{ + Label: label, Stem: stem, Script: script, Case: cas, + Ok: true, Runs: []RunMetrics{}, AllocRuns: []int64{}, + Skipped: &reason, + } } // TestCounts is the driver's SUMMARY counts for one namespace. diff --git a/scripts/ci/go/internal/measure/summary.go b/scripts/ci/go/internal/measure/summary.go index a05279c..93f5384 100644 --- a/scripts/ci/go/internal/measure/summary.go +++ b/scripts/ci/go/internal/measure/summary.go @@ -76,3 +76,15 @@ func CheckSummary(sum Summary) string { } return "" } + +// summariesAgree compares two runs of the same namespace for the +// shim cross-check: same tests, no failures/errors either side, +// and assertions on both sides. :pass itself is excluded — two +// namespaces assert per edge of a random graph, so their tallies +// legitimately differ run to run. (Cross-runtime drift detection +// in report-compare stays strict with explicit exceptions; this +// check only guards counter-run divergence.) +func summariesAgree(a, b Summary) bool { + return a.Test == b.Test && a.Fail == 0 && a.Error == 0 && + b.Fail == 0 && b.Error == 0 && a.Pass > 0 && b.Pass > 0 +} diff --git a/scripts/ci/go/internal/measure/summary_test.go b/scripts/ci/go/internal/measure/summary_test.go index a594fd8..7b86640 100644 --- a/scripts/ci/go/internal/measure/summary_test.go +++ b/scripts/ci/go/internal/measure/summary_test.go @@ -50,6 +50,29 @@ func TestParseSummaryMalformed(t *testing.T) { } } +func TestSummariesAgree(t *testing.T) { + base := Summary{Test: 1, Pass: 26} + if !summariesAgree(base, Summary{Test: 1, Pass: 22}) { + t.Fatal("pass-only variance (random-graph) must agree") + } + if !summariesAgree(base, base) { + t.Fatal("identical must agree") + } + for _, other := range []Summary{ + {Test: 2, Pass: 26}, // different tests + {Test: 1, Pass: 26, Fail: 1}, // failures + {Test: 1, Pass: 26, Error: 1}, // errors + {Test: 1, Pass: 0}, // no assertions + } { + if summariesAgree(base, other) { + t.Fatalf("must disagree: %+v vs %+v", base, other) + } + } + if summariesAgree(Summary{Test: 1, Fail: 1}, base) { + t.Fatal("failed first side must disagree") + } +} + func TestCheckSummary(t *testing.T) { ok := Summary{Test: 2, Pass: 10} if err := CheckSummary(ok); err != "" { diff --git a/scripts/ci/go/internal/measure/table.go b/scripts/ci/go/internal/measure/table.go index aaba32b..afd5dd4 100644 --- a/scripts/ci/go/internal/measure/table.go +++ b/scripts/ci/go/internal/measure/table.go @@ -22,6 +22,10 @@ func PrintTable(out io.Writer, results []CaseResult, failures []string) { fmt.Fprintf(out, "FAIL %s\n", errText) continue } + if res.Skipped != nil { + fmt.Fprintf(out, "%s: skipped (%s)\n", tag, *res.Skipped) + continue + } s := res.Summary insn := "None" if s.Instructions != nil { @@ -100,18 +104,25 @@ func AppendSummary(path string, report Report, bmf BMF) error { runtimeVersion = fmt.Sprintf(", runtime: %s", *report.Runtime.RuntimeVersion) } ok := 0 + skipped := 0 for _, r := range report.Results { - if r.Ok { + if r.Skipped != nil { + skipped++ + } else if r.Ok { ok++ } } + skippedNote := "" + if skipped > 0 { + skippedNote = fmt.Sprintf(", %d skipped", skipped) + } lines := []string{ "## Perf measurement (`algoperf measure`)", "", fmt.Sprintf("- %s (%s), java: %s%s", report.TimestampUTC, cpuModel, javaVersion, runtimeVersion), fmt.Sprintf("- runtime: %s, perf counters: %s, repeat: %d, jobs: %d, alloc_repeat: %s", report.Runtime.Name, perfUsed, report.Config.Repeat, report.Config.Jobs, allocRepeat), - fmt.Sprintf("- %d ok, %d failed", ok, len(report.Failures)), + fmt.Sprintf("- %d ok%s, %d failed", ok, skippedNote, len(report.Failures)), "- Primary signals: min CPU time, max peak RSS, min allocated " + "bytes, median instructions (load-independent); wall time is " + "secondary. Compare same-machine ratios only.", @@ -122,7 +133,7 @@ func AppendSummary(path string, report Report, bmf BMF) error { perNsWall := map[string]float64{} perNsInsn := map[string][]*pyjson.Num{} for _, res := range report.Results { - if !res.Ok { + if !res.Ok || res.Skipped != nil { continue } key := res.Label From 15824e1a9cba8bf4f891a47c5c49f67e98f6f0c3 Mon Sep 17 00:00:00 2001 From: Hao Hu Date: Tue, 3 Nov 2026 05:44:46 +0100 Subject: [PATCH 6/6] Issue #1: skip problem-056 (7.4g EpsilonGC need) Round-2 JVM shard 3 failed its callgrind phase with OOM (same exit-3 signature as matrix-exp). Measured locally: 7.4g heap used; the next-heaviest namespace needs 0.67g, so the 2g budget stays and problem-056 joins the skip-list with the measured reason. AI-Tool: Muse Code 1.4.14-R7309.1 (Meta Muse Spark) --- docs/1/specification.md | 13 ++++++++----- perf/sweep_skips.json | 3 ++- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/1/specification.md b/docs/1/specification.md index 97a3d05..25499d7 100644 --- a/docs/1/specification.md +++ b/docs/1/specification.md @@ -149,11 +149,14 @@ Invocations: excluded from BMF) — the T1211 policy. Tune from CI data. - `perf/sweep_skips.json` excludes named namespaces with a reviewed reason (recorded in the report as `skipped`, never - silent). Seeded with one entry: - `maths.matrix-exponentiation-test` exceeds both legs' - budgets (native run >1800s on CI, JVM EpsilonGC >2g heap) - while yielding no Bencher signal (its callgrind phase - would always skip); `ci.yml` still covers it. + silent). Seeded with two entries, both still covered by + `ci.yml`: `maths.matrix-exponentiation-test` (native run + >1800s on CI, JVM EpsilonGC >2g heap, and its callgrind + phase would always skip — no Bencher signal either way) + and `project-euler.problem-056.sol1-test` (JVM EpsilonGC + needs 7.4g measured; raising the 2g budget would OOM the + runners at jobs=2 — the next-heaviest namespace needs + 0.67g, so 2g stays). - The shim cross-check compares `:test`/`:fail`/`:error` plus `:pass > 0` (not exact `:pass`): the two random-graph namespaces legitimately tally different assertion counts diff --git a/perf/sweep_skips.json b/perf/sweep_skips.json index 0eaea6f..69fdf69 100644 --- a/perf/sweep_skips.json +++ b/perf/sweep_skips.json @@ -1,3 +1,4 @@ { - "clojure-algorithms.maths.matrix-exponentiation-test": "exceeds the sweep budgets: native run needs >1800s on CI (big-int fib benchmark), JVM EpsilonGC needs >2g heap. Still covered by ci.yml." + "clojure-algorithms.maths.matrix-exponentiation-test": "exceeds the sweep budgets: native run needs >1800s on CI (big-int fib benchmark), JVM EpsilonGC needs >2g heap. Still covered by ci.yml.", + "clojure-algorithms.project-euler.problem-056.sol1-test": "JVM EpsilonGC needs 7.4g heap (measured), exceeding the 2g budget; raising it would OOM the runners at jobs=2. Still covered by ci.yml." }