Repository navigation
Conversation
The 13 callgrind-slow benchmarks (9 JVM + 4 clojure-rs) each get a dedicated slow leg with a 3600s/case budget, so they recover without stalling the fast path: - snip measure gains --exclude (comma-separated, label/stem match), --callgrind-xmx (EpsilonGC heap for the JVM callgrind recipe) and --callgrind-include-exempt (run exempt labels instead of skipping); --only gains the same comma/label-stem matching (superset of the old label-substring behavior; the pilot's --only selects identically). - perf-callgrind main shards exclude exactly the slow set (verified 1:1 against the real label/stem universe); each slow leg pins one benchmark, reports its Bencher subset, and fails when empty. - report-callgrind merges slow artifacts too and fails on any remaining callgrind skip, so no benchmark stays cross-check-exempt. - beam_search pairs --callgrind-include-exempt with --callgrind-xmx 8g (its allocation exceeds the default 2g EpsilonGC heap). - Documents the cross-repo comparison rule (sum of per-case MIN Ir); benchmark names and the clojure aggregation are unchanged. AI-Tool: Muse Code (Meta Muse Spark)
|
Bencher Report| Project | clojure-code-snippet |
| Branch | fix/40-comparability |
| Commit | bba3c27 |
| Report | Testbed | Adapter | Status |
|---|---|---|---|
| 1 | ubuntu-latest-clojure-rs | JSON | ✅ No Alerts |
| 2 | ubuntu-latest-jvm | JSON | ✅ No Alerts |
Report 1: 1 benchmark
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
| Benchmark | callgrind_allocated_bytes | Measure (units) x 1e9 | callgrind_instructions | Measure (units) x 1e9 |
|---|---|---|---|---|
| graph/shortest_path/dijkstrashortreach | 📈 view plot | 60.83 units x 1e9 | 📈 view plot 🚷 view threshold | 983.16 units x 1e9 |
Report 2: 1 benchmark
⚠️ WARNING: No Threshold found!Without a Threshold, no Alerts will ever be generated.
Click here to create a new Threshold
For more information, see the Threshold documentation.
To only post results if a Threshold exists, set the--ci-only-thresholdsflag.
| Benchmark | callgrind_allocated_bytes | Measure (units) x 1e9 | callgrind_instructions | Measure (units) x 1e12 |
|---|---|---|---|---|
| string_related/automata/count_strings | 📈 view plot | 9.03 units x 1e9 | 📈 view plot 🚷 view threshold | 2.45 units x 1e12 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #40 — the clojure-side comparability work: slow-leg recovery so
no benchmark lacks Ir, with the aggregation rule documented (clojure
already uses Σ-min, unchanged).
Changes
scripts/ci/go/internal/measure/filter.go: comma-list splitting,label/stem substring matching,
--callgrind-xmxvalidation.cmd/snip/measure.go+exec.go+report.go: new--exclude,--callgrind-xmx(default2g),--callgrind-include-exemptflags;--onlyuses the same matcher;ToolVersion5→6.test.yml: main shards exclude the 13 slow (benchmark, runtime) pairs;new
perf-callgrind-slowjob (13 legs,--timeout 3600,--jobs 1for beam_search + dijkstrashortreach, beam_search with--callgrind-xmx 8g);report-callgrindmerges slow artifacts andfails on any remaining
callgrind_skipped.Validation
snip measure --self-test: 0 failed (incl. 6 new filter checks);go test ./...: 11 pkgs ok; gofmt/go vet/diff-check clean; YAML parses.--onlypattern proven to match exactly its benchmark in thereal
Discoveruniverse; exclude lists script-checked against theslow matrix. No benchmark renames (bmf.go diff is comment-only).
CI-time impact
Main shards get faster (stop burning ~11 CPU-h of 900s timeouts); slow
legs add parallel jobs bounded per-leg — wall time ≈ slowest slow leg.
Sequencing: land #45 first so trie/edit_distance legs shrink. If any
pre-#45 case exceeds 3600s, CI goes red by design.
AI-Tool: Muse Code (Meta Muse Spark)