[codspeed]

Summary

This PR sets up continuous performance tracking with CodSpeed. CodSpeed supports the JVM through a fork of JMH (Java Microbenchmark Harness), so the benchmarks are JMH classes that call the Clojure functions directly. The measurements use the walltime instrument.

The CodSpeed workflow passed on this PR and reported all 50 benchmarks (run).

Note on runners

The benchmark job currently runs on ubuntu-latest. I first targeted CodSpeed macro runners (codspeed-macro), but no macro runner picked up the job, so they don't seem to be enabled for this organization yet. On shared GitHub runners, walltime results vary a lot more than on macro runners, and CodSpeed shows a "Walltime results may be unreliable" warning about it. Once macro runners are enabled, change runs-on: ubuntu-latest to runs-on: codspeed-macro in .github/workflows/codspeed.yml. See also walltime usage outside macro runners.

What's included

• third-party/codspeed-jvm: git submodule pinned to v0.2.1 of CodSpeedHQ/codspeed-jvm. It provides the CodSpeed JMH fork, which isn't published to Maven Central yet.
• bench/: a small standalone Gradle project (with its own wrapper) that keeps the JVM benchmark tooling out of the deps.edn setup:
  • settings.gradle.kts swaps upstream JMH for the CodSpeed fork through a composite build.
  • build.gradle.kts uses the me.champeau.jmh plugin. It adds ../src, ../resources and the fixtures to the benchmark classpath, so the Clojure code is loaded straight from source.
  • src/jmh/clojure/.../fixtures.clj builds deterministic inputs from seeded java.util.Random: random vectors, text, graphs and grids.
  • src/jmh/java/...: 50 benchmarks in 8 groups:
    • Sorts: merge, quick, heap, shell, counting, tim, insertion
    • Searches: binary search, bisect, linear search, quick select
    • Strings: KMP, Z-function, Rabin-Karp, Boyer-Moore-Horspool, Levenshtein, Jaro-Winkler, Manacher
    • Hashes: SHA-256, SHA-1, MD5, CRC32, Adler-32, djb2
    • Data compression: Huffman, LZ77, LZW, Burrows-Wheeler, RLE
    • Dynamic programming: LCS, edit distance, knapsack, LIS, matrix chain order, max subarray, Floyd-Warshall
    • Graphs: Dijkstra (adjacency list and binary grid), BFS, Kruskal, Tarjan, Kosaraju
    • Maths: three prime sieves, primality test, prime factorisation, totient, Fibonacci, Karatsuba
• .github/workflows/codspeed.yml runs on pushes to main, on pull requests and on workflow_dispatch. It fetches only the JMH fork and its native hooks (not the example submodules bundled with codspeed-jvm), builds the benchmark jar, and runs it through CodSpeedHQ/action@v5 in walltime mode. It authenticates with OIDC.
• README: CodSpeed badge and a "Benchmarks" section explaining how to run the benchmarks locally.

The existing CI (cljfmt and the test suite) is unchanged and passes. fixtures.clj passes cljfmt.

Running locally

git submodule update --init third-party/codspeed-jvm
git -C third-party/codspeed-jvm submodule update --init jmh-fork/jmh-core/native-instrument-hooks/instrument-hooks
(cd bench && ./gradlew jmh -Pbench=Sorts)   # -Pbench is an optional regex filter

Next steps

• Enable CodSpeed macro runners for the organization and switch the job to codspeed-macro to get stable measurements.
• Once merged, the run on main becomes the baseline, and later pull requests get a performance report comment.
• To add a benchmark, add a @Benchmark method to one of the classes in bench/src/jmh/java/clojure_algorithms/bench/. If it needs input data, add a generator to fixtures.clj.
• Every benchmark currently runs 3 x 1s of warmup and 5 x 1s of measurement in a single fork. The benchmark step takes about 14 minutes on ubuntu-latest. You can change this with the @Warmup, @Measurement and @Fork annotations on each class.

If you'd like more coverage, comment @codspeedbot add benchmarks for the matrix and backtracking namespaces and I'll add them.

[codspeed]

Congrats! CodSpeed is installed 🎉

🆕 50 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

Detected benchmarks

• burrowsWheelerTransform500: 253.2 µs
• huffmanEncode4k: 622.7 µs
• lempelZivCompress4k: 1.2 ms
• lz77Compress1k: 952.9 µs
• runLengthEncode4k: 533.7 µs
• editDistance300: 39.5 ms
• floydWarshall60: 78.5 ms
• knapsack100Items: 16.6 ms
• longestCommonSubsequence150: 8.4 ms
• longestIncreasingSubsequence10k: 2.5 ms
• matrixChainOrder60: 11 ms
• maxSubarraySum10k: 199.8 µs
• breadthFirstSearch5000Nodes: 6.7 ms
• dijkstra2000Nodes: 8 ms
• dijkstraBinaryGrid40x40: 4.8 ms
• kosarajuScc2000Nodes: 3.1 ms
• kruskalMst2000Nodes: 7.3 ms
• tarjanScc2000Nodes: 2 ms
• adler32_4k: 130.9 µs
• crc32_4k: 54 µs
• ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

---