Proposal
Add a manifest feature that vendors zlib-ng and statically links it in ZLIB_COMPAT mode, replacing the Makefile's -lz. dist currently links whatever system libz the builder happens to have.
Why it qualifies for the distribution
It is measured, byte-identical, and was declined upstream on taste rather than on correctness — the manifest's admission criterion.
lh3#28 measured it on a 96-core Graviton4 against hg38, 20M pairs as one interleaved .gz, hyperfine x3, SAM byte-identical:
| -t |
zlib |
zlib-ng |
faster |
| 16 |
155.67 |
155.44 |
tied |
| 32 |
79.98 |
79.80 |
tied |
| 64 |
42.90 |
42.37 |
1.3% |
| 96 |
43.45 |
40.70 |
7.0% |
Upstream's response was that -t 96 is a rare use case and it does not belong in the main documentation. That is a reasonable position for upstream and exactly the kind of thing this distribution exists to carry.
The build logic already exists
The bioconda recipe for upstream releases does precisely this: recipes/minibwa/build.sh fetches zlib-ng 2.3.3, builds it with -DZLIB_COMPAT=ON -DBUILD_SHARED_LIBS=OFF, and swaps the Makefile's -lz for the static archive:
LIBS="-lpthread ${ZLIBNG_LIBDIR}/libz.a -lm"
So this is largely a matter of moving working, shipping build logic into a feature branch.
Interaction with single-copy-parser — please read before implementing
These two features substantially cancel, and the distribution probably wants one of them rather than both.
single-copy-parser speeds up the parse/copy that happens after decompression. It is worth roughly 10% of wall at high thread counts against stock zlib, and roughly 0 against zlib-ng, because once inflate is fast the read stage hides under mapping and there is nothing left to win. That is stated in lh3#34, whose own option (1) was "drop it — zlib-ng is the better read-stage lever".
Two further points favour zlib-ng if only one is kept:
- Upstream's objection to the parser was "not worth the added complexity" — it adds a second FASTQ parser while
kseq.h remains in use by l2bit.c and fastmap.c. zlib-ng adds no new code to maintain.
- Past roughly 192 threads, single-threaded inflate dominates the read stage regardless, so the parser only ever addressed half the problem.
Whichever feature sits earlier in manifest order will absorb the credit for the shared win, so the ordering should be chosen deliberately rather than by accident.
Caveat: a new kind of manifest feature
Every current feature is a source patch merged onto upstream. This one changes the build's dependency surface — it needs cmake and ninja and fetches a tarball — so assembly and the byte-identity gate both get heavier. SAM output should remain byte-identical, since zlib-ng in ZLIB_COMPAT mode emits identical decompressed bytes; that is worth confirming rather than assuming.
Suggested sequence
The high-thread regime is the only place either feature acts, and neither has been measured there in this fork. Measure that axis first, then decide between the two with numbers for both halves of the trade.
Proposal
Add a manifest feature that vendors zlib-ng and statically links it in
ZLIB_COMPATmode, replacing the Makefile's-lz.distcurrently links whatever system libz the builder happens to have.Why it qualifies for the distribution
It is measured, byte-identical, and was declined upstream on taste rather than on correctness — the manifest's admission criterion.
lh3#28 measured it on a 96-core Graviton4 against hg38, 20M pairs as one interleaved
.gz, hyperfine x3, SAM byte-identical:Upstream's response was that
-t 96is a rare use case and it does not belong in the main documentation. That is a reasonable position for upstream and exactly the kind of thing this distribution exists to carry.The build logic already exists
The bioconda recipe for upstream releases does precisely this:
recipes/minibwa/build.shfetches zlib-ng 2.3.3, builds it with-DZLIB_COMPAT=ON -DBUILD_SHARED_LIBS=OFF, and swaps the Makefile's-lzfor the static archive:So this is largely a matter of moving working, shipping build logic into a feature branch.
Interaction with
single-copy-parser— please read before implementingThese two features substantially cancel, and the distribution probably wants one of them rather than both.
single-copy-parserspeeds up the parse/copy that happens after decompression. It is worth roughly 10% of wall at high thread counts against stock zlib, and roughly 0 against zlib-ng, because once inflate is fast the read stage hides under mapping and there is nothing left to win. That is stated in lh3#34, whose own option (1) was "drop it — zlib-ng is the better read-stage lever".Two further points favour zlib-ng if only one is kept:
kseq.hremains in use byl2bit.candfastmap.c. zlib-ng adds no new code to maintain.Whichever feature sits earlier in manifest order will absorb the credit for the shared win, so the ordering should be chosen deliberately rather than by accident.
Caveat: a new kind of manifest feature
Every current feature is a source patch merged onto upstream. This one changes the build's dependency surface — it needs cmake and ninja and fetches a tarball — so assembly and the byte-identity gate both get heavier. SAM output should remain byte-identical, since zlib-ng in
ZLIB_COMPATmode emits identical decompressed bytes; that is worth confirming rather than assuming.Suggested sequence
The high-thread regime is the only place either feature acts, and neither has been measured there in this fork. Measure that axis first, then decide between the two with numbers for both halves of the trade.