add AVX2/AVX-512 to ksw_extd2 with runtime dispatch - #20
Conversation
5ee0228 to
cd10f54
Compare
ksw_extd2 (dual-gap banded extension) is the main base-alignment cost for long reads (~50-66% of HiFi/ONT CPU) and was 128-bit only. This adds AVX2 and AVX-512BW versions of the gap-fill path, selected at runtime via __builtin_cpu_supports (one binary; falls back to SSE4.2/NEON). Only the banded approx+cigar path is widened; other flags call the original kernel. Output is bit-identical to the SSE4.2 build on all four HG002 zenodo sets vs GRCh38, across sse/avx2/avx512 (~4.4M alignment records, 0 diffs). End-to-end on a c7i (user CPU, vs SSE4.2): set readlen mean/median avx2 avx512 WGS-1M 151 / 151 1.00x 1.00x HiC-1M 151 / 151 1.00x 1.00x SBX-HG002 241 / 227 1.00x 1.00x HiFi-10k 18881 / 18047 1.15x 1.19x ONT-10k 19810 / 18180 1.11x 1.14x The win needs wide-band DP -- long, divergent reads where gap-fill bands are wide; it is absent on short/mid reads and on low-divergence data of any length (confirmed by simulation: flat with small indels at every length, ~1.09x only at 16 kb with large indels). MINIBWA_EXTD2_ISA=sse|avx2|avx512 forces a tier for testing.
cd10f54 to
fde95fe
Compare
|
I like the way AVX2/AVX512 is implemented but this will considerably increase the burden of maintenance in the long term. Given that the short-read performance remains the same and minimap2 still dominates long-read alignment, I will not merge now. I will keep the PR open until there is a real need. |
|
I agree there's a tension between maintenance burden and trying to make |
|
Independently reproduced on a fresh c7i.4xlarge (Xeon 8488C), this time as wall-clock of the mapping phase rather than user CPU, five reps per tier interleaved with order reversal, Same shape as the user-CPU numbers above, slightly lower as expected for a wall-clock metric that includes the unwidened parts of the run. SAM output byte-identical across all three tiers on HiFi and WGS, re-confirming the bit-identity check. Two notes for whoever picks this up: The short-read rows were measured with #64 speeds up the 128-bit |
ksw_extd2(dual-gap banded extension) is minibwa's main base-alignment cost for long reads (~50–66% of HiFi/ONT CPU) and was 128-bit only. This adds AVX2 and AVX-512BW implementations of the gap-fill path, selected at runtime via__builtin_cpu_supportsso a single binary auto-uses the widest available ISA and falls back to SSE4.2/NEON otherwise. Only the banded approx+cigar path is widened; every other flag combination calls the original kernel.Output is bit-identical to the SSE4.2 build on all four HG002 zenodo sets vs GRCh38, across sse/avx2/avx512 (~4.4M alignment records, 0 diffs).
End-to-end on a c7i (Sapphire Rapids), user CPU vs SSE4.2:
The win needs wide-band DP — long, divergent reads where the gap-fill bands are wide; it is absent on short/mid reads and on low-divergence data of any length (confirmed by simulation: flat with small indels at every length, ~1.09x only at 16 kb with large indels).
MINIBWA_EXTD2_ISA=sse|avx2|avx512forces a tier for testing.