Fix BGZF truncation at embedded EOF markers - #712
Conversation
|
Tested this patch against #711 on real data It fixes the problem and reproduces pre regression behaviour exactly. Built master and this PR from source (Ubuntu 24.04, isa-l / libdeflate / libhwy from apt). 1. Real dataA WES sample straight off bcl convert, not manually concatenated The R1 file contains two BGZF streams and 24,910,823 read pairs (verified with
The master figure is exactly the number of read pairs in the first BGZF stream, confirming it halts at the first EOF block. 2. Minimal reproducerTwo samll files of 200 read pairs each, compressed separately and concatenated, so the correct answer is 400: bgzip -c a_R1.fastq > a.bgz # 200 pairs
bgzip -c b_R1.fastq > b.bgz # 200 pairs
cat a.bgz b.bgz > merged_R1.fastq.gz # 400 pairs
A single stream 3. Equivalet with v1.1.0v1.1.0 is the last release before the regression. Same file, same flags. These counts are R1+R2 combined, as reported in the JSON:
Identical on every metric, and the final read in the output file is the same in both. Read order and pairing are also preserved: the first read of stream 2 appears in the output immediately after the last read of stream 1, and R1/R2 name order is identical. Note; Concatenated plain gzip was never affected, only BGZF |
|
Looking forward to having this PR merged. |
|
thanks, merged. |
No description provided.