Skip to content

Fix BGZF truncation at embedded EOF markers - #712

Merged
sfchen merged 1 commit into
OpenGene:masterfrom
KimYannn:fix/bgzf-embedded-eof
Sep 1, 2026
Merged

sfchen merged 1 commit into
OpenGene:masterfrom
KimYannn:fix/bgzf-embedded-eof

Conversation

@KimYannn

Copy link
Copy Markdown
Member

No description provided.

@aalnawfal1992

Copy link
Copy Markdown

Tested this patch against #711 on real data It fixes the problem and reproduces pre regression behaviour exactly. Built master and this PR from source (Ubuntu 24.04, isa-l / libdeflate / libhwy from apt).

1. Real data

A WES sample straight off bcl convert, not manually concatenated The R1 file contains two BGZF streams and 24,910,823 read pairs (verified with zcat | wc -l).

build read pairs seen
master 24,854,834 stops at the end of stream 1
this PR 24,910,823 reads the whole file

The master figure is exactly the number of read pairs in the first BGZF stream, confirming it halts at the first EOF block.

2. Minimal reproducer

Two samll files of 200 read pairs each, compressed separately and concatenated, so the correct answer is 400:

bgzip -c a_R1.fastq > a.bgz          # 200 pairs
bgzip -c b_R1.fastq > b.bgz          # 200 pairs
cat a.bgz b.bgz > merged_R1.fastq.gz # 400 pairs
build read pairs seen expected
master 200 400
this PR 400 400

A single stream bgzip file (200 pairs, no embedded EOF marker) reads correctly as 200 on both builds, so the patch does not regress the normal case and does not hang. That was worth checking, since break -> continue means the trailing EOF marker no longer terminates the loop and markDone() is not called on that path.

3. Equivalet with v1.1.0

v1.1.0 is the last release before the regression. Same file, same flags. These counts are R1+R2 combined, as reported in the JSON:

metric v1.1.0 this PR
reads before filtering 49,821,646 49,821,646
reads after filtering 49,446,918 49,446,918
low_quality 366,304 366,304
too_many_N 4,422 4,422
too_short 3,996 3,996
adapter trimmed reads 125,966 125,966
adapter trimmed bases 6,376,503 6,376,503

Identical on every metric, and the final read in the output file is the same in both.

Read order and pairing are also preserved: the first read of stream 2 appears in the output immediately after the last read of stream 1, and R1/R2 name order is identical.

Note; Concatenated plain gzip was never affected, only BGZF

@KimYannn
KimYannn marked this pull request as ready for review August 24, 2026 02:59
@omegahh

omegahh commented Aug 31, 2026

Copy link
Copy Markdown

Looking forward to having this PR merged.

@sfchen
sfchen merged commit 6563faa into OpenGene:master Sep 1, 2026
2 checks passed
@sfchen

sfchen commented Sep 1, 2026

Copy link
Copy Markdown
Member

thanks, merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants