Reject truncated BGZF blocks instead of treating them as EOF - #717
Open
thierrygosselin wants to merge 2 commits into
Open
thierrygosselin wants to merge 2 commits into
thierrygosselin wants to merge 2 commits into
Conversation
Distinguish normal end-of-file from incomplete BGZF headers and bodies. Report input read errors explicitly instead of treating them as EOF. Validate block sizes before subtracting the header length, preventing unsigned underflow for undersized blocks. Preserve support for concatenated BGZF streams and complete streams without a terminal EOF marker.
Test incomplete headers, payloads, trailers, and undersized blocks using synthetic BGZF fixtures with one and four processing threads. Verify that valid streams, concatenated streams, and complete streams without a terminal EOF marker preserve the input reads. All 14 cases pass with the fix. The unchanged BGZF reader incorrectly returns success for all eight malformed-input cases.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The BGZF reader currently treats incomplete block headers and bodies as normal end-of-file. Consequently, truncated input can produce incomplete output while fastp exits successfully.
This PR distinguishes these failures from clean EOF.
Changes
This is separate from the concatenated-stream fix in #712 and does not depend on #716.
Tests
Added synthetic regression tests covering:
Each case runs with
-w 1and-w 4.On macOS ARM64, all 14 cases pass with the patch. The unchanged BGZF reader incorrectly exits successfully for all eight malformed-input cases.
Scope
This patch addresses block framing and read failures. It does not implement comprehensive gzip-header, CRC, or decompression-error validation. Linux execution and injected input-stream I/O errors have not been tested.