read FASTQ records with a single copy instead of two - #1
Draft
nh13 wants to merge 1 commit into
Draft
Conversation
kseq copies every field twice: once from the decode buffer into a kstring_t, then again from the kstring_t into the mb_bseq1_t the mapping pipeline keeps. On short reads at high thread counts the read step is a large fraction of wall, and that second copy is pure overhead. fr_fastq slices each record directly out of its own refillable buffer and the caller copies each field once into mb_bseq1_t. The parser is not tied to a decompressor: it pulls bytes through a small callback, which bseq.c wires to gzread, so an in-process .gz file and a decompressed plaintext pipe on stdin both work unchanged. The record grammar mirrors kseq's so output is byte-identical; verified on two-file and interleaved paired-end, single-end, PAF, and the plaintext and gzip stdin pipes.
nh13
added
the
feature
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces kseq on the read path with a single-copy FASTA/FASTQ parser.
kseq copies every field twice — decode buffer into a
kstring_t, then thekstring_tinto themb_bseq1_tthe pipeline keeps (kseq2bseq). This addsfr_fastq, which slices each record out of its own refillable buffer so the caller copies each field once intomb_bseq1_t. The record grammar mirrors kseq's (leading junk skipped, name to first whitespace, comment the rest of the line with a single trailing\rdropped, multi-line seq/qual concatenated, CRLF normalised), so output is byte-identical. The parser pulls bytes through a one-function callback;bseq.cwires it togzread, so a.gzfile and a decompressed plaintext stdin pipe both work unchanged.Verification: a differential harness (kseq vs fr_fastq on identical bytes, with the refill capped to 1-3 byte reads to force a buffer underrun at every record boundary) over empty records, CRLF, multi-line seq/qual, leading junk, blank lines, missing trailing newline, qual shorter/longer than seq,
@/+as the first qual byte, and header-only EOF — all identical. End-to-end byte-identical SAM/PAF on two-file and interleaved paired-end, single-end,-y, and plaintext/gzip stdin pipes, plus a small--Kchunk-boundary stress.Performance is regime-dependent: the gain is the removed copy of every base, so it shows up only when the read stage is on the critical path. With a slow system zlib at high thread counts (read-bound) wall drops ~10% on a 96-thread GRCh38 run; with a fast inflate (zlib-ng) the read stage hides under mapping and the effect is ~0.
This is a fork branch for discussion — see the linked issue on lh3/minibwa. Note
kseq.his still used byl2bit.candfastmap.c, so as-is this addsfr_fastqalongside kseq rather than replacing it.