Repository navigation
Document CRAM input and test CRAM/BAM coverage equivalence - #1180
Merged
Merged
Conversation
Add a "CRAM input" section to the coverage documentation: CRAM files need the reference FASTA given with -f/--fasta. Without it, and with REF_PATH unset, HTSlib 1.21 and earlier download every reference sequence from the EBI server (the multi-hour slowdown in #759), while later versions stop with a decode error. The section also covers REF_PATH/REF_CACHE and the errors a mismatched FASTA produces. A note beside the duplicate-marking guidance states that base quality recalibration does not change coverage (#1105). Back those claims with tests: the marked-duplicate test now runs on CRAM as well as BAM, with reads carrying Phred 2 base qualities and a CRAM that decodes only through the FASTA passed to CNVkit; the CRAM/BAM comparison now requires exact equality in both the depth and --count paths rather than agreement to one decimal place.
These commands read CRAM files when given -f/--fasta, as doc/pipeline.rst describes, but their command summaries, input arguments, and the batch -n/--normal option named only BAM and bedGraph input.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1180 +/- ##
=======================================
Coverage 76.39% 76.39%
=======================================
Files 75 75
Lines 8418 8418
Branches 1487 1487
=======================================
Hits 6431 6431
Misses 1556 1556
Partials 431 431
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents CRAM input, which the code has supported since 6ceef3b but the user documentation never mentioned, and answers the BAM/CRAM/BQSR questions in #1105 with tests behind each claim.
Fixes #1105
Documentation
A new "CRAM input" subsection under
coverageindoc/pipeline.rstcovers:-f/--fastais needed to read CRAM. It is accepted bybatch(including together with-r/--reference),coverageandautobin, and is forwarded to every pysam andsamtools bedcovcall. CNVkit builds a missing.craiitself; CRAM indexing does not require the reference.-fFASTA (by name), thenREF_CACHEandREF_PATH(by the@SQM5checksum), then the local file in the@SQURtag (by name). WhenREF_PATHis unset, HTSlib 1.21 and earlier (pysam 0.23 and earlier) fall back to the EBI reference server and download every sequence into~/.cache/hts-ref. This is the multi-hour slowdown reported in CNVkit coverage takes too long on CRAM file #759. HTSlib 1.22 and later (pysam 0.24 and later) removed that default and stop withUnable to fetch reference/Failure to decode slice. WhenREF_PATHis set but lacks the sequence, every version fails the same way. The[W::cram_populate_ref] Creating reference cache directorywarning appears only when HTSlib creates the default cache directory, so its absence does not rule out downloads.seq_cache_populate.plprints the template to assign toREF_CACHE. The documentation recommends assigning the same template toREF_PATH, because HTSlib 1.21 selects the EBI default wheneverREF_PATHis empty, regardless ofREF_CACHE.MD5 checksum reference mismatch, and CNVkit stops with an error rather than computing coverage from misread alignments. A@SQ length mismatchwarning, as in Could not retrieve index file for CRAM files #779, indicates a different assembly. It can appear while decoding still succeeds, when the sequences agree over the region read, so it should be read as a sign of the wrong FASTA.--countpaths. Duplicate flags are honored in CRAM exactly as in BAM. A new paragraph beside the existing duplicate-marking note states that CNVkit does not read base qualities. GATK ApplyBQSR rewrites only those scores, and apart from GATK's default filter for malformed reads (WellformedReadFilter, the only default onReadWalker), it passes every read through unchanged. MarkDuplicates output and recalibrated output therefore yield the same coverage. For the same reason, CRAM's lossy quality-score compression has no effect on coverage.The HTSlib behavior above was checked against the
cram_populate_refsource at the 1.21 and 1.22 tags. It was also observed directly with pysam 0.23.3 (HTSlib 1.21) and pysam 0.24.1 (HTSlib 1.24).Tests
test_coverage_skips_marked_duplicatesnow runs on CRAM as well as BAM, and half of the surviving reads carry Phred 2 at every base. The CRAM is written against a FASTA that is then renamed, so itsURtag names a missing file. Reads run withREF_PATHandREF_CACHEpointed at a nonexistent directory, so decoding depends only on the FASTA passed to CNVkit, and no request ever goes to the network. This isolation is necessary: the synthetic poly-A sequence is served by the EBI MD5 server, and without it, a version ofbedcovthat dropped the FASTA still passed.test_coverage_cramnow requires exact equality, in both the depth and--countpaths. It previously usedassert_array_almost_equal(..., decimal=1)on the depth path only. The measured difference is 0.0. The fixture has two duplicate-flagged reads inside the chrM target; counting them would raise that target's depth from 82.5 to 84.5. The row count is now asserted absolutely, so an empty result cannot pass. The fixture CRAM'sURtags name an absolute path on the machine that wrote it, so where that path exists, this test cannot show that the FASTA argument is used. The docstring notes this, and points to the test above, which does show it.Each test was checked against a deliberately broken implementation:
bedcovcounts duplicates for CRAM onlytest_coverage_cram (by_count=False)and thefmt='cram'subtest fail; BAM subtests passbedcovno longer receives the FASTAfmt='cram'subtest fails; BAM subtest passesBoth tests pass under pysam 0.23.3 (the minimum pin) and pysam 0.24.1.
CLI help
The
batch,coverageandautobincommand summaries and input arguments, andbatch -n/--normal, now name CRAM. Previously they named only BAM and bedGraph.Not changed
OSError: truncated file, which suggests file corruption rather than a missing or mismatched-f. This is documented here through the HTSlib messages that precede it; a clearer error message is left for a separate change.autobinpositional help has claimed bedGraph input since 83c16bb, butautobincallsensure_bam_indexunconditionally and fails on.bed.gz. This is left for a separate change, because it requires choosing between rejecting bedGraph input and supporting it.