Repository navigation
Estimate autobin depths from bedGraph input - #1182
Merged
Merged
Conversation
autobin's help has listed bedGraph (.bed.gz) input since bedGraph support was added to coverage (#984), but do_autobin always opened the sample as an alignment file and failed with "file does not contain alignment data". batch failed the same way whenever it sized bins automatically from a bedGraph normal. A bedGraph records neither read counts, read length, nor chromosome lengths, so the index-based estimates for wgs depth and hybrid antitarget depth are unavailable. For a bedGraph, every depth is now measured in the same seeded random sample of regions that the target depth already uses: the targets, the accessible regions (wgs), or the accessible regions on targeted chromosomes minus the targets (hybrid antitargets). The hybrid target depth of a bedGraph covers all targets; the BAM path restricts it to the accessible chromosomes only because it subtracts target reads from each chromosome's total. Each sampled region is trimmed to a centered 10 kb window, so that sampling megabase-scale accessible regions reads about 1 Mb of per-base depth. wgs on a bedGraph requires accessible regions. hybrid without them, or with none on a targeted chromosome, estimates only the target depth and logs a warning. In batch, a hybrid bedGraph normal without --access takes its accessible regions from --fasta, as batch already does for wgs. BAM and CRAM estimates are unchanged. Two error cases change for every input. A region set with no positive-width region, as when the targets and accessible regions share no chromosome names, now yields an undefined depth and reaches the existing chromosome-name error (#421) instead of an IndexError. An empty set of accessible regions passed to do_autobin is treated as none given, as the command line already does, so hybrid on a BAM falls back to the BAM index instead of failing. The pipeline documentation gains a section on bedGraph input.
bedGraph input accepts a .tbi or a .csi tabix index, but the file was opened with pysam.TabixFile's default index lookup, which finds only .tbi. A bedGraph indexed only with CSI, as mosdepth writes its per-base output, therefore passed the index check and then failed to open. The check, renamed find_bedgraph_index, now returns the index it found, and that path is passed to TabixFile explicitly.
sample_midsize_regions keeps the regions between the 25th and 75th percentile of size. With exactly two regions, of different size, both percentiles fall strictly between the two sizes and no region is kept, so autobin reported the chromosome-name error (#421) for input whose names matched. When the percentile range keeps no region, all regions of nonzero size are now eligible. Any region set that previously yielded a depth selects the same regions as before; BAM, CRAM, and bedGraph input with such a region set, which raised that error, now yields a depth.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1182 +/- ##
==========================================
+ Coverage 76.39% 76.69% +0.29%
==========================================
Files 75 75
Lines 8418 8465 +47
Branches 1487 1499 +12
==========================================
+ Hits 6431 6492 +61
+ Misses 1556 1543 -13
+ Partials 431 430 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
autobin, andbatchwhenever it sizes bins automatically, failed on bedGraph (.bed.gz) input with "file does not contain alignment data", although the help of all three commands lists bedGraph since it became acoverageinput (#984). This adds bedGraph support toautobinfor all three protocols, fixes the opening of CSI-indexed bedGraph files, and fixes a sampler edge case found along the way.Fixes #1181
How depths are estimated from a bedGraph
For a BAM or CRAM file,
autobinreads per-chromosome read counts from the index and estimates the read length; the wgs depth and the hybrid off-target depth come from those. A bedGraph records neither, nor chromosome lengths. Every bedGraph depth is therefore measured in the same seeded sample of regions (sample_midsize_regions) that the on-target depth already uses, reading the bedGraph throughcoverage.bedgraph_to_basecount:ValueErrorotherwise)Each sampled region is trimmed to a centered 10 kb window (
BEDGRAPH_SAMPLE_WIDTH). Accessible and off-target regions reach megabases each: 100 midsize regions of the hg19 access file span about 49 Mb, andbedgraph_to_basecountparses lines in Python at roughly 0.2 to 1.3 million per second, so untrimmed sampling of a 30x per-base WGS bedGraph would take minutes. Trimmed, a sample reads about 1 Mb. Targets are almost always shorter than the window, so the on-target estimate is unaffected by it.The bedGraph hybrid on-target depth covers all targets. The BAM path restricts it to chromosomes shared with the accessible regions only because it subtracts each chromosome's target reads from that chromosome's total; for a bedGraph the restriction would discard data, and with accessible regions from
access(which omits chrM and alternative contigs) it would drop chrM targets or, for a mitochondrial panel, fail with the chromosome-name error.Without accessible regions, or with none off-target on a targeted chromosome, hybrid on a bedGraph estimates only the on-target bin size and logs a warning. In
batch, a hybrid bedGraph normal without--accesstakes its accessible regions from--fasta, asbatchalready does for wgs; with neither, the default off-target bin size applies.A chromosome absent from the bedGraph counts as zero depth, as it does in
coverage.Measurements
On the test sample (
na12878-chrM-Y-trunc),my-targets.bed, amplicon:samtools depth -J -G 0x704The fixture bedGraph counts the duplicate-flagged reads that the BAM path excludes; a bedGraph of the same reads gives the identical estimate, since both use the same seeded region sample. A synthetic bedGraph with known depths (300x targets, 30x elsewhere) yields exactly 300 and 30 for the on- and off-target depths, and 30 for wgs.
The off-target estimators differ in kind (BAM subtracts target reads from index totals; a bedGraph is measured directly), so their values are not expected to agree closely.
CSI-indexed bedGraph files
find_bedgraph_index(formerlyvalidate_bedgraph_index) accepted a.csiindex, but the file was then opened withpysam.TabixFile's default lookup, which finds only.tbi. mosdepth indexes its per-base output with CSI, so that output passed the check and failed to open incoverageas well. The located index is now passed toTabixFileexplicitly.Sampler edge case
sample_midsize_regionskeeps regions between the 25th and 75th percentile of size. With exactly two regions of different size, both percentiles fall strictly between the two sizes and nothing is kept, which surfaced as the chromosome-name error (#421). All regions of nonzero size are now eligible in that case. A randomized comparison over several thousand region sets confirmed that every set which previously kept regions keeps exactly the same ones.Behavior changes for BAM and CRAM
No BAM or CRAM depth or bin size changes for input that previously succeeded:
do_autobinoutput was compared exactly against master across seven region sets, and the frozen regression test passes. Three error cases now behave differently for every input type:IndexError.do_autobinis treated as none given, as the command line already did; BAM hybrid then uses the index instead of failing.Documentation
doc/pipeline.rstgains a "bedGraph input" section (tabix index requirement, ignored read-selection options, missing chromosomes, and the read filters to apply for coverage comparable to a BAM, including mosdepth's defaults), and theautobin,coverage, andbatchsections describe the bedGraph behavior. Theautobinhelp for the positional argument and-gis updated. Thecoveragemodule docstring, which named theantitargetcommand, is corrected.