Repository navigation
Warn when zero-coverage bins will bias segmentation - #1183
Merged
Merged
Conversation
Bins with no reads carry a placeholder log2 value near -20. Unless --drop-low-coverage is given they are segmented as measurements, and at WGS bin density with Poisson dropout this silently reports copy-neutral regions as megabase-scale losses (CBS) or fragments the profile (haar). Dropping them by default is not an option, because in germline and high-purity tumor samples a zero-read run is a genuine homozygous deletion. do_segmentation now logs one warning per sample when at least 1% of autosomal bins are at the floor, estimating the resulting downward shift in segment means and recommending larger bins. Sex chromosomes are excluded so a female chrY does not trigger it. Two predicates are extracted so the warning cannot diverge from the code it describes: CopyNumArray.low_coverage_mask from drop_low_coverage, and GenomicArray.autosome_filter from autosomes, without the latter's fall-back to the whole array. Output is unchanged. Adds segmentation tests on test/formats/wgs-chr17.cnr and runs the haar noise-floor assertions at WGS density. Documents the trade-off in the tumor, germline, sex and segment documentation.
The diploid_parx_genome docstrings and examples named 'hg19', 'hg38' and 'mm10', none of which skgenome.genomebuild registers; passing any of them raises KeyError. Name 'grch37' and 'grch38' instead, as the CLI help and documentation already do.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1183 +/- ##
==========================================
+ Coverage 76.69% 76.76% +0.07%
==========================================
Files 75 75
Lines 8465 8491 +26
Branches 1499 1503 +4
==========================================
+ Hits 6492 6518 +26
Misses 1543 1543
Partials 430 430
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
--diploid-parx-genome accepted only grch37 and grch38, so a user whose data and annotation files are named for UCSC hg38 got a KeyError for the name they would naturally type. UCSC hg19 and hg38 share chromosome X and Y coordinates with GRCh37 and GRCh38, so the same PAR coordinates apply. GenomeBuild gains an aliases field, resolved through a lookup table separate from REGISTERED_BUILDS so that the historical cnvlib.params.PSEUDO_AUTSOMAL_REGIONS keys are unchanged. The error for an unknown build now lists aliases too. The documentation states the accepted names, that chromosome naming style does not matter, and that a build which does not match the data cannot be detected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Bins with no mapped reads carry a placeholder log2 value near -20 (
params.NULL_LOG2_COVERAGE).--drop-low-coverageremoves them, but it is opt-in, so by default the placeholders reach the segmenters as though they were measurements. Afterfix, these bins sit at roughly -20 minus log2(median depth) minus the reference log2, so in a.cnrthey fall anywhere below -15 rather than at exactly -20.On
test/formats/wgs-chr17.cnr(165,626 bins of median width 267 bp), 12.9% of bins are at the floor. The fraction matches Poisson zero-coverage at about 2 reads per bin, not biology. Segmenting atthreshold=1e-6, asbatch -m wgsdoes:--drop-low-coverage--drop-low-coveragechr17 is copy-neutral in this sample. By default CBS reports 78% of the chromosome as copy loss, and its segment count gives no sign of trouble. Haar fragments instead: its FDR threshold is scaled by a median-based noise estimate, which is blind to the contamination, while the windowed means it thresholds are not.
Why this warns instead of dropping
Dropping zero-coverage bins by default was the original behavior and was made opt-in in 2015 because it is harmful for germline samples. #683 documents the clinical case: confirmed CDKN2A homozygous deletions in high-purity tumor samples had zero reads across the gene, and
--drop-low-coverageerased them. Injecting a 40-bin zero-depth deletion intotest/formats/regression/p2-9_2.cnrreproduces this: with the bins dropped, CBS reports the region as copy-neutral inside a 119 Mb segment, and haar omits it.Per bin, Poisson dropout and a genuine deletion are indistinguishable: both have zero depth and a placeholder log2. Other options were measured and rejected:
fixno bin is at exactly -20, andwgs-chr17.cnrhas no depth column, so neither rule ever fires on the input that exhibits the problem. Where they do fire, the bins they select are exactly the deletion bins.fix. This repairs CBS at a weight multiplier of 0.01, but haar stays fragmented (more than 8,700 segments) until the weight is exactly 0. Zero-weight bins are already dropped, so this reduces to always-drop and also changes.cnroutput.wgs-chr17.cnr, keeps injected deletions of 20 or more bins, and changes no committed fixture. It also silently loses homozygous deletions spanning fewer than 10 targets in a degraded panel, which is the size of CDKN2A, so it is not adopted.Change
do_segmentationlogs one warning per sample whenskip_lowis off and at least 1% of finite autosomal bins are flagged as low coverage. The warning gives the count, the fraction, and the expected downward shift in segment means, computed as fraction x (mean of measured bins - mean of floor bins). It recommends recomputing coverage with larger bins (autobin), and names--drop-low-coveragetogether with its cost. Onwgs-chr17.cnrthe estimate is 2.53 log2 units, against a measured CBS median shift of 2.505.--processes.p2-9_2.cnr). Over autosomes, the targeted fixtures top out at 0.28% (amplicon.cnr).CopyNumArray.low_coverage_mask()is extracted fromdrop_low_coverage(), so the warning and the filter it recommends use one predicate.GenomicArray.autosome_filter()is extracted fromautosomes(). It returns an all-False mask whereautosomes()would fall back to the whole array, so an input with no recognizable autosomes is not assessed rather than having its chrY counted.autosomes()behavior is unchanged.wgs-chr17profile shifts CBS segment means by about 0.2 log2 and multiplies the haar segment count about 80-fold.segment, tumor, germline and sex analysis now explains the trade-off. It also corrects the claim that even cell lines always contain enough normal cells for the -15 rule to hold.Genome build names
Two further commits concern the names accepted by
--diploid-parx-genome.The first corrects
diploid_parx_genomedocstring examples that namedhg19,hg38andmm10. At the time none of these was a registered build name, and each raisedKeyError.The second accepts the UCSC names
hg19andhg38as aliases ofgrch37andgrch38, so a user whose files follow UCSC naming can type the name they would naturally use. PAR detection takes the X and Y chromosome names from the data and compares only coordinates, sochrXversusXnaming does not matter. The alias is valid because the coordinates are identical: the PAR coordinates inskgenome.genomebuildmatch the NCBI assembly region reports for GRCh37.p13 and GRCh38 (unchanged through GRCh38.p14), which list hg19 and hg38 as synonyms, and the PAR sequences downloaded from UCSC and Ensembl are byte-identical.cnvkit.py call --diploid-parx-genomeoutput is byte-identical forgrch38,hg38andHG38.GenomeBuildgains analiasesfield. Lookup goes through a table derived from it, separate fromREGISTERED_BUILDS, so the historical keys ofcnvlib.params.PSEUDO_AUTSOMAL_REGIONSstaygrch37andgrch38.hg18(NCBI36, with different PAR coordinates) still raises.doc/sex.rstlists the accepted names and notes that CNVkit cannot detect a build that does not match the data. PAR1 differs between the builds only near its boundaries, but the two PAR2 regions do not overlap, so naming the wrong build misses the real PAR2 and treats a stretch of X-specific sequence as autosomal.Not included: mouse assemblies (PAR coordinates still have to be sourced), and validating the build name during argument parsing; an unknown name still fails later with a
KeyError.Numerical output
Unchanged. A baseline comparison of CBS, haar and
nonesegmentation, with and without--drop-low-coverage, onwgs-chr17.cnr,amplicon.cnrandregression/p2-9_2.cnris byte-identical. The same holds fordrop_low_coverage()output. The full test suite, including the slow frozen-pipeline regression test, passes.Tests
FloorBinWarningTests:wgs-chr17.cnr, with the shift sentence;skip_lowand on the targeted fixtures;HaarWgsNoiseFloorTestsruns the existing haar noise-floor assertions onwgs-chr17.cnrafterdrop_low_coverage(). Both assertions hold there, which confirms that haar's over-segmentation on the raw array comes from the floor bins rather than from WGS bin density. On the raw array both fail: 32,305 segments at q=1e-4 against a limit of 11,042.Known limitations
The warning informs but does not change results. A pipeline that does not surface stderr will still produce biased segments for heavily affected samples. If #300 replaces the placeholder with a smoothed floor, the threshold and the shift estimate should be measured again.