Skip to content

Warn when zero-coverage bins will bias segmentation - #1183

Merged
etal merged 3 commits into
masterfrom
etal/fix/floor-bins-segmentation
Oct 14, 2026
Merged

etal merged 3 commits into
masterfrom
etal/fix/floor-bins-segmentation

Conversation

@etal

@etal etal commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Problem

Bins with no mapped reads carry a placeholder log2 value near -20 (params.NULL_LOG2_COVERAGE). --drop-low-coverage removes them, but it is opt-in, so by default the placeholders reach the segmenters as though they were measurements. After fix, these bins sit at roughly -20 minus log2(median depth) minus the reference log2, so in a .cnr they fall anywhere below -15 rather than at exactly -20.

On test/formats/wgs-chr17.cnr (165,626 bins of median width 267 bp), 12.9% of bins are at the floor. The fraction matches Poisson zero-coverage at about 2 reads per bin, not biology. Segmenting at threshold=1e-6, as batch -m wgs does:

segments median segment log2 Mb with log2 < -1
CBS, default 44 -2.505 63.4
CBS, --drop-low-coverage 42 +0.501 0.0
haar, default 30,009 -1.158 16.0
haar, --drop-low-coverage 51 +0.111 0.0

chr17 is copy-neutral in this sample. By default CBS reports 78% of the chromosome as copy loss, and its segment count gives no sign of trouble. Haar fragments instead: its FDR threshold is scaled by a median-based noise estimate, which is blind to the contamination, while the windowed means it thresholds are not.

Why this warns instead of dropping

Dropping zero-coverage bins by default was the original behavior and was made opt-in in 2015 because it is harmful for germline samples. #683 documents the clinical case: confirmed CDKN2A homozygous deletions in high-purity tumor samples had zero reads across the gene, and --drop-low-coverage erased them. Injecting a 40-bin zero-depth deletion into test/formats/regression/p2-9_2.cnr reproduces this: with the bins dropped, CBS reports the region as copy-neutral inside a 119 Mb segment, and haar omits it.

Per bin, Poisson dropout and a genuine deletion are indistinguishable: both have zero depth and a placeholder log2. Other options were measured and rejected:

  • Drop only bins at exactly -20, or only depth == 0. After fix no bin is at exactly -20, and wgs-chr17.cnr has no depth column, so neither rule ever fires on the input that exhibits the problem. Where they do fire, the bins they select are exactly the deletion bins.
  • Down-weight floor bins in fix. This repairs CBS at a weight multiplier of 0.01, but haar stays fragmented (more than 8,700 segments) until the weight is exactly 0. Zero-weight bins are already dropped, so this reduces to always-drop and also changes .cnr output.
  • A softer floor value (Use "safe_log2" instead of NULL_LOG2_COVERAGE #300). Clipping the floor to -5 still leaves 22.5 Mb of false loss. The floor would have to be -3 or higher.
  • Drop floor runs shorter than 10 bins when at least 1% of bins are at the floor. This reproduces the clean segmentation on wgs-chr17.cnr, keeps injected deletions of 20 or more bins, and changes no committed fixture. It also silently loses homozygous deletions spanning fewer than 10 targets in a degraded panel, which is the size of CDKN2A, so it is not adopted.

Change

  • do_segmentation logs one warning per sample when skip_low is off and at least 1% of finite autosomal bins are flagged as low coverage. The warning gives the count, the fraction, and the expected downward shift in segment means, computed as fraction x (mean of measured bins - mean of floor bins). It recommends recomputing coverage with larger bins (autobin), and names --drop-low-coverage together with its cost. On wgs-chr17.cnr the estimate is 2.53 log2 units, against a measured CBS median shift of 2.505.
  • The warning runs before the per-arm split, so it is emitted once per sample, including under --processes.
  • Sex chromosomes are excluded from the count, because a female sample's chrY is legitimately at the floor (22 of 45 chrY bins in p2-9_2.cnr). Over autosomes, the targeted fixtures top out at 0.28% (amplicon.cnr).
  • CopyNumArray.low_coverage_mask() is extracted from drop_low_coverage(), so the warning and the filter it recommends use one predicate.
  • GenomicArray.autosome_filter() is extracted from autosomes(). It returns an all-False mask where autosomes() would fall back to the whole array, so an input with no recognizable autosomes is not assessed rather than having its chrY counted. autosomes() behavior is unchanged.
  • The 1% threshold: injecting floor bins at 1% into the clean wgs-chr17 profile shifts CBS segment means by about 0.2 log2 and multiplies the haar segment count about 80-fold.
  • The documentation for segment, tumor, germline and sex analysis now explains the trade-off. It also corrects the claim that even cell lines always contain enough normal cells for the -15 rule to hold.

Genome build names

Two further commits concern the names accepted by --diploid-parx-genome.

The first corrects diploid_parx_genome docstring examples that named hg19, hg38 and mm10. At the time none of these was a registered build name, and each raised KeyError.

The second accepts the UCSC names hg19 and hg38 as aliases of grch37 and grch38, so a user whose files follow UCSC naming can type the name they would naturally use. PAR detection takes the X and Y chromosome names from the data and compares only coordinates, so chrX versus X naming does not matter. The alias is valid because the coordinates are identical: the PAR coordinates in skgenome.genomebuild match the NCBI assembly region reports for GRCh37.p13 and GRCh38 (unchanged through GRCh38.p14), which list hg19 and hg38 as synonyms, and the PAR sequences downloaded from UCSC and Ensembl are byte-identical. cnvkit.py call --diploid-parx-genome output is byte-identical for grch38, hg38 and HG38.

  • GenomeBuild gains an aliases field. Lookup goes through a table derived from it, separate from REGISTERED_BUILDS, so the historical keys of cnvlib.params.PSEUDO_AUTSOMAL_REGIONS stay grch37 and grch38.
  • The error for an unknown build lists the aliases. hg18 (NCBI36, with different PAR coordinates) still raises.
  • doc/sex.rst lists the accepted names and notes that CNVkit cannot detect a build that does not match the data. PAR1 differs between the builds only near its boundaries, but the two PAR2 regions do not overlap, so naming the wrong build misses the real PAR2 and treats a stretch of X-specific sequence as autosomal.

Not included: mouse assemblies (PAR coordinates still have to be sourced), and validating the build name during argument parsing; an unknown name still fails later with a KeyError.

Numerical output

Unchanged. A baseline comparison of CBS, haar and none segmentation, with and without --drop-low-coverage, on wgs-chr17.cnr, amplicon.cnr and regression/p2-9_2.cnr is byte-identical. The same holds for drop_low_coverage() output. The full test suite, including the slow frozen-pipeline regression test, passes.

Tests

  • FloorBinWarningTests:
    • warns once on wgs-chr17.cnr, with the shift sentence;
    • stays silent with skip_low and on the targeted fixtures;
    • ignores sex-chromosome floor bins, including on input with no autosomes;
    • ignores non-finite bins;
    • omits the shift sentence when the shift would print as 0.00 or when every bin is at the floor.
  • HaarWgsNoiseFloorTests runs the existing haar noise-floor assertions on wgs-chr17.cnr after drop_low_coverage(). Both assertions hold there, which confirms that haar's over-segmentation on the raw array comes from the floor bins rather than from WGS bin density. On the raw array both fail: 32,305 segments at q=1e-4 against a limit of 11,042.

Known limitations

The warning informs but does not change results. A pipeline that does not surface stderr will still produce biased segments for heavily affected samples. If #300 replaces the placeholder with a smoothed floor, the threshold and the shift estimate should be measured again.

etal added 2 commits October 13, 2026 16:04
Bins with no reads carry a placeholder log2 value near -20. Unless
--drop-low-coverage is given they are segmented as measurements, and at
WGS bin density with Poisson dropout this silently reports copy-neutral
regions as megabase-scale losses (CBS) or fragments the profile (haar).
Dropping them by default is not an option, because in germline and
high-purity tumor samples a zero-read run is a genuine homozygous
deletion.

do_segmentation now logs one warning per sample when at least 1% of
autosomal bins are at the floor, estimating the resulting downward shift
in segment means and recommending larger bins. Sex chromosomes are
excluded so a female chrY does not trigger it. Two predicates are
extracted so the warning cannot diverge from the code it describes:
CopyNumArray.low_coverage_mask from drop_low_coverage, and
GenomicArray.autosome_filter from autosomes, without the latter's
fall-back to the whole array. Output is unchanged.

Adds segmentation tests on test/formats/wgs-chr17.cnr and runs the haar
noise-floor assertions at WGS density. Documents the trade-off in the
tumor, germline, sex and segment documentation.
The diploid_parx_genome docstrings and examples named 'hg19', 'hg38'
and 'mm10', none of which skgenome.genomebuild registers; passing any
of them raises KeyError. Name 'grch37' and 'grch38' instead, as the CLI
help and documentation already do.
@codecov

codecov Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.76%. Comparing base (d4906a0) to head (f145ae9).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1183      +/-   ##
==========================================
+ Coverage   76.69%   76.76%   +0.07%     
==========================================
  Files          75       75              
  Lines        8465     8491      +26     
  Branches     1499     1503       +4     
==========================================
+ Hits         6492     6518      +26     
  Misses       1543     1543              
  Partials      430      430              
Flag Coverage Δ
unittests 76.76% <100.00%> (+0.07%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

--diploid-parx-genome accepted only grch37 and grch38, so a user whose
data and annotation files are named for UCSC hg38 got a KeyError for the
name they would naturally type. UCSC hg19 and hg38 share chromosome X
and Y coordinates with GRCh37 and GRCh38, so the same PAR coordinates
apply.

GenomeBuild gains an aliases field, resolved through a lookup table
separate from REGISTERED_BUILDS so that the historical
cnvlib.params.PSEUDO_AUTSOMAL_REGIONS keys are unchanged. The error for
an unknown build now lists aliases too. The documentation states the
accepted names, that chromosome naming style does not matter, and that
a build which does not match the data cannot be detected.
@etal
etal merged commit b72dca1 into master Oct 14, 2026
15 checks passed
@etal
etal deleted the etal/fix/floor-bins-segmentation branch October 14, 2026 23:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant