Skip to content

ci: add trustworthy coverage ratcheting toward 90% #10867

Description

@ReubenBond

Goal

Turn the existing coverage report into a trustworthy non-regression ratchet which can advance Orleans from 78.54% to above 90% without encouraging exclusions or denominator manipulation.

Coverage report: https://github.com/dotnet/orleans/runs/98679059570

Current behavior

  • The complete PR CI matrix merges coverage from providers, CodeGen, .NET 8/10, Linux, Windows, and macOS.
  • The repository summarizer de-duplicates physical source lines and reports 78.54% (99,056/126,115).
  • Raw Cobertura aggregation reports lower values because linked/shared source appears in multiple assemblies.
  • Branch coverage is not published canonically.
  • The coverage check currently reports success unconditionally and has no trusted main-branch baseline or changed-line gate.
  • Only loaded maintained source appears in the report, so compiled-but-unloaded source needs an explicit universe check.

Work

  • Produce trusted same-matrix main and PR coverage artifacts with SDK/tool/config fingerprints.
  • Define canonical line and branch de-duplication for linked/shared source.
  • Verify the exact expected matrix artifact set before merging.
  • Build a compiled-source manifest so missing assemblies/files cannot silently improve coverage.
  • Add report-only overall, per-package, changed-line, and changed-branch comparisons.
  • Calibrate normal variance, then fail regressions.
  • Allow baselines to move upward automatically; require reviewed scope changes for downward movement.
  • Fail unexplained changes to source/module filters, compile inclusion, exclusions, or coverage attributes.
  • Add tests for missing artifacts, duplicate lines, source-universe drift, exclusions, and check conclusions.
  • Raise the repository floor incrementally as coverage work lands, ending above 90%.

Acceptance criteria

  • Coverage comparison is reproducible between main and pull requests.
  • The check fails verified coverage regressions after a report-only calibration period.
  • Canonical overall line, branch, package, and changed-code metrics are published.
  • Missing test partitions or maintained source fail the coverage job instead of shrinking the denominator.
  • The ratchet cannot be lowered without an explicit reviewed change and rationale.
  • Final enforced repository line coverage exceeds 90%.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions