test(abc): the test module is ungated, so a minimal-feature build reports ~2,600 spurious failures

Summary

src/metrics/abc.rs's mod tests carries no feature gate, so 2,598 of
3,118 tests in it fail under cargo test -p big-code-analysis --no-default-features --features rust. Every per-language test in the module
is compiled regardless of whether its grammar is enabled.

Why it matters

.claude/rules/testing.md requires a #[cfg(any(feature = ...))] union gate
on any test whose fixture table is feature-dependent, so a build enabling
none of its rows drops the test rather than failing it. Individual modules
inside abc.rs now honour that — the newer ones added across 2026-09 carry
both the union gate and a non-vacuity guard. The enclosing module does
not, so the discipline is being applied test-by-test inside a container that
defeats it.

The practical effect is that a minimal-feature build cannot be used to verify
anything about ABC: the signal is buried under ~2,600 unrelated failures that
are bootstrap artifacts rather than regressions — the same class of
misleading-red as the missing-corpora case #1171 fixed.

A second trap in the same area

A workspace-wide cargo test --no-default-features --features X does not
isolate features at all: big-code-analysis-cli and big-code-analysis-web
depend on the library with default features, and cargo unifies them, so
everything is silently re-enabled. Three different feature sets reported an
identical 4 passed; 3428 filtered out before this was noticed — the
uniform-number tell from .claude/rules/testing.md.

Only -p big-code-analysis isolates. Any feature-gate verification run
without it proves nothing, and proves it while looking green. Worth a line in
.claude/rules/testing.md alongside the existing gate guidance.

Suggested shape

Gate mod tests on the union of every language feature its contents name, or
split the per-language sections into already-gated child modules and gate the
parent on their union. Verify against a subset enabling none of them — note
rust,typescript is not such a subset for any table with a TypeScript or TSX
row, so a single non-listed language (--features go) is the reproducer.

chore: polish items from the 2.1.0 delta review #1220 / PR fix: Objc block nargs, diagnostic-prefix lexer, and 2.1.0 delta-review polish #1221 — the same union-gate omission, twice in one batch.
test(features): 20 sweeps lack the cfg-union half and fail spuriously #1286 — the non-vacuity guard this pairs with.

RESOLVED

All three items below are done on fix/1472-gate-per-language-tests
(12 commits, not pushed). Detail in the closing comment. In short:

The 2,594 ungated tests — re-measured as 3,025 across src/,
big-code-analysis-ast/src/ and tests/; 3,235 markers now, all but
a handful derived.
The gate that derives the marker — utils/check-test-lang-gates.py,
wired into make pre-commit / make ci. It reproduces all 167 gates
written by hand in this tree, which is what licensed generating the
rest.
Subject.carries_tests — fixed, with the matcher on the
qualified module path.

The feature-matrix CI legs now run the suite rather than only
compiling it, which is what #1285 was waiting for.

One correction to the body below: the "30 of 3,482" figure does not
reproduce. 80 of 3,469 src/ tests carried a feature cfg of their
own.

Update — the CI half is closed; three items remain

PR #1476 closed the CI half of this issue. .github/workflows/ci.yml's
feature-matrix job now documents that it runs clippy and not the suite,
and a new utils/check-feature-gates.py runs in each leg asserting that a
union-gated test subject is absent under a feature set enabling none of
its rows. What remains:

1. The 2,594 ungated per-language lib tests (the original issue)

Unchanged. Measured on fix/batch-2026-09-14: --no-default-features --features rust fails 2,642 of 3,238 tests, 2,594 of them in the lib
target. Root cause is a contract violation, not a missing cfg in the
abstract:

panicked at big-code-analysis-ast/src/node.rs:84:
invariant: the caller checked LANG::is_enabled, or reached this through
AnyParser: LanguageDisabled(Python)

ParserTrait::new's own doc says the same and directs callers to check
LANG::is_enabled first. Only 30 of 3,482 #[test] fns in src/ do.
By module (halved for nextest's double-reporting): abc 525, loc 363,
cognitive 346, halstead 211, cyclomatic 207, npm 187, npa 180, wmc 167,
nargs 136, nexits 111, nom 77, tokens 36, mi 7.

Two approaches were considered and rejected during #1476; record them so
they are not re-proposed:

Gate each test. Semantically right, ~2,600 attributes. The hazard is
that a cross-language parity test needs all(feature = …) where a
per-language test needs one feature, and a mechanical rewrite that gets
that backwards produces a test that silently stops running.
Make the helpers self-skip on is_enabled(). Only 3 of ~13 helpers
take a runtime LANG; the rest take a type parameter, and ParserTrait
exposes no language accessor, so it would need an additive change to a
published trait. The disqualifier is independent of that: it would turn
~2,600 tests into silent no-ops under partial features, which
.claude/rules/testing.md treats as worse than no test.

2. `check-feature-gates.py` cannot see a missing gate

A scanner of declared markers reports a clean tree when the marker is
deleted, because deleting it deletes the subject. That is the
hidden_literal_supertypes shape (fixed in 0e15c49c on #1476), and only
its compile half — unused imports at -D warnings — was caught, by clippy.

The gate catches the defeated direction (a union present but weakened,
e.g. a non-feature disjunct), verified both ways during #1476. Catching the
missing direction needs a different gate: derive what the marker should
be from the union of the item's inner #[cfg(feature = …)] rows and compare
it against what is declared. That is the natural companion to item 1, since
the same derivation would drive a mechanical gating pass.

3. A latent false pass in `Subject.carries_tests`

utils/check-feature-gates.py:265 classifies a mod as test-bearing only
when it carries its own bare cfg(test) predicate:

return (self.kind == "mod" and self.is_cfg_test) or (
    self.kind == "fn" and self.is_test_fn
)

A union-gated mod nested inside an already-#[cfg(test)] parent has no
test in its own attribute, so it is classified compile-time only and
never compared against the nextest listing — a false pass in exactly the
#1220 shape the gate exists to catch. No current subject hits it (all 34 are
top-level #[cfg(test)] mod), so it is latent.

The naive repair is wrong: subject_matcher anchors on the bare name
((?:^|::)name(?:::|$)), so admitting any mod makes a subject named
tests match every ::tests:: path in the crate and fail every disjoint
leg. Narrowing it needs the subject's full module path rather than its bare
name.

Also noticed, adjacent

src/test_support.rs:321 — assert_csharp_fixture_spells carries no
#[cfg(feature = …)] while its sibling assert_perl_fixture_spells (:329)
carries #[cfg(feature = "perl")]. Both are correct today: the C# callers
are ungated and the Perl ones are gated. It becomes dead_code at
-D warnings the moment a C# test module is gated the way
perl_statement_modifier_parity is — i.e. the first time item 1 is worked
on. Adding the gate before then would break the build.

docs(ci): say why feature-matrix runs no tests

ci(feature-matrix): gate union-gated test reachability

CI half closed in PR #1476; two new items folded in

The body is updated. Summary of what changed and what is left.

Closed. .github/workflows/ci.yml now says why feature-matrix runs
clippy and not the suite, and utils/check-feature-gates.py runs in each
leg asserting a union-gated subject is absent under a feature set enabling
none of its rows. Verified both directions against the real tree —
perturbing a union with a non-feature disjunct gives:

$ cargo clippy --all-targets --no-default-features --features go … -D warnings
CLIPPY_EXIT=0

$ cargo nextest run --no-default-features --features go -E 'test(/the_1184_constructs/)'
panicked: "at least one language feature must be enabled for this test to mean anything"
1 test run: 0 passed, 1 failed

Clippy green, test broken — the gap the gate now closes.

Two items folded in from the whole-branch review of #1476, both
verified rather than taken on report:

Subject.carries_tests has a latent false pass (item 3 in the body).
A union-gated mod nested inside an already-#[cfg(test)] parent is
classified compile-time only and never checked. Latent — all 34 current
subjects are top-level. The naive repair breaks the matcher; details in
the body.
assert_csharp_fixture_spells lacks the feature gate its Perl sibling
has. Harmless today, becomes dead_code at -D warnings the first
time anyone gates a C# test module — i.e. the first time item 1 is worked
on. Worth fixing with that work, not before it.

Also recorded in the body: the two approaches to the ungated-tests half
that were considered and rejected during #1476 (per-test gating, and
helpers self-skipping on is_enabled()), with the reasoning, so they are
not re-proposed from scratch.

The honest framing for whoever picks this up: items 1 and 2 are the same
piece of work. Deriving what each item's gate should be — the union of its
inner #[cfg(feature = …)] rows — is both the missing-gate detector and the
thing that would drive a mechanical gating pass over the 2,594.

All three items done, on `fix/1472-gate-per-language-tests`

12 commits, 75 files, +7,465/−100. Not pushed.

1. The ungated per-language tests

Re-measured first, because the issue's figures were themselves copied
from a ci.yml comment: across src/, big-code-analysis-ast/src/ and
tests/, 3,025 tests named a grammar without gating on it — 2,840
in src/, 148 in the ast crate, 37 under tests/. (The "30 of 3,482"
in the body does not reproduce; 80 of 3,469 src/ tests carried a
feature cfg of their own.)

3,235 markers now, all but a handful derived by the new gate.

2. The gate that derives the marker

utils/check-test-lang-gates.py. For every item in a test scope it
collects the languages the body reaches — concrete *Parser, *Code
and LANG::* spellings, directly and through same-module helpers that
hardcode a parser — then evaluates the item's real cfg with each
needed feature turned off. A gate still satisfied there would admit the
item into a build lacking the grammar, which is the defect. Evaluating
the predicate under a one-feature-off assignment is what makes this
sound without symbolic implication, and it handles any / all /
not / nesting for free.

The language table is read out of the mk_langs! invocation rather than
copied, so Tsx → typescript and Ccomment / Preproc →
c-family-helpers cannot drift.

What licensed generating 3,225 markers: the derivation reproduces
all 167 gates written by hand in this tree, and a RepositoryTest
keeps it that way. Wired into make pre-commit and make ci (a pure
source scan, ~2.5 s); --fix writes markers, --show prints the
derivation.

3. `Subject.carries_tests`

Fixed, with the matcher moved onto the qualified module path as the body
predicted. Verified against real legs rather than fixtures alone: go
now checks 34 of 43 subjects, rust,typescript 30, c 21.

A real bug the pass found

In the root crate feature = "c-family-helpers" is false by
default: root cpp forwards to big-code-analysis-ast/cpp and never
enables the root's own copy, and all-languages does not list it. Six
Ccomment / Preproc tests would have been gated out of every default
build. --all-features hides this — only the default build catches it,
which is why the nextest list --all-features set-diff stayed clean
throughout. Root-crate markers now expand to
any(c, c-family-helpers, cpp, mozcpp).

One deliberate non-goal: imports

Imports are not gated. Whether one is live is a function of the
feature set, and the three shapes that matter are invisible to a
scanner: a trait reached through its methods names nothing, a glob binds
an unknowable set, and a macro is invoked from item position. Gating
them is wrong too wide (unused) and wrong too narrow — a narrowed
use super::*; took a name out from under a still-compiling test
(E0599). A crate-level
cfg_attr(not(feature = "all-languages"), allow(unused_imports)) covers
the noise instead, and only in partial builds; the default and
--all-features builds report unused imports exactly as before.

Five cases the derivation cannot attribute are gated by hand, each with
its reason in a comment: the corpus tests pick a language per file from
a glob; suppression_test.rs picked one from a path extension (now a
LANG parameter, which reads better anyway); ParserTrait is reached
only through its methods; assert_csharp_fixture_spells is
pub(crate) with callers in another file.

Verification

Check	Result
Derivation vs. the 167 hand-written gates	167/167 reproduced
`nextest list --all-features --workspace`, before/after	identical — 5,773 tests, 0 lost, 0 added
`make pre-commit`	`BCA_GATE: pass`
Gate self-tests	43 pass

Nine feature legs, each cargo clippy --all-targets -- -D warnings
and a full run, all green — where --no-default-features --features rust used to fail in the thousands:

no-default-features   321      ast, none        189
rust                  591      ast, go          192
c                     396      ast, rust+ts     221
python                542
tcl                   458
rust,typescript       877

-p throughout: a workspace-wide run unifies the cli/web crates'
default features back on and proves nothing while looking green (#1457).

Review

Three fresh-context reviewers went over the branch. An independent
re-derivation of the markers found no wrong-language marker and no
test unreachable in any CI configuration. Everything else they found
is fixed in the follow-up commits: three macro-branch bugs (a crash, a
--fix that would write into a transcriber body, a marker landing on
the item above), a 4.7× scan slowdown, an IndexError on exotic line
separators, a non-vacuity guard that counted production items, a
not/any/all prefix crash, ~190 lines of unreachable import-gating
code, and two over-narrow gates (one of them pre-existing).

Follow-up worth its own issue

Nothing detects a gate that is too narrow. The --all-features
set-diff cannot see it (an over-narrow gate still runs there), and the
per-leg runs only catch gates too wide — a marker that over-gates a test
off a leg is indistinguishable from a green run. A per-leg test-count
floor would close it; that is a separate piece of work.

Closing: all three items done, plus the over-gate direction

The three items in the body were finished as described in the comment
above. #1478 then found the check needed a second direction and several
of its rules were wrong; that work landed in 5ec5b225 on the same
branch and is summarised there. Net effect on this issue's subject:

3,500 gated test items, derived rather than hand-written
16 gates accepted as deliberately wider than their bodies, each with
a marker naming which features and why; a marker that stops being
load-bearing now fails
check-feature-gates.py's nested-mod false pass (item 3) fixed
the feature-matrix legs run the suite instead of only compiling it

Verified: 15 CI legs green (30/30 clippy and nextest),
nextest list --all-features unchanged, BCA_GATE: pass.

Branch fix/1472-gate-per-language-tests is not pushed.

test(features): add a gate deriving each test's language cfg

test(features): gate every per-language test on its grammar

fix(features): check a union-gated mod nested in a test scope

Navigation Menu

Repository navigation

test(abc): the test module is ungated, so a minimal-feature build reports ~2,600 spurious failures #1472

Summary

Why it matters

A second trap in the same area

Suggested shape

Related

RESOLVED

Update — the CI half is closed; three items remain

1. The 2,594 ungated per-language lib tests (the original issue)

2. `check-feature-gates.py` cannot see a missing gate

3. A latent false pass in `Subject.carries_tests`

Also noticed, adjacent

CI half closed in PR #1476; two new items folded in

All three items done, on `fix/1472-gate-per-language-tests`

1. The ungated per-language tests

2. The gate that derives the marker

3. `Subject.carries_tests`

A real bug the pass found

One deliberate non-goal: imports

Verification

Review

Follow-up worth its own issue

Closing: all three items done, plus the over-gate direction

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test(abc): the test module is ungated, so a minimal-feature build reports ~2,600 spurious failures #1472

Description

Summary

Why it matters

A second trap in the same area

Suggested shape

Related

RESOLVED

Update — the CI half is closed; three items remain

1. The 2,594 ungated per-language lib tests (the original issue)

2. check-feature-gates.py cannot see a missing gate

3. A latent false pass in Subject.carries_tests

Also noticed, adjacent

Activity

dekobon commented on Sep 15, 2026

CI half closed in PR #1476; two new items folded in

dekobon commented on Sep 16, 2026

All three items done, on fix/1472-gate-per-language-tests

1. The ungated per-language tests

2. The gate that derives the marker

3. Subject.carries_tests

A real bug the pass found

One deliberate non-goal: imports

Verification

Review

Follow-up worth its own issue

dekobon commented on Sep 16, 2026

Closing: all three items done, plus the over-gate direction

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

2. `check-feature-gates.py` cannot see a missing gate

3. A latent false pass in `Subject.carries_tests`

All three items done, on `fix/1472-gate-per-language-tests`

3. `Subject.carries_tests`