G3.EX: 38 of 44 non-pass rows were classifier gaps, not defects — feature refusals need a re-run, not a skip #3182

What the first full G3.EX sweep measured

981 example targets, timeout 120, on lambda-vector (x86_64, RTX 4090). Evidence: evidence/dogfood/0.67.0/examples.tsv.

The 44 fail + timeout rows were then re-run one at a time with their own required-features and their logs kept. That is what this ticket is about: of the 44, only 6 rows were genuine defects. The rest were the classifier answering a question it was not built to answer.

The three gaps, in order of how many rows they cost

1. A feature the example names at RUNTIME, not in required-features (9 rows). cuda_training_benchmark prints "This benchmark requires the 'cuda' feature" and exits 1; with --features cuda it exits 0. Same for cublas_fp8_7b_reproducer (cuda), compressed_kv and compressed_index (compression), semantic_embeddings (embeddings), tensor_example (tensor), showcase_shell (shell-autocomplete). The sweep already reads required-features from cargo metadata; these examples declare none and gate themselves in code instead.

The right behaviour is not a new skip class. It is a re-run: parse the feature the example names, run it again with that feature, and classify the SECOND run. A skip here would hide the case where the example is broken with its feature — which is exactly what gpu_info turned out to be (#3179).

2. No class for "runs until you stop it" (20 rows). api_server binds a listener and serves; pipeline_tui and calculator_tui are TUIs; buggy_server is a server; a dozen profile_*/bench_*/measure_* examples are benchmarks with no bound. Killing these after 120 s and calling it timeout — a class the skill defines as "a defect" — is wrong on all twenty. They need either a long-running class asserted from the target's own declaration (so it cannot be claimed post hoc by anything that happens to hang), or a smoke mode.

The distinction that must not be lost: a server that is still serving at 120 s is healthy; an inference example still on token 1 at 120 s is not. A class that cannot tell those apart is worse than no class.

3. NEEDS_ARGS_RE and NEEDS_DATA_RE are too tight (6 rows). ^(Usage|error: the following required arguments) misses --model-dir required (ssc_eval, ssc_preflight) and .nth(1).unwrap() on argv (check_layer4). [Mm]odel not found misses No model found. (qa_serve) and Failed to load transformer model (finetune_real).

The six genuine defects, for the record

Why this is a ticket and not a silent widening

Widening a classifier until the red goes away is how a gate becomes theatre. Every class added here must cite the line that put the row in it — the skill already requires this for needs-hardware and needs-data — and the feature case must re-run rather than skip. PR #3163 (the needs-data class) is the template: a hermetic fixture in tests/fixtures/dogfood_examples/, a --selftest row per class, and a mutation row proving the class can still turn RED.

Done

--selftest carries a case row for each of: a runtime-feature refusal that passes on re-run, a runtime-feature refusal that still fails with the feature (must stay a defect), a declared long-running target, --model-dir required, and No model found. A full sweep on the release commit then reports genuine defects only, and the count matches a hand audit of its non-pass rows.