Declare selftest fixture tiers so packaged fixtures stop going amber

AGENTS.md

@@ -144,7 +144,7 @@ Optionally: a factory method in `src/Reactor/Elements/Dsl.cs`, fluent modifiers
 |---|---|---|
 | Algorithm, pure function, hook bookkeeping, D3 math | Unit test (xUnit) | `tests/Reactor.Tests/` |
 | Element mount/update against real WinUI controls | Selftest fixture | `tests/Reactor.AppTests.Host/SelfTest/Fixtures/` |
-| Behaviour that differs under MSIX identity (`ms-appx:`, `Package.Current`, MRT, `PackageRuntime.IsPackaged` branches) | Selftest fixture gated with `PackagedIdentityFixtures.RequirePackagedTier` | same folder; runs for real in `tests/Reactor.PackagedTests` |
+| Behaviour that differs under MSIX identity (`ms-appx:`, `Package.Current`, MRT, `PackageRuntime.IsPackaged` branches) | Selftest fixture gated with `PackagedIdentityFixtures.RequirePackagedTier` **and** declared `SelfTestTier.Packaged` | same folder; runs for real in `tests/Reactor.PackagedTests` |
 | Real user input, UIA properties, cross-process | E2E test (winapp ui) | `tests/Reactor.AppTests/Tests/` |
 
 Start with unit tests. Use selftests only when you need a live WinUI control. E2E is the slowest tier.
@@ -156,6 +156,13 @@ adds only MSIX properties plus a `Package.appxmanifest`. The whole corpus runs u
 **Gotcha worth not re-deriving:** the manifest's `uap5:AppExecutionAlias` is load-bearing — launching
 the alias stub inherits stdout while keeping package identity, which AUMID activation cannot do
 (it is brokered, so stdout can't be redirected at all).
+**Second gotcha:** a packaged fixture needs *two* declarations, not one. The
+`RequirePackagedTier` gate decides whether the body asserts; `SelfTestFixtureRegistry.TierRequirements`
+decides whether the unpackaged host runs it at all. Skip the second and the fixture self-skips
+into the amber skip inventory on every unpackaged run forever (issue #1154). Consequence worth
+knowing: **`--list-fixtures` is tier-dependent**, and both hosts print
+`# Total not-applicable fixtures:` / `# Not applicable fixture list:` after `# Total failures:`
+so the exclusion is an assertable fact rather than a silent absence.
 
 ### Console-mutating tests need collection isolation
 

TESTING.md

@@ -132,7 +132,7 @@ The reliable tell is the module label. A run that started prints the handshake-d
 | An algorithm, pure function, record equality, hook bookkeeping, D3 math — anything that doesn't need a WinUI window | **Unit test** in `tests/Reactor.Tests/` |
 | How an element mounts/updates against a real WinUI control, layout math against real Yoga+XAML, reconciler behavior end-to-end, assertions via `VisualTreeHelper` | **Selftest fixture** in `tests/Reactor.AppTests.Host/SelfTest/Fixtures/` (registered in `SelfTestFixtureRegistry`, wrapped by a `[TestMethod]` in `SelfTestBatch`) |
 | Real user input (clicks, keystrokes, tab navigation), UIA properties as seen by assistive tech, cross-process behavior, XAML Island interop | **E2E test** in `tests/Reactor.AppTests/Tests/` |
-| Anything that changes under **MSIX package identity** — `ms-appx:` resolution, `Package.Current`, MRT lookups, `PackageRuntime.IsPackaged` branches | **Selftest fixture** as above, gated with `PackagedIdentityFixtures.RequirePackagedTier` so it self-skips in the unpackaged tier |
+| Anything that changes under **MSIX package identity** — `ms-appx:` resolution, `Package.Current`, MRT lookups, `PackageRuntime.IsPackaged` branches | **Selftest fixture** as above, gated with `PackagedIdentityFixtures.RequirePackagedTier` **and** declared `SelfTestTier.Packaged` so the unpackaged tier does not run it |
 
 Rule of thumb: start with a unit test. Drop to selftest only when you need a live control. Reach for E2E only when you need cross-process UIA — E2E is the slowest and flakiest tier.
 
@@ -302,12 +302,20 @@ why SKIPPED is not just a politer green. Always put an issue number in the reaso
 one; the reason string is all a reader of the skip report gets. And if you are reaching for a skip
 to silence a flake, fix the flake instead — a skip makes the flake invisible rather than absent.
 
+**"This tier structurally cannot run it" is not on that list, and must not be expressed as a skip.**
+A skip is a per-run observation; tier applicability is a fixed property of the fixture, so a skip
+would restate the same permanent fact on every run and accumulate in the amber inventory — which is
+what [#1154](https://github.com/microsoft/microsoft-ui-reactor/issues/1154) was. Declare the tier in
+`SelfTestFixtureRegistry.TierRequirements` instead and the fixture is simply not selected; see §3.
+
 For raw-TAP consumers (the AOT job pipes `--self-test` straight to a `.tap` artifact and greps
 `^not ok `), the Host emits a `# Total skipped fixtures: N` trailer — placed *after*
 `# Total failures:` so the abort discriminator below is unaffected — followed by
 `# Skipped fixture list: <names>` when non-zero. Each fully-skipped fixture also gets its own
-`# Fully skipped fixture: <name> - N check(s) skipped, 0 assertions ran` line as it happens. The
-three prefixes are deliberately distinct so a grep for one does not match the others.
+`# Fully skipped fixture: <name> - N check(s) skipped, 0 assertions ran` line as it happens.
+Alongside them, and reporting a different thing, come `# Total not-applicable fixtures: N` and
+`# Not applicable fixture list: <names>` (§3) — fixtures that deliberately did **not** run here.
+All five prefixes are deliberately distinct so a grep for one does not match the others.
 
 One fixture, `SelfTestVerdict_OnlySkips_PositiveControl`, is **expected** to be Skipped on every
 run. It asserts nothing on purpose: it is the positive control that proves the SKIPPED verdict
@@ -520,20 +528,58 @@ while still inheriting stdout — so the TAP contract and flags from tier 2 are
 
 ### Writing a packaged fixture
 
-Fixtures live in the shared corpus (`tests/Reactor.AppTests.Host/SelfTest/Fixtures/`) and gate
-themselves:
+Fixtures live in the shared corpus (`tests/Reactor.AppTests.Host/SelfTest/Fixtures/`). Two steps,
+and both are load-bearing. **Gate** the fixture body:
 
 ```csharp
 if (!PackagedIdentityFixtures.RequirePackagedTier(H, this)) return;
 ```
 
-They run for real here and emit a single TAP skip in the unpackaged tier. Register them with the
-**`Packaged_`** prefix — that is what the shim's `IdentityDependentFixtures_Actually_Asserted`
-guard uses to decide which fixtures must never skip.
+and **declare** its tier in `SelfTestFixtureRegistry.TierRequirements`, beside its entry in
+`AllFixtures`:
+
+```csharp
+["Packaged_MyNewThing"] = SelfTestTier.Packaged,
+```
+
+Register it with the **`Packaged_`** prefix — that is what the shim's
+`IdentityDependentFixtures_Actually_Asserted` guard uses to decide which fixtures must never skip.
+
+The two steps do different jobs and neither replaces the other. The **declaration** governs
+*selection*: an undeclared fixture is offered to every tier, so the unpackaged host runs it, hits
+the gate, and emits a skip that lands in the run's amber skip inventory — a permanent entry
+describing a condition that is structural rather than incidental
+([#1154](https://github.com/microsoft/microsoft-ui-reactor/issues/1154)). The **gate** governs
+whether the body *asserts*: if this tier were ever launched without identity, it skips, and
+`IdentityDependentFixtures_Actually_Asserted` turns that into a red. Selection cannot do that job,
+because selection cannot observe identity — it keys off the entry assembly, and the unpackaged
+binary is a different binary.
+
+So `--list-fixtures` is **tier-dependent**: the unpackaged host does not list, and does not run,
+the fixtures declared `SelfTestTier.Packaged`. Both hosts print what they excluded, after
+`# Total failures:`:
+
+```
+# Total not-applicable fixtures: 3
+# Not applicable fixture list: Packaged_IdentityGuard, Packaged_SettingsStoreRoundTrip, …
+```
+
+That trailer exists because the fix for #1154 was a *removal*, and success and catastrophe produce
+the same observation when you fix something by removing it: "no amber" is what you get whether the
+filter works or whether somebody deleted the packaged corpus. Both shims assert on the trailer,
+and they demand opposite things — `SelfTestBatch.NotApplicableFixtures_AreExcludedFromThisTier`
+requires a non-empty list and that none of those names reached discovery or the run;
+`PackagedSelfTestBatch.EveryFixture_IsApplicableToThePackagedTier` requires **zero**. A tier probe
+stuck on one answer would otherwise look correct from whichever side agreed with it. For the same
+reason a *missing* trailer is never read as a count of zero — that would let a host which stopped
+reporting satisfy the packaged assertion by silence.
 
 The gate keys off the entry assembly, not `PackageRuntime.IsPackaged`: a fixture that skipped
 whenever identity was missing would report green if this tier ever ran without it.
 
+If a fixture needs the *absence* of identity, `SelfTestTier.Unpackaged` is the mirror declaration;
+nothing uses it yet.
+
 ### Scope and knobs
 
 The whole corpus runs under identity (~5 min, on its own CI runner).

tests/Reactor.AppTests.Host/Program.cs

@@ -12,8 +12,14 @@
 {
     // Fast path: emit the selftest fixture registry, one name per line, and exit.
     // Used by Reactor.SelfTests to discover fixtures without launching WinUI.
-    foreach (var name in SelfTestFixtureRegistry.AllFixtures)
+    //
+    // Deliberately the CURRENT TIER's corpus, not the whole registry: a fixture this host
+    // cannot run must not get a test case that could only ever report "skipped" (issue #1154).
+    // The two wrappers' list parsers both drop `#` lines, so the trailer below is inert to
+    // discovery while still naming the exclusions for a human running this by hand.
+    foreach (var name in SelfTestFixtureRegistry.FixturesForCurrentTier)
         Console.WriteLine(name);
+    SelfTestRunner.WriteNotApplicableTrailer();
     return;
 }
 

tests/Reactor.AppTests.Host/SelfTest/Fixtures/PackagedIdentityFixtures.cs

@@ -10,16 +10,26 @@ namespace Microsoft.UI.Reactor.AppTests.Host.SelfTest.Fixtures;
 /// </summary>
 /// <remarks>
 /// <para>These live in the shared fixture corpus rather than in a packaged-only source
-/// set, so the two hosts stay a single body of tests and <c>--list-fixtures</c> agrees
-/// across both tiers. What differs is the <see cref="IsPackagedTier"/> gate below.</para>
+/// set, so the two hosts stay a single body of tests. What differs is <i>selection</i>:
+/// they are declared <c>SelfTestTier.Packaged</c> in
+/// <c>SelfTestFixtureRegistry.TierRequirements</c>, so the unpackaged host neither lists
+/// nor runs them — <c>--list-fixtures</c> is deliberately tier-dependent (issue #1154).
+/// They previously ran everywhere and self-skipped, which restated a permanent structural
+/// fact as a per-run observation and left three entries in the amber skip inventory on
+/// every unpackaged run.</para>
+/// <para><b>The gate below is not redundant with that declaration.</b> Selection cannot
+/// observe identity — it keys off which binary is running, and the packaged host binary is
+/// a different binary. The gate is what catches the packaged host being launched
+/// <i>without</i> identity: a broken registration, a stale alias resolving to something
+/// else, someone running the .exe out of the build output. There the fixtures skip and
+/// <c>PackagedSelfTestBatch.IdentityDependentFixtures_Actually_Asserted</c> turns that into
+/// a red, rather than the suite reporting green while measuring nothing.</para>
 /// <para><b>Why the gate keys off the entry assembly.</b> A fixture that merely skipped
-/// whenever <c>PackageRuntime.IsPackaged</c> was false would be worse than useless: if
-/// the packaged tier ever launched the app without identity — a broken registration, a
-/// stale alias resolving to something else, someone running the .exe out of the build
-/// output — every identity check would quietly skip and the suite would report green
-/// while measuring nothing. That is precisely the failure mode this tier exists to
-/// remove. Keying off the entry assembly instead makes the requirement structural: the
-/// packaged host binary <i>must</i> have identity, and says so by failing.</para>
+/// whenever <c>PackageRuntime.IsPackaged</c> was false would be worse than useless: it
+/// would treat the missing-identity case as an excuse rather than a fault, which is
+/// precisely the failure mode this tier exists to remove. Keying off the entry assembly
+/// instead makes the requirement structural: the packaged host binary <i>must</i> have
+/// identity, and says so by failing.</para>
 /// </remarks>
 internal static class PackagedIdentityFixtures
 {
@@ -48,10 +58,17 @@ internal static class PackagedIdentityFixtures
     /// <c>false</c>.
     /// </summary>
     /// <remarks>
-    /// The skip's check name is derived from <paramref name="fixture"/> rather than passed
+    /// <para>The skip's check name is derived from <paramref name="fixture"/> rather than passed
     /// in, so callers cannot invent three different spellings for the same concept and the
     /// name always points at the fixture a reader has to go look at. Call it as
-    /// <c>RequirePackagedTier(H, this)</c>.
+    /// <c>RequirePackagedTier(H, this)</c>.</para>
+    /// <para><b>On the normal unpackaged path this is unreachable</b>, because the fixture is
+    /// declared <c>SelfTestTier.Packaged</c> and never selected there (issue #1154). It stays
+    /// because it guards a case selection cannot see: the <i>packaged</i> host running without
+    /// package identity. There the skip is what
+    /// <c>PackagedSelfTestBatch.IdentityDependentFixtures_Actually_Asserted</c> converts into a
+    /// failure — so this must keep skipping rather than assert, or that guard would never see
+    /// the condition it exists to report.</para>
     /// </remarks>
     internal static bool RequirePackagedTier(Harness h, SelfTestFixtureBase fixture)
     {