Skip to content

chore: merge test/node-conformance into tests/, and gate the partition - #60

Merged
Wahbeh-Mohammad merged 1 commit into
mvpfrom
55-merge-testnode-conformance-into-tests-and-gate-the-partition-that-keeps-the-two-suites-apart
Aug 31, 2026
Merged

chore: merge test/node-conformance into tests/, and gate the partition#60
Wahbeh-Mohammad merged 1 commit into
mvpfrom
55-merge-testnode-conformance-into-tests-and-gate-the-partition-that-keeps-the-two-suites-apart

Conversation

@Wahbeh-Mohammad

Copy link
Copy Markdown
Contributor

Closes #55.

What changed

This repository has two top-level test trees. Their names differed by one
character. test/node-conformance/ held 14 node --test files. bun run test:node runs these files against the built dist/. tests/conformance/xcut/
holds Bun test files. bun run test runs these files. Both trees are necessary.
The separation between them is also necessary. Only the names were a problem. No
file recorded that the two names differed.

This change moves the Node test tree to tests/node-conformance/. One top-level
tree now holds every test that crosses a process, a network, or a runtime
boundary. Each subdirectory holds one test runner.

Before this change, the file system kept the two trees apart. The Node test tree
was outside the reach of every Bun command. After this change, one path in five
files keeps the two trees apart. These five files must agree:

File Content
bunfig.toml [test] pathIgnorePatterns keeps the Node test tree out of bun test
package.json the test:node glob is the only command that runs the Node test tree
eslint.config.js the .mjs override supplies the Node globals
.claude/skills/ci-preflight/run-ci.mjs three globs in the --node-floor path
tests/node-conformance/README.md the membership rule and the tree paths

Each failure of this arrangement is silent:

  • Bun does not report an error for a node:test import. Bun collects these
    files. Bun runs these files. Bun reports these files as passed. The report
    shows no evidence about Node.
  • Bun accepts an unknown [test] key. Bun shows no warning. The key
    testPathIgnorePatterns looks correct. Bun does not read this key.
  • node --test exits with code 0 when a glob matches no file.

Measured with Bun 1.3.14, with bun run test:

Configuration Files Exit code
pathIgnorePatterns present 164 0
pathIgnorePatterns absent 178 1

The 14 additional files are the Node test tree. Bun runs these files and proves
nothing about Node. The exit code 1 is an accident. Thirteen of the 14 files
pass on Bun. The fourteenth file fails on an unrelated timer assertion. That
failure does not identify the cause. Do not use the exit code as a control.

This change adds scripts/verify-test-partition.mjs. This script reads the five
files. This script runs no test suite. This script blocks CI when the five files
disagree. The script makes seven checks. Five checks come from issue #55. Two
more checks guard rules that CLAUDE.md states and that no tool enforced. The
root test script must name both trees. The [test] root key must stay
"packages".

Other changes:

  • .github/workflows/ci.yml gets two blocking steps. One step runs the new
    gate. One step runs bun run test:scripts.
  • The second step closes docs/open-items.md H13. bun run test:scripts now
    runs in CI.
  • The trigger for H13 had occurred before this change. scripts/knowledge.test.mjs
    failed on main from commit 36c3f96. That commit added a correction to
    docs/knowledge/deliberate-deviations.md. The correction cites CFG-1. A
    topic file with no requirement ID received its first requirement ID. The
    --list-topics count moved from 16 to 15.
  • CLAUDE.md and .claude/skills/knowledge-lookup/SKILL.md quote the same
    count. Both documents were also incorrect. All three values now move together.
    The assertion message names the two documents.
  • Three more values in the same paragraph were incorrect. The corpus is 592 KB
    and not 518 KB. The roll-up count is 255 and not 256. The substantive count is
    386 and not 385. A fourth group of 4 uncited IDs was absent from the text. A
    second assertion now pins these values against --coverage.
  • docs/open-items.md H20 records a gap that this change does not close. The
    80% coverage floor measures the Bun run only. bun run test:node adds nothing
    to this floor.

These items stay unchanged, by decision:

  • docs/superpowers/plans/, docs/superpowers/specs/, docs/validation-prompts/
    and .changeset/*.md keep the old paths. These files are dated records. The
    paths were correct on the date of each record.
  • The node-conformance CI job calls bun run test:node. This job does not
    name a path. The steps of this job are unchanged.
  • tests/tsconfig.json gets a comment only. The compiler options are unchanged.
    This file sets no allowJs. Therefore tsc does not read the .mjs subtree.
    That subtree tests the shipped artifact and receives the fewest static checks.
    CI runs that subtree on two Node versions. This is the compensating control.
  • No changeset. A consumer cannot see this change.

Verification:

  • node .claude/skills/ci-preflight/run-ci.mjs --clean --node-floor passes all
    18 steps.
  • bun test ./tests collects 7 files. Before the change it collected 21 files.
    This result is the same on Bun 1.3.14 and on Bun 1.4.0.
  • bun run test:node runs 137 tests across all 14 files. This result is the
    same on Node 26 and on the Node 20.3.0 floor.
  • bun run test:scripts runs 81 tests.

Reviews done

Three review passes ran in parallel against one frozen snapshot. Each pass ran
as a separate agent. Each pass received the same snapshot and its own
instructions. No pass received the results of the other passes. No pass could
edit a file. Each pass used the knowledge-lookup skill to read the applicable
rules and cited rule IDs in its findings.

Pass Subject Result
1 — Shape the change against the requirement The shape is correct. No redesign.
2 — Adversarial the gate in use 3 major and 8 minor defects.
3 — Reader the repository after the change 1 blocker, 6 major, and 14 minor defects.

Pass 1 approved the shape. Therefore the findings of pass 2 and pass 3 are
applicable. Pass 1 also confirmed three earlier decisions. The gate belongs in
scripts/ and not in a skill directory. The new static tier in run-ci.mjs
is correct. The H13 work belongs in this change. Without the H13 work, CI never
runs the 21 tests of the new gate.

Pass 3 reported one blocker. No test started the gate as a program. A person
could delete process.exit(1), and all 64 tests still passed. The test file
scripts/verify-seam-1.test.mjs states this risk in its own header. This change
adds a test harness. The harness copies the gate into a temporary tree and runs
the gate as a program. The harness reads the exit code and the output.

Pass 2 reported six conditions where the first version of the gate passed and
the partition was broken. Each condition now has a check and a test:

Condition Before After
A pattern names a directory in the Bun test tree pass check 4
The eslint entry is deleted and the comment remains pass check 5
A test file is named retry.mjs or orphan.test.ts pass check 3
The ignore glob widens to tests/** pass check 4
The root test script names one subtree only pass check 6
[test] root is removed pass check 7

The first condition is the most important. Bun applies pathIgnorePatterns
during the directory walk. A pattern that names a directory removes the whole
subtree. That pattern matches no file path. The first version of the gate tested
file paths only.

Four more defects from pass 2 are also corrected. The ** operator now spans
whole path segments, as Bun does. The TOML reader now respects quoted strings.
The array reader now stops at the next key. The directory reader now uses
withFileTypes and does not fail on a broken symbolic link.

Two defects appeared during the correction work. The new tests found both. A
general comment remover deleted the correct files: entry, because the string
packages/*/scripts/*.mjs contains the comment delimiters. A substring test
accepted ./tests/conformance, which is the exact narrowing that the rule
forbids.

Pass 1 and pass 3 disagreed about one fact. Pass 1 stated that the test names
follow the pattern of both sibling files. Pass 3 stated that the test names
deviate from testing.md:14 and from scripts/verify-seam-1.test.mjs. Pass 3
is correct. verify-seam-1.test.mjs uses the required form. Only
verify-sse-37.test.mjs does not. All test names now use the required form.

Pass 3 also found incorrect statements in the documents. The ### HARD RULE
heading contained the rest of the ## Commands section, including the gate
list. The gate list claimed to be complete and omitted two blocking CI steps.
The README of the Node test tree listed 6 of 14 files. This change removes that
table. Nothing regenerated the table, and the phase checklists already record
the same information.

Pass 2 disproved one statement in this change. The comment in bunfig.toml
stated that both configurations exit with code 0. That statement is incorrect.
The corrected measurement is in the table above.

Evidence for the blocker correction: a person removes process.exit(1) from the
gate, and one test fails. Before this review, the same removal caused no test to
fail.

The repository had two top-level test trees whose names differed by one
character. `test/node-conformance/` held 14 `node --test` files run by
`bun run test:node` against the built dist/; `tests/conformance/xcut/` held
the Bun-run cross-package suites added by Phase 9. Both trees are necessary
and so is the split between them. Only the names were the problem, and no
file recorded that they differed at all.

Moves the Node tree to `tests/node-conformance/`, so one top-level tree holds
everything crossing a process, a network, or a runtime boundary, one
subdirectory per runner. Styleguide 11-testing scopes that rule to process and
network boundaries; reading a runtime boundary the same way is this repo's
extension, and it is now stated as one rather than cited as if the styleguide
said it.

What the move costs, and what pays for it:

Before, the file system held the separation — the Node tree sat outside
anything `bun test` could reach, so nothing could drift. Now one path written
into five files holds it, and every way that breaks is silent. Bun does not
error on a `node:test` import: it collects those files, runs them, and reports
them PASSING while proving nothing about Node. Bun accepts an unrecognized
`[test]` key with no warning, so `testPathIgnorePatterns` reads as configured
and does nothing. `node --test` over a glob matching nothing exits 0.

Measured on `bun run test`, pinned Bun 1.3.14:

    with pathIgnorePatterns   →  164 files, exit 0
    without it                →  178 files, exit 1

The 14 extra files are the Node suite collected by a runner that cannot prove
anything about Node. That run goes red only by accident — 13 of the 14 pass
silently and the 14th trips an unrelated timer assertion pointing nowhere near
the cause. The exit code is not a control.

So `scripts/verify-test-partition.mjs` reads the five files that must agree
(bunfig.toml, package.json, eslint.config.js, ci-preflight/run-ci.mjs, and the
tree's README) and blocks CI on disagreement. It runs neither suite. Seven
checks: the issue's five, plus two guarding rules CLAUDE.md states and nothing
enforced — the root script must name both trees whole, and `[test] root` must
stay "packages".

Adversarial review found six ways the first draft passed while the partition
was broken; each is now a check with a test:

  - Bun applies pathIgnorePatterns while WALKING, so a pattern naming a
    directory prunes the subtree without matching any file path. Adding
    `tests/conformance/fixtures` silently dropped a real test file and every
    full-path check stayed green. Directory prefixes now count.
  - The comment added above eslint.config.js's `files:` entry quoted the glob
    it documents, so deleting the entry left the check satisfied by the
    sentence explaining the guarantee.
  - A case named `retry.mjs` or `orphan.test.ts` was ignored by Bun, unmatched
    by test:node, and exempted by the check's own extension filter.
  - `**` compiled to a bare `.*`, matching paths Bun does not ignore — a gate
    green-lighting a config Bun reads differently.
  - A glob narrowed to one of 14 cases satisfied "matches at least one file".
  - `statSync` per entry threw a bare ENOENT stack trace on a broken symlink.

Closes open-items H13 — `bun run test:scripts` now runs in CI. Its trigger had
already fired: `knowledge.test.mjs` had been failing on main since 36c3f96,
whose Phase 10 correction to docs/knowledge/deliberate-deviations.md cites
CFG-1 and so gave a previously ID-less topic its first requirement ID, moving
the `--list-topics` count from 16 to 15. A gate whose own logic degrades still
exits 0, so nothing else in the run would have noticed. The same count is
quoted in CLAUDE.md and knowledge-lookup/SKILL.md; all three now move together
and the assertion's failure message names the other two. Three neighbouring
counts in that paragraph had rotted the same way (592 KB not 518, 255 not 256,
386 not 385, and a fourth bucket of 4 cited nowhere that went unmentioned) and
are now pinned by a second canary against `--coverage`.

Records what the audit found and this change does not fix: the 80% coverage
floor measures only the Bun run, and `test:node` contributes nothing to it
(open-items H20, RECORDED with a trigger).

Deliberately unchanged: docs/superpowers/plans/ and specs/, the validation
prompts, and .changeset/*.md keep their `test/node-conformance/` paths — they
are dated records and were correct when written. The node-conformance CI job
calls `bun run test:node`, not a path, so its steps are untouched.
tests/tsconfig.json gains a comment only; it sets no allowJs, so tsc still
never opens the .mjs subtree, and that subtree remains the one testing the
shipped artifact with the fewest static checks over it. CI running it on two
Node versions is the compensating control. No changeset — nothing here is
visible to a consumer.

Verified: node .claude/skills/ci-preflight/run-ci.mjs --clean --node-floor,
all 18 steps. `bun test ./tests` collects 7 files, not 21, on 1.3.14 and 1.4.0.
`bun run test:node` runs 137 tests across all 14 files on Node 26 and on the
20.3.0 floor. test:scripts is 81 cases. Commenting out the gate's
`process.exit(1)` fails a test; before the review it passed all 64.
@Wahbeh-Mohammad Wahbeh-Mohammad self-assigned this Aug 30, 2026
@Wahbeh-Mohammad
Wahbeh-Mohammad merged commit 12b2242 into mvp Aug 31, 2026
3 checks passed
@Wahbeh-Mohammad
Wahbeh-Mohammad deleted the 55-merge-testnode-conformance-into-tests-and-gate-the-partition-that-keeps-the-two-suites-apart branch August 31, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant