Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ harness. See `scripts/README.md` for the full recipe and how to add a library.
- **Currently green (cross-checked vs Node):** `decimal.js` (22,624), `big.js` (47,456; ~7 min — heavy arbitrary-precision arithmetic on the tree-walker), `acorn` (13,507), `prismjs` (2,563), `uglify-js` (4,233; ~15 min — full compress DSL transform/mangle/codegen corpus), `highlight.js` (731; ~30 min for the full auto-detection competition set).
- **`bignumber.js`** is wired but blocked on a jsse bug it surfaced (strict-mode `return <call-returning-non-object>` in a constructor returns that value instead of `this`, jsse#238); it goes green once that engine bug is fixed.
- **`luxon`** is wired at 3.7.2 with an exact 1,152-test Node cross-check. Node is green; jsse currently passes 1,045, with the remaining Intl/system-zone gaps tracked in jsse#262–#265.
- **`zod`** is wired at v4.4.3 in separate normal and jitless processes with an exact 2,184-case Node cross-check. Node is green; jsse currently reports 2,176 passing (including one visible jitless-only skip), with residuals tracked in jsse#309–#310 and jsse#313–#315.
- **`moment`** is wired at 2.30.1 with an exact 3,871-test Node cross-check. Node is green with 162,868 assertions; jsse executes every test and records 198 visible failing assertions, tracked in jsse#311.

### Acorn (`./scripts/run-acorn-tests.sh` — thin wrapper over the harness)
Expand Down
95 changes: 95 additions & 0 deletions docs/specs/2026-07-17-zod-library-harness-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Zod library harness design

## Scope

Add Zod's runtime-focused v4 classic suite as one reproducible real-world
library corpus under issue #279. Pin the latest stable release available when
the issue was assigned, v4.4.3. Its native Vitest run contains 1,092 runtime
tests across 79 files; the library harness runs every registered test in both
normal and global `jitless` modes for an exact 2,184-case lock.

Typecheck-only Vitest projects are outside the JavaScript-engine corpus: they
exercise TypeScript rather than ECMAScript execution. Zod's v3 and v4-mini
suites are also outside this issue, whose normal/jitless requirement targets
the v4 classic validator.

## Approaches considered

1. Add separate `zod` and `zod-jitless` library configurations. This reuses the
runner unchanged, but permits the two configurations to drift and makes the
issue's one-command expected-count lock awkward.
2. Extend the runner with prefixed bundle variants and optional process
isolation. This keeps one source corpus and count lock while isolating the
heap and event loop as well as module state. This is the selected approach.
3. Concatenate two independently scoped IIFE copies into one process. This was
prototyped first, but sustained async work in the normal copy polluted the
jitless copy and exposed continuation-liveness failures before the corpus
could finish.

## Design

Add `scripts/libs/zod.sh` and a generator that discovers the sorted
`packages/zod/src/v4/classic/tests/*.test.ts` set during the cached prepare
step. The generator writes a static entry and a narrow `vitest` compatibility
module. Esbuild aliases upstream imports of `vitest` to that module and bundles
all test files, Zod source, assertion support, and fixture dependencies into a
single IIFE. No filesystem or package resolution remains at runtime.

The compatibility module maps `describe`, `test`/`it`, `beforeEach`, and
`afterEach` onto the shared in-process TAP runner. It reuses a pinned published
expect matcher core, adds Vitest-compatible inline snapshot serialization, and
treats `expectTypeOf` as a runtime no-op because its assertions are enforced by
the separately excluded TypeScript project. The runner builds two files from
the same IIFE, one under each mode prefix, and launches them in independent
engine processes. Each copy registers the full upstream suite with a mode
suffix. Each wrapper sets Zod's documented config `jitless` value immediately
before invoking the original body, after upstream `beforeEach` hooks and before
upstream cleanup. Normal and jitless therefore share source bytes and assertion
logic without sharing mutable schemas, probe caches, GC state, or host jobs.

The force-test-harness prelude is used on both JSSE and Node. Node therefore
executes the identical bundle and matcher adapter, while the expected count is
independently established by a green native Vitest run at the pinned tag. The
2,184 lock prevents missing files or registrations from becoming a false pass.

Node-only imports in the upstream tests are replaced at bundle time with small
portable modules only where JSSE's host floor lacks that surface. These
replacements must run identically on Node, remain in `scripts/`, and preserve
the tested validator behavior. Host-only cases that cannot be represented
without materially implementing a new platform API will be excluded
explicitly and documented rather than hidden as passing tests.

The static-corpus portability patches are guarded against upstream drift. The
10 MiB base64 throughput fixture is scaled symmetrically to 64 KiB, still a
large-input validation without dominating a tree-walker run. Its artificial
500 ms async-refinement delay becomes a zero-delay timer on both engines while
retaining the asynchronous boundary (#310). Vitest's per-file cache and
prototype cleanup are reproduced explicitly. One jitless-only async function
refinement remains a visible `# SKIP` for the continuation bug in #309.

## Failure handling

A matcher throw, test/hook rejection, missing summary, nonzero failure count,
timeout, count mismatch, or engine-count disagreement fails the library run.
Any Zod failures that remain after narrow harness corrections will stay visible
and be reduced to follow-up engine issues. Engine behavior is changed only for
spec-backed defects, with focused tests and relevant test262 coverage.

## Verification

- Run native Vitest without typechecking to confirm 1,092/1,092 at v4.4.3.
- Run the generated corpus on Node, then JSSE with the Node cross-check, and
require exactly 2,184 registered tests on each engine.
- Run the shared harness and Node-shim self-tests when their seams are used.
- Run formatting, Clippy, release build/tests, relevant RegExp, BigInt, Proxy,
Reflect, and Symbol test262 areas for any engine changes, then full test262.
- Update the library harness documentation with the pin, exact count, result,
and any explicitly tracked residual failures.

## Result

Both JSSE and Node register exactly 2,184 cases. Node is green. JSSE reports
2,176 passing and eight failing cases: the same four failures in normal and
jitless mode, reduced to Date parsing (#313), array integrity levels (#314),
and Node-compatible JSON parse diagnostics (#315). The optional-chain parser
gap surfaced during bundle startup is fixed with focused test262 coverage.
24 changes: 24 additions & 0 deletions scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@ and (optionally) overrides hook functions:
| `LIB_ESBUILD_EXTRA` | bash array of extra esbuild flags (e.g. `(--main-fields=main,module)`) |
| `LIB_SHIM` | extra per-lib shim file (relative to `scripts/`) layered after `node-shim.js` |
| `LIB_SHIMS` | ordered array of additional shim files; use when a library needs more than `LIB_SHIM` |
| `LIB_BUNDLE_PREFIXES` | ordered prefix files; when set, build one isolated copy of the bundle after each prefix (for the same corpus in multiple modes) |
| `LIB_SEPARATE_BUNDLES` | with bundle prefixes, run each prefixed copy in a separate engine process and concatenate their output before verdict evaluation (default `0`) |
| `LIB_ENV` | host-process environment assignments applied to both engine runs (e.g. `("TZ=America/New_York")`). Reaches each engine's **native** layer only — jsse's Rust `Date`/`Intl` and Node's ICU read the OS `TZ`/`LANG`; **not** reflected in jsse's JS-visible `process.env`, which the `--node` shim leaves `{}`. Don't use it for values a library reads from `process.env` in JS. |
| `LIB_EXPECT_COUNT` | if set, both engines must report exactly this count (belt-and-suspenders against silent bundling drift) |
| `LIB_TIMEOUT` | seconds; wrap each engine run so a hang/slow suite reports cleanly |
Expand Down Expand Up @@ -234,9 +236,31 @@ assertions).
| `highlight.js` | 11.11.2 | ✅ 731 (cross-checked) | 536 markup + 195 auto-detection fixtures across 192 grammars; ~30 min |
| `js-sha256` | v0.11.1 | ✅ 916 (cross-checked) | Pure-JS SHA-224/SHA-256 and HMAC vectors; string, Buffer, TypedArray, and ArrayBuffer inputs |
| `luxon` | 3.7.2 | ⚠️ 1,045 / 1,152 | exact count cross-checked; Node is 1,152 / 1,152; blocked on #262–#265 |
| `zod` | v4.4.3 | ⚠️ 2,176 / 2,184 | normal + jitless, exact count cross-checked; Node is 2,184 / 2,184; residuals tracked in #313–#315 |
| `moment` | 2.30.1 | ⚠️ 198 failing assertions across 3,871 tests | exact registered-test count cross-checked; Node is green with 162,868 assertions; residual tracked in #311 |
| `bignumber.js` | v9.1.2 | ⚠️ blocked | see below; green on Node today |

### Zod normal and jitless corpus

`gen-zod-entry.js` statically imports all 79 v4 classic runtime test files.
Native Vitest at v4.4.3 reports 1,092 tests; the harness runs the identical IIFE
once normally and once with Zod's global `jitless` option, locking the combined
count at 2,184. The two modes use separate engine processes so module caches,
GC state, and pending host jobs cannot leak across the boundary. Node runs the
same two generated files and is green at 2,184 / 2,184.

The adapter uses Jest's published matcher core plus Vitest's pretty-printer;
type-only `expectTypeOf` calls remain runtime no-ops. Node-only test imports are
replaced by symmetric, bundle-local portability modules. The upstream 10 MiB
base64 throughput input is bounded to 64 KiB, and one artificial 500 ms async
delay is changed to a zero-delay timer (#310). One jitless-only async function
refinement is visibly skipped under #309; all other registered bodies run.

JSSE currently reports 2,176 passing and eight failures: Date parsing (#313),
array outputs that fail `Object.isFrozen` after Zod freezes them (#314), and the
Node-specific text of a `JSON.parse` error snapshot (#315), each repeated in
normal and jitless mode. These remain failing assertions rather than skips.

### PrismJS token-stream fixtures

`scripts/gen-prism-entry.js` embeds Prism core, dependency-ordered grammar
Expand Down
105 changes: 105 additions & 0 deletions scripts/gen-zod-entry.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
#!/usr/bin/env node
// Generate a static esbuild entry for Zod's v4 classic Vitest runtime suite.
//
// The source suite is discovered only during the cached prepare step. Every
// sorted import is then embedded in one bundle, leaving no runtime filesystem
// dependency. The library runner prefixes identical normal and jitless copies
// of the resulting IIFE, then executes them in independent engine processes.

"use strict";

const fs = require("fs");
const path = require("path");

if (process.argv.length !== 3) {
console.error("usage: node gen-zod-entry.js <output-entry>");
process.exit(2);
}

const output = path.resolve(process.argv[2]);
const packageRoot = path.dirname(output);
const testRoot = path.join(packageRoot, "src/v4/classic/tests");
const utilSource = path.join(packageRoot, "src/v4/core/util.ts");

const testFiles = fs
.readdirSync(testRoot)
.filter((name) => name.endsWith(".test.ts"))
.sort();

if (testFiles.length !== 79) {
throw new Error(`expected 79 Zod classic test files, found ${testFiles.length}`);
}

// Vitest normally gives every test file a fresh module graph. A single bundle
// cannot, so make the one observable memoized probe resettable by the adapter.
// The adapter restores the original accessor before each mode-tagged test;
// validator behavior is unchanged, while both JIT and jitless paths remain
// reachable and jitless-allows-eval.test.ts retains its upstream meaning.
const util = fs.readFileSync(utilSource, "utf8");
const cacheNeedle = 'Object.defineProperty(this, "value", { value });';
if (util.split(cacheNeedle).length !== 2) {
throw new Error("Zod cached() implementation changed; update the isolation patch");
}
fs.writeFileSync(
utilSource,
util.replace(cacheNeedle, 'Object.defineProperty(this, "value", { value, configurable: true });')
);

// prototypes.test.ts cleans up an enumerable prototype extension by assigning
// undefined. That is harmless with Vitest's per-file module isolation, but in
// a static suite it poisons later $constructor() calls, which bind every
// enumerable prototype key. Delete the temporary property instead.
const prototypeTest = path.join(testRoot, "prototypes.test.ts");
const prototypeSource = fs.readFileSync(prototypeTest, "utf8");
const prototypeNeedle = "z.ZodType.prototype._classic = undefined;";
if (prototypeSource.split(prototypeNeedle).length !== 2) {
throw new Error("Zod prototype cleanup changed; update the isolation patch");
}
fs.writeFileSync(
prototypeTest,
prototypeSource.replace(prototypeNeedle, "delete (z.ZodType.prototype as any)._classic;")
);

// The upstream "big base64" case is a throughput benchmark: it validates two
// independently generated 10 MiB byte buffers. Preserve its large-input
// semantics while bounding it for the tree-walker and the repeated CI corpus.
// 64 KiB remains well beyond the validators' ordinary examples without making
// one regex assertion dominate the entire library run.
const stringTest = path.join(testRoot, "string.test.ts");
const stringSource = fs.readFileSync(stringTest, "utf8");
const tenMegabytes = "randomBytes(1024 * 1024 * 10)";
if (stringSource.split(tenMegabytes).length !== 3) {
throw new Error("Zod big base64 test changed; update the bounded fixture patch");
}
fs.writeFileSync(
stringTest,
stringSource.replaceAll(tenMegabytes, "randomBytes(64 * 1024)")
);

// This test's delay is artificial; its assertion is about collecting multiple
// async validation errors, not elapsed time. A 500 ms delay can be lost by the
// Node-host timer pump late in the jitless corpus (issue #310). Preserve the
// asynchronous timer boundary but make it immediately eligible on both engines.
const asyncTest = path.join(testRoot, "async-parsing.test.ts");
const asyncSource = fs.readFileSync(asyncTest, "utf8");
const delayedRefinement = "setTimeout(() => resolve(false), 500);";
if (asyncSource.split(delayedRefinement).length !== 2) {
throw new Error("Zod delayed refinement changed; update the issue #310 patch");
}
fs.writeFileSync(
asyncTest,
asyncSource.replace(delayedRefinement, "setTimeout(() => resolve(false), 0);")
);

const imports = [
'// Generated by scripts/gen-zod-entry.js.',
'import { URL as PortableURL } from "whatwg-url";',
"if (typeof globalThis.URL === \"undefined\") globalThis.URL = PortableURL;",
];

for (const name of testFiles) {
imports.push(`import ${JSON.stringify(`./src/v4/classic/tests/${name}`)};`);
}

fs.writeFileSync(output, `${imports.join("\n")}\n`);
console.log(`gen-zod-entry: imported ${testFiles.length} Zod classic test files`);
58 changes: 58 additions & 0 deletions scripts/libs/zod.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# zod — TypeScript-first schema validation, exercised in normal and jitless
# modes from the same statically bundled v4 classic runtime suite.
#
# Native Vitest at the pinned tag reports 1,092 passing runtime tests across
# 79 files (typechecking disabled). The runner executes every upstream test in
# both modes and locks the combined 2,184-case count on both engines.
# Normal and jitless execute in separate processes so the stress corpus also
# isolates heap, host-job, and async-continuation state between modes.

LIB_REPO="https://github.com/colinhacks/zod.git"
LIB_REF="v4.4.3"
LIB_ENTRY="packages/zod/jsse-entry.ts"
LIB_ESBUILD_PLATFORM="browser"
LIB_ESBUILD_EXTRA=(
"--conditions=@zod/source"
"--keep-names"
"--alias:vitest=./packages/zod/jsse-vitest.ts"
"--alias:node:crypto=./packages/zod/jsse-portability.ts"
"--alias:node:util=./packages/zod/jsse-portability.ts"
"--alias:recheck=./packages/zod/jsse-portability.ts"
"--alias:@seriousme/openapi-schema-validator=./packages/zod/jsse-portability.ts"
"--alias:@web-std/file=./packages/zod/jsse-portability.ts"
)
LIB_SHIMS=("zod-host-shim.js" "node-test-harness-force.js" "node-test-harness.js")
LIB_BUNDLE_PREFIXES=("zod-normal-mode.js" "zod-jitless-mode.js")
LIB_SEPARATE_BUNDLES=1
LIB_EXPECT_COUNT="2184"
LIB_TIMEOUT="3600"

lib_prepare() {
# The runtime corpus needs only its matcher/serializer and a bundled URL
# implementation. Test-only Node modules are replaced by the symmetric
# portability seam copied below; Zod itself has no runtime dependencies.
node -e "const p=require('./package.json'); p.devDependencies={}; p.scripts={}; p.sideEffects=true; require('fs').writeFileSync('package.json', JSON.stringify(p,null,2)+'\n')"
node -e "const f='packages/zod/package.json'; const p=require('./'+f); p.sideEffects=true; require('fs').writeFileSync(f, JSON.stringify(p,null,2)+'\n')"
npm install --ignore-scripts --no-audit --no-fund --no-save \
expect@29.7.0 @vitest/pretty-format@4.1.5 whatwg-url@17.1.0

cp "$SCRIPT_DIR/zod-vitest-shim.ts" packages/zod/jsse-vitest.ts
cp "$SCRIPT_DIR/zod-test-portability.ts" packages/zod/jsse-portability.ts
node "$SCRIPT_DIR/gen-zod-entry.js" "$LIB_ENTRY"
}

lib_verdict() {
local out="$1" rc="$2" line P F T summaries=0 total=0 failed=0
while IFS= read -r line; do
if [[ "$line" =~ PASS:\ ([0-9]+)[[:space:]]+FAIL:\ ([0-9]+)[[:space:]]+TOTAL:\ ([0-9]+) ]]; then
P="${BASH_REMATCH[1]}"; F="${BASH_REMATCH[2]}"; T="${BASH_REMATCH[3]}"
summaries=$((summaries + 1))
total=$((total + T))
failed=$((failed + F))
fi
done < <(grep -oE 'PASS: [0-9]+[[:space:]]+FAIL: [0-9]+[[:space:]]+TOTAL: [0-9]+' "$out" || true)
if [ "$rc" -eq 0 ] && [ "$summaries" -eq 2 ] && [ "$failed" -eq 0 ] && [ "$total" -gt 0 ]; then
echo "PASS $total"; return 0
fi
echo "FAIL $total"; return 1
}
Loading
Loading