[KS] DX checks: dev loop, API docs coverage, templates, CLI, reproducible builds (#170, #171)

docs/e2e-flows.md

@@ -1105,9 +1105,10 @@ the same and links to the runner's page.
 
 An audit is a gate step that checks the release artifact itself instead of driving the app. A check registered
 with `audit: <name>` is verified by that step, so no flow lists it (coverage reports `audit-flow` if one does). The
-gate runs each audit step once, after the build check, and judges every audit check for the target's platform: a
+gate runs each audit step once, one at a time, after the build check, and judges every audit check for the target's platform: a
 `failed` or `unsupported` result, or no result, fails a blocking check. A scoped run (`--flow`) runs no audits.
-`bun run e2e:audit [--artifact <app>] [--package <zip or folder>]` runs only the audits for this machine's target and prints each result.
+`bun run e2e:audit [--artifact <app>] [--package <zip or folder>] [--audit <name>]…` runs only the audits for this machine's
+target (with `--audit`, only those steps) and prints each result.
 
 `bundle` (`scripts/e2e/audit/bundle.ts`, registered in `checks/packaging-distribution.yaml`) audits release bundle
 hygiene (#153):
@@ -1152,6 +1153,88 @@ the distribution ZIP, or the folder holding the one `<name>-<version>-<arch>.zip
 Spark makes no DMG, so there is no DMG layout check. App translocation at launch is not checked: it needs a notarized
 app launched from its quarantined location.
 
+#### Developer experience audits
+
+Four audit steps, registered in `checks/developer-experience.yaml` (#170, #171), check the developer's side of a release.
+The `templates` and `dev-loop` steps build and run apps, so run them under the native build lock on a shared Mac.
+
+`api-docs` (`scripts/api-docs.ts`) reads the public surface the way `docs/api-public-surface.json` snapshots it.
+`docs/api-undocumented.json` lists the known gaps. `tests/api-docs-coverage.test.ts` (part of `bun run test`) fails on an
+undocumented export or page-less subpath that is not listed, and on a listed one that is now documented, so the list only
+shrinks. The audit fails while the list is not empty.
+
+| Check | What it verifies |
+|---|---|
+| `DX-DOCS-01` | Every public export has a TSDoc comment on its declaration (what an editor shows). |
+| `DX-DOCS-02` | No public export, parameter or return type is `any`. |
+| `DX-DOCS-03` | Every public subpath has a `docs/*.md` page that imports it (`@legendapp/spark/<subpath>` in quotes or backticks). |
+
+`cli` (`scripts/e2e/audit/cli.ts`) runs the SDK's `spark` bin (`packages/desktop/bin/spark.cjs`) headless in a scratch
+directory, only with arguments that fail or print help before any side effect.
+
+| Check | What it verifies |
+|---|---|
+| `DX-CLI-01` | `spark --help` names every command of the CLI's grammar (`packages/cli/src/cli-arguments.ts`) and `dev`; `-h`, no arguments and each command's `--help` exit 0; `spark dev --help` in the Kitchen Sink adds `expo start`'s options; `spark --version` prints the SDK version. |
+| `DX-CLI-02` | An unknown command or option, missing or extra arguments, a group without its subcommand (`spark sdk`), invalid values, conflicting build modes and running outside an app each exit 1 with one line on stderr that names the problem, and no stack trace. |
+
+`templates` (`scripts/e2e/audit/templates.ts`) packs the SDK (`scripts/pack.ts`), then works in a scratch directory with
+`CI=1`, no TTY, no stdin and a scratch `SPARK_HOME`, so the developer's registered SDKs and runners are untouched. It
+unregisters every app it built from LaunchServices (`lsregister -u`) and removes the directory. It took 8 to 10
+minutes on an M4 Max: the packages come from the npm registry and the two release builds compile React Native twice.
+
+| Check | What it verifies |
+|---|---|
+| `DX-TPL-01` | `spark create --packages <packed manifest>` makes the macOS, Windows and universal templates; each depends on the packed SDK, installs and type-checks (`tsc --noEmit`). |
+| `DX-TPL-02` | `spark build` of the macOS template makes a release app with a Hermes bytecode `main.jsbundle` that `codesign --verify --deep --strict` accepts. |
+| `DX-REPRO-01` | A second release build, after deleting `.spark/` and the generated `macos/` project, has byte-identical JavaScript (`*.jsbundle`, `*.hbc`, `*.js`). |
+| `DX-REPRO-02` | Every other file in the app is byte-identical too. Each difference is listed; for a Mach-O file, whether its `LC_UUID` or other load commands differ. See [Reproducible builds](#reproducible-builds). |
+
+`dev-loop` (`scripts/e2e/audit/dev-loop.ts`) runs `spark dev --clear --no-open --max-workers 2` for the Kitchen Sink and
+launches its dev build in driver mode (`scripts/testing/dev-session.ts`; build it first with `cd examples/kitchen-sink &&
+bun run rebuild:macos`). It edits `screens/devtools/FastRefreshMarker.tsx` and `desktop.config.json` and always restores them.
+
+| Check | What it verifies |
+|---|---|
+| `DX-LOOP-01` | From starting `spark dev --clear` (an empty Metro cache) to the dev build's first settled render, within the cold start budget. |
+| `DX-LOOP-02` | Three edits each reach the running runtime within the Fast Refresh budget, as hot updates: a global set before them survives (a reload clears it). |
+| `DX-LOOP-03` | After the edits the dev build is still compatible (`.spark/session.json`) and the same app process runs: no native rebuild. A native configuration change (the app's `version`) makes `spark dev` report "Native configuration or host changed." with a rebuild offered, within the native change budget; reverting it makes the build compatible again. |
+| `DX-LOOP-04` | A syntax error shows React Native's error overlay (LogBox, in the app's window) with the `SyntaxError`; the fix closes it and reaches the app without a reload. |
+
+##### Dev loop budgets
+
+The budgets (`DEV_LOOP_BUDGETS` in `scripts/e2e/audit/dev-loop.ts`) are for the reference MacBook Air (M1, 8 GB) and
+apply on every Mac. They were not measured on the M1: they are three times what a 16-inch MacBook Pro (M4 Max, Mac16,5,
+macOS 27) measured, rounded up. Replace them with values measured on the reference machine.
+
+| Budget | M4 Max measured | M1 budget |
+|---|---|---|
+| Cold start (`DX-LOOP-01`) | 10.8 s, 13.2 s | 45 s |
+| Fast Refresh, per edit (`DX-LOOP-02`) | 0.3 to 0.6 s | 2 s |
+| Native change detected (`DX-LOOP-03`) | 0.8 s | 5 s |
+
+DevTools in the dev loop has its own checks (`checks/devtools.yaml`).
+
+#### Reproducible builds
+
+`DX-REPRO-01` and `DX-REPRO-02` build the macOS template twice. `spark build` prepares the project in place (it patches
+`node_modules` and writes `macos/`, `ios/`, `react-native.config.js` and configuration), so each build starts from an APFS
+clone of the project as `spark create` left it, at the same path. With identical inputs the JavaScript bundle and
+`spark-runtime.json` are byte-identical. Built from an already-prepared project instead, `spark-runtime.json`'s
+`fingerprint` differs, because the first build changed its inputs.
+
+The native output is not bit-reproducible, because of Apple's tools. On Xcode 27 (macOS 27, arm64), four files differ. For
+each of them, `DX-REPRO-02` checks the parts that should still match:
+
+| File | Why it differs | What must still match |
+|---|---|---|
+| The main executable (`Contents/MacOS/<name>`) | The linker puts two symbol pointers in the other order, so some instructions and `__objc_stubs` refer to other slots. `LC_UUID` is a hash of the output, and the code signature is a hash of the file. | Every load command apart from `LC_UUID`, and the symbols (`nm -m`). |
+| `Assets.car` | `actool` stamps the build time and gives each rendition a unique name, which also changes its digest. | Every rendition's other attributes (`assetutil --info`). |
+| `*.nib` | `ibtool` writes the same objects in another order. | The objects, each with its class and values, with references as the referenced object's class. |
+| `_CodeSignature/CodeResources` | It holds the hashes of the files above. | Nothing more: the files it seals are compared themselves. |
+
+Any other difference, or a difference in what these compare, fails `DX-REPRO-02`. The frameworks' binaries (Hermes and
+the rest) and every other resource are byte-identical.
+
 ### Budgets
 
 `e2e/budgets.yaml` (schema: `e2e/schema/budgets.schema.json`) holds baselines per

e2e/checks/developer-experience.yaml

@@ -0,0 +1,78 @@
+# Developer experience (#170, #171): gate audit steps, not flows (docs/e2e-flows.md#audits). DevTools has its own
+# registry (checks/devtools.yaml, epic #267).
+area: developer-experience
+prefix: DX
+checks:
+  DX-LOOP-01:
+    title: Cold start of the dev loop, spark dev --clear for the Kitchen Sink to the first settled render of its dev build, is within the reference MacBook Air (M1, 8 GB) budget
+    platforms: [macos]
+    blocking: true
+    spec: "#170 Dev loop"
+    audit: dev-loop
+  DX-LOOP-02:
+    title: Fast Refresh delivers each source edit to the running app as a hot update, with no reload of the JavaScript runtime, within the M1 budget per edit
+    platforms: [macos]
+    blocking: true
+    spec: "#170 Fast Refresh"
+    audit: dev-loop
+  DX-LOOP-03:
+    title: JavaScript edits never require a native rebuild (the dev build stays compatible and the same process keeps running); a native configuration change makes spark dev report that the build needs a rebuild, within the M1 budget, and reverting it makes the build compatible again
+    platforms: [macos]
+    blocking: true
+    spec: "#170 Native change detection requires rebuild"
+    audit: dev-loop
+  DX-LOOP-04:
+    title: A syntax error in an edit shows React Native's error overlay in the running app; fixing it dismisses the overlay and Fast Refresh resumes without a reload
+    platforms: [macos]
+    blocking: true
+    spec: "#170 Error overlay"
+    audit: dev-loop
+  DX-DOCS-01:
+    title: Every public export (docs/api-public-surface.json) has a TSDoc comment on its declaration
+    blocking: true
+    spec: "#171 Every export documented"
+    audit: api-docs
+  DX-DOCS-02:
+    title: No public export, parameter or return type is any
+    blocking: true
+    spec: "#171 Every export typed"
+    audit: api-docs
+  DX-DOCS-03:
+    title: Every public subpath has a docs page that imports it
+    blocking: true
+    spec: "#171 Every export documented"
+    audit: api-docs
+  DX-CLI-01:
+    title: spark --help names every command; -h, no arguments and each command's --help exit 0; spark dev --help in an app adds expo start's options; spark --version prints the SDK version
+    blocking: true
+    spec: "#171 CLI commands"
+    audit: cli
+  DX-CLI-02:
+    title: CLI misuse (unknown command or option, missing or extra arguments, a group without its subcommand, invalid values, conflicting modes, no app) exits 1 with one line on stderr that names the problem and no stack trace
+    blocking: true
+    spec: "#171 CLI commands"
+    audit: cli
+  DX-TPL-01:
+    title: spark create makes each template (macOS, Windows, universal) from the packed SDK, headless (CI=1, no TTY), and it installs and type-checks
+    platforms: [macos]
+    blocking: true
+    spec: "#171 Templates build; headless CI"
+    audit: templates
+  DX-TPL-02:
+    title: spark build of the macOS template from the packed SDK makes a release app headless, with Hermes bytecode and a signature codesign verifies
+    platforms: [macos]
+    blocking: true
+    spec: "#171 Templates build; headless CI"
+    audit: templates
+  DX-REPRO-01:
+    title: Two clean release builds of the macOS template have byte-identical JavaScript bundles
+    platforms: [macos]
+    blocking: true
+    spec: "#171 Reproducible builds"
+    audit: templates
+  DX-REPRO-02:
+    title: Two clean release builds of the macOS template are byte-identical apart from what docs/e2e-flows.md#reproducible-builds explains, and each difference is reported
+    platforms: [macos]
+    blocking: true
+    spec: "#171 Reproducible builds"
+    audit: templates

e2e/schema/checks.schema.json

@@ -59,7 +59,11 @@
           "audit": {
             "enum": [
               "bundle",
-              "package"
+              "package",
+              "api-docs",
+              "cli",
+              "templates",
+              "dev-loop"
             ],
             "description": "The gate audit step that verifies this check against the release build, instead of a flow (see docs/e2e-flows.md#audits)."
           }

packages/cli/src/cli-arguments.ts

@@ -7,9 +7,9 @@ const options = {
   device: { type: "string" }, project: { type: "string" }, platform: { type: "string" },
   packages: { type: "string" }, port: { type: "string" }, runner: { type: "boolean" },
   dev: { type: "boolean" }, release: { type: "boolean" }, preview: { type: "boolean" },
-  force: { type: "boolean" }, dsym: { type: "string", multiple: true }, "submission-id": { type: "string" }, "ad-hoc": { type: "boolean" }, help: { type: "boolean", short: "h" },
+  force: { type: "boolean" }, dsym: { type: "string", multiple: true }, "submission-id": { type: "string" }, "ad-hoc": { type: "boolean" }, help: { type: "boolean", short: "h" }, version: { type: "boolean" },
 } as const;
-const commands: Record<string, { flags: readonly string[]; arguments: readonly [number, number] }> = {
+export const commands: Record<string, { flags: readonly string[]; arguments: readonly [number, number] }> = {
   create: { flags: ["example", "package-manager", "universal", "platform", "packages"], arguments: [1, 1] },
   "add desktop": { flags: ["project", "packages"], arguments: [0, 0] },
   "sdk export": { flags: ["packages", "runtime"], arguments: [1, 1] },
@@ -35,9 +35,10 @@ export function cliArguments(args: string[]) {
   const grouped = ["sdk", "add", "updates"].includes(positionals[0]);
   const command = positionals.slice(0, grouped ? 2 : 1).join(" ");
   if (!command || (grouped && positionals.length === 1 && values.help)) {
-    if (Object.keys(values).some(key => key !== "help")) throw new Error("Choose a command before specifying options. Run spark --help.");
+    if (Object.keys(values).some(key => key !== "help" && key !== "version")) throw new Error("Choose a command before specifying options. Run spark --help.");
     return parsed;
   }
+  if (grouped && positionals.length === 1) throw new Error(`Choose a subcommand: ${Object.keys(commands).filter(name => name.startsWith(`${command} `)).map(name => `spark ${name}`).join(", ")}.`);
   const contract = commands[command];
   if (!contract) throw new Error(`Unknown command: ${command}. Run spark --help.`);
   for (const key of Object.keys(values)) if (key !== "help" && !contract.flags.includes(key)) throw new Error(`--${key} is not an option for spark ${command}`);

packages/cli/src/dev-command.ts

@@ -2,7 +2,7 @@ import { spawnProcess } from "./process.ts";
 import path from "node:path";
 import { prepareConfig, selectTarget, supportedPlatforms } from "@legendapp/spark-desktop-config/config.cjs";
 import { dev } from "./dev.ts";
-import { findProject } from "./local.ts";
+import { appRoot, findProject } from "./local.ts";
 import { hostPlatform } from "./platform.ts";
 import { nodeCommand } from "./windows.ts";
 
@@ -44,7 +44,8 @@ export async function devCommand(args: string[]) {
   const options = devArguments(args);
   const start = path.resolve(options.project ?? process.cwd());
   if (options.expo.includes("--help") || options.expo.includes("-h")) {
-    console.log("spark dev options:\n  --project <directory>    Application directory\n  --platform <platform>    Initial launch target; all declared platforms stay available\n  --runner-binary <path>  Register and use a Spark Runner\n  --app-arg <value>        Pass a desktop launch argument (repeatable)\n  --no-open                Wait for a launch key (explicit Expo launch flags still apply)\n\nAll other options belong to expo start:\n");
+    console.log("spark dev options:\n  --project <directory>    Application directory\n  --platform <platform>    Initial launch target; all declared platforms stay available\n  --runner-binary <path>  Register and use a Spark Runner\n  --app-arg <value>        Pass a desktop launch argument (repeatable)\n  --no-open                Wait for a launch key (explicit Expo launch flags still apply)\n\nAll other options belong to expo start" + (appRoot(start) ? ":\n" : "; run spark dev --help inside an app to list them."));
+    if (!appRoot(start)) return;
     const child = spawnProcess(nodeCommand(start, "expo", "expo", ["start", "--help"]), { cwd: start, stdin: "inherit", stdout: "inherit", stderr: "inherit" });
     process.exitCode = await child.exited;
     return;

packages/cli/src/index.ts

@@ -59,7 +59,8 @@ async function main() {
       process.exit(process.exitCode);
     }
   }
-  if (values.help || !command) {
+  if (values.version && !command) console.log(VERSION);
+  else if (values.help || !command) {
     console.log(`Legend Spark
 
   spark create MyApp  Create an app (--universal for shared Settings; --platform selects a target)
@@ -76,6 +77,7 @@ async function main() {
 Inside an app: npm run dev, npm run build, npm run package (or pnpm, yarn, bun)
 
 Advanced: updates init <feedURL>, credentials, doctor, analyze, open [app], build --dev, build --preview, package --ad-hoc (test package, not notarized)
+Version: spark --version
 Crash reports: symbolicate <report.json> --dsym <MyApp.app.dSYM or folder> (prints the report with symbols and source lines)
 Windows: dev and build --dev; production builds are not yet supported.
 SDK transfer: sdk export <directory> [--runtime <Spark Runner directory>], sdk import <directory>

packages/cli/src/local.ts

@@ -21,16 +21,20 @@ export function findFramework(start = import.meta.dirname): string | undefined {
   }
 }
 
-export function findProject(start: string): string {
-  let dir = path.resolve(start);
-  while (true) {
+/** The app that contains `start`, or undefined outside an app. */
+export function appRoot(start: string): string | undefined {
+  for (let dir = path.resolve(start); ; dir = path.dirname(dir)) {
     if ((existsSync(path.join(dir, "app.json")) || existsSync(path.join(dir, "desktop.config.json"))) && existsSync(path.join(dir, "package.json"))) return dir;
-    const parent = path.dirname(dir);
-    if (parent === dir) throw new Error("No spark app found. Run this command inside your app, or create one with spark create MyApp.");
-    dir = parent;
+    if (path.dirname(dir) === dir) return undefined;
   }
 }
 
+export function findProject(start: string): string {
+  const root = appRoot(start);
+  if (!root) throw new Error("No spark app found. Run this command inside your app, or create one with spark create MyApp.");
+  return root;
+}
+
 export function readRuntime(app: string): Runtime | undefined {
   try {
     const windows = existsSync(path.join(app, "spark-runtime.json"));