Skip to content

Commit 5065db8

Browse files
ci: retire Spec Coverage and Engine Split Metric; run the showcase smoke in the Nightly Tiers live tier (#26529)
<!-- claude-projects-attribution: {"github_login":"os-steve"} --> _Requested by **Steve** · [project thread](https://claude.ai/code/project/chan_01FdL1XeUyvVNxLreh3F4Qi9?thread=cmsg_01FdL1XeUyvVNxLreh3F4Qi988VUoRMhEaekZbB638HuLE)_ Fixes #26445 Clause-②: no Metadata-protocol: none Ruling-ref: 6433804956 ## 维护者速读 - 改了什么:删掉两条没人读的定时工作流(Spec Coverage 每天、Engine Split Metric 每周);Showcase Smoke 不再是独立工作流,它的整套 Playwright 冒烟改由夜跑 `Nightly Tiers` 里的一个 `*.live.test.ts` 运行。 - 为什么改:裁决 B(6433804956)。三条工作流红了都不开卡、没人看;并进夜跑后,showcase 冒烟红了会由夜跑的 report 步骤开卡。 - 风险与回滚:夜跑分片 1 多约 8 分钟(构建 Console + 装 Chromium + 冒烟本身),仍在 45 分钟上限内;回滚即 revert 本 PR,三条工作流原样回来。 - 先看:`.github/workflows/test-nightly-tiers.yml`(新增准备步骤、诊断上传、分片腿顺序)、`examples/app-showcase/test/showcase-smoke.live.test.ts`、`examples/app-showcase/vitest.config.ts`。 - 两分钟验证:本 PR 改了 `test-nightly-tiers.yml`,所以它自己会在 PR 上跑一次 Nightly Tiers(分支上不写看板);看 `Nightly tiers (1/2)` 里 showcase 那一腿的 `53 passed`。 - 需要你做的:这是 Tier H(`.github/workflows/**`、`scripts/pm/**`),请批准后由席位落地。 ## What changes **Retired (deleted):** - `.github/workflows/coverage-nightly.yml` (Spec Coverage, nightly). Its only product was a `coverage-report` artifact; nothing reads it. `packages/spec`'s `test:coverage` script stays as an on-demand developer command (it is not a `check:*` gate and no workflow calls it after this PR). - `.github/workflows/engine-split-metric.yml` (Engine Split Metric, weekly, report-only). **The ADR-0076 D7 metric script stays in `scripts/`, unwired from any schedule**: `scripts/check-engine-split-ratio.mjs` runs on demand (its header now says so), and its `--self-test` stays wired in `lint.yml`, so the instrument stays correct while unrun. Keeping it means ADR-0076 still names a script that exists; its sentence naming the deleted workflow needs a one-line note, which is a separate governed PR (this one does not touch `docs/adr/**`). - `.github/workflows/showcase-smoke.yml` (Showcase Smoke, nightly). **Showcase smoke moved under Nightly Tiers:** - New `examples/app-showcase/test/showcase-smoke.live.test.ts`. The `*.live.test.*` name puts it in the nightly `live` tier (`scripts/nightly-tiers.mjs`), so no pull request or merge-queue run collects it. It does not restate any assertion: it runs `playwright test` with the package's own `playwright.config.ts` (all three specs under `e2e/`: the 49 nav surfaces, the ADR-0085 detail shapes, the bulk capability gate), streams the `list` reporter to stdout for the stall guard, and passes only when Playwright's JSON report has no failing test, no run-level error, and at least one test ran. A red names each failing spec (`file › title`) and, for a whole-run red, says to read it as environment first. - `examples/app-showcase/vitest.config.ts` reads `OS_TEST_TIERS` through `scripts/nightly-tiers.mjs` (`readTierMode`, `isNightlyTierFile`, imported, not re-spelled): under `queue` the tier files are excluded by exact path and the default include is unchanged; under `nightly` they are the include. `turbo.json`'s `@objectstack/example-showcase#test` already hashed `OS_TEST_TIERS`; it now also lists `scripts/nightly-tiers.mjs` (+ `.d.mts`) as inputs, as `@objectstack/cli#test` does. - `.github/workflows/test-nightly-tiers.yml`: - new step "Prepare the showcase browser smoke": on the shard carrying `@objectstack/example-showcase` only, `pnpm console:build` and `playwright install --with-deps chromium` (the two prerequisites the retired workflow had). `continue-on-error`, so a failed Console build cannot take the shard's CLI slice down with it; the smoke then fails as a whole-run red and the night still files a card. - new step "Publish the showcase smoke's Playwright diagnostics" (`examples/app-showcase/test-results/`, `always()`, `if-no-files-found: ignore`), replacing the retired workflow's two diagnostics uploads. - the leg loop runs the file-level slices first and the whole-package leg last (`for LEG in $SLICES __whole__`), with a comment on why it differs from ci.yml here: the loop stops at the first red leg, and the cheap whole package (the smoke) must not stop the shard's slice of the CLI e2e tier. - header comments that named the two retired workflows are reworded. **Pins and readers updated:** - `scripts/check-stall-guard-budget.mjs`: self-test battery 10 registers one case per real (workflow, job) with a guarded step; `coverage-nightly`'s job was one, so its floor goes 13 → 12 with a comment saying why. - `scripts/pm/dispatch-gates.self-test.mjs`: the live-tree case that required `scripts/check-engine-split-ratio.mjs --days 90` in workflow text drops that key (same treatment as `check-prerelease-pin-watch.mjs --verbose` when its workflow left); the fixture row quoting it stays. - `docs/qa/platform-checklist/areas/platform-core.json`: the showcase item's `source` names the live test and `test-nightly-tiers.yml` instead of the deleted workflow. - Comment pointers in `half-state-patrol.yml`, `shard-timings-refresh.yml` and `examples/app-showcase/playwright.config.ts`. Readers grepped (file names, workflow names, job names, artifact names, `test:smoke`, `test:coverage`): `dependabot.yml`, the required-context set, the aggregator roster, `check:self-test-wired` and `scripts/ci/scheduled-full-run.mjs` hold none of the three. `scheduled-full-run.mjs` uses `'Spec Coverage'` only as an example of a name outside its map, which stays true. ## Tests Local, on this branch (worktree `objectstack-issue-26445`), closure built with `pnpm turbo run build --filter=@objectstack/example-showcase^...` and `pnpm console:build`: - Live tier, real stack (`OS_TEST_TIERS=nightly vitest run` in `examples/app-showcase`, Chromium from `/opt/pw-browsers`): `53 passed (2.3m)` from Playwright, `Test Files 1 passed (1)`, `Tests 1 passed (1)`, at `0d08883a`. - Negative control: `scripts/ablation-replace.mjs` injected a failure into the Task Board surface only (anchor 1 → 0, blob `cf29f07379cd` → `26b53aa6a54e`). Result: Playwright `52 passed`, Task Board failed twice (with its retry), and the vitest test failed with `1 of 53 showcase smoke test(s) failed, 0 run-level error(s)` naming `showcase-smoke.spec.ts › surface renders cleanly: Task Board`. Restored: blob equals HEAD, `git diff HEAD` empty. - Two reds on the way, both fixed in this PR and kept in the test's comments: vitest's worker sets `NODE_ENV=test`, which disarms the dev-admin seed, so global setup's sign-in got 401 (the wrapper now drops vitest's worker env); a randomly picked port that was busy made `serve --dev` shift one port up while Playwright waited on the original until its 180 s timeout (the wrapper now takes a port from the OS). - Queue tier: `pnpm --filter @objectstack/example-showcase test` → `Test Files 39 passed (39)`, `Tests 484 passed (484)` (the live file is not collected). `pnpm --filter @objectstack/example-showcase typecheck` exit 0. - `node scripts/nightly-tiers.mjs --check`: `@objectstack/example-showcase owns 1 nightly-tier file(s); vitest collects 39 under queue, 1 under nightly`; CLI unchanged (512 / 85). `OS_TEST_TIERS=bogus` is refused by the config. - Partition of the nightly document (`--packages` → `partition-test-shards.mjs --shard k/2`): shard 1 = `@objectstack/cli 1/2` + `@objectstack/example-showcase` (kept whole, "not the run's floor"), shard 2 = `@objectstack/cli 2/2`. `nightly-tiers.mjs --self-test` and `partition-test-shards.mjs --self-test` exit 0. ## Acceptance notes - **Measured on a runner by this PR itself:** `test-nightly-tiers.yml` is in its own `pull_request` paths filter, so this PR runs Nightly Tiers on a hosted runner (branch runs never write to the board). That run is the first reading of the prepare step and the smoke under turbo's strict env. - Cost: shard 1 took ~24.6 min on the last scheduled night (run 40275583120). The retired workflow spent ~4.6 min on the Console build and ~1.4 min on the smoke (run 40286891669), so shard 1 should land near 33 min against its 45-minute limit. The partitioner weighs the showcase by its queue-tier timing (48 s) and does not know the live run's cost; if shard 1 gets tight, the lever is the matrix, not this file. - Not changed, read-only for this card: `.github/workflows/ci.yml` L810 still says "coverage-nightly workflow" in a comment. Left as dated readings: `docs-deploy.yml` (cron-hour survey "on the day this landed"), `lint.yml` L1495 (a 2026-08-21 measurement), `scripts/docs-audit/README.md`, `check-pnpm-acquisition.mjs`, `check-node-version.mjs`, `check-stall-guard-budget.mjs`'s "measured when this landed" line, and `dispatch-gates.mjs` comments that cite the old workflows as history. - Follow-up owed (separate governed PR): ADR-0076's D7 line names `.github/workflows/engine-split-metric.yml`; a one-line note that the script now runs on demand. - Root `test:smoke` (turbo) and the package's `test:smoke` stay: `pnpm test:smoke` is still how a person runs the suite by hand, and the QA checklist items cite it. ## Verification Gate list derived with `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` at the final head: 86 commands at `57f8caa3`, each run with its exit code captured before any pipe: **86 exit 0**. `dispatch-gates --ran`: "86 derived famil(ies) accounted for — 86 run, 0 NOT-MEASURED (a DERIVED zero)". Two were red on the way and are fixed in this PR: `check:stall-guard-budget` (battery 10 floor, see above) and `check:pm-dispatch-gates` (1 of 1935 fast-tier cases, the engine split live key). The dispatch-gates slow tier (`node scripts/pm/check-dispatch-gates.mjs --slow`, push/schedule only in CI) also ran: `✓ dispatch-gates self-test: 2200 cases pass`, exit 0. After merging `origin/main` (`2094edeb`, which brought the shard-matrix reader into `partition-test-shards.mjs`) the derivation is unchanged (86) and the overlapping gates were re-run: `partition-test-shards.mjs --self-test`, `nightly-tiers.mjs --self-test`, `nightly-tiers.mjs --check` all exit 0, and the nightly partition is unchanged. Families CI runs (Lint & Repo Gates, TypeScript Type Check, Test Core, Build Core, the UI lanes, Nightly Tiers on this PR) are read from this PR's check-runs. Changeset: none. `skip-changeset` — workflows, root scripts, `turbo.json` and the private `@objectstack/example-showcase` publish nothing. ## Human Minimum - [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic. (not touched) - [x] 2. Every name I added uses the casing for its kind. - [x] 3. No formatter ran; `pnpm lint` passes (CI). - [x] 4. One issue, one branch off an up-to-date `main`, one pull request. - [x] 5. Nothing publishes; `skip-changeset`. - [x] 6. `packages/spec` not touched. - [x] 7. No hand edits to `content/docs/references/`, `packages/*/CHANGELOG.md` or `content/docs/releases/`. - [x] 8. Title, body and comments are in English (the maintainer brief is Chinese by convention). - [x] The *Before Submitting* list in CONTRIBUTING.md is done. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01ADHkmTHHuvZqTM7XKWfv8Q --- _Generated by [Claude Code](https://claude.ai/code/session_01ADHkmTHHuvZqTM7XKWfv8Q)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 16ca320 commit 5065db8

15 files changed

Lines changed: 323 additions & 201 deletions

‎.github/workflows/coverage-nightly.yml‎

Lines changed: 0 additions & 48 deletions
This file was deleted.

‎.github/workflows/engine-split-metric.yml‎

Lines changed: 0 additions & 41 deletions
This file was deleted.

‎.github/workflows/half-state-patrol.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -178,7 +178,7 @@ on:
178178
# action, and never by reading a cadence out of this file.
179179
workflow_dispatch: {}
180180
# Changes to the patrol itself get exercised before they merge — the same
181-
# posture as engine-split-metric.yml. On a pull_request run the sweep still
181+
# posture as test-nightly-tiers.yml. On a pull_request run the sweep still
182182
# executes (that is the point: the transport, the flags and the rendering are
183183
# proven on a real runner), but the anchor write is skipped and the rendered
184184
# body goes to the run's step summary instead. A PR must never rewrite the

‎.github/workflows/shard-timings-refresh.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -271,7 +271,7 @@ jobs:
271271
persist-credentials: false
272272

273273
# Restore-only: scheduled runs read main's store cache; the per-push
274-
# workflows own saving it. Same shape as coverage-nightly.yml.
274+
# workflows own saving it. Same shape as test-nightly-tiers.yml.
275275
- name: Setup workspace
276276
uses: ./.github/actions/setup-workspace
277277
with:

‎.github/workflows/showcase-smoke.yml‎

Lines changed: 0 additions & 90 deletions
This file was deleted.

‎.github/workflows/test-nightly-tiers.yml‎

Lines changed: 58 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ name: Nightly Tiers
1313
#
1414
# ## Shape
1515
#
16-
# `coverage-nightly.yml`'s: `schedule` + `workflow_dispatch`, node 22, pnpm,
16+
# The house nightly shape: `schedule` + `workflow_dispatch`, node 22, pnpm,
1717
# RESTORE-ONLY caches (a scheduled run reads main's pnpm store and Build Core's
1818
# turbo seeds; the per-push workflows own saving them, and a nightly that saved
1919
# would churn the rationed 10 GB pool the way PR-side saves once did — see
@@ -41,6 +41,14 @@ name: Nightly Tiers
4141
# `--shard k/N`) only when a second tier-owning package arrives heavy enough
4242
# for the partitioner's own balance line to say so.
4343
#
44+
# The second tier-owning package is `@objectstack/example-showcase`, and it is
45+
# not that heavy: one `*.live.test.ts` file (#26445), the showcase browser
46+
# smoke, which used to be its own scheduled workflow (Showcase Smoke) and
47+
# filed nothing when red. The partitioner keeps it whole ("not the run's
48+
# floor") and places it on one shard; that shard also builds the Console SPA
49+
# and installs Chromium for it (≈ 5 min, the "Prepare the showcase browser
50+
# smoke" step), which no other shard pays.
51+
#
4452
# ## What lands where — the house idiom of `platform-checklist-watchdog.yml`
4553
#
4654
# On green this workflow files nothing, edits nothing and closes nothing. On
@@ -75,9 +83,7 @@ on:
7583
schedule:
7684
# Nightly. The minute is offset off the top of the hour ON PURPOSE (scheduled
7785
# workflows queue behind everyone else's `:00` cron) and off every sibling
78-
# patrol's minute, so two of them never contend for the same runner minute;
79-
# 05:29 sits after `coverage-nightly` (05:00) has started and well before
80-
# `showcase-smoke` (07:00).
86+
# patrol's minute, so two of them never contend for the same runner minute.
8187
#
8288
# ⚠️ Two readers, one fact. This value is restated as `PATROL_SCHEDULE` in
8389
# the `env:` block below, where the report step computes the deadline the
@@ -205,6 +211,31 @@ jobs:
205211
echo 'Items on this shard (a package name, or a package plus a k/n file-level slice):'
206212
cat "$RUNNER_TEMP/shard-packages.txt"
207213
214+
# The showcase browser smoke (`examples/app-showcase/test/showcase-smoke.live.test.ts`)
215+
# drives the Console against a real backend through Playwright, and needs
216+
# two things `turbo run test` does not produce: the Console SPA the backend
217+
# serves at /_console (`packages/console/dist`, built by the `build:dist`
218+
# task — `pnpm console:build`), and a Chromium this `@playwright/test` can
219+
# launch. Only the shard the partitioner gave that package pays for them.
220+
# The test's own `^build` closure is the `test` task's, as for every
221+
# package here.
222+
#
223+
# `continue-on-error`, on purpose: the same shard carries a slice of the
224+
# CLI's e2e tier, and a Console build that fails one night must not take
225+
# that slice down with it. The night is still red, and still names the
226+
# file — the smoke then fails every test at once, and its failure text
227+
# says to read a whole-run red as the environment first; this step's red
228+
# in the log is the environment.
229+
- name: Prepare the showcase browser smoke
230+
continue-on-error: true
231+
run: |
232+
if ! grep -qE '^@objectstack/example-showcase( |$)' "$RUNNER_TEMP/shard-packages.txt"; then
233+
echo "@objectstack/example-showcase is not on this shard — nothing to prepare."
234+
exit 0
235+
fi
236+
pnpm console:build
237+
pnpm --filter @objectstack/example-showcase exec playwright install --with-deps chromium
238+
208239
# The same leg loop as ci.yml's "Run this shard's tests", under the
209240
# nightly setting, with vitest's JSON reporter added beside the default
210241
# one: `--outputFile.json` is RELATIVE so each package writes its own
@@ -220,6 +251,13 @@ jobs:
220251
# ⛔ A failing leg STOPS the remaining ones, as in ci.yml: a second full
221252
# leg on a job that is already red only risks turning an informative red
222253
# into a killed job with no report at all.
254+
#
255+
# One deliberate difference from ci.yml: the file-level SLICES run first
256+
# and the whole-package leg last. The whole packages here are the cheap
257+
# ones (the partitioner keeps a package whole when it is not the run's
258+
# floor) — today the showcase browser smoke — and a red in one of them
259+
# must not stop the shard's slice of the CLI's e2e tier from running.
260+
# The other way round, a red slice skips a few minutes of smoke.
223261
- name: Run this shard's nightly tiers
224262
env:
225263
OS_TEST_TIERS: nightly
@@ -246,7 +284,7 @@ jobs:
246284
REPORTER="--reporter=default --reporter=json --outputFile.json=.nightly-tiers/vitest-report.json"
247285
STATUS=0
248286
LOGS=""
249-
for LEG in __whole__ $SLICES; do
287+
for LEG in $SLICES __whole__; do
250288
if [ "$LEG" = __whole__ ]; then
251289
[ -n "$FILTERS" ] || continue
252290
LOG="$RUNNER_TEMP/nightly-tiers-packages.log"
@@ -299,6 +337,21 @@ jobs:
299337
if-no-files-found: ignore
300338
retention-days: 14
301339

340+
# The showcase smoke's screenshots, traces and error context, which
341+
# Playwright writes under its default `outputDir` beside its config.
342+
# `always()` and `ignore`: a red smoke leaves them, and so does a green
343+
# one that passed only on its retry (`retries: 1` under CI) — the flaky
344+
# first attempt is the evidence a time-dependent red needs. A clean green,
345+
# and every shard without the showcase, leave nothing and upload nothing.
346+
- name: Publish the showcase smoke's Playwright diagnostics
347+
if: always()
348+
uses: actions/upload-artifact@v7
349+
with:
350+
name: showcase-smoke-diagnostics-${{ matrix.shard }}
351+
path: examples/app-showcase/test-results/
352+
if-no-files-found: ignore
353+
retention-days: 7
354+
302355
# The same two questions ci.yml's guard answers about a red suite — was
303356
# every test vitest COUNTED actually run, and did every scheduled package
304357
# report at all — asked of this shard's log against this shard's schedule.

‎docs/qa/platform-checklist/areas/platform-core.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -347,7 +347,8 @@
347347
"traps": ["hydration-race", "single-datapoint", "stale-console-bundle"],
348348
"source": [
349349
"examples/app-showcase/e2e/showcase-smoke.spec.ts#SURFACES (SURFACES)",
350-
".github/workflows/showcase-smoke.yml",
350+
"examples/app-showcase/test/showcase-smoke.live.test.ts",
351+
".github/workflows/test-nightly-tiers.yml",
351352
"objectui: packages/app-shell/src/chrome/ErrorBoundary.tsx (route-level fallback + recovery actions)",
352353
"examples/app-showcase/src/ui/apps/index.ts (nav truth)"
353354
],

‎examples/app-showcase/playwright.config.ts‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,10 @@ declare const process: { env: Record<string, string | undefined> };
1212
* to run `playwright test`; locally it reuses an already-running :3000.
1313
*
1414
* Run: pnpm --filter @objectstack/example-showcase exec playwright test
15-
* (or via the ci/showcase-smoke workflow). Non-blocking by design.
15+
* (`pnpm test:smoke`). On CI it runs nightly, never on a pull request:
16+
* `test/showcase-smoke.live.test.ts` runs this config in the nightly
17+
* `live` tier (test-nightly-tiers.yml), whose report job files a card
18+
* when it is red.
1619
*
1720
* ## Where the server is, and which browser drives it
1821
*

0 commit comments

Comments
 (0)