Repository navigation
feat: storybook with deterministic visual regression baselines - #444
Conversation
Storybook (@storybook/nextjs-vite 10.6.5) builds against the app's own Next 16 / React 19, with stories for every component in components/ and per-state stories (loading / connected / error) for each composite. Determinism: every story runs on an injected frozen clock (.storybook/fake-clock.ts), with server actions, the wallet context and the discovery service mocked and every NEXT_PUBLIC_* pinned, so the live-accrual counter and every other value render identically on each run. CI: a new storybook-visual job builds Storybook and runs a Playwright full-page screenshot diff per story inside the pinned Playwright container, at zero pixel tolerance. It uploads the diffs as an artifact, and on failure re-renders candidate baselines for review. Self-tests check that a one-line CSS regression fails the comparison and that the clock holds the counter still, with a positive control. The new steps are registered in .github/ci-gates.json. The Next 16 friction and how it was resolved is documented in docs/STORYBOOK.md.
✅ Deploy Preview for streampay-org ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
A committed pixel baseline needs one deterministic state. Playwright's animations:"disabled" pauses/cancels at capture time instead of removing the animation, so the captured frame still depends on where in its cycle an infinite animation (pulse, wallet-spin) or a restarted transition happened to be, and two captures of the same settled render are not byte-identical. Declaring animation/transition/caret away pins every element to its static specified style - the same mechanism the app's own prefers-reduced-motion block uses for .live-dot.
The 59 PNGs in frontend/visual/__screenshots__/linux are the candidate set from the Storybook job of run 40926388369 (attempt 2, head 1cb573b), rendered inside mcr.microsoft.com/playwright:v1.64.1-noble — the only renderer these baselines are comparable against — and taken from the visual-baselines-linux artifact (id 12919894826). Before committing, every file was screened: PNG signature, IHDR first and IEND last with no trailing bytes, no unrecognised chunks, no tEXt/zTXt/iTXt payload chunks, and no duplicate content except one pair noted below. The set is exactly the 59 story ids the suite asks for — no extra, no missing. streams-streamform--default.png and streams-streamform--submitting.png are byte-identical: the Submitting story's play function does reach the disabled "Streaming…" button (its waitFor passes), but the render captured a moment later is the untouched form, and the DOM snapshot CI writes at failure shows the empty recipient and the enabled button. That is a story-level gap, not a missing baseline, and it is called out in the PR rather than hidden by retouching an image.
The self-test still fails on any byte of difference between the two captures — that part is unchanged. What it could not say was *what* moved: run 40926388369 reports "render drifted while the clock was frozen, Received: -1" and nothing else, which leaves every candidate (a late async update, a remount, an animation the pin missed, a height change that rewrites the whole fullPage image) equally plausible. So the test now records the window it is asserting over: a MutationObserver over the document for the whole 2.5 s (childList, characterData and attributes, with old values), the scrollHeight and the live animation set each capture saw, and — when the buffers differ — both PNGs plus that log as report attachments, which CI uploads in visual-diffs. The summary goes into the failure message itself, so the CI log alone distinguishes "the DOM changed" from "the DOM was still and the pixels moved anyway".
Run 40941155592 proves the committed baselines: every one of the 59 stories matched byte for byte in a fresh render of a different commit, and the two self-tests' setup is right — only the live-accrual capture still drifts. It also shows the evidence capture failing on its own: fs.promises.writeFile into testInfo.outputDir hit ENOENT because Playwright creates that directory when it writes its own artifacts, not before the test body runs, so the write threw and replaced the drift failure with an unrelated error. tsc caught a second one: `const log = (host.x = [])` infers never[]. Attachments now go out as `body` instead of touching the filesystem, the whole capture is inside a try/catch whose only effect is a note on the failure message, and the DOM-mutation log is inlined there too, so the CI log carries the diagnosis even if the report never gets read.
The instrumented run answered its own question. CI reported, at head 635b7a7: 60 passed / 1 failed, and the failure message carried the whole window — render drifted while the clock was frozen: 86948 vs 86957 bytes, scrollHeight 1063 -> 1064, animations none -> none, 4 DOM mutations [+3ms input#_r_1_ @Style = "caret-color: transparent !important;" (was "") ;; +56ms ... = "" ;; +2565ms ... ;; +2623ms ...] No component re-rendered. The only four changes in the window are the caret-hiding style Playwright writes on the *focused element* before it takes a screenshot and restores right after — once per capture. Two consequences, both fixed here: - the capture read scrollHeight and the animation list in `Promise.all` alongside the screenshot, so the measurement could land on either side of that toggle while a `fullPage` shot was also resizing the viewport. 1063 vs 1064 is one pixel of document height, and one pixel of height rewrites the whole PNG — the byte difference is the harness, not the app. The page is now snapshotted in a single `evaluate` *before* the image, never in parallel with it. - a focused input means every capture styles the page differently during the shot than after it. Focus is dropped and the scroll reset once, before the first measurement. The fix normalises the *harness* only, and only up front: nothing is re-normalised between the captures, so if the app itself moves focus or scrolls inside the window that is still drift. Those two values are now reported in the failure message along with height and animations. The comparison is unchanged — byte-for-byte PNG equality, maxDiffPixels nowhere, retries 0 — and the positive control (tick the fake clock, the counter must move) is unchanged.
The previous commit's instrumentation was supposed to explain the drift. It did, and the answer is not what it looked like. Decoding the two PNGs the test attached (run 40946791424, job 123705891314, attachment files 051e6f1c… at 86,948 B and 3f2bde30… at 86,957 B): 051e6f1c… IHDR 1024 x 1063 3f2bde30… IHDR 1024 x 1064 over the shared 1024 x 1063 area: 0 differing channel bytes the extra row: 10,14,22 … — the page background, one solid row So the render did not move. Not one pixel of the 1,088,512 they share differs; the second image is simply one row taller and the extra row is empty. The caret-colour records the mutation log caught are Playwright styling the page for its own screenshot and undoing it afterwards — real, visible, and irrelevant, because PIN_RENDERED_STATE already forces `caret-color: transparent` on everything and the pixels never changed. Chasing them would have been the wrong fix, and blurring the element in the last commit changed nothing about the numbers: still 86948 vs 86957, still 1063 -> 1064. What does change the page is `fullPage`. A document's scroll height is never smaller than its viewport, and a full-page capture resizes the viewport to the document to frame it, then restores it. That round trip leaves the document one CSS pixel taller here — deterministically, in both runs — so `scrollHeight` is 1064 where the content is still 1063, and the next full-page image carries one more row of background. Two full-page captures of the same page are therefore not byte-comparable no matter what the component does, which is a property of the tool, not of the code under test. Fix: pin the viewport to the page height + 60 px once, up front, and take viewport screenshots. Nothing resizes during a capture any more, and the test loses no coverage: the viewport contains the entire page, the comparison is still byte-for-byte with no tolerance, retries stay 0, and a render that really moved would fail exactly as before. The evidence block now also reports the scroll offset and the focused element so the next surprise explains itself too. Also verified while reading this run: with main merged into the branch (globals.css +306 lines, StreamList and page.tsx changed) all 59 story baselines still matched — 60 passed / 1 failed, and the `--update-snapshots=changed` pass re-rendered 59 passed with nothing changed. The committed baselines are current.
|
Landed: the gate is green at I pushed five commits onto this branch as a maintainer, so you are not on the hook for the parts that were repo plumbing rather than your code. Everything below was measured in CI, not on a laptop: the baselines in this repo only mean something if the same Playwright image rendered them. The baselines are committed ( Determinism, measured twice rather than assumed.
The self-test that kept failing was the harness, not your component. The frozen-clock test asserted two captures 2.5 s apart were byte-identical and got Fix ( One thing I would like you to fix, and it does not block this merge: Where it stands: all four workflows are green at |
Summary
Closes #93
Adds a Storybook documentation site for the frontend component set, plus a visual-regression gate in CI. A UI change now shows up in the PR as an image diff instead of something a reviewer has to spot by eye. Storybook builds against the app's own Next 16 / React 19. The friction that combination caused, and how each problem was resolved, is documented in
docs/STORYBOOK.md.What's included
Storybook
@storybook/nextjs-vite@10.6.5, using the app's ownnext@16,react@19andreact-dom@19(no second copy of either). It reuses thevite@8the vitest suites already pin, so the lockfile gains no second bundler.components/**/*.stories.tsx.DashboardStreamDetailClientStreamForm/BatchStreamFormWalletButton/WalletModalNetworkBadge,ClaimSchedulerSettings,DiscoveryConsentModal,StreamList, Toast, Charts, panelsen/de) wraps every story in the realLocaleProvider.Deterministic rendering
.storybook/fake-clock.ts):Dateis frozen at2026-01-15T12:00:00Zfor every story. The live-accrual ticker still fires each second, but every tick sees the same instant, so the "Available now" counter renders identical digits on every run. Real timers are left untouched, so React and Storybook scheduling are unaffected.sb.mock.NEXT_PUBLIC_*value is pinned, including an RPC URL pointing at a closed port.mcr.microsoft.com/playwright:v1.64.1-noble, atmaxDiffPixels: 0. A CI step fails if@playwright/testand the image tag drift apart.CI: new
storybook-visualjobnpm ci→ check that the Playwright version matches the container →npm run build-storybook.npx playwright test: one full-page screenshot per story, compared againstfrontend/visual/__screenshots__/linux/.visual-diffsartifact (expected / actual / diff PNGs, plus the HTML report).visual-baselines-linux, so baselines can be updated without Docker. CI never writes or commits a baseline itself..github/ci-gates.json, so removing one later trips the workflow-integrity gate.Demo: the gate catches a real style change
frontend/visual/detector.visual.tsruns on every PR as a self-test of the gate:.status-badgeand requires the comparison against the committed baseline to fail. The resulting diff images are in thevisual-diffsartifact on every run.First-run baselines
The baselines were rendered in the pinned container and committed in a separate commit with a review note. All 59 PNGs were inspected before committing.
Next 16 / React 19 friction and resolutions
app/actions.ts("use server") importsserver-only, which throws in a browser bundleapp/__mocks__/actions.ts.__mocks__is a private App Router folder, so it is never routed.lib/discovery/service.tsimportsnode:cryptolib/discovery/__mocks__/service.tsprovides only the two exports the modal uses.sb.mock(..., { spy: true })rewrites type re-exports before types are stripped, which breakslib/stellar.tsfetchAssetMetaalready checks first.vite-plugin-storybook-nextjsper-keyNEXT_PUBLIC_*values override Storybook'senvoption, so a local.env.localleaked into renders**globs skip.storybook/, so the config was never type-checkedtsconfig.jsonincludes.storybook/**explicitly.finishedin Storybook 10, notcompletedfinishedand fails onerrored/aborted.No application source code was changed. The diff is limited to tooling config, stories, mocks, the visual suite, CI and docs.
How to verify