Add property-based testing: Hypothesis (Python) and Bombadil (TypeScript) - #1385
Open
raman325 wants to merge 16 commits into
Open
Add property-based testing: Hypothesis (Python) and Bombadil (TypeScript)#1385raman325 wants to merge 16 commits into
raman325 wants to merge 16 commits into
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mption Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Installs @antithesishq/bombadil (pinned exact, 0.x) and adds a dependency-free static harness (pbt/harness/) that mounts lcm-slot and lcm-lock-codes against a scripted mock hass, plus a zero-dependency static file server (pbt/serve.mjs) to serve it. This is the harness a follow-up task will drive with Bombadil property specs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wvy9UfQtNuvz1qqbEFYoB7
Adds pbt/spec.ts (properties: no PIN leakage, chip count matches model, pushed names eventually render, suspended state eventually shows its banner, plus a shadow-DOM-aware click action generator) and pbt/run.mjs (spawns the static server, drives `bombadil browser test`, propagates its exit code). Wires yarn test:pbt, extends lint to ./pbt, and adds pbt/spec.ts to the tsconfig include so type-aware eslint can check it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wvy9UfQtNuvz1qqbEFYoB7
Root tsconfig uses moduleResolution: node (node10), which cannot see Bombadil's package.json exports subpaths, so pbt/spec.ts type-checked as implicit-any everywhere. pbt/tsconfig.json extends the root config with moduleResolution: bundler (contained to pbt/, root untouched) and points eslint's type-aware parser at it for pbt/**/*.ts. Switching resolution strategies surfaces a real bug in @antithesishq/bombadil 0.6.1's own shipped types: actions.d.ts re-exports randomRange from random.d.ts, which never declares it. node10 resolution never reached these files at all, silently masking it. skipLibCheck avoids failing our build on a declaration file we don't own. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wvy9UfQtNuvz1qqbEFYoB7
- harness.js chaos-rename: digit-free labels (letters instead of the numeric revision) so a rename can never collide with a live secret PIN and falsely trip maskedPinNeverLeaks; harden the property itself with digit-boundary regex matching instead of raw substring includes - deepText: join shadow-root chunks with a newline so adjacent text nodes can't fuse into a spurious digit run across a shadow boundary - run.mjs: listen for the bombadil child process's 'error' event (e.g. ENOENT) so a failed launch resolves the exit promise instead of hanging forever and leaking the static server - package.json: test:pbt now uses && so a failed build doesn't property-test a stale bundle - namesEventuallyRendered: read modelNames inside the eventually thunk so the pending obligation tracks the current model, not a stale snapshot from when the formula was first evaluated - harness.js chaos-add-slot: reuse the lowest free slot in 1..6 instead of max+1, which saturated permanently and let chaos coverage decay - spec.ts extractors: optional-chain window.__lcmHarness with safe defaults so a sample taken before harness.js runs doesn't crash - serve.mjs: resolve ROOT with fileURLToPath instead of raw pathname (percent-encoding breaks paths with spaces), and answer /favicon.ico with 204 before attempting a file read, since a 404 there could trip bombadil's default noHttpErrorCodes property in headed runs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wvy9UfQtNuvz1qqbEFYoB7
… frontend Bombadil's noHttpErrorCodes default property caught a real harness defect: clicking a lock-codes chip triggers history.pushState to a virtual Home Assistant route (e.g. /config/integrations/integration/lock_code_manager#config_entry=...), and Bombadil's reload action then requests that route directly from pbt/serve.mjs, which 404s. The real HA frontend is a single-page app that serves its shell for every route; the harness server now mirrors that by falling back to pbt/harness/index.html for any extensionless path, while still 404ing honestly on missing assets (identified by extension). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wvy9UfQtNuvz1qqbEFYoB7
github-actions
Bot
added
dependencies
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1385 +/- ##
=======================================
Coverage 96.98% 96.98%
=======================================
Files 53 53
Lines 6495 6495
Branches 470 470
=======================================
Hits 6299 6299
Misses 196 196
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed change
Adds a property-based testing layer on both stacks, complementing the example-based suites:
Python (Hypothesis)
tests/properties/runs inside the normalpytest tests/invocation: 18 pure properties covering the PIN generator (length/charset/never-unsafe), the provider slot-tag codec (round-trips across canonical/compact/slot-only/legacy formats, parser totality), and theSlotSyncManager.calculate_in_synccontract (all documented branches).RuleBasedStateMachinedrives the realBaseLockwrite orchestration (async_internal_set_usercode/async_internal_clear_usercode) andLockUsercodeUpdateCoordinatoragainstMockLCMLockwith a dict oracle — random interleavings of writes, deletes, external keypad changes, refreshes, and connection faults, asserting convergence, idempotency (WriteResult.NO_CHANGE), duplicate rejection, and fail-loud disconnected writes.dev(default) runs 15 examples per property; CI setsHYPOTHESIS_PROFILE=cifor 200. Full-suite marginal cost is about one second.parse_slot_num(float('inf'))raisedOverflowErrorthrough a helper documented to absorb conversion errors — fixed in this PR.TypeScript (Bombadil)
pbt/contains a static harness that mounts the builtlcm-slotandlcm-lock-codescards with a scripted mockhassand a light-DOM "chaos panel" (external PIN changes, availability/sync flips, slot add/clear), plus a Bombadil spec with temporal-logic properties: masked PINs never appear in the DOM, chip count always matches the model, pushed data eventually renders, suspended state eventually shows its banner — on top of Bombadil's defaults (no uncaught exceptions, console errors, or HTTP errors).yarn test:pbtbuilds the bundle and runs a time-boxed headless exploration locally; a new nightly + manual-dispatch workflow (bombadil.yml) runs 5-minute explorations and uploads the trace on violation. Never runs on PRs.Counterexample triage policy is documented in
AGENTS.md: a found counterexample is a deliverable — fix the code, or narrow a strategy only for documented contract edges, never silence by rerunning.Type of change
Additional information
🤖 Generated with Claude Code
https://claude.ai/code/session_01Wvy9UfQtNuvz1qqbEFYoB7