feat(detector): serve detector only from per-session provider bundles; always-PoW fallback - #2744
Merged
Merged
Conversation
HughParry
force-pushed
the
feat/detector-bundle-pool-encryption
branch
3 times, most recently
from
June 22, 2026 14:42
459dece to
b716672
Compare
…lback
Adds the provider-side building blocks for the precomputed, per-session
detector bundle pool (bumblebee-style):
- DetectorBundlePool: load precomputed {id}.js/{id}.json pairs into memory,
uniform-random per-session pick, hot-swap replace() for admin push.
- Redis short-TTL session->bundle mapping (cache:detector:{id}, 60s).
- Frictionless fallback to a real PoW challenge when the pool is initialised
but empty. Dark until the pool is initialised at boot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HughParry
force-pushed
the
feat/detector-bundle-pool-encryption
branch
2 times, most recently
from
June 23, 2026 10:30
525e857 to
85483e7
Compare
…resolution Wires the detector bundle pool end-to-end (Part 2): - Boot: initialise the pool from PROSOPO_DETECTOR_POOL_DIR ONLY when the dir exists (absent => legacy path; present-but-empty => PoW fallback). - Client assign endpoint (POST .../client/detector/assign): picks a random bundle, stores the short-TTL session->bundle binding, returns the obfuscated detector inline; returns useProviderBundle:false when no pool. Excluded from the Prosopo-User header requirement (runs before the account exists). - Admin replace-pool endpoint (hot-swap the in-memory pool). - decryptPayload resolves the assigned bundle (single deterministic decrypt with its own keypair + inner cipher config) and falls back to the legacy key pool. innerConfig threaded through getBotScore + regenerated decode bundles. - Client (customDetectBot): resolves provider, asks for a bundle, loads it via blob import or falls back to the bundled detector; threads detectorSessionId. - types: ApiParams.detectorSessionId, assign/replace paths + bodies. All dark unless a pool dir is provisioned, so existing detection is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HughParry
force-pushed
the
feat/detector-bundle-pool-encryption
branch
from
June 23, 2026 11:14
85483e7 to
41bd528
Compare
HughParry
marked this pull request as draft
June 23, 2026 11:41
…; always-PoW fallback The detector now lives ONLY in the provider-served pool bundles — no bundled/ inlined detector and no legacy detector-key pool. Each session's bundle encrypts everything it produces (score, SIMD readings, behavioural data) with its own RSA keypair + inner ChaCha20-Poly1305 cipher; the provider decrypts each payload with that exact bundle. - Pool is always initialised at boot (missing/empty dir ⇒ empty pool), so the three states collapse to two: bundles present ⇒ per-session serving; no bundles ⇒ always PoW (covers no-pool, empty-pool, and client `detectorUnavailable`). - `detectorSessionId → bundleId` Redis binding resolved at the frictionless hop; the bundleId is promoted onto the durable session record so later hops (SIMD attach, PoW/puzzle/image solution submit) decrypt with the same bundle. - Client: removed the inlined `@prosopo/detector` runtime import (type-only now); when no provider bundle can be obtained/run it signals `detectorUnavailable` and the provider serves PoW. - All server decrypt paths resolve the session bundle and pass its inner cipher; legacy key-pool brute force + env fallback removed. Decrypt failure fails closed. - Session record + schema gain `bundleId`; frictionless request gains `detectorUnavailable`. Tests updated for the bundle-only model. Note: temporary [POOL-DEBUG] logging retained for local validation; strip before merge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HughParry
changed the title
HughParry
marked this pull request as ready for review
June 24, 2026 12:09
… detector dep
Fixes two CI failures from the detector-only-on-providers change:
- e2e: the no-detector PoW fallback built session params from the client's
(empty) token, and sendCaptcha rejects a falsy token ("Session parameters
must be set before sending a pow captcha") → 400 → widget stuck. The bypass
session legitimately has no detection token, so synthesise a unique one.
- lint:refs: `@prosopo/detector` is now a type-only import in
procaptcha-frictionless, so it must be a tsconfig reference, not a runtime
dependency. Removed it from package.json dependencies and the tsconfig
references (type still resolves via the workspace); lockfile updated.
Adds a regression test asserting the no-detector fallback never passes an empty
token to sendPowCaptcha.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sopo/detector
The previous change removed @prosopo/detector from procaptcha-frictionless's deps
and tsconfig references (it's now type-only) — but `typeof import("@prosopo/detector")`
still needs the package's .d.ts, so `tsc --build` failed in CI ("Cannot find module
'@prosopo/detector'") because detector is no longer built as a project reference.
Declare the detector's default-export signature locally from shared @prosopo/types
primitives instead. This satisfies both the ref-linter (no detector dep/ref) and
`tsc --build` (no detector build needed), with no behavioural change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolved conflicts (additive on both sides unless noted): - database/redisCache.ts: keep both the detector-bundle cache methods (cacheDetectorBundle/getDetectorBundle) and main's session-escalation cache methods (cache/get/invalidateCachedSessionEscalation). - types/provider/api.ts (GetFrictionlessCaptchaChallengeRequestBody): adopt main's boundedString(INPUT_LIMITS.*) hardening AND keep the feature's detectorSessionId/detectorUnavailable fields (bounded for consistency; token stays bounded-but-empty-allowed for the detectorUnavailable case). - provider/api/captcha.ts: combine main's express type imports (NextFunction/Request/Response for asyncHandler) with the feature's assignDetectorBundle import. - getFrictionlessCaptchaChallenge/handler.ts: take the feature's prebuilt shortCircuitInput (a superset of main's inline literal — same fields plus requestId + detectorUnavailable). - procaptcha-frictionless/package.json: take main's @prosopo/api 3.5.6 / @prosopo/common 3.1.41; do NOT re-add @prosopo/detector (feature made DetectorType local/type-only). - unit test imports: keep main's AccessPolicyType import + feature's fuller vitest import. - package-lock.json: regenerated (--package-lock-only) from main's lockfile against the merged package.json set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…h, 3-arg detector - remove all [POOL-DEBUG] rollout logging (provider + client, 6 files) - delete orphaned getBehavioralData brute-force decrypt loop + its test (replaced by resolve-by-bundleId single-key decrypt) - detector signature 5-arg -> 3-arg: drop vestigial env + randomProviderSelectorFn params (DetectorType + customDetectBot call site) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conflicts resolved by keeping both sides' additive optional params (feature: detectorSessionId/detectorUnavailable/bundleId; main: currentUrl): - getFrictionlessCaptcha(): detectorSessionId, detectorUnavailable, currentUrl - createSession(): bundleId, currentUrl (incl. realigning the 2nd/non-conflicted call site) - types api-schema, handler destructure, submitPoW escalation: keep both - procaptcha-frictionless package.json: main's version bumps, dropped @prosopo/detector (detector is provider-served, not bundled) - package-lock.json regenerated Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Version-bump-only conflicts: take main's bumped @prosopo/* versions; keep @prosopo/detector omitted (provider-served). package-lock regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Previous regen on macOS dropped the linux @rollup/rollup-* binaries, breaking CI's vite/rollup builds. Reconciled in place from main's lockfile (all platforms preserved), detector dep still removed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Commits a throwaway-key detector-pool bundle (packages/provider/fixtures/detector-pool) and sets PROSOPO_DETECTOR_POOL_DIR for the integration test + frictionless cypress provider startups. Makes frictionless run real scored detection instead of the empty-pool PoW fallback, fixing ipValidation + frictionless.cy.ts. Keys are throwaway/test-only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pool-encryption # Conflicts: # package-lock.json # packages/procaptcha-frictionless/package.json # packages/procaptcha-frictionless/src/customDetectBot.ts # packages/provider/src/api/captcha/getFrictionlessCaptchaChallenge/handler.ts # packages/provider/src/api/captcha/getFrictionlessCaptchaChallenge/shortCircuit.ts # packages/provider/src/api/startProviderApi.ts
Since the detector moved to per-session provider bundles, every decrypt path resolves a bundle's own keypair. Nothing reads the Mongo detector-key pool anymore, so pushing/rotating keys through the admin API was already a no-op — live-looking code with no effect. Remove the whole surface: - admin endpoints UpdateDetectorKey / RemoveDetectorKey (+ their unit tests) - ProviderApi.updateDetectorKey / .removeDetectorKey - ClientTaskManager.updateDetectorKey / .removeDetectorKey - IProviderDatabase.storeDetectorKey / .getDetectorKeys / .removeDetectorKey - the `detector` collection registration, DetectorRecordSchema, DetectorSchema and the DetectorKey type - UpdateDetectorKeyBody / RemoveDetectorKeyBodySpec / UpdateDetectorKeyResponse - rate-limit entries for both admin paths Also corrects the assignDetectorBundle doc comment, which still described a fallback to "the bundled detector and the legacy key-pool decryption path" — both removed earlier in this branch. The `detector` collection is left on disk; no migration drops it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SdWox4Yt4P2sNqESzSY2q9
…-decrypt path Pool push: - dedicated 128 MB body limit on ReplaceDetectorPool (~86 MB for 100 bundles); all other routes keep the 1 MB backstop. Not raised further because express.json buffers into one string and V8 caps strings at 512 MiB (~620 bundles) — past that needs a chunked push, not a bigger number. - a pushed pool is written to the pool dir (staged + renamed so a crash can't leave a partial set) and reloaded at boot, instead of vanishing with the process on any restart. The response reports `persisted` so an in-memory-only push can be alarmed on. - provider1/provider2 each mount their own host volume for the pool. Never bake it into the image: prosopo/provider is public on Docker Hub and every bundle ships its own RSA private key. Separate dirs because the two containers run different image versions. Release stamping: - bundles record the release they were built from; providers skip bundles from another release (PROSOPO_DETECTOR_POOL_RELEASE, unset = no check). The widget no longer carries a detector and runs whatever we serve, so this was the only thing that could catch a pool built off the wrong tag. Failed decryption: - now a first-class decision (DECRYPTION_FAILED, 3 image rounds) evaluated before every other check. decryptPayload substitutes userAgent/userId undefined, baseBotScore 1 and timestamp 0 on failure, so these sessions were being decided by those placeholders: the UA check compares a real hash to undefined and can never match, so they surfaced as USER_AGENT_MISMATCH sized at 6 rounds, and on any sitekey with autoBanScoreThreshold set the score of 1 hard-blocked them with a 401 first. A payload we cannot read is absence of evidence, not evidence of a bot. - timestampDecayFunction drops its now-unreachable decryptionFailed arm. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SdWox4Yt4P2sNqESzSY2q9
provider.Caddyfile had no `encode` directive and Caddy does not compress by default, so every proxied response went out uncompressed. That was cheap when the largest bodies were small JSON replies; it stops being cheap now that /detector/assign returns ~830 KB of obfuscated JS inline, once per frictionless session. Measured on the pool fixture: gzip 2.6x (827 -> 319 KiB), zstd 3.1x (-> 271 KiB) — obfuscated output compresses well because the obfuscator's string-array encoding is highly repetitive. At 15k sessions/hour that takes one provider from ~9.15 TB/month of egress to ~3 TB. Placed above `reverse_proxy` inside the `route` block, where directive order is literal, so it wraps the proxied response. Verified with `caddy adapt` against prosopo/caddy:2.5.7-dns-3455: the encode handler lands immediately before reverse_proxy with prefer ["zstd","gzip"]. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SdWox4Yt4P2sNqESzSY2q9
…pool-encryption # Conflicts: # packages/procaptcha-frictionless/src/customDetectBot.ts
The stage-into-a-sibling-and-rename approach was destructive against the
provider's bind-mounted pool directory. `rmSync(dir, {recursive:true})`
successfully deleted the mount's CONTENTS, then `renameSync` failed with EBUSY
on the mount point itself — so a push wiped the on-disk pool and reported
SUCCESS, with only the `persisted:false` flag hinting anything was wrong. Unit
tests on a tmpdir never hit it because a tmpdir isn't a mount point.
Verified on staging: a 20-bundle push now returns persisted:true, writes 20
js/json pairs (0600) to both provider volumes on all three pronodes, and
reloads count:20 after a container restart.
Write each bundle to a per-file temp name and rename it into place within the
same directory (still atomic, reader never sees a partial bundle), then prune
bundles no longer present. Never touch the directory itself. Two tests pin the
invariant that made this fail: the pool dir's inode is unchanged across a
persist, and no .tmp files are left behind.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SdWox4Yt4P2sNqESzSY2q9
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SdWox4Yt4P2sNqESzSY2q9
Brings the Vite 8 / Vitest 4 upgrade (#2850) and follow-ups into the detector-pool branch. Conflicts: - procaptcha-frictionless/package.json: took main's newer workspace versions but kept this branch's removal of @prosopo/detector — the pool loads bundles dynamically and declares the detector signature locally in detectorLoader.ts. - customDetectBotSimd.test.ts: kept main's vitest 4 fix (ProviderApi mock must be a function expression, not an arrow, because customDetectBot calls it with `new`) and re-added this branch's assignDetectorBundle mock. - package-lock.json: regenerated.
The widget threw on load and never defined window.procaptcha, so every e2e
run timed out waiting for it.
Cause is a Rolldown chunk cycle between the shared browser chunk and
utilCryptoChunk. It only appears on this branch: the detector is no longer
bundled into the widget, which reshapes the module graph enough for the two
chunks to reference each other. Whichever side evaluated first then read the
other's module-scope bindings before they were assigned — first
`init_dist` (the nodePolyfills global shim, declared with `var`, so hoisted
undefined), and behind it `isHex` reading `.test` of an undefined regex.
Putting util-crypto in the shared browser chunk gives both a single
evaluation order. It also preserves the reason that chunk exists: one module
instance, one componentsCache.
Verified by loading the built bundle in headless chromium:
main 26 chunks, 0 cycles, PASS
pool 28 chunks, 1 cycle, FAIL ("init_dist is not a function")
fixed 25 chunks, 0 cycles, PASS
manualChunks was tried first to isolate the polyfill shims into their own
chunk; Rolldown ignores the returned name for modules it has already
co-located, so that had no effect and was dropped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GagGdoZ8Lyj876K5EsDwRb
Conflicts were the procaptcha-frictionless dependency block in package.json and its mirror in package-lock.json: main's release bumps versus this branch having dropped @prosopo/detector as a runtime dependency. Took main's version bumps and kept the detector dependency removed — procaptcha-frictionless imports DetectorType type-only now, so lint:refs requires it be a tsconfig reference rather than a package.json dependency. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RJsacV3Y8wrzn8xexXPyeP
Fallout from merging v3.6.77 — main added tests against an API this branch changed, which git merged cleanly but tsc rejected: - "updates the detector key" / "removes a detector key" cover ProviderApi.updateDetectorKey/.removeDetectorKey and the matching AdminApiPaths entries. This branch removed that whole surface in 4865a7a, since per-session bundles carry their own keypair and the Mongo detector-key pool is gone. Dropped both tests. - "includes the optional context when supplied" passes currentUrl and iframeUrl positionally. This branch inserted detectorSessionId and detectorUnavailable ahead of them, so both arguments were landing on the wrong parameters — iframeUrl onto a boolean, which is what tsc caught. Pass undefined for the two new parameters so the URLs land where the test's assertions expect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RJsacV3Y8wrzn8xexXPyeP
routingFrictionless has failed since the merge that brought it in from main (it was authored in #2876, where no detector pool exists). The spec asserts the frictionless-phase captchaType directly, but this branch's runNoDetectorPowFallback answers on an empty pool at handler.ts:439 — before setRoutingContext at handler.ts:614 — so sendCaptcha sees no routing context and returns the PoW baseline without ever invoking route(). CI confirms it: the provider logs `empty_detector_pool_pow_fallback` three times, then the image and puzzle cases fail with "expected 'pow' to equal 'image'/'puzzle'". The pow case only passed because PoW is already the baseline. Point the step at the committed fixture pool, as the frictionless step already does, so the fallback is correctly declined and the request reaches the routing machine. escalation/postPowPuzzle don't need this — they assert the post-PoW escalation.captchaType and are content to start from PoW. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XJ7KiTDKiu3mxQ4iuLFK2y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.