Skip to content

AI: TypeSafe judges ad and tracker hosts; smart capture asks about DNS-only hosts - #5

Merged
Neoo-Blue merged 1 commit into
mainfrom
typesafe-judge
Sep 19, 2026
Merged

Neoo-Blue merged 1 commit into
mainfrom
typesafe-judge

Conversation

@Neoo-Blue

Copy link
Copy Markdown
Owner

What

TypeSafe domain judge. Smart capture and the domain tester's "Ask the assistant" can ask TypeSafe's Jev model (System One API) two typed questions per host: is it ad or tracking infrastructure (noul), and how likely is a DNS block to break something (choice: low / medium / high). It works with no chat model configured. When a chat model is configured, it takes over if TypeSafe fails. Chat, briefs and reviews are unchanged.

  • internal/ai/typesafe.go: client, questions, verdict mapping. Retries 429/5xx with backoff; a bad key stops the whole batch after one request.
  • Config ai.typesafe.{enabled,api_key}: masked in the API, kept on backup restore, and settable from Settings > Assistant > TypeSafe domain judge.
  • A 40% or higher probability of "high" breakage is reported as high, so the host goes to a person instead of being blocked automatically.

Smart capture on DNS-only nodes. The heuristics measure referrers, response sizes and paths, so a host seen only as DNS lookups scored 0 and never reached the model. On the reference Pi node, 11,794 of 11,839 candidates were at 0.0 and nothing had ever been judged.

  • DNS-only candidates are escalated once, while their status is candidate.
  • The verdict stands on its own instead of being blended with that 0, and it is capped below auto_block_score. A verdict on the name alone can only send a host to review, never block it.
  • Review rows with an AI verdict and nothing new to measure keep that verdict for a person.
  • PendingCandidates puts unjudged rows ahead of review rows, so a review backlog cannot starve the 500-row window.
  • A quiet interval no longer erases a host's earlier HTTP evidence (withStoredHTTP).
  • classify() uses errors.As, so wrapped provider errors are recognised.

Behaviour change for nodes without TypeSafe

With smart_capture.use_ai on, DNS-only candidates now reach the chat model too: one fast-model call per pass (40 hosts) while the backlog lasts. The judge prompt now says that a third_party_ratio of 0 with no HTTP evidence means "not measured", not "first-party".

Testing

  • go vet ./... and go test -race ./... pass locally.
  • New internal/ai/typesafe_test.go (httptest) covers verdict mapping, a DNS-only state carrying no zeros, the heuristic score staying out of the state, and a bad key aborting after one request.
  • New internal/adblock/smart_test.go covers DNS-only escalation, the capped final score, review verdicts kept across passes, and HTTP history surviving a quiet pass. Disabling withStoredHTTP makes it fail.
  • Live run over 120 real hostnames from the Pi: AppsFlyer 0.91, Crashlytics 0.85, AdBlock telemetry 0.81; first-party services and status pages below 0.5.
  • Deployed to the Pi as v1.32.1-typesafe. The first pass logged judge: 40/40 TypeSafe verdicts (2 to review, 38 dismissed, 0 blocked).

🤖 Generated with Claude Code

…S-only hosts

TypeSafe's System One API (Jev) answers the domain judge's two questions per host
as typed probabilities: is it ad or tracking infrastructure (noul) and how likely
is a block to break something (choice). It runs with no chat model configured and
falls back to the chat model when it fails. Settings > Assistant > TypeSafe domain
judge holds the switch and the key (masked in the API, kept across backup restore).

Smart capture's heuristics only measure HTTP signals, so a host seen only in DNS
scored zero and was never escalated; on a DNS-only node nothing was ever judged.
Such hosts are now judged once, the verdict stands alone rather than blended with
that zero, and it can only queue for review, never block. Unjudged candidates are
ordered ahead of review rows, a quiet interval no longer erases a host's earlier
HTTP evidence, and classify() unwraps provider errors so a bad key stops a batch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Neoo-Blue
Neoo-Blue merged commit bc080f3 into main Sep 19, 2026
3 checks passed
@Neoo-Blue
Neoo-Blue deleted the typesafe-judge branch September 19, 2026 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant