Repository navigation
Alpha 2 features: wildcard DNS rules, visible blocks, silent pin learning, list fixes - #7
Merged
Merged
Conversation
A page that treats a failed request as blocked, such as an ad-block test, counted the proxy's empty 403 as loaded. A request the filter engine blocks (and, in absolute form, a request to a DNS-blocked host) is now answered by its Sec-Fetch-Dest header, which browsers send and apps do not: - document (a top-level navigation): a short plain-text 403 page, not cached, saying that Tollgate blocked it. - any other value (script, image, empty for fetch and XHR, iframe, websocket, ...): no response, as on a blocked host's connection: the HTTP/2 stream is reset with INTERNAL_ERROR and the connection stays open, HTTP/1.1 closes the connection. - no header (apps): the empty 403 with Access-Control-Allow-Origin, as before, so SDKs that retry network errors do not loop. Stats and blocked-log events are unchanged. A request on a connection whose host has become passed through now closes the connection also when it gets no response, as the 403 it replaces did.
detect_format in the filter crate tells a hosts file from adblock syntax by the majority of a list's rule lines: an address followed by a name, or a single host name, is a hosts line; anything else is adblock syntax. Comments and blank lines are left out, and a list without rule lines, or with as many of each, gives no verdict. The FFI exports it as detect_list_format, so the app can correct a custom list added with the wrong type. On the default lists no adblock rule line has a hosts shape and every StevenBlack rule line has one.
StevenBlack hosts is a fifth built-in list, on for existing installs too, since lists.json stores only the lists switched off. The updater records the built-in list ids; when they change, it marks a pending settings change, so the next launch, foreground or background refresh downloads the new list and compiles it from the cached copies instead of waiting for the daily update. Installs from before the record count as changed once. A custom list with the same address as an enabled built-in list is skipped, and its row in Settings says so, so the owner can delete it. A custom list whose content disagrees with its type is compiled as the type its content has: a hosts file under Request rules or Domain rules becomes a Hosts file, adblock rules under Hosts file become Domain rules. After a successful compile the corrected type is saved in lists.json (unless the owner changed that list meanwhile), the list screen reloads it, and a warning line says what was changed.
Turning HTTPS filtering on used to clear every learned pin, to forget hosts learned while the certificate was untrusted. Pins learned with a trusted certificate were lost too whenever the owner switched filtering off and on, and each pinning app then failed again until relearned. The app now records the reason in UserDefaults when it turns filtering off itself because iOS does not trust the certificate (the trust check and the check after enabling). Only then does the next enable clear the pins, through the same restart as before (Connect On Demand paused, tunnel stopped, file removed, then started again), and removing the file clears the record. If the file cannot be removed, the record stays and the next enable tries again.
Some pinning apps refuse an intercepted connection without a TLS alert: the X app for iOS and Messenger cancel it inside their certificate check, and the socket just closes, sometimes after a close_notify (device log of 2026-09-30: X hung up 13 times on one host within 5 seconds, Messenger 147 times within 45). Pin learning only listened for alerts, so such apps failed on every attempt. The intercepted client stream is now wrapped in a FlightWatch that follows the TLS records the proxy writes, to tell when the record carrying our certificate has been written in full (not after a HelloRetryRequest alone, and never for a resumed session), and whether the client was still connected after that. A handshake that then ends with end of stream, a reset or close_notify is a silent refusal: it is counted (tls_silent_refusals, also in the FFI stats) and reported to the policy. The policy learns a silent refusal pin under its own, stricter rule, since clients also hang up for other reasons (a race lost to another network, suspension, a preconnect they did not need): refusals of the host in three different seconds within a minute, none while a client completed a handshake with our certificate for the host in the last minute, and none from a burst: refusals on four hosts within 10 seconds forget the pending refusals, take back silent pins from that window and suppress silent learning for a minute, restarted while the burst lasts. A network change drops silent pins from the last minute and the pending refusals. Silent pins are saved and listed like the others. Alert learning is unchanged.
The tunnel copies the core's new tlsSilentRefusals counter into TunnelStats, and the Home screen's Diagnostics section shows it under Certificate rejections, so an app that fails without an alert can be spotted.
A passthrough host's CONNECT waited for the DNS lookup and the TCP connection before its 200 (about 40 ms on the device). It is now answered at once and the host is dialed in the spawned task while hyper hands over the client connection, holding the passthrough slot as before. When the dial fails, the client connection is closed and the client sees its TLS handshake fail, instead of getting 502. Locally, with a 40 ms lookup, CONNECT to 200 went from 41.5 ms to 0.2 ms; CONNECT to a completed TLS handshake with the origin went from 41.9 ms to 41.8 ms, since the handshake still needs the upstream.
A DoH connection idle for over 30 s was dropped, because after device sleep it is probably dead, so the first lookup after a pause paid a new TCP and TLS handshake (about 40 ms to Cloudflare and 60 ms to Quad9 measured here). The resolver now tells sleep from idle time with tollgate_common::clock::Reading, which reads a clock that counts sleep and one that does not (CLOCK_MONOTONIC_RAW and CLOCK_UPTIME_RAW on Apple platforms, CLOCK_BOOTTIME and CLOCK_MONOTONIC on Linux). A connection is used again after up to 120 s (MAX_IDLE) of idle time while awake; any sleep over 1 s (MAX_SLEEP) still forces a new connection. Nothing is sent to keep connections open. Measured from here: queries on Cloudflare connections idle for 30 to 240 s were all answered, and one idle for 600 s failed at once although Cloudflare left another open for over 20 minutes; Quad9 answered after 30 to 170 s but once failed at once after 90 s, and closed idle connections after 180 s, which the client sees. A query that fails at once is retried on a new connection, so only a connection lost silently costs the warm deadline. TLS sessions were already resumed by rustls' default session cache; a resumed TLS 1.3 handshake takes the same round trip, so nothing changes there.
The AdGuard DNS filter blocks families of hosts with 407 wildcard rules such as ||log*.example.com^ and ||tracking.*.example.com^, which the parser skipped, so those hosts resolved and connected. A rule with a * in its name now becomes a pattern that means what adblock means for a request to https://host/: || starts the match at the host or one of its labels, | and :// at the host, no anchor anywhere in it, ^ ends it at the end of the host, and without a caret it may end anywhere. Wildcard exceptions are parsed the same way. Every kind of exception wins over a wildcard block, and $important is not supported on patterns. domains.bin moves to format version 2: the header grows to 40 bytes and holds the length of a wildcard block section after the exception section. Version 1 files still load, because after an app update the tunnel keeps the previous compile until the lists are compiled again. Each pattern is filed under its literal head or tail in a trie, so a lookup matches only the patterns whose literal fits the host. On the phone's DNS lists a lookup of a host no list blocks takes about 100 ns longer, and one the hashes block about 250 ns less, since wildcard exceptions are no longer tried one by one.
# Conflicts: # core/crates/tollgate-ffi/tests/swift_bindings.rs
A build that reads the lists differently (the wildcard section of domains.bin, a new built-in list, hosts-file detection) left existing installs on the files the previous build compiled until the daily update. The app now records its build number at launch and, when another build ran before, compiles the lists from the cached copies at the next opportunity, as it already does when the built-in lists change.
Adds E23 to E30 to the alpha 2 checklist and a design-spec revision for wildcard DNS rules and domains.bin version 2, the recompile after an update, answers to blocked requests, passthrough answered first, DoH reuse, silent certificate refusals, the fifth built-in list, hosts-file detection and when learned pins are cleared. Updates the checks that expected a 502 from an unreachable passthrough host or a 403 for requests a URL rule blocks in Firefox.
DNS lists in the wildcard domains format write one *.name per line for a name and its subdomains. Since wildcard rules became patterns, each such line was the pattern *.name*, which has no literal head or tail, so every lookup tried all of them: a list of 75,000 cost about 1 ms per DNS query and proxied CONNECT and 5 MiB of the tunnel's heap, twice during a reload, and *.name* also matched a.name.other.example. An unanchored *.name, with or without a caret and with no other *, is now the hashed name, like ||name^, which blocks name itself too, as the adblock editions of those lists do. *.zip, whose name has no dot, is skipped like ||zip^. A compile also keeps at most 4,096 patterns, 64 of them with a * at both ends (the AdGuard DNS filter has 417 and 1): exceptions first, since leaving one out would block what a list unblocks, then blocks in list order. The rest are counted as skipped lines and logged.
A host was learned from silent refusals in 3 different seconds within 60 s, and pending refusals were kept no longer. A pinning app may reach a second host only once or twice each time it is used: the X app hung up on its second pinned host in two seconds only, 4 s apart, so that host was never learned however often the app was opened. Refusals are now counted over REJECTION_WINDOW_SECS (10 minutes), the window of the alert rule, and the next use of such an app completes the count. A client's successful handshake with our certificate now guards its host for the same 10 minutes, since a browser can use one connection for minutes without a new handshake and its rare hang-ups would otherwise add up. Refusals within one second still count once, and a success, a burst, a network change and forgetting a host still clear the pending refusals. SILENT_WINDOW_SECS is now SILENT_RECENT_SECS: the 60 s in which pins learned from silent refusals are taken back on a wake or a network change.
The X app offers an X25519 share beside X25519MLKEM768, which the proxy does not support, so the proxy answers with a ServerHello for X25519 at once; the device log shows a 122-byte ServerHello and no second ClientHello on every refused connection. The flight watch docs and the X-like test client claimed it got a HelloRetryRequest, so the test that learns a pin end to end went through a handshake X never performs. The X-like client now sends the X25519 share as the hybrid's component, and a test checks that it gets a full handshake on X25519. A separate client that offers only the unsupported share still forces a HelloRetryRequest, for the hang-up before the certificate, and a new test counts a hang-up after the certificate that follows one.
A custom list with the address of an enabled built-in list was skipped whatever file it fed. A copy of EasyPrivacy typed Domain rules, which puts its hosts in the DNS blocklist where the built-in request list puts none, lost them after the update and was labelled the same list. Only a custom list that feeds the same file as the built-in list is now a duplicate; a hosts file typed Request rules is compiled once more beside the built-in list, its type is corrected, and it is skipped from the next run. A custom StevenBlack list was skipped as soon as StevenBlack became built in, while the built-in list had no cached copy yet, so if its first download failed the compile went ahead without either and about 34,000 hosts blocked before the update were not until a later update. A built-in list with no cached copy now starts from the cached copy of a custom list with its address, of any type, so that compile needs no download and a failed one falls back to it; the next successful download replaces it. E23 now expects the same number of enabled lists when Added by you has a StevenBlack list, which is no longer counted, and describes that list's first compile after the update.
E24 step 3 builds a name a rule does not match by putting a letter before the first label, which still matches rules that start with ||*, and it cannot tell a pattern apart from a block of the domain after the last * that the list also has. The step now asks for rules that start with a letter or digit after || and whose domain is not blocked on its own. E25 step 3 expected a page saying it was not found, while the address answers 404 with the site's usual page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Eight follow-ups from alpha 1, built on four parallel branches, merged, then reviewed adversarially (four reviewers; nine confirmed findings fixed).
Blocking
*only in exceptions, so 407 block rules in the AdGuard DNS filter were skipped. They now apply with adblock's meaning:||at a label boundary,*any run of characters including dots,^at the end of the host.domains.binmoves to format 2 with an indexed pattern section; format 1 files still load. Patterns are capped at 4,096 (64 of them open at both ends), and*.namelines from wildcard-domain lists are read as plain names. A differential test against adblock-rust agrees on 99% of 336,000 generated hosts; each difference is a known adblock-rust matching limit, where this follows the list authors' meaning.Sec-Fetch-Dest: documentgets a short page saying Tollgate blocked it; any other browser request gets no response (HTTP/2 stream reset, HTTP/1.1 close), as with an in-browser blocker. Requests without the header, from apps, keep the empty 403, so SDKs that retry network errors do not loop.Apps that refuse Tollgate's certificate
Latency
200now arrives at once; the connection is usable no sooner, since TLS still waits for the server.Docs
docs/experiments/alpha2.md, and an alpha 2 revision in the design spec covering these decisions.Testing
cargo fmtandcargo clippy -D warningsare clean.🤖 Generated with Claude Code