Conversation
Google now serves clients it classifies as automated a SERP whose organic
hrefs are replaced by encrypted tokens: a protobuf blob only Google can
decrypt, minted fresh on every impression. Captured live, every organic
href on the page looks like this:
<a href="/goto?url=CAESegHrOzAVXrOxTz8MiFgIt6wo...">
ParseHTML returns those verbatim, so on such a SERP every result comes back
with an unusable URL. Because domain, display_url, favicon and the result id
are all derived from that URL, one changed attribute empties the whole
record: `domain` and `display_url` come back "", and `id` — MD5 over the
URL — changes on every request for the same result, since the token is
per-impression. Rank tracking that matches on domain silently stops
matching, and callers that dedupe by id see every result as new.
The destination cannot be recovered from the token itself. But Google
cannot stop showing the user which site a result came from, and it does not:
the breadcrumb in <cite> and the source name in span.VuuXrf are both still
there. This change derives a result's identity from those when the href is
opaque, so link obfuscation degrades one optional field instead of nulling
the record.
Measured on a live fully-wrapped SERP (kept as testdata):
cite alone 7/9 domains
cite + VuuXrf 9/9 domains, zero network calls
cite carries the exact host; the two it misses are Reddit/video blocks that
render a comment count where the breadcrumb would be, and VuuXrf covers
those. Verified against the true destinations by resolving the tokens.
- core: add IsUsableResultURL / IsWrappedGoogleURL. The check is positive
("an absolute http(s) URL that is not a Google redirect") and does not
depend on the token's byte signature, so it survives a re-encoding. It
covers all three shapes seen in the wild: a bare token, an absolute
https://www.google.com/goto?url=… wrapper, and a relative /goto?url=…
one. The middle shape is a valid https URL, so a scheme check alone
passes it and every result then reports domain "www.google.com"; the
relative shape has no host, so a host check alone misses it.
- core: derive domain/display_url/favicon from DomainFromAttribution when
the URL is unusable, and key the result id on domain+title instead of the
per-impression token so identity stops moving between requests.
- core: reshape the engine breadcrumb into the existing display_url form
("techradar.com › vpn › best-vpn") rather than passing through Google's
"https://www.techradar.com › VPN › VPN Services", so one result list does
not mix two shapes.
- google: add Cite and SourceName selectors, populated by both the rod and
goquery paths.
- The brand→domain map is deliberately small. A wrong guess would attribute
a result to a site it does not belong to, which is worse than an empty
domain, so unknown brands stay empty.
Results keep their raw URL: this recovers attribution, it does not invent a
destination.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Google now serves clients it classifies as automated a SERP whose organic hrefs are replaced by encrypted tokens: a protobuf blob only Google can decrypt, minted fresh on every impression. Captured live, every organic href on the page looks like this: