Fix OLD (Oldham) scraper#439
Open
symroe wants to merge 1 commit into
Open
Conversation
… block committees.oldham.gov.uk's WAF rejects wreq's Firefox133 TLS fingerprint with a connection reset. Python requests (OpenSSL TLS) is not blocked. The endpoint is a plain XML API (ModGov ASMX) with no JavaScript or bot-detection that would require browser emulation, so requests is appropriate here. playwright was also tried and failed (Chromium headless TLS fingerprint also rejected by this WAF), confirming requests as the correct fallback.
Member
Author
Re-scrape after 16b5c4cInitial fix: added
Photos are sourced from Generated by LGSF maintenance bot (2026-07-19) Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
committees.oldham.gov.ukruns a WAF that rejects wreq's Firefox133 TLS fingerprint with a TCP connection reset. From Lambda, this manifests as a TCP connection timeout (harder block on Lambda's IP range). The ModGov ASMX endpoint is fully accessible to Python's standard OpenSSL TLS stack and to browser clients, but wreq's TLS client-hello is actively refused.http_lib = "playwright"was tried first (Chromium headless TLS fingerprint) and also failed withnet::ERR_CONNECTION_RESET, indicating this WAF blocks both Firefox133 and headless Chromium fingerprints but accepts Python's OpenSSL TLS stack.What was fixed
councillors.py: addedhttp_lib = "requests"— Python requests uses OpenSSL's standard TLS handshake which passes the WAF, and the ModGov ASMX endpoint returns plain XML with no JavaScript or interactive content that would require browser emulation.verify_requests = Falseretained from the previous fix (self-signed/untrusted cert on this host).Scrape results
Verified from live endpoint (
committees.oldham.gov.uk/mgWebService.asmx/GetCouncillorsByWard):Opened by LGSF maintenance bot (2026-07-19)
Generated by Claude Code