Fix EXE (Exeter) scraper#437
Open
symroe wants to merge 1 commit into
Open
Conversation
committees.exeter.gov.uk times out from Lambda with a wreq request-level timeout (TCP/TLS succeeds, HTTP request gets no response). The same page returns 200 OK from non-Lambda IPs. Adding http_lib = "playwright" uses headless Chromium whose Chrome TLS fingerprint bypasses the application- layer block that drops wreq's Firefox-fingerprint requests. Co-Authored-By: Claude <noreply@anthropic.com>
Member
Author
Re-scrape after 6062d4eAdded Verified from live site (proxy environment, 200 OK):
Exact email and photo counts will be confirmed after the next Lambda run with Playwright. Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
committees.exeter.gov.ukaccepts TCP/TLS connections from Lambda but silently drops HTTP requests —wreqtimes out after 30 seconds withkind: Request, source: TimedOuton the HTML list page (mgMemberIndex.aspx). The same page returns HTTP 200 OK from non-Lambda IPs. This is an application-layer block on wreq's Firefox TLS fingerprint.A previous fix (PR #387, June 2026) added
http_lib = "playwright"but was closed without merging. EXE is still failing as of 2026-07-14 with the same timeout pattern.What was fixed
councillors.py: addedhttp_lib = "playwright"— headless Chromium's Chrome TLS fingerprint bypasses the application-layer block for both the list page and all individual councillor detail page fetches.Scrape results
Verified from live site (our proxy environment returns 200 OK):
mailto:links)Opened by LGSF maintenance bot (2026-07-16)
Generated by Claude Code