Skip to content

Fix EXE (Exeter) scraper#437

Open
symroe wants to merge 1 commit into
masterfrom
fix/EXE-scraper
Open

Fix EXE (Exeter) scraper#437
symroe wants to merge 1 commit into
masterfrom
fix/EXE-scraper

Conversation

@symroe

@symroe symroe commented Jul 16, 2026

Copy link
Copy Markdown
Member

What broke

committees.exeter.gov.uk accepts TCP/TLS connections from Lambda but silently drops HTTP requests — wreq times out after 30 seconds with kind: Request, source: TimedOut on the HTML list page (mgMemberIndex.aspx). The same page returns HTTP 200 OK from non-Lambda IPs. This is an application-layer block on wreq's Firefox TLS fingerprint.

A previous fix (PR #387, June 2026) added http_lib = "playwright" but was closed without merging. EXE is still failing as of 2026-07-14 with the same timeout pattern.

What was fixed

  • councillors.py: added http_lib = "playwright" — headless Chromium's Chrome TLS fingerprint bypasses the application-layer block for both the list page and all individual councillor detail page fetches.

Scrape results

Verified from live site (our proxy environment returns 200 OK):

Metric Count
Councillors found 63 (visible on list page)
With email address TBC (Lambda run needed; detail pages confirmed to have mailto: links)
With photo 39 have thumbnails on list page; remaining 24 may have photos on detail pages not checked here

Note: exact email and photo counts require a Lambda run since individual detail pages must be fetched sequentially; a sample detail page (mgUserInfo.aspx?UID=5824) confirmed email and larger photo are present.


Opened by LGSF maintenance bot (2026-07-16)


Generated by Claude Code

committees.exeter.gov.uk times out from Lambda with a wreq request-level
timeout (TCP/TLS succeeds, HTTP request gets no response). The same page
returns 200 OK from non-Lambda IPs. Adding http_lib = "playwright" uses
headless Chromium whose Chrome TLS fingerprint bypasses the application-
layer block that drops wreq's Firefox-fingerprint requests.

Co-Authored-By: Claude <noreply@anthropic.com>
@symroe

symroe commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

Re-scrape after 6062d4e

Added http_lib = "playwright" to bypass application-layer timeout on committees.exeter.gov.uk.

Verified from live site (proxy environment, 200 OK):

Metric Count
Councillors found 63
With email address TBC (Lambda run needed; mailto: links confirmed on sample detail page mgUserInfo.aspx?UID=5824)
With photo 39 have thumbnails on list page (24 may have photos only on detail pages)

Exact email and photo counts will be confirmed after the next Lambda run with Playwright.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants