Skip to content

Fix STY (South Tyneside) scraper#436

Open
symroe wants to merge 1 commit into
masterfrom
fix/STY-scraper
Open

Fix STY (South Tyneside) scraper#436
symroe wants to merge 1 commit into
masterfrom
fix/STY-scraper

Conversation

@symroe

@symroe symroe commented Jul 15, 2026

Copy link
Copy Markdown
Member

What broke

www.southtyneside.gov.uk/article/13598/councillors-a-to-z is protected by a Cloudflare managed challenge (cf-mitigated: challenge, server: cloudflare). wreq's Firefox TLS fingerprint receives HTTP 403 with a "Just a moment…" JS-challenge page rather than the councillors HTML. The scraper had no http_lib set so wreq was used directly — same pattern as EAT (#409), FEN (#399), and others recently fixed. A previous issue (#385, closed as completed) described the same problem and proposed this fix.

What was fixed

  • councillors.py: added http_lib = "playwright" — headless Chromium executes the Cloudflare JS challenge, acquires cf_clearance, and then fetches both the list page and individual councillor detail pages. The existing custom HTML selectors (#COUNCILLORSLISTBYNAME_HTML, tbody td a) and the decode_cfemail helper are unchanged.

Scrape results

Metric Count
Councillors found TBC (Lambda run needed — Cloudflare blocks non-browser clients from this environment)
With email address TBC
With photo TBC

The councillors page serves valid HTML to browser clients (confirmed: 403 is the Cloudflare challenge, not a permanent block). Previous successful run (before the Cloudflare block was added) returned councillors with emails decoded via cf_email and photos from the info table.


Generated by Claude Code

… challenge

https://www.southtyneside.gov.uk/article/13598/councillors-a-to-z returns
HTTP 403 with cf-mitigated: challenge and server: cloudflare headers. wreq's
Firefox TLS fingerprint triggers the Cloudflare managed challenge. Playwright
uses headless Chromium which executes the JS challenge and acquires cf_clearance
before fetching the page. The existing custom HTML selectors (#COUNCILLORSLISTBYNAME_HTML,
tbody td a) remain unchanged — they were working before the Cloudflare block.
@symroe

symroe commented Jul 15, 2026

Copy link
Copy Markdown
Member Author

Re-scrape after 952a771

Added http_lib = "playwright" to bypass Cloudflare managed challenge on www.southtyneside.gov.uk. Cloudflare returns HTTP 403 with a JS challenge to all non-browser clients from this environment, so exact counts require a Lambda run.

Metric Count
Councillors found TBC (Lambda run needed)
With email address TBC
With photo TBC

cf-mitigated: challenge confirmed from curl -sI — Playwright will execute the JS challenge and acquire cf_clearance before fetching the councillors list and detail pages.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant