CMD (Camden) scraper failing — Cloudflare Managed Challenge blocking playwright

[symroe]

Error (2026-07-11)

Page.goto: Timeout 30000ms exceeded.
navigating to "https://democracy.camden.gov.uk/mgWebService.asmx/GetCouncillorsByWard"

Background

Previous issues #310, #337, #343 all covered the same Cloudflare challenge block; #343 was closed 2026-06-26 as "completed" but the scraper is still failing today with the same pattern.

Current state

The scraper has use_proxy = True + http_lib = "playwright". As noted in issue #343: the PROXY_URL is not passed to PlaywrightHTTPClient, so use_proxy = True is silently ignored when playwright is the HTTP client. Playwright connects directly from Lambda and hits the Cloudflare Managed Challenge, which times out after 30 seconds.

From our environment: democracy.camden.gov.uk returns HTTP 403 with a Cloudflare "Just a moment..." managed challenge page (confirmed: cf-mitigated: challenge, cType: 'managed').

Fix patterns ruled out

1. HTTPS migration — already HTTPS
2. verify_requests = False — not a cert issue
3. Removing use_proxy = True — has no effect; playwright already ignores the proxy setting (confirmed in base.py lines 71–74)
4. http_lib = "playwright" without proxy — already set; Cloudflare Managed Challenge requires a non-datacenter IP that can pass Cloudflare's bot score check

What needs to happen

This requires either:

• A framework change to pass proxy_url to PlaywrightHTTPClient so that requests route through a residential/non-datacenter IP (same fix needed as CMD (Camden) scraper failing — Cloudflare Managed Challenge blocking playwright #343 described)
• Or Camden Council to add the Lambda IP range to their Cloudflare allowlist

[symroe]

Still failing — 2026-07-17

Same error, same pattern:

Page.goto: Timeout 30000ms exceeded.
navigating to "https://democracy.camden.gov.uk/mgWebService.asmx/GetCouncillorsByWard", waiting until "networkidle"

Duration: 34.7s. The Cloudflare Managed Challenge is still blocking playwright running directly from Lambda's datacenter IP. No change since the last report — still requires a framework-level fix to route playwright through a non-datacenter IP, or Camden to allowlist Lambda's IP range.

---

Generated by Claude Code

[symroe]

2026-08-17 maintenance check — still failing

As of the latest run (2026-08-15), CMD continues to fail. PR #448 is open for this council, but the site is still blocking.

Error (2026-08-15T20:23:17Z):

playwright._impl._errors.TimeoutError: Page.goto: Timeout 30000ms exceeded.
Call log:
  - navigating to "https://democracy.camden.gov.uk/mgWebService.asmx/GetCouncillorsByWard", waiting until "networkidle"

The Cloudflare Managed Challenge on Camden's ModGov instance continues to block the Playwright request before the page loads. The 30-second timeout is hit on every attempt. This is consistent with Cloudflare serving a JS challenge that Playwright doesn't resolve in time from Lambda's IP range.

---

Generated by Claude Code