Fix MAV (Malvern Hills) scraper#435
Open
symroe wants to merge 1 commit into
Open
Conversation
…ov.uk The scraper was hitting http://moderngov.malvernhills.gov.uk which returns an HTTP 301 redirect to HTTPS. wreq does not follow cross-protocol redirects, so it received the redirect body (HTML "Document Moved" page) instead of the ModGov XML, resulting in 0 councillors found. The HTTPS endpoint is fully operational — confirmed: GET returns valid XML (31 councillors, 29 with email, 31 with photo).
Member
Author
Re-scrape after cbabd42Changed
2 councillors have no email address in the XML. All 31 have Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
moderngov.malvernhills.gov.ukonly serves its ModGov endpoint over HTTPS, butmetadata.jsonpointedbase_urlathttp://. When wreq sends a GET to the HTTP URL it receives an HTTP 301 redirect body (<h1>Object Moved</h1>…) rather than the ModGov XML — wreq does not follow cross-protocol redirects. The framework parses this HTML as XML, finds no<councillor>elements, and raisesValueError: Not many councillors found (0). The HTTPS endpoint is fully operational:GET https://moderngov.malvernhills.gov.uk/mgWebService.asmx/GetCouncillorsByWardreturns valid councillor XML with 31 councillors.What was fixed
metadata.json: changedbase_urlfromhttp://moderngov.malvernhills.gov.uktohttps://moderngov.malvernhills.gov.ukScrape results
Verified from the live HTTPS endpoint:
Generated by Claude Code