19 — Warn loudly that USE_SITEMAP replaces link-following - #17
Open
MirjamOdile wants to merge 2 commits into
Open
19 — Warn loudly that USE_SITEMAP replaces link-following#17MirjamOdile wants to merge 2 commits into
MirjamOdile wants to merge 2 commits into
Conversation
USE_SITEMAP swaps the spider class for a SitemapSpider with no LinkExtractor, so any spider configured with it against an incomplete sitemap silently loses every content section the sitemap doesn't list — no error, healthy-looking crawl, missing sections — and real sitemaps are routinely incomplete. Until a hybrid crawl exists (documented follow-up), the crawl now warns at start when the spider has follow rules, and CLAUDE.md states the two are mutually exclusive instead of implying they compose. docs/requests/19-use-sitemap-replaces-link-following.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was broken: the
USE_SITEMAPsetting sounds additive ("also read the site's page list") but actually swaps the spider for a sitemap-only crawler with no link-following at all. Real sitemaps are routinely incomplete, so any spider built with this setting silently never crawls the sections the sitemap doesn't list — no error, a healthy-looking crawl, missing content. The manual actively claimed the two compose. (Near-miss in the wild: a sitemap listing 1,410 article URLs but none of the site's 647 research/project pages, reachable only by links.)What it does now: the crawl prints an unmissable warning at launch — "USE_SITEMAP replaces link-following: ONLY sitemap-listed URLs will be crawled; this spider's follow rules are ignored" — and the manual now states the trade-off plainly in all three places it discusses sitemaps. The footgun still exists, but it can no longer fire silently.
Details
cli/crawl.py: warning printed whenUSE_SITEMAProutes to the sitemap spider and the spider also has follow rules.CLAUDE.md: §7.1 + two §5 mentions corrected (they previously read as "sitemap + link-following compose").docs/requests/19-use-sitemap-replaces-link-following.md.Behavior changes
USE_SITEMAPspiders that have follow rules. Nothing about the crawl itself changes — this PR only makes an existing trade-off visible.Verified
Docs + a launch-time message only; no crawl logic touched. Lint/import checked. The underlying behavior it warns about (SitemapSpider has no LinkExtractor) is verified by reading the spider classes and by the near-miss described in the request doc.
Merge note: touches
cli/crawl.py, as do PRs 17/18/21 — merging in numeric order minimizes conflicts.