docs

Author: vdusek

docs/guides/request_loaders.mdx

@@ -136,7 +136,7 @@ The <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> is
 The `SitemapRequestLoader` is designed specifically for sitemaps that follow the standard Sitemaps protocol. HTML pages containing links are not supported by this loader - those should be handled by regular crawlers using the `enqueue_links` functionality.
 :::
 
-The loader supports filtering URLs using glob patterns and regular expressions, allowing you to include or exclude specific types of URLs. The <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> provides streaming processing of sitemaps, ensuring efficient memory usage without loading the entire sitemap into memory.
+The loader supports filtering URLs using glob patterns and regular expressions, allowing you to include or exclude specific types of URLs. By default, the loader also keeps only URLs whose host matches their parent sitemap (`enqueue_strategy='same-hostname'`), matching the `enqueue_links` default. Pass `enqueue_strategy='all'` to disable this filter, or `'same-domain'` / `'same-origin'` for other scopes. The <ApiLink to="class/SitemapRequestLoader">`SitemapRequestLoader`</ApiLink> provides streaming processing of sitemaps, ensuring efficient memory usage without loading the entire sitemap into memory.
 
 <RunnableCodeBlock className="language-python" language="python">
     {SitemapExample}

src/crawlee/_utils/robots.py

@@ -91,11 +91,7 @@ def is_allowed(self, url: str, user_agent: str = '*') -> bool:
         return bool(self._robots.can_fetch(str(check_url), user_agent))
 
     def get_sitemaps(self) -> list[str]:
-        """Get the list of same-host sitemap URLs from the robots.txt file.
-
-        Sitemap entries pointing to a different host than the robots.txt file are filtered out, as required by the
-        robots.txt specification.
-        """
+        """Get the list of same-host sitemap URLs from the robots.txt file."""
         same_host_sitemaps: list[str] = []
         for sitemap_url in self._robots.sitemaps:
             if matches_enqueue_strategy('same-hostname', target_url=sitemap_url, origin_url=self._original_url):