Skip to content

chore: Fix flaky tests for SitemapRequestLoader #1717

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Pijukatel merged 1 commit into from
Feb 6, 2026
+10 −6
Merged

chore: Fix flaky tests for SitemapRequestLoader #1717

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions tests/unit/request_loaders/test_sitemap_request_loader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,9 @@ async def test_sitemap_traversal(server_url: URL, http_client: HttpClient) -> No

while not await sitemap_loader.is_finished():
item = await sitemap_loader.fetch_next_request()
assert item is not None

await sitemap_loader.mark_request_as_handled(item)
if item:
await sitemap_loader.mark_request_as_handled(item)

assert await sitemap_loader.is_empty()
assert await sitemap_loader.is_finished()
Expand All @@ -85,6 +85,9 @@ async def test_is_empty_does_not_depend_on_fetch_next_request(server_url: URL, h
await sitemap_loader.mark_request_as_handled(item)

assert await sitemap_loader.is_empty()

await asyncio.sleep(0.1)

assert await sitemap_loader.is_finished()


Expand Down Expand Up @@ -192,12 +195,13 @@ def transform_request(request_options: RequestOptions) -> RequestOptions | Reque

while not await sitemap_loader.is_finished():
request = await sitemap_loader.fetch_next_request()
assert request is not None
assert request.user_data.get('transformed') is True

extracted_urls.add(request.url)
if request:
assert request.user_data.get('transformed') is True

extracted_urls.add(request.url)

await sitemap_loader.mark_request_as_handled(request)
await sitemap_loader.mark_request_as_handled(request)

assert len(extracted_urls) == 5
assert extracted_urls == {
Expand Down
Loading