Skip to content

Fix rendered crawl startup, document capture, and CI fixtures - #487

Open
PrzemyslawKlys wants to merge 10 commits into
v2-speedygonzalesfrom
fix/crawl-document-capture
Open

PrzemyslawKlys wants to merge 10 commits into
v2-speedygonzalesfrom
fix/crawl-document-capture

Conversation

@PrzemyslawKlys

@PrzemyslawKlys PrzemyslawKlys commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Rendered crawls prepare their browser session before the first navigation, so request routes and initialization scripts apply to the destination. Capture HTML, title, URL and document identity together after interactions, and retry a replaced execution context within the existing capture deadline. A committed non-HTTP destination is skipped without carrying over the previous response metadata. Public session navigation and configured form login retain their existing behavior.

The HTTP and browser fixtures now wait for complete request headers and observe when a stalled read has started before testing cancellation. Unknown-length response-limit checks use a generated stream, and the PowerShell fixtures use syntax supported by Windows PowerShell's C# compiler. The script recipe fixture gives its script step the tested 100 ms limit while page navigation uses the normal deadline. This includes the fixture fixes from #494 and #496 so CI can exercise them with the browser correction.

Validation: all 1,573 tests pass in the full .NET 10 suite. The HTTP, structure and split-packet header cases pass on .NET 10, .NET 8 and .NET Framework 4.7.2. All 25 public crawl cases, both public routing/login-isolation cases and three recipe timeout/navigation/error cases pass in PowerShell 5.1 and 7.

@codecov

codecov Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.45455% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 58.08%. Comparing base (0b55f02) to head (bd7bb25).

Files with missing lines Patch % Lines
Sources/HtmlTinkerX/HtmlCrawler.Browser.cs 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@                  Coverage Diff                  @@
##           v2-speedygonzales     #487      +/-   ##
=====================================================
+ Coverage              58.05%   58.08%   +0.03%     
=====================================================
  Files                    486      487       +1     
  Lines                  36054    36021      -33     
  Branches                7249     7252       +3     
=====================================================
- Hits                   20930    20923       -7     
+ Misses                 12757    12734      -23     
+ Partials                2367     2364       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@PrzemyslawKlys
PrzemyslawKlys force-pushed the fix/crawl-document-capture branch from abd51a0 to c0213f7 Compare October 5, 2026 22:01
@PrzemyslawKlys PrzemyslawKlys changed the title Capture rendered crawl content from the current document Prepare crawler sessions before navigation and capture the current document Oct 5, 2026
@PrzemyslawKlys PrzemyslawKlys changed the title Prepare crawler sessions before navigation and capture the current document Fix rendered crawl startup, document capture, and CI fixtures Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant