Process completed crawl pages during the crawl

[PrzemyslawKlys]

Crawl consumers process extracted pages as they arrive through HtmlCrawlOptions.PageObserver and IHtmlCrawlPageObserver. The crawler awaits the callback after extraction and optional asset downloads, before committing the page's checkpoint. It delivers successful and failed result pages in crawl order; skipped candidates and checkpoint-loaded pages are excluded.

Invoke-HtmlCrawl -StreamPages sends these pages to the PowerShell pipeline instead of returning the final crawl result. Default output stays the same. Callback failures and cancellation propagate, and external sinks should tolerate a retry of an uncommitted callback. Pipeline output acknowledges queueing rather than completion of downstream processing; export files may be committed after a page is emitted.

This layer builds on streamed chunk export #497. Content release #501 uses the observer to keep streamed copies available while releasing the crawl's retained bodies. The stack order is #497, #492, then #501.

Validation: the rebased layer passes 10 related observer, runspace and structure checks on .NET 10 and .NET 8, and eight on .NET Framework 4.7.2. Both public pipeline cases pass in PowerShell 5.1 and 7, including a response held until the first page reaches the pipeline. The standalone callback contract has an independent source review; the subsequent content-release interaction was also reviewed. Current CI remains a merge gate.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.54%. Comparing base (62b4c2f) to head (0668a14).

Additional details and impacted files

@@                      Coverage Diff                       @@
##           perf/crawl-streamed-chunks     #492      +/-   ##
==============================================================
+ Coverage                       58.19%   58.54%   +0.35%     
==============================================================
  Files                             488      488              
  Lines                           36107    36117      +10     
  Branches                         7284     7286       +2     
==============================================================
+ Hits                            21013    21146     +133     
+ Misses                          12728    12576     -152     
- Partials                         2366     2395      +29     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.