Release persisted crawl page bodies on request

[PrzemyslawKlys]

Set HtmlCrawlOptions.RetainPageContent = false or use Invoke-HtmlCrawl -ReleasePageContent to release HTML, text, Markdown and HTTP cache bodies after their full records have been saved.

$crawl = Invoke-HtmlCrawl -Url 'https://example.com/docs/' -OutPath './crawl' -IncludeMarkdown -ReleasePageContent

The result keeps metadata and structured data. Exports hydrate stored bodies one page at a time, and the final manifest remains self-contained. Resume, conditional refresh, copied datasets and repeated saves preserve content. Private page identities keep reordered pages and cached skipped records attached to their own stored bodies.

With -StreamPages, emitted copies keep their content while the crawl releases its retained copies. A C# observer can call page.CreateSnapshot() during the callback to retain the current strings. It is a shallow copy: collections and structured data remain shared. It neither reloads released content nor keeps a hidden dependency on its source content file.

The option requires an output or resume path. Keep the stored dataset available while using a result with released bodies, or load it with HtmlCrawler.LoadResultAsync for an editable result with retained content. Metadata, deduplication data and queued streamed copies still consume memory; loading a manifest or refresh source can materialize its bodies.

Merge streamed chunk export #497, then page observers #492, before this layer. The updated stack passes 50 focused .NET 10 cases and all three streaming cases on each PowerShell version. Earlier validation covers persistence, refresh and resumed checkpoints on all three supported .NET targets, with mutation proof and independent content-lifetime review. Current hosted CI remains a merge gate.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-06T04:22:01.372863Z	a071c44	New commits
🔒 Security Review	✅ Completed 2026-10-05T21:47:27.081362Z	c401849	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[codecov]

Codecov Report

❌ Patch coverage is 97.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.09%. Comparing base (17885ed) to head (1103435).

Files with missing lines	Patch %	Lines
Sources/HtmlTinkerX/HtmlCrawler.StoredContent.cs	94.87%	1 Missing and 1 partial ⚠️
Sources/HtmlTinkerX/HtmlCrawler.Persistence.cs	95.23%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@                       Coverage Diff                       @@
##           feature/crawl-page-observer     #501      +/-   ##
===============================================================
+ Coverage                        58.97%   59.09%   +0.12%     
===============================================================
  Files                              490      491       +1     
  Lines                            36178    36267      +89     
  Branches                          7301     7322      +21     
===============================================================
+ Hits                             21335    21433      +98     
+ Misses                           12413    12406       -7     
+ Partials                          2430     2428       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b6893d41e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".