Skip to content

Preserve sitemap and page publication dates - #503

Open
PrzemyslawKlys wants to merge 3 commits into
v2-speedygonzalesfrom
feature/sitemap-entry-dates
Open

PrzemyslawKlys wants to merge 3 commits into
v2-speedygonzalesfrom
feature/sitemap-entry-dates

Conversation

@PrzemyslawKlys

@PrzemyslawKlys PrzemyslawKlys commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Source discovery needs reliable dates before it selects or orders pages. This adds sitemap entries with modification dates while preserving document order and keeping child sitemaps distinct from page URLs. The existing URL-only API shares that parser and excludes image-extension locations.

HtmlPublicationDates.Parse reads declared publication and modification dates from page metadata and matching Article/WebPage JSON-LD. It keeps those dates separate, rejects other-page identities, and retains unambiguous dates across matching graph records. Structured crawler metadata preserves these dates after article selection removes the document head.

A private downstream consumer uses the APIs for bounded discovery and source chronology. Its package repin follows the HtmlTinkerX release containing these changes.

Validation: 17 discovery-parser tests and 29 publication/structured-metadata tests pass on each of net472, net8.0, and net10.0. Independent read-only reviews covered both contracts; all actionable findings were fixed and confirmed. Downstream selector integration and the full 122-test consumer suite pass against a local candidate package. A live source check also confirms that language-query URLs retain dates declared for their canonical article URLs. Repository CI runs on this PR.

@codecov

codecov Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.64516% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.19%. Comparing base (0b55f02) to head (729e322).

Files with missing lines Patch % Lines
Sources/HtmlTinkerX/HtmlPublicationDates.cs 80.00% 0 Missing and 8 partials ⚠️
Sources/HtmlTinkerX/HtmlDiscoveryParser.cs 81.25% 1 Missing and 2 partials ⚠️
...rces/HtmlTinkerX/HtmlCrawler.StructuredDocument.cs 66.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@                  Coverage Diff                  @@
##           v2-speedygonzales     #503      +/-   ##
=====================================================
+ Coverage              58.05%   58.19%   +0.14%     
=====================================================
  Files                    486      488       +2     
  Lines                  36054    36108      +54     
  Branches                7249     7272      +23     
=====================================================
+ Hits                   20930    21013      +83     
+ Misses                 12757    12713      -44     
- Partials                2367     2382      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@PrzemyslawKlys PrzemyslawKlys changed the title Expose sitemap entry modification dates Preserve sitemap and page publication dates Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant