fix(audit): analyze XHTML and mixed-case HTML content types

[kevin9327]

Problem

crawlPage asks for two document types and accepts one:

const response = await fetch(url, {
  headers: {
    "User-Agent": CRAWL_USER_AGENT,
    Accept: "text/html,application/xhtml+xml",
  },
  ...
const contentType = response.headers.get("content-type") ?? "";
const isHtml = contentType.includes("text/html");

application/xhtml+xml never contains text/html, and the substring test is
case-sensitive, so Content-Type: TEXT/HTML fails it too — media types are
case-insensitive per RFC 9110.

isHtml === false is not a small mislabel. The body is never handed to
analyzeHtml, so the row is persisted with an empty title, empty meta
description, no headings, no images, no structured data and no links, and
runPageReporters returns early on !page.isHtml — the page reports no
issues at all. Because no links are extracted, nothing is added to the
frontier either: on a site that serves XHTML the audit stops at the start URL.

Fix

Lowercase the header and accept the second type the request already asked for.

Tests

New site-audit-workflow-helpers.test.ts, stubbing fetch the way
scrape.test.ts does. Three content types that must be analyzed, and one that
must not — application/pdf is the pin that this is not a widening; it passes
before and after.

Against main:

 ✓ analyzes a page served as text/html; charset=utf-8
 × analyzes a page served as application/xhtml+xml; charset=utf-8
 × analyzes a page served as TEXT/HTML
 ✓ records a non-HTML document without analyzing it

AssertionError: expected { isHtml: false, title: '', …(1) } to deeply equal { isHtml: true, …(2) }
-   "isHtml": true,
-   "linkCount": 1,
-   "title": "A page title",
+   "isHtml": false,
+   "linkCount": 0,
+   "title": "",

Tests  2 failed | 2 passed (4)

With the fix:

Tests  4 passed (4)

How I tested

Windows 11, Node 22.

npx vitest run src/server/workflows/   → 3 files, 9 tests passed
npx tsc --noEmit                       → clean
npx prettier --check <the two files>   → clean
npx oxlint src/server/workflows/       → 0 warnings, 0 errors

I did not run the whole suite or e2e.

Scope

Only the two types the crawler advertises. I did not add application/xml or
text/xml: those are not requested, and a sitemap or feed answering an
Accept: text/html request is not a page the analyzer should read.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.