Write per-crawl stats: HTTP status counts + sitemap accounting for auditing

[MirjamOdile]

Problem

Assessing a fleet's health needs each crawl's page liveness (HTTP status mix)
and a coverage denominator (how many URLs the sitemap declared, how many were
rule-eligible). Sampling that after the fact is slow (re-fetching ~1000 URLs
per spider) and drifts — the sitemap you fetch today isn't the one the crawl
saw. Scrapy already counts every response status during the crawl; the data
just isn't persisted anywhere.

Change

Every completed production crawl persists its own numbers to
data/<project>/_audit/crawl_stats/<spider>.json:

• BaseDBSpiderMixin.closed() dumps item count, request count, and the HTTP
  status histogram Scrapy already tracks (filtered on the
  downloader/response_status_count/ prefix so nothing else leaks in).
• Sitemap spiders additionally count, while parsing, the sitemap's page URLs
  (sitemap_total) and those that survive date/deny filters AND match an
  allow rule (eligible) — the coverage denominator, recorded from the real
  crawl at the crawl's own point in time. <sitemapindex> entries (sub-sitemap
  refs) are never counted as pages.

Guards, so a wrong number can never masquerade as a right one:

• Item-capped runs never write — skipped by the CLOSESPIDER_ITEMCOUNT
  setting, not the close reason: a --limit/health run that finishes UNDER
  its cap still closes with reason == "finished" and must not overwrite a
  real crawl's stats.
• Resumed (checkpoint) runs never claim a denominator — a resume restores
  the request queue from disk but every in-memory counter restarts at zero.
  Detection reads the scheduler's own persisted state
  (JOBDIR/requests.queue/active.json — non-empty exactly when a run
  continues an interrupted crawl); the writer then withholds
  sitemap_total/eligible and stamps "resumed": true, keeping leg-only
  items/status.
• All write errors are caught — stats-writing can never fail a crawl.

Verification

17 unit tests: writer happy-path and negative paths (non-finished reasons,
item-capped runs, resumed runs, write failure), sitemap counter fidelity
(match-all fallback, sitemapindex exclusion, relative-loc resolution), and
resume detection state matrix. Two additional round-trip tests (auto-skipped
in trees without a consumer) prove the file parses back into liveness/coverage
readings. Full unit suite green.

Known limitations (documented, deliberate)

• A resumed crawl produces leg-only items/status (marked "resumed": true).
  The full fix — accumulating counters across legs in the checkpoint dir — is
  a worthwhile follow-up, but must handle the dupefilter-clearing corruption
  recovery (which makes a resumed leg re-parse sitemaps → double-count risk).
• Incremental (DeltaFetch) re-crawls shrink the sample: the histogram reflects
  only newly fetched URLs.

[MirjamOdile]

Heads-up: one fix-up commit appended since you last looked (Sitemap counter: unique page locs only, media excluded).

What was broken: the sitemap page-counter this PR introduces (it records "how many pages does the sitemap list" for the audit's coverage target) over-counted in two ways —
image/attachment <loc>s (WordPress image sitemaps) were counted as content pages, and the same URL repeated across sub-sitemap files was counted twice. Net effect: inflated coverage
targets, e.g. one site showing 3,742 phantom "missing" pages that were just photos.

What it does now: the counter counts each real page once — unique locs only, media extensions excluded (mirroring the audit's fetched-path filter). Measurement-only: nothing about
what gets crawled changes.

The commit also updates one test expectation in the writer tests for an audit-side change riding on the quality-tool PR (5xx no longer count as dead in liveness); the test skips cleanly
while the quality tool isn't merged.

Request doc: docs/requests/23-sitemap-crawl-denominator.md.