Add the isolated Trafilatura HTML provider

[salmonumbrella]

What changed

Docbank now has a local Trafilatura provider for exact supplied HTML and XHTML bytes. It exposes no URL, redirect, linked-resource, feed, sitemap, remote-metadata, or arbitrary option surface. Accepted output must cover the locally parsed visible text exactly; section provenance is complete only when Docbank verifies the supplied DOM paths and headings itself.

On Linux, the default runner seals the digest-verified bridge executable, launches it inside private user, network, PID, and mount namespaces, and applies no-new-privileges plus a seccomp boundary. IP and Unix sockets, io_uring networking paths, and the amd64 x32 syscall ABI are denied. Output is bounded while streaming, cancellation and overflow tear down the full process tree, and launcher setup failures remain distinct from bridge exit codes.

Why

HTML extraction should stay local and never turn into ambient web fetching or invented structure. Running a parser binary also needs a real isolation boundary; “local process” is not enough if it can reach the network, host sockets, unrelated processes, or arbitrary files.

Usage

For library use on Linux, construct the provider with trafilatura.New(profile). The profile pins the exact bridge executable, its SHA-256, immutable runtime identity, and finite limits; the built-in runner supplies the isolation boundary. macOS and Windows require a separately audited injected runner and otherwise fail closed.

This PR does not add daemon, API, or CLI selection. The approved S1–S3 application-surface work will construct and register providers with the R10 runtime registry.

Part of #176 (R17). Stacks on #208.

[roborev-ci]

roborev: Combined Review (5f6ed2c)

Medium-severity issues remain in retry reconciliation and timestamp validation.

Medium

• cmd/docbank/daemon.go:184 — Both backfill loops retain retry entries indefinitely when a failed target is repaired or deleted during backoff. Once the stale entry is due, waitDelay returns zero while listings remain empty, causing tight database polling; the checksum job also never completes. Reconcile retry entries after each complete cursor pass by removing keys absent from current missing targets, and retain a positive idle delay when no listed target is actionable.

• internal/processing/source_metadata.go:798 — Eight-digit timestamps are accepted without calendar-date validation. Invalid values such as 20241340 can break canonicalization and cause endless source-metadata retries instead of an extraction warning. Validate with time.Parse("20060102", value) and return false for invalid dates so the collector emits unparseable_timestamp.

---

Reviewers: 2 done | Synthesis: codex, 9s | Total: 19m26s

[roborev-ci]

roborev: Combined Review (784c27e)

Review verdict: Three medium-severity issues should be addressed before merging.

Medium

• internal/processing/source_metadata.go:174 — Extracted metadata is not bounded to the limits enforced by MarshalSourceMetadataV1. Oversized list entries, custom-property names, or aggregate records can extract successfully but fail canonicalization, causing permanent background retries. Enforce per-entry, label, and total encoded-size limits during collection, dropping excess metadata with bounded warnings.

• document/trafilatura/provider.go:534 — Complete evidence uses section headings as locator names, but repeated headings are valid HTML and evidence validation rejects duplicate named units. This can classify an exact Trafilatura response as malformed despite unique verified source paths. Use the verified source path—or a bounded stable encoding—as the unique locator name, retaining the heading separately for display.

• internal/store/extraction.go:502 — Every extraction after a lexical generation exists invokes legacy lexical migration, scanning and validating the global segment/build catalog before publishing another generation. Sequential bulk ingestion can therefore become quadratic. Batch, debounce, or incrementally update lexical generation publication so each ingestion batch performs at most one global rebuild.

---

Reviewers: 2 done | Synthesis: codex, 8s | Total: 23m42s

[roborev-ci]

roborev: Combined Review (f687a2f)

Medium-severity reliability and resource-exhaustion issues remain; no Critical or High findings were reported.

Medium

• Duplicate provider jobs and charges — document/docling/client.go:217
  Poll exhaustion while a submitted task remains pending is classified as a retryable capacity failure. The worker clears the non-resumable provider snapshot and resubmits the document, potentially creating duplicate Docling jobs and charges. Return an ambiguous-submission error after polling exhaustion, or persist the task ID and continue polling it.

• Provider deadlines incorrectly treated as terminal cancellation — document/datalab/client.go:761, document/docling/client.go:496, document/marker/client.go:646
  Provider-imposed operation deadlines are classified as cancellation, even when a request was submitted and its outcome is unknown. Distinguish parent cancellation from internal deadlines; classify pre-egress timeouts as transient and post-egress timeouts as ambiguous or resumable.

• Deeply nested XML can exhaust daemon memory — internal/processing/source_metadata.go:336
  Each XML text chunk is copied into every open ancestor, making memory consumption proportional to input size multiplied by nesting depth. Limit XML depth and accumulated text, or retain text only for relevant leaf elements, with a bounded-resource test using deeply nested synthetic input.

---

Reviewers: 2 done | Synthesis: codex, 9s | Total: 24m45s

[roborev-ci]

roborev: Combined Review (ec355a5)

Medium-severity issues remain in authorization enforcement, retry cleanup, and metadata extraction limits.

Medium

• document/bridge/client.go:428 — Result decoding applies artifact-count and cumulative-byte ceilings only when optional profile limits are nonzero, ignoring active authorization limits. This can cause unauthorized artifacts or excessive data to be downloaded and retained before outer validation rejects the response.

  • Fix: Validate artifact count and roles against the authorization, check declared evidence/Markdown/artifact sizes against authorization.MaxTotalResultBytes before downloads, and enforce the actual cumulative limit during decoding.

• cmd/docbank/daemon.go:184 — Failed backfill keys remain in retries after their targets disappear. Once due, stale entries can cause continuous database polling; the checksum job may also never complete.

  • Fix: During each full cursor scan, reconcile retries with currently missing targets and remove stale keys before calculating the wait delay.

• internal/processing/source_metadata.go:337 — XML character data is copied into every open ancestor buffer, allowing deeply nested OOXML or XMP input to amplify memory usage by depth and potentially exhaust the daemon.

  • Fix: Enforce XML-depth and cumulative-text budgets, buffer only recognized metadata elements, and limit each value while parsing.

• internal/processing/source_metadata.go:174 — List collection can accept entries exceeding MaxSourceMetadataValueBytes and produce records exceeding MaxSourceMetadataEncodedBytes. Canonical marshaling then rejects them, causing repeated metadata-publication failures.

  • Fix: Enforce per-entry and aggregate encoded-size limits during collection, omitting oversized values with a warning.

---

Reviewers: 2 done | Synthesis: codex, 13s | Total: 17m0s

[roborev-ci]

roborev: Combined Review (821ee1e)

High

• internal/processing/source_metadata.go:323 — extractXMLText stores strings.Builder values in a growing slice. Appending a nested element can copy a builder after it has been written, causing a panic on the next write. Store pointers to elements/builders or accumulate only into bounded active fields, and add coverage for indented nested XML.

Medium

• document/bridge/client.go:428 — External artifacts are fetched before enforcing per-invocation MaxArtifacts, AllowedArtifactRoles, and MaxTotalResultBytes. Preflight all descriptors against the invocation authorization, including cumulative declared size, before downloading anything.
• document/llamaparse/client.go:161 — Resumed operations incorrectly use the post-submission checkpoint time as CompletedAt, even though completion occurs after polling and retrieval. Set state.completedAt to acceptedAt after successful result retrieval and validation.
• internal/processing/source_metadata.go:174 — List metadata bypasses per-entry and aggregate encoded-size limits, while splitValues at line 767 allocates an unbounded list before checking its count. Parse incrementally with count, per-value, and total-byte budgets, omitting overflow with a bounded warning.
• document/trafilatura/native_runner_linux.go:82 — The isolated renderer retains the daemon user’s filesystem access because its mount namespace still exposes the complete host filesystem. Build a minimal private root, expose only necessary runtime files read-only, provide isolated writable temporary storage, and include filesystem confinement in fail-closed attestation.

---

Reviewers: 2 done | Synthesis: codex, 12s | Total: 27m0s