Add broad-format Docling rendition support

[salmonumbrella]

What changed

Docbank now has a fixed-route Docling provider for every format the selected profile declares and the local media inspector can authorize—not just PDF. PDF results can keep complete natural page evidence, including blank pages; other supported families currently publish bounded Markdown with degraded provenance until their natural-unit mapping can be proved locally.

The provider verifies the sealed upload before egress, uses only same-origin async submit, poll, and result routes, refuses redirects and ambient cookies, and stays on the same task after Docling returns an ID. Structured evidence is accepted only from the supported Docling v1 schema; malformed, partial, drifted, or unlocated output fails closed or degrades explicitly.

Why

Docling supports a broad file surface, but provider acceptance alone cannot define Docbank authority. We should process every locally inspectable and bounded format while refusing to invent page, slide, sheet, or section provenance that Docbank cannot verify.

Usage

For library use, construct the client with docling.New(profile, secrets, httpClient). An operator_network profile can point at Docling Serve over loopback HTTP without a credential; hosted profiles require HTTPS and may resolve a named API-key binding.

Docbank connects to the service the operator runs; it does not install or launch Docling. Daemon, API, and CLI registration is intentionally deferred to S1–S3.

Part of #176 (R7). Stacks on #196.

[roborev-ci]

roborev: Combined Review (f7e011b)

Verdict: Changes need revision due to one high-severity security issue and four medium-severity reliability/data-loss issues.

High

• Windows upload spool inherits permissive access controls — document/upload/spool_windows.go:28
  Windows ignores the supplied Unix mode bits, so the spool directory and source file inherit the parent DACL. Another local user with access to the configured spool parent could copy sensitive document contents before deletion. Use internal/winsecurity.MkdirPrivateAt or MkdirPrivatePinnedAt, ensure the file receives the restricted DACL, and add an integration test with an Everyone-accessible parent.

Medium

• Docling evidence silently omits supported content — document/docling/client.go:545
mapEvidence consumes only texts, ignoring table cells and body ordering while marking evidence complete. PDF table content can disappear, and table-only output may produce no readable rendition. Traverse all supported content with provenance, or classify unhandled/no-readable-text output as unusable so Markdown fallback is selected.

• String-list metadata can exceed serialization limits — internal/processing/source_metadata.go:176
  Entries and aggregate encoded records are not bounded, so oversized metadata can cause MarshalSourceMetadataV1 to fail and trigger permanent backfill retries. Enforce per-entry and aggregate limits during extraction, omitting excess values with a warning.

• Invalid eight-digit dates are accepted — internal/processing/source_metadata.go:793
  Values such as 20241340 become 2024-13-40, then fail canonical validation and retry indefinitely. Validate converted dates with time.Parse("2006-01-02", value) and emit unparseable_timestamp when invalid.

• Bridge polling discards provider retry delays — document/bridge/client.go:179
  Retryable polling errors replace the response with a synthetic running state, losing RetryAfter. This can poll too quickly and exhaust the budget. Preserve the classified retry delay and bound it by the operation deadline.

---

Reviewers: 2 done | Synthesis: codex, 11s | Total: 18m39s

[roborev-ci]

roborev: Combined Review (8f6b7dc)

Verdict: Changes requested — two high-severity and four medium-severity issues remain.

High

• document/bridge/client.go:261 — Upload identity is not verified. The bridge checks only upload length, not SHA-256. A modified stream of the same length can be disclosed to the provider and produce results and an idempotency key tied to the wrong authorized source. Spool and hash the full stream before any network request, reject identity mismatches, and submit only verified bytes.

• internal/processing/source_metadata.go:337 — Nested XML can exhaust daemon memory. Character data is copied into every open ancestor’s builder without depth or aggregate-memory limits, allowing a bounded OOXML/XMP payload to amplify into gigabytes. Extract only recognized leaf values and enforce explicit XML-depth and cumulative-text limits.

Medium

• document/bridge/client.go:124 — Bridge jobs can outlive authorization. Render applies only client.totalTimeout and ignores authorization.ExpiresAt, so document transmission or result acceptance can occur after expiry. Parse the expiry, clamp the operation context to it, and check the local clock before each outbound request and before accepting completion.

• document/bridge/client.go:390 — Artifact authorization is checked after download. Artifacts are fetched before validating MaxArtifacts, allowed roles, or MaxTotalResultBytes, letting a faulty bridge trigger excessive downloads and allocations. Validate the complete manifest and summed declared sizes before resolving any artifact.

• document/docling/client.go:545 — Evidence marked complete can omit table text. mapEvidence reads only Docling’s texts collection and ignores separate text-bearing collections such as tables. Map all supported content using reading order, or mark evidence with unhandled collections as partial/degraded.

• internal/store/upgrade.go:301 — Schema-v3 detection can accept unknown layouts. Exact columns are checked for only two tables, while other required tables are merely checked for presence, risking silent loss of unknown state during JSONL rebuilding. Match shipped v3 layouts against an exact schema fingerprint and reject all others.

---

Reviewers: 2 done | Synthesis: codex, 13s | Total: 20m47s

[roborev-ci]

roborev: Combined Review (4dbccff)

Changes requested: three high-severity security/resource-control issues and three medium-severity data-consistency/retry issues remain.

High

• Unbounded bridge artifact downloads before validation — document/bridge/client.go:390
  Artifact count, roles, and cumulative size are validated only after downloading, allowing a malicious bridge to force allocations beyond MaxTotalResultBytes. Prevalidate metadata and enforce a remaining aggregate-byte budget before each fetch.

• Windows upload spools may expose source documents through inherited ACLs — document/upload/spool_windows.go:29
  Unix-style modes do not block inherited Windows ACLs. Create spool directories atomically with protected owner-only DACLs, such as winsecurity.MkdirPrivatePinnedAt, and similarly restrict source files before writing document bytes.

• Bridge rendering continues after authorization expiry — document/bridge/client.go:124
  Only the initial authorization is checked; polling and artifact retrieval can continue after ExpiresAt. Bound the operation by the earliest applicable deadline, recheck expiry before every request, and reject results received after expiry without trusting provider timestamps.

Medium

• Staged-rendition retries can fail immutable-record comparison — internal/store/processing_catalog.go:313
  Stored records include derived artifact MD5 values while the original normalized record may not, potentially preventing publication retries permanently. Exclude derived MD5 data from equality checks or canonicalize both records identically, and add retry-after-staging-failure coverage.

• Terminal legacy extraction failures remain in a tight retry loop — internal/store/extraction.go:480
  Invalid UTF-8 and oversized text remain ready after lexical cutover, causing repeated processing and migration attempts. Remove terminal failures from the active queue while retaining their results; requeue only after an extractor-version change or explicit repair.

• Historical versions receive current source metadata — internal/store/source_metadata.go:194
  Older content versions are combined with the node’s current path, name, and latest provenance, falsely attributing later facts to historical versions. Associate metadata with the introducing version or omit current-only facts from historical responses.

---

Reviewers: 2 done | Synthesis: codex, 15s | Total: 19m44s

[roborev-ci]

roborev: Combined Review (c18828b)

Medium-severity issues found in bridge upload integrity, authorization expiry enforcement, and API contract alignment.

Medium

• document/bridge/client.go:258 — Upload content is not verified against the authorized digest. Multipart submission checks only byte length, so same-length altered content could be processed and attributed to the authorized Metadata().SHA256. Spool and hash the read-once upload, compare its SHA-256 with the authorized metadata, and submit only verified bytes.

• document/bridge/client.go:124 — Bridge operations can continue after authorization expires. Execution is bounded by TotalTimeout but not authorization.ExpiresAt, allowing uploads, polling, and artifact retrieval after temporal disclosure authority has ended. Set the deadline to the earliest of the caller deadline, total timeout, and authorization expiry; recheck expiry before each request and before returning a completed result rather than relying on provider-controlled receipt timestamps.

• document/bridge/openapi.yaml:175 — The OpenAPI schema permits responses rejected by the runtime client. receipt is unconstrained, and error messages may be 1024 bytes although runtime validation allows only 160 safe characters. Define complete, closed receipt and usage schemas with required fields, align error constraints with runtime validation, and add contract tests against the client decoder.

---

Reviewers: 2 done | Synthesis: codex, 9s | Total: 17m56s

[roborev-ci]

roborev: Combined Review (eabac32)

Verdict: Four medium-severity issues remain involving authorization preflight, metadata bounds, ODS expansion, and XML resource usage.

Medium

• Bridge artifacts fetched before authorization validation — document/bridge/client.go:387
  Artifact count, roles, and aggregate size are validated only after download, allowing a nonconforming bridge to trigger large or unauthorized transfers. Preflight artifact metadata before fetching and enforce a running aggregate-byte limit during transfer.

• Metadata collector permits canonically invalid values — internal/processing/source_metadata.go:174
  Oversized list entries and source labels can pass collection but fail canonical encoding, causing repeated backfill failures. Enforce canonical limits during collection, dropping or truncating invalid values with a warning.

• ODS repeated rows and columns are not counted — document/media/inspect.go:534
  Inspection ignores table:number-columns-repeated and table:number-rows-repeated, so compact documents can exceed the logical-cell limit and cause unexpectedly expensive processing. Parse both attributes using checked arithmetic and reject expanded counts over the limit.

• XML metadata extraction can consume quadratic memory — internal/processing/source_metadata.go:321
  Character data is copied into every open ancestor builder, making extraction quadratic with nesting depth and text size. Capture text only for recognized fields and impose nesting-depth and aggregate-text limits.

---

Reviewers: 2 done | Synthesis: codex, 15s | Total: 24m6s

[roborev-ci]

roborev: Combined Review (d57efbe)

Summary: The changed provider, upload, metadata, consent, backup, and derivative-management paths do not introduce a plausible privilege gain, unauthorized disclosure, or security-control bypass.

No issues found.

---

Reviewers: 2 total (1 done, 1 skipped) | Synthesis: codex | Total: 16m30s

[roborev-ci]

roborev: Combined Review (70f8de7)

Code requires changes: one High-severity transport-security issue and four Medium-severity resource/authorization-limit issues were found.

High

• Plaintext provider origins expose documents and credentials — document/docling/client.go:678, document/bridge/client.go:546
  The operator_network trust boundary permits non-loopback http:// origins. Requests can then expose document contents and reusable credentials (document/docling/client.go:520, document/bridge/client.go:535) to interception or redirection. Require HTTPS for non-loopback origins. For local plaintext services, verify every resolved address is loopback through the restricted-egress transport and reject credentials over all other HTTP connections.

Medium

• Authorization expiry is not enforced during active bridge operations — document/bridge/client.go:124
  Polling and artifact downloads may continue after ExpiresAt. Bound the operation context by the earliest caller deadline, total timeout, or authorization expiry, and test expiry during an in-progress job.

• ODS repetition attributes can bypass the cell limit — document/media/inspect.go:534
  Cell inspection ignores table:number-columns-repeated and table:number-rows-repeated, allowing a compact document to represent a logical sheet exceeding MaxCells. Count repeated rows and columns using overflow-safe arithmetic and add coverage for repeated cells beyond the policy limit.

• Nested XML metadata can cause quadratic resource use — internal/processing/source_metadata.go:321
extractXMLText appends each text segment to every open element, causing quadratic CPU and memory growth for deeply nested XMP or OOXML. Enforce nesting and accumulated-text limits, or buffer only relevant elements, returning bounded partial metadata with a warning.

• Oversized source metadata can cause endless backfill retries — internal/processing/source_metadata.go:174
  Collection does not enforce codec limits for labels, list entries, or total encoded size, so valid oversized fields can fail marshaling and repeatedly trigger automatic backfill. Apply codec bounds during collection, omit oversized values with warnings, and verify that partial evidence remains marshalable.

---

Reviewers: 2 done | Synthesis: codex, 13s | Total: 21m3s