Skip to content

feat(processing): reactive FIFO processing pipeline and worker-thread PDF merge - #1704

Open
manuc66 wants to merge 1 commit into
masterfrom
split/processing-pipeline
Open

feat(processing): reactive FIFO processing pipeline and worker-thread PDF merge#1704
manuc66 wants to merge 1 commit into
masterfrom
split/processing-pipeline

Conversation

@manuc66

@manuc66 manuc66 commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Problem

While a scan is post-processed (PDF merge, uploads), the listen and adf-autoscan loops await the whole delivery before polling the printer again. A slow destination or a heavy merge can make the loop miss the next scan event or trip the printer's userActionTimeout / waitScanNewPageRequest timeouts. In single-scan this is one iteration and does not matter.

Solution

Producer/consumer in the same process:

  • Explicit staged pipeline (postProcessing): generate PDF → announce → deliver → cleanup, threaded through a shared context — the seam where future pre-processing stages (OCR, --post-command) plug in.
  • FIFO processing queue (src/queue/processingQueue.ts): captured scans are drained in capture order by one background worker; the loop returns to polling immediately.
  • PDF merge off the event loop (src/pdfMergeWorker.ts): jsPDF runs in a worker thread, so health checks and printer HTTP responses stay responsive during a big merge.
  • single-scan keeps its synchronous contract (exits only once delivery finished), routed through processScanProcessingJob.

Durability (explicitly not part of this PR)

The queue is in memory. A crash mid-job leaves captured files on disk (cleanup only runs once delivery finished) and delivery can be redone by hand — no worse than the previous synchronous behavior, and no breaking change for existing Paperless/Nextcloud users.

Review checklist

  • Durability trade-off: the queue is in memory — a crash mid-job leaves captured files on disk and delivery can be redone by hand. Confirm this is acceptable and documented before merging (a conditional durable inbox is planned as a follow-up).
  • Memory (partial): the PDF merge is off the event loop (worker thread). The remaining whole-file fs.readFile reads (webhook sha256, S3 upload) belong to the S3/webhook PRs.
  • Non-blocking delivery: webhook/upload sending no longer blocks the scan loop — delivery runs on the background drain. Verify with the real-services harness.
  • Error isolation: outbox/write errors are confined to the drain worker and no longer increment the listener's anti-crash errorCount. Verify the listener keeps running when a delivery target is unreachable.
  • Startup: the startup flushOutbox blocking is the webhook PR's concern; here the queue guarantees the scan loop is never blocked by a slow delivery.

Merge this PR first — it is the foundation. The S3-delivery and webhook/outbox PRs follow.

Testing

  • test/processingQueue.test.ts: enqueue returns before the work completes (capture not blocked), FIFO order through a real delivery, awaited single-job path.
  • Full suite green (674 passing) including the adapted listenCmd/postProcessing suites.

@manuc66
manuc66 force-pushed the split/processing-pipeline branch from 9b6f10e to a32816e Compare September 5, 2026 05:25
… PDF merge

Port of PR #1699 onto master, adapted to the current codebase (no S3 or
webhook yet): the staged post-processing pipeline, the in-memory FIFO
processing queue, and the jsPDF merge moved to a worker thread.

In listen and adf-autoscan mode, captured scans are dropped onto a
single background drain so the loop keeps polling the printer while
uploads or post-processing run; single-scan keeps its synchronous
contract (exits only once delivery finished). Scan order is preserved.
Durability is intentionally out of scope (in-memory queue).

Also ships docs/processing-pipeline.md and the CHANGELOG entry.
@manuc66
manuc66 force-pushed the split/processing-pipeline branch from a32816e to 95c0724 Compare September 5, 2026 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant