Skip to content

Save verified 10M-row CSV results and recover interrupted promotions - #3

Merged
RAHoebe merged 10 commits into
analysis_integrationfrom
feat/data-query-production-foundations
Sep 8, 2026
Merged

RAHoebe merged 10 commits into
analysis_integrationfrom
feat/data-query-production-foundations

Conversation

@RAHoebe

@RAHoebe RAHoebe commented Sep 8, 2026 •

Copy link
Copy Markdown
Collaborator

Remote query results can now be saved directly to their original OMERO context as the exact worker CSV, with a linked full SQL/typed-parameter recipe and searchable verified provenance. Fresh source/group/session checks and OMERO's effective annotation permission apply to querying, downloading and saving. Modified, expired or no-longer-authorized results require a new authorized query.

Signed persistent journals and tagged OMERO creations make saves recoverable after process or host crashes, including a lost creation reply, interrupted upload, partial attachment links and interrupted cleanup. Authorized retries reuse complete verified saves or reconcile incomplete ones. An explicit administrator dry-run/apply command supports expired sessions without storing administrator credentials. CSV/recipe upload is chunked; the notebook runtime transfers CSV buffers and removes temporary CSVs after parsing.

An optional coordinated profile supports 10M rows, 2 GiB results and 15-minute worker queries with aligned broker/Gunicorn/proxy timeouts. Existing manual upload limits, Methods v1/v2, notebook ctx.query(), tokens, local/remote policy and old workers remain supported. Transport/correlation/audit additions are optional and additive. Target branch remains analysis_integration; worker PR.

Validation: 146 backend tests, 230 frontend tests, 16 notebook SDK tests, runtime/browser smoke and verified wheel; 17 actual HTTP mixed-version/rollback cases; live OMERO permission matrix. Real OMERO query/download/promotion and checksum-matched browser loading pass at 10M rows for all three formats. Exact row/byte boundaries, 18 real engine OOM/restart cases, 19 process-kill checkpoints, explicit operator reconciliation, ten promotion VM power cuts and six cache-publication VM cuts pass. The completed three-repeat 10M matrix covers all 540 requests: 468 successes, 72 expected admission rejections and zero unexpected failures; all 72 single-client 1M/4M regressions pass. Normal runs have no kernel OOM kills and remain within cache quotas. Additional first-journal-directory and durable cache-entry power cuts pass. The upgraded local Chrome notebook/query/save workflow also passes.

See the runbook and tested implementation pair for reproducible commands, isolated Hyper-V provisioning, the observed OMERO Lucene-index recovery procedure, deployment order and rollback. Preserve state/audit/signing keys and reconcile interrupted saves before retiring this recovery code. Target production deployments still require their own hardware/storage/OMERO acceptance.

Completed release evidence includes per-batch timings, memory, cache usage, failures and hashes of raw evidence.

@RAHoebe RAHoebe changed the title Save authorized worker CSV results with verified provenance Save verified 10M-row CSV results and recover interrupted promotions Sep 8, 2026
@RAHoebe
RAHoebe marked this pull request as ready for review September 8, 2026 12:51
@RAHoebe
RAHoebe merged commit 2fe8c3a into analysis_integration Sep 8, 2026
11 checks passed
@RAHoebe
RAHoebe deleted the feat/data-query-production-foundations branch September 8, 2026 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant