Repository navigation
Save verified 10M-row CSV results and recover interrupted promotions - #3
Merged
RAHoebe merged 10 commits intoSep 8, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remote query results can now be saved directly to their original OMERO context as the exact worker CSV, with a linked full SQL/typed-parameter recipe and searchable verified provenance. Fresh source/group/session checks and OMERO's effective annotation permission apply to querying, downloading and saving. Modified, expired or no-longer-authorized results require a new authorized query.
Signed persistent journals and tagged OMERO creations make saves recoverable after process or host crashes, including a lost creation reply, interrupted upload, partial attachment links and interrupted cleanup. Authorized retries reuse complete verified saves or reconcile incomplete ones. An explicit administrator dry-run/apply command supports expired sessions without storing administrator credentials. CSV/recipe upload is chunked; the notebook runtime transfers CSV buffers and removes temporary CSVs after parsing.
An optional coordinated profile supports 10M rows, 2 GiB results and 15-minute worker queries with aligned broker/Gunicorn/proxy timeouts. Existing manual upload limits, Methods v1/v2, notebook
ctx.query(), tokens, local/remote policy and old workers remain supported. Transport/correlation/audit additions are optional and additive. Target branch remainsanalysis_integration; worker PR.Validation: 146 backend tests, 230 frontend tests, 16 notebook SDK tests, runtime/browser smoke and verified wheel; 17 actual HTTP mixed-version/rollback cases; live OMERO permission matrix. Real OMERO query/download/promotion and checksum-matched browser loading pass at 10M rows for all three formats. Exact row/byte boundaries, 18 real engine OOM/restart cases, 19 process-kill checkpoints, explicit operator reconciliation, ten promotion VM power cuts and six cache-publication VM cuts pass. The completed three-repeat 10M matrix covers all 540 requests: 468 successes, 72 expected admission rejections and zero unexpected failures; all 72 single-client 1M/4M regressions pass. Normal runs have no kernel OOM kills and remain within cache quotas. Additional first-journal-directory and durable cache-entry power cuts pass. The upgraded local Chrome notebook/query/save workflow also passes.
See the runbook and tested implementation pair for reproducible commands, isolated Hyper-V provisioning, the observed OMERO Lucene-index recovery procedure, deployment order and rollback. Preserve state/audit/signing keys and reconcile interrupted saves before retiring this recovery code. Target production deployments still require their own hardware/storage/OMERO acceptance.
Completed release evidence includes per-batch timings, memory, cache usage, failures and hashes of raw evidence.