Skip to content

FIX: Deadlock caused by fork in multiprocess - #571

Open
scranford1 wants to merge 3 commits into
developfrom
bugfix/multiprocess-deadlock
Open

scranford1 wants to merge 3 commits into
developfrom
bugfix/multiprocess-deadlock

Conversation

@scranford1

@scranford1 scranford1 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

The use of fork can lead to deadlocking in child processes due to how the parent process is cloned. By swapping to spawn, we remove the deadlock risk from the process. Using spawn, introduces a significant performance hit and means that mutated global variables are no longer inherited as a new python interpreter is launched.

@scranford1 scranford1 self-assigned this Sep 23, 2026
@scranford1

Copy link
Copy Markdown
Contributor Author

@mpu-creare Curious to get your opinion on this. In unit testing, we hit deadlocking in test_parrallel.py test_parallel_process_zarr_async but not every time. Spawn resolves this but has significant impact on performance and how we can use global variables.

@mpu-creare

Copy link
Copy Markdown
Contributor

Frankly, we never use this functionality so I'd vote for simplifying by just deprecating this.

@scranford1

Copy link
Copy Markdown
Contributor Author

Frankly, we never use this functionality so I'd vote for simplifying by just deprecating this.

I have deprecated the Process node and skipped related unit tests.

@scranford1
scranford1 marked this pull request as ready for review September 24, 2026 16:28
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants