New OWEwoksWidget to run tasks in a process #82

Activity

1. poautran commented on Jan 23, 2026

   

   poautran

   on Jan 23, 2026

   MemberAuthor

   More actions

   In GitLab by @woutdenolf on Jan 23, 2026, 15:32 GMT+1:

   Yes please. I think I proposed this already but got rejected.
2. poautran commented on Jan 23, 2026

   

   poautran

   on Jan 23, 2026

   MemberAuthor

   More actions

   In GitLab by @loichuder on Jan 23, 2026, 15:37 GMT+1:

   For the record, here are two working process-based executors (courtesy of @tvincent):

   • Based on Qobject: https://gitlab.esrf.fr/levantin/txs/-/blob/master/txs/app/ProcessWorker.py?ref_type=heads
   • Based on Future: https://gitlab.esrf.fr/ui/tomovis/-/blob/main/tomovis/subprocess_executor.py?ref_type=heads
3. poautran

   added

   gitlab issue

   feature request

   on Feb 14, 2026
4. loichuder

   self-assigned this

   on Feb 17, 2026
5. woutdenolf commented on Aug 3, 2026

   

   woutdenolf

   on Aug 3, 2026

   Member

   More actions

   Multi-processing is almost implemented. See #434
6. woutdenolf

   removed

   feature request

   gitlab issue

   on Oct 6, 2026
7. payno commented on Oct 7, 2026

   

   payno

   on Oct 7, 2026

   Member

   More actions

   Yes, the processing happens here, and it supports cancellation. For now, it's up to each task to decide how to handle aborting. We could probably implement something more generic for this.
8. payno

   updated priority medium and progress triage on Oct 7, 2026
9. woutdenolf commented on Oct 7, 2026

   

   woutdenolf

   on Oct 7, 2026

   Last edited by woutdenolf

   Member

   More actions

   For reference, the vocabulary I propose

   Word	Applies when	What it does
   cancel	Not started yet	Task never runs
   abort	Running	Task is killed, whether it cooperates or not
   stop	Running	Task is asked to finish early, it may ignore the request, unclear whether an exception is raised or not

   Task can only do the last one, so stop. Futures can potentially do all three (perhaps call that halt). Or maybe abort and stop should always cancel when still pending.

   Note that Celery uses revoke(terminate=False) for our cancel and revoke(terminate=True) for our cancel+abort.

   https://docs.celeryq.dev/en/2.5-archived/userguide/workers.html#revoking-tasks
10. loichuder commented on Oct 9, 2026

    

    loichuder

    on Oct 9, 2026

    Member

    More actions

    Unassigning myself since I had a look and I don't understand how to implement this with the new architecture.
11. loichuder

    removed their assignment

    on Oct 9, 2026
12. loichuder commented on Oct 9, 2026

    

    loichuder

    on Oct 9, 2026

    Last edited by loichuder

    Member

    More actions

    I mean, Claude is telling me that

    class OWEwoksWidgetProcess(
        OWEwoksBaseWidget,
        **ow_build_opts,
        concurrency=Concurrency.PROCESS,
        max_workers=1,
        submit_policy=SubmitPolicy.ALWAYS,
    ):

    is enough.

    But how to handle aborting in this case?
13. woutdenolf commented on Oct 10, 2026

    

    woutdenolf

    on Oct 10, 2026

    Member

    More actions

    ewoksorange main now runs tasks in a subprocess (#434). No dedicated widget class is needed: pass concurrency="process" as a class argument.

    import time
    
    from ewokscore import Task
    from ewoksorange.gui.owwidgets.base import OWEwoksBaseWidget
    from ewoksorange.gui.owwidgets.meta import ow_build_opts
    
    
    class MyTask(Task, input_names=["duration"], output_names=["result"]):
        def run(self):
            for _ in range(self.inputs.duration):
                if self.cancelled:
                    return
                time.sleep(1)
            self.outputs.result = "done"
    
    
    class OWMyTask(
        OWEwoksBaseWidget,
        **ow_build_opts,
        ewokstaskclass=MyTask,
        concurrency="process",  # inputs and outputs must be picklable
        max_workers=1,
        submit_policy="always",  # or "drop_if_busy"
    ):
        name = "My task"
    
        def __init__(self, *args, **kwargs):
            super().__init__(*args, **kwargs)
            self._future = None
            self.task_executor.started.connect(self._on_started)
    
        def _on_started(self, future):
            self._future = future
    
        def _on_abort_clicked(self):
            if self._future is not None:
                self._future.abort()  # see below: cooperative for now

    The TaskFuture (returned by execute_ewoks_task() or emitted by task_executor.started) has:

    • future.cancel(): the task never runs (only works while it is still queued).
    • future.abort(): today this is cooperative.
      • It sets task.cancelled in the worker process and calls task.cancel().
      • The task decides whether and when to stop. One that ignores the request runs to completion.

    What is still missing for this issue

    • abort should kill the task, whether it cooperates or not. That is the whole point of running it in a process.
    • The current cooperative behavior is really stop, following the vocabulary in New OWEwoksWidget to run tasks in a process #82 (comment):
      • cancel: not started yet, the task never runs
      • abort: running, the task is killed
      • stop: running, the task is asked to finish early
    • The renaming has started:
      • Rename task cancel to request_stop ewokscore#514: Task.cancel → Task.request_stop
      • Adapt to Task.cancel now being deprecated. #490: adapt ewoksorange to it
    • Implementation note for a real abort:
      • A single ProcessPoolExecutor worker can't be terminated without breaking the pool (BrokenProcessPool).
      • So killing a task probably needs one multiprocessing.Process per task, like the executors from @tvincent linked above.
14. payno commented on Oct 12, 2026

    

    payno

    on Oct 12, 2026

    Member

    More actions

    Ok so in addition to the renaming we need to add a abort API but only at ewoksorange level.