Adjoint plugin - how can I collect the task_id of the forward and backward runs

[remypa]

Hi,

I am using the adjoint plugin, and I'm looking for the way to collect the task_ids of the forward and backward runs. I would need it to be able to use web.delete to delete the data stored server-side (and avoid paying the corresponding storage fees....).

I could not find any other way than patching tidy3d/plugins/adjoint/web.py to add the task_id to the calls to JaxSimulationData.from_sim_data().

Am I missing the normal / non-hacky way of doing this ?

Thanks

[tylerflex]

Hi @remypa

I think this is actually an oversight on our part. Probably the task_id should be passed in from_sim_data() but we didn't do so.

is this what you did?

        # construct JaxSimulationData with no data (yet)
        self_dict["simulation"] = jax_sim
        self_dict["data"] = ()
        self_dict["task_id"] = task_id # <- Add this line

I can add this fix for a future version, thanks for bringing this to our attention! You can submit a form here to get some free credits for the bug find :).

• Tyler

[tylerflex]

actually I think this is not the fix, I needed to pass task_id in to all calls of from_sim_data().. but I'll fix it one way or another in #1674

[majedo-net]

Hello, is there any solution to this same issue in the latest versions that are no longer using Jax? I see a comment on #1674 that there was no way (at the time) to retrieve task_id of completed web.run calls? Is that still the case? Simulation data from adjoint runs with permittivity grids for topology optimization seem to occupy several GBs on web storage even for relatively small design region (~5000 variables) so it would be nice to delete these during each optimization iteration.

[marcorudolphflex]

For the current autograd path, web.run(...) still does not return the forward/adjoint task_ids on the returned SimulationData.

The practical workaround is to use a new folder_name and ideally an unique task_name, then delete the matching forward and adjoint tasks after value_and_grad(...) completes:

import tidy3d.web as web

folder = "my_topopt_cleanup"
task_name = f"iter_{i}"

value, grad = value_and_grad(
    lambda params: objective(params, folder_name=folder, task_name=task_name)
)(params)

for task in web.get_tasks(folder=folder):
    task_id = task.get("task_id") or task.get("taskId")
    name = task.get("task_name") or task.get("taskName")

    if name == task_name or name.startswith(f"{task_name}_adjoint_"):
        web.delete(task_id)

a simpler cleanup would be

from tidy3d.web.core.task_core import Folder
Folder.get(folder_name).delete()

Important: do the deletion after the gradient call returns, not immediately after the forward web.run, because the server-side adjoint task depends on the forward task.

[majedo-net]

Thank you! This is working well for me. However the second suggestion using:
from tidy3d.web.core.task_core import Folder Folder.get(folder_name).delete()
gives the following error:
tidy3d.web.core.exceptions.WebError: error=Only empty folder can be deleted.

The first method using task ids from the folder works great.

[marcorudolphflex]

Great!
Ah sorry, I missed that - still useful as a post-cleanup I guess.