Add basic per-actor external volume flow by msau42 · Pull Request #405 · agent-substrate/substrate

Michelle Au (msau42) · 2026-07-08T01:01:24Z

Initial part of #232.

Extends the ActorTemplate API to add support for per-actor external volumes and adds control plane and atelet hooks following the Actor lifecycle.

Callouts to volume operations are abstracted with a Volume interface. Right now, only a mock volume plugin for testing has been implemented, but we will add CSI support next.

Tests pass
Appropriate changes to documentation are included in the PR - Not going to update documentation until we add CSI support.

Michelle Au (msau42) · 2026-07-08T15:46:21Z

+	Capacity resource.Quantity `json:"capacity" protobuf:"bytes,1,opt,name=capacity"`
+	// storageClassName refers to the StorageClass to create the volume from.
+	// +required
+	StorageClassName string `json:"storageClassName" protobuf:"bytes,2,opt,name=storageClassName"`


open for discussion: can we reference k8s objects or should we create our own api?

I don't think there's a cleaner way. In theory we could have an ateapi level api which just acts as an indirect, but I don't see a reason at this moment.

Michelle Au (msau42) · 2026-07-08T15:48:09Z

  map<string, string> match_labels = 1;
 }

+message ExternalVolume {


discuss: should ExternalVolume be its own standalone object rather than embedded fields into the Actor?

Right now ExternalVolume is tied to an Actor but we are also thinking about the scenario where you want to share a volume across multiple actors.

Michelle Au (msau42) · 2026-07-08T15:56:14Z

 	}
 	err = s.persistence.CreateActor(ctx, actor)
 	if err != nil {
+		// Cleanup created volumes if DB write fails


Discuss: In general, how should we approach errors when an operation only partially succeeds? Should we rollback all the partially completed operations? What happens if the rollback fails?

Or we leave the system in the partial state and require users to cleanup by calling Delete? Even then, if deletion fails and Create is called again, we could still have partially leftover resources from a previous invocation.

My current thoughts:

If there is a pair of "Create" and "Delete" operations, then we can leave the partial state and clean it up on the Delete. This does imply we will have to add some "Creating" and possibly "Deleting" state to Actor before we start creating resources for it. I already have marked that as a todo.

For operations like "Resume", there isn't a corresponding delete operation if the resume failed. It's possible resume could be called on a different node. That means we could have partial state left on the first node. In those cases we can try to rollback the state as much as possible but that would still be a best effort operation. This has a few implications:

We probably need to have some process on the node that monitors oprhaned resources and tries to clean them up and/or alert

We need to make sure these orphaned resources don't get reused if a new actor with the same name gets created. This potentially implies that we should have some sort of actor uuid and be using that in our actor directories.

Tim Hockin (thockin)

I didn't get too far into the reading of this yet, but wanted to send a couple questions before EOD

Tim Hockin (thockin) · 2026-07-09T00:19:39Z

+	Capacity resource.Quantity `json:"capacity" protobuf:"bytes,1,opt,name=capacity"`
+	// storageClassName refers to the StorageClass to create the volume from.
+	// +required
+	StorageClassName string `json:"storageClassName" protobuf:"bytes,2,opt,name=storageClassName"`


I don't think there's a cleaner way. In theory we could have an ateapi level api which just acts as an indirect, but I don't see a reason at this moment.

Tim Hockin (thockin) · 2026-07-09T00:27:07Z

+
+// VolumePlugin abstracts storage operations.
+type VolumePlugin interface {
+	CreateVolume(ctx context.Context, name string, capacity string, storageClass string) (volumeID string, err error)


Do you see this as "kubernetes PVC is the plugin", or "CSI is the plugin", or "NFS is the plugin" ?

What I really want to understand is what credentials are granted to the API server to create and destroy volumes, and what additional layers are in between. E.g. if I need to create a million k8s PVCs and PVs, that's trouble, but if I need to give ateapi binaries the ability to delete EBS volumes in the same context as k8s, that's also trouble.

My thought is that substrate will directly call CSI grpc APIs (so ateapi needs to be authorized to communicate with CSI drivers). We will not be creating PVC objects, although we will need to have a substrate API reference to the volume IDs (which I have added in this PR).

Michelle Au (msau42) · 2026-07-09T01:09:45Z

 			Type:        "bind",
-			Source:      ateompath.DurableDirVolumeMountPoint(atespace, actorID, vm.GetName()),
+			Source:      srcPath,
+			Options:     []string{"bind", "rw"},


double check this. durabledir didn't set any options.

This adds a new externalVolumeTemplate where users specify the capacity, storage driver and storage properties they would like via StorageClass. When an actor is created, we will also provision a volume for the actor according to the template. The volume is deleted when the actor is deleted.

Adds external volumes to the Actor object and integrates control plane Actor workflows with the corresponding volume operations: CreateActor -> CreateVolume DeleteActor -> DeleteVolume ResumeActor -> AttachVolume Pause/Suspend Actor -> DetachVolume

Volume operations are integrated into the actor workflows: - Run/Restore: MountVolume - Checkpoint: UnmountVolume Volumes are mounted under the actor directory and passed as bind mounts to OCI.

This allows for different storage backends to be supported. Right now it only contains the mock volume plugin, which is only intended for testing purposes right now.

It makes the file counter path configurable so we can test it against different volume types. And adds an optional file verification step to test the presence of a file on an existing volume.

Julian Gutierrez Oschmann (juli4n) · 2026-07-10T22:36:56Z

+	// for each actor. The volume only lives as long as the actor and is deleted
+	// when the actor is deleted.
+	// +optional
+	ExternalVolumeTemplate *ExternalVolumeTemplate `json:"externalVolumeTemplate,omitempty" protobuf:"bytes,3,opt,name=externalVolumeTemplate"`


I don't think the protobuf directives control anything in CRDs. We should just remove them from this and other fields.

Tim Hockin (thockin)

I read this. It seems broadly correct. I am still worried about permissions, but not enough to block this.

I swear I had another comment, and it was important, but I forgot what it was. Oh well, Friday. :)

Tim Hockin (thockin) · 2026-07-10T15:30:46Z

+	// for each actor. The volume only lives as long as the actor and is deleted
+	// when the actor is deleted.
+	// +optional
+	ExternalVolumeTemplate *ExternalVolumeTemplate `json:"externalVolumeTemplate,omitempty" protobuf:"bytes,3,opt,name=externalVolumeTemplate"`


Do we need the word "Template" in here? All of these are templates

Tim Hockin (thockin) · 2026-07-11T00:47:05Z


 func (s *CallAteletPauseStep) RetryBackoff() *wait.Backoff { return nil }

+type DetachVolumesForPauseStep struct {


Could use comments on why this is different than suspend

Tim Hockin (thockin) · 2026-07-11T00:47:28Z


 func (s *CallAteletSuspendStep) RetryBackoff() *wait.Backoff { return nil }

+type DetachVolumesStep struct {


Could use comments on how this is different from pause

Michelle Au (msau42) commented Jul 8, 2026

View reviewed changes

Michelle Au (msau42) force-pushed the mock-mount-dev branch from 3a4a327 to ccc7a2d Compare July 8, 2026 17:05

Michelle Au (msau42) changed the title ~~Add basic external volume flow~~ Add basic per-actor external volume flow Jul 8, 2026

Michelle Au (msau42) force-pushed the mock-mount-dev branch 2 times, most recently from c2286ea to 96fa815 Compare July 8, 2026 22:50

Haven Xia (HavenXia) reviewed Jul 9, 2026

View reviewed changes

Comment thread cmd/ateapi/internal/controlapi/volumes.go Outdated

Tim Hockin (thockin) reviewed Jul 9, 2026

View reviewed changes

Michelle Au (msau42) commented Jul 9, 2026

View reviewed changes

Zheng Lammerts (zlammerts-svg) added this to Substrate Milestones (go/substrate-milestones) Jul 9, 2026

github-project-automation Bot moved this to Todo in Substrate Milestones (go/substrate-milestones) Jul 9, 2026

Zheng Lammerts (zlammerts-svg) removed this from Substrate Milestones (go/substrate-milestones) Jul 9, 2026

Michelle Au (msau42) force-pushed the mock-mount-dev branch 2 times, most recently from 95b3702 to 11a558c Compare July 10, 2026 00:19

Michelle Au (msau42) added 5 commits July 10, 2026 00:24

ateapi handling of external volumes.

a9c90db

Adds external volumes to the Actor object and integrates control plane Actor workflows with the corresponding volume operations: CreateActor -> CreateVolume DeleteActor -> DeleteVolume ResumeActor -> AttachVolume Pause/Suspend Actor -> DetachVolume

atelet handling of external volumes.

79bd689

Volume operations are integrated into the actor workflows: - Run/Restore: MountVolume - Checkpoint: UnmountVolume Volumes are mounted under the actor directory and passed as bind mounts to OCI.

Allow ateom to see host mounts

ff871f7

Volume plugin interface and mock volume implementation.

8e69611

This allows for different storage backends to be supported. Right now it only contains the mock volume plugin, which is only intended for testing purposes right now.

Michelle Au (msau42) force-pushed the mock-mount-dev branch from 11a558c to 4890a34 Compare July 10, 2026 01:03

Update counter demo with external volume using mock volume plugin.

ffc96b8

It makes the file counter path configurable so we can test it against different volume types. And adds an optional file verification step to test the presence of a file on an existing volume.

Michelle Au (msau42) force-pushed the mock-mount-dev branch from 4890a34 to ffc96b8 Compare July 10, 2026 18:02

Julian Gutierrez Oschmann (juli4n) reviewed Jul 10, 2026

View reviewed changes

Tim Hockin (thockin) reviewed Jul 11, 2026

View reviewed changes


		func (s CallAteletPauseStep) RetryBackoff() wait.Backoff { return nil }

		type DetachVolumesForPauseStep struct {


		func (s CallAteletSuspendStep) RetryBackoff() wait.Backoff { return nil }

		type DetachVolumesStep struct {

Uh oh!

Conversation

Michelle Au (msau42) commented Jul 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michelle Au (msau42) Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Tim Hockin (thockin) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tim Hockin (thockin) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Michelle Au (msau42) Jul 8, 2026 •

edited

Loading