What problem does this solve?
Some model extensions generate one mesh from a structured scene, not from one image. WorldSculpt is a concrete case: its published inference path expects a scene directory with transforms.json, RGB frames, instance masks, and per-instance metric 3D AABBs (with optional pose/scale metadata); it reconstructs objects and composes a scene GLB. Multiple image ports alone do not provide those annotations.
Today Modly's model-extension workflow submits an image to /generate/from-image and invokes generate(image_bytes, params, ...). A process extension can receive a filesystem path, but it cannot use the model-extension model_sources download flow. Encoding the scene directory in a string/text parameter would require a fake text connection and a placeholder image request, while leaving users without a clear required scene input. This blocks an honest WorldSculpt node whose Pixal3D base and WorldSculpt adapter weights are managed from Models/Extensions.
Proposed solution
Add an opt-in, versioned scene-directory input contract for model nodes. The manifest should declare a required scene/directory input distinct from image; the workflow UI should present a folder input/picker and route one selected local scene directory through the job API and extension subprocess without fabricating an image. An optional scene-generation entry point (or equivalent versioned dispatch) should receive the validated directory path, params, progress, and cancellation, while existing generate(image_bytes, ...) nodes stay unchanged. Older hosts must reject unsupported scene-input manifests explicitly instead of silently treating them as image nodes.
The host should enforce path boundaries and access rules (canonicalize paths, reject missing/non-directory paths and unsafe symlink escapes, never mutate the selected input, and use a separate output directory). The extension must also resolve every frame/mask path referenced by transforms.json within the selected scene root before upstream code opens it, rejecting traversal and symlink escapes. WorldSculpt-specific validation would check cameras, instance IDs and metric AABBs, and reject incomplete reconstruction rather than return a partial GLB.
Keep weights in the normal Models/Extensions lifecycle: the model node can declare model_sources for AlayaLab/WorldSculpt, with required checkpoint checks. Reusing the Pixal3D base without duplicate downloads depends separately on the pending shared-weight-group contract; this issue should not reimplement it. Setup/Repair should not silently download model weights.
Acceptance criteria
- A model extension can declare a required scene-directory input; the UI and saved workflows represent it without a fake image or text edge.
- The selected scene path reaches the correct extension node through API/job/subprocess dispatch; legacy image nodes retain their current behavior.
- Invalid, missing, or escaping scene paths and nested frame/mask references fail before inference with actionable errors. The host does not mutate selected input, and outputs are written separately.
- The existing Models/Extensions download/readiness flow continues to manage node-private
model_sources; shared Pixal3D base reuse is handled by the separate shared-weight-group work.
- A host contract fixture proves scene-path transport, progress/cancellation, failures, and legacy compatibility without requiring WorldSculpt's GPU weights. Unsupported older hosts fail closed on the new manifest contract.
- Separately, real WorldSculpt acceptance verifies valid scene input produces a loadable
scene.glb and that missing annotations or failed object reconstruction cannot return a partial success.
- Manifest, workflow UI, transport, path-security, downloader, and compatibility tests cover the new contract.
This request is about a general extension input contract, not hard-coding WorldSculpt's scene schema into Modly. It should be additive to, or reuse, the port schema discussed in #228, not compete with it. Shared Pixal3D weight reuse is covered separately by #343; neither issue should assume the other has merged.
Alternatives considered
inputs: ["image", ...] / extra_image_paths: provides RGB views, not calibrated cameras, masks, or object boxes.- A string folder parameter on a text-only model node: technically possible, but requires a spurious text connection and placeholder image upload; it does not express a required scene input.
- A separate process extension: has natural file-path I/O, but currently lacks UI-managed model weight sources and shared-weight references.
Confirmation
What problem does this solve?
Some model extensions generate one mesh from a structured scene, not from one image. WorldSculpt is a concrete case: its published inference path expects a scene directory with
transforms.json, RGB frames, instance masks, and per-instance metric 3D AABBs (with optional pose/scale metadata); it reconstructs objects and composes a scene GLB. Multiple image ports alone do not provide those annotations.Today Modly's model-extension workflow submits an image to
/generate/from-imageand invokesgenerate(image_bytes, params, ...). A process extension can receive a filesystem path, but it cannot use the model-extensionmodel_sourcesdownload flow. Encoding the scene directory in a string/text parameter would require a fake text connection and a placeholder image request, while leaving users without a clear required scene input. This blocks an honest WorldSculpt node whose Pixal3D base and WorldSculpt adapter weights are managed from Models/Extensions.Proposed solution
Add an opt-in, versioned scene-directory input contract for model nodes. The manifest should declare a required scene/directory input distinct from
image; the workflow UI should present a folder input/picker and route one selected local scene directory through the job API and extension subprocess without fabricating an image. An optional scene-generation entry point (or equivalent versioned dispatch) should receive the validated directory path, params, progress, and cancellation, while existinggenerate(image_bytes, ...)nodes stay unchanged. Older hosts must reject unsupported scene-input manifests explicitly instead of silently treating them as image nodes.The host should enforce path boundaries and access rules (canonicalize paths, reject missing/non-directory paths and unsafe symlink escapes, never mutate the selected input, and use a separate output directory). The extension must also resolve every frame/mask path referenced by
transforms.jsonwithin the selected scene root before upstream code opens it, rejecting traversal and symlink escapes. WorldSculpt-specific validation would check cameras, instance IDs and metric AABBs, and reject incomplete reconstruction rather than return a partial GLB.Keep weights in the normal Models/Extensions lifecycle: the model node can declare
model_sourcesforAlayaLab/WorldSculpt, with required checkpoint checks. Reusing the Pixal3D base without duplicate downloads depends separately on the pending shared-weight-group contract; this issue should not reimplement it. Setup/Repair should not silently download model weights.Acceptance criteria
model_sources; shared Pixal3D base reuse is handled by the separate shared-weight-group work.scene.glband that missing annotations or failed object reconstruction cannot return a partial success.This request is about a general extension input contract, not hard-coding WorldSculpt's scene schema into Modly. It should be additive to, or reuse, the port schema discussed in #228, not compete with it. Shared Pixal3D weight reuse is covered separately by #343; neither issue should assume the other has merged.
Alternatives considered
inputs: ["image", ...]/extra_image_paths: provides RGB views, not calibrated cameras, masks, or object boxes.Confirmation