TRELLIS.2 macOS — quality-first runtime

Run TRELLIS.2 image-to-3D generation natively on Mac.

This is an isolated, quality-first Apple Silicon runtime for Microsoft's TRELLIS.2 image-to-3D model. It starts from the PyTorch/MPS port by shivampkumar, pins the Metal stack from pedronaugusto, and carries local fixes for decode-time mesh holes and MPS cascade sampling.

It is intentionally independent from LiTo: their latent spaces cannot be mixed without retraining. Use TRELLIS.2 where a static, UV-mapped PBR mesh is the desired deliverable; keep LiTo for its image alignment and view-dependent appearance.

Results

The parent port reports ~5m13 cold-start for a 512 pipeline on an M4 Pro 24GB, with an ~18GB memory peak. That is a useful compatibility indication, not a performance promise for the M3 Max. This repository records every run locally so that the M3 Max result becomes the source of truth.

Output is a GLB with base-color, metallic, and roughness textures — ready for use in 3D applications.

Example

Input image → Generated 3D mesh (~400K vertices, ~800K triangles) with Metal-baked PBR textures:

Requirements

• macOS 27 on Apple Silicon (M3 Max / 36GB is the primary target)
• Python 3.11+
• 24GB+ unified memory (36GB recommended for the 1024 cascade experiment)
• ~15GB disk space for model weights (downloaded on first run)
• Xcode command-line tools and the Metal Toolchain

Quick Start

# From this dedicated sibling repository
cd /Users/jean/Documents/Lab/active/trellis2-macos

# Download the Xcode Metal Toolchain so setup can build the
# Metal-accelerated runtime and texture baker.
xcodebuild -downloadComponent MetalToolchain

# Log into HuggingFace (needed for gated model weights)
hf auth login

# Request/accept access to these gated models in the browser:
#   https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m
#   https://huggingface.co/briaai/RMBG-2.0

# Run setup (creates venv, installs deps, clones & patches TRELLIS.2,
# builds the pinned Metal backends; it fails rather than silently choosing a
# lower-quality texture path)
bash setup.sh

# Activate the environment
source .venv/bin/activate

# Generate a 3D model from an image
python scripts/doctor.py --require-metal
python scripts/model_access.py --require
python generate.py path/to/image.png --output results/example/asset

SKIP_METAL=1 exists for diagnostics only. It is not a quality-equivalent configuration: it can fall back to Python sparse convolution and a softer texture baker.

SKIP_METAL=1 bash setup.sh

setup.sh pre-clones Git dependencies into deps/ so all network I/O happens up front and checks out the exact revisions again on every invocation. If a dependency checkout becomes inconsistent, preserve it for diagnosis and use a new clone rather than discarding an existing working directory.

Source and Metal dependencies are pinned by commit. The initial Hugging Face download requires network access and the DINOv3/RMBG model permissions; every generation can then be made fail-closed and fully local with --offline.

Usage

# Basic usage
python generate.py photo.png

# With options
python generate.py photo.png --seed 123 --output my_model --pipeline-type 512

# Once all model weights are cached, prohibit all network access
python generate.py photo.png --offline --output results/offline/asset

# Make silent MPS fallback fail while investigating a backend
python generate.py photo.png --strict-mps --no-texture --output results/strict/asset

# Run a reproducible corpus manifest (starts actual generation unless --dry-run)
python scripts/benchmark.py --image photo.png --pipeline 512 --steps 12

# All options
python generate.py --help

Option	Default	Description
--seed	42	Random seed for generation
--output	output_3d	Output filename (without extension)
--pipeline-type	512	Pipeline resolution: 512, 1024, 1024_cascade
--texture-size	1024	PBR texture resolution: 512, 1024, 2048
--no-texture	—	Skip texture baking, export geometry only
--strict-mps	—	Reject implicit CPU fallback, for validation only
--offline	—	Read the local Hugging Face cache only; fail if anything is missing
--pbr-face-target	200000	Face cap for the textured GLB; 0 tests a full-detail bake
--pbr-alpha-mode	auto	Preserve predicted transparency, or use opaque for solid assets
--pbr-geometry-mode	preserve	Keep bake-input triangles immutable; legacy restores the historical cleanup path for comparison
--pbr-orientation-mode	preserve	Preserve source winding, or opt into radial orientation for roughly closed solid characters/props
--pbr-radial-max-residual-conflicts	8	Expert, radial-only absolute conflict budget; does not relax the fractional budget
--pbr-radial-max-residual-conflict-fraction	1e-4	Expert, radial-only fractional conflict budget; does not relax the absolute budget
--primary-azimuth	0	Camera azimuth of the positional image when experimental multi-view mode is active
--view IMAGE AZIMUTH	—	Add a conditioning view; repeat up to three times
--blend-temperature	2.0	Spatial concentration of each view in the shared 3D latent
--multiview-texture-mode	blend	Use all views for appearance, or primary to use only the positional image
--mesh-postprocess	none	Opt into validated UDF topology reconstruction with udf
--udf-resolution	512	Power-of-two repair grid resolution
--udf-band	1.0	UDF narrow-band width in voxels
--udf-project-back	0.9	Ratio used to project repaired vertices back to the decoded surface

Every run writes the decoded geometry to <output>_raw.obj before texture baking and keeps <output>.obj as a byte-identical compatibility copy. It also writes <output>.generation.json; inspect topology with python scripts/asset_inspect.py <output>_raw.obj.

Experimental multi-view conditioning

--view activates an opt-in research path that keeps one shared diffusion trajectory and evaluates the released mono-image model once per view at every step. Positive per-view velocities are blended spatially before the original classifier-free guidance calculation. Without --view, the established single-image code path is unchanged and the multi-view backend is not imported.

# Example: two three-quarter views that share the right side of the object.
# Azimuth is the camera position in raw TRELLIS Z-up coordinates:
# 0° = +X, 90° = +Y, 180° = -X, 270° = -Y.
python generate.py front-right.png \
  --primary-azimuth 45 \
  --view rear-right.png 135 \
  --blend-temperature 0.75 \
  --multiview-texture-mode blend \
  --mesh-postprocess udf \
  --pbr-alpha-mode opaque \
  --pipeline-type 512 \
  --output results/multiview/asset

Start with two RGBA views at the same framing, focal length and elevation, with one side visible in both images. Direct 512 and 1024 pipelines are supported; 1024_cascade deliberately fails before model loading until the basic orientation and quality gate has passed. The azimuth is used to assign model predictions to regions of the latent; it is not an extra camera token understood by the published weights. Consequently, this mode depends on the mono-image model canonicalizing all views consistently and should not yet be treated as production multi-view reconstruction. The generation JSON records all input paths, normalized azimuths and the blend temperature.

--multiview-texture-mode primary changes only the texture latent: sparse structure and shape remain conditioned by every supplied view. It can reduce appearance conflicts when turnaround images agree on silhouette but disagree on small colors, seams or facial details. Unseen appearance is still inferred by the mono-image checkpoint; this is not literal camera projection.

--mesh-postprocess udf is a separate, fail-open geometry stage. It writes a candidate and a <output>_postprocess.json report, then selects the candidate for the GLB preview only if topology and surface-preservation gates pass. A candidate that adds boundary edges, small closed hole loops or vertex-disconnected islands is rejected even if it reduces non-manifold edges. The raw OBJ is never overwritten. The PBR GLB remains a delivery preview: the legacy Metal baker can perform additional repair and decimation, so use the raw/repaired OBJ plus the JSON report when judging geometry for sculpting or rigging.

For an opaque character or prop, use --pbr-alpha-mode opaque if the automatic alpha prediction creates transparent freckles or apparently missing patches. This changes only the GLB material interpretation; it does not fill geometry or alter the raw/repaired OBJ. Leave the default auto for genuinely transparent assets.

TRELLIS output winding is not necessarily an outward-normal reference. For a roughly closed, approximately star-shaped character or solid prop, --pbr-orientation-mode radial first makes each manifold edge-connected component locally consistent, then chooses its direction from an area-weighted score around the mesh bounding-box centre. This changes only face index order in the simplified PBR proxy; decoded/raw/repaired OBJ geometry remains intact. The generation JSON records reversal counts and component diagnostics. Keep the default preserve for open surfaces, deep cavities, strongly concave objects, clothing shells and overlapping accessories. The radial heuristic can still choose the wrong side around eyes, mouths, armpits, fingers or other concave regions, so inspect Blender's Face Orientation overlay before rigging. A tiny non-orientable remnant may be accepted only when both strict budgets hold: at most 8 unresolved manifold edges and at most 0.01% of manifold adjacencies. The JSON report records the count, fraction and whether the tolerance was used. Anything above either limit stops before publishing a partially corrected GLB. Asset-specific investigations can override both budgets explicitly; changing only one never bypasses the other, defaults stay strict, and each effective value plus its default/cli origin is recorded in the generation report. An override accepts known local seams—it does not repair them—so geometry validation remains exact. The corner-normal gate is derived from the observed residual fraction, not the CLI ceiling, is hard-capped at 0.025%, and never permits a fully opposed radial face.

# Opt-in export preparation for a solid character-like asset.
python generate.py character-front-right.png \
  --primary-azimuth 45 \
  --view character-rear-right.png 135 \
  --blend-temperature 0.75 \
  --pbr-geometry-mode preserve \
  --pbr-orientation-mode radial \
  --pbr-alpha-mode opaque \
  --output results/multiview/character

The default --pbr-geometry-mode preserve simplifies the PBR proxy once. Any explicit orientation pass runs next, then those triangles become immutable during Metal UV unwrapping and texture bake. In this mode, rasterized UV positions are sampled directly: the baker's legacy closest-point BVH reprojection is disabled because it can jump to a nearby sheet on dense, non-manifold geometry. The generation report records direct_uv_surface or source_bvh under pbr_texture_projection. The GLB is written to a temporary path and published only after its triangle multiset and face winding match that final proxy exactly after UV seams and axis conversion are neutralized. Explicit vertex normals must also be present, finite, non-zero and unit length; radial exports recompute them from the final UV-split triangles. Fully opposed faces and opposed corners are allowed only within tightly bounded singularity budgets; larger disagreement fails validation. A rejected Metal candidate falls back to the geometry-preserving KDTree baker. legacy retains the pinned o_voxel cleanup/simplification path for diagnosis and cannot be combined with radial orientation. A legacy candidate that changes geometry is saved separately as <output>_legacy_rejected.glb and cannot replace the validated final GLB.

What Was Ported

TRELLIS.2 depends on several CUDA-only libraries. This port replaces each of them with a backend that runs on Apple Silicon:

Original (CUDA)	Replacement	Purpose
flex_gemm	mtlgemm (Pedro Naugusto's Metal port) with backends/conv_none.py fallback	Sparse 3D convolution. The Metal port is the default now; the pure-PyTorch gather-scatter path is the fallback for machines without the Metal Toolchain.
o_voxel._C hashmap	backends/mesh_extract.py	Mesh extraction from dual voxel grid (pure Python)
flash_attn	PyTorch SDPA	Scaled dot-product attention for sparse transformers (padded, not fused — room for improvement)
cumesh	mtlmesh Metal backend	Decode-time hole filling, face cleanup and simplification. The runtime keeps a narrow fallback only if an obsolete binary fails.
nvdiffrast	mtldiffrast (Metal) with pure-Python fallback	Differentiable rasterization for texture baking

Additionally, all hardcoded .cuda() calls throughout the codebase were patched to use the active device instead.

Technical Details

Sparse 3D Convolution (backends/conv_none.py): Implements submanifold sparse convolution by building a spatial hash of active voxels, gathering neighbor features for each kernel position, applying weights via matrix multiplication, and scatter-adding results back. Neighbor maps are cached per-tensor to avoid redundant computation.

Mesh Extraction (backends/mesh_extract.py): Reimplements flexible_dual_grid_to_mesh using Python dictionaries instead of CUDA hashmap operations. Builds a coordinate-to-index lookup table, finds connected voxels for each edge, and triangulates quads using normal alignment heuristics.

Attention (patched full_attn.py): Adds an SDPA backend to the sparse attention module. Pads variable-length sequences into batches, runs torch.nn.functional.scaled_dot_product_attention, then unpads results.

Texture Baking: The pinned Metal stack is mtldiffrast, mtlbvh, mtlmesh, mtlgemm, and the Apple o_voxel fork. The default textured export caps at 200K faces for current Metal UV/rasterization/export stability. Its preserve adapter keeps native Metal UV/rasterization, disables the baker's cleanup, second simplification and non-manifold orientation pass, and samples the already-rasterized surface without a redundant closest-point reprojection. An optional, deterministic radial pass can establish a character-oriented winding reference before baking; an exact post-export geometry-and-winding gate then prevents later damage or inversions from replacing it. The raw OBJ always preserves the decoded mesh and --pbr-face-target 0 remains a deliberate full-detail experiment.

Performance

The figures below are an upstream-port reference on an M4 Pro (24GB), pipeline 512, with cached weights and the full Metal stack. They are not a claim for the M3 Max. The benchmark harness writes M3 Max measurements locally and is the only basis for choosing a default.

Stage	Time
Pipeline load (first call per process)	103s
Sparse structure sampling (12 steps)	80s
Shape SLat sampling (12 steps)	22s
Texture SLat sampling (12 steps)	12s
Shape SLat decoder (VAE forward)	~20s
Tex SLat decoder (VAE forward)	~7s
flexible_dual_grid_to_mesh (pure Python)	~8s
fast_simplification (858K → 200K faces)	~1s
Texture bake (Metal, 1024²)	~15s
Total wall-clock (cold start)	5m 13s
Generation + bake only (excluding pipeline load)	3m 20s

Sampling steps that also touch sparse convolution tend to be dominated by attention, which remains SDPA-padded in the conservative default. Fused Metal attention is deliberately not selected until it has passed the same quality corpus on this Mac.

Memory usage peaks at around 18GB unified memory during generation.

First-ever run adds ~15GB of HuggingFace weight downloads (TRELLIS.2, DINOv3, RMBG-2.0) — network-bound, not included above. The pipeline load time is dominated by deserializing those weights from disk; if you batch multiple images in one Python process you pay load once.

SKIP_METAL=1 is for diagnosis only; it is not a comparable quality or performance configuration.

Limitations

• 1024 cascade is experimental: it includes the MPS cache fix from upstream PR #167, but it has to be qualified on this M3 Max before it becomes the default.
• 1536 is not targeted yet: no Apple implementation has established an acceptable quality/stability baseline at that resolution.
• Sparse attention is not fused in the default quality backend: SDPA preserves the conservative baseline. Fused Metal attention and PyTorch FlexAttention are benchmark candidates, not assumed improvements.
• Pre-simplified before texture bake: The mesh is decimated once from ~800K to ~200K faces for current Metal UV/rasterization/export stability. Preserve mode skips the redundant BVH projection; legacy mode can still build it. The GLB preserves that proxy exactly, not the full-resolution topology. Use the OBJ output for the decoded full-resolution mesh.
• No training support: Inference only.

See the quality and validation protocol for the acceptance gates and benchmark corpus.

On mtlgemm / flex_gemm

setup.sh installs mtlgemm as part of the Metal stack. It is used both for the sparse-convolution diffusion path and for the texture baker's grid_sample_3d. Without it, generate.py can fall back to conv_none.py for diffusion and a torch.nn.functional.grid_sample texture path. Neither fallback is accepted as the production quality configuration.

License

The porting code and documentation distributed by this repository are released under the MIT License. This license does not grant rights to dependencies cloned by setup.sh, downloaded model weights, input images, or generated assets. See THIRD_PARTY_NOTICES.md for the complete upstream inventory, pinned revisions, licenses, and redistribution conditions.

In particular, self-hosted RMBG-2.0 weights are non-commercial unless you have a commercial agreement with BRIA. It is therefore unsuitable for a commercial deployment as configured unless that agreement is in place or the background-removal component is replaced/disabled.

Credits

• TRELLIS.2 by Microsoft Research — the original model and codebase
• DINOv3 by Meta — image feature extraction
• RMBG-2.0 by BRIA AI — background removal
• @pedronaugusto — mtldiffrast, mtlbvh, mtlmesh, and the CPU fork of o_voxel that together provide the Metal texture-baking path used by this repo
• ComfyUI-Trellis2 by visualbruno — inspiration for the experimental spatial multi-view velocity blending design