Add canonical 3D surgical deformation bridge

CONTRIBUTORS.md

@@ -15,6 +15,7 @@ Thanks to everyone who has contributed to LandmarkDiff.
 | [lshariprasad](https://github.com/lshariprasad) | histogram_match_skin tests ([#263](https://github.com/dreamlessx/LandmarkDiff-public/pull/263)) |
 | [dagangtj](https://github.com/dagangtj) | SafetyResult dataclass improvements ([#235](https://github.com/dreamlessx/LandmarkDiff-public/pull/235)), file logging handler ([#236](https://github.com/dreamlessx/LandmarkDiff-public/pull/236)), api_client error messages ([#237](https://github.com/dreamlessx/LandmarkDiff-public/pull/237)) |
 | [PredictiveManish](https://github.com/PredictiveManish) | OpenAPI specification ([#340](https://github.com/dreamlessx/LandmarkDiff-public/pull/340)), batch processing notebook ([#334](https://github.com/dreamlessx/LandmarkDiff-public/pull/334)) |
+| [mimosapudical](https://github.com/mimosapudical) | Canonical 3D surgical deformation bridge, FLAME integration, and held-out multi-view validation ([#418](https://github.com/dreamlessx/LandmarkDiff-public/issues/418)) |
 | [srikar117](https://github.com/srikar117) | Dark mode for Gradio interface ([#204](https://github.com/dreamlessx/LandmarkDiff-public/issues/204), in progress) |
 | [passionworkeer](https://github.com/passionworkeer) | Docker architecture feedback ([#5](https://github.com/dreamlessx/LandmarkDiff-public/issues/5)) |
 

docs/SHARED_3D_SURGERY.md

@@ -0,0 +1,204 @@
+# Shared 3D Surgical Deformation
+
+This module adds a geometry-first alternative to LandmarkDiff's independent
+per-view 2D procedure presets.
+
+## What changes
+
+Independent 2D mode applies a separate landmark preset to every uploaded view.
+Shared 3D mode instead:
+
+1. fits one patient-specific canonical face geometry from the available views;
+2. estimates a full 3D scaled-orthographic camera for each view;
+3. applies one surgical deformation in canonical 3D;
+4. reprojects that same deformation into every camera;
+5. reuses the existing TPS/image pipeline for rendering.
+
+The goal is not to make frontal and profile images look identical. The goal is
+for both images to be projections of the same underlying anatomical edit.
+
+## Geometry backends
+
+### Dependency-free canonical fallback
+
+The default backend uses LandmarkDiff's 478-point MediaPipe topology. It is
+intended for tests, demos, and environments where FLAME assets are unavailable.
+It must not be interpreted as the official FLAME statistical head model.
+
+### Official FLAME backend
+
+LandmarkDiff can load a user-supplied official FLAME model asset:
+
+```python
+from landmarkdiff.flame_model import FlameModel
+
+model = FlameModel.from_flame_assets(
+    "/path/to/flame2023.pkl",
+    "/path/to/mediapipe_to_flame.json",
+)
+```
+
+The correspondence may be either:
+
+- JSON: a 478-entry direct vertex list, or a possibly partial object mapping
+  MediaPipe ids to FLAME vertex ids;
+- NPZ direct map: a 478-entry `vertex_indices` array; or
+- standard barycentric NPZ embedding with `landmark_indices`,
+  `lmk_face_idx`, and `lmk_b_coords`.
+
+In the standard format, `landmark_indices` are MediaPipe landmark ids. The
+FLAME surface location is represented by a triangle id plus barycentric
+coordinates. LandmarkDiff evaluates those barycentric locations directly for
+camera and shape fitting. Partial embeddings, including the commonly used
+selected-landmark form, are supported: mapped points are reprojected from the
+3D edit while unmapped MediaPipe points retain their original coordinates for
+the downstream 478-point TPS pipeline.
+
+FLAME model files and third-party correspondence assets are not redistributed
+by this repository. Some public trackers publish MediaPipe-to-FLAME embeddings
+under licenses that prohibit redistribution, so callers must provide an
+embedding they are entitled to use.
+
+The first FLAME backend uses the neutral FLAME template/topology and identity
+shape basis. Global head pose is represented by the per-view camera. Expression
+and jaw articulation remain neutral in this first surgical-geometry version.
+
+## Supported 3D procedures
+
+The current canonical handle sets cover:
+
+- rhinoplasty
+- alarplasty
+- mentoplasty
+- genioplasty
+
+Shared 3D rejects unsupported procedures explicitly instead of silently falling
+back to independent 2D.
+
+## Held-out-view benchmark
+
+The benchmark is designed to avoid circular evaluation. Ground-truth synthetic
+surgery is generated by an analytic Gaussian 3D field, while Shared 3D uses
+sparse surgical handles plus Laplacian propagation.
+
+The training views are 0°, 30°, and 60°. The default held-out view is 80°.
+The held-out pre-op image may be used to estimate its camera, but its post-op
+target is never used by the fit or deformation.
+
+Fallback backend:
+
+```bash
+python scripts/benchmark_surgery3d.py \
+  --procedure mentoplasty \
+  --out artifacts/surgery3d_benchmark
+```
+
+Official FLAME backend:
+
+```bash
+python scripts/benchmark_surgery3d.py \
+  --flame-model /path/to/flame2023.pkl \
+  --correspondence /path/to/mediapipe_to_flame.npz \
+  --procedure mentoplasty \
+  --out artifacts/surgery3d_benchmark_flame
+```
+
+Primary metric:
+
+- held-out reprojection NME
+
+Secondary diagnostics:
+
+- canonical vertex RMSE, converted to millimetres using fitted IPD;
+- held-out camera rotation error;
+- front/profile chin displacement ratio for a qualitative sanity check.
+
+The benchmark also writes a front / 45° / profile comparison grid and a
+canonical before/after 3D deformation visualization.
+
+### Synthetic validation results
+
+Using the dependency-free canonical backend, the sweep-selected regularization
+(`lambda_smooth=30`, `lambda_locality=0.3`) was evaluated on 27 held-out-view
+conditions spanning three procedures, three intensities (40/60/100), and three
+held-out yaw angles (20°/45°/85°).
+
+| Procedure | Wins vs. independent 2D | Mean relative NME change |
+| --- | ---: | ---: |
+| mentoplasty | 9 / 9 | -12.0% |
+| genioplasty | 9 / 9 | -47.2% |
+| rhinoplasty | 6 / 9 | -1.1% |
+| **Overall** | **24 / 27** | **-20.1%** |
+
+Median relative NME change across all 27 cases was -18.1%. Maximum synthetic
+camera-rotation error was approximately 0.0001°.
+
+The three losses are all near-frontal rhinoplasty cases at 20° yaw, where Shared
+3D is about 23% worse in NME than the existing 2D preset. This is an important
+trade-off rather than a hidden failure: the existing rhinoplasty preset is
+explicitly hand-tuned for frontal-image alar/dorsum narrowing, while the Shared
+3D path is designed to preserve one anatomical edit across changing viewpoints.
+A separate regularization sweep did not remove this near-frontal regression
+without changing the deformation model itself.
+
+These numbers are synthetic geometry validation, not clinical outcome accuracy.
+They demonstrate the cross-view geometry behavior of the method and should not
+be interpreted as evidence of patient-specific surgical prediction quality.
+
+## Numerical implementation
+
+The 3D surgical deformation solves a Laplacian objective with a small locality
+penalty:
+
+[
+\sum_h w_h\|\Delta v_h-d_h\|^2
++ \lambda_s\|L\Delta V\|^2
++ \lambda_l\|\Delta V\|^2.
+]
+
+The locality term prevents the constant-vector nullspace of the graph Laplacian
+from turning a local procedure into an almost global translation.
+
+For FLAME-scale meshes the system is solved matrix-free with preconditioned
+conjugate gradient rather than constructing and factorizing a dense 5K-by-5K
+matrix.
+
+## Validation results
+
+Using the dependency-free canonical backend, held-out-view validation was run
+across 27 combinations of:
+
+- procedures: mentoplasty, genioplasty, rhinoplasty;
+- intensities: 40, 60, 100;
+- held-out yaw: 20°, 45°, 85°.
+
+With the sweep-selected regularization `lambda_smooth=30` and
+`lambda_locality=0.3`:
+
+- Shared 3D wins 24 / 27 held-out cases;
+- mean relative held-out NME change is -20.1%;
+- genioplasty wins 9 / 9 cases, mean -47.2%;
+- mentoplasty wins 9 / 9 cases, mean -12.0%;
+- rhinoplasty wins 6 / 9 cases, mean -1.1%;
+- maximum synthetic camera-rotation error is about 0.0001°.
+
+The three losses are all rhinoplasty at 20° yaw, where Shared 3D is about 23%
+worse than the independent 2D preset. The same rhinoplasty deformation improves
+at 45° and 85° yaw. This is treated as a method limitation rather than hidden by
+procedure-specific tuning.
+
+A focused 70° / 80° held-out matrix gives 6 / 6 wins with about 23.7% mean
+relative NME improvement.
+
+## Current limitations
+
+- The real FLAME path requires an externally supplied MediaPipe-to-FLAME
+  correspondence map.
+- A projected vertex-color PLY preview is available; UV-unwrapped or multi-view
+  photometric texture fitting is not part of the current geometry bridge.
+- The 3D handle magnitudes are method parameters and should be clinically
+  calibrated before being interpreted as patient-specific surgical predictions.
+- Real clinical multi-view validation remains separate from the synthetic
+  held-out geometry benchmark.
+- Near-frontal rhinoplasty remains a documented trade-off: the current 3D
+  deformation wins at 45°/85° in the synthetic validation but loses at 20°.

landmarkdiff/__init__.py

@@ -16,12 +16,15 @@
     "curriculum",
     "data",
     "data_version",
+    "deformation3d",
     "displacement_model",
     "ensemble",
     "evaluation",
     "experiment_tracker",
     "face_verifier",
     "fid",
+    "flame_fitting",
+    "flame_model",
     "hyperparam",
     "inference",
     "landmarks",
@@ -35,6 +38,7 @@
     "morphometry",
     "postprocess",
     "safety",
+    "surgery3d",
     "synthetic",
     "validation",
 ]