POSe estimation with Explicit/Implicit Differentiable OptimizatioN
POSEIDON brings differentiable pose estimation to deep keypoint models by integrating Perspective-n-Point (PnP) solving with implicit gradients — diving deep into spatial reasoning for robust 3D localization.
 |
 |
|---|---|
| YOLO-NAS-POSE S with COCO-POSE weights | YOLO-NAS-POSE S with Random weights |
This project enhances the YOLO-NAS architecture to support vision-based landing (VBL) by injecting differentiable pose estimation into the learning process. Instead of only optimizing bounding box and keypoint quality, we directly supervise the 3D pose using a differentiable PnP solver in the loss function.
Traditional object detection models are not designed to meet aerospace-grade accuracy tolerances. This project aims to bridge that gap by:
- Embedding camera pose estimation directly into the learning objective
- Enabling models to learn keypoint configurations that are optimal for 6-DoF localization
- Leveraging implicit gradients for backpropagation through the PnP optimization
-
✅ YOLO-NAS based backbone for real-time inference
-
✅ Predict 2D keypoints of known 3D landmarks (e.g., runway corners)
-
✅ Differentiable P3P (Perspective-Three-Point) solver using:
- 🌀 A Novel Parametrization of the Perspective-Three-Point Problem for a Direct Computation of Absolute Camera Position and Orientation (Kneip and al. CVPR 2011)
-
✅ Pose-aware loss combining PnP and OKS (but not tested yet)