Repository navigation
feat: add depth estimation task implementation - #1552
Conversation
msluszniak
left a comment
There was a problem hiding this comment.
HF configs fail the spec schema:
family:depth-anything-v2/-v3don't match^[a-z0-9_]+$, should bedepth_anything_v2/_v3.capabilities:depth-estimationisn't in the enum. Neither are gemma-4-multimodal'saudio-text-to-text/image-text-to-text, so let's add all three to the spec.- Variants carry
quantized/default, which the schema disallows (also in the blazeface and rfdetr-keypoint Vulkan configs).
Cards:
4. ZipDepth's "Repository structure" has no sizes on the config.json lines.
5. ZipDepth Core ML is fp32, while other CV Core ML builds are fp16 (fp32 never reaches the ANE). If fp16 hurts accuracy, please note that on the card.
6. Please add an "XNNPACK int8" section with quality numbers to both Depth Anything cards, like the other int8 repos have.
msluszniak
left a comment
There was a problem hiding this comment.
Code suggestions:
download-libs.js: theFEATURE_MAPcomment names only Depth Anything V2; V3 and ZipDepth ship the same three backends.- Docs still describe only V2:
- Intro: add ZipDepth (inverse depth) and DA3 (depth, larger is farther).
- Options table: the "Depth Anything V2 preset" column doesn't fit ZipDepth (
'linear',{ alpha: 1 / 255, beta: 0 }). IMAGENET_NORMis used by V3 too.
- Demo: V3 outputs depth, and
1 - normalizedsquashes near-range contrast (the vase merges with the table). Consider inverting before normalizing:const values = isDepth ? depth.data.map((d) => (d > 0 ? 1 / d : NaN)) : depth.data;
- Demo: use the option's
familyinstead ofObject.values(DEPTH_V3_MODELS).includes(selectedModel).
Optional follow-up: a depthKind: 'inverse' | 'depth' field on DepthEstimatorModel would save apps from special-casing V3.
|
Tested both iOS and Android in demo apps on all possible models and worked really cool! |
|
FIxed both the HF and applied the code suggestion. Also improved the performance of models after profiling (around 10% inference time improvement). |
Description
This PR introduces new depth estimation task implementation along with a test suite and a simple example screen in computer vision app to test the new feature end-to-end. Alongside it we upload 3 depth estimation models to SWM HuggingFace repo to use with this new task with different performance/quality tradeoffs; all accessible in the new collection https://huggingface.co/collections/software-mansion/depth-estimation.
Introduces a breaking change?
Type of change
Tested on
Testing instructions
Screenshots
Related issues
Closes #1372
Checklist
Additional notes