Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
275 changes: 275 additions & 0 deletions examples/hand_detector/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,275 @@
# MediaPipe Palm Detector

This example reconstructs the MediaPipe palm detector as a PyTorch module,
quantizes it with TICO WrapQ, and exports static Circle models with an NHWC
input boundary.

The example is organized around four entry points:

```text
convert.py Convert source TFLite weights and graph metadata to PyTorch artifacts.
export.py Export floating-point or calibrated quantized Circle models.
analyze.py Run reusable numerical quantization analyses.
verify.py Verify torch.export and Circle artifacts.
```

Model-independent numerical analysis lives in `tico.quantization.analysis`.
The example only supplies palm-detector model loading, data normalization, and
output-boundary selection.

## Model interface

The exported model accepts one image tensor:

```text
shape: [1, 192, 192, 3]
layout: NHWC
range: [0, 1]
dtype: float32 before quantized export
```

It returns:

```text
regressors: [1, 2016, 18]
classifiers: [1, 2016, 1]
```

The PyTorch implementation uses NCHW internally. `NHWCInputAdapter` keeps the
external ABI explicit while Circle layout optimization removes redundant
internal layout transitions.

## Setup

Build and install TICO first:

```bash
./ccex build
./ccex install --cpu_only
```

Install the example dependencies:

```bash
python -m pip install -r examples/hand_detector/requirements.txt
```

Run commands from the repository root with `python -m`. This keeps package
imports stable and avoids depending on the current working directory.

## Convert the source TFLite model

```bash
python -m examples.hand_detector.convert \
/path/to/hand_detector.tflite
```

The default outputs are:

```text
examples/hand_detector/hand_detector_spec.json
examples/hand_detector/hand_detector_float.pt
```

The converter currently supports the operator subset used by the supplied palm
detector. It is not a general TFLite-to-PyTorch frontend.

## Export Circle models

### Floating point

```bash
python -m examples.hand_detector.export float \
--output examples/hand_detector/hand_detector_float.circle
```

The command verifies the NHWC input, layout optimization, and the two
`RESIZE_BILINEAR` operators unless `--skip-verification` is supplied.

### Quantized

Use representative tensors produced by the same preprocessing path as runtime
inputs:

```bash
python -m examples.hand_detector.export quantized \
--calibration-dir /path/to/calibration_npy \
--bits 8 16
```

The quantization policies are:

| Tensor role | UINT8 | INT16 |
|---|---|---|
| Image and activations | per-tensor asymmetric | per-tensor symmetric |
| Conv/depthwise weights | per-channel asymmetric | per-channel symmetric |
| PReLU slope | per-channel asymmetric | per-channel symmetric |
| Convolution bias | INT32 | INT64 |

Synthetic inputs are available only for smoke tests:

```bash
python -m examples.hand_detector.export quantized \
--synthetic-calibration-samples 32 \
--bits 8
```

## Quantization analysis

### A/B/C/D ablation

The standard profiles isolate the major quantization error sources:

```text
A output-only
B weight-only with floating-point activations and outputs
C internal activation-only with floating-point weights and outputs
D full quantization
```

Run all four profiles from one calibrated candidate:

```bash
python -m examples.hand_detector.analyze ablation \
--calibration-dir /path/to/calibration_npy \
--evaluation-dir /path/to/evaluation_npy \
--bits 8
```

The same API is reusable from Python:

```python
from tico.quantization.analysis import (
QuantizationAblation,
QuantizationBoundaries,
QuantizationProfile,
SiteSelector,
)
```

A model adapter only needs to define which observer sites represent final model
outputs. Parameter and internal-activation profiles are derived from observer
roles.

### Output clipping

Compare MinMax, fixed percentile, and calibration-L1 clipping while leaving all
internal model computation in floating point:

```bash
python -m examples.hand_detector.analyze output-clipping \
--calibration-dir /path/to/calibration_npy \
--evaluation-dir /path/to/evaluation_npy \
--bits 8
```

This reports calibration and evaluation MAE, selected ranges, affine qparams,
saturation, and integer-code utilization. The L1 candidate is selected only
from calibration outputs.

### Activation observer sweep

Keep per-channel MinMax weight quantization fixed and compare activation range
estimators:

```bash
python -m examples.hand_detector.analyze observer-sweep \
--calibration-dir /path/to/calibration_npy \
--evaluation-dir /path/to/evaluation_npy \
--bits 8 \
--percentiles 99.9 99.99 99.999
```

`PercentileObserver` uses bounded sampling, so it does not retain every value
from every activation tensor.

## Calibration and evaluation data

Supported NumPy shapes are:

```text
[192, 192, 3]
[1, 192, 192, 3]
[3, 192, 192]
[1, 3, 192, 192]
```

Integer arrays are converted to float32 and divided by 255. Floating-point
arrays are assumed to already use the model input range.

Calibration and evaluation may point to the same directory for numerical-floor
analysis, but policy selection and final reporting should use disjoint data.
Use offsets to split a naturally sorted directory:

```bash
python -m examples.hand_detector.analyze observer-sweep \
--calibration-dir /path/to/npy \
--calibration-offset 0 \
--calibration-limit 200 \
--evaluation-dir /path/to/npy \
--evaluation-offset 200 \
--evaluation-limit 79 \
--require-disjoint
```

For frames extracted from video, split by source video or capture session rather
than adjacent frame number.

## Verification

Verify the PyTorch export graph:

```bash
python -m examples.hand_detector.verify torch
```

Verify a floating-point Circle model:

```bash
python -m examples.hand_detector.verify circle \
examples/hand_detector/hand_detector_float.circle
```

Verify a quantized Circle model:

```bash
python -m examples.hand_detector.verify quantized \
examples/hand_detector/exported/hand_detector_uint8.circle \
--bits 8
```

## Internal support modules

Implementation helpers are under `_support/` and are not separate user-facing
commands:

```text
_support/circle.py
_support/conversion.py
_support/data.py
_support/quantization.py
_support/tflite_flatbuffer.py
_support/verify_circle_layout.py
_support/verify_circle_resize.py
_support/verify_quantized_circle.py
```

## Tests

Run reusable analysis tests:

```bash
python -m unittest discover -s test/quantization/analysis -v
python -m unittest discover -s test/quantization/wrapq -p "test_control.py" -v
python -m unittest discover -s test/quantization/wrapq/observers \
-p "test_percentile.py" -v
```

Run the model example tests:

```bash
python -m examples.hand_detector.test_hand_detector
```

See `docs/layout_optimization.md` for the Circle layout-region optimization design
and `THIRD_PARTY_NOTICES.md` for source-model attribution.
14 changes: 14 additions & 0 deletions examples/hand_detector/THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
MediaPipe Hand Tracking Models
------------------------------

This project includes PyTorch conversions of the MediaPipe Hand Tracking
palm detection and hand landmark models.

The original models are licensed under the Apache License, Version 2.0.

Modifications:
- Converted the original TensorFlow Lite models to PyTorch modules.
- Adapted tensor layouts and operator implementations for PyTorch.
- Added quantization and model-debugging support.
- The converted and quantized models may produce results different from
the original MediaPipe models.
15 changes: 15 additions & 0 deletions examples/hand_detector/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Copyright (c) 2026 Samsung Electronics Co., Ltd. All Rights Reserved
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""MediaPipe palm-detector conversion and quantization example."""
15 changes: 15 additions & 0 deletions examples/hand_detector/_support/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Copyright (c) 2026 Samsung Electronics Co., Ltd. All Rights Reserved
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Internal helpers for the hand-detector example entry points."""
Loading
Loading