[bpacholek]

Part of #182: Follow face. Reframe's Face mode is left as it is (see below), so the issue stays open for it.

Follow face found no face in 4K street footage: the detector sees the whole frame at 128 × 128, where a face a few dozen pixels of a 720p proxy is a few pixels.

What changed

• FaceDetector.detect takes a region of the picture, and gives its boxes in fractions of the whole picture.

• When the mask is on a face too small for the whole frame, FaceTracks also looks at overlapping squares of the full frame, then merges the boxes as before. FaceTracking.tiles sizes the squares so the face is about 20 px of the detector's 128: 12 squares for a face 1/20 of a 720p frame wide, at most 15 in 16:9. With a bigger face nothing changes.

• The squares also find other people, and things like faces (a sign, a car's rear, a dog) scoring as high as real faces (0.53 to 0.80 for both on the footage). So FaceTracking.follow no longer takes the nearest face whatever the distance. The same face must be:

  • within 2.5 of its widths of where it was last seen (at least 0.05 of the frame), a little more while it is lost, at most 1.5 times that;
  • half to twice its size, compared with the face seen, not the mask, which can be drawn any size.

  Beyond that the sample has no face, and track fills it as before.

Checked

• On the desktop, with MediaPipe's own Python package, on 15 frames of three street clips at 720p: the whole frame (480 px, as before) found no face in any; 4 × 3 squares found faces in 9.
• On the emulator, with a temporary test (not committed) on a 13.5 s street clip at 720p, the mask on a woman's face (about 50 px) at 5 s: the path follows her from 3.9 s until she leaves the frame at 7.4 s (35 of 135 samples, checked by drawing the boxes on the frames). Analysis took 14 s on the emulator.
  • Before the reach rule, the same run "found" 75 samples: the path jumped to a waiter, children at a table, a box on the pavement and a cyclist.
• FaceDetectorTest.findsASmallFaceInARegion (ml:vision): the test portrait at 70 px in a 1280 × 720 frame is found in a region (0.93) and not in the whole frame.
• FaceTracksTest.aSmallFaceIsFollowedFromTheMaskOnIt (core:engine): the portrait moving across a 9:16 frame at 0.3 scale is followed within 0.02 of the frame. It took 4.7 s for 30 samples, against 2.6 s for the close face.
• FaceTrackingTest: smallFacesAreLookedForInTilesOfThePicture and aLostFaceDoesNotJumpToSomeoneElse.
• FaceTracksTest and ReframerTest (all four) pass on the emulator.

Not done: Reframe's Face mode. It has no mask to start from and follows the surest face. With the squares, a sign or a car scores as high as a person, so it could frame those. It keeps to the whole frame. Options for it: BlazeFace full range (made for faces up to 5 m), or requiring a face to be seen in several samples in a row before following it.

🤖 Generated with Claude Code