Skip to content

fix(tracking): prevent overshoot from stale camera frames - #1396

Open
stefanleyb wants to merge 1 commit into
pollen-robotics:mainfrom
stefanleyb:fix/face-tracking-frame-pose-sync
Open

stefanleyb wants to merge 1 commit into
pollen-robotics:mainfrom
stefanleyb:fix/face-tracking-frame-pose-sync

Conversation

@stefanleyb

Copy link
Copy Markdown

Issue

Closes #1331

Description

Face tracking was projecting a detected face from the head pose at processing time, although the camera frame was captured earlier. While the head was moving, this made the target escalate and caused repeated overshoot/correction.

This records each producer-side camera frame timestamp by IPC buffer offset, retains a short measured head-pose history, and interpolates the pose at capture time before projecting the face into world coordinates. A detector configured for timestamps drops a frame when its timestamp is unavailable instead of silently using a later pose. No public API or daemon route changes.

Testing

  • ruff check and mypy pass for the changed source files.
  • 68 focused unit tests pass, covering timestamp mapping, face tracking, backend pose history, and media-server pose/watchdog/WebRTC paths.
  • Tested on one Reachy Mini Wireless running the 1.10.0 release. In three stationary-face positions, measured target overshoot fell from 12.1–14.9° to 0.7–3.9° and settling time fell from 2.46–5.04 s to 1.14–1.76 s.
  • With instrumentation disabled, a normal daemon tracking run reacquired the operator after natural movement and brief occlusion, then held direct gaze.

Tested on

  • Reachy Mini Wireless
  • Reachy Mini Lite
  • MuJoCo simulation (--sim)
  • Mockup simulation (--mockup-sim)
  • Not applicable (e.g. docs, typo)

AI assistance

  • No AI involvement
  • AI helped with wording or boilerplate
  • AI wrote code here, and I ran and reviewed it
  • An agent produced this PR, and I have not run it myself

Assisted-by: OpenAI Codex: GPT-5

@stefanleyb stefanleyb changed the title fix(tracking): project faces from frame-time head pose fix(tracking): prevent overshoot from stale camera frames Sep 1, 2026
@stefanleyb

Copy link
Copy Markdown
Author

Videos of Before/after, where the overreach of the initial movement is visible in the current version (including blocking the movement because Reachy goes to its limits), which makes it a bit weird and unnatural.
Similar video with the new version, where Reachy directly looks at the user, builds connection and shows curiosity.

https://github.com/user-attachments/assets/7d5fd8d3-0929-41fb-9b25-f8036fa52f24
https://github.com/user-attachments/assets/07bf23fe-d940-45d9-b543-2bdb3d09c494

@RemiFabre

Copy link
Copy Markdown
Member

This is a very good input, thank you. We're reworking how the face tracking works. Would you be ok to test our new implementation for feedback?

@stefanleyb

Copy link
Copy Markdown
Author

@RemiFabre yes, I would definitely be happy to test the new version! I am working on a version where the body moves also, and I think I finally managed to have a clean "natural-looking" movement for the body. At some point, body movement could be a parameter when launching face tracking instead of an app built on top of head-only movement. Feel free to ping me here or on discord (stephaneleyb_29208).

stefanleyb added a commit to stefanleyb/reachy_mini_daemon_layers that referenced this pull request Sep 15, 2026
The layer records overlay 08d9e40ca but never said where that commit comes
from. It is pollen-robotics/reachy_mini#1396, so a reader can follow the fix
to its source — and knows the layer becomes unnecessary if the PR merges.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Head tracking still not working

2 participants