Skip to content

fix(motion): keep recorded-move loading off the daemon event loop - #1358

Open
Dariushuangg wants to merge 1 commit into
pollen-robotics:mainfrom
Dariushuangg:fix/recorded-move-blocking-load
Open

Dariushuangg wants to merge 1 commit into
pollen-robotics:mainfrom
Dariushuangg:fix/recorded-move-blocking-load

Conversation

@Dariushuangg

@Dariushuangg Dariushuangg commented Aug 16, 2026

Copy link
Copy Markdown

Issue

No issue filed yet, so the problem is described here per the template.
Happy to split it out if you'd rather track it separately.

Playing any recorded move on a robot that can't reach huggingface.co takes the
whole daemon down. The client sees a disconnect, not a failed download.

play_recorded_move_dataset builds RecordedMoves(dataset_name) per request.
The constructor calls snapshot_download(local_files_only=True) and, on
LocalEntryNotFoundError, falls back to a networked snapshot_download().
That is a synchronous call inside an async def route, so it blocks the uvicorn
event loop: /api/daemon/status, /api/state/full and the state WebSocket all
stop answering until it returns.

Reproduce on a robot with an empty HF cache and huggingface.co blocked (or just
firewall it):

# hangs, and takes every other endpoint with it
curl -m 150 http://<robot>:8000/api/move/recorded-move-datasets/list/pollen-robotics/reachy-mini-emotions-library
curl -m 6   http://<robot>:8000/api/daemon/status   # HTTP 000 while the above runs

The cache is empty in the first place because the wireless launcher starts the
daemon without --preload-datasets, so preload_default_datasets() never runs,
and its failures are swallowed with a warning either way.

Description

Load libraries through a memoized get_recorded_moves() and await it via
asyncio.to_thread in both HTTP routes. _async_play_recorded_move (the WebRTC
data-channel path) already did exactly this — the two HTTP routes never got the
same treatment.

Three effects:

  • The event loop stays free. A cold cache still makes that request slow,
    but the daemon keeps serving everything else, so a blocked HF surfaces as one
    failed move instead of a dead robot.
  • A warm library isn't re-parsed per request. Building the emotions library
    reads 172 files; it now happens once per dataset.
  • Concurrent cold-cache requests share one download. A per-dataset lock
    collapses N in-flight requests into a single build; other datasets stay
    unblocked.

Deliberately not changed here, to keep this to one concern — happy to open
follow-ups:

  • The network fallback is kept, so behaviour on a reachable HF is unchanged. An
    alternative is cache-only playback returning 503, but that regresses robots
    that legitimately download on first use.
  • --preload-datasets still isn't passed by the wireless launcher.
  • Anne-Charlotte/music (surfaced in the desktop app's Dances tab) isn't in
    DEFAULT_DATASETS, so it is never preloaded.

Testing

tests/unit_tests/test_recorded_moves_cache.py, 3 cases, no hardware or
network (snapshot_download stubbed):

  • a library is built once and reused
  • distinct datasets get distinct libraries
  • 8 concurrent requests for a cold dataset produce exactly 1 download
$ uv run pytest tests/unit_tests/test_recorded_moves_cache.py -q
3 passed in 298.17s

$ ruff check .                     # All checks passed!
$ mypy src/reachy_mini/motion/recorded_move.py \
       src/reachy_mini/daemon/app/routers/move.py \
       src/reachy_mini/daemon/backend/abstract.py
Success: no issues found in 3 source files

No route signatures or response models changed, so docs/source/API/openapi.json
is unaffected.

Scope of the hardware evidence: the failure below was measured on a physical
Reachy Mini Wireless, and the recovery numbers come from warming that robot's HF
cache. This patch itself has not been run on the robot — it is covered by the
unit tests above. Flagging that explicitly so a reviewer can decide whether to
ask for an on-robot run before merging.

Tested on

  • Reachy Mini Wireless
  • Reachy Mini Lite
  • MuJoCo simulation (--sim)
  • Mockup simulation (--mockup-sim)
  • Not applicable (e.g. docs, typo)

AI assistance

  • No AI involvement
  • AI helped with wording or boilerplate
  • AI wrote code here, and I ran and reviewed it
  • An agent produced this PR, and I have not run it myself

Annex — measurements on a physical robot

Reachy Mini Wireless, daemon 1.8.3, Debian 13 aarch64, CN network (mobile ISP).

HF reachability, measured from the robot over SSH (not from the dev machine —
a local TUN proxy silently tunnels the dev machine's traffic and reports a false
success):

$ curl -m 12 -o /dev/null -w "%{http_code} %{time_total}s\n" https://huggingface.co
000 12.001134s
$ curl -m 12 -o /dev/null -w "%{http_code} %{time_total}s\n" https://hf-mirror.com
200 0.225316s
$ getent hosts huggingface.co
2a03:2880:f102:183:face:b00c:0:25de     huggingface.co

DNS returns a Facebook-owned IPv6 block, so the connection never establishes.
~/.cache/huggingface did not exist on the robot.

Before — cold cache, HF unreachable:

list endpoint    HTTP 000, timed out at 150.005s
/api/daemon/status   HTTP 000 (6s timeout)  <- collateral damage
/api/state/full      HTTP 000 (6s timeout)  <- collateral damage

After — same robot, cache warmed via HF_ENDPOINT=https://hf-mirror.com
(emotions 172 files / 8.6M, dances 21 files / 484K):

list endpoint                        HTTP 200 in 0.958s (85 moves)
POST .../emotions-library/curious1   HTTP 200 in 0.81s
/api/daemon/status polled at 3.3 Hz throughout playback:
    25 ok / 0 failed, max latency 0.999s

The "after" figures are with a warm cache and therefore show the path this patch
keeps fast; the patch's distinct contribution is that the "before" row no longer
takes the other two endpoints down with it.

Playing a recorded move rebuilt RecordedMoves on every request. Its
constructor calls snapshot_download, which falls back to a networked
download when the HF cache is cold - a blocking call inside an async
route, so it stalled the uvicorn event loop.

Where huggingface.co is unreachable that fallback never returns quickly,
and a single emotion took down every other endpoint and the state stream
with it: /api/daemon/status, /api/state/full and the WebSocket all stop
answering, so clients report the robot as disconnected rather than
surfacing a failed download.

Load libraries through a memoized get_recorded_moves() and await it via
asyncio.to_thread in both HTTP routes, matching what the WebRTC
data-channel path already did. A per-dataset lock collapses concurrent
cold-cache requests into one download instead of one per request, and the
memoization stops a warm library being re-parsed on every playback.

Assisted-by: Claude:claude-opus-5
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Dariushuangg

Copy link
Copy Markdown
Author

my robot is purchase and run by Seeed Studio, which is located in China. That's why there's hugging face reachability problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant