PoPe API Inference Service

Small FastAPI service that queues Record3D inference jobs, runs egoallo/3_record3d_inference.py one at a time using a single background worker, and lets clients poll job status until completion.

It also serves a lightweight browser UI at / where you can drag and drop a .r3d file, watch the job status update, inspect or download the output artifacts, and launch trajectory visualization as a separate post-run action.

Note: Job history is kept in memory only. Restarting the service will lose all existing job records.

What This Repo Assumes

This service is a wrapper around the adapted egoallo inference script for Record3D videos recorded on iphone with Lidar. For a real run, the machine still needs:

• a working Python environment for egoallo
• the egoallo repo checked out locally
• the EgoAllo checkpoint/data downloaded
• the SMPL-H model file placed where the upstream script expects it

The service is easiest to run when it lives next to a local egoallo checkout like this:

api-inference/
├── egoallo/
├── main.py
├── pyproject.toml
├── service/
└── tests/

For the /jobs/{job_id}/scene endpoint (which returns 3D scene JSON for the frontend viewer), the frontend repo must be a sibling of api-inference so the service can run frontend/scripts/export_3d.py to build the scene from inference outputs.

If you keep egoallo somewhere else, set INFERENCE_SCRIPT_PATH and INFERENCE_WORKDIR explicitly.

Quick Start

These steps are the recommended "fresh machine" flow for someone who wants the web UI and the API to work without guessing.

1. Clone the repos

git clone https://github.com/modellingpeople/api-inference.git
cd api-inference
git clone https://github.com/modellingpeople/egoallo.git

2. Create and activate a Python environment

Use the same environment for both egoallo and this service.

egoallo's upstream repo says to use Python 3.12 or newer. That is the safest choice for real inference runs.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

3. Install egoallo and this service

cd egoallo
pip install -e .
cd ..

pip install '.[dev]'
pip install pyliblzfse

Notes:

• pyliblzfse is needed by the Record3D script to read LiDAR depth data correctly.
• The service defaults INFERENCE_PYTHON_BIN to the same Python interpreter that starts uvicorn, so always launch the service from this activated environment.

4. Install the extra EgoAllo inference dependencies

The upstream egoallo README calls out extra inference dependencies, especially JAX and jaxls.

CPU example:

pip install jax==0.6.1 jaxlib==0.6.1
pip install git+https://github.com/brentyi/jaxls.git

If you want NVIDIA GPU acceleration, follow the upstream egoallo instructions instead of the CPU example above:

pip install "jax[cuda12]==0.6.1"
pip install git+https://github.com/brentyi/jaxls.git

5. Download the EgoAllo checkpoint and example data

From the upstream repo:

cd egoallo
bash download_checkpoint_and_data.sh
cd ..

This is important because 3_record3d_inference.py defaults to checkpoint paths inside the egoallo repo.

6. Place the SMPL-H model where the script expects it

The upstream script expects this file by default:

egoallo/data/smplh/neutral/model.npz

You need to download the "Extended SMPL+H model" from the MANO / SMPL-H source linked in the upstream egoallo README, then place the .npz file at exactly that path.

Example:

mkdir -p egoallo/data/smplh/neutral
# then copy your downloaded model.npz to:
# egoallo/data/smplh/neutral/model.npz

7. Start the service

From the api-inference repo root:

source .venv/bin/activate
uvicorn main:app --host 0.0.0.0 --port 8000

8. Open the web UI

open http://127.0.0.1:8000/

If you are not on macOS, open that URL in your browser manually.

9. Run your first job

You have two options:

• Drag and drop a local .r3d file into the browser UI.
• Submit a server-local path to an existing .r3d file through the UI or API.

When a job finishes, the UI shows the output directory and lets you inspect artifacts like:

• stdout.log
• stderr.log
• generated .npz output files
• any other files written by 3_record3d_inference.py

If you want the upstream --visualize-traj experience, launch it after the job reaches succeeded. The UI exposes a button for this, and the API exposes a dedicated endpoint.

By default, each job writes into:

./runs/<job_id>/

Uploaded .r3d files are stored under a per-upload UUID subdirectory to avoid filename collisions:

./uploads/<upload_uuid>/<original_filename>.r3d

Configuration

If your local layout matches:

api-inference/
├── egoallo/
└── ...

then the service should discover the inference script automatically.

Optional overrides:

export INFERENCE_SCRIPT_PATH=/absolute/path/to/3_record3d_inference.py
export INFERENCE_PYTHON_BIN=/absolute/path/to/python
export OUTPUT_ROOT=/absolute/path/to/runs
export UPLOAD_ROOT=/absolute/path/to/uploads
export INFERENCE_WORKDIR=/absolute/path/to/egoallo/repo
export VISUALIZATION_SCRIPT_PATH=/absolute/path/to/service/visualize_record3d.py
export VISUALIZATION_HOST=127.0.0.1

Defaults:

• INFERENCE_SCRIPT_PATH: searches ./egoallo/3_record3d_inference.py, ../egoallo/3_record3d_inference.py, and the path relative to the installed package directory
• INFERENCE_PYTHON_BIN: the same interpreter that started the service
• INFERENCE_WORKDIR: the parent directory of the resolved inference script
• OUTPUT_ROOT: ./runs
• UPLOAD_ROOT: ./uploads
• VISUALIZATION_SCRIPT_PATH: resolved from inside the installed service package directory (not relative to CWD)
• VISUALIZATION_HOST: 127.0.0.1
• DEFAULT_TRAJ_LENGTH: not configurable via environment variable — the built-in default is 256; override per job via traj_length in the request body

The service passes --output-dir directly to 3_record3d_inference.py, so each job gets its own deterministic output folder under OUTPUT_ROOT.

API Examples

Open the browser UI:

open http://127.0.0.1:8000/

Submit a job using an existing local path on the server (omit traj_length to use the default of 256; omit guidance_mode to use no_hands):

curl -X POST http://127.0.0.1:8000/jobs \
  -H 'Content-Type: application/json' \
  -d '{
    "r3d_path": "/path/to/recording.r3d",
    "traj_length": 512,
    "guidance_mode": "no_hands"
  }'

guidance_mode can be no_hands (default), hamer_wrist, or hamer_reproj2. HaMeR modes require hamer_helper and are slower but improve hand pose estimation.

Poll a job:

curl http://127.0.0.1:8000/jobs/<job_id>

Upload a .r3d file (omit traj_length to use the default of 256; omit guidance_mode to use no_hands):

curl -X POST http://127.0.0.1:8000/jobs/upload \
  -F "file=@/path/to/recording.r3d" \
  -F "traj_length=512" \
  -F "guidance_mode=no_hands"

Get 3D scene JSON for a succeeded job (used by the frontend viewer; requires frontend repo as sibling):

curl http://127.0.0.1:8000/jobs/<job_id>/scene

List artifacts for a finished job:

curl http://127.0.0.1:8000/jobs/<job_id>/artifacts

Launch trajectory visualization after the job succeeds:

curl -X POST http://127.0.0.1:8000/jobs/<job_id>/visualize

Check service health (worker liveness and queue depth):

curl http://127.0.0.1:8000/health

Troubleshooting

ModuleNotFoundError from egoallo or one of its dependencies

You are almost certainly starting the service from the wrong Python environment. Activate the same environment where you installed egoallo, then start uvicorn again.

The service starts, but inference fails immediately

Check the job artifacts in the UI or under runs/<job_id>/:

• stderr.log
• stdout.log

The most common causes are:

• missing EgoAllo checkpoint files
• missing egoallo/data/smplh/neutral/model.npz
• missing JAX or jaxls
• launching the service from a Python environment that does not have the egoallo dependencies installed

Record3D depth or point cloud processing fails

Make sure pyliblzfse is installed in the active environment:

pip install pyliblzfse

I do not keep egoallo nested inside this repo

Set these before starting the service:

export INFERENCE_SCRIPT_PATH=/absolute/path/to/egoallo/3_record3d_inference.py
export INFERENCE_WORKDIR=/absolute/path/to/egoallo