Server memory grows unbounded during Pet Recognition reset/backfill (~0.4 GiB per 1,000 jobs)

[bernhardberger]

Version: gallery-server v5.7.0 (Docker), gallery-ml v5.7.0-cuda on a remote ML host (RTX 4090), Postgres 14-vectorchord0.4.3-pgvectors0.2.0, Valkey 9.

Setup: ~84k assets (images + external read-only libraries), 5 users. Pet detection rfdetr-nano (minScore 0.75), pet recognition pet-recognition-base (maxDistance 0.55, minFaces 1).

What happened: I started a full reset (PUT /api/jobs/petRecognition {"command":"start","force":true}). While detection jobs ran, the RSS of immich_server rose roughly linearly, about 0.4 GiB per 1,000 completed jobs. It reached 17.9 GiB of a 19.5 GiB limit and was not released on its own. Restarting the server container freed the memory immediately. The resumed queue then finished normally, with no failed jobs.

Expected: memory stays flat during a long pet detection/recognition run.

Workaround: pause the queue, restart immich_server and resume it whenever memory passes a threshold.

Notes: ML ran in a separate remote container, so the growth is in the server process itself. I haven't tested whether face detection/recognition resets show the same pattern at the same scale. Happy to provide a heap snapshot if you tell me which worker/flag to use.

[Deeds67]

Hey, thanks for reporting it.

We've started looking. The pet detection and recognition job handlers don't keep state between jobs, and the request path to the ML service (which is shared with face detection, smart search and OCR) didn't grow in an isolated test, so we haven't found the cause yet. A few things would help narrow it down:

1. Which process grows? immich_server runs two Node processes: immich-api (HTTP) and immich, which runs the background jobs. While memory is climbing, could you run this a couple of times a few minutes apart and paste the output?

docker exec immich_server sh -c 'for p in /proc/[0-9]*; do printf "%s  %s\n" "$(grep VmRSS $p/status 2>/dev/null)" "$(tr "\0" " " < $p/cmdline)"; done' | grep -i immich

2. Allocator. Near the top of the immich_server startup log, is there a line "skipping libmimalloc - path not found ..."?
3. Where are your files stored? Are the preview/thumbnail files on local disk, or on S3-compatible storage?
4. Were other queues busy at the same time? For example thumbnail generation, face detection or smart search — and what concurrency do you have set for the Pet Detection queue?
5. Have you seen this with other jobs? During earlier large runs — the initial face detection or smart search over your library, for example — did immich_server memory also climb like this, or is pet detection the first time you've noticed it?

On the heap snapshot: we'll likely take you up on it. The background jobs run in a worker thread inside the immich process, so the usual signal-triggered snapshot wouldn't capture them. We'll follow up with exact steps rather than send you on a detour. When you do share one, please send it privately rather than attaching it here: a heap snapshot can contain file paths, config values and other data from your instance.

[bernhardberger]

Thanks! I re-ran the full pet reset (force: true, ~83k assets) to collect this. I sampled once a minute and restarted the server whenever the container passed 12 GiB, the same safeguard as the first run.

1. Which process: it's the immich (jobs) process. immich-api stays flat.

12:14:29 cgroup=3.67GiB  immich-api=333MiB immich=3207MiB   petDetection waiting=77441
12:16:32 cgroup=7.02GiB  immich-api=334MiB immich=6403MiB   waiting=67393
12:18:34 cgroup=11.03GiB immich-api=304MiB immich=10122MiB  waiting=58089
12:19:36 cgroup=13.0GiB  immich-api=286MiB immich=11825MiB  waiting=53637  -> restart
12:21:08 cgroup=3.54GiB  immich-api=512MiB immich=2924MiB   waiting=49330
12:28:19 cgroup=12.92GiB immich-api=283MiB immich=11514MiB  waiting=27997  -> restart
12:29:56 cgroup=2.78GiB  immich-api=500MiB immich=2228MiB   waiting=24899
12:38:06 cgroup=11.6GiB  immich-api=269MiB immich=10293MiB  waiting=0 (queue drained)
12:48:21 cgroup=11.37GiB immich-api=303MiB immich=10169MiB  (after 10 min idle, not released)

• The growth rate was consistent across all three segments: 0.32–0.40 MiB per completed pet-detection job.
• The memory is almost entirely anonymous. A mid-run sample of /proc/<pid>/status for immich showed VmRSS 6.25 GB, RssAnon 6.16 GB, RssFile 85 MB, RssShmem 0, 29 threads.
• It isn't released after the queue drains. Only a restart frees it.

2. Allocator: there's no "skipping libmimalloc" line in the startup log, and /usr/lib/x86_64-linux-gnu/libmimalloc.so.3 exists in the image. So I assume mimalloc is loaded.

3. Storage: no S3; everything is on the filesystem. Thumbnails/previews are on local disk (LXC rootfs). Originals and uploads are on an NFSv4 mount, and the external libraries are read-only bind mounts of that same NFS share.

4. Other queues: effectively only petDetection was running. A few samples showed brief thumbnailGeneration, facialRecognition or petRecognition activity, but nothing sustained. Concurrency: petDetection 2, faceDetection 2, smartSearch 2, thumbnailGeneration 3, metadataExtraction 5, ocr 1, classification 4, the rest at defaults. ML runs on a remote CUDA machine-learning container (gallery-ml:v5.7.0-cuda), so inference itself isn't in this process.

5. Other jobs: I don't have comparable measurements. The initial face detection and smart search for this library ran long ago, and we weren't tracking server memory then. I didn't notice it before, but that's not strong evidence.

For the heap snapshot, happy to follow your steps and send it privately.

[photoserver-pro]

Sorry, uh, if I'm barging into this technical discussion inappropriately. ;-)
I've made the following observation, but I can't back it up with numbers yet. So for now this is just the user experience, without technical validation.
**When the following processes are running on my end (face detection, smart search, auto-classification), Noodle becomes extremely slow for users who aren't the owners of the images. For the admin who owns the images, this slowdown isn't noticeable. **

Since I have a lot of RAM (48 GB), I've never actually run into a real RAM issue. But I am seeing these performance problems.
I'm just putting it out there as a thought that might be related. I'll try to reproduce this over the weekend, though, and validate it with data. Maybe there are permission-checking processes running here that then collide with these three ML-driven and DB Heavy tasks and compete for resources.

[Deeds67]

What's happening: it's a bug in Node.js, not in the pet detection code. In Node versions 24.16 through 24.19, when the server uploads an image in a request, the runtime keeps that image in memory after the request finishes. Every machine learning job uploads its preview image to the ML service this way, so each job left one preview behind, about 0.35 MiB. That matches your 0.32–0.40 MiB per job and your RssAnon figure. The v5.7.0 server image ships Node 24.18.1. Node fixed the bug in 24.20.0.

This affects every machine learning job, not just pets: face detection, smart search and OCR too. A full pet reset is simply the largest bulk run, which is why it showed up there.

How we confirmed it: we reproduced the growth on v5.7.0 against a copy of a real library, at 0.36 MiB per completed job. A heap snapshot of the jobs worker showed exactly one retained image per job. We then ran the same test on a server image with Node 24.21.0 and memory stayed flat: about 1 MiB retained after 4,000 jobs.

The fix: #1146 updates the server's base image to the one with Node 24.21.0, the same update upstream Immich has already made. It will ship in the next server release.

Until then, restarting immich_server when memory gets high is still the right workaround, as you've been doing. The queue picks up where it left off.

No heap snapshot needed. Thanks for offering, and for the very clear data.