Skip to content

chore(deps): update base-image to v202609151314 (Node 24.21) to fix ML job memory growth - #1146

Closed
Deeds67 wants to merge 1 commit into
mainfrom
fix/base-image-node-blob-leak
Closed

Deeds67 wants to merge 1 commit into
mainfrom
fix/base-image-node-blob-leak

Conversation

@Deeds67

@Deeds67 Deeds67 commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator

Fixes #1144.

Problem

The server's jobs process grows by roughly 0.35 MiB per machine learning job and never releases it. On a large pet detection reset this reaches tens of GiB (the reporter saw ~0.4 GiB per 1,000 jobs; only a restart frees it).

Cause

Node 24.16 through 24.19 never release a Blob once fetch has streamed it as a request body. MachineLearningRepository.getFormData() sends every /predict request as FormData with the preview image in a Blob, so each ML request (face detection, smart search, OCR, pets) keeps one preview in memory. Our current base image, 202608300913, ships Node 24.18.1. Node 24.20.0 fixed the bug.

Evidence:

  • Heap snapshot of the jobs worker thread on a clone of a real library (v5.7.0): after 3,184 successful pet detection jobs, exactly 3,184 retained Node / BackingStore objects (1.16 GiB), each owned by a Blob held by a native global handle, surviving a forced GC.
  • Minimal reproduction, for await (const _ of blob.stream()); on 400 KB blobs, 2,000 iterations: 24.15.0 → 25 MiB, 24.18.1 → 763 MiB, 24.19.0 → 812 MiB, 24.20.0 → 21 MiB, 24.21.0 → 9 MiB.

Fix

Cherry-pick upstream's base-image bump (immich-31585, already on the rolling v3.3.0 branch) to main ahead of the rebase. base-server-prod:202609151314 ships Node 24.21.0. The server code is unchanged.

Validation

On a clone of the personal instance running this branch's RC image (fix-base-image-node-blob-leak-rc1, Node 24.21.0):

  • Leak: 4,015 successful pet detection jobs (in-pod stub ML): jobs-worker arrayBuffers 6 → 7 MiB after forced GC. On v5.7.0 the same test gave 10 → 1,169 MiB for 3,184 jobs.
  • Real ML service: one smart search and one single-asset face detection both succeeded.

Not covered by the clone (read-only S3): thumbnail generation and video transcoding under the new libvips/ffmpeg in the base image. Those rely on this PR's e2e CI.

Cherry-picked from upstream immich-31585 ahead of the v3.3.0 rebase.

The new base image ships Node 24.21.0 instead of 24.18.1. Node 24.16
through 24.19 never release a Blob after fetch streams it as a request
body, and every machine learning request sends the preview image as a
Blob in FormData. The server therefore kept one preview in memory per ML
request until restart, about 0.35 MiB per job on a large pet detection
reset. Node 24.20.0 fixed this.
@Deeds67 Deeds67 added the changelog:fix Bug fix for changelog label Sep 25, 2026
Deeds67 pushed a commit that referenced this pull request Sep 28, 2026
Backport of upstream immich-31854, shipped in Immich v3.2.4. Supersedes #1146:
the new base image also ships a Node release past the 24.16-24.19 Blob stream
leak (#1144).

(cherry picked from commit 6786b82)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Daniel Dietzler <mail@ddietzler.dev>
Deeds67 added a commit that referenced this pull request Sep 28, 2026
* fix(mobile): sync status page goes blank when the counts query fails

Backport of upstream immich-31644 (cherry-pick of immich-31642), shipped in
Immich v3.2.3.

(cherry picked from commit 91d34f9)

Co-authored-by: Santo Shakil <santoshakil@users.noreply.github.com>

* chore(deps): update base-image to v202609281550

Backport of upstream immich-31854, shipped in Immich v3.2.4. Supersedes #1146:
the new base image also ships a Node release past the 24.16-24.19 Blob stream
leak (#1144).

(cherry picked from commit 6786b82)

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Daniel Dietzler <mail@ddietzler.dev>

* chore: record Immich v3.2.4 as the upstream base

With the two v3.2.3/v3.2.4 changes backported, main now contains everything
in upstream v3.2.4. Neither release touches server/src/schema/, so the
revert-to-immich script and the migration coverage are unchanged.

---------

Co-authored-by: immich-push-o-matic[bot] <179150890+immich-push-o-matic[bot]@users.noreply.github.com>
Co-authored-by: Santo Shakil <santoshakil@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Daniel Dietzler <mail@ddietzler.dev>
@Deeds67

Deeds67 commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator Author

Merged in #1151

@Deeds67 Deeds67 closed this Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog:fix Bug fix for changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Server memory grows unbounded during Pet Recognition reset/backfill (~0.4 GiB per 1,000 jobs)

1 participant