Skip to content

STORM-3871: remove a topology's dependency artifact blobs when it is cleaned up - #9011

Merged
rzo1 merged 2 commits into
masterfrom
fix/blobstore-dependency-cleanup
Aug 24, 2026
Merged

STORM-3871: remove a topology's dependency artifact blobs when it is cleaned up#9011
rzo1 merged 2 commits into
masterfrom
fix/blobstore-dependency-cleanup

Conversation

@rzo1

@rzo1 rzo1 commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Fixes #7653 / STORM-3871.

Nimbus.rmDependencyJarsInTopology read only StormTopology.get_dependency_jars() and never get_dependency_artifacts(), so every blob uploaded through --artifacts was left behind when the topology was cleaned up. Of the five places in the tree that read those two fields, this was the only one looking at jars but not artifacts. The blobs are then unrecoverable: a dep- key carries no topology id, so once the owning -stormcode.ser is gone topoIdsToClean can never rediscover them. That matches the report, where blobs from days earlier are still present.

Cleanup now collects candidates from both fields and deletes a key only when a cluster-wide scan of the other topologies dependency lists shows nothing else references it. That scan is what makes artifact reclamation safe: before c78ab74 artifact keys were deterministic and deliberately shared between topologies (STORM-2016: "don t remove artifacts from blobstore since it s shared across topologies"), and a cluster upgraded across that commit still holds such blobs.

If the scan itself fails, and one unreadable topology blob is enough since Utils.deserialize throws RuntimeException, the pass falls back to reclaiming only keys whose shape proves they belong to a single upload (the UUID that applyUUIDToFileName stamps), and logs what it kept and why. Without that fallback a single corrupt blob would orphan every dying topology dependencies permanently, which is worse than the behaviour being fixed.

Tests cover the success path, a legacy key shared by another live topology, and the degraded path. storm-server: 494 tests, 0 failures.

@rzo1 rzo1 added this to the 3.1.0 milestone Aug 23, 2026
@rzo1 rzo1 self-assigned this Aug 23, 2026
@reiabreu

Copy link
Copy Markdown
Contributor

Disclaimer: this comment was generated with the help of an LLM.

The added cleanup tests are thorough — the reference-counting, the fail-safe fallback, and the UUID-shape edge cases are all well covered. One small coverage gap: referencedDependencyKeys treats a KeyNotFoundException (a candidate topology with no code blob) as contributing no references and continues the scan, which is a different branch from the read-failure path that doCleanupReclaimsOnlyProvablyUniqueDependencyBlobsWhenTheReferencesCannotBeRead exercises (an IOException that aborts the scan and triggers the fallback). A test with a candidate topology whose code blob is missing — asserting the scan still succeeds and the other topologies' references are honored — would close that branch. Non-blocking.

rzo1 added a commit that referenced this pull request Aug 24, 2026
… code blob

Review feedback on #9011: a KeyNotFoundException while collecting dependency
references means the candidate topology contributes no references, unlike a
read failure it does not abort the scan or trigger the conservative fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rzo1
rzo1 force-pushed the fix/blobstore-dependency-cleanup branch from 1ea0dc0 to 5819a2b Compare August 24, 2026 08:18
rzo1 and others added 2 commits August 24, 2026 10:54
… code blob

Review feedback on #9011: a KeyNotFoundException while collecting dependency
references means the candidate topology contributes no references, unlike a
read failure it does not abort the scan or trigger the conservative fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rzo1
rzo1 force-pushed the fix/blobstore-dependency-cleanup branch from 5819a2b to 7b190a6 Compare August 24, 2026 08:55
@rzo1
rzo1 merged commit 4a1c21a into master Aug 24, 2026
7 checks passed
@rzo1
rzo1 deleted the fix/blobstore-dependency-cleanup branch August 24, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[STORM-3871] Storm blobstore leak space

2 participants