STORM-3871: remove a topology's dependency artifact blobs when it is cleaned up - #9011
Conversation
|
Disclaimer: this comment was generated with the help of an LLM. The added cleanup tests are thorough — the reference-counting, the fail-safe fallback, and the UUID-shape edge cases are all well covered. One small coverage gap: |
… code blob Review feedback on #9011: a KeyNotFoundException while collecting dependency references means the candidate topology contributes no references, unlike a read failure it does not abort the scan or trigger the conservative fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1ea0dc0 to
5819a2b
Compare
… code blob Review feedback on #9011: a KeyNotFoundException while collecting dependency references means the candidate topology contributes no references, unlike a read failure it does not abort the scan or trigger the conservative fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5819a2b to
7b190a6
Compare
Fixes #7653 / STORM-3871.
Nimbus.rmDependencyJarsInTopologyread onlyStormTopology.get_dependency_jars()and neverget_dependency_artifacts(), so every blob uploaded through--artifactswas left behind when the topology was cleaned up. Of the five places in the tree that read those two fields, this was the only one looking at jars but not artifacts. The blobs are then unrecoverable: adep-key carries no topology id, so once the owning-stormcode.seris gonetopoIdsToCleancan never rediscover them. That matches the report, where blobs from days earlier are still present.Cleanup now collects candidates from both fields and deletes a key only when a cluster-wide scan of the other topologies dependency lists shows nothing else references it. That scan is what makes artifact reclamation safe: before c78ab74 artifact keys were deterministic and deliberately shared between topologies (STORM-2016: "don t remove artifacts from blobstore since it s shared across topologies"), and a cluster upgraded across that commit still holds such blobs.
If the scan itself fails, and one unreadable topology blob is enough since
Utils.deserializethrowsRuntimeException, the pass falls back to reclaiming only keys whose shape proves they belong to a single upload (the UUID thatapplyUUIDToFileNamestamps), and logs what it kept and why. Without that fallback a single corrupt blob would orphan every dying topology dependencies permanently, which is worse than the behaviour being fixed.Tests cover the success path, a legacy key shared by another live topology, and the degraded path. storm-server: 494 tests, 0 failures.