Fix protobuf temp directory leak in ProtoBufUtils - #19378
Open
mayankshriv wants to merge 1 commit into
Open
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #19378 +/- ##
=========================================
Coverage 67.55% 67.55%
Complexity 1430 1430
=========================================
Files 3486 3486
Lines 224100 224130 +30
Branches 35370 35376 +6
=========================================
+ Hits 151392 151420 +28
Misses 60678 60678
- Partials 12030 12032 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ProtoBufUtils.getFileCopiedToLocal() created temporary directories
with prefix "pinot-protobuf" that were never cleaned up. Every call
to ProtoBufMessageDecoder.init(), ProtoBufRecordReader.init(), or
ProtoBufCodeGenMessageDecoder.init() leaked one directory under
java.io.tmpdir.
For descriptor file callers (ProtoBufMessageDecoder, ProtoBufRecordReader):
replace the copy-to-temp-then-open pattern with readDescriptorFileBytes()
which streams bytes directly via PinotFS.open() - no temp files at all.
For the JAR caller (ProtoBufCodeGenMessageDecoder): resolve the JAR
to a local File inline. Local JARs are used directly without copying.
Remote JARs are copied to a temp directory that intentionally persists
for the decoder's lifetime because the JVM may lazily resolve classes
from the JAR at decode time via the URLClassLoader chain. Since
StreamMessageDecoder does not extend Closeable, there is no lifecycle
hook to clean up, but this is a one-time-per-consumer-init cost and
consumers are long-lived.
Removed dead methods: getFileCopiedToLocal(), getDescriptorFileInputStream(),
createLocalFile(), withLocalFile(), and FileAction from ProtoBufUtils.
Renamed loadClass(File) to createClassLoader(File) for clarity.
Testing:
- Added ProtoBufTempFileLeakTest with 5 tests covering all three code
paths. Each test snapshots pinot-protobuf* directories in java.io.tmpdir
before the operation, performs the operation with functional correctness
assertions (decoding a message and checking field values), then asserts
no new temp directories remain afterward.
- testMessageDecoderInitDoesNotLeakTempDir: simple descriptor via
ProtoBufMessageDecoder, verifies decode of sample record fields.
- testMessageDecoderComplexDescriptorDoesNotLeakTempDir: complex
nested descriptor via ProtoBufMessageDecoder.
- testCodeGenDecoderInitDoesNotLeakTempDir: simple JAR via
ProtoBufCodeGenMessageDecoder, verifies decode after init cleanup.
- testCodeGenDecoderComplexJarDoesNotLeakTempDir: complex JAR with
nested/repeated/map types via ProtoBufCodeGenMessageDecoder.
- testRecordReaderLifecycleDoesNotLeakTempDir: full ProtoBufRecordReader
lifecycle (init, read, close) with delimited protobuf data file.
- All 5 tests confirmed to fail before the fix (each leaking 1 temp
directory) and pass after. Full module suite: 172/172 tests pass.
mayankshriv
force-pushed
the
fix-protobuf-temp-dir-leak
branch
from
August 28, 2026 14:23
8a225a4 to
d3a58b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ProtoBufUtils.getFileCopiedToLocal()created temporary directories with prefixpinot-protobufthat were never cleaned up. Every call toProtoBufMessageDecoder.init(),ProtoBufRecordReader.init(), orProtoBufCodeGenMessageDecoder.init()leaked one directory underjava.io.tmpdir.Fix
Descriptor file callers (
ProtoBufMessageDecoder,ProtoBufRecordReader): replaced the copy-to-temp-then-open pattern withreadDescriptorFileBytes()which streams bytes directly viaPinotFS.open()- no temp files at all.JAR caller (
ProtoBufCodeGenMessageDecoder): resolve the JAR to a localFileinline. Local JARs are used directly without copying. Remote JARs are copied to a temp directory that intentionally persists for the decoder's lifetime because the JVM may lazily resolve classes from the JAR at decode time via theURLClassLoaderchain. SinceStreamMessageDecoderdoes not extendCloseable, there is no lifecycle hook to clean up, but this is a one-time-per-consumer-init cost and consumers are long-lived.Removed dead methods:
getFileCopiedToLocal(),getDescriptorFileInputStream(),createLocalFile(),withLocalFile(), andFileActionfromProtoBufUtils. RenamedloadClass(File)tocreateClassLoader(File)for clarity.Testing
Added
ProtoBufTempFileLeakTestwith 5 tests covering all three code paths. Each test snapshotspinot-protobuf*directories injava.io.tmpdirbefore the operation, performs the operation with functional correctness assertions (decoding a message and checking field values), then asserts no new temp directories remain afterward.testMessageDecoderInitDoesNotLeakTempDirProtoBufMessageDecodertestMessageDecoderComplexDescriptorDoesNotLeakTempDirProtoBufMessageDecodertestCodeGenDecoderInitDoesNotLeakTempDirProtoBufCodeGenMessageDecodertestCodeGenDecoderComplexJarDoesNotLeakTempDirtestRecordReaderLifecycleDoesNotLeakTempDirProtoBufRecordReaderlifecycle (init, read, close)All 5 tests confirmed to fail before the fix (each leaking 1 temp directory) and pass after. Full module suite: 172/172 tests pass.