Repository navigation
fix(s3stream): preserve pending multipart buffers through copy and release - #3652
Open
ashsamantaray-stripe wants to merge 3 commits into
Open
ashsamantaray-stripe wants to merge 3 commits into
ashsamantaray-stripe wants to merge 3 commits into
Conversation
…lease Committed-By-Agent: codex Co-authored-by: codex <noreply@openai.com>
ashsamantaray-stripe
requested review from
1sonofqiu,
Gezi-lzq and
superhx
as code owners
October 12, 2026 09:21
…arts Committed-By-Agent: codex Co-authored-by: codex <noreply@openai.com>
Committed-By-Agent: codex Co-authored-by: codex <noreply@openai.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Upstream #3528 followed the original #3527 and is already in this PR's base. It replaced the deferred handoff and addressed several review concerns before this PR was opened.
ProxyWriterreplays small copy and write operations intoMultiPartWriteronce the object exceeds the single-upload threshold. A range read for the current multipart part can still be pending when the writer is released.MultiPartWriter.release()currently frees that part buffer and completes before the read finishes, so the read can later append to a released buffer. A write queued behind the read also remains backed by the caller's buffer aftercopyOnWrite()returns.Replay can also leave a 28 MiB write buffered before an 8 MiB copy. The copy path then subtracts 28 MiB from the minimum part size and constructs an invalid range end.
Repeated PutObject close calls can submit an empty overwrite. Release can also return while a PutObject replay still owns buffers and its upload is pending.
What
Already fixed by upstream #3528
copyOnWrite().Fixed in this PR
close()idempotent and makerelease()wait for active replay and upload cleanup.How
Release joins the already submitted part uploads with cleanup of the current, unsubmitted part. The current part releases its buffer only after its ordered range-read chain finishes. Pending writes retain their own buffer reference until that chain appends or rejects them;
copyOnWrite()replaces those references with private copies. Access to the part buffer and pending-write list is synchronized across caller and range-read completion threads.When a part is submitted while a range read is pending, it snapshots cached buffers before the writer drops its reference to that part. Copy replay flushes an existing part that already meets the minimum size before deriving a new copy range.
The PutObject close path returns the same future on later calls. Release waits for that future, which completes after the replay and storage write have cleaned up their owned buffers.
Reviewer notes
Start with
MultiPartWriter.ObjectPartfor pending buffer ownership, thenMultiPartWriter.release()for cleanup completion andcopyWrite0()for the range boundary.ProxyWriter.close()andrelease()cover the PutObject lifecycle.ProxyWriterTestexercises both paths.Verification:
./gradlew --build-cache metadata:S3UnitTest core:S3UnitTest s3stream:test;./gradlew --build-cache rat checkstyleMain checkstyleTest spotlessJavaCheck;./gradlew :s3stream:spotbugsMain.