compression: copy a mostly-empty compressed frame out instead of shrinking it - #323
Merged
Merged
Conversation
…nking it `compressed_output` shrank a working buffer that was more than half empty with `shrink_to_fit`. For a large input that buffer comes from mmap, and glibc serves the shrink with mremap, leaving a chunk too small to raise its dynamic mmap threshold when it is freed. Every later compression of a large input therefore mapped a fresh buffer, and flate2's Rust backends zero-fill the whole spare output capacity on each call, faulting in every page of it. Copy the compressed bytes into a buffer of their own and drop the working buffer instead. Freeing the full-size buffer raises the threshold, so later calls reuse heap memory, and the frame now retains exactly its compressed size. Signed-off-by: Iain McGinniss <309153+iainmcgin@users.noreply.github.com>
iainmcgin
marked this pull request as ready for review
September 26, 2026 01:53
rpb-ant
approved these changes
Sep 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#291 shrinks a compressed frame's working buffer with
shrink_to_fitwhen it is more than half empty. For a large input that buffer is above glibc's mmap threshold, so the shrink is anmremap, and freeing the few-KB chunk it leaves never raises glibc's dynamic threshold. Every later compression of a large input then maps a fresh buffer, and flate2's Rust backends zero-fill its whole spare capacity, faulting in every page.On a c7i.metal-24xl (turbo off, pinned), gzip of a 1 MiB compressible input:
main(#291,shrink_to_fit)This copies the compressed bytes into a buffer of their own and drops the working buffer, in the helper gzip and zstd share, which raises the threshold (up to glibc's 32 MiB cap) so later calls reuse heap memory. A queued frame now retains exactly its compressed size; the retention tests assert that. They would also pass with
shrink_to_fit, since the allocator effect is not testable in-process.