Repository navigation
Conversation
ZSTD_decompressStream() only recognized a legacy frame when the current call started with the full magic number and at least 5 bytes. When the first call supplied 1 to 4 bytes, the early magic check rejected the legacy prefix, and on later calls the legacy detection looked at the current input instead of the bytes already buffered in headerBuffer. Accept legacy magic prefixes in the early check, detect legacy frames from headerBuffer, and replay the buffered bytes to the legacy decoder before the rest of the input. Fixes meta#4802
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4802
Root cause
With legacy support enabled,
ZSTD_decompressStream()only recognized a v0.4 to v0.7 frame when a single call started with the magic number and carried at least 5 bytes. Two things broke a smaller first call:headerBufferandZSTD_getFrameHeader_advanced()checks that they match a supported magic number. That check only knew the zstd and skippable magics, so a legacy prefix returnedUnknown frame descriptorright away.ZSTD_isLegacy()on the current input instead of on the bytes already buffered inheaderBuffer, so the magic was never seen there either.Fix
headerBuffer, replay those buffered bytes to the legacy stream decoder, then pass the rest of the input as before.When the whole header comes in one call, the decoder gets the same bytes in the same order, so that path is unchanged.
Side effect: with legacy support compiled in,
ZSTD_getFrameHeader()given 1 to 4 bytes of a legacy frame now asks for 5 bytes instead of returningprefix_unknown, as it already does for a zstd or skippable prefix. With 5 or more bytes it still returns an error for a legacy frame, as before.Test
tests/legacy.c:testStreamingAPI()now takes a chunk size and is also run feeding the input one byte at a time, so every frame header is split across calls. The helper also stops callingZSTD_initDStream()on every iteration (needsInitwas never cleared), which would otherwise reset the stream between chunks.Before the fix (MSVC build,
ZSTD_LEGACY_SUPPORT=1):After the fix:
The reproducer from the issue now reports
consistentfor first calls of 1 to 8 bytes.zstreamtest -T30sandfuzzer -T30s(withZSTD_LEGACY_SUPPORT=5) pass, andtests/legacy.calso passes with legacy support disabled.