Repository navigation
legacy/v0.7: decode RLE blocks in the streaming path - #4819
Open
ManoharPaturi wants to merge 1 commit into
Open
ManoharPaturi wants to merge 1 commit into
ManoharPaturi wants to merge 1 commit into
Conversation
The v0.7 decoder supports RLE blocks in its one-shot path, expanding them with ZSTDv07_generateNxBytes(), but the block stage of ZSTDv07_decompressContinue() rejects them with a generic error. A frame containing an RLE block therefore decodes through ZSTD_decompress() and fails through ZSTD_decompressStream(), partway through the frame. Transfer the block's regenerated size from the header stage, where ZSTDv07_getcBlockSize() already extracts it into blockProperties, to the block stage, and expand RLE blocks there with the same helper the one-shot path uses. The v0.7.0 reference decoder shipped with the same inconsistency, so no frame produced by the v0.7 line itself contains an RLE block; the asymmetry is nonetheless reachable by any third party encoder for the format, and the one-shot path defines what the format accepts. Add a crafted frame to the legacy test, one RLE block expanding to 100000 bytes: before this change the streaming leg fails with Error (generic), after it the streaming output matches the one-shot output.
Author
|
@Cyan4973 these legacy decoder fixes (#4819 among the v0.2 through v0.7 batch) have been green on all checks for a few weeks now with no reviewer assigned yet. Each one is a narrow correctness fix in one legacy format with a round-trip regression test in tests/legacy.c. Could you or another maintainer take a look when the queue allows? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The v0.7 decoder expands RLE blocks in its one-shot path, through ZSTDv07_generateNxBytes() fed with the regenerated size that ZSTDv07_getcBlockSize() extracts into blockProperties. The block stage of ZSTDv07_decompressContinue() instead rejects them with a generic error.
A frame containing an RLE block therefore decodes completely through ZSTD_decompress() and fails partway through ZSTD_decompressStream(). Demonstrated with a crafted frame holding a single RLE block of byte 0x41 expanding to 100000 bytes, on unfixed dev:
The change carries the regenerated size from the header stage to the block stage in a new dctx field, and expands RLE blocks there with the same helper the one-shot path uses. After the change the streaming decoder returns the same 100000 bytes as the one-shot decoder, and the frame is added to tests/legacy.c behind the ZSTD_LEGACY_SUPPORT level 7 gate; on unfixed dev the new test fails with Error (generic) and with the change it passes, along with the existing legacy tests at every level from 1 to 7.
For background: the v0.7.0 reference decoder shipped with the same inconsistency between its two paths, which is how it reached the legacy copy, and no encoder from the v0.7 line emits RLE blocks. The format's one-shot path defines what is accepted, so third party encoders producing RLE blocks are currently readable in one call but not through the streaming interface.