Repository navigation
compress: avoid using incomplete dictionary FSE tables - #4786
atharvmantri wants to merge 1 commit into
Conversation
|
Hi @atharvmantri! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
Summary
Guard the optimal-parser dictionary-statistics fast path on all four entropy repeat modes, not only the Huffman table mode.
Why
FSE_getMaxNbBits()intentionally returns a faketableLog + 1cost for zero-frequency symbols. A custom dictionary with incomplete FSE coverage can therefore reach the strictassert(bitCost < scaleLog)checks inZSTD_rescaleFreqs()through the publicZSTD_createCDict_advanced()/ZSTD_compress_usingCDict()APIs, as described in #4681.When any FSE table is incomplete, the existing no-dictionary-statistics initialization is the safe path. This avoids consuming incomplete dictionary tables without changing the valid-table path.
Validation
git diff --checkpasses.Fixes #4681.