Describe the bug
It's not a logical bug. Just the ZSTD_createCDict_byReference would output differently if the input data is directly adjacent to the dict. Making the output non-deterministic if the input is randomly allocated.
I found there's a opt-in param ZSTD_c_deterministicRefPrefix , that solves the problem for refPrefix. They share exactly same mechanism. But it doesn't apply to the CDict.
And the comment https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L2275 said
* If you really care about determinism when using a dictionary or prefix,
* like when doing delta compression, you should select this option.
So I believe it might also need to apply to the new API ZSTD_createCDict_byReference
To Reproduce
Compressing one 256 KB random buffer with a random 64 KB ZSTD_createCDict_byReference() dictionary,
contiguous vs separated:
level 2 | byReference | MISMATCH
level 3 | byReference | MISMATCH
level 4 | byReference | MISMATCH
level 6 | byReference | MISMATCH
level 2 | byReference, deterministicRefPrefix=1 | MISMATCH
level 2 | byCopy (control), default | IDENTICAL
Expected behavior
Opt-in deterministic mode
Screenshots and charts
If applicable, add screenshots and charts to help explain your problem.
Desktop (please complete the following information):
- OS: [Ubuntu]
- Version [24.04]
- Compiler [gcc13.3.0]
- Flags [any flag]
- Other relevant hardware specs [AVX2 , 16 cores]
- Build system [Any]
Additional context
I'll create a PR with unit test.
Describe the bug
It's not a logical bug. Just the
ZSTD_createCDict_byReferencewould output differently if the input data is directly adjacent to the dict. Making the output non-deterministic if the input is randomly allocated.I found there's a opt-in param ZSTD_c_deterministicRefPrefix , that solves the problem for refPrefix. They share exactly same mechanism. But it doesn't apply to the CDict.
And the comment https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L2275 said
So I believe it might also need to apply to the new API
ZSTD_createCDict_byReferenceTo Reproduce
Compressing one 256 KB random buffer with a random 64 KB
ZSTD_createCDict_byReference()dictionary,contiguous vs separated:
Expected behavior
Opt-in deterministic mode
Screenshots and charts
If applicable, add screenshots and charts to help explain your problem.
Desktop (please complete the following information):
Additional context
I'll create a PR with unit test.