Skip to content

Non-deterministic output with ZSTD_createCDict_byReference #4738

Description

@cdcxd

Describe the bug

It's not a logical bug. Just the ZSTD_createCDict_byReference would output differently if the input data is directly adjacent to the dict. Making the output non-deterministic if the input is randomly allocated.

I found there's a opt-in param ZSTD_c_deterministicRefPrefix , that solves the problem for refPrefix. They share exactly same mechanism. But it doesn't apply to the CDict.

And the comment https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L2275 said

 * If you really care about determinism when using a dictionary or prefix,
 * like when doing delta compression, you should select this option. 

So I believe it might also need to apply to the new API ZSTD_createCDict_byReference

To Reproduce

Compressing one 256 KB random buffer with a random 64 KB ZSTD_createCDict_byReference() dictionary,
contiguous vs separated:

level 2 | byReference     | MISMATCH
level 3 | byReference     | MISMATCH
level 4 | byReference     | MISMATCH
level 6 | byReference     | MISMATCH 
level 2 | byReference, deterministicRefPrefix=1               | MISMATCH
level 2 | byCopy (control), default                           | IDENTICAL

Expected behavior
Opt-in deterministic mode

Screenshots and charts
If applicable, add screenshots and charts to help explain your problem.

Desktop (please complete the following information):

  • OS: [Ubuntu]
  • Version [24.04]
  • Compiler [gcc13.3.0]
  • Flags [any flag]
  • Other relevant hardware specs [AVX2 , 16 cores]
  • Build system [Any]

Additional context

I'll create a PR with unit test.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions