[filtede98]

With ZSTD_d_refMultipleDDicts the DCtx keeps a hashset of DDict pointers.
ZSTD_DCtx_reset(ZSTD_reset_session_and_parameters) and
ZSTD_DCtx_refDDict(dctx, NULL) clear the dictionary but leave the set
populated. Later ZSTD_DCtx_refDDict() calls then add to a set that still
holds pointers to DDicts the caller may have freed, and a frame that asks for
one of those dictIDs makes ZSTD_DCtx_selectFrameDDict() read the freed DDict.

Sequence (ASan build of dev):

ZSTD_DCtx_setParameter(dctx, ZSTD_d_refMultipleDDicts, ZSTD_rmd_refMultipleDDicts);
ZSTD_DCtx_refDDict(dctx, ddictA);
ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters);
ZSTD_freeDDict(ddictA);
ZSTD_DCtx_setParameter(dctx, ZSTD_d_refMultipleDDicts, ZSTD_rmd_refMultipleDDicts);
ZSTD_DCtx_refDDict(dctx, ddictB);
ZSTD_decompressDCtx(dctx, ..., frame_with_dictID_of_A, ...);

ERROR: AddressSanitizer: heap-use-after-free
#0 ZSTD_getDictID_fromDDict zstd_ddict.c:243
#1 ZSTD_DCtx_selectFrameDDict zstd_decompress.c:363
#2 ZSTD_decodeFrameHeader zstd_decompress.c:709

The same happens with ZSTD_DCtx_refDDict(dctx, NULL) in place of the reset.

The patch frees the set in both places. Added a test in tests/fuzzer.c that
uses only the public API and decodes a frame after the DDict was freed;
it fails under ASan without the lib change and passes with it.

Ran tests/fuzzer -T20s with ASan/UBSan: all tests pass.

I dropped the load factor expression change that this PR used to contain,
since #4760 covers it.

[GeekQAQ]

This PR also fixes a functional problem that does not need any ZSTD_freeDDict(): after a reset with parameters, a DCtx in multiple-DDict mode still selects DDicts that were referenced before the reset, even though the new configuration does not reference them.

Reproducer (public API plus ZSTD_STATIC_LINKING_ONLY for the experimental parameter; no DDict is freed before the last decode): stale_ddict_after_reset.c (below). It references dictionaries A (dictID 1) and B (dictID 2) in multiple-DDict mode, decodes a frame that needs B, calls ZSTD_DCtx_reset(dctx, ZSTD_reset_session_and_parameters), enables multiple-DDict mode again and references only C (dictID 3). It then decodes the same frame and compares with a fresh DCtx configured the same way.

zstd 1.5.7, frame dictID = 2
fresh DCtx (C only):         error: Dictionary mismatch
reused DCtx before reset:    ok, 74 bytes
reused DCtx after reset (C only): ok, 74 bytes

Same result on dev at 49cf517. With this PR (81c4414), the reused DCtx returns Dictionary mismatch after the reset, like the fresh one. All builds with ASan/UBSan, no sanitizer reports in this program.

So the reused DCtx still decodes frames for dictionaries that the current configuration does not reference. For an application that reuses one DCtx across sessions, each with its own set of dictionaries, the reset does not isolate the sessions.

The documentation seems to point both ways, which may be worth settling in the docs together with this fix:

• zstd.h, advanced dictionary API: "Dictionaries are sticky, they remain valid when same context is reused, they only reset when the context is reset with ZSTD_reset_parameters or ZSTD_reset_session_and_parameters."
• ZSTD_DCtx_refDDict(): "The memory for the table is allocated on the first call to refDDict, and can be freed with ZSTD_freeDCtx()."

My reading is that the first sentence makes the PR's behavior the intended one (the table is dropped by a parameter reset and by refDDict(NULL)).

stale_ddict_after_reset.c

/* After ZSTD_DCtx_reset(..., ZSTD_reset_session_and_parameters), a DCtx in multiple-DDict mode still
 * selects a DDict that was referenced before the reset. No DDict is freed before the last decode. */
#define ZSTD_STATIC_LINKING_ONLY
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <zstd.h>
#include <zdict.h>

static void *make_dict(unsigned id, unsigned seed, size_t *len) {
    static unsigned char content[8192], samples[64 * 256];
    static size_t sizes[64];
    for (size_t i = 0; i < sizeof content; i++) content[i] = (unsigned char)("abcdefghij"[(i * seed + i / 7) % 10]);
    for (int s = 0; s < 64; s++) {
        sizes[s] = 256;
        for (int k = 0; k < 256; k++) samples[s * 256 + k] = content[(s * 97 + k) % sizeof content];
    }
    ZDICT_params_t p = {0};
    p.compressionLevel = 3;
    p.dictID = id;
    void *buf = malloc(16384);
    *len = ZDICT_finalizeDictionary(buf, 16384, content, sizeof content, samples, sizes, 64, p);
    if (ZDICT_isError(*len)) { fprintf(stderr, "%s\n", ZDICT_getErrorName(*len)); exit(1); }
    return buf;
}

static void decode(const char *tag, ZSTD_DCtx *d, const void *src, size_t n) {
    char out[256];
    size_t r = ZSTD_decompressDCtx(d, out, sizeof out, src, n);
    if (ZSTD_isError(r)) printf("%-28s error: %s\n", tag, ZSTD_getErrorName(r));
    else printf("%-28s ok, %zu bytes\n", tag, r);
}

int main(void) {
    size_t la, lb, lc;
    void *a = make_dict(1, 3, &la), *b = make_dict(2, 5, &lb), *c = make_dict(3, 11, &lc);
    ZSTD_DDict *A = ZSTD_createDDict(a, la), *B = ZSTD_createDDict(b, lb), *C = ZSTD_createDDict(c, lc);

    const char *text = "a frame compressed with dictionary B, a frame compressed with dictionary B";
    char frame[512];
    ZSTD_CCtx *cc = ZSTD_createCCtx();
    size_t n = ZSTD_compress_usingDict(cc, frame, sizeof frame, text, strlen(text), b, lb, 3);
    printf("zstd %s, frame dictID = %u\n", ZSTD_versionString(), ZSTD_getDictID_fromFrame(frame, n));

    /* Fresh DCtx: multiple-DDict mode, only C referenced. */
    ZSTD_DCtx *fresh = ZSTD_createDCtx();
    ZSTD_DCtx_setParameter(fresh, ZSTD_d_refMultipleDDicts, ZSTD_rmd_refMultipleDDicts);
    ZSTD_DCtx_refDDict(fresh, C);
    decode("fresh DCtx (C only):", fresh, frame, n);

    /* Reused DCtx: A and B referenced and used, then reset with parameters, then configured like the fresh one. */
    ZSTD_DCtx *reused = ZSTD_createDCtx();
    ZSTD_DCtx_setParameter(reused, ZSTD_d_refMultipleDDicts, ZSTD_rmd_refMultipleDDicts);
    ZSTD_DCtx_refDDict(reused, A);
    ZSTD_DCtx_refDDict(reused, B);
    decode("reused DCtx before reset:", reused, frame, n);
    ZSTD_DCtx_reset(reused, ZSTD_reset_session_and_parameters);
    ZSTD_DCtx_setParameter(reused, ZSTD_d_refMultipleDDicts, ZSTD_rmd_refMultipleDDicts);
    ZSTD_DCtx_refDDict(reused, C);
    decode("reused DCtx after reset (C only):", reused, frame, n);

    ZSTD_freeDCtx(fresh); ZSTD_freeDCtx(reused); ZSTD_freeCCtx(cc);
    ZSTD_freeDDict(A); ZSTD_freeDDict(B); ZSTD_freeDDict(C);
    free(a); free(b); free(c);
    return 0;
}