[Algunenano]

Updates contrib/zstd to a rebase of the ClickHouse/zstd fork onto upstream dev (49cf5179, 2026-10-05; the old fork was 199 non-merge commits behind), plus pending upstream PRs and one new patch of ours. Fork branch: ClickHouse/dev-20261021 (8388bfae). The overlap-copy patches of the old fork are kept; its unwind-info patches are upstream already.

Changes on top of upstream dev:

• Decompress: optimize sequence decompression (fast-path inlining, ABI scalarization, branch elision) meta/zstd#4751: sequence decoding. Most of the decompression gain on x86.
• dfast: pipelined extDict block compressor meta/zstd#4764: double-fast compressor, the strategy of ZSTD(3).
• opt: tabulate the optimal parser's per-code price components meta/zstd#4732: optimal parser (ZSTD(16) and above).
• decompress: preserve the frame pointer in the Huffman x86-64 asm meta/zstd#4811: keeps the frame pointer in the Huffman x86-64 asm, so profilers can unwind through it. Neutral for speed.
• legacy/v0.5: fix streaming regression from the wrap history fix meta/zstd#4816: fixes a regression of upstream dev in streaming decompression of legacy v0.5 frames (Destination buffer is too small once a frame crosses its window).
• New: a branchless copy of matches with an offset below 16 (NEON tbl, SSSE3 pshufb, AVX2 vpshufb). It replaces ZSTD_overlapCopy8, whose branch on offset < 8 mispredicts on numeric columns where offsets like 4, 8 and 12 alternate.

Compressed output is byte-identical to the current fork at levels 1, 3, 6, 9 and 16, and the checksum of all columns of ClickBench hits is the same after decompression.

ClickBench hits (100M rows in one ZSTD(3) part, no columns cache, median of 20 hot runs, non-LTO local build) on AMD EPYC 9R14 (Zen 4), speedup vs current master:

Query	16 threads on 8 cores	64 threads on 32 cores
SELECT count() FROM hits WHERE NOT ignore(SearchPhrase)	1.22x	1.19x
same with URL	1.14x	1.12x
same with Title	1.17x	1.13x
same with ClientIP	1.21x	1.14x
same with RegionID	1.14x	1.13x
same with UserID	1.08x	1.03x
SELECT sum(cityHash64(*)) FROM hits	1.06x	1.06x
SELECT COUNT(DISTINCT SearchPhrase) FROM hits	1.12x	1.11x

zstd CLI on the same data, single core, ZSTD(3), ClickHouse block sizes, vs current master: compression 1.018x, decompression 1.175x (geomean over 7 numeric columns and the separate characters and sizes streams of 4 String columns). The short-offset copy alone, vs the same set without it: ClientIP 1.15x, UserID 1.12x, RegionID 1.085x, ResolutionWidth 1.065x, and the characters streams of long strings 1-2% slower; in ClickHouse it is 1.12x on the ClientIP scan, 1.08x on UserID, 1.06x on RegionID, 0.98x on URL and Title, 1.01x on sum(cityHash64(*)) and 1.02x on COUNT(DISTINCT SearchPhrase) (16 threads). On Graviton4 it is +5.6% CLI decompression geomean, ClientIP +45%, UserID +14%, and 1.20x on the ClientIP scan in ClickHouse.

Earlier measurements of the same set without the short-offset copy, vs current master:

• Intel Xeon 6975P-C: SearchPhrase / URL / Title scans 1.24x / 1.17x / 1.20x with 16 threads on 8 cores, COUNT(DISTINCT SearchPhrase) 1.12x. With ZSTD(3), the INSERT INTO ... SELECT of the whole table uses 2% less CPU and OPTIMIZE FINAL of 5 parts (20M rows) takes 36.5 s instead of 37.5 s.
• Graviton4 (c8g.16xlarge): string scans 1.02x to 1.04x, the rest unchanged, insert and merge unchanged.
• Per PR, CLI: Decompress: optimize sequence decompression (fast-path inlining, ABI scalarization, branch elision) meta/zstd#4751 is +9% decompression on Intel and neutral on Graviton4; dfast: pipelined extDict block compressor meta/zstd#4764 is +2.7% ZSTD(3) compression on Intel and neutral on Graviton4; opt: tabulate the optimal parser's per-code price components meta/zstd#4732 is +13% (Intel) and +17% (Graviton4) at ZSTD(16).

Evaluated and not included:

• lazy: speed up the row matchfinder (levels 5-12) by 2-8%, identical output meta/zstd#4841: only helps levels 5-12 on x86. It also miscounts match lengths on big-endian targets, which corrupts data silently.
• decompress: keep the repeat-offset triple in registers on x86-64 meta/zstd#4733: +4% alone on x86, and slower combined with Decompress: optimize sequence decompression (fast-path inlining, ABI scalarization, branch elision) meta/zstd#4751.
• Add more prefetching to decompression. meta/zstd#3828 and FSE: fuse symbol spreading with table building on the fast path meta/zstd#4836: no gain on top of Decompress: optimize sequence decompression (fast-path inlining, ABI scalarization, branch elision) meta/zstd#4751.
• Optimize ZSTD compression level 6 parameters meta/zstd#4646: compresses worse at level 6.
• AArch64 only: AArch64: Improve sequence decoding meta/zstd#4813 (0.98x decompression), huf: Tune X1/X2 decoder selection for AArch64 meta/zstd#4803 and [AArch64] Optimize ZSTD_isRLE() with 64-byte NEON vector unrolling meta/zstd#4736 (neutral), Improving AArch64 Performance with SVE for Decompression Functions meta/zstd#4715 (no SVE code with our flags).

Related: #124405

Changelog category (leave one):

• Performance Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Update zstd to the latest upstream development version with pending performance improvements. Decompression of ZSTD(3) data is up to 20% faster on x86.

[clickhouse-gh]

Workflow [PR], commit [e37a4a6]

Summary: ❌

• Performance Comparison: Performance dashboard

job_name	test_name	status	info	comment
Stateless tests (arm_asan_ubsan, targeted)		FAIL
	05407_distributed_plan_exchange_compression_default	FAIL	cidb
	05407_distributed_plan_exchange_compression_default	FAIL	cidb
	05407_distributed_plan_exchange_compression_default	FAIL	cidb
	05407_distributed_plan_exchange_compression_default	FAIL	cidb
Stateless tests (amd_tsan, s3 storage, targeted)		FAIL
	05407_distributed_plan_exchange_compression_default	FAIL	cidb
Stateless tests (arm_binary, s3 storage, DBReplicated, parallel, 2/2)		FAIL
	05407_distributed_plan_exchange_compression_default	FAIL	cidb
Performance Comparison (amd_release, master_head, 4/6)		FAIL	Performance dashboard
	hashing_serialized_values #0	FAIL	dashboard

---

AI Review

Summary

PR #127206, reviewed at e37a4a6a7df6f7a9711ab2d20ad7cedd1b864883, updates zstd with compression and decompression optimizations. No actionable findings remain; both earlier findings are fixed. AArch64 ASan validation passed 115,200 focused short-offset round trips, the basic tests, 309 randomized tests, 1,648 randomized streaming tests, and the legacy suite including both v0.5 wrapping regressions.

Missing context / blind spots

• ⚠️ Full CI for the reviewed head is still running. Local runtime checks used AArch64/GCC with ASan; the x86 intrinsic and assembly paths were inspected but not executed locally. Completion of the head's x86 and sanitizer CI would close this platform-validation gap.

Final Verdict

Status: ✅ Approve. No required code changes.

[clickhouse-gh]

Build profile diff (arm_release)

Commit e37a4a6a7df6f7a9711ab2d20ad7cedd1b864883 was not compared: the CI logs cluster did not answer (exception 241, code 500: { "meta": [ { "name": "file", "type": "LowCardinality(String)" …).

See the job log for details.