GH-48231 [C++][Parquet] Add FSST encoding support for Parquet by ArnavBalyan · Pull Request #48232 · apache/arrow

ArnavBalyan · 2025-11-23T14:08:01Z

Rationale for this change

Adds support for FSST Encoding for Parquet

What changes are included in this PR?

FSST Encoder/Decoder and third party dependencies for Parquet

Are these changes tested?

Unit tests, tested with live jobs

Are there any user-facing changes?

Yes, new encoding

GitHub Issue: [C++][Parquet] Add FSST Support for Parquet #48231

github-actions · 2025-11-23T14:08:26Z

⚠️ GitHub issue #48231 has been automatically assigned in GitHub to PR creator.

ArnavBalyan · 2025-11-24T05:36:03Z

cc @julienledem @emkornfield will fix builds soon thanks

wgtmac · 2025-11-25T02:10:02Z

Thanks for creating the PoC implementation! I haven't yet checked the detail about the FSST algorithm. IMHO it is generally fine to directly depend on https://github.com/cwida/fsst for PoC and benchmark. I'm not sure how much effort is required to write our own FSST implementation. We care about maintainability and are strict with adding a new 3rd party dependency, especially when we have already depended on xsimd to optimize vectorization.

cc @pitrou

ArnavBalyan · 2025-11-25T02:38:36Z

Thanks for taking a look, yeah this is something we briefly discussed in the Parquet sync, generally https://github.com/cwida/fsst should be reliable for FSST. Re-implementing it might require some duplication, will discuss in Parquet sync if we can get a consensus on the dependency.

emkornfield · 2025-11-25T03:12:54Z

Re-implementing it might require some duplication, will discuss in Parquet sync if we can get a consensus on the dependency.

Apologies, i haven't had a chance to look at this yet, but a reminder the sync is not an official place to come to consensus (official decisions should be discussed and finalized on the mailing list). Another option is to vendor/copy most of the FSST library in the source tree. This also impacts Arrow should probably be brought up on both mailing lists.

ArnavBalyan · 2025-11-25T03:21:06Z

Re-implementing it might require some duplication, will discuss in Parquet sync if we can get a consensus on the dependency.

Apologies, i haven't had a chance to look at this yet, but a reminder the sync is not an official place to come to consensus (official decisions should be discussed and finalized on the mailing list). Another option is to vendor/copy most of the FSST library in the source tree. This also impacts Arrow should probably be brought up on both mailing lists.

Sure that works too! Just wanted to get a consensus with the community, will start a mail thread instead. Let me check the vendor/copy option, should be 6-7 files from FSST if we opt to duplicate relevant code.

ArnavBalyan · 2025-11-25T11:44:50Z

Eliminated the direct dependency on fsst, which is working well. Will check on email thread the feedback from community, and update if needed.

ArnavBalyan · 2025-11-26T13:41:35Z

cc @wgtmac could you please re-run the test. Just checking if it's not related by fsst by any chance thanks!

kou · 2025-11-27T01:28:12Z

If we bundle a dependency (copy to cpp/src/arrow/vendored/), we don't need to change this file.
See also:

arrow/cpp/src/arrow/CMakeLists.txt

Lines 447 to 483 in 79d6458

set(ARROW_VENDORED_SRCS

vendored/base64.cpp

vendored/datetime.cpp

vendored/double-conversion/bignum-dtoa.cc

vendored/double-conversion/bignum.cc

vendored/double-conversion/cached-powers.cc

vendored/double-conversion/double-to-string.cc

vendored/double-conversion/fast-dtoa.cc

vendored/double-conversion/fixed-dtoa.cc

vendored/double-conversion/string-to-double.cc

vendored/double-conversion/strtod.cc

vendored/musl/strptime.c

vendored/uriparser/UriCommon.c

vendored/uriparser/UriCompare.c

vendored/uriparser/UriEscape.c

vendored/uriparser/UriFile.c

vendored/uriparser/UriIp4.c

vendored/uriparser/UriIp4Base.c

vendored/uriparser/UriMemory.c

vendored/uriparser/UriNormalize.c

vendored/uriparser/UriNormalizeBase.c

vendored/uriparser/UriParse.c

vendored/uriparser/UriParseBase.c

vendored/uriparser/UriQuery.c

vendored/uriparser/UriRecompose.c

vendored/uriparser/UriResolve.c

vendored/uriparser/UriShorten.c)

if(APPLE)

list(APPEND ARROW_VENDORED_SRCS vendored/datetime/ios.mm)

endif()

set_source_files_properties(vendored/datetime.cpp PROPERTIES SKIP_UNITY_BUILD_INCLUSION

ON)

arrow_add_object_library(ARROW_VENDORED ${ARROW_VENDORED_SRCS})

# Disable DLL exports in vendored uriparser library

foreach(ARROW_VENDORED_TARGET ${ARROW_VENDORED_TARGETS})

target_compile_definitions(${ARROW_VENDORED_TARGET} PRIVATE URI_STATIC_BUILD)

endforeach()

kou · 2025-11-27T01:28:59Z

If we want to bundle a dependency, could you use cpp/src/arrow/vendored/ instead of cpp/thirdparty/?
https://github.com/apache/arrow/tree/main/cpp/src/arrow/vendored

emkornfield · 2025-11-27T02:07:35Z

I think before we spend a lot of time reviewing this we should try to close out on the overall design on the parquet mailing list. Could we maybe mark this as a draft?

Standalone C++ harness comparing FSST (the vendored PR apache#48232 codec), the OnPair implementation, zstd level 1 and lz4 on the same string corpora, reporting compression ratio and encode/decode throughput under realistic bit-packed accounting. Includes a dict-encode-then-OnPair variant for repetitive columns and an OnPair-auto path that picks the dictionary budget per column to minimize size (storing the resulting code width). The Rust bench-fsst-onpair generator produces the corpora (18 TPC-H string columns plus synthetic URL and JSON/variant columns) via --dump-corpora so the harness reads identical inputs.

Standalone C++ harness comparing FSST (the vendored PR apache#48232 codec), the OnPair implementation, zstd level 1 and lz4 on the same string corpora, reporting compression ratio and encode/decode throughput under realistic bit-packed accounting. Decode is bulk whole-column over the bit-packed code stream, so ratio and decode are mutually consistent. Includes a dict-encode-then-OnPair variant for repetitive columns and an OnPair-auto path that picks the dictionary budget per column to minimize size (storing the resulting code width). The Rust bench-fsst-onpair generator produces the corpora (18 TPC-H string columns plus synthetic URL and JSON/variant columns) via --dump-corpora so the harness reads identical inputs.

ArnavBalyan · 2026-08-15T10:33:46Z

This work is continued in #50872

Add FSST support

7274e4b

ArnavBalyan requested review from AlenkaF, raulcd, rok and wgtmac as code owners November 23, 2025 14:08

ArnavBalyan marked this pull request as draft November 23, 2025 14:08

github-actions Bot added Component: Parquet Component: C++ Component: Python awaiting review Awaiting review labels Nov 23, 2025

ArnavBalyan changed the title ~~GH-48231 [C++] [Parquet] Add FSST encoding support for Parquet~~ GH-48231 [C++][Parquet] Add FSST encoding support for Parquet Nov 23, 2025

update

9ba9672

ArnavBalyan added 10 commits November 24, 2025 08:10

cmake build

12eef2e

update

68cc6f2

update

05506fa

update

5935b9e

update

ff3b8f7

update

f5e2bdb

update

7c105d4

update

511776f

update

0344d45

update

f9b43c4

ArnavBalyan added 2 commits November 25, 2025 07:31

vendor fsst

c49f19c

update

603a076

ArnavBalyan added 2 commits November 25, 2025 09:27

lint

e19cc5e

lint

d4176ae

ArnavBalyan marked this pull request as ready for review November 25, 2025 11:43

ArnavBalyan requested review from assignUser, jonkeane and kou as code owners November 25, 2025 11:43

kou reviewed Nov 27, 2025

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Nov 27, 2025

ArnavBalyan marked this pull request as draft November 28, 2025 06:22

CurtHagenlocher mentioned this pull request Aug 13, 2026

feat(parquet): FSST, written as encoding 11 because ALP already holds 10 clast-project/engineered-wood#149

Merged

ArnavBalyan closed this Aug 15, 2026

Navigation Menu

Uh oh!

GH-48231 [C++][Parquet] Add FSST encoding support for Parquet - #48232

GH-48231 [C++][Parquet] Add FSST encoding support for Parquet#48232
ArnavBalyan wants to merge 16 commits into
apache:mainfrom
ArnavBalyan:arnavb/fsst-support

ArnavBalyan commented Nov 23, 2025 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 23, 2025

Uh oh!

ArnavBalyan commented Nov 24, 2025

Uh oh!

wgtmac commented Nov 25, 2025

Uh oh!

ArnavBalyan commented Nov 25, 2025

Uh oh!

emkornfield commented Nov 25, 2025

Uh oh!

ArnavBalyan commented Nov 25, 2025 •

edited

Loading

Uh oh!

ArnavBalyan commented Nov 25, 2025

Uh oh!

ArnavBalyan commented Nov 26, 2025

Uh oh!

kou Nov 27, 2025

Uh oh!

kou Nov 27, 2025

Uh oh!

emkornfield commented Nov 27, 2025

Uh oh!

ArnavBalyan commented Aug 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	set(ARROW_VENDORED_SRCS
	vendored/base64.cpp
	vendored/datetime.cpp
	vendored/double-conversion/bignum-dtoa.cc
	vendored/double-conversion/bignum.cc
	vendored/double-conversion/cached-powers.cc
	vendored/double-conversion/double-to-string.cc
	vendored/double-conversion/fast-dtoa.cc
	vendored/double-conversion/fixed-dtoa.cc
	vendored/double-conversion/string-to-double.cc
	vendored/double-conversion/strtod.cc
	vendored/musl/strptime.c
	vendored/uriparser/UriCommon.c
	vendored/uriparser/UriCompare.c
	vendored/uriparser/UriEscape.c
	vendored/uriparser/UriFile.c
	vendored/uriparser/UriIp4.c
	vendored/uriparser/UriIp4Base.c
	vendored/uriparser/UriMemory.c
	vendored/uriparser/UriNormalize.c
	vendored/uriparser/UriNormalizeBase.c
	vendored/uriparser/UriParse.c
	vendored/uriparser/UriParseBase.c
	vendored/uriparser/UriQuery.c
	vendored/uriparser/UriRecompose.c
	vendored/uriparser/UriResolve.c
	vendored/uriparser/UriShorten.c)
	if(APPLE)
	list(APPEND ARROW_VENDORED_SRCS vendored/datetime/ios.mm)
	endif()
	set_source_files_properties(vendored/datetime.cpp PROPERTIES SKIP_UNITY_BUILD_INCLUSION
	ON)
	arrow_add_object_library(ARROW_VENDORED ${ARROW_VENDORED_SRCS})
	# Disable DLL exports in vendored uriparser library
	foreach(ARROW_VENDORED_TARGET ${ARROW_VENDORED_TARGETS})
	target_compile_definitions(${ARROW_VENDORED_TARGET} PRIVATE URI_STATIC_BUILD)
	endforeach()

Uh oh!

Conversation

ArnavBalyan commented Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Nov 23, 2025

Uh oh!

ArnavBalyan commented Nov 24, 2025

Uh oh!

wgtmac commented Nov 25, 2025

Uh oh!

ArnavBalyan commented Nov 25, 2025

Uh oh!

emkornfield commented Nov 25, 2025

Uh oh!

ArnavBalyan commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ArnavBalyan commented Nov 25, 2025

Uh oh!

ArnavBalyan commented Nov 26, 2025

Uh oh!

kou Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

kou Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

emkornfield commented Nov 27, 2025

Uh oh!

ArnavBalyan commented Aug 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ArnavBalyan commented Nov 23, 2025 •

edited

Loading

ArnavBalyan commented Nov 25, 2025 •

edited

Loading