GH-48231 [C++][Parquet] Add FSST encoding support for Parquet

[ArnavBalyan]

Rationale for this change

• Adds support for FSST Encoding for Parquet

What changes are included in this PR?

• FSST Encoder/Decoder and third party dependencies for Parquet

Are these changes tested?

• Unit tests, tested with live jobs

Are there any user-facing changes?

• Yes, new encoding

• GitHub Issue: [C++][Parquet] Add FSST Support for Parquet #48231

[Copilot]

Pull request overview

Adds FSST (Fast Static Symbol Table) encoding support to Arrow’s C++ Parquet implementation, including a new “symbol table page” concept in the Parquet page stream and metadata, plus Python (pyarrow) exposure/tests and documentation updates.

Changes:

• Implement FSST encoding/decoding for Parquet BYTE_ARRAY columns, including symbol table page serialization/deserialization and metadata plumbing.
• Extend Parquet metadata/page model (thrift, enums, readers/writers) to represent and handle symbol table pages and FSST encoding stats.
• Add pyarrow bindings/tests and C++ docs/tests, and vendor the upstream CWI FSST implementation into the Arrow tree.

Reviewed changes

Copilot reviewed 44 out of 47 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
python/pyarrow/tests/parquet/test_metadata.py	Adds assertions for symbol table metadata; adds FSST metadata test.
python/pyarrow/tests/parquet/test_basic.py	Adds FSST roundtrip coverage and invalid-type error check.
python/pyarrow/parquet/core.py	Documents FSST as a supported column encoding option.
python/pyarrow/includes/libparquet.pxd	Exposes FSST/ALP encodings and symbol table metadata APIs to Cython.
python/pyarrow/_parquet.pyx	Exposes symbol table metadata properties; maps FSST/ALP encoding names.
LICENSE.txt	Adds license notice for vendored FSST sources.
docs/source/cpp/parquet.rst	Documents FSST support and configuration details.
cpp/src/parquet/types.h	Adds FSST/ALP encodings, SYMBOL_TABLE_PAGE, and FSST-related enums.
cpp/src/parquet/types.cc	Adds stringification and checksum eligibility for new enums/page type.
cpp/src/parquet/thrift_internal.h	Adds thrift conversion traits/helpers for SymbolTableType.
cpp/src/parquet/properties.h	Adds fsst_offset_encoding writer property (global + per-column).
cpp/src/parquet/properties.cc	Ensures fsst_offset_encoding round-trips through builder copy.
cpp/src/parquet/properties_test.cc	Adds tests for fsst_offset_encoding defaults/overrides.
cpp/src/parquet/parquet.thrift	Extends format with FSST/ALP encodings, SYMBOL_TABLE_PAGE, symbol table header and metadata fields.
cpp/src/parquet/metadata.h	Adds symbol table page metadata accessors and builder Finish overload.
cpp/src/parquet/metadata.cc	Implements symbol table metadata validation, storage, and encoding stats.
cpp/src/parquet/metadata_test.cc	Updates metadata tests for new Finish signature/fields.
cpp/src/parquet/meson.build	Adds FSST implementation sources to Meson build.
cpp/src/parquet/fsst_internal.h	Declares Parquet adapter for the vendored FSST codec.
cpp/src/parquet/fsst_internal.cc	Implements FSST symbol table training/serialization/compress/decompress.
cpp/src/parquet/file_serialize_test.cc	Adds tests validating symbol table page writing and metadata.
cpp/src/parquet/file_reader.cc	Includes symbol table offset in read ranges; adjusts crypto context.
cpp/src/parquet/file_deserialize_test.cc	Adds serde tests for symbol table page header/body validation.
cpp/src/parquet/encoding.h	Extends encoder API for per-page encoding and FSST symbol table access; declares FSST factories.
cpp/src/parquet/encoding_test.cc	Adds FSST encoder/decoder roundtrip and validation tests.
cpp/src/parquet/encoder.cc	Implements FSST encoder and MakeFsstEncoder factory.
cpp/src/parquet/decoder.cc	Implements FSST decoder, MakeFsstDecoder, and integrates decode path.
cpp/src/parquet/column_writer.h	Adds PageWriter hook for symbol table pages.
cpp/src/parquet/column_writer.cc	Writes symbol table pages, enforces dictionary/symbol-table exclusivity, wires FSST writer flow.
cpp/src/parquet/column_reader.cc	Reads symbol table pages, configures FSST decoding, validates physical/logical counts, resets state across row groups.
cpp/src/parquet/column_reader_test.cc	Adds tests for FSST def-level consistency and whole-page skipping.
cpp/src/parquet/column_page.h	Adds SymbolTablePage page type wrapper.
cpp/src/parquet/CMakeLists.txt	Adds FSST sources, compiler flags, and AVX512 compilation options to CMake build.
cpp/src/parquet/arrow/arrow_reader_writer_test.cc	Adds Arrow-level FSST roundtrip, metadata, nested, and dictionary-output tests.
cpp/src/generated/parquet_types.h	Regenerates thrift-derived C++ types for new enums/structs/fields.
cpp/src/arrow/vendored/fsst/README.md	Documents vendoring origin/commit and local integration location.
cpp/src/arrow/vendored/fsst/LICENSE	Adds upstream MIT license file for vendored FSST.
cpp/src/arrow/vendored/fsst/libfsst.hpp	Vendored upstream FSST implementation header.
cpp/src/arrow/vendored/fsst/libfsst.cpp	Vendored upstream FSST implementation source.
cpp/src/arrow/vendored/fsst/fsst.h	Vendored upstream FSST C API header.
cpp/src/arrow/vendored/fsst/fsst_avx512.cpp	Vendored upstream AVX512 acceleration source.
cpp/src/arrow/vendored/fsst/fsst_avx512_unroll4.inc	Vendored upstream AVX512 include fragment (unroll=4).
cpp/src/arrow/vendored/fsst/fsst_avx512_unroll3.inc	Vendored upstream AVX512 include fragment (unroll=3).
cpp/src/arrow/vendored/fsst/fsst_avx512_unroll2.inc	Vendored upstream AVX512 include fragment (unroll=2).
cpp/src/arrow/vendored/fsst/fsst_avx512_unroll1.inc	Vendored upstream AVX512 include fragment (unroll=1).

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 44 out of 47 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

cpp/src/parquet/properties.h:622

• The fsst_training_data_pages docstring says “Positive values defer that many pages before training”, but the implementation buffers and uses that many pages to train (training occurs once that many pages are collected). This wording is misleading and could cause misconfiguration.