Add codec pipeline integration tests and design doc - #19309
Open
xiangfu0 wants to merge 1 commit into
Open
Conversation
This was referenced Aug 19, 2026
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 19, 2026 09:51
88e1d40 to
026ac0b
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 20, 2026 09:08
026ac0b to
4f0c32d
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 21, 2026 00:49
4f0c32d to
c8810ab
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 21, 2026 04:16
c8810ab to
948d51c
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 24, 2026 09:41
948d51c to
f31545b
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 25, 2026 04:55
f31545b to
17259a1
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## xiangfu0/codex/codec-stack/06-reload-enable #19309 +/- ##
==============================================================================
Coverage 57.53% 57.53%
Complexity 7 7
==============================================================================
Files 2688 2688
Lines 164285 164285
Branches 26676 26676
==============================================================================
Hits 94523 94523
+ Misses 61750 61749 -1
- Partials 8012 8013 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 25, 2026 20:25
17259a1 to
43e5df3
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 25, 2026 21:20
43e5df3 to
231bb99
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 25, 2026 21:49
231bb99 to
2d41839
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 26, 2026 09:20
2d41839 to
99b615f
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 27, 2026 09:09
99b615f to
64d2e75
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 28, 2026 09:15
64d2e75 to
4ed1bd6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Stacked on #19308 and the final test/documentation slice of the split of #18229.
Integration coverage
CodecPipelineIntegrationTestbuilds and queries an offline table with representative pipelines covering all eight built-in codecs on INT/LONG raw columns:LZ4,ZSTD(3),SNAPPY,GZIP.DELTA,SNAPPYandDELTA,GZIP.DELTA,LZ4,DELTA,ZSTD(3),DELTADELTA,LZ4,T64,GORILLA,T64,LZ4, andGORILLA,ZSTD(3).Both SSE and MSE exercise sums, filters, batched point lookups across real chunk boundaries, and cross-codec consistency. The test also verifies dictionary coexistence and a raw STRING column with
ZSTD(3).The test opens every generated segment to prove the config is not ignored: V7 columns must use
FixedByteChunkSVForwardIndexReaderV7and expose the exact canonical spec; legacy-compatible numeric/string columns must retain the expected legacy reader andChunkCompressionType; the dictionary column must remain dictionary encoded.Design documentation
Adds
docs/design/codec-pipeline-v7.mdcovering:pinot-segment-local.Safety and verification
This slice changes only integration tests and documentation. The final reactor run builds all 63 required modules and runs all 88 integration cases. Spotless, Checkstyle, license format, and license check cover every affected module in the stack.
Stack
#19284 → #19285 → #19305 → #19306 → #19307 → #19308 → #19309 (this PR)
Review and merge parent-first.