Skip to content

Add codec pipeline integration tests and design doc - #19309

Open
xiangfu0 wants to merge 1 commit into
xiangfu0/codex/codec-stack/06-reload-enablefrom
xiangfu0/codex/codec-stack/07-integration-docs
Open

Add codec pipeline integration tests and design doc#19309
xiangfu0 wants to merge 1 commit into
xiangfu0/codex/codec-stack/06-reload-enablefrom
xiangfu0/codex/codec-stack/07-integration-docs

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Context

Stacked on #19308 and the final test/documentation slice of the split of #18229.

Integration coverage

CodecPipelineIntegrationTest builds and queries an offline table with representative pipelines covering all eight built-in codecs on INT/LONG raw columns:

  • Legacy-compatible compression-only specs: LZ4, ZSTD(3), SNAPPY, GZIP.
  • V7 compression-handler chains: DELTA,SNAPPY and DELTA,GZIP.
  • Transform/packing pipelines including DELTA,LZ4, DELTA,ZSTD(3), DELTADELTA,LZ4, T64, GORILLA, T64,LZ4, and GORILLA,ZSTD(3).

Both SSE and MSE exercise sums, filters, batched point lookups across real chunk boundaries, and cross-codec consistency. The test also verifies dictionary coexistence and a raw STRING column with ZSTD(3).

The test opens every generated segment to prove the config is not ignored: V7 columns must use FixedByteChunkSVForwardIndexReaderV7 and expose the exact canonical spec; legacy-compatible numeric/string columns must retain the expected legacy reader and ChunkCompressionType; the dictionary column must remain dictionary encoded.

Design documentation

Adds docs/design/codec-pipeline-v7.md covering:

  • The exact bounded grammar (4096-character cap and no leading-zero numeric arguments).
  • Structural config normalization versus semantic executor/V7 canonicalization.
  • Actual runtime placement in pinot-segment-local.
  • V7 framing, exact byte/bit order, codec-library framing, corruption defenses, and golden fixtures.
  • Offline and committed-realtime support versus the mutable consuming representation.
  • Migration/rollback limits, mixed-version safety, resource bounds, and context-owned reusable decode scratch.

Safety and verification

This slice changes only integration tests and documentation. The final reactor run builds all 63 required modules and runs all 88 integration cases. Spotless, Checkstyle, license format, and license check cover every affected module in the stack.

Stack

#19284#19285#19305#19306#19307#19308#19309 (this PR)

Review and merge parent-first.

@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 88e1d40 to 026ac0b Compare August 19, 2026 09:51
@xiangfu0 xiangfu0 added documentation Improvements or additions to documentation testing Related to tests or test infrastructure test-coverage Adds or improves test coverage index Related to indexing (general) labels Aug 19, 2026
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 026ac0b to 4f0c32d Compare August 20, 2026 09:08
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 4f0c32d to c8810ab Compare August 21, 2026 00:49
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from c8810ab to 948d51c Compare August 21, 2026 04:16
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 948d51c to f31545b Compare August 24, 2026 09:41
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from f31545b to 17259a1 Compare August 25, 2026 04:55
@codecov-commenter

codecov-commenter commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.53%. Comparing base (d0afba2) to head (4ed1bd6).

Additional details and impacted files
@@                              Coverage Diff                               @@
##             xiangfu0/codex/codec-stack/06-reload-enable   #19309   +/-   ##
==============================================================================
  Coverage                                          57.53%   57.53%           
  Complexity                                             7        7           
==============================================================================
  Files                                               2688     2688           
  Lines                                             164285   164285           
  Branches                                           26676    26676           
==============================================================================
  Hits                                               94523    94523           
+ Misses                                             61750    61749    -1     
- Partials                                            8012     8013    +1     
Flag Coverage Δ
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (?)
java-25 57.53% <ø> (ø)
lane-a 100.00% <ø> (ø)
lane-b 0.00% <ø> (ø)
temurin 57.53% <ø> (ø)
unittests 57.53% <ø> (ø)
unittests1 57.53% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 17259a1 to 43e5df3 Compare August 25, 2026 20:25
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 43e5df3 to 231bb99 Compare August 25, 2026 21:20
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 231bb99 to 2d41839 Compare August 25, 2026 21:49
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 2d41839 to 99b615f Compare August 26, 2026 09:20
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 99b615f to 64d2e75 Compare August 27, 2026 09:09
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 64d2e75 to 4ed1bd6 Compare August 28, 2026 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation index Related to indexing (general) test-coverage Adds or improves test coverage testing Related to tests or test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants