Skip to content

Add T64 and GORILLA packing transform codecs - #19306

Merged
xiangfu0 merged 1 commit into
masterfrom
xiangfu0/codex/codec-stack/04-t64-gorilla
Aug 25, 2026
Merged

Add T64 and GORILLA packing transform codecs#19306
xiangfu0 merged 1 commit into
masterfrom
xiangfu0/codex/codec-stack/04-t64-gorilla

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Context

Stacked on #19305 and part of the split of #18229.

What changed

  • Adds T64, a frame-of-reference packing transform over fixed 64-value blocks.
  • Adds GORILLA, an XOR/window packing transform with an MSB-first bitstream.
  • Registers both frozen codec names and enforces that a packing transform is the last transform before any compression stages.
  • Makes all fixed-width frame fields explicitly big-endian.
  • Requires exact frame consumption: T64 rejects nonzero tail slots and trailing bytes; GORILLA rejects nonzero final padding and trailing bytes.
  • Uses reusable 64-value block scratch for T64 instead of allocating a full chunk-sized long[].
  • Corrects the public wire documentation for contained-window reuse and width-minus-one encoding.

Safety and scope

T64/GORILLA remain runtime-only in this slice; production forward-index creation/loading is first wired in #19307.

Verification

  • INT/LONG round trips across empty, singleton, partial-block, boundary, and every supported bit width.
  • Happy-path bounded decodeInto, mixed executor chains, corruption/trailing/padding rejection, and encoded-size bounds.
  • Independent known-byte fixtures for T64 INT/LONG and GORILLA.
  • Little-endian caller-view regressions proving big-endian persisted decoding.
  • Spotless, Checkstyle, license format, and license check for pinot-segment-local.

Stack

#19284#19285#19305#19306 (this PR)#19307#19308#19309

Review and merge parent-first.

@xiangfu0 xiangfu0 added feature New functionality index Related to indexing (general) serialization Related to data serialization and deserialization performance Related to performance optimization labels Aug 19, 2026
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/04-t64-gorilla branch from 1924ed6 to 7014dcc Compare August 20, 2026 09:07
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/04-t64-gorilla branch from 7014dcc to bdc7c33 Compare August 21, 2026 00:49
@xiangfu0 xiangfu0 added the timeseries-compression Related to time-series data compression label Aug 21, 2026
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/04-t64-gorilla branch from bdc7c33 to d251cca Compare August 21, 2026 04:16
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/04-t64-gorilla branch from d251cca to 3129e00 Compare August 24, 2026 09:40
@xiangfu0
xiangfu0 requested review from Jackie-Jiang and a lite review from Copilot August 25, 2026 03:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/04-t64-gorilla branch from 3129e00 to 8eecfc9 Compare August 25, 2026 03:36
@codecov-commenter

codecov-commenter commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.34568% with 31 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.26%. Comparing base (f40e6d4) to head (f31fb3e).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...segment/local/io/codec/GorillaCodecDefinition.java 92.48% 8 Missing and 8 partials ⚠️
...not/segment/local/io/codec/T64CodecDefinition.java 92.10% 7 Missing and 8 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master   #19306       +/-   ##
=============================================
+ Coverage     39.01%   67.26%   +28.25%     
- Complexity     1423     1424        +1     
=============================================
  Files          3482     3484        +2     
  Lines        223388   223793      +405     
  Branches      35194    35289       +95     
=============================================
+ Hits          87161   150543    +63382     
+ Misses       128313    61267    -67046     
- Partials       7914    11983     +4069     
Flag Coverage Δ
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (?)
java-25 67.26% <92.34%> (+28.25%) ⬆️
lane-a 100.00% <ø> (ø)
lane-b 0.00% <ø> (ø)
temurin 67.26% <92.34%> (+28.25%) ⬆️
unittests 67.26% <92.34%> (+28.25%) ⬆️
unittests1 57.49% <0.00%> (?)
unittests2 39.12% <92.34%> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Base automatically changed from xiangfu0/codex/codec-stack/03-delta-codecs to master August 25, 2026 20:25
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/04-t64-gorilla branch from 8eecfc9 to f31fb3e Compare August 25, 2026 20:25
@xiangfu0
xiangfu0 merged commit cf58531 into master Aug 25, 2026
12 checks passed
@xiangfu0
xiangfu0 deleted the xiangfu0/codex/codec-stack/04-t64-gorilla branch August 25, 2026 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New functionality index Related to indexing (general) performance Related to performance optimization serialization Related to data serialization and deserialization timeseries-compression Related to time-series data compression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants