Skip to content

Add DELTA and DELTADELTA transform codecs - #19305

Merged
xiangfu0 merged 1 commit into
masterfrom
xiangfu0/codex/codec-stack/03-delta-codecs
Aug 25, 2026
Merged

Add DELTA and DELTADELTA transform codecs#19305
xiangfu0 merged 1 commit into
masterfrom
xiangfu0/codex/codec-stack/03-delta-codecs

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Context

Stacked on #19285 and part of the split of #18229.

What changed

  • Adds DELTA and DELTADELTA as typed-layout-preserving INT/LONG transforms.
  • Keeps the output headerless and the same width/count as the input so transforms can be chained before packing or compression.
  • Defines intentional two's-complement modular arithmetic for overflow-adjacent values.
  • Registers both frozen codec names in the closed runtime registry.
  • Makes persisted decoding explicitly big-endian, independent of a caller view's byte order.

Safety and scope

Registering DELTA/DELTADELTA makes them available to the bounded runtime API, but production forward-index creation/loading is not wired until #19307. This slice changes no segment format or production I/O path.

Verification

  • INT/LONG round trips for empty, boundary, monotonic, negative, sign-changing, and overflow-adjacent data.
  • Allocating and bounded decodeInto paths, transform chains, and malformed-width rejection.
  • Stable hand-authored decode fixtures that pin DELTA and DELTADELTA wire bytes independently of the encoders.
  • Little-endian caller-view regressions proving big-endian persisted decoding.
  • Spotless, Checkstyle, license format, and license check for pinot-segment-local.

Stack

#19284#19285#19305 (this PR)#19306#19307#19308#19309

Review and merge parent-first.

@xiangfu0 xiangfu0 added feature New functionality index Related to indexing (general) serialization Related to data serialization and deserialization performance Related to performance optimization labels Aug 19, 2026
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/03-delta-codecs branch from 7aa46f4 to 6de5c07 Compare August 20, 2026 09:07
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/03-delta-codecs branch from 6de5c07 to 432f34d Compare August 21, 2026 00:49
@xiangfu0 xiangfu0 added the timeseries-compression Related to time-series data compression label Aug 21, 2026
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/03-delta-codecs branch from 432f34d to e4af8ac Compare August 21, 2026 04:16
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/03-delta-codecs branch from e4af8ac to 489c6ae Compare August 24, 2026 09:40
Base automatically changed from xiangfu0/codex/codec-stack/02-runtime to master August 25, 2026 02:36
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/03-delta-codecs branch from 489c6ae to b81db76 Compare August 25, 2026 03:23
@xiangfu0
xiangfu0 requested review from Jackie-Jiang and a lite review from Copilot August 25, 2026 03:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@codecov-commenter

codecov-commenter commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.58537% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.21%. Comparing base (4824a5f) to head (b81db76).

Files with missing lines Patch % Lines
...gment/local/io/codec/BaseDeltaCodecDefinition.java 87.80% 3 Missing and 2 partials ⚠️
...ment/local/io/codec/DeltaDeltaCodecDefinition.java 97.97% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             master   #19305    +/-   ##
==========================================
  Coverage     67.20%   67.21%            
- Complexity     1418     1424     +6     
==========================================
  Files          3479     3482     +3     
  Lines        223075   223280   +205     
  Branches      35135    35166    +31     
==========================================
+ Hits         149920   150080   +160     
- Misses        61184    61255    +71     
+ Partials      11971    11945    -26     
Flag Coverage Δ
integration 100.00% <ø> (+100.00%) ⬆️
integration1 100.00% <ø> (?)
integration2 0.00% <ø> (ø)
java-25 67.21% <96.58%> (+<0.01%) ⬆️
lane-a 100.00% <ø> (+100.00%) ⬆️
lane-b 0.00% <ø> (ø)
temurin 67.21% <96.58%> (+<0.01%) ⬆️
unittests 67.21% <96.58%> (+<0.01%) ⬆️
unittests1 57.64% <0.00%> (-0.06%) ⬇️
unittests2 39.01% <96.58%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Jackie-Jiang Jackie-Jiang removed the performance Related to performance optimization label Aug 25, 2026
@Jackie-Jiang Jackie-Jiang added performance Related to performance optimization release-notes Referenced by PRs that need attention when compiling the next release notes labels Aug 25, 2026
/// [element_size bytes: first delta (second - first), if count > 1]
/// [(count-2) * element_size bytes: delta-of-deltas, if count > 2]
/// ```
final class DeltaDeltaCodecDefinition

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need DELTADELTA, or just chain 2 DELTA?
What is the pros and cons of both approaches?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are close but not interchangeable, so we kept both — DELTA,DELTA stays legal in the DSL (the validator accepts chained value-preserving transforms), while DELTADELTA is the dedicated form. Trade-offs:

Not the same encoding at index 1. DELTADELTA emits [v0, v1−v0, second differences…] (the classic Gorilla-style form). Composing DELTA twice emits [v0, (v1−v0)−v0, second differences…] — the second slot carries an epoch-magnitude outlier for timestamp columns. Both round-trip exactly, but when a frame-of-reference packing stage follows (DELTADELTA,T64,…), that one outlier forces max bit-width for the whole first 64-value block; the dedicated codec keeps every post-v0 value small.

Cost per stage. The executor runs each stage as a full pass with its own intermediate buffer, so DELTA,DELTA is two passes + one extra scratch buffer per chunk on encode and decode; DELTADELTA does the second difference in a single pass.

Frozen-name surface (the real argument for dropping it). Codec names are permanent on-disk contracts once V7 headers ship. Dropping DELTADELTA later would be a format break, whereas adding it later would have been compatible — so keeping it is a commitment. We think it earns it as the canonical transform for the headline use case (near-regular timestamps), but if you prefer to minimize the frozen surface, removing it from the registry in this PR and re-adding it later is fully backward-compatible — happy to do that instead.

🤖 Addressed by Claude Code

@xiangfu0
xiangfu0 merged commit f40e6d4 into master Aug 25, 2026
12 of 13 checks passed
@xiangfu0
xiangfu0 deleted the xiangfu0/codex/codec-stack/03-delta-codecs branch August 25, 2026 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New functionality index Related to indexing (general) performance Related to performance optimization release-notes Referenced by PRs that need attention when compiling the next release notes serialization Related to data serialization and deserialization timeseries-compression Related to time-series data compression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants