Add DELTA and DELTADELTA transform codecs - #19305
Conversation
7aa46f4 to
6de5c07
Compare
6de5c07 to
432f34d
Compare
432f34d to
e4af8ac
Compare
e4af8ac to
489c6ae
Compare
489c6ae to
b81db76
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19305 +/- ##
==========================================
Coverage 67.20% 67.21%
- Complexity 1418 1424 +6
==========================================
Files 3479 3482 +3
Lines 223075 223280 +205
Branches 35135 35166 +31
==========================================
+ Hits 149920 150080 +160
- Misses 61184 61255 +71
+ Partials 11971 11945 -26
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| /// [element_size bytes: first delta (second - first), if count > 1] | ||
| /// [(count-2) * element_size bytes: delta-of-deltas, if count > 2] | ||
| /// ``` | ||
| final class DeltaDeltaCodecDefinition |
There was a problem hiding this comment.
Do we need DELTADELTA, or just chain 2 DELTA?
What is the pros and cons of both approaches?
There was a problem hiding this comment.
They are close but not interchangeable, so we kept both — DELTA,DELTA stays legal in the DSL (the validator accepts chained value-preserving transforms), while DELTADELTA is the dedicated form. Trade-offs:
Not the same encoding at index 1. DELTADELTA emits [v0, v1−v0, second differences…] (the classic Gorilla-style form). Composing DELTA twice emits [v0, (v1−v0)−v0, second differences…] — the second slot carries an epoch-magnitude outlier for timestamp columns. Both round-trip exactly, but when a frame-of-reference packing stage follows (DELTADELTA,T64,…), that one outlier forces max bit-width for the whole first 64-value block; the dedicated codec keeps every post-v0 value small.
Cost per stage. The executor runs each stage as a full pass with its own intermediate buffer, so DELTA,DELTA is two passes + one extra scratch buffer per chunk on encode and decode; DELTADELTA does the second difference in a single pass.
Frozen-name surface (the real argument for dropping it). Codec names are permanent on-disk contracts once V7 headers ship. Dropping DELTADELTA later would be a format break, whereas adding it later would have been compatible — so keeping it is a commitment. We think it earns it as the canonical transform for the headline use case (near-regular timestamps), but if you prefer to minimize the frozen surface, removing it from the registry in this PR and re-adding it later is fully backward-compatible — happy to do that instead.
🤖 Addressed by Claude Code
Context
Stacked on #19285 and part of the split of #18229.
What changed
DELTAandDELTADELTAas typed-layout-preserving INT/LONG transforms.Safety and scope
Registering DELTA/DELTADELTA makes them available to the bounded runtime API, but production forward-index creation/loading is not wired until #19307. This slice changes no segment format or production I/O path.
Verification
decodeIntopaths, transform chains, and malformed-width rejection.pinot-segment-local.Stack
#19284 → #19285 → #19305 (this PR) → #19306 → #19307 → #19308 → #19309
Review and merge parent-first.