Skip to content

Add V7 raw forward index format for codec pipelines - #19307

Open
xiangfu0 wants to merge 1 commit into
masterfrom
xiangfu0/codex/codec-stack/05-v7-format
Open

Add V7 raw forward index format for codec pipelines#19307
xiangfu0 wants to merge 1 commit into
masterfrom
xiangfu0/codex/codec-stack/05-v7-format

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds a self-describing V7 fixed-byte forward-index format with explicit magic and the canonical codecSpec in its header.
  • Routes every non-null codecSpec, including plain LZ4, SNAPPY, GZIP, and ZSTD, through V7. Legacy compression configuration still selects legacy formats.
  • Fails table-config validation unless codec pipelines target an enabled RAW, single-value INT/LONG forward index with a valid pipeline and bounded V7 chunk settings.
  • Reconciles legacy-to-V7, V7 spec changes, unchanged V7, and V7-to-legacy transitions during reload.
  • Preserves the historical reader path for old fixed-byte versions, including legacy version 7, unless the new discriminator is present.

Compatibility

This is an on-disk upgrade boundary. Upgrade every component that may build or read segments before enabling codecSpec. Before a binary rollback, remove codecSpec and reload or regenerate affected V7 forward indexes into a legacy format.

Verification

  • Real codec round trips, corrupt-input cases, released legacy fixtures, factory routing, table validation, and reload transitions.
  • Real offline and realtime clusters with no mocks, both query engines, Kafka force-commit/load, and direct V7 reader verification.
  • Spotless, Checkstyle, license formatting, and license checks on all affected modules.

Stack

#19306 is merged. This PR now owns V7 selection, validation, reload reconciliation, and end-to-end coverage. Child PRs #19308 and #19309 need to be rebased parent-first with duplicated behavior removed.

@xiangfu0 xiangfu0 added feature New functionality index Related to indexing (general) index-spi Related to index SPI interfaces serialization Related to data serialization and deserialization needs-attention Used for sensitive changes - allows searching PRs post release to narrow down causes for regression. labels Aug 19, 2026
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/05-v7-format branch from d906d6d to 04dd66f Compare August 20, 2026 09:07
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/05-v7-format branch from 04dd66f to 669848c Compare August 21, 2026 00:49
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/05-v7-format branch from 669848c to e784709 Compare August 21, 2026 04:16
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/05-v7-format branch from e784709 to fa07c2c Compare August 24, 2026 09:40
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/05-v7-format branch from fa07c2c to 74c773d Compare August 25, 2026 04:55
@codecov-commenter

codecov-commenter commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 4.82574% with 355 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.52%. Comparing base (5e914c9) to head (00bceb0).

Files with missing lines Patch % Lines
.../forward/FixedByteChunkSVForwardIndexReaderV7.java 0.00% 145 Missing and 1 partial ⚠️
...riter/impl/FixedByteChunkForwardIndexWriterV7.java 0.00% 143 Missing ⚠️
.../local/segment/index/forward/ForwardIndexType.java 4.54% 20 Missing and 1 partial ⚠️
.../impl/fwd/SingleValueFixedByteRawIndexCreator.java 0.00% 10 Missing ⚠️
...ment/index/forward/ForwardIndexCreatorFactory.java 28.57% 5 Missing and 5 partials ⚠️
...ocal/segment/index/loader/ForwardIndexHandler.java 37.50% 5 Missing and 5 partials ⚠️
...ment/index/readers/forward/ChunkReaderContext.java 0.00% 7 Missing and 1 partial ⚠️
...gment/index/forward/ForwardIndexReaderFactory.java 46.15% 5 Missing and 2 partials ⚠️

❗ There is a different number of reports uploaded between BASE (5e914c9) and HEAD (00bceb0). Click for more details.

HEAD has 4 uploads less than BASE
Flag BASE (5e914c9) HEAD (00bceb0)
unittests 2 1
java-25 6 5
temurin 6 5
unittests2 1 0
Additional details and impacted files
@@              Coverage Diff              @@
##             master   #19307       +/-   ##
=============================================
- Coverage     67.55%   57.52%   -10.03%     
+ Complexity     1430        7     -1423     
=============================================
  Files          3486     2688      -798     
  Lines        224100   164280    -59820     
  Branches      35370    26674     -8696     
=============================================
- Hits         151392    94505    -56887     
- Misses        60678    61758     +1080     
+ Partials      12030     8017     -4013     
Flag Coverage Δ
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-25 57.52% <4.82%> (-10.03%) ⬇️
lane-a 100.00% <ø> (ø)
lane-b 0.00% <ø> (ø)
temurin 57.52% <4.82%> (-10.03%) ⬇️
unittests 57.52% <4.82%> (-10.03%) ⬇️
unittests1 57.52% <4.82%> (-0.12%) ⬇️
unittests2 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/05-v7-format branch from 74c773d to 4fe389f Compare August 25, 2026 20:25
Base automatically changed from xiangfu0/codex/codec-stack/04-t64-gorilla to master August 25, 2026 21:20
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/05-v7-format branch 3 times, most recently from 7520945 to 598b44a Compare August 25, 2026 22:54
@xiangfu0 xiangfu0 added the backward-incompat Introduces a backward-incompatible API or behavior change label Aug 25, 2026
@xiangfu0 xiangfu0 added the upgrade-incompat PR may introduce incompatibility during upgrade of an installation label Aug 25, 2026
@xiangfu0
xiangfu0 requested review from Jackie-Jiang and yashmayya and a lite review from Copilot August 26, 2026 01:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/05-v7-format branch 3 times, most recently from 236b5a3 to 00bceb0 Compare August 28, 2026 09:11
The legacy fixed-byte layout cannot encode a codec pipeline or self-identify a new reader format. Add a V7 header with explicit magic and the canonical codec specification so readers can dispatch safely and validate every frame.

Route every non-null codecSpec to V7 and reserve legacy formats for legacy compression configuration. Validate RAW single-value INT/LONG columns and exact chunk bounds during table-config validation, and reconcile format changes during reload.

V7 is forward-only across mixed versions: upgrade all segment builders and readers before enabling codecSpec. Before a binary rollback, remove codecSpec and rewrite affected forward indexes to a legacy format.

Cover corruption, released legacy fixtures, validation, reload transitions, and offline/realtime clusters with real metadata and segments.
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/05-v7-format branch from 00bceb0 to 175ac1a Compare August 29, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backward-incompat Introduces a backward-incompatible API or behavior change feature New functionality index Related to indexing (general) index-spi Related to index SPI interfaces needs-attention Used for sensitive changes - allows searching PRs post release to narrow down causes for regression. serialization Related to data serialization and deserialization upgrade-incompat PR may introduce incompatibility during upgrade of an installation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants