Validate codecSpec and support codec-aware forward-index reloads

[xiangfu0]

Context

Stacked on #19307 and part of the split of #18229.

Summary

Completes the activation begun in #19307 by adding semantic table-config validation and reload/rewrite decisions for legacy-to-V7, V7-to-legacy, and V7 spec changes.

Semantic validation and scope enforcement

• Validates the resolved effective ForwardIndexConfig against the closed runtime registry and V7 shape constraints.
• Rejects codecSpec on disabled forward indexes and OPEN_STRUCT children rather than silently discarding it.
• Allows realtime tables to use the normal mutable consuming representation; the codec is applied when an immutable segment is built or committed.
• Keeps a compression-only spec that maps to one legacy ChunkCompressionType on the established raw format.

Reload and rewrite behavior

• Legacy raw → equivalent legacy-compatible codecSpec: no rewrite.
• Legacy raw → V7-requiring transform/chain/non-default option: rewrite to V7.
• V7 → a different canonical V7 spec: rewrite; semantically equivalent canonical specs are no-ops.
• V7 → supported legacy compression (PASS_THROUGH, SNAPPY, ZSTANDARD, LZ4, or GZIP): rewrite to the legacy format for rollback.
• RAW codec rewrites are decided independently of dictionary add/remove and dictionary-optimization outcomes, so combined changes complete in one reload.
• Historical DELTA/DELTADELTA enum values remain only as defensive compatibility if an old invalid config reaches reload; current validation rejects them as user-configurable rollback targets.

Compression statistics

V7 has no single legacy compression enum. Compression statistics now preserve and report the tracked raw uncompressed size with a nullable legacy type, including segment creation and reload, without changing the persisted metadata/API shape.

Verification

• Table-config validation, disabled-forward, immutable/realtime, and OPEN_STRUCT coverage.
• Legacy↔V7, V7 spec change, dictionary add/remove, and heuristic-decline reload tests.
• Actual V7 segment creation/reload tests with compression statistics enabled.
• Spotless, Checkstyle, license format, and license check on all affected modules.

Stack

#19284 → #19285 → #19305 → #19306 → #19307 → #19308 (this PR) → #19309

Review and merge parent-first.

[xiangfu0]

Pushed aafe8c6fd7 correcting the CompressionCodec Javadoc this PR adds. Two issues, both found while assessing whether compressionCodec could be deprecated in favour of codecSpec:

1. The DELTA/DELTADELTA guidance described a migration that is not possible. The previous wording ("For new SV INT/LONG columns, prefer codecSpec=..." / "migrating to codecSpec changes on-disk semantics") implied users have a working compressionCodec: DELTA config to migrate from. They do not: both constants are declared (false, false), so ForwardIndexType.validateForwardIndexEnabled rejects them on the dictionary branch (isApplicableToDictEncodedIndex()) and on the raw branch (isApplicableToRawIndex()) alike. I verified this empirically rather than by inspection — a table config with compressionCodec: DELTA on a RAW single-value INT column, the exact shape the codec targets, fails validation:

IllegalStateException: Compression codec: DELTA is not applicable to raw column: intCol

Same for DELTADELTA and for LONG. Note this makes TableConfigUtils.validateGorillaCompressionCodecIfPresent (the SV / INT-LONG checks for DELTA and DELTADELTA) effectively unreachable on the accept path — the applicability check fires first. That is pre-existing and out of scope here, but worth knowing.

The Javadoc now states plainly that these values are rejected by table-config validation and points to codecSpec="DELTA,LZ4" as the supported route, rather than framing it as a migration.

2. The six constants with no codecSpec equivalent had no Javadoc at all, which read as an oversight next to the four annotated ones. PASS_THROUGH, MV_ENTRY_DICT and the CLP family now each say so explicitly and why:

• PASS_THROUGH — no equivalent; the DSL has no identity codec and rejects a blank spec, and this is the default for METRIC columns, so it stays the only way to state "uncompressed".
• MV_ENTRY_DICT — no equivalent; codecSpec applies only to RAW forward indexes.
• CLP family — no equivalent; whole-index formats for STRING columns rather than chunk codecs, validated against stored type instead of the applicability flags.

I also fixed the malformed {@link CLPForwardIndexCreatorV2) reference in that comment (mismatched brace/paren, and those classes are not reachable from pinot-spi anyway) by using code spans.

Net effect: the guidance now covers all 12 constants and matches what validation actually does. Comment-only change, no behavior difference.

Verification: pinot-spi compiles; IndexCombinationValidationTest, TableConfigUtilsTest, ForwardIndexTypeTest all green (169 tests); spotless/checkstyle/license clean. #19309 has been rebased onto the new head so the chain stays intact.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 71.42857% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.58%. Comparing base (00bceb0) to head (d0afba2).

Files with missing lines	Patch %	Lines
.../segment/index/openstruct/OpenStructIndexType.java	66.66%	0 Missing and 1 partial ⚠️
...r/api/resources/SegmentCompressionStatsReader.java	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@                              Coverage Diff                              @@
##             xiangfu0/codex/codec-stack/05-v7-format   #19308      +/-   ##
=============================================================================
+ Coverage                                      67.56%   67.58%   +0.01%     
  Complexity                                      1430     1430              
=============================================================================
  Files                                           3488     3488              
  Lines                                         224459   224464       +5     
  Branches                                       35427    35429       +2     
=============================================================================
+ Hits                                          151663   151696      +33     
+ Misses                                         60730    60710      -20     
+ Partials                                       12066    12058       -8     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	67.58% <71.42%> (+0.01%)	⬆️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	67.58% <71.42%> (+0.01%)	⬆️
unittests	67.57% <71.42%> (+0.01%)	⬆️
unittests1	57.53% <0.00%> (+<0.01%)	⬆️
unittests2	39.39% <71.42%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.