[PEP] Add per-index-type storage size breakdown

[johnsolomonj]

What

Extend GET /tables/{table}/size and GET /tables/{table}/metadata to expose how much disk space each index type (forward index, inverted index, bloom filter, range index, star-tree, text, vector, dictionary) consumes per replica.

Problem

#18185 added tier-level storage breakdown and per-column compression stats. The total segment size per tier is still a black box — operators cannot tell which index types contribute most to storage cost, making informed index configuration decisions impossible.

Prior art

GET /tables/{tableName}/segments/{segmentName}/metadata already returns per-column, per-index-type sizes (ColumnMetadataImpl.getIndexSizeMap(), from #15591) — but only for V3 segments (reads v3/index_map) and packed indexes (excludes directory-backed Lucene text/HNSW vector indexes). This PEP extends the concept to V1/V2, directories, and table-level aggregation.

It's also broken for remoteTier segments: SegmentMetadataImpl.init() silently no-ops when v3/index_map isn't present locally at load time, and tier backends aren't guaranteed to reproduce it. The read path doesn't branch on tier — it's a hidden dependency on local directory completeness. The reload-time refresh below inherited the same dependency and needed a fix.

Solution

Collect per-index sizes at segment write time and persist to metadata.properties — the single source for all segment types (V1/V2/V3, server/minion). The controller fans out via the same bounded path from #18185, aggregates by index type, and exposes a new indexSizeBreakdown field on existing API responses.

Collection — write time

No changes to seal() ordering. Sizes are collected inside flushColIndexes() just before writeMetadata(), when all index files exist on disk in V1 format:

flushColIndexes()
  ├─ seal all index creators
  ├─ collectIndexSizes()  ← stat V1 files; store in memory
  └─ writeMetadata()      ← single write including indexSizeInBytes keys

• Regular index files: stat using IndexType.getFileExtensions(colMeta). For V3 add MAGIC_MARKER_SIZE_BYTES (8) per file — V3 prefixes every packed entry in columns.psf with an 8-byte magic marker. Not applied to directories.
• External Lucene/vector dirs (storeInSegmentFile=false): TextIndexUtils.hasTextIndex() / VectorIndexUtils.hasVectorIndex() + FileUtils.sizeOfDirectory().

Persisted to metadata.properties:

column.message.indexSizeInBytes.inverted_index = 12345678
column.message.indexSizeInBytes.text_index     = 34567890
column.user_id.indexSizeInBytes.forward_index  = 8901234

Collection — reload time

SegmentPreProcessor.process() updates indexSizeInBytes.* only for indexes an IndexHandler actually adds or removes this reload — it never clears and re-derives every key from scratch. An earlier version did, and that was the root cause of a silent data-loss bug on tiered segments (see Prior art): re-deriving requires the local directory to be byte-for-byte complete (v3/index_map for packed entries, individual files otherwise), which a remoteTier segment's tier-materialized copy doesn't guarantee.

• Added: size comes from the writer's in-memory record at write time (SingleFileIndexDirectory's IndexEntry._size, marker-inclusive for non-v1 targets) or raw file length for v1. Reading it back from the freshly-written index_map right after this reload's own packing step is equivalent — it's authored locally, this reload.
• Removed: the key is cleared.
• Untouched (including a pure tier-migration move with no index changes): left as-is in metadata.properties — already correct from an earlier write, nothing to re-derive.

Rule: scope by "did this reload touch it," not by "is index_map non-empty." A freshly-written index_map only proves the entries just written are correct — it says nothing about untouched entries, which depend on the old manifest having loaded correctly at open time, the exact dependency this fix avoids. Format-agnostic (V1/V2/V3 identical), no tier detection. metadata.properties is the single read source for untouched entries. Doesn't affect CRC (.fwd/.dict only).

API response

"indexSizeBreakdown": {
  "forward_index":  { "sizePerReplicaInBytes": 32000000000 },
  "inverted_index": { "sizePerReplicaInBytes": 18000000000 },
  "bloom_filter":   { "sizePerReplicaInBytes":  2000000000 },
  "star_tree":      { "sizePerReplicaInBytes":  4000000000 },
  "dictionary":     { "sizePerReplicaInBytes":  1500000000 },
  "text_index":     { "sizePerReplicaInBytes":  8000000000 },
  "vector_index":   { "sizePerReplicaInBytes": 12000000000 }
}

indexSizeBreakdown is a table-level aggregate — it does not respect the columns= filter, consistent with columnCompressionStats from #18185. The existing columnIndexSizeMap already provides per-column filtered sizes for loaded segments; indexSizeBreakdown adds V1/V2, directory-backed indexes, and table-level aggregation.

Flag

tableIndexConfig.indexSizeStatsEnabled (default false) gates both collection and API inclusion. indexSizeBreakdown appears in responses only when ?includeIndexSizeStats=true is passed — same pattern as ?includeColumnStats=true in #18185.

Modules touched

Module	Change
pinot-segment-spi	New metadata.properties key constants; new accessor in ColumnMetadata
pinot-segment-local	collectIndexSizes() in flushColIndexes(); reload update in SegmentPreProcessor
pinot-common	New IndexSizeBreakdownInfo DTO
pinot-server	Read from ColumnMetadataImpl, include in response
pinot-controller	Aggregate by index type, expose on existing APIs

Notes

• forwardIndexAndDictionaryStorageSizePerReplicaInBytes from #18185 combines forward index and dictionary for compression ratio. indexSizeBreakdown breaks them out separately for a complete per-index-type cost picture.
• Pinot Console UI changes and backfilling existing segments are out of scope.

Implementation

#19255