perf(rest): lazy-decode snapshots in commitTableResponse when snapshot-loading-mode=refs

[technicolorbeat]

Proposed Change

When a client commits changes via Transaction.Commit(), the REST server returns a commitTableResponse containing the full table metadata, including all snapshots. Today, iceberg-go eagerly materializes every snapshot even though common commit paths either discard the returned table or only inspect the snapshot referenced by the current or target branch.

For tables with long snapshot histories (1000+ snapshots), parsing and allocating the complete snapshot history can add material latency and memory pressure to each commit. The exact improvement should be established with repository benchmarks rather than assumed.

Proposed change:

When snapshot-loading-mode=refs is configured, add a parse-time deferred-snapshot path for REST commit responses:

• Intercept the snapshots array before the normal metadata parser materializes every Snapshot; applying laziness after a full ParseMetadataBytes call does not achieve the optimization.
• Retain the original raw snapshot JSON for later materialization.
• Eagerly materialize every snapshot targeted by refs (branches and tags), not only current-snapshot-id. This keeps CurrentSnapshot() and SnapshotByName() fast, including commits to non-main branches.
• Synchronously validate and index the deferred snapshot data sufficiently to preserve the current parse-time error contract. Deferred decoding must not turn malformed metadata into a silent "snapshot not found" result.
• On SnapshotByID(id), consult the eager referenced-snapshot index first and materialize full history only on a miss.
• Materialize full history when Snapshots() is called.
• Use pointer-owned, thread-safe lazy state so metadata copies do not copy a used lock or sync.Once value.

The implementation should remain internal to metadata parsing. REST code should select the parsing mode for the raw commit response rather than mutate an already parsed metadata object.

Correctness requirements:

• CurrentSnapshot() and lookups of any named branch or tag must not materialize unreferenced history.
• Snapshots() and historical SnapshotByID() lookups must return the complete snapshot history from the commit response.
• Concurrent access must materialize history at most once and must be race-free.
• JSON serialization must preserve the complete snapshot list even before materialization; it must never serialize only the eager subset.
• Metadata equality, cloning, builders, format-version upgrades, and other operations that require complete metadata must preserve their existing logical behavior.
• Existing snapshot validation must remain synchronous, or the design must provide an explicit error-returning path. Lazy getters must not silently swallow deferred decode errors.
• Default behavior remains unchanged when snapshot-loading-mode is unset or set to all.

Performance validation:

Add benchmarks using realistic snapshots with non-empty summaries at multiple history sizes (for example 10, 100, 1,000, and 10,000 snapshots). At minimum, measure time and allocations for:

• the existing eager commit-response parse;
• deferred parse without snapshot access;
• CurrentSnapshot() and named-ref access;
• the first historical SnapshotByID() lookup;
• the first Snapshots() call;
• repeated access after materialization.

The intended result is that commit-and-exit and referenced-snapshot-only paths avoid constructing historical Snapshot objects. This remains an O(response size) client-side scan and does not reduce server serialization or network transfer; eliminating those costs would require a REST protocol change.

Backward compatibility:

• No breaking public API changes are required.
• Full snapshot history remains available on demand; metadata must never expose silent truncation.
• The optimization is initially gated by snapshot-loading-mode=refs for compatibility and rollout.

Related:

• REST: Add support for ?snapshots query parameter in loadTable #1748 — added the snapshot-loading-mode catalog property
• feat(rest): support snapshot-loading-mode catalog property in loadTable #1749 — implemented ?snapshots=refs on LoadTable (merged)
• REST: lazy-load unreferenced snapshots when snapshot-loading-mode=refs #1792 — lazy-load unreferenced snapshots omitted from LoadTable(...?snapshots=refs) responses

#1792 is related but distinct: it requires a follow-up network request and context-aware catalog behavior, while this issue concerns deferred local materialization of snapshot bytes already present in a commit response.

Spec reference:

No spec change is required for this client-side optimization. A future protocol-level commit-response projection could additionally reduce response size and server serialization, but is outside this issue.

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

Proposal document

Not applicable — client-side optimization.

Specifications

• Table
• View
• REST
• Puffin
• Encryption
• Other