fix: preserve clone auto-increment continuation

[gouhongshen]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

Fixes #27092

What this PR does / why we need it:

Root cause

Fresh table clones copied the source row from mo_catalog.mo_increment_columns into the destination allocator. That row's offset is the upper bound of a reserved allocation batch (10,000 in the reproducer), not the source session's next visible value (4). Because a fresh clone has a new table ID and an independent allocator, transferring that reservation made its first generated ID 10,001.

The persisted table definition separately carries explicit schema lower bounds such as ALTER TABLE ... AUTO_INCREMENT = 100. Those are user-visible state and must still be preserved.

Changes

• For a fresh clone, reconstruct visible auto-increment state from the copied row maximum and the larger of the source/destination schema lower bounds. Do not copy the source allocator's reserved batch upper bound.
• Continue copying raw allocator state for ALTER COPY paths that retain the column-ID space, where existing CN caches make that state part of the ownership handoff.
• Continue copying allocator state for hidden internal auto-increment columns; hidden index-table allocator handling is unchanged.
• Add unit coverage for the fresh-clone and ALTER COPY ownership boundaries, including a source schema lower bound larger than the destination's.
• Add BVT coverage for table clone, database clone, both Data Branch forms, empty sources, deleted maximum IDs, explicit AUTO_INCREMENT, explicit transactions, and historical snapshots.
• Update the table-clone and temporary-table-clone BVT expectations to assert contiguous IDs from the highest copied row instead of discarded source reservation batches.

Issue-to-test proof

• The exact issue reproduction reports source next 4, clone next 5, and first cloned-table insert 4 for table, database, and both Data Branch clone forms.
• Empty sources insert 1; a source whose largest row was deleted seeds the independent clone from the copied maximum; explicit AUTO_INCREMENT = 100 inserts 100; snapshot clone asserts snapshot-visible rows plus exact 4 / 5 inserted/next values.
• A permanent source cloned into a temporary table preserves copied IDs 10 and 11, then assigns 12 to the first destination insert.
• Unit tests prove that fresh visible columns discard the persisted reservation while preserving the larger source/destination schema floor; ALTER COPY and hidden internal columns retain allocator state required by their ownership model.
• The existing chained-clone BVT proves subsequent clone and delete/insert sequences continue at 40001 through 40006, without introducing a new 10,000-value gap at each clone.

Verification

Latest semantic rebase: base 0d7f4dd69ac86a2d8e8746af568a7de8aaa4287a / head a9f360b768c09d2682fc960812306f1c7595cc94 was rebased onto main at 117714dd98de39aadc1e55e49f584dba58c51a10, producing exact head bc71e21cdf903b0dc2c3a9331c79ef847988db11.

• git range-diff 0d7f4dd69a..a9f360b768 origin/main..HEAD: all four patches are equivalent.
• Stable aggregate patch ID before and after every rebase: ff530dfc6cb58fd343bc0663b95c917a95aa40fc.
• git diff --check origin/main..HEAD: passed; final diff remains exactly the intended six clone files.
• Repository CGo wrapper, exact non-empty selection, -v, and -count=1: all eight focused clone allocator tests plus incoming TestDedupLoadCleansUpAfterPanic, TestDedupLoadWaiterGetsSuccessfulOwnerValue, and TestDedupLoadCleansUpAfterLoadCancel passed on exact head bc71e21cdf. Each incoming concurrency test also passed independently under -race -count=100 after adaptive measurement selected the 100-run cap.
• Full pkg/sql/compile and pkg/sql/colexec/table_clone packages passed on patch-equivalent prior head 93146ffceb; the two later base increments change only fileservice and objectio tests, so that package evidence remains semantically valid. The changed fileservice regression passed on head a9f360b768, and the three changed objectio tests passed in normal and focused race modes on head bc71e21cdf.
• The incoming fix: preserve insert metadata lock during index creation #27792 two-CN regular/fulltext index regression and pessimistic DDL atomicity BVT retain exact merged-base evidence. This PR changes neither INSERT planning, ObjectRef ownership, metadata locking, nor their fixtures, so those topology/BVT results remain semantically valid.
• On the prior combined fix(frontend): publish ANALYZE stats by table generation #27758 tree, all 39 incoming stats publication/admission/generation regressions passed together with the eight clone allocator tests. Full pkg/sql/compile, pkg/sql/colexec/table_clone, pkg/frontend, pkg/sql/plan, pkg/vm/engine, and pkg/vm/engine/disttae packages passed.
• The test(morpc): isolate factory timeout budget #27812 MORPC factory-timeout lifecycle test and the full pkg/common/morpc package passed on the prior patch-equivalent head; the current base change does not touch MORPC inputs.
• Earlier incoming focused regressions passed on their combined code trees: eight prepared FULLTEXT/IVFFLAT tests, two single-remote reader ownership tests, fourteen prepared-numeric planner/frontend tests, the real TestIssue27294PreparedNumericOverloads execution test, and the MORPC deadline-contract test. The relevant full compile, table-clone, planner, table-function, IVFFLAT, frontend, and MORPC packages passed.
• Exact BVT workflow 33192377421 attempt 1 on superseded patch-equivalent head a9f360b768 passed all three changed clone BVT files (65/65, 74/74, and 163/163). Attempt 2 was still active when the required branch rewrite superseded that head. Fresh CI is required for the new base.
• Earlier base-sensitive validation remains recorded: cross-account Data Branch, view-metadata decoder, snapshot quota, vector-index JSON, unsigned metadata, ranking window, FK SET NULL, SQLSTATE/transaction/Starlark, account-drop/DAG-reclaim/cross-account snapshot, timestamp function/binder/CTAS, authentication/clock/configuration/cancellation, proxy fixture, variance protocol, planner, and message regressions passed as those inputs arrived; the relevant full planner, vector-index, frontend, Data Branch, executor, aggregate, group, error, compile, proxy, message, and table-clone packages passed.

The incoming base adds ten merged changes after the prior reviewed base, with no overlap in this PR's six files:

• edcc672d64 (Fix prepared fulltext score threshold; push prepared distance bounds #27765) carries prepared FULLTEXT score guards and prepared IVFFLAT distance bounds. Its NULL/reuse, literal/parameter parity, per-row-expression, plan-shape, and engine guard regressions passed locally; the full planner, table-function, and IVFFLAT packages passed.
• 52e307962f (fix(compile): preserve in-memory rows on single remote scans #27796) changes single-remote scan reader ownership inside pkg/sql/compile. Both new reader-selection regressions, the eight clone allocator tests, and the full compile/table-clone packages passed together, proving the remote scan path does not alter clone allocator ownership.
• 8b6a44e83c (fix: accept fractional prepared numeric overloads #27556) adds sparse prepared-numeric plan metadata and runtime category rebinding. Its focused planner/frontend and real execute-path regressions passed; the full planner/frontend/compile packages passed.
• 90534b6c61 (fix(morpc): normalize expired deadline errors #27711) normalizes expired MORPC deadlines while preserving explicit cancellation. Its focused deadline contract and full MORPC package passed.
• 67ac89c058 (test: cover multi-insert prepare and privileges #27741) adds four multi-insert PREPARE/privilege BVT assets only. It changes no production or clone input. Its exact merged-base BVT evidence remains valid.
• 4cbdc2cf46 (fix(frontend): publish ANALYZE stats by table generation #27758) makes ANALYZE stats publication table-generation-aware and adds scoped cancellation/admission ownership. Its 39 focused regressions and the affected full packages passed with this PR's clone tests. This PR changes no stats lifecycle state or ANALYZE BVT input.
• 2e1be09278 (test(morpc): isolate factory timeout budget #27812) isolates a MORPC factory-timeout regression from queue-admission and process-wide worker timing. It changes test code only; the exact test and full MORPC package passed on the patch-equivalent combined head.
• 52336df805 (fix: preserve insert metadata lock during index creation #27792) preserves the INSERT write target's metadata lock by deep-copying scan-local ObjectRef values. Its focused alias-isolation UT passed together with this PR's eight clone tests, and the full compile/table-clone packages passed. This PR does not touch INSERT plans or metadata lock ownership.
• 0d7f4dd69a (test(fileservice): remove async callback flush deadline #27789) removes an async callback flush deadline from one fileservice lifecycle test and adds a direct pending-queue assertion. It changes test code only; the exact regression passed on head a9f360b768.
• 117714dd98 (test(objectio): synchronize dedupLoad waiter admission #27782) replaces scheduler sleeps in three objectio dedupLoad waiter tests with a bounded admission barrier. It changes test code only. All three changed tests passed with the eight clone allocator tests on head bc71e21cdf; each also passed independently under -race -count=100.

The incoming FULLTEXT/IVFFLAT, multi-insert, ANALYZE, concurrent-index, and pessimistic DDL public-path results remain exact merged-base evidence because this PR changes none of their planner/index/stats/lock execution inputs or case files. Fresh CI will independently exercise the combined head.

CI investigation

• [Bug]: mo-tester sorts rows by ignored columns, causing false BVT failures #27164 remains the sole unresolved unrelated blocker pending fresh CI on head bc71e21cdf. Its latest evidence belongs to superseded patch-equivalent head a9f360b768.
• Workflow 33192377421 attempt 1 reproduced the same root cause in PROXY BVT. snapshot/sys_restore_system_table_to_sys_account.sql completed 313 statements with 311 successful and exactly two failed. Its mo_stored_procedure comparisons at SQL rows 167 and 183 expected ignored proc_id values 1920000/1920001, received ignored values 9/10, and mo-tester cross-paired test_if_hit_second_elseif with test_if_hit_else at row:0,column:1.
• All three changed clone BVT files passed again inside the failed producer: clone_auto_increment_state.sql 65/65, table_clone.sql 74/74, and temporary_table_clone.sql 163/163. Same-head Ubuntu UT, shared build, SCA, coverage UT, and PESSIMISTIC BVT passed.
• The aggregate Coverage failure was derivative. Its direct job log records prerequisites_ready: false for generation 33192377421-1 and exits before coverage processing.
• The six-file clone-only diff changes neither the failing snapshot case nor mo-tester. Its only production change is confined to constructTableClone; the failing case contains no clone or Data Branch path and has identical base/head blob 20efac5d8d440ef3a86e459d0d0553c921c2e398. No PR code change was made. Repeated exact-head evidence is recorded on [Bug]: mo-tester sorts rows by ignored columns, causing false BVT failures #27164.
• Failed jobs from workflow 33192377421 attempt 1 were rerun exactly once. Attempt 2 remained active when the required branch rewrite superseded its head and will not be rerun again.
• [Bug]: pkg/proxy coverage UT crashes on concurrent test-server map writes #27771 remains resolved as a blocker for this PR. Its fix, merged PR fix(proxy): synchronize test CN server global variables #27772, remains in the current base.
• [Bug]: Ubuntu UT job cascades across packages after cluster and checkpoint timeouts #26519 remains resolved as a blocker for this PR; its broader deferred issue remains open independently.
• The checked-allocator failure formerly tracked by [Bug]: TestLazyCacheFSReadsMagicPrefixedRemoteObjectsWithoutLocalChecksum leaks IOEntry data in UT Coverage #27145 remains fixed by merged test: release lazy cache read vector #27147.

Residual risks

A fresh clone intentionally owns an independent allocator. If the source's highest generated rows were deleted, the clone starts after the highest row it actually copied rather than inheriting a source CN's unused cached range. Explicit schema lower bounds remain preserved. There are no catalog, wire-format, or API changes in this PR, and the fix does not read or mutate source CN allocator caches.

[gouhongshen]

One unrelated CI blocker remains pending fresh CI on rebased head bc71e21cdf. Its latest evidence belongs to superseded patch-equivalent head a9f360b768.

• [Bug]: mo-tester sorts rows by ignored columns, causing false BVT failures #27164 reproduced unchanged in workflow 33192377421 attempt 1. PROXY BVT completed snapshot/sys_restore_system_table_to_sys_account.sql with 313 statements, 311 successful and exactly two failed. Both failures were the mo_stored_procedure comparisons at SQL rows 167 and 183: expected ignored proc_id values 1920000/1920001, actual ignored values 9/10, and mo-tester again reported row:0,column:1 after cross-pairing test_if_hit_second_elseif with test_if_hit_else.

The aggregate Coverage failure was derivative: its direct log records prerequisites_ready: false for generation 33192377421-1 and exits before coverage processing. Same-head ordinary Ubuntu UT, shared build, SCA, coverage UT, and PESSIMISTIC BVT passed. All three changed clone BVT cases passed inside the failed producer: clone_auto_increment_state.sql 65/65, table_clone.sql 74/74, and temporary_table_clone.sql 163/163. The six-file clone-only diff changes neither the snapshot case nor mo-tester; its only production change is confined to constructTableClone, while the failing case contains no clone or Data Branch path and has identical base/head blob 20efac5d8d440ef3a86e459d0d0553c921c2e398. No PR code change was made. Exact-head evidence is recorded on #27164.

Failed jobs from workflow attempt 1 were rerun exactly once. Attempt 2 remained active when the required branch rewrite superseded its head and will not be rerun again.

#27771 remains resolved as a blocker for this PR. Merged PR #27772 remains in the current base.

The required conflict-free rebase onto main at 117714dd98 is patch-equivalent across all four PR commits and produced head bc71e21cdf; the stable aggregate patch ID remains ff530dfc6c. All eight focused clone allocator tests and the three incoming objectio waiter-admission tests passed on the exact head; each incoming concurrency test also passed independently under -race -count=100. Fresh CI is starting.

[XuPeng-SH]

Deep review completed on exact head 9cf7087.

The fresh-clone versus ALTER-copy allocator ownership split is consistent: visible columns on a new table are rebuilt from the transaction-consistent copied-row maximum plus schema/session floors, while retained-ID ALTER paths and hidden internal allocators preserve their fenced state. I also verified the issue reproduction against a locally built exact-head server (source next 4, clone insert 4, clone next 5), explicit AUTO_INCREMENT=100, and the clone-of-clone empty-source counterexample. Focused checked-allocator tests, the full table_clone package, and focused race tests passed. No blocking code issue found.