fix: fence DDL visibility across CN admission

[ck89119]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

Fixes #27743

What this PR does / why we need it:

A successful client DDL commit could be followed by a fresh proxy-routed connection on another CN whose applied logtail was still older than the DDL commit. The new connection could therefore fail catalog lookup with no such table.

This PR adds a two-sided DDL visibility fence:

• a CN publishes DDLVisibilityBarrierReady only after QueryService is listening, synchronizes the maximum published frontier, and only then opens public ingress;
• startup rollback and ordinary shutdown stop periodic heartbeats, publish DDLVisibilityBarrierReady=false and ingress-ready=false, wait for the authoritative inventory to observe the withdrawal, and only then close QueryService;
• a DDL commit synchronizes every barrier-ready CN generation, refreshes authoritative membership after fan-out, and retries until the generation/address set is stable;
• if an RPC from an older membership snapshot fails, the sender authoritatively revalidates the exact service/generation/address tuple: a withdrawn or replaced target restarts fan-out, while a still-current target preserves the partial-barrier error;
• DDL no-op commits synchronize the transaction's observed catalog frontier;
• automatic DDL fencing uses the new cancellable SyncCommitV2 method, gated by MORPC v36, so rolling upgrades never invoke the legacy receiver's fatal wait path; upstream v35 remains exclusively assigned to scaled-variance wire state;
• live pre-v36→v36 activation establishes a distributed DDL-producer cut: the control plane concurrently sends the complete CN target set; every CN blocks/drains local DDL, keeps its barrier receiver reachable, waits for all targets to report Prepared, applies the converged frontier, waits for all targets to report Fenced, then restores ingress and releases DDL;
• activation failures remain ingress-withdrawn with DDL blocked while retaining the v36 receiver capability; retries resolve hidden CNs through authoritative raw inventory and fence the recovery RPC by generation/query-address identity;
• periodic heartbeat snapshot/send is serialized with activation publication, preventing stale same-generation readiness from overtaking a withdrawal; activation also preserves pre-start ingress=false until all public listeners are live;
• the receiver propagates logtail lag, timeout, and cancellation instead of acknowledging an incomplete fence.

The startup handshake and post-fan-out membership check close both admission orderings: a CN published before the final check is included in the DDL retry, while a CN published afterwards catches up from the frontier already stored by the completed fan-out.

Validation:

• deterministic membership-addition, generation-replacement, stale-target withdrawal/replacement, persistent-target failure, revalidation failure, mixed-version, startup-frontier, concurrent DDL-vs-activation linearization, distributed Prepared/Fenced convergence, activation fail-closed/retry, shutdown-withdrawal ordering, withdrawal-failure, lag/cancel, and negative-path unit tests;
• full tests for frontend, CN service, query service client, cluster service, HAKeeper/logservice metadata, and transaction client owners/consumers;
• focused race tests with -count=20 and full package race tests for shared-state owners;
• go vet for all affected production packages;
• generated protobuf outputs regenerated from source with the repository generator;
• a two-CN embedded topology check that creates DDL on CN-0 and immediately reads the table through CN-1.

[aptend]

Deep-reviewed exact head 3df6a5c. There is one blocking rolling-upgrade correctness gap; details inline. Local build-only tests, go vet, owning-package tests, and focused race tests passed.

[iamlinjunhong]

Reviewed exact head 3df6a5c against the current origin/main. One confirmed P1 rolling-upgrade correctness issue remains; details inline. I inspected the complete diff, protocol-version update path, CN startup/admission/heartbeat/shutdown lifecycle, DDL commit fan-out, generated protobuf changes, and tests. No additional P0/P1 findings.

[aptend]

Re-reviewed the complete diff and the increment from my prior reviewed head 3df6a5c to exact head 7340575, including both resolved threads and the author replies. The original missing v34→v35 activation fence is now present, but two activation-ordering races remain blocking. Existing focused normal/race tests, build, and vet pass; an exact-head startup-window counterexample fails.

[aptend]

Re-reviewed the full diff and all prior threads at exact head 6b1e497. The previous stale-heartbeat ordering and pre-start ingress issues are closed: heartbeat snapshot/send is serialized with activation, and activation restores the captured ingress state. Two blocking issues remain: failed live activation cannot be retried once the target is withdrawn from admission, and this branch collides with the MORPC v35 already on current main. Exact-head package tests, focused race tests, go vet, and diff checks passed; the withdrawn-target counterexample fails as described inline.

[aptend]

Deep re-review of exact head 8dc075e. I read all reviews, inline comments, author replies, and resolved/unresolved thread state; compared the increment since 6b1e497; and rechecked the complete DDL visibility protocol, startup/admission/heartbeat/shutdown lifecycle, DDLCommitGate, fan-out, protobuf, and protocol-version diff. Both remaining blockers are materially closed. Activation recovery now authoritatively refreshes and resolves the raw CN inventory, so an ingress/admission-hidden or draining pending CN stays reachable; the request carries the selected generation/address and the receiver rejects a stale replacement before touching activation state. Current main’s v35 remains the scaled-variance contract, while this feature consistently uses v36 for activation, sender gating, SyncCommitV2, latest version, protobuf, and tests. Startup readiness publication is also serialized and directly heartbeated while holding the activation owner lock, avoiding stale publication and self-deadlock. Q1-Q3 audit found bounded target/result state, context-bounded refresh/RPC/wait paths, release of all responses, and fail-closed DDL/ingress ownership across retry and shutdown. Exact-head full tests for all eight owning packages pass; focused activation/recovery races pass five runs; go vet and diff checks pass. Two exact-head CI jobs are still queued/running, with no failure reported. No blocking correctness, rolling-upgrade, lifecycle, concurrency, or performance issue remains.