Synchronizer: Add --skip-catalog-sync flag for DR-oriented syncs - #263
Open
saidixith002 wants to merge 1 commit into
Open
Synchronizer: Add --skip-catalog-sync flag for DR-oriented syncs#263saidixith002 wants to merge 1 commit into
saidixith002 wants to merge 1 commit into
Conversation
Allows skipping create/overwrite/remove of catalog objects on the target while still synchronizing catalog-roles and grants for catalogs that already exist there. Useful when target catalogs are pre-created with DR-specific storage locations that shouldn't be clobbered.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
--skip-catalog-syncflag tosync-polaristhat skips creation, overwrite, and removal of catalog objects on the target, while still synchronizing catalog-roles andgrants for catalogs that already exist there.
This complements the existing
--skip-iceberg-contentflag (added in #256) and is aimed at the same class of problem: disaster-recovery setups where the synchronizershouldn't be allowed to mutate certain top-level resources on the target.
Motivation
In a DR setup, target catalogs are often pre-created with intentionally different storage locations (
allowedLocations,default-base-location,roleArn, etc.) than thesource. Today, if a catalog exists on both source and target and the synchronization strategy is
CREATE_AND_OVERWRITEorREPLICATE(or--diff-onlydetects adifference), the synchronizer will
dropCatalogCascade+createCatalogon the target, copying the source catalog's storage config verbatim and silently clobbering theDR-specific location.
--skip-catalog-synclets operators keep principal, principal-role, catalog-role, and grant synchronization fully active while guaranteeing catalog objects themselves arenever created, overwritten, or removed on the target.
Changes
PolarisSynchronizer: newskipCatalogSyncconstructor parameter/field.syncCatalogs()now checks this flag before each of the create/overwrite/remove operations.SyncPolarisCommand: new--skip-catalog-syncCLI option, wired into thePolarisSynchronizerconstructor call.Behavior when --skip-catalog-sync is set
synchronize catalog-roles/grants against, so these are also excluded from catalog-role sync for that catalog.
synchronized against the existing target catalog.
Iceberg namespace/table sync is governed independently by
--skip-iceberg-content, so the two flags can be combined to synchronize only principals, principal-roles,catalog-roles, and grants, without ever touching catalog objects or Iceberg content on the target.
Backward compatibility
--skip-catalog-syncdefaults tofalse. No behavior changes for existing invocations unless the flag is explicitly passed.Testing
Added
PolarisSynchronizerSkipCatalogSyncTest, covering the skipped case (create/overwrite/remove all skipped on the target, source-only catalogs excluded from catalog-rolesync, catalogs existing on both sides still get catalog-role sync) and the non-skipped case (existing behavior unchanged). Updated
PolarisSynchronizerSkipIcebergContentTestcall sites for the new constructor parameter; all existing tests continue to pass.