Skip to content

Synchronizer: Add --skip-catalog-sync flag for DR-oriented syncs - #263

Open
saidixith002 wants to merge 1 commit into
apache:mainfrom
saidixith002:feat/skip-catalog-sync
Open

Synchronizer: Add --skip-catalog-sync flag for DR-oriented syncs#263
saidixith002 wants to merge 1 commit into
apache:mainfrom
saidixith002:feat/skip-catalog-sync

Conversation

@saidixith002

Copy link
Copy Markdown
Contributor

Summary

Adds a --skip-catalog-sync flag to sync-polaris that skips creation, overwrite, and removal of catalog objects on the target, while still synchronizing catalog-roles and
grants for catalogs that already exist there.

This complements the existing --skip-iceberg-content flag (added in #256) and is aimed at the same class of problem: disaster-recovery setups where the synchronizer
shouldn't be allowed to mutate certain top-level resources on the target.

Motivation

In a DR setup, target catalogs are often pre-created with intentionally different storage locations (allowedLocations, default-base-location, roleArn, etc.) than the
source. Today, if a catalog exists on both source and target and the synchronization strategy is CREATE_AND_OVERWRITE or REPLICATE (or --diff-only detects a
difference), the synchronizer will dropCatalogCascade + createCatalog on the target, copying the source catalog's storage config verbatim and silently clobbering the
DR-specific location.

--skip-catalog-sync lets operators keep principal, principal-role, catalog-role, and grant synchronization fully active while guaranteeing catalog objects themselves are
never created, overwritten, or removed on the target.

Changes

  • PolarisSynchronizer: new skipCatalogSync constructor parameter/field. syncCatalogs() now checks this flag before each of the create/overwrite/remove operations.
  • SyncPolarisCommand: new --skip-catalog-sync CLI option, wired into the PolarisSynchronizer constructor call.

Behavior when --skip-catalog-sync is set

  • Catalogs that only exist on the source (would be created): skipped entirely, with a warning logged. Since no matching catalog exists on the target, there is nothing to
    synchronize catalog-roles/grants against, so these are also excluded from catalog-role sync for that catalog.
  • Catalogs that exist on both source and target (would be overwritten): the catalog object itself (drop + recreate) is skipped, but catalog-roles and grants are still
    synchronized against the existing target catalog.
  • Catalogs that would be removed (target-only, under the REPLICATE strategy): removal is skipped, and the target catalog is left untouched.

Iceberg namespace/table sync is governed independently by --skip-iceberg-content, so the two flags can be combined to synchronize only principals, principal-roles,
catalog-roles, and grants, without ever touching catalog objects or Iceberg content on the target.

Backward compatibility

--skip-catalog-sync defaults to false. No behavior changes for existing invocations unless the flag is explicitly passed.

Testing

Added PolarisSynchronizerSkipCatalogSyncTest, covering the skipped case (create/overwrite/remove all skipped on the target, source-only catalogs excluded from catalog-role
sync, catalogs existing on both sides still get catalog-role sync) and the non-skipped case (existing behavior unchanged). Updated PolarisSynchronizerSkipIcebergContentTest
call sites for the new constructor parameter; all existing tests continue to pass.

Allows skipping create/overwrite/remove of catalog objects on the target
while still synchronizing catalog-roles and grants for catalogs that
already exist there. Useful when target catalogs are pre-created with
DR-specific storage locations that shouldn't be clobbered.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant