feat: Synchronizer: pluggable CredentialWriter for principal credentials

polaris-synchronizer/README.md

@@ -89,16 +89,22 @@ java -jar cli/build/libs/polaris-synchronizer-cli.jar create-omnipotent-principa
 Upon finishing execution, the tool will output the principal name and client credentials for this
 principal. **Make sure to note these down as they will be necessary for the migration step.**
 
+By default, credentials are logged to the console:
+
 **Example Output:**
 ```
 ======================================================
-Omnipotent Principal Credentials:
+Principal Credentials:
 name = omnipotent-principal-XXXXX
 clientId = ff7s8f9asbX10
 clientSecret = <client-secret>
 ======================================================
 ```
 
+> :warning: Credential output is pluggable via the `--credential-output-type` and `--credential-output-properties`
+> options, see [Configuring Credential Output](#configuring-credential-output) below. Console output should be
+> securely managed; client credentials should only ever be stored in a secure vault.
+
 Additionally, at the end of execution the command will output a list of catalogs for which catalog setup failed.
 **These catalogs may experience failure during migration**.
 
@@ -136,7 +142,7 @@ for subsequent steps.
 **Example Output:**
 ```
 ======================================================
-Omnipotent Principal Credentials:
+Principal Credentials:
 name = omnipotent-principal-YYYYY
 clientId = 0af20a3a0037a40d
 clientSecret = <client-secret>
@@ -150,14 +156,48 @@ clientSecret = <client-secret>
 > grants for the catalog, and assign it to the principal used to run this tool (presumably, a principal with the `servic_admin`
 > principal role). Then, re-running `create-omnipotent-principal` should be able to create the relevant entities for that catalog.
 
+### Configuring Credential Output
+
+Both `create-omnipotent-principal` and `sync-polaris` (when run with `--sync-principals`) generate new principal
+credentials. Where those credentials are output is controlled by two options, available on both commands:
+
+* `--credential-output-type`: One of `CONSOLE` (default), `FILE`, or `CUSTOM`.
+  * `CONSOLE`: Logs credentials to the console, matching the output shown above. This is the default, so existing
+    invocations are unaffected unless you opt in to a different type.
+  * `FILE`: Writes credentials as [JSON Lines](https://jsonlines.org/) (one JSON object per principal, per line) to
+    a file. The file is created with owner-only read/write permissions, since it contains plaintext secrets.
+  * `CUSTOM`: Discovers a user-supplied class implementing `CredentialWriter` via `java.util.ServiceLoader`, allowing
+    you to plug in your own storage backend (e.g. a secrets manager) without modifying this tool. Your implementation
+    must be registered as a service provider (declared in a
+    `META-INF/services/org.apache.polaris.tools.sync.polaris.access.CredentialWriter` file on the classpath) and
+    placed on the classpath when running the CLI.
+* `--credential-output-properties`: Properties to configure the selected type.
+  * For `FILE`:
+    * `json-file`: (required) path to the file to write credentials to.
+    * `append`: (default: `false`) if `true`, appends to an existing file instead of truncating it on each run.
+  * For `CUSTOM`:
+    * `custom-impl`: the fully-qualified classname of your registered `CredentialWriter` service provider.
+
+**Example:** Write credentials generated by `create-omnipotent-principal` to a JSON Lines file instead of the console:
+```
+java -jar cli/build/libs/polaris-synchronizer-cli.jar create-omnipotent-principal \
+--polaris-api-connection-properties base-url=http://localhost:8181 \
+--polaris-api-connection-properties oauth2-server-uri=http://localhost:8181/api/catalog/v1/oauth/tokens \
+--polaris-api-connection-properties credential=<client_id>:<client_secret> \
+--polaris-api-connection-properties scope=PRINCIPAL_ROLE:ALL \
+--credential-output-type FILE \
+--credential-output-properties json-file=./omnipotent-principal-credentials.jsonl
+```
+
 ### Step 3: Running the Migration/Synchronization
 
 Running the synchronization requires minimal reconfiguration, can be run idempotently, and will attempt to only copy over the
 diff between the source and target Polaris instances. This can be achieved using the `sync-polaris` command.
 
 > :warning: If you want to migrate principals and their assignments to principal-roles as well, run the tool with the
 > `--sync-principals` flag. Please note that this will reset the client credentials for that principal on the target 
-> Polaris instance. The new credentials will be logged to stdout, ONLY for each newly created or overwritten principal. 
+> Polaris instance. The new credentials will be output via the configured `--credential-output-type`, by default this
+> means they are logged to the console, ONLY for each newly created or overwritten principal.
 > Please note that this output should be securely managed, client credentials should only ever be stored in a secure vault.
 
 **Example** Running the synchronization between source Polaris instance using a bearer token, and a target Polaris instance

polaris-synchronizer/api/src/main/java/org/apache/polaris/tools/sync/polaris/PolarisSynchronizer.java

@@ -32,6 +32,7 @@
 import org.apache.polaris.core.admin.model.Principal;
 import org.apache.polaris.core.admin.model.PrincipalRole;
 import org.apache.polaris.core.admin.model.PrincipalWithCredentials;
+import org.apache.polaris.tools.sync.polaris.access.CredentialWriter;
 import org.apache.polaris.tools.sync.polaris.catalog.BaseTableWithETag;
 import org.apache.polaris.tools.sync.polaris.catalog.ETagManager;
 import org.apache.polaris.tools.sync.polaris.catalog.MetadataNotModifiedException;
@@ -62,6 +63,8 @@ public class PolarisSynchronizer {
 
   private final ETagManager etagManager;
 
+  private final CredentialWriter credentialWriter;
+
   private final boolean haltOnFailure;
 
   private final boolean diffOnly;
@@ -76,6 +79,7 @@ public PolarisSynchronizer(
       PolarisService source,
       PolarisService target,
       ETagManager etagManager,
+      CredentialWriter credentialWriter,
       boolean diffOnly,
       SynchronizationReport report,
       boolean skipIcebergContent) {
@@ -86,6 +90,7 @@ public PolarisSynchronizer(
     this.source = source;
     this.target = target;
     this.etagManager = etagManager;
+    this.credentialWriter = credentialWriter;
     this.diffOnly = diffOnly;
     this.report = report;
     this.skipIcebergContent = skipIcebergContent;
@@ -154,10 +159,9 @@ public void syncPrincipals() {
     for (Principal principal : principalSyncPlan.entitiesToCreate()) {
       try {
         PrincipalWithCredentials createdPrincipal = target.createPrincipal(principal);
-        clientLogger.info("Created principal {} on target. Target credentials: {}:{} - {}/{}",
+        credentialWriter.writeCredentials(createdPrincipal);
+        clientLogger.info("Created principal {} on target. - {}/{}",
                 principal.getName(),
-                createdPrincipal.getCredentials().getClientId(),
-                createdPrincipal.getCredentials().getClientSecret(),
                 ++syncsCompleted,
                 totalSyncsToComplete
         );
@@ -174,10 +178,9 @@ public void syncPrincipals() {
       try {
         target.dropPrincipal(principal.getName());
         PrincipalWithCredentials overwrittenPrincipal = target.createPrincipal(principal);
-        clientLogger.info("Overwrote principal {} on target. Target credentials: {}:{} - {}/{}",
+        credentialWriter.writeCredentials(overwrittenPrincipal);
+        clientLogger.info("Overwrote principal {} on target. - {}/{}",
                 principal.getName(),
-                overwrittenPrincipal.getCredentials().getClientId(),
-                overwrittenPrincipal.getCredentials().getClientSecret(),
                 ++syncsCompleted,
                 totalSyncsToComplete
         );

polaris-synchronizer/api/src/main/java/org/apache/polaris/tools/sync/polaris/access/CredentialWriter.java

@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.tools.sync.polaris.access;
+
+import org.apache.polaris.core.admin.model.PrincipalWithCredentials;
+
+/**
+ * Generic interface to output newly generated/rotated principal credentials. This allows the
+ * destination of the credentials to be completely independent from the tool.
+ *
+ * <p>Implementations should be fully configured and ready to use once constructed; obtaining and
+ * applying any configuration is the responsibility of {@code CredentialWriterFactory}, not of
+ * callers of this interface.
+ */
+public interface CredentialWriter extends AutoCloseable {
+
+  /**
+   * Outputs the given principal's credentials.
+   *
+   * @param principalWithCredentials the principal and its associated credentials
+   */
+  void writeCredentials(PrincipalWithCredentials principalWithCredentials);
+}

polaris-synchronizer/api/src/test/java/org/apache/polaris/tools/sync/polaris/PolarisSynchronizerSkipIcebergContentTest.java

@@ -299,6 +299,7 @@ public void testSkipIcebergContentSkipsIcebergSyncButStillSyncsCatalogRoles() {
             source,
             target,
             new NoOpETagManager(),
+            null,
             false,
             new SynchronizationReport(),
             true);
@@ -324,6 +325,7 @@ public void testIcebergContentSyncedWhenNotSkipped() {
             source,
             target,
             new NoOpETagManager(),
+            null,
             false,
             new SynchronizationReport(),
             false);
@@ -356,6 +358,7 @@ public void testTableScopedGrantsStillAttemptedWhenIcebergContentSkipped() {
             source,
             target,
             new NoOpETagManager(),
+            null,
             false,
             new SynchronizationReport(),
             true);

polaris-synchronizer/build.gradle.kts

@@ -78,6 +78,9 @@ tasks.named<RatTask>("rat").configure {
 
     // Rat can't scan binary images
     excludes.add("**/*.png")
+
+    // ServiceLoader provider-config files are plain classnames with no comment syntax for a header
+    excludes.add("**/META-INF/services/**")
 }
 
 tasks.named<Wrapper>("wrapper") {

polaris-synchronizer/cli/build.gradle.kts

@@ -37,6 +37,7 @@ dependencies {
   implementation("org.slf4j:log4j-over-slf4j:2.0.17")
   implementation("org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:1.7.1")
   implementation("org.apache.commons:commons-csv:1.13.0")
+  implementation("com.fasterxml.jackson.core:jackson-databind:2.18.3")
   runtimeOnly("ch.qos.logback:logback-classic:1.5.17")
 
   testImplementation("org.junit.jupiter:junit-jupiter-params:5.10.0")

polaris-synchronizer/cli/src/main/java/org/apache/polaris/tools/sync/polaris/ConsoleCredentialWriter.java

@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.tools.sync.polaris;
+
+import org.apache.polaris.core.admin.model.PrincipalWithCredentials;
+import org.apache.polaris.tools.sync.polaris.access.CredentialWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Implementation that logs principal credentials to the console. */
+public class ConsoleCredentialWriter implements CredentialWriter {
+
+  private final Logger consoleLog = LoggerFactory.getLogger("console-log");
+
+  @Override
+  public void writeCredentials(PrincipalWithCredentials principalWithCredentials) {
+    consoleLog.info(
+        "\n======================================================\n"
+            + "Principal Credentials:\n"
+            + "\tname = {}\n"
+            + "\tclientId = {}\n"
+            + "\tclientSecret = {}\n"
+            + "======================================================",
+        principalWithCredentials.getPrincipal().getName(),
+        principalWithCredentials.getCredentials().getClientId(),
+        principalWithCredentials.getCredentials().getClientSecret());
+  }
+
+  @Override
+  public void close() {}
+}

polaris-synchronizer/cli/src/main/java/org/apache/polaris/tools/sync/polaris/CreateOmnipotentPrincipalCommand.java

@@ -32,6 +32,7 @@
 import org.apache.polaris.core.admin.model.PrincipalRole;
 import org.apache.polaris.core.admin.model.PrincipalWithCredentials;
 import org.apache.polaris.tools.sync.polaris.access.AccessControlService;
+import org.apache.polaris.tools.sync.polaris.access.CredentialWriter;
 import org.apache.polaris.tools.sync.polaris.service.PolarisService;
 import org.apache.polaris.tools.sync.polaris.service.impl.PolarisApiService;
 import org.slf4j.Logger;
@@ -83,13 +84,34 @@ public class CreateOmnipotentPrincipalCommand implements Callable<Integer> {
       })
   private int concurrency;
 
+  @CommandLine.Option(
+          names = {"--credential-output-type"},
+          defaultValue = "CONSOLE",
+          description = "One of { CONSOLE, FILE, CUSTOM }. Default: CONSOLE. Controls how the newly generated " +
+                  "principal credentials are output."
+  )
+  private CredentialWriterFactory.Type credentialWriterType;
+
+  @CommandLine.Option(
+          names = {"--credential-output-properties"},
+          description = "Properties to initialize credential output." +
+                  "\nFor type FILE:" +
+                  "\n\t- " + JsonFileCredentialWriter.JSON_FILE_PROPERTY + ": The JSON Lines file to write principal credentials to." +
+                  "\n\t- " + JsonFileCredentialWriter.APPEND_PROPERTY + ": (default: false) Whether to append to an existing file instead of overwriting it." +
+                  "\nFor type CUSTOM:" +
+                  "\n\t- " + CredentialWriterFactory.CUSTOM_CLASS_NAME_PROPERTY + ": The classname of the CredentialWriter service provider to load via ServiceLoader."
+  )
+  private Map<String, String> credentialWriterProperties;
+
   @Override
   public Integer call() throws Exception {
     polarisApiConnectionProperties.putIfAbsent(PolarisApiService.ICEBERG_WRITE_ACCESS_PROPERTY,
             String.valueOf(withWriteAccess));
 
     try (PolarisService polaris = PolarisServiceFactory.createPolarisService(
-            PolarisServiceFactory.ServiceType.API, polarisApiConnectionProperties)) {
+            PolarisServiceFactory.ServiceType.API, polarisApiConnectionProperties);
+         CredentialWriter credentialWriter =
+                 CredentialWriterFactory.createCredentialWriter(credentialWriterType, credentialWriterProperties)) {
 
       AccessControlService accessControlService = new AccessControlService((PolarisApiService) polaris);
 
@@ -171,16 +193,7 @@ public Integer call() throws Exception {
               "Encountered issues creating catalog roles for the following catalogs: {}",
               failedCatalogs.stream().map(Catalog::getName).toList());
 
-      consoleLog.info(
-              "\n======================================================\n"
-                      + "Omnipotent Principal Credentials:\n"
-                      + "\tname = {}\n"
-                      + "\tclientId = {}\n"
-                      + "\tclientSecret = {}\n"
-                      + "======================================================",
-              principalWithCredentials.getPrincipal().getName(),
-              principalWithCredentials.getCredentials().getClientId(),
-              principalWithCredentials.getCredentials().getClientSecret());
+      credentialWriter.writeCredentials(principalWithCredentials);
 
     }