rename config to secrets; add warning message for old versions

Author: themisvaltinos

docs/integrations/engines/duckdb.md

@@ -17,7 +17,7 @@
 | `catalogs`         | Mapping to define multiple catalogs. Can [attach DuckDB catalogs](#duckdb-catalogs-example) or [catalogs for other connections](#other-connection-catalogs-example). First entry is the default catalog. Cannot be defined if using `database`. |  dict  |    N     |
 | `extensions`       | Extension to load into duckdb. Only autoloadable extensions are supported.                                                                                                                                                                      |  list  |    N     |
 | `connector_config` | Configuration to pass into the duckdb connector.                                                                                                                                                                                                |  dict  |    N     |
-| `secrets_config`   | Configuration for authenticating external sources (e.g., S3) using DuckDB secrets.                                                                                                                                                                                                |  dict  |    N     |
+| `secrets`   | Configuration for authenticating external sources (e.g., S3) using DuckDB secrets.                                                                                                                                                                                                |  dict  |    N     |
 
 #### DuckDB Catalogs Example
 
@@ -142,11 +142,11 @@ If a connector, like Postgres, requires sensitive information in the path, it mi
 
 DuckDB can read data directly from cloud services via extensions (e.g., [httpfs](https://duckdb.org/docs/extensions/httpfs/s3api), [azure](https://duckdb.org/docs/extensions/azure)).
 
-The `secrets_config` option allows you to configure DuckDB's [Secrets Manager](https://duckdb.org/docs/configuration/secrets_manager.html) to authenticate with external services like S3. This is the recommended approach for cloud storage authentication in DuckDB v0.10.0 and newer, replacing the [legacy authentication method](https://duckdb.org/docs/stable/extensions/httpfs/s3api_legacy_authentication.html) via variables.
+The `secrets` option allows you to configure DuckDB's [Secrets Manager](https://duckdb.org/docs/configuration/secrets_manager.html) to authenticate with external services like S3. This is the recommended approach for cloud storage authentication in DuckDB v0.10.0 and newer, replacing the [legacy authentication method](https://duckdb.org/docs/stable/extensions/httpfs/s3api_legacy_authentication.html) via variables.
 
 ##### Secrets Configuration Example for S3
 
-The `secrets_config` accepts a list of secret configurations, each defining the necessary authentication parameters for the specific service:
+The `secrets` accepts a list of secret configurations, each defining the necessary authentication parameters for the specific service:
 
 === "YAML"
 
@@ -160,7 +160,7 @@ The `secrets_config` accepts a list of secret configurations, each defining the
             remote: "s3://bucket/data/remote.duckdb"
           extensions:
             - name: httpfs
-          secrets_config:
+          secrets:
             - type: s3
               region: "YOUR_AWS_REGION"
               key_id: "YOUR_AWS_ACCESS_KEY"
@@ -189,7 +189,7 @@ The `secrets_config` accepts a list of secret configurations, each defining the
                     extensions=[
                         {"name": "httpfs"},
                     ],
-                    secrets_config=[
+                    secrets=[
                         {
                             "type": "s3",
                             "region": "YOUR_AWS_REGION",

docs/integrations/engines/motherduck.md

@@ -104,4 +104,4 @@ Congratulations \- your SQLMesh project is up and running on MotherDuck\!
 | `token`            | The optional MotherDuck token. If not specified, the user will be prompted to login with their web browser. | string | N        |
 | `extensions`       | Extension to load into duckdb. Only autoloadable extensions are supported.                                  | list   | N        |
 | `connector_config` | Configuration to pass into the duckdb connector.                                                            | dict   | N        |
-| `secrets_config`   | Configuration for authenticating external sources (e.g. S3) using DuckDB secrets.                           | dict   | N        |
+| `secrets`   | Configuration for authenticating external sources (e.g. S3) using DuckDB secrets.                           | dict   | N        |

sqlmesh/core/config/connection.py

@@ -202,7 +202,7 @@ class BaseDuckDBConnectionConfig(ConnectionConfig):
         catalogs: Key is the name of the catalog and value is the path.
         extensions: A list of autoloadable extensions to load.
         connector_config: A dictionary of configuration to pass into the duckdb connector.
-        secrets_config: A list of dictionaries used to generate DuckDB secrets for authenticating with external services (e.g. S3).
+        secrets: A list of dictionaries used to generate DuckDB secrets for authenticating with external services (e.g. S3).
         concurrent_tasks: The maximum number of tasks that can use this connection concurrently.
         register_comments: Whether or not to register model comments with the SQL engine.
         pre_ping: Whether or not to pre-ping the connection before starting a new transaction to ensure it is still alive.
@@ -213,7 +213,7 @@ class BaseDuckDBConnectionConfig(ConnectionConfig):
     catalogs: t.Optional[t.Dict[str, t.Union[str, DuckDBAttachOptions]]] = None
     extensions: t.List[t.Union[str, t.Dict[str, t.Any]]] = []
     connector_config: t.Dict[str, t.Any] = {}
-    secrets_config: t.List[t.Dict[str, t.Any]] = []
+    secrets: t.List[t.Dict[str, t.Any]] = []
 
     concurrent_tasks: int = 1
     register_comments: bool = True
@@ -286,21 +286,27 @@ def init(cursor: duckdb.DuckDBPyConnection) -> None:
                 except Exception as e:
                     raise ConfigError(f"Failed to set connector config {field} to {setting}: {e}")
 
-            if self.secrets_config and version.parse(duckdb.__version__) >= version.parse("0.10.0"):
-                # For older versions, legacy authentication is used by setting through the connector_config
-                # https://duckdb.org/docs/stable/extensions/httpfs/s3api_legacy_authentication.html
-
-                for secrets in self.secrets_config:
-                    secret_settings: t.List[str] = []
-                    for field, setting in secrets.items():
-                        secret_settings.append(f"{field} '{setting}'")
-
-                    if secret_settings:
-                        secret_clause = ", ".join(secret_settings)
-                        try:
-                            cursor.execute(f"CREATE SECRET ({secret_clause});")
-                        except Exception as e:
-                            raise ConfigError(f"Failed to create secret: {e}")
+            if self.secrets:
+                duckdb_version = duckdb.__version__
+                if version.parse(duckdb_version) < version.parse("0.10.0"):
+                    from sqlmesh.core.console import get_console
+
+                    get_console().log_warning(
+                        f"DuckDB version {duckdb_version} does not support secrets-based authentication (requires 0.10.0 or later).\n"
+                        "To use secrets, please upgrade DuckDB. For older versions, configure legacy authentication via `connector_config`.\n"
+                        "More info: https://duckdb.org/docs/stable/extensions/httpfs/s3api_legacy_authentication.html"
+                    )
+                else:
+                    for secrets in self.secrets:
+                        secret_settings: t.List[str] = []
+                        for field, setting in secrets.items():
+                            secret_settings.append(f"{field} '{setting}'")
+                        if secret_settings:
+                            secret_clause = ", ".join(secret_settings)
+                            try:
+                                cursor.execute(f"CREATE SECRET ({secret_clause});")
+                            except Exception as e:
+                                raise ConfigError(f"Failed to create secret: {e}")
 
             for i, (alias, path_options) in enumerate(
                 (getattr(self, "catalogs", None) or {}).items()

sqlmesh/dbt/target.py

@@ -195,7 +195,7 @@ def to_sqlmesh(self, **kwargs: t.Any) -> ConnectionConfig:
         if self.settings is not None:
             kwargs["connector_config"] = self.settings
         if self.secrets is not None:
-            kwargs["secrets_config"] = self.secrets
+            kwargs["secrets"] = self.secrets
         return DuckDBConnectionConfig(
             database=self.path,
             concurrent_tasks=1,

tests/core/test_config.py

@@ -550,7 +550,7 @@ def test_connection_config_serialization():
         "pre_ping": False,
         "pretty_sql": False,
         "connector_config": {},
-        "secrets_config": [],
+        "secrets": [],
         "database": "my_db",
     }
     assert serialized["default_test_connection"] == {
@@ -561,7 +561,7 @@ def test_connection_config_serialization():
         "pre_ping": False,
         "pretty_sql": False,
         "connector_config": {},
-        "secrets_config": [],
+        "secrets": [],
         "database": "my_test_db",
     }
 

tests/core/test_connection_config.py

@@ -426,7 +426,7 @@ def test_duckdb(make_config):
         type="duckdb",
         database="test",
         connector_config={"foo": "bar"},
-        secrets_config=[
+        secrets=[
             {
                 "type": "s3",
                 "region": "aws_region",
@@ -436,7 +436,7 @@ def test_duckdb(make_config):
         ],
     )
     assert config.connector_config
-    assert config.secrets_config
+    assert config.secrets
     assert isinstance(config, DuckDBConnectionConfig)
     assert not config.is_recommended_for_state_sync
 

tests/integrations/jupyter/test_magics.py

@@ -759,16 +759,16 @@ def test_info(notebook, sushi_context, convert_all_html_output_to_text, get_all_
         "Models: 18",
         "Macros: 8",
         "",
-        "Connection:\n  type: duckdb\n  concurrent_tasks: 1\n  register_comments: true\n  pre_ping: false\n  pretty_sql: false\n  extensions: []\n  connector_config: {}\n  secrets_config: None",
-        "Test Connection:\n  type: duckdb\n  concurrent_tasks: 1\n  register_comments: true\n  pre_ping: false\n  pretty_sql: false\n  extensions: []\n  connector_config: {}\n  secrets_config: None",
+        "Connection:\n  type: duckdb\n  concurrent_tasks: 1\n  register_comments: true\n  pre_ping: false\n  pretty_sql: false\n  extensions: []\n  connector_config: {}\n  secrets: None",
+        "Test Connection:\n  type: duckdb\n  concurrent_tasks: 1\n  register_comments: true\n  pre_ping: false\n  pretty_sql: false\n  extensions: []\n  connector_config: {}\n  secrets: None",
         "Data warehouse connection succeeded",
     ]
     assert get_all_html_output(output) == [
         "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Models: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">18</span></pre>",
         "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Macros: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span></pre>",
         "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>",
-        '<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,\'DejaVu Sans Mono\',consolas,\'Courier New\',monospace">Connection:  type: duckdb  concurrent_tasks: <span style="color: #008080; text-decoration-color: #008080; font-weight: bold">1</span>  register_comments: true  pre_ping: false  pretty_sql: false  extensions: <span style="font-weight: bold">[]</span>  connector_config: <span style="font-weight: bold">{}</span>  secrets_config: <span style="color: #800080; text-decoration-color: #800080; font-style: italic">None</span></pre>',
-        '<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,\'DejaVu Sans Mono\',consolas,\'Courier New\',monospace">Test Connection:  type: duckdb  concurrent_tasks: <span style="color: #008080; text-decoration-color: #008080; font-weight: bold">1</span>  register_comments: true  pre_ping: false  pretty_sql: false  extensions: <span style="font-weight: bold">[]</span>  connector_config: <span style="font-weight: bold">{}</span>  secrets_config: <span style="color: #800080; text-decoration-color: #800080; font-style: italic">None</span></pre>',
+        '<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,\'DejaVu Sans Mono\',consolas,\'Courier New\',monospace">Connection:  type: duckdb  concurrent_tasks: <span style="color: #008080; text-decoration-color: #008080; font-weight: bold">1</span>  register_comments: true  pre_ping: false  pretty_sql: false  extensions: <span style="font-weight: bold">[]</span>  connector_config: <span style="font-weight: bold">{}</span>  secrets: <span style="color: #800080; text-decoration-color: #800080; font-style: italic">None</span></pre>',
+        '<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,\'DejaVu Sans Mono\',consolas,\'Courier New\',monospace">Test Connection:  type: duckdb  concurrent_tasks: <span style="color: #008080; text-decoration-color: #008080; font-weight: bold">1</span>  register_comments: true  pre_ping: false  pretty_sql: false  extensions: <span style="font-weight: bold">[]</span>  connector_config: <span style="font-weight: bold">{}</span>  secrets: <span style="color: #800080; text-decoration-color: #800080; font-style: italic">None</span></pre>',
         "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Data warehouse connection <span style=\"color: #008000; text-decoration-color: #008000\">succeeded</span></pre>",
     ]