|
18 | 18 | | `extensions` | Extension to load into duckdb. Only autoloadable extensions are supported. | list | N | |
19 | 19 | | `connector_config` | Configuration to pass into the duckdb connector. | dict | N | |
20 | 20 | | `secrets` | Configuration for authenticating external sources (e.g., S3) using DuckDB secrets. | dict | N | |
| 21 | +| `file_systems` | Configuration for registering `fsspec` filesystems to the DuckDB connection. secrets. | dict | N | |
21 | 22 |
|
22 | 23 | #### DuckDB Catalogs Example |
23 | 24 |
|
@@ -256,4 +257,37 @@ After configuring the secrets, you can directly reference S3 paths in your catal |
256 | 257 |
|
257 | 258 | Refer to the official DuckDB documentation for the full list of [supported S3 secret parameters](https://duckdb.org/docs/stable/extensions/httpfs/s3api.html#overview-of-s3-secret-parameters) and for more information on the [Secrets Manager configuration](https://duckdb.org/docs/configuration/secrets_manager.html). |
258 | 259 |
|
259 | | -> Note: Loading credentials at runtime using `load_aws_credentials()` or similar deprecated functions may fail when using SQLMesh. |
| 260 | +> Note: Loading credentials at runtime using `load_aws_credentials()` or similar deprecated functions may fail when using SQLMesh. |
| 261 | +
|
| 262 | +##### File system configuration example for Microsoft Onelake |
| 263 | + |
| 264 | +The `file_systems` accepts a list of file systems to register in the DuckDB connection. This is especially useful for Azure Storage Accounts, as it adds write support for DuckDB which is not natively supported by DuckDB (yet). |
| 265 | + |
| 266 | + |
| 267 | +=== "YAML" |
| 268 | + |
| 269 | + ```yaml linenums="1" |
| 270 | + gateways: |
| 271 | + ducklake: |
| 272 | + connection: |
| 273 | + type: duckdb |
| 274 | + catalogs: |
| 275 | + ducklake: |
| 276 | + type: ducklake |
| 277 | + path: myducklakecatalog.duckdb |
| 278 | + data_path: abfs://MyFabricWorkspace/MyFabricLakehouse.Lakehouse/Files/DuckLake.Files |
| 279 | + extensions: |
| 280 | + - ducklake |
| 281 | + file_systems: |
| 282 | + - protocol: abfs |
| 283 | + storage_options: |
| 284 | + account_name: onelake |
| 285 | + account_host: onelake.blob.fabric.microsoft.com |
| 286 | + client_id: {{ env_var('AZURE_CLIENT_ID') }} |
| 287 | + client_secret: {{ env_var('AZURE_CLIENT_SECRET') }} |
| 288 | + tenant_id: {{ env_var('AZURE_TENANT_ID') }} |
| 289 | + # anon: False # To use azure.identity.DefaultAzureCredential authentication |
| 290 | + ``` |
| 291 | + |
| 292 | + |
| 293 | +Refer to the documentation for `fsspec` [fsspec.filesystem](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.filesystem) and `adlfs` [adlfs.AzureBlobFileSystem](https://fsspec.github.io/adlfs/api/#api-reference) for a full list of storage options. |
0 commit comments