Merge branch 'main' into add_fabric_warehouse

Author: fresioAS

docs/concepts/models/python_models.md

@@ -163,15 +163,15 @@ def execute(
 ) -> pd.DataFrame:
 
     # pre-statement
-    context.fetchdf("SET GLOBAL parameter = 'value';")
+    context.engine_adapter.execute("SET GLOBAL parameter = 'value';")
 
     # post-statement requires using `yield` instead of `return`
     yield pd.DataFrame([
         {"id": 1, "name": "name"}
     ])
 
     # post-statement
-    context.fetchdf("CREATE INDEX idx ON example.pre_post_statements (id);")
+    context.engine_adapter.execute("CREATE INDEX idx ON example.pre_post_statements (id);")
 ```
 
 ## Optional on-virtual-update statements

docs/guides/configuration.md

@@ -149,6 +149,16 @@ Because the VSCode extension establishes a long-running process connected to the
 
 Therefore, we do not recommend using DuckDB as a state store with the VSCode extension.
 
+### Environment variables
+
+The VSCode extension is based on a [language server](https://en.wikipedia.org/wiki/Language_Server_Protocol) that runs in the background as a separate process. When the VSCode extension starts the background language server, the server inherits environment variables from the environment where you started VSCode. The server does *not* inherit environment variables from your terminal instance in VSCode, so it may not have access to variables you use when calling SQLMesh from the CLI.
+
+If you have environment variables that are needed by the context and the language server, you can use one of these approaches to pass variables to the language server:
+
+- Open VSCode from a terminal that has the variables set
+- Use environment variables pulled from somewhere else dynamically (e.g. a `.env` file) in your config 
+- Set the environment variables in the python environment that the extension uses. You can find detailed instructions [here](https://code.visualstudio.com/docs/python/environments#_environment-variables)
+
 ### Python environment woes
 
 The most common problem is the extension not using the correct Python interpreter.

docs/guides/vscode.md

@@ -18,6 +18,7 @@
 | `extensions`       | Extension to load into duckdb. Only autoloadable extensions are supported.                                                                                                                                                                      |  list  |    N     |
 | `connector_config` | Configuration to pass into the duckdb connector.                                                                                                                                                                                                |  dict  |    N     |
 | `secrets`   | Configuration for authenticating external sources (e.g., S3) using DuckDB secrets.                                                                                                                                                                                                |  dict  |    N     |
+| `filesystems`   | Configuration for registering `fsspec` filesystems to the DuckDB connection.                                                                                                                                                                                                |  dict  |    N     |
 
 #### DuckDB Catalogs Example
 
@@ -256,4 +257,36 @@ After configuring the secrets, you can directly reference S3 paths in your catal
 
 Refer to the official DuckDB documentation for the full list of [supported S3 secret parameters](https://duckdb.org/docs/stable/extensions/httpfs/s3api.html#overview-of-s3-secret-parameters) and for more information on the [Secrets Manager configuration](https://duckdb.org/docs/configuration/secrets_manager.html).
 
-> Note: Loading credentials at runtime using `load_aws_credentials()` or similar deprecated functions may fail when using SQLMesh.
+> Note: Loading credentials at runtime using `load_aws_credentials()` or similar deprecated functions may fail when using SQLMesh.
+
+##### File system configuration example for Microsoft Onelake
+
+The `filesystems` accepts a list of file systems to register in the DuckDB connection. This is especially useful for Azure Storage Accounts, as it adds write support for DuckDB which is not natively supported by DuckDB (yet).
+
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    gateways:
+      ducklake:
+        connection:
+          type: duckdb
+          catalogs:
+            ducklake:
+            type: ducklake
+            path: myducklakecatalog.duckdb
+            data_path: abfs://MyFabricWorkspace/MyFabricLakehouse.Lakehouse/Files/DuckLake.Files
+        extensions:
+          - ducklake
+        filesystems:
+          - fs: abfs
+            account_name: onelake
+            account_host: onelake.blob.fabric.microsoft.com
+            client_id: {{ env_var('AZURE_CLIENT_ID') }}
+            client_secret: {{ env_var('AZURE_CLIENT_SECRET') }}
+            tenant_id: {{ env_var('AZURE_TENANT_ID') }}
+            # anon: False # To use azure.identity.DefaultAzureCredential authentication 
+    ```
+
+
+Refer to the documentation for `fsspec` [fsspec.filesystem](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.filesystem) and `adlfs` [adlfs.AzureBlobFileSystem](https://fsspec.github.io/adlfs/api/#api-reference) for a full list of storage options. 

docs/integrations/engines/duckdb.md

@@ -9,21 +9,21 @@ SQLMesh supports integrations with the following tools:
 * [Kestra](https://kestra.io/plugins/plugin-sqlmesh/tasks/cli/io.kestra.plugin.sqlmesh.cli.sqlmeshcli)
 
 ## Execution engines
-SQLMesh supports the following execution engines for running SQLMesh projects:
+SQLMesh supports the following execution engines for running SQLMesh projects (engine `type` in parentheses - example usage: `pip install "sqlmesh[databricks]"`):
 
-* [Athena](./engines/athena.md)
-* [Azure SQL](./engines/azuresql.md)
-* [BigQuery](./engines/bigquery.md)
-* [ClickHouse](./engines/clickhouse.md)
-* [Databricks](./engines/databricks.md)
-* [DuckDB](./engines/duckdb.md)
-* [Fabric](./engines/fabric.md)
-* [MotherDuck](./engines/motherduck.md)
-* [MSSQL](./engines/mssql.md)
-* [MySQL](./engines/mysql.md)
-* [Postgres](./engines/postgres.md)
-* [GCP Postgres](./engines/gcp-postgres.md)
-* [Redshift](./engines/redshift.md)
-* [Snowflake](./engines/snowflake.md)
-* [Spark](./engines/spark.md)
-* [Trino](./engines/trino.md)
+* [Athena](./engines/athena.md) (athena)
+* [Azure SQL](./engines/azuresql.md) (azuresql)
+* [BigQuery](./engines/bigquery.md) (bigquery)
+* [ClickHouse](./engines/clickhouse.md) (clickhouse)
+* [Databricks](./engines/databricks.md) (databricks)
+* [DuckDB](./engines/duckdb.md) (duckdb)
+* [Fabric](./engines/fabric.md) (fabric)
+* [MotherDuck](./engines/motherduck.md) (motherduck)
+* [MSSQL](./engines/mssql.md) (mssql)
+* [MySQL](./engines/mysql.md) (mysql)
+* [Postgres](./engines/postgres.md) (postgres)
+* [GCP Postgres](./engines/gcp-postgres.md) (gcppostgres)
+* [Redshift](./engines/redshift.md) (redshift)
+* [Snowflake](./engines/snowflake.md) (snowflake)
+* [Spark](./engines/spark.md) (spark)
+* [Trino](./engines/trino.md) (trino)