Merge branch 'main' into add_fabric_warehouse

Author: fresioAS

README.md

@@ -145,8 +145,7 @@ python -m venv .venv
 source .venv/bin/activate
 pip install 'sqlmesh[lsp]' # install the sqlmesh package with extensions to work with VSCode
 source .venv/bin/activate # reactivate the venv to ensure you're using the right installation
-sqlmesh init duckdb # get started right away with a local duckdb instance
-sqlmesh plan # see the plan for the changes you're making
+sqlmesh init # follow the prompts to get started (choose DuckDB)
 ```
 
 </details>
@@ -163,13 +162,12 @@ python -m venv .venv
 .\.venv\Scripts\Activate.ps1
 pip install 'sqlmesh[lsp]' # install the sqlmesh package with extensions to work with VSCode
 .\.venv\Scripts\Activate.ps1 # reactivate the venv to ensure you're using the right installation
-sqlmesh init duckdb # get started right away with a local duckdb instance
-sqlmesh plan # see the plan for the changes you're making
+sqlmesh init # follow the prompts to get started (choose DuckDB)
 ```
 </details>
 
 
-Follow the [quickstart guide](https://sqlmesh.readthedocs.io/en/stable/quickstart/cli/#1-create-the-sqlmesh-project) to learn how to use SQLMesh. You already have a head start!
+Follow the [quickstart guide](https://sqlmesh.readthedocs.io/en/stable/quickstart/cli/) to learn how to use SQLMesh. You already have a head start!
 
 Follow the [crash course](https://sqlmesh.readthedocs.io/en/stable/examples/sqlmesh_cli_crash_course/) to learn the core movesets and use the easy to reference cheat sheet. 
 

docs/concepts/models/python_models.md

@@ -163,15 +163,15 @@ def execute(
 ) -> pd.DataFrame:
 
     # pre-statement
-    context.fetchdf("SET GLOBAL parameter = 'value';")
+    context.engine_adapter.execute("SET GLOBAL parameter = 'value';")
 
     # post-statement requires using `yield` instead of `return`
     yield pd.DataFrame([
         {"id": 1, "name": "name"}
     ])
 
     # post-statement
-    context.fetchdf("CREATE INDEX idx ON example.pre_post_statements (id);")
+    context.engine_adapter.execute("CREATE INDEX idx ON example.pre_post_statements (id);")
 ```
 
 ## Optional on-virtual-update statements

docs/guides/configuration.md

@@ -149,6 +149,16 @@ Because the VSCode extension establishes a long-running process connected to the
 
 Therefore, we do not recommend using DuckDB as a state store with the VSCode extension.
 
+### Environment variables
+
+The VSCode extension is based on a [language server](https://en.wikipedia.org/wiki/Language_Server_Protocol) that runs in the background as a separate process. When the VSCode extension starts the background language server, the server inherits environment variables from the environment where you started VSCode. The server does *not* inherit environment variables from your terminal instance in VSCode, so it may not have access to variables you use when calling SQLMesh from the CLI.
+
+If you have environment variables that are needed by the context and the language server, you can use one of these approaches to pass variables to the language server:
+
+- Open VSCode from a terminal that has the variables set
+- Use environment variables pulled from somewhere else dynamically (e.g. a `.env` file) in your config 
+- Set the environment variables in the python environment that the extension uses. You can find detailed instructions [here](https://code.visualstudio.com/docs/python/environments#_environment-variables)
+
 ### Python environment woes
 
 The most common problem is the extension not using the correct Python interpreter.

docs/guides/vscode.md

@@ -18,6 +18,7 @@
 | `extensions`       | Extension to load into duckdb. Only autoloadable extensions are supported.                                                                                                                                                                      |  list  |    N     |
 | `connector_config` | Configuration to pass into the duckdb connector.                                                                                                                                                                                                |  dict  |    N     |
 | `secrets`   | Configuration for authenticating external sources (e.g., S3) using DuckDB secrets.                                                                                                                                                                                                |  dict  |    N     |
+| `filesystems`   | Configuration for registering `fsspec` filesystems to the DuckDB connection.                                                                                                                                                                                                |  dict  |    N     |
 
 #### DuckDB Catalogs Example
 
@@ -256,4 +257,36 @@ After configuring the secrets, you can directly reference S3 paths in your catal
 
 Refer to the official DuckDB documentation for the full list of [supported S3 secret parameters](https://duckdb.org/docs/stable/extensions/httpfs/s3api.html#overview-of-s3-secret-parameters) and for more information on the [Secrets Manager configuration](https://duckdb.org/docs/configuration/secrets_manager.html).
 
-> Note: Loading credentials at runtime using `load_aws_credentials()` or similar deprecated functions may fail when using SQLMesh.
+> Note: Loading credentials at runtime using `load_aws_credentials()` or similar deprecated functions may fail when using SQLMesh.
+
+##### File system configuration example for Microsoft Onelake
+
+The `filesystems` accepts a list of file systems to register in the DuckDB connection. This is especially useful for Azure Storage Accounts, as it adds write support for DuckDB which is not natively supported by DuckDB (yet).
+
+
+=== "YAML"
+
+    ```yaml linenums="1"
+    gateways:
+      ducklake:
+        connection:
+          type: duckdb
+          catalogs:
+            ducklake:
+            type: ducklake
+            path: myducklakecatalog.duckdb
+            data_path: abfs://MyFabricWorkspace/MyFabricLakehouse.Lakehouse/Files/DuckLake.Files
+        extensions:
+          - ducklake
+        filesystems:
+          - fs: abfs
+            account_name: onelake
+            account_host: onelake.blob.fabric.microsoft.com
+            client_id: {{ env_var('AZURE_CLIENT_ID') }}
+            client_secret: {{ env_var('AZURE_CLIENT_SECRET') }}
+            tenant_id: {{ env_var('AZURE_TENANT_ID') }}
+            # anon: False # To use azure.identity.DefaultAzureCredential authentication 
+    ```
+
+
+Refer to the documentation for `fsspec` [fsspec.filesystem](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.filesystem) and `adlfs` [adlfs.AzureBlobFileSystem](https://fsspec.github.io/adlfs/api/#api-reference) for a full list of storage options. 

docs/integrations/engines/duckdb.md

@@ -9,21 +9,21 @@ SQLMesh supports integrations with the following tools:
 * [Kestra](https://kestra.io/plugins/plugin-sqlmesh/tasks/cli/io.kestra.plugin.sqlmesh.cli.sqlmeshcli)
 
 ## Execution engines
-SQLMesh supports the following execution engines for running SQLMesh projects:
+SQLMesh supports the following execution engines for running SQLMesh projects (engine `type` in parentheses - example usage: `pip install "sqlmesh[databricks]"`):
 
-* [Athena](./engines/athena.md)
-* [Azure SQL](./engines/azuresql.md)
-* [BigQuery](./engines/bigquery.md)
-* [ClickHouse](./engines/clickhouse.md)
-* [Databricks](./engines/databricks.md)
-* [DuckDB](./engines/duckdb.md)
-* [Fabric](./engines/fabric.md)
-* [MotherDuck](./engines/motherduck.md)
-* [MSSQL](./engines/mssql.md)
-* [MySQL](./engines/mysql.md)
-* [Postgres](./engines/postgres.md)
-* [GCP Postgres](./engines/gcp-postgres.md)
-* [Redshift](./engines/redshift.md)
-* [Snowflake](./engines/snowflake.md)
-* [Spark](./engines/spark.md)
-* [Trino](./engines/trino.md)
+* [Athena](./engines/athena.md) (athena)
+* [Azure SQL](./engines/azuresql.md) (azuresql)
+* [BigQuery](./engines/bigquery.md) (bigquery)
+* [ClickHouse](./engines/clickhouse.md) (clickhouse)
+* [Databricks](./engines/databricks.md) (databricks)
+* [DuckDB](./engines/duckdb.md) (duckdb)
+* [Fabric](./engines/fabric.md) (fabric)
+* [MotherDuck](./engines/motherduck.md) (motherduck)
+* [MSSQL](./engines/mssql.md) (mssql)
+* [MySQL](./engines/mysql.md) (mysql)
+* [Postgres](./engines/postgres.md) (postgres)
+* [GCP Postgres](./engines/gcp-postgres.md) (gcppostgres)
+* [Redshift](./engines/redshift.md) (redshift)
+* [Snowflake](./engines/snowflake.md) (snowflake)
+* [Spark](./engines/spark.md) (spark)
+* [Trino](./engines/trino.md) (trino)